The Paper
"Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs" comes from Gabrielle Kaili-May Liu, Avi Caciularu, Gal Yona, Idan Szpektor, and Arman Cohan at Yale NLP and Google, posted to arXiv in June 2026. The paper's central claim is that standard RLHF and fine-tuning approaches fail to produce models that accurately represent their own uncertainty - and that a new reward signal derived from metacognitive self-assessment can close that gap. The code is available at github.com/yale-nlp/RLMF.
The Problem Before This Paper
Hallucination research has mostly focused on factual accuracy - whether a model's answer is correct. But there is a second, underexplored failure mode: miscalibrated verbal uncertainty. A model can be wrong and sound confident. It can be right and express unnecessary doubt. These two failure modes are distinct. The first is a knowledge problem. The second is a metacognition problem - the model's expressed beliefs do not match its actual epistemic state. Prior work on calibration typically measured confidence post-hoc using softmax probabilities or numerical probability estimates, which do not translate directly to the hedging language that appears in real model outputs. Approaches like SFT on calibrated examples or RLHF with general human feedback did not specifically target the alignment between verbal expressions like "I think" or "I'm not sure" and ground-truth correctness. No method before RLMF used metacognitive self-judgment as the reward signal itself.
What They Built
RLMF is a two-stage training pipeline built on GRPO-style reinforcement learning. The first stage uses metacognitive data selection: the model judges the quality of its own responses, and those self-judgments determine which examples enter the training set - filtering for cases where the model's expressed confidence will have a nonzero learning signal. The second stage trains the model via RL where the reward function directly measures alignment between expressed confidence and actual correctness. If the model says "I'm certain" and is wrong, the reward is negative. If it says "I'm not sure" and is wrong, the reward is neutral or positive. The decoupled design separates the faithful calibration objective from accuracy objectives, allowing each to be optimized without interference.
RLMF reward signal (faithful calibration):
R = f(expressed_confidence, correct)
high R: "I'm unsure" + wrong, or "I'm certain" + correct
low R: "I'm certain" + wrong (overconfident hallucination)
neutral: "I'm unsure" + correct (underconfident but harmless)
The metacognitive data selection stage is not just a preprocessing filter - it actively shapes which training examples surface. Rather than naive active learning (selecting examples by loss magnitude or model uncertainty), the model's own self-performance judgments are used to rank candidates. This introduces a self-referential quality signal that standard data curation pipelines lack. The method is evaluated on Qwen3 and Llama 3 across factual QA, multiple-choice reasoning, and math benchmarks, with comparisons against SFT baselines, standard GRPO, and the DR-GRPO variant.
Faithful calibration (FC) target:
FC = P(correct | high expressed confidence)
- P(correct | low expressed confidence)
Goal: maximize FC gap while preserving accuracy
Key Findings
- RLMF beats standard RL by up to 63% on faithful calibration metrics across SimpleQA, TriviaQA, PopQA, MMLU, SciQ, and MATH - while preserving base accuracy on those benchmarks.
- Metacognitive data selection outperforms naive active learning. Using the model's own self-judgments to curate training examples produces better-calibrated models than selecting by loss magnitude or uncertainty alone.
- The decoupled two-stage design is critical. Jointly optimizing accuracy and calibration in a single RL objective caused interference; separating them into sequential stages resolved it.
- Verbal uncertainty expression transfers across task types. Models trained with RLMF on factual QA generalized calibrated hedging behavior to mathematical reasoning and science tasks without domain-specific fine-tuning.
- The method achieves state-of-the-art faithful calibration without degrading task accuracy - addressing the core tension that prior calibration methods faced: reducing overconfidence while not inducing excessive hedging that makes models less useful.
Results
RLMF surpasses standard GRPO and DR-GRPO variants by up to 63% on faithful calibration across six benchmarks. The gains are consistent across both Qwen3 and Llama 3 base models, suggesting the reward design is architecture-agnostic. On SimpleQA and PopQA - open-domain factual tasks where LLMs are most prone to confident hallucination - RLMF shows the largest calibration improvements. On MATH and SciQ, where models tend toward underconfidence, RLMF corrects in the opposite direction, raising expressed confidence on correct answers without inflating it on incorrect ones. Against the SFT baseline, RLMF consistently outperforms on calibration while matching or exceeding accuracy, indicating that behavioral cloning on calibrated examples is not sufficient - the RL feedback loop is necessary to drive genuine alignment between expressed and intrinsic uncertainty.
Why This Matters for AI and Automation
- Miscalibrated confidence is a deployment risk, not just an academic metric. In any agentic system where an LLM's output determines whether to escalate to a human, execute a transaction, or proceed autonomously, overconfident wrong answers are more dangerous than simply wrong answers. RLMF targets that specific failure mode.
- This connects directly to the reliability science framing from Beyond pass@1 (Week 16). That paper measured reliability from outside - via repeated sampling and external evaluation. RLMF builds reliability from inside - training the model to signal its own unreliability accurately. Both approaches are needed for production-grade agents.
- Autodata (Week 20) and RLMF form a natural pipeline. Autodata generates high-quality training examples through an agentic loop. RLMF can then train on those examples with a reward signal that ensures the model not only learns the right answers but also learns when it does not know the right answer. Quality data in, calibrated uncertainty out.
- For RAG systems specifically, calibrated uncertainty enables smarter retrieval decisions. An agent that knows it is uncertain can trigger a retrieval step. An overconfident agent skips retrieval on wrong answers. RLMF makes that self-assessment trustworthy.
My Take
The 63% improvement over standard RL is the number that should get attention, but the more interesting result is the transfer across task types. Calibrated verbal uncertainty generalizing from factual QA to math reasoning suggests the model is learning something about epistemic self-awareness as a skill - not just memorizing which topics to hedge on. That is a qualitatively different outcome than domain-specific calibration fine-tuning. The metacognitive data selection piece is elegant but underspecified in the abstract - the exact mechanism by which self-judgments are converted into a training signal deserves scrutiny, since self-assessment quality is itself model-dependent and can compound errors if the base model's self-judgment is systematically biased. The decoupled two-stage design is a practical insight worth generalizing: whenever you have two objectives that interfere under joint optimization, separating them into sequential stages with independent reward functions is a cleaner engineering choice than trying to balance loss weights. The open question is scalability - these results use Qwen3 and Llama 3, but whether the faithful calibration gains hold at GPT-5 or Gemini 3 Pro scale, where base calibration is already higher, is not addressed. The gains may compress significantly at the frontier.
Discussion Question
RLMF trains models to express calibrated uncertainty verbally - but verbal uncertainty and behavioral uncertainty are different things. A model might say "I'm not sure" and still execute a tool call or make a decision as if it were certain. How would you design an agentic system that uses a model's expressed uncertainty as a genuine signal for when to pause, escalate, or retrieve - rather than just a stylistic output feature?
Read the Paper on arXiv →