9 of 10 widely used benchmarks score 'I do not know' as zero, so language models learn to bluff.
The usual explanation is that a language model invents facts because it only predicts the next word, and that connecting it to your own documents makes the problem go away. A fuller version says training and testing reward a confident guess over an honest 'I do not know', and that the fixes with evidence are grounded answers with citations, permission to abstain, and scoring that punishes confident errors.
- 9 of 10benchmarks with no credit for abstaining
- 26% vs 75%wrong answers on SimpleQA, declining vs guessing model
- 74.5%citations that support their sentence
- Models state false things confidently because training and testing reward a guess over 'I do not know'.Kalai and colleagues at OpenAI and Georgia Tech show the errors start in pretraining: facts with no pattern, such as birthdays, can only be memorised, so if 20 percent of them appear once in the training text, expect the base model to get at least 20 percent wrong, even with error-free data. The errors then survive because of grading. Under binary scoring, one point for a right answer and nothing for a wrong one or an abstention, declining is never the best move. Of the 10 widely used benchmarks the authors reviewed, 9 score this way. Only WildBench gives abstention partial credit, and the authors note its rubric can still rate 'I do not know' below a flawed answer.Why Language Models Hallucinate (Observation 1, Table 2)
- Connecting the model to your documents removes the problem.Retrieval raised the share of correct answers in a study by researchers at Google, UC San Diego and Duke, but it also made models less willing to decline. Claude 3.5 Sonnet abstained on 84.1 percent of questions without retrieved text and on 52 percent with it; Gemini 1.5 Pro went from 100 percent to 18.6 percent; GPT-4o barely moved, from 34.4 to 31.2. On the Musique question set, 55.4 percent of questions came with passages that did not hold enough to answer, and the larger models often answered wrongly instead of declining when the passages fell short. A retrieval miss does not look like an error in production; it looks like a fluent answer.Sufficient Context: A New Lens on Retrieval Augmented Generation Systems (Section 4.2, Figure 1)
- Answers that cite their sources can be trusted.Grounding has evidence behind it: in the original RAG paper by Lewis and colleagues, human raters comparing 452 pairs of generated quiz questions judged the retrieval model more factual in 42.7 percent of pairs and the plain model in 7.1 percent. A citation is still a claim to be checked. A Stanford audit of four generative search engines, Bing Chat and Perplexity among them, found 51.5 percent of generated sentences fully supported by their citations, and 74.5 percent of citations supporting the sentence they were attached to. Roughly 1 citation in 4 did not back its sentence.Evaluating Verifiability in Generative Search Engines
- Letting a model decline cuts false answers without costing much accuracy.OpenAI's GPT-5 system card gives SimpleQA scores without web access for two small reasoning models. o4-mini: 24 percent correct, 75 percent wrong. gpt-5-thinking-mini: 22 percent correct, 26 percent wrong. The card prints no declined share; if every question is answered right, answered wrong or declined, the remainders are 1 percent and 52 percent. Two points of accuracy went with a wrong-answer rate about a third the size (26 against 75). This is one vendor comparing two models that differ in more than their willingness to decline, and the trade is not always this cheap: in the Sufficient Context study, fine-tuning a Mistral model to decline more often cost it correct answers. On an accuracy-only leaderboard the first model still ranks higher, which is the incentive problem in the first check.GPT-5 System Card (Table 8, SimpleQA)
- Scoring that penalises confident errors fixes the incentive.The arithmetic holds; the paper offers it as a proposal, not a measured result. State a confidence target t in the instructions and charge t/(1−t) points per wrong answer, so a guess pays only when the model is more than t sure. At t = 0.9 a wrong answer costs 9 points, and an answer that is 85 percent likely to be right scores 0.85 − 0.15 × 9 = −0.5 against 0 for declining. (The paper prints a penalty of 2 for t = 0.75; its own formula gives 3.) The authors argue the rule must go into the main benchmarks, because adding separate hallucination tests leaves the incentive in place.Why Language Models Hallucinate (Section 4.2, explicit confidence targets)
the line to remember
Models bluff because bluffing scores well; score a wrong answer below 'I do not know', ground each claim in a passage, and check the passage supports it.
For your product
Accuracy alone is the wrong number to buy on. Ask a vendor for three figures measured on your own questions: how often the system is right, how often it is wrong, and how often it declines. In acceptance tests, score a wrong answer as worse than a declined one, include questions your documents cannot answer so you see what happens when search comes back thin, and sample the cited passages to confirm they support the sentence. A system that sometimes declines and is rarely wrong is usually worth more than one that always answers.
Sources: Why Language Models Hallucinate (Kalai et al., OpenAI and Georgia Tech) · Sufficient Context: A New Lens on Retrieval Augmented Generation Systems (Joren et al., UC San Diego, Duke University and Google) · Evaluating Verifiability in Generative Search Engines (Liu, Zhang and Liang, Stanford University, Findings of EMNLP) · Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., Facebook AI Research, UCL and NYU, NeurIPS) · GPT-5 System Card (OpenAI)