Hallucination

score = right − wrong × t / (1 − t); answer only if confidence > t

The grading Kalai and colleagues propose: state a confidence target t and charge t/(1−t) points for each wrong answer. A guess then pays only when the model is more than t sure, so honest abstention is rewarded instead of punished.

Worked example: With t = 0.9 a wrong answer costs 9 points. A guess that is 80% likely to be right is worth 0.8 × 1 − 0.2 × 9 = −1.0, so “I don't know”, worth 0, is the better reply.

H = −Σ p(c) · ln p(c), summed over clusters c of answers that mean the same

Semantic entropy: sample several answers, group those with the same meaning, and measure how spread out the groups are. Zero means every answer agrees; high values flag likely confabulation.

Worked example: Five answers 1887, 1891, 1887, 1902, 1884 form clusters with shares 0.4, 0.2, 0.2 and 0.2, so H ≈ 1.33 out of a possible 1.61; five answers that all mean “Paris” give H = 0.

calibrated: accuracy of answers given with confidence p ≈ p

A threshold on confidence only helps if the confidence means something. Check it by grouping answers by stated confidence and comparing each group's accuracy.

Worked example: If a model says 90% on 100 answers and 70 are right, it overstates its confidence by 20 points; SimpleQA found that models consistently overstate theirs.