A system is calibrated when the confidence it states matches how often it turns out to be right. Take every case where it said it was 80 per cent sure; if it was correct in about 80 per cent of them, it is well calibrated. This is a different property from accuracy, and the two come apart in both directions. A system can be accurate and badly calibrated, by being right most of the time while claiming a certainty it has not earned. It can be inaccurate and well calibrated, by being wrong often and saying so. Only one of those two is safe to rely on, and the accurate one is not it.
The answer, in one line
A system is calibrated when its stated confidence matches how often it turns out to be right. If it says 80 per cent across a thousand cases and is correct in about 800 of them, it is well calibrated.
How it is measured#
The standard summary is expected calibration error. Group the predictions into confidence bins, compare the average stated confidence in each bin against the accuracy actually achieved in that bin, and average the gaps, weighted by how many predictions fall in each. Zero is perfect. The measure is a summary and behaves like one: it can hide compensating errors, where a system is overconfident in one band and underconfident in another, so it is read alongside a reliability diagram rather than on its own.
A second measure answers a different question. AUC, the area under the receiver operating characteristic curve, asks whether the confidence signal separates the cases where the system is right from the cases where it is wrong. Calibration asks whether the number is honest in absolute terms; AUC asks whether it is useful for ranking. A system can discriminate well and be badly calibrated, which is a fixable problem, and the reverse, which is not.
What the numbers are#
Xiong and colleagues tested five models across eight datasets. The figures for plain verbalised confidence deserve the space, because the summary "models are overconfident" understates them. Average expected calibration error: 0.520 for GPT-3, 0.461 for Vicuna, 0.436 for LLaMA 2, 0.377 for GPT-3.5 and 0.180 for GPT-4. Even GPT-4's average AUROC of 62.7 per cent sits close to the 50 per cent chance line.
The shape of the numbers is as informative as their size. Stated confidences arrive "as multiples of 5 and with most values ranging between the 80% to 100% range", which the authors suggest means the models "might be imitating human expressions when verbalizing confidence". A system asked how sure it is produces a plausible human-sounding percentage. That is a text-generation act rather than a measurement.
One distinction decides how to read all of this. Internal confidence signals, derived from the model's own probabilities, are considerably better calibrated than the number the model writes in a sentence. The user almost always sees the written one.
The second gap, which is the one that hurts#
Even a calibrated model does not produce a calibrated reader. Steyvers, Tejeda, Kumar, Belem, Karny, Hu, Mayer and Smyth measured both ends on the same items with 301 participants. Model confidence separated correct from incorrect answers at an AUC of 0.751 for GPT-3.5 and 0.781 for GPT-4o. Participants reading those models' explanations reached 0.589 and 0.592, which the authors describe as "only slightly better than random guessing".
They name the shortfall the calibration gap, alongside a discrimination gap, and attribute the human side of it to overconfidence: people "generally believe that LLMs are more accurate than they actually are". Their most practically awkward finding is about length. Longer explanations significantly raised participant confidence while leaving discrimination unchanged, at a mean participant AUC of 0.54. More detail made readers surer without making them righter, which is the opposite of what an interface designer would predict.
Why it belongs in a governance conversation#
Calibration is usually treated as a machine-learning property and filed with the engineers. It decides something organisational. Every oversight arrangement that asks a person to check the cases where the system is unsure depends on the system knowing when it is unsure, and on that knowledge reaching the person in a form they read correctly. Both links are measurably weak, and neither is required by law: nothing in Regulation (EU) 2024/1689 obliges a system to tell the person in front of it how confident it is in the specific output.
So a confidence score is not a control until somebody has checked that it is calibrated on their own data, and checked that the people acting on it read it as intended. The estate's fuller treatment is on how an AI agent should communicate uncertainty.
Related SuperSkills research#
On what to do with the signal, how an AI agent should communicate uncertainty, does explaining an AI decision help and how to know when AI is wrong. On the reading side, over-reliance, automation bias and why AI sounds so confident. On the underlying behaviour, what an AI hallucination is.
Key research and primary sources
- Xiong, M., Hu, Z., Lu, X., Li, Y., Fu, J., He, J. and Hooi, B. (2024). Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. ICLR 2024.
- Steyvers, M., Tejeda, H., Kumar, A., Belem, C., Karny, S., Hu, X., Mayer, L. W. and Smyth, P. (2025). What large language models know and what people think they know. Nature Machine Intelligence, 7, 221-231.
- Zhang, Y., Liao, Q. V. and Bellamy, R. K. E. (2020). Effect of confidence and explanation on accuracy and trust calibration in AI-assisted decision making. FAT* 2020.
- Zhou, K., Hwang, J. D., Ren, X. and Sap, M. (2024). Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty. ACL 2024.
About this research#
Rahim Hirji is the author of SuperSkills (Kogan Page, 2026), keynote speaker on AI and human capability, and founder of The SuperSkills Intelligence Company. Calibration and expected calibration error are standard terms in statistics and machine learning. The calibration gap and the discrimination gap are Steyvers and colleagues' terms. Nothing on this page is a SuperSkills coinage. The expected calibration error values from the Steyvers paper are absent here on purpose: they sit inside a figure rather than in the text, so only the AUC figures, which are stated in the body, are quoted.
Explainer · SS-2026-172 · Graded against the published rubric
Hirji, R. (2026). Calibration. The SuperSkills evidence base, SS-2026-172. https://thesuperskills.com/research/what-is-calibration. Last reviewed 4 September 2026.
An evidence review by Rahim Hirji, not peer-reviewed research. For a material claim, cite the underlying study as well; every study here carries its own permanent link.
How citations and IDs work