Does explaining an AI's reasoning help?

Sometimes, and not reliably. Explanations can improve understanding and help people calibrate when to rely on a system. They can also increase reliance on wrong answers, because a fluent reason makes an output feel checked when it has not been.

The evidence supports neither a simple yes nor a simple no, so this page does not close the question. What it can settle is the distinction almost everyone asking it is missing.

Explanation accuracy is not decision accuracy

NIST separates the two by name, and the separation does most of the useful work here.

In the report's own terms, explanation accuracy is a distinct concept from decision accuracy. Decision accuracy is whether the system's judgement is correct. Explanation accuracy is whether the explanation correctly describes how the system reached it. Regardless of the system's decision accuracy, the explanation may or may not accurately describe how it came to its conclusion. Graded entry.

Four properties are set out: that an explanation is given, that it is meaningful to its audience, that it is accurate about the process, and that the system operates only within the conditions it was designed for. NIST is explicit that the first two alone do not require an explanation to reflect what the system actually did.

Which produces four combinations rather than two. Right answer with a true explanation. Right answer with a false explanation. Wrong answer with a false explanation. And wrong answer with a true explanation, which is the most useful of the four and the least discussed, because a faithful account of how a system reached a bad conclusion is what an auditor needs above all else.

Why explanations can make things worse

The failure mode is not that explanations are false. It is that they satisfy the impulse that would otherwise have produced a check.

Automation bias is the documented tendency to under-question automated advice, established across decades of human factors work. Skitka and colleagues, Parasuraman and Manzey. An explanation gives a reviewer something to accept rather than something to verify, and accepting a reason feels like scrutiny in a way that accepting a bare answer does not.

This is the same mechanism as Fisher's search result: access to an external source was experienced as personal knowledge. Graded entry. Here, an account of the reasoning is experienced as having followed the reasoning.

The EU AI Act legislates against exactly this, requiring that overseers remain aware of the tendency to over-rely on system output. Graded entry. Whether awareness is sufficient protection is a separate matter, and the literature on debiasing suggests it usually is not.

What an explanation is actually good for

Three uses survive the above, and they are narrower than the general claim.

Spotting the wrong basis. If the explanation shows the output rests on a feature that should be irrelevant, that is genuine information. It is also available without the answer itself being checkable.

Auditing after the fact. A faithful account of process is what an investigation needs, worth having whether or not it helps anyone in the moment.

Knowing the system is out of range. NIST's fourth principle, knowledge limits, is arguably the most useful of the four and the least implemented: a system saying it is outside the conditions it was designed for tells you more than any account of its reasoning.

What an explanation cannot do is warrant the conclusion. The only thing that does that is checking the conclusion against something independent of the system.

What this page does not establish

NIST's report is a framework rather than an experiment. It reports no effect on human decision quality and should not be cited as though it did.

The claim that explanations increase misplaced trust is supported by the automation bias literature and by the mechanism, and the direct experimental evidence on explanation specifically is mixed, with results going both ways depending on task, interface and expertise. That mixture is the reason this question stays partly open rather than being closed in either direction.

There is also a real argument that this page should not flatten. Explanations serve purposes other than helping the immediate reviewer: contestability, redress, regulatory compliance and the right of an affected person to be told why. A system that helps nobody decide faster can still be the right requirement.

Key sources

• Phillips, P. J. et al. (2021). Four Principles of Explainable Artificial Intelligence. NISTIR 8312. Graded entry.
• Skitka, L. J., Mosier, K. L. and Burdick, M. (1999). Does automation bias decision-making? Graded entry.
• Parasuraman, R. and Manzey, D. H. (2010). Complacency and Bias in Human Use of Automation. Graded entry.
• Fisher, M., Goddu, M. K. and Keil, F. C. (2015). Searching for Explanations. Graded entry.
• European Union (2024). Regulation (EU) 2024/1689, Article 14. Graded entry.

Related SuperSkills research

On confident prose as a false signal, why does AI sound so confident. On the bias underneath, automation bias and automation complacency. On who can act on a doubt, who can override an AI system. On telling when it is wrong, how do I know when AI is wrong. On auditing afterwards, how do you audit an AI-assisted decision.

About this research

Rahim Hirji is the author of SuperSkills (Kogan Page, 2026), keynote speaker on AI and human capability, and founder of The SuperSkills Intelligence Company. The four principles and the explanation-accuracy distinction are NIST's, and this page applies them rather than adding to them.

How this research works  ·  Reviewed quarterly  ·  Found an error? Tell me and it is corrected on the page.