Automation bias

Automation bias is the tendency to accept what an automated system tells you without applying the scrutiny you would apply to the same claim from a person. It is an established concept from human-factors research, not a SuperSkills term, and it is considerably older than artificial intelligence: it was named and measured in cockpits, hospitals and military command systems long before anyone asked a chatbot for advice. Two things make it more dangerous than it sounds. It appears in novices and experts alike, so seniority is not protection. And it splits into two kinds of error, only one of which anyone designs against.

The two kinds of error, and why the split matters

Skitka, Mosier and Burdick demonstrated automation bias experimentally in 1999 and separated it into errors of commission, where a person acts on a recommendation that is wrong, and errors of omission, where a person misses something the system did not flag.

Almost every oversight process ever written is designed to catch the first. A reviewer checks the output, questions the recommendation, signs it off. Omission errors are invisible by construction: nothing appears on the screen to check, no alert is raised, no decision is presented. The absence is the error, and absences do not arrive in an inbox. That is why an organisation can have a well-designed review process, high compliance with it, and still accumulate exactly the failures the process cannot see.

What the evidence shows

Parasuraman and Manzey, reviewing decades of work across aviation, medicine and the military in 2010, established the durability of the effect. It appears in experienced operators as readily as in novices, it resists training, and it worsens under time pressure and high workload. Their term for the underlying state is automation complacency: attention withdraws from a system that is performing well, which is a rational allocation of a scarce resource right up to the moment it is not.

The most vivid contemporary demonstration is the jagged technological frontier. Dell'Acqua and colleagues gave 758 consultants access to GPT-4 in 2023. Inside the model's competence they were dramatically better. On a task deliberately placed just outside it, consultants using AI performed worse than consultants with no AI at all. They did not fail through carelessness. They failed because confident, fluent output does not signal which side of the frontier it is on.

The most uncomfortable finding is about the remedy. Dzindolet and colleagues, in 2003, found that trust mediates reliance, as you would expect. But they also found that explaining why an aid might err increased reliance on it, even when that restored trust was unwarranted. If your governance rests on transparency producing appropriate scepticism, this study says the mechanism can run backwards.

And calibration does not settle in a sensible place on its own. Dietvorst, Simmons and Massey described algorithm aversion in 2015: after seeing an algorithm err, people abandon it even when it outperforms them. Logg, Minson and Moore described algorithm appreciation in 2019: for many estimates people weight algorithmic advice more heavily than human advice, with domain experts the notable exception. Trust moves for reasons unrelated to accuracy, in both directions.

What it looks like outside the laboratory

Marine accident investigators have documented the mechanism in its cleanest form. A 2021 joint study by the UK Marine Accident Investigation Branch and its Danish counterpart, following groundings by ships using electronic charts, interviewed 155 deck officers and observed 31 ships. Their finding is worth quoting: distrust of the instrument, "which is traditionally expected of OOWs, is challenged, because such discrepancies are rarely encountered."

The system was not unreliable. It was reliable enough, often enough, that the habit of checking had no occasion to be exercised and eventually stopped existing. That is automation bias described as a decay process rather than a personality flaw, and it explains why training alone does not fix it. The officers had been trained. What they lacked was reasons to doubt.

Where the evidence is uncertain

Most of the foundational work concerns deterministic automation: systems that behave the same way every time and fail in characteristic ways an operator can learn. Generative AI is different. It is probabilistic, its competence is jagged rather than bounded, and its failures are fluent rather than obviously odd. The direction of the finding almost certainly transfers. The magnitude may not, and could plausibly be worse, since a wrong answer that reads well is harder to catch than an alarm that does not sound.

The trust literature is also mostly laboratory work with bounded tasks and clear right answers. Real decisions are longer, more ambiguous and rarely scored, which is precisely the environment in which nobody finds out whether the bias operated.

The SuperSkills view

Automation bias is usually treated as a warning about individuals. It is better understood as a fact about design. If the effect appears in experts, resists training, worsens under load and can be made worse by explanation, then no amount of telling people to think critically will address it. The only lever that reliably moves is the structure of the work: who decides, at what point, with what time, on what stated basis, with what authority to refuse.

This is why I argue that a human in the loop is not a control. Placing a person at the end of an automated process, with no time budget and no stated grounds for disagreement, does not counteract automation bias. It creates ideal conditions for it. The person is tired, the output is fluent, the queue is long, and nothing in the design requires them to reconstruct the reasoning. What you get is a signature.

The practical countermeasure is unglamorous. Decide in advance what would make you reject the output, before you see it. Specify the categories of case where the system is known to be weak. Measure the disagreement rate and investigate when it approaches zero, because a process in which nobody ever disagrees is indistinguishable from a process in which nobody is looking.

Related SuperSkills research

The mechanism underneath it is cognitive offloading. The design response is human and AI decision making and Human at the Start. The wider argument is AI and human judgement, and the organisational accumulation is capability debt.

Key research and primary sources

• Parasuraman, R. and Manzey, D. H. (2010). Complacency and Bias in Human Use of Automation: An Attentional Integration. Human Factors, 52(3).
• Skitka, L. J., Mosier, K. L. and Burdick, M. (1999). Does automation bias decision-making? International Journal of Human-Computer Studies, 51(5).
• Dzindolet, M. T. et al. (2003). The role of trust in automation reliance. International Journal of Human-Computer Studies, 58(6).
• Parasuraman, R. and Riley, V. (1997). Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors, 39(2).
• Dell'Acqua, F. et al. (2023). Navigating the Jagged Technological Frontier. Harvard Business School and BCG working paper.
• Dietvorst, B. J., Simmons, J. P. and Massey, C. (2015). Algorithm Aversion. Journal of Experimental Psychology: General, 144(1).
• Marine Accident Investigation Branch and Danish Maritime Accident Investigation Board (2021). Application and Usability of ECDIS.

About this research

Rahim Hirji is the author of SuperSkills: The Seven Human Skills for the Age of AI (Kogan Page, 2026) and the founder of The SuperSkills Intelligence Company. Automation bias, automation complacency, algorithm aversion and algorithm appreciation are established concepts from human-factors and decision research and were not coined by him; this page defines them because the SuperSkills argument rests on them and because misattribution is common. The graded evidence is in the evidence base.