What are the ironies of automation?

The ironies of automation are the problems that automation creates for the person left standing beside it. Lisanne Bainbridge set them out in five pages in Automatica in 1983: automating the routine parts of a job hands the operator the exceptions, the emergencies and the watching, while taking away the daily practice that built the competence to handle any of them. It is now among the most cited papers in human factors, and most of what is currently argued about human oversight of AI is a restatement of it.

A paper about chemical plants, applied to something else#

Bainbridge was writing about industrial process control: plants and power stations where a computer holds variables steady and a person sits in a control room. Nothing in the paper concerns language models, and the estate treats the application to them as an argument by analogy and not as a measurement. That caveat is worth making early, because the paper is quoted so confidently in AI governance documents that a reader could easily take it for evidence about AI. It is evidence about what happens to people when a machine takes the routine half of their work, which is a claim general enough to be useful and specific enough to be wrong in places.

What makes it durable is the structure of the argument. Bainbridge starts from the designer, and she gives the designer the least flattering motive available: the operator is seen as unreliable and inefficient, so the design tries to remove them. Her observation is that the attempt fails in a particular way. The designer removes everything they can work out how to automate, and whatever is left over stays with the person. So the human job is defined by the limits of the engineering and not by what a human being happens to be good at.

The sequence Bainbridge sets out#

She does not number them, and readers who cite "the four ironies" or "the five ironies" are imposing an order she did not write. Barry Strauch, reviewing the paper for IEEE Transactions on Human-Machine Systems in 2018, narrates them in sequence instead, and this is that sequence with his page citations to the original.

1. The more automated the system, the more it depends on the person. "The more advanced a control system is," Bainbridge wrote, "the more crucial may be the contribution of the human operator" (p. 775).
2. The skills needed for take-over are the skills automation has been practising instead of you. Her line is the one everybody quotes: "a formerly experienced operator may now be an inexperienced one" (p. 775).
3. Take-over arrives at the worst moment. "When manual take-over is needed there is likely to be something wrong with the process, so that unusual actions will be needed to control it, and one can argue that the operator needs to be more rather than less skilled, and less rather than more loaded, than average" (p. 775).
4. The monitoring assignment contradicts its own premise. "The automatic control system has been put in because it can do the job better than the operator, but yet the operator is being asked to monitor that it is working effectively" (p. 776).
5. So does the judgement assignment. "If the computer is being used to make the decisions because human judgment and intuitive reasoning are not adequate in this context, then which of the decisions is to be accepted? The human monitor has been given an impossible task" (p. 776).
6. The resulting job is a bad one. A job both boring and responsible is, in her words, "one of the worst types", and it offers no opportunity for the person "to acquire or maintain the qualities required to handle the responsibility" (p. 776).

Those quotations are reproduced from Strauch's retrospective, with the page numbers he gives. The Automatica original sits behind Elsevier's paywall and was not read at source for this page, so anyone quoting from here is quoting Bainbridge at one remove and should say so.

How far the paper travelled, counted rather than asserted#

Strauch put a number on the reach: "As of early November 2016, Google Scholar listed 1800 scholarly works that had cited Ironies of Automation." He set it beside two comparators from the same literature, Wiener and Curry's 1980 paper on flight-deck automation at 564 citations and Norman's 1990 paper at 488, and noted that ten further works cited Bainbridge in the fortnight he was counting.

A citation count measures circulation and says nothing about whether the argument is right, or even whether the citing authors read it. This estate holds several cases of a paper being cited widely for a claim it does not make, and the traffic around Bainbridge has produced at least one of its own: see the vigilance decrement, where a sentence of hers about half an hour of attention has become, through repetition, a finding that nobody measured in the form it is now quoted.

What Strauch added in 2018, and what he conceded#

Strauch's position is that the ironies are unresolvable and not merely unresolved, for as long as two things hold: that no developer can guarantee a system free of anomalies, and that regulators, companies and the public will therefore insist a human being remains involved. He names several ironies that had emerged since 1983. Automation can disguise a shortcoming an operator already had, so that the shortcoming surfaces only in the emergency. A minor anomaly can become catastrophic through the operator's attempt to resolve it using the same automation that helped cause it. And a person can become fully qualified to operate an automated system without acquiring the expertise to understand what it does, which he calls an irony that could not have been recognised in Bainbridge's time.

He is careful about the counter-evidence, and the care is the reason to cite him. His own account records that fatal commercial jet accidents in the United States have become rare, against two to three major air transport accidents a year around 1983, and he allows in the same paragraph that this may be a short-term statistical aberration before arguing that operations have probably become safer. A paper arguing that the ironies remain unresolved, printing the figure that most obviously cuts against it, is doing the thing this research asks of a source.

Where the argument runs out#

Bainbridge reports no data. The paper is a theoretical analysis of process control and it is graded here on that basis. Strauch's method is a set of accident narratives selected because they illustrate the ironies, which can show that the mechanism occurs and cannot show how often it does. Neither establishes a rate, and no page on this site should use either to say how common any of this is.

The transfer to generative AI breaks in one specific place, and it breaks in the direction that makes things worse. Bainbridge's operator knew when control had been handed back: an alarm sounded, a loop dropped out, the plant misbehaved visibly. A person reading model output gets no such signal. The output looks the same whether the model is inside its competence or outside it, which is the finding the estate files under the jagged frontier. Her operator was also, once, a practised operator. A graduate reviewing AI-drafted work today may never have produced that kind of work unaided at all, so there is no earlier competence to decay: the condition described at the missing rungs and synthetic seniority is a different starting point from the one Bainbridge assumed, and her remedies assume the practice existed.

Her own remedies, and the one that misfired#

Bainbridge proposed four things: better alarms and displays; automatic shutdown on failure where shutting down does not itself destabilise the process; periodic hands-on control by the operator to keep the skill alive; and simulator practice where hands-on control is impractical. Her line about the third has survived as well as any in the paper. If letting the operator take manual control for a short period in each shift sounds laughable, then simulator practice has to be provided instead.

The first remedy has a well-documented failure of its own. Because a person cannot watch indefinitely, Bainbridge suggested the watching be done by an automatic alarm system connected to sound signals. Medicine ran that experiment at scale and the result is alarm fatigue: an intensive care study recorded more than 2.5 million alarms in five units in a single month, and clinicians learned to disregard them. Handing the monitoring to an alarm moves the problem to whoever has to decide which alarms matter.

What an organisation can actually do with this#

Four things follow, and three of them are design decisions rather than exhortations.

• Keep the practice on purpose. If people are expected to take over, they need to do the work often enough to be able to. This is Bainbridge's periodic manual control, and in a knowledge organisation it looks like a proportion of work done unaided by design, recorded as a cost.
• Make the hand-back visible. Automation that fails quietly is worse than automation that fails loudly, which is her own point about graceful degradation being a virtue in a person and a hazard in a machine. Where a system cannot signal that it has left its competence, somebody has to check on a schedule that does not depend on noticing.
• Size the watching against what a person can do. An oversight arrangement that assumes continuous attention has assumed something the evidence does not support. The relevant limits are at the vigilance decrement and automation complacency.
• Count the training as a cost of the automation. The final line of the paper, in Bainbridge's framing, is that the most successful automated systems, the ones that rarely need a human, may need the greatest investment in human training. A business case that books the savings and not that investment has understated the price.

Key sources

• Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775 to 779. DOI 10.1016/0005-1098(83)90046-8. Paywalled at Elsevier and not read at source for this page; all quotations here are as reproduced in Strauch (2018) with his page citations. Graded entry.
• Strauch, B. (2018). Ironies of Automation: Still Unresolved After All These Years. IEEE Transactions on Human-Machine Systems, 48(5), 419 to 433. DOI 10.1109/THMS.2017.2732506. The copy read for this page is IEEE's accepted-article proof, which carries the DOI, the manuscript dates and the full text, and not the final pagination. Graded entry.
• Endsley, M. R. and Kiris, E. O. (1995). The Out-of-the-Loop Performance Problem and Level of Control in Automation. Human Factors, 37(2), 381 to 394. Graded entry.
• Parasuraman, R. and Manzey, D. H. (2010). Complacency and Bias in Human Use of Automation: An Attentional Integration. Human Factors, 52(3). Graded entry.

Related SuperSkills research#

The two mechanisms underneath this are the vigilance decrement and the out-of-the-loop performance problem, and the remedy that failed is alarm fatigue. On what the oversight job actually consists of, the invisible work of oversight and meaningful human oversight. On why a person in the loop is not by itself a control, human in the loop is not a safeguard and the moral crumple zone. On the skill half of the argument, deskilling and how fast skills decay.