How do you design a stop button people will actually use?

By making stopping cheaper than not stopping, and by keeping the control's false alarm rate low enough that pressing it stays a rational act. The regulation supplies the button. Everything that decides whether a human being reaches for it sits outside the regulation: how often the system has cried wolf, whether the person can tell a real condition from a spurious one in the seconds available, whether they can plausibly be blamed for a stop that turns out to be unnecessary, and whether anyone ever looks at how often it was used. Industries with far higher stakes have been getting this wrong for four decades, and the record of how they got it wrong is public.

The regulation buys you the mechanism and stops there#

Article 14 of Regulation (EU) 2024/1689 requires high-risk AI systems to be designed so that natural persons can effectively oversee them. Paragraph 4 lists what those persons must be enabled to do. The fifth item is the button.

to intervene in the operation of the high-risk AI system or interrupt the system through a 'stop' button or a similar procedure that allows the system to come to a halt in a safe state.

The paragraph before it is the interesting one, because the same regulation already anticipates the problem. Oversight personnel must be enabled "to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias), in particular for high-risk AI systems used to provide information or recommendations for decisions to be taken by natural persons". A regulator has written the failure mode into the statute alongside the control, and has left the mechanism for addressing it entirely to the deployer. Compliance is a design question dressed as a procurement one.

Most organisations reading this are not in scope for Article 14 and will build stop controls anyway, because an agent that acts on their behalf needs one. The scoping question and the design question come apart here. The design question has an evidence base, and it does not come from AI.

Hospitals have measured what happens to a control that fires too often#

The best-quantified version of this problem is clinical alarms. Drew and colleagues recorded every alarm from bedside physiologic monitors across five adult intensive care units at the University of California, San Francisco, for the 31 days of March 2013. The count: 2,558,760 unique alarms, of which 381,560 were audible, giving "an audible alarm burden of 187/bed/day". Nurse scientists annotated 12,671 arrhythmia alarms against a defined protocol with 95 per cent inter-rater agreement, and found 88.8 per cent of them were false positives. Of the 168 true ventricular tachycardia alarms, 93 per cent "were not sustained long enough to warrant treatment".

Read the second figure alongside the first. Nearly nine in ten alarms were wrong, and of the ones that were right, nine in ten did not require anybody to do anything. A clinician who ignored every alarm on that unit would have been correct almost every time, and the one occasion on which they were not is the one the system exists for.

The consequence is on the record. The Joint Commission's Sentinel Event Alert 50, dated 8 April 2013, describes what people do in response: "In response to this constant barrage of noise, clinicians may turn down the volume of the alarm, turn it off, or adjust the alarm settings outside the limits that are safe and appropriate for the patient, all of which can have serious, often fatal, consequences." Its database held 98 alarm-related events between January 2009 and June 2012, of which 80 resulted in death, 13 in permanent loss of function and five in unexpected additional care or extended stay. The Commission's own footnote should travel with those numbers: reporting is voluntary, represents "only a small proportion of actual events", and no conclusion about frequency or trend should be drawn from it. The figure is a floor with an unknown ceiling, which is worse than a rate rather than better.

The Commission's contributing-factor tally is the design brief. Alarm signals inappropriately turned off, 36. Absent or inadequate alarm system, 30. Alarm signals not audible in all areas, 25. Improper alarm settings, 21. Every one of those is a decision somebody made about a control that was supposed to be the safety net.

Parasuraman and Riley named this in 1997 and nobody in AI governance cites it#

The vocabulary for the whole problem already exists. Parasuraman and Riley set out four ways humans and automation interact, and the third of them describes the stop button exactly.

Disuse, or the neglect or underutilization of automation, is commonly caused by alarms that activate falsely. This often occurs because the base rate of the condition to be detected is not considered in setting the trade-off between false alarms and omissions.

The second sentence contains the engineering. When the condition being detected is rare, a detector with an excellent false positive rate still produces mostly false positives, because there is so little true signal to be right about. An agent-monitoring alarm tuned to catch an event that occurs once a quarter will fire wrongly many times before it fires correctly once, and by then the person watching will have learned, accurately, that it means nothing. Neglecting the base rate is a specification failure that then produces an entirely rational human response.

Their fourth category names the organisational version: "Automation abuse, or the automation of functions by designers and implementation by managers without due regard for the consequences for human performance, tends to define the operator's roles as by-products of the automation." A stop button added at the end of a design, to a person whose role was never designed, is that category.

The person holding the button cannot do the job the button assumes#

Lisanne Bainbridge published the definitive four pages on this in 1983, and every sentence of it applies to an AI agent.

We know from many 'vigilance' studies (Mackworth, 1950) that it is impossible for even a highly motivated human being to maintain effective visual attention towards a source of information on which very little happens, for more than about half an hour.

She then closes the loop that most oversight policy leaves open: "This raises the question of who notices when the alarm system is not working properly. Again, the operator will not monitor the automatics effectively if they have been operating acceptably for a long period." A well-behaved system trains its supervisor out of supervising. And the moment the button is needed is the worst possible moment to need a person: "When manual take-over is needed there is likely to be something wrong with the process, so that unusual actions will be needed to control it, and one can argue that the operator needs to be more rather than less skilled, and less rather than more loaded, than average."

Her central irony describes the AI oversight arrangement without alteration: "the automatic control system has been put in because it can do the job better than the operator, but yet the operator is being asked to monitor that it is working effectively." The estate's argument that human in the loop is not a safeguard, and that supervising work you cannot do is presence rather than scrutiny, is a restatement of Bainbridge with a language model in place of a process plant.

One second of silence in Tempe#

The clearest documented case of a stop control that existed and was not used is the crash in Tempe, Arizona, on 18 March 2018, investigated by the US National Transportation Safety Board as HWY18MH010 and reported as NTSB/HAR-19/03. Three design decisions in that report deserve to be read by anyone specifying an agent.

The system saw the pedestrian in good time. "The ADS detected the pedestrian 5.6 seconds before impact. Although the ADS continued to track the pedestrian until the crash, it never accurately classified her as a pedestrian or predicted her path." The vehicle's own manufacturer-fitted safety net had been removed: "Because ATG disengaged the Volvo ADASs during ATG ADS operation, the Volvo ADASs were not active at the time of the crash." The NTSB found that deactivating the forward collision warning and automatic emergency braking "without replacing their full capabilities removed a layer of safety redundancy".

The third decision is the one this page is about, and the NTSB's wording is worth quoting in full.

When the system detected an emergency situation, it initiated action suppression. That was a 1-second period during which the ADS would suppress braking while (1) the system verified the nature of the detected hazard and calculated an alternative path, or (2) the vehicle operator took control of the vehicle. No alert was given to the operator when action suppression was initiated. ATG stated that it implemented action suppression because of concerns about false alarms, the ADS identifying a hazardous situation when none existed, that would cause the vehicle to engage in unnecessary extreme maneuvers. The primary countermeasure in an emergency situation was the vehicle operator, who was expected to recognize the hazard, to take control of the vehicle, and to intervene appropriately.

The design named a human as the primary countermeasure and then withheld from that human the single piece of information she needed to act as one. It did so for a defensible reason: false alarms would have made the vehicle behave erratically. That is the base rate trade-off from Parasuraman and Riley, made explicitly by an engineering team, resolved in favour of the machine's composure, and paid for by somebody who was not in the room.

The NTSB's probable cause is a finding about the operator, correctly so: she was visually distracted by her phone, and "had the vehicle operator been attentive, she would likely have had sufficient time to detect and react to the crossing pedestrian to avoid the crash or mitigate the impact". The contributing factors are a finding about everybody else, and they name the mechanism this page is about: "lack of adequate mechanisms for addressing operators' automation complacency, all a consequence of its inadequate safety culture".

The cost of stopping falls on the person who stops#

Alarm design is only half of it. The other half is what happens to the person afterwards, and the offshore oil industry has the clearest written record of that, because a regulator was forced to make stopping compulsory.

The US Bureau of Safety and Environmental Enforcement requires, in 30 CFR 250.1930, that safety and environmental management procedures "grant all personnel the responsibility and authority, without fear of reprisal, to stop work or decline to perform an assigned task when an imminent risk or danger exists". The phrase "without fear of reprisal" is in the regulation because it was not true. In the rulemaking preamble at 78 FR 20423, BSEE records the comment it received: that stop-work authority "should remain voluntary rather than mandatory", and that "in past OCS accidents, the SWA program did not function as designed because personnel hesitated to implement this provision due to fear of reprisal". That is an industry telling its regulator, on the record, that the authority existed and went unexercised.

Weber, MacGregor, Provan and Rae asked the people holding it why. Ten focus groups across the liquefied petroleum gas industry, published in Safety Science in 2018, and their title is a participant's sentence: "We can stop work, but then nothing gets done." The full quotation carries the mechanism: "We've got the ASW. We can stop work, there's no drama. But then nothing gets done. So you end up going back to the way you were doing [the work]." Their conclusion is the design principle: stopping "does not solely hinge on the willingness of individual workers to stop, but also depends on contextual factors surrounding the stop work decision".

Toyota's production system is the one widely documented arrangement that solves this deliberately, and it solves it by inverting who bears the cost. In the company's own description of jidoka, "the machine or equipment can detect the abnormality and stop automatically, or the operator can stop the line by pulling the stop cord themselves", and pulling the cord lights the andon board "so that workers can call the person in charge when there is an abnormality". Stopping summons help rather than scrutiny. The person who stops the line has performed the expected act, and somebody senior arrives to take the problem off them. Note what Toyota does and does not claim: the page documents that the mechanism exists. It reports no pull rate and makes no behavioural claim, and neither should anybody citing it.

Five properties that decide whether a stop control is real#

These are a synthesis of the record above rather than a validated instrument. They are stated as questions because each one has an answer that can be produced before deployment and checked afterwards.

• Detectability. Can the holder tell, in the time available, that the condition has occurred? Bainbridge's half hour is the ceiling on continuous monitoring, and Tempe is what a 1.2-second decision window looks like when the alert was withheld. If the answer requires reading an output the person could not have produced, the control is nominal.
• Base rate. What proportion of the alerts that prompt a stop will be wrong? This is a number a deployer can compute before shipping and almost never does. Above some threshold, ignoring the alert becomes the rational policy, and the person ignoring it is behaving correctly. Drew's 88.8 per cent is what the far end looks like.
• Cost to the stopper. What happens to the person who stops something that turns out to be fine? BSEE had to write "without fear of reprisal" into federal regulation, and it still had to make the programme mandatory. If a false stop costs the individual more than a missed stop costs them, the control will be used approximately never.
• Visibility. Is the stop logged, and does anyone review the count? A control that has never been used is either unnecessary or broken, and nothing distinguishes the two without a record. Article 12 of the same regulation requires the logging; nothing requires anybody to look. This is the same argument the estate makes about auditing an AI-assisted decision.
• Standing. Is the authority held by a named person with the seniority to survive using it? An authority granted to everybody is held by nobody, which is the finding of who can override an AI system. Name the role in the deployment record, before the system goes live.

Two of these are technical and three are organisational, which matches where the failures actually occurred. Nothing in the clinical, offshore or vehicle record turned on the button being hard to press.

Expect the control to be unpopular, and decide now that this is acceptable#

There is one direct experiment on interventions that force a person to engage rather than accept. Buçinca, Malaya and Gajos tested three cognitive forcing designs against two simple explainable-AI approaches and a no-AI baseline with 199 participants. Cognitive forcing significantly reduced overreliance. It also produced the least favourable subjective ratings of any design tested, and the benefit was larger for participants higher in Need for Cognition. Effective friction is disliked, unevenly, by the people it protects.

Anybody adding a stop control with teeth should expect the satisfaction score to fall and should write down, in advance, that the fall is the price. Otherwise the control is removed at the first efficiency review by somebody who has a number showing it is unpopular and no number showing what it prevented. That asymmetry, between a measurable irritation and an unmeasurable avoided harm, is the same one that makes the work of oversight invisible.

Where this sits in my own argument#

"Rules Before Tools" (2025) put the case that advantage comes from redesigned processes, named accountable owners and guardrails rather than from chasing model releases, and that the rules and decision rights should be fixed before the tools slot into them. A stop control is the smallest possible test of whether an organisation has done that: it is one rule, one owner, one guardrail, and most deployments cannot answer who holds it. "The Decision You Never Made" (2025) argued that the consequential choices about AI in most organisations were never made by anyone and accumulated out of individual convenience. A stop button that exists in the interface and nowhere in the operating model is that argument in miniature.

Attribution note. Disuse, misuse and abuse are Parasuraman and Riley's terms. The ironies of automation are Bainbridge's. Cognitive forcing functions are Buçinca, Malaya and Gajos's. Jidoka and andon are Toyota's. Stop-work authority is industrial safety vocabulary. Drift versus design is mine. The five properties above are a synthesis and are not claimed as a coinage.

Where the alarm evidence stops transferring#

It does not claim that clinical alarm evidence transfers directly to AI agents. Intensive care alarms are high-frequency, high-volume and physiological. Agent stop controls will be low-frequency and consequential, which is a different regime with a different failure profile. What transfers is the mechanism, not the rate.

It does not claim the Joint Commission figures measure the size of the problem. Reporting to that database is voluntary and the Commission says explicitly that no conclusions should be drawn about frequency or trend. The widely repeated estimate that between 85 and 99 per cent of alarm signals do not require clinical intervention is quoted by the Commission from a 2011 AAMI publication that was not read for this page, so it does not appear above as a figure. Drew's measured 88.8 per cent is used instead, and it applies only to the 12,671 annotated arrhythmia alarms rather than to all alarms.

It does not claim the Tempe crash was caused by the absence of an alert. The NTSB determined the probable cause was the operator's failure to monitor while visually distracted. The action suppression design is cited here as a documented decision about alerting under uncertainty, not as the cause.

It does not claim the five properties are validated. No study has tested them together, and no field evidence exists that a stop control built to them is used more often, because nobody has run that experiment. Treat them as a structure for a design review.

It states no application date for the EU provisions, because the implementation timetable has been amended and the published texts consulted for this estate have not agreed on it. A reader who needs a date should take it from the current Official Journal text.

Key sources

• European Union (2024). Regulation (EU) 2024/1689, Article 14: Human oversight. Read at the European Commission's AI Act Service Desk.
• Drew, B. J., Harris, P., Zegre-Hemsey, J. K., Mammone, T., Schindler, D., Salas-Boni, R. et al. (2014). Insights into the Problem of Alarm Fatigue with Physiologic Monitor Devices. PLoS ONE, 9(10), e110274. Full text.
• The Joint Commission (2013). Sentinel Event Alert 50: Medical device alarm safety in hospitals, 8 April 2013. PDF.
• Parasuraman, R. and Riley, V. (1997). Humans and Automation: Use, Misuse, Disuse, Abuse. Human Factors, 39(2), 230-253.
• Bainbridge, L. (1983). Ironies of Automation. Automatica, 19(6), 775-779.
• National Transportation Safety Board (2019). Collision Between Vehicle Controlled by Developmental Automated Driving System and Pedestrian, Tempe, Arizona, March 18, 2018. NTSB/HAR-19/03. Report.
• Bureau of Safety and Environmental Enforcement. 30 CFR 250.1930, stop work authority, and the rulemaking preamble at 78 FR 20423.
• Weber, D. E., MacGregor, S. C., Provan, D. J. and Rae, A. (2018). "We can stop work, but then nothing gets done." Safety Science, 108, 149-160.
• Bucinca, Z., Malaya, M. B. and Gajos, K. Z. (2021). To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI. PACM HCI, 5(CSCW1).
• Toyota Motor Corporation. Toyota Production System, on jidoka and the andon.

Related SuperSkills research#

On the authority itself, who can override an AI system, when should I override AI and allocating AI decision rights. On why presence is not oversight, human in the loop is not a safeguard, meaningful human oversight and the invisible work of oversight. On agents, who manages AI agents, letting an agent act on your behalf and AI agents and human judgement. On the underlying bias, automation bias and automation complacency. On deliberate friction, which decisions should become slower.

About this research#

Rahim Hirji is the author of SuperSkills (Kogan Page, 2026), keynote speaker on AI and human capability, and founder of The SuperSkills Intelligence Company. Every source above was read at the document itself: the Article 14 text at the European Commission's own service desk, the NTSB report at ntsb.gov, the regulation at eCFR and the rulemaking comment in the Federal Register. The Parasuraman and Riley definition was read in the published abstract at SAGE and no claim on this page rests on the body of that paper, which is paywalled. Two Deepwater Horizon survey figures that circulate in this area were searched for at the Chemical Safety Board's own report and are not there, so they do not appear here.

How this research works  ·  Reviewed quarterly  ·  Found an error? Tell me and it is corrected on the page.