When should an organisation reverse an AI deployment?

A deployment is a decision, and decisions can be withdrawn. In practice almost nothing is built that way: approval is treated as a gate that opens once, the fallback is documented rather than maintained, and the question of what would cause the system to be switched off is asked for the first time on the day it needs to be.

Resilience engineering answered most of this before the technology existed. What follows is that literature, and the boundary where it stops.

The five conditions#

NIST's framework requires mechanisms and assigned responsibilities to supersede, disengage or deactivate systems performing inconsistently with intended use, and names when that may be necessary. Graded entry.

A system reaching the end of its lifetime. Detected risks exceeding tolerance thresholds. Mitigation beyond the organisation's capacity. Feasible mitigations failing regulatory, legal or normative standards. And impending risk detected during monitoring for which timely mitigation cannot be implemented.

The structural point sits underneath the list. These thresholds belong to continual monitoring rather than to a one-off approval, which means the numbers have to be set while the system is working and everyone is pleased with it. Set afterwards, they are set by whoever is defending the decision.

This is voluntary guidance rather than a standard with conformity assessment, and it contains no evidence that any of it improves outcomes.

Why the fallback is the hard part#

Switching a system off returns the task. It does not return the capability, because the capability decayed while the system was running, and for cognitive work the decay is quick. The measured rates are here.

This is Bainbridge's irony in its organisational form. Ironies of Automation (1983). The people expected to take over are the ones whose practice was removed by the thing they are taking over from.

So a fallback that exists as a document is not a fallback. What makes it real is that somebody has recently done it.

How much redundancy#

Enough independent capability to detect failure, maintain critical operations within acceptable limits, and recover when the system is unavailable. The amount is a function of consequence, recoverability and acceptable downtime, which is business continuity reasoning rather than anything about AI.

There is no universal percentage. Any figure presented as one should be treated as invented until its derivation is shown, and figures of this kind circulate freely.

Perrow adds the warning that matters most here, because the instinctive fix is the wrong one. In systems that are interactively complex and tightly coupled, serious failures are a structural property, and adding warnings and safeguards increases complexity and can make the system less safe. Normal Accidents (1984). Answering an automation risk by adding a second automated check is the move his book is about.

Weick and Sutcliffe come at it from the other direction and are worth holding alongside rather than instead: high-reliability organisations do sustain performance under uncertainty, through preoccupation with failure, reluctance to simplify, sensitivity to operations, commitment to resilience and deference to expertise rather than rank. Managing the Unexpected. The two traditions genuinely disagree about whether such systems can be managed safely, and this page does not resolve that.

Practising without the system#

The frequency question has a principled answer and no number. Practice has to be often enough that the capability survives, which is set by the decay rate rather than by the calendar.

Aviation is the only industry that has converted this into a scheduled requirement at fixed intervals, and it derived those intervals from its own accident history. What professions can learn from aviation. That is a sound basis for aviation and it is not evidence about law firms or hospitals, and it gets borrowed as though it were.

What transfers is the principle rather than the number: the practice is scheduled, it is on the manual task, and it happens before it is needed.

How quickly an organisation recovers is unmeasured#

No organisation-level measurement exists of how quickly unaided capability returns after a system fails. Relearning beats first learning for individuals at every interval tested, which is encouraging, and it has never been tested on professional judgement. The individual evidence is here. Recovering a routine that several people ran together is a harder problem again, because the coordination decayed alongside the skills.

The NIST framework is guidance, not outcome evidence. Perrow and Weick are analytical traditions built from accident cases, and they contradict each other on the central question. None of this constitutes a demonstration that reversible deployment produces better results than the alternative.

And there is a cost the other way that this page should not pretend away. Maintaining a fallback, practising without the system and holding redundant capability are all expensive, and an organisation that does all three for every system will be slower and poorer than one that does not. The judgement about where it is warranted is not one the literature makes for anybody.

29 and 30 September 2026: a pilot with one alert, and a plan put on hold#

Two reports in one week show the decision this page is about being taken in opposite directions. Carly Page at The Register reported on 30 September figures obtained by Liberty Investigates under freedom of information and first reported by the Guardian: British Transport Police’s six-month trial of live facial recognition “scanned more than 500,000 faces at London railway stations and generated just one alert. That turned out to be a false positive”. The deployments took “almost 100 hours of police officers’ time and cost more than £320,000.” The force’s response: “the trial has been extended until November and expanded from Network Rail stations onto the London Underground.” A Times of India summary of the Guardian’s report, read at inkl, adds that the force says the first phase taught it how to use the technology in a railway setting and that three confirmed alerts have followed since the extension began. Jasleen Chaggar of Big Brother Watch called the results “meagre”. Neither report says that a threshold for success or failure was set before the trial began, and this page turns on that point: a threshold set afterwards is set by whoever is defending the decision. A pilot that returns one false alert in half a million scans may still deserve extending, if the watchlist was small by design or the conditions were unrepresentative. What the public record lacks is the number, agreed in advance, that would have ended it. The second report runs the other way. Joe Rossignol at MacRumors, relaying Bloomberg, wrote on 29 September that Apple “recently considered laying off about 5,000 support employees, but it has put those plans on hold indefinitely”, having believed that AI agents could take over some of the work. No reason is given and the report carries no comment from the company. If it is accurate, this is the cheapest reversal there is, the one made before the capability has gone, and it leaves the people who could take the work back still employed to do so. The two cases make the same demand of any organisation: write down what would stop the system, and who decides, while everyone is still pleased with it.

Key sources

• National Institute of Standards and Technology (2023). AI Risk Management Framework Playbook, MANAGE 2.4. Graded entry.
• Perrow, C. (1984). Normal Accidents: Living with High-Risk Technologies. Basic Books. In the essential works.
• Weick, K. E. and Sutcliffe, K. M. (2001). Managing the Unexpected. Jossey-Bass. In the essential works.
• Bainbridge, L. (1983). Ironies of Automation. Automatica, 19(6), 775-779. In the essential works.

Related SuperSkills research#

On what the oversight role actually costs, the invisible work of oversight. On testing what remains, the capability audit. On what is being preserved, organisational capability. On buying rather than building it, preserving capability across vendors. On the industry that schedules the practice, what professions can learn from aviation.

About this research#

Rahim Hirji is the author of SuperSkills (Kogan Page, 2026), keynote speaker on AI and human capability, and founder of The SuperSkills Intelligence Company. Normal accidents are Perrow's, high-reliability organising is Weick and Sutcliffe's, and the five conditions are NIST's. None is a SuperSkills coinage.

How this research works  ·  Reviewed quarterly  ·  Found an error? Tell me and it is corrected on the page.