When should an organisation reverse an AI deployment?

A deployment is a decision, and decisions can be withdrawn. In practice almost nothing is built that way: approval is treated as a gate that opens once, the fallback is documented rather than maintained, and the question of what would cause the system to be switched off is asked for the first time on the day it needs to be.

Resilience engineering answered most of this before the technology existed. What follows is that literature, and the boundary where it stops.

The five conditions

NIST's framework requires mechanisms and assigned responsibilities to supersede, disengage or deactivate systems performing inconsistently with intended use, and names when that may be necessary. Graded entry.

A system reaching the end of its lifetime. Detected risks exceeding tolerance thresholds. Mitigation beyond the organisation's capacity. Feasible mitigations failing regulatory, legal or normative standards. And impending risk detected during monitoring for which timely mitigation cannot be implemented.

The structural point sits underneath the list. These thresholds belong to continual monitoring rather than to a one-off approval, which means the numbers have to be set while the system is working and everyone is pleased with it. Set afterwards, they are set by whoever is defending the decision.

This is voluntary guidance rather than a standard with conformity assessment, and it contains no evidence that any of it improves outcomes.

Why the fallback is the hard part

Switching a system off returns the task. It does not return the capability, because the capability decayed while the system was running, and for cognitive work the decay is quick. The measured rates are here.

This is Bainbridge's irony in its organisational form. Ironies of Automation (1983). The people expected to take over are the ones whose practice was removed by the thing they are taking over from.

So a fallback that exists as a document is not a fallback. What makes it real is that somebody has recently done it.

How much redundancy

Enough independent capability to detect failure, maintain critical operations within acceptable limits, and recover when the system is unavailable. The amount is a function of consequence, recoverability and acceptable downtime, which is business continuity reasoning rather than anything about AI.

There is no universal percentage. Any figure presented as one should be treated as invented until its derivation is shown, and figures of this kind circulate freely.

Perrow adds the warning that matters most here, because the instinctive fix is the wrong one. In systems that are interactively complex and tightly coupled, serious failures are a structural property, and adding warnings and safeguards increases complexity and can make the system less safe. Normal Accidents (1984). Answering an automation risk by adding a second automated check is the move his book is about.

Weick and Sutcliffe come at it from the other direction and are worth holding alongside rather than instead: high-reliability organisations do sustain performance under uncertainty, through preoccupation with failure, reluctance to simplify, sensitivity to operations, commitment to resilience and deference to expertise rather than rank. Managing the Unexpected. The two traditions genuinely disagree about whether such systems can be managed safely, and this page does not resolve that.

Practising without the system

The frequency question has a principled answer and no number. Practice has to be often enough that the capability survives, which is set by the decay rate rather than by the calendar.

Aviation is the only industry that has converted this into a scheduled requirement at fixed intervals, and it derived those intervals from its own accident history. What professions can learn from aviation. That is a sound basis for aviation and it is not evidence about law firms or hospitals, and it gets borrowed as though it were.

What transfers is the principle rather than the number: the practice is scheduled, it is on the manual task, and it happens before it is needed.

What this page does not establish

No organisation-level measurement exists of how quickly unaided capability returns after a system fails. Relearning beats first learning for individuals at every interval tested, which is encouraging, and it has never been tested on professional judgement. The individual evidence is here. Recovering a routine that several people ran together is a harder problem again, because the coordination decayed alongside the skills.

The NIST framework is guidance, not outcome evidence. Perrow and Weick are analytical traditions built from accident cases, and they contradict each other on the central question. None of this constitutes a demonstration that reversible deployment produces better results than the alternative.

And there is a cost the other way that this page should not pretend away. Maintaining a fallback, practising without the system and holding redundant capability are all expensive, and an organisation that does all three for every system will be slower and poorer than one that does not. The judgement about where it is warranted is not one the literature makes for anybody.

Key sources

• National Institute of Standards and Technology (2023). AI Risk Management Framework Playbook, MANAGE 2.4. Graded entry.
• Perrow, C. (1984). Normal Accidents: Living with High-Risk Technologies. Basic Books. In the essential works.
• Weick, K. E. and Sutcliffe, K. M. (2001). Managing the Unexpected. Jossey-Bass. In the essential works.
• Bainbridge, L. (1983). Ironies of Automation. Automatica, 19(6), 775-779. In the essential works.

Related SuperSkills research

On what the oversight role actually costs, the invisible work of oversight. On testing what remains, the capability audit. On what is being preserved, organisational capability. On buying rather than building it, preserving capability across vendors. On the industry that schedules the practice, what professions can learn from aviation.

About this research

Rahim Hirji is the author of SuperSkills (Kogan Page, 2026), keynote speaker on AI and human capability, and founder of The SuperSkills Intelligence Company. Normal accidents are Perrow's, high-reliability organising is Weick and Sutcliffe's, and the five conditions are NIST's. None is a SuperSkills coinage.

How this research works  ·  Reviewed quarterly  ·  Found an error? Tell me and it is corrected on the page.