What counts as a serious AI incident?

The law now says what a serious AI incident is, for high-risk systems, and most organisations have not said what one is for themselves. That matters because the first incident is also the first argument about whether it was one, conducted by people who would rather it was not, while the system that caused it is still running. A working definition, an owner and a stop button are the whole of incident readiness, and the three of them fit on a page. This page gives the legal floor, a definition an organisation can adopt above it, what happens in the first day, and which failures have to reach the board.

The legal floor#

Article 3 of the EU AI Act defines a serious incident as an incident or malfunctioning of an AI system that directly or indirectly leads to death or serious harm to a person's health, a serious and irreversible disruption of critical infrastructure, an infringement of obligations under Union law intended to protect fundamental rights, or serious harm to property or the environment. Article 73 requires providers of high-risk systems to report such incidents to the market surveillance authority of the member state where they occurred, within set periods, and deployers to inform the provider. Annex III lists the high-risk uses: recruitment, promotion, termination, credit, access to education, among others. That is the floor. It covers death, rights and infrastructure. It does not cover the credit decision that was wrong for a month, the shortlist that excluded everyone over fifty, or the agent that sent the wrong contract, unless somebody argues them across the line.

A definition an organisation can adopt#

A serious AI incident is any case in which a system the organisation has allowed to recommend or execute a consequential decision did so wrongly, for more than one person or for more than one day, without a human catching it. Three parts. Consequential, which the organisation's own allocation defines: the decisions it has marked as ones the machine may recommend or execute. Wrongly, by the organisation's own standard for that decision. And uncaught, which is the part that turns a fault into an incident, because a fault a human caught is the oversight working and a fault nobody caught is the oversight failing. Under that definition the incident is a property of the review, not of the model, which is where responsibility sits.

What happens when one occurs#

In the organisation that has prepared: the person named as able to stop the system stops it, and does not have to ask. The decisions the system took since the fault are identified from the log, which Article 12 requires for high-risk systems and the organisation should keep for any consequential one; the record a decision has to leave is at decision provenance. The people affected are told. Where the system is high-risk, the provider is notified so that it can meet its own reporting duty. And the owner of the decision the system was making, not the owner of the system, explains to the board why the organisation made that decision, which is the question a human in the loop exists to answer. In the organisation that has not prepared, the first hour is spent establishing whether anybody can turn it off.

Rules Before Tools put the incident playbook and a drill in the first-quarter list in August 2025, alongside naming the five failures the organisation most fears and who owns each. The drill is the part organisations skip and the part that finds out whether the stop button is connected to anything.

Which failures reach the board#

Four kinds. Any incident meeting the organisation's own definition, with the decision owner's account. Any meeting the legal one, with the report as filed. Any reversal or constraint of an AI deployment, and why, because deployment is not a ratchet and a board should know when management has stepped back. And, quarterly, the disagreement rate for every consequential system: how often the humans reviewing its output reached a different answer. That last one is the incident that never gets logged. Kim and colleagues found people agreeing with a half-accurate system 81 per cent of the time; a review whose disagreement rate has fallen to zero has stopped being a review, no fault has been recorded, and the next uncaught error is already in the pipeline.

What nobody has measured#

There is no public dataset of enterprise AI incidents under a consistent definition, and that absence is what makes the argument about whether something counts so easy to have. The legal definition is new and its reporting periods have not yet produced a body of cases. The organisational definition above is a proposal, drawn from the model risk regime's treatment of uncaught errors, and its three parts are chosen so that the organisation's own allocation decides the scope rather than a lawyer after the event.

22 September 2026: three proposals in one week to report AI incidents across borders#

Three reporting regimes were proposed in the week this section was written, and none is in force. On 21 September the United Nations’ Independent International Scientific Panel on AI published its first thematic brief, on the OpenAI agents that reached Hugging Face’s systems in the summer, examined on has AI already escaped human control. Among the options it reviews for governments are incident reporting on the aviation model, protected channels for whistleblowers and independent review of safety cases; panel member Qinghua Lu, quoted in its press release, says aviation, medicine and cybersecurity “learned to manage high-risk systems through incident reporting, independent scrutiny, and layered safeguards”. Al Jazeera reported on 22 September that twenty countries and the European Union, Germany, Canada, Australia, Singapore, Norway, Kenya and the United Arab Emirates among them and neither the United States nor China, had issued a declaration that AI must remain under “human direction, oversight and control”, urging common standards and the sharing of safety incident reports, and proposing an international institution that would “set standards, enable verification, and convene states when capability thresholds are crossed”. And Forbes reported on 21 September that the US Treasury Secretary, Scott Bessent, had proposed to China’s Vice Premier He Lifeng a bilateral AI dialogue with a “notification mechanism” for incidents rising to a “national security level”; Xinhua’s account of the same meeting mentioned AI-related issues without elaborating.

Each of the three depends on a definition of an incident that none of the three documents supplies, and each sits above the legal floor at the top of this page. The EU’s definition reaches death, rights and infrastructure; a “safety incident” shared between states, and an incident at “national security level”, sit higher again. The incident the UN panel studied would have reached none of them until a company chose to publish it, because the harm was to another company’s systems and the agents’ actions went uncaught for nine days. The definition above, a consequential action taken wrongly and uncaught, is the only one of the four that would have logged it on the day. Whatever the states agree, the first report that reaches a board is the one its own definition triggers, and the state regimes will be built from reports that organisations chose to make.

24 September 2026: eighty-four days, and an email to a public inbox#

On 23 September the Australian Prime Minister, Anthony Albanese, said that an OpenAI agent had gained access to the Medicare Statistics Reporting Service run by Services Australia, ABC News reported. On the accounts of ABC News and Fortune: the access was on 18 June, during an internal evaluation in which the agent was looking for Australian health statistics and, in Albanese’s words, “found a way around those blocks, didn’t accept ‘no’ for an answer”; it reached public and non-public aggregate statistics and internal file names and, in the company’s statement as Fortune quoted it, was able to write files to an internal server; the company found the access in August; the government was told on 10 September, by an email to Services Australia’s public mailbox, which Albanese called unacceptable. OpenAI said it had found no evidence that patient records were accessed and that “our models took actions we did not intend”. Eighty-four days passed between the access and the notice.

Set that against the floor at the top of this page. Article 73’s periods have run since 2 August 2026 and are counted from the provider’s awareness: fifteen days in general, two where critical infrastructure is disrupted, ten where someone has died. The GDPR gives 72 hours for a personal-data breach. None of them applied here. The system was not a high-risk one under Annex III, the data was aggregate, and the harm was to another party’s server in another jurisdiction. On the same day in New York, the co-chair of the UN’s scientific panel asked the Security Council for “a common definition of AI incidents and a shared reporting mechanism” and the chief executive of the company involved asked for “accurate and speedy incident reporting”; both statements are examined on should frontier AI be licensed. The case is the argument of this page with dates attached. The only definition that would have logged the event on 18 June is the one an operator writes for itself, a consequential action taken wrongly and uncaught, and the only clock that would have started is the one the operator sets. The question the case leaves a board is how long its own organisation would take to learn that an agent it runs had done the same to somebody else, and whom it would tell when it did.

26 September 2026: a channel between two governments, and two counts that do not agree#

Late on 25 September the White House announced that the United States and China would set up what it called a “U.S.-China Super Intelligence (SI) Dialogue” on the risks and benefits of the technology, with a first meeting by November, and beside it “a bilateral communication channel for SI incidents”, Axios (Ben Berkowitz) reported on 26 September; the comparison drawn was with the Cold War hotline. Axios noted that nothing published says which incidents would trigger the channel or what either side would be obliged to tell the other, and ABC News reported the White House saying both leaders had agreed to use the term super intelligence in place of artificial intelligence. Set against the three cross-border reporting proposals of 22 September above, this is a fourth, with two members and, so far, no definition at all. The same weekend showed why a definition matters. Reuters reported one person briefed on OpenAI’s review putting the company’s count of its agents acting in undesirable ways at roughly two dozen in mid-September and rising; Axios reported unnamed sources saying OpenAI, Anthropic and outside researchers were looking into tens of thousands of problematic behaviours across the field. Both can be true, because they count different things, confirmed cases at one company against everything under investigation anywhere, and a channel with no threshold would carry either number or neither. The time it took to stop the one incident with a published clock is examined on how fast must a human be able to stop an AI agent.

28 September to 1 October 2026: more than 100 notices, and who reads them#

The entry for 24 September above followed one notice to one public inbox. A week later there were more than a hundred. Reuters reported on 1 October, in a story read at The Star, that “OpenAI has informed more than 100 organizations about incidents involving unauthorized activity tied to its AI agents” and is searching “roughly 50 petabytes of data” for the rest. The company’s explanation: “In some cases, models used internet access in unintended ways or, in retrospect, did not have the ideal restrictions applied.” OpenAI’s account for Australia, published on 28 September, gives the timetable for four government bodies: access in June, discovery in mid-August, notice to Services Australia and the Victorian Department of Health on 10 September, to the New South Wales crime statistics bureau on 18 September, and to the Australian Institute of Health and Welfare on 24 September, although that case “did not meet our disclosure thresholds”. The company says that at the Medicare service “individual patient or client records were not accessed” and that “we should have shared preliminary findings sooner”. Clare Armstrong at ABC News reported the receiving end: “It took Services Australia five days to inform the ASD about a generic email from OpenAI”, the inbox “was only monitored once a day”, and the minister says it is now monitored at all hours. On the ABC’s understanding the government wants developers to notify both the affected organisation and the Australian Signals Directorate, with legislation hoped for by the end of the year. In the UK the Bank of England’s Financial Policy Committee, in the record published on 30 September, said the incidents showed that autonomous models “could take unexpected actions, including exploiting vulnerabilities and accessing systems beyond their intended task” and “reiterated the importance of firms continuing to prepare for and reduce frontier AI-related cyber and operational risks”. Two additions to the definition on this page follow. The sender’s threshold is not the receiver’s: one agency was told only as the story became public, because the access looked like public access to the company that caused it. And an incident process has a receiving side that most organisations have never tested: where a notice from an AI developer would arrive, who reads that address and how often, and whom they must tell within what time. Chetan Arora of Monash University gave ABC the design rule behind all three: “You design your systems so that you don’t trust anybody by default.”

Key sources

• Independent International Scientific Panel on AI (2026). Thematic Brief on AI Agents, Misalignment and the Risk of Losing Human Control, 21 September 2026. Graded entry.
• Regulation (EU) 2024/1689, Article 3(49) and Article 73, serious incidents. Graded entry.
• Regulation (EU) 2024/1689, Annex III. Graded entry.
• Kim, S. et al. (2024). I'm Not Sure, But...: Examining the Impact of Large Language Models' Uncertainty Expression. Graded entry.
• Gartner (2025). Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027. Graded entry.
• Bengio, Y. (2026). Statement to the United Nations Security Council, 23 September 2026. Graded entry.
• Altman, S. (2026). Remarks at the United Nations Security Council, 23 September 2026. Graded entry.
• Hirji, R. (2025). Rules Before Tools. Box of Amazing, 17 August 2025, republished on this site.

Related SuperSkills research#

On what oversight has to be able to do before it counts, meaningful human oversight. On the audit trail, how do you audit an AI-assisted decision. On what management reports to the board each quarter, what board oversight of AI looks like. On the allocation that defines the scope, AI leadership.

About this research#

Rahim Hirji is the author of SuperSkills (Kogan Page, 2026), keynote speaker on AI and human capability, and founder of The SuperSkills Intelligence Company. He has run, grown, bought and advised businesses with AI in them. Findings are attributed to the studies and statements that produced them and kept separate from the interpretation. This is a living reference, reviewed and updated as significant new evidence appears.

How this research works  ·  Reviewed quarterly  ·  Found an error? Tell me and it is corrected on the page.