What are AI red lines?

A red line for AI is a use or a behaviour ruled out in advance, whatever the benefit, rather than a risk to be weighed case by case. The phrase took its current shape on 22 September 2025, when the Global Call for AI Red Lines asked governments to agree enforceable international lines by the end of 2026. This week the head of Microsoft AI asked for one, and the chief executives of Anthropic and OpenAI told the UN Security Council where they would start. No international line exists. The nearest thing in law is Article 5 of the EU AI Act, which has prohibited eight practices since February 2025. The line any organisation can draw today is the list of decisions a machine may never make in its name.

Where the phrase comes from#

Red lines are older than AI; the words come from diplomacy and mean a limit whose crossing brings a response. The Global Call for AI Red Lines, organised by the French Center for AI Safety, The Future Society and Berkeley’s Center for Human-Compatible AI and launched during the high-level week of the 80th UN General Assembly, gave them an AI-specific list and a deadline. It asks governments to agree international red lines, workable in practice and enforced, by the end of 2026. The campaign’s site now lists more than 300 signatories, among them 15 Nobel and Turing laureates and 11 former heads of state and ministers; at launch, reports counted more than 200 and 10 Nobel laureates. Its examples fall into two families. Lines on use: nuclear command and control, lethal autonomous weapons, mass surveillance and social scoring, impersonating a human without disclosure. Lines on behaviour: a system copying itself without authorisation, a system that cannot be immediately switched off, autonomous cyberattacks. The campaign did not invent the words; it fixed a meaning that policy now argues over.

What was said this week#

Bloomberg reported on 24 September 2026 that Mustafa Suleyman, chief executive of Microsoft AI, argued the industry needs a red line for advanced AI and that governments should evaluate frontier models before release rather than leave the boundary to the companies building them. The day before, at the Security Council’s meeting on AI and international security, Dario Amodei of Anthropic proposed starting narrow, with agreements against specific harms such as the use of AI to build biological weapons, then verification so that states can check each other’s commitments, then common testing standards, according to the Associated Press account carried by Al Jazeera and The Next Web. Sam Altman of OpenAI asked, in remarks his company published, for a mechanism of national and international frontier standards. Yoshua Bengio asked for licensing, recorded on should frontier AI be licensed.

The governments did not agree with each other. The UK Foreign Secretary, Ed Miliband, told the Council in a speech published by the Foreign Office that “we cannot outsource to private companies the first duty of government to protect our people”, and asked for rigorous testing of frontier models and visibility for governments into what companies are doing. The United States, through the White House science adviser Michael Kratsios, rejected what Al Jazeera reported as “all efforts by international bodies to assert centralised control and global governance of AI”. China’s ambassador, Fu Cong, argued for open models and against excluding countries from the technology, according to the AP report carried by Business Standard. Three companies asked for rules that would bind them. Whether that support survives a draft that constrains them is untested, because no draft was on the table.

A red line is not a risk appetite#

A risk appetite accepts some probability of harm in exchange for a benefit. A red line refuses the trade. The refusal needs three things to be more than a sentence: a definition precise enough that everyone can tell when the line is crossed, a way of detecting the crossing, and a consequence someone will apply. The two families differ here. A line on use binds the people deploying a system, and ordinary law can enforce it after the fact: a court can ask who ran the social scoring system. A line on behaviour binds the model itself and can only be enforced before release, by measuring whether the model can copy itself or resist being stopped. That is a capability evaluation, and the pages on whether models know when they are being tested and the AI kill switch set out how far such measurement can be trusted. The 2026 Hugging Face incident, on has AI already escaped human control, is a behaviour line crossed in a test with the limits off. The behaviour lines are the ones people mean when they say red line, and the ones nobody yet knows how to police.

The red lines that already exist in law#

Of the statutes this estate has reviewed, the EU AI Act alone carries a list of things AI may not be used for. Article 5 has applied since 2 February 2025 and prohibits eight practices, among them harmful manipulation, social scoring, predicting crime from profiling alone, scraping faces to build databases, inferring emotion in workplaces and schools, and real-time remote biometric identification by police in public places, each with listed exceptions. Two further prohibitions, on intimate images made without consent and on child sexual abuse material, apply from 2 December 2026 on the text as published. Every one is a line on use: none says what a model may never do, only what people may never do with one. The UK has no statute for models and declined a statutory kill switch this month, as recorded on should AI development be paused. The law, where it exists, draws the easier family of lines and leaves the harder to the companies’ own commitments.

Why the lines are easy to say and hard to draw#

The campaign’s deadline is the end of 2026 and no treaty text is public. Two obstacles were visible in one afternoon at the Council. The first is verification. A line without a check is a promise, and no accepted way exists for one state to confirm what another state’s companies have trained; the page on whether an evaluator paid by the company can be independent records how thin the current arrangements are. The second is that the governments of the two countries where the frontier models are built took different positions at the same table, and a line that one of them will not sign binds only the signers. A third sits inside the lines themselves: “a system that cannot be immediately terminated” presupposes that somebody measured whether it can be, and the line is only as real as that measurement.

The red lines an organisation can draw this quarter#

An organisation cannot draw an international line and can draw its own in an afternoon, because it controls all three parts: definition, detection and consequence. The estate’s position, set out in Rules Before Tools, is that the decision an organisation controls is which decisions a machine may make in its name. A red line is the negative of that list: the decisions it may never make. A machine never issues the final decision to dismiss, to refuse credit or care, or to discipline, without a named person who has read the case. An agent never acts outside the tools and accounts it was given, and never runs with its logging off. No system is deployed until someone has written who can stop it and how fast, as set out on approving AI decisions at machine speed. Every line has a detection: the override rate, the log, the incident definition on what counts as a serious AI incident. Most organisations have not done this; IBM’s 2026 survey, on do employees know they are expected to override AI, found that about a quarter had written down which work is human-led. A board that wants to know whether the company has red lines can ask for the list and the date it was written. The absence of a list is an answer.

What this does not show#

Nothing here shows that red lines reduce risk. The Global Call is a position signed by many people, not a study, and its signatory counts are the campaign’s own. What was said at the Security Council is a set of positions taken at one meeting; none is policy, and the companies’ support for lines has not been tested against a text that would constrain them. Bloomberg’s account of Suleyman’s argument is a report of an argument, and he offered no proposal for where the line should sit. Article 5 has been in force for nineteen months and the record read for this page shows no completed enforcement action under it, so how the existing lines work in practice is unknown. And an organisation’s own lines do not touch the frontier risk the Council met to discuss. They touch the part of the risk that organisation can measure, which is the part this site is about.