Will AI replace programmers?

Nothing measured so far shows replacement, and the productivity evidence points somewhere more interesting. The two most cited coding experiments disagree by seventy-five percentage points, and the disagreement is not a contradiction. They measured different work, and the line between them is the line between writing code and understanding a system.

Seventy-five points apart, and both are sound#

Peng and colleagues randomised 95 professional programmers, recruited through Upwork, to implement an HTTP server in JavaScript with or without GitHub Copilot. Among those who finished, the assisted group averaged 71 minutes against 161, a 55.8 per cent reduction with a confidence interval running from 21 to 89 per cent. The authors state they did not examine code quality, the task is greenfield rather than work inside a mature codebase, and they are employed by the vendor. Graded entry.

METR randomised experienced open-source developers on real tasks in their own mature repositories and measured them 19 per cent slower with AI tools permitted. Graded entry.

Put the designs side by side and the numbers stop fighting. One is a fresh artefact built to a specification by people who have never seen the codebase, because there is no codebase. The other is a change to a system the developer has lived inside for years, where the expensive knowledge is already in their head and the model has none of it. Production got faster. Comprehension did not, and comprehension was the bottleneck.

That split is the useful thing to carry into any decision about software teams, and it sharpens the boundary Dell'Acqua's consultants ran into. Graded entry.

METR has since unsettled its own figure#

In February 2026 METR reported a second study, 57 developers across 143 repositories and more than 800 tasks, and said the data now gives an unreliable signal. Between 30 and 50 per cent of developers declined to submit tasks they did not want to do without AI. The raw results reverse, a speedup of -18 per cent for returning developers and -4 per cent for new recruits, and every confidence interval crosses zero. They think developers are probably more sped up in 2026 than in 2025, and say their own data is only very weak evidence for it. Graded entry.

So the 19 per cent describes early-2025 tooling under a design its authors have replaced. It is still the cleanest measurement anybody has of experienced developers in real repositories, and no longer a statement about today. Both halves of that need saying together.

The blackout task, and what it exposed#

The most pointed result in software is not about speed. Sankaranarayanan ran 78 novice programmers through a custom IDE in three conditions: manual, unrestricted AI, and AI scaffolded to withhold direct answers. Both AI groups beat the manual control on functional utility and did not differ from each other.

Then the AI was taken away and they were given a maintenance task on what they had built. Unrestricted AI users failed at 77 per cent. The scaffolded group failed at 39 per cent. The author's term for the first group is fragile experts: high functional output masking low corrective competence. Graded entry.

The limits are real and the author states them. One session, one blackout task, novice programming, no longitudinal follow-up. What makes it worth carrying is that the two AI groups were indistinguishable on the thing a manager would have measured, and separated by thirty-eight points on the thing that decides whether the code can be maintained next year.

Three kinds of debt, and AI moves them in different directions#

Storey proposes that a codebase now carries three debts rather than one. Technical debt lives in the code. Cognitive debt lives in the people, as the erosion of shared understanding across a team. Intent debt lives in the artefacts, as the absence of captured rationale, goals and constraints. The argument is that generative AI may reduce the first while accelerating the other two, because code can be produced faster than a team can build the understanding needed to change it safely. Graded entry.

This is a framework paper with an illustrative anecdote, and it offers nothing empirical about prevalence or magnitude. It earns its place here because it names the mechanism the two measured results above are both touching, and because it is the same argument this estate makes about organisations generally at capability debt.

What the labour-market data says, which is less than either side claims#

Employment of 22 to 25 year olds in AI-exposed occupations sits about 19 per cent below where it would have been had it tracked their less-exposed peers, and the divergence runs through reduced hiring rather than separations. Software is among the exposed occupations. It is also observational, it is not causal, and the authors describe their own findings as canaries rather than causal estimates. The figure and its four common misreadings are set out at the AI employment gap.

Nothing in it shows programmers being replaced. It shows fewer doors opening for people who have not yet become programmers, which is a different problem with different remedies.

What is not established#

Code quality, almost everywhere. Peng declined to measure it. METR measured time, not defect rates. No study on this estate follows AI-assisted code into production and counts what broke.

Anything about senior developers over time. Sankaranarayanan's participants were novices and the blackout was thirty minutes. Whether an experienced engineer loses corrective competence over years of assisted work is unmeasured, and that is the question a business actually needs answered.

And the direction of travel. METR believe the picture is improving and decline to put a number on it. Anyone confident about where developer productivity sits in 2027 is ahead of everybody who has measured it.

If you run a software team#

Expect the gains where work is new and specified, and do not assume them where the work is changing something load-bearing. Measure defects and time to change, not lines produced or tokens consumed, because the two results above separate at the point where those diverge. And treat the blackout result as a design instruction rather than a warning: the scaffolded group did the same work and kept the competence, so the difference was in how the tool was configured and not in whether it was used.

Key sources

• Peng, S., Kalliamvakou, E., Cihon, P. and Demirer, M. (2023). The Impact of AI on Developer Productivity: Evidence from GitHub Copilot. arXiv:2302.06590. Graded entry.
• METR (2025). Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. Graded entry.
• Becker, J., Rush, N., Cunningham, T., Rein, D. and Mahamud, K. (2026). We are Changing our Developer Productivity Experiment Design. METR, 24 February 2026. Graded entry.
• Sankaranarayanan, S. (2026). Mitigating "Epistemic Debt" in Generative AI-Scaffolded Novice Programming. arXiv:2602.20206. Graded entry.
• Storey, M.-A. (2026). From Technical Debt to Cognitive and Intent Debt. ACM Queue, preprint arXiv:2603.22106. Graded entry.

Related SuperSkills research#

The productivity question across all work is at does AI actually make people more productive, and the METR result in full at what is the METR study. On the boundary between the two coding results, the jagged frontier. On what the blackout task is measuring, capability debt and desirable difficulty. On the hiring side, the AI employment gap and the missing rungs.

About this research#

Written by Rahim Hirji, author of SuperSkills (Kogan Page, 2026), keynote speaker on AI and human capability, and founder of The SuperSkills Intelligence Company. "Fragile experts" is Sankaranarayanan's phrase and the triple debt model is Storey's; neither is claimed here. Every figure is kept with the study design that produced it.

How this research works  ·  Reviewed quarterly  ·  Found an error? Tell me and it is corrected on the page.