What is the illusion of competence?

The illusion of competence is the gap between how capable people feel and how capable they are. Fluent material feels learned, a legible explanation feels understood, and a good output feels like proof of a good performer. The gap is invisible from the inside, and that invisibility makes it the central measurement problem in this research: nobody can self-report their way out of it, so it has to be tested.

Confidence and durability point in opposite directions#

Bjork and colleagues set out the finding that organises the rest: the study conditions producing the highest confidence produce the least durable learning, and the conditions that feel like they are going badly produce the most. Graded entry. Learners asked to choose their own method reliably choose the one that feels best, which is reliably the weaker one.

Fisher and colleagues then showed the effect can be induced by a tool. Searching the internet inflated people's estimates of their own unaided knowledge, and it did so even on questions the search had never touched. Graded entry. Access to an external source was being experienced as personal knowledge. That study used a search engine in 2015, on a task where the person still had to read, choose and interpret.

The gap that only appears when the tool is taken away#

Sankaranarayanan gave 78 participants a programming task in three conditions: manual, unrestricted AI, and a scaffolded version built to make the user do part of the thinking. Both AI groups beat the manual control on the work itself and did not differ from each other. Then the AI was removed and participants had to maintain what they had built. The unrestricted group failed at 77 per cent, against 39 per cent for the scaffolded group. Graded entry.

Two groups, indistinguishable while the tool was present, separated by a factor of two the moment it was not. Bastani's school trial has the same shape: grades up 48 per cent with unrestricted access, then 17 per cent below a control group that never had it, once it was withdrawn. Graded entry.

Huemmer's three-wave study puts the metacognitive layer on top, tracking how verification effort declines as confidence in the tool grows. It belongs here for its shape and not for its weight: it is a pilot of twenty-three people recruited at one institution, with no control group, and its authors say in their companion paper that they performed no inferential tests at all. Graded entry.

Why self-report is the wrong instrument#

Almost every published survey of AI's effect on skills asks people whether they think their skills have degraded. EY's 2025 survey found 37 per cent worried about it, rising to 43 per cent in the UK. Graded entry. BCG asked 70 executives whether they were observing deskilling and half said yes. Graded entry.

Those are useful figures about worry and about the executive agenda. They are close to worthless as measurements of capability, because the illusion of competence is the claim that people cannot see this in themselves. Asking someone whether AI has degraded their judgement asks them to report on the one thing the effect obscures. The Polish endoscopists whose unassisted detection rate fell six percentage points were not reporting a decline. Somebody measured it. Graded entry.

A fourth frame, done properly, and it still cannot rescue self-report#

A report published in April 2026 shows how far this goes, and deserves working through at length, because it arrives carrying this page's own idea. The Clearing, a commercial site for engineers who feel worn down by AI, surveyed 2,147 of them and found that 63 per cent "report measurable decline in at least one core skill they had before adopting AI tools", that 71 per cent agree with "I often feel like a middleman between AI output and actual results", and that 58 per cent "cannot fully explain the code they shipped last month without AI assistance". It names the pattern the Competence Illusion, and defines it as "when AI tools make you productive without making you more competent. You ship more. You understand less." That is the same phenomenon this page defines, arrived at independently, and the report is open about its method. Graded entry.

The figures still cannot be used here, and the reason is where the respondents came from. The data was collected through the site's own AI Fatigue Quiz, from people recruited via its newsletter and three programming subreddits, and the report says so itself: results "reflect the experiences of engineers who found and chose to participate in the survey". So the sample is people who went looking for a quiz about AI fatigue, and the 63 per cent is then read as how common AI fatigue is. The word measurable is carrying weight it has not earned, too, because nothing in the report measures a skill; all seven skill-area figures are perceptions of decline, the very instrument the section above rules out.

Set that beside Dixon's survey of 1,250 US workers, which found about 3 per cent had lost a job to AI, and the pair teaches the fault better than either does alone. Dixon's frame excluded the outcome he was measuring, because every respondent was in work, so anyone displaced and still unemployed could appear in neither the numerator nor the denominator. The Clearing's frame selects on the outcome. One number is too low by construction and the other too high, and neither error is visible in the number itself. Graded entry.

What I take from the report is the word rather than the percentage. The middleman problem is their phrase, it is better than anything I had for the same thing, and an engineer who has become the reviewer of work they used to author is describing a change of role and not a change of workload. A figure does not become more citable for pointing the way an argument already goes, and this site's commitment to checking sampling frames has to cut towards its own conclusions as readily as against them. The 63 per cent agrees with me. It is still not evidence.

A third case arrived on 3 September 2026 and it is the one worth learning to spot, because the other two can be diagnosed from the document and this one cannot. unitQ, a company that sells an AI research platform, issued a release reporting that in a study of 160 research, design and insights professionals, 42 per cent said AI had reduced or slowed hiring or contributed to roles going unfilled, and 28 per cent said early-career researchers would feel the impact before anyone else. Forty per cent warned of deskilling or of shallow, homogenised output. Graded entry.

Read that against the two above and the difference lands somewhere unfamiliar: there is no frame here at all, good or bad. The release gives no field dates, no recruitment source, no screening criteria, no geography and no margin of error, and its only two outbound links go to the company's home page and to the page that sells the platform. There is no report to open. The study was run on the vendor's own product, which the release says plainly and which is fair enough on its face; what follows from it is that the instrument, the sample and the sales pitch are one object.

Two smaller things compound it. The 40 and 51 per cent figures are prefaced "Among research and insights professionals", a subgroup of the 160 that excludes the designers the 160 includes, and the size of that subgroup is never given, so anybody writing "40 per cent of 160 professionals" has misreported the source while quoting it accurately. And the 42 and 28 per cent carry no stated base at all. The reader is left to assume 160 because 160 is the number in the same sentence.

Then there is the sentence that decides how this one should be treated. The release says of itself: "the method was published alongside the finding". It was not. That is a different kind of problem from an openly bad frame, because a careful reader who checks whether the working has been shown is told that it has. Dixon excluded the outcome from his frame and said so in enough detail that anyone could see it. The Clearing selected on the outcome and printed the sentence that gives it away. unitQ asserts a method and supplies none, and a reader who trusts the assertion has no second line of defence.

A fourth case, published 21 September 2026, has none of the first three faults. IBM's Institute for Business Value, working with Oxford Economics, fielded two paired surveys between April and June 2026: 1,500 CHROs and senior workforce-strategy executives across 21 geographies and 23 industries, and a separate sample of 8,800 full-time employees across 28 countries. The method is stated, the dates are stated, the two populations are kept distinct, and the release does not claim to have measured anything it has not. Graded entry.

The headline figures are exactly the shape this page has been describing throughout. 71 per cent of CHROs rank supervising, validating and overriding AI outputs as a top workforce priority, against 29 per cent of employees who rank it as important. 60 per cent of employees say they worry AI could erode their own skills, most often naming critical thinking. 46 per cent of CHROs rank skills erosion among their organisation's top concerns. And 80 per cent of CHROs believe AI is creating invisible employee work, validation and error-correction that goes uncounted in any workload measure. The release carried the widest wire pickup of the four cases on this page, syndicated by PR Newswire, Nasdaq and Newswire.ca among others, all dated 21 September.

A transparent frame and a large one do not change what kind of figure comes out the other end. Every number above is a stated priority or a stated worry, collected once, from people describing their own organisation. None of it is a before-and-after measurement of anyone's critical thinking, and the 80 per cent figure on invisible work is itself an executive's estimate of a cost nobody has counted, which is a second layer of the same problem sitting on top of the first. Cleaning up a sampling frame removes three ways a survey can mislead. It does not touch the fourth, which is that self-report was never able to see the thing this page is about.

Four cases make a class, and the class is the point of this section rather than any of the numbers in it. A sampling frame can exclude the thing being counted, select on the thing being counted, refuse to describe itself, or, as here, do none of those and still only be able to report what people believe about their own capability. The third is the commonest outside academic publishing and the hardest to catch, because catching it means noticing an absence rather than an error. The fourth is the hardest to dismiss, because there is nothing wrong with the survey. It belongs on this page because a confident number, however well sampled, produces the illusion of competence in whoever reads it if nobody asks what kind of number it is.

What defeats it#

The illusion is a claim about introspection rather than about performance, so any measurement taken without the assistance defeats it: unaided work samples, live decisions made without the tool, and asking someone to explain and defend an output rather than produce one. That is the argument set out in assessing capability rather than output.

The uncomfortable implication for anyone running an AI programme is that the people best placed to report a problem are the least able to detect it, and the dashboard showing output quality will not show it either. That is why capability debt accrues silently, and why the diagnostic question is not how much AI a team uses but what happens when it is removed.

Key sources

• Bjork, E. L. and Bjork, R. A. (2011). Making Things Hard on Yourself, But in a Good Way: Creating Desirable Difficulties to Enhance Learning. Graded entry.
• Fisher, M., Goddu, M. K. and Keil, F. C. (2015). Searching for Explanations: How the Internet Inflates Estimates of Internal Knowledge. Journal of Experimental Psychology: General. Graded entry.
• Sankaranarayanan, S. (2026). Mitigating "Epistemic Debt" in Generative AI-Scaffolded Novice Programming. Graded entry.
• Budzyn, K. et al. (2025). Endoscopist deskilling risk after exposure to artificial intelligence in colonoscopy. The Lancet Gastroenterology and Hepatology, 10(10), 896-903. Graded entry.
• The Clearing (2026). AI Fatigue in 2026: The State of the Engineer. Cited here for its vocabulary and declined as a measurement. Graded entry.
• Dixon, J. C. (2026). I surveyed workers to see if AI had caused job losses. The Conversation. Graded entry.
• unitQ (2026). Professionals say AI is already reshaping research hiring. PR Newswire, 3 September 2026. Carried here as an example of an undisclosed frame and not as a measurement. Graded entry.
• IBM Institute for Business Value, with Oxford Economics (2026). AI Puts Critical Thinking at the Center of Workforce Priorities. IBM, 21 September 2026. Carried here as a well-fielded frame that still cannot substitute for a measurement. Graded entry.

Related SuperSkills research#

The illusion of competence is why capability debt is not reported by the people carrying it. The mechanisms it conceals are retrieval practice, productive struggle and desirable difficulty. See also automation complacency, the Google effect, am I becoming dependent on AI, assessing capability rather than output and synthetic seniority.

About this research#

Rahim Hirji is the author of SuperSkills (Kogan Page, 2026), keynote speaker on AI and human capability, and founder of The SuperSkills Intelligence Company. The illusion of competence is an established concept in metacognition research and is not a SuperSkills coinage. Findings are attributed to the studies that produced them and kept separate from the interpretation.

How this research works  ·  Reviewed quarterly  ·  Found an error? Tell me and it is corrected on the page.