Each entry gives the study, how it was done, what it found, what it supports and, as plainly, what it does not. Grades describe what kind of evidence a source is; how to read them is on the main page. Every entry has a permanent link and a copyable citation.
Peer-reviewed#
Bloom, B. S. (1984). The 2 Sigma Problem: The Search for Methods of Group Instruction as Effective as One-to-One Tutoring
Educational Researcher, 13(6), 4 to 16
- Method
- A synthesis of two doctoral dissertations from Bloom's group at Chicago (Anania; Burke) that compared conventional classroom instruction, mastery learning and one-to-one tutoring on the same material, with the tutored students as the reference point.
- Finding
- Students tutored one to one, with mastery-learning corrective procedures, scored on average about two standard deviations above students taught conventionally in a class of about thirty; mastery learning in class scored about one standard deviation above. Bloom framed the research problem as finding group methods that could match tutoring.
- What it supports
- The origin of the two-sigma benchmark for tutoring that education and AI-tutoring literature still cite, and that it rests on a small number of controlled studies in Bloom's own programme.
- What it does not support
- That tutoring in general produces two standard deviations. The estimate comes from studies in Bloom's group under mastery-learning conditions, and later reviews of the wider tutoring literature found effects well under half that size (see VanLehn 2011).
Peer-reviewed#
Balzer, W. K., Doherty, M. E. and O'Connor, R. (1989). Effects of cognitive feedback on performance
Psychological Bulletin, 106(3), 410-433
- Method
- Review of the experimental literature on feedback in multiple-cue judgement tasks, separating feedback about outcomes from feedback about the judge's own use of cues.
- Finding
- Outcome feedback, being told whether the answer was right, produced little improvement and sometimes made judgement worse. Cognitive feedback, showing how the judge weighted the available information against how the environment actually weights it, produced improvement.
- What it supports
- That the kind of feedback matters more than the presence of feedback, and that knowing the result is not the same as learning from it.
- What it does not support
- That cognitive feedback is available in real work. It requires a model of the environment that most professional settings do not have.
Practitioner framework#
Collins, A., Brown, J. S. and Newman, S. E. (1989). Cognitive apprenticeship: teaching the crafts of reading, writing and mathematics
In Resnick, L. B. (ed.), Knowing, Learning and Instruction, Lawrence Erlbaum
- Method
- Instructional framework proposed from the study of traditional apprenticeship, setting out six methods: modelling, coaching, scaffolding, articulation, reflection and exploration.
- Finding
- Argues that the defining feature of apprenticeship is making expert thinking visible, and that schooling fails at cognitive tasks because the reasoning stays inside the expert's head.
- What it supports
- Where the idea comes from and what it instructs, and it names the mechanism by which watching a senior person work transfers anything.
- What it does not support
- That the methods produce measured gains. It is a framework argued from observation, with little controlled testing.
Peer-reviewed#
Ericsson, K. A., Krampe, R. T. and Tesch-Romer, C. (1993). The Role of Deliberate Practice in the Acquisition of Expert Performance
Psychological Review, 100(3), 363-406
- Method
- Two studies of violinists and pianists in Berlin.
- Finding
- Sets out deliberate practice: effortful, targeted activity at the edge of current ability, with feedback, sustained over years.
- What it supports
- That expert performance is built through a specific kind of effortful practice rather than exposure.
- What it does not support
- How much of the difference between performers practice explains. See the Macnamara and Maitra re-examination below.
Peer-reviewed#
McDaniel, M. A., Whetzel, D. L., Schmidt, F. L. and Maurer, S. D. (1994). The Validity of Employment Interviews: A Comprehensive Review and Meta-Analysis
Journal of Applied Psychology, 79(4)
- Method
- Meta-analysis of 245 validity coefficients from 86,311 individuals.
- Finding
- Structured interviews predict job performance substantially better than unstructured interviews.
- What it supports
- That structure, rather than the interview itself, is what carries the predictive weight.
- What it does not support
- A single stable figure for the gap. Estimates of the size of the structure effect vary considerably between meta-analyses.
Peer-reviewed#
Arthur, W., Bennett, W., Stanush, P. L. and McNelly, T. L. (1998). Factors that influence skill decay and retention: A quantitative review and analysis
Human Performance, 11(1)
- Method
- Meta-analysis of 189 independent data points from 53 articles.
- Finding
- Skill loss ran from d of -0.01 immediately after training to d of -1.4 after more than 365 days of non-use. Physical, natural and speed-based tasks decayed less than cognitive, artificial and accuracy-based tasks.
- What it supports
- That skill decay is measurable, that it is a function of the interval, and that cognitive skills go first.
- What it does not support
- How fast any particular professional skill decays, or how quickly it can be regained.
Peer-reviewed#
Giedd, J.N., Blumenthal, J., Jeffries, N.O., Castellanos, F.X., Liu, H., Zijdenbos, A., Paus, T., Evans, A.C. and Rapoport, J.L. (1999). Brain development during childhood and adolescence: a longitudinal MRI study
Nature Neuroscience, 2(10), 861-863
- Method
- Longitudinal MRI, 243 scans from 145 healthy participants, 89 male and 46 female, at the US National Institute of Mental Health.
- Finding
- Cortical grey matter in frontal regions peaks in pre-adolescence and then thins through synaptic pruning, with the prefrontal cortex among the last areas to mature.
- What it supports
- That the prefrontal cortex matures later than other regions. This is real, replicated and not in dispute.
- What it does not support
- Any threshold age. The paper reports a slower trajectory, not an endpoint, and names no age at which development completes. Everything downstream that cites it for a cut-off is citing something it does not contain.
Peer-reviewed#
Benkard, C. L. (2000). Learning and Forgetting: The Dynamics of Aircraft Production
American Economic Review, 90(4), 1034-1054, September 2000. DOI 10.1257/aer.90.4.1034. Circulated first as NBER Working Paper 7127, May 1999, DOI 10.3386/w7127. PROVENANCE: the abstract was read at source on the NBER working-paper record on 21 September 2026. The full text of the published article is behind the AEA paywall, the NBER PDF could not be retrieved, and no figure from it is quoted here. A depreciation estimate of 0.96 monthly, implying 61 per cent annual survival of the stock of experience, circulates widely and is reported by Bongers 2017; it has NOT been confirmed against Benkard's own text and must not be used until it has.
- Method
- A cost dataset for a commercial aircraft firm, used to estimate learning dynamics in the production of a wide-body airliner. Method detail beyond the abstract has not been read at source.
- Finding
- In the author's own summary, the data are 'inconsistent with the simple learning hypothesis, and particularly the prediction that a firm's unit cost must decline with its cumulative production'. Instead 'strong support is found for the hypothesis of organizational forgetting', a model in which unit costs depend on past production experience 'but where that experience depreciates over time', and that 'some, but not all, of a firm's production experience transfers from one generation of an aircraft to the next'.
- What it supports
- That organisational experience depreciates rather than accumulating indefinitely, and that the transfer of accumulated experience across product generations is partial. It is the paper that put organisational forgetting into the economics literature.
- What it does not support
- Any rate, in this research, because no number from it has been confirmed at source. Nothing about knowledge work, and nothing about individual people: the unit is a firm's production programme. The partial-transfer finding is about successive aircraft built by the same firm and transfers to a change of supplier or of model only as an analogy.
Peer-reviewed#
Maguire, E. A., Gadian, D. G., Johnsrude, I. S., Good, C. D., Ashburner, J., Frackowiak, R. S. J. and Frith, C. D. (2000). Navigation-related structural change in the hippocampi of taxi drivers
PNAS, 97(8), 4398-4403
- Method
- Cross-sectional structural MRI. 16 right-handed male London taxi drivers with more than 1.5 years driving, against scans of 50 healthy right-handed male non-taxi-drivers.
- Finding
- Posterior hippocampi were significantly larger in taxi drivers than in controls, and hippocampal volume correlated with time spent driving a taxi, positively in the posterior and negatively in the anterior hippocampus.
- What it supports
- That sustained, effortful spatial practice is associated with measurable structural difference in the adult brain, and that the association scales with how long the practice has continued.
- What it does not support
- Causation. It is cross-sectional, so it cannot separate the practice building the brain from a particular brain selecting into the job. It says nothing about what happens when the practice stops, nothing about AI, and nothing about knowledge work.
Compiled review#
Anderson, L. W. and Krathwohl, D. R. (eds.) (2001). A Taxonomy for Learning, Teaching, and Assessing: A Revision of Bloom's Taxonomy of Educational Objectives
Longman, New York
- Method
- A revision of Bloom's 1956 taxonomy by a group of cognitive psychologists and curriculum specialists, restating the categories as verbs and adding a second knowledge dimension.
- Finding
- Six cognitive process categories in ascending order: remember, understand, apply, analyse, evaluate, create.
- What it supports
- That there is a long-established and widely taught vocabulary for ordering cognitive demand, which any newer ladder of AI use is implicitly competing with.
- What it does not support
- That the order is strictly hierarchical in practice, or that it transfers to human-AI interaction. It describes what a learner is asked to do, not what a tool is asked to do, and the two come apart as soon as a model can perform the higher categories on request.
Peer-reviewed#
Kalyuga, S., Chandler, P., Tuovinen, J. and Sweller, J. (2001). When Problem Solving Is Superior to Studying Worked Examples
Journal of Educational Psychology, 93(3), 579-588, September 2001. Verified at source on the ERIC record EJ640537, 18 September 2026, which carries the author abstract in full; the full text is behind the publisher. NOTE that ERIC's citation_author metadata field returns 'Sweller, John' alone while its own record body lists all four authors, which is the second metadata source on this paper's family to name one author where there are four. No figure from this study is used here or on any page, because only the abstract could be read.
- Method
- Experiments with mechanical trade apprentices, given either worked examples to study or the equivalent problems to solve, tested at successive levels of domain experience.
- Finding
- In the authors' own words, inexperienced trainees benefited most from worked examples; with more experience in the domain, worked examples became redundant and problem solving proved superior.
- What it supports
- That the reversal is a measured result and not only a framework. It is the single cleanest demonstration that removing support can become the better instruction for the same person who needed it earlier.
- What it does not support
- Any magnitude, because only the abstract was readable. Also nothing about knowledge work: the domain is mechanical trades and the task is bounded, with a correct answer the experimenter already holds.
Narrative review#
Kalyuga, S., Ayres, P., Chandler, P. and Sweller, J. (2003). The Expertise Reversal Effect
Educational Psychologist, 38(1), 23-31. DOI 10.1207/S15326985EP3801_4. Read in full at source 18 September 2026 from the University of Wollongong Research Online copy, because the publisher's own page is abstract-only. AUTHOR ORDER, and it matters because two orders are in circulation: the article's byline and its running head both read Kalyuga, Ayres, Chandler, Sweller, and Kalyuga is given a separate affiliation (Educational Testing Centre, UNSW) from the other three (School of Education, UNSW). The Wollongong repository's own Publication Details line reverses this to 'Sweller, J., Ayres, P. L., Kalyuga, S. & Chandler, P. A.', which is the form its cover sheet generates from its institutional author record. The article is what ships here. Same class of fault as Nature's page metadata returning only the corresponding author.
- Method
- Review of eleven experiments on interactions between instructional format and learner prior knowledge, roughly half of them by these authors. No new data. No systematic search protocol is reported.
- Finding
- Instructional guidance that helps inexperienced learners can lose its effect and then reverse it as the same learners gain domain knowledge. The authors' explanation is redundancy: once a learner's own schemas supply the guidance, the external guidance has to be cross-referenced against them, and that costs working memory. Reported across split attention, redundancy, modality, worked examples, isolated elements and imagination. Their conclusion is that without tailoring to learner experience, 'the effectiveness of instructional designs is likely to be random'.
- What it supports
- That the sign of a support's effect on learning depends on who is receiving it, established forty years before the AI question and in a literature with no stake in it. It is the strongest independent corroboration the research holds for treating 'does AI help' as a malformed question.
- What it does not support
- Anything about AI, which did not exist in this form when it was written. Nothing in it tests a generative system, and the guidance it studies is a worked example or a labelled diagram, both of which display the reasoning. Also not a systematic review: the studies are selected by the authors, most of the reversals are their own results, and no effect sizes are pooled.
Peer-reviewed#
Gogtay, N., Giedd, J.N., Lusk, L., Hayashi, K.M., Greenstein, D., Vaituzis, A.C., Nugent, T.F., Herman, D.H., Clasen, L.S., Toga, A.W., Rapoport, J.L. and Thompson, P.M. (2004). Dynamic mapping of human cortical development during childhood through early adulthood
Proceedings of the National Academy of Sciences, 101(21), 8174-8179
- Method
- A densely sampled subset of THIRTEEN participants from the NIMH longitudinal project, each scanned roughly every two years.
- Finding
- Maps the sequence in which cortical regions mature, with higher-order association cortices maturing after lower-order sensorimotor regions.
- What it supports
- A developmental sequence, in thirteen people.
- What it does not support
- A population age of maturity. Thirteen participants cannot establish one, and the paper does not claim to. This is the study behind the Time Magazine coverage in which the number 25 first appears in public.
Peer-reviewed#
Cepeda, N. J., Pashler, H., Vul, E., Wixted, J. T. and Rohrer, D. (2006). Distributed Practice in Verbal Recall Tasks: A Review and Quantitative Synthesis
Psychological Bulletin, 132(3)
- Method
- Meta-analysis of 839 assessments across 317 experiments in 184 articles.
- Finding
- Spacing and retention interval act jointly. The gap between practice sessions that produces best retention increases as the target retention interval increases.
- What it supports
- That when practice happens changes how much survives, independently of how much practice there is.
- What it does not support
- Application to procedural or professional skill. The synthesis covers verbal recall.
Peer-reviewed#
Huckman, R. S. and Pisano, G. P. (2006). The Firm Specificity of Individual Performance: Evidence from Cardiac Surgery
Management Science, 52(4), 473-488, April 2006. DOI 10.1287/mnsc.1050.0464. PROVENANCE: the publisher-supplied abstract was read at source on the RePEc record on 21 September 2026. The full text is paywalled and was not opened, so no sample size, coefficient or interval is quoted here.
- Method
- Observational study of cardiac surgeons who operate at more than one hospital within narrow periods of time, using patient mortality as the outcome measure and recent procedure volume at each hospital as the exposure. Sample size and specification not read at source.
- Finding
- In the authors' words, 'the quality of a surgeon's performance at a given hospital improves significantly with increases in his or her recent procedure volume at that hospital but does not significantly improve with increases in his or her volume at other hospitals'. They conclude that 'surgeon performance is not fully portable across hospitals (i.e., some portion of performance is firm specific)' and offer 'preliminary evidence suggesting that this result may be driven by the familiarity that a surgeon develops with the assets of a given organization'.
- What it supports
- That a measurable part of an expert's performance belongs to the pairing of that expert with a particular organisation, and accrues through recent repetition inside it. The comparison is within-surgeon across sites, which is what makes it an argument about the organisation rather than about who is good.
- What it does not support
- Any magnitude, in this research, because the tables were not read. The asset-familiarity explanation is the authors' own and they call it preliminary. It is one clinical speciality in one health system in the 1990s, it concerns freelancing surgeons rather than outsourced corporate functions, and it says nothing about AI.
Peer-reviewed#
Roediger, H. L. III and Karpicke, J. D. (2006). Test-Enhanced Learning: Taking Memory Tests Improves Long-Term Retention
Psychological Science, 17(3), 249-255. DOI 10.1111/j.1467-9280.2006.01693.x
- Method
- Two experiments, 120 and 180 participants, reading prose passages on general science topics. Restudying compared with being tested, with final recall measured at 5 minutes, 2 days or 1 week.
- Finding
- The winner reverses with delay. At 5 minutes restudying beat testing, 81 per cent against 75 per cent. At one week testing beat restudying, 56 per cent against 42 per cent. In the second experiment repeated study led at 5 minutes, 83 against 71 per cent, and trailed badly at one week, 40 against 61 per cent.
- What it supports
- That the study method producing the best immediate performance produces the worst durable retention, and that a measurement taken close to the learning will rank the methods in exactly the wrong order.
- What it does not support
- Anything about AI. It is prose recall in a laboratory, and the transfer to professional judgement is by analogy.
Peer-reviewed#
Kapur, M. (2008). Productive Failure
Cognition and Instruction, 26(3), 379-424. DOI 10.1080/07370000802212669
- Method
- Randomised comparison, 309 eleventh-grade physics students in India working on Newtonian kinematics. One group solved ill-structured problems in groups before individual well-structured problems; the other solved well-structured problems throughout.
- Finding
- The group given ill-structured problems struggled visibly and produced poor solutions during the collaborative phase, then outperformed the other group on individual near-transfer and far-transfer measures afterwards.
- What it supports
- That struggle which looks like failure at the time can produce better subsequent transfer than a smooth path through the same material, and that judging a learning design by how well it is going is unreliable.
- What it does not support
- A precise magnitude. The design, sample and direction of the result are confirmed from the publisher's abstract and the author's own presentation of the study, but the full text sits behind a paywall that could not be read, so no post-test figures are quoted here.
Compiled review#
American Red Cross Advisory Council on First Aid, Aquatics, Safety and Preparedness (2009). Scientific Review: CPR Skill Retention
American Red Cross
- Method
- Systematic review of 47 articles on CPR skill retention across healthcare and lay populations, with retest intervals from six weeks to 24 months.
- Finding
- Substantial skill degradation occurs within the first year after training, with declining retention from six to twelve months unless there is refresher training.
- What it supports
- That a life-critical, heavily trained procedural skill decays on a timescale of months without practice.
- What it does not support
- Any link to patient outcomes. Decay was measured on manikins, not in resuscitations.
Peer-reviewed#
Bjork, E. L. and Bjork, R. A. (2011). Making Things Hard on Yourself, But in a Good Way: Creating Desirable Difficulties to Enhance Learning
In Psychology and the Real World, Worth Publishers
- Method
- Synthesis of decades of laboratory work on spacing, interleaving and retrieval practice.
- Finding
- Conditions that make study feel harder improve long-term retention; conditions that make it feel fluent improve immediate performance and worsen retention. Learners systematically mistake fluency for learning.
- What it supports
- That the subjective sense of learning is an unreliable guide to whether learning occurred. Directly relevant, because AI makes work feel fluent.
- What it does not support
- That AI-assisted work is equivalent to a fluent study condition. That inference is ours, not the authors'.
Peer-reviewed#
Cook, D. A., Hatala, R., Brydges, R., Zendejas, B., Szostek, J. H., Wang, A. T., Erwin, P. J. and Hamstra, S. J. (2011). Technology-enhanced simulation for health professions education: a systematic review and meta-analysis
JAMA, 306(9), 978-988
- Method
- Systematic review and meta-analysis of 609 studies of simulation-based training in the health professions.
- Finding
- Simulation training produced large effects on knowledge, skills and behaviours against no intervention, and smaller effects on patient outcomes.
- What it supports
- That rehearsing decisions in a setting built for rehearsal is an effective way to build professional capability, and that it is the best-evidenced delivery structure available.
- What it does not support
- That the effect transfers outside clinical training, or that simulation without feedback and repetition does anything. The comparison in most studies is against no training at all.
Peer-reviewed#
Karpicke, J. D. and Blunt, J. R. (2011). Retrieval Practice Produces More Learning than Elaborative Studying with Concept Mapping
Science, 331(6018), 772-775. DOI 10.1126/science.1199327
- Method
- Two experiments, 80 and 120 undergraduates, comparing retrieval practice with elaborative study by concept mapping, tested one week later.
- Finding
- Retrieval practice scored 0.67 against 0.45 for concept mapping, about a 50 per cent advantage in long-term retention, d = 1.50. In the second experiment 101 of 120 students, 84 per cent, did better after retrieval practice than after elaborative study.
- What it supports
- That effortful recall outperforms a more elaborate and more comfortable-feeling study method, on the same material, in the same students.
- What it does not support
- That concept mapping is worthless. A published comment by Mintzes and colleagues (Science, 2011, 334(6055), 453) disputes the instructional fidelity of the concept-mapping condition, and anyone citing this should note it.
Compiled review#
VanLehn, K. (2011). The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems
Educational Psychologist, 46(4), 197 to 221
- Method
- A review of experimental comparisons of human tutoring, intelligent tutoring systems of different granularity, and no tutoring, computing mean effect sizes across the studies gathered.
- Finding
- Human tutoring averaged an effect size of 0.79 standard deviations over no tutoring; step-based intelligent tutoring systems 0.76; substep-based systems 0.40; answer-based systems 0.31. VanLehn concluded that the two-sigma effect of one-to-one human tutoring reported by Bloom was not observed in the wider literature, and that well-designed computer tutors were nearly as effective as human ones.
- What it supports
- That the standard against which AI tutors are now measured, human one-to-one tutoring, produces an effect closer to 0.8 standard deviations than to 2 across the experimental literature, and that computer tutors had approached it a decade before large language models.
- What it does not support
- What the effect is for any particular subject, age group or tutor; the studies reviewed are heterogeneous and the review is fifteen years old. The comparison is on immediate learning measures and says nothing about retention or about generative AI.
Peer-reviewed#
Woollett, K. and Maguire, E. A. (2011). Acquiring 'the Knowledge' of London's Layout Drives Structural Brain Changes
Current Biology, 21(24), 2109-2114, 20 December 2011
- Method
- Longitudinal structural MRI over four years. 79 male trainee London taxi drivers studying for the Knowledge, and 31 male non-taxi-driver controls, scanned before and after. Average-IQ adults, real training rather than a laboratory task.
- Finding
- In those who qualified, acquiring an internal spatial representation of London was associated with a selective increase in grey matter volume in the posterior hippocampi, with concomitant changes to their memory profile. In the authors' words, no structural brain changes were observed in trainees who failed to qualify or in control participants. The gain in posterior hippocampus came alongside costs elsewhere in the memory profile.
- What it supports
- The strongest available evidence that the practice itself does the work rather than selection. Same starting cohort, same training, and the structural change appears only in those who completed it. It also shows the trade: the capability gained is paid for with capability elsewhere, which is skill substitution observed in tissue rather than argued.
- What it does not support
- Anything about AI, and anything about removal. This is acquisition, over four years, in one domain. It does not show that the structure regresses when the practice is delegated to a machine, and no study has shown that. The frequently circulated claim that GPS use produces measurable cognitive decline or early-onset dementia is a prediction rather than a finding, and cannot be traced to a study of this kind.
Institutional survey#
Bureau d'Enquetes et d'Analyses (2012). Final Report on the accident on 1 June 2009 to the Airbus A330-203, flight AF 447
BEA, France
- Method
- Statutory accident investigation. Flight recorder analysis and training record review. 25 new safety recommendations.
- Finding
- The investigation cited the lack of practical training in high-altitude manual handling and in the procedure for speed anomalies among the contributing factors.
- What it supports
- That a formal investigation attributed part of an accident to manual handling practice that had not been maintained.
- What it does not support
- A general rate of skill loss across the pilot population. It is one accident.
Regulator guidance#
Federal Aviation Administration, Flight Standards Service (2013). SAFO 13002: Manual Flight Operations
Safety Alert for Operators, dated 4 January 2013 (1/4/13)
- Method
- A regulator's safety alert encouraging operators to promote manual flight operations when appropriate. Fetched and read at source on 4 October 2026. It reports no data.
- Finding
- States that continuous use of autoflight systems could lead to degradation of a pilot's ability to recover quickly from an undesired state, and asks operators to set policies giving pilots appropriate opportunities to exercise manual flying skills, with guidance on when automation is preferable, for example in high-workload conditions.
- What it supports
- That the regulator of the most automated profession asked operators in writing, in 2013, to schedule practice without the automation.
- What it does not support
- That the practice works, or how often it is needed. It is guidance rather than a rule and carries no measured effect of the periods it recommends.
Practitioner method#
Klein, G., Hintze, N. and Saab, D. (2013). Thinking inside the box: the ShadowBox method for cognitive skill development
Proceedings of the 11th International Conference on Naturalistic Decision Making
- Method
- A training method in which a trainee works through a scenario making the same calls an expert made, then compares their choices and their reasons against the expert panel's, with small field evaluations reported by the developers.
- Finding
- Reported gains in the developers' own field studies were about 28 per cent with 59 Marines and about 21 per cent with 30 Army officers.
- What it supports
- That there is a usable structure for training the reasoning behind a decision rather than the decision itself, by making the expert's cue selection visible and comparable.
- What it does not support
- That the gains generalise. The evaluations are small, run by the method's developers, and use scenario scores rather than performance at work.
Peer-reviewed#
Moonen-van Loon, J. M. W., Overeem, K., Donkers, H. H. L. M., van der Vleuten, C. P. M. and Driessen, E. W. (2013). Composite reliability of a workplace-based assessment toolbox for postgraduate medical education
Advances in Health Sciences Education, 18(5)
- Method
- Generalisability study of 12,779 workplace-based assessments from 953 medical residents.
- Finding
- A reliability coefficient of 0.80 required eight mini-CEX observations, nine DOPS or nine multi-source feedback rounds. Combined in a portfolio the requirement fell to seven, eight and one respectively.
- What it supports
- That observing someone at work can reach defensible reliability, and roughly how many observations that takes.
- What it does not support
- That these scores predict later performance or patient outcomes. It measures consistency, not criterion validity. A single observation is not reliable.
Institutional modelling#
PARC/CAST Flight Deck Automation Working Group (2013). Operational Use of Flight Path Management Systems: Final Report
Federal Aviation Administration
- Method
- Working-group synthesis of accident and incident data, operator surveys and prior research. 28 findings, 18 recommendations.
- Finding
- Identified vulnerabilities in manual handling after transition from automated control, and in the definition, development and retention of those skills. Also found that pilots sometimes rely too much on automated systems and may be reluctant to intervene.
- What it supports
- That a regulator examined automation dependence in a whole industry and named skill retention as a finding.
- What it does not support
- A measured rate of skill decay. It is a synthesis of findings, not a controlled study.
Peer-reviewed#
Tannenbaum, S. I. and Cerasoli, C. P. (2013). Do team and individual debriefs enhance performance? A meta-analysis
Human Factors, 55(1), 231-245
- Method
- Meta-analysis of 46 studies of structured debriefs, for individuals and teams, in field and simulated settings.
- Finding
- Properly conducted debriefs improved performance by around 20 to 25 per cent over control, with an average effect size of about 0.67.
- What it supports
- That a structured review of what happened and why, done to a method, reliably improves subsequent performance. It is the best-evidenced routine available for building judgement from experience.
- What it does not support
- That any meeting called a debrief does this. The effect depends on structure, on self-discovery rather than being told, and on reviewing process rather than outcome.
Peer-reviewed#
Casner, S. M., Geven, R. W., Recker, M. P. and Schooler, J. W. (2014). The Retention of Manual Flying Skills in the Automated Cockpit
Human Factors, 56(8)
- Method
- 16 airline pilots flew routine and non-routine scenarios in a Boeing 747-400 simulator with automation level varied.
- Finding
- Instrument scanning and manual control were mostly intact even where pilots reported little recent practice. The cognitive tasks, tracking position without a map, deciding the next navigational step and recognising instrument failures, showed frequent and significant problems.
- What it supports
- That the hands survive disuse better than the judgement does, in the profession with the most automation experience.
- What it does not support
- A general rule for knowledge work. The sample is 16 pilots in a simulator.
Compiled review#
Macnamara, B. N., Hambrick, D. Z. and Oswald, F. L. (2014). Deliberate Practice and Performance in Music, Games, Sports, Education, and Professions: A Meta-Analysis
Psychological Science, 25(8), 1608-1618
- Method
- Meta-analysis of 88 studies and 157 effect sizes relating accumulated deliberate practice to performance.
- Finding
- Deliberate practice explained 12 per cent of the variance in performance overall, 95% CI [9%, 15%], leaving 88 per cent unexplained. By domain: 26 per cent for games, 21 per cent for music, 18 per cent for sports, 4 per cent for education and under 1 per cent for the professions.
- What it supports
- That accumulated practice is one contributor among several rather than the dominant explanation of expert performance, and that its contribution varies sharply by domain.
- What it does not support
- That practice does not build professional expertise. The professions estimate rests on 7 effect sizes, is not statistically significant (p = .62), and the occupations sampled were computer programming, military aircraft piloting, soccer refereeing and insurance selling. It is not evidence about law, medicine, consulting or analysis. It gets quoted as though it were.
Peer-reviewed#
Rowland, C. A. (2014). The Effect of Testing Versus Restudy on Retention: A Meta-Analytic Review of the Testing Effect
Psychological Bulletin, 140(6), 1432-1463. DOI 10.1037/a0037559
- Method
- Meta-analysis, 159 effect sizes from 61 studies published between 1975 and 2013, random-effects model.
- Finding
- A reliable testing effect, g = 0.50 with a confidence interval of 0.42 to 0.58. With feedback the effect rises to g = 0.73; without feedback it falls to g = 0.39.
- What it supports
- That the testing effect survives aggregation across four decades of studies, at a moderate effect size, and that feedback roughly doubles it.
- What it does not support
- That the effect size transfers to workplace learning. The constituent studies are overwhelmingly laboratory and classroom work on verbal material.
Peer-reviewed#
Hartshorne, J.K. and Germine, L.T. (2015). When does cognitive functioning peak? The asynchronous rise and fall of different cognitive abilities across the life span
Psychological Science, 26(4), 433-443
- Method
- Cross-sectional analysis of large web-based and standardisation samples across a wide age range.
- Finding
- Different cognitive abilities peak at different ages, some in the late teens or early twenties, others not until the forties or fifties. There is no single age at which cognitive functioning peaks.
- What it supports
- That a single maturity age is the wrong shape of answer, whatever number is put in it. Abilities do not arrive together.
- What it does not support
- That age is irrelevant. It shows the timing is ability-specific rather than absent.
Peer-reviewed#
Hogarth, R. M., Lejarraga, T. and Soyer, E. (2015). The two settings of kind and wicked learning environments
Current Directions in Psychological Science, 24(5), 379-385
- Method
- Theoretical account with experimental illustrations, distinguishing environments by the quality of the feedback they return to the learner.
- Finding
- In a kind environment feedback is quick, accurate and drawn from a complete sample, so experience teaches. In a wicked one it is delayed, partial, biased by the learner's own actions, or drawn from a sample the learner's decisions censored, so experience can teach the wrong lesson with confidence.
- What it supports
- That the value of experience depends on the structure of the feedback, not on the amount of experience, and that some domains cannot be learned by repetition alone.
- What it does not support
- Which real occupations sit where. The distinction is a lens, not a measurement, and no instrument scores an environment for kindness.
Peer-reviewed#
Murre, J. M. J. and Dros, J. (2015). Replication and Analysis of Ebbinghaus' Forgetting Curve
PLoS ONE, 10(7)
- Method
- Single-subject relearning experiment replicating Ebbinghaus across intervals from 20 minutes to 31 days, 10 replications per interval.
- Finding
- Relearning to criterion took less time than original learning at every retention interval tested, confirming the savings effect Ebbinghaus reported in 1885.
- What it supports
- That something survives apparent forgetting, and that it shows up as faster relearning rather than as recall.
- What it does not support
- That professional skills behave like nonsense syllables, or that savings hold at the scale of a career. It is one subject and verbal material.
Peer-reviewed#
Storm, B. C. and Stone, S. M. (2015). Saving-Enhanced Memory: The Benefits of Saving on the Learning and Remembering of New Information
Psychological Science, 26(2), 182-188. DOI 10.1177/0956797614559285
- Method
- Three experiments on the effect of saving a digital file on memory for subsequently studied material. GRADED FROM THE PUBLISHED ABSTRACT ONLY: the full text is paywalled and could not be read at the primary source, so no sample sizes or statistics are recorded here.
- Finding
- Saving one file before studying a new file significantly improved memory for the contents of the new file. The effect was not observed when the saving process was deemed unreliable, or when the contents of the to-be-saved file were not substantial enough to interfere with memory for the new file.
- What it supports
- That cognitive offloading can improve rather than degrade subsequent memory, and that the benefit depends on the external store being trusted.
- What it does not support
- Anything quantified. This entry rests on the abstract alone, which is a weaker basis than every other entry in this base and is stated as such. It also predates generative AI and concerns file saving rather than a system that can fabricate its own contents.
Peer-reviewed#
Hamilton, E. R., Rosenberg, J. M. and Akcaoglu, M. (2016). The Substitution Augmentation Modification Redefinition (SAMR) Model: A Critical Review and Suggestions for Its Use
TechTrends, 60(5), 433-441
- Method
- Critical review of the SAMR model against the educational technology literature.
- Finding
- Names three problems: the absence of context, a rigid hierarchical structure that implies higher is better, and an emphasis on product over process. The authors also record that SAMR is largely absent from the peer-reviewed literature despite heavy practitioner use, and that its theoretical and foundational evidence is thin.
- What it supports
- That a widely adopted framework can spread through practice with almost no evidence behind it, and that popularity is not a proxy for validity. Held here as the standard against which any framework in this research, including this research's own, should be judged.
- What it does not support
- That SAMR is useless in practice. The authors offer suggestions for its use rather than a case for abandoning it, and their objection is to how it is applied more than to what it contains.
Compiled review#
Simons, D. J., Boot, W. R., Charness, N., Gathercole, S. E., Chabris, C. F., Hambrick, D. Z. and Stine-Morrow, E. A. L. (2016). Do Brain-Training Programs Work?
Psychological Science in the Public Interest, 17(3)
- Method
- Systematic review applying pre-specified best-practice standards to every study cited by commercial brain-training companies as evidence of efficacy.
- Finding
- Extensive evidence that training improves performance on the trained tasks, less evidence for closely related tasks, and little evidence that training improves distantly related tasks or everyday cognitive performance. No cited study met all best-practice standards.
- What it supports
- That near transfer is real and far transfer, which is what any claim to have trained attention requires, is essentially unsupported.
- What it does not support
- That practising a specific skill is useless. The failure is transfer, not learning.
Peer-reviewed#
Bongers, A. (2017). Learning and forgetting in the jet fighter aircraft industry
PLOS ONE, 12(9), e0185364, published 28 September 2017. DOI 10.1371/journal.pone.0185364. Open access, CC BY. Read in full at source 21 September 2026. PROVENANCE: the two coefficient tables are published as images and are not in the article text, so every figure below is taken from the author's running prose. The paper states no sample size, standard errors or R-squared anywhere in its text.
- Method
- Learning curves estimated from United States Department of Defense flyaway cost, converted to constant dollars with the DoD procurement deflator, for three fighter programmes: the F/A-18E/F Super Hornet, 554 units acquired in fiscal 1997 to 2013; the F-22A Raptor, 182 units in fiscal 2000 to 2009; and the F-35A Lightning II, 135 units in fiscal 2007 to 2015. Data are annual procurement lots, so the per-programme series runs from nine to seventeen observations. The standard log-log learning curve is fitted by ordinary least squares; the forgetting model, in which the stock of experience carries over at a rate lambda, is fitted by nonlinear least squares. Estimates repeated on lot-adjusted data by the Loerch method, which the author reports as making no significant difference.
- Finding
- Baseline learning curves of 86 per cent for the Super Hornet, 85.4 per cent for the Raptor and 91 per cent for the Lightning II, so learning rates of about 14 per cent for the first two and about 9 per cent for the third. In the forgetting model the persistence parameter was not significantly different from zero for the F-22A and F-35A: in the author's words, 'the depreciation of experience in the production of these units is total, on an annual basis'. For the Super Hornet she estimates 0.072, 'indicating that, on an annual basis, only 7.2% of the experience is maintained'. She attributes the difference to production rate, noting that Benkard 'argues that the low aircraft production rate is the main factor that explains the observed high forgetting. Our results seem to confirm that.'
- What it supports
- That accumulated organisational production experience can depreciate almost completely within a year, and that the rate at which the work is actually performed is a candidate explanation for how much survives. It is a measurement of organisational forgetting rather than individual skill decay: the unit of analysis is a production programme.
- What it does not support
- Anything about professional judgement, advice or knowledge work; the outcome is unit cost in aircraft assembly. The data are annual lots rather than continuous production and the author warns that the total-depreciation result 'may be due to the small number of units produced and the annual nature of the data used'. She declares two specifications unusable, the Raptor forgetting model as inconsistent and the F-35A column with both controls as implying rising costs, and notes that F-35A learning is overestimated because flyaway cost covers only units bought by the DoD while other countries also bought the aircraft. No sample size or standard error is reported in the text, so the precision behind 0.072 cannot be assessed from the paper as published in HTML.
Regulator guidance#
Federal Aviation Administration, Flight Standards Service (2017). SAFO 17007: Manual Flight Operations Proficiency
Safety Alert for Operators, dated 4 May 2017 (5/4/17)
- Method
- A regulator's safety alert on manual flight proficiency. Fetched and read at source on 4 October 2026. It reports no data.
- Finding
- Encourages training and line-operations policies that ensure proficiency in manual flight operations is developed and maintained for air carrier pilots, to be built into operational policy, ground training, flight training and proficiency checks. Lists SAFO 13002 as related guidance.
- What it supports
- That the 2013 request was repeated and widened to training and proficiency checks four years later.
- What it does not support
- Any measurement of skill retention or of the effect of the policy. Guidance on the law, not the law itself.
Peer-reviewed#
Przybylski, A. K. and Weinstein, N. (2017). A Large-Scale Test of the Goldilocks Hypothesis: Quantifying the Relations Between Digital-Screen Use and the Mental Well-Being of Adolescents
Psychological Science, 28(2), 204-215
- Method
- Preregistered analysis of a representative sample of English 15-year-olds. Sampling frame 298,080; 120,115 provided usable data, 100,850 on paper and 19,265 online.
- Finding
- Links between digital screen time and mental wellbeing are described by quadratic rather than linear functions, with inflection points at 1 hour 40 minutes for weekday video-game play and 1 hour 57 minutes for weekday smartphone use, rising to 3 hours 41 minutes and 4 hours 17 minutes for other measures and to between 3 hours 35 minutes and 4 hours 50 minutes at weekends. Average Cohen's d for engagement beyond the inflection points was minus 0.18, accounting for 1 per cent or less of variance, against d = 0.54 for regularly eating breakfast and 0.58 for regular sleep. The authors state that moderate use is not intrinsically harmful and may be advantageous.
- What it supports
- That dose-response in this literature is not linear, and that the negative effects of heavy use are less than a third the size of the positive associations with sleep and breakfast.
- What it does not support
- Causation, and not displacement. The paper names the displacement hypothesis as the field's dominant assumption and calls for future work systematically analysing what is being displaced or amplified, which the field did not then do. Cross-sectional, self-reported exposure, English 15-year-olds only.
Compiled review#
National Academies of Sciences, Engineering, and Medicine (2018). How People Learn II: Learners, Contexts, and Cultures
The National Academies Press, Washington DC
- Method
- Consensus study report synthesising research on learning across the lifespan.
- Finding
- Defines metacognition as the ability to monitor and regulate one's own cognitive processes and to consciously regulate behaviour, including affective behaviour, and identifies calibration as the accuracy of a learner's monitoring.
- What it supports
- That the established definition contains a control component as well as a knowledge component. Monitoring alone is not metacognition.
- What it does not support
- Anything about AI. The report predates general availability of these systems and makes no claim about them.
Argued perspective#
Caulfield, M. (2019). SIFT (The Four Moves)
Hapgood, 19 June 2019. An earlier version, Four Moves and a Habit, appeared in Web Literacy for Student Fact Checkers (2017)
- Method
- A practitioner method, not a study. Four moves: Stop, Investigate the source, Find better coverage, Trace claims to the original context.
- Finding
- Turns Wineburg and McGrew's lateral reading result into four actions a person can perform in under a minute, and is now taught in university libraries worldwide.
- What it supports
- That the fact-checker behaviour can be reduced to a teachable sequence. It is the most widely adopted of the source-evaluation frameworks and the closest external comparison to any source rule offered here.
- What it does not support
- Its own effectiveness. Caulfield offers it as a practical method and does not present a trial of it, and it inherits its evidence from the lateral reading literature rather than generating any.
Argued perspective#
Ericsson, K. A. and Harwell, K. W. (2019). Deliberate practice and proposed limits on the effects of practice on the acquisition of expert performance: why the original definition matters and recommendations for future research
Frontiers in Psychology, 10, 2396
- Method
- Argued reply in a reviewed venue, re-examining which activities in the meta-analytic literature meet the original definition of deliberate practice.
- Finding
- Argues that most studies aggregated under the term measured practice of other kinds, and that the low variance figures follow from that misclassification rather than from a limit on practice itself.
- What it supports
- That the size of the practice effect is contested on definitional grounds, and that a figure quoted from either side needs its definition quoted with it.
- What it does not support
- That the meta-analytic estimate is wrong. It reports no new data and does not resolve the dispute.
Peer-reviewed#
Macnamara, B. N. and Maitra, M. (2019). The role of deliberate practice in expert performance: revisiting Ericsson, Krampe and Tesch-Romer (1993)
Royal Society Open Science, 6, 190327
- Method
- Direct replication and re-analysis of the 1993 study.
- Finding
- Accumulated practice explained considerably less of the difference between performers than the original is usually taken to claim.
- What it supports
- That the quality and design of practice matters more than the count. Included here deliberately, because it complicates the argument this research relies on.
- What it does not support
- That practice does not matter. It does; the simple dose-response reading is what fails.
Peer-reviewed#
Orben, A. and Przybylski, A. K. (2019). The association between adolescent well-being and digital technology use
Nature Human Behaviour, 3(2), 173-182
- Method
- Specification curve analysis across three nationally representative datasets: the US Youth Risk and Behaviour Survey 2007-2015 at 74,814 adolescents, Monitoring the Future 2008-2016 at 268,672, and the UK Millennium Cohort Study at 11,872, a total of 355,358. 372 justifiable specifications identified for YRBS, 40,966 for MTF and 603,979,752 for MCS, of which 20,004 were run.
- Finding
- The association between digital technology use and adolescent wellbeing is negative but small, explaining at most 0.4 per cent of the variation in wellbeing, which the authors state is too small to warrant policy change. In YRBS, regularly eating potatoes was associated with wellbeing 0.9 times as negatively as technology use; in MCS, wearing glasses was 1.5 times as negatively. Bullying ran 4.3 times more negative and marijuana 2.7 times, both YRBS. Sleep and breakfast ranged from 1.7 to 44.2 times more positive across all three datasets.
- What it supports
- That an entire public debate was conducted on an effect too small to act on, and that analytic flexibility rather than data availability was doing the work in the studies that found more.
- What it does not support
- Causation in either direction. The authors state it is possible the associations they document, and those previously documented, are spurious, and that noisy self-report measurement could itself have diminished a real effect. Two figures often attached to this paper are not in it: 1.45 times for glasses, and 3,221,225,472 analyses, which comes from Odgers and Jensen describing it. Read in the Oxford ORA accepted manuscript, the published full text being paywalled.
Peer-reviewed#
Orben, A. and Przybylski, A. K. (2019). Screens, Teens, and Psychological Well-Being: Evidence From Three Time-Use-Diary Studies
Psychological Science, 30(5), 682-696
- Method
- Exploratory and confirmatory specification analyses across three nationally representative datasets from Ireland, the United States and the United Kingdom, N = 17,247 after exclusions, using time-use diaries as well as retrospective self-report.
- Finding
- Little evidence of substantial negative associations between digital screen engagement and adolescent wellbeing, whether measured across the day or before bedtime. Correlations between diary-recorded and retrospectively self-reported engagement were 0.18 in Ireland, 0.08 and 0.05 on US weekdays and weekend days, and 0.18 in the UK. Retrospective self-report consistently produced the most negative correlations. Extrapolating from median effects, an adolescent would need to report 63 hours 31 minutes more technology use a day to lower wellbeing by half a standard deviation, or 11 hours 14 minutes taking the maximum effect size in the specification set.
- What it supports
- That the two standard instruments for measuring screen exposure barely agree with each other, and that the more negative results come from the weaker instrument, which is what common method variance would predict.
- What it does not support
- That diaries are ground truth. They remain recall-based and the authors note brief or concurrent uses may not be recorded. Still cross-sectional, and it measures totals and timing rather than the kind of activity displaced.
Peer-reviewed#
Wineburg, S. and McGrew, S. (2019). Lateral Reading and the Nature of Expertise: Reading Less and Learning More When Evaluating Digital Information
Teachers College Record, 121(11), 1-40
- Method
- Think-aloud study of 45 experienced internet users evaluating unfamiliar websites: 10 PhD historians, 10 professional fact-checkers and 25 Stanford undergraduates.
- Finding
- Historians and undergraduates read vertically, staying on the page and judging it by its own appearance. Fact-checkers left almost immediately, opened new tabs and read laterally, judging the source by what the rest of the web said about it. The fact-checkers reached sounder conclusions in less time.
- What it supports
- That evaluating a source is a behaviour rather than a checklist, that the behaviour is learnable, and that domain expertise does not confer it: PhD historians performed like undergraduates.
- What it does not support
- Anything about AI output specifically. It predates generative models, and a model's answer has no source page to leave. The transferable part is the move away from judging a text by its own surface.
Peer-reviewed#
Dahmani, L. and Bohbot, V. D. (2020). Habitual use of GPS negatively impacts spatial memory during self-guided navigation
Scientific Reports, 10, 6310, 14 April 2020. DOI 10.1038/s41598-020-62877-0. Read in full at source 5 September 2026
- Method
- Behavioural, cross-sectional with an unplanned longitudinal arm. 50 healthy regular drivers in Montreal aged 19 to 35 (18 women, 32 men; mean age 27.6), driving at least four days a week, from 60 recruited. Lifetime GPS experience measured by the McGill GPS questionnaire; navigation measured on two virtual radial-arm mazes plus a map-drawing score and the Santa Barbara Sense of Direction scale. 13 of the 50 returned a mean of 3.23 years later. Effect sizes are Pearson r with bootstrapped one-tailed BCa 95 per cent intervals. NO NEUROIMAGING: hippocampal dependence is inferred from prior validation of the tasks, not measured in this sample.
- Finding
- Cross-sectionally, greater lifetime GPS experience was associated with lower use of hippocampus-dependent spatial strategies (r = -0.22 on the first probe trial), lower navigation strategy scores (r = -0.20), poorer map drawing (r = -0.22) and fewer landmarks noticed (r = -0.26). In the 13-person follow-up, hours of GPS use since first testing tracked a steeper decline in spatial memory strategy use (r = -0.68) and in map drawing (r = -0.52).
- What it supports
- The only study to follow the same people while their GPS use rose. Its strongest internal argument against reverse causation is that heavier GPS users did not report a poorer sense of direction (r = 0.07 against the SBSOD), so the obvious alternative, that weak navigators reach for the satnav, has no support in the data.
- What it does not support
- Anything about the brain, because no scan was taken. Anything about dementia, Alzheimer's or atrophy: those words appear nowhere in the paper. And little with confidence about the longitudinal effect, which rests on 13 people from an unplanned follow-up. The authors' own words: they caution against any strong conclusions as spurious correlations are possible. Their discussion elsewhere uses notably firmer causal language than that sentence licenses. Nothing here transfers to reasoning: spatial memory is not judgement.
Compiled review#
Odgers, C. L. and Jensen, M. R. (2020). Annual Research Review: Adolescent mental health in the digital age: facts, fears, and future directions
Journal of Child Psychology and Psychiatry, 61(3), 336-348
- Method
- Annual research review of the evidence on adolescent digital technology use and mental health, covering 29 studies in its main table.
- Finding
- Most research to date has been correlational, focused on adults rather than adolescents, and has produced a mix of conflicting small positive, negative and null associations. The most recent and rigorous large-scale preregistered studies report small associations that offer no way of distinguishing cause from effect and are unlikely to be of clinical or practical significance, explaining less than 0.5 per cent of the variance. Of the 29 studies reviewed, only two included objective or informant-rated measures of screen use, and the correlation between objectively measured and retrospectively reported screen time is estimated at about 0.20.
- What it supports
- That the field's own review verdict is that the evidence does not support causal claims or even strong consistent correlational patterns.
- What it does not support
- Anything new. It is a review, not primary data, it does not test displacement of any specific activity, and it does not address AI. Read in the NIH author manuscript.
Peer-reviewed#
Parry, D. A., Davidson, B. I., Sewall, C. J. R., Fisher, J. T., Mieczkowski, H. and Quintana, D. S. (2021). A systematic review and meta-analysis of discrepancies between logged and self-reported digital media use
Nature Human Behaviour, 5(11), 1535-1547
- Method
- Systematic review and meta-analysis using robust variance estimation. 106 effect sizes included overall; the self-report against logged comparison draws on 66 effect sizes from 44 studies with a total sample of 52,007.
- Finding
- The correlation between self-reported and logged digital media use is positive but medium, r = 0.38, 95 per cent CI 0.33 to 0.42. For problematic use it falls to r = 0.25 across 40 effect sizes from 19 studies. Over-reporting and under-reporting occur in similar proportions, and fewer than 10 per cent of self-reports fall within 5 per cent of the equivalent logged value. The authors conclude that self-report measures may not be a valid stand-in for more objective measures and ask for pause in drawing wide-reaching knowledge or policy conclusions from studies relying solely on them.
- What it supports
- That the independent variable in most of the screen-time literature is wrong by a measured amount, which is the strongest single reason not to carry that method into questions about AI use.
- What it does not support
- That self-report is useless, or that logs are ground truth: the authors note potential biases in log data too. They state it is an open question whether the discrepancy is random or systematic error. Nothing in it concerns AI. Read in the University of Bath accepted manuscript alongside the published abstract.
Peer-reviewed#
Shorey, S., Lau, T. C., Lau, S. T. and Ang, E. (2021). Entrustable professional activities in health care education: a scoping review
Medical Education, 55(11), 1247-1260
- Method
- Scoping review of the published literature on entrustable professional activities, the units of professional work a trainee is progressively permitted to perform unsupervised.
- Finding
- Adoption is wide across health professions education and the supervision levels are used to structure graduated responsibility, while validity evidence for the entrustment decisions themselves is mixed and uneven.
- What it supports
- That one profession has built an explicit, staged mechanism for handing over responsibility as capability is demonstrated, and that the mechanism is in real use.
- What it does not support
- That entrustment decisions are reliable, or that the approach transfers outside clinical training.
Peer-reviewed#
Sinha, T. and Kapur, M. (2021). When Problem Solving Followed by Instruction Works: Evidence for Productive Failure
Review of Educational Research, 91(5), 761-798. DOI 10.3102/00346543211019105
- Method
- Meta-analysis of 53 studies and 166 comparisons of problem-solving-before-instruction against instruction-before-problem-solving.
- Finding
- A moderate effect favouring problem solving first, Hedges g = 0.36 with a confidence interval of 0.20 to 0.51, rising to between 0.37 and 0.58 where the design followed the productive failure principles closely. The effect reverses for second to fifth graders and for domain-general skills, where instruction first wins.
- What it supports
- That letting people struggle before teaching them beats teaching them first, at moderate effect size, for older learners on domain-specific content.
- What it does not support
- That struggle is universally good. The authors report the reversal for young children and for general skills themselves, and an earlier meta-analysis by Darabi and colleagues rested on only 12 studies.
Compiled review#
Verhaeghen, P. (2021). Mindfulness as Attention Training: Meta-Analyses on the Links Between Attention Performance and Mindfulness Interventions, Long-Term Meditation Practice, and Trait Mindfulness
Mindfulness, 12(3)
- Method
- Three meta-analyses covering 109 effect sizes from 40 intervention studies, 59 effect sizes from 18 long-term meditator studies, and 197 effect sizes from 28 trait studies.
- Finding
- Average effects were small to moderate, Hedges g of 0.29 for interventions and 0.32 for long-term practice, concentrated in inhibition and executive control rather than sustained attention.
- What it supports
- That something measurable happens, and that it is smaller and narrower than the popular claim.
- What it does not support
- That the effect survives comparison with an active control. The published breakdown does not separate active from passive controls, so demand effects cannot be excluded.
Peer-reviewed#
Brynjolfsson, E., Li, D. and Raymond, L. (2023). Generative AI at Work
Quarterly Journal of Economics, 140(2), 889-942. DOI 10.1093/qje/qjae044. Advance Access 4 February 2025. Earlier version NBER Working Paper 31161
- Method
- Staggered rollout of a GPT-3-based conversational assistant across 5,172 customer-support agents in 133 teams at a single Fortune 500 business-process software firm, most of them working from the Philippines. Three million chats observed, 1.2 million of them post-deployment. The system was fine-tuned on past agent conversations, with chats by top performers deliberately up-weighted in training.
- Finding
- Resolutions per hour rose 15 per cent on average, 15.2 per cent in the preferred specification with agent and tenure fixed effects. Less skilled and less experienced workers gained a 30 per cent increase in issues resolved per hour, rising to 36 per cent for the lowest skill quintile, while the most skilled saw no significant productivity change and small declines in conversation quality and customer satisfaction. Customer sentiment improved by half a standard deviation and requests to speak to a manager fell about 25 per cent. During unplanned outages, agents with longer AI exposure still handled chats faster than their pre-AI baseline, but only those who had adhered closely to the suggestions.
- What it supports
- That a model trained on the behaviour of a firm's best workers can transfer measurable parts of that behaviour to its newest ones, immediately and at scale, raising the floor far more than the ceiling. The outage evidence shows some of the gain survives the tool being switched off, conditional on the worker having engaged with it rather than passed it through.
- What it does not support
- Whether those novices became experts. It measures output over months in one firm, one occupation and a stable product environment, and the authors say so. Wages, labour demand and hiring composition were not observed. The outage estimates are the authors' own noisiest, because outages are rare and may not be comparable chats.
Peer-reviewed#
Liang, W., Yuksekgonul, M., Mao, Y., Wu, E. and Zou, J. (2023). GPT detectors are biased against non-native English writers
Patterns, 4(7)
- Method
- Seven widely used GPT detectors evaluated against TOEFL essays by non-native English speakers and essays by US eighth-grade students.
- Finding
- Detectors misclassified more than half of the non-native essays as AI-generated, an average false positive rate of 61.22 percent, while classifying US eighth-grade essays with near-perfect accuracy. The proposed mechanism is that detectors rely on perplexity, and second-language writing is more predictable.
- What it supports
- That AI detection carries a severe and systematic bias against second-language writers, and that the bias is structural rather than a tuning problem.
- What it does not support
- That every detector now on the market performs identically. The study tested tools available at the time, and vendors dispute the generalisation.
Argued perspective#
Lo, L. S. (2023). The CLEAR path: A framework for enhancing information literacy through prompt engineering
The Journal of Academic Librarianship, 49(4), 102720
- Method
- A proposed framework, not an experiment. Sets out five principles for writing prompts, Concise, Logical, Explicit, Adaptive and Reflective, for use in information literacy teaching.
- Finding
- Offers the most widely cited peer-reviewed prompt framework in education. Every element concerns the quality of the instruction given to the model: brevity, order, specificity, iteration and review of the output.
- What it supports
- That the published prompt frameworks optimise the output. Useful here as the comparison case: CLEAR, CO-STAR and Google's TCREI all ask what the model should be told, and none asks what the person should keep doing themselves.
- What it does not support
- That following CLEAR improves learning, or output quality. It is a framework proposal in a library science journal, with no trial behind it, and its author does not claim one.
Argued perspective#
Mollick, E. and Mollick, L. (2023). Assigning AI: Seven Approaches for Students, with Prompts
Wharton School Research Paper, SSRN 4475995, revised September 2023. Also arXiv:2306.10052, June 2023. Not peer-reviewed in either venue
- Method
- A practical framework paper. Seven roles a student or teacher can put an AI into, each with a rationale, an example prompt, an example output, the risks, and guidance. No study, no sample, no measured outcome.
- Finding
- The seven roles are mentor, giving feedback; tutor, giving direct instruction; coach, prompting metacognition; teammate, arguing the other side; student, whom you teach in order to find out what you do not know; simulator, for practice; and tool, for the mechanical part. Each carries a named pedagogical risk, and the one attached to tool is 'outsourcing thinking, rather than work'.
- What it supports
- That there are at least seven distinct uses beyond the answer machine, which is a useful correction to a debate conducted as though there were one. The underlying pedagogical claims rest on established learning science, including Bjork on desirable difficulties and the retrieval-practice literature.
- What it does not support
- That any of it works in this form. The authors say so themselves: the approaches are 'still in their infancy and largely untested' and should be approached 'with a spirit of experimentation'. Established priors applied to an unevaluated delivery mechanism. The prompts are also anchored to mid-2023 model capability and should not be presented as current.
Peer-reviewed#
Sackett, P. R., Zhang, C., Berry, C. M. and Lievens, F. (2023). Revisiting the design of selection systems in light of new findings regarding the validity of widely used predictors
Industrial and Organizational Psychology, 16
- Method
- Meta-analytic re-correction of prior personnel-selection meta-analyses, addressing systematic overcorrection for range restriction.
- Finding
- Work sample validity falls from the widely quoted .54 to .33. Structured interviews fall from .51 to .42 and become the strongest single predictor. Unstructured interviews fall from .38 to .19. General cognitive ability falls from .51 to .31.
- What it supports
- That the numbers most often cited for assessment methods were inflated, and by how much.
- What it does not support
- That these methods do not work. Work samples and structured interviews remain among the strongest predictors available. It also offers no validity data for AI-era assessment formats.
Peer-reviewed#
Shin, M., Kim, J., van Opheusden, B. and Griffiths, T. L. (2023). Superhuman artificial intelligence can improve human decision-making by increasing novelty
Proceedings of the National Academy of Sciences, 120(12), e2214840120, published 21 March 2023. DOI 10.1073/pnas.2214840120
- Method
- 5,857,513 move decisions by professional Go players, 1950 to 2021, from the Games of Go on Disk database. Each human move was scored by KataGo against the program's best counterfactual move (Decision Quality Index: 100 per cent minus the win-rate gap). Novelty: a move creating a sequence never seen earlier in the record, within the first 60 moves. Read from the arXiv version on 15 October 2026.
- Finding
- Human decision quality 'improved comparatively little before the advent of superhuman AI' and rose significantly after it (2016 to 2017). Novel moves became more frequent and occurred earlier, and after the advent novel moves contributed significantly more to the increase in decision quality than non-novel moves. The improvement survived removing moves that matched the AI's choice (about 40 per cent), held in moves 21 to 50, and appeared in responses to opponents' deviations from AI play.
- What it supports
- That a whole profession's measured decision quality rose after a machine surpassed it, and that the rise was not explained by copying the machine alone. The strongest long-run evidence that expert human judgement can improve, not only atrophy, after superhuman AI.
- What it does not support
- Causation: the authors say superhuman AI 'may have' prompted exploration. Decision quality is judged by an AI, so part of the gain may be play converging on machine preferences. Go is a closed game with a near-perfect scorer, unlike most work. Individual AI use is not observed.
Peer-reviewed#
Scarfe, P., Watcham, K., Clarke, A. and Roesch, E. (2024). A real-world test of artificial intelligence infiltration of a university examinations system: A 'Turing Test' case study
PLOS ONE, 19(6), e0305354. DOI 10.1371/journal.pone.0305354. Published 26 June 2024. Open access, data at OSF
- Method
- 33 fake student accounts submitted wholly GPT-4-written answers into the live examinations system of the School of Psychology and Clinical Language Sciences at the University of Reading, summer 2023. Five undergraduate modules across all years, about 5 per cent of submissions on each. Two formats, four 200-word answers in a 2.5-hour window and one 1,500-word essay in an 8-hour window, both unsupervised at home. Markers were staff and trained postgraduates, marking anonymously and entirely unaware of the study.
- Finding
- 94 per cent of the AI submissions were not detected, and 97 per cent went undetected on the stricter test of a marker actually mentioning AI. Across the five modules there was an 83.4 per cent probability, by resampling, that the AI submissions would outscore an equal-sized random draw of real students, an advantage of just over half a classification boundary. AI lost on one module only, a finalist module carrying three AI submissions.
- What it supports
- That wholly machine-written work passed through a real examinations system substantially undetected and scored above the human median, under blind marking, in a live setting rather than a demonstration.
- What it does not support
- How much real cheating occurs, which the authors state plainly: 'we have no way to estimate the proportion of students in our sample who used AI'. It also flatters detection rather than damning it. In their words the 6 per cent detection rate 'likely overestimates our ability to detect real-world use of AI to cheat in exams', because a real student would not take so naively obvious an approach. The comparison group may itself contain AI-assisted work. And the 83.4 per cent is a resampling probability against a median, NOT a count of head-to-head comparisons won, which is how it is usually restated.
Peer-reviewed#
Adinoff, B. and Nunes, J.C. (2025). Challenging the 25-year-old 'mature brain' mythology: implications for the minimum legal age for non-medical cannabis use
The American Journal of Drug and Alcohol Abuse, 51(5), 577-583
- Method
- Review of the neuroscience and policy literature behind the age-25 threshold.
- Finding
- Argues the mature-brain-at-25 claim is not supported by the underlying neuroscience and should not be used as a basis for age thresholds in policy.
- What it supports
- That the challenge to this claim is in the peer-reviewed literature rather than confined to science journalism.
- What it does not support
- Anything about AI, learning or capability. It is cited here for the status of the claim, not for the subject.
Peer-reviewed#
Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakci, O. and Mariman, R. (2025). Generative AI Without Guardrails Can Harm Learning: Evidence from High School Mathematics
Proceedings of the National Academy of Sciences, 122(26), e2422633122, published 25 June 2025. DOI 10.1073/pnas.2422633122. A correction was issued on 20 August 2025, issue dated 26 August, DOI 10.1073/pnas.2518204122, PMID 40833419: a production error had printed the wrong affiliation for Osbert Bastani. It changes no author, no method, no figure and no finding. Recorded here because two search queries reaching this research ask specifically for the correction, and a reader who finds one and does not read it could reasonably assume the result was retracted. It was not. Correction read at source 12 September 2026. On 26 October 2026 the notice's title, DOI (10.1073/pnas.2518204122) and dates (online 20 August 2025, print 26 August 2025, volume 122 issue 34) were re-confirmed from Crossref, which types it as a correction and an update to the original; the notice text itself could not be re-read, PNAS and PubMed both returning bot checks.
- Method
- Field experiment, nearly 1,000 high-school students, three arms: unrestricted GPT-4, a hints-only tutor, and a control.
- Finding
- Grades rose 48 percent with unrestricted access and 127 percent with the tutor while the tool was present. With access removed, the unrestricted group scored 17 percent LOWER than students who never had it. The guardrailed tutor largely removed the harm.
- What it supports
- That the design of the interface, not the presence of AI, decides whether people learn. The single most useful result in this literature.
- What it does not support
- What a guardrailed interface should look like for professional work. It was school mathematics over a bounded period.
Peer-reviewed#
Choi, S., Kang, H., Kim, N. and Kim, J. (2025). How Does Artificial Intelligence Improve Human Decision-Making? Evidence from the AI-Powered Go Program
Accepted for publication in the Strategic Management Journal, per the authors' January 2025 version on arXiv (2310.08704)
- Method
- 749,190 moves (first 30 of each game) by 1,241 professional players in 24,973 major games, 2015 to 2019. Move quality is the gap to the best move according to Leela Zero. Treatment: the public release of Leela in February 2017, the first freely available AI Go program stronger than top players. Read from the arXiv version on 15 October 2026.
- Finding
- After the release the gap to the best move shrank by about 0.76 percentage points (about 30.5 per cent), errors fell about 33.7 per cent and the largest critical mistake about 21.9 per cent. Moves matched the program's top suggestions more often. No pre-release trend; a weaker 2015 program (Crazy Stone) produced no improvement. Younger players gained about 10.9 per cent more than older ones; gains were largest in the opening.
- What it supports
- That the improvement in professional Go followed access to a program players could train with, rather than AlphaGo's 2016 match, which players could watch but not use.
- What it does not support
- Individual use, which is unobserved; there is no clean control group, so the authors treat estimates as lower bounds. Gains concentrated in the opening leave open how much is preparation rather than judgement. Some reported figures are inconsistent between sections of the paper.
Vendor research#
Coursera; foreword by Greg Hart, CEO (2025). Global Skills Report 2025
Coursera, June 2025
- Method
- Platform data from more than 170 million Coursera learners across 100+ countries, covering calendar 2024 enrolments, combined with external indicators including World Bank Human Capital Index and labour force participation data. A new AI Maturity Index draws on IMF and OECD metrics; the exact learner-data window and ranking weights are not published on the page.
- Finding
- GenAI enrolments grew 195% year on year and passed 8 million in total, with 700 GenAI courses averaging 12 enrolments per minute. Latin America recorded 425% year-on-year growth in GenAI enrolments, the highest of any region, and India recorded over 1.3 million GenAI enrolments in 2024. Women are 46% of Coursera's learner base but 32% of GenAI enrolments. Switzerland, the Netherlands, Sweden and Singapore rank first to fourth on skill proficiency; Singapore ranks first in Asia-Pacific on both proficiency and AI maturity.
- What it supports
- That demand for generative AI courses on one large platform rose steeply in 2024 and is unevenly distributed by region and gender.
- What it does not support
- Enrolments are not completions or competence, and platform users are not a population sample. Country rankings measure Coursera learners' performance, not national skill levels. Coursera sells the courses being counted.
Peer-reviewed#
Kestin, G., Miller, K., Klales, A., Milbourne, T. and Ponti, G. (2025). AI tutoring outperforms in-class active learning: an RCT introducing a novel research-based design in an authentic educational setting
Scientific Reports, 15, published 3 June 2025
- Method
- Randomised crossover experiment in Harvard's largest introductory physics course, Fall 2023. Of 233 enrolled students, 194 were eligible on consent and completion. Each student experienced both conditions across two consecutive weeks, one topic taught by in-class active learning and one by a purpose-built AI tutor at home, with pre-tests and post-tests for each. The tutor used GPT-4 with expert-crafted question-specific prompts, pre-written answers, instructional video and a structured scaffold.
- Finding
- Median post-test score 4.5 in the AI condition against 3.5 in the active-learning condition, from a combined pre-test median of 2.75; median learning gain over double. Mann-Whitney z = -5.6, p below 10 to the minus 8. Linear regression effect size 0.63, described by the authors as an underestimate because of a ceiling effect; quantile regression gives 0.73 to 1.3 standard deviations. Median time on task 49 minutes against 60 assumed for the class, with no correlation between time on task and score. Engagement 4.1 against 3.6 and motivation 3.4 against 3.1; enjoyment and growth mindset showed no significant difference.
- What it supports
- That a heavily engineered AI tutor, built by subject experts to follow established pedagogy, can outperform a well-run active-learning class on immediate post-test performance at the understanding, applying and analysing levels, in less time.
- What it does not support
- That a general chatbot does this. Accuracy depended on pre-written answers and instructor-written prompts. Retention was not measured; post-tests followed the lessons immediately. One course, one institution, two topics. The authors state they do not presume the result holds where complex synthesis or higher-order critical thinking is required.
Vendor research#
LinkedIn Learning (2025). 2025 Workplace Learning Report: The rise of career champions
LinkedIn Learning, early 2025
- Method
- Survey of 937 L&D and HR professionals and 679 learners across North America, South America, Asia-Pacific and Europe, plus interviews with talent leaders; fieldwork dates are not stated on the page. Combined with LinkedIn platform data described as 1 billion members, 14 million jobs and 5 million profile updates per minute, current to September 2024.
- Finding
- 49% of L&D professionals agree that their executives are concerned employees do not have the right skills to execute the business strategy. 71% of L&D professionals say they are exploring, experimenting with or integrating AI into their work. Organisations classed as career development champions are 42% more likely to describe themselves as frontrunners in generative AI adoption (51% versus 36% of others), 32% more likely to be deploying AI training programmes and 88% more likely to offer career-enhancing gig opportunities.
- What it supports
- That L&D leaders report a skills concern at executive level and that organisations investing in career development also report earlier generative AI adoption, as an association.
- What it does not support
- It cannot show that career development causes AI adoption; the champion classification and the adoption stage are both self-reported by the same respondents. LinkedIn sells the learning products the report recommends, and the 937-person sample is not described as representative.
Compiled review#
Mansfield, K. L., Ghai, S., Hakman, T., Ballou, N., Vuorre, M. and Przybylski, A. K. (2025). From social media to artificial intelligence: improving research on digital harms in youth
The Lancet Child and Adolescent Health, 9(3), 194-204. Personal View, not primary research
- Method
- Personal View setting out the methodological failures of the social media harms literature and what should be done differently for AI. No new data.
- Finding
- Self-reported screen time is problematic as a measure, being imprecise and prone to bias, and as a construct, being unidimensional, homogenous and of little validity, failing to distinguish social, educational, entertainment, work and informational uses. On AI, the authors write that using self-reported frequency or duration of adolescent AI use as the exposure measure of interest is perhaps even more concerning than counting total time spent on social media, and that only behavioural data on exposure to a range of AI applications would provide the detail needed. They record that health policy decisions have been implemented on inconsistent, non-causal or ungeneralisable evidence of online harms.
- What it supports
- That the researchers who built the screen-time evidence base have published the warning against transplanting its method to AI, in advance and in a clinical journal.
- What it does not support
- Anything empirical. It is a commentary with no new data and no head-to-head methodological comparison. Its prescription is finer-grained measurement of which AI in what context, which is adjacent to but not the same as asking which human practice was displaced. Publisher full texts returned empty bodies; read in the Oxford ORA accepted version.
Peer-reviewed#
Mousley, A. and colleagues (2025). Topological turning points across the human lifespan
Nature Communications, November 2025
- Method
- Diffusion MRI from 3,802 people aged 0 to 90, analysed for structural network topology across the lifespan.
- Finding
- Four topological turning points, at approximately ages 9, 32, 66 and 83, dividing life into five epochs. The adolescent epoch runs from about 9 to about 32. The largest overall shift in trajectory occurs around 32, not in the mid-twenties.
- What it supports
- That structural network reorganisation continues well past the mid-twenties, and that the nearest thing to a boundary at the end of adolescence sits around 32.
- What it does not support
- That 32 is the new 25. The authors describe turning points in network topology, not a moment of cognitive completion, and reading it as a new threshold would repeat the original error with a different number.
Simulation model#
Peterson, A. J. (2025). AI and the problem of knowledge collapse
AI & Society, 40(5), 3249-3269. DOI 10.1007/s00146-024-02173-x. Received 9 July 2024, accepted 17 December 2024, published online 19 January 2025, issue dated June 2025. University of Poitiers. PROVENANCE, and it runs in both directions. The VERSION OF RECORD IS NOT OPEN ACCESS: its abstract, endnotes, references and full Appendix A were read at source on 17 September 2026 and the BODY IS PAYWALLED and was not read. The model detail, the simulation figures and the language-model counts below come from the author's own preprint, arXiv:2404.03502v2 of 22 April 2024, which the published paper names in its own acknowledgements alongside hal-04534111. Anybody citing the arXiv preprint alone is understating it, because it has been peer reviewed and published since; anybody citing the journal version for the model detail should know that detail was read in the preprint.
- Method
- An agent-based simulation plus a small illustrative language-model audit. The simulation: 25 individuals with types drawn from a lognormal distribution, 100 rounds, each round choosing to learn the expensive way, to learn through a discounted AI-assisted process, or not at all. The expensive path samples the true distribution, modelled as a Student's t with 10 degrees of freedom; the AI path samples the same distribution truncated at 0.75 standard deviations either side of the mean in the default setting. Public knowledge is a kernel density estimate over the 100 most recent samples, and distance from the truth is Hellinger distance. Agents update on observed returns at a learning rate of 0.05 and have NO foresight: the paper says they 'cannot foresee the true future value of their innovation options'. Generations turn over every 10 rounds. The audit: four models (GPT-3.5-turbo, Claude-3-sonnet, Gemini-pro, Llama2-70b) asked what human well-being depends on, under five prompt phrasings, against a reference list of 2,693 entities.
- Finding
- THE SIMULATION. 'For our default model, after nine generations, when there is no AI discount the public distribution has a Hellinger distance of just 0.09 from the true distribution. When AI-generated content is 20% cheaper (discount rate is 0.8), the distance increases to 0.22, while a 50% discount increases the distance to 0.40.' Those ratios are the 2.3 and 3.2 in circulation. THE CONDITION THE NUMBER DEPENDS ON: 'If there is no generational change, there is at worst only a reduction in the tails of public knowledge outside the truncation limits. In this case the distribution is stable and does not "collapse".' How often turnover happens, every 3, 5, 10 or 20 rounds, has little effect; whether it happens at all decides the result. Truncation matters as much as price: at two standard deviations 'the effect is minimal', at a quarter of one 'the impact is large'. Faster updating by agents can offset a moderate discount and does not prevent collapse. THE AUDIT: Aristotle drew 4,779 mentions and Martha Nussbaum 1,080, against 83 for Avicenna and Ibn Sadi combined, 62 for Al-Ghazali and 52 for Al-Farabi, while Martin Seligman drew 392; the corpus contains 30,315 uses of the word 'the', so Aristotle appears once for every seven of those. Asking region by region across 34 named regions raised evenness from 0.55 to 0.86 on Pielou's index. DEFINITION, from the published appendix: 'the progressive narrowing over time (or over technological representations) of the set of human working knowledge and the current human epistemic horizon relative to the set of broad historical knowledge.'
- What it supports
- That there is a specified mechanism by which cheap central answers could narrow what a society holds, with the conditions written down and testable, and that the narrowing compounds only across generational turnover. Separately, that at one point in time four models answered a question about human well-being with a heavily concentrated set of names, and that prompting region by region widened it substantially. Peterson also separates his subject from model collapse in his own words: his 'interest is in the inverse of this concern', because 'Humans, unlike LLMs trained by researchers, have agency in deciding among possible inputs.'
- What it does not support
- That knowledge collapse is occurring. Peterson never claims it and the paper is conditional throughout: AI 'can paradoxically harm public understanding', reliance 'could lead to' collapse, dependence 'may lead to' a reduction in the long tails, the concern is 'plausible'. The 2.3 is a simulation output under default parameters and not a measurement of anybody's beliefs. The distribution standing for knowledge is a stated metaphor: 'we make no claim that "truth" is in some deep way distributed 1-D Gaussian', and the truncation device is 'meant to be metaphorical'. The language-model audit measures concentration at one moment and therefore cannot show narrowing over time, which is what the term denotes, and the author says it 'is not intended as a general purpose benchmark'. Its published table of diversity scores carries rows for three of the four models named and none for Llama2-70b, unexplained. The number of responses generated for that corpus is stated nowhere. And the phenomenon may be unobservable in principle: 'we cannot know if the current tails of knowledge are correct or too thin'. The preprint's informal definition, 'the set of information available to humans', is WIDER than the published one, so the two versions do not say the same thing.
Peer-reviewed#
Rasalkar, K., Tripathy, S., Sinha, S., Mukherjee, B., Takkella, N., Dadel, E. V., Sundriyal, M. and Prasad, S. (2025). Enhancing medical assessment strategies: a comparative study between structured, traditional and hybrid viva-voce assessment
BMC Medical Education, 25(1), article 835, 4 June 2025. DOI 10.1186/s12909-025-07428-9. AUTHOR CORRECTION, 19 September 2026: this entry read 'Prasad, S. et al.' from an earlier session. Swetanka Prasad is the EIGHTH and last author, not the first, and the entry id is left unchanged only because it is referenced elsewhere. Checked against the publisher record at Crossref. Third misattribution caught in this research since 4 September 2026, and the first in which the research itself was the source of the error.
- Method
- 151 medical students assessed by two examiners across three viva formats, compared on reliability and perceived fairness.
- Finding
- Traditional unstructured viva showed significant inter-examiner variability. Structured formats improved fairness and coverage. The best format reached a reliability of 0.663, which is moderate rather than high.
- What it supports
- That oral examination can be made fairer by structuring it, and that unstructured viva has a measurable examiner problem.
- What it does not support
- That oral assessment is highly reliable. Even the best format tested was only moderate, at a single institution.
Vendor research#
Udemy Business (2025). 2026 Global Learning & Skills Trends Report
Udemy, September 2025
- Method
- Consumption data from Udemy Business learners at more than 17,000 enterprise customers worldwide, comparing 1 July 2024 to 30 June 2025 with the prior year on total consumption and percentage growth by course topic. Supplemented by an employee survey whose sample size and fieldwork dates are not given in the release.
- Finding
- Udemy reports 11 million GenAI course enrolments to date. Consumption of Microsoft Copilot content rose 3,400% year on year and GitHub Copilot content 13,534%. Consumption of AI ethics and governance courses rose 98%, decision-making 38%, critical thinking 37% and adaptive skills overall 25%. In the accompanying survey, 88% of employees agree effective leadership is critical to their organisation's AI initiatives, but 48% believe their managers are ready for the AI era.
- What it supports
- That enterprise learners on one platform shifted time towards AI tool training and, at the same time, towards judgement and ethics content in 2024 to 2025.
- What it does not support
- Percentage growth from a small base overstates scale (the Copilot figures have no base numbers). Consumption is not skill gain, and customers of a learning vendor are not representative employers. Udemy sells the content.
Simulation model#
Acemoglu, D., Kong, D. and Ozdaglar, A. (2026). AI, Human Cognition and Knowledge Collapse
NBER Working Paper 34910. DOI 10.3386/w34910. The page displays an issue date of February 2026 while its own citation metadata gives 2 March 2026; both are printed here rather than the tidier one. Read at source on 17 September 2026: the abstract and the acknowledgements, which name the Hewlett Foundation, the Stone Foundation and the MIT Gen-AI Consortium as funders, with Acemoglu separately acknowledging Hewlett, Schmidt Sciences and the Smith Richardson Foundation. The full working paper was not read.
- Method
- A dynamic economic model of learning and decision making. Successful decisions need community-level general knowledge and individual context-specific knowledge together, as complements. Costly human effort jointly produces a private signal about a person's own context and a 'thin' public signal that accumulates into the community's stock, which is where the learning externality comes from. Agentic AI supplies context-specific recommendations that substitute for the effort. No data; the results are properties of the model.
- Finding
- 'When human effort is sufficiently elastic and agentic recommendations exceed an accuracy threshold, the economy can tip into a knowledge-collapse steady state in which general knowledge vanishes ultimately, despite high-quality personalized advice.' Two further results carry the policy weight. Welfare is non-monotone in agentic accuracy, so there is an interior welfare-maximising level of precision and a more accurate agent is not always better. And greater capacity to aggregate human-generated general knowledge 'unambiguously raises welfare and increases resilience to knowledge collapse'.
- What it supports
- That the erosion argument can be stated as an externality with a tipping point, in the standard machinery of economics, by authors whose work in this area is taken seriously. The externality framing is the contribution: the individual who takes the agent's recommendation is not making a mistake, and the loss falls on a stock nobody owns.
- What it does not support
- Anything about the world. It is a model and its numbers are properties of its assumptions, the abstract states the tipping result conditionally, and the full paper has not been read here. It is also NOT the origin of the term: Andrew Peterson published 'AI and the Problem of Knowledge Collapse' in April 2024 and in AI & Society in January 2025, and the two papers model different objects. Peterson narrows the diversity of what a population holds; this narrows the public stock of general knowledge through a learning externality. The research's own glossary credited the coinage here until 17 September 2026 and was corrected.
Working paper#
Brosch, H., Langer, C., Lergetporer, P., Mueller, S., Pfeifer, H. and Tschöpe, N. (2026). Task Expansion with Generative AI: Experimental Evidence on Middle-Skilled, Early-Career Workers
Stanford Digital Economy Lab working paper, listed 7 October 2026. Abstract read at source 9 October 2026; the paper itself was not read
- Method
- Field experiment with 673 final-year IT apprentices at twelve German vocational schools. Access to AI was randomly assigned for occupation-specific tasks, some of them beyond the apprentices' formal training.
- Finding
- 'Crucially, productivity increases by 26–31 percentage points on tasks beyond apprentices' formal training, providing direct evidence of AI-enabled task expansion.' 'These productivity gains do not come at the cost of reduced immediate task comprehension.' Gains tended to be larger where performance without AI was lowest and among apprentices with above-median AI literacy.
- What it supports
- That AI let early-career workers complete tasks beyond their training, with no loss of comprehension measured straight afterwards.
- What it does not support
- Anything about skill months later or without the tool: comprehension was measured immediately. The productivity measure is not defined on the page read, and no funding statement appears there. Not peer-reviewed.
Working paper#
Chen, E., Judicke, S., Beigh, K., Tang, X., Wang, I., Yuan, N., Xiao, Z., Li, C., Li, S., Luttmer, R., Singh, S., Yampolsky, M., Parikh, N., Zhao, Y., Chen, M., Huang, S., Mohanty, A., Johnson, G., Mackey, J., Lin, J. and Koedinger, K. (2026). Supporting or Replacing Students' Reasoning? Cognitive Offloading in AI-Assisted Proof Learning
arXiv:2610.31481v1, submitted 22 October 2026. Preprint; no journal reference listed when read on 2 November 2026. The authors describe it as a substantially revised and extended analysis of the classroom study in arXiv:2602.18807
- Method
- Classroom field study in one undergraduate proof-based mathematics course at Carnegie Mellon University (308 enrolled). Of 155 consenting students, 153 had grade and usage records: 75 randomised to early access to the AI tutor (a general chatbot plus a homework-feedback interface requiring a proof attempt first) after Midterm 1, 78 to access after Midterm 2. Homework effect from a mixed model; Midterm 2 effect by OLS with heteroskedasticity-consistent standard errors and Midterm 1 as covariate; usage analyses on 148 students with Midterm 3 as outcome, adjusting for self-efficacy, Midterm 2 and homework-feedback use. The randomised comparison is access timing for both interfaces together. The usage and mediation analyses are observational, because students chose how much to use the tool. The term is not stated, and data collection stopped at Midterm 3 for budget reasons.
- Finding
- While only the early group had access, their homework scores were 2.56 points higher (95% CI 0.17 to 4.94, p = .036), but Midterm 2 showed no reliable difference (B = -2.47, 95% CI -5.60 to 0.66, p = .122). Across the period when everyone had access, each additional chatbot message was associated with a 0.0302-point lower Midterm 3 score (3.02 per 100 messages, 95% CI -5.67 to -0.36 per 100, p = .026). Lower self-efficacy predicted heavier chatbot use, and the indirect path from self-efficacy through chatbot use to Midterm 3 was 0.622 exam points per self-efficacy point (bootstrap 95% CI 0.121 to 1.560). Homework-feedback requests showed no reliable association with Midterm 3 (B = 0.084, p = .131).
- What it supports
- That in this course early access to an AI tutor lifted homework without a detectable exam benefit, and that heavier use of the open chatbot went with lower later exam scores among students who started from lower self-efficacy.
- What it does not support
- That the chatbot caused the lower exam scores: the usage analyses are observational and unmeasured differences could drive them. The authors say offloading is their interpretation, since mental effort and use of AI-generated reasoning were not measured. The feedback interface was not randomised separately, so the contrast with the chatbot is a contrast of associations. One course at one university, consenting students only, and Midterm 2 was an ordinary course exam and not a purpose-built retention test.
Working paper#
Cruces, G., Fernandez Meijide, D., Galiani, S., Galvez, R. and Lombardi, M. (2026). Does generative AI narrow education-based productivity gaps? Evidence from a randomized experiment
NBER Working Paper 34851; CEPR Discussion Paper 21299; arXiv:2608.04198
- Method
- Randomised online experiment, 1,174 adults aged 25 to 45, workplace-style problem-solving task with or without a generative AI assistant, followed by an unassisted module.
- Finding
- AI improved performance for everyone and more for the less educated. Without AI, higher-education participants outperformed lower-education participants by 0.548 standard deviations; with AI the gap fell to 0.139, closing about three-quarters of it. Treated participants did not perform worse once AI was removed, and lower-education participants retained part of their improvement, although a sizeable gap re-emerged.
- What it supports
- That assisted use does not automatically leave people worse off than unassisted controls when the tool is taken away, and that AI can compress an education-based performance gap while it is present.
- What it does not support
- That skill formed. One session with an immediate unassisted module tests transfer within a sitting, not skill formation over time, and the studies that found post-removal deficits taught a body of knowledge and removed the tool afterwards. The equity gain is also partly transient by the authors' own account, since a sizeable gap re-emerges without the assistant.
Working paper#
Eastwood, M., Narne, H., Hilby, J., Denny, P., Aggarwal, A. and Kapoor, A. (2026). Guardrails or Roadblocks? Effects of Pedagogical Style and Context Awareness in AI Teaching Assistants for Programming
arXiv 2609.29995, submitted 24 September 2026. Abstract read at source 28 September 2026
- Method
- Randomised controlled trial with 132 students in an introductory programming course, assigned to one of four AI teaching assistants varying on pedagogical style (Socratic or direct) and contextual awareness (with or without the full problem context).
- Finding
- Students rated the Socratic assistant with full context least favourably; that condition showed more interaction stress, more use of outside language models and less demonstrated comprehension, though not all differences reached statistical significance.
- What it supports
- That a more guarded tutor is not automatically a better one, and that students route around support they find restrictive.
- What it does not support
- Effects on learning outcomes beyond the course; the bypass finding is a pattern rather than a significant result; the paper is a preprint.
Working paper#
Habibullah, A., Alshoibi, Y., Alshiekh, M., Khan, S. and Khan, N. (2026). Where LLM Graders Succeed and Break: Evidence from Two Computer-Science Exams
arXiv 2609.29333, submitted 24 September 2026. Abstract read at source 28 September 2026
- Method
- 171 model configurations run over a computer vision exam sat by 570 students and independently double-marked by two humans, with a replication on a machine learning exam sat by 1,038 students; ablations on prompt wording; a LoRA fine-tuning adapter trained on about 3,900 pooled marked answers.
- Finding
- The best configuration's mean absolute error was 1.64 marks out of 35, below the 2.61 marks by which the two human markers differed from each other. A 'strict grader' preamble made 14 of 17 open-weight models fail badly and three stop grading; the phrase 'never give partial credit' on its own was enough to stop models grading. The failures reproduced on the second exam with exam-specific direction. The adapter brought five small open models to parity with a human marker and reduced prompt sensitivity.
- What it supports
- That on structured computer science exams a language model can mark within the range of human disagreement, and that its accuracy is a property of the prompt and tuning rather than of the model.
- What it does not support
- Anything about essays, proofs, portfolios or reflective writing; whether markers who stop marking lose diagnostic ability; the paper is a preprint and has not been peer-reviewed.
Argued perspective#
Ke, Y., Jin, L., Ong, J.C.L., Thirunavukarasu, A.J., Car, J., Cheung, C.Y., Tham, Y.C., Ting, D.S.W., Ong, M.E.H., Compton, S., Narayan, A., Keane, P.A., Wong, T.Y., Bates, D.W., Tan, P. and Liu, N. (2026). AI-induced never-skilling in medical education
Nature Medicine, 32(6), 1997-2006, published 22 May 2026. DOI 10.1038/s41591-026-04438-y
- Method
- A Perspective, not primary research. Sixteen authors across sixteen institutions argue a three-part taxonomy of how AI can interrupt the formation of clinical competence, and propose an untested three-phase protective framework. The clinical evidence it leans on is borrowed, chiefly Budzyn and colleagues on colonoscopy.
- Finding
- Separates three distinct failures. Deskilling is the degradation of established competence in clinicians already trained. Mis-skilling is the acquisition of incorrect reasoning patterns through uncritical adoption of erroneous or biased AI output. Never-skilling is the failure to form foundational competence during training, when AI substitutes for the cognitive effort that would have built it. The authors predict a state they call false proficiency: competence that appears real but depends on the AI remaining available.
- What it supports
- That the three failures are different problems requiring different responses, and that the entry-level case is not simply deskilling applied to younger people. Never-skilling has no baseline to return to, which is what makes it a distinct category rather than a matter of degree.
- What it does not support
- That never-skilling occurs. The authors disclaim this themselves and do so more than once: 'Direct causal evidence linking AI exposure during training to competency failure in medical trainees does not exist', and the abstract concedes that direct evidence from medical training is absent. It is a risk model, explicitly not an established phenomenon. Prevalence, severity and reversibility are all stated as unknown, and the proposed framework is untested. Cite it for the taxonomy and as the origin of the terms, never as evidence of harm.
Working paper#
Liu, G., Christian, B., Dumbalska, T., Bakker, M. A. and Dubey, R. (2026). AI Assistance Reduces Persistence and Hurts Independent Performance
arXiv:2604.04721, submitted 6 April 2026, revised 5 August 2026 (v4). Preprint, not peer reviewed
- Method
- A series of randomised controlled trials on human-AI interactions, N = 1,222, across mathematical reasoning and reading comprehension. AI assistance is available during a practice phase and then withdrawn, and performance is measured unassisted.
- Finding
- AI assistance improves performance in the short term, and people then perform significantly worse without AI and are more likely to give up. The authors report that these effects emerge after only brief interactions, approximately 10 minutes. They attribute the loss of persistence to AI conditioning people to expect immediate answers, denying them the experience of working through challenges on their own, and note that persistence is one of the strongest predictors of long-term learning.
- What it supports
- That a withdrawal effect can be produced causally, in a randomised design, and that it appears far faster than anyone had assumed. It also moves the mechanism from knowledge to persistence, which is a different and more portable claim.
- What it does not support
- Anything about sustained professional practice. These are short online tasks and the measured effect is a within-session carry-over rather than skill decay, so it cannot show whether the effect compounds, persists beyond the session or transfers to complex work. A preprint, not peer reviewed.
Working paper#
Liu, J., Sweet, T., Chen, M. H., Engelberg, J., Masters, M. C., Clark, M., Persaud, A., Lancaster, A., Hollingsworth, J. K. and Rice, J. K. (2026). The Effects of Course-Integrated AI Tutoring on Student Performance and Engagement: A Randomized University Trial
Annenberg Institute EdWorkingPaper 26-1598, October 2026, manuscript dated 30 September 2026. Abstract and paper read at source 9 October 2026
- Method
- Group-randomised trial at instructor level within course blocks, Fall 2025, University of Maryland, College Park: 2,379 undergraduates and 30 instructors. The intervention was access to a GPT-4o-based virtual study assistant tied to course materials. The headline estimate comes from an exact-match subsample of 22 instructors and 1,353 students.
- Finding
- 'Among sections of the same course, tutor access reduced final grades by 0.37 standard deviations' (4.12 points) 'and learning management system participation by 0.90 standard deviations'. Implied effects were 5.10 points lower for first-generation students and 2.23 for others. 'Only 14.63% of treated students ever used the VSA.' In the full sample 'there is not significant evidence of adverse treatment effects'.
- What it supports
- That giving students access to a university's own AI tutor did not raise grades in this trial and, among sections of the same course, went with lower grades and much lower recorded participation.
- What it does not support
- That using the tutor lowers learning: the effect is of access, under 15 per cent of students used it, the outcome is course grades and not an independent test, and the full-sample estimate is not significant. The provost who commissioned the trial is a co-author. No funding statement. Not peer-reviewed.
Compiled review#
Multiple Freedom of Information investigations: The Times, The Student Eye and The Scotsman (2026). Recorded penalties for AI misuse in UK universities
Three separate FOI investigations covering different institutions and different academic years. They must not be aggregated
- Method
- Freedom of Information requests to universities, compiled by three outlets. The Times to the 24 Russell Group members for 2024-25; The Student Eye to the University of Bristol, May 2025; The Scotsman to Scottish institutions via Miles Briggs MSP, March 2025, covering 2023-24.
- Finding
- Russell Group: 2,053 recorded punishments in 2024-25 against roughly 700 the year before, from about 350,000 students, with four members disclosing expulsions (UCL, Imperial, Glasgow and Leeds). Bristol: 526 penalties in 2023-24, against 153 the previous year and 7 in 2021-22. Scotland: 1,051 cases in 2023-24 against 131 in 2022-23, of which Abertay alone recorded 351 cases with 342 upheld.
- What it supports
- That formal academic penalties for AI misuse have risen steeply and are now numbered in thousands, and that a student who assumes this is theoretical is wrong.
- What it does not support
- A national picture, a trend in behaviour, or one dataset. Seven of the 24 Russell Group universities do not record AI investigations at all, so 2,053 is a floor across an incomplete sample. The three sets cover DIFFERENT YEARS, 2024-25 for the Russell Group and 2023-24 for Scotland and Bristol, and they overlap institutionally, so they cannot be added together. Most importantly the universities' own position is that much of the rise reflects newly created recording categories rather than more cheating: Abertay introduced 'unacceptable AI use' as a category only in 2023 and accounts for a third of the Scottish total. Cases, penalties and upheld findings are three different counts and are routinely conflated in coverage.
Working paper#
Northcutt, C., Hasmani, I., Feng, K., Khangi, T., Plesner, A. and Mueller, J. (Handshake AI Research) (2026). StudentBench: AI and human tutoring yield equivalent GRE learning gains
arXiv 2609.28470, submitted 23 September 2026. Preprint, not peer reviewed. Handshake AI funded the work. Read at source 24 September 2026
- Method
- Randomised controlled trial with 2,383 adult participants recruited from Handshake's own platform, mostly aged 18 to 23, from about 70,000 invited. Three arms: one hour of text-based AI tutoring (thirteen models tested, with lesson planning, conversational tutoring and generated practice problems), one hour with an expert human GRE tutor on live video, or one hour of educational videos as control. A 27-question GRE pre-test, then a post-test of different questions with no tutor present. Equivalence tested by two one-sided tests with bounds of plus or minus 0.25 pooled standard deviations, about 4.09 percentage points. A secondary analysis of 2,028 pairwise expert evaluations.
- Finding
- Against control, AI tutoring added 6.86 percentage points on the quantitative section (95 per cent CI 4.02 to 9.69) and 5.47 on the verbal (2.46 to 8.47). AI and human tutoring gains were statistically equivalent within the preset bounds, p = .015. In five of seven GRE domains the best-performing AI tutor exceeded the human tutors on average. On the authors' costing, with human tutors at 75 dollars an hour, the smallest model to reach equivalence did so at 918 times lower cost per percentage point of gain (0.0052 dollars against 4.81). Faster AI responses correlated with more student messages, more correct practice and larger gains.
- What it supports
- That in one large randomised trial, one hour of AI tutoring produced immediate test gains on standardised questions indistinguishable, within a quarter of a standard deviation, from one hour with a paid expert human tutor, and that the result held across several models.
- What it does not support
- Anything about retention or transfer: the post-test was immediate and the authors write that they 'measured immediate learning gains without evaluating whether they persist over months'. There was no arm in which students worked through practice problems alone, so the gain either tutor adds over self-study is unknown. Participants were paid adults literate in English on one platform; the human tutors were the ones the company recruited; the funder supplies human experts to AI developers. Equivalence on average is not identity, and a GRE item is one kind of learning.
Working paper#
Poudyal, B. (2026). Mapping the Authorized Boundary: A Comparative Policy-Vignette Study of Generative AI Governance in Australian Higher Education
arXiv 2609.29689, submitted 30 August 2026 and listed 25 September 2026. Abstract read at source 28 September 2026
- Method
- Fifteen standard vignettes of student generative AI use tested against the published policies of 20 Australian universities, 300 university-case classifications, coded twice, distinguishing binding instruments from the whole official policy environment.
- Finding
- 120 combinations (40.0 per cent) clearly prohibited, 97 (32.3 per cent) potential breaches, 27 (9.0 per cent) permitted with conditions and 56 (18.7 per cent) indeterminate; none clearly permitted. Coder agreement 57.3 per cent, Cohen's kappa 0.395. Written policies regulate retention of AI-generated text more clearly than process-only assistance.
- What it supports
- That the written rules of twenty universities cannot be read consistently even by trained coders, and say least about the uses students make most.
- What it does not support
- What happens in tutorials and departments, where indeterminate cases may be resolved; anything outside Australia; the paper is a preprint.
Working paper#
Sankaranarayanan, S. (2026). Mitigating "Epistemic Debt" in Generative AI-Scaffolded Novice Programming using Metacognitive Scripts
arXiv 2602.20206, 22 February 2026, revised 31 March 2026
- Method
- Between-subjects experiment, 78 participants recruited via Prolific and UserInterviews.com, using a custom Cursor IDE plugin backed by Claude 3.5 Sonnet. Three conditions: manual control, unrestricted AI, and scaffolded AI. Followed by a 30-minute AI-blackout maintenance task. Preprint, not peer reviewed.
- Finding
- Both AI groups outperformed the manual control on functional utility (p < .001) and did not differ from each other (p = .64). On the subsequent AI-blackout maintenance task, unrestricted AI users failed at 77 per cent against 39 per cent for the scaffolded group. The author describes "fragile experts": developers whose high functional utility masks critically low corrective competence.
- What it supports
- That the gap between producing output and being able to repair it can be created inside a single session, and that interface design changes the size of that gap. The scaffolded condition roughly halved the failure rate.
- What it does not support
- A durable effect on capability. It is a single session with a single blackout task, in novice programming, with no longitudinal follow-up. The author puts "epistemic debt" in quotation marks in his own title and builds explicitly on Kirschner, so it is not offered as a new construct.
Institutional survey#
Schwartz, H. L. and Diliberti, M. K., RAND (2026). More Students Use AI for Homework, and More Believe It Harms Critical Thinking: Selected Findings from the American Youth Panel
RAND research report RRA4742-1, 17 March 2026; discussed in a RAND interview with Diliberti published 30 September 2026. Both read at source 2 October 2026
- Method
- RAND American Youth Panel survey of 1,214 young people aged 12 to 29, fielded in December 2025.
- Finding
- 67 per cent of students endorsed the statement 'The more students use AI for their schoolwork, the more it will harm their critical thinking skills.' With the exception of using AI to get answers for homework, most did not regard their school-related uses as cheating. In the September interview Diliberti said only about a third of students reported schoolwide rules on AI and that many did not know.
- What it supports
- That most US students surveyed use AI while believing it harms their thinking, and that few report clear school rules.
- What it does not support
- Any effect on learning: these are self-reports. The exact share reporting schoolwide rules was not visible on the report page read. United States only.
Working paper#
Shakiba, A. (2026). When AI Can Build It, What Should We Assess? Rethinking Evidence of Engineering Competence
arXiv 2610.36413, submitted 26 October 2026. Abstract read at source 30 October 2026
- Method
- An account of a 13-week engineering course that permitted AI, with 87 students in 17 teams and 957 individual repositories, using an engineering accountability workflow and six evidence channels linked by collect, cross-check and escalate logic. A single course, not peer reviewed.
- Finding
- Students could hand production to AI but stayed responsible for planning, verification, review, integration and explanation. AI changes which rubric criteria carry evidential weight. 'Automation makes evidence collection easier to scale than evidence interpretation.'
- What it supports
- That one course has tried assessing how students checked and explained AI-produced work rather than the work itself, and found the reading of evidence, not its collection, to be the constraint.
- What it does not support
- Learning outcomes against a comparison group, or that the approach transfers beyond one course.
Working paper#
Shen, J. H. and Tamkin, A. (2026). How AI Impacts Skill Formation
arXiv:2601.20245v1 [cs.CY], submitted 28 January 2026, CC BY-NC-ND 4.0. Pre-registered at osf.io/pk6a5; code and annotations at github.com/safety-research/how-ai-impacts-skill-formation. Work conducted under the Anthropic Fellows Program, and summarised by Anthropic on 29 January 2026 at anthropic.com/research/AI-assistance-coding-skills. Preprint, not peer reviewed. Read at source 14 September 2026
- Method
- Between-subjects randomised experiment, 52 participants, 26 per arm, learning Trio, a Python library for asynchronous programming none of them had used, through a self-guided tutorial with starter code and a concept brief. Maximum 35 minutes on two coding tasks, then a 14-question, 27-point quiz taken without AI, covering debugging, code reading and conceptual understanding; code-writing questions were excluded by design to keep syntax errors out of the score. The treatment arm had an AI assistant in the sidebar, based on GPT-4o, with access to the participant's code and able to produce the full correct solution on request. Recruitment was through a third-party crowd-worker platform at a flat fee of 150 US dollars, screened for more than a year of weekly Python use, some prior exposure to AI coding assistance, and no prior Trio use. Screen recordings of 51 participants were annotated by hand. Four pilots preceded the main study.
- Finding
- The assisted arm averaged 50 per cent on the quiz against 67 per cent for the hand-coding arm, a difference of 4.15 points on 27, Cohen's d of 0.738 at p = 0.010, holding at d = 0.725, p = 0.016 with warm-up time as a covariate. The widest gap was on debugging questions and the narrowest on code reading; the authors' explanation is that the control arm hit more errors and got better at resolving them, and the error counts support it, with a median of 1.0 total errors in the AI arm against 3.0 without. The AI arm finished about two minutes faster and that difference was not statistically significant. Inside the treatment arm, six annotated interaction patterns split cleanly: wholesale delegation (n=4), progressive reliance (n=4) and iterative AI debugging (n=4) averaged under 40 per cent, while generation-then-comprehension (n=2), hybrid code-and-explanation (n=3) and conceptual inquiry (n=7) averaged 65 per cent or higher. Explanations were the most common query type at 79, asked by 21 of 25; code generation accounted for 51 queries from 16 of 25, and four of those participants asked for nothing else.
- What it supports
- That withdrawing the assistance and testing unaided produces a measurable comprehension penalty in professional developers meeting unfamiliar material, on a pre-registered primary outcome. The interaction-pattern split is the most useful thing in it, and points at how the tool is used rather than whether it is used, which is where the guardrailed arms of the school and programming trials arrived from the other direction.
- What it does not support
- That AI use degrades professional expertise. One library, one tutorial, one hour, a preprint, and a sample of 52. The interaction-pattern analysis was NOT pre-registered and the authors state that it draws no causal link, so it cannot be quoted as evidence that asking for explanations causes better retention. The assistant was a chat sidebar, and the authors expect an agentic tool to produce a larger effect, so this is a floor. Whether an immediate quiz score predicts longer-term skill is, in their words, an important question the study does not resolve. The conflict of interest runs against the finding rather than towards it, since the work was done under an AI company's fellowship and reports harm, but it is disclosed here either way. ONE DESCRIPTION IN CIRCULATION DOES NOT COME FROM THE PAPER. Anthropic's own write-up calls the participants '52 (mostly junior) software engineers' and secondary coverage repeats it; the paper's balance table reports 15 of 26 in the treatment arm and 14 of 26 in the control at seven or more years of experience, and two per arm at one to three years. The result is therefore about developers meeting unfamiliar material, not about juniors, which widens it rather than weakening it. Checked at both sources on 14 September 2026.
Institutional survey#
Stephenson, R. and Armstrong, C. (2026). Student Generative AI Survey 2026
HEPI Report 199, Higher Education Policy Institute with Kortext, published 12 March 2026. The third annual edition; the 2024 and 2025 editions were written by Josh Freeman
- Method
- 1,054 full-time undergraduates polled through the Savanta panel in December 2025, weighted on gender, institution type and year of study, margin of error about 3 per cent. The 2024 wave was polled by UCAS rather than Savanta, a change inside the trend line.
- Finding
- 95 per cent report using AI in at least one way and 94 per cent say they have used generative AI to help prepare assessed work, against 89 per cent in 2025 and about 53 per cent in 2024. What they use it for is mostly comprehension: explaining concepts 61 per cent, summarising an article 49, suggesting research ideas 40, structuring thoughts 39. Including AI-generated text directly in assessed work is 12 per cent, up from 8 in 2025 and 3 in 2024. 68 per cent think AI skills are essential, and only 36 per cent feel encouraged by their institution to use AI.
- What it supports
- That AI use among full-time UK undergraduates is close to universal in preparation, and that institutional encouragement lags a long way behind student behaviour.
- What it does not support
- That 94 per cent of students put AI into work that gets marked. THIS IS THE MISREADING THE FIGURE INVITES and the distinction is the whole point of the survey: the 94 covers everything from explaining a concept to drafting, and the number for including AI-generated text directly is 12. Anyone writing 'uses AI on assessed work' has converted a 12 per cent finding into a 94 per cent one with one preposition. It is also self-report, an ever-done lifetime measure rather than current practice, a commercial panel rather than a census, full-time students only, and HEPI warn that new response options were added in 2026 which mechanically makes the 94 easier to reach.
Working paper#
Stromberg, D., Lei, V. and Wu, Y. (2026). The Generative AI Learning Penalty: Evidence from Chinese Secondary Education
CEPR Discussion Paper 21577, CEPR Press, published 2 June 2026. Working paper, not peer reviewed
- Method
- Thirty months of panel data on 26,811 Chinese students in grades 7 to 12, combining monthly closed-book exams, high-school and college entrance exams, and homework scores and completion time across nine subjects. Staggered AI adoption in a difference-in-differences design. The outcome measures are closed-book and invigilated, so they record unaided performance by construction.
- Finding
- AI adoption raises homework scores by 18 per cent and reduces completion time by 30 per cent, and lowers monthly exam scores by 20 per cent within six months. High-stakes entrance-exam scores fall by 18 and 24 per cent, with the full penalty emerging only after about two years. Losses are largest in social science, then STEM, then languages, and are especially large for junior students, high-achieving students and boys. They concentrate among roughly 80 per cent of AI users whose behaviour is consistent with homework outsourcing, indicated by very short completion time coupled with high homework scores. Users who maintain similar completion time to non-users experience small losses.
- What it supports
- That the gap between assisted output and unaided capability can be measured at scale over years rather than minutes, and that it widens rather than closing. It also locates the damage in delegation rather than in access: the students who kept working at their normal pace were largely spared.
- What it does not support
- Causation with the confidence of a randomised trial. Adoption is self-selected and staggered rather than assigned, the outsourcing split is inferred from time-on-homework rather than observed, and a two-year lag makes contemporaneous confounds harder to exclude. One country, one school system, secondary students. Nothing about professional work. Not peer reviewed.
Institutional survey#
Tench, B., Weinstein, E. and James, C., with Starks, A. and Konrath, S. (2026). An AI Policy Isn't a Playbook: Educators Need Agency, Not Just New Rules
Center for Digital Thriving, Harvard Graduate School of Education, September 2026. Report read at source 2 October 2026
- Method
- Survey of 719 teachers and 299 principals through the RAND American Educator Panels, April to May 2025; interviews with 12 educators and with 31 young people aged 15 to 19, April to June 2026.
- Finding
- Of the AI dilemmas educators described in their own words, 74 per cent of teacher descriptions (171) and 69 per cent of principal descriptions (85) were coded as cheating-related. Students reported being falsely accused, and some 'have begun to cap their effort, or perform mediocrity on purpose to not get flagged'.
- What it supports
- That cheating dominates how US educators describe their AI problem, and that some students say they respond to detection by doing worse work on purpose.
- What it does not support
- How common deliberate under-performance is: it rests on 31 interviews. The percentages are shares of the dilemmas described, not of all educators surveyed. United States only.
Peer-reviewed#
Topaz, M., Roguin, N., Gupta, P., Zhang, Z. and Peltonen, L.-M. (2026). Fabricated citations: an audit across 2.5 million biomedical papers
The Lancet, 407, 1779-1781. DOI 10.1016/S0140-6736(26)00603-3. Online 7 May 2026. Correspondence rather than a full article. Led from Columbia University School of Nursing and Data Science Institute
- Method
- The CITADEL pipeline over 2.5 million papers in the PubMed Central Open Access collection, January 2023 to 18 February 2026, extracting 125.6 million references of which 97.1 million carried a verifiable identifier. Title and identifier were compared; the 30,812 mismatches were routed by a language model into categories, then each flagged title was searched in PubMed, Crossref, OpenAlex and Google Scholar. A reference counted as fabricated only if it appeared in none of them.
- Finding
- 4,046 fabricated references across 2,810 papers. The rate of papers carrying at least one rose from 1 in 2,828 in 2023 to 1 in 458 in 2025 and 1 in 277 in the first seven weeks of 2026, a twelvefold increase. Review articles ran 57 per cent higher than other paper types. At the time of the audit 98.4 per cent of affected papers had received no publisher action.
- What it supports
- That references to studies which do not exist are entering the peer-reviewed literature at a rising rate, measured rather than asserted, and that the correction machinery is not catching them.
- What it does not support
- A rate for PubMed. The audit covers the PubMed Central OPEN ACCESS collection, and Nature published a correction to its own coverage on 13 May 2026 making exactly this distinction. The 2026 figure is seven weeks and the authors mark it as an incomplete observation period. It is a floor rather than a ceiling, because 28.5 million references, 22.7 per cent, lacked verifiable identifiers and were excluded. It does not establish intent: Topaz has said 91 per cent of affected papers carried only one or two, many likely honest mistakes by authors who did not check AI output. Cochrane and Northwestern have both criticised the method publicly, on transparency and on not separating citations that matter to a conclusion from those that do not. NOTE, the paper itself could not be read at source, so no limitation is quoted here as the authors' own words.
Where this evidence is used
The research that rests most on this section:
- How is judgement trained?
- How humans learn with AI
- Should there be AI-free periods at work?
- Is screen time the same argument as AI use?
- Is an AI tutor as good as a human tutor?
- Can you regain a skill you have lost?
- Does the brain mature at 25?
- What happens to medical and legal training?
The other sections
- Judgement, thinking and offloading 79
- Human and AI collaboration 44
- Empathy, creativity and what stays human 36
- Work, jobs and the labour market 94
- The seven human capabilities 38
- The institutional record 180
- Professions and sectors 25
- Frontline, clinical and physical work 24
- The international picture 29
The whole base is also machine-readable at evidence.json under CC BY 4.0.
Reference · SS-2026-449 · How I use AI here
Cite this page
Hirji, R. (2026). Learning, practice and expertise. The SuperSkills evidence base, SS-2026-449. https://thesuperskills.com/research/evidence/learning-and-expertise. Last reviewed 3 November 2026.
An evidence review by Rahim Hirji, not peer-reviewed research. For a material claim, cite the underlying study as well; every study here carries its own permanent link.
How citations and IDs work