What is an after action review?

An after action review is a structured discussion held soon after a piece of work, covering what was intended, what happened, why the two differ, and what to do differently. Across 46 studies, properly conducted debriefs improved subsequent performance by around 20 to 25 per cent. It is the best-evidenced judgement routine available, and the qualifier in that sentence carries most of the weight.

Where it comes from#

The form was developed by the US Army in the 1970s and spread to aviation, emergency medicine and surgery. The military version is deliberately unranked: the review is led by a facilitator rather than by the senior officer present, and the point is to establish what happened rather than to allocate blame. That design decision is not decoration, and the organisations that adopt the format while keeping the hierarchy tend to get the document without the effect.

What the evidence says#

Tannenbaum and Cerasoli's 2013 meta-analysis covered 46 studies of debriefs for individuals and teams, in field and simulated settings. Performance improved by around a fifth against control, with an average effect size near 0.67. The analysis also identified what the effect depends on: an explicit structure, active self-discovery by participants rather than being told the lesson, a focus on the process rather than the outcome, and multiple sources of information about what actually occurred.

That last condition is the one most often missing. A review conducted from a single person's memory of a complicated event is reviewing a reconstruction, and the reconstruction is already organised around how it turned out.

Why process rather than outcome#

Because the evidence on feedback says outcome information teaches very little. Balzer's review found that being told whether a judgement was right produced little improvement and sometimes made things worse, while being shown how you weighted the available information produced improvement. A review that asks whether the project succeeded is supplying outcome feedback. A review that asks what each person believed at the time, what they were weighing, and what they would have needed to see to decide otherwise is supplying something closer to the kind that works.

This is also why the review is worth much more when a decision journal exists. Without a record made in advance, the question of what you thought at the time cannot be answered honestly by anybody.

The four questions#

• What did we intend to happen?
• What actually happened, according to more than one person?
• Why do those differ?
• What do we sustain and what do we change?

Two conditions turn that list into the thing the research measured. It happens soon enough that memory has not reorganised, and stating a mistake carries no cost for the person stating it. Where the second is untrue, the review will produce agreement instead of information, and agreement is what a review that never changes anything looks like from inside.

What this has to do with AI#

A debrief needs a decision to review. Where the first draft, the shortlist or the recommendation comes from a machine and the person edits it, there is often no moment at which anybody committed to a view, so there is nothing for the review to examine. Adding the question of what the tool proposed, what was changed and why, is the smallest useful adaptation, the same record asked for by the shared prompt review asks for.

What this does not establish#

That holding reviews improves judgement in general. The meta-analysis measured performance on subsequent tasks, much of it in training and simulation settings with clear outcomes, which is a friendlier environment than most professional work. Nothing here shows the effect holds where results arrive years later or never resolve. And the figure applies to debriefs conducted to a method: a meeting called a retrospective with none of the four conditions present is not the intervention that was measured.

Key sources

• Tannenbaum, S. I. and Cerasoli, C. P. (2013). Do team and individual debriefs enhance performance? A meta-analysis. Human Factors, 55(1).
• Balzer, W. K., Doherty, M. E. and O'Connor, R. (1989). Effects of cognitive feedback on performance. Psychological Bulletin, 106(3).

Related SuperSkills research#

• How is judgement trained
• What is a decision journal
• Kind and wicked learning environments