What is decision provenance?

Decision provenance is a record of the pipeline behind an automated decision: what fed into it, what was decided, and what that decision then set in motion elsewhere. The term belongs to Jatinder Singh, Jennifer Cobbe and Chris Norval, who proposed it in IEEE Access in January 2019, and the move they made was to take provenance, a well-established idea about tracing where data came from, and point it at decisions instead.

Why an established idea needed a new name#

Provenance was already a working concept in databases and scientific computing, where it answers a narrow question: which sources and transformations produced this value. Singh, Cobbe and Norval were looking at something bigger, what they call systems-of-systems, where the information flows that drive a decision cross technical and organisational boundaries and become, in their word, opaque. Their argument is that the opacity people complain about in algorithmic decision-making often sits in those flows rather than in the model, and that the flows are the tractable part.

They name five things provenance of this kind is meant to support: oversight, audit, compliance, risk mitigation and user empowerment. Their paper is a proposal with an implementation agenda attached and it reports no new data. It was peer reviewed in IEEE Access, volume 7, pages 6562 to 6574, and it carries the weight of an argument in a reviewed venue rather than of a measurement.

Not the same thing as an explanation#

The distinction that gets lost is between provenance and explainability. An explanation attempts to say why a model produced the output it produced. It arrives after the fact, usually from a second system whose account of the first cannot be independently checked. Provenance is a record of what actually flowed: which data entered, which version of which component acted on it, which human touched it and when, what the decision triggered downstream. It makes no claim about reasoning at all.

That narrowness is the point. Provenance is verifiable in a way an explanation is not, because it is a log of events rather than a story about them. An organisation that cannot say why its model scored somebody a four can still say, with evidence, what went in, who reviewed it, what changed afterwards and who was affected.

Where the regulation has already arrived#

Two instruments in this estate's evidence base ask for something close to this without using the term. Article 12 of the EU AI Act requires high-risk systems to be built so that events are automatically recorded across the system lifetime, with providers keeping the logs for at least six months under Article 19. Article 14 goes further in a different direction: the people overseeing such a system must be able to interpret its output, to remain alert to their own tendency to over-rely on it, and to disregard, override or reverse it.

The two articles are wired together in the text rather than merely adjacent in it. Article 12(3)(d), for the biometric systems in Annex III point 1(a), requires the logs to include "the identification of the natural persons involved in the verification of the results, as referred to in Article 14(5)". The record is required to name the human overseer. Exercising the Article 14 power in the first place means being able to reconstruct how a decision was reached, which is what Article 12 records and what Singh, Cobbe and Norval were writing about five years earlier. New York City's Local Law 144 makes the same dependency visible from the other end: its rules define the trigger for a bias audit partly as using a simplified output to overrule conclusions derived from other factors, including human decision-making. Identifying when that has happened requires a trail.

The connection to capability#

A provenance record is a necessary condition for oversight and nowhere near a sufficient one. Somebody has to be able to read it and to disagree with what it shows, and that is a capability question rather than an engineering one. An organisation can hold a complete audit trail and still have nobody left who could form an independent view of whether the decision at the end of it was right, which is the arrangement described at the moral crumple zone: accountability assigned to a person who has been placed beyond the point of being able to exercise it.

So the useful test for a board is two-part. Can we reconstruct how this decision was reached, and is there somebody here who could have reached a different one. The first is decision provenance. The second is what this research spends most of its time on, and the two fail independently.

Key sources

• Singh, J., Cobbe, J. and Norval, C. (2019). Decision Provenance: Harnessing Data Flow for Accountable Systems. IEEE Access, 7, 6562 to 6574. DOI 10.1109/ACCESS.2018.2887201, published 16 January 2019, open access under CC BY. The arXiv record's own comment field gives the volume as 9, which is wrong; the publisher record and the authors' institutional repository both give volume 7, and IEEE Access volume 9 is 2021. Graded entry.
• European Parliament and Council (2024). Article 12, record-keeping, Regulation (EU) 2024/1689. Graded entry.
• European Parliament and Council (2024). Article 14, human oversight, Regulation (EU) 2024/1689. Graded entry.
• Council of the City of New York (2021). Local Law 144, automated employment decision tools. Graded entry.

Related SuperSkills research#

The oversight standard this supports is meaningful human oversight, and the practical version is how to audit an AI-assisted decision. On who carries the check, who owns verification and who can override an AI system. On the failure this is meant to prevent, the moral crumple zone and why human in the loop is not a safeguard. On keeping the record honest about its own sources, the source rule.