When the credit AI declines a qualified applicant. When the medical AI misses a diagnosis that a physician would have caught. When the hiring AI filters out candidates who would have been interviewed under the prior process. Who made the decision? Who is accountable for its consequences? The AI produced an output. The organization deployed the AI. The governance program approved the deployment. The affected individual experienced a consequence. The accountability chain from output to consequence is the governance design question most AI programs have not answered.
Why the Question Is Harder Than It Sounds
Accountability for AI decisions is complicated by the distributed nature of the decisions that produce a consequential AI output. The data scientists who built the model made choices about training data, model architecture, and optimization targets. The product team that deployed the model made choices about the deployment context, the decision threshold, and whether human review would be required. The business leader who commissioned the system made choices about its scope and its integration into consequential workflows. The procurement team that selected the AI vendor made choices about which system's outputs the organization would rely on. The AI produced the output. A chain of human decisions produced the AI that produced the output.
Attributing accountability in this chain is genuinely complex. The model's output was determined by choices that were made at multiple points by multiple people, many of whom were not present at the moment the consequential output was produced and many of whom did not anticipate the specific scenario in which the output caused harm. Clear accountability in a chain of distributed decisions requires that accountability was explicitly assigned at each decision point, not reconstructed after the fact.
AI accountability cannot be assigned retrospectively. It must be designed prospectively — at the moment each consequential design and deployment decision is made — so that when the system produces a harmful output, the accountability chain already exists.
The Three Accountability Gaps
The Gap Between Model Output and Human Decision
Many AI deployments operate in a human-in-the-loop model where the AI produces a recommendation and a human makes the final decision. This model is supposed to allocate accountability clearly: the human who makes the final decision is accountable for the decision. In practice, the allocation is more complicated. When the AI recommendation is presented to the human decision-maker without explanation, the human who confirms the AI's recommendation is rubber-stamping rather than deciding. When the workflow is designed to make overriding the AI's recommendation difficult — requiring documented justification, additional approval, or flagging the override in performance metrics — the structural pressure tilts toward the AI's recommendation. The human is nominally accountable for a decision the system was designed to make.
The Gap Between Deployment Decision and Harm
The people who approved the AI system's deployment — the governance committee, the executive sponsor, the board — made a decision at a point in time that the system was acceptable for deployment. They did not make a decision that each specific output the system would produce was acceptable. When a specific output causes harm, the deployment decision is the retrospective accountability target. Whether the deployment decision-makers understood the risk of the specific harm that occurred, whether that risk was disclosed to them before approval, and whether the approval process included the authority to condition deployment on specific safeguards are the governance design questions that determine whether the accountability attribution is fair.
The Gap Between AI Vendor and AI User
Foundation model vendors increasingly disclaim responsibility for outputs produced by applications built on their models. The organization that builds an application on a foundation model has accepted responsibility for the application's outputs through the vendor's terms of service. The foundation model vendor produced the underlying capability. The organization deployed it in a context the vendor may not have anticipated. The organization integrated it into workflows that produce consequential outputs. The accountability for the consequential output sits with the organization, regardless of whether the foundation model's behavior contributed to it.
Designing Accountability Into the AI Program
Accountability for AI decisions requires that accountability is assigned at each decision point in the AI lifecycle — and that the assignment is documented, not assumed. The data scientists who define the training data are accountable for the governance of that data. The product team that sets the decision threshold is accountable for the implications of that threshold for false positive and false negative rates and their consequences. The executive who approves deployment is accountable for the risk characterization presented at approval and for the conditions under which deployment was approved. The operations team that monitors the system is accountable for detecting when performance has drifted from the approved baseline.
This accountability chain requires documentation: who made each decision, based on what information, under what governance process, with what conditions attached. When harm occurs, the chain produces a factual account of the decisions that produced the harmful output. Without the chain, accountability is attributed based on who is most visibly associated with the system rather than on who made the decisions that produced the harm.
The EU AI Act's requirements for high-risk AI systems include human oversight, transparency, and record-keeping obligations that create the factual basis for accountability attribution. These requirements are governance design requirements. Organizations that implement them as documentation exercises produce records. Organizations that implement them as accountability design requirements produce systems that can demonstrate who was responsible for what.
Design the accountability chain before the system is deployed. The harmful output that reveals a gap in the chain is an expensive moment to design it.
Name the accountable person for each AI decision point before deployment. When the system gets it wrong, the accountability chain already exists. That is the governance standard.
Enterprise practitioner perspective. Not legal advice. Part of the Deep Trust Governance Series by Verisq. Get the free weekly Breach Digest.