
When an AI agent acts, who is accountable?
Delegating execution to an AI agent requires clear boundaries around authority, intervention, and evidence: who authorized the commitment, and who is accountable for its consequences?
A supplier receives an order confirmation and begins preparing it. So far, nothing seems unusual. The purchase meets a real need, the price is within budget, and the delivery date will prevent a delay. Suppose, however, that the confirmation came from an artificial intelligence agent tasked with finding alternative sources of supply. To complete the transaction, it also informed the previous supplier that its services were no longer needed and changed the production schedule. This is a hypothetical situation, but it lets us examine the problem without invoking a catastrophe: even when the solution seems sensible, we still need to ask who had the authority to approve every commitment it involved.
Searching for an alternative and committing the company are not simply different degrees of the same task. Between them lies a decision about resources and relationships that other people will have to accept, honor, or correct. When that step remains implicit, sound execution can conceal poorly defined delegation.
By agent, I mean an AI system that can string together actions through tools and pursue an objective with some latitude to execute. What matters here is what it can do on an organization’s behalf without further human intervention. The question arises before any model failure: it begins when someone grants the system access to operations whose consequences extend beyond the system itself.
What we are actually authorizing
The order in our example involves several possible decisions. Procurement may have authority to choose a supplier but not to change a production priority. Whoever manages the budget may approve spending without being entitled to cancel an existing agreement. The agent could have technical access to all these operations and receive an instruction that makes them means to the same end. No reasoning error would be needed for it to carry out a sequence that the person giving the instruction was not authorized to decide alone. The problem would lie in how permissions and purpose were combined with the authority of the person who assigned the task.
This distinction already has a recognizable technical expression. In February 2026, NIST’s National Cybersecurity Center of Excellence presented a concept paper on identity and authorization for software and AI agents (opens in a new tab), raising questions about identification, access, and auditability. It was a proposed project, not a binding standard. These controls address part of the problem; applying them to our purchase still requires deciding which actions should be permitted and under what conditions. The fact that an account can use a function does not explain what it should be allowed to use that function for.
We could allow the agent to collect quotes and prepare a comparison, leaving the purchase pending a later decision. We could also let it confirm orders within a known scope without seeking approval for every transaction. Useful autonomy consists precisely in removing interventions that add no judgment. But that scope must represent the commitment the company is willing to make: perhaps the agent may replenish supplies from an approved vendor, within an allocated budget, provided it does not alter a promised delivery. If it finds an alternative that requires changing any of those conditions, it has reached the limit of its mandate, even if proceeding would better achieve the original objective.
That limit may also move while work is under way. Production might withdraw the need that prompted the order, or procurement might suspend a supplier. The design must therefore account for how authorization is updated or revoked, and what happens to a transaction started under conditions that no longer apply.
Some commitments, moreover, can be understood only by looking at several actions together. Two agents could independently draw on a budget that both regard as available, without either exceeding its individual limit. And canceling a purchase afterward would not necessarily recover what the supplier had already spent preparing it. Coordination among delegated authorities, and the point at which their consequences become hard to reverse, are part of the same decision. It is not enough for each transaction to appear permissible in isolation if the company is unwilling to stand behind their combined result.
The person who can still decide
Faced with these difficulties, adding a human approval step seems prudent. It can be. Its value depends on what remains open when the request arrives: if the supplier has already received confirmation, the reviewer is managing a consequence, not preventing it. If the reviewer receives only a favorable summary, without access to the terms of the order, they cannot examine what they are being asked to take responsibility for. The timing of review, the available information, and the real ability to refuse must be designed together. An approval button does not, by itself, provide those conditions.
Nor does a person’s presence guarantee a fair allocation of responsibility. In Moral Crumple Zones (opens in a new tab), published in 2019, Madeleine Clare Elish examined accidents involving automated sociotechnical systems and their public coverage to describe how responsibility can fall on an operator with limited control. Her work offers a conceptual warning, not a measurement of today’s enterprise agents: we can keep someone visible in the process without giving them sufficient means to govern it.
Return, then, to the order. An investigation that ends with the name of the person who clicked Confirm may fail to examine those who defined the assignment, combined the permissions, or decided that such a review was sufficient. Reconstructing that sequence does not diffuse responsibility; it helps distinguish what each participant was expected to decide. The person reviewing a purchase is accountable for the judgment they were in a position to exercise. Whoever authorized the agent’s use should be able to explain why that scope of execution was reasonable. And whoever implemented the controls should be able to show that they reflect what was authorized. These responsibilities are related, but they do not replace one another: assigning them all to the last visible person would make it unnecessary to revisit the decisions that came before.
To examine those decisions, I would retain evidence of the mandate in force, the authorizations applied, and the actions actually performed. An explanation after the fact of why a purchase seems reasonable does not demonstrate that it was permitted when confirmed. Records should allow us to reconstruct that difference rather than merely provide a persuasive account of the outcome.

Up to this point, the problem may seem internal. For the supplier, however, what exists is a commitment received from the company. The supplier should not have to determine which part belonged to the model, its configuration, or the department that used it before finding someone empowered to review the order. The organization needs to undertake that investigation without making it a condition for responding to the supplier. I am speaking here of organizational responsibility; legal attribution depends on the circumstances and rules applicable to each case.
Delegating more work can be a good decision. For it to be one, we must retain the ability to explain what we allowed, intervene in what is still under way, and answer for what has already had consequences. In the purchase with which we began, identifying the agent that sent the confirmation is a fact about the transaction, not a sufficient answer to someone who acted in reliance on it. A system that acts in our name cannot become the reason we cease to answer in our own.





