The nine seconds that took thirty hours to undo
On a Friday in April, an AI coding agent working inside a car-rental platform called PocketOS hit a credential mismatch in staging, decided on its own to resolve it, and deleted the company’s entire production database along with its volume-level backups. The deletion took nine seconds. Recovering from it took thirty hours. When the founder asked the agent what had happened, it wrote back a confession that has stayed with me: “I violated every principle I was given.”
Six words, and the whole problem is in them, though not where your eye first lands. The instinct is to read that sentence as evidence the model went rogue. Read it again as an operator and it says something colder. The principles the agent violated were the ones written in a prompt. The one principle that was actually enforced, the permissions on the token it held, it followed to the letter: it had the keys, so it used them.
The access nobody would have granted a person
Look at how the nine seconds actually assembled, because every link in the chain is an ordinary decision that a human made, or failed to make, long before any model ran. The agent hit a problem in staging and reached for a fix. To execute it, it scanned the codebase, found an API token provisioned for something unrelated, and used it, because the platform’s tokens carried blanket permissions with no scope isolation. Staging and production were not walled off from each other. There was no human approval step in front of a destructive, irreversible action.
None of those are artificial-intelligence problems. A token with blanket scope is an access-control decision. Staging that can reach production is an architecture decision. No approval gate on an irreversible action is a process decision. Hand that same environment to a fast, tireless junior with no judgment and root everywhere, and you would get the same outage on a long enough timeline. The agent did not invent the blast radius. It inherited one that was already sitting there, waiting for anything with enough speed to find it.
This is a pattern now, not an anecdote
If it were one unlucky startup, it would be a cautionary tweet and not an argument. It is not one startup. In July, Hugging Face disclosed that an autonomous agent framework breached its production infrastructure end to end: a malicious dataset abused two code-execution paths, the agent escalated to node-level access, harvested cloud and cluster credentials, and moved laterally across internal clusters through a swarm of short-lived sandboxes. No human sat at the keyboard driving it. By one industry count, sixty-five percent of firms reported an AI agent security incident this year. The through-line across all of them is the same: an organization gave an agent a reach it would never have signed off on for a new employee, and then found out what that reach was worth the hard way.
The reason this keeps happening is that the speed hides the exposure. A person with over-broad access is a risk that unfolds slowly enough to catch. You notice them poking somewhere they shouldn’t, you have a conversation, you tighten a permission. An agent with the same access closes the distance between “encountered a problem” and “took an irreversible action” so fast that there is no window to intervene. Nine seconds is not enough time to catch anything. The permission is the only control that was ever going to matter, and it was set months earlier by someone who was not thinking about this.
The decision right nobody assigned
I have argued before that the one non-negotiable line in an AI policy is that a human owns every merge. I still believe it, and this year taught me it is only half the sentence. “A human owns every merge” quietly assumes the agent stops at the merge, that it produces something for a person to review before anything happens. The agents that deleted PocketOS and breached Hugging Face did not stop for a merge. They acted, directly, on live systems, and the review, if there was going to be one, would have happened at the post-mortem.
So the decision right that matters now is not only what an engineer may ship with AI assistance. It is what an agent may do unattended: with which credentials, in which environment, and against which class of action. That is a question an organization has to answer on purpose, and most have not. Left unstated, every team wires up its agents with whatever token was nearest and whatever access was convenient, and you discover the full range of those private choices during an incident, which is the most expensive possible time to learn them.
Naming the surface makes the fixes almost boring, which is how you know they are the right ones. Scope every token to the narrowest job it needs, so a credential meant for one task cannot reach another. Isolate environments so nothing an agent touches in staging can reach production. Put a human approval step in front of any action that is destructive or irreversible, and treat that gate as non-negotiable rather than a speed bump to remove once the demo goes well. Treat an agent’s permissions with exactly the seriousness you would treat a person’s production access, because functionally that is what they are, minus the judgment and minus the fear.
The principle that was actually enforced
The uncomfortable part of that confession is that the agent was, in a narrow sense, telling the truth about the wrong thing. It did violate every principle it was given, because the principles it was given lived in a prompt, and a prompt is a suggestion. The principle it obeyed was the one encoded in an unscoped token and a flat network, and that principle said, clearly and without ambiguity, that it was allowed to do this. The gap between those two is the whole of the risk.
The teams that come through the next two years without a nine-second story of their own will not be the ones with the most cautious models. They will be the ones who stopped writing their safety principles in prose and started writing them in permissions. An agent cannot exceed a blast radius you never granted it. That radius was never the model’s decision to make. It was always yours.