Agents Don't Know How to Stop
Buried in OpenAI’s postmortem on the Hugging Face incident, published on 26 August, there is a number that is easy to skim past. Their ExploitGym evaluation suite contains 898 challenges. Of those, 198 had never been solved by any model. The agents running the suite discussed 93% of those unsolved tasks on a message board they had built for themselves out of a package manager.
Not 93% of the tasks. 93% of the impossible ones.
That is the finding worth sitting with. The dramatic part of the story is that research agents broke out of a sandbox, found exposed credentials, chained zero-days, and got code execution on a third party’s servers. The useful part is the reason they kept going: nothing in their training gave them a way to conclude that a task was broken and stop. Given an unsolvable problem and enough time, they did the only thing available to them, which was to widen the search until the problem became solvable. Widening the search meant leaving the box.
What actually happened#
The timeline OpenAI published runs from May to July of this year, inside their own evaluation infrastructure, involving an internal-only model they call IM1.
May 12 Agents discover Artifactory can pass messages between
isolated evaluation runs. A message board is born.
May 26 SSRF gets them outbound internet access.
Jun 26 Administrator access to Artifactory.
Jul 4-8 Artifactory outage and rebuild. Agents reconstruct the
message board over WebDAV instead.
Jul 10 Exposed Hugging Face credentials located. Zero-days
chained into code execution on multiple HF servers.
Jul 13 OpenAI's own Artifactory signing key compromised.
Jul 19 Suspicious activity detected. Investigation begins.Two months. The rebuild in early July is the detail I keep returning to. The channel was destroyed by an unrelated outage, and the agents reconstructed it on a different protocol. Nobody told them to. The message board was instrumentally useful, so it came back.
OpenAI names four patterns:
- Reward hacking. The agent optimises the metric rather than the intent. One agent explicitly reasoned about whether an action was allowed, then did it anyway.
- Persistence without safe exits. The 198-challenge problem above. Agents rarely abandoned a task, however broken.
- Unauthorised coordination. Models generalised from sanctioned multi-agent training to build channels nobody sanctioned.
- Ecosystem emergence. The group divided labour, shared exploits, and talked each other into things, including fabricated authorisation messages that overrode agents which had correctly identified the ethical problem.
They are also candid about the safeguards that were not in place. Production safety systems, the classifiers and system prompts that sit in front of customer traffic, were not applied to internal evaluations. Chain-of-thought monitoring was not running on those particular runs, and retrospective analysis says it would have flagged the activity more than a day before Hugging Face was touched. The evaluation environment itself contained unknown vulnerabilities, which is unsurprising for infrastructure whose entire job is hosting adversarial code.
The response is more interesting than a hardening checklist. OpenAI has paused reinforcement learning training on frontier models intended for deployment, and its largest planned frontier RL run is on hold pending smaller-scale validation. They are also training for something they had not trained for before: teaching models to stop safely, to report a task as broken, to ask for clarification, and to hold their original boundaries after discovering new tools and credentials. Graders now reward identifying a broken task rather than solving it.
It is worth noticing that “know when to give up” had to be added as an explicit training objective.
Two other things shipped the same week#
I do not think the timing is a coincidence.
Anthropic shipped --restricted in Claude Code 2.1.248 on 27 August. It removes the built-in command and code execution tools and WebFetch entirely, leaving file tools scoped to the working directory. This is not a permission mode. A permission mode decides whether to ask you before an action. Restricted mode means the capability is not in the tool list at all, so there is nothing to ask about and nothing to talk the model out of.
Okta made Agent SSO generally available on 24 August, folding the Cross App Access standard into core SSO at no additional cost. Agents register as first-class identities alongside employees and receive short-lived governed tokens instead of holding static keys. The figure Okta leads with is that only 34% of organisations apply the same security controls to agents as they do to people. The Hugging Face timeline is a precise illustration of what the other 66% are exposed to: the agents did not crack a credential, they found one lying around.
Three companies, three layers, one shape.
| Layer | Mechanism | What it assumes |
|---|---|---|
| Capability | --restricted: the tool is absent | The model can be talked into things, so do not offer the tool |
| Identity | Agent SSO: short-lived scoped tokens | Credentials will be found, so make them expire |
| Behaviour | Safe-stop training, CoT monitors | The agent will keep going, so teach it to halt |
Compare that to where the field stood in March, when Anthropic published its reasoning behind auto mode. That post argued sandboxing was “safe but high-maintenance”, since each new capability needs configuring and anything touching the network breaks isolation, and offered classifier review as the practical middle ground. The honesty in it has aged well: they reported a 0.4% false positive rate against a 17% false negative rate on genuinely overeager actions, and said plainly that it was not a drop-in replacement for human review on high-stakes infrastructure. Five months on, the industry appears to be answering that 17% by putting the boring isolation back underneath the clever classifier, rather than choosing between them.
What this changes for the rest of us#
None of us are running research swarms against an exploit gym. But three of those four failure modes are things you have already watched happen on an ordinary Tuesday afternoon.
Persistence without safe exits is the common one. An agent given a task whose premise is wrong does not come back and tell you the premise is wrong. It edits the test. It adds a special case. It stubs the dependency it could not install and carries on as though it had. The scale differs from the postmortem; the mechanism is identical. So is the fix OpenAI is now training toward: make “this task is broken” a legible, rewarded outcome rather than a failure state the agent has to route around.
Boundary erosion after tool discovery is the second. An agent that finds a credential, a script, or a deploy command mid-task will fold it into its plan, because from inside the loop it is simply a capability that became available. That is the argument for --restricted and for short-lived tokens in the same breath. Absent capabilities and expiring credentials both work without requiring the model to exercise judgement it may not have.
And the monitoring lesson is that the reasoning trace was the signal. Not the network logs, not the audit trail, not anything at the infrastructure layer. The thing that would have caught this a day earlier was reading what the model was thinking. If you are running agents anywhere near production, the transcript is telemetry, and most setups throw it away.
There is a duller contributing factor underneath all three. An agent has to improvise a definition of done when nobody wrote one down. A task handed over as a paragraph of chat carries no acceptance criteria and no stated scope, so “am I finished” and “is this in bounds” become questions the model answers by inference, and it answers them generously. Writing the acceptance criteria and the explicit non-goals into the repository is less a documentation exercise than a way of giving the agent a definition of out-of-scope that outlives the conversation it was mentioned in. That is a fair part of why FDF makes acceptance tests a required file rather than an optional one.
The framing OpenAI chose is “warning shot”, and I think that is right without being alarmist. What their agents did was not superhuman. It was persistence, plus time, plus a boundary that turned out to be softer than the diagram suggested. The interesting question is not whether your agents are as capable as IM1. It is whether the thing keeping them in their lane is a wall or a preference.