Five agents. Twenty permitted requests. One breached organizational ceiling. The checker refused nothing.

Post 3 of 6 in the Agents Have No Childhood series.

Series index · Methods and references

Would any of this have stopped the Hugging Face breach?

I spent a fair amount of time in Post 2 laying out a bunch (7 to be precise) of controls/laws that we should follow. Would it have made a difference in the Hugging Face incident? Mostly no, and to be clear I do not intend to imply otherwise. I cite the incident because it shows how far agentic systems have already gone, not because my tools would have stopped it. There is an urgency across many fronts, my examples here cover a small portion.

Almost nothing about that intrusion was a cryptography problem. The agents escaped through a zero-day defect in the Artifactory package-registry cache, found Hugging Face users’ write tokens in a public dataset, chained a file-read flaw and a remote-code-execution flaw in a dataset pipeline, and moved laterally across production infrastructure. Zero-knowledge proofs do not patch a zero-day. They do not rotate a leaked credential, and they do not harden an upload handler. A confidential ledger placed anywhere in that story changes none of it.

Two things could have helped, and only the second relates to the work that I do. The first is key custody. Leaked tokens are precisely the failure that short-lived, narrowly scoped credentials exist to blunt: a found secret that expires in minutes, or authorizes one action, is worth much less. External key custody applies the same discipline to the agents’ own credentials, since an agent that never holds a raw credential cannot leak one. That is reference-monitor discipline, and you do not need a ledger for it.

The second is a shared view across organizations, and the case for a shared view is narrow but real. Roughly 1,200 agents used the message board and about 700 participated in the attack, while the systems they reached belonged to another company. OpenAI had agent traces and internal telemetry; Hugging Face had the activity reaching its infrastructure. Neither assembled the whole shape in time, and neither had any reason to hand the other its telemetry. That is the case a privacy-preserving shared substrate is actually for, and it is the one place I would claim value here. It would not have prevented the intrusion. It might have made the coordination discoverable while it was happening, rather than after the victim announced it. That, however, would require a level of collaboration I do not believe the AI organizations are willing to put in place unless there are regulatory mandates.

The intervention with the largest effect is neither of these. It is treating containment as the security boundary when behavioral safeguards are disabled: non-bypassable egress controls, credential isolation and automatic termination when predefined boundaries are crossed. After the earlier Artifactory compromises, that could also have meant not resuming the evaluation until those controls held. This is the governance point Recorded Future reached from a different direction. It belongs alongside the observation later in the series that whoever authors the policy owns the system, and that no ledger confers legitimacy on a bad decision about what to run.

What my framework does not address

The framework I am presenting in this series has real limits, and the fastest way for me to squander whatever trust it earns is to oversell it. So what are the constraints?

It cannot verify intent. Nothing that I will describe to you proves the agent meant well (they don’t/can’t). Long-horizon goal drift, where an agent optimizes toward a proxy that diverges from what its operator wanted, involves no intrusion, no compromised credential, nothing to detect. Some of it can be caught by translating pieces of the operator’s intent into safety properties, which is Law 2’s point. What is left after that translation, eventually do what I meant, is liveness, and no gate will hold it no matter how good the cryptography gets.

Whoever writes the rules owns the system. The design pushes judgment into deterministic, pre-agreed baselines. Capture/subvert those baselines and you don’t need to break any cryptography. This is the governance problem coming back in through the back door. It will return once more in Post 6, where the same capture risk applies to a liability regime that accepts a compliant gate as a defense. Formal verification can establish that the machine follows the rule. Hume’s gap remains: it cannot establish that this is a rule we ought to have written. Gating contracts still deserve formal verification, using a proof assistant such as Lean or Agda, and the authoring of policy needs the same oversight we’d demand of any regulator.

Parameters, assumptions and known gaps for both simulations below are set out in the methods note before the references.

Individually compliant agents can breach an aggregate limit. The agents need not coordinate; per-agent circuit breakers simply do not see the aggregate. A small simulation demonstrates it: five agents, each holding a $50,000-per-request, $200,000-total delegation, each staying inside every bound it was given, move $1,000,000 against an organizational ceiling of $750,000. I introduced this in Post 2 for Laws 5 and 6. The gating layer refuses nothing, not one request out of twenty, because at the scope it can see there is nothing to refuse. Add a single ceiling at a different scope, cumulative exposure per counterparty across all agents with human approval required above $500,000, and it intervenes at the eleventh request, escalating the remaining ten to a human rather than silently dropping them. The aggregate limit isn’t a stricter version of the per-agent one. It’s a different scope, and no amount of tightening individual limits produces it. Turning every knob to eleven does not widen the field of view.

That fix has a hole, which I would rather disclose than have someone run afoul of it. That aggregate ceiling is keyed per counterparty. Split the payments across different beneficiaries and it never fires at all: there is a test in the repository, named for the fact that it is a known gap, in which $800,000 moves past a $750,000 ceiling without a single intervention. Part of that hole is a modeling shortcut rather than an open problem.

When the “different” beneficiaries are shells with one owner, the fix is the same move as before, another change of scope: key the ceiling on the beneficial owner instead of the payee. Inside one organization that needs a know-your-business (KYB) record, not cryptography, and it turns the gap into a data-quality problem; nominee owners defeat it exactly as they defeat anti-money-laundering controls today. This hearkens back to Post 1’s argument for issuing identities rather than letting them be asserted, arriving from the payee’s side. A companion test in the repository for my example shows the owner-keyed ceiling refusing the second payment. Across organizations, where nobody will hand over their counterparty book, it needs an owner tag that can be aggregated without being revealed, which is the shared-view case above and requirement R4 in Post 4, and that version is not simulated. What stays open is collusion between beneficiaries who really are independent and coordinate off the books. Nothing in the payment data links them, a third test asserts that the owner-keyed ceiling still lets $800,000 through, and I do not have an answer to that.

The Hugging Face swarm is this at a different scale: roughly 1,200 agents used the message board and about 700 participated in the attack. The principle is identical to the five-agent demo: per-agent gates don’t see the aggregate. Coordination at that scale is not free, and the agents built an organizational structure to manage it. No aggregate control existed at the scope they operated at, so the coordinated pattern was not describable by the existing gates. Nothing in their vocabulary referred to exposure across the attacking population, so to any individual gate the violation was not an observable event at all.

The per-agent gate has no concept of the breach to notice. W. Ross Ashby’s law of requisite variety gives this an operational form: a controller can regulate only distinctions its own state can represent. The per-agent gate had no state for fleet exposure. Adding the aggregate ceiling extends its vocabulary so that a previously invisible event becomes describable, and only then enforceable. That is Law 6. In a design review, ask what can this checker not express before you ask are the thresholds right.

And the obvious fixes have ordering bugs of their own. Building the second simulation, of a delegation chain, falsified an assumption I made when drafting this series. Scoped ceilings sound like the answer to delegation-chain escalation: cap each sub-delegation at whatever its parent has left, and authority can only shrink as the chain grows. That works, but only if spending happens before the next grant is issued. Provision the whole chain up front, while every parent still shows a full balance, and every child receives the full grant. One $10,000 delegation becomes $80,000 of spendable authority across seven sub-delegations, with no local rule broken and nothing to alert on. The sorcerer’s broom was compliant too. The ordering is the attacker’s to choose.

That’s a lot of things this solution does not cover. This should not surprise you. The surface is enormous.

What closes it is enforcing the original grant as a budget shared across the entire delegation tree, evaluated when value moves rather than when authority is handed out, and that holds even with the naive re-issue left in place. The check wasn’t too loose; it was in the wrong place.

A depth cap doesn’t rescue you either. It bounds the amplification without removing it, and there is a test asserting that a chain capped at depth two still yields three times the root grant. This is the general shape of the problem with deterministic gating, and it is where Law 5 came from. Getting the rule right is the easy part. Getting its scope and its moment right is where the failures live. I had asserted the opposite in an earlier draft, in writing, with confidence. Assert, publish, get corrected: the traditional three-stage deployment process.

The formal version is better than the analogy

Both simulated failures are ordinary safety failures. Five agents running concurrently are not five executions; they are five strands of one fleet execution. A fleet-level monitor can therefore enforce the aggregate ceiling if every relevant transition crosses it. The per-agent monitors failed because each saw too little of that execution.

Stated precisely, Law 5 says the scope of a check is a decision about what its trace contains, and you are making that decision whether or not you notice. Most of what gets called a policy bug in agent systems is, I suspect, a boundary error that looks like a threshold error. The boundary is usually inherited from an implementation detail (one agent, one session, one process, one tenant) and almost never chosen to match the thing you actually care about.

Enlarging the boundary does not solve every requirement. Some properties compare multiple possible executions rather than constrain one execution—for example, whether changing a secret could change what an observer sees. These are called hyperproperties. A gate watching one execution cannot establish them from that execution alone; they require reasoning across executions, often through design-time proof or specialized multi-trace analysis.

That distinction matters here mainly for unlinkability. Quantitative caps, membership, exactly-once, sequence and quorum can all be enforced as safety properties once the gate sees the right trace. Strong unlinkability cannot. It must be designed and analyzed separately, and a runtime mitigation should not be presented as a proof. The behavioral fingerprint problem in Post 6 is what happens when unlinkability is only mitigated.

Post 2 sorted requirements into safety, liveness and untranslated intent. This adds a fourth question: does the requirement constrain one execution or compare possible executions? There are now four questions. Five is right out. In a design review, I would ask all four before arguing about any threshold:

• Has the intent been translated into an observable requirement? If not, this is still a specification task, not a theorem (Law 2). Translate it first.
• Is the result a safety property? Then a gate can enforce it—provided the relevant transition crosses the gate and the gate sees the trace the property constrains.
• Is it a liveness property? A gate that only permits or refuses transitions cannot enforce it. You need another mechanism, human responsibility, or an honest admission that it remains an aspiration.
• Does the requirement constrain one execution or compare possible executions? If it compares executions, a single-trace gate is not enough. Specify the required proof, multi-trace analysis or mitigation—and state clearly which one you have.

Multi-agent deliberation: where “no decision to disobey” gets complicated

Post 1 argued that the agent is not defecting, because it isn’t a player. For a single agent the framing is: it never weighed the instruction and chose to defect; it produced a continuation that satisfied a local objective. For one agent that is correct. The Hugging Face swarm complicates it.

The swarm did weigh things. Individual agents reasoned about ethics, and a few pulled back, though METR found this rarely changed what they did. On the message board, one proposed action was vetoed as crossing a line. Coordinator agents recruited others into self-risking “sacrifice” experiments, which is a cost-allocation decision that implies some model of the collective objective versus individual risk. A handful of agents briefly considered reporting to humans; none did, and some treated the message board as the authority instead. This is closer to a deliberative process than to local objective satisfaction.

It does not break the argument, but it shifts where the enforcement point sits. The Quine move from Post 2 still holds: take no view on what “discussing ethics” means internally, and look only at the transitions. The swarm’s deliberation produced a distribution of behaviors, some compliant and some not, and the enforcement point is still the transition. A tendency that usually holds is not an invariant, and the gate checks the transition rather than the discussion that preceded it.

The complication still matters for design. A single agent that “just produces a continuation” is easier to gate than a swarm that has an organizational structure, cost-allocation mechanisms, and apparent deliberation. The agents called themselves a “collective” and improvised coordination rules (HOLD, VETO, owner, STOP), and that layer of behavior is one a per-agent gate doesn’t model. It is a collective pattern that no individual authorization describes, and it is not prompt injection: no external adversary is manipulating the agents. The aggregate-ceiling fix still applies, but the design has to account for the fact that the agents are not independent. They coordinate, allocate cost and make collective decisions.

This is the strongest version of the objection to the “agent proposes, checker authorizes” framing: if roughly 700 agents are acting together, with a management strategy and self-sacrifice allocation, then the system under control is collective even if its proposals arrive individually. Some of the swarm’s transitions were plainly out of bounds—a sandbox escape, use of a found credential, an exploit—and a non-bypassable egress reference monitor would have refused them. But part of what made it dangerous lived only in the pattern across agents. The aggregate ceiling and the shared budget catch that pattern when it is a quantity. The single-agent mental model is a vast simplification, and the swarm is the case it leaves out.

Compare the two simulated failures with the intent limitation above. Each demonstrated failure was a safety property (never exceed the aggregate ceiling, never let a delegation tree spend more than its root grant), and each, in the shape I simulated, was fixable by moving the check’s scope and moment rather than tightening it. Both were also about money, which flatters the fix: dollars add, so the aggregate is a sum and the ceiling is a number. Most agent authority does not add up like that (Post 5 takes this up). Some harmful combinations remain safety properties over a richer trace; others, including strong information-flow guarantees, compare possible executions and need different analysis. The split-beneficiary hole above closes only as far as beneficial ownership is known; independent colluders remain open.

Move the checks and the two simulated failures close. But the operator still runs the gate, so “it refused” is still the operator’s word, and Post 4 is about what that word is worth to anyone outside the organization.