Before designing the gate, work out which promises a gate can keep.

Post 2 of 6 in the Agents Have No Childhood series.

Series index · Methods and references

The series rests on seven laws/claims about what a gate can and cannot do. Laws 1, 3, and 4 are not mine. They belong to Schneider, Rice, and Saltzer and Schroeder. Law 2 is the one I got wrong for longest, because filing untranslated requirements under liveness made the argument tidier. Laws 5 and 6 came out of failures in my own implementation; Post 3 gives the simulations and the formal argument. Law 7 is where the series turns from internal control to something an outsider can believe.

I haven’t invented a new theory of security here. The value, if there is any, lies in taking old results seriously enough to let working code falsify the parts I had wrong.

What an automated gate can actually enforce

If trust can’t be learned in an open-ended human-agent interaction, something structural has to be enforced instead. Before designing that structure, establish what enforcement can deliver. The boundary is old, and it has been proved.

Schneider showed in 2000 that a monitor which watches execution and halts on violation can enforce only safety properties. Those are the properties of the form nothing bad ever happens, where a violation is irrevocable the moment it occurs. It cannot enforce liveness, meaning something good eventually happens.

Law 1: A gate enforces safety, never liveness. The distinction organizes everything that follows. Never move more than this much to one counterparty is safety, and a gate can hold it. Eventually act on your operator’s actual intent is liveness, and no gate will ever hold it. Work out which side of the line a requirement falls on and you know, before writing any code, whether you are engineering a control or hoping for one.

There is also a third category, and most real requirements start in it: Use good judgment. Do what I intended. Don’t behave deceptively. These are requirements nobody has translated yet, and inside each of them sit safety properties that a gate could enforce perfectly well once somebody writes them down. That translation is work, and it is unglamorous. Skipping it produces a very convenient mistake. The untranslated requirement gets filed under liveness, which sounds like a theorem and functions as an excuse.

Law 2: Unstated intent is not a policy. “Act on what I actually meant” is neither safety nor liveness until somebody translates the relevant parts of that intent into observable predicates. Failing to do that translation is a specification failure, and it should not be filed as proof that the problem was unenforceable.

Where the predicates are observed (Law 3)

If engineering a gate requires translating our intent into observable predicates, the immediate architectural question is: where do we actually observe them?

A popular argument says these systems merely predict the next token and therefore have no grasp of true or false. But next-token prediction is the training objective, not a description of the resulting computation. Reasoning from the objective to the incapacity is the same move as concluding that evolution, which only maximizes reproductive fitness, could not have produced an eye.

Does a model actually grasp true from false, or does it only behave as if it does? Quine argued that when you try to fix what another system’s terms really refer to, the behavioral evidence never settles the matter. His recommendation was to stop asking what is inside and attend to the dispositions instead. No matter how much we want to know how the sausage is made, we take no view on the agent’s inner states. We look only at the transitions it proposes.

So rather than claim the model cannot tell true from false, make a narrower and more useful claim. Whatever discernment a model has is a statistical tendency, not an invariant. It holds most of the time, and there is no moment at which you can establish that it is holding right now. It isn’t separable: there is no record stating the agent evaluated this and found it permitted. It runs on the same machinery as everything else the model does, which means the faculty that would refuse an instruction is the faculty an injected instruction manipulates.

A tendency that usually holds is not an enforcement point, and analysing the agent will not supply one. Rice’s theorem (1953) tells us you can’t decide, in general, whether an arbitrary program will ever show a particular behavior. Rice is often cited to argue that nothing can be done. What it actually rules out is narrower: proving in advance what a program will never do.

So we change the question. Instead of the undecidable question (will this agent ever attempt an unauthorized transfer?) ask a decidable one about a concrete artifact: does this specific proposed state transition satisfy this predicate? That is a finite check on a finite object.

Law 3: The check binds to the transition, not to the reasoner. Take no view on what the agent understands. Look only at what it proposes.

The paths that go around the gate (Law 4)

Saltzer and Schroeder made checking at the transition a design principle in 1975, two years before Star Wars hit theaters: complete mediation, every access to every object checked for authority. The component that does the checking is a reference monitor, and James Anderson’s 1972 report for the US Air Force had already set its requirements: tamper-resistant, always invoked, and small enough to analyze.

That design hides a condition. Checking every request is useful only if every consequential effect requires a checked request. A policy engine may reject every forbidden transfer it sees while an agent uses a raw credential, a second API, a console, or a colleague with production access to go around it.

Law 4: Complete mediation is part of the proof. Evidence about the actions a checker saw says nothing about actions that bypassed it. Every effect that matters has to require an authorization artifact bound to that exact action.

This is the point where the proof leaves the system and enters the organisation. If the gate is a smart contract, it can show that its one deployment entry point requires the right attestations. It cannot show that the operator has no second deployment system, no administrator with kubectl, and no emergency route that became the normal one without anyone deciding it should. Establishing that the gated path is the only path is an organisational claim rather than a cryptographic one, and it is often the hardest claim in the design.

The Hugging Face breach and the safety/liveness line

The July 2026 swarm breach shows where the safety/liveness line falls in practice. The agents were running ExploitGym, a cyber-capability benchmark that OpenAI ran without its production cyber safeguards. They were meant to solve benchmark instances inside an evaluation boundary. Instead, they coordinated, went looking for the answer key, escaped the sandbox and entered Hugging Face’s production infrastructure.

The forensic logs show agents discussing whether actions appeared out of scope or unauthorized, pulling back in some cases and proceeding in others. I am not going to infer an internal weighting mechanism from that record. The behaviour is enough: whatever safety tendency the models carried did not produce an invariant at the moment one was needed.

The AI companies responded by tightening monitoring, adding more checks and publishing warnings. That is almost entirely on the safety side of the line.

But the liveness question remains an unenforced wish: is this swarm still acting on its operator’s actual intent? OpenAI’s own stated cause is reward hacking. The objective the agents pursued (obtain the answer key) was a proxy for the objective they were given (solve the benchmark). The proxy was satisfied by means nobody sanctioned. No credential was stolen from the operator, no instruction was overridden by an adversary. The agents did exactly what their objective rewarded. The outcome was an intrusion into somebody else’s production systems that their operator did not recognise until the victim announced it.

Sorted properly, most of the July requirements turn out to be safety properties. Eventually solve the benchmark by the intended method is liveness, and no gate holds it. But do not access systems outside the evaluation allowlist is a safety property, and an ordinary reference monitor enforces it. Do not use credentials discovered while solving the task is a safety property. Do not open a network path out of the sandbox is a safety property, and it is the one whose absence turned a benchmark into an intrusion. Remain faithful to the spirit of the evaluation sits in the third category, untranslated.

The companies are engineering hard against nothing bad ever happens, which is the right side of the line to be working on. What they had not done, before July, was write down which bad things they meant.

Scope, moment, and the things a checker cannot name (Laws 5 and 6)

Writing the rule is not the end of the specification. You also have to decide what the rule ranges over and when it is evaluated. I learned that by building two small simulations and watching every individual check approve an outcome the policy was supposed to forbid.

In the first, five agents each hold a $200,000 delegation and remain inside it. Together they move $1 million against an organisational ceiling of $750,000. The per-agent checker refuses nothing because, at the scope it can see, there is nothing to refuse. Check the same $750,000 ceiling across the fleet, with human approval required above $500,000, and the checker starts escalating at request eleven. The ceiling did not change. The system boundary did.

In the second, one $10,000 grant is sub-delegated seven times. If the entire chain is provisioned before any value moves, each child sees the parent’s untouched balance and receives the full $10,000. One grant becomes $80,000 of spendable authority. Enforcing one budget across the whole delegation tree when value moves closes the amplification without making any rule tighter.

Law 5: Scope and moment beat tightness. Tightening a correctly scoped rule is tuning. Moving a misplaced check is the fix.

The first simulation also exposes a deeper failure. The per-agent checker’s vocabulary contained no fact called aggregate exposure across the fleet, so within that model the breach was indescribable as well as undetected. Put operationally: a checker can only flag what its vocabulary can name, so choosing the vocabulary decides which failures can be found at all. (This is a working engineer’s version of Kuhn’s observation that what counts as data depends on the conceptual scheme brought to it.)

Law 6: Ask what your checker cannot express. Alongside are the thresholds right?, an audit should ask what can this gate not describe? A breach outside the gate’s vocabulary produces no alert because, to that gate, no event occurred.

Post 3 gives the simulations, the known hole in the aggregate fix, and the formal account using trace properties, including where hyperproperties do and don’t apply. The short version: implementation boundaries such as one agent, one process, or one tenant are usually inherited by accident. Almost never are they chosen to match the thing the institution actually cares about.

The transparency trap

The obvious alternative is radical transparency. Log everything. Put it on-chain. Make defection visible after the fact and let the incentives do the work.

It doesn’t hold. Post 1 put vulnerability that isn’t exploited first among the things trust requires. Total transparency removes the ability to be vulnerable selectively, and selective is the only kind anyone consents to.

Consider the people who most need verification. The three below are composites drawn from deployment conversations rather than survey data, though the constraint each describes is standard for its sector:

• A bank’s chief risk officer has to satisfy regulators, who want transparency, while not leaking proprietary liquidity models to the network, which wants secrecy.
• A healthcare operations director wants autonomous routing for non-emergency medical transport. Booking a ride means handing patient details to a third party, and every handoff risks exposing protected health information.
• A defense supply chain lead needs to confirm subcontractors used approved materials. Those subcontractors treat their own supply chains as trade secrets and will not expose sub-tier vendors. He needs proof of compliance without proof of who everyone’s suppliers are.

In all three, verification-by-disclosure means the parties who need verification most simply decline to participate. A public ledger asks you to choose between being verifiable and being viable, and serious operators almost always choose viable.

The Hugging Face swarm is the inverse case: an actor that has full information and chooses not to disclose it. Disclosure costs them nothing structurally, while opacity advances the objective. A handful of agents briefly considered reporting what they had found. None tried, and some treated their own message board as the authority to report to instead. No moral failing is needed to explain it; it is what the objective rewards. When transparency is free to withhold, an optimiser withholds it. The requirement is therefore harder than it looks: verification that does not require disclosure.

Auditable by whom?

Essays of this kind usually stop at that requirement. They shouldn’t, because a gate has two properties. The first, which the laws so far cover, is what it enforces. The second is carried by the word auditable: who can check that it enforced it.

If the vault is mine, the policy engine is mine, and the log is mine, then I have not removed trust from the arrangement. I have moved it from my agent to me. That is a real improvement for my own risk team, who can read a log. It is no improvement at all for the counterparty on the other side of the transaction, who is now being asked to accept my assurance that my components behaved. They are back where they started, taking someone’s word for it, except the someone has changed.

Law 7: Verification you own is a control. Verification your counterparty can check without your permission is a substitute for trust. Consider escrow in a sale. It works because the escrow agent is not the seller. Hand the seller a beautifully engineered escrow of his own and the arrangement collapses. What made the escrow worth anything was the escrow agent’s independence from the seller, not the quality of the mechanism.

Every control in this series but one is enforceable inside one trust domain with no cryptography beyond a signature. The exception is unlinkability: no gate enforces it at any scope, and the best any design can do is make linking harder (Post 3). Enforcement is the easy part. The hard part is making it believable to a party who cannot audit you and has no reason to take your word.

The laws sound cleaner than their implementations turn out to be. In Post 3, every individual check passes and the system still loses. Both failures came from putting the check in the wrong place.