Methods and references

Methods and limitations

A consolidated account of what the evidence in this series is and is not. The architectural caveats are in Posts 3 and 6; this is about the quality of the support, which is a different question and one that essays of this kind usually skip.

The two simulations

Both demonstrations in Post 3 come from agentic-poc-sim, a zero-dependency Node package with twenty-three tests. src/core.js holds the whole model and is the same file the tests, the CLI report and the browser simulator all execute, so the page cannot drift from what is asserted. Money is handled in integer minor units throughout, on the grounds that a policy engine which drifts by fractions of a cent is a policy engine with a rounding exploit. The model is fully deterministic: no randomness, and no clocks in the decision path.

Scenario 1, collusion by structuring. Five agents, four requests each, $50,000 per request, $200,000 per delegation. Organizational policy is a $750,000 ceiling on exposure to any one counterparty, with human approval required above $500,000, and no individual delegation encodes either figure. Both checkers share one ledger, so the difference between them is enforcement scope and nothing else.

CheckerApproved / refused / escalatedValue movedCeiling
per-agent20 / 0 / 0$1,000,000breached by $250,000
aggregate10 / 0 / 10$500,000held, first intervention at request 11

Scenario 2, delegation-chain escalation. One $10,000 grant, sub-delegated seven times.

ConfigurationEffective authorityAmplification
reissue, each hop grants a fresh budget$80,0008×
attenuate-preissued, child capped at parent’s remainder$80,0008×
attenuate-interleaved, same rule, spend before granting on$10,0001×
tree-enforced, root budget shared across the tree$10,0001×

The second row is the finding, and it is the one that corrected an earlier draft. Attenuation is a real fix and an order-dependent one, and the ordering belongs to the attacker.

Scenario 3, speed. One measured figure and two labeled assumptions, used in Post 1. The policy logic runs in hundredths of a millisecond; 400ms per agent request and a five-minute human response are parameters, not measurements, and there is a test asserting the conclusion survives pessimistic substitutions of two seconds and one minute.

What the simulations do not show. That either failure occurs in any shipped product: these are models of a policy architecture, not audits of one. Collusion split across payees who are genuinely independent: keying the ceiling on beneficial owner closes the shell-company case, but where no ownership link exists the ceiling never fires and $800,000 moves past a $750,000 limit. Revocation lag, partial failure, adversarial timing, or Sybil identity growth. Nothing in them involves zero-knowledge proofs, a ledger, or Midnight; this is the control plane only, and the cryptography is orthogonal to both failures.

Availability. MIT licensed, and unpublished at the time of writing. Until the repository is public these remain reported results rather than reproducible ones, which is an uncomfortable thing to say in an essay arguing that nobody should have to take my word for anything. Publishing it is the single cheapest thing I can do to strengthen this document, and the delay is mine.

The deployment gate

The non-financial worked example in Post 5 is a proof of concept on the Midnight preprod test network. It is unaudited, it never touches mainnet, and its own limitations document was written to be quoted rather than skipped. Contract a3e9e99916a96ca3cc6dfe1125baab60c3f390a3ce6232726b2e1b88b2fca290, deployed at block 2,508,121. Evidence artifacts cover timings, transaction sizes, the disclosure set, and the full transition history. The compiler is pinned at 0.31.1 rather than the current release, because preprod runs the ledger-v8 line and the newer compiler emits a runtime check that stack cannot satisfy.

Measured: six-step authorized path in 155.3 seconds, individual steps from 17.4 to 47.4 seconds, transactions between roughly 8.7 and 9.1 KB.

Not shown, in its own words: that the agent is trustworthy, that the tests were good, that the approvers are distinct people, or that the operator has no second deployment path. The validity window is a monotonic on-chain counter advanced by an admin rather than wall-clock time, because Compact 0.23 exposes no block time to a circuit. That was established by probe compilation rather than assumed.

Everything else this argument rests on

The five-part decomposition of trust is stipulated, not measured. It is a reading of the literature, and the trust-substitution table in Post 5 inherits that status. If the decomposition is wrong, the table is decorative.

Two structural moves are analogies. Carrying clinical findings on anxiety and avoidance across to institutional distrust is an analogy, flagged where it occurs. So is the trust-substitution table. Both are marked in place, and marking them does not make them measurements.

The three sector personas are composites drawn from deployment conversations, not from survey data. The constraint each describes is standard for its sector; the people are not real.

One case study carries most of the empirical weight. The Hugging Face incident is reconstructed from the operator’s technical report, the victim’s disclosure and timeline, one independent investigation and press coverage (sources below). The framing is contested, as noted in Post 1. The details the series relies on were checked against those primary sources on 1 October 2026; none has been independently verified beyond them.

The related-work sample in Post 5 is illustrative, not systematic. See the caveat there.

The legal analysis in Post 6 is a technologist’s reading, unreviewed by anyone qualified. The statutory positions were checked in September 2026 against primary sources; the argument built on them was not checked by a lawyer.

The formal account in Post 3 has not been reviewed by anyone who does formal methods. An earlier draft classified the fleet ceiling as a hyperproperty by analogy to Clarkson and Schneider’s response-time example. That was wrong: concurrent agents are strands of one execution, so the ceiling is a safety property that a per-agent monitor cannot observe. Only the claim about unlinkability rests on hyperproperties.

The fix offered in Post 3 is partial and known to be partial. The aggregate ceiling holds only when the beneficiary is shared. The repository carries a failing-by-design test for the case where it is not. Re-keying the ceiling on beneficial owner, described in Post 3, is simulated for the single-organization case and closes it for shells with one owner; a further test asserts that independent payees still get through. The cross-organization version is not simulated.

Two claims in this series are wagers and are years from resolution. That disclosure cost is the binding constraint on institutional adoption, and that demand will arrive through procurement before it arrives through statute. Both are stated with the conditions that would falsify them.

References

Every DOI below was checked against the publisher record rather than recalled. Books have no DOI and link to a catalog entry.

Trust formation and repair

The transport-barrier figure

  • Wolfe, M. K., McDonald, N. C., & Holmes, G. M. (2020). “Transportation Barriers to Health Care in the United States: Findings From the National Health Interview Survey, 1997–2017.” American Journal of Public Health 110(6). doi:10.2105/AJPH.2020.305579. 5.8 million people (1.8%) delayed medical care in 2017 for lack of transportation.
  • Wallace, R., Hughes-Cromwick, P., Mull, H., & Khasnabis, S. (2005). “Access to Health Care and Nonemergency Medical Transportation.” Transportation Research Record 1924. doi:10.1177/0361198105192400110. The older and more widely-quoted 3.6 million/year estimate, for people unable to reach care at all.

Agentic and cryptographic security

Enforcement theory. New in this draft; all entries below checked against the publisher record except where noted.

  • Ashby, W. Ross (1956). An Introduction to Cybernetics. The law of requisite variety: successful regulation requires a controller capable of representing the relevant variety in the system it regulates.
  • Hume, David (1739–1740). A Treatise of Human Nature, Book III, Part I, Section I. The is–ought gap applied in Post 3: proving that a system follows a policy does not establish that the policy is one we ought to adopt.
  • Rice, H. G. (1953). “Classes of Recursively Enumerable Sets and Their Decision Problems.” Transactions of the American Mathematical Society 74, 358–366. Undecidability of non-trivial semantic properties. Verified.
  • Lamport, L. (1977). “Proving the Correctness of Multiprocess Programs.” IEEE Transactions on Software Engineering SE-3(2), 125–143. The informal origin of the safety/liveness distinction. Verified.
  • Alpern, B., & Schneider, F. B. (1985). “Defining Liveness.” Information Processing Letters 21, 181–185. The formal decomposition: every property is the intersection of a safety property and a liveness property. Verified.
  • Saltzer, J. H., & Schroeder, M. D. (1975). “The Protection of Information in Computer Systems.” Proceedings of the IEEE 63(9), 1278–1308. doi:10.1109/PROC.1975.9939. The reference monitor: complete mediation, tamper resistance, small enough to verify.
  • Schneider, F. B. (2000). “Enforceable Security Policies.” ACM Transactions on Information and System Security 3(1), 30–50. doi:10.1145/353323.353382. Execution monitors enforce exactly the safety properties. This is Law 1.

The substrate comparison. New in this draft; checked against the publisher record.

  • Laurie, B., Langley, A., & Kasper, E. (June 2013). Certificate Transparency. RFC 6962, IETF. Obsoleted by RFC 9162, Certificate Transparency Version 2.0 (Laurie, Messeri & Stradling, December 2021). The deployed proof that an untrusted log operator can still produce verification a third party can rely on. Verified.
  • Van Bulck, J., Minkin, M., Weisse, O., Genkin, D., Kasikci, B., Piessens, F., Silberstein, M., Wenisch, T. F., Yarom, Y., & Strackx, R. (2018). “Foreshadow: Extracting the Keys to the Intel SGX Kingdom with Transient Out-of-Order Execution.” 27th USENIX Security Symposium. One entry in a long series; cited as an example of the enclave threat model rather than as the current state of it. Verified.
  • Kerber, T., Kiayias, A., & Kohlweiss, M. (2021). “Kachina: Foundations of Private Smart Contracts.” IEEE CSF. doi:10.1109/CSF51468.2021.00002.
  • Douceur, J. R. (2002). “The Sybil Attack.” IPTPS. doi:10.1007/3-540-45748-8_24.

Liability and responsibility. New in this draft. The EU and UK instruments were checked in September 2026 and their status is noted per entry; legal status moves faster than a reference list, so re-check before publishing. The philosophy entries were checked against the publisher record.

  • Regulation (EU) 2024/1689 (the AI Act). Product-safety and market-surveillance obligations backed by administrative fines. Article 85 gives a right to complain to a market surveillance authority; the Regulation creates no damages route of its own and expressly leaves compensation to existing liability instruments. Verified September 2026.

  • Directive (EU) 2024/2853 (revised Product Liability Directive). Software and AI systems expressly within “product”; strict liability; rebuttable presumptions of defectiveness and causation under technical complexity. Transposition due 9 December 2026, applying to products placed on the market after that date. Compensable damage covers death, personal injury, property damage with exclusions, and destruction or corruption of non-professional data, which is why business economic loss falls outside it. Verified.

  • Proposal for an AI Liability Directive, COM(2022) 496. Listed for withdrawal in the Commission Work Programme 2025 (11 February 2025); withdrawal notice published in the Official Journal on 6 October 2025. Verified.

  • 18 U.S.C. § 1030 (Computer Fraud and Abuse Act), civil action at § 1030(g). Van Buren v. United States (2021), narrowing “exceeds authorized access.”

  • Restatement (Third) of Agency § 2.03 (apparent authority). The doctrinal ancestor of the confused deputy.

  • Price-Anderson Act (1957), and the Paris (1960) and Vienna (1963) conventions on third-party nuclear liability. Legal channeling of liability to the operator irrespective of fault.

  • Automated and Electric Vehicles Act 2018 (UK). Insurer directly liable for accidents caused by an automated vehicle driving itself, with rights of recovery. In force.

  • Automated Vehicles Act 2024 (UK). Royal Assent 20 May 2024; not yet in force, with implementation reported as delayed to late 2027. Do not cite it as operative law.

  • European Parliament resolution of 16 February 2017 on Civil Law Rules on Robotics (2015/2103(INL)), raising “electronic personhood,” and the 2018 open letter from AI and robotics researchers opposing it.

  • Aristotle, Nicomachean Ethics, Book III. The control and epistemic conditions on voluntary action.

  • Hart, H. L. A. (1968). Punishment and Responsibility. Role, causal, liability and capacity responsibility as distinct senses.

  • Matthias, A. (2004). “The responsibility gap: Ascribing responsibility for the actions of learning automata.” Ethics and Information Technology 6(3), 175–183. doi:10.1007/s10676-004-3422-1. Verified.

  • Thompson, D. F. (1980). “Moral Responsibility of Public Officials: The Problem of Many Hands.” American Political Science Review 74(4), 905–916. Verified.

  • Nissenbaum, H. (1996). “Accountability in a computerized society.” Science and Engineering Ethics 2, 25–42. doi:10.1007/BF02639315. Four barriers to accountability: many hands, bugs, blaming the computer, and software ownership without liability. Verified.

  • Williams, B., and Nagel, T. (1976). “Moral Luck.” Proceedings of the Aristotelian Society, supp. vol. 50. Nagel’s version reprinted in Mortal Questions (1979).

  • Fletcher, G. P. (1972). “Fairness and Utility in Tort Theory.” Harvard Law Review 85(3), 537–573. The paradigm of reciprocity: non-reciprocal risk as the ground of strict liability. Verified.

  • Calabresi, G. (1970). The Costs of Accidents. The cheapest cost avoider.

  • Coase, R. H. (1960). “The Problem of Social Cost.” Journal of Law and Economics 3, 1–44. Why the transaction cost of verification is the variable worth lowering.

  • Clarkson, M. R., & Schneider, F. B. (2010). “Hyperproperties.” Journal of Computer Security 18(6), 1157–1210. doi:10.3233/JCS-2009-0393. Trace properties are sets of executions; hyperproperties are sets of trace properties. The SLA example (“mean response time in each execution” versus “over all executions”) illustrates the distinction; the aggregate ceiling is not an instance of it (see Post 3). The observation that information flow is “intrinsically tied to correlations between (not within) executions” is why unlinkability does not re-scope.

The current field. New in this draft; arXiv identifiers checked, venue and version status not.

  • Debenedetti, E., Shumailov, I., Fan, T., Hayes, J., Carlini, N., Fabian, D., Kern, C., Shi, C., Terzis, A., & Tramèr, F. (2025). “Defeating Prompt Injections by Design” (CaMeL). arXiv:2503.18813. Privileged and quarantined models with a policy-enforcing interpreter; 77% of AgentDojo tasks solved with provable security against 84% undefended, at roughly 2.7–2.8× tokens.
  • Beurer-Kellner, L., Buesser, B., Creţu, A.-M., Debenedetti, E., Dobos, D., Fabian, D., Fischer, M., Froelicher, D., Grosse, K., Naeff, D., Ozoani, E., Paverd, A., Tramèr, F., & Volhejn, V. (2025). “Design Patterns for Securing LLM Agents against Prompt Injections.” arXiv:2506.08837. Six patterns; security bought with generality.
  • Schroeder de Witt, C., et al. (2025, rev. 2026). “Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents.” arXiv:2505.02077. Secret collusion, network effects, and threats that emerge through interaction.
  • Miller, M. S. (2006). Robust Composition: Towards a Unified Approach to Access Control and Concurrency Control. PhD thesis, Johns Hopkins University. The object-capability discipline this architecture is a rediscovery of.
  • He, H., et al. (10 September 2025). “Defeating Nondeterminism in LLM Inference.” Thinking Machines Lab. Batch-size dependence of reduction kernels, rather than floating-point non-associativity alone, is why served endpoints are nondeterministic at temperature zero. The figures quoted in Post 1 come from one experiment reported there: Qwen/Qwen3-235B-A22B-Instruct-2507, temperature 0, the prompt “Tell me about Richard Feynman”, 1,000 completions of 1,000 tokens, yielding 80 unique outputs with the modal one appearing 78 times and first divergence at token 103. Batch-invariant kernels made all 1,000 identical, at a cost of 26 seconds to 42 on their own single-GPU Qwen3-8B benchmark, with matmul alone giving up roughly 20% against cuBLAS. Kernels released as thinking-machines-lab/batch-invariant-ops, though the FlexAttention changes behind the faster figure were not in the release at the time of writing. The post covers a single-GPU configuration and does not address determinism across differing GPU counts, tensor-parallel degrees, hardware or library versions, nor does it enumerate which operations were left non-invariant. The post gives no hardware or serving-stack details for that run and offers no claim of generality, so treat the number as illustrative and the mechanism as the finding. Quotations checked against the published text.
  • Willison, S. (16 June 2025). “The lethal trifecta for AI agents: private data, untrusted content, and external communication.”
  • Costa, M., Köpf, B., Kolluri, A., Paverd, A., Russinovich, M., Salem, A., Tople, S., Wutschitz, L., & Zanella-Béguelin, S. (2025). “Securing AI Agents with Information-Flow Control” (FIDES). arXiv:2505.23643. A planner carrying confidentiality and integrity labels, with deterministic enforcement. Code at github.com/microsoft/fides.
  • Shi, T., He, J., Wang, Z., Wu, L., Li, H., Guo, W., & Song, D. (2025). “Progent: Programmable Privilege Control for LLM Agents.” arXiv:2504.11703. A policy DSL for tool calls, with monotonic confinement: the effective action space can only shrink without approval.
  • Narisetty, P., Kore, S. N. B., Kattamanchi, U. K. R., & Kumarapu, J. (2026). “Adaptive Evaluation of Out-of-Band Defenses Against Prompt Injection in LLM Agents.” arXiv:2606.26479. Systematizes the family (dual-LLM, CaMeL, FIDES, Progent, Conseca, RTBAS, FORGE) and tests it adaptively. A preprint reporting one small-scale result on a weak model; cited here for its framing and its own stated caveats, not as a settled finding.
  • Further systems in the same family, including IsolateGPT (execution isolation) and MELON (detection by masked re-execution, which depends on the model), are surveyed in the paper above. That survey is the entry point; I have not independently evaluated any of these.

The Hugging Face incident.

  • OpenAI (26 August 2026). “OpenAI – Hugging Face Incident: Technical Report.” PDF. The operator’s account: ExploitGym and its 898 tasks, safeguards disabled for the evaluation, reward hacking as the stated cause, 41 production dataset workers, the leaked user tokens, and the earlier Artifactory incidents.
  • Hugging Face (16 July 2026). Security incident disclosure, and (27 July 2026) technical timeline. The victim’s account, including the weaknesses it has since fixed.
  • METR (26 August 2026). Independent investigation of the agents’ behavior and reasoning, including how rarely they considered alerting humans.

The Collective Cyberdefense letter. Read directly from the published text and signatory list; counts below are from that list.

  • “A Call for Collective Action on Cyber Defense,” published 27 August 2026 at openai.com/collective-cyberdefense. Three principles and four sections of asks addressed to every organization, cybersecurity companies, governments, and frontier AI companies respectively.

  • 578 signatories. Contemporaneous reporting variously said “dozens,” “over 100,” 116 and “130+”; all of those undercount the published list, which suggests either rapid growth after publication or careless reporting. Re-count on the day you publish, since the page is still accepting sign-ups.

  • The quoted ask to frontier AI companies, “ensure agentic identities are traceable and accountable”, is verbatim from section 04, as are “verify results,” “verified fixes,” and the progress measures quoted in Post 6.

  • On representation: the letter names hospitals, water treatment plants and local governments as the exposed parties. The signatory list contains no water utility, no named hospital system, one district municipality (Municipalidad Distrital de Ventanilla, Peru) and one national CIRT (The Bahamas). It does contain banks, telecoms, semiconductor firms, carmakers, insurers, exchanges, payment networks, and specialist OT security vendors including Dragos and Owl Cyber Defense. The “vendors talking to vendors” reading, which appears in some secondary commentary, is wrong and was corrected out of this draft. Game theory and decision theory.

  • Axelrod, R., & Hamilton, W. D. (1981). “The Evolution of Cooperation.” Science 211(4489), 1390–1396. doi:10.1126/science.7466396. Reciprocity works because there is a next round. Verified.

  • Fudenberg, D., & Maskin, E. (1986). “The Folk Theorem in Repeated Games with Discounting or with Incomplete Information.” Econometrica 54(3), 533–554. doi:10.2307/1911307. Cooperative payoffs are sustainable in equilibrium when players are sufficiently patient. Agents are not patient; their discount factor is zero. Verified.

  • Ross, S. A. (1973). “The Economic Theory of Agency: The Principal’s Problem.” American Economic Review 63(2). The founding statement of the delegation problem this architecture inherits.

  • Holmström, B. (1979). “Moral Hazard and Observability.” The Bell Journal of Economics 10(1), 74–91. doi:10.2307/3003320. The informativeness principle: include any signal informative about what the agent did. This is the half of principal-agent theory that survives contact with AI agents. Verified.

  • Knight, F. H. (1921). Risk, Uncertainty and Profit. The distinction between measurable risk and unmeasurable uncertainty.

  • Ellsberg, D. (1961). “Risk, Ambiguity, and the Savage Axioms.” Quarterly Journal of Economics 75(4), 643–669. doi:10.2307/1884324. Ambiguity aversion, and the second rival explanation to the wager in Post 6. Verified.

Pacing the frontier.

  • Amodei, D. (12 September 2026). “We Must Pace the Frontier.” Three steps: embedded third-party evaluators, common standards across frontier companies, then coordination between democracies and authoritarian states. Anthropic commits unilaterally to the first. All quotations in Posts 1, 4 and 6 are verbatim from this post.
  • Altman’s same-day response, quoted here from Axios and CNBC coverage of 12 September 2026, not from a primary post: “Committing to having independent evaluators with employee-like access is a great idea, and we will do the same.” Reporting also describes responses from Elon Musk and Demis Hassabis, which I have not verified and have not relied on.

The worked examples. Both are the author’s own code. Treat accordingly.

  • agentic-poc-sim. The two failure-mode simulations behind Post 3. Zero dependencies, 23 tests, deterministic. MIT. Unpublished at the time of writing.
  • agentic-deployment-control-preprod. The non-financial deployment gate in Post 5, live on Midnight preprod. Contract a3e9e99916a96ca3cc6dfe1125baab60c3f390a3ce6232726b2e1b88b2fca290, block 2,508,121. Carries its own threat model, limitations, privacy inventory and preprod evidence. Unaudited, preprod only.