The Interruption
Problem

I. The asymptote

Large language models are probabilistic machines. This is not a defect to be engineered away; it is the mathematical character of the technology. Each token is a draw from a distribution, each step in a reasoning chain inherits the uncertainty of the steps before it, and in any sufficiently long sequence of dependent actions, error is not a possibility but an eventuality. A system that is 99 percent reliable per step completes a twenty-step task correctly about 82 percent of the time. At 95 percent per step, the twenty-step task succeeds barely more than a third of the time. The arithmetic is unforgiving, and it explains a pattern that has now been observed across thousands of enterprise deployments: models keep improving, benchmarks keep climbing, and production systems keep failing in ways that no benchmark predicted.

The industry's own diagnosticians have converged on this reading. Gartner projects that by 2028, 40 percent of enterprise AI failures will trace not to model capability but to inadequate evaluation and monitoring of agentic systems. Contemporary analyses of production deployments put agent failure rates in real workflows between 70 and 95 percent, a figure that sounds implausible until one recognizes that it measures end-to-end task completion rather than per-response quality. The reliability curve bends toward 100 percent and never arrives. What changes as models improve is not whether the residual exists but where it hides, and how expensive it is when it surfaces.

This residual has a specific economic character. It is not distributed evenly across tasks. It concentrates in the moments that matter most: the irreversible action, the ambiguous edge case, the situation the training distribution never contained. An agent that drafts internal summaries can be wrong at 5 percent with no consequence. An agent that issues refunds, files insurance claims, screens job candidates, or speaks to patients on the telephone converts the same 5 percent into chargebacks, regulatory exposure, and litigation. The question that determines whether autonomous systems can be deployed at scale is therefore not how good the model is. It is what happens in the gap.

II. The production paradox

The gap is now visible in the adoption statistics, which describe an industry in a peculiar state: near-universal experimentation and near-universal hesitation, simultaneously.

Roughly 79 percent of enterprises report having adopted AI agents in some form, yet only about 11 percent run them in production. McKinsey's 2026 survey work finds 88 percent of organizations using AI in at least one function while only 23 percent are scaling agentic AI anywhere in the enterprise, and its authors conclude that the barrier is not the technology but governance, accountability, and operational ownership. Gartner's forecast is blunter still: more than 40 percent of agentic AI projects will be canceled by 2027, with escalating cost, unclear return, and weak risk controls named as the causes. Only 21 percent of organizations report a mature governance model for autonomous agents. And in one of the stranger sentences ever to appear in an analyst forecast, Gartner projects more than 2,000 legal claims involving harm from autonomous AI systems, what it calls death-by-AI claims, by the end of 2026.

The same period has seen extraordinary spending on the problem, but the spending is asymmetric in a revealing way. Enterprise budgets for AI evaluation and observability now average roughly 310 thousand dollars annually at mid-market firms and 2.4 million dollars at Fortune 500 companies, with 71 percent of enterprises increasing those budgets for 2026. The tooling this money buys is genuinely sophisticated: tracing platforms, evaluation harnesses, drift monitors, quality scorers. The venture market has capitalized the category at well over two billion dollars, with the leading evaluation vendor raising 121 million dollars and the leading framework company valued above one billion on the strength of its observability product.

Yet all of this infrastructure shares a single architectural commitment. It observes. It records what the agent did, scores how well the agent did it, and reports the findings after the fact. None of it can act. When an agent is mid-execution and beginning to go wrong, with a customer on the other end of the transaction, the most heavily funded stack in enterprise software offers a rearview mirror of increasing resolution and no steering input of any kind. The industry has built an elaborate apparatus for knowing about failure and almost nothing for interrupting it.

This is the interruption problem, and it is the actual bottleneck beneath the adoption paradox. Enterprises are not declining to deploy agents because they lack information about agent quality. They are declining because deployment currently means accepting unbounded downside: an error, multiplied by machine speed and transaction volume, discovered only in retrospect. The missing layer is the one that converts unbounded risk into bounded risk, and its absence is measured in that gulf between 79 percent adoption and 11 percent production.

III. What a control layer is

The engineering discipline that studies this problem predates artificial intelligence by half a century. Supervisory control theory, developed by Thomas Sheridan and colleagues at MIT from the 1970s onward, describes the architecture by which humans govern semi-autonomous systems: the machine executes, the human monitors at a level of abstraction above the execution, and, critically, the human retains calibrated authority to intervene, redirect, or halt. The theory's central finding is that the intervention channel is not an accessory to automation but a precondition of it. Automation without an intervention channel is not autonomy; it is abdication.

Applied to AI agents, a supervisory control layer has a definite shape. It sits in the action path rather than beside it, which is the decisive architectural distinction from observability. Consequential actions, defined by policy rather than by hardcoded lists, pause at a gate before execution. The pending action, its parameters, and its context are routed to an accountable human, who approves, modifies, or denies it, after which execution resumes incorporating the verdict. The intervention itself, who decided, on what basis, with what change to the outcome, is recorded automatically in a form that can later reconstruct the decision. Around this primitive sit the operational necessities: a fleet-level view of running agents, policies that tune the gating threshold as evidence accumulates, and a fail-open guarantee ensuring that an outage of the control layer never becomes an outage of the systems it supervises.

Two findings from the research literature complicate the picture in productive ways, and any serious version of this layer must be built around them rather than in ignorance of them.

The first is that human oversight, naively implemented, decays into ritual. The phenomenon is documented under several names: automation bias in the human-factors literature, and what recent work on AI evaluation calls the rubber-stamp effect, in which reviewers under time pressure come to trust the machine's judgment and approve its outputs even when they are wrong. Studies of human verification of AI-generated labels find that anchored reviewers drift toward blind agreement, and that when those contaminated judgments are then used to calibrate automated judges, the errors compound. Lisanne Bainbridge named the underlying paradox in her 1983 paper Ironies of Automation: the more reliable the automation, the less practiced the human supervisor, and the less capable of intervening precisely when intervention matters most. A control layer that merely inserts a human produces the appearance of oversight. One that works must manage the human's attention as deliberately as it manages the machine's actions, through calibrated gating volumes, injected vigilance tests that verify reviewers actually catch seeded errors, and measurement of oversight quality as a first-class metric.

The second is that the review function itself is partially automatable, and pretending otherwise produces bad economics. Contemporary research on model-based evaluation finds agreement between LLM judges and human raters reaching Cohen's kappa values of 0.84 to 0.92 on well-specified rubrics, near-perfect concordance at a fraction of the cost, while the same literature documents systematic judge biases, verbosity preference, self-preference, hallucinated rationales, that require continuous human calibration. The mature architecture is therefore a triage pyramid: deterministic rules catch cheap failures instantly, model-based judges score everything at scale, and human judgment is reserved for the apex, the cases where liability, ambiguity, regulation, or rubric-setting demand a person. The human's role in this architecture is not volume review. It is accountability, calibration, and the resolution of the tail. That division of labor is what allows a supervisory layer to scale economically as agent volume grows, rather than reproducing the cost structure of the manual processes agents were meant to replace.

There is also a moral dimension, articulated most sharply in Madeleine Clare Elish's work on what she called the moral crumple zone: the tendency of sociotechnical systems to assign blame to the nearest human when automation fails, regardless of how little actual control that human held. A control layer, honestly built, is the corrective to this pattern rather than an instance of it. It gives the accountable human real authority at the moment of decision, real context on which to exercise it, and a real record of what they could and could not have known. Oversight without authority is scapegoating infrastructure. Authority with records is governance.

IV. The historical pattern

Every prior wave of automation that acquired the capacity to act in the world converged on the same institutional response, generally in the same sequence, and generally only after a visible disaster.

Industrial machinery came first. The emergency stop is now so ubiquitous that it reads as furniture, but it is a legal artifact: IEC 60204 and its national counterparts require the emergency stop function on essentially all machinery, and equipment does not achieve certification without it. The entire discipline of supervisory control and data acquisition, the software layer through which industrial operators monitor and command physical processes, encodes the same settlement. The industry that automated first concluded, a century ago, that sensing without the capacity to intervene is half a system.

Algorithmic trading provides the closest software precedent, and the most instructive failure. Trading algorithms were among the first software agents entrusted with autonomous, high-value, irreversible actions, and for years firms operated them with rich monitoring and no mandatory interruption capability. On August 1, 2012, Knight Capital deployed a misconfigured trading system that no one could stop, and lost approximately 440 million dollars in roughly 45 minutes while its staff watched the losses accumulate on their monitoring screens. The regulatory response was structural rather than rhetorical: SEC Rule 15c3-5 mandated pre-trade risk controls in the execution path, exchanges built kill-switch functionality into market infrastructure, and inline risk gates became a permanent, unremarkable feature of a market that once considered them friction. Aviation ran the same arc earlier: no autopilot is certified without an instantaneous disconnect under the pilot's hand, and the entire modern doctrine of flight-deck automation is organized around the management of authority between human and machine rather than the elimination of the human.

The sequence is stable across domains. Automation gains agency. It operates for a period under observation but without enforced interruption. A failure occurs whose defining feature is that everyone could see it and no one could stop it. Regulation then mandates the interruption capability, and the inline control layer becomes standing infrastructure, invisible precisely because it is universal.

AI agents are, at this writing, the first consequential automation in roughly a century to be deployed at scale ahead of that settlement. They act in payment systems, communication channels, hiring pipelines, and clinical workflows, and the standard deployment ships with telemetry and without a brake. What is historically unusual about the present moment is not that the settlement is missing. It is that, for the first time, the regulation is arriving before the sector's Knight Capital rather than after it.

V. The regulatory environment, read honestly

The law is the strongest tailwind behind a supervisory control layer, and also the source of its most genuine headwinds. Both halves deserve an unsentimental reading.

The tailwind is concrete and dated. The European Union's AI Act, Regulation 2024/1689, entered full application for high-risk systems on August 2, 2026. Its Article 14 requires that high-risk AI systems be designed for effective human oversight, including, in the regulation's own language, the ability of a human to intervene in or interrupt the system through a stop button or a similar procedure. Article 12 requires automatic recording of events across the system's lifetime, sufficient to permit post-hoc reconstruction of individual AI-assisted decisions, and deployers must retain those logs for a minimum of six months per entry, in a form that remains readable, intact, and correlatable, accessible on demand to national market surveillance authorities. Penalties scale to seven percent of global turnover. Legal commentary on early enforcement has already identified the operative failure mode: organizations rarely fail these reviews because their algorithms were irresponsible, they fail because their audit trail could not answer hard questions, and ad hoc logs assembled after the fact are treated as equivalent to no system at all. The regulation, in other words, does not merely permit a control layer. It describes one, down to the button.

The Article 14 jurisprudence contains a further subtlety with direct product implications. Oversight must be meaningful rather than symbolic, and analyses of compliant record-keeping emphasize that records in which the final outcome differs from the AI's original proposal are not an edge case but a regulatory expectation. A log showing uniform approval is, to a competent examiner, evidence of the rubber-stamp effect rather than of oversight. Compliance under this standard therefore cannot be satisfied by documentation alone; it requires an operational mechanism that demonstrably changes outcomes, which is precisely what a documentation-first governance industry cannot supply.

The American landscape supplies a second, more chaotic tailwind. Texas's Responsible Artificial Intelligence Governance Act took effect January 1, 2026, and grants an affirmative defense to organizations that substantially comply with a recognized risk framework such as the NIST AI Risk Management Framework, which makes documented oversight not merely a compliance cost but a litigation shield. Colorado repealed its 2024 AI Act and replaced it with SB 26-189, effective 2027, centering on consumer notices, adverse-outcome explanations within thirty days, and meaningful human review rights, a formulation that converts oversight from a policy into a staffed workflow, since a right to demand human review presupposes a human and a workflow to receive the demand. California layered multiple statutes effective January 2026, Illinois and New York City regulate AI in employment decisions, and insurance regulators are moving through the NAIC model framework. The pattern across jurisdictions is fragmentary in detail and consistent in direction: the human review requirement keeps appearing, attached to different triggers, in different sectors, with different deadlines running through 2030.

Now the headwinds, which are real and should be stated as plainly as the tailwinds. First, the regulatory timeline itself is unstable. The European Commission's Digital Omnibus package, proposed in late 2025, contemplated delaying the Annex III high-risk obligations, possibly to December 2027, and while nothing had passed into law as of mid-2026, leaving August 2026 the enforceable date, the negotiation itself injects planning uncertainty into every European compliance budget. Second, the American patchwork faces a federal preemption campaign: a December 2025 executive order directed federal agencies to press against state AI laws and asked the FTC to opine on when state requirements are preempted by federal deceptive-practices law. No preemption has been enacted, and the order itself exempts several regulatory categories, but any thesis resting exclusively on Colorado or Texas is resting on contested ground, which argues for anchoring the compliance case on the EU regime, on sectoral regulators in finance, insurance, and healthcare whose authority predates the AI debate, and on the procurement channel described below, with state law treated as acceleration rather than foundation. Third, the technical standards that will operationalize the logging requirements, prEN 18229-1 on AI logging and human oversight and ISO/IEC 24970 on AI system logging, remain unfinished drafts. This cuts both ways: builders today are constructing to a regulation that defines outcomes without specifying methods, which is risk, and builders who construct to the drafts now will be conformant when the standards land while their competitors retrofit under pressure, which is opportunity. Fourth, there is a genuine tension between the logging mandate and data protection law: comprehensive decision records can themselves accumulate personal data that GDPR disciplines or forbids, and the emerging practice, retaining hashes and classifications rather than raw content, with the customer rather than the vendor as custodian of the records, is as much a legal architecture as a technical one.

The honest synthesis is that regulation makes the supervisory layer inevitable while making its precise specification unstable. That combination punishes two strategies, waiting for clarity and building to a single statute, and rewards one: building the operational capability, interruption, accountable decision, reconstructable record, that every plausible version of the rules requires, in a schema flexible enough to render against whichever text finalizes.

VI. The procurement channel

Parallel to public regulation, and arguably faster-moving, a private regulatory system has assembled itself inside enterprise purchasing, and it may prove the more immediate forcing function because its penalties are denominated in lost revenue rather than eventual fines.

In October 2025, the Cloud Security Alliance released AI-CAIQ, an artificial-intelligence extension of the consensus questionnaire that has governed cloud vendor security review for a decade, mapped to a controls matrix of 243 controls across 18 domains. The Shared Assessments SIG questionnaire added AI-specific modules in its 2026 revision. The consequence, documented across practitioner surveys and legal advisories through 2026, is that essentially every meaningful vendor security questionnaire now contains a scored block on AI governance: model inventory and provenance, training-data rights, output monitoring procedures, hallucination controls, subprocessor transparency, and alignment with the NIST framework and ISO/IEC 42001, the AI management-system standard that procurement analysts describe as migrating from differentiator to baseline in financial services, healthcare, and the public sector. SOC 2, the previous gate, is now assumed rather than evaluated. The canonical complaint of 2026, quoted across security-industry commentary, is that the questionnaire grew an AI section and the vendors cannot answer it.

The structural significance of this channel is that it propagates. A regulation binds the entities in its jurisdiction; a questionnaire binds everyone who sells to anyone who uses it, which in B2B software approaches everyone. This is the mechanism by which SOC 2 became universal without any statute requiring it, and the mechanism is now loaded with the oversight question. For any vendor whose product takes consequential actions, the question describe what happens when your model is wrong has become a deal-gating event, and the available answers divide cleanly. A policy document describes intentions. An operational control layer produces evidence: which actions were gated, who decided, what changed, within what response times, with what demonstrated reviewer vigilance. The governance-documentation industry, which is substantial and growing, sells the former. The latter is what the questionnaire is actually asking about, and at present, almost no one sells it.

VII. The economics of the judgment layer

The market context can be sized from three directions, each with a different degree of speculation, and the honest ordering runs from the narrow and certain to the broad and conjectural.

The narrow base is the spending that already exists. The evaluation and observability budgets cited earlier, 310 thousand dollars at mid-market and 2.4 million at the Fortune 500 on average, growing at 71 percent of enterprises, establish that reliability tooling for AI systems is a funded line item today, not a category awaiting invention. The analyst bucket in which oversight infrastructure formally sits, AI trust, risk, and security management, is estimated at roughly 4.9 billion dollars in 2025 with projections toward 39.8 billion by 2035, growth driven disproportionately by model governance and compliance, its largest segment, and by European demand growing above 22 percent annually under direct Act pressure. A single quarter of European enforcement activity in 2026 was credited with an estimated 180 million dollar surge in compliance-platform procurement.

The intermediate frame is Gartner's own category call, which is unusual in that the analyst named the layer before the market consolidated it. Gartner forecasts that guardian agents, its term for the oversight and control layer of agentic systems, will capture 10 to 15 percent of the agentic AI market by 2030, on the explicit reasoning that as multi-agent systems interact at machine speed, human oversight in its traditional form cannot keep pace and must be re-implemented as infrastructure. Against an agentic AI market projected to grow from roughly 7.6 billion dollars in 2025 to 50 billion or more by 2030, with Gartner separately forecasting that 40 percent of enterprise applications will embed task-specific agents by the end of 2026, up from under 5 percent a year prior, the guardian layer alone describes a five to seven billion dollar category within the decade. IDC's projection of up to 1.3 billion AI agents in operation by 2028 gives the same conclusion in units rather than dollars: a population of autonomous actors that large cannot be governed by staffing, only by infrastructure.

The broad frame is a labor argument rather than a software argument. Every workforce in economic history has carried a supervisory layer, quality control, management, audit, compliance, typically absorbing 5 to 15 percent of total labor cost. As enterprises transfer work from employees to agents, that supervisory function does not disappear; it currently has no vendor, and it hides inside headcount, inside the 100-percent-manual-review regimes that quietly convert AI products back into drafting tools, and inside the deployment hesitancy measured in Section II. If agents perform even a conservative slice of the work projected for them by the early 2030s, a supervisory function priced at a small fraction of the value of that work describes an annual market in the high single-digit billions, of which the compliance-shaped portion counted by the analysts is only the visible fraction.

One property distinguishes the economics of this layer from those of the observability tools it superficially resembles: its unit costs are designed to fall. Every human intervention is a labeled correction on a specific workflow, which becomes both a permanent regression test and calibration data for the automated tiers of the triage pyramid. Gating thresholds tighten as approval evidence accumulates; the share of actions requiring human attention declines while the record of oversight, the thing regulation and procurement actually demand, grows richer. The same mechanism answers the most common objection to the entire category, that improving models will erase the need for supervision. Model improvement moves the escalation threshold; it does not remove it, because the residual is mathematical, the accountability requirement is legal, and, as the industrial precedent shows, the emergency stop did not come off the machinery when the machinery improved. What model improvement does do is periodically destabilize every deployed system at once, since each major model release changes behavior across the installed base, and an accumulated failure history against which a new model can be replayed before cutover converts the layer's supposed obsolescence event into its moment of maximum utility.

VIII. The counterargument, taken seriously

The strongest case against this thesis is not that supervision is unnecessary but that it will be absorbed: by the model providers, who could ship oversight as a platform feature; by the frameworks, whose interrupt primitives already exist; or by the automation of judgment itself, as model-based evaluation closes the gap with human review.

Each absorption path runs into a structural limit. The model providers face a conflict of position: oversight of an agent must be credibly independent of the agent, both to regulators, who will not accept the supervised system as its own supervisor, and to enterprises running models from multiple vendors, which is now the norm. The framework primitives are real but are functions, not governance: an interrupt call gives a developer a pause, while an organization needs fleet-level visibility across heterogeneous frameworks, policy administered by people who do not read code, an approval surface in the tools where accountable staff already work, and a record that survives audit, none of which a primitive supplies and all of which constitute the actual product. And the automation of judgment, far from dissolving the layer, is the layer's own internal economics: the kappa statistics cited earlier are an argument for the triage pyramid, not against the apex, because the functions that remain human under every regulatory text now in force, accountability, calibration, and the ambiguous tail, are precisely the functions the judges cannot absorb, and the vigilance research shows that even those functions fail without infrastructure deliberately built to keep them honest.

A subtler objection deserves the last word, because it is the one the field's own scholarship raises. If automation bias is as corrosive as the literature says, a control layer risks becoming what Elish warned of, a moral crumple zone at industrial scale, manufacturing accountable humans to absorb blame for systems they cannot meaningfully govern. The objection is correct about the failure mode and wrong about the conclusion. The choice on offer is not between flawed oversight and some purer alternative; it is between oversight built with the failure mode designed in, measured vigilance, calibrated volumes, genuine authority, reconstructable records, and the two regimes actually observable in the market today: symbolic oversight that certifies nothing, and no interruption capability at all. The scholarship is not an argument against the layer. It is the specification for building it properly.

IX. Synthesis

The argument of this essay reduces to five propositions. Autonomous systems built on probabilistic models carry an irreducible failure residual that concentrates precisely in their most consequential actions. The infrastructure industry has funded the observation of that residual at extraordinary scale while leaving the interruption of it essentially unbuilt, an omission explained by architecture, by the recency of agents that act, and by an ideological season that treated the human presence as a defect. A century of precedent across every prior automation wave indicates that the interruption capability becomes mandatory infrastructure, historically after disaster, and that the settlement is stable once reached. Law and private procurement are, unusually, arriving ahead of the disaster this time, in fragmentary and unstable form, but converging from every direction on the same operational demands: the ability to intervene, an accountable decision, and a record that can reconstruct it. And the economics of supplying those demands improve with scale rather than degrading, because every act of human judgment the layer captures makes the layer require less of it.

What follows from these propositions is a claim about the shape of the coming stack. Between the systems that act and the organizations answerable for their actions, a supervisory control layer will exist, as surely as the emergency stop exists on the factory floor and the risk gate exists in the market's execution path. The open questions are only who builds it, whether it is built before or after this sector's 440-million-dollar afternoon, and whether it is built to the standard the scholarship demands, oversight that is real, measured, and on the record, or to the standard the compliance economy will tolerate, which is paper. The gap between those two standards is where the next important infrastructure company is currently unclaimed.