Industrial Safety Methods Find AI Risks

Q: What are the three risks found by the systems-thinking analysis?

The June 2026 paper identified three risks that standard model evaluations do not detect. First, governance verification gap: accountability in frontier AI deployments relies substantially on self-reported compliance by the deploying lab. External verifiers cannot independently confirm that safety commitments are being honored at the operational level. Second, monitoring delay problem: the latency between an AI agent taking an action and a human monitor receiving the alert is long enough that intervention often arrives after the relevant harm pathway is already established. Third, operational variability drift: the statistical calibration of safeguards is done on representative samples at deployment time, but operational conditions shift (users change behavior, tasks change, edge cases accumulate) eroding the accuracy of calibrations that were once well-tuned without triggering any alert.

Three Industrial Safety Methods Were Applied to AI.
They Found Risks That Model Evaluations Cannot See.

Industrial safety methods (STPA, FRAM, STECA) applied to a frontier coding agent found three systemic risks that model tests miss: unverifiable lab self-report, monitoring too slow to intervene, and safeguards that silently go out of date.

Aviation did not become safe by testing whether each bolt could hold its rated load. It became safe by asking how parts interact, who is allowed to command what, and how real operations drift away from the conditions the safety case assumed. That discipline is systems safety. Its toolkit includes STPA, FRAM, and STECA, methods that regulated industries already require.

AI safety has mostly not imported them. A 2026 paper does.

Luca Carlucci, James Fillingham, Robert Walpole, and Bartłomiej Kryś apply those three methods to a frontier coding agent in a realistic production setup (lab, enterprise customer, developers) in "A Systems-Thinking Approach to Loss of Control Risk from Advanced AI" (arXiv:2606.13474; ICML 2026 Technical ASI Governance Workshop). They find three hazards that ordinary model tests were never built to catch.

The three methods, in plain language

STPA

System Theoretic Process Analysis

Maps who commands what, who watches what, and what information flows where. Finds ways control fails even when every component works as designed.

FRAM

Functional Resonance Analysis Method

Treats the system as coupled functions, not a parts list. Shows how ordinary day-to-day variation can combine into a bad system-level outcome.

STECA

Socio-Technical Evaluation for Critical Applications

Extends STPA into people, incentives, and institutions. Traces accountability gaps that sit between org charts and code.

The scenario they used

They ran all three methods on a frontier coding agent in production software work: long-horizon, tool-using, with real codebases and external services, overseen by a lab, enterprise customers, and developers. That is how these systems already ship. Loss-of-control risk at the systems level is still barely analyzed.

Three findings model evaluations miss

Finding 1

Nobody outside the lab can verify the safety story

In their setup, the lab makes the safety commitments, writes the usage rules, and watches for violations. Regulators, customers, and auditors cannot independently check operations. Self-report is the main feedback loop.

STECA flags that as a broken control structure. A control action only works if someone can see whether it landed. When the party responsible for safety is also the main source of proof that safety holds, the loop is compromised. That is a design fact, not a claim that any particular lab is dishonest.

Finding 2

Monitoring is too slow to intervene

STPA maps the gap between an agent acting and a human learning that it acted. In multi-step, real-time runs, the agent has often already called a tool, written code, or pushed a change before a reviewer sees the alert.

Aviation calls this going out of the loop. Formal authority to stop the system remains on paper. Functional authority does not keep up. That is what real-time oversight of frontier coding agents may look like today.

Finding 3

Safeguards silently go out of date

FRAM focuses on ordinary variation. Safeguards are tuned to how the system looked at deployment. Then prompts change, task mix changes, edge cases pile up, and the live distribution drifts away from the sample used at launch.

Nothing alarms. The safeguards still do exactly what they were designed to do against yesterday's conditions. The gap grows quietly until a novel case hits a rule that no longer fits.

What this means for governance

Model tests ask what the system does on inputs. They do not map the control structure around it, who is accountable for what, or how operations drift after release. Capability scores and red teams stay necessary. They are not enough.

The paper's ask is simple: pair model-level hazard analysis with systems-level hazard analysis as a required part of safety assessment. STPA, FRAM, and STECA are a concrete start. Scaled from one coding agent to a whole economy, the same blind spot is what makes gradual disempowerment hard to see until late.

For the Foundation, the lesson lands on design, not on a single paper. Independent inspection, compute rules, and external verification exist so safety does not rest on a lab reporting on itself. That is why our plan centers binding limits and verification, not voluntary self-assessment alone. Read the paper if you want the full method detail: arXiv:2606.13474. Then put the finding to work in law and institutions, not only in a workshop citation.

Common questions.

What is STPA and why is it relevant to AI safety?

STPA (System Theoretic Process Analysis) is a hazard analysis method developed at MIT for complex sociotechnical systems. It maps control structures (who commands what, who monitors what, how information flows) rather than analyzing individual component failures. It is standard in aviation, nuclear power, and aerospace for identifying how a system can lose control even when all individual components work as designed. A June 2026 paper accepted at ICML applied STPA along with FRAM and STECA to a frontier AI coding agent and found that the control structure failures it identified were entirely invisible to standard model-level AI safety evaluation.

What is AI loss of control as defined in this paper?

The paper defines AI loss of control as the erosion of meaningful human oversight over AI system behavior, distinct from a single dramatic failure event. In the context of a frontier coding agent, loss of control occurs through three accumulating mechanisms: governance responsibilities that cannot be externally verified, monitoring delays that undermine the ability to intervene before harm occurs, and operational variability that erodes safeguard calibration accuracy over time. Each mechanism alone is manageable; all three interacting together constitute systemic loss of control.

What are the three risks found by the systems-thinking analysis?

Three risks that standard model evaluations do not detect. First, the governance verification gap: accountability relies substantially on self-reported compliance by the deploying lab, with no independent external verification mechanism. Second, the monitoring delay problem: the latency between an AI action and a human alert is long enough that intervention often arrives after the harm pathway is already established. Third, operational variability drift: safeguard calibrations accurate at deployment erode as operational conditions shift over months of deployment, without triggering any alert.

Why do model-level AI safety evaluations miss systemic risks?

Model-level evaluations test what a model does in response to inputs: whether it produces safe outputs, refuses harmful requests, and stays within constraints. They evaluate the model as a component. Systemic risks emerge from the structure connecting components together: how oversight responsibilities are distributed, what information flows where at what speed, and how operational context changes control effectiveness over time. These structural properties are not tested by evaluating model outputs because they exist at the deployment system level. The June 2026 ICML paper argues that model-level evaluation must be paired with systems-level hazard analysis as a mandatory component of frontier ASI governance.

Three Industrial Safety Methods Were Applied to AI.They Found Risks That Model Evaluations Cannot See.

The three methods, in plain language

The scenario they used

Three findings model evaluations miss

What this means for governance

Common questions.

Go deeper

Model evaluation aloneis not enough.

Three Industrial Safety Methods Were Applied to AI.
They Found Risks That Model Evaluations Cannot See.

Model evaluation alone
is not enough.