The precautionary principle emerged from environmental law and is now embedded in international instruments. Its best-known formulation is Principle 15 of the 1992 Rio Declaration: where there are threats of serious or irreversible damage, lack of full scientific certainty shall not be used as a reason for postponing cost-effective measures to prevent it. Variants appear throughout European Union law and in treaties on climate, biodiversity, and hazardous substances. The principle exists precisely for situations where waiting for proof means waiting too long.

The principle holds that where an activity threatens serious or irreversible harm, the absence of full scientific certainty must not be used to postpone protective action. That rule is the strongest legal foundation for acting on AI risk before a catastrophe demonstrates the point.

Why it fits AI so well

The case for AI risk has an awkward feature: the most serious scenarios have not happened, and cannot be demonstrated in advance without running the experiment. Critics use this to dismiss the concern, no proof, no problem. The precautionary principle is the direct answer. It was built for threats that are grave and potentially irreversible but not yet certain, and it inverts the burden: the question is not whether catastrophe is proven, but whether the possibility is serious enough to warrant protective action given what is at stake.

Two features of frontier AI make it close to a textbook case for the principle. The potential harm is on the highest end of the severity scale, plausibly catastrophic and irreversible. And the science is genuinely uncertain, with credible experts assigning meaningful probability to loss-of-control scenarios and no reliable method to rule them out. Grave, irreversible, uncertain: this is exactly the profile the precautionary principle was written to address.

What precaution requires in practice

  • Shifting the burden of proof. Developers of the most capable systems should bear the burden of demonstrating safety before deployment, rather than society bearing the burden of proving danger after the fact, the model already used for pharmaceuticals and nuclear plants.
  • Acting before certainty. Governance should not wait for a demonstrated catastrophe to justify binding limits, because for irreversible risks the demonstration is the disaster.
  • Proportionality to the stakes. The scale of precaution should match the scale of potential harm; the highest-severity risks justify the strongest measures.
  • Reversibility as a priority. Where possible, keep options open and avoid steps that foreclose future correction, a principle directly relevant to a technology that could entrench itself.

The objections

The precautionary principle has real critics, and honest advocacy engages them. The strongest objection is that, taken to an extreme, precaution can paralyze: almost any activity poses some conceivable catastrophic risk, and a principle that says 'act against all unproven grave threats' could justify blocking beneficial innovation on speculative grounds. Critics also note that precaution has its own costs (forgone benefits, including the possibility that AI itself could avert other catastrophes) and that these belong in the calculation.

These points do not defeat the principle; they discipline its use. The answer is that precaution is proportional action against threats that are both serious and credibly supported, weighing the costs of action against the costs of inaction, not 'ban everything that could conceivably go wrong'. Frontier AI qualifies not because someone can imagine a bad outcome, but because credible experts assign real probability to catastrophic, irreversible loss of control. The principle asks for measures proportionate to that specific, well-supported risk, not for a blanket halt to technology.

For an irreversible risk, 'wait for proof' is the opposite, not caution. The precautionary principle exists so that we are not required to suffer the catastrophe to earn the right to prevent it.

The foundation for acting now

The precautionary principle matters for ASI governance because it dissolves the most common argument for delay, that we should not regulate until the risks are proven. In law and in ethics, that argument fails for exactly the class of threat AI represents. Grave, irreversible, and uncertain harms are the paradigm case for acting before certainty, with the burden on those creating the risk to show it is contained. This does not settle what the measures should be; that is the work of treaty design, verification, and thresholds. But it settles the prior question of whether it is legitimate to act at all under uncertainty. It is, and the principle explains why.