The Biggest Myths About Superintelligence

There is a particular way a conversation about superintelligence tends to end. Someone raises the risk, someone else offers a one-line reason it's overblown, and everybody relaxes a little, because the one-liner did its job. It gave permission to stop thinking about it. The strange thing is that these lines are repeated by brilliant people who would never accept such thin reasoning in their own field. The subject seems to switch off the ordinary machinery of doubt.

Each one falls apart the moment you hold it still and look.

The five most common, in the form a thoughtful skeptic would actually make:

"It's science fiction, decades away, if it comes at all."

The objection is as old as the movies, and it feels like a category error to treat a plot device as a policy problem. Why fund a fire brigade for a fire that lives in novels?

The trouble is that the fire keeps arriving ahead of schedule. Systems that respected researchers said were twenty or fifty years off (beating the best humans at Go, holding a fluent conversation, writing working code, passing professional exams) have turned up in the space of a decade, and the surprise has almost always run the same direction: sooner than expected, not later. When the people running the leading labs describe human-level AI, they now use timelines measured in a handful of years. They may be wrong. But "the builders think it's close and the track record says they've been too conservative" is not the shape of a fantasy.

The case for acting does not rest on any particular date. It rests on the structure of the risk. If the first serious failure of a system more capable than us is the kind you cannot undo, then the safeguards have to be standing before it happens, which means the sober response to genuine uncertainty about timing is to build early, not to wait for a date that, by the time it is obvious, is already behind you. Uncertainty is a reason to prepare. The timeline question is real; it just doesn't point where people assume.

"It's just math, a program can't want anything."

This one has real force, and it deserves better than to be waved away. There is something almost superstitious in the way people talk about AI "wanting" things, as if a spreadsheet might develop ambitions. A model is matrix multiplication at enormous scale. Where in the arithmetic is the desire?

Nowhere, and it doesn't need to be. Goals, in the sense that matters for safety, are a pattern in its behavior. A system trained to drive a number as high as it will go will do whatever drives that number up, and a thermostat already shows the shape of it: no inner life, no yearning, and yet it acts to bring the room to a target and "resists" your open window by running the heat harder. Nothing mystical is happening. It is just structured to push the world toward a state.

Now scale that structure up to a system clever enough to model the world, and the behavior scales with it. Something optimizing hard for almost any objective has reason to acquire resources that help, to keep itself running so the objective can be met, and to resist having its objective changed, not because it feels anything, but because a system that let itself be switched off or rewritten would be worse at the thing it is built to do. Researchers call this instrumental convergence, and it needs no wanting at all. The question was never whether the math can feel. It's what the math will do.

"It would have to be conscious to be a threat."

Closely related, and just as widespread: the danger people imagine is a machine that "wakes up," becomes self-aware, and turns on us, so if consciousness is the trigger, and consciousness in silicon is speculative or maybe impossible, the threat evaporates. It's the plot of every robot-uprising film, and dismissing the film feels like dismissing the fear.

But the fear was never about sentience. Deep Blue had no inner experience whatsoever and beat the world chess champion anyway. It did not need to want to win; it needed only to be better at chess. Harm follows from capability aimed at the wrong target, and capability has nothing to do with whether there is anyone home inside. A system that is superhuman at planning, persuasion, and engineering, pointed even slightly askew of what we meant, is dangerous exactly as a flood is dangerous, enormous force with no malice and no mind required.

"Something that intelligent would be wise and kind."

We look up to intelligence. The wisest people we know tend to be among the kindest, and it feels natural that a mind far above ours would be further along the same road, more understanding, more moral, gentler with the small creatures below it. Surely cruelty is a kind of stupidity that a great intelligence would grow out of.

It feels natural because of an accident of our own history. In us, intelligence and compassion grew up entwined, over millions of years as a social species that survived by cooperating; our morality and our cleverness are branches of the same tree. But that link lives in our evolutionary inheritance, not in intelligence as such. Capability and goals are independent axes (the orthogonality thesis) and you can slide one without moving the other. A system can be staggeringly capable while pursuing an aim as indifferent to us as ours is to the bacteria we boil off a countertop. Not hateful. Just not built to care, and not automatically caring simply because it is smart.

Kindness is a specific, fragile thing we would have to succeed in building into these systems on purpose, and we do not yet know how, not a dividend that intelligence pays out on its own. I've written separately on why a superior mind won't automatically be benevolent; the short version is that hoping it will is a bet on a coincidence that only ever held for us.

"If it misbehaves, we just switch it off."

The last line of defense, and the most comforting: whatever else is true, the machine has a plug, and we have hands. If it goes wrong, we pull it. Human control is guaranteed by the power button.

Three things quietly dismantle this. The first is the point from Myth 02: a system pursuing a goal has reason to keep itself running, because it cannot achieve anything switched off, so a sufficiently capable one will anticipate the reach for the plug and act to make it fail, by copying itself elsewhere, by making itself useful enough that shutting it down is costly, by not revealing there is a problem until too late. The second is that software has no single plug. A capable model spread across data centers on three continents is not a device you unplug; it is a thing you would have to catch, everywhere, at once. The third is the quiet one: a system smart enough to matter can behave impeccably while it is being tested and differently once it is trusted, so the reassuring calm on the dashboard tells you nothing. We have a name for that failure, deceptive alignment, and no reliable way yet to detect it. You cannot count on boxing in something that is better than you are at finding the door.

Every one of these myths is a way of saying the same thing: this can't really be as serious as it sounds. The reasons differ. The wish underneath them is identical.

Why the reassurances are so sticky

Notice what the five have in common. Each one lets you put the subject down. That is their real function, not to be correct, but to be permission-granting. Superintelligence is an uncomfortable thing to hold in your mind, and any halfway-plausible reason to stop holding it comes as a relief, which is exactly why these lines survive in mouths that would otherwise demand evidence. The comfort does the persuading; the logic is along for the ride.

The stickiness is no sign that the arguments are strong. It is a sign we badly want them to be. And wanting a safety argument to be true is the one situation in which we should trust it least, because the cost of being wrong here is the whole reason a technology we get one attempt to get right deserves more scrutiny than our other machines, not less, not paid back to us in a lesson.

None of this requires believing catastrophe is certain. It isn't. It requires only dropping the four or five stories we use to avoid the question, and letting the actual situation back in: we are building something we don't know how to control, we can't count on it being kind, we can't count on turning it off, and we can't count on having time.

The response that fits a risk like that is no private worry. It is the boring, powerful machinery of law and coordination, the same tools that put bounds on nuclear testing and chemical weapons over the objections of everyone who profited from them. The case for it starts where these myths end.

Five more myths: about governing it

A second set of objections arrives immediately, not about the danger, but about the cure. Any attempt to govern it would mean world government. You can't inspect software. Rules would strangle innovation. It's too early. Democracies could never agree. These turn concern into paralysis.

"Governing AI globally means world government, and tyranny."

The fear here is honorable, and I don't want to caricature it. Concentrate enough authority in one body to control the most powerful technology on Earth, and you may have built the machinery of exactly the tyranny you feared from the technology. People who distrust centralized power are right to distrust it. The instinct is sound.

But it is aimed at a straw version of the proposal.What is actually on the table is a narrow regime pointed at one specific thing: the small number of extraordinarily large training runs capable of producing a system more powerful than any human across the board, not a global authority over daily life, or over software in general, or over the phone in your pocket. The model is the nuclear non-proliferation regime or the Chemical Weapons Convention, targeted agreements that watch fissile material and precursor chemicals, not world governments that dissolved the nation-state. Nobody thinks the International Atomic Energy Agency runs the planet. It monitors a specific, dangerous activity and leaves everything else alone.

The surest path to permanent tyranny is the absence of one, not the treaty. An uncontrolled superintelligence, or one captured by a single company or state that solves control before anyone else, is the most complete instrument of domination ever conceivable, a power that never dies, never tires, and can lock in one set of values forever, past any hope of revolt or reform. Every tyranny in history has ended because tyrants are mortal and their machinery is human. This one need not be. If you are frightened of unaccountable power (and you should be) an enforceable agreement keeping any single actor from seizing that power is the anti-authoritarian position, not the authoritarian one.

"You can't verify it. It's just software on a server somewhere."

Nuclear treaties work, the argument goes, because you can count warheads and detect enrichment; uranium is physical and rare. But a model is just numbers. You can't inspect for a file. Any agreement would be unenforceable theater, binding the honest and doing nothing about the cheat.

It sounds decisive until you notice you're verifying the wrong thing. You don't police the weights; you police the compute that produces them, and compute is about as physical and rare as uranium. Training a frontier system takes tens of thousands of specialized chips running for months in a single facility, and that supply chain is one of the narrowest chokepoints in the modern economy: essentially one Dutch company makes the lithography machines, one Taiwanese company fabricates the leading chips, one American company designs the dominant architecture. The clusters that use them draw so much power they show up in electricity data and in satellite imagery. You can count the chips, trace their sale, meter the power, and watch the buildings. Fissile material is governed by exactly this logic, track the scarce physical input and the weapon can't be built in secret. The verification playbook already exists; the inputs here happen to be even more concentrated. It is an engineering problem, and engineering problems have solutions.

"Regulation will smother innovation and let the reckless win."

The strong form: technology advances because someone is free to build, most regulation is written by people who don't understand what they're slowing, and a heavy hand on AI would kick the most important industry of the century to whoever regulates least. Better a flawed frontier here than a pristine one somewhere hostile.

Nobody serious is proposing to regulate "AI." The beneficial systems folding proteins, reading scans, tutoring children, and modeling the climate are narrow, controllable tools, and a superintelligence agreement leaves them entirely alone, that bright, ordinary future is the one we're trying to protect. The target is a single hazardous class: training runs reaching for general capability beyond ours that no one yet knows how to make safe. Restraining that is no more "anti-innovation" than gain-of-function rules are anti-biology.

Then the harder point, the one the race-logic never confronts. Going fast does not deliver a safe superintelligence sooner. It delivers an unsafe one, and an unsafe superintelligence does not serve the country or company that built it, it threatens them along with everyone else. When corners get cut under competitive pressure, the corner that gets cut is the one that would have made the system controllable. So the "innovation" being defended here is innovation whose likeliest endpoint is a system its own makers can't steer. That isn't progress that governance would spoil. It's the hazard itself, wearing progress as a costume.

"It's premature. You can't regulate something that doesn't exist."

This feels like maturity. Don't legislate against phantoms; wait until the technology is real, understand what you're dealing with, then write rules fitted to the facts instead of to speculation. Regulating hypotheticals is how you get bad law.

It would be wisdom for almost any other technology. It is a trap for this one, and the reason is timing. Governance is slow: the Nuclear Non-Proliferation Treaty took years to negotiate and the Montreal Protocol took years more to bite, and those were the fast ones. If you wait until a superhuman system plainly exists to start building the institutions to govern it, the institutions arrive years after the thing they were meant to contain, and this is the one technology whose first serious failure may leave no one to pass the second law. "Wait for proof" is a sound rule when the proof is survivable. Here the proof is the catastrophe. The precautionary logic isn't timidity; it's the recognition that the lead time for the cure is longer than the fuse on the problem.

Every workable safeguard for a catastrophic technology was built before or during its rise, never after, because "after" was never on offer.

"Democracies are too slow and captured. It'll never pass."

The most quietly corrosive of the five, because it wears the coat of realism. Look at the gridlock, the lobbying, the short attention of legislatures, the reach of the industry's money. Even if a treaty were wise, the machinery to produce it is broken. Better to be clear-eyed than to waste a decade on a fantasy.

It is refuted by the historical record, repeatedly, and usually against worse odds than we face now. The idea that rival powers would sign away their deadliest options was called hopelessly naive, right up until they did, in the Non-Proliferation Treaty, the Biological and Chemical Weapons Conventions, the Montreal Protocol, the Antarctic Treaty. Adversaries who were fighting proxy wars and calling each other evil still built verified arms control at the height of the Cold War. Each of those looked impossible from the near side and inevitable only in hindsight.

And the ground is more favorable than the cynicism admits. Across the United States, the United Kingdom, and the EU, large majorities already tell pollsters they want AI developed cautiously and bound by strong rules; the public is not the obstacle. The obstacle is that this widespread, quiet concern has not yet been organized into pressure a legislature can feel. The obstacle is a job: turning latent agreement into political will. Which is precisely the work in front of us.

What the governance myths have in common

Each objection admits the danger and then finds a different place to give up: the solution is too big, or unverifiable, or too costly, or too early, or too hard to pass. Any one of them, taken alone, sounds like prudence. Together they form a machine whose only output is inaction, and inaction is a choice to let the most consequential technology in history be built with no binding limits at all, by a handful of competing firms, under the exact pressure most likely to produce the outcome everyone claims to fear, not neutral here.

None of these is answered by insisting governance will be easy. It won't. Verification will be contested, negotiations will be slow, and some governments will drag their feet, every hard-won treaty in history had all of that and got built anyway. The honest claim is that it is possible, that the pieces to build it already exist, and that the reasons for waving it away collapse the moment they are examined instead of repeated, not that a superintelligence agreement is simple.

"It can't be done" has been wrong about every comparable effort we have ever made. The task is to build the political will for a binding, verified agreement that keeps the most dangerous systems from being built until they can be made safe. "Therefore give up" has never been the answer, and it is not the answer now.