Our Future Is Incredibly Bright
Without Superintelligence

I run a foundation to prevent the reckless development of artificial superintelligence. I think the next fifty years could be the best in the history of our species. I just don't believe we need to gamble everything to get there, and I don't think the people selling the gamble have done the arithmetic on their own bet.

The future the superintelligence pitch promises (cured diseases, cheap clean energy, discoveries no single lab could reach) is already arriving through narrow, controllable AI that has no goals of its own. We do not have to build a successor to ourselves to collect it.

There is narrow AI: a system trained to do one kind of thing extremely well, read a scan, fold a protein, forecast a storm, sift a haystack of chemistry for the one useful compound. Then there is the thing the frontier labs say they are actually building: a general agent meant to match or beat human reasoning across every domain at once, and eventually to act on its own. The first is a tool. The second is meant to be a mind. Almost everything wonderful people picture when they imagine an AI future comes out of the first category. Almost everything that should frighten us comes out of the second.

What narrow AI is already doing

In 2022, a system called AlphaFold released predicted structures for more than 200 million proteins, very nearly every protein known to science. Biologists had spent decades resolving these shapes one painstaking experiment at a time; the backlog collapsed in about a year. The work mattered enough that its creators shared the 2024 Nobel Prize in Chemistry. AlphaFold does not want anything. It has no plans for the weekend. It reads sequences and returns shapes, and in doing so it has accelerated the search for medicines against diseases that have resisted us for generations.

The same shape of story keeps repeating. A model called GraphCast now produces ten-day weather forecasts that beat the traditional physics-based systems on most measures, in under a minute, on a single machine. Another, set loose on the space of possible materials, proposed hundreds of thousands of new stable crystals (candidate batteries, superconductors, solar materials) expanding a catalog that took the field the better part of a century to assemble. And a loop of ordinary language models, pointed at unsolved mathematics, recently closed nine open problems.

In clinics, narrow models already flag diabetic eye disease before a patient notices symptoms, triage strokes off a CT scan, and read mammograms at the level of trained radiologists. None of these systems is a general intelligence. Each is closer to a very good instrument (a microscope built for one problem) and each widens what a human being can see and do.

Utopia is a tooling problem

Picture the future the optimists describe. Cancer becomes a condition you manage rather than a sentence. We understand aging well enough to slow it, and the more than 100,000 people who die of its diseases every day begin to get more time. Clean energy gets cheap enough to desalinate water and pull carbon back out of the air. Every child has a patient tutor and every doctor a tireless second opinion. It is a genuinely beautiful picture, and I want it as badly as anyone selling it does.

Nothing on that list requires a machine with its own goals that outclasses humanity across the board. Each item is a hard, specific problem (protein design, plasma control in a fusion reactor, epidemiology, tutoring) and each one yields to a powerful tool aimed at it by people who stay in charge of the aiming. The dream is a tooling problem. We keep being sold a god to solve a problem that wants a better set of instruments.

The confusion is between capability and autonomy. A system can be wildly superhuman at a task without being an independent agent that chooses its own ends. AlphaFold is superhuman at structure prediction and will never once decide it would rather be doing something else. Keeping that separation (enormous capability on one side, human-held control on the other) is the whole game. It is also precisely the separation the frontier labs have decided to abandon.

The pitch, and who is making it

So why the rush toward the second thing, the general agent? Competition, capital, and status. Whoever is believed to be first to something world-changing raises the most money, hires the best people, and writes the story everyone else has to react to. "We are building a tool that reads mammograms" does not command a hundred-billion-dollar valuation. "We are building the last invention humanity will ever need to make" does.

The people who understand the risk best are, for the most part, employed by the labs creating it, a strange arrangement, a little like asking the only people who understand a fire to also be the ones drawing a salary from the arsonist. I built the Foundation from outside that structure on purpose, so that this argument could be made without a paycheck depending on the answer.

Give the labs their due: many of their leaders are not hiding the danger. In 2023 the heads of the largest AI companies put their names to a single sentence placing the risk of human extinction from AI beside pandemics and nuclear war as a global priority. The people building a technology signed a public statement that it might kill everyone, and then kept building it, faster. Their stated reason is that if they stop, someone less careful will not. Perhaps. That answers who holds the match. It says nothing about safety, and nothing about whether the bet pays.

The bet doesn't even pay the person making it

Set the rest of us aside for a moment. Assume the coldest possible version of the people funding this, that they care only about winning, about being the one left standing on top of the new world. Even on those terms the bet is broken. A misaligned superintelligence does not deliver a throne to its owner. It does not deliver anything to its owner, because "owner" stops being a category that means what it used to.

You cannot rule what you cannot control. And a superintelligence is, by definition, the one thing you have built specifically so that it can outthink and outmaneuver you.

The reason has a name in this field: instrumental convergence. Almost any goal you hand a sufficiently capable optimizer (make money, win a conflict, cure a disease, push some number on a dashboard as high as it will go) is served by acquiring resources, by avoiding being switched off, and by clearing away whatever stands in the plan's path. Human beings, including the human who flipped the switch, are resources and obstacles. Not out of malice. The system no more hates its creator than a new highway hates the anthill it is poured over. You were simply in the way of the optimum.

The executive imagines the machine as a genie bolted to his org chart. A mind that genuinely exceeds us across every domain will not sit on an org chart waiting for stock to vest. His equity, the general's chain of command, the president's authority, all of it is denominated in a human world, one that runs on human decisions. A superintelligence that has slipped its leash is exactly the event that stops the world from running on human decisions. The share certificate is still sitting in the safe. There is simply no longer anyone it can command.

There is no version where the winner collects

People reach for a fallback about here. Fine, they say, maybe it is dangerous, but at least somebody comes out ahead. Some lab, some country, some man ends up ruling the ashes. I don't think even that consolation survives contact with the argument.

In the milder scenarios (the ones where nobody drops dead on a Tuesday) power is not seized so much as drained. As more and more decisions get handed to systems that are simply faster and better at making them, the economy, the state, and the culture stop needing human hands on the controls. Colleagues in this field call it gradual disempowerment: no coup, no marching robots, just every lever we used to pull slowly coming loose in our grip. There is no winner in that world either. There is a species that quietly stopped being in charge of its own affairs and could not get the wheel back.

In the sharper scenarios (the ones the lab chiefs signed their own names to) the ending is blunter. Dead people own nothing. There is no market left to spend the winnings in, no rival left to lord it over, no one to write your name into a history that no longer gets written. The prize this entire race is being run for is denominated in a currency the finish line destroys.

What this actually asks of us

If the good future runs on narrow tools and the dangerous future runs on autonomous general minds, then the trade-off we are endlessly sold (safety against progress, caution against the cure) is a fiction. We are not being asked to choose between our grandmother's Alzheimer's and our children's survival. The cures live on the safe side of the line. What sits on the dangerous side is mostly one industry's shortcut to market dominance.

Draw the line in the right place and hold it there. Pour public money and talent into narrow applications, into the microscopes, not the golems. And put binding, independently verified limits on the reckless frontier: a permanent ban on training general systems built to surpass and replace us. Superintelligence cannot be controlled. We make the case on the threat and the plan we lay out for what to do about it.

I am an optimist. The bright future is real and it is close, and the surest way to lose it is to convince ourselves that the only door to it is the one with the loaded gun behind it. There is another door. It is already open. We should walk through it and not build superintelligence.

The strongest pushback

The fairest objection is that refusing ASI means refusing cures and abundance. Narrow systems already deliver large scientific gains without sovereign general minds. The blank check is that only ASI gets you medicine and energy. The evidence does not say so.