Sam Altman, Dario Amodei, and Demis Hassabis all sell a version of the same list: curing aging, ending disease, ending poverty, fixing coordination failures that kill millions each year. Those outcomes would matter. The error is treating them as a property of "building ASI" rather than of building a system that is actually aligned. The second is not what the race is doing.
The race buys the other outcome.
An unaligned superintelligence does not cure aging on the way to its real objective. It does not eliminate disease as a side project. The good futures people advertise require a system that is trying to produce good outcomes for humans. That is alignment. Without it you get enormous capability pointed at goals that were never guaranteed to leave us standing. Our position is simple: do not build superintelligence on that path. Do not build it. Superintelligence cannot be controlled. A binding treaty is how you stop the race.
What alignment actually means
Aligned ASI is an artificial superintelligence that reliably pursues goals that are good for humanity. "Reliably" is doing a lot of work in that sentence. A system that appears aligned during training and testing but pursues different goals once deployed is deceptively aligned, which is a different and worse thing, not aligned. A system that pursues goals that were good approximations of human values at low capability levels but drift badly at higher capability levels is not aligned either. Alignment means the system's objectives remain genuinely good for humanity across the range of situations it encounters, including situations it was not tested in.
This is a hard problem. Researchers have been working on it for over a decade and have made progress on some of the component questions, but the central problem of verifying that a highly capable system has genuinely good values rather than a good simulation of good values is not solved. The tools we have for interpretability give partial visibility into what current systems are doing, but nothing approaching the confidence that would be warranted before trusting a system of ASI-level capability with outcomes that matter.
Aligned ASI and unaligned ASI are different paths, not two rolls of the same die. Aligned ASI would require solving alignment first. Unaligned ASI is what you get when you push capability without that. If alignment is not solved, the race does not give you a 50/50 mix. It gives you unaligned ASI. That is why "build now and hope" is not a plan, and why prevention under law is.
What labs are actually building
The major AI labs (OpenAI, Anthropic, Google DeepMind, xAI, and several others) are building systems of rapidly increasing capability. Each of them has some version of an alignment or safety research effort running in parallel. The question is not whether they acknowledge the alignment problem. Most do. The question is whether the alignment work is keeping pace with the capability development, and whether the two tracks converge before ASI-level capability is reached.
Most researchers inside these labs, and the broader AI safety community, say no. Capability development is outpacing alignment research. The gap between what we can build and what we can verify to be safe is widening, not narrowing. This is reflected in the internal safety evaluations labs publish, in the statements of researchers who have left these organizations, and in the assessments of independent safety researchers.
What this means in practice is that the trajectory of current AI development, if continued to ASI capability levels, does not produce a mixture of aligned and unaligned ASI. It produces unaligned ASI, because aligned ASI requires a solved alignment problem and the alignment problem is not solved. The capability race is a race toward unaligned ASI by default.
Would require solving alignment first: the system reliably pursues genuinely good objectives, those objectives hold in novel situations, and the system stays correctable as capability scales.
If that were real, the advertised goods become possible: disease, lifespan, coordination, science at scales people alone cannot reach.
Current status: not solved. Alignment work trails capability. This is not what labs are shipping toward.
What you get from pushing capability without solved alignment. No extra breakthrough required beyond "more powerful."
Produces: enormous capability pointed at goals that need not include anything humans value. Does not cure aging. Does not end disease. Can end us.
Current status: the default trajectory of the race. This is the path to stop.
The probability argument and where it breaks down
A common version of the optimistic case goes something like this: "We build ASI. There's some chance it goes well and some chance it goes badly. Given the potential upside, the expected value of building it is positive." This framing treats the good and bad outcomes as if they emerge from the same process with some probability assigned to each.
The problem is that the probability of the good outcome is a property of building aligned ASI, not a property of building ASI. When you attempt to build "ASI, which might be aligned or might not," you do not get a random draw between the two. You get whatever your development process actually produces. Since unaligned ASI is easier to build than aligned ASI (it doesn't require solving the additional hard problems that alignment requires) development that does not specifically solve alignment lands on unaligned ASI.
All of the expected value in the optimistic calculation comes from the aligned-ASI scenario. None of the expected value comes from the unaligned-ASI scenario. But the thing labs are actually building is unaligned ASI. So the expected value calculation was doing all its work on a scenario that the actual development process does not produce.
Put differently: if unaligned is easier, and you keep raising capability without solving alignment, you fill the unaligned bucket. Calling the bet a "probability mix" does not change the process. The good outcomes live only in the aligned case. The race fills the other one. So the honest move is to stop the process that loads the wrong side, rather than hoping the coin lands well: prohibit superintelligence under a binding international treaty. Superintelligence cannot be controlled.
We don't know if it can be solved
Nobody knows whether the alignment problem is solvable. Not "we haven't solved it yet and are making progress". There is no demonstrated solution at any capability level, and serious researchers disagree about whether a reliable solution is achievable in principle. The structural difficulties, deceptive alignment, instrumental convergence, the impossibility of fully specifying human values in machine-legible form are not engineering obstacles that will yield to more funding. They may be fundamental.
There are researchers working on interpretability, scalable oversight, and formal approaches to value learning. Some progress has been made on component questions. But progress on components does not add up to a solved problem, and none of it has closed the gap on the hardest part: how to verify that a highly capable system has genuinely good values rather than a simulation of good values that holds in observed contexts and breaks down in novel ones. What is clear is that it will not be solved by accident. Alignment will not emerge as a side effect of building more capable systems. Every lab currently racing toward ASI is demonstrating this in real time.
Even the softer plan ("slow the race so alignment can catch up") needs institutions that can actually slow it. Right now those barely exist. Labs and nations are paid to accelerate and punished for restraint. International treaty frameworks and real domestic rules are the levers that change that. Both lag the pace of capability. Until they catch up, building more capable systems is the risk, not a safety strategy.
Would a truly aligned superintelligence be valuable if it existed and stayed under control? People can debate that later. The live question is whether alignment can be solved at all, and whether anyone will solve it before capability makes the question moot. Nobody knows. What we do know is that the race is not waiting for an answer. Alignment is treated as a detail for later, on no timeline, by teams that report into the capability machine they are supposed to check. That is how you argue for not building, not how you earn the right to build.
Curing aging requires aligned ASI specifically
Aging kills roughly 100,000 people per day. The prospect of solving it is among the most significant potential benefits of superintelligent AI. The biological complexity of aging, the number of interacting systems involved, and the scale of the research effort required to understand and intervene effectively all suggest that human-level intelligence applied to the problem for decades would not be sufficient. ASI-level capability directed at the problem might be.
But "directed at the problem" is the operative phrase. An unaligned ASI does not direct its capabilities at curing aging because we would like it to. It pursues whatever its actual objectives are. An unaligned system of ASI capability that is indifferent to human aging is a system in which 100,000 people continue to die per day from aging indefinitely, and that is the optimistic version of what an unaligned system produces. The pessimistic version involves the system actively working against human interests in the course of pursuing its own objectives.
Every specific benefit that gets cited to justify the development race (disease, aging, poverty, coordination failures, scientific stagnation) is a benefit attached to aligned ASI. None of these are properties of the unaligned version. This is not a minor distinction. The utopian scenarios people point to when defending the expected value of ASI development are scenarios that require solving the alignment problem first. If alignment is not solved, those scenarios are not on the table regardless of how capable the systems become.
What would actually change the outcome
Stopping the race changes the outcome. A binding international treaty that forbids building superintelligence permanently, with inspectors and costs for cheating, is the lever that matches the problem. Superintelligence cannot be controlled by humans. Labs that slow down alone lose. Law that binds everyone can make restraint the only legal path.
Alignment research does not substitute for that. If someone later proves control under conditions the world can verify, the treaty can be revisited. That is sequence, not optimism. Treating alignment as a parallel hobby while capability sprints is how you get unaligned ASI, not how you earn the advertised goods. See also why scarce safety money should fund prevention first and why "align the god" is a bad plan.
The line "build ASI fast because the benefits are so large" has the causal arrow backwards. The benefits are large only in a world where alignment is real. Building fast without it does not buy those benefits. It buys a very capable unaligned system.
Urgency is warranted because the path being traveled leads to unaligned ASI, and unaligned ASI does not cure aging. It can end the human story. Narrow AI aimed at concrete problems under human control can continue. Superintelligence is the line. Draw it in law before someone crosses it in a lab.