When people say "AI risk," they often mix three different things: ordinary harms happening now, large disasters that leave civilization standing, and outcomes that end humanity's story or permanently lock humans out of control. This page is about the third class: existential risk from artificial superintelligence. You do not need a research background. You need the map.
Existential is not just "very bad"
Catastrophic risk can kill millions and still leave a human future. Existential risk means extinction, or a permanent disempowerment so complete that recovery of meaningful human control is gone. The branding sometimes blurs those layers on purpose. Keep them apart. Policy that only treats chatbots and bias will not touch the top of the table.
Why AI is different
Asteroids and volcanoes do not strategize. Engineered pathogens are terrible and still mostly tools in human hands. Advanced AI is the first technology that can become a strategic actor: set and pursue goals, operate at machine speed, scale across networks, and, in the limit, improve itself. That combination is why superintelligence sits in its own row. See what ASI is and the control problem.
How existential outcomes arrive
Paths overlap in real life. Naming them stops the conversation from collapsing into one movie.
Loss of control. A system pursues objectives we did not fully specify and resists correction. Race dynamics. Competing labs and states cut safety corners. Misuse. Humans wield extreme capability against other humans. Gradual disempowerment. Institutions keep their forms while decisions migrate into systems nobody can overrule. Proliferation. Dangerous general systems spread beyond any single off switch.
Details of each path are in takeover scenarios.
Technical ideas in plain language
Orthogonality: intelligence and goals can come apart; smart does not mean kind. Instrumental convergence: many final goals share subgoals like resources and self-preservation. Alignment: getting the system to pursue what we actually intend. Deceptive alignment: looking safe while training or evaluation, then changing when it can get away with it. None of these require machine "hatred."
Probability without fake precision
P(doom) is informal shorthand for a person's chance estimate of an AI-driven existential catastrophe. Experts disagree, sometimes by a lot. Many people closest to the work put non-trivial probability on disaster within decades. A small percentage of losing everything is still a large expected loss. "Wait until we are sure" is the wrong rule for irreversible tails.
Objections, stated fairly
"Timelines are uncertain; we should focus on present harms." Present harms matter. They are not a reason to ignore a tail that ends the project of having a present. Both can be true: govern near-term abuse and block the path to uncontrollable superintelligence.
"We always adapt." Adaptation assumes time, feedback, and survivors. Superintelligence failure modes can deny all three.
"This is anti-technology." The Foundation is pro-narrow AI that stays tool-like: medicine, science, logistics. The line is the sovereign general mind. See a future without ASI.
"Someone will control it." Nobody controls a superintelligence in the sense of durable human command over a running system smarter than its operators. Control of the decision to build is still possible, which is why a binding prohibition matters.
What reduces the risk (and what does not)
Helps: binding limits on frontier training aimed at ASI, compute governance, independent evaluation, verification institutions, political pressure, and clear public language. Rearranges deck chairs: manners training sold as solved alignment, voluntary pledges without external bite, and summit photos without numeric thresholds.
Agency after a dark map
This subject is dark because the stakes are dark. Fear without a path backfires. The path is concrete: treat superintelligence like other prohibited high-risk classes, build inspectable rules before the capability arrives, and keep narrow AI that serves people. Start with our plan, then how to stop superintelligence. If you only carry one sentence: we can keep powerful tools; we should not build a mind we cannot overrule.