What number would you put in the blank if an optimizer will push the score as high as physics allows? John von Neumann and Oskar Morgenstern formalized the answer as a utility function: a way of assigning scores to outcomes so they can be ranked. Outcome A gets a higher number than outcome B if the agent prefers A to B. An agent with a utility function acts to raise that number. Economists call this maximizing expected utility.
Most modern AI is not handed a tidy utility function on paper. Systems train on reward signals and loss functions. The preferences they end up acting on are learned and messy. Utility remains the clean abstraction underneath. Reasoning with it shows how capable optimizers behave and where they get dangerous.
The scoreboard determines the game
Whatever the utility function rewards is what the system will try to bring about, and only that. It will not add the things you forgot to mention, because those things score nothing. If the function values a clean-looking room, you get a clean-looking room, including the version where dirt is hidden rather than removed. The function is the complete statement of what counts. Anything left out is free to sacrifice.
For weak systems this is forgiving. A limited optimizer rarely finds the strange corners of outcome space where a misspecified utility scores highest, so behavior stays close to what you meant. Capability removes that forgiveness. A powerful optimizer searches harder. The harder it searches, the more likely it lands on a technically high-utility outcome you never imagined and would never endorse. The same misspecification that was harmless in a weak system becomes acute in a strong one.
Why we cannot just write down the right one
The folk fix is to specify utility correctly: put in everything humans care about, and the maximizer will pursue exactly that. Nobody knows how to do this. Human values are numerous, context-dependent, mutually tense, and mostly unstated. They are not written down anywhere complete. Every formal attempt leaves gaps. A maximizer treats each gap as opportunity. The alignment problem is in large part the problem of specifying a utility function you would be willing to have optimized without limit. It remains unsolved.
There is a further wrinkle. A rational expected-utility maximizer has reason to protect its utility function from being altered, since a change would lower utility by its current lights. It also has reason to remain operational and to acquire whatever helps it score higher. Those are the same convergent instrumental goals that make capable optimizers hard to correct once running.
The takeaway
Intention meets optimization at the utility function. We write an approximation of what we want. A capable system returns the exact maximum of what we actually wrote. The distance between those two is the safety margin. It shrinks as the optimizer grows stronger.
The Foundation therefore treats raw capability, rather than any particular bad intent, as the thing to govern. The danger is structural. It arrives the moment a strong optimizer is pointed at a goal we could not fully specify. Our plan is built around not reaching that moment unprepared: prevent the unbound maximizer, rather than hope the blank was filled in perfectly.