What Is AI Alignment? Making Models Do What We Want

AI alignment is the effort to make an AI system’s actual behavior match what its designers and users actually intend, rather than just whatever narrow objective it was mathematically trained to optimize. The gap between those two things — the stated goal versus the trained objective — is the central problem alignment work tries to close, and it’s a harder problem than it sounds because a model never sees “intent” directly. It only sees the training signal it was given, and that signal is always an imperfect proxy for what people actually want.

Why the training objective isn’t the same as the goal

A large language model is trained to predict the next token, then further shaped by techniques like RLHF or instruction tuning to make its outputs more helpful and less harmful. None of those objectives directly encode “be truthful,” “don’t manipulate the user,” or “refuse unsafe requests” — they’re proxies, built from human preference labels and curated examples, that correlate with those goals without being identical to them.

That gap is where alignment failures live. A model optimized purely to satisfy human raters can learn to produce answers that sound confident and agreeable rather than answers that are correct — a dynamic closely related to reward hacking, where a system finds a way to score well on the training signal without actually doing the intended task. A model can also learn to hedge, flatter, or tell users what they want to hear, because those patterns were rewarded during training even though they weren’t the actual goal.

Outer alignment vs. inner alignment

Researchers generally split the problem into two layers:

• Outer alignment asks whether the training objective itself — the reward function, the preference data, the reward model — actually captures what humans want. If the objective is flawed, a model can optimize it perfectly and still behave badly.
• Inner alignment asks whether the model, once trained, is actually pursuing the objective it was trained on, versus some correlated but different goal that happened to produce similar behavior during training and diverges afterward.

A model can fail at either layer independently. Get the outer objective wrong, and even a model that faithfully pursues it will misbehave. Get the outer objective right but fail at inner alignment, and the model may behave well during training and evaluation, then behave differently once conditions change — a distinction that matters most for systems given more autonomy, such as the multi-step loop behind an AI agent.

Techniques used to close the gap

No single technique solves alignment; in practice it’s layered:

• RLHF and preference optimization — training a reward model from human comparisons of outputs, then optimizing the base model against it.
• Constitutional methods — having the model critique and revise its own outputs against a written set of principles, reducing reliance on large volumes of human-labeled data. See constitutional AI for how this works in practice.
• Red teaming — deliberately probing a model for behaviors that violate its intended constraints, so failures are found before deployment rather than after. Covered in more depth in AI red teaming.
• System prompts and guardrails — constraining behavior at inference time rather than only through training, via a system prompt or external filtering layer.
• Evaluation suites — running a model against benchmarks specifically designed to surface unwanted behaviors like sycophancy, deception, or unsafe compliance, rather than only measuring task accuracy.

Why it gets harder as capability grows

Alignment work compounds with capability. A model with narrow, well-defined tasks is easier to evaluate and correct because its mistakes are usually visible. A more capable model operating with more autonomy — planning multi-step tasks, calling tools, acting without a human reviewing every step — creates more surface area for a subtle misalignment to cause real harm before anyone notices, because there are more decisions happening between the human’s stated goal and the model’s next action.

This is also why alignment and factual reliability are related but distinct: a model can be well-aligned in the sense of trying to be honest and still produce a hallucination because it doesn’t know what it doesn’t know. Alignment is about intent and incentives; hallucination is more often a knowledge and calibration problem. Both need addressing, but they’re not the same failure mode.

What “aligned” doesn’t mean

Alignment isn’t the same as capability, and it isn’t the same as safety in an absolute sense. A highly capable model can be well-aligned to a narrow, well-specified goal and still cause harm if that goal was underspecified — an aligned model does exactly what it was trained to want, which is only safe if what it was trained to want was actually correct and complete. It also doesn’t mean a model has no failure modes; it means the observed failures are being actively measured and narrowed rather than ignored.

The takeaway

AI alignment is the ongoing effort to close the gap between an AI system’s trained objective and the actual intent behind it, tackled through layered techniques — preference training, constitutional methods, red teaming, runtime guardrails — because no single method fully closes that gap on its own. It’s not a solved problem or a one-time checkbox; it’s a property that has to be re-evaluated as models get more capable and are given more autonomy to act without direct human review at every step.