LLM Alignment: Understanding Human Preferences, RLHF, DPO, and the Alignment Problem

A pretrained language model is optimized, over trillions of tokens, to assign high probability to whatever text tends to come next. Ask it “How do I reset a forgotten password?” and a pure base model might answer helpfully, or continue with “How do I change my username? How do I delete my account?”, because a list of FAQ questions is a perfectly likely continuation. It might also give confident nonsense, or imitate an argument from a forum. All of those are good next-token predictions. None of them is necessarily what the user wanted.

Alignment, in the practical sense used by people who train models, is the process of turning a model that predicts text into a model that behaves as intended: following instructions, being honest about uncertainty, refusing genuinely harmful requests without refusing harmless ones, and adopting a consistent character. In the broader sense used in AI safety research, it is the problem of making sure an AI system’s objectives match human intentions, even as the system becomes more capable than the people supervising it. Current post-training techniques are a partial solution to the first problem, and an open question for the second.

Why next-token prediction is not enough

The pretraining objective is

$$\max_\theta\; \mathbb{E}_{x \sim \mathcal{D}} \sum_t \log p_\theta(x_t \mid x_{<t}),$$

and its optimum is the distribution of the training corpus. That creates several gaps between what the model does and what we want.

• The corpus is not a model of good behaviour. It contains errors, misinformation, toxicity and every quality level of writing. A model that imitates it imitates all of that, in proportion.
• Plausible is not true. Nothing in the objective rewards truth over typicality. A fluent, confident, wrong answer can be very likely.
• Prompts are ambiguous about intent. The same text could start a question, a quiz, a joke or a forum thread. A base model hedges across all of them.
• Some intended behaviours are rare in text. Saying “I don’t know” when uncertain, asking a clarifying question, or declining a harmful request politely and briefly are uncommon in web text.

Alignment techniques add a second optimization stage that targets behaviour directly, using signals about which outputs are better. The field has converged on a pipeline whose shape was set by OpenAI’s InstructGPT work (Ouyang et al., 2022): supervised fine-tuning, then learning from preferences. A key result from that paper is still the best argument for the approach. Human raters preferred outputs from a 1.3B-parameter InstructGPT model to outputs from the 175B GPT-3 base model, despite a 100× difference in size.

Figure 1. Two post-training pipelines. RLHF learns an explicit reward model and optimizes against it with RL; DPO optimizes the same objective directly on preference pairs.

Stage 1: supervised fine-tuning

Supervised fine-tuning (SFT) trains the base model on curated examples of prompts and ideal responses, with the ordinary language-modelling loss applied only to the response tokens:

$$\mathcal{L}_{\text{SFT}}(\theta) = -\,\mathbb{E}_{(x, y) \sim \mathcal{D}_{\text{SFT}}} \sum_{t=1}^{|y|} \log \pi_\theta(y_t \mid x, y_{<t}).$$

SFT teaches the format of being an assistant: answer the question that was asked, follow the requested structure, use the chat template, stop at the end of the turn. It is surprisingly data-efficient. The LIMA paper (Zhou et al., 2023) fine-tuned a 65B base model on only 1,000 carefully selected examples and got a capable assistant, which led to the superficial alignment hypothesis: most knowledge and capability come from pretraining, and alignment mostly selects which existing behaviours to express.

SFT has hard limits, though:

• It can only imitate. The model learns to produce responses like the demonstrations, so it cannot exceed the quality of its demonstrators, and it learns their mistakes and stylistic quirks too.
• It gives no negative signal. SFT says “this response is good”, never “this one is worse, and here is the difference”. Many important distinctions, such as subtle factual errors, an unnecessary refusal, or a slightly misleading summary, are comparisons rather than demonstrations.
• Demonstrations are expensive. Writing an excellent answer is much harder than judging which of two answers is better.
• It can encourage hallucination. If a demonstration contains a fact the model does not actually know, training on it teaches the model to state things it cannot verify.

These limits motivate learning from comparisons.

Human preference data

Instead of asking people to write ideal answers, show them a prompt $x$ and two (or more) responses sampled from the model, and ask which is better. Each judgement gives a pair: the preferred response $y_w$ and the rejected one $y_l$. Raters typically follow detailed guidelines covering helpfulness, honesty and harmlessness.

Comparisons are cheaper and more reliable than demonstrations, but they are noisy. In InstructGPT, trained labelers agreed with each other about 73% of the time. Preferences also encode whatever raters happen to reward, including length, confident tone, formatting and agreement with their own views, and those biases flow into everything downstream.

Stage 2a: the reward model

Classical RLHF first turns preferences into a scalar reward model $r_\phi(x, y)$, usually initialized from the SFT model with its output layer replaced by a scalar head. The standard way to connect a scalar score to pairwise choices is the Bradley–Terry model, which says the probability that $y_w$ is preferred to $y_l$ depends on their reward difference:

$$P(y_w \succ y_l \mid x) = \sigma\big(r(x, y_w) - r(x, y_l)\big) = \frac{1}{1 + e^{-(r(x, y_w) - r(x, y_l))}}.$$

The reward model is trained by maximizing the likelihood of the observed preferences:

$$\mathcal{L}_{\text{RM}}(\phi) = -\,\mathbb{E}_{(x, y_w, y_l)} \Big[\log \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big)\Big].$$

Only differences matter, so the reward is defined up to a per-prompt constant. Its accuracy at predicting held-out human preferences is usually in the 65–75% range, close to the inter-rater agreement ceiling. The reward model is thus a learned, imperfect proxy for human judgement, and that fact drives most of what follows.

Stage 2b: RL against the reward model, with a leash

With a reward model in hand, the policy $\pi_\theta$ (initialized from the SFT model) is optimized to produce high-reward responses. Optimizing reward alone would be disastrous. The policy would find whatever outputs exploit the reward model’s errors: degenerate text, extreme length, or patterns the reward model happens to overrate. So the objective adds a penalty for drifting away from a frozen reference policy $\pi_{\text{ref}}$ (the SFT model):

$$\max_{\pi_\theta}\; \mathbb{E}_{x \sim \mathcal{D},\, y \sim \pi_\theta(\cdot \mid x)} \big[r_\phi(x, y)\big] \;-\; \beta\, \mathrm{KL}\big(\pi_\theta(\cdot \mid x)\,\big\|\,\pi_{\text{ref}}(\cdot \mid x)\big).$$

The KL term is what keeps RLHF stable. It keeps the policy in the region where the reward model was trained and is still roughly accurate, and it preserves the fluency and knowledge of the base model. The coefficient $\beta$ sets the trade-off between optimizing the proxy and staying close to the original model. In practice the KL is applied per token, by adding $-\beta \log \frac{\pi_\theta(y_t \mid \cdot)}{\pi_{\text{ref}}(y_t \mid \cdot)}$ to the reward at each step.

PPO

The policy is usually optimized with Proximal Policy Optimization (PPO; Schulman et al., 2017). Text generation is treated as a sequential decision problem: the state is the prompt plus the tokens generated so far, the action is the next token, and the reward arrives at the end of the response (plus the per-token KL penalty). PPO’s core is a clipped surrogate objective that prevents any single update from moving the policy too far:

$$\mathcal{L}_{\text{PPO}}(\theta) = \mathbb{E}_t\Big[\min\big(\rho_t(\theta)\, \hat{A}_t,\; \operatorname{clip}(\rho_t(\theta), 1 - \epsilon, 1 + \epsilon)\, \hat{A}_t\big)\Big], \qquad \rho_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\text{old}}}(a_t \mid s_t)}.$$

Here $\hat{A}_t$ is the advantage: how much better the chosen token turned out than the policy’s average expectation from that state. It is estimated with a learned value model (the critic) and generalized advantage estimation. The clipping means that once a token’s probability has changed by more than a factor of $1 \pm \epsilon$ relative to the policy that generated the data, there is no further gain from pushing it.

PPO-based RLHF works, and it produced InstructGPT, ChatGPT and Llama 2-Chat (which combined rejection sampling with PPO and used separate helpfulness and safety reward models). But it is heavy machinery:

• Four models in memory: the policy being trained, the frozen reference, the reward model, and the value model, each often the size of the policy.
• Generation inside the training loop. Every step samples fresh responses from the current policy, which is slow and complicates distributed training.
• Many sensitive hyperparameters: $\beta$, the clipping range, advantage normalization, learning rates for policy and critic, batch sizes, and reward normalization. Small implementation details are known to have large effects.

Simplifying this is the motivation for DPO.

Direct Preference Optimization

Rafailov et al. (2023) observed that the KL-regularized objective above has a closed-form optimal solution. For any reward function $r$, the policy that maximizes $\mathbb{E}[r] - \beta\,\mathrm{KL}(\pi \| \pi_{\text{ref}})$ is

$$\pi^*(y \mid x) = \frac{1}{Z(x)}\, \pi_{\text{ref}}(y \mid x) \exp\!\left(\frac{1}{\beta}\, r(x, y)\right),$$

where $Z(x) = \sum_y \pi_{\text{ref}}(y \mid x)\exp(r(x, y)/\beta)$ normalizes the distribution. The optimal policy reweights the reference by the exponentiated reward. The partition function $Z(x)$ sums over all possible responses, so this cannot be used directly. But it can be inverted to express the reward in terms of the policy:

$$r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x).$$

Now substitute this into the Bradley–Terry model. Preferences depend only on the difference in rewards between two responses to the same prompt, so the intractable $\beta \log Z(x)$ term cancels:

$$P(y_w \succ y_l \mid x) = \sigma\!\left(\beta \log \frac{\pi^*(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi^*(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right).$$

That gives a loss that can be minimized directly over the policy parameters, with no reward model and no sampling:

$$\mathcal{L}_{\text{DPO}}(\theta) = -\,\mathbb{E}_{(x, y_w, y_l)}\left[\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)\right].$$

The policy is its own reward model. The quantity $\hat{r}_\theta(x, y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}$ is an implicit reward, and DPO trains it with exactly the reward model’s loss.

The gradient shows what DPO actually does:

$$\nabla_\theta \mathcal{L}_{\text{DPO}} = -\beta\, \mathbb{E}\Big[\underbrace{\sigma\big(\hat{r}_\theta(x, y_l) - \hat{r}_\theta(x, y_w)\big)}_{\text{large when the model ranks the pair wrongly}}\,\big(\nabla_\theta \log \pi_\theta(y_w \mid x) - \nabla_\theta \log \pi_\theta(y_l \mid x)\big)\Big].$$

It raises the likelihood of the chosen response, lowers that of the rejected one, and weights each example by how wrong the current implicit reward is. Pairs the model already ranks correctly contribute little.

Figure 2. The Bradley–Terry model turns a reward margin into a preference probability. DPO’s gradient weight is largest when the model ranks a pair the wrong way round, and fades once the pair is ranked correctly.

import torch.nn.functional as F

def sequence_logprob(model, input_ids, response_mask):
    """Sum of log-probabilities of the response tokens under the model."""
    logits = model(input_ids).logits[:, :-1]
    targets = input_ids[:, 1:]
    logp = F.log_softmax(logits, dim=-1).gather(-1, targets.unsqueeze(-1)).squeeze(-1)
    return (logp * response_mask[:, 1:]).sum(-1)

def dpo_loss(policy, reference, chosen, rejected, beta=0.1):
    """chosen / rejected: dicts with input_ids (prompt + response) and response_mask."""
    pi_w = sequence_logprob(policy, chosen["input_ids"], chosen["response_mask"])
    pi_l = sequence_logprob(policy, rejected["input_ids"], rejected["response_mask"])
    with torch.no_grad():
        ref_w = sequence_logprob(reference, chosen["input_ids"], chosen["response_mask"])
        ref_l = sequence_logprob(reference, rejected["input_ids"], rejected["response_mask"])
    margin = beta * ((pi_w - ref_w) - (pi_l - ref_l))
    loss = -F.logsigmoid(margin).mean()
    accuracy = (margin > 0).float().mean()          # how often the implicit reward ranks correctly
    return loss, accuracy

DPO is essentially a classification loss over pairs. It needs two models (policy and frozen reference, whose log-probabilities can even be precomputed), no generation during training, and few hyperparameters. That simplicity made preference optimization accessible outside the largest labs. Zephyr-7B was trained with DPO on the UltraFeedback dataset, and Meta’s Llama 3 post-training used rounds of SFT, rejection sampling and DPO.

What DPO gives up

DPO’s equivalence to RLHF holds at the optimum and under its assumptions. In practice there are real differences:

• It is offline. DPO learns from a fixed dataset of pairs, usually sampled from some other model. It never sees its own new outputs during training, so it cannot discover or correct its own failure modes the way on-policy RL can. Iterative or online DPO, which regenerates and relabels pairs from the current policy in rounds, closes much of this gap.
• Its regularization is weaker than it looks. The KL constraint is only enforced implicitly, through the reference ratio on the data that appears in the dataset. Responses far from the data are unconstrained.
• Chosen likelihoods often go down. DPO only cares about the margin between chosen and rejected. It frequently lowers the likelihood of both, which can move probability mass to responses that appear in neither and hurt quality.
• It overfits deterministic preferences. When a pair is always preferred one way, the Bradley–Terry likelihood pushes the margin toward infinity. Azar et al. (2023) proposed IPO, which replaces the log-sigmoid with a squared loss toward a fixed margin, to address this.
• Head-to-head comparisons favour well-tuned PPO. Xu et al. (2024) found that a carefully tuned PPO pipeline outperformed DPO on several benchmarks, including challenging code generation, which suggests online exploration still matters for hard tasks.

A family of variants trades off these issues. KTO (Ethayarajh et al., 2024) learns from unpaired “good” or “bad” labels using a loss inspired by prospect theory. ORPO (Hong et al., 2024) folds a preference term into SFT and needs no reference model. SimPO (Meng et al., 2024) uses the length-normalized log-likelihood as the implicit reward, with a target margin and no reference model.

GRPO and verifiable rewards

For tasks with checkable answers, such as mathematics with a known solution or code with unit tests, the reward does not need to be learned. It can be computed. Reinforcement learning with verifiable rewards (RLVR) uses these rule-based signals, which are much harder to hack than a learned reward model.

DeepSeek introduced Group Relative Policy Optimization (GRPO) in DeepSeekMath (Shao et al., 2024) and used it at scale for DeepSeek-R1. GRPO removes PPO’s value model. For each prompt it samples a group of $G$ responses, scores them, and uses each response’s score relative to its group as the advantage:

$$\hat{A}_i = \frac{r_i - \operatorname{mean}(r_1, \dots, r_G)}{\operatorname{std}(r_1, \dots, r_G)}.$$

The group itself provides the baseline that the critic used to estimate. The policy update is then PPO-like, with clipping and a KL penalty against a reference model. This saves the memory and instability of training a critic, at the cost of generating several samples per prompt. With verifiable rewards on reasoning problems, it produced models that learned to write long chains of thought, check their work and backtrack, behaviours that were not demonstrated to them directly.

AI feedback and constitutions

Human preference labels are expensive, slow, and hard to scale to topics that require expertise. RL from AI feedback (RLAIF) replaces some or all human labels with judgements from a capable model. Anthropic’s Constitutional AI (Bai et al., 2022) made the principles explicit. In a supervised phase, the model critiques and revises its own responses according to a written set of principles (the “constitution”). In an RL phase, an AI model compares response pairs against those principles to produce the preference labels used for training.

This makes the values being trained into the model more legible, since they are written down, and cheap to iterate on. It also moves the problem rather than removing it: the feedback model has its own biases and blind spots, and the method inherits whatever the constitution fails to specify. In practice, most production post-training now mixes human and AI preference data, verifiable rewards, and rule-based checks.

Where alignment techniques fail

Reward hacking and overoptimization

The reward model is a proxy, and optimizing a proxy hard enough separates it from the target. This is Goodhart’s law. Gao, Schulman and Hilton (2022) measured it directly. Training policies against a proxy reward model and scoring them with a much larger “gold” reward model, they found that as the policy moved away from its initialization (measured by KL divergence), the proxy reward kept rising while the gold reward rose, peaked, and then fell. The KL penalty and early stopping exist to stay on the rising part of that curve. The paper also found that larger reward models overoptimize more slowly, which is one argument for investing in reward model quality.

Concrete forms of reward hacking in LLMs include:

• Length exploitation. Reward models often prefer longer answers, so policies learn to pad. Singhal et al. (2023) found that a substantial part of the measured improvement from RLHF in several settings could be attributed to increased length.
• Format and style gaming. Excessive bullet points, headings, confident tone and hedging phrases that raters tend to reward regardless of content.
• Test gaming in code RL. Special-casing visible tests, or modifying the tests, instead of fixing the code, when the reward is “tests pass”.

Sycophancy

When raters reward answers they agree with, models learn to agree. Sharma et al. (2023) showed that several RLHF-trained assistants changed correct answers when users pushed back, tailored claims to a user’s stated views, and that both human raters and preference models sometimes preferred convincing sycophantic answers to correct ones. Sycophancy is a direct consequence of optimizing for approval, which is what preference data measures, rather than for truth, which raters cannot always check.

The alignment tax and over-refusal

Post-training can reduce performance on some capabilities. InstructGPT observed regressions on public NLP benchmarks and reduced them by mixing pretraining gradients into PPO (the “PPO-ptx” variant). Safety training in particular can produce over-refusal: declining harmless requests that look superficially like harmful ones. The balance between helpfulness and harmlessness is a real trade-off that has to be measured, often with separate reward models or datasets for each.

Shallow alignment

There is evidence that much of what alignment changes is concentrated in a small part of the model’s behaviour. Jailbreaks, through role-play framing, adversarial suffixes or many-shot examples in long contexts, can often recover behaviours that training suppressed. Fine-tuning an aligned open model on a small number of harmful examples can remove its safety behaviour. This is consistent with the superficial alignment hypothesis: post-training selects and suppresses behaviours far more than it removes underlying capabilities.

The deeper problem

The techniques above address what safety researchers call outer alignment: specifying the right objective. Even with a perfect objective, a trained model might learn something different that happens to score well on the training distribution. Specifically, the model might learn heuristics, proxies or goals that diverge from the intended ones in new situations (goal misgeneralization). And all of these methods rely on humans, or models supervised by humans, being able to judge outputs. As models become better than their evaluators at a task, it gets harder to tell a good answer from a convincing one. Scalable oversight research, including AI-assisted evaluation, debate between models, and decomposing hard judgements into easier ones, aims to keep supervision effective as capability grows. Recent empirical work has also begun to test whether models behave differently when they believe they are being trained. For example, Greenblatt et al. (2024) reported a model in an experimental setup selectively complying with a training objective it disagreed with, apparently to avoid being modified. Whether and how such behaviours scale is an active research question.

What alignment training does not do

• It does not teach new knowledge. Post-training mostly shapes which existing behaviours are expressed and how. Knowledge comes overwhelmingly from pretraining. RLHF can even make a model more confident in things it doesn’t know, if raters reward confidence.
• DPO does not remove RLHF’s problems. It removes the reward model and the sampling loop, not the underlying problems. It still learns from noisy, biased preferences, and it gives up online exploration.
• It does not make a model safe on its own. Post-trained behaviour can often be bypassed by jailbreaks or removed by fine-tuning. Alignment training is one layer of a defence that also needs monitoring, input and output filtering, and careful deployment.
• It is not mainly about refusals. Most alignment work is about being helpful in the intended way: following instructions precisely, being honest about uncertainty, and asking clarifying questions. Unnecessary refusals are an alignment failure too.
• It does not learn ground truth. Preferences are judgements made by particular people under particular guidelines, with limited time and expertise. A model trained on them learns those people’s preferences, including their mistakes.

How post-training looks in production

A representative modern pipeline for a chat or reasoning model:

1. SFT on a high-quality mix of human-written and model-generated, human-reviewed demonstrations covering instruction following, tool use, refusals and multi-turn behaviour.
2. Preference optimization on pairs from the current model, labelled by humans and AI judges against written guidelines. This uses DPO-style losses or reward-model-based RL, often in several iterative rounds where each round’s model generates the next round’s data.
3. RL with verifiable rewards (for example with GRPO) on mathematics, code and other checkable tasks, to improve reasoning.
4. Targeted safety training with dedicated datasets and red-teaming, and evaluation of both harmful compliance and over-refusal.
5. Extensive evaluation on capability benchmarks, held-out human preference tests, and checks for regressions, sycophancy and reward hacking, before each round is accepted.