AI fundamentals, part 7: Alignment — RLHF and why models behave (mostly)

Part 7from the AI fundamentals series · 8 parts in all

Parts 1–6 produced a model that predicts text disturbingly well. But raw next-token prediction gives you an internet simulator, not an assistant: ask it a question and it may answer — or continue with five more fake questions. The process that turns a predictor into a helpful, honest, harmless assistant is alignment, and its workhorse is RLHF: Reinforcement Learning from Human Feedback.

Stage 1: Supervised fine-tuning (SFT)

First, collect a few thousand to a few million demonstrations — real conversations where humans wrote good assistant replies — and continue training the base model on them with ordinary cross-entropy (our part-3 loss). This teaches the format: "when I see a user question, I produce a helpful answer, then stop." SFT alone gets you 80% of the way; the failure mode is that it can only imitate, never exceed, the demonstrator.

def sft_loss(model, prompt_ids, response_ids):
    # Train only on the response tokens. The prompt is context we condition on,
    # not text we learn to produce.
    full = prompt_ids + response_ids
    logits = gpt_forward(full[:-1], model.params)
    # mask: loss only where the target is INSIDE the response
    targets = full[1:]
    mask = [i >= len(prompt_ids) - 1 for i in range(len(targets))]
    nll = cross_entropy_masked(logits, targets, mask)
    return nll

Stage 2: the reward model

Next, humans compare pairs of assistant answers to the same prompt and pick the better one (thumb-level feedback is far more consistent than trying to write perfect answers). From these comparisons we train a reward model — the same transformer with a scalar head — to predict which answer humans would prefer:

def reward_loss(rm, prompt, better, worse, margin=1.0):
    # Bradley-Terry pairwise loss: raise the score gap between the preferred
    # and rejected answers. rm scores any (prompt, response) pair.
    rb = rm.score(prompt, better)        # scalar
    rw = rm.score(prompt, worse)         # scalar
    return -np.log(np.sigmoid(rb - rw + margin))   # push the gap, softly

The reward model is the whole point of the exercise: it compresses human judgment into a differentiable signal the model can optimize against — including on new answers no human ever rated.

Stage 3: RL against the reward

Now the policy (the LLM) generates answers, the reward model scores them, and policy gradient (usually PPO) nudges the model toward higher-reward outputs:

def ppo_objective(policy, rm, prompts, kl_coef=0.05):
    # PPO, conceptually: maximize reward, minus a penalty for drifting too far
    # from the SFT model. That KL term is not decoration - without it the policy
    # finds 'reward hacks': outputs the RM scores highly but humans hate.
    total = 0
    for p in prompts:
        resp, logprobs = policy.generate_with_logprobs(p)
        r = rm.score(p, resp)
        kl = policy.kl_vs_reference(p, resp)      # distance from the SFT policy
        total += r - kl_coef * kl                 # reward, tethered
    return total

That KL tether is the key insight of the whole field: an optimizer chasing a learned proxy reward will find its loopholes — verbosity, flattery, confident hedging, sycophancy — unless it's anchored to the model that used to be honest. Reward hacking is not a bug you fix once; it's the ongoing tension of the technique.

DPO: RLHF without the RL

The 2023 simplification that took over: skip the reward model and PPO entirely. DPO (Direct Preference Optimization) derives a closed-form loss that optimizes preferences directly on the policy itself:

def dpo_loss(policy, ref, prompt, chosen, rejected, beta=0.1):
    # One pair, one loss. pi/ref log-ratios replace the reward model entirely.
    dc = policy.logprob_of(prompt, chosen)   - ref.logprob_of(prompt, chosen)
    dj = policy.logprob_of(prompt, rejected) - ref.logprob_of(prompt, rejected)
    return -np.log(np.sigmoid(beta * (dc - dj)))   # widen the relative gap

Stable, single-stage, and how most open models are tuned today — at some cost in extractable performance versus a carefully run PPO pipeline.

Why models refuse — and why they sometimes shouldn't

Alignment data includes refusals, so refusals are behavior the model learned — which is why they can be miscalibrated (refusing benign requests; over-refusal is the most common alignment complaint in production). And none of this is a cryptographic guarantee: jailbreaks are optimization against the same signal, finding inputs where the learned behavior fails. Treat alignment as reliability engineering, not proof — which is exactly the mindset of the final part: what all of this means for the software you'll actually ship.