Alignment: making it useful, and what that costs
Everything so far produced a base model: a system trained to continue text, and nothing else. It is not an assistant, it has no notion of a question, and asking it one will usually get you more questions. The gap between that object and something like ChatGPT is not a bigger network — it is a second and third stage of training with different data and a different objective. This chapter runs the first of those stages, and then measures what it broke.
What a base model actually is
A base model has been optimised for exactly one thing: given some text, predict what comes next in a corpus that looks like the training data. It has no goals, no notion of being addressed, and no concept of a "reply". Prompt it with "Q: who speaks?\nA:" and a plausible continuation might be another question, a stage direction, or a line of dialogue — because in the text it learned from, that is what tends to follow.
This surprises people who meet chat models first, but it is worth taking seriously, because the base model is where essentially all the knowledge lives. Pretraining is where the compute goes — months, thousands of GPUs, trillions of tokens. What follows is comparatively tiny, and it teaches behaviour, not facts.
The three stages, and their relative sizes
| Stage | Data | Objective | Rough scale |
|---|---|---|---|
| 1. Pretraining | Trillions of tokens of scraped text | Next-token prediction | Months; almost all the compute |
| 2. Supervised fine-tuning (SFT) | Thousands to millions of written demonstrations | Next-token prediction, on demonstrations only | Hours to days |
| 3. Preference optimisation (RLHF / DPO) | Human comparisons between candidate responses | Maximise a preference signal, stay near the SFT model | Hours to days |
Stage 2: supervised fine-tuning
Keep training the same weights with the same loss function, but on curated prompt → good response pairs written in a consistent format. Nothing about the algorithm changes; only the data does. That is what makes SFT so approachable, and it is exactly what the lab below runs.
The format is doing more work than the content. By seeing thousands of examples where a marker is followed by a helpful answer that then stops, the model learns a mode: "when I see this shape, produce an answer, then emit the end token." Real systems use explicit chat templates with role markers and special tokens; the Q: / A: convention in this lab is the same idea at its smallest.
Stage 3: learning from preferences
SFT can only teach imitation of responses someone wrote down, and "which of these two answers is better" is far cheaper to collect — and often more informative — than "write the ideal answer". RLHF trains a reward model on human comparisons, then optimises the language model against that reward with reinforcement learning (usually PPO), penalised for drifting too far from where it started. DPO reaches a similar result by deriving a loss that can be optimised directly on the preference pairs, with no separate reward model and no RL loop — which is why much of the open-weight ecosystem now uses it.
The alignment tax
Fine-tuning moves the weights toward the fine-tuning data. Anything the previous weights were good at, and the new data does not exercise, tends to get slightly worse. Measured on the original corpus, loss goes up. That regression is the alignment tax, and the lab measures it directly: validation loss before fine-tuning, validation loss after, difference.
The underlying mechanism is catastrophic forgetting, a general property of gradient descent on neural networks: the parameters have no memory of which past behaviour they were responsible for, so training on a narrow distribution overwrites whatever was stored in the directions it touches. The effect is dramatic here because the instruction data is a few hundred nearly identical lines, which is about as narrow as a distribution gets.
The standard mitigations are all forms of "move less":
- A much lower learning rate than pretraining — often 10–100× lower. The lab uses this.
- Fewer steps. Fine-tuning that runs too long is the most common way to ruin a good base model.
- Mixing in pretraining data during fine-tuning, so the old distribution keeps being reinforced.
- Parameter-efficient methods such as LoRA, which freeze the original weights and train small low-rank adapters alongside them. The base model is mathematically untouched and the adapter is a few megabytes rather than a full model copy — which is why nearly all community fine-tunes are LoRAs.
- KL penalties in stage 3, which explicitly punish drifting away from the reference model.
You should see the after-tuning output adopt the answer format sharply, and you should see the loss on Shakespeare rise. Both are the point. In a frontier lab that trade-off is the central engineering tension of the whole post-training stage: make it agreeable without making it dumber.
What this does not give you
Scaling this demo up produces something that answers in the right format. It does not, on its own, produce a system that is truthful, safe or robust, and it is worth being precise about why:
- Format is not correctness. A model can learn to answer confidently and be wrong; SFT rewards the shape of a good answer. Confident wrongness is a behaviour that fine-tuning on confident-sounding demonstrations actively encourages.
- The reward is a proxy. In stage 3, the model optimises a learned reward model, not human values. Optimise any proxy hard enough and you find its flaws — reward hacking, sycophancy, and answers that pattern-match to "good" without being good.
- Alignment is shallow relative to pretraining. A thin behavioural layer over a vast base model can be argued around, which is the structural reason jailbreaks keep working.
- Evaluating open-ended quality is unsolved. There is no held-out loss for "is this helpful?", which is why the lab asks you to judge the output yourself. That is not a shortcut in the demo; it is genuinely the state of the art.
Vocabulary
- Base model
- A pretrained next-token predictor. All the knowledge, none of the assistant behaviour.
- SFT
- Supervised fine-tuning: continued training on curated prompt/response demonstrations.
- RLHF
- Reinforcement learning from human feedback: train a reward model on comparisons, optimise against it.
- DPO
- Direct preference optimisation: the same preference signal as a plain loss, with no reward model or RL loop.
- Alignment tax
- The regression on general performance caused by fine-tuning for behaviour.
- Catastrophic forgetting
- Training on a narrow distribution overwrites capabilities the new data does not exercise.
- LoRA
- Low-rank adapters trained alongside frozen weights, so the base model is preserved exactly.
Check yourself
If you can answer these without re-reading, the lab below will make sense. If you cannot, the relevant section is worth a second pass — that is a better use of your time than clicking buttons.
- Why does a base model respond to a question with more questions?
- Which stage contains almost all of the compute, and which stage contains almost all of the assistant behaviour?
- What changes between pretraining and supervised fine-tuning — the algorithm, the data, or both?
- Explain the alignment tax in terms of catastrophic forgetting.
- Why does LoRA make the tax easier to manage than full fine-tuning?
- Why is "the model produced a well-formatted, confident answer" weak evidence that fine-tuning worked?
Where this comes from
- Ouyang et al., "Training Language Models to Follow Instructions with Human Feedback" (2022) — InstructGPT: the RLHF pipeline that produced ChatGPT.
- Rafailov et al., "Direct Preference Optimization" (2023) — Preference learning without reinforcement learning.
- Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models" (2021) — How to fine-tune without touching the base weights.
- Bai et al., "Constitutional AI" (2022) — Using model-generated critiques in place of some human feedback.