Sampling: the same weights can sound bored or deranged
Training is over. The weights are frozen and will not change again in this chapter. Everything that follows concerns one narrow question — given a probability distribution over the next token, which token do you actually emit? — and the answer changes the output more than almost anything else you can do to a finished model. Sampling is where a large fraction of "this model feels bad" complaints actually originate.
What the model gives you, and what it does not
A forward pass ends with a vector of V logits for the final position. Softmax turns that into a probability distribution. And there the model stops — a probability distribution is not text. Something outside the model must collapse it to one concrete token, append that token to the context, and run the whole forward pass again for the next one.
tokens = encode(prompt)
for _ in range(max_new_tokens):
logits = model(tokens)[-1] # distribution for the next position
next_id = decode_strategy(logits) # <- this chapter
tokens.append(next_id)
text = decode(tokens)One consequence worth internalising: generation cost is quadratic-ish and sequential. Each new token requires a forward pass over the whole context, and the tokens cannot be produced in parallel because each depends on the last. (Production systems avoid recomputing everything with a KV cache, storing the keys and values of previous positions so each step only processes the new token — which is why serving LLMs is memory-bandwidth-bound.)
Greedy decoding, and why the obvious answer is wrong
The obvious strategy: always take the highest-probability token. It is deterministic, reproducible, and it maximises the probability of each token given what came before. It is also, for open-ended text, reliably bad — and the way it fails is very specific.
Greedy decoding falls into loops. "I went to the store to buy some milk and then I went to the store to buy some milk and then…" — once a phrase repeats, the context now contains evidence that this phrase follows this context, which makes it more likely still. It is a self-reinforcing attractor, and there is no randomness available to escape it.
There is a deeper reason. Real human text is not the most probable text. Measured against a language model, human writing sits in a band of moderate surprise, constantly making choices that were not the single likeliest. Always choosing the maximum produces text that is bland and degenerate in a way that is immediately recognisable once you have seen it. This is the central finding of "The Curious Case of Neural Text Degeneration", and it is why every open-ended generation system samples.
Temperature: reshaping the distribution
Dividing logits by T before the softmax rescales the gaps between them. Small T magnifies differences, so the leader pulls further ahead; large T compresses them toward uniform. The name is borrowed from statistical physics, where the same expression describes how a system occupies energy states — high temperature, more disorder.
| Temperature | Behaviour | Use for |
|---|---|---|
| 0 | Argmax. Deterministic, prone to loops | Arithmetic, extraction, anything with one right answer |
| 0.2–0.5 | Conservative; mostly the obvious continuation | Code, factual answers, summarisation |
| 0.7–0.9 | The usual default. Varied but coherent | General text, dialogue |
| 1.2+ | The tail starts winning. Novel, then incoherent | Brainstorming — and observing failure |
| 2.0 | Near-uniform noise | Nothing, except seeing what noise looks like |
The useful range is narrower than people expect, and it narrows further for weak models. Your browser-trained character model has genuinely uncertain distributions, so raising the temperature amplifies real confusion rather than adding pleasant variety. Expect the coherent window here to sit lower than the 0.7–1.0 you would use with a production model.
Truncation: deleting the tail before you roll
Temperature reshapes the whole distribution, which means it can never fully suppress absurd options — it only makes them rarer. But the tail of a vocabulary distribution is thousands of tokens each with tiny probability, and collectively that tail can hold real mass. Sample enough tokens and you will eventually draw one of them, and one badly chosen token derails everything after it. Truncation methods remove the tail from consideration entirely, then renormalise what remains.
top-k
Keep the k highest-probability tokens, discard the rest, renormalise, sample. Simple and effective. Its weakness is that k is fixed while the model's certainty is not: after "The capital of France is" there is really only one sensible token, and k = 40 invites 39 wrong ones back into the running. After "He opened the door and saw", forty options is too few.
top-p (nucleus sampling)
Sort by probability and keep the smallest set whose cumulative probability reaches p. The number of candidates now adapts to the model's confidence: a sharp distribution yields two or three candidates, a flat one yields hundreds. top_p = 0.9 is the most common default in production systems, frequently combined with a generous top_k as a hard ceiling.
Repetition penalty
A direct patch on the loop problem: divide the logits of tokens already generated, making them progressively less attractive. It works, and it is a blunt instrument — pushed too high it forbids the ordinary repetition that real language depends on, and you get text that avoids common words unnaturally. Values much above 1.2 usually do more harm than good.
Reading the output fairly
Your model has roughly fifty thousand parameters and has read a fraction of one play. Judge it accordingly: the achievement is not meaning, it is shape. Look for correctly formed words, plausible letter statistics, capitalised speaker names followed by colons, line breaks in roughly the right places. All of that was learned from raw characters with no notion of words at all.
Because the sampler is seeded, the same settings and the same prompt produce the same output every time — which is what makes the comparisons in this chapter fair. Change one control at a time. If you change temperature and the prompt together, you have learned nothing about either.
The sampler is exposed as editable code for the same reason attention was: the mechanism is small enough to read in full. Everything above — the penalty loop, the argmax special case, the temperature division, the max-subtraction that keeps exp() from overflowing — is visible in about thirty lines. Change it and the very next generation uses your version, with no retraining.
Vocabulary
- Decoding strategy
- The rule that turns a probability distribution into one emitted token. Not part of the weights.
- Greedy decoding
- Always take the argmax. Deterministic, loop-prone, right for tasks with one correct answer.
- Temperature
- Divides logits before softmax. Below 1 sharpens, above 1 flattens.
- top-k
- Keep the k most likely tokens and renormalise. Fixed candidate count.
- top-p / nucleus
- Keep the smallest set reaching cumulative probability p. Candidate count adapts to confidence.
- Repetition penalty
- Down-weights already-generated tokens to break loops.
- KV cache
- Stored keys and values for previous positions, so each generated token needs only one new position of work.
Check yourself
If you can answer these without re-reading, the lab below will make sense. If you cannot, the relevant section is worth a second pass — that is a better use of your time than clicking buttons.
- Why is greedy decoding prone to loops, and why is the loop self-reinforcing?
- Why is the most probable text not the most human-like text?
- What does temperature do to the gaps between logits, and what is temperature 0 really doing?
- Give a prompt where top-k = 40 is clearly too many candidates, and one where it is too few.
- Both top-p and temperature reduce the influence of unlikely tokens. What does truncation achieve that temperature cannot?
- You changed no weights but the output got much better. What did you change, and what does that tell you about complaints that a model "got worse"?
Where this comes from
- Holtzman et al., "The Curious Case of Neural Text Degeneration" (2019) — Nucleus sampling, and the evidence that likely text is dull text.
- Fan et al., "Hierarchical Neural Story Generation" (2018) — Where top-k sampling comes from.
- Anthropic and OpenAI API docs on temperature / top_p — The same three controls, in production.