Embeddings

An embedding table is a lookup table (and also a matrix multiply)

The embedding table is a matrix E of shape [V, d]: one row per vocabulary entry, each row a vector of d numbers. Token id i becomes row i. That is the entire operation.

# view 1: a lookup
x = E[token_id]                    # shape [d]

# view 2: a matrix multiply by a one-hot vector
onehot = [0, 0, ..., 1, ..., 0]    # shape [V], the 1 at position token_id
x = onehot @ E                     # shape [d] — identical result

The second view is why embeddings are trainable like everything else: a lookup is a linear layer, so gradients flow into E exactly as they would into any weight matrix. The only implementation difference is that the gradient is sparse — a training step touches only the rows for tokens that actually appeared in the batch.

Where the meaning comes from

Nothing about the initialisation is meaningful — E starts as small random numbers, typically drawn from a normal distribution with standard deviation around 0.02. Structure appears only because gradient descent finds that placing tokens with similar predictive behaviour near each other reduces loss. Tokens that can be substituted for one another in context end up close together, because moving one moves the predictions of both in the same helpful direction.

That is the whole of the famous "king − man + woman ≈ queen" result: it is not evidence of reasoning, it is evidence that co-occurrence statistics have consistent linear structure. In this chapter you are working at character level, so expect the learned geometry to cluster vowels, consonants, punctuation and capitals rather than concepts.

Choosing d, and the residual stream

The embedding width d is the width of the entire model. Every layer reads a [T, d] array and writes a [T, d] array; nothing in a transformer changes that shape. Practitioners call this persistent array the residual stream, and it is the most useful mental model in the architecture: a bus of T slots, each d numbers wide, that every layer reads from and adds to.

Too small a d and everything competes for room — the model cannot represent enough distinct features to separate what it needs to separate. Too large and you are paying for capacity the data cannot fill. The tiny d = 48 used here is chosen so a full training run finishes in your browser in seconds; every phenomenon in this campaign is real at this size, just smaller.

The order problem

Here is the structural fact that forces the rest of this chapter. Attention — the whole of Chapter 3 — computes, for every pair of positions, a compatibility score from their vectors, and then a weighted sum. Every one of those operations treats the input as a set. Permute the input positions and the outputs permute identically; nothing anywhere in the computation depends on where a token sits.

So without extra help, "dog bites man" and "man bites dog" are the same input to the model. This is not a subtle degradation — it deletes syntax entirely. Recurrent networks never had this problem because they consumed tokens one at a time, in order; the transformer traded that away for parallelism and had to buy order back separately.

The fix is to add a second vector — depending only on position, not on the token — into the residual stream before the first layer:

Addition, not concatenation, which surprises people. It works because d dimensions is a lot of room: the model is free to learn to keep the two kinds of information in roughly separate subspaces, and the first attention layer can read out whichever it needs.

Three ways to supply position

Learned absolute (what GPT-2 does)

A second trainable table P of shape [T_max, d]. Position 0 gets a learned vector, position 1 gets another, and so on. Simple, effective, and it costs T_max × d parameters. Its weakness: nothing is defined beyond T_max, so the model cannot process a sequence longer than the longest it was trained on — the vectors for those positions were never learned.

Sinusoidal (the 2017 original)

Fixed, not learned: each dimension is a sine or cosine of position at a different frequency, geometrically spaced from very fast to very slow. Reading the pattern across dimensions gives a unique, smoothly varying signature per position, rather like a binary counter made continuous.

It costs zero parameters and is defined for every position, so it extrapolates past the training length in principle. In practice models trained on it still degrade beyond their training length, just less abruptly.

None (the control)

The experiment below. It is not there for realism — it is there so the cost of removing position is a number you measured rather than a claim you accepted.

Reading the experiment honestly

Expect the order-blind model to be clearly worse, but not catastrophic — and understand why. Even with no position information at all, a model can still learn the unigram distribution: that e is common, q is rare, and spaces are everywhere. That alone gets you a long way down from the uniform baseline. What it cannot learn is anything conditional on arrangement, which is most of what language is.

The gap is measured in nats — the natural-log units of cross-entropy loss, which Chapter 6 converts into something more interpretable. For now, treat differences of 0.1 nats as meaningful and 0.01 as noise: these are short runs from a random initialisation, and run-to-run variance is real. If you want to be rigorous about a small difference, run each configuration more than once.

Vocabulary

Embedding table

A [V, d] matrix of trainable parameters; row i is the vector for token i.

d (model dimension)

The width of every vector in the model, from the embedding to the final layer.

Residual stream

The [T, d] array that every layer reads from and adds back into.

Permutation equivariance

The property that reordering inputs simply reorders outputs — why attention needs positional information.

Positional encoding

A vector that depends only on position, added to the token embedding.

RoPE

Rotary position embedding: encodes relative distance by rotating queries and keys. The current default.

Check yourself

If you can answer these without re-reading, the lab below will make sense. If you cannot, the relevant section is worth a second pass — that is a better use of your time than clicking buttons.

1. Why is an embedding lookup mathematically the same as multiplying by a one-hot vector, and why does that matter for training?
2. What does permutation equivariance mean, and what does it do to "dog bites man"?
3. Why can position information be added to the token embedding rather than concatenated to it?
4. What is the specific weakness of learned absolute position embeddings when you want a longer context?
5. Why does a model with no positional information still beat the uniform baseline by a wide margin?

Where this comes from

• Vaswani et al., "Attention Is All You Need" (2017), §3.5 — The original sinusoidal formulation and the argument for it.
• Su et al., "RoFormer: Rotary Position Embedding" (2021) — RoPE, now the default in open-weight models.
• Press et al., "Train Short, Test Long" (2021) — ALiBi, and a clear account of why extrapolation is hard.