The Block

Anatomy of a block

x = x + Attention(LayerNorm(x))   # communication between positions
x = x + MLP(LayerNorm(x))         # computation within each position

Two lines, repeated L times. The whole of GPT is that loop plus an embedding at the front and a linear layer at the back. Note the division of labour that the two sub-layers imply, because it is the cleanest way to think about a transformer:

• Attention is the only place positions talk to each other. Remove it and each position is processed in complete isolation.
• The MLP is the only place non-trivial per-position computation happens. It sees one position at a time and knows nothing about its neighbours.
• Everything is added into the residual stream, never overwriting it. Each layer contributes an increment.

Residual connections: the reason depth works at all

Writing x = x + f(x) instead of x = f(x) looks trivial. It is the single most important structural idea in deep learning since backpropagation, and the reason networks went from about 20 layers to hundreds.

The argument is about gradients. In a plain stack, the gradient reaching layer 1 is a product of the Jacobians of every layer above it. Multiply forty matrices whose scale is slightly below 1 and the result is effectively zero — the early layers stop learning. Slightly above 1 and it explodes instead. With a residual connection, the derivative of x + f(x) with respect to x is 1 + f'(x): there is always a path of exact gradient straight through, and the learned part is a correction on top of it.

In the race below, the no-residual configuration will be dramatically worse than everything else, at only two layers. Scale that intuition: at forty layers it does not train at all.

Normalisation, and the pre-LN / post-LN argument

LayerNorm takes a vector, subtracts its mean, divides by its standard deviation, and then applies a learned per-dimension gain and bias:

Being per-position is what makes it suitable here: it behaves identically at batch size 1 and batch size 512, and identically at any sequence length, with no running statistics to maintain between training and inference. The ε in the denominator is not decoration — without it a constant vector divides by zero.

Where to put it

This is a genuinely interesting piece of history: the original paper used post-LN, and getting those models to train required a carefully tuned learning-rate warmup that nobody could fully explain. The 2020 analysis showed why — post-LN produces enormous gradients at initialisation near the output layers — and the field moved to pre-LN essentially overnight. At two layers you should expect the difference in the race to be small; the choice is about what happens at depth 40, not depth 2.

The MLP: where most of the parameters live

MLP(x) = W_down · activation(W_up · x)
         W_up:   [d, 4d]
         W_down: [4d, d]

Project up to four times the width, apply a non-linearity, project back down. That is it. This runs independently at every position — no mixing, no context — and it accounts for roughly two-thirds of the parameters in a standard transformer: 8d² for the MLP against 4d² for attention's four projection matrices.

Two-thirds of the parameters doing purely per-position work seems strange until you ask what else could store knowledge. Attention decides where to look; it has no capacity to hold facts. A productive interpretation, supported by real experiments, is that the MLP acts as a key-value memory: W_up rows detect patterns in the incoming vector, the activation gates which fired, and W_down rows write out the associated content. Editing specific factual associations in a model by modifying particular MLP weights is a working technique, which is strong evidence for that reading.

The expansion ratio

4× has been the default since 2017, and it is mostly convention that has held up under testing. Narrower saves parameters and costs quality; wider adds parameters with diminishing returns. Race 2×, 4× and 8× below and note that the differences are much smaller than the residual ablation — this is a tuning choice, not a structural one.

GELU versus ReLU

ReLU is max(0, x): hard zero below the threshold, identity above, and exactly zero gradient for any negative input — a unit that goes negative for all inputs is dead forever. GELU is a smooth version, roughly x · Φ(x), which passes a small negative gradient instead of none and is differentiable everywhere. It is a modest, consistent improvement, and it is what GPT-2 and BERT used. Current models often use SwiGLU, a gated variant that is better again for the same compute.

The one thing the activation cannot be is linear. Without a non-linearity, stacked matrix multiplications collapse into a single matrix multiplication and depth buys you literally nothing.

Running the race honestly

Read the results with appropriate scepticism. These are short runs from random initialisations, and run-to-run variation from the seed alone is real. A gap of 0.3 nats is a finding; a gap of 0.02 is noise dressed as a finding. If a comparison matters to you, run it again.

Vocabulary

Block

One attention sub-layer plus one MLP sub-layer, each with normalisation and a residual connection.

Residual connection

x = x + f(x). Gives gradients a direct path and makes the residual stream a shared channel across depth.

LayerNorm

Per-position normalisation to zero mean and unit variance, with learned gain and bias.

pre-LN / post-LN

Whether normalisation happens inside the residual branch (pre, stable) or after the addition (post, original).

MLP / feed-forward

Per-position up-projection, non-linearity, down-projection. About two-thirds of the parameters.

Expansion ratio

How much wider the MLP hidden layer is than d. Conventionally 4×.

Check yourself

If you can answer these without re-reading, the lab below will make sense. If you cannot, the relevant section is worth a second pass — that is a better use of your time than clicking buttons.

1. Which sub-layer moves information between positions, and which one does not?
2. State the gradient argument for residual connections in one sentence.
3. Why is LayerNorm computed across the feature dimension rather than across the batch?
4. What did pre-LN fix that made learning-rate warmup less critical?
5. Why do two-thirds of a transformer's parameters sit in a component that cannot see other positions?
6. What happens to a deep network if the activation function is linear?

Where this comes from

• He et al., "Deep Residual Learning" (2015) — Residual connections, and the degradation problem they solved.
• Xiong et al., "On Layer Normalization in the Transformer Architecture" (2020) — The analysis that settled the pre-LN vs post-LN question.
• Geva et al., "Transformer Feed-Forward Layers Are Key-Value Memories" (2021) — Evidence for the memory interpretation of the MLP.