Transformer architecture

The minimum architecture you need to read a paper. Three ideas: tokens become vectors, attention lets every token look at every other one, and you stack many such layers.

What a transformer actually does

A transformer is a stack of layers. At every layer, every word in the input gets to look at every other word and pull whatever information it needs. By the time the input has passed through 80 such layers, each word's representation has absorbed context from the whole sentence (or document, or codebase).

That mechanism — every-token-looks-at-every-token — is called attention, and it's the one architectural idea that powers every major LLM since 2017. GPT, Claude, Gemini, Llama, Qwen: same skeleton, different sizes and refinements.

That's the whole concept. Below: the math behind attention, why residual connections matter, and the small list of refinements (RoPE, multi-head, MoE) that have shown up since the original paper.

When you'd reach for it

You don't really "use" a transformer — you use a model that is one. But understanding the architecture helps when you need to:

• Read papers — every modern AI paper assumes you know what attention is.
• Reason about model limits — context-window cost is quadratic in attention; that's a transformer fact, not a quirk.
• Compare architectures — when someone says "Mamba is sub-quadratic" or "Mixture-of-Experts is sparse," that's a transformer point of reference.
• Interview or pitch credibly — knowing the three letters Q, K, V is table stakes for AI engineering roles.

If you'll never read a paper, never tune a model, and never argue about architectures, you can stop after the concept section.

How it's actually built

The 30,000-foot picture:

Inside each layer:

Just two operations stacked many times: attention and a feed-forward MLP, with residual connections (the +x skip-around). Plus normalisation to keep things numerically stable.

Attention, finally

Attention answers: for each token, how much should I look at every other token?

Every token gets three vectors derived from its embedding:

• Query (Q) — what am I looking for?
• Key (K) — what do I contain that others might look for?
• Value (V) — if attended to, what information do I pass forward?

For each pair of tokens (i, j):

score(i, j) = Q_i · K_j / sqrt(d_k)        (scaled dot product)

Take softmax across all j to get weights summing to 1. The output for token i is a weighted sum of value vectors:

output_i = Σ_j  softmax_j(score(i, j)) · V_j

That's it. Attention is a learned, content-conditional weighted average. It lets distant tokens influence each other in one step — the key reason transformers replaced RNNs.

Three subtleties worth knowing

Under the hood (optional)

A scaled-down transformer block in PyTorch-style code: layer norm, attention, MLP, residual connections. About 12 lines. The forward method is the literal recipe — normalise → attention → add to input → normalise → MLP → add to input → output. Skip if you don't code.

class Block(nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        self.ln1 = nn.LayerNorm(d_model)
        self.attn = MultiHeadAttention(d_model, n_heads, causal=True)
        self.ln2 = nn.LayerNorm(d_model)
        self.mlp = MLP(d_model, d_ff=4*d_model)
 
    def forward(self, x):
        x = x + self.attn(self.ln1(x))   # residual + attention
        x = x + self.mlp(self.ln2(x))    # residual + MLP
        return x

Further reading

• Attention Is All You Need (Vaswani et al. 2017) — the original paper. Surprisingly readable.
• The Illustrated Transformer (Jay Alammar) — best visual explainer of the architecture.
• Karpathy — Let's build GPT, from scratch — implement a real (small) transformer in 2 hours.
• 3Blue1Brown — Visualising attention — beautiful video walkthrough of what attention "sees".
• Anthropic — Mathematical Framework for Transformer Circuits — for the deeply curious.