Architecture & Representation¶
What this page solves
What is inside a large model: how text becomes vectors, how attention lets tokens look at each other, and how blocks stack into a GPT. With this page, training, inference and multimodality share one skeleton. Required; prerequisites: matrix products, PyTorch tensors.
How far to go¶
| Level | Learn | Done when |
|---|---|---|
| Orientation | Tokens, embeddings, context, attention, generation | Input-to-output drawn; F0 done |
| Foundation | Residuals, normalisation, positions, the training loop | Small GPT built in F4; every tensor shape explained |
| Advanced | Long context, MoE, state-space models, linear attention | A comparison experiment designed for one question |
Core knowledge¶
Tokenisation: BPE and similar algorithms split text into sub-word units. Vocabulary size sets the embedding size and sequence length; how Chinese, code and numbers are split affects cost and capability (digits split badly make arithmetic harder).
Embeddings: each token looks up a vector; the output layer maps vectors back to vocabulary probabilities. Many models tie input embeddings and output weights.
Self-attention: each token produces Q, K, V; softmax(QKᵀ/√d)·V lets each position gather information from others by content similarity. A causal mask keeps generation from seeing the future. Multiple heads attend to different relations.
Transformer block: attention + feed-forward network (FFN, most of the parameters), each wrapped in a residual connection and normalisation. Modern LLMs use pre-norm (RMSNorm) and SwiGLU for stabler training.
Positional encoding: attention itself is order-blind. Absolute encodings add a vector per position; RoPE rotates Q and K by position so attention depends only on relative distance — today's default.
Long context: attention is O(n²) and the KV cache grows linearly. Remedies: sliding windows, sparse attention, FlashAttention (same complexity, far less memory traffic), RoPE extrapolation and interpolation.
Mixture of experts (MoE): replace the FFN with many experts and let a router activate a few per token. Large total parameters, little compute per token; hard parts are load balancing and memory (all experts must fit).
Alternatives: state-space models (Mamba) and linear attention compress history into a fixed-size state, O(1) per inference step and cheaper on long sequences, but weaker at exact long-range recall; hybrids are spreading.
Unified multimodal representation: images become patches, audio becomes frames, encoders turn them into tokens of the same width as text, and everything enters one Transformer.
Debate · autoregressive vs. diffusion¶
Autoregressive generation, token by token, is simple to train and natural for language — the LLM default. Diffusion denoises in parallel and dominates images, video and robot actions; diffusion language models are explored for parallel decoding. The trade-off is the simplicity of sequential generation against the efficiency and multi-modality of parallel generation.
Curated resources¶
-
Main track · Karpathy Zero to Hero
First two makemore lectures → "Let's build GPT"; micrograd first if gradients are unclear.
-
Text alternative · LLMs-from-scratch
Chapters 2–5 as notebooks, from tokenisation to GPT pretraining; choose this or the previous one.
-
Interface reference · HF LLM Course
Chapter 2 on tokenizers and model loading, for when an interface question comes up.
Follow-along experiments¶
Self-check¶
Why divide attention scores by √d?
The variance of Q·K grows with d; large values push softmax towards one-hot and kill gradients. Dividing by √d keeps the variance near 1.
How is the causal mask implemented?
Set the upper triangle (future positions) of the score matrix to −∞ before softmax so their weights become 0.
Where are most parameters in a Transformer block?
In the FFN: two linear layers with an inner width around 4× the hidden size (SwiGLU ≈ 8/3× with three matrices) — about twice the attention projections.
Why is MoE large in parameters but cheap at inference? And memory?
Each token visits only a few experts, so compute follows active parameters; but all expert weights must be loaded, so memory follows total parameters.