The transformer architecture, introduced in "Attention is All You Need" (2017), powers modern LLMs and most of contemporary deep learning. Understanding transformers is foundational to working with LLMs.
This page covers how they work and why they matter.
Pre-transformer sequence models (RNNs, LSTMs) processed tokens sequentially. Information had to flow through hidden states.
Transformers process all tokens in parallel. Each token "attends" directly to every other token via the attention mechanism.
For each token:
Result: each token's output is informed by all input tokens, weighted by relevance.
Run attention multiple times in parallel with different projections. Concatenate.
Different heads learn different relationship types (syntax, semantics, coreference, etc.).
Typical: 12-128 heads.
Self-attention is O(n²) in sequence length. This is the dominant cost for long contexts.
Many efficiency variants try to reduce this (sparse attention, linear attention, etc.). Standard attention with hardware-aware implementations (FlashAttention) remains common.
Convert tokens to vectors. Typically 1024-12288 dimensions for modern LLMs.
Tokens have no inherent position in self-attention. Positional encodings inject position info.
Variants:
Per layer:
Modern variants tweak normalization placement (pre-norm vs post-norm), use SwiGLU instead of standard FFN, etc.
Stack N transformer blocks. Modern LLMs: 32-128+ layers.
Final layer norm + linear projection to vocabulary size.
For generation: sample from the resulting distribution.
Bidirectional attention. Used for understanding tasks (classification, NER, embedding).
Causal (left-to-right) attention. Used for generation.
Modern LLMs (GPT, Claude, Llama, etc.) are decoder-only.
Encoder processes input; decoder generates output. Used for translation, summarization.
Less common for LLMs but still used (T5, FLAN).
Treat image patches as tokens. Same architecture; different input pipeline.
Combine text and image (or audio, etc.) tokens. Input modality has its own encoding; the rest of the architecture is shared.
Self-supervised on huge text corpora.
Objective:
Pretraining is expensive ($1M-100M+ for state-of-the-art LLMs).
Adapt pretrained model to specific task with labeled data.
Train reward model from human preferences; train policy to maximize reward.
Aligns model with human preferences. Powers ChatGPT, Claude, etc.
Variant where principles ("be helpful, harmless") guide self-critique. Anthropic's approach.
Memory-efficient attention. Critical for long contexts.
Multiple FFN "experts"; route each token to a subset.
Effective parameter count > active parameters per inference.
Models: Mixtral, DBRX, GPT-4 (rumored), DeepSeek-V2.
Share keys/values across query heads. Reduces memory.
Local attention beyond a window; reduces compute for long context.
Original transformers: 512-2K tokens. Modern: 8K-1M+ tokens.
Achieved through:
Process the prompt. Compute K and V for all tokens. Cache them.
This is the parallelizable, fast phase.
Generate tokens one at a time. Each token requires reading the entire KV cache.
Memory-bandwidth bound. Why LLM inference is "slow."
Stores K and V from previous tokens. Reused across decode steps.
Memory grows linearly with context length.
Empirical findings (Kaplan et al., Chinchilla):
Roughly: bigger model + more data → better performance, predictably.
Some abilities appear suddenly at scale:
Why exactly is unclear. Architecture + scale + training data interact.
Plausible but wrong outputs. Models predict tokens, not truth.
Even small LLMs are expensive at scale.
Limited; long-context models exist but quality degrades with length.
Without retrieval/tools, models reason from training data only.
Symbolic / mathematical reasoning is fragile. Better with chain-of-thought, tools.
Models don't know recent events without retrieval.
These advantages compounded; transformers eclipsed RNNs across NLP within a few years.
For now, transformers remain dominant.
For practitioners:
For researchers: the architecture is mature; innovation happens at training, data, and adjacent components.