Roy van Rijn
Roy van Rijn's interactive visual explainer for developers showing how LLMs work, from tokenization and embeddings to attention, transformers, training, KV cache, and quantization.
Last verified:
What is Roy van Rijn?
Roy van Rijn's Anatomy of an LLM is an interactive visual guide that explains how large language models work from the inside out. It follows the complete journey of text through an LLM: starting with raw text, converting it to tokens, transforming tokens into vector embeddings, passing through transformer blocks with attention mechanisms, producing logits, applying sampling strategies, and finally generating output text.
Roy van Rijn pricing
Pricing model: Freemium
Free - The guide is completely free to access on Roy van Rijn's website with no paywall, subscription, or paid tiers mentioned.
Roy van Rijn pros
- Interactive visuals make abstract LLM concepts concrete and inspectable
- Follows one complete chain from text to tokens to vectors to attention to output
- Uses small numbers first for easier understanding before scaling up
- Covers 15 comprehensive chapters from tokenization to quantization
- Live tokenizer demo with OpenAI's o200k_base encoding
- Explains multi-head attention with visual Q/K/V matrix inspector
- Interactive RoPE visualization showing relative position rotation
- Shows modern decoder block details including Llama-style dimensions
- Explains KV cache with compute-memory tradeoff visualization
- Quantization selector shows bit-width vs memory/quality tradeoffs
- Scroll top-to-bottom as narrative or jump to specific chapters
- Covers both pretraining and post-training (instruction tuning, RLHF)
- Explains optimizers (SGD, Momentum, Adam) with trajectory visualization
- Backpropagation chapter shows forward/backward pass step-by-step
- No code required - understand transformers visually and intuitively
Roy van Rijn cons
- Uses toy/scaled examples rather than actual production model weights
- No downloadable resources or offline version mentioned
- Does not cover encoder-decoder architectures in depth
- No interactive coding exercises or hands-on implementation
- Visualizations are simplified for intuition, not exact reproductions
- No certificate or formal certification upon completion
- Single-author content without peer review or citations
- No discussion of specific model comparisons (GPT vs Llama vs others)
Frequently asked questions about Roy van Rijn
What is the main goal of this guide?
The goal is not to memorize every formula, but to understand what changes at each step of LLM processing and why that step exists. By the end, you should be able to trace the full path from text to tokens to vectors to transformer blocks to logits to sampling to output, making the black box much smaller.
How do I use this interactive guide?
You can scroll from top to bottom as a single narrative, or jump between chapters to explore a specific concept. Each chapter has interactive controls you can experiment with, such as switching sentences in the tokenizer demo or selecting tokens to inspect attention links.
Why not just use whole words as tokens instead of subwords?
Whole words are too rigid. New names, typos, code, inflections, compound words, and multilingual text would constantly produce words the model has never seen. Subword tokens are the compromise: common text stays compact while unusual text can still be built from smaller pieces.
What is the difference between token IDs and embeddings?
Token IDs are just labels like row numbers in a table - token 15339 is not meaningfully close to 15340. Embeddings turn those labels into vectors: lists of learned numbers where tokens appearing in similar contexts end up with related vectors, creating a high-dimensional coordinate system.
Why do we need activation functions in neural networks?
Without activation functions, stacking layers would still behave like one large linear transformation that could be collapsed into a single matrix. The activation function breaks linearity, letting the network build curved, conditional, non-linear transformations instead of only scaling and rotating vectors.
What is the difference between logits and probabilities?
A logit is not a probability - it is just an unnormalized score. Higher usually means the model thinks the token fits better, but the numbers do not add up to 100%. Softmax converts logits into probabilities that sum to 1, then sampling chooses the next token.
How does attention work with Q, K, and V?
For each token, the model creates three learned views: query (what this token is looking for), key (what this token can be matched on), and value (the information this token can contribute). The model compares queries to keys to produce attention scores/weights, then uses those weights to mix the value vectors.
What is RoPE and why do we need it?
RoPE (Rotary Positional Embeddings) gives attention position information by rotating query and key vectors based on token positions. Since attention compares tokens by content but language depends on order (dog bites man vs man bites dog), RoPE makes the Q·K dot product sensitive to relative position.
What is the KV cache and why is it important?
The KV cache stores key and value vectors for each token during attention so they don't need to be recomputed for every new token during autoregressive generation. This saves massive compute (62.8x less repeated attention work in the example) but uses memory that grows with context length.
What is quantization and when should I use it?
Quantization reduces memory pressure by representing weights with fewer bits (8-bit, 4-bit instead of 16/32-bit). The tradeoff is less precision = less memory = often faster/cheaper inference = some approximation error. For local inference, quantization can be the difference between a model that doesn't fit in memory and one that runs comfortably.