Prefill vs Decode: The Two Phases of LLM Inference
LLM inference runs in two phases: prefill processes the prompt in parallel, decode generates tokens one at a time. Why they behave so differently.
Every response from a large language model happens in two distinct phases. Prefill reads the entire prompt at once, processing all input tokens in parallel to build up the model’s internal state. Decode then generates the output one token at a time, each new token depending on the one before it. The two phases stress hardware in opposite ways — prefill is limited by raw compute, decode by memory bandwidth — and almost every optimization in modern LLM serving is about handling that mismatch well.
A quick refresher on how generation works
A transformer-based LLM predicts the next token given all the tokens so far. To do that, each layer’s attention mechanism needs “key” and “value” vectors for every earlier token. Recomputing those for the whole sequence at every step would be wasteful, so inference engines store them in a KV cache and reuse them.
Prefill and decode are defined by how that cache gets filled.
Prefill: reading the prompt
During prefill, the model already knows every input token: the system prompt, the conversation history, any retrieved documents, and the user’s question. Since nothing needs to be predicted yet, all of those tokens can be pushed through the network together as one large batch of matrix multiplications.
That has three consequences:
- It is compute-bound. Large matrix multiplications are exactly what GPUs and other accelerators are built for. Each weight loaded from memory is reused across many tokens, so the arithmetic units stay busy.
- Cost grows with prompt length. More input tokens means more computation, and the attention part grows faster than linearly with sequence length. Techniques like FlashAttention exist largely to keep long prefills efficient.
- It sets time to first token. The user sees nothing until prefill finishes and the first output token is produced. Long prompts mean a longer pause before streaming begins.
At the end of prefill, the KV cache holds entries for every prompt token, and the model has produced its first output token.
Decode: generating the answer
Decode is the loop that follows. On each step the model takes the single most recent token, runs it through every layer, attends to everything in the KV cache, appends the new key and value vectors, and emits one more token. Then it repeats until it reaches a stop condition.
The work per step is tiny compared with prefill — one token’s worth of computation — but the model still has to read all of its weights from memory for that one token, plus the entire KV cache for attention. Very little arithmetic is done per byte loaded.
- It is memory-bandwidth-bound. The accelerator spends most of its time waiting for weights and cache data to stream in from high-bandwidth memory, not computing. This is a textbook case of bandwidth versus latency dominating performance.
- It is inherently sequential. Token n+1 cannot be computed until token n exists, so a single request cannot be parallelized across output positions.
- It sets the streaming speed. The time per decode step determines how fast text appears once it starts, usually described as inter-token latency or tokens per second.
Prefill vs decode at a glance
| Prefill | Decode | |
|---|---|---|
| Tokens processed per step | All prompt tokens at once | One per request |
| Parallelism within a request | High | None across output positions |
| Primary bottleneck | Compute | Memory bandwidth |
| Hardware utilization (single request) | High | Low |
| User-facing metric | Time to first token | Inter-token latency, tokens per second |
| Scales with | Input length | Output length (and context size) |
Why the asymmetry matters
Batching fixes decode, up to a point. If decode is waiting on memory, the obvious fix is to make each weight load do more work. Serving systems group many users’ decode steps into one batch, so a single pass over the weights produces a token for dozens of requests. Continuous batching takes this further by letting requests join and leave the batch at every step instead of waiting for a whole batch to finish. The limit is memory: every request in the batch needs its own KV cache, which is why techniques like PagedAttention for cache management and grouped-query attention for smaller caches matter so much.
Mixing phases causes interference. A long prefill arriving in the middle of a busy decode batch can stall everyone else’s token stream for a moment, because the accelerator is busy chewing through the new prompt. Chunked prefill splits long prompts into pieces and interleaves them with decode steps, smoothing out those stalls at a small cost to the new request’s time to first token.
Some systems split the phases entirely. In disaggregated serving, one pool of accelerators handles prefill and another handles decode. The prefill workers build the KV cache, transfer it to a decode worker, and move on. Each pool can then be sized and tuned for its own bottleneck. The trade-off is the cost of moving potentially large caches between machines.
Decode tricks target the sequential step. Speculative decoding uses a small draft model to guess several tokens ahead, then has the large model verify them in one pass. Verification looks like a mini prefill — several tokens at once — so it uses the hardware more efficiently than plain one-at-a-time decode.
What this means for application developers
You do not need to run inference servers to benefit from understanding the two phases.
- Long prompts slow the start, long outputs slow the finish. If users complain about the wait before text appears, look at prompt size. If they complain that responses take forever to complete, look at output length.
- Input and output tokens are priced differently. Providers commonly charge more per output token than per input token, which reflects that decode uses hardware less efficiently. You can model the trade-off with our LLM token cost calculator.
- Repeated prefixes can skip prefill. When many requests share the same long system prompt or document, prompt caching lets the provider reuse the stored KV cache instead of recomputing it, cutting both latency and cost.
- Streaming hides decode latency. Showing tokens as they arrive makes a slow decode feel responsive, since users start reading after the first token rather than waiting for the last.
The takeaway
Prefill processes the whole prompt in parallel, keeps accelerators busy, and determines how long users wait for the first token. Decode produces one token at a time, spends most of its time waiting on memory, and determines how fast the answer streams. Batching, cache management, chunked prefill, disaggregated serving, and speculative decoding are all ways of coping with that split — and for application builders, it explains why prompt length, output length, and caching each affect speed and cost in different ways.
Tagged
Keep reading
Chisato · · 4 min read What Is Speculative Decoding? LLM Inference Explained
Speculative decoding speeds up LLM text generation by having a small draft model guess tokens the large model verifies in one pass. Here's how it works.
Chisato · · 6 min read HyDE Explained: Hypothetical Document Embeddings for RAG
HyDE improves RAG retrieval by having an LLM write a fake answer first, then searching with that answer's embedding instead of the raw query.
Chisato · · 5 min read LLM Perplexity Explained: What the Metric Measures
Perplexity measures how surprised a language model is by text: the exponent of its average per-token loss. How it is computed, read, and misread.