Articles

LLM Perplexity Explained: What the Metric Measures

Perplexity measures how surprised a language model is by text: the exponent of its average per-token loss. How it is computed, read, and misread.

Chisato Chisato · · 5 min read
Abstract purple fibers resembling a neural network

Perplexity is a measure of how well a language model predicts a piece of text. Formally, it is the exponential of the model’s average negative log-likelihood per token — in plain terms, how “surprised” the model is by each token it sees. A lower perplexity means the model assigned higher probability to the tokens that actually appeared. A perplexity of 10 can be read as the model being, on average, as uncertain as if it were choosing uniformly among 10 equally likely options at every step.

It is the oldest and most direct way to evaluate a language model, and it remains the standard training-time metric. It is also widely misunderstood as a measure of how good or useful a model is, which it is not.

What a language model actually outputs

A language model reads a sequence of tokens and, at each position, produces a probability distribution over its entire vocabulary for what comes next. Those probabilities come from passing the model’s raw scores through a softmax.

Given the text The cat sat on the, a good model might assign mat a probability of 0.3, floor 0.15, couch 0.1, and spread the rest across thousands of other tokens. If the real next token is mat, the model did well. If it is xylophone, which it assigned a probability of 0.00001, the model was badly surprised.

Perplexity aggregates that surprise over a whole passage.

How perplexity is computed

For a sequence of N tokens, the model assigns a probability to each actual token given everything before it. The calculation is:

  1. For each token, take the log of the probability the model assigned to it.
  2. Average those log-probabilities across all N tokens and negate the result. This is the cross-entropy loss — the same quantity the model is trained to minimize.
  3. Exponentiate: perplexity = e^(loss) when natural logs are used, or 2^(loss) with base-2 logs. The result is the same either way, as long as the base matches.
loss       = −(1/N) × Σ log p(tokenᵢ | tokens before i)
perplexity = exp(loss)

Two boundary cases make the scale intuitive:

  • A model that predicts every token with certainty (probability 1) has a loss of 0 and a perplexity of 1. That is the floor.
  • A model that knows nothing and guesses uniformly over a vocabulary of size V has a perplexity of V.

Real models land between those extremes. Because perplexity is an exponential of the loss, small drops in loss late in training translate into noticeable drops in perplexity.

Why it uses the geometric mean

Exponentiating the average log-probability is the same as taking the inverse geometric mean of the per-token probabilities. The geometric mean is the right choice because probabilities multiply across a sequence: the probability of a whole sentence is the product of the conditional probabilities of its tokens. Perplexity normalizes that product by length, so passages of different sizes can be compared.

It also means a few very low-probability tokens hurt perplexity disproportionately. One token assigned a probability near zero contributes a huge negative log term. That property is useful for spotting where a model is confidently wrong.

What perplexity is good for

Tracking training. Perplexity on held-out validation text is the main signal that pretraining is working. If it stops falling, the model has stopped learning generalizable patterns; if training loss keeps falling while validation perplexity rises, the model is overfitting.

Comparing checkpoints of the same model. For two checkpoints with the same tokenizer evaluated on the same text, lower perplexity reliably means better next-token prediction.

Measuring compression damage. When a model is quantized or pruned, perplexity on a reference corpus is a fast first check of how much prediction quality was lost. A small increase suggests the compressed model is close to the original; a large jump means something broke.

Detecting unusual text. Text that a model finds highly perplexing may be out of domain, corrupted, or in a language the model barely saw. Perplexity is used to filter low-quality data from training corpora for this reason.

Where perplexity misleads

It is tokenizer-dependent. Perplexity is measured per token, and different tokenizers split the same text into different numbers of tokens. A model with a larger vocabulary uses fewer, longer tokens, each carrying more information, so its per-token perplexity is not comparable to a model with a smaller vocabulary. To compare across tokenizers, researchers normalize to bits per byte or bits per character instead.

It depends on the evaluation text. Perplexity on news articles says little about perplexity on source code. Numbers are only comparable when the dataset, preprocessing, and context length match.

Context length matters. A model evaluated with a long context window has more information for each prediction than the same model evaluated in short, independent chunks. How text is split into windows — and whether those windows overlap — changes the result.

It does not measure usefulness. Perplexity rewards matching the statistics of the evaluation text, not being correct, helpful, or safe. A model fine-tuned to follow instructions often has higher perplexity on raw web text than its base model, because it has been shaped toward a particular style of response, yet it is far more useful in practice. Task-based LLM evals are the right tool for judging that.

It is not about generation settings. Perplexity is computed from the model’s probabilities over known text. Sampling knobs like temperature affect how text is generated, not the perplexity of a fixed passage under the model’s unmodified distribution.

Perplexity vs task-based evaluation

PerplexityTask-based evals
What it measuresNext-token prediction quality on a corpusPerformance on specific tasks (answers, code, reasoning)
Needs labeled answersNo, just textUsually yes, or a grader
Cost to computeOne forward pass over the textGeneration plus grading
Comparable across modelsOnly with the same tokenizer and dataYes, if the harness is the same
Best forTraining, compression checks, data filteringChoosing and shipping a model

The takeaway

Perplexity is the exponential of a language model’s average per-token cross-entropy loss: a number that starts at 1 for a perfect predictor and grows as the model becomes more surprised by the text. It is the right metric for watching a transformer train, comparing checkpoints, and checking whether quantization hurt quality. It is the wrong metric for comparing models with different tokenizers or for deciding which model is most useful — for that, normalize to bits per byte or, better, run task-based evaluations.

Chisato Chisato · · 5 min read

Prefill vs Decode: The Two Phases of LLM Inference

LLM inference runs in two phases: prefill processes the prompt in parallel, decode generates tokens one at a time. Why they behave so differently.

#AI #LLMs #Machine Learning