Evaluation & Benchmarking

Why evaluating an LLM is harder than evaluating a classifier#

A traditional classifier has one right answer per input, so evaluation is straightforward: compare the prediction to the label, compute accuracy/precision/recall (see this site's Machine Learning tutorials). An LLM's output is open-ended free text — there's often no single "correct" answer, multiple different phrasings can be equally good, and quality itself is often subjective (is this summary "good"? that depends on what the reader needed). Evaluating LLMs well means combining several different, complementary methods, because none of them alone gives a trustworthy full picture.

Perplexity: a training-level, not task-level, metric#

Perplexity measures how well a language model predicts a held-out piece of text — roughly, how "surprised" the model is by the actual next token, averaged over a sequence. Lower perplexity means the model assigned higher probability to what actually came next, indicating it's a better statistical model of that kind of text.

Perplexity = 2^(average negative log-probability of each token)

Perplexity is useful for comparing base language models during pretraining or measuring how well a model fits a particular domain's text, but it says nothing directly about whether the model is useful for a specific task — a model can have excellent perplexity on generic web text and still fail badly at, say, following a specific instruction format or answering factually. It's a foundational research metric, not something most application teams need to compute themselves.

Benchmark suites: standardized, comparable tasks#

Benchmark suites are fixed sets of tasks with known correct answers, used to compare different models' capabilities on a standardized, reproducible basis — things like multi-subject academic-style question answering, grade-school and competition math problems, and coding tasks with automatically checkable test cases. Their strength is objectivity and comparability across models and over time; their weakness is that a benchmark score doesn't necessarily predict performance on your specific application's actual use case, and once a benchmark becomes well-known, there's a real risk that its problems (or close variants) leak into later training data, inflating scores in a way that no longer reflects genuine generalization.

Human evaluation#

Human evaluation has people directly judge model outputs — rating helpfulness, correctness, tone, or comparing two models' answers side by side and picking the better one ("pairwise comparison"). It's the closest available proxy for real-world usefulness and the only method that reliably captures subjective quality (tone, clarity, appropriateness) that automated metrics struggle to score at all. Its costs are real too: it's slow, expensive to run at scale, and subject to inter-rater disagreement — different human raters, or even the same rater on different days, can score the same output differently.

LLM-as-judge#

LLM-as-judge uses a separate (often more capable) LLM to score or compare outputs, following a rubric provided in its own prompt — a way to get evaluation that's faster and cheaper than pure human review, at a scale human review can't match.

You are an evaluator. Compare the two summaries below and decide which
one better captures the key points of the source article, without adding
information not present in the source. Respond with "A", "B", or "Tie",
followed by a one-sentence reason.

Source article: [...]
Summary A: [...]
Summary B: [...]

It's genuinely useful for rapid iteration — checking dozens of prompt variants against a rubric far faster than a human panel could — but it carries its own well-documented biases: judge models often favor longer answers regardless of actual quality, can favor a particular writing style, and (in some setups) can be biased toward whichever answer appears first in the prompt. Treating LLM-as-judge scores as a directional signal to combine with periodic human spot-checks, rather than as ground truth on their own, is the standard way production teams use it responsibly.

Comparison table#

Common mistakes#

• Relying on a single benchmark score as proof a model will perform well on your specific application — benchmark performance and real-world task performance can diverge significantly, especially for narrow, domain-specific use cases.
• Treating LLM-as-judge output as ground truth with no human spot-checking at all — judge models have documented, systematic biases (favoring longer or more confidently-worded answers) that can silently skew results if never validated against human judgment.
• Using perplexity to compare models meant for different tasks or trained on different data distributions — it's only a meaningful comparison between models evaluated on the same held-out text.