Evaluating Prompts Systematically

Why "it looks good when I tried it" isn't evaluation#

Testing a prompt by trying it a few times and reading the output is exactly how most prompts get shipped — and exactly why prompt quality regressions are so common in production. LLM output is non-deterministic (even at low temperature, small input variations can shift output meaningfully), so a handful of manual tries tells you almost nothing about how a prompt performs across the actual range of real inputs it will see, and gives you no way to notice when a "small tweak" quietly makes things worse on cases you didn't happen to re-check.

Building an evaluation dataset#

A prompt evaluation dataset is a set of representative inputs, ideally paired with either a known-correct answer or a clear rubric for what a good answer looks like. It should deliberately include:

• Typical cases — the bulk of what the prompt will actually see in production.
• Edge cases — ambiguous, unusual, or boundary inputs that are disproportionately likely to reveal a prompt's weak spots (an empty input, a wildly off-topic request, a case with two plausible right answers).
• Known past failures — every time a prompt produces a bad output in production, that exact input becomes a permanent addition to the evaluation dataset, so the same mistake gets caught automatically the next time the prompt changes.

[
  {
    "input": "The battery life is disappointing, but the camera quality is excellent.",
    "expected_category": "mixed",
    "notes": "Known edge case: previously misclassified as purely positive"
  },
  {
    "input": "Fast shipping and works exactly as described.",
    "expected_category": "positive"
  },
  {
    "input": "",
    "expected_category": "invalid_input",
    "notes": "Empty input -- should be rejected, not classified"
  }
]

A/B testing prompts#

A/B testing compares two prompt variants against real (or held-out) traffic, measuring which one performs better on a defined metric — task success rate, user satisfaction rating, downstream conversion, or a scored rubric. The core discipline is changing exactly one meaningful thing between variant A and variant B; changing several elements of a prompt at once (tone, examples, and format instructions, all in one revision) makes it impossible to attribute a measured difference to any specific change.

Variant A (current):
"Summarize this support ticket in 2-3 sentences."

Variant B (candidate):
"Summarize this support ticket in 2-3 sentences. Include the customer's
specific request and any error messages they mentioned, if present."

-- Run both against the same 200 held-out tickets, score each summary
-- against a rubric (or human review), compare aggregate scores.

Regression testing prompts#

Just as application code gets a test suite that runs before every change ships, a prompt in active use benefits from an evaluation suite that runs automatically whenever the prompt changes — checking the new version against the full evaluation dataset (including every previously-recorded failure case) before it replaces the old one in production.

def evaluate_prompt(prompt_template, eval_dataset, judge_fn):
    results = []
    for case in eval_dataset:
        output = run_prompt(prompt_template, case["input"])
        passed = judge_fn(output, case)
        results.append({"input": case["input"], "passed": passed})
    pass_rate = sum(r["passed"] for r in results) / len(results)
    return pass_rate, results

# Before shipping a prompt change, confirm it doesn't regress previously-fixed cases
pass_rate, results = evaluate_prompt(new_prompt, eval_dataset, judge_fn)
assert pass_rate >= baseline_pass_rate, "New prompt regresses on the eval set"

judge_fn can be an exact-match check for structured tasks, a rule-based check (does the output contain a required field, stay under a length limit), or an LLM-as-judge call (see this site's LLM tutorials on evaluation) for more open-ended tasks where there's no single correct string to match against.

Common mistakes#

• Evaluating a prompt change only against a handful of examples picked because they happen to be easy to check — the exact cases most likely to reveal a regression are usually the edge cases and past failures, not the easy typical ones.
• Changing multiple things in a prompt at once and then A/B testing the result — if the new variant wins or loses, there's no way to know which specific change actually caused it.
• Treating an evaluation dataset as a one-time setup task instead of a living artifact — every real production failure should get added to it, so the same regression can never silently ship again unnoticed.