LLM Evals in 2 Hours

Two hours is enough to get the whole discipline, because the discipline is small. Most of what's written about evals is tooling noise around five ideas. This walks the five ideas in order, marks every claim as works or doesn't, and ends with a 30-minute exercise that produces a real artefact.

The whole thing in ten claims

If you read nothing else, read this. Every claim below is defended later with a source and a number.

1. Benchmarks tell you which model is smartest. Evals tell you whether your system works. Confusing the two is the single most common failure mode in the field right now.
2. Looking at your data beats any framework. Run 100 examples, read every output by hand, write down what went wrong. That habit outperforms every tool you could buy.
3. Build evaluators only for failures you have actually observed. Imagined failure modes produce metrics nobody acts on.
4. Off-the-shelf metrics ("helpfulness", "coherence", BLEU, ROUGE) are worse than useless on real products. They give a number that moves without telling you anything, which is worse than having no number.
5. Three layers, cheapest first: deterministic code checks → LLM-as-judge → human review. Never send to a judge what a regex can catch.
6. One judge, one dimension, binary output. Three to five sharp judges beat twenty-to-thirty noisy ones. Likert 1–5 scales are a trap.
7. An uncalibrated judge is worse than no judge — it manufactures false confidence. Calibrate against 50–100 human-labelled examples (including bad ones) and hit high-80s to 90s agreement before you trust it.
8. Judges are measurably biased: ~10–15 points of win-rate swing from answer order, 15–30 points of inflation for longer answers, 10–25% self-preference. Engineering problems with known mitigations, not reasons to give up.
9. For agents, the final answer is not the unit of evaluation. Score the trajectory and the tool calls too — a 20-step agent at 95% per-step reliability succeeds end-to-end ~36% of the time.
10. Eval speed is a product feature. Airbnb went from weeks to same-day mostly by caching judge scores so runs became deterministic — boring systems engineering, not better models.

The mental model: benchmarks are not evals

A benchmark answers which model is smartest in general. MMLU, SWE-bench, GPQA. Useful once a quarter when you pick a base model. Irrelevant to whether your support assistant hallucinates refund policy.

An eval answers does my system do its job on my traffic. It is built from your failure modes, on your data, with your definition of correct. It is a property of your product, not of the model. You cannot download it.

The three-layer stack

Airbnb's architecture, and the consensus shape across every serious team: escalate only what the cheaper layer can't decide.

The loop that works: error analysis

This is the core. Everything else is scaffolding around it. It comes from Hamel Husain and Shreya Shankar and it is the closest thing the field has to a settled method.

How it actually goes

1. Sample ~100 real traces. Production traffic if you have it, realistic inputs if you don't. Fewer than 100 and your taxonomy is still moving.
2. Open coding. Read each output. Write a free-form sentence about what's wrong. No categories yet — categories imposed early are the ones you imagined, not the ones you have. Do this yourself. Delegating open coding to an LLM saves an hour and costs you the product intuition the whole exercise exists to build.
3. Axial coding. Now group the notes into 5–10 named failure modes. An LLM is genuinely useful here — clustering notes you wrote is fine; writing the notes is not.
4. Count. Pivot table, failure mode by frequency. This is your priority queue, and the answer to "what should we work on" for the next month.
5. Fix the top one. Usually a prompt change. Then re-sample and see whether the distribution moved.

The worked example

Airbnb's fictionalised support-policy assistant. 100 test inputs through a first prototype produced four failure modes — and the counts immediately tell you what to build:

Judges you can actually trust

This is where most teams' evals quietly stop meaning anything. A judge that isn't calibrated still returns a confident number every run, that number goes on a dashboard, and decisions get made from it. Airbnb's phrasing is the one to remember: a judge without calibration is worse than no judge at all, because it gives you false confidence.

Four design rules, in order of how much they matter

1. One judge, one dimension

No "God evaluator" scoring overall quality. One judge for faithfulness, one for tone, one for conciseness. Compound rubrics can't be debugged — when the score drops you don't know which thing broke. Airbnb's target: 3–5 sharply calibrated judges, not 20–30 noisy ones.

2. Binary, not Likert

Pass/fail. Not 1–5. Likert scales invite the question "what separates a 3 from a 4?", which neither your annotators nor your model can answer consistently; they push mass to the middle; and they need larger samples to detect a real difference. Binary forces the rubric to be sharp, which is the actual goal. This is the 2026 consensus and one of the highest-leverage changes you can make to an existing eval suite.

3. The rubric is the work

Ambiguity in a rubric means humans can't agree either, and if humans can't agree you cannot automate. Airbnb's rule: if your experts disagree on labels, stop. Fix the human disagreement first. Automating an ambiguous standard just scales the ambiguity.

A rubric that works looks like this — criterion, explicit fail conditions, worked examples on both sides:

READABILITY — score 1 or 0.

Score 1 when the text is clean, natural language a guest would read
without friction.

Score 0 when ANY of these hold:
  · tone is too formal, too casual, or robotic
  · uses internal terminology ("listing entity", "host ops")
  · formatting issues: sentence fragments, missing end punctuation
  · grammatical errors
  · needlessly complex sentence structure

PASS: "This place is a 10-minute walk from the beach and has a full
       kitchen."
FAIL: "The listing entity provides beach proximity (approx. 10 min
       ambulatory) and comprehensive culinary facilities"
       → internal terminology, robotic tone.

4. Judge with a different model than you generate with

Self-preference is measured at 10–25%. Using the same model as generator and judge builds that bias directly into your release gate. Use a different family, and give the judge chain-of-thought before its verdict.

The calibration protocol — and where the value actually is

You are measuring your judge against a human. That means a 2×2, and the two off-diagonal cells are worth more than the two on-diagonal ones.

1. Build a golden set of 50–100 examples, human-labelled by someone who knows the domain. Include bad examples. A golden set of only good outputs tests nothing — you're measuring whether the judge can say yes.
2. Run the judge against it.
3. Measure agreement. Cohen's kappa or Krippendorff's alpha (they correct for agreement-by-chance, which raw accuracy doesn't). Target high 80s to 90s. Track TPR and TNR separately.
4. Read the disagreements. The step people skip and where the information is. Airbnb's faithfulness judge started at 78%; reading disagreements revealed it was marking accurate paraphrases as unfaithful. Rubric refinement plus new few-shot examples took it to 88%.
5. Recalibrate on a schedule. Failure modes evolve, models get silently updated, rubrics age faster than the system under test.

The biases, with measured magnitudes

Technique catalogue: what works, what doesn't

Every named technique you'll hear, placed by what it costs to run against how much it actually tells you about your product.

G-Eval, specifically — because you'll be asked about it

Three mechanisms stacked: (1) it turns your natural-language criterion into explicit evaluation steps automatically, (2) a form-filling prompt lets one pass assess several fields — input, output, expected output, retrieved context, (3) it weights the score by token-level log-probabilities instead of taking the raw integer, giving finer granularity than a model that emits "4" every time.

The original paper reported 0.514 Spearman correlation with human judgment on summarisation — the best of its baselines on coherence, consistency, fluency and relevance. That's a real result, and also worth reading honestly: 0.51 is a moderate correlation, not a substitute for human labels.

Verdict: a good structured default for subjective dimensions, and strictly better than asking a model "rate this 1–5". Not a replacement for calibrating against humans, and no answer to any of the bias problems.

Tooling, in one paragraph

Experienced teams converge on two tools, not one: a lightweight library for CI gating (DeepEval, Promptfoo, or RAGAS if you're RAG-heavy) plus a platform for human annotation, regression tracking and dashboards (Braintrust, LangSmith, Arize/Phoenix). LangSmith is the path of least resistance inside LangChain; RAGAS is the specialist for retrieval metrics; Promptfoo is strongest for CLI red-teaming and cross-model matrices. But note the ordering: none of these matter until you've done error analysis, because every one of them will happily hand you generic metrics that mean nothing about your product.

Agents, speed, and production

Agents: the final answer is the wrong unit

Start from the arithmetic that governs everything.

Three levels, all of which need evals:

Practically: tracing is the backbone. Reconstruct execution paths from spans by tree traversal, then attach evals to specific subagents. Without traces you can see that something failed but not where — and "where" is the only actionable part.

Making evals fast enough to iterate on

The second Airbnb piece is really a systems-engineering story, and the most transferable part of the whole set. Their problem wasn't model quality — an eval cycle took weeks, so nobody could iterate.

1. Separate the two noise sources. Epistemic uncertainty (the judge/model is limited) versus aleatoric uncertainty (the task itself is ambiguous). Conflating them makes you sample more data to fix a problem more data can't fix.
2. Cache both axes. References keyed by sample ID + generation config; judge scores keyed by sample + model output + judge config + metric. Identical inputs return cached results, so evaluation becomes deterministic and runs become comparable. This is the whole trick.
3. Bounded mutation instead of retraining. Micro-adapters — LoRA patches of rank <50, under an hour on one GPU — with rules to fuse co-triggering patches and auto-unload unused ones. Days → an hour.
4. End-to-end validation on representative inputs, stratified across high-volume segments, long-tail locales and prior incidents. Component tests pass while the seams fail.

Offline evals, online evals, guardrails — three different jobs

The statistics you can't skip

• ~92 test cases gets you reliable detection of a medium effect (w=0.5) at α=0.005 with 0.90 power. Most benchmarks sit at 261–599 per split, comfortably above. This is why "100 examples" keeps appearing — it's not folklore.
• Bootstrap your confidence intervals. Resample with replacement, 500–1,000+ replicates, recompute the metric. If two systems' 95% CIs don't overlap, the difference is real. If they overlap, you're looking at noise. Watch the cost: bootstrapping an LLM judge naively multiplies your API bill by the replicate count — bootstrap over cached scores, not fresh calls.
• A 100% pass rate means your eval is too easy. A suite sitting around 70% is doing more work for you than one at 100%. Optimising the pass rate rather than the product is a real and common failure.

Anti-pattern cheat sheet

The consolidated "doesn't work" list. Most of these cost a quarter each.

The 30-minute exercise

Reading this gets you the map. This gets you the skill. Pick any LLM feature you or your team already ship — a summariser, an extractor, an agent step, anything with real traffic.

1. (10 min) Open code 20 outputs. Real ones. One free-form sentence per output on what's wrong, in a spreadsheet. Don't categorise yet. Don't let a model do this.
2. (5 min) Axial code. Group your notes into named failure modes. You'll get 3–6. Count them. You now have a prioritised list nobody in the org has.
3. (5 min) Sort them by layer. Which are catchable in code? Which need a judge? Which are just prompt bugs you can fix right now? Usually at least one is a ten-minute fix.
4. (10 min) Write one binary rubric for the most frequent judge-shaped failure. Criterion, explicit fail conditions, one passing and one failing example — the readability rubric above is the template.

What you should have at the end:

• A counted failure taxonomy for one real feature
• One binary rubric, written down
• At least one fix you can ship today
• A concrete sense of what your golden set needs to contain

---

Sources

• Eval-Driven Development: Lessons from Evaluating GenAI at Scale — Airbnb Engineering. The three-layer architecture, calibration protocol, rubric design, the worked support-assistant example, the 78%→88% calibration story.
• From Weeks to a Day: How We Made LLM Evaluation Fast Enough to Iterate On — Airbnb Engineering. Caching for determinism, epistemic vs aleatoric noise, micro-adapters, "debt at the seams".
• LLM Evals: Everything You Need to Know — Hamel Husain & Shreya Shankar. Open/axial coding, sample sizes and saturation, binary over Likert, TPR/TNR judge validation, CI vs production, the "don't do this" list.
• G-Eval: The Definitive Guide — Confident AI. Mechanism (auto-CoT, form-filling, log-prob weighting), the 0.514 Spearman result, DAG evals.
• Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge and Self-Preference Bias in LLM-as-a-Judge — the bias magnitudes and the perplexity-familiarity mechanism.
• LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods — the systematic survey of the space.
• Agent Evaluation: A Detailed Guide — Cameron Wolfe. Trajectory, per-turn and tool-call evaluation.
• Experiments, Online Evaluations, and Guardrails — when to use each — LangWatch. The three-jobs distinction.
• LLM-as-a-Judge in 2026: How It Works, When It Fails — what G-Eval left unsolved; agent-as-judge and distilled evaluators.
• LLM evaluation with confidence intervals — bootstrap methodology and sample-size power analysis.