Evals at scale

The medium-track lesson covered the "why" of evals. This is the "how" once you have real production traffic, multiple model variants in flight, and a team that needs eval results in CI before merging.

What evals at scale actually do

Evals at small scale are a script and a golden set. Evals at scale are infrastructure: golden tests run on every PR, real production traffic gets sampled and scored continuously, regressions trip alerts, and interesting failures flow back into the test suite. The same loop, but always-on and instrumented.

The point isn't more tests — it's trustable tests. Without them, every model change, prompt tweak, or library upgrade is a coin flip. With them, you can ship with the confidence that you'd know within hours if something regressed.

That's the whole concept. Below: the components of a mature eval setup, the discipline LLM-as-judge requires to be usable, and the observability layer underneath.

When you'd reach for it

Once you have real production traffic and people depending on the system:

• Multiple model variants in flight — Sonnet vs Haiku, frontier vs fine-tuned, cost vs quality experiments.
• A team that needs CI gates — PRs that drop a metric beyond threshold get flagged automatically.
• Customer-facing AI where a 0.4% regression matters because someone notices.
• Drift over time — production traffic evolves; static evals go stale.

You don't need this on day one of a prototype. The investment compounds once the product is real.

How it's actually built

A mature eval setup is its own piece of software:

You don't have to build all of this on day one. But every piece pays for itself when something breaks.

Golden sets vs. live eval

• Golden sets are static. Great for catching regressions and comparing variants — but they get stale.
• Live evals sample production traffic (e.g. 1%) and score with the same harness. Surface drift, edge cases, and emerging failures.

Healthy pattern: golden set in CI for known cases, live eval for unknown cases, and a feedback loop that promotes interesting live failures into the golden set.

Where it bites in real life

Observability and tracing

Non-negotiable in production. Every call traced with: inputs, outputs, model, latency, token counts, cost, eval scores, and a unique ID you can search. Tools: Langfuse, LangSmith, Braintrust, Helicone — or roll your own with structured logs into Postgres/DuckDB. Pick one before you need it.

Under the hood (optional)

A pairwise LLM-as-judge with order randomisation: randomly flip A/B before showing them to the judge, then translate the judge's pick back to the original positions. ~15 lines. Skip if you don't code — the takeaway is always randomise position to mitigate bias.

import random
 
def pairwise_score(question, answer_a, answer_b, judge_model):
    if random.random() < 0.5:
        a, b, swapped = answer_a, answer_b, False
    else:
        a, b, swapped = answer_b, answer_a, True
 
    prompt = f"""Compare two answers to the question. Pick the better one.
 
Question: {question}
 
Answer A: {a}
Answer B: {b}
 
Output JSON: {{ "winner": "A" or "B" or "tie", "reason": str }}"""
 
    j = json.loads(call_llm(judge_model, prompt))
    if j["winner"] == "tie":
        return "tie"
    pick = j["winner"]
    return ("A" if not swapped else "B") if pick == "A" else ("B" if not swapped else "A")

Further reading

• Hamel Husain — A Field Guide to Rapidly Improving AI Products — production-grade eval thinking.
• Braintrust — Eval techniques — concrete patterns for LLM-as-judge, scoring, CI integration.
• LangSmith — Evaluation — managed traces + evals.
• Anthropic — Evaluating prompts — first-party eval tooling.
• Chatbot Arena (LMSYS) — public Elo ranking from millions of pairwise votes.