02 Measure quality

Facilitator companion to the learner lesson 02 Measure quality.

Timing

~15 minutes total.

• 5 min — per-tier accuracy and outcome breakdown on the winning config's Leaderboard detail view.
• 4 min — the llm_judge secondary signal, and where it disagrees with correctness.
• 6 min — drill into two or three individual LangFuse traces for wrong/low-scoring questions.

Talk track

• Reframe the goal: winning the Arena tells you a config beat the rest in aggregate; this module is about how it wins and where it's weakest — same idea as knowing not just that a candidate passed an interview, but which questions they nailed.
• Say explicitly that this module produces no new data — everything here was already captured by the correctness and llm_judge evaluators in Module 01. This is a reading exercise, not a re-run.
• On per-tier accuracy: point out that a config can look strong overall while being shaky on the hardest tier, and that's exactly what an aggregate leaderboard number hides.
• On the outcome breakdown: walk through the categories out loud — sandbox-rejected SQL, a ClickHouse error, an empty result, a wrong result set — and that each is a different kind of problem with a different fix, not one undifferentiated "failure."
• On llm_judge: it's binary-correctness's finer-grained sibling — a config can be correct by execution accuracy while writing SQL a reviewer would still flag (an unnecessary subquery, a fragile date comparison). Find one live disagreement between correctness and llm_judge if you can, it's the clearest way to make the distinction land.
• Close by picking two or three wrong/low-scoring questions and reading their full traces end to end live — prompt sent, generated SQL, error or result — looking out loud for a pattern (a phrasing, a join, a date filter the model keeps mishandling). This is the same trace-reading skill Module 04 reuses on production traffic.

Common failures

• LangFuse evaluators not configured — if Module 01 fell back to --grade local, there is no llm_judge score to read here, and the outcome breakdown may be thinner (local grading covers correctness but not every category the LangFuse evaluators produce). Say so plainly and focus the module on correctness + trace drill-down only.
• No wrong answers to drill into — if the winning config scored 100% on the demo grid, pick a non-winning config's failures instead; the trace-reading skill is the point, not finding a flaw in the winner specifically.
• ClickHouse not seeded / harness never ran — the Leaderboard detail view (per-tier accuracy, outcome breakdown) is empty. This means Module 01 didn't finish; go back and re-run it before continuing.
• Clicking through to a trace 404s or loads the wrong project — usually means the browser is pointed at a different LangFuse project than the one in .env, or LANGFUSE_BASE_URL/keys don't match the account that ran Module 01's harness.

Reset steps

• If the Leaderboard detail view is empty, re-seed and re-run the cheap subset: scripts/arena.sh up, then python -m eval.harness --run-id demo2 --models qwen3.7-flash,gpt-5.6-luna --prompts P1_zeroshot,P3_dialect.
• Use a fresh --run-id for the re-run so it's unambiguous which Leaderboard rows and which LangFuse Experiment you're reading in the room.
• If only the dashboard rendering is stuck (data exists in ClickHouse), restart the local servers without re-seeding: scripts/arena.sh stop && scripts/arena.sh serve.
• If evaluators were the gap, this module's depth depends on fixing that before the next session — there's no in-session substitute for the LangFuse-side scores.