02 Measure quality

Starting point

Module 01 complete: the Arena has run, the Leaderboard is populated, and you have a winning config_id (<model>__<prompt>, e.g. claude-sonnet-5__P1_zeroshot).

Why

Winning the Arena tells you a config beat the rest on cost per correct answer in aggregate. It doesn't tell you how it wins, where it's weakest, or whether its SQL is merely correct or actually good. Before you build on this model (improve it, release it), it's worth understanding it — the same way you'd want to know not just that a candidate passed an interview, but which questions they nailed and which they barely scraped by on. LangFuse already has everything you need for this: the evaluators from Module 01 scored every item, and every item has a full trace. This module is about reading that detail, not producing new data.

Concepts — under the hood

Trace → observations → scores. LangFuse structures every run of the agent the same way:

• A trace is one agent run — one model × prompt × question execution, named agent_run and tagged with the config_id.
• Observations are the spans inside that trace. The only one is llm_call, a generation observation carrying the prompt, the completion, token usage, cost, and timing for the model call. There's no separate observation for the SQL execution step — the SQL runs as a plain ClickHouse call with no LangFuse span of its own; the generated SQL and its result set land on the trace root's input/output instead.
• Scores are what the evaluators from Module 01 attach to the trace/dataset item after the fact: correctness (binary execution accuracy), llm_judge (graded LLM-as-a-judge SQL quality), and an outcome category.

Tiers. Each of the 20 golden questions in arena-golden has a tier from 1 (simplest — single-table counts and filters) to 5 (hardest — multi-table joins, funnels, margin calculations). Per-tier accuracy exists because a config's overall number can hide a tier-5 collapse behind strong tier-1/2 performance.

Outcome categories. eval/grading.py classifies every non-pending row into one of these, in order of how "far" the answer got:

Each implies a different fix: a sql_policy_rejected run needs a better system prompt about staying read-only; sql_exec_error usually means a dialect gap (see P3_dialect); empty_but_expected and wrong_result usually mean a filter, join, or aggregation logic error.

Goal

Comfortable reading per-tier accuracy and the outcome breakdown for your winning config, understanding what the secondary llm_judge signal adds on top of raw correctness, and able to drill from a leaderboard row into the exact LangFuse trace behind any one question.

Step 1 — Read per-tier accuracy and the outcome breakdown

Open http://localhost:5174 → Leaderboard and click into your winning config's row. Alongside accuracy, latency, and cost-per-correct-answer, each config shows:

• Per-tier accuracy — questions in arena-golden are grouped by difficulty tier; a config that looks strong overall can still be shaky on the hardest tier, and that's exactly the kind of gap an aggregate number hides.
• Outcome breakdown — not every non-correct answer fails the same way. Some SQL is rejected by the sandbox, some returns a ClickHouse error, some returns an empty result, some just returns the wrong result set. Each is a different kind of problem with a different fix.

How to read it. Per-tier accuracy is a small table or bar set, one row per tier 1–5 — scan right to left for where the number drops off; a config that's near-perfect on tiers 1–2 and falls off a cliff at tier 4–5 is telling you it handles simple lookups fine but struggles with joins and multi-step aggregation. The outcome breakdown is a count per category (correct, sql_policy_rejected, sql_exec_error, empty_but_expected, wrong_result) — a pile of sql_exec_error points at dialect problems, a pile of wrong_result points at logic problems, and they call for different fixes.

Step 2 — Read the llm_judge secondary signal

The correctness score is binary: does the result set match, yes or no. The llm_judge score you configured in Module 01 is a secondary, finer-grained signal — an LLM-as-a-judge rating of SQL quality on top of that binary outcome. A config can be correct by execution accuracy while still writing SQL a reviewer would flag (an unnecessary subquery, a fragile date comparison, a join that happens to produce the right rows for this data but wouldn't generalize). Use llm_judge to spot that gap between "passes" and "well-written."

Step 3 — Drill into individual traces

Click through from a leaderboard row to its per-question results, then click any one question to open its LangFuse trace. Each trace carries the full path for that question: the prompt sent to the model, the generated SQL, the model's llm_call generation (prompt, completion, token counts, cost), span timings for every step, and — if the question failed — the ClickHouse error that came back. This is the same trace-reading skill you'll use again in Module 04 once questions start arriving from real users instead of the golden dataset.

Pick two or three questions your winning config got wrong (or scored low on llm_judge) and read their traces end to end. You're looking for a pattern: a phrasing, a join, a date filter the model consistently mishandles.

How to verify you are done

• You can state your winning config's accuracy on at least one specific tier, not just its overall number.
• You can point to at least one question where correctness and llm_judge disagree, or explain why they don't for your run.
• You've opened at least one LangFuse trace and can walk through prompt → generated SQL → result or error for that question.

Exercise — break and diagnose a failure pattern

Turn the trace-reading from Step 3 into a written artifact you can hand off to Module 03:

1. From your winning config's per-question results, pick 2–3 questions that are either correctness = 0 or scored low on llm_judge.

2. For each one, open its LangFuse trace and fill in a row of this table:

   Question	What it generated	Why it failed	Outcome category
   (question text)	(the SQL it produced, in short)	(your read: wrong join, missing date filter, misread phrasing, …)	(sql_exec_error / wrong_result / …)

3. Look across your 2–3 rows for a repeated pattern — the same kind of join, the same date-filter mistake, the same phrasing the model consistently misreads. One pattern, not just a list of unrelated bugs, is what you want here.

Wrap-up

You now know not just that your config won, but how — where it's strong, where it's weak, and what its failures actually look like at the trace level. That detail is exactly what turns into action next.

End state

A detailed quality picture for your winning config. Continue to 03 Improve continuously to turn what you found into new golden data.