03 Improve continuously

Starting point

Module 02 complete: you have a winning config and a detailed quality picture for it — per-tier accuracy, the llm_judge signal, and trace-level detail on where it succeeds and fails.

Why

Selecting and measuring a model once is not the end of the story — it's a loop, not a pipeline. No golden dataset anticipates every way real users will phrase a question, so production feedback should make the next Arena run sharper. This module builds that mechanism: real questions, once a human has approved them, become new golden questions that Module 01 measures against. The moment you release in Module 04, every 👍/👎 a real user clicks starts feeding straight into this same loop.

Concepts — under the hood

The human-in-the-loop flywheel. Every module up to now has run against a fixed 20-question golden set. That set is a snapshot of what you anticipated users would ask — it will never anticipate everything. The fix isn't a bigger golden set written up front; it's a mechanism that grows the golden set from what actually happens in production, with a human checkpoint in the middle so bad SQL never becomes "ground truth" by accident:

Two pieces are doing work here:

• Annotation queue — a review inbox for traces. LangFuse can route flagged traces (e.g. tagged serving with a low or negative user_feedback score) into a queue a human works through one at a time, without hand-hunting through the full trace list. It exists because some judgments automation can't make: whether a question was even answerable, whether the SQL a model produced is acceptable even if not what a human would have written, what the correct SQL actually is. The queue is where a human makes that call and records it.
• Compounding golden data — every item you promote is permanent: it stays in arena-golden and gets re-measured on every future Arena run, for every model and prompt in the grid, not just the one that surfaced it. Today's production failure becomes tomorrow's regression test. Run this loop for a few weeks and the golden set stops being 20 questions someone guessed at design time and starts being a record of what your users actually ask — which is exactly the distribution you want your benchmark measuring.

Goal

At least one real question, reviewed and approved by a human, promoted into the arena-golden LangFuse dataset — and a benchmark re-run that includes it.

Step 1 — Know where feedback comes from

Once you release in Module 04 and the Chat tab is live, every 👍/👎 a user clicks is written back automatically as a user_feedback score on the corresponding LangFuse trace — nothing to build here, it's wired into the serving API from the start.

Step 2 — Triage with a LangFuse Annotation Queue

In the LangFuse UI, set up an Annotation Queue to route interesting traces to a human reviewer:

1. Annotation Queues → New Queue — name it something like production-review.
2. Configure it to pull in traces tagged serving, prioritizing ones with a low or negative user_feedback score (these are the turns most likely to reveal a real gap — a phrasing the prompt didn't handle, a join the model got wrong, a question outside the schema's coverage). Set this up now so it's ready the moment real traffic starts arriving.
3. A reviewer opens each queued trace, reads the question, the generated SQL, and the result, and — for the ones where the right SQL is clear even if the model didn't produce it — writes the correct golden_sql and approves the item.

What you should see. A configured queue with a filter (tag serving, sorted by user_feedback ascending) and, once traffic exists, a growing worklist of individual traces rather than a raw trace table — each one waiting on a human decision, not a score.

Step 3 — Export approved items

Export the reviewer's approved items as a JSON array shaped like this:

reviewed.json

[
  {
    "id": "prod-001",
    "question": "How many customers placed an order in the last 30 days?",
    "golden_sql": "SELECT count(DISTINCT customer_id) FROM v_orders WHERE order_ts >= now() - INTERVAL 30 DAY",
    "tier": 2,
    "ordered": false
  }
]

• id — a unique id for the new golden question.
• question — the natural-language question, verbatim.
• golden_sql — the reviewer-approved correct SQL against the v_* views.
• tier — difficulty tier (matches the tiers used elsewhere in arena-golden).
• ordered — whether row order matters when grading this question's result set.

Step 4 — Promote to the golden dataset

source .env && python -m scripts.promote_to_golden reviewed.json

For each reviewed item, this snapshots the golden result set by running golden_sql against ClickHouse right now, and adds it to LangFuse's arena-golden dataset with the same shape the benchmark harness already uses — no separate migration step.

What you should see. One prepared <id>: <N> golden row(s) line per item, then a summary line: promoted <N> question(s) into the 'arena-golden' dataset — re-run python -m eval.harness to measure them. In LangFuse, open Datasets → arena-golden and confirm the new item is listed with its question, expected_output, and metadata.golden_sql.

Step 5 — Re-run the Arena

python -m eval.harness

The leaderboard now measures the promoted, real-world questions alongside the original golden set. If a config that looked strong in Module 01 stumbles on these new questions, that's the loop working: production surfaced a gap, review confirmed it, and the Arena now catches it — before it ships again.

How to verify you are done

• reviewed.json contains at least one approved item with a golden_sql.
• python -m scripts.promote_to_golden reviewed.json ran without error and the item appears in the arena-golden dataset in LangFuse.
• Re-running python -m eval.harness produces leaderboard results that include the promoted question.

Exercise — close the loop on one of your own failures

Module 02 asked you to write down 2–3 questions your winning config got wrong, along with why. Now put one of them through the whole flywheel and predict the outcome before you look:

1. Pick one question from your Module 02 table where the outcome was sql_exec_error, empty_but_expected, or wrong_result — not sql_policy_rejected, since that's a prompt-behavior problem, not a missing golden question.

2. Write its correct SQL by hand against the v_* views, and add it to reviewed.json with a fresh id (e.g. prod-fix-01) and an appropriate tier.

3. Promote it:

   source .env && python -m scripts.promote_to_golden reviewed.json

4. Before re-running, predict: will your winning config_id now pass this question? Same model, same prompt, same failure mode it had before — is there any reason promoting the question changes the model's behavior on it?

5. Re-run just your winner against it to check:

   python -m eval.harness --models <winner-model> --prompts <winner-prompt>

6. Verify against the leaderboard: did your prediction hold? And separately — is the arena-golden dataset now one item bigger than it was at the start of Module 02?

Wrap-up

This is the whole point of routing everything through LangFuse: the same traces that Module 04 will watch in production are the traces that feed back into Module 01's Arena, with a human decision in the middle where it matters.

End state

A feedback mechanism ready to close the loop — annotation queue, promotion script, and a re-run that already includes your first promoted question. Continue to 04 Release to production to ship the winner live and start feeding this loop with real traffic.