03 Improve continuously

Facilitator companion to the learner lesson 03 Improve continuously.

Timing

~20 minutes total.

• 3 min — where feedback comes from (user_feedback scores written by the serving API — nothing to build, just explain the wiring in advance of Module 04).
• 6 min — set up a LangFuse Annotation Queue and triage a couple of sample traces.
• 4 min — write/export reviewed.json.
• 4 min — python -m scripts.promote_to_golden reviewed.json.
• 3 min — re-run the Arena and see the promoted question show up.

Talk track

• Open with the framing: selecting and measuring a model once is not the end of the story, it's a loop. No golden dataset anticipates every way real users phrase a question, so production feedback has to make the next Arena run sharper.
• Close the loop explicitly, forward and backward: real 👍/👎 clicks from Module 04's Chat tab land as user_feedback scores on traces in this same LangFuse project; this module's Annotation Queue triages those traces; approved ones get promoted into arena-golden; Module 01's Arena measures against that updated dataset next time it runs. Say the sentence "this is the same LangFuse project, start to finish" again here — it's the whole thesis of the workshop.
• On the Annotation Queue: this is set up before real traffic exists, precisely so it's ready the moment Module 04 goes live. Configure it now to prioritize low or negative user_feedback scores — those are the turns most likely to reveal a real gap.
• Walk through the reviewer's job concretely: read the question, the generated SQL, the result; for the ones where the right SQL is clear even though the model got it wrong, write the correct golden_sql and approve.
• Show reviewed.json's shape on screen (id, question, golden_sql, tier, ordered) before running promote_to_golden — it's a small, readable file and seeing it demystifies the script.
• After promote_to_golden, re-run python -m eval.harness and point out: if a config that looked strong in Module 01 stumbles on the new question, that's the loop working — production surfaced a gap, review confirmed it, and the Arena now catches it before it ships again.

Common failures

• Annotation queue not set up in advance — if this is the first time anyone's configured one, budget extra time; walk it live rather than skipping ahead with "it's in the README."
• No real traces yet to triage — this module usually runs before Module 04 in a workshop setting, so there's no production traffic. Use the Arena's own Module 01 traces (a wrong or low-llm_judge answer from Module 02's drill-down) as stand-in "production" traces for the live demo of the queue and review flow.
• reviewed.json malformed or missing a field — promote_to_golden.py expects id, question, golden_sql, and reads tier/ordered with defaults (tier=3, ordered=False) if omitted, but a missing golden_sql or id will raise a KeyError. Double check the JSON shape against the example in the learner module before running the script.
• golden_sql doesn't run cleanly against ClickHouse — the script snapshots the golden result set by executing golden_sql right then; a typo or a reference to a raw table instead of a v_* view fails immediately with a ClickHouse error, not silently.
• LangFuse credentials missing/wrong — promote_to_golden.py needs the same LANGFUSE_* env vars as the harness; without source .env first, it errors before it gets to ClickHouse.
• Re-run doesn't seem to include the new question — confirm the promotion actually reported "promoted N question(s)" and that you re-ran python -m eval.harness (or a subset covering the relevant tier) after promoting, not before.

Reset steps

• If ClickHouse itself needs re-seeding first: scripts/arena.sh up.
• Re-run a cheap subset that will pick up the promoted item: python -m eval.harness --run-id demo2 --models qwen3.7-flash,gpt-5.6-luna --prompts P1_zeroshot,P3_dialect (use a fresh --run-id each time you re-promote and re-run, so the Leaderboard rows stay easy to tell apart).
• If promote_to_golden.py failed partway, fix reviewed.json and simply re-run source .env && python -m scripts.promote_to_golden reviewed.json — it's safe to re-run; each item is added to arena-golden independently.
• TODO: confirm whether there is a supported way to remove a wrongly-promoted item from the arena-golden LangFuse dataset for a clean re-demo, or whether the reset is simply "promote a different, correct item and mention the old one in passing" — the repo has no delete-from-dataset script today.