01 Select the base model

Facilitator companion to the learner lesson 01 Select the base model.

Timing

~20 minutes total.

• 8 min — LangFuse evaluator setup (LLM Connection, correctness code evaluator, llm_judge LLM-as-a-judge evaluator). Do this as a live walkthrough on your own projector, not just a link to the README — it is the most fiddly step in the session.
• 8 min — python -m eval.harness --run-id demo running the full grid (talk over the wait; it's a good moment for the "why cost per correct answer" framing).
• 4 min — open the Leaderboard, read the winner, name the config_id.

Talk track

• Frame this as the foundational decision of the workshop: which model powers the agent, decided by evidence instead of a public leaderboard someone else ran on a different workload.
• Emphasize where grading happens: inside LangFuse, not inside the harness. The harness only orchestrates the grid and pulls scores back into ClickHouse's eval_runs table. This is the same LangFuse project connected in Module 00 — no new tool introduced here.
• Name the headline metric explicitly and why it isn't raw accuracy: cost per correct answer — quality per dollar for this specific task. Point out that cost is computed from live OpenRouter pricing, refreshed at the start of each run, not a stale number baked into config.yaml.
• Show the grid vocabulary on screen: six models split proprietary vs open-weight (claude-sonnet-5, gpt-5.6-luna, gemini-flash-lite / deepseek-v4-flash, qwen3.7-flash, glm-4.7-flash) × prompts (P1_zeroshot … P3_dialect), and that a config_id is <model>__<prompt>.
• Live-click from a Leaderboard row into a per-question result, then into the LangFuse trace behind it — this is the drill-down habit Module 02 goes deep on.
• Land on the winner and say its config_id out loud — every module from here on refers back to it.

Common failures

• LangFuse evaluators not configured — the harness will hang waiting on correctness/llm_judge scores (bounded by --eval-timeout, default 180s) and then report unscored items. Fallback for a live room: grade locally instead — python -m eval.harness --limit 2 --grade local --no-judge — a fast smoke test that bypasses LangFuse evaluators entirely.
• Placeholder OPENROUTER_API_KEY — every call in the grid fails with a 401. Confirm this in Module 00, but if it slipped through, this is where it becomes visible: an entire run with zero correct answers across every config.
• A model slug drifted off OpenRouter's catalog — config.yaml's model ids (e.g. anthropic/claude-sonnet-5) are pinned to what was live on OpenRouter at authoring time; OpenRouter deprecates/renames slugs. If a config errors out immediately with a "model not found"-style failure, check https://openrouter.ai/api/v1/models for the current slug and compare against config.yaml. The dashboard's /api/models endpoint reflects the live catalog, so a mismatch there is a fast way to spot drift before running the harness.
• ClickHouse not seeded — if Module 00 didn't finish cleanly, the harness's queries against v_* views return empty results or errors; every config comes back with the same outcome. Re-run scripts/arena.sh up.
• Run looks stuck — the grid is models × prompts (6 × 3 = 18 configs by default); it can take several minutes end to end. Use --models/--prompts to shrink it for a live demo (see Reset steps).

Reset steps

• Confirm ClickHouse is seeded: scripts/arena.sh up (idempotent; safe to re-run).
• Re-run a cheap subset instead of the full grid: python -m eval.harness --run-id demo2 --models qwen3.7-flash,gpt-5.6-luna --prompts P1_zeroshot,P3_dialect
• Always give a fresh run a new --run-id (e.g. demo2, demo3) so it shows up as its own row set on the Leaderboard and its own Experiment in LangFuse, rather than merging into a previous run.
• If LangFuse evaluators are the blocker in the moment, unblock with python -m eval.harness --limit 2 --grade local --no-judge and explain to the room that this is the safety valve, then fix the evaluator setup at the break.
• If the dashboard itself looks wedged (not the harness), scripts/arena.sh stop then scripts/arena.sh serve restarts just the local servers without touching the seeded data.