01 Select the base model

Facilitator companion to the learner lesson 01 Select the base model.

Timing

~20 minutes with the recommended subset; 45–60 minutes if you run the full grid live.

• 8 min — Langfuse evaluator setup (LLM Connection, correctness code evaluator, llm_judge LLM-as-a-judge evaluator definition, which emits agent-arena-llm-judge). Do this as a live walkthrough on your own projector, not just a link to the README — it is the most fiddly step in the session.
• 8 min — run a two-model, one-prompt subset (talk over the wait; it's a good moment for the "why cost per correct answer" framing). A full grid normally takes 35–45 minutes and should be pre-run or left for self-paced work.
• 4 min — open the Leaderboard, read the winner, name the config_id.

Talk track

• Frame this as the foundational decision of the workshop: which model powers the agent, decided by evidence instead of a public leaderboard someone else ran on a different workload.
• Emphasize where grading happens: inside Langfuse, not inside the harness. The harness orchestrates the grid and records complete Experiment Items; the leaderboard reads them through the Langfuse Public API. This is the same Langfuse project connected in Module 00 — no new tool or second results store is introduced here.
• Clarify dataset cardinality: the repo has 20 YAML questions, while q019 and q020 are few-shot holdouts, so a clean arena-golden experiment dataset has 18 items.
• Name the headline metric explicitly and why it isn't raw accuracy: cost per correct answer — quality per dollar for this specific task. Point out that cost is computed from live OpenRouter pricing, refreshed at the start of each run, not a stale number baked into config.yaml.
• Show the grid vocabulary on screen: six models split proprietary vs open-weight (claude-sonnet-5, gpt-5.6-luna, gemini-flash-lite / deepseek-v4-flash, qwen3.7-flash, glm-4.7-flash) × prompts (P1_zeroshot … P3_dialect), and that a config_id is <model>__<prompt>.
• Live-click from a Leaderboard row into a per-question result, then into the Langfuse trace behind it — this is the drill-down habit Module 02 goes deep on.
• Land on the winner and say its config_id out loud — every module from here on refers back to it.

Common failures

• Langfuse evaluators not configured — the harness waits only until --eval-timeout (default 180s), then exits non-zero and names the missing scores. It waits for exact score agent-arena-llm-judge, emitted by evaluator definition llm_judge. Run python -m scripts.provision_langfuse_evaluators for that OpenRouter-backed judge, fix the correctness evaluator target/filter, then run a small two-model, one-prompt grid with a fresh --run-id; there is deliberately no local results fallback because Langfuse is the evaluation store.
• Placeholder OPENROUTER_API_KEY — every call in the grid fails with a 401. Confirm this in Module 00, but if it slipped through, this is where it becomes visible: an entire run with zero correct answers across every config.
• A model slug drifted off OpenRouter's catalog — config.yaml's model ids (e.g. anthropic/claude-sonnet-5) are pinned to what was live on OpenRouter at authoring time; OpenRouter deprecates/renames slugs. If a config errors out immediately with a "model not found"-style failure, check https://openrouter.ai/api/v1/models for the current slug and compare against config.yaml. The dashboard's /api/models endpoint reflects the live catalog, so a mismatch there is a fast way to spot drift before running the harness.
• ClickHouse not seeded — if Module 00 didn't finish cleanly, the harness's queries against v_* views return empty results or errors; every config comes back with the same outcome. Re-run scripts/arena.sh up.
• Run looks stuck — the grid is models × prompts (6 × 3 = 18 configs by default); it can take several minutes end to end. Use --models/--prompts to shrink it for a live demo (see Reset steps).

Reset steps

• Confirm ClickHouse is seeded: scripts/arena.sh up (idempotent; safe to re-run).
• Re-run a cheap subset instead of the full grid: python -m eval.harness --run-id demo2 --models qwen3.7-flash,gpt-5.6-luna --prompts P1_zeroshot,P3_dialect
• Always give a fresh run a new --run-id (e.g. demo2, demo3) so it shows up as its own row set on the Leaderboard and its own Experiment in Langfuse, rather than merging into a previous run.
• If Langfuse evaluators are the blocker, verify both target Experiments and dataset filter arena-golden, then re-run the cheap subset above with a fresh --run-id.
• If the dashboard itself looks wedged (not the harness), scripts/arena.sh stop then scripts/arena.sh serve restarts just the local servers without touching the seeded data.