Instructor track
Timing, talk tracks, common failures, and reset steps for facilitators.
This is the Instructor track: the facilitator's companion to the Learner track. Each module mirrors its learner counterpart and adds timing guidance, talk tracks, common failure modes, and reset steps for running the session live.
The workshop follows one continuous arc: select the base model → measure offline → release and detect → investigate with a human → prove and monitor the improvement.
Total timing
| Module | Learner lesson | Instructor notes | Outcome | Budget |
|---|---|---|---|---|
| 00 | Setup | notes | Connect OpenRouter, ClickHouse, and Langfuse; seed the Agent Arena data. | 20 min |
| 01 | Select the base model | notes | Run the contest and select a configuration by cost per correct answer. | 20 min |
| 02 | Measure offline | notes | Read the winner's offline scores and traces before release. | 15 min |
| 03 | Release and detect | notes | Release the stale policy and capture the evaluator-pass/user-fail disagreement. | 15 min |
| 04 | Investigate | notes | Complete an evidence-first human diagnosis and corrected-output handoff. | 20 min |
| 05 | Close the loop | notes | Expand the golden dataset, prove the fix, and enable calibrated online protection. | 25 min |
That is 115 minutes — about 1 hour 55 minutes — of hands-on work for the full session, plus transitions between modules.
Room setup
- One
.envper learner (or per pair), populated from their own OpenRouter, Langfuse Cloud, and ClickHouse Cloud accounts — credentials are not shared across learners, because Module 00 also provisions a dedicatedarena_roread-only ClickHouse user per seed. - A projector view of your own Langfuse Cloud project, kept separate from learner projects, for experiment, production-trace, annotation, and evaluator walkthroughs in Modules 01–05.
- A terminal with
scripts/arena.sh statusready to run at any point — it reports whether the dashboard API and web UI are up, and prints row counts for eachv_*view.scripts/arena.sh upseeds the ClickHouse business dataset and starts the local services; benchmark results live in Langfuse (with no Aurora, ClickPipes, or ClickStack in this workshop).
Prerequisites checklist
Confirm each of these before Module 00 starts, ideally the day before:
- Learners have created (or been given) accounts on OpenRouter, Langfuse Cloud, and ClickHouse Cloud, with real API keys — not placeholder values.
- Learners have cloned the repo and can create a Python virtualenv
(
python -m venv .venv && source .venv/bin/activate && pip install -r requirements.txt). - Node.js is installed for the web Chat tab (
cd web && npm install), needed starting in Module 00 (the dashboard) and again in Module 03 (the Chat tab). - You have read
eval/langfuse_evaluators/README.md— Module 01's one-time Langfuse evaluator setup is the most fiddly step in the whole session and is worth rehearsing. - You can run
source .env && scripts/arena.sh upend to end yourself once before the session, so you know what "seeded correctly" looks like when a learner's run doesn't match it.