01 Select the base model

Starting point

Module 00 complete: .env sourced, the arena database seeded, LangFuse connected, and the local dashboard reachable at http://localhost:5174 with an empty Leaderboard tab.

Why this is the foundational decision

This is the decision the whole workshop is built around. Before you ship a serious agent, you have to answer a foundational question: which model should power it? Models differ enormously in capability and price, and the best choice depends on your specific task — not on a public leaderboard someone else ran on a different workload. Guessing is costly in both directions: overpay for a frontier model you don't need, or ship a cheap one that quietly gets your real questions wrong.

So instead of guessing, you run a contest: the Arena. A grid of models and prompt strategies all answer the same golden questions, LangFuse grades every answer as an experiment, and the metric that crowns the winner is not raw accuracy but cost per correct answer — quality per dollar for your use case. Concretely, this module answers, with evidence: across a grid of models and prompt strategies, which configuration gets the most correct answers per dollar? "Correct" means execution accuracy — the generated SQL returns the same result set as the golden SQL, not SQL that merely looks plausible. Grading happens inside LangFuse, not inside the harness — LangFuse hosts the evaluators, and the harness just orchestrates runs and pulls the resulting scores back into ClickHouse. Everything after this module (measure, improve, release) assumes you've made this choice on evidence.

Concepts — under the hood

LangFuse's data model for this evaluation. A Dataset (arena-golden) holds the 20 golden questions, each with its expected result set. Every model × prompt configuration you run is one Experiment — a LangFuse Dataset Run — against that same dataset, so every config is graded on the exact same questions. The evaluators you set up in Step 1 (correctness, llm_judge) then attach Scores to each dataset item within that run. One dataset, many experiments, one score per item per experiment — which is what lets the Leaderboard compare configs apples-to-apples.

The prompt strategies are contestants too. The grid isn't just models — it's model × prompt, because how you ask matters as much as who you ask. From config.yaml and agents/prompts.py:

The roster: proprietary vs open-weight. The six contestants split evenly along a second axis that matters just as much as the model name — whether the weights are closed (a vendor API you can only call) or open (a model you could self-host, fine-tune, or keep entirely inside your own data boundary). Open-weight models are often far cheaper per token, while proprietary frontier models may lead on raw capability — but "may" is exactly what this Arena is built to test for your task, not assume. Running both sides through the same golden dataset lets cost-per-correct-answer tell you whether you actually need to pay for the frontier, or whether a cheap open-weight model gets you there for a fraction of the price. The roster below is deliberately low-cost: NL→SQL is a simple enough task that even the priciest contestant here is a mid-tier model, not a frontier one.

Why cost-per-correct, and why execution accuracy. "Correct" is decided by execution accuracy: does running the generated SQL produce the same result set as the golden SQL? That's the honest signal — it doesn't care whether the SQL is byte-for-byte different from the golden query, only whether it answers the question right. The headline ranking metric is then

cost_per_correct_answer = total cost of the run ($) / number of correct answers

which rewards a model that's nearly as accurate but much cheaper over a frontier model that's marginally better but far more expensive — the metric a real cost- conscious team would actually optimize for.

Goal

A populated Leaderboard ranking at least a few model × prompt configurations by cost-per-correct-answer, with every ranking backed by a LangFuse trace you can drill into, and a crowned winner: one config_id.

Step 1 — Set up LangFuse evaluators (one time)

Do this once, in the LangFuse UI, following the repo's eval/langfuse_evaluators/README.md:

1. LLM Connection — Settings → LLM Connections → add an OpenAI-compatible provider named OpenRouter: base URL https://openrouter.ai/api/v1, API key = your OPENROUTER_API_KEY. Pick a cheap judge model that supports structured output (e.g. openai/gpt-5.6-luna).
2. Code evaluator correctness — Evaluators → Set up Evaluator → Code → paste in eval/langfuse_evaluators/correctness_evaluator.py → Target: Experiments → filter dataset = arena-golden. This evaluator compares the agent's result set (from the trace) against the golden result set (the dataset item's expected_output) and emits an execution-accuracy correctness score (0/1) plus an outcome category. It has no network egress — the SQL already ran inside the agent; the evaluator only compares result sets.
3. LLM judge llm_judge — Evaluators → Set up Evaluator → LLM-as-a-judge → Custom → use the system/eval prompts and variable mappings from eval/langfuse_evaluators/llm_judge_prompt.md → Target: Experiments, dataset arena-golden → numeric score llm_judge. This is a secondary signal that rates SQL quality, on top of the primary correctness score — you'll lean on it in Module 02.

Step 2 — Run the contest

source .env && python -m eval.harness --run-id demo

What you should see. The harness first prints a summary line (run_id=demo configs=6x3 ...), then one line per question as it runs, e.g.:

claude-sonnet-5__P1_zeroshot q001 pending 812ms $0.00021

Every row starts pending — the SQL ran and the result set was captured on the trace, but the LangFuse evaluators haven't scored it yet. Once every config has finished running, the harness switches to waiting: grading via LangFuse evaluators — waiting on N traces..., printing a countdown as correctness/llm_judge scores land, and finishes with ✓ pulled evaluator scores for all N traces into eval_runs. That pending → scored handoff is the harness handing grading off to LangFuse and then pulling the verdict back into eval_runs — exactly the split described above.

This runs the full model × prompt grid (every model in config.yaml against every prompt strategy, P1_zeroshot through P3_dialect) as a LangFuse Dataset Run (Experiment) against the arena-golden dataset. --grade langfuse is the default: the harness waits for the correctness and llm_judge evaluators to score every item, then pulls the scores back into ClickHouse's eval_runs table.

A single config is <model>__<prompt>, e.g. claude-sonnet-5__P1_zeroshot. The available names come straight from config.yaml:

• Models: the six contestants in the roster table above — three proprietary (claude-sonnet-5, gpt-5.6-luna, gemini-flash-lite) and three open-weight (deepseek-v4-flash, qwen3.7-flash, glm-4.7-flash)
• Prompts: P1_zeroshot, P2_fewshot, P3_dialect

Useful flags:

• --models qwen3.7-flash,gpt-5.6-luna / --prompts P1_zeroshot,P3_dialect — restrict the grid to a CSV subset instead of running everything.
• --run-id <name> — tag the run so it's easy to find on the Leaderboard and in LangFuse's Experiments view.

Safety valve — no LangFuse evaluators needed

If you want to sanity-check the harness end-to-end before wiring up evaluators (or LangFuse is unavailable), grade locally instead:

python -m eval.harness --limit 2 --grade local --no-judge

--grade local uses the in-process grading in eval/grading.py (and, unless --no-judge is passed, eval/judge.py for the LLM-judge score) instead of waiting on LangFuse evaluators. --limit 2 caps it to two questions per config, so it's fast to run as a smoke test.

Step 3 — Crown the winner

Open http://localhost:5174 → Leaderboard. Every model × prompt config is ranked by cost-per-correct-answer — the headline metric. A cost × accuracy chart and a "best value" ranking sit above the table.

Cost is computed from live OpenRouter pricing — the harness refreshes model prices from OpenRouter's /models endpoint at the start of each run, so cost-per-correct-answer reflects what the model actually costs today, not a stale number baked into config.yaml.

How to read it. Sort order is cost-per-correct-answer ascending — the winner is the top row, not the row with the highest accuracy. Watch the cost × accuracy chart for a cheap model sitting near an expensive one on the accuracy axis: that gap, at a fraction of the cost, is the whole reason this metric exists instead of a plain accuracy leaderboard.

The top row is your winner: note its config_id. Module 02 goes deep on exactly how good it is, not just that it won.

How to verify you are done

• The Leaderboard table shows at least one model × prompt row with a cost-per-correct-answer value (not empty).
• Clicking a config's row shows per-question results, and clicking a question opens its LangFuse trace with the generated SQL visible.
• You can name the config_id (<model>__<prompt>) the Arena crowned as the winner.

Exercise — predict, then verify

Before you open the Leaderboard for real, make a prediction and write it down:

1. Looking only at the model list and the prompt table above (not the Leaderboard), guess which model × prompt config you think will win on cost-per-correct-answer. Write down the config_id and one sentence on why (e.g. "cheapest model paired with the dialect prompt, because most failures are dialect mistakes, not reasoning mistakes").
2. Now open the Leaderboard and check. Were you right?
3. Whatever the result, answer this: did a cheaper model beat (or come close to) a frontier model? If so, that gap — cheap-and-nearly-as-good beating expensive-and-slightly-better — is the point of ranking by cost-per-correct-answer instead of by raw accuracy. If a frontier model won outright, note by how much it beat the next-cheapest config — that margin is what would justify its price in a real deployment decision.

Wrap-up

You now have evidence, not a guess, for which model and prompt strategy is worth running — ranked by cost per correct answer and backed by LangFuse traces for every run. Take note of your winning config_id; you'll use it in every module from here on.

End state

A ranked leaderboard and a crowned config_id. Continue to 02 Measure quality to see how good that winner really is.