AgentArena — LangFuse Workshop

Welcome to the AgentArena playbook. AgentArena is a head-to-head contest for LLM agents: over the course of this workshop you put a roster of LLMs into a contest for one specific job — natural-language-to-SQL over a ClickHouse-backed e-commerce dataset — and crown a winner using evidence, not a public leaderboard. LangFuse is connected from the very first module, not bolted on at the end: it runs the contest, measures the winner's quality, drives continuous improvement, and carries through into production.

Why this workshop

When you build a serious agent application, one of the first and most consequential decisions is which model to use. Models vary enormously in both capability and price, and the right choice depends on your specific task — not on someone else's public leaderboard. Guess here and you lose in one of two directions: overpay for a frontier model you didn't need, or ship a cheap one that quietly gets your real workload wrong.

The disciplined answer is to run the contest yourself: put a grid of models and prompt strategies through LangFuse experiments against your own golden dataset, and let cost per correct answer — quality per dollar for your use case — pick the winner. That decision is the foundation everything else rests on; there's no point measuring, improving, or releasing an agent built on the wrong model.

This workshop makes that concrete with an NL→SQL chatbot as the use case, and one flow running through all four core modules: select the base model → measure its quality → continuously improve → release to production. LangFuse is the thread that ties every step together — the same project, traces, and datasets from Module 00 all the way to production.

This playbook is dual-track. The Learner track is the lesson you follow in the room. The Instructor track is the facilitator's companion for the same module: timing, talk track, common failures, and reset steps.

Modules