04 Release to production

Starting point

Module 03 complete: you have a winning config_id from the Arena, a detailed quality picture for it, and an improvement loop (annotation queue, promote_to_golden) ready to receive real feedback.

Why

Every module so far has used the exact same agent core (agents/) and read-only SQL sandbox (agents/sqlguard.py) that the benchmark harness uses — the Arena and the serving API share one code path, so what you measured in Modules 01–02 is what you're about to run live. And once this chatbot is fielding real questions, the question shifts from "which config is best?" to "is it still working, and for whom?" LangFuse is where that question gets answered — no separate observability stack required, because it's the same project, the same traces, that ran the Arena in Module 01. Dev and production aren't two different tools; they're the same LangFuse project at two points in one continuous loop.

Concepts — under the hood

One LangFuse project, two audiences. Nothing about the wiring changes between dev and prod — only the traffic source does:

This is the payoff of wiring LangFuse in during Module 00: the correctness/llm_judge evaluators, the trace-reading skill from Module 02, even the tags-and-filter habits you built reading the Arena's traces — none of it is dev-only tooling you now have to replace with something production-grade. It's the same UI, the same project, pointed at different traffic.

Sessions. A session groups every trace sharing a session_id into one conversation. The Chat tab generates a fresh session_id per page load automatically (web/src/chat/Chat.jsx sets it as a module-level constant evaluated once when the page loads, not persisted — a refresh starts a new session); a direct /ask call generates one per request unless you pass your own. Sessions are the unit for "did this conversation go well" — a user might ask three follow-up questions, get the first one wrong, and still leave satisfied if the correction landed; that story only shows up at the session level, not on any single trace.

The /ask + /feedback contract. POST /ask runs the agent live and returns:

POST /feedback takes that trace_id plus a value (1.0 for 👍, 0.0 for 👎) and calls create_score(name="user_feedback", ...) on it — the exact score Module 03's annotation queue filters and sorts by. A click in the Chat tab and a curl to /feedback do the identical thing.

Goal

A running serving API that answers a real question with SQL, a result set, cost, latency, and a trace_id — plus comfortable navigating LangFuse's Traces, Sessions, and feedback views for that traffic, and knowing what to watch for as a signal something needs attention.

Step 1 — Start the serving API

source .env && uvicorn serving.api:app --port 8100

What you should see. Uvicorn logs Application startup complete on port 8100. If it exits immediately, check .env for CLICKHOUSE_*, OPENROUTER_*, and LANGFUSE_* — the API loads config and connects to ClickHouse at startup, before it will serve a single request.

Step 2 — Ask it a question

curl -s localhost:8100/ask -H 'content-type: application/json' \
  -d '{"question":"How many customers are there?","config_id":"claude-sonnet-5__P1_zeroshot"}'

Swap config_id for whatever the Arena crowned in Module 01 — remember, a config_id is <model>__<prompt> (e.g. claude-sonnet-5__P1_zeroshot, qwen3.7-flash__P3_dialect). The response includes the generated SQL, the result columns and rows, cost_usd, latency_ms, an outcome, and a trace_id you can use to attach feedback (see Module 03).

Step 3 — Or use the web Chat tab

cd web && npm install && npm run dev

Set VITE_SERVING_BASE to point the web UI's Chat tab at your serving API (it defaults to http://localhost:8100, which matches the command above):

VITE_SERVING_BASE=http://localhost:8100 npm run dev

The Chat tab lets you type a question, pick a config_id from a dropdown, and see the SQL/result/cost/latency for each turn — plus 👍/👎 buttons on every answer, which feed directly into Module 03's improvement loop.

Step 4 — Open your LangFuse project

Log in to your LangFuse Cloud project — the same one from Module 00 that ran the Arena in Module 01.

Step 5 — Read a trace

Go to Traces and filter by tag serving — this isolates production chat turns from the Arena's experiment runs sitting in the same project. Each trace is one chat_turn: the question, the generated SQL, the model's llm_call generation (prompt, completion, token counts, cost), timing for every span, and — if the query failed — the error that came back from ClickHouse. Click into any trace to see the full prompt-to-SQL-to-result path for that one question, the same skill you used in Module 02 on the Arena's traces.

How to read it. This is structurally the same trace view from Module 02 — root span plus an llm_call generation — with two differences to notice: the tag is serving instead of a config_id-only tag, and there's no correctness/llm_judge score attached, because no evaluator runs against production traffic (there's no golden answer to grade against). The only score you'll see here is user_feedback, and only on turns a user rated.

Step 6 — Read a session

Go to Sessions to see whole conversations grouped together (every /ask call sharing a session_id — automatically generated per Chat tab session, or one you pass explicitly). This is the view for "did this user's conversation actually go well end to end," not just "was this one turn correct."

Step 7 — Check cost and latency

Both Traces and Sessions surface cost and latency per turn (and LangFuse aggregates them across a time range) — the same two numbers Module 01's leaderboard optimized in aggregate, now visible per real question in production. Watch for configs that were cheap and fast on the Arena's golden questions but drift expensive or slow on the long tail of real user phrasing.

Step 8 — Check user feedback scores

Every 👍/👎 a user clicks in the Chat tab is written back as a user_feedback score (1.0 / 0.0) attached to that trace's id. On the Traces list, sort or filter by the user_feedback score to jump straight to the turns your users flagged — good ones are candidates for promoting into the golden dataset; bad ones are exactly what you want to triage first.

Step 9 — Know what to watch for

• Negative feedback clusters — several 👎 on structurally similar questions usually means a prompt gap (a phrasing, a join, a date filter) that the arena-golden dataset doesn't yet cover.
• Cost or latency drift — a config that looked cheap in Module 01 but creeps up in production usually means real questions are longer or more complex than the Arena's sample, or a provider's pricing changed (worth re-running Module 01 — it always pulls live OpenRouter prices).
• Error outcomes — traces whose outcome isn't a clean success (SQL rejected by the sandbox, a ClickHouse error, an empty result) are worth reading end to end; some are prompt issues, some are genuinely unanswerable questions.

How to verify you are done

• The /ask request returns a 200 with generated SQL, a result set, cost_usd, latency_ms, and a trace_id.
• In LangFuse, a chat_turn trace appears with a child llm_call generation carrying model name, token usage, and cost, grouped under a session id.
• (If you started the web UI) the Chat tab renders the same answer with 👍/👎 buttons.
• You can find your /ask traffic by filtering Traces on tag serving, open one trace and point to its generated SQL, token usage, and cost, and open its corresponding session to see all its turns grouped together.
• You can find (or produce, via a 👍/👎 in the Chat tab) a trace with a user_feedback score.

Exercise — diagnose a live failure

Ask your serving API a question deliberately outside its comfort zone — either genuinely hard (a multi-table margin calculation) or genuinely out of scope (something the schema has no data for) — then diagnose it the same way you diagnosed Module 02's failures, but on real traffic this time:

1. Pick or write a hard question, e.g.:

   curl -s localhost:8100/ask -H 'content-type: application/json' \
     -d '{"question":"What is the average order margin by product category for repeat customers in the last quarter?","config_id":"claude-sonnet-5__P1_zeroshot"}'

   (Or type it into the Chat tab if you started it in Step 3.)

2. Note the outcome and trace_id in the response.

3. Open that trace_id in LangFuse Traces and read it end to end: the question, the generated SQL, and — if it failed — the ClickHouse error or the empty/wrong result.

4. Diagnose it the way you would in Module 02: is this a sql_exec_error (dialect gap), an empty_but_expected/wrong_result (logic gap), or a sql_policy_rejected (prompt-behavior gap)?

5. Give it a 👎 (via /feedback or the Chat tab button) so it's flagged for the Module 03 annotation queue.

Wrap-up

Every turn — whether it comes from curl or the Chat tab — is now traced in the exact same LangFuse project that ran the Arena and measured the winner's quality. Monitoring here isn't a new tool bolted on at the end; it's the same tracing that's been running since Module 00, now pointed at real traffic. And every signal you see here — negative feedback, cost or latency drift, error outcomes — feeds directly into Module 03's review loop.

End state

The winner is live, traced, and monitored, and the loop is now fully closed: production feedback (👎s, error outcomes, cost/latency drift) flows into Module 03's annotation queue, a human turns the ones worth fixing into new golden questions, promote_to_golden grows arena-golden, and the next Arena re-run in Module 01 measures every config against them — including the one currently live. This is the last module in the Learner track. Loop back to Module 03 as real feedback arrives, or revisit 01 Select the base model if your requirements, data, or the model roster change.