03 Release and detect

Starting point

Module 02 is complete. Record the winning config_id selected from the actual Arena run, then export it from the lab root (ClickHouse_Demos/workshops/agent_arena):

source .env
export WINNER_CONFIG_ID="${WINNER_CONFIG_ID:-qwen3.7-flash__P2_fewshot}"

The verified workshop run selected qwen3.7-flash__P2_fewshot; keep your room's winner if it differs. You will use the same model and prompt that you measured—not a special failure-only agent.

Why a passing evaluator can still miss user value

An online evaluator measures only the dimension it was designed to measure. Here, sql-execution-success answers an important operational question: did the agent produce SQL that ClickHouse could execute? It does not know whether the SQL follows the current business definition of active customer.

That creates a realistic monitoring gap:

Operational evaluation catches broken SQL, timeouts, and execution errors. Semantic or user evaluation asks whether an executable answer is useful and aligned with business meaning. Neither replaces the other. A thumbs-down is a prioritization signal, not ground truth; a human will investigate it in Module 04.

Goal

Create one real chat_turn trace where the operational evaluator passes but a user marks the answer down. Record the trace and the two conflicting counts for the human investigation.

Step 1 — Prove the seeded incident is reproducible

Run the preflight from the lab root before starting the demo server:

source .env
export WINNER_CONFIG_ID="${WINNER_CONFIG_ID:-qwen3.7-flash__P2_fewshot}"
.venv/bin/python -m schema.gen_schema_context
.venv/bin/python -m scripts.check_online_eval_scenario \
  --config-id "$WINNER_CONFIG_ID"

The command executes both definitions and asks the selected configuration three paraphrases. It must print different stale_count and current_count values, three classification_N=policy-v1 lines, and:

If a paraphrase returns ok/unknown, the preflight retries only that same paraphrase and configuration once. It does not retry policy-v2 or provider/model/agent failures, and a second ok/unknown remains blocked.

OK: seeded online-evaluation incident is reproducible

If the counts are equal or any classification is not policy-v1, stop: the contrast would not be visible in this data/model run.

The current business definition is this exact SQL:

SELECT uniqExact(customer_id) FROM v_orders
WHERE order_ts >= now() - INTERVAL 30 DAY
AND status NOT IN ('cancelled', 'returned')

The old policy-v1 definition instead counts customers who signed up in the last 90 days. The SQL the model generates in this module is therefore valid relative to the explicit policy-v1 instructions. The point is not that the model is unintelligent; the deployed policy context is stale while the SQL-execution evaluator is too narrow to notice.

Step 2 — Provision the operational evaluator

Provisioning is idempotent, so it is safe to run again:

source .env
.venv/bin/python -m scripts.provision_online_evaluators --operational

Expect the output to name evaluator sql-execution-success and enabled rule agent-arena-sql-execution-online.

Step 3 — Start the seeded stale release

In the first terminal, from the lab root, start the server with the old policy explicitly selected and leave it running:

source .env
AGENT_ARENA_POLICY_VERSION=policy-v1 \
  .venv/bin/uvicorn serving.api:app --port 8100

Do not omit AGENT_ARENA_POLICY_VERSION. The service otherwise defaults to the current policy-v2, which correctly excludes cancelled and returned orders.

Step 4 — Ask and rate through Chat

In another terminal, start the dashboard if it is not already running:

scripts/arena.sh serve

Open http://localhost:5174, select the Chat tab and $WINNER_CONFIG_ID, then ask:

How many active customers do we have?

Read the generated SQL and result. It should follow the seeded 90-day signup definition from policy-v1. Click 👎 on this answer and wait until the UI says feedback sent.

This Chat root trace is the single authoritative incident you will score and hand to Module 04.

Step 5 — Find and verify the Chat trace

In Langfuse, open Tracing and filter for user-thumbs = false. Open the newest root chat_turn whose question is How many active customers do we have?, whose config matches $WINNER_CONFIG_ID, and whose metadata shows policyversion=policy-v1. Copy its trace ID and trace URL, then set the ID locally:

export CHAT_TRACE_ID="<paste the Chat trace ID>"
.venv/bin/python -m scripts.verify_online_scores "$CHAT_TRACE_ID" \
  sql-execution-success=true user-thumbs=false

Verify that user-thumbs is a Boolean false, not a numeric or text score. The serving source calls the metadata field policy_version; the OpenTelemetry adapter sanitizes it to the emitted Langfuse key policyversion.

Step 6 — Reproduce with an unrated curl

The raw API call is a mandatory command-level reproduction and diagnostic. It creates a separate trace, but it is not the feedback incident and must not be rated:

source .env
export WINNER_CONFIG_ID="${WINNER_CONFIG_ID:-qwen3.7-flash__P2_fewshot}"
ASK_BODY=$(.venv/bin/python -c \
  'import json,sys; print(json.dumps({"question": sys.argv[1], "config_id": sys.argv[2]}))' \
  "How many active customers do we have?" "$WINNER_CONFIG_ID")
CURL_RESPONSE=$(curl -fsS http://localhost:8100/ask \
  -H 'content-type: application/json' -d "$ASK_BODY")
printf '%s\n' "$CURL_RESPONSE" | .venv/bin/python -m json.tool
CURL_TRACE_ID=$(printf '%s\n' "$CURL_RESPONSE" | .venv/bin/python -c \
  'import json,sys; data=json.load(sys.stdin); assert data.get("policy_version") == "policy-v1"; assert data.get("outcome") == "ok"; trace_id=data.get("trace_id"); assert isinstance(trace_id, str) and trace_id; print(trace_id)')

The response must have outcome: "ok", a non-empty trace_id, and policy_version: "policy-v1".

Wait for the asynchronous evaluator and verify only its operational score:

.venv/bin/python -m scripts.verify_online_scores "$CURL_TRACE_ID" \
  sql-execution-success=true

Do not call /feedback for CURL_TRACE_ID and do not put it in the worksheet. It is only a reproducible API diagnostic; CHAT_TRACE_ID remains the handoff trace.

Step 7 — Compare against the current policy

Execute the current-policy SQL through the same read-only ClickHouse client as the agent, then compare its single result with the stale result in CURL_RESPONSE:

.venv/bin/python - <<'PY'
from arena.config import load_config
from agents.chclient import ROClickHouseClient

sql = """SELECT uniqExact(customer_id) FROM v_orders
WHERE order_ts >= now() - INTERVAL 30 DAY
AND status NOT IN ('cancelled', 'returned')"""
result = ROClickHouseClient(load_config().clickhouse).query(sql)
print(result.rows[0][0])
PY

This is the exact query you just executed:

SELECT uniqExact(customer_id) FROM v_orders
WHERE order_ts >= now() - INTERVAL 30 DAY
AND status NOT IN ('cancelled', 'returned')

The different count is the user-visible failure. The SQL ran; it answered the wrong business definition. Record that count beside the authoritative Chat trace evidence.

Investigation worksheet

Keep this handoff for Module 04:

How to verify you are done

• The preflight showed different stale/current counts and all three seeded questions classified as policy-v1.
• The service ran with AGENT_ARENA_POLICY_VERSION=policy-v1 and /ask returned outcome: "ok".
• The authoritative Chat trace has sql-execution-success=true and user-thumbs=false in Langfuse.
• The mandatory curl diagnostic returned outcome: "ok", produced a distinct CURL_TRACE_ID, and was not rated or handed off.
• You recorded the authoritative Chat trace ID/URL and both counts without sharing credentials.
• You can explain why a successful SQL execution did not prove semantic correctness.

Continue to Module 04 — Investigate to turn this signal into a human-reviewed diagnosis.