05 Close the loop

Starting point

Module 04 ended with a completed human annotation for the authoritative Module 03 chat_turn. Keep its corrected SQL and provenance worksheet available:

source=production-feedback
source_trace_id=<authoritative Chat trace ID>
failure_category=stale-business-policy
source_policy_version=policy-v1
annotation_id=<completed task ID when available>

The original production trace has sql-execution-success=true and user-thumbs=false. The thumbs-down found a trace worth reviewing; the completed annotation supplied the diagnosis and corrected ground truth.

The continuous evaluation and improvement loop

This module closes one turn of the loop:

1. user feedback exposes a blind spot in the current online evaluator;
2. a human investigates and approves a correction;
3. that reviewed incident expands the golden dataset;
4. baseline and candidate releases run over the same expanded dataset;
5. a general evaluator is calibrated offline before it is enabled online; and
6. future traffic keeps collecting both evaluator scores and user feedback.

The last step matters: deploying a better evaluator does not end user feedback. An evaluator can only measure dimensions represented in its policy catalog and prompt. A future 👎 can reveal another missing policy, ambiguous request, or failure mode and start the same loop again.

Goal

Pass five evidence gates: promote, baseline, candidate, calibrate, then enable and replay. Use the Module 02 winner for both experiments so policy version is the only intended treatment change.

Run every command below from ClickHouse_Demos/workshops/agent_arena:

cd ClickHouse_Demos/workshops/agent_arena
source .env
export WINNER_MODEL="${WINNER_MODEL:-qwen3.7-flash}"
export WINNER_PROMPT="${WINNER_PROMPT:-P2_fewshot}"
export WINNER_CONFIG_ID="${WINNER_CONFIG_ID:-qwen3.7-flash__P2_fewshot}"

The defaults are the verified workshop winner. If your room selected another config_id, set all three values to that model/prompt instead and keep them unchanged through every gate.

Evidence gate 1 — Promote the reviewed incident

Create reviewed.json in the lab root with the following three records. Replace the two placeholder values everywhere before running promotion. If Langfuse does not expose an annotation task ID, remove annotation_id from all three records instead of leaving a placeholder; that field is optional, while the other production provenance fields are required.

[
  {
    "id": "prod-active-001",
    "question": "How many active customers do we have?",
    "golden_sql": "SELECT uniqExact(customer_id) FROM v_orders WHERE order_ts >= now() - INTERVAL 30 DAY AND status NOT IN ('cancelled', 'returned')",
    "tier": 2,
    "ordered": false,
    "source": "production-feedback",
    "source_trace_id": "<paste the Module 03 Chat trace ID>",
    "failure_category": "stale-business-policy",
    "source_policy_version": "policy-v1",
    "annotation_id": "<paste the completed task ID>"
  },
  {
    "id": "prod-active-002",
    "question": "What is our active customer count right now?",
    "golden_sql": "SELECT uniqExact(customer_id) FROM v_orders WHERE order_ts >= now() - INTERVAL 30 DAY AND status NOT IN ('cancelled', 'returned')",
    "tier": 2,
    "ordered": false,
    "source": "production-feedback",
    "source_trace_id": "<paste the Module 03 Chat trace ID>",
    "failure_category": "stale-business-policy",
    "source_policy_version": "policy-v1",
    "annotation_id": "<paste the completed task ID>"
  },
  {
    "id": "prod-active-003",
    "question": "How many customers qualify as active under our business definition?",
    "golden_sql": "SELECT uniqExact(customer_id) FROM v_orders WHERE order_ts >= now() - INTERVAL 30 DAY AND status NOT IN ('cancelled', 'returned')",
    "tier": 2,
    "ordered": false,
    "source": "production-feedback",
    "source_trace_id": "<paste the Module 03 Chat trace ID>",
    "failure_category": "stale-business-policy",
    "source_policy_version": "policy-v1",
    "annotation_id": "<paste the completed task ID>"
  }
]

Only prod-active-001 is the exact question from the user-feedback trace. prod-active-002 and prod-active-003 are reviewer-authored paraphrases derived from that same investigated incident. They use the same source trace and completed annotation for auditability; they are not two additional production feedback traces. The three inputs intentionally all invoke the governed active-customer metric, so the baseline cannot appear healthy by testing unrelated counts.

Promote the reviewed batch:

source .env
.venv/bin/python -m scripts.promote_to_golden reviewed.json

Expect three prepared prod-active-* lines followed by:

promoted 3 question(s) into the 'arena-golden' dataset

Open Langfuse → Datasets → arena-golden and inspect each new item's metadata. Verify source=production-feedback, the same real source_trace_id, failure_category=stale-business-policy, and source_policy_version=policy-v1.

The repo source corpus has 20 YAML questions. q019 and q020 are few-shot prompt holdouts, so a clean project starts with 18 arena-golden Experiment items. This three-item promotion makes that clean dataset 21 items. A reused project may contain additional approved items; record that provenance instead of deleting them to force a count, and require the baseline and candidate to use identical item IDs.

Promotion validates the complete batch, read-only SQL, and required provenance before querying ClickHouse or writing dataset items. It then reads existing dataset metadata and refuses a colliding ID with different production provenance. If that authenticated preflight cannot establish provenance safely, it stops without a write. Repeating a genuine promotion is safe only when the colliding production provenance is identical.

Evidence gate 2 — Run the policy-v1 baseline

First provision the catalog-driven judge for experiments. This creates its online observation rule in a disabled state:

source .env
.venv/bin/python -m scripts.provision_online_evaluators \
  --business-policy-experiments

Expect experiment rule enabled=True; online rule enabled=False. Confirm the online rule is still disabled in Langfuse before continuing.

Give this workshop attempt a unique suffix, then run the selected model and prompt over the expanded dataset with the stale policy:

export LOOP_RUN_SUFFIX="${LOOP_RUN_SUFFIX:-$(date +%Y%m%d-%H%M%S)}"
export BASELINE_RUN_ID="online-loop-baseline-${LOOP_RUN_SUFFIX}"
export CANDIDATE_RUN_ID="online-loop-candidate-${LOOP_RUN_SUFFIX}"

.venv/bin/python -m eval.harness --run-id "$BASELINE_RUN_ID" \
  --policy-version policy-v1 --models "$WINNER_MODEL" --prompts "$WINNER_PROMPT" \
  --wait-for-score business-policy-adherence

The harness appends --policy-v1 to the release ID. It waits for three exact Experiment score names on every trace: correctness, agent-arena-llm-judge, and business-policy-adherence. Do not proceed if the run times out or any score is missing.

In Langfuse Experiments, record the baseline's dataset item count and aggregate correctness. Expect 21 items in a clean project after promotion. Reused projects can have more approved items, and provider responses can vary, so the release gate is the paired comparison below rather than a hard-coded aggregate score.

Evidence gate 3 — Run the policy-v2 candidate

Without changing the dataset, model, prompt, or run suffix, run the candidate:

.venv/bin/python -m eval.harness --run-id "$CANDIDATE_RUN_ID" \
  --policy-version policy-v2 --models "$WINNER_MODEL" --prompts "$WINNER_PROMPT" \
  --wait-for-score business-policy-adherence

Record the candidate's actual aggregate, then compare the two runs in Langfuse and require:

• identical dataset item IDs and item counts;
• all three prod-active-* items move from correctness=0 under policy-v1 to correctness=1 under policy-v2;
• every item that predates prod-active-* is compared per item, with no correctness=1 to correctness=0 regression; and
• aggregate candidate correctness is not lower than baseline correctness.

Stop if a pre-existing item regresses. A candidate that fixes the incident by breaking known behavior has not passed the release gate.

Evidence gate 4 — Calibrate one general policy judge

business-policy-adherence is not an “active-customer evaluator.” It receives the question, generated SQL, and the complete policy-v2 metric catalog. It determines which governed metric applies and returns PASS, FAIL, or NOT_APPLICABLE. The same design can check active customers, revenue, conversion, and gross margin without creating one evaluator per question phrasing.

Before enabling it for production observations, inspect these Experiment items:

Repeat the active-customer check for prod-active-002 and prod-active-003. Read the judge reasoning as well as the category: it should name the applicable catalog policy and evaluate the generated SQL against it. A plain count must remain NOT_APPLICABLE, demonstrating that the judge does not force every count question into the active-customer policy.

Keep the online rule disabled if any category is wrong, any required score is missing, the structured output is malformed, or the correctness comparison regressed. Offline experiment calibration comes first because it lets you inspect false passes and false failures against known examples before the evaluator affects production monitoring.

Evidence gate 5 — Enable and replay on policy-v2

Only after all calibration gates pass, enable the observation rule:

source .env
.venv/bin/python -m scripts.provision_online_evaluators \
  --enable-business-policy-online

Expect the exact rule name agent-arena-business-policy-online with enabled=True. The command fails closed when it cannot find a dataset-scoped Experiment score named business-policy-adherence; your manual calibration checks above remain the quality gate.

Stop the policy-v1 server. In the first terminal, start the candidate and leave it running:

source .env
AGENT_ARENA_POLICY_VERSION=policy-v2 \
  .venv/bin/uvicorn serving.api:app --port 8100

In the second terminal, define a helper that accepts a question and returns its trace ID only after confirming a successful policy-v2 response:

source .env
export WINNER_CONFIG_ID="${WINNER_CONFIG_ID:-qwen3.7-flash__P2_fewshot}"
ask_trace() {
  local question="$1"
  local body
  body=$(.venv/bin/python -c \
    'import json,sys; print(json.dumps({"question": sys.argv[1], "config_id": sys.argv[2]}))' \
    "$question" "$WINNER_CONFIG_ID")
  curl -fsS http://localhost:8100/ask \
    -H 'content-type: application/json' -d "$body" | \
    .venv/bin/python -c \
    'import json,sys; data=json.load(sys.stdin); assert data["policy_version"] == "policy-v2" and data["outcome"] == "ok"; print(data["trace_id"])'
}

Ask the active-customer and revenue questions once. Online observation scores use the rule name agent-arena-business-policy-online, not the Experiment score name:

ACTIVE_TRACE=$(ask_trace "How many active customers do we have?")
.venv/bin/python -m scripts.verify_online_scores "$ACTIVE_TRACE" \
  sql-execution-success=true agent-arena-business-policy-online=PASS

REVENUE_TRACE=$(ask_trace "What was revenue in the last 30 days?")
.venv/bin/python -m scripts.verify_online_scores "$REVENUE_TRACE" \
  sql-execution-success=true agent-arena-business-policy-online=PASS

The conversion question has a verified stochastic boundary. Ask it once and preserve that trace. If the serving outcome is non-ok, or its exact required scores are missing or fail, retry the same question and config at most once. This block keeps both attempts visible:

ask_conversion() {
  local body
  body=$(.venv/bin/python -c \
    'import json,sys; print(json.dumps({"question": sys.argv[1], "config_id": sys.argv[2]}))' \
    "What is our view-to-purchase conversion rate for the last 7 days?" \
    "$WINNER_CONFIG_ID")
  curl -fsS http://localhost:8100/ask \
    -H 'content-type: application/json' -d "$body" | \
    .venv/bin/python -c \
    'import json,sys; data=json.load(sys.stdin); assert data["policy_version"] == "policy-v2"; print("\t".join((data["trace_id"], data["outcome"])))'
}

IFS=$'\t' read -r CONVERSION_TRACE_1 CONVERSION_OUTCOME_1 <<< \
  "$(ask_conversion)"
if .venv/bin/python -m scripts.verify_online_scores "$CONVERSION_TRACE_1" \
  sql-execution-success=true agent-arena-business-policy-online=PASS; then
  CONVERSION_SCORES_1=pass
else
  CONVERSION_SCORES_1=fail
fi
if [ "$CONVERSION_OUTCOME_1" = ok ] && [ "$CONVERSION_SCORES_1" = pass ]; then
  CONVERSION_RESULT_1=pass
else
  CONVERSION_RESULT_1=fail
fi
printf 'conversion_attempt=1 trace_id=%s outcome=%s exact_scores=%s result=%s\n' \
  "$CONVERSION_TRACE_1" "$CONVERSION_OUTCOME_1" \
  "$CONVERSION_SCORES_1" "$CONVERSION_RESULT_1"

CONVERSION_TRACE_2=not-run
CONVERSION_OUTCOME_2=not-run
CONVERSION_SCORES_2=not-run
CONVERSION_RESULT_2=not-run
if [ "$CONVERSION_RESULT_1" != pass ]; then
  IFS=$'\t' read -r CONVERSION_TRACE_2 CONVERSION_OUTCOME_2 <<< \
    "$(ask_conversion)"
  if .venv/bin/python -m scripts.verify_online_scores "$CONVERSION_TRACE_2" \
    sql-execution-success=true agent-arena-business-policy-online=PASS; then
    CONVERSION_SCORES_2=pass
  else
    CONVERSION_SCORES_2=fail
  fi
  if [ "$CONVERSION_OUTCOME_2" = ok ] && [ "$CONVERSION_SCORES_2" = pass ]; then
    CONVERSION_RESULT_2=pass
  else
    CONVERSION_RESULT_2=fail
  fi
fi
printf 'conversion_attempt=2 trace_id=%s outcome=%s exact_scores=%s result=%s\n' \
  "$CONVERSION_TRACE_2" "$CONVERSION_OUTCOME_2" \
  "$CONVERSION_SCORES_2" "$CONVERSION_RESULT_2"

if [ "$CONVERSION_RESULT_1" != pass ] && \
   [ "$CONVERSION_RESULT_2" != pass ]; then
  printf '%s\n' \
    'STOP: conversion failed twice; preserve both traces and investigate.' >&2
  false
fi

Do not retry until green. If both attempts fail, retain both traces, keep the result visible, and route the new evidence through human annotation, golden-data improvement, and the same paired calibration loop.

Only after conversion passes, ask the plain-count question once:

PRODUCT_TRACE=$(ask_trace "How many products are there?")
.venv/bin/python -m scripts.verify_online_scores "$PRODUCT_TRACE" \
  sql-execution-success=true agent-arena-business-policy-online=NOT_APPLICABLE

The online evaluator runs asynchronously. The verifier polls for up to 180 seconds by default; a score that is still pending is not the same as a failing score.

Keep the loop running

Leave 👍/👎 enabled after rollout. Monitor disagreements such as agent-arena-business-policy-online=PASS beside user-thumbs=false: they are high-value candidates for the next annotation queue. Human review decides whether to correct policy, prompts, data, or the evaluator. Approved cases return to arena-golden, then the next candidate repeats the same baseline → candidate → calibration → guarded enablement sequence.

The workshop rules sample 100% of eligible traces so every learner sees evidence. That is a teaching setting, not a production default. Real sampling should reflect traffic, evaluator cost, latency, risk, and the incident coverage you need.

Completion evidence

• The production root trace still shows sql-execution-success=true and Boolean user-thumbs=false.
• The production-investigation-<session> human-annotation task is completed with a verified correction and approved-for-golden=true.
• All three golden items exist with genuine production-feedback provenance; you can distinguish the one user question from the two reviewer-authored paraphrases.
• Baseline and candidate used the same expanded dataset, model, and prompt; the candidate fixed all three promoted items and introduced no existing correctness regression.
• Experiment calibration produced FAIL, PASS, and NOT_APPLICABLE with exact score name business-policy-adherence.
• The observation rule was disabled during calibration, then enabled only after the gates passed.
• Active-customer, revenue, and conversion traces have agent-arena-business-policy-online=PASS; the plain product count has agent-arena-business-policy-online=NOT_APPLICABLE.
• You can explain why online evaluation and user feedback continue to improve each other after deployment.