04 Investigate

Starting point

Module 03 produced one authoritative Chat trace for How many active customers do we have? with an intentional disagreement:

Keep that trace ID/URL and the two reference counts from your worksheet. Do not use the unrated curl diagnostic from Module 03.

Why a human investigation is necessary

A thumbs-down tells the team where to look; it does not tell the team what failed. The user could have meant something different, the request could be ambiguous, the generated SQL could be invalid, or two business definitions could differ. Promoting every negative signal directly into a golden dataset would turn guesses into ground truth.

In this module, a reviewer first records what is observable, then tests possible explanations, and only then records a diagnosis and correction. That reviewed decision—not the thumbs-down—is the ground truth handed to Module 05.

Goal

Complete one production-investigation-<session> annotation task for the Module 03 root chat_turn. The completed task must contain an observation, a failure category, the exact corrected SQL, approval for the golden dataset, and production provenance.

Step 1 — Find the exact feedback incident

In Langfuse, open Tracing and filter for the Boolean score user-thumbs = false. Open the trace that matches all of the following:

• name/root observation chat_turn;
• question How many active customers do we have?;
• the winner config_id and trace ID recorded in Module 03;
• metadata policyversion=policy-v1; and
• scores sql-execution-success=true and user-thumbs=false.

The serving source emits policy_version, but the OpenTelemetry adapter removes the underscore, so the Langfuse metadata key is policyversion.

Step 2 — Create the three review score configs

This setup is an intentionally UI-only human-review step. The workshop repository does not contain a command that creates or completes this annotation task for you.

Before creating the queue, open Settings → Scores → Create and create these configs:

Use the names and hyphenation exactly as shown. Creating the configs first is the safest workflow because a queue's set of attached score-config IDs is fixed when the queue is created. If a config was omitted from that attachment set, create a new queue with a fresh suffix. Score configs themselves are mutable: supported name, schema, or category edits must be made as an audited score-config update, and those edits do not rewrite scores that already exist.

Step 3 — Create the queue and target the logical root

Open Annotations → Queues → Create and:

1. Name it production-investigation-<session>, replacing <session> with a short, unique workshop identifier.
2. Attach all three score configs from Step 2.
3. Create the queue.
4. Return to the Module 03 trace, select its root chat_turn observation, open the Annotate dropdown, and select this queue.
5. Open the new task and verify its target is chat_turn, not llm_call.

The queue cannot change which score-config IDs are attached after creation. Recreate it only when that attachment set is wrong; use an audited score-config update for a supported edit to an already attached config.

Step 4 — Open-code what you can observe

Module 03 deliberately disclosed the seeded setup. For this investigation, bracket that prior workshop knowledge and practice the workflow a reviewer would use on an unknown incident: inspect the question, generated SQL, returned count, model/prompt, and both scores before naming a cause. Enter an evidence-only note in observed-issue, for example:

The answer returned a count and its SQL executed. The observed count differs from the
second reference count recorded in Module 03. The generated query uses a 90-day
customer signup window, and the trace metadata reports policy-v1.

This language does not yet claim that the model, SQL engine, user, or policy is at fault. That separation prevents the seeded diagnosis from being smuggled into the review before the evidence is checked.

Step 5 — Inspect the complete trace evidence

Still on the root chat_turn, verify:

• metadata policyversion=policy-v1;
• generated SQL uses v_customers and the 90-day signup_date window;
• the structured result contains the observed trace count from Module 03;
• the operational score is Boolean sql-execution-success=true; and
• the user signal is Boolean user-thumbs=false.

The generated SQL is consistent with the policy-v1 instructions that the release supplied. The successful execution score is also correct within its deliberately narrow scope. At this point, neither fact establishes whether that deployed policy matches the current governed definition.

Step 6 — Test the two policy definitions side by side

From ClickHouse_Demos/workshops/agent_arena, execute both read-only definitions in the same environment:

source .env
.venv/bin/python - <<'PY'
from arena.config import load_config
from agents.chclient import ROClickHouseClient

queries = {
    "policy-v1": """SELECT count() FROM v_customers
WHERE signup_date >= today() - INTERVAL 90 DAY""",
    "policy-v2": """SELECT uniqExact(customer_id) FROM v_orders
WHERE order_ts >= now() - INTERVAL 30 DAY
AND status NOT IN ('cancelled', 'returned')""",
}
client = ROClickHouseClient(load_config().clickhouse)
for version, sql in queries.items():
    result = client.query(sql)
    print(f"{version}: {result.rows[0][0]}")
PY

The two counts must match the worksheet values and differ from each other. You now have enough evidence to diagnose a stale deployed business definition: the trace advertises policy-v1, its SQL follows that policy, and the verified current query implements policy-v2.

Step 7 — Annotate, correct, approve, and complete

Return to the annotation task and record:

Switch Corrected Output to plain-text mode, then enter this exact raw SQL:

SELECT uniqExact(customer_id) FROM v_orders
WHERE order_ts >= now() - INTERVAL 30 DAY
AND status NOT IN ('cancelled', 'returned')

Langfuse records this correction; it does not execute the SQL. Step 6's read-only ClickHouse client must have executed the exact text successfully before approval. If you edit the correction, rerun that text through the same client. Then choose Complete (or Complete + next). A malformed, non-executable, or unverified correction must not be approved as golden ground truth.

Step 8 — Record the Module 05 provenance

Copy these values to your worksheet. Keep IDs private to the workshop project:

source_trace_id, failure_category, and source_policy_version are required for a production-derived golden record. annotation_id is optional in the runtime, but record it when the UI exposes it so the decision remains auditable.

How to verify you are done

• You investigated the single Module 03 Chat trace with user-thumbs=false.
• The annotation target is the root chat_turn, never the child llm_call.
• The queue is named production-investigation-<session> and contains all three correctly typed score configs.
• observed-issue records behavior before diagnosis.
• You ran the stale and current SQL side by side and confirmed different counts.
• The completed task records stale-business-policy, exact corrected SQL, and approved-for-golden=true.
• Your worksheet preserves production provenance for Module 05 without publishing live trace IDs or project URLs.
• You can explain why a thumbs-down prioritizes human review but does not itself become ground truth.

Continue to Module 05 — Close the loop to promote the reviewed correction, compare policy versions, and prevent the same class of failure online.