07 Test, fail, and fix

Facilitator companion to the learner lesson 07 Test, fail, and fix.

Timing

About 20 minutes. This is the incident lab; protect enough runway for the diagnosis to land before module 08 and the wrap-up. The lab runs fault 01 (5-15 min diagnosis). Faults 02 (5-10 min) and 03 (10-20 min, and only with the full roughly 30-million-row dataset) remain available as optional extra incidents for a fast room. Set a hard-stop at rehearsal so the wrap-up is not squeezed.

Talk track

• This is the payoff: use everything built so far to run a real incident.
• Describe the symptom, not the cause, and let the agents converge from telemetry.
• Consider showing several attendees' diagnoses side by side on the projector.
• The lab runs fault 01. If time permits an extra round, add fault 02 (and fault 03 only when the full dataset was pre-seeded); use the shared stash-and-switch reset below between faults.

Answer key

Do not share this section with learners; it lives in the playbook only and is never committed to the app repo. Each fault is one small change on its own branch off build-workshop-v1; the fix is to revert it. Run one fault at a time. The lab's incident is fault 01 (5-15 min, curveball — the backend is innocent). Optional extras for a fast room: fault 02 (5-10 min, warm-up, the trace tells all) and fault 03 only when the full dataset is available (10-20 min, deeper ClickHouse reasoning).

Shared reset for every fault (the stash preserves participant edits and avoids a blocked branch switch):

git stash push --include-untracked -m "module-07-fix"
git switch build-workshop-v1
docker compose --env-file .env.workshop \
  -f docker-compose.workshop.yml -f docker-compose.otel.yml up -d --build

Fault 02 — zone stats 500 (fault/02-zone-stats-500)

Commit message participants see: "backend: align zone stats column names with API params" (touches backend/app/query_builders.py zone_stats_sql only). It groups by pickup_zone_id, a column that does not exist on taxi_trips, instead of pickup_location_id.

• Symptom: the map choropleth is empty (polygons render, but every zone sits in the lightest bucket) and the "Query Nms" caption is missing; bursts of 500s on GET /api/metrics/zone_stats (React Query retries three times). All other cards are fine.
• Where the signal lives: a ClickStack clickhouse.query span with error=True, error.category="query_failed", and db.statement containing pickup_zone_id AS zone_id; the recorded exception is ClickHouse Code 47 UNKNOWN_IDENTIFIER. Matching ERROR log: "ClickHouse query failed (category=query_failed, elapsed_ms=...): ... | sql=...".
• Diagnosis path: find the 500 span -> read db.statement -> UNKNOWN_IDENTIFIER on pickup_zone_id -> DESCRIBE taxi_trips (or compare sibling query builders) -> the real column is pickup_location_id.
• Fix: change pickup_zone_id back to pickup_location_id in zone_stats_sql (or git revert the fault commit); rebuild the backend.
• Reset: the shared reset above.

Fault 03 — slow dashboard (fault/03-slow-dashboard)

Commit message: "backend: window the trend series by bucket in HAVING" (touches timeseries_sql). It moves the time-window predicate from WHERE to a HAVING on the bucket alias, so WHERE becomes 1. ClickHouse can no longer prune by the (car_type, pickup_datetime) primary key and full-scans taxi_trips with two quantileTDigest states, blowing the 5s max_execution_time.

• Symptom: only the "What's happening now?" trend card fails — it spins, then shows "504 Query timed out...". Every sibling card stays fast.
• Where the signal lives: a clickhouse.query span with error.category="timeout", db.elapsed_ms around 5000, and db.statement showing WHERE 1 ... GROUP BY ts HAVING ts >= ...; the exception is Code 159 TIMEOUT_EXCEEDED. The contrast with the fast 200 spans on the other endpoints is the tell.
• Diagnosis path: one slow card among fast siblings -> open the timeout span -> read the SQL -> the time filter is in HAVING and WHERE has no pickup_datetime range -> a primary-key-pruning / predicate-pushdown anti-pattern (siblings put the window in WHERE).
• Fix: restore the time window to WHERE (revert the fault commit); rebuild the backend.
• Reset: the shared reset above.
• Instructor caveat: severity scales with data volume and service size. On the full seeded dataset (~30M rows) on a workshop-tier service (which may be waking from idle) the 5s timeout fires reliably; on the tiny sample seed it is merely slower, not a hard 504. The client send_receive_timeout and the server max_execution_time are both 5s, so a socket-timeout race can occasionally surface as a 500 instead of the clean 504 — a property of the baseline settings, not the fault.

Fault 01 — map not loading (fault/01-map-not-loading)

Commit message: "frontend: serve map geojson from /static asset path" (touches the fetch path in frontend/src/ui/ZoneMap.tsx). It fetches /static/taxi_zones.geojson, which does not exist. Key nuance: nginx's SPA fallback (try_files ... /index.html) returns HTTP 200 with the HTML document rather than a 404, so r.ok passes and r.json() throws a SyntaxError like Unexpected token '<'.

• Symptom: the map card is dead — base tiles render but there are no NYC polygons or choropleth, and a red inline error shows. Everything else is fine.
• Where the signal lives: nothing in backend ClickStack — the asset never reaches FastAPI. The browser console shows the JSON parse error naming /static/taxi_zones.geojson; the Network tab shows that request returning text/html with status 200; the nginx access log shows the fallback.
• Diagnosis path: backend traces are clean -> pivot to the browser console / Network tab -> a .geojson request returning 200 text/html -> recognize the SPA-fallback gotcha -> the file actually lives at the web root, /taxi_zones.geojson.
• Fix: revert the fetch path (two lines) or git revert the fault commit; rebuild the frontend.
• Reset: the shared reset above.
• Teaching point: not every failure surfaces in backend traces, and a 404 can masquerade as a 200 behind an SPA fallback — so read the actual response body and content-type, not just the status code.
• Note (browser SDK): with the HyperDX browser SDK enabled (module 05 overlay), the frontend now surfaces this in ClickStack itself — the ZoneMap parse failure is captured as a console.error span under ServiceName=nyc-taxi-frontend, paired with the resource span for the text/html 200. So the agent can localize it from telemetry without leaving ClickStack; the browser console / Network tab is the fallback, not the only path.

Sample agent response (fault 01, via the ClickStack MCP):

Expected diagnosis: the agent ties the failure to the geojson request returning 200 text/html (SPA fallback), the resulting JSON.parse error captured under nyc-taxi-frontend, and recommends shipping the asset to /static/ or guarding the .json() parse.

Common failures

• Not enough telemetry baseline for the agent to localize; module 05 must have run early.
• Attendees jump to the fix before confirming root cause from evidence.
• Fault symptom not yet visible because the stack was just brought up — or, for fault 03, because too little historical data is loaded. Measured in the dry run: on the default single-month seed (~3.17M rows) the WHERE-vs-HAVING fault is not observable — the full scan runs in ~300-560ms, indistinguishable from the baseline. Seed several months before demoing fault 03, or stick with fault 01 (which reproduces instantly) and treat fault 03 as an at-scale illustration.
• Fault 01 produces no backend trace at all, and the bad request returns 200 text/html via the SPA fallback rather than a 404; attendees may search traces forever. Nudge them to the browser console / Network tab, where the JSON parse error and the text/html response show.
• Forgetting --build after checking out a fault branch, so the old image keeps running.

Reset steps

• The shared stash-and-switch reset restores the complete app without discarding a participant's attempted fix.
• Re-inject a fault by checking out one of fault/01-map-not-loading / fault/02-zone-stats-500 / fault/03-slow-dashboard and rebuilding with the workshop plus otel compose files.
• Per-fault symptoms, signals, and fixes are in the Answer key section above. Keep the answer key in this playbook only; it is never committed to the app repo.