05 ClickStack
Instructor notes for module 05 — timing, talk track, common failures, and reset steps.
macOS terminal: Run workshop commands in Terminal using zsh or bash.
Facilitator companion to the learner lesson 05 ClickStack.
Timing
About 15 minutes. Telemetry must accumulate before the module 07 incident lab, so start the collector as early as this module allows and keep it running.
Talk track
- Managed ClickStack stores telemetry in ClickHouse Cloud; HyperDX is the hosted UI.
The local collector only forwards data. Show both an end-to-end request trace and a
fresh
DEBUG ... ClickHouse query okrecord in the Log source. - Have learners start the collector in Step 1 before they launch Managed ClickStack in Step 2. This gives telemetry time to arrive and makes the first hosted-UI view useful.
- Show one end-to-end request trace on the projector.
Common failures
- OTLP host ports 4317/4318 already in use: have them set
OTEL_GRPC_HOST_PORT/OTEL_HTTP_HOST_PORTin.env.workshop(preflight suggests values, e.g. 24317/24318) and, from anywhere inside the cloned repository, runcd "$(git rev-parse --show-toplevel)/workshops/build_workshop/app" && ./preflight.sh; the back end reaches the collector in-network, so the remap is safe. - Collector overlay not started (forgot the second
-f docker-compose.otel.yml), so nothing arrives in HyperDX. OTLP_AUTH_TOKENmismatch, or missingCLICKHOUSE_*in.env.workshop(the collector reusesCLICKHOUSE_HOST/PORT/USER/PASSWORDto reach the Cloud service).- No traffic generated yet; have them click through both dashboards.
- On Docker Desktop, the optional
--profile container-logsscraper cannot mount the host log path (daemon in a VM); this is expected. Backend logs still travel over OTLP. Usedocker compose ... logsonly to diagnose the forwarder if HyperDX is empty. - After the overlay recreates the backend, the app used to 502 (frontend nginx cached the old backend IP). The frontend now re-resolves per request, so it self-heals within ~10s and needs no manual restart. Confirmed fixed in the dry run.
- Dry-run reference: telemetry pipeline verified end to end —
nyc-taxi-backendtraces withclickhouse.querychild spans carryingdb.statement/db.elapsed_ms/db.rows_returned. Confirmed in the 2026-07 clean-room dry run.
Reset steps
- Restart with both compose files:
docker compose --env-file .env.workshop -f docker-compose.workshop.yml -f docker-compose.otel.yml up -d --build. - In the Cloud console, open the service, choose ClickStack, and relaunch the hosted UI if the SSO session expired.