05 Benchmark and cutover

Facilitator companion to the learner lesson 05 Benchmark and cutover.

Timing

About 45 minutes across five steps: adding the ClickHouse dashboards (Step 1, a couple of minutes via the scripted import), running the benchmark (Step 2, seven queries times three runs times two engines — a few minutes, mostly unattended but short enough to just watch), the cutover itself (Step 3, interactive — stop the producer, catch up the delta, refresh dbt, start the ClickHouse producer, each a deliberate step in order), parity verification (Step 4, quick), and teardown (Step 5).

TODO: no per-step timing is given in the lab's own material beyond the 45-minute total for the module — confirm the split during rehearsal, particularly whether Step 3's --resume catch-up pass is reliably fast enough (seconds to minutes, per the lab's own description) to not need any buffer.

Talk track

• This is the module that turns "prepared" into "migrated." Modules 03 and 04 proved the data and the pipeline; this module proves a number (the benchmark) and proves the write path actually moves (the cutover).
• The cutover order is the content here, not a formality: stop the Snowflake producer, run --resume to close the gap, refresh dbt, then start the ClickHouse producer. Each step depends on the one before it — running them out of order is exactly how a silent parity failure happens (see Common failures).
• Say explicitly why --resume is fast here when module 03's original migration was not: it watermarks on max(pickup_at) already in ClickHouse and only pulls the delta, so a gap that has been open since module 01 closes in seconds to minutes, not another 40-50 minute bulk transfer.
• Tie the cutover back to agg_hourly_zone_trips's empty state from module 04: it fills for the first time in the whole lab once the ClickHouse producer starts, because its filter only ever matched live-producer rows. This is the payoff to the question partners have been asking since module 04.
• This is the last module before the written assessment — remind the room to keep migration-plan.md and the benchmark CSV somewhere reachable after Step 5's teardown deletes both cloud environments.

Common failures

• A partner imports the dashboard ZIP manually through the Superset UI instead of running add_clickhouse_connection.sh. The committed export has its ClickHouse host redacted to your-instance.clickhouse.cloud; the script patches the URI from .env before importing, but a manual UI import uses the placeholder host as-is and the connection will not connect. Have them edit the connection afterward to point at the real CLICKHOUSE_HOST and credentials.
• A partner skips the --resume catch-up pass in Step 3 and cuts over anyway. ClickHouse permanently drops whatever rows landed in the gap between module 03's original migration and the moment the producer stopped — a silent parity failure. Step 4's parity check is built to catch this, but only if it is run; a partner who jumps straight to the benchmark write-up without running Step 4 will not notice the dropped rows.
• The Snowflake producer was already stopped, back in module 01 or 02, by a tidy-minded partner. Cutover has nothing to measure a gap against if the producer never ran continuously — this breaks the entire cutover demonstration, not just this step. If it happened, the honest fix is to restart the producer, let it write for a few minutes to create a real gap, then proceed; there is no way to retroactively demonstrate a gap that was never open.
• Parity check fails (difference greater than 0.01%). Run the catch-up pass again and re-check: python scripts/02_migrate_trips.py --resume then bash scripts/01_verify_migration.sh.
• Superset shows 403 Forbidden. The session cookie expired — log out, log back in at http://localhost:8088, then re-run superset/add_clickhouse_connection.sh.
• The benchmark shows N/A for a query, most often Q7. The benchmark script could not connect to ClickHouse — confirm CLICKHOUSE_HOST is set (source .clickhouse_state) and the service is running.

Reset steps

• Parity check fails: re-run python scripts/02_migrate_trips.py --resume then bash scripts/01_verify_migration.sh.
• Cutover needs to be undone (reverse cutover): docker stop nyc_taxi_ch_producer, then bring the Snowflake producer back up from workshop_public/snowflake_migration_lab/01-setup-snowflake/superset with docker-compose --env-file ../.env up -d producer.
• Full environment reset: source .env && ./teardown.sh from workshop_public/snowflake_migration_lab/03-migrate-to-clickhouse/ destroys the ClickHouse Cloud service and the ClickHouse producer container (if cutover happened); Snowflake is not touched by this script — tear it down separately with source .env && ./teardown.sh from workshop_public/snowflake_migration_lab/01-setup-snowflake/.
• Before running either teardown, confirm migration-plan.md and the benchmark CSV (scripts/benchmark_results_<timestamp>.csv) are saved somewhere reachable — both cloud environments are gone after this, and module 06 needs exactly those two files and nothing else.