01 Source environment

Facilitator companion to the learner lesson 01 Source environment.

Timing

About 45 minutes total. Step 2 (./setup.sh) is the only unattended stretch — roughly 5-10 minutes, most of it the TABLE(GENERATOR) synthetic data seed — and it is worth talking over rather than watching in silence; it is a good moment for the talk track below. The remaining time (Steps 1 and 3-5) needs the partner at the keyboard: configuring credentials, verifying the producer and Superset came up, starting the dbt refresh loop, and reading through the seven queries.

Talk track

• This module exists to build a source worth migrating: a VARIANT column, a CDC stream, two scheduled tasks, and a BI layer reading on top of all of it — not a toy table. Every one of those becomes a specific migration decision in module 02.
• Walk the Medallion shape once on the diagram: RAW -> STAGING -> ANALYTICS inside NYC_TAXI_DB, and the two objects that keep it moving independently of dbt — TRIPS_CDC_STREAM and the two scheduled tasks.
• Point at the query library (Step 5) even though partners will not translate the queries until module 02 — each of the seven already carries a comment sketching the ClickHouse side, so partners can start pattern-matching now.
• State clearly, and repeat at the end of the module: leave the producer and the dbt refresh loop running. Module 05's cutover measures the exact gap the producer creates between Snowflake and ClickHouse during the migration, and a partner who tidies up by stopping it here silently destroys that demonstration three modules later.

Common failures

• terraform init fails with a provider error. Confirm Terraform >= 1.6 and that the machine has internet access to the Terraform registry.
• snowsql connection refused. Verify SNOWFLAKE_ORG and SNOWFLAKE_ACCOUNT; test directly with snowsql -a ${SNOWFLAKE_ORG}-${SNOWFLAKE_ACCOUNT} -u ${SNOWFLAKE_USER}.
• dbt run fails with relation not found. Run ./setup.sh --skip-seed first — it creates the database structure — and confirm profiles.yml points at NYC_TAXI_DB.
• Superset shows connection refused. Superset needs about 60 seconds after docker-compose up to initialize; check docker logs nyc_taxi_superset if it is still down after that.
• The data seed takes longer than the partner expects. The TABLE(GENERATOR) insert generates 50M rows in roughly 10-12 minutes; the follow-up UPDATE that populates the TRIP_METADATA JSON column on all 50M rows can add another 15-20 minutes on a SMALL warehouse. This is normal Snowflake behavior for a large VARIANT column update, not a stall — say so before a partner starts troubleshooting a script that is working correctly.
• A partner stops the trip producer once this module looks "done." It is not — modules 02 through 05 all depend on it staying live, and module 05's cutover specifically measures the gap it creates. This is the single most damaging tidy-minded mistake in the whole workshop; say it explicitly, more than once.

Reset steps

• Re-provisioning without repaying the ~10-minute data load: ./setup.sh --skip-seed (infrastructure already exists and TRIPS_RAW already has data).
• Iterating on a Terraform or SQL change only: ./setup.sh --skip-seed alone.
• Iterating on dbt models only, skipping Superset entirely: ./setup.sh --skip-seed --skip-superset.
• Testing Terraform changes without re-running dbt: add --skip-dbt.
• After a dbt model schema change, force a full incremental rebuild: --full-refresh (combine with --skip-seed to avoid also repaying the data load).
• Full reset only if the environment is unrecoverable: source .env && ./teardown.sh from workshop_public/snowflake_migration_lab/01-setup-snowflake/, then a plain ./setup.sh re-run. This is destructive and repays the full ~10-12 minute seed — do not reach for it as a first response to a stuck partner.