BigQuery MigrationClickHouse Workshops

Run of show

Beat-by-beat timings matching the story flow's spec, what to say at each one, and what to cut when the clock runs out.

Read this alongside the learner track, not instead of it -- the "what to say" column below is a prompt, not a script, and it only makes sense next to the module it points at.

The beats

Nine rows: pre-work, then eight timed beats. The eight timed beats sum to 170 minutes, plus whatever breaks the room needs -- 10 + 15 + 25 + 25 + 30 + 25 + 15 + 25 = 170.

BeatTitleModuleTimePurpose
--Pre-work00-setupbeforeaccount, service
0Why is this slow?01-why-is-this-slow10mestablish pain in their language
1You're already querying it02-query-in-place15mearly win; covers late signups
2The lazy migration03-the-lazy-migration25mdeliberate anticlimax
3Shrinking the table04-challenge-compression25mfirst genuine "oh", walked through step by step
4The sort key05-challenge-sort-key30mintellectual core
5The impossible dashboard06-challenge-dashboard25mbuying trigger; no BigQuery equivalent
6Ask your data a question07-ask-your-data15mpayoff, not a magic trick
7Your real migration08-your-real-migration25mcredibility beat (cited benchmark), then convert

What to say

Pre-work. In the session's first ten minutes, ask each participant to name their own slowest query -- the one a customer or colleague already waits on. Write handles and one-line descriptions somewhere you can read back from at beat 7. This is the room's own pain, not a benchmark you ran for them, and it lands harder at the end than anything on a slide.

Beat 0 -- Why is this slow? State the puzzle flatly and stop: BigQuery pruned its scan down to 4.46 MB and still took 0.98 seconds; the identical question against ClickHouse read every row with no pruning at all and answered in 52 milliseconds. Do not explain the gap yet. Module 01 holds both numbers open on purpose, all the way to the last module -- if you resolve the puzzle here, beat 7 has nothing left to pay off.

Beat 1 -- You're already querying it. This module does not depend on beat 0, by design -- say so out loud, because it is what covers a late signup or a slow account-creation flow without them losing the thread. gcs() reads the bucket directly, no ingestion, and the count comes back 4,295,584. Frame it as "this already works, and it is still not what you want behind a dashboard" -- the next beat is why.

Beat 2 -- The lazy migration. Deliver this one flatly, not as a twist. The naive, BigQuery-inferred schema loads, allow_nullable_key = 1 is required and not a trick, the funnel query returns the right numbers, and that is the whole module. Do not oversell the anticlimax and do not apologize for it either -- undersell it. This module's entire job is signalling that this is not a sales demo, and that credibility is exactly what beat 7 spends later. If you make beat 2 exciting, you spend the credibility beat 7 needs before it arrives.

Beat 3 -- Shrinking the table. Same rows the naive table already holds, five concrete changes run one at a time -- type sizing, LowCardinality, codecs, sort order, then de-duplicating event_params -- each with the compressed-byte count that shows whether it helped. This is a walkthrough, not a challenge: everyone runs the same DDL and lands on the same final table, bq.events_tuned, which beats 4 and 5 both build on. Let the sort-key step land as a real trade-off, not a mistake to fix -- the table gets larger there on purpose, and that surprise is worth sitting in for a moment before moving on.

Beat 4 -- The sort key. The longest beat and the one to protect on the clock. Open by demonstrating the measurement discipline yourself before anyone times their own query: copy a query_id, read read_rows off system.query_log, and show why the console's own clock cannot answer this (network round trip alone is often larger than the entire gap being measured). Budget floor-helper attention here over any other beat.

Beat 5 -- The impossible dashboard. Open with the sentence that frames the whole beat: no sort key rescues a query that groups every row in the window, because there is no single equality predicate to lead on. The lever is a materialized view on AggregatingMergeTree, and the single most common failure is an empty result because the view never saw the rows already sitting in the table before it existed -- flag the backfill before anyone builds, not after they ask why their dashboard is empty.

Beat 6 -- Ask your data a question. Let the room run the same three prompts against both their own tuned schema and against bq.events_naive, and let the difference speak for itself. If the console agent is not enabled on a given trial service, run it live yourself against your own service and put the generated SQL on screen -- this module degrades to instructor-led without losing its point.

Beat 7 -- Your real migration. Open with the module's own cost-performance benchmark, not this workshop's dataset: concede that ClickHouse Cloud and BigQuery Enterprise are close at 1 billion rows, then show the gap widen through 10 billion and 100 billion. This is the credibility beat the old cost module used to carry, made with someone else's published, real-billing numbers rather than this workshop's own sample, which is deliberately too small to make that case honestly. Then close by reading back the slowest queries you captured in the first ten minutes. The exit offer -- send us your schema and volumes -- is the actual conversion moment, and it lands because it is now pointed at a query someone in the room named themselves, not at this workshop's dataset.

When you are behind

The recommendation, if the clock forces a cut, is to cut beat 4 (the sort key, 30m) before cutting beat 5 (the impossible dashboard, 25m). Two reasons: the workshop's objective is pipeline, and beat 5 is the beat with no BigQuery equivalent at all -- it is the capability argument, not a tuning win, and it is what a prospect remembers walking out. Beat 4's lesson is not lost entirely if it is cut, either -- beat 3's walkthrough already put sort order on screen, including the step where changing it made the table larger, so the room has felt a piece of what a sort key trades off before beat 4 would have made it explicit. Cutting the larger 30-minute block also recovers more clock than cutting beat 5 would.

This inverts the source one-pager's own stated rule, which says to cut beat 5 before beat 4 and separately calls the impossible dashboard the highest-priority module because it has no BigQuery answer. As written, that combination sacrifices the buying trigger first and keeps the beat that BigQuery can, in principle, be tuned to answer differently. This page does not silently pick a side: the recommendation above is a reasoned inversion of the source document, not a correction of an error in it.

There is a real opposing reading, and it deserves a paragraph rather than a footnote: nobody fully appreciates what an incrementally-maintained materialized view buys them without having just spent thirty minutes feeling what a sort key can and cannot do -- beat 5's "no sort key rescues this" line lands only if the room has internalized what a sort key does rescue, which is beat 4's entire content. Cut beat 4 and beat 5 opens on an assertion the room has to take on faith instead of one they just proved to themselves. Both readings are defensible. This is a delivery decision you make with the room in front of you -- read the pace and the energy at the 75-minute mark, not this page, when the moment to cut actually arrives.

Before beat 0: capture the BigQuery screenshots

Module 01 does not ask learners to run BigQuery queries themselves -- querying BigQuery, even against a public dataset, bills to whichever GCP project runs it, and this workshop promises learners no Google Cloud project of their own (see 00-setup and the workshop's own "What you need"). Instead, the module shows two screenshots of a live BigQuery console run that you capture once, ahead of the session, the same time you build the backup service below.

  1. Open the BigQuery console. Go to console.cloud.google.com/bigquery, signed in with any Google account. Select or create a GCP project -- any project works, since bigquery-public-data.ga4_obfuscated_sample_ecommerce is public and the query cost is billed to whichever project runs it, not to the dataset's owner. If the project has no billing account attached, BigQuery will prompt you to add one; on-demand queries are free up to 1 TB scanned per month, and both queries below scan under a quarter of a gigabyte combined, so this costs nothing in practice.
  2. Run the single-user lookup. Paste module 01's first query (the one filtering on user_pseudo_id = '3272961.4196485002') into the query editor and run it. Wait for the 20-row result.
  3. Screenshot the query and the "Execution details" tab together. Below the results grid, click the Execution details tab -- it reports elapsed time, slot time consumed, bytes shuffled, bytes spilled to disk, and, per stage, records read and written. Capture a screenshot showing the query text and this tab. Crop out the left-hand Explorer sidebar and the top project-switcher bar before saving -- both show your GCP project name, and the sidebar repeats it a second time in its resource tree. Save as bq-single-user-lookup.png.
  4. Repeat for the one-day dashboard aggregate (module 01's second query, the one filtering on _TABLE_SUFFIX = '20201201'). Save as bq-dashboard-aggregate.png.
  5. Drop both files into site/public/screenshots/bigquery-migration/, cropped to remove the sidebar/project-switcher, resized to about 1280px wide, and palette-quantized to stay under the site's 200 KB screenshot budget -- then wire them into 01-why-is-this-slow.mdx (and its translations) as plain ![alt](path) embeds. See that directory's README.md for the exact filenames and what each should show.

The module's quoted numbers (3.34s / 229,493,155 bytes / $0.0013045, and 0.98s / 4,455,256 bytes / $0.0000253) are the median of three runs recorded in MEASUREMENTS.md. Your own capture will not match those to the decimal -- BigQuery's elapsed time varies run to run -- but it should land in the same order of magnitude. If a single run looks like an outlier, run it twice more and screenshot the middle one, the same way the recorded figures were produced.

What to have open before you start

A backup ClickHouse Cloud service, in a browser tab you are not sharing your screen from yet, with all three tables already built and loaded -- the naive table, bq.events_tuned from beat 3's walkthrough, and the funnel materialized view with its backfill already run. If your live demo service has a problem mid-session (a stalled ClickPipe, a fat-fingered DROP TABLE, a network blip), switch to the backup and keep going without losing the beat you are on. Building it live, under time pressure, in front of the room is the failure mode this is meant to prevent.

ในหน้านี้

TH