Elasticsearch Observability Migration
Migrate a production-shaped Elasticsearch observability stack to ClickHouse Cloud and ClickStack through a measured parallel run.
Migrate logs, traces, and metrics from Elasticsearch, Kibana, Filebeat, and Elastic APM to ClickHouse Cloud, the OpenTelemetry Collector, and HyperDX. You will build both sides, make the architecture decisions, prove parity during a dual-write window, and cut over.
Why this workshop
An observability migration is not a file copy. Elasticsearch data streams, mappings, ILM, ingest pipelines, Kibana dashboards, and alerting each need an explicit ClickHouse design. This lab makes those decisions visible and testable against two live workloads.
What you will build
During the parallel run, two OpenTelemetry Collectors send the same live signals to both backends. Automated checks compare counts and enrichment before the final cutover.
Modules
| Module | Time | Learner | Instructor | Outcome |
|---|---|---|---|---|
| 00 | 15 min | Setup | notes | Tools, Cloud account, and artifacts ready |
| 01 | 30 min | Build the source | notes | Two realistic workloads flowing into Elasticsearch |
| 02 | 60–90 min | Analyze and design | notes | Worksheet, query translation, and migration ADR complete |
| 03 | 2–3 hours | Execute the migration | notes | Parallel run validated and ClickHouse-only cutover complete |
| 04 | 60–90 min | Validate knowledge | notes | Assessment complete and reviewed |
Time and cost
Budget 4–6 hours. The local path needs Docker with at least 16 GB available RAM. The optional EC2 path incurs AWS charges until cleanup. A ClickHouse Cloud free trial is sufficient for the target side; delete the service when the workshop ends.
What this lab does not cover
- Historical reindexing from an existing production Elasticsearch cluster
- Production identity, network, or multi-tenant access-control design
- A complete alert-notification integration
- Long-term capacity planning or formal performance certification