06 AI SRE

Starting point

You are on build-workshop-v1 — no checkout needed. Budget about 15 minutes.

Prerequisites: telemetry flowing into ClickStack (module 05), your coding agent working with MCP support (module 00).

Why

Your agent is already good at writing SQL. Give it your observability data and the right tools, and it can do SRE work: build a dashboard that surfaces the app's health and an alert that fires when something goes wrong. This module wires the agent to ClickStack and puts it to work.

Goal

Your agent, connected to the ClickStack MCP, has saved an SRE dashboard and at least one alert against your telemetry.

Step 1 — Verify the ClickStack connection from Module 00

Do not add another MCP server here. Use the clickstack connection already configured and authorized in Module 00. Ask for a health check that exercises its tools end to end:

Using the clickstack MCP, check the health of the nyc-taxi-backend service over the last 24h — request volume, error rate, and latency — and tell me if anything looks off.

Expected: a grounded summary with request volume, error rate, latency, and last activity. If the tool asks for a service, select the ClickHouse service used by this workshop.

Step 2 — Have the agent build an SRE dashboard

ClickStack can build the dashboard for you from a plain-language description. In ClickStack, open Dashboards in the left sidebar and click Generate with AI. Describe what you want — the generator inspects your telemetry first and builds tiles from what it actually finds (if the data lacks something you asked for, it tells you rather than inventing generic tiles):

explore the app's traces and logs, then build and save an SRE dashboard showing request rate, error rate, and p95 latency per service

Dashboards → Generate with AI: describe the dashboard and let HyperDX inspect the data and build it.

The AI works through your telemetry — sampling logs and traces, checking for error spans, then assembling and verifying each tile — and summarizes what it built, with a link to open the saved dashboard:

The generator grounds every tile in real data and explains what it found before saving.

Open the saved dashboard to see the result: request rate, error rate, and p95 latency per service, alongside total-request/error counts, overall p95, a log-warnings tile, and a top-endpoints table — all reading live from the nyc-taxi-backend service.

The finished SRE dashboard, saved under Dashboards and populated from live telemetry.

Step 3 — Add an alert

Ask the agent to save an alert (via clickstack_save_alert) on the error-rate signal, with a threshold it can justify from the baseline it just observed.

The agent can save the alert, but it cannot create the notification channel the alert delivers to — there is no MCP tool for that. Pre-create a webhook in the HyperDX UI first, then let the agent's saved alert target it:

1. Open one of your dashboard charts — the error-rate tile is a good pick — and click its Alert button (the bell icon on the tile).
2. In the alert's notification settings, choose Generic Webhook as the channel.
3. You need a URL to point it at. For a throwaway endpoint that needs zero setup, open webhook.site in another tab; it hands you a unique URL and shows every request it receives in real time. Copy that URL and paste it in as the webhook URL.
4. Save. The channel now exists, so the agent's saved alert can target it — and when the alert fires you can watch the payload arrive live on your webhook.site page.

Step 4 (optional) — AI Notebooks: investigate inside HyperDX

The coding-agent-plus-MCP path above works from your editor. ClickStack also has an in-product AI investigation surface: AI Notebooks (beta, Managed ClickStack only).

1. A team admin enables it once: HyperDX - Team Settings -> Security Policies -> turn on Generative AI. A Notebooks entry then appears in the left sidebar.
2. Open a new notebook and ask in plain language, for example: "What are the slowest backend operations in the last 30 minutes, and did any of them error?"
3. The agent queries your logs, traces, and metrics and emits interactive tiles — its reasoning, the queries it executed, charts, and summaries. You can branch an investigation from any intermediate tile, and mix in manual search, timeseries, table, or markdown tiles alongside the AI ones.

Same telemetry, two AI surfaces: the MCP path builds durable artifacts (dashboards, alerts) from your editor; Notebooks is for interactive investigation inside the observability product itself. In module 07 you can diagnose the fault with either.

How to verify you are done

• A saved SRE dashboard appears in HyperDX with the health signals populated.
• A saved alert exists with a sensible threshold.
• You can articulate what the dashboard would look like during an incident.
• Optional: a Notebook exists with one plain-language investigation and its tiles.

Wrap-up

You now have an agent that can not only query telemetry but build durable observability artifacts from it. That is exactly what you will lean on when something breaks.

End state

An AI-built SRE dashboard and alert are in place. Continue to 07 Test, fail, and fix with the editor agent you configured here.