Data observability tool for the data analytics stack
Build a data observability command center called Pulse — a single-pane incident feed and analytics dashboard over an analytics data stack (dbt/Elementary metadata plus native data-warehouse platform health), with AI-generated root causes, chat alerting, and per-tool cost & reliability deep dives. Triggers A scheduled ingest scan (every ~15 min) that builds all state and drives alerting. A separate daily cron that pre-computes the heavy cost/usage analytics and caches them so page loads never run expensive scans. Space-authenticated web and API routes for the dashboard and its actions (view, resolve/reopen, notify). What it does, at a high level Ingest (cron): fetch live warehouse/dbt metadata and native platform-health telemetry, triage it all into one severity-sorted incident feed (blast radius from the dbt DAG, correlated-signal grouping, platform incidents like credit burn, ingestion backlog, storage anomalies, warehouse contention), enrich each incident with an AI root cause + suggested fix, persist the state to a shared volume, and post new/escalating incidents to a team chat channel (deduped; live-branch only). When a live source can't be read, degrade cards to an honest "unavailable" state rather than faking data. Deep-dive refresh (daily cron): run the heavy warehouse-usage scans (spend by product/user/warehouse, LLM/Cortex usage, dbt models overdue vs their own inferred run cadence, orphan tables with no dbt model) and cache the JSON to a volume. Dashboard (webpage route): a dark command-center UI, client-side routed with an Overview (unified incident feed, source tiles, trend sparklines, expandable incidents with AI root cause and evidence) plus Snowflake and dbt deep-dive pages. Render state inlined for instant paint, then poll a read API for refresh. Read APIs (routes): one serving the incident-feed state, one serving the cached deep-dive analytics. Action APIs (routes): resolve/reopen an incident (append-only history) and push an on-demand chat notification for a single incident. Take the acting user from the spoof-proof authenticated-email header, never from the client. Store all shared state in named volumes (exclusive writers, atomic writes). Keep cost rates and platform thresholds (credit budget, queue/storage/pipe thresholds) as clearly-marked constants. Where a live connector doesn't exist yet, badge that source as simulated. For the data warehouse, use Snowflake (via its SQL API) — or your team's warehouse. For AI root causes, use Anthropic Claude — or another LLM provider. For alerting, post to Slack — or your team's chat tool. Tools used: Snowflake (SQL API v2, ACCOUNT_USAGE + Elementary/dbt metadata), Anthropic Claude (Messages API), Slack (chat.postMessage, chat.getPermalink)
What this prompt builds
Pulse is a single command-center dashboard that unifies reliability signals across the whole data stack — dbt runs, tests, freshness and schema health, native Snowflake platform health (query performance, warehouse cost, ingestion backlog, storage growth), and BI usage — into one incident feed. Every issue is auto-triaged by severity, escalated when its blast radius (how many downstream models and dashboards depend on it) is large, and grouped into a single incident instead of a wall of disconnected alerts. For each incident, Pulse correlates upstream failures, schema changes, deploy history, and warehouse-level signals, then uses Claude to generate a plain-English root-cause explanation and a suggested fix right on the incident card. New or escalating incidents post to Slack, and anyone can acknowledge or re-notify directly from the dashboard, so there's always a visible trail of who's on it and when.
The problem
Data pipelines today run across a handful of disconnected tools — Snowflake, dbt, Fivetran, Omni, and more — and observability is scattered across all of them. A dbt test failure shows up in dbt Cloud, an ingestion delay shows up in Fivetran's own dashboard, warehouse cost and query health live inside Snowflake's UI, and BI usage lives in a fourth tool entirely. When something breaks, whoever's on the hook has to manually stitch together clues from all of these to figure out what actually happened, how far it's spread, and who's affected — often after a stakeholder has already noticed a number looks wrong. We even had raw SQL-level test and freshness monitoring already built on top of Elementary and our warehouse, but nobody could casually check it day to day because it wasn't unified anywhere. The pain lands on the data and analytics engineering team, and by extension anyone downstream who depends on a dashboard or report built on that warehouse.
Solution and impact
Pulse is a single command-center dashboard that unifies reliability signals across the whole data stack into one incident feed: dbt run, test, freshness, and schema health (via Elementary and a custom observability layer already built on top of it), native Snowflake platform health (query performance, warehouse cost burn, Snowpipe ingestion backlog, storage growth, pulled straight from Snowflake's own account-usage views), and Fivetran/Omni tiles that are simulated for now and ready to go live the moment those connectors are provisioned. Every issue gets auto-triaged with a consistent severity, escalated automatically when its blast radius — how many downstream models and dashboards depend on it — is large, and grouped into a single incident instead of a wall of disconnected alerts. For every incident, Pulse automatically correlates upstream failures, schema changes, deploy history, and warehouse-level signals like query queuing or a stalled Snowpipe, then has Claude turn that evidence into a plain-English root-cause explanation and one concrete suggested fix, right on the incident card. It posts new or escalating incidents to Slack, and lets anyone acknowledge an incident or re-notify Slack directly from the dashboard, so there's a visible trail of who's on it and when. The impact: what used to mean manually cross-referencing dbt Cloud, Snowflake's query history, and Slack threads now takes one glance at Pulse — the incident is already found, already explained, and already routed. It's also built entirely on connectors already in our space (Snowflake, Slack, Anthropic), so it went from idea to a working prototype in a single day.