dbt-sentinel 0.6.1__tar.gz

dbt_sentinel-0.6.1/PKG-INFO

@@ -0,0 +1,195 @@
+Metadata-Version: 2.4
+Name: dbt-sentinel
+Version: 0.6.1
+Summary: AI-grounded root-cause analysis for failing dbt tests
+Keywords: dbt,data-quality,observability,analytics-engineering,llm
+Author: Qamar Raza
+Author-email: Qamar Raza <raza.qamar@gmail.com>
+License-Expression: MIT
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Database
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Dist: click==8.1.7
+Requires-Dist: duckdb==1.1.3
+Requires-Dist: httpx==0.28.1
+Requires-Dist: python-dotenv>=1.2.2
+Requires-Dist: rich==13.9.4
+Requires-Dist: google-cloud-bigquery>=3.43.0 ; extra == 'bq'
+Requires-Dist: streamlit>=1.40 ; extra == 'ui'
+Requires-Python: >=3.12
+Project-URL: Homepage, https://github.com/qraza/dbt-sentinel
+Project-URL: Repository, https://github.com/qraza/dbt-sentinel
+Project-URL: Issues, https://github.com/qraza/dbt-sentinel/issues
+Provides-Extra: bq
+Provides-Extra: ui
+Description-Content-Type: text/markdown
+
+# dbt-sentinel
+
+**dbt tells you a test failed. It doesn't tell you why.** `dbt-sentinel` reads dbt's
+run artifacts after a build, pulls the compiled SQL and a sample of the rows that
+actually failed, and asks an LLM — grounded strictly in that evidence — for a root-cause
+explanation and a concrete fix. It flags low-confidence answers instead of bluffing.
+
+[![CI](https://github.com/qraza/dbt-sentinel/actions/workflows/ci.yml/badge.svg)](https://github.com/qraza/dbt-sentinel/actions/workflows/ci.yml)
+![Python 3.12](https://img.shields.io/badge/python-3.12-blue)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+
+![sentinel analyze demo](docs/demo.gif)
+
+Runs in CI too — on every pull request it posts its grounded diagnosis as a comment:
+
+![PR comment](docs/pr-comment.png)
+
+## What this is
+
+A data-quality companion for dbt. When `dbt build` reports a failing test, you normally
+get a name and a row count — then you go digging. `dbt-sentinel` closes that gap: it joins
+`run_results.json` and `manifest.json` to find each failure, runs the test's own compiled
+SQL against the warehouse to sample the offending rows, and sends that concrete evidence to
+an LLM with a prompt engineered to stay grounded in it. The output is a structured verdict —
+root cause, suggested fix, confidence, and the evidence behind it — in the terminal and,
+optionally, markdown for a PR comment. It reads what dbt already produced; it never re-runs
+your tests or mutates data (the warehouse is opened read-only).
+
+It also tracks failures across runs — each one is flagged **new**, **recurring**, or
+**regressed**, and `sentinel history <test-id>` prints a test's full timeline, so you can
+see whether something just broke or has been broken for weeks.
+
+## Quickstart
+
+```bash
+uv sync
+export ANTHROPIC_API_KEY="sk-ant-..."
+export ANTHROPIC_MODEL="claude-opus-5"
+
+uv run sentinel analyze \
+  --target-dir path/to/dbt/target \
+  --db path/to/warehouse.duckdb \
+  --markdown report.md
+```
+
+If all tests pass, it says so and exits. If not, you get a diagnosis per failure.
+
+## How it works
+```
+dbt build ─► run_results.json (which tests failed, row counts)
+manifest.json (compiled SQL, guarded model/column, test type)
+│
+▼
+parse.py join artifacts → FailingTest records
+│
+▼
+context.py run the test's compiled SQL against DuckDB,
+│ sample the offending rows (read-only, capped)
+▼
+analyze.py grounded prompt → LLM → {root_cause, fix,
+│ confidence, evidence}
+▼
+report.py Rich terminal panel + markdown
+(via cli.py: sentinel analyze)
+```
+The design choice that keeps it small and robust: **it reads dbt's artifacts rather than
+re-running tests.** Everything it needs — compiled SQL, failing-row count, guarded model —
+is already in the JSON dbt writes on every build.
+
+## Grounding is the whole point
+
+An LLM asked "why did my dbt test fail?" with no context invents a plausible-sounding
+story. `dbt-sentinel` feeds the model only real evidence — the compiled SQL, the column
+schema, and a sample of the actual failing rows — and tells it to ground every claim in
+that evidence or report low confidence.
+
+The difference on a real failure (a taxi-trip speed test where `avg_speed_mph` was computed
+with a `600` multiplier instead of `60`):
+
+**Naive prompt — test name + row count only, no evidence:**
+
+> Most likely: division by zero or NULL producing infinite/NULL speed. With 1.8M failing
+> rows this suggests a systemic data issue. Run these diagnostic queries to pinpoint the
+> cause… [three SQL queries checking null/zero durations and negative distances].
+
+**Grounded (dbt-sentinel):**
+
+> The `avg_speed_mph` formula uses a multiplier of 600 instead of 60. A trip of 9.24 miles
+> in 52 minutes yields (9.24/52)×60 = 10.66 mph (plausible), but the bug computes
+> (9.24/52)×600 = 106.62 — which exactly matches the failing row. The same 10× inflation
+> holds across all sampled rows. **Confidence: high.**
+
+The naive answer guesses the wrong cause and hands the work back to you as queries to run.
+The grounded answer runs them, finds the real bug, and proves it against a row. See
+[`docs/example-analysis.md`](docs/example-analysis.md) for the full output.
+
+## Warehouses
+
+Sampling runs behind a small adapter interface, so the engine is a flag, not a rewrite.
+DuckDB works out of the box; BigQuery needs the optional extra and Application Default
+Credentials.
+
+```bash
+# DuckDB (default)
+uv run sentinel analyze --target-dir path/to/target --db warehouse.duckdb
+
+# BigQuery
+uv sync --group bq
+gcloud auth application-default login
+uv run sentinel analyze --target-dir path/to/target --bq-project my-project --bq-location EU
+```
+
+BigQuery works against the free sandbox — no billing account required. Credentials are
+never handled by dbt-sentinel itself; the client reads them from ADC.
+
+## Dashboard
+
+A single-page Streamlit view over the recorded history — latest run, per-test failure
+trend, and the stored root cause. It reads dbt-sentinel's own history database, so it
+needs no warehouse connection and no API key.
+
+```bash
+uv sync --group ui
+uv run --group ui streamlit run app/dashboard.py
+```
+
+![dashboard](docs/dashboard.png)
+
+## Design decisions
+
+**Read artifacts, don't re-run tests.** dbt writes the artifacts on every build; parsing
+them needs no dbt invocation and works against any completed run, including CI.
+
+**Sample rows via the test's compiled SQL.** Some models are ephemeral and aren't queryable
+as tables. The test's compiled SQL has that logic inlined, so it returns the offending rows
+regardless of materialization.
+
+**Ground, then surface confidence.** The prompt contains only evidence; the model is told to
+say "insufficient evidence" rather than speculate, and confidence is shown, not hidden.
+
+**Read-only, always.** The warehouse connection inspects; it never mutates.
+
+## Limitations
+
+- DuckDB and BigQuery supported; other engines need a new Warehouse adapter.
+- One dbt project per run.
+- Models with extended thinking can spend the whole token budget before emitting text,
+  so `max_tokens` is set generously; too low a value yields an unparseable (low-confidence)
+  result rather than an answer.
+- Diagnosis quality depends on the model and on how much signal the sampled rows carry;
+  low-signal failures correctly return low confidence.
+
+## Development
+
+```bash
+uv sync --group dev
+uv run ruff check .
+uv run pytest -v
+```
+
+CI runs the same lint + tests on every push. Tests use committed fixtures and a mocked API —
+no warehouse, no API key needed.
+
+---
+
+Built by [Qamar Raza](https://github.com/qraza).
+

dbt_sentinel-0.6.1/README.md

@@ -0,0 +1,167 @@
+# dbt-sentinel
+
+**dbt tells you a test failed. It doesn't tell you why.** `dbt-sentinel` reads dbt's
+run artifacts after a build, pulls the compiled SQL and a sample of the rows that
+actually failed, and asks an LLM — grounded strictly in that evidence — for a root-cause
+explanation and a concrete fix. It flags low-confidence answers instead of bluffing.
+
+[![CI](https://github.com/qraza/dbt-sentinel/actions/workflows/ci.yml/badge.svg)](https://github.com/qraza/dbt-sentinel/actions/workflows/ci.yml)
+![Python 3.12](https://img.shields.io/badge/python-3.12-blue)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+
+![sentinel analyze demo](docs/demo.gif)
+
+Runs in CI too — on every pull request it posts its grounded diagnosis as a comment:
+
+![PR comment](docs/pr-comment.png)
+
+## What this is
+
+A data-quality companion for dbt. When `dbt build` reports a failing test, you normally
+get a name and a row count — then you go digging. `dbt-sentinel` closes that gap: it joins
+`run_results.json` and `manifest.json` to find each failure, runs the test's own compiled
+SQL against the warehouse to sample the offending rows, and sends that concrete evidence to
+an LLM with a prompt engineered to stay grounded in it. The output is a structured verdict —
+root cause, suggested fix, confidence, and the evidence behind it — in the terminal and,
+optionally, markdown for a PR comment. It reads what dbt already produced; it never re-runs
+your tests or mutates data (the warehouse is opened read-only).
+
+It also tracks failures across runs — each one is flagged **new**, **recurring**, or
+**regressed**, and `sentinel history <test-id>` prints a test's full timeline, so you can
+see whether something just broke or has been broken for weeks.
+
+## Quickstart
+
+```bash
+uv sync
+export ANTHROPIC_API_KEY="sk-ant-..."
+export ANTHROPIC_MODEL="claude-opus-5"
+
+uv run sentinel analyze \
+  --target-dir path/to/dbt/target \
+  --db path/to/warehouse.duckdb \
+  --markdown report.md
+```
+
+If all tests pass, it says so and exits. If not, you get a diagnosis per failure.
+
+## How it works
+```
+dbt build ─► run_results.json (which tests failed, row counts)
+manifest.json (compiled SQL, guarded model/column, test type)
+│
+▼
+parse.py join artifacts → FailingTest records
+│
+▼
+context.py run the test's compiled SQL against DuckDB,
+│ sample the offending rows (read-only, capped)
+▼
+analyze.py grounded prompt → LLM → {root_cause, fix,
+│ confidence, evidence}
+▼
+report.py Rich terminal panel + markdown
+(via cli.py: sentinel analyze)
+```
+The design choice that keeps it small and robust: **it reads dbt's artifacts rather than
+re-running tests.** Everything it needs — compiled SQL, failing-row count, guarded model —
+is already in the JSON dbt writes on every build.
+
+## Grounding is the whole point
+
+An LLM asked "why did my dbt test fail?" with no context invents a plausible-sounding
+story. `dbt-sentinel` feeds the model only real evidence — the compiled SQL, the column
+schema, and a sample of the actual failing rows — and tells it to ground every claim in
+that evidence or report low confidence.
+
+The difference on a real failure (a taxi-trip speed test where `avg_speed_mph` was computed
+with a `600` multiplier instead of `60`):
+
+**Naive prompt — test name + row count only, no evidence:**
+
+> Most likely: division by zero or NULL producing infinite/NULL speed. With 1.8M failing
+> rows this suggests a systemic data issue. Run these diagnostic queries to pinpoint the
+> cause… [three SQL queries checking null/zero durations and negative distances].
+
+**Grounded (dbt-sentinel):**
+
+> The `avg_speed_mph` formula uses a multiplier of 600 instead of 60. A trip of 9.24 miles
+> in 52 minutes yields (9.24/52)×60 = 10.66 mph (plausible), but the bug computes
+> (9.24/52)×600 = 106.62 — which exactly matches the failing row. The same 10× inflation
+> holds across all sampled rows. **Confidence: high.**
+
+The naive answer guesses the wrong cause and hands the work back to you as queries to run.
+The grounded answer runs them, finds the real bug, and proves it against a row. See
+[`docs/example-analysis.md`](docs/example-analysis.md) for the full output.
+
+## Warehouses
+
+Sampling runs behind a small adapter interface, so the engine is a flag, not a rewrite.
+DuckDB works out of the box; BigQuery needs the optional extra and Application Default
+Credentials.
+
+```bash
+# DuckDB (default)
+uv run sentinel analyze --target-dir path/to/target --db warehouse.duckdb
+
+# BigQuery
+uv sync --group bq
+gcloud auth application-default login
+uv run sentinel analyze --target-dir path/to/target --bq-project my-project --bq-location EU
+```
+
+BigQuery works against the free sandbox — no billing account required. Credentials are
+never handled by dbt-sentinel itself; the client reads them from ADC.
+
+## Dashboard
+
+A single-page Streamlit view over the recorded history — latest run, per-test failure
+trend, and the stored root cause. It reads dbt-sentinel's own history database, so it
+needs no warehouse connection and no API key.
+
+```bash
+uv sync --group ui
+uv run --group ui streamlit run app/dashboard.py
+```
+
+![dashboard](docs/dashboard.png)
+
+## Design decisions
+
+**Read artifacts, don't re-run tests.** dbt writes the artifacts on every build; parsing
+them needs no dbt invocation and works against any completed run, including CI.
+
+**Sample rows via the test's compiled SQL.** Some models are ephemeral and aren't queryable
+as tables. The test's compiled SQL has that logic inlined, so it returns the offending rows
+regardless of materialization.
+
+**Ground, then surface confidence.** The prompt contains only evidence; the model is told to
+say "insufficient evidence" rather than speculate, and confidence is shown, not hidden.
+
+**Read-only, always.** The warehouse connection inspects; it never mutates.
+
+## Limitations
+
+- DuckDB and BigQuery supported; other engines need a new Warehouse adapter.
+- One dbt project per run.
+- Models with extended thinking can spend the whole token budget before emitting text,
+  so `max_tokens` is set generously; too low a value yields an unparseable (low-confidence)
+  result rather than an answer.
+- Diagnosis quality depends on the model and on how much signal the sampled rows carry;
+  low-signal failures correctly return low confidence.
+
+## Development
+
+```bash
+uv sync --group dev
+uv run ruff check .
+uv run pytest -v
+```
+
+CI runs the same lint + tests on every push. Tests use committed fixtures and a mocked API —
+no warehouse, no API key needed.
+
+---
+
+Built by [Qamar Raza](https://github.com/qraza).
+

dbt_sentinel-0.6.1/pyproject.toml

@@ -0,0 +1,56 @@
+[project]
+name = "dbt-sentinel"
+version = "0.6.1"
+description = "AI-grounded root-cause analysis for failing dbt tests"
+license = "MIT"
+keywords = ["dbt", "data-quality", "observability", "analytics-engineering", "llm"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Database",
+    "Topic :: Software Development :: Quality Assurance",
+]
+readme = "README.md"
+authors = [
+    { name = "Qamar Raza", email = "raza.qamar@gmail.com" }
+]
+requires-python = ">=3.12"
+dependencies = [
+    "click==8.1.7",
+    "duckdb==1.1.3",
+    "httpx==0.28.1",
+    "python-dotenv>=1.2.2",
+    "rich==13.9.4",
+]
+
+[project.optional-dependencies]
+bq = ["google-cloud-bigquery>=3.43.0"]
+ui = ["streamlit>=1.40"]
+
+[project.urls]
+Homepage = "https://github.com/qraza/dbt-sentinel"
+Repository = "https://github.com/qraza/dbt-sentinel"
+Issues = "https://github.com/qraza/dbt-sentinel/issues"
+
+[project.scripts]
+sentinel = "dbt_sentinel.cli:main"
+
+[build-system]
+requires = ["uv_build>=0.11.3,<0.12.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+bq = [
+    "google-cloud-bigquery>=3.43.0",
+]
+dev = [
+    "pytest>=9.1.1",
+    "ruff>=0.16.0",
+]
+ui = [
+    "pandas>=3.0.5",
+    "streamlit>=1.61.1",
+]
+
+

dbt_sentinel-0.6.1/src/dbt_sentinel/__init__.py

@@ -0,0 +1,43 @@
+"""dbt-sentinel — AI-grounded root-cause analysis for failing dbt tests.
+
+Typical library use:
+
+    from dbt_sentinel import parse, open_warehouse, gather_context, analyze
+
+    for test in parse("target"):
+        ctx = gather_context(test, open_warehouse(duckdb_path="wh.duckdb"))
+        print(analyze(ctx).root_cause)
+"""
+
+from .analyze import Analysis, analyze, build_prompt
+from .context import FailureContext, gather_context
+from .parse import FailingTest, parse
+from .report import AnalyzedFailure, build_markdown, render_terminal
+from .warehouse import (
+    BigQueryWarehouse,
+    DuckDBWarehouse,
+    QueryResult,
+    Warehouse,
+    open_warehouse,
+)
+
+__version__ = "0.6.1"
+
+__all__ = [
+    "Analysis",
+    "AnalyzedFailure",
+    "BigQueryWarehouse",
+    "DuckDBWarehouse",
+    "FailingTest",
+    "FailureContext",
+    "QueryResult",
+    "Warehouse",
+    "__version__",
+    "analyze",
+    "build_markdown",
+    "build_prompt",
+    "gather_context",
+    "open_warehouse",
+    "parse",
+    "render_terminal",
+]

dbt_sentinel-0.6.1/src/dbt_sentinel/analyze.py

@@ -0,0 +1,170 @@
+"""Turn a failing test's gathered context into a grounded, root-cause analysis.
+
+This is the part that makes dbt-sentinel more than a wrapper around "ask an LLM
+why my test failed". The model is given *only* concrete evidence -- the test
+definition, the compiled SQL, the column schema, and a capped sample of the
+actual offending rows -- and is instructed to ground every claim in that
+evidence and to report low confidence (rather than invent a cause) when the
+sample doesn't support a conclusion.
+
+The LLM is asked to return strict JSON so the result is structured and testable:
+    {"root_cause": ..., "suggested_fix": ..., "confidence": "high|medium|low",
+     "evidence": ...}
+
+The Anthropic call is a thin httpx POST, matching the taxi repo's llm helper, so
+it's trivially mockable in tests -- no network in CI.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+from dataclasses import dataclass
+from typing import Any
+
+import httpx
+
+from .context import FailureContext
+
+ANTHROPIC_URL = "https://api.anthropic.com/v1/messages"
+# Set ANTHROPIC_MODEL to whatever model your key can use. Check the exact string
+# your taxi repo's cli/llm.py already uses -- that's the source of truth for your
+# setup -- rather than trusting this default.
+DEFAULT_MODEL = os.environ.get("ANTHROPIC_MODEL", "claude-sonnet-4-5")
+API_VERSION = "2023-06-01"
+
+SYSTEM_PROMPT = (
+    "You are a senior analytics engineer reviewing a failed dbt data-quality "
+    "test. You are given the test definition, the compiled SQL that ran, the "
+    "column schema, and a sample of the rows that FAILED the test. "
+    "Diagnose the most likely root cause and propose a concrete fix. "
+    "Ground every statement in the evidence provided. If the evidence is "
+    "insufficient to be sure, say so plainly and set confidence to 'low' rather "
+    "than guessing or inventing tables, columns, or values that are not shown. "
+    "Respond with ONLY a JSON object with keys: root_cause (string), "
+    "suggested_fix (string), confidence (one of 'high', 'medium', 'low'), "
+    "evidence (string: which specific columns/values led you to the conclusion)."
+)
+
+
+@dataclass
+class Analysis:
+    """A structured, grounded verdict on a single failing test."""
+
+    root_cause: str
+    suggested_fix: str
+    confidence: str  # "high" | "medium" | "low"
+    evidence: str
+    raw: str = ""  # the model's raw text, kept for debugging / display
+
+    @property
+    def is_confident(self) -> bool:
+        return self.confidence.lower() in {"high", "medium"}
+
+
+def build_prompt(ctx: FailureContext) -> str:
+    """Assemble the grounded user prompt from gathered context.
+
+    Deliberately includes only evidence -- no external knowledge, no model name
+    guessing. Everything here is something dbt or the warehouse actually
+    produced for this specific failure.
+    """
+    t = ctx.test
+    columns = ", ".join(f"{name} ({dtype})" for name, dtype in ctx.columns) or "(unknown)"
+    sample = json.dumps(ctx.sample_rows, indent=2, default=str)
+
+    kind = t.test_type or "singular (hand-written) test"
+    kwargs = json.dumps(t.test_kwargs) if t.test_kwargs else "(none)"
+
+    return (
+        f"A dbt test failed.\n\n"
+        f"Test name: {t.test_name}\n"
+        f"Test type: {kind}\n"
+        f"Test arguments: {kwargs}\n"
+        f"Guarded model: {t.model_name}\n"
+        f"Guarded column: {t.column_name or '(whole-row / singular test)'}\n"
+        f"Failing row count: {t.failure_count}\n"
+        f"dbt message: {t.message or '(none)'}\n\n"
+        f"Columns in the failing rows: {columns}\n\n"
+        f"Sample of the rows that FAILED the test (capped):\n{sample}\n\n"
+        f"Compiled SQL that dbt executed for this test:\n{t.compiled_sql}\n\n"
+        f"Sampling note: {ctx.note}\n"
+    )
+
+
+def _parse_response(text: str) -> Analysis:
+    """Parse the model's JSON reply, tolerating stray prose around the object."""
+    raw = text.strip()
+    start, end = raw.find("{"), raw.rfind("}")
+    if start != -1 and end != -1 and end > start:
+        try:
+            data: dict[str, Any] = json.loads(raw[start : end + 1])
+            return Analysis(
+                root_cause=str(data.get("root_cause", "")).strip(),
+                suggested_fix=str(data.get("suggested_fix", "")).strip(),
+                confidence=str(data.get("confidence", "low")).strip().lower(),
+                evidence=str(data.get("evidence", "")).strip(),
+                raw=raw,
+            )
+        except json.JSONDecodeError:
+            pass
+    # Model didn't return usable JSON -- fail safe as low confidence, not a crash.
+    return Analysis(
+        root_cause="Could not parse a structured analysis from the model response.",
+        suggested_fix="",
+        confidence="low",
+        evidence="",
+        raw=raw,
+    )
+
+
+def analyze(
+    ctx: FailureContext,
+    *,
+    api_key: str | None = None,
+    model: str = DEFAULT_MODEL,
+    max_tokens: int = 8192,
+    client: httpx.Client | None = None,
+) -> Analysis:
+    """Send grounded context to the Anthropic API and return a structured verdict.
+
+    Args:
+        ctx: the gathered failure context (test + sample offending rows).
+        api_key: Anthropic API key; falls back to ANTHROPIC_API_KEY env var.
+        model: model string to call.
+        client: optional httpx.Client, mainly for injection in tests.
+    """
+    key = api_key or os.environ.get("ANTHROPIC_API_KEY")
+    if not key:
+        raise RuntimeError(
+            "No Anthropic API key. Set ANTHROPIC_API_KEY or pass api_key=."
+        )
+
+    payload = {
+        "model": model,
+        "max_tokens": max_tokens,
+        "system": SYSTEM_PROMPT,
+        "messages": [{"role": "user", "content": build_prompt(ctx)}],
+    }
+    headers = {
+        "x-api-key": key,
+        "anthropic-version": API_VERSION,
+        "content-type": "application/json",
+    }
+
+    owns_client = client is None
+    client = client or httpx.Client(timeout=60)
+    try:
+        resp = client.post(ANTHROPIC_URL, headers=headers, json=payload)
+        resp.raise_for_status()
+        body = resp.json()
+    finally:
+        if owns_client:
+            client.close()
+    # Anthropic returns {"content": [{"type": "text", "text": "..."}], ...}
+    text = "".join(
+        block.get("text", "")
+        for block in body.get("content", [])
+        if block.get("type") == "text"
+    )
+    return _parse_response(text)