dbt-scribe 0.1.0__tar.gz

dbt_scribe-0.1.0/.env.example

@@ -0,0 +1,6 @@
+# Copy this file to .env and fill in the keys for the provider(s) you use.
+# Only the key for your configured provider is required.
+
+ANTHROPIC_API_KEY=sk-ant-...
+OPENAI_API_KEY=sk-...
+GOOGLE_API_KEY=...

dbt_scribe-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,30 @@
+name: CI
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Check out repository
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+
+      - name: Install package
+        run: python -m pip install -e ".[dev]"
+
+      - name: Run ruff
+        run: ruff check .
+
+      - name: Run mypy
+        run: mypy dbt_scribe
+
+      - name: Run pytest
+        run: pytest

dbt_scribe-0.1.0/.gitignore

@@ -0,0 +1,68 @@
+# =============================================================================
+# dbt-scribe — .gitignore
+# =============================================================================
+
+# ── Python ────────────────────────────────────────────────────────────────────
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+*.pyd
+
+# Virtual environments
+.venv/
+venv/
+env/
+ENV/
+
+# Distribution / packaging
+dist/
+build/
+*.egg-info/
+*.egg
+.eggs/
+MANIFEST
+
+# Type checking
+.mypy_cache/
+.dmypy.json
+dmypy.json
+.pytype/
+
+# Test / coverage
+.pytest_cache/
+.coverage
+.coverage.*
+coverage.xml
+htmlcov/
+
+# ── dbt-scribe specific ───────────────────────────────────────────────────────
+# LLM response cache — keyed on compiled SQL + config fingerprint
+.dbt-scribe-cache/
+
+# DuckDB files from test projects
+*.duckdb
+*.duckdb.wal
+
+# ── Environment / secrets ─────────────────────────────────────────────────────
+.env
+.env.local
+.env.*.local
+
+# ── Editors and OS ────────────────────────────────────────────────────────────
+.DS_Store
+Thumbs.db
+.idea/
+.vscode/
+*.swp
+*.swo
+*~
+
+# ── Claude Code ───────────────────────────────────────────────────────────────
+.claude/
+
+# ── Misc ──────────────────────────────────────────────────────────────────────
+*.log
+
+# Claude Code local settings
+.claude/

dbt_scribe-0.1.0/CONTEXT.md

@@ -0,0 +1,91 @@
+# CONTEXT.md — dbt-scribe
+
+## What this project is
+
+`dbt-scribe` is a Python CLI that analyses a dbt Core project and uses an LLM
+(Anthropic Claude, OpenAI, or Google Gemini) to automatically generate:
+
+- **Documentation** — model and column descriptions in YAML, plus long-form docs
+  blocks in Markdown (`*__docs.md` files), following dbt's two-tier convention
+- **Tests** — named generic tests in YAML (`not_null`, `unique`, `accepted_values`,
+  `relationships`) and singular SQL tests in `tests/`, inferred from column semantics
+
+The tool is non-destructive by default: it only fills in what is missing and never
+overwrites existing descriptions or tests unless `--force` is explicitly passed.
+A `{{ doc("...") }}` reference is treated as a filled description and is preserved.
+
+## Problem it solves
+
+Writing thorough dbt documentation and tests manually is time-consuming. A typical
+staging model with 15 columns takes 30–45 minutes to document properly when following
+strict conventions (English descriptions, two-tier docs, named tests, shared blocks,
+four-section mart template, etc.). This debt accumulates quickly across multiple
+projects and degrades portfolio quality.
+
+Existing tools (`dbt-osmosis`, `dbt-codegen`, `dbt Assist`) either perform
+mechanical propagation without LLM understanding, generate empty boilerplate, or
+require dbt Cloud. `dbt-scribe` fills the gap: LLM-powered generation, local,
+configurable per project, compatible with dbt Core.
+
+## How it works
+
+1. **Bootstrap** — validates that the current directory is a dbt project root
+   (`dbt_project.yml` + `target/manifest.json` + `dbt-scribe.yml` must all be present)
+2. **Manifest parsing** — reads `target/manifest.json` to extract compiled SQL
+   (Jinja2-resolved), column lists, lineage, adapter type, and fully-qualified node names
+3. **YAML parsing** — reads existing `.yml` files to determine what is already documented
+4. **Analysis** — detects the layer (staging / intermediate / marts) and infers column
+   types (pk, fk, enum, timestamp, boolean, metric, shared, text)
+5. **Generation** — calls the configured LLM provider with structured Jinja2 prompts;
+   all responses are JSON for reliable parsing
+6. **Writing** — creates `.yml` files from scratch or merges into existing ones;
+   creates or updates `*__docs.md` files; writes singular test SQL files
+
+> **Prerequisite:** `dbt compile` must be run before `dbt-scribe` so that
+> `target/manifest.json` is up to date. The tool validates this at startup.
+
+## Configuration
+
+Each dbt project that uses `dbt-scribe` has its own `dbt-scribe.yml` at its root.
+This file is versioned with the dbt project and controls: LLM provider and model,
+documentation rules (two-tier, shared columns, mart template), test generation patterns
+(PK/FK/enum column name regexes), coverage thresholds, and layer conventions.
+
+A default config is generated by `dbt-scribe init`.
+
+## Design principles
+
+- **Manifest-first** — compiled SQL from `manifest.json` is the only source of truth;
+  raw `.sql` files (which contain unresolved Jinja2) are never parsed directly
+- **Non-destructive** — existing documentation and tests are preserved unless `--force`
+- **Convention-aware** — the tool understands dbt layer conventions and adapts its
+  output accordingly (different doc templates for staging vs. marts, etc.)
+- **Provider-agnostic** — an `LLMProvider` abstraction isolates all generator code
+  from the specifics of Anthropic, OpenAI, or Google SDKs
+- **JSON output from LLM** — all prompts instruct the model to return JSON only,
+  eliminating fragile regex-based text parsing
+
+## Supported dbt adapters
+
+DuckDB (default), BigQuery, PostgreSQL. Auto-detected from `manifest.json` metadata.
+
+## Target users (V1)
+
+Solo analytics engineers and small teams running dbt Core locally, with a three-layer
+medallion architecture (staging / intermediate / marts). The tool is designed to be
+dropped into any existing dbt project with minimal setup.
+
+## Repository
+
+- GitHub: https://github.com/jeremy6680/dbt-scribe
+- Author: Jeremy Marchandeau
+- License: MIT
+- Part of the Web2Data portfolio (web2data.jeremymarchandeau.com)
+
+## Related projects
+
+- `w2d-scaffold` — project scaffolding CLI; `dbt-scribe init` may eventually be called
+  automatically when scaffolding a new `data` project type
+- Metrigator — shares the same "audit and automatically improve project quality" philosophy
+- TasteBase, BrickMetrics, DraftLab — dbt projects in the portfolio that serve as
+  real-world test targets for `dbt-scribe`

dbt_scribe-0.1.0/DECISIONS.md

@@ -0,0 +1,363 @@
+# DECISIONS.md — dbt-scribe
+
+Architectural and technical decisions log.
+Each entry documents what was decided, why, and what alternatives were considered.
+
+---
+
+## ADR-001 — Multi-provider LLM abstraction from V1
+
+**Date:** 2026-05-07  
+**Status:** Accepted
+
+**Decision:** Support three LLM providers from V1 (Anthropic Claude, OpenAI, Google
+Gemini) through a shared `LLMProvider` abstract interface.
+
+**Rationale:** Provider preference is a legitimate user choice (cost, access, model
+quality). Implementing the abstraction upfront costs little (one interface + three
+simple adapters) and avoids a painful refactor later. All providers receive identical
+prompts (structured JSON, temperature 0.2), making the abstraction natural.
+
+**Providers:**
+
+| Provider              | Default model              | Environment variable | SDK package    |
+| --------------------- | -------------------------- | -------------------- | -------------- |
+| `anthropic` (default) | `claude-sonnet-4-20250514` | `ANTHROPIC_API_KEY`  | `anthropic`    |
+| `openai`              | `gpt-4o`                   | `OPENAI_API_KEY`     | `openai`       |
+| `google`              | `gemini-2.5-pro`           | `GOOGLE_API_KEY`     | `google-genai` |
+
+**Note on Google SDK:** The originally planned `google-generativeai` package was
+deprecated before implementation. The `google-genai` package (Google's replacement,
+`from google import genai`) is used instead. The interface is functionally equivalent.
+
+**Consequence:** Three SDK dependencies instead of one. All three are installed by
+default; optional extras may be introduced in V2 if package size becomes a concern.
+
+**Alternative rejected:** Claude-only in V1, abstraction in V2. Rejected because the
+interface is trivial to write now and would cause an API-breaking change later.
+
+---
+
+## ADR-002 — Structured JSON output from LLM
+
+**Date:** 2026-05-07  
+**Status:** Accepted
+
+**Decision:** All LLM calls must return valid JSON only. The system prompt explicitly
+forbids any text outside the JSON structure (no preamble, no markdown fences,
+no explanation).
+
+**Rationale:** JSON parsing is reliable and deterministic. Free-text responses would
+require fragile regex-based extraction that breaks on minor model output variations.
+
+**Consequence:** Prompts must be carefully engineered to prevent the model from adding
+explanatory text. A JSON parse failure triggers a retry (up to 3 attempts).
+
+---
+
+## ADR-003 — One LLM call per model, not per column
+
+**Date:** 2026-05-07  
+**Status:** Accepted
+
+**Decision:** Generate documentation and tests for all columns of a model in a single
+LLM call.
+
+**Rationale:** A single call gives the model full inter-column context (e.g., it can
+infer that `home_score` and `away_score` are paired concepts). It also reduces API
+cost and latency compared to one call per column.
+
+**Consequence:** Prompts can be long for wide models. If a model exceeds ~50 columns,
+the call will be split into two sequential calls to stay within context limits.
+
+---
+
+## ADR-004 — `ruamel.yaml` instead of `PyYAML`
+
+**Date:** 2026-05-07  
+**Status:** Accepted (deferred to Phase 2 — `PyYAML` used in Phase 1 for simplicity)
+
+**Decision:** Use `ruamel.yaml` for all YAML read/write operations in the final tool.
+
+**Rationale:** `PyYAML` does not preserve comments or key ordering in round-trip
+operations. dbt YAML files use structured comments (`# ── Primary key ──`) as
+visual separators that must be preserved when merging into existing files.
+
+**Consequence:** `ruamel.yaml` has a slightly more verbose API than `PyYAML`.
+Phase 1 uses `PyYAML` to keep the initial scope small; migration to `ruamel.yaml`
+is scheduled for Phase 2 (Step tracked in NEXT_STEPS.md backlog).
+
+---
+
+## ADR-005 — Cache key = SHA-256(compiled_sql + config_fingerprint)
+
+**Date:** 2026-05-07  
+**Status:** Accepted (deferred to Phase 2)
+
+**Decision:** LLM response cache keys are computed from a composite hash of the
+compiled SQL and the relevant sections of `dbt-scribe.yml`.
+
+**Rationale:** Hashing compiled SQL alone is insufficient — if `dbt-scribe.yml`
+changes (new patterns, new `default_owner`, new thresholds), cached results may no
+longer match the expected output even if the SQL is unchanged.
+
+**Implementation:**
+
+```python
+config_fingerprint = json.dumps({
+    "llm": config.llm.model_dump(),
+    "docs": config.docs.model_dump(),
+    "tests": config.tests.model_dump(),
+    "conventions": config.conventions.model_dump(),
+}, sort_keys=True)
+cache_key = hashlib.sha256((compiled_sql + config_fingerprint).encode()).hexdigest()
+```
+
+**Consequence:** `.dbt-scribe-cache/` directory must be added to `.gitignore`.
+A config change invalidates all cache entries for affected models.
+
+---
+
+## ADR-006 — `dbt-scribe.yml` lives in the dbt project root
+
+**Date:** 2026-05-07  
+**Status:** Accepted
+
+**Decision:** The configuration file is versioned inside the target dbt project,
+not in a global user directory.
+
+**Rationale:** Each dbt project has its own conventions (adapter, coverage thresholds,
+shared column names, owner details). The config must evolve with the project and be
+visible to anyone cloning the repo.
+
+**Consequence:** `dbt-scribe` looks for `dbt-scribe.yml` in the current working
+directory. A missing config file triggers a `BootstrapError` with an explicit
+suggestion to run `dbt-scribe init`.
+
+---
+
+## ADR-007 — `sqlglot` applied to compiled SQL only (never to raw `.sql` files)
+
+**Date:** 2026-05-07  
+**Status:** Accepted
+
+**Decision:** `sqlglot` is used exclusively to parse the compiled SQL extracted from
+`manifest.json`. Raw `.sql` files (which contain unresolved Jinja2 templates) are
+never passed to `sqlglot`.
+
+**Rationale:** `sqlglot` is an excellent multi-dialect SQL parser, but dbt `.sql`
+files contain Jinja2 (`{{ ref('...') }}`, `{{ var('...') }}`, custom macros) that
+`sqlglot` cannot handle. Using compiled SQL (where Jinja2 is fully resolved by dbt)
+eliminates this friction entirely.
+
+**Consequence:** `sql_parser.py` does not exist. All SQL parsing logic lives in
+`manifest_parser.py` (extraction from manifest) and `analyzer.py` (calling `sqlglot`
+on the extracted compiled SQL).
+
+**Alternative rejected:** Naive Jinja2 pre-processing (replace `{{ ... }}` with
+placeholders). Rejected because it is fragile and impossible to test exhaustively.
+
+---
+
+## ADR-008 — YAML Writer creates missing files from scratch
+
+**Date:** 2026-05-07  
+**Status:** Accepted
+
+**Decision:** `dbt-scribe` creates `.yml` files from scratch when they do not exist,
+rather than requiring a pre-existing skeleton from `dbt-codegen`.
+
+**Rationale:** Makes the tool fully autonomous. Users can run `dbt-scribe generate`
+on a dbt project that has no YAML documentation at all and get a complete, correctly
+structured result. `dbt-codegen` remains compatible but is no longer required.
+
+**Consequence:** The YAML Writer must know the canonical structure expected per layer
+(section comments, column ordering, conditional `persist_docs`, tags inferred from
+the manifest `fqn`).
+
+**Alternative rejected:** Require a pre-existing empty `.yml` file. Rejected because
+it introduces unnecessary friction (two-tool workflow for a task that should be one command).
+
+---
+
+## ADR-009 — `manifest.json` is a mandatory prerequisite
+
+**Date:** 2026-05-07  
+**Status:** Accepted
+
+**Decision:** `target/manifest.json` is required for all commands except `init`.
+There is no degraded mode that operates without the manifest in V1.
+
+**Rationale:** Raw `.sql` files contain Jinja2 that cannot be reliably parsed without
+running dbt. The manifest, produced by `dbt compile`, contains fully-resolved compiled
+SQL — the only reliable source for column extraction.
+
+**User impact:** The user must run `dbt compile` before using `dbt-scribe`. This is
+documented prominently in the README and enforced by the bootstrap check with an
+explicit error message including the command to run.
+
+**Alternative rejected:** Best-effort mode on raw SQL (Jinja2 replaced by placeholders).
+Rejected because results would be unreliable and difficult to test. The `dbt compile`
+prerequisite is a low-friction, well-understood workflow step.
+
+**V2 consideration:** A `--no-manifest` flag could enable partial analysis on raw SQL
+for cases where compiling the project is not possible (e.g., missing warehouse credentials).
+
+---
+
+## ADR-010 — `{{ doc("...") }}` reference treated as a filled description
+
+**Date:** 2026-05-07  
+**Status:** Accepted
+
+**Decision:** The YAML Writer treats a column description containing `{{ doc("...") }}`
+as "already documented" and will not overwrite it unless `--force` is passed.
+
+**Rationale:** A `doc()` reference is intentional documentation. Overwriting it
+automatically would silently break carefully hand-crafted YAML files.
+
+**Implementation:** `is_description_set(description)` in `yaml_parser.py` returns
+`True` for any non-empty string OR any string matching the pattern `{{\s*doc\(`.
+
+**Consequence:** `--force` is the only mechanism to replace an existing `doc()` reference
+with a generated inline description.
+
+---
+
+## ADR-011 — Run from dbt project root, no `--project-dir` flag in V1
+
+**Date:** 2026-05-07  
+**Status:** Accepted
+
+**Decision:** `dbt-scribe` must be invoked from the dbt project root directory.
+No `--project-dir` flag is provided in V1.
+
+**Rationale:** Keeps all path resolution relative to `os.getcwd()`, consistent with
+how `dbt` itself works. Simpler bootstrap logic.
+
+**Consequence:** The bootstrap check validates `dbt_project.yml` in the current
+directory and exits with a clear error if it is absent.
+
+**Alternative rejected:** Auto-discovery (walking up parent directories to find
+`dbt_project.yml`). Rejected because implicit behavior is hard to debug and
+could produce surprising results in nested project structures.
+
+**V2:** A `--project-dir` flag will be added to support CI workflows that invoke
+tools from a repository root that is not the dbt project root.
+
+---
+
+## ADR-012 — `model_root` read from `dbt_project.yml`, not hardcoded
+
+**Date:** 2026-05-08  
+**Status:** Accepted
+
+**Decision:** The path prefix used to locate model YAML files (default `"models"`) is
+read from the `model-paths` key of `dbt_project.yml` at startup, rather than being
+hardcoded as the string `"models"`.
+
+**Rationale:** dbt's `model-paths` config is the authoritative source for where models
+live. Hardcoding `"models"` would silently create files in the wrong directory (or miss
+existing ones) for any project that sets `model-paths: [dbt_models]` or similar —
+without raising an error, causing documentation corruption.
+
+**Implementation:** `_read_model_root()` in `cli.py` reads `dbt_project.yml` (already
+required by `_bootstrap()`) and extracts `model-paths[0]`, defaulting to `"models"` if
+the key is absent. The value is stored in `ScribeConfig.model_root` via `model_copy()`
+and threaded through to `yaml_writer`, `docs_writer`, and `resolver`.
+
+**Consequence:** `ScribeConfig` gains a `model_root: str` field. It is not
+user-configurable in `dbt-scribe.yml` — it is always derived from `dbt_project.yml` to
+avoid duplicating config the user already maintains.
+
+**Alternative rejected:** A `model_root` key in `dbt-scribe.yml`. Rejected because it
+would require users to keep two files in sync.
+
+---
+
+## ADR-013 — Non-retryable HTTP status codes bypass the retry loop
+
+**Date:** 2026-05-08  
+**Status:** Accepted
+
+**Decision:** `LLMProvider.complete()` does not retry exceptions whose `status_code`
+attribute is in `{400, 401, 403, 404}`. These are re-raised immediately.
+
+**Rationale:** Retrying authentication errors (401), permission errors (403), or
+not-found errors (404) is wasteful and misleading — the result is identical on every
+attempt, and the 0 + 2 + 4 second backoff adds 6 seconds of delay before the same
+failure. The real root cause (missing API key, wrong model name) is also obscured by
+the final "LLM call failed after 3 attempts" message.
+
+**Implementation:** Provider-agnostic duck typing: `getattr(exc, "status_code", None)`.
+The Anthropic, OpenAI, and Google SDKs all expose `status_code` on their HTTP exception
+classes. No provider SDK is imported in `base_generator.py`.
+
+**Consequence:** Only transient errors (network timeout, 429 rate limit, 5xx) benefit
+from the retry loop.
+
+---
+
+## ADR-014 — sqlglot column extraction fallback when manifest columns dict is empty
+
+**Date:** 2026-05-09
+**Status:** Accepted
+
+**Decision:** When `manifest.json` has an empty `columns` dict for a node (which happens
+when no YAML existed at `dbt compile` time), extract column names from the compiled SQL
+using sqlglot. Try the adapter dialect first (e.g. `bigquery` for backtick quoting), then
+fall back through `bigquery → duckdb → postgres → None` until one succeeds.
+
+**Rationale:** dbt only populates `columns` in the manifest from existing YAML declarations.
+On a greenfield project with no YAML, the dict is always empty — which is exactly the
+situation `dbt-scribe` is designed to fix. The multi-dialect fallback handles BigQuery's
+backtick quoting which sqlglot rejects without the correct dialect.
+
+**Consequence:** `manifest_parser.py` now imports sqlglot. Column names extracted from SQL
+are lowercased and have no data_type or description (None). This is acceptable — the
+generator only needs column names to produce descriptions and tests.
+
+---
+
+## ADR-015 — Mart docs block assembled in Python, not by LLM
+
+**Date:** 2026-05-09
+**Status:** Accepted
+
+**Decision:** The four-section mart docs block structure is assembled in Python in
+`docs_generator._assemble_mart_docs_block()`. The LLM is only asked to provide the
+content of two sections (`description_and_motivation` and `known_limitations`) as
+separate JSON fields. The section headers, blank lines, and stakeholder sections are
+added by code.
+
+**Rationale:** Prompt-only enforcement of the four-section template proved unreliable —
+the LLM consistently returned a `docs_block_content` key with free-form markdown
+regardless of how the prompt was worded. Moving structure to code guarantees the template
+is always respected. The fallback in `_assemble_mart_docs_block` handles the case where
+the LLM still returns `docs_block_content` by using it as the description section content.
+
+**Consequence:** `docs_mart.j2` now requests separate JSON fields. The LLM response shape
+changed from `{model_description, docs_block_content, columns}` to
+`{model_description, description_and_motivation, known_limitations, columns}`.
+The Python assembly guarantees the four sections regardless of LLM output.
+
+---
+
+## ADR-016 — Layer detection uses alternate spelling fallbacks
+
+**Date:** 2026-05-09
+**Status:** Accepted
+
+**Decision:** `detect_layer()` in `analyzer.py` accepts common alternate spellings of
+layer folder names (`mart` vs `marts`, `int` vs `intermediate`) in addition to exact
+config matches.
+
+**Rationale:** Real dbt projects use `mart/` (without s) while the CDC and default config
+specify `marts/`. Requiring an exact match caused `Layer.UNKNOWN` on the test project,
+which silently routed mart models through the staging prompt and bypassed
+`_assemble_mart_docs_block`. Alternate spelling fallbacks make the tool robust to the
+most common naming variations without requiring config changes.
+
+**Consequence:** Users with non-standard folder names should still set `marts_prefix`
+correctly in `dbt-scribe.yml` — the fallbacks are a safety net, not a replacement for
+correct config.