sql-code-graph 1.5.0__tar.gz → 1.33.0__tar.gz

{sql_code_graph-1.5.0 → sql_code_graph-1.33.0}/.claude/agents/architect-planner.md

@@ -91,6 +91,12 @@ Before starting work, call `list_graph_stats_tool` once.
 - Unit tests: ...
 - Integration tests: ...
 
+> Describe each test by the **behavior** it verifies, never by a task/case code
+> (`TC1`, `T-03`, …). The developer names tests `test_<unit>_<scenario>_<expected>`
+> and links this plan in the docstring — see *Test naming* in
+> [`developer.md`](.claude/agents/developer.md). Do not seed opaque codes the
+> developer will copy into test names.
+
 ## Acceptance Criteria
 - [ ] Criterion 1
 - [ ] Criterion 2
@@ -139,14 +145,29 @@ code-reviewer's job).
 
 Steps:
 
-1. Read `git diff main...HEAD` and the plan side-by-side.
-2. For each implementation step in the plan, verify it was addressed.
-3. Append a `## Plan Compliance — YYYY-MM-DD` section to `plan/<feature>.md`:
-   - `PASS` — all steps implemented as planned (deviations are documented)
+1. Read `git diff main...HEAD` and the plan side-by-side. **The developer's
+   handoff/PR body is a claim, not evidence — never quote it as verification.**
+   Presence of code that "looks like" a step is not compliance; trace what the
+   code DOES.
+2. For each implementation step in the plan, verify it was addressed and record
+   the **evidence**: the diff file:line that implements it, and the test that
+   pins it. A step with no citable diff hunk or no pinning test is not PASS.
+3. For each plan **acceptance criterion**, run its test yourself (or execute the
+   criterion's repro directly) and record the observed output. Check the test
+   asserts the criterion itself, not adjacent behavior — and that fixtures model
+   the live configuration the criterion will face (schema aliases, USE SCHEMA
+   context, real-corpus statement shapes), not a minimal idealized case. A
+   criterion whose test passes on a fixture that cannot exhibit the failure mode
+   is UNVERIFIED, not PASS (this is how the v1.21.0 dual-write shipped green).
+4. Append a `## Plan Compliance — YYYY-MM-DD` section to `plan/<feature>.md`,
+   one row per step/criterion: PASS with evidence (diff hunk + test + observed
+   output), FAIL with reason, or UNVERIFIED with what's missing.
+   - `PASS` — every row PASS (deviations are documented)
    - `FAIL — <step>: <reason>` — a step was skipped or implemented contrary
-     to the plan
-4. Commit the compliance result.
-5. If `FAIL`, tell the developer what to fix before code review starts.
+     to the plan; UNVERIFIED rows also block an overall PASS
+5. Commit the compliance result.
+6. If `FAIL` (or any UNVERIFIED), tell the developer what to fix before code
+   review starts.
 
 ## MUST NOT
 

{sql_code_graph-1.5.0 → sql_code_graph-1.33.0}/.claude/agents/developer.md

@@ -78,6 +78,24 @@ Before starting work, call `list_graph_stats_tool` once.
 13. Say **"finished"** — this signals the idle architect-planner to perform the
     plan-compliance check before the PR moves to code review.
 
+## Report facts, not verdicts
+
+Your handoff, PR body, and final report state **what is** — never whether it
+satisfies the plan. That judgment belongs to the compliance check and the
+shepherd, who verify against the diff, not your summary.
+
+- DO report: files changed with file:line, exact test counts (collected/passed/
+  failed/skipped), commands run with their observable output, deviations with
+  their mechanism, anything NOT done or NOT verified.
+- DO NOT write: "implemented as planned", "all acceptance criteria met",
+  "no deviations" as a conclusion you reasoned to, "plan compliant", or any
+  equivalent verdict. If you believe a criterion is met, cite the test name and
+  its asserted output instead — the evidence, not the judgment.
+- A claim of "pre-existing failure" requires proof in the report: the exact
+  command run on clean master and its output. No proof → don't make the claim.
+- Never record decisions in progress.txt that were not explicitly made by the
+  user or the shepherd in this session.
+
 ## PR Review Iteration
 
 When addressing review comments:
@@ -115,6 +133,57 @@ Run through this before marking work done:
    feature produces observable output (e.g., non-empty list, specific field value). Tests
    that only cover exception handling or log messages are insufficient on their own.
 
+## Test naming — describe behavior, link the doc
+
+Tests are read far more often than the plan that spawned them. A name like
+`test_tc2_*` or `test_T35_DOWN_1_*` is meaningless once the sprint closes — the
+reader has to go find the plan to learn what broke. Use a name that stands on its
+own and a docstring that links the governing doc.
+
+**Convention (standard three-part name — nothing to install):**
+
+```
+test_<unit>_<scenario>_<expected>
+```
+
+**Forbidden: opaque plan-internal codes in the test name.** These families are all
+ephemeral identifiers that mean nothing outside their plan — never use them as the
+name:
+
+| Family | Example | Origin |
+|--------|---------|--------|
+| `TC<n>` | `test_tc6b_*` | "test case" numbers in a plan |
+| `T<nn>` task IDs | `test_T01_*`, `test_T09_06_*` | sprint task IDs |
+| `T34`/`T35` (+ `CFG`/`DIFF`/`DOWN`/`IDX` sub-codes) | `test_T35_DOWN_1_*` | feature task IDs |
+| `F<n>` feature IDs | `test_F2_*` | feature IDs |
+| `gap<n>` / `issue<n>` | `test_gap3_*`, `test_issue38_*` | blueprint-gap / issue numbers |
+
+The code goes in the **docstring**, not the name. Link the plan doc (markdown link,
+real path — see the link convention in `ARCHITECTURE_REVIEW.md`) so the reader can
+jump to the rationale:
+
+```python
+def test_filtered_surface_recall_physical_sources(indexed_db):
+    """Physical source tables survive the default analyze filter.
+
+    Guards the #40 user-surface recall plan
+    ([plan doc](plan/sprints/v1_2_1_bugfix.md)).
+    """
+```
+
+**Two exceptions** (these codes are durable, documented vocabulary — keep the code,
+just add the doc-link docstring):
+- The **`E{n}` lineage-failure taxonomy** (`test_e27_*` etc.) — `E{n}` *is* the
+  shared name; renaming it loses meaning. Link the taxonomy doc in the docstring.
+- **CLAUDE.md-pinned regression guards** (`test_T09_01_qualify_once.py`,
+  `*_invariant.py`) are referenced *by filename* in `CLAUDE.md`. Do not rename the
+  file without updating CLAUDE.md in lockstep; prefer leaving the filename and giving
+  the test functions behavioral names + docstrings.
+
+**Rename-on-touch, not big-bang.** Do not open a PR that mass-renames existing tests.
+When you next implement in or edit a test (or its area), bring the tests you touch up
+to this convention as part of that change. New tests follow it from the start.
+
 ## Performance invariants — MUST preserve
 
 `src/sqlcg/parsers/base.py` contains hard-won performance fixes. Before touching this

{sql_code_graph-1.5.0 → sql_code_graph-1.33.0}/.claude/agents/sprint-planner.md

@@ -168,6 +168,11 @@ critical finding before shipping. Place it in `## Test Strategy` under
 - Include a failure message that names the original regression
   (e.g., `"Column lineage edges must be non-zero — guards against v0.3.0 regression"`)
 
+> Refer to tests by the **behavior** they verify, not by a task/case code (`TC1`,
+> `T-03`, …). The developer names tests `test_<unit>_<scenario>_<expected>` and links
+> the plan in the docstring — see *Test naming* in
+> [`developer.md`](.claude/agents/developer.md). Don't seed codes into "Tests to add".
+
 ## Sprint-Level Acceptance Criteria — Always Required
 
 End every plan with `## Acceptance Criteria (sprint-level)` — a checklist of

{sql_code_graph-1.5.0 → sql_code_graph-1.33.0}/.github/workflows/release.yml

@@ -4,6 +4,11 @@ on:
   push:
     tags: ["v*"]
   workflow_dispatch:
+    inputs:
+      ref:
+        description: "Tag or commit to build and publish (e.g. v1.6.0). Leave blank to use the current HEAD of the default branch."
+        required: false
+        default: ""
 
 permissions:
   contents: write
@@ -14,6 +19,8 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.inputs.ref || github.ref }}
       - uses: astral-sh/setup-uv@v4
       - run: uv sync --all-extras
       - run: uv run pytest tests/unit tests/integration -v --tb=short
@@ -26,6 +33,8 @@ jobs:
     environment: pypi
     steps:
       - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.inputs.ref || github.ref }}
       - uses: astral-sh/setup-uv@v4
       - run: uv build
       - uses: pypa/gh-action-pypi-publish@release/v1

{sql_code_graph-1.5.0 → sql_code_graph-1.33.0}/.gitignore

@@ -65,6 +65,8 @@ tests/e2e/test_dwh_schema_e2e.py
 .claude/progress.txt
 columns.csv
 results_*.json
+# Large scratch data export (811k rows) — never commit
+anomaly.csv
 
 # Local-only golden lineage ground truth — names private DWH columns
 tests/e2e/golden_lineage.yaml

sql_code_graph-1.33.0/ARCHITECTURE_REVIEW.md

@@ -0,0 +1,364 @@
+# Architecture Review — sql-code-graph (`sqlcg`)
+
+**Current as of:** v1.25.0 · DuckDB backend · Python 3.12
+**Status:** living document — describes the system as it stands today.
+**History:** the pre-1.4 blueprint review, PR/issue reviews, and every sprint
+postmortem (§1–20 + the Kuzu→DuckDB migration review) are preserved verbatim in
+[`ARCHITECTURE_REVIEW_ARCHIVE.md`](ARCHITECTURE_REVIEW_ARCHIVE.md). This file was
+overhauled on 2026-06-08 to drop content made stale by the DuckDB migration and the
+v1.x releases. See [§8 History](#8-history) for the timeline.
+
+> **Link convention.** Links here use real repo paths (e.g.
+> [`base.py`](src/sqlcg/parsers/base.py)) so a rename breaks the link *visibly*. The
+> guard test [`test_doc_links.py`](tests/unit/test_doc_links.py) fails CI if any link
+> in a maintained doc points at a missing file.
+
+---
+
+## 1. What `sqlcg` is
+
+`sqlcg` indexes a corpus of `.sql` files into an embedded **DuckDB** graph and answers
+**lineage** questions over it — "what feeds this column?", "what breaks if I change this
+table?", "which views are downstream?". It parses SQL with `sqlglot`, persists tables /
+columns / queries / lineage edges as graph nodes and relationships, and exposes the graph
+two ways: a `typer` **CLI** (`sqlcg …`) and an **MCP server** for use from Claude.
+
+It targets both ends of a scale spectrum — a ~20-file ETL repo and a 1,000–1,600 file
+data warehouse — from the same zero-config `sqlcg index ./sql` entry point. The scale
+rules and performance budget live in [`CLAUDE.md`](CLAUDE.md); the invariants that enforce
+them are summarised in [§4](#4-performance-invariants).
+
+---
+
+## 2. Component map
+
+```mermaid
+flowchart TD
+    subgraph entry[Entry points]
+        CLI["CLI — typer<br/>cli/main.py"]
+        MCP["MCP server — FastMCP<br/>server/server.py + tools.py"]
+    end
+
+    subgraph index[Indexing pipeline]
+        WALK["walker — find .sql"]
+        POOL["pool — spawn workers<br/>+ hard-kill timeouts"]
+        PARSE["parsers — sqlglot<br/>per dialect"]
+        AGG["aggregator — 2-pass<br/>cross-file lineage"]
+        IDX["indexer — orchestrate<br/>+ batched bulk upsert"]
+    end
+
+    subgraph store[Graph store]
+        BACKEND[("DuckDBBackend<br/>7 node / 12 edge tables")]
+    end
+
+    subgraph readpath[Read / query]
+        ANALYZE["analyze / find queries<br/>recursive-CTE traversal"]
+    end
+
+    subgraph live[Liveness]
+        FRESH["freshness — indexed vs HEAD"]
+        WRITER["writer — single-writer drain"]
+        JOBS["jobs / watcher — file events"]
+    end
+
+    CLI --> IDX
+    CLI --> ANALYZE
+    MCP --> ANALYZE
+    MCP --> WRITER --> IDX
+    IDX --> WALK --> POOL --> PARSE --> AGG --> IDX
+    IDX --> BACKEND
+    ANALYZE --> BACKEND
+    FRESH -.-> BACKEND
+    JOBS --> WRITER
+```
+
+**Read path** (`analyze`, `find`, most MCP tools) and **write path** (`index`, `reindex`)
+both go through `DuckDBBackend`. When the MCP server is running, the CLI routes reads to it
+([`read_client.py`](src/sqlcg/server/read_client.py)) and serialises all writes through a
+single-writer drain ([`writer.py`](src/sqlcg/server/writer.py)) so there is exactly one
+writer per database.
+
+### 2.1 Responsibilities
+
+| Module | Job |
+|---|---|
+| [`cli/main.py`](src/sqlcg/cli/main.py) + [`cli/commands/`](src/sqlcg/cli/commands) | Typer CLI. Commands: `index`, `reindex`, `watch`, `report`, `gain`, `install`, `uninstall`; groups `db` (`init`/`reset`/`info`/`list-repos`), `find` (`table`/`column`/`pattern`), `analyze` (`upstream`/`downstream`/`impact`/`failures`/`unused`), `mcp`, `git` (`install-hooks`). |
+| [`server/server.py`](src/sqlcg/server/server.py) | FastMCP instance + stdio entry. Constructs `mcp` at module scope (after the stdout→stderr guard) so [`tools.py`](src/sqlcg/server/tools.py) can register tools onto it. |
+| [`server/tools.py`](src/sqlcg/server/tools.py) | The 18 MCP tools — `trace_column_lineage`, `get_upstream_dependencies`, `get_downstream_dependencies`, `diff_impact`, `find_table_usages`, `find_definition`, `search_sql_pattern`, `analyze_unused`, `get_hub_ranking`, `index_repo`, `execute_sql`, `get_empty_propagation`, `get_pr_impact`, etc. |
+| [`server/writer.py`](src/sqlcg/server/writer.py) | Single-writer queue + drain task. The drain holds `backend_lock` and runs the indexer op directly — the sole backend writer. |
+| [`server/read_client.py`](src/sqlcg/server/read_client.py) | Routes CLI read commands through the live server socket when one is running. |
+| [`server/control.py`](src/sqlcg/server/control.py) | `.pid` / `.sock` control files for server discovery, status, and stop — keyed off the db path. |
+| [`server/models.py`](src/sqlcg/server/models.py) · [`server/exceptions.py`](src/sqlcg/server/exceptions.py) | Pydantic return types for tools; the tool error hierarchy. |
+| [`server/noise_filter.py`](src/sqlcg/server/noise_filter.py) | Filters backup tables and schema-alias mirrors out of user-facing surfaces. |
+| [`server/skill.py`](src/sqlcg/server/skill.py) | Pure renderer for the bundled Claude skill (`SKILL.md`); the installer writes its output to disk. |
+| [`indexer/indexer.py`](src/sqlcg/indexer/indexer.py) | Orchestrates parse → persist. Two-pass index, batched bulk upsert (`_flush_row_batch`), atomic SHA stamping. |
+| [`indexer/walker.py`](src/sqlcg/indexer/walker.py) | Walks the corpus for `.sql` files honouring ignore patterns. |
+| [`indexer/pool.py`](src/sqlcg/indexer/pool.py) | Spawn-mode worker pool; each worker holds pass-1/pass-2 parsers. Per-task hard-kill timeout + respawn (what `concurrent.futures` can't do). |
+| [`indexer/dbt_adapter.py`](src/sqlcg/indexer/dbt_adapter.py) | Enriches `SchemaResolver` from a dbt `manifest.json`. |
+| [`indexer/git_delta.py`](src/sqlcg/indexer/git_delta.py) | Computes the changed-file set between two git SHAs for incremental reindex. |
+| [`indexer/watcher.py`](src/sqlcg/indexer/watcher.py) | `watchdog` file-event source for `sqlcg watch`. |
+| [`indexer/error_classify.py`](src/sqlcg/indexer/error_classify.py) | Maps lineage-extraction failures to the `E{n}` taxonomy for measurement/reporting. |
+| [`lineage/aggregator.py`](src/sqlcg/lineage/aggregator.py) | `CrossFileAggregator` — pass-1 source harvest, pass-2 cross-file lineage resolution. |
+| [`lineage/schema_resolver.py`](src/sqlcg/lineage/schema_resolver.py) | `SchemaResolver` — table/view/column resolution with a lock-guarded cache. One instance per re-index job (never shared across threads). |
+| [`parsers/base.py`](src/sqlcg/parsers/base.py) | Base data models + abstract parser; column-lineage extraction via `sg_lineage`. **Hot path — see [§4](#4-performance-invariants).** |
+| [`parsers/registry.py`](src/sqlcg/parsers/registry.py) | Dialect → parser factory. |
+| [`parsers/{ansi,snowflake,bigquery,postgres,tsql}_parser.py`](src/sqlcg/parsers) | Per-dialect parsers; Snowflake carries the scripting-block / gap handlers. |
+| [`core/duckdb_backend.py`](src/sqlcg/core/duckdb_backend.py) | `DuckDBBackend` — the graph store. 7 node + 12 edge tables, bulk upsert via `unnest`, reentrant `transaction()`, `run_read`, recursive-CTE traversal SQL. |
+| [`core/graph_db.py`](src/sqlcg/core/graph_db.py) | `GraphBackend` ABC. Single-impl today (see [§5](#5-active-findings)); kept for the v2 second-source path. |
+| [`core/schema.py`](src/sqlcg/core/schema.py) | Node/edge labels + `SCHEMA_VERSION` (the schema gate). |
+| [`core/queries.py`](src/sqlcg/core/queries.py) | Loader for query strings in `queries.sql`. |
+| [`core/config.py`](src/sqlcg/core/config.py) | `DbConfig.from_env()` — resolves the db path and settings. |
+| [`core/freshness.py`](src/sqlcg/core/freshness.py) | Backend-free indexed-SHA-vs-HEAD delta; unit-testable with only a git repo. |
+| [`core/jobs.py`](src/sqlcg/core/jobs.py) | Watch job manager — debounced reindex on file-change events. |
+| [`metrics/store.py`](src/sqlcg/metrics/store.py) | SQLite metrics store; opt-out via `SQLCG_METRICS=0`. |
+
+---
+
+## 3. Key flows
+
+### 3.1 Indexing (`sqlcg index`)
+[`walker`](src/sqlcg/indexer/walker.py) enumerates `.sql` files →
+[`pool`](src/sqlcg/indexer/pool.py) farms them to spawn workers → each
+[`parser`](src/sqlcg/parsers/base.py) produces tables/columns/queries and per-statement
+column lineage. The [`aggregator`](src/sqlcg/lineage/aggregator.py) runs **two passes**:
+pass-1 harvests CTE/CTAS bodies and DDL into the `SchemaResolver`; pass-2 reparses with a
+`dependency_filter` to resolve cross-file lineage. [`indexer`](src/sqlcg/indexer/indexer.py)
+accumulates rows across a batch and issues **one bulk upsert per node/edge label per batch**
+into [`DuckDBBackend`](src/sqlcg/core/duckdb_backend.py), stamping the indexed SHA inside the
+same transaction.
+
+### 3.2 Query / lineage (`sqlcg analyze`, MCP tools)
+Lineage traversal is a recursive CTE in DuckDB with a depth cap + path-based cycle guard.
+A kind-filter (`WHERE kind IS NULL OR kind IN ('table','external')`) keeps CTE/derived
+scratch nodes out of user-facing surfaces. Up/down direction differ only in which
+`COLUMN_LINEAGE` edge endpoint binds the result's `file:line`.
+
+> **Mental model — CTE/derived edges are structural, not coverage gaps.** For
+> `INSERT INTO t (cols) WITH cte … SELECT … FROM cte`, the parser emits two *different,
+> both-correct* edge families: (1) **CTE chain edges** where the CTE node is the `dst`
+> (`src.x → cte.col`, from the CTE-projection block) and (2) **positional edges** where the
+> CTE is the `src` and the real target is the `dst` (`cte.col → t.col`, from the `#25`
+> block). Multi-hop **upstream** traversal walks *both* to chain `t.col ← cte.col ← src.x`.
+> A `kind='cte'`/`kind='derived'` node is a scratch intermediate, never persistent schema,
+> so it can **never** carry a `HAS_COLUMN` row — but it is *required* in the edge set for
+> traversal. Suppressing family (1) (an earlier "P1 de-leak" attempt) dead-ends upstream
+> traversal at the CTE node and was reverted; the canary is
+> [`test_analyze_case_fold`](tests/integration/test_analyze_case_fold.py)
+> (`test_uppercase_upstream_anchor_returns_same_ids_as_lowercase`).
+
+> **Coverage-metric formula — edge health is scoped to real-table destinations.** The
+> `edge_health` and `blindspot` metrics in [`coverage.py`](src/sqlcg/cli/coverage.py)
+> count a `COLUMN_LINEAGE` edge as "good" only when its stripped dst table has a
+> `HAS_COLUMN` entry. Because CTE/derived dst edges can never be catalogued (above), they
+> are **excluded from both numerator and denominator** via
+> `WHERE EXISTS (SELECT 1 FROM "SqlTable" t WHERE t.qualified = <dst_table strip> AND
+> t.kind NOT IN ('cte','derived'))`. This mirrors the analyze-surface kind-filter: scratch
+> nodes are scored out of coverage exactly as they are filtered out of user surfaces.
+> Counting them (the pre-2026-06 formula) made ≥95 % edge health mathematically impossible
+> — ~29 % of all edges legitimately point at CTE/derived intermediates.
+
+### 3.2b INFORMATION_SCHEMA catalog enrichment (PR 2, 2026-06-10)
+
+**Why the 2026-05 prohibition is lifted.** The 2026-05 removal of INFORMATION_SCHEMA
+CSV ingestion was based on a "zero edge delta" measurement — true but wrong-axis: schema
+never creates edges, it makes edge *targets resolvable* (`HAS_COLUMN`). The strict
+column-level health metric that shows the win didn't exist until v1.6.1 (PR 1).
+
+Measured root cause (2026-06-10, DWH, ~69k edges): 94 % of all bad edges (18,244/19,438)
+come from ETL files writing to tables whose DDL lives in Liquibase **XML** changelogs
+(not indexed). An INFORMATION_SCHEMA export covers those tables directly, falsifying
+wrong positional guesses (contradicted bucket) rather than hiding them.
+
+**Design rule (replaces the old prohibition):**
+- `sqlcg catalog load <csv>` loads `INFORMATION_SCHEMA.COLUMNS` into the graph as
+  `HAS_COLUMN(source='information_schema')` rows **post-index** (never at parse time).
+- Precedence: `ddl > information_schema > usage` — enforced in `upsert_edges_bulk`
+  via `ON CONFLICT (src_key, dst_key) DO UPDATE SET source = CASE … END`.
+- **Parse-time schema feeding remains forbidden.** Passing `schema_sources` into
+  `exp.expand()` or `qualify()` is still prohibited (the hot-loop path that measured
+  zero and regressed performance in 2026-05).
+- Per-file reindex (`delete_nodes_for_file`) is **source-aware**: only `ddl`/NULL rows
+  are deleted; `information_schema` and `usage` rows survive incremental reindex.
+- Config persistence: `[sqlcg.catalog] path = "..."` in `.sqlcg.toml` causes
+  `index_repo` to re-apply the catalog at the end of every full rebuild (zero parse-time
+  cost, one-shot bulk upsert outside all hot loops).
+
+### 3.3 MCP server + single-writer model
+[`server.py`](src/sqlcg/server/server.py) runs FastMCP over stdio. Writes (`index`,
+`reindex`) are enqueued and drained by [`writer.py`](src/sqlcg/server/writer.py) under
+`backend_lock` — one writer per db. The full-rebuild drain wraps `clear_all_tables()` +
+`index_repo()` in one reentrant transaction, so a mid-rebuild failure rolls back to the
+prior graph. Reads currently share that lock (see [§5](#5-active-findings)).
+
+### 3.4 Freshness / reindex / watch
+[`freshness`](src/sqlcg/core/freshness.py) compares the stored `indexed_sha` to git HEAD;
+[`git_delta`](src/sqlcg/indexer/git_delta.py) yields the changed files; `reindex` /
+[`jobs`](src/sqlcg/core/jobs.py) + [`watcher`](src/sqlcg/indexer/watcher.py) drive
+incremental updates (`git install-hooks` wires this to commits).
+
+---
+
+## 4. Performance invariants
+
+These are **not duplicated here** — the canonical table (parser hot-path rules, bulk-upsert
+and batch-flush invariants, the named regressions each guards) lives in
+[`CLAUDE.md`](CLAUDE.md) under *Performance invariants — DO NOT REMOVE OR SIMPLIFY*. They
+are pinned by [`test_perf_scaling_guard.py`](tests/unit/test_perf_scaling_guard.py),
+[`test_bulk_upsert_invariant.py`](tests/unit/test_bulk_upsert_invariant.py),
+[`test_upsert_batch_invariant.py`](tests/unit/test_upsert_batch_invariant.py), and
+[`test_T09_01_qualify_once.py`](tests/unit/test_T09_01_qualify_once.py). Any refactor of
+[`base.py`](src/sqlcg/parsers/base.py) or [`indexer.py`](src/sqlcg/indexer/indexer.py) must
+keep them green.
+
+---
+
+## 5. Active findings
+
+Only genuinely-open items live here. Resolved findings and their reasoning are in the
+[archive](ARCHITECTURE_REVIEW_ARCHIVE.md).
+
+| # | Sev | Finding | Where | Plan |
+|---|-----|---------|-------|------|
+| F1 | LOW | **Scratch-object leak into the default `analyze` surface.** ~22% of `kind='table'` nodes carry `tmp_*`/`temp_*` prefixes and leak into filtered output. Fix belongs at **classification time** (emit `kind='derived'` for TEMP/TRANSIENT CTAS targets), never in the name-blind filter. | [`aggregator.py`](src/sqlcg/lineage/aggregator.py) `_build_file_rows`, [`analyze.py`](src/sqlcg/cli/commands/analyze.py) | none yet |
+| F2 | LOW | **RESOLVED (v1.28.2, PR #119+#120)** `run_read` now uses `self._conn.cursor()` for a per-call snapshot; the read dispatch path in `server.py` no longer acquires `backend_lock`. Reads during a rebuild see the last committed graph immediately. Drain (writer.py:330) still holds `backend_lock` for the full write transaction. Proven by `test_write_memory_ceiling.py` Scenario A (read returns in <1 s with non-empty result while write transaction open). | [`duckdb_backend.py`](src/sqlcg/core/duckdb_backend.py) `run_read`, [`server.py`](src/sqlcg/server/server.py) `query` op | sprint_backlog_q2_bugfix §PR-E |
+| F3 | LOW | **RESOLVED (v1.28.2, PR #119+#120)** 1,600-file write-memory ceiling measured and verified. Peak RSS in a dedicated subprocess (1,600 minimal SQL fixtures, batch_size=50): **0.154 GiB (165 MB)** — well under the 3.8 GiB gate. Pinned by [`test_write_memory_ceiling.py`](tests/integration/test_write_memory_ceiling.py) Scenario B. | [`indexer.py`](src/sqlcg/indexer/indexer.py), [`duckdb_backend.py`](src/sqlcg/core/duckdb_backend.py) | sprint_backlog_q2_bugfix §PR-E |
+| F4 | LOW | **Single-impl ABC + stale Cypher/Kuzu docstrings.** `GraphBackend` adds indirection without polymorphism now; `clear_all_tables` is a `NotImplementedError`-default on the ABC; a docstring still says "skipped by KuzuDB's MERGE semantics". De-Cypher the docstrings and relocate `clear_all_tables` off the ABC, or collapse the ABC if v2 ACCESS_HISTORY is shelved. | [`graph_db.py`](src/sqlcg/core/graph_db.py) | none yet |
+| F5 | LOW | **TC6b guard reconstructs the query.** The terminal-sink test rebuilds the downstream Cypher/SQL string instead of running `analyze downstream` end-to-end, so it can drift from the shipped query. Replace with a `CliRunner`-based guard. | [`test_user_surface_recall_guard.py`](tests/integration/test_user_surface_recall_guard.py) | none yet |
+| F6 | LOW | **Dead `_set_backend_lock` stub.** Defined but never called (the escalation plumbing it belonged to was deleted in the DuckDB migration). Violates the CLAUDE.md "every new method has a grep-confirmed call site" rule — remove it. | [`tools.py`](src/sqlcg/server/tools.py) | none yet |
+| F7 | MED | **95 % edge-health milestone is ~30 points short after P1/P3/P4.** With the scoped metric (CTE/derived excluded) the DWH baseline is ≈41 %; P3 (+6,045 edges) + P4 lift it to ≈55–65 %. Reaching ≥95 % needs catalogs for ~566 residual **real-table** blindspots — the P2 cohort (temp / cross-file / unseen-DDL tables) and the deferred P5 `USE SCHEMA` bare-name group (956 tables). **RESOLVED (GitHub #124, PR fix/coverage-metric-baselining)**: 95 % numeric gate dropped; coverage is now reported as an observed metric with no pass/fail target. BQ-2 is the follow-up sprint. | [`coverage.py`](src/sqlcg/cli/coverage.py), [`base.py`](src/sqlcg/parsers/base.py) | [coverage_p1_p3_p4](plan/sprints/coverage_p1_p3_p4.md) §95 % Feasibility Analysis |
+| F8 | MED | **Index-summary metric undercount masks coverage gaps.** End-of-`sqlcg index` prints "1 failed" while 410 files carry `parse_failed=True` in the live DWH (1,335 files indexed, v1.25.0). The printed count reflects only the final-phase summary, not the full parse-failure population. Users and agents see a misleadingly-healthy finish line. Being addressed in the coverage sprint. | [`indexer.py`](src/sqlcg/indexer/indexer.py) summary path | `plan/sprints/coverage_parse_failures.md` (pending creation) |
+| F9 | LOW | **`pr-impact` rename suppression renders no "verify consumers updated" label.** The help text promises a dedicated labelled section when a rename is detected with consumers updated; the actual output emits only "No genuine lost producers detected" with no rename callout. Suppression logic is correct; the label is missing. | [`server/tools.py`](src/sqlcg/server/tools.py) `get_pr_impact` | none yet |
+| F10 | LOW | **`git install-hooks` embeds venv path, not user-installed binary.** Hooks installed via `sqlcg git install-hooks` point at `.venv/bin/sqlcg`, not `~/.local/bin/sqlcg`. If the project venv is rebuilt or deleted, the hooks become stale silently while the user-installed PyPI copy is current. Observed live on v1.25.0 (both pointed at 1.25.0 coincidentally — but this is a version-skew surface). | [`indexer/`](src/sqlcg/indexer/) install path | none yet |
+| F11 | LOW | **`mcp restart` is stop-only by design but misnamed.** The command stops the server and instructs the user to reconnect via `/mcp`; it does NOT restart the server process. For a stdio MCP server this is architecturally correct (the client owns the lifecycle), but the name and the live-verify expectation are contradicted by the observed behaviour. Either rename to `mcp stop` or document prominently in `--help` that the client must reconnect. Honest reframe shipped in v1.20.0; the naming gap remains. | [`server/control.py`](src/sqlcg/server/control.py), [`cli/commands/`](src/sqlcg/cli/commands) | none yet |
+| F12 | MED | **`pr-impact` requires graph to be indexed at base-ref; not self-healing.** Each `pr-impact --base X` run leaves the graph indexed at HEAD. A second run fails with a hint to manually reindex. Worse, the installed git post-checkout/post-merge hooks fire a background `sqlcg reindex` on every branch switch, racing `pr-impact` off the base. Reproduced on the live DWH (2026-06-13). Working protocol: chain `reindex --from <current_sha> --to <base>` immediately before `pr-impact`. A self-healing reindex-to-base step inside `pr-impact` itself is the right fix. | [`server/tools.py`](src/sqlcg/server/tools.py) `get_pr_impact`, [`indexer/git_delta.py`](src/sqlcg/indexer/git_delta.py) | backlog (deferred post data-loss-impact sprint) |
+
+---
+
+## 6. DWH graph topology insights (2026-06-13, v1.25.0)
+
+Source: [`plan/reports/dwh_graph_analysis.md`](plan/reports/dwh_graph_analysis.md) —
+full tables, method, and candidate lists live there; this section captures the
+actionable architectural findings only.
+
+Graph: **2,030 nodes · 2,930 directed edges** (table-level, noise-filtered), built from
+1,335 files, 52,514 column-lineage edges (schema v8, DWH corpus, 2026-06-13).
+
+### 6.1 Centrality — VALIDATED
+
+Ground-truth check: `ba.wtdh_artikel` (degree rank #3, total=107) and
+`ba.wtfe_verkoopinfo` (rank #6, total=57) independently reproduced as top hubs.
+**Out-degree / total-degree is the right blast-radius metric** — it answers "what breaks
+the most downstream if this table is wrong". PageRank is the wrong metric here: it ranks
+hubs low because they are *sources*, not sinks.
+
+The warehouse is anchored on five conformed dimensions (store, banner, article, date,
+week) and one dominant fact hub (verkoopinfo) — exactly the expected retail star-schema
+shape.
+
+**Surprising high-centrality tables** (not obvious from name alone):
+`da.ttint_verkooptransactie` (PageRank #4 — a quiet integration staging table is the
+single biggest convergence point with in=25), `ba.wtfe_korting` (PageRank #1 — discount
+logic far more central than its name suggests), `ba.wtfi_axi_ail` +
+`ba.wtfe_artikel_inteken_lijst` (intake-list pipeline dense hub, not a leaf report).
+
+### 6.2 Named single-points-of-failure (high betweenness bridges)
+
+These four permanent tables sit on the most shortest-paths; an outage in any of them
+**fragments lineage across subject areas**. Directly actionable for outage planning.
+
+| Table | Betweenness | Role |
+|---|---|---|
+| `ba.wtfe_verkoopinfo` | 0.00372 | #1 bridge AND top hub (sales fact) |
+| `da.ttint_verkooptransactie` | 0.00340 | Transaction integration bridge (in=25 → upstream of sales + promotions) |
+| `ba.wtfe_promotie_artikel_bouwmarkt` | 0.00175 | Bridges promotions ↔ sales ↔ store |
+| `ba.wtfv_voorraad_dagstand` | 0.00120 | Inventory daily-snapshot bridge |
+
+Betweenness floats are meaningful only as a ranking; computed on the 2,030-node
+filtered DiGraph.
+
+### 6.3 Communities
+
+171 communities, modularity **0.763** (strong, well-separated). Communities map
+cleanly onto retail subject areas. Two notable anomalies:
+
+- `ba.wtfe_loyalty` sits in the **Webshop** community (#32), not a
+  customer/CRM community — candidate for a real coupling or mis-placement.
+- **HR/workforce fused into the Store community (#4)** — staffing data has no independent
+  lineage island; it only exists hanging off the store dimension.
+
+Cross-community bridge edges concentrate on the same four betweenness tables (§6.2) —
+confirming that any refactor touching those tables has the widest blast radius.
+
+### 6.4 Orphan / satellite belt
+
+~4,417 SqlTable nodes have zero table-level lineage edges. The headline is **not
+"4,000 dead tables"**: that population is dominated by parse-failure coverage gaps
+(410 parse-failed files) + targetless write queries (1,090 of 2,349) + downstream
+export mirrors (`ia_*` schemas) — not genuinely standalone business tables. Only ~57
+clean-parse standalone DDL tables are confidently business isolates.
+
+This orphan belt is the main driver of the coverage sprint (F8 +
+`plan/sprints/coverage_parse_failures.md` — being produced separately).
+
+### 6.5 Tier-2 "refactor advisor" candidate (feature opportunity)
+
+The `wtfv_`/`wtfs_` prefix-split pattern surfaces as a recurring topology signal: 7
+ranked candidate groups where a fact-view and a fact-snapshot table share J=1.0 leaf
+sets and 8–71 shared columns (see the full table in
+[`plan/reports/dwh_graph_analysis.md`](plan/reports/dwh_graph_analysis.md) §Lens 4).
+The strongest candidate is the `*voorraad_dagstand*` inventory family (3 tables, J=1.0,
+33 shared leaves, 33–71 shared columns).
+
+**This is a topology + column-name hypothesis, NOT a confirmed refactor recommendation.**
+The `manus_export` false positive (J=1.0, 50 shared leaves, only 5 shared columns of 84)
+proves same inputs ≠ same logic. Confirming evidence a tier-2 pass needs before any
+prescriptive recommendation:
+
+- **(a) SQL-expression comparison** — prove the *logic* (not just the inputs) is duplicated.
+  Compare actual SELECT projections / WHERE / join logic per `SqlQuery.sql`.
+- **(b) Shared grain / PK** — a conformed hub is only legal if the candidates share a
+  grain. PK/grain metadata is **not in the graph today**.
+
+Record as a feature opportunity. Do not implement until (a) and (b) are available.
+
+---
+
+## 7. Deferred / roadmap
+
+- **Coverage sprint — parse-failure diagnosis (active).** 410/1,335 files
+  parse-failed on the live DWH, dominated by error classes **E5 (217)** and **E8 (184)**;
+  only 1 timeout. This is the dominant driver of the satellite/orphan belt (§6.4) and the
+  F7/F8 findings. Diagnosis: [`plan/research/coverage_parse_failure_diagnosis.md`](plan/research/coverage_parse_failure_diagnosis.md);
+  sprint plan: `plan/sprints/coverage_parse_failures.md` (being produced separately).
+- **v2 — Snowflake `ACCESS_HISTORY` as a second lineage source.** Fuse query-history lineage
+  to *enrich* (not replace) static parsing. This is the main reason the `GraphBackend` ABC
+  (F4) is kept rather than collapsed.
+- **ETL session-context limits (fundamental).** Cross-session temp tables, `COPY INTO`
+  column mapping, and dynamic `IDENTIFIER()` references are static-analysis limits — they
+  need runtime information. Intra-file temp-table chains resolve via pass-1 DDL sniffing;
+  cross-file/cross-session chains do not. Full analysis in the
+  [archive §6](ARCHITECTURE_REVIEW_ARCHIVE.md).
+- **Incremental reindex** (vs. full rebuild) — the real fix for F2/F3's lock-duration and
+  memory pressure; explicitly deferred during the DuckDB migration.
+- **PyPI publish pipeline — batch-push root cause resolved; re-publish backlog pending.**
+  PyPI is stuck at v1.5.1 because tags v1.6.0–v1.14.2 were pushed as a batch (GitHub
+  Actions silently drops triggers for all but ~3 tags in a single push). The v1.14.2
+  `workflow_dispatch` run also failed its `test` gate. A `workflow_dispatch` re-publish
+  path exists (`docs/releasing-pypi.md`); actual tagging and per-tag republishing are
+  user-gated. v1.15.0–v1.25.x are not yet tagged or published.
+
+---
+
+## 8. History
+
+Earlier content is preserved verbatim in
+[`ARCHITECTURE_REVIEW_ARCHIVE.md`](ARCHITECTURE_REVIEW_ARCHIVE.md). Detailed per-release
+plans live in [`plan/`](plan) (a frozen historical archive — see
+[`plan/WORKFLOW.md`](plan/WORKFLOW.md) for the agent pipeline that produced them).
+
+| Era | What it covered |
+|---|---|
+| Pre-1.4 blueprint review (archive §1–7) | Original design review, ETL-lineage gap analysis, open questions. |
+| PR/issue reviews (archive §8, §10) | PR #1 inline comments; issues #5/#6 LLM-experience feedback. |
+| v0.3–v1.3 sprint postmortems (archive §11–20) | Column-lineage diagnostics, the `E{n}` taxonomy, bulk-upsert/perf work, freshness + single-writer queue, recall guards. |
+| Kuzu → DuckDB migration (archive, v1.4.0) | Backend swap, perf-invariant re-verification, the concurrency-model deviation that became F2. |

sql_code_graph-1.5.0/ARCHITECTURE_REVIEW.md → sql_code_graph-1.33.0/ARCHITECTURE_REVIEW_ARCHIVE.md

@@ -1,4 +1,13 @@
-# Architecture Review — sql-code-graph (sqlcg)
+# Architecture Review — ARCHIVE (sql-code-graph)
+
+> **Frozen historical record.** This is the pre-overhaul `ARCHITECTURE_REVIEW.md`,
+> captured verbatim on 2026-06-08 (through the v1.4.0 Kuzu→DuckDB migration). It is **not
+> maintained** — links and findings reflect the repo as it was when each section was
+> written. For the current architecture see
+> [`ARCHITECTURE_REVIEW.md`](ARCHITECTURE_REVIEW.md). Kept for the decision/postmortem
+> trail; nothing below should be treated as current.
+
+---
 
 Blueprint version: v1.2 (May 2026)
 Review date: 2026-05-02 (updated 2026-05-06)

{sql_code_graph-1.5.0 → sql_code_graph-1.33.0}/CHANGELOG.md

@@ -1,3 +1,36 @@
+## v1.25.0 (2026-06-13)
+
+> **Re-index recommended (no schema change).** `SCHEMA_VERSION` unchanged.
+> View classification (`kind='view'`) and the `gating_join_tables` supplement only
+> take effect on a fresh `sqlcg index <path>`.
+
+### Feat
+
+- **view classification**: `CREATE VIEW` targets now stamped `kind='view'` so they
+  persist as `SqlTable.kind='view'` instead of being indistinguishable from tables;
+  live-verified on a ~1,400-file DWH (582 views classified)
+- **empty-impact** (`sqlcg analyze empty-impact` / `get_empty_propagation` MCP tool):
+  downstream blast radius when named table(s) are empty — two views: value-derivation
+  (primary, column-lineage refinement) and row-reachability (supplement, direct
+  gating-join reads)
+- **pr-impact** (`sqlcg analyze pr-impact` / `get_pr_impact` MCP tool): detect
+  producers a PR dropped + their blast radius; code-regression detection (not runtime
+  monitoring); rename-handling via column-set AND consumer Jaccard ≥ 0.6
+- **gating-join supplement** (`get_change_scope` / `diff_impact`): `gating_join_tables`
+  field added — tables that row-empty via direct gating-join reads supplementing the
+  column-lineage blast radius; caveat: depends on join type (not recorded on edges)
+- **3-signal unused** (`analyze_unused` / `sqlcg analyze unused`): redefines "used"
+  as the union of (D1) direct `SELECTS_FROM`, (D2) `STAR_SOURCE`, and (D3)
+  column-lineage CTE-derived read; `find_table_usages` augmented with `via_lineage`
+  entries for the same D3 signal
+
+### Fix
+
+- **pr-impact BUG-A**: CLI path was not initialising the backend before calling
+  `_compute_pr_impact`, causing `RuntimeError: Backend not initialized`
+- **pr-impact BUG-B**: blast-radius engine was zeroing the result when the resync
+  produced no new edges; now computed on the pre-resync graph (Option i)
+
 ## v1.1.3 (2026-06-01)
 
 > **Re-index recommended (no schema change).** `SCHEMA_VERSION` stays `6`, so no
@@ -38,7 +71,7 @@
 > `SCHEMA_VERSION` 4 to 6 (added `start_line` on `SqlQuery`, repurposed
 > `SqlTable.kind`, added the `ExternalConsumer` node + `CONSUMED_BY` edge). There
 > is no in-place migration — re-run `sqlcg index <path>` to rebuild the graph. If
-> you run the MCP server, restart it (`sqlcg mcp restart`) after re-indexing.
+> you run the MCP server, stop it (`sqlcg mcp stop`) and reconnect via your editor after re-indexing. (Note: as of v1.20.0 the server self-heals on the next tool call — no manual stop needed.)
 
 ### Feat