PyPI - sql-code-graph - Versions diffs - 1.0.2__tar.gz → 1.1.0__tar.gz - Mend

sql-code-graph 1.0.2tar.gz → 1.1.0tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (416) hide show

{sql_code_graph-1.0.2 → sql_code_graph-1.1.0}/ARCHITECTURE_REVIEW.md RENAMED Viewed

@@ -2827,3 +2827,124 @@ removes the skill, mirroring the F1 sentinel teardown.
   persists the prerequisite `indexed_sha`.
 - `max_closure_depth=3` is a reasoned default, not yet calibrated against real
   DWH chain depth — revisit if a deep chain is observed under-resolving.
+---
+## 17. v1.1.0 — Live Graph Freshness & Daemon Reindex (Cluster A)
+Review date: 2026-05-31 · Reviewer: architect-reviewer · Branch:
+`fix/v1.0.2-bugfix` · Verdict: **APPROVED-WITH-FOLLOWUPS**
+Resolves the §16.3 deferred staleness FACT and closes the silent-stale-graph
+class of failure. Plan: [`v1.1.0_live_graph_freshness.md`](plan/v1.1.0_live_graph_freshness.md);
+sprint breakdown: [`sprint_12_v1.1.0.md`](plan/sprint_12_v1.1.0.md). GitHub
+issues [#28], [#29], [#30]. Four PRs (21ecd2e, abcf41c, e3bead4/add000d/f02b279,
+b54153f), all merged on the branch.
+### 17.1 Freshness model (#30)
+A new backend-free helper [`core/freshness.py`](src/sqlcg/core/freshness.py)
+computes a `Freshness(indexed_sha, head_sha, stale_by_commits, dirty, branch)`
+from a repo root + the stored SHA. All git calls funnel through one private
+`_git(root, *args)` that returns `None` on any failure — **never raises**
+(matches the defensive pattern in `git_delta.py`/`watcher.py`). `stale_by_commits`
+is `git rev-list --count <indexed>..<head>`; an indexed SHA unknown to git
+(rebased/shallow, or the `+dirty` sentinel) yields `None` ("distance unknown"),
+never a false `0`. The helper does not import the backend — the caller passes
+`indexed_sha` in — so it unit-tests against a real temp git repo with no Kuzu.
+Surfaced in three places, all reusing the single helper:
+- `sqlcg db info` prints `render_freshness_line` (silent on empty/non-git DB).
+- MCP `db_info` tool extends `DbInfoResult` with the four optional fields;
+  Neo4j's `NotImplementedError` on `get_indexed_sha()` is guarded → reports
+  `indexed_sha: null` without crashing.
+- `mcp status` JSON (see 17.3).
+`index --include-working-tree` (#30, PR-B) folds uncommitted changes and
+overwrites the stored SHA with a `"<head>+dirty"` sentinel via a fresh backend
+context. The sentinel is intentionally **not** a valid SHA, so freshness reports
+"commit distance unknown, working tree dirty" — accurate rather than guessed.
+Default committed-only behaviour is unchanged.
+### 17.2 Daemon control-channel architecture (#29)
+The MCP server is a single process running one `anyio` event loop holding the
+KuzuDB write lock for its whole lifetime (no lock lease — verified no safe Kuzu
+API exists; a reader/writer split was explicitly rejected). v1.1 adds a
+**Unix-domain-socket control channel** on that same event loop, not a second
+process. Control-file paths derive from `get_db_path()` (`KuzuConfig`-owned),
+never hardcoded `~/.sqlcg`, so two servers on two DBs do not collide:
+`graph.db` → `graph.db.sock` + `graph.db.pid` (both `0o600`). See
+[`server/control.py`](src/sqlcg/server/control.py) (paths + pid record +
+`is_pid_alive` + cleanup) and `_control_socket_task` in
+[`server/server.py`](src/sqlcg/server/server.py).
+Protocol (newline-delimited JSON, one request → one response):
+- `status` → `{running, pid, db_path, indexed_sha, head_sha, stale_by_commits,
+  connected_clients, uptime}`. `connected_clients` is best-effort `1` (stdio
+  transport = one client by design; documented approximation, not a gate).
+- `stop` → replies `{"ok": true}`, sets `stop_event` (see 17.5).
+- `reindex {root, from, to, dialect}` → see 17.4.
+CLI side (`mcp status`/`stop`/`restart` in
+[`cli/commands/mcp.py`](src/sqlcg/cli/commands/mcp.py)) connect-and-probe the
+socket. R3 stale-socket handling: on `ConnectionRefusedError`/`FileNotFoundError`,
+fall through to the PID-file probe (`is_pid_alive`) — `running:true,
+degraded:"socket unavailable"` if the PID lives, `{"running": false}` otherwise.
+The socket task is guarded behind `sys.platform != "win32"`; Windows keeps the
+direct-write + staleness-cue path (IPC deferred to v1.2).
+### 17.3 Daemon-writer reindex (#28 — the real fix)
+The silent-stale-graph failure (a second process opening the locked DB,
+swallowed by the hook's `|| true`) is fixed by routing the delta **through the
+process that already holds the lock**. The `reindex` socket op calls
+`Indexer.resync_changed(root, from, to, db, dialect)` **unchanged** (positional
+match verified against [`indexer.py`](src/sqlcg/indexer/indexer.py) line 428),
+run via `anyio.to_thread.run_sync` (R1: off the event-loop thread so tool calls
+are not blocked) behind a single `anyio.Lock` (`reindex_lock`, R2: the Kuzu
+connection is not thread-safe — see finding 3.3; concurrent notifies serialise).
+`reindex --notify` ([`reindex.py`](src/sqlcg/cli/commands/reindex.py)) probes the
+socket and on any connect failure falls through to the existing direct-write path
+unchanged (R3). The git hooks ([`git.py`](src/sqlcg/cli/commands/git.py)) now pass
+`--notify` and emit a one-line **stderr cue** on failure while staying non-fatal
+to the checkout — killing the "silent" mode. Idempotency sentinels unchanged (R9).
+**Perf invariants intact (verified):** neither `parsers/base.py` nor
+`indexer.py` changed anywhere in Cluster A (`git diff --stat 21ecd2e~1 dcba863`
+on both paths is empty). The reindex op reuses `resync_changed`, so the
+bulk-upsert and once-per-statement-scope invariants are inherited; an integration
+test (`test_reindex_via_server.py` Scenario A) pins that `resync_changed` is
+called exactly once. DWH regression gate **PASSED**: 175s (≤240s budget),
+45,652 edges (≥44,939 baseline — no coverage regression).
+### 17.4 Deviation: `os._exit(0)` on stop (accepted for v1.1)
+The MCP `stdio_server` runs stdin `readline` in a thread via
+`anyio.to_thread.run_sync` with `abandon_on_cancel=False`. When stdin is a
+parent-held pipe, that blocking read **cannot be interrupted** from inside the
+subprocess (the parent holds the write end open), so a clean cancel-scope
+unwind cannot drain the thread. The `stop` path therefore does explicit cleanup
+**first** — `shutdown_backend()` then `cleanup_control_files()` in `_stop_watcher`
+([`server.py`](src/sqlcg/server/server.py) ~L218-255) — and then calls
+`os._exit(0)`. R8 (backend shutdown + socket removal before exit) holds because
+the teardown runs ahead of the hard exit, not in `main()`'s `finally` (which
+`os._exit` skips). This is **documented honestly, not faked, and contains no
+TODO in a happy path** (commit f02b279). Acceptable for v1.1.
+`restart` is `stop` + client-respawn guidance, **not** a true re-parent: the
+editor owns the stdio process lifecycle and v1.1 has no launcher to re-spawn it
+(R5). Also documented, not faked.
+### 17.5 Tracked for v1.2
+- **True graceful stop / true `restart`** — needs a launcher model that owns the
+  stdio process so the blocking-pipe-read can be unblocked and `os._exit(0)`
+  retired in favour of clean unwind (R5 + 17.4).
+- **Windows IPC** (named pipes) for lifecycle + notify-reindex (R7).
+- **Per-file / SQL-only dirty tracking** — v1.1 reports whole-tree dirty only.
+- **Socket auth token** for multi-user machines (R6: socket is `0o600`,
+  owner-only, but any same-uid local process can send `stop`/`reindex`).
+- **Freshness on every tool result by default** — v1.1 ships it gated/off to
+  protect hot-path latency; default-on pending an overhead measurement.
+- **`connected_clients` fidelity** — best-effort `1` today; no real counter.

sql_code_graph-1.1.0/CHANGELOG.md ADDED Viewed

@@ -0,0 +1,102 @@
+## v1.1.0 (2026-06-01)
+> **⚠️ Upgrade note — re-index required.** The graph schema advanced from
+> `SCHEMA_VERSION` 4 to 6 (added `start_line` on `SqlQuery`, repurposed
+> `SqlTable.kind`, added the `ExternalConsumer` node + `CONSUMED_BY` edge). There
+> is no in-place migration — re-run `sqlcg index <path>` to rebuild the graph. If
+> you run the MCP server, restart it (`sqlcg mcp restart`) after re-indexing.
+### Feat
+- **#28/#29/#30**: live-graph freshness & daemon reindex — Unix-socket control channel,
+  `mcp status/stop/restart`, reindex through the running server, post-merge hook routing
+  via `ORIG_HEAD`, `index --include-working-tree` + dirty sentinel, and a freshness signal
+  (indexed SHA vs HEAD) surfaced in `db info` / the `db_info` MCP tool
+- **#31**: source location — `file`/`line`/`expression` on lineage edges (`start_line` on `SqlQuery`)
+- **#32**: meaningful confidence — `1.0` for plainly-parsed facts, with a `reason` on inferred edges
+- **#33**: CLI/MCP output parity + node-kind tagging (`table` / `cte` / `derived` / `external`)
+- **#34**: `analyze_unused` segregates presentation/terminal tables from dead code
+- **#35**: external downstream lineage injection — `ExternalConsumer` node + `CONSUMED_BY` edge
+- **hygiene**: config-path footgun warning, find/analyze noise-filter parity, cause discoverability,
+  and MCP config-root reconciliation
+- **skill**: bundled Claude skill teaches the new provenance + node-kind fields (F2)
+- **indexer**: batch-scoped upsert — bulk writes once per batch instead of per file
+  (~13,400 → ~270 backend round-trips on the 1,340-file DWH corpus; upsert 169s → 52s)
+### Fix
+- **#31**: preserve newline count in Snowflake preprocessing (multi-line `start_line` bug)
+- **#28**: socket timeout fix, symbolic-ref `--to` resolution, hook binary path
+- **index**: resolve walker root once so relative paths yield absolute paths (sibling of #24)
+- **analyze_unused**: external-consumer warning clarifies the edge is created anyway;
+  debug log on the no-Repo File-node fallback
+### Perf
+- **indexer**: raise default per-file parse timeout 5s→10s
+## v1.0.2 (2026-05-30)
+### Fix
+- **parser**: restore COALESCE/func lineage + CTE-chain hops broken by eb19f29 (#25)
+- **observability**: make pool-path timeouts visible + route warnings to log file (#13/#27d PR-07)
+- **analyze**: query-time bare-name fallback for unqualified INSERT targets (#26 PR-06)
+- **analyze**: wire NoiseFilter into CLI + add *_bck_* glob (#27a/b PR-08)
+- **install**: restore uvx cold-cache message dropped in #23 rewrite
+- **parser**: preserve skip markers for INSERT-column-list expressions (#25 regression)
+- **parser**: INSERT positional mapping overrides SELECT alias attribution (#25)
+- **install**: write via claude mcp add + ~/.claude.json fallback, remove settings.json (#23)
+- **server**: capture fd 1 before _configure_mcp_logging to fix stdio transport (#22)
+- **find**: lowercase name/ref before graph query to match C2 normalization (#12)
+- **reindex**: resolve root to absolute before ignore matching (#24)
+## v1.0.1 (2026-05-30)
+### Fix
+- **indexer**: abort immediately on Ctrl-C instead of flushing a partial graph
+### Perf
+- **parser**: recover 3 once-per-statement lineage regressions from 4234e5d
+## v1.0.0 (2026-05-30)
+### Feat
+- **trust-layer**: `Judgement` discriminator (fact vs. heuristic) on risk/dead-code
+  outputs; `analyze_unused` + `get_hub_ranking` tools; golden-harness answer-anchors
+- **F1 living-codebase resync**: incremental branch-change resync via
+  `Indexer.resync_changed` + cross-file pass-2 affected closure; standalone
+  `sqlcg reindex`; `indexed_sha` metadata (schema v4); `post-merge` hook
+- **F2 bundled skill**: `sqlcg install` provisions a `SKILL.md` encoding the
+  fact/heuristic boundary; `sqlcg mcp best-practices` CLI command
+- **index**: `--profile` flag emitting per-stage and per-file timing
+- **analyze**: `failures` subcommand surfacing per-file parse-failure cause
+### Refactor
+- remove the INFORMATION_SCHEMA-CSV / `load-schema` path (zero measured edge delta)
+- compact the generated `SKILL.md` (~40% smaller, fact/heuristic-sorted tool table)
+## v0.2.1 (2026-05-05)
+### Fix
+- **benchmarks**: add bench_*.py to pytest python_files so bench_indexer.py is collected
+- **tests**: use tracked variable in fake_run and regenerate cli docs
+- **tests**: strip ANSI codes in test_cli_help assertion
+## v0.2.0 (2026-05-05)
+### Feat
+- **walker**: default to git ls-files to avoid indexing untracked files
+- **phase-9+10**: git-aware graph freshness + PyPI publishing
+- **phase-8**: metrics and feedback layer
+### Fix
+- **tools**: move inline Cypher queries to queries.py and fix decorator order (C1, C2)
+- **walker**: address code-review findings C2/W1/W2/W4/S1

{sql_code_graph-1.0.2 → sql_code_graph-1.1.0}/CLAUDE.md RENAMED Viewed

@@ -108,9 +108,11 @@ Any refactor that touches [`base.py`](src/sqlcg/parsers/base.py) or [`indexer.py
 | **`body_scope` is built once per statement BEFORE the column loop** (with a schema-free qualify retry on failure), never lazily inside it | `_extract_column_lineage` | Building it inside the loop re-ran expand+qualify+build_scope for *every* column whenever qualify failed (the failure path). Regressed in `4234e5d` |
 | In the INSERT column-list aliasing path, **`body.copy()` + strip-WITH happen once** before the loop; only the single projection is swapped per column | `_extract_column_lineage` INSERT block | A full-body deepcopy per column is O(cols × body_size). Regressed in `4234e5d` |
 | **`_upsert_parsed_file` uses `upsert_nodes_bulk`/`upsert_edges_bulk`** (Phase A→B→C), never `upsert_node`/`upsert_edge` per row | `indexer.py` `_upsert_parsed_file` | Measured: bulk=181s vs per-node=1020s+ on 1,340-file DWH (~10 execute() calls per file vs ~14,500 total). Commit `4234e5d` accidentally regressed this by rewriting the method during C2 normalization. |
+| **`_flush_row_batch` accumulates rows across all files in a batch and calls each `upsert_*_bulk` once per batch** (not once per file) | `indexer.py` `_flush_batch`/`_upsert_file_batch` → `_flush_row_batch` | Measured: per-file flush = ~13,400 `execute()` on 1,340-file DWH (10/file); per-batch flush = ~270 (10/batch). v1.1.1. Guard: [`test_upsert_batch_invariant.py`](tests/unit/test_upsert_batch_invariant.py) |
-Tests covering these invariants live in [`test_T09_01_qualify_once.py`](tests/unit/test_T09_01_qualify_once.py)
-and [`test_bulk_upsert_invariant.py`](tests/unit/test_bulk_upsert_invariant.py) (each pins
+Tests covering these invariants live in [`test_T09_01_qualify_once.py`](tests/unit/test_T09_01_qualify_once.py),
+[`test_bulk_upsert_invariant.py`](tests/unit/test_bulk_upsert_invariant.py), and
+[`test_upsert_batch_invariant.py`](tests/unit/test_upsert_batch_invariant.py) (each pins
 one *named* regression). If any fail after a refactor, the invariant was broken — do not mark
 the test as pre-existing and move on.

{sql_code_graph-1.0.2 → sql_code_graph-1.1.0}/PKG-INFO RENAMED Viewed

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sql-code-graph
-Version: 1.0.2
+Version: 1.1.0
 Summary: SQL code graph analyzer and lineage tracer
 Project-URL: Homepage, https://github.com/Warhorze/sql-code-graph
 Project-URL: Repository, https://github.com/Warhorze/sql-code-graph
@@ -219,7 +219,7 @@ After indexing, `sqlcg db info` shows non-zero `STAR_EXPANSION lineage edges`, a
 | **Search & meta** | |
 | `search_sql_pattern(query)` | Full-text search across indexed SQL |
 | `list_dialects_and_repos()` | List indexed repos and dialects (catalogue) |
-| `db_info()` | Graph health, node counts, parse quality breakdown, warnings |
+| `db_info()` | Graph health, node counts, parse quality breakdown, warnings, freshness (indexed SHA vs HEAD) |
 | `execute_cypher(query)` | Raw Cypher query against the graph |
 | `submit_feedback(...)` | Report a false positive/negative to improve metrics |
@@ -228,6 +228,12 @@ After indexing, `sqlcg db info` shows non-zero `STAR_EXPANSION lineage edges`, a
 > `table.column`. Each returned node carries both `name` (the bare column) and
 > `table` (the owning `schema.table`), so results are navigable without a second lookup.
+> **Provenance fields**: lineage edges now carry `file`, `line`, and `expression`
+> (where the lineage was derived from), a `confidence` of `1.0` for plainly-parsed
+> facts (lower for inferred edges, with a `reason`), and a `table_kind`
+> (`table` / `cte` / `derived` / `external`) so CTE and derived aliases are
+> distinguishable from real tables.
 > **LLM agent tip**: call `db_info()` before lineage queries to check that
 > `SqlColumn > 0` and `warnings` is empty. If `parse_quality["scripting_block"]`
 > is high, column lineage will be limited for those files — use table-level tools
@@ -243,16 +249,21 @@ sqlcg db init                          # initialise graph database
 sqlcg index <path> --dialect snowflake # index SQL files (snowflake is the tested dialect)
 sqlcg index <path> --dialect auto      # read dialect from .sqlcg.toml
 sqlcg index <path> --profile           # index + print per-stage timing and slowest files
+sqlcg index <path> --include-working-tree  # also index uncommitted changes (marks graph dirty)
 sqlcg reindex <path> --from <sha> --to <sha>  # incremental resync of only changed files
 sqlcg analyze unused                   # tables with no query references
 sqlcg analyze upstream/downstream      # trace lineage from the CLI
 sqlcg find table/column/pattern        # search the graph
 sqlcg watch <path>                     # watch for file changes
+sqlcg db info                          # graph stats + freshness (indexed SHA vs HEAD)
 sqlcg git install-hooks                # install post-checkout + post-merge resync hooks
 sqlcg gain                             # show usage metrics
 sqlcg report                           # generate FP/error report
 sqlcg mcp best-practices               # print the fact/heuristic boundary for the MCP tools
 sqlcg mcp start                        # start MCP server manually
+sqlcg mcp status                       # server status JSON (via control socket)
+sqlcg mcp stop                         # stop the running MCP server gracefully
+sqlcg mcp restart                      # stop the server (client must respawn it)
 sqlcg version                          # show installed version
 ```

{sql_code_graph-1.0.2 → sql_code_graph-1.1.0}/README.md RENAMED Viewed

@@ -181,7 +181,7 @@ After indexing, `sqlcg db info` shows non-zero `STAR_EXPANSION lineage edges`, a
 | **Search & meta** | |
 | `search_sql_pattern(query)` | Full-text search across indexed SQL |
 | `list_dialects_and_repos()` | List indexed repos and dialects (catalogue) |
-| `db_info()` | Graph health, node counts, parse quality breakdown, warnings |
+| `db_info()` | Graph health, node counts, parse quality breakdown, warnings, freshness (indexed SHA vs HEAD) |
 | `execute_cypher(query)` | Raw Cypher query against the graph |
 | `submit_feedback(...)` | Report a false positive/negative to improve metrics |
@@ -190,6 +190,12 @@ After indexing, `sqlcg db info` shows non-zero `STAR_EXPANSION lineage edges`, a
 > `table.column`. Each returned node carries both `name` (the bare column) and
 > `table` (the owning `schema.table`), so results are navigable without a second lookup.
+> **Provenance fields**: lineage edges now carry `file`, `line`, and `expression`
+> (where the lineage was derived from), a `confidence` of `1.0` for plainly-parsed
+> facts (lower for inferred edges, with a `reason`), and a `table_kind`
+> (`table` / `cte` / `derived` / `external`) so CTE and derived aliases are
+> distinguishable from real tables.
 > **LLM agent tip**: call `db_info()` before lineage queries to check that
 > `SqlColumn > 0` and `warnings` is empty. If `parse_quality["scripting_block"]`
 > is high, column lineage will be limited for those files — use table-level tools
@@ -205,16 +211,21 @@ sqlcg db init                          # initialise graph database
 sqlcg index <path> --dialect snowflake # index SQL files (snowflake is the tested dialect)
 sqlcg index <path> --dialect auto      # read dialect from .sqlcg.toml
 sqlcg index <path> --profile           # index + print per-stage timing and slowest files
+sqlcg index <path> --include-working-tree  # also index uncommitted changes (marks graph dirty)
 sqlcg reindex <path> --from <sha> --to <sha>  # incremental resync of only changed files
 sqlcg analyze unused                   # tables with no query references
 sqlcg analyze upstream/downstream      # trace lineage from the CLI
 sqlcg find table/column/pattern        # search the graph
 sqlcg watch <path>                     # watch for file changes
+sqlcg db info                          # graph stats + freshness (indexed SHA vs HEAD)
 sqlcg git install-hooks                # install post-checkout + post-merge resync hooks
 sqlcg gain                             # show usage metrics
 sqlcg report                           # generate FP/error report
 sqlcg mcp best-practices               # print the fact/heuristic boundary for the MCP tools
 sqlcg mcp start                        # start MCP server manually
+sqlcg mcp status                       # server status JSON (via control socket)
+sqlcg mcp stop                         # stop the running MCP server gracefully
+sqlcg mcp restart                      # stop the server (client must respawn it)
 sqlcg version                          # show installed version
 ```

sql-code-graph 1.0.2__tar.gz → 1.1.0__tar.gz

sql-code-graph 1.0.2tar.gz → 1.1.0tar.gz