sql-code-graph 1.35.3__tar.gz → 1.36.1__tar.gz

{sql_code_graph-1.35.3 → sql_code_graph-1.36.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sql-code-graph
-Version: 1.35.3
+Version: 1.36.1
 Summary: SQL code graph analyzer and lineage tracer
 Project-URL: Homepage, https://github.com/Warhorze/sql-code-graph
 Project-URL: Repository, https://github.com/Warhorze/sql-code-graph
@@ -85,7 +85,8 @@ for that project.
 
 1. **Initialize**: `sqlcg db init`
 2. **Index**: `sqlcg index ./sql --dialect snowflake`
-3. **Keep fresh**: `sqlcg git install-hooks` (optional)
+3. **Keep fresh**: `sqlcg git install-hooks` (auto-reindex on branch switch), or
+   `sqlcg watch ./sql` to re-index automatically as you edit files (both optional)
 
 ## Full setup (recommended)
 
@@ -303,7 +304,8 @@ sqlcg analyze empty-impact <table>...  # downstream blast radius when named tabl
 sqlcg analyze pr-impact --base <ref>   # detect producers a PR dropped + their blast radius (code-regression detection)
 sqlcg analyze upstream/downstream      # trace lineage from the CLI
 sqlcg find table/column/pattern        # search the graph
-sqlcg watch <path>                     # watch for file changes
+sqlcg watch <path>                     # watch a directory and re-index on SQL file changes
+sqlcg viz                              # generate a self-contained graph-explorer HTML
 sqlcg db info                          # graph stats + freshness (indexed SHA vs HEAD)
 sqlcg git install-hooks                # install post-checkout + post-merge resync hooks
 sqlcg gain                             # show usage metrics
@@ -316,7 +318,7 @@ sqlcg mcp restart                      # stop the server (client must respawn it
 sqlcg version                          # show installed version
 ```
 
-### Staying on the latest build (v1.5.0)
+### Staying on the latest build
 
 The installed package, the CLI, and the running MCP server all report the **same**
 version (`sqlcg.__version__`). After upgrading, an editor may still be talking to an
@@ -325,7 +327,7 @@ server's `version` and a `stale_by_version` flag that is `true` when the live se
 differs from the installed build. Re-running `sqlcg install` stops the stale server so
 your editor respawns it on the new build, so you never debug against an outdated server.
 
-### Reads while the server is running (v1.2.0)
+### Reads while the server is running
 
 DuckDB takes an exclusive lock on the database file, so while the MCP server is
 live it holds that lock (other processes cannot open the file, even read-only).
@@ -336,6 +338,34 @@ running they open the database directly, exactly as before. If the server is
 mid-reindex the read waits for it to finish rather than failing with
 "Database is locked".
 
+### Visualising the graph (`viz`)
+
+`sqlcg viz` generates a **self-contained graph-explorer HTML** from the live
+graph — every node, edge, the force-graph library, and the facet data are inlined
+into a single file, so it opens by double-click in a browser with no server or
+external resources.
+
+```bash
+sqlcg viz                              # writes table_graph.html in the current dir
+sqlcg viz --out lineage.html           # choose the output path
+```
+
+In the browser the filter composes schema ∩ kind ∩ tag (multi-select per facet)
+plus a job dropdown; an edge renders only when both endpoints are visible. Table,
+view, and temp nodes are on by default; CTE and derived nodes are off (toggleable).
+
+Two optional CSV facets let you colour and slice the graph by your own
+conventions — both are `pattern,label[,color]` with a required header, patterns
+are case-insensitive `fnmatch` globs (`*`, `?`) matched against the qualified
+table name:
+
+```bash
+sqlcg viz --tags tags.csv              # legend swatches: colour + filter the graph
+sqlcg viz --jobs jobs.csv              # populate the job dropdown (filter only)
+```
+
+Omit `--tags` to hide the tag legend; omit `--jobs` to leave the dropdown empty.
+
 ## Supported dialects
 
 sqlcg is built on [sqlglot](https://github.com/tobymao/sqlglot), so other dialects

{sql_code_graph-1.35.3 → sql_code_graph-1.36.1}/README.md

@@ -49,7 +49,8 @@ for that project.
 
 1. **Initialize**: `sqlcg db init`
 2. **Index**: `sqlcg index ./sql --dialect snowflake`
-3. **Keep fresh**: `sqlcg git install-hooks` (optional)
+3. **Keep fresh**: `sqlcg git install-hooks` (auto-reindex on branch switch), or
+   `sqlcg watch ./sql` to re-index automatically as you edit files (both optional)
 
 ## Full setup (recommended)
 
@@ -267,7 +268,8 @@ sqlcg analyze empty-impact <table>...  # downstream blast radius when named tabl
 sqlcg analyze pr-impact --base <ref>   # detect producers a PR dropped + their blast radius (code-regression detection)
 sqlcg analyze upstream/downstream      # trace lineage from the CLI
 sqlcg find table/column/pattern        # search the graph
-sqlcg watch <path>                     # watch for file changes
+sqlcg watch <path>                     # watch a directory and re-index on SQL file changes
+sqlcg viz                              # generate a self-contained graph-explorer HTML
 sqlcg db info                          # graph stats + freshness (indexed SHA vs HEAD)
 sqlcg git install-hooks                # install post-checkout + post-merge resync hooks
 sqlcg gain                             # show usage metrics
@@ -280,7 +282,7 @@ sqlcg mcp restart                      # stop the server (client must respawn it
 sqlcg version                          # show installed version
 ```
 
-### Staying on the latest build (v1.5.0)
+### Staying on the latest build
 
 The installed package, the CLI, and the running MCP server all report the **same**
 version (`sqlcg.__version__`). After upgrading, an editor may still be talking to an
@@ -289,7 +291,7 @@ server's `version` and a `stale_by_version` flag that is `true` when the live se
 differs from the installed build. Re-running `sqlcg install` stops the stale server so
 your editor respawns it on the new build, so you never debug against an outdated server.
 
-### Reads while the server is running (v1.2.0)
+### Reads while the server is running
 
 DuckDB takes an exclusive lock on the database file, so while the MCP server is
 live it holds that lock (other processes cannot open the file, even read-only).
@@ -300,6 +302,34 @@ running they open the database directly, exactly as before. If the server is
 mid-reindex the read waits for it to finish rather than failing with
 "Database is locked".
 
+### Visualising the graph (`viz`)
+
+`sqlcg viz` generates a **self-contained graph-explorer HTML** from the live
+graph — every node, edge, the force-graph library, and the facet data are inlined
+into a single file, so it opens by double-click in a browser with no server or
+external resources.
+
+```bash
+sqlcg viz                              # writes table_graph.html in the current dir
+sqlcg viz --out lineage.html           # choose the output path
+```
+
+In the browser the filter composes schema ∩ kind ∩ tag (multi-select per facet)
+plus a job dropdown; an edge renders only when both endpoints are visible. Table,
+view, and temp nodes are on by default; CTE and derived nodes are off (toggleable).
+
+Two optional CSV facets let you colour and slice the graph by your own
+conventions — both are `pattern,label[,color]` with a required header, patterns
+are case-insensitive `fnmatch` globs (`*`, `?`) matched against the qualified
+table name:
+
+```bash
+sqlcg viz --tags tags.csv              # legend swatches: colour + filter the graph
+sqlcg viz --jobs jobs.csv              # populate the job dropdown (filter only)
+```
+
+Omit `--tags` to hide the tag legend; omit `--jobs` to leave the dropdown empty.
+
 ## Supported dialects
 
 sqlcg is built on [sqlglot](https://github.com/tobymao/sqlglot), so other dialects

{sql_code_graph-1.35.3 → sql_code_graph-1.36.1}/docs/getting-started.md

@@ -56,12 +56,12 @@ cd sql-code-graph
 uv sync
 ```
 
-Verify the install (your exact version may differ — anything **1.32.0 or
+Verify the install (your exact version may differ — anything **1.36.0 or
 newer** is current):
 
 ```bash
 sqlcg version
-# sqlcg version 1.32.0
+# sqlcg version 1.36.0
 ```
 
 ### 3.2 Initialise the graph database
@@ -72,7 +72,7 @@ where the graph file lives.
 
 ```bash
 sqlcg db init
-# Database initialised at /home/<you>/.sqlcg/graph.db (schema v9)
+# Database initialised at /home/<you>/.sqlcg/graph.db (schema v11)
 ```
 
 ### 3.3 Index your repo (point at the ROOT)
@@ -379,6 +379,11 @@ Then run Question 1 (`find table`) on the hub tables you know — validate that
 the graph reflects reality on your known ground truth before relying on it for
 unknown territory.
 
+To see the whole warehouse at once, run `sqlcg viz` — it writes a
+self-contained `table_graph.html` you open in a browser to explore the graph
+visually, filtering by schema, node kind, and your own tag/job CSV facets. See
+the `viz` section in the [README](../README.md#visualising-the-graph-viz).
+
 ---
 
 ## 6. Where to go next
@@ -390,3 +395,5 @@ unknown territory.
 | [`docs/cli.md`](cli.md) | Full CLI reference (every command and flag) |
 | `sqlcg analyze failures` | Lists files that failed to parse with their dominant error-code bucket — start here when coverage is lower than expected |
 | `sqlcg git install-hooks` | Install post-checkout + post-merge hooks that auto-reindex on branch switch |
+| `sqlcg watch <path>` | Watch a directory and re-index on SQL file changes — keeps the graph fresh while you edit |
+| `sqlcg viz` | Generate a self-contained graph-explorer HTML to explore lineage in a browser |

sql_code_graph-1.36.1/plan/metrics/identifier_live_acceptance.md

@@ -0,0 +1,84 @@
+# Generic variable-name resolution — live-DWH acceptance (PR-3)
+
+Feature: [`feature_generic_var_name_resolution.md`](../sprints/feature_generic_var_name_resolution.md)
+(shipped v1.36.0, PR #169). `IDENTIFIER($var)` table names built from `SET`
+string-concat session variables are now resolved at `schema.table` grain via a
+bounded, dialect-agnostic constant-fold of the variable's RHS AST.
+
+## Method
+
+Two isolated sqlcg builds, **same** live DWH corpus (1335 files indexed,
+`/home/ignwrad/Projects/dwh`, dialect snowflake, identical `indexed_sha
+fdf1b551…`), each into its own `SQLCG_DB_PATH`; `gain --json` captured from each.
+
+| Build | Commit | Version | JSON |
+|-------|--------|---------|------|
+| BEFORE (no IDENTIFIER feature) | `449df6d` (#172) | 1.35.3 | [`identifier_live_before_v1.35.3.json`](identifier_live_before_v1.35.3.json) |
+| AFTER (feature merged) | `c480c15` (#169) | 1.36.0 | [`identifier_live_after_v1.36.0.json`](identifier_live_after_v1.36.0.json) |
+
+Index timings: BEFORE 2:42.76, AFTER 2:41.07 wall (both < 3-min budget). Raw
+edges 44093 → 45248; degraded files 174 → 161 (E8 161 → 148, 13 fewer).
+
+## Verdict: WIN CONFIRMED
+
+### 1. The win
+
+| Metric | Before | After | Delta |
+|--------|-------:|------:|------:|
+| `resolvable_write_col_edges` | 36936 | 37713 | **+777** |
+| `transitive_col_edges` | 4182 | 4558 | +376 |
+| `zero_edge_write_queries` | 1028 | 969 | **−59** |
+| `total_write_queries` | 2323 | 2323 | 0 |
+| SELECTS_FROM edges | 6065 | 6239 | **+174** |
+| `good_edges` (catalogued) | 52682 | 53811 | +1129 |
+| `total_edges` | 60830 | 61983 | +1153 |
+| `total_tables` | 6552 | 6673 | +121 |
+| `catalogued_tables` | 5800 | 5864 | +64 |
+
+**121 new `schema.table` nodes** appear in AFTER and not BEFORE — exactly the
+IDENTIFIER($var) targets now resolving. **119 of 121 carry ≥1 real edge.** The
+DWH has **136 files** containing `identifier($…)` (matches the plan's ~136). The
+write side surfaces as `resolvable_write_col_edges` +777 and SELECTS_FROM +174
+(the dynamic DML writes flow through column lineage / SELECTS_FROM, not
+INSERTS_INTO — which is 0 in both builds). `zero_edge_write_queries` fell by 59,
+i.e. 59 formerly-dropped write queries now produce a real edge.
+
+### 2. No regression
+
+`good_edges_strict` 48151 → 48907 (**+756**, UP). `good_edges_scoped` 37471 →
+37502 (+31, UP). Health percentages move trivially (strict 79.16 → 78.90,
+scoped 98.56 → 98.53) — the denominator grows with the new resolved targets;
+absolute good edges rose in both. No drop.
+
+### 3. No catalog collision (env-parameterization safety)
+
+- Non-empty-catalog table nodes = **3 in both** builds (unchanged). The feature
+  added **zero** literal `catalog.schema.table` nodes; every resolved IDENTIFIER
+  target landed at empty-catalog `db.name`, matching the catalog-less identity
+  used by DDL / plain-SQL. The research's "zero literal catalog.schema.table
+  refs" holds.
+- No two distinct real tables collapsed into one node. The only `(db,name)`
+  values with >1 node are per-file local CTE/temp names (`cte_remove_duplicates`,
+  `tmp_*`, `agg`, …) — kind=`cte`, a pre-existing pattern, not IDENTIFIER tables.
+- Spot-checks (real SELECTS_FROM edges in AFTER, absent in BEFORE, correct grain):
+  - `EMP.APPLICATION_FIELDS` → `emp.application_fields` (cat='', SELECTS_FROM=2)
+  - `FAC.VERZAMELFACTUUR` → `fac.verzamelfactuur` (cat='', SELECTS_FROM=3)
+  - `ods.ig_wecycle` → `ods.ig_wecycle` (cat='', SELECTS_FROM=3)
+
+### 4. Negative cases still hold
+
+No phantom table nodes from non-table identifier positions. No node named
+`identifier`, no `$`-bearing or unfolded `(…)` table introduced by the feature.
+The only `$`/paren junk nodes (`@ma.msstg_*/${internal.job.name}/` Snowflake
+stage paths, `ods workaround artikel (planning)`) are **identical in BEFORE** —
+pre-existing, not from this feature. `wmswarehouse*` matches are real data
+tables, not `ALTER WAREHOUSE` artifacts. The structural sink-walk (visiting only
+`exp.Table` positions) correctly auto-excludes ALTER WAREHOUSE / CALL identifiers.
+
+## Why the win is ~777/121 not the full ~187 refs
+
+The +777 `resolvable_write_col_edges` and 121 new nodes cover the resolvable
+share. The plan's Non-Goals exclude the ~40 scalar-subquery / genuinely
+runtime-dynamic delta vars (honest give-up) and chain depth > 1. Of 136 files,
+the resolved fraction lands as 121 distinct target tables (119 with edges); the
+remainder are OPAQUE-prefix / subquery-form sinks that correctly stay dropped.

sql_code_graph-1.36.1/plan/metrics/identifier_live_after_v1.36.0.json

@@ -0,0 +1,31 @@
+{"total_calls": 1721, "last_7d_calls": 1721, "index_runs": 5, "feedback_tp": 0, 
+"feedback_total": 0, "top_tools": [{"name": "execute_sql", "count": 580}, 
+{"name": "trace_column_lineage", "count": 452}, {"name": 
+"list_dialects_and_repos", "count": 148}], "execute_sql_ratio": 0.34, 
+"parse_quality": {"sqlglot": 5166, "scripting": 20, "dynamic_sql": 3}, 
+"coverage": {"catalogued_tables": 5864, "total_tables": 6673, "good_edges": 
+53811, "total_edges": 61983, "phantom_edges": 12241, "blindspot_tables": 631, 
+"good_edges_strict": 48907, "edge_health_strict_pct": 78.9, "good_edges_scoped":
+37502, "total_edges_scoped": 38063, "edge_health_scoped_pct": 98.53, 
+"phantom_confirmed": 11701, "phantom_contradicted": 352, "phantom_unverified": 
+188, "top_blindspot_tables": [{"table": "ba.wtfa_kpi_datum_klant", "bad_edges": 
+70}, {"table": "ba.wtfs_voorraad_dagstand", "bad_edges": 55}, {"table": 
+"ba.wtfs_voorraad_zondagstand", "bad_edges": 55}, {"table": 
+"da.rtgmd_postnl_customer_usage", "bad_edges": 43}, {"table": 
+"da.rtgmd_postnl_athome_3s", "bad_edges": 33}, {"table": 
+"wtfe_kpi_voorraad_artikel_voorraadlocatie", "bad_edges": 32}, {"table": 
+"da.rtgmd_postnl_facturen", "bad_edges": 30}, {"table": "da.hthyb_pages", 
+"bad_edges": 23}, {"table": "da.rtgmd_dhb_extra_kosten", "bad_edges": 19}, 
+{"table": "da.hthyb_page_brandplacement", "bad_edges": 17}], 
+"blindspot_tables_for_80pct": 18, "files_indexed": 1335, "indexed_sha": 
+"fdf1b551a34601a6cf3ce1c8b9f76e27ce2753e6", "db_path": 
+"/tmp/sqlcg-after-graph.db", "index_timestamp": 1781478657.7558403, 
+"degraded_parse_total": 5189, "degraded_parse_queries": 27, 
+"degraded_parse_by_dir": {"etl": {"degraded": 8, "total": 2776}, "ddl": 
+{"degraded": 19, "total": 2413}}, "zero_edge_write_queries": 969, 
+"total_write_queries": 2323, "cte_key_collisions": 0, 
+"rescuable_unqualified_edges": 125, "info_schema_has_column_rows": 100193, 
+"cte_source_gap_writes": 0, "resolvable_write_col_edges": 37713, 
+"transitive_col_edges": 4558, "qualify_failed_statements": 9, 
+"tool_version_stale": false, "indexed_tool_version": "1.36.0", 
+"running_tool_version": "1.36.0"}}

sql_code_graph-1.36.1/plan/metrics/identifier_live_before_v1.35.3.json

@@ -0,0 +1,32 @@
+{"total_calls": 1721, "last_7d_calls": 1721, "index_runs": 5, "feedback_tp": 0, 
+"feedback_total": 0, "top_tools": [{"name": "execute_sql", "count": 580}, 
+{"name": "trace_column_lineage", "count": 452}, {"name": 
+"list_dialects_and_repos", "count": 148}], "execute_sql_ratio": 0.34, 
+"parse_quality": {"sqlglot": 5166, "scripting": 20, "dynamic_sql": 3}, 
+"coverage": {"catalogued_tables": 5800, "total_tables": 6552, "good_edges": 
+52682, "total_edges": 60830, "phantom_edges": 12241, "blindspot_tables": 628, 
+"good_edges_strict": 48151, "edge_health_strict_pct": 79.16, 
+"good_edges_scoped": 37471, "total_edges_scoped": 38020, 
+"edge_health_scoped_pct": 98.56, "phantom_confirmed": 11701, 
+"phantom_contradicted": 352, "phantom_unverified": 188, "top_blindspot_tables": 
+[{"table": "ba.wtfa_kpi_datum_klant", "bad_edges": 70}, {"table": 
+"ba.wtfs_voorraad_dagstand", "bad_edges": 55}, {"table": 
+"ba.wtfs_voorraad_zondagstand", "bad_edges": 55}, {"table": 
+"da.rtgmd_postnl_customer_usage", "bad_edges": 43}, {"table": 
+"da.rtgmd_postnl_athome_3s", "bad_edges": 33}, {"table": 
+"wtfe_kpi_voorraad_artikel_voorraadlocatie", "bad_edges": 32}, {"table": 
+"da.rtgmd_postnl_facturen", "bad_edges": 30}, {"table": "da.hthyb_pages", 
+"bad_edges": 23}, {"table": "da.rtgmd_dhb_extra_kosten", "bad_edges": 19}, 
+{"table": "da.hthyb_page_brandplacement", "bad_edges": 17}], 
+"blindspot_tables_for_80pct": 17, "files_indexed": 1335, "indexed_sha": 
+"fdf1b551a34601a6cf3ce1c8b9f76e27ce2753e6", "db_path": 
+"/tmp/sqlcg-before-graph.db", "index_timestamp": 1781478465.9151304, 
+"degraded_parse_total": 5189, "degraded_parse_queries": 27, 
+"degraded_parse_by_dir": {"etl": {"degraded": 8, "total": 2776}, "ddl": 
+{"degraded": 19, "total": 2413}}, "zero_edge_write_queries": 1028, 
+"total_write_queries": 2323, "cte_key_collisions": 0, 
+"rescuable_unqualified_edges": 125, "info_schema_has_column_rows": 100193, 
+"cte_source_gap_writes": 0, "resolvable_write_col_edges": 36936, 
+"transitive_col_edges": 4182, "qualify_failed_statements": 9, 
+"tool_version_stale": false, "indexed_tool_version": "1.35.3", 
+"running_tool_version": "1.35.3"}}