sql-dag-flow 0.4.8__tar.gz → 0.4.9__tar.gz

{sql_dag_flow-0.4.8/src/sql_dag_flow.egg-info → sql_dag_flow-0.4.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sql-dag-flow
-Version: 0.4.8
+Version: 0.4.9
 Summary: A sophisticated SQL lineage visualization tool for Medallion Architectures.
 Author-email: Flavio Sandoval <dsandovalflavio@gmail.com>
 License: MIT
@@ -89,15 +89,17 @@ Specially optimized for the **Medallion Architecture** (Bronze, Silver, Gold) an
 ### 📊 Discovery & Analysis Tools
 *   **Impact Analysis**: Visualize blast radius before making changes. Highlights downstream models, column usage, and risk levels.
 *   **Diff View on Refresh**: Automatically summarizes added, removed, and modified nodes/edges after code changes.
-*   **Column Usage Tracking (Improved in v0.4.8 🔧)**: Schema Preview shows which specific columns are used by downstream consumers. Now tracks both qualified (`u.name`) and unqualified (`name`) column references for accurate counts.
+*   **Column Usage Tracking (Improved in v0.4.9 🔧)**: Schema Preview shows which specific columns are used by downstream consumers. Uses `sqlglot.optimizer.qualify_columns` for precise resolution of unqualified column references.
 *   **Staleness Detection**: Automatically flags inactive models (`Last Modified > 90d` = Stale) to help clean up legacy pipelines.
 *   **Business Rule Extraction**: Automatically detects and displays WHERE filters, CASE logic, HAVING clauses, and aggregations from each SQL model.
 *   **Complexity Scoring**: Weighted metric per node (JOINs×3, CTEs×2, Subqueries×3, Filters×1, CASE×2, Aggregations×1, UNIONs×2) with color-coded badges (🟢 Low, 🟡 Medium, 🟠 High, 🔴 Very High). Toggleable via ⚡ button.
 *   **Node Comparison**: Select exactly 2 nodes and compare them side-by-side. Highlights differences across metadata, schema columns (shared vs unique), dependencies (Venn-style), complexity scores (with delta indicators), business rules, and SQL content (synced scroll).
 *   **Statistics Panel**: Centered popup with layer distribution bars, edge/source/sink/orphan counts, project/dataset tree, and architecture health validation (now detects **circular dependencies**).
-*   **Schema Extraction (Improved in v0.4.8 🔧)**: Backend AST-based extraction via `sqlglot` handles DDL, CTAS, `CREATE VIEW AS`, CTEs, window functions, CASE expressions, and `SELECT *`.
-*   **Batch Hide from Toolbar (New in v0.4.8 🆕)**: Select multiple nodes → click "Hide" in the selection toolbar to hide them all at once.
-*   **Discovery Mode Fix (v0.4.8 🔧)**: Ghost nodes from Discovery Mode are now hidden when their connected source nodes are hidden, preventing orphan ghost nodes.
+*   **Schema Extraction**: Backend AST-based extraction via `sqlglot` handles DDL, CTAS, `CREATE VIEW AS`, CTEs, window functions, CASE expressions, and `SELECT *`.
+*   **Column-Level Lineage (New in v0.4.9 🆕)**: Traces how each output column derives from source columns. Shows transformation chain (e.g., `order_timestamp ← orders_raw.order_date via CAST(... AS DATETIME)`).
+*   **SQL Syntax Validation (New in v0.4.9 🆕)**: Detects SQL parse errors and displays structured warnings with line/column references. Shows ⚠️ badge on nodes with syntax issues.
+*   **Batch Hide from Toolbar**: Select multiple nodes → click "Hide" in the selection toolbar to hide them all at once.
+*   **Discovery Mode Fix**: Ghost nodes from Discovery Mode are now hidden when their connected source nodes are hidden, preventing orphan ghost nodes.
 
 ### 🎨 Linear-Inspired UI (New in v0.4.6 ✨)
 *   **Design Token System**: ~80 CSS custom properties for consistent theming across all components.
@@ -142,7 +144,7 @@ Install easily via `pip`:
 pip install sql-dag-flow
 ```
 
-To update to the latest version (**v0.4.8**):
+To update to the latest version (**v0.4.9**):
 
 ```bash
 pip install --upgrade sql-dag-flow

{sql_dag_flow-0.4.8 → sql_dag_flow-0.4.9}/README.md

@@ -64,15 +64,17 @@ Specially optimized for the **Medallion Architecture** (Bronze, Silver, Gold) an
 ### 📊 Discovery & Analysis Tools
 *   **Impact Analysis**: Visualize blast radius before making changes. Highlights downstream models, column usage, and risk levels.
 *   **Diff View on Refresh**: Automatically summarizes added, removed, and modified nodes/edges after code changes.
-*   **Column Usage Tracking (Improved in v0.4.8 🔧)**: Schema Preview shows which specific columns are used by downstream consumers. Now tracks both qualified (`u.name`) and unqualified (`name`) column references for accurate counts.
+*   **Column Usage Tracking (Improved in v0.4.9 🔧)**: Schema Preview shows which specific columns are used by downstream consumers. Uses `sqlglot.optimizer.qualify_columns` for precise resolution of unqualified column references.
 *   **Staleness Detection**: Automatically flags inactive models (`Last Modified > 90d` = Stale) to help clean up legacy pipelines.
 *   **Business Rule Extraction**: Automatically detects and displays WHERE filters, CASE logic, HAVING clauses, and aggregations from each SQL model.
 *   **Complexity Scoring**: Weighted metric per node (JOINs×3, CTEs×2, Subqueries×3, Filters×1, CASE×2, Aggregations×1, UNIONs×2) with color-coded badges (🟢 Low, 🟡 Medium, 🟠 High, 🔴 Very High). Toggleable via ⚡ button.
 *   **Node Comparison**: Select exactly 2 nodes and compare them side-by-side. Highlights differences across metadata, schema columns (shared vs unique), dependencies (Venn-style), complexity scores (with delta indicators), business rules, and SQL content (synced scroll).
 *   **Statistics Panel**: Centered popup with layer distribution bars, edge/source/sink/orphan counts, project/dataset tree, and architecture health validation (now detects **circular dependencies**).
-*   **Schema Extraction (Improved in v0.4.8 🔧)**: Backend AST-based extraction via `sqlglot` handles DDL, CTAS, `CREATE VIEW AS`, CTEs, window functions, CASE expressions, and `SELECT *`.
-*   **Batch Hide from Toolbar (New in v0.4.8 🆕)**: Select multiple nodes → click "Hide" in the selection toolbar to hide them all at once.
-*   **Discovery Mode Fix (v0.4.8 🔧)**: Ghost nodes from Discovery Mode are now hidden when their connected source nodes are hidden, preventing orphan ghost nodes.
+*   **Schema Extraction**: Backend AST-based extraction via `sqlglot` handles DDL, CTAS, `CREATE VIEW AS`, CTEs, window functions, CASE expressions, and `SELECT *`.
+*   **Column-Level Lineage (New in v0.4.9 🆕)**: Traces how each output column derives from source columns. Shows transformation chain (e.g., `order_timestamp ← orders_raw.order_date via CAST(... AS DATETIME)`).
+*   **SQL Syntax Validation (New in v0.4.9 🆕)**: Detects SQL parse errors and displays structured warnings with line/column references. Shows ⚠️ badge on nodes with syntax issues.
+*   **Batch Hide from Toolbar**: Select multiple nodes → click "Hide" in the selection toolbar to hide them all at once.
+*   **Discovery Mode Fix**: Ghost nodes from Discovery Mode are now hidden when their connected source nodes are hidden, preventing orphan ghost nodes.
 
 ### 🎨 Linear-Inspired UI (New in v0.4.6 ✨)
 *   **Design Token System**: ~80 CSS custom properties for consistent theming across all components.
@@ -117,7 +119,7 @@ Install easily via `pip`:
 pip install sql-dag-flow
 ```
 
-To update to the latest version (**v0.4.8**):
+To update to the latest version (**v0.4.9**):
 
 ```bash
 pip install --upgrade sql-dag-flow

{sql_dag_flow-0.4.8 → sql_dag_flow-0.4.9}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "sql-dag-flow"
-version = "0.4.8"
+version = "0.4.9"
 description = "A sophisticated SQL lineage visualization tool for Medallion Architectures."
 readme = "README.md"
 requires-python = ">=3.8"

{sql_dag_flow-0.4.8 → sql_dag_flow-0.4.9}/src/sql_dag_flow/main.py

@@ -107,15 +107,24 @@ def get_filtered_graph(data: dict = Body(...)):
 def get_path():
     return {"path": CURRENT_DIRECTORY}
 
-@app.get("/export")
-def export_data_dictionary(dialect: str = "bigquery"):
-    """Generates a Markdown data dictionary report."""
+@app.post("/export")
+def export_data_dictionary(data: dict = Body(...)):
+    """Generates a Markdown data dictionary report for visible nodes only."""
+    dialect = data.get("dialect", "bigquery")
+    visible_node_ids = data.get("visible_node_ids", None)  # None = export all
+    
     if not os.path.exists(CURRENT_DIRECTORY):
         raise HTTPException(status_code=400, detail="Directory not found")
     
     tables = parse_sql_files(CURRENT_DIRECTORY, dialect=dialect)
     nodes, edges, cycles = build_graph(tables, discovery_mode=False)
     
+    # Filter to only visible nodes if list provided
+    if visible_node_ids is not None:
+        visible_set = set(visible_node_ids)
+        nodes = [n for n in nodes if n['id'] in visible_set]
+        edges = [e for e in edges if e['source'] in visible_set and e['target'] in visible_set]
+    
     lines = []
     lines.append(f"# Data Dictionary")
     lines.append(f"")
@@ -195,6 +204,17 @@ def export_data_dictionary(dialect: str = "bigquery"):
                 lines.extend(rules_items[:6])
                 lines.append(f"")
             
+            # Schema columns
+            schema = d.get('schema', [])
+            if schema:
+                lines.append(f"**Schema ({len(schema)} columns):**")
+                lines.append(f"")
+                lines.append(f"| Column | Type |")
+                lines.append(f"|--------|------|")
+                for col in schema:
+                    lines.append(f"| `{col.get('name', '?')}` | {col.get('type', 'UNKNOWN')} |")
+                lines.append(f"")
+            
             # Column consumers
             col_consumers = d.get('column_consumers', {})
             if col_consumers:
@@ -207,6 +227,30 @@ def export_data_dictionary(dialect: str = "bigquery"):
                     lines.append(f"| `{col}` | {consumer_labels} |")
                 lines.append(f"")
             
+            # Column lineage
+            col_lineage = d.get('column_lineage', {})
+            if col_lineage:
+                lines.append(f"**Column Lineage:**")
+                lines.append(f"")
+                lines.append(f"| Output Column | Source | Transform |")
+                lines.append(f"|---------------|--------|-----------|")
+                for col, sources in sorted(col_lineage.items()):
+                    for src in sources:
+                        src_ref = f"{src.get('source_table', '')}.{src.get('source_column', '')}" if src.get('source_table') else src.get('source_column', '')
+                        transform = src.get('transform', '') or '—'
+                        lines.append(f"| `{col}` | `{src_ref}` | {transform} |")
+                lines.append(f"")
+            
+            # Syntax warnings
+            syntax_warnings = d.get('syntax_warnings', [])
+            if syntax_warnings:
+                lines.append(f"**⚠️ Syntax Issues ({len(syntax_warnings)}):**")
+                lines.append(f"")
+                for w in syntax_warnings:
+                    loc = f"Line {w.get('line', '?')}, Col {w.get('col', '?')}" if w.get('line') else ""
+                    lines.append(f"- {w.get('description', 'Unknown error')} {f'({loc})' if loc else ''}")
+                lines.append(f"")
+            
             lines.append(f"---")
             lines.append(f"")
     

{sql_dag_flow-0.4.8 → sql_dag_flow-0.4.9}/src/sql_dag_flow/parser.py

@@ -3,6 +3,7 @@ import re
 import time
 import sqlglot
 from sqlglot import exp
+from sqlglot.optimizer.qualify_columns import qualify_columns as sqlglot_qualify_columns
 import networkx as nx
 
 
@@ -506,6 +507,30 @@ def parse_sql_files(directory, allowed_subfolders=None, dialect="bigquery"):
                         "header_meta": header_meta,
                         "last_modified_days": days_ago
                     }
+                except sqlglot.errors.ParseError as pe:
+                    # Feature 4: Capture structured syntax errors
+                    syntax_warnings = []
+                    for err in pe.errors:
+                        syntax_warnings.append({
+                            "description": err.get("description", str(pe)),
+                            "line": err.get("line"),
+                            "col": err.get("col"),
+                            "highlight": err.get("highlight", "")
+                        })
+                    print(f"Syntax error in {filepath}: {pe}")
+                    tables[filename_base] = {
+                        "id": filename_base,
+                        "label": filename_base,
+                        "layer": layer,
+                        "type": "unknown",
+                        "project": "n/a",
+                        "dataset": "n/a",
+                        "path": filepath,
+                        "dependencies": {},
+                        "error": str(pe),
+                        "syntax_warnings": syntax_warnings,
+                        "content": sql_content
+                    }
                 except Exception as e:
                     print(f"Error parsing {filepath}: {e}")
                     tables[filename_base] = {
@@ -520,6 +545,144 @@ def parse_sql_files(directory, allowed_subfolders=None, dialect="bigquery"):
                         "error": str(e),
                         "content": sql_content
                     }
+    # ===== Second Pass: Qualify Columns & Column Lineage =====
+    # Build a global schema dict for qualify_columns and lineage
+    global_schema = {}  # {dataset: {table: {col: type}}}
+    for tid, tdata in tables.items():
+        schema_cols = tdata.get("schema", [])
+        if not schema_cols:
+            continue
+        ds = tdata.get("dataset", "default")
+        label = tdata.get("label", tid)
+        if ds not in global_schema:
+            global_schema[ds] = {}
+        col_dict = {}
+        for c in schema_cols:
+            col_dict[c["name"]] = c.get("type", "UNKNOWN")
+        global_schema[ds][label] = col_dict
+        # Also add without dataset prefix for simpler lookups
+        if "default" not in global_schema:
+            global_schema["default"] = {}
+        global_schema["default"][label] = col_dict
+    
+    # Re-extract column_references using qualify_columns for precision
+    for tid, tdata in tables.items():
+        if tdata.get("error") or not tdata.get("content"):
+            continue
+        try:
+            parsed = sqlglot.parse_one(tdata["content"], read=dialect)
+            
+            # Try to qualify columns using the global schema
+            try:
+                qualified_ast = sqlglot_qualify_columns(parsed.copy(), schema=global_schema, dialect=dialect)
+            except Exception:
+                qualified_ast = parsed  # Fallback to unqualified
+            
+            # Re-extract column references from qualified AST
+            alias_map = {}
+            defined_ctes = set()
+            for cte in qualified_ast.find_all(exp.CTE):
+                cn = cte.alias_or_name
+                if cn:
+                    defined_ctes.add(cn)
+            
+            for t in qualified_ast.find_all(exp.Table):
+                t_name = t.name
+                t_full = t_name
+                if t.db:
+                    t_full = f"{t.db}.{t_name}"
+                    if t.catalog:
+                        t_full = f"{t.catalog}.{t.db}.{t_name}"
+                if t.alias:
+                    alias_map[t.alias] = t_full
+                alias_map[t_name] = t_full
+            
+            column_references = {}
+            target_name = tdata.get("label", tid)
+            unqualified_columns = set()
+            
+            for col in qualified_ast.find_all(exp.Column):
+                col_name = col.name
+                col_table = col.table
+                if col_table and col_table in alias_map:
+                    source = alias_map[col_table]
+                    if source not in column_references:
+                        column_references[source] = set()
+                    column_references[source].add(col_name)
+                elif not col_table:
+                    unqualified_columns.add(col_name)
+            
+            # Fallback for any remaining unqualified columns
+            if unqualified_columns:
+                source_tables = [
+                    v for k, v in alias_map.items()
+                    if v != target_name
+                    and v.split('.')[-1] != target_name
+                    and v not in defined_ctes
+                    and v.split('.')[-1] not in defined_ctes
+                ]
+                source_tables = list(set(source_tables))
+                if source_tables:
+                    for src in source_tables:
+                        if src not in column_references:
+                            column_references[src] = set()
+                        column_references[src].update(unqualified_columns)
+            
+            column_references = {k: sorted(list(v)) for k, v in column_references.items()}
+            tdata["column_references"] = column_references
+            
+        except Exception:
+            pass  # Keep original column_references
+    
+    # ===== Column-Level Lineage =====
+    # For each table, trace how each output column derives from source columns
+    for tid, tdata in tables.items():
+        if tdata.get("error") or not tdata.get("content"):
+            continue
+        schema_cols = tdata.get("schema", [])
+        if not schema_cols:
+            continue
+        
+        column_lineage = {}  # col_name -> [{source_table, source_column, transform}]
+        sql_content = tdata["content"]
+        
+        for col_info in schema_cols:
+            col_name = col_info["name"]
+            if col_name == "*":
+                continue
+            try:
+                from sqlglot.lineage import lineage as sqlglot_lineage
+                node = sqlglot_lineage(
+                    col_name, sql_content,
+                    schema=global_schema,
+                    dialect=dialect
+                )
+                sources = []
+                for child in node.downstream:
+                    source_col = child.name
+                    # Extract the source table from the expression
+                    source_table = ""
+                    if child.source and isinstance(child.source, exp.Table):
+                        source_table = child.source.name
+                        if child.source.db:
+                            source_table = f"{child.source.db}.{child.source.name}"
+                    
+                    transform = ""
+                    if child.expression and str(child.expression) != source_col:
+                        transform = str(child.expression)
+                    
+                    sources.append({
+                        "source_table": source_table,
+                        "source_column": source_col,
+                        "transform": transform
+                    })
+                if sources:
+                    column_lineage[col_name] = sources
+            except Exception:
+                pass  # Lineage not available for this column
+        
+        if column_lineage:
+            tdata["column_lineage"] = column_lineage
     
     return tables