sql-dag-flow 0.4.7__tar.gz → 0.4.9__tar.gz

{sql_dag_flow-0.4.7/src/sql_dag_flow.egg-info → sql_dag_flow-0.4.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sql-dag-flow
-Version: 0.4.7
+Version: 0.4.9
 Summary: A sophisticated SQL lineage visualization tool for Medallion Architectures.
 Author-email: Flavio Sandoval <dsandovalflavio@gmail.com>
 License: MIT
@@ -89,12 +89,17 @@ Specially optimized for the **Medallion Architecture** (Bronze, Silver, Gold) an
 ### 📊 Discovery & Analysis Tools
 *   **Impact Analysis**: Visualize blast radius before making changes. Highlights downstream models, column usage, and risk levels.
 *   **Diff View on Refresh**: Automatically summarizes added, removed, and modified nodes/edges after code changes.
-*   **Column Usage Tracking**: Schema Preview shows which specific columns are used by downstream consumers.
+*   **Column Usage Tracking (Improved in v0.4.9 🔧)**: Schema Preview shows which specific columns are used by downstream consumers. Uses `sqlglot.optimizer.qualify_columns` for precise resolution of unqualified column references.
 *   **Staleness Detection**: Automatically flags inactive models (`Last Modified > 90d` = Stale) to help clean up legacy pipelines.
 *   **Business Rule Extraction**: Automatically detects and displays WHERE filters, CASE logic, HAVING clauses, and aggregations from each SQL model.
 *   **Complexity Scoring**: Weighted metric per node (JOINs×3, CTEs×2, Subqueries×3, Filters×1, CASE×2, Aggregations×1, UNIONs×2) with color-coded badges (🟢 Low, 🟡 Medium, 🟠 High, 🔴 Very High). Toggleable via ⚡ button.
+*   **Node Comparison**: Select exactly 2 nodes and compare them side-by-side. Highlights differences across metadata, schema columns (shared vs unique), dependencies (Venn-style), complexity scores (with delta indicators), business rules, and SQL content (synced scroll).
 *   **Statistics Panel**: Centered popup with layer distribution bars, edge/source/sink/orphan counts, project/dataset tree, and architecture health validation (now detects **circular dependencies**).
-*   **Schema Preview**: Extracts columns from DDL, CTAS (`CREATE TABLE AS SELECT`), and `CREATE VIEW` statements.
+*   **Schema Extraction**: Backend AST-based extraction via `sqlglot` handles DDL, CTAS, `CREATE VIEW AS`, CTEs, window functions, CASE expressions, and `SELECT *`.
+*   **Column-Level Lineage (New in v0.4.9 🆕)**: Traces how each output column derives from source columns. Shows transformation chain (e.g., `order_timestamp ← orders_raw.order_date via CAST(... AS DATETIME)`).
+*   **SQL Syntax Validation (New in v0.4.9 🆕)**: Detects SQL parse errors and displays structured warnings with line/column references. Shows ⚠️ badge on nodes with syntax issues.
+*   **Batch Hide from Toolbar**: Select multiple nodes → click "Hide" in the selection toolbar to hide them all at once.
+*   **Discovery Mode Fix**: Ghost nodes from Discovery Mode are now hidden when their connected source nodes are hidden, preventing orphan ghost nodes.
 
 ### 🎨 Linear-Inspired UI (New in v0.4.6 ✨)
 *   **Design Token System**: ~80 CSS custom properties for consistent theming across all components.
@@ -139,7 +144,7 @@ Install easily via `pip`:
 pip install sql-dag-flow
 ```
 
-To update to the latest version (**v0.4.6**):
+To update to the latest version (**v0.4.9**):
 
 ```bash
 pip install --upgrade sql-dag-flow

{sql_dag_flow-0.4.7 → sql_dag_flow-0.4.9}/README.md

@@ -64,12 +64,17 @@ Specially optimized for the **Medallion Architecture** (Bronze, Silver, Gold) an
 ### 📊 Discovery & Analysis Tools
 *   **Impact Analysis**: Visualize blast radius before making changes. Highlights downstream models, column usage, and risk levels.
 *   **Diff View on Refresh**: Automatically summarizes added, removed, and modified nodes/edges after code changes.
-*   **Column Usage Tracking**: Schema Preview shows which specific columns are used by downstream consumers.
+*   **Column Usage Tracking (Improved in v0.4.9 🔧)**: Schema Preview shows which specific columns are used by downstream consumers. Uses `sqlglot.optimizer.qualify_columns` for precise resolution of unqualified column references.
 *   **Staleness Detection**: Automatically flags inactive models (`Last Modified > 90d` = Stale) to help clean up legacy pipelines.
 *   **Business Rule Extraction**: Automatically detects and displays WHERE filters, CASE logic, HAVING clauses, and aggregations from each SQL model.
 *   **Complexity Scoring**: Weighted metric per node (JOINs×3, CTEs×2, Subqueries×3, Filters×1, CASE×2, Aggregations×1, UNIONs×2) with color-coded badges (🟢 Low, 🟡 Medium, 🟠 High, 🔴 Very High). Toggleable via ⚡ button.
+*   **Node Comparison**: Select exactly 2 nodes and compare them side-by-side. Highlights differences across metadata, schema columns (shared vs unique), dependencies (Venn-style), complexity scores (with delta indicators), business rules, and SQL content (synced scroll).
 *   **Statistics Panel**: Centered popup with layer distribution bars, edge/source/sink/orphan counts, project/dataset tree, and architecture health validation (now detects **circular dependencies**).
-*   **Schema Preview**: Extracts columns from DDL, CTAS (`CREATE TABLE AS SELECT`), and `CREATE VIEW` statements.
+*   **Schema Extraction**: Backend AST-based extraction via `sqlglot` handles DDL, CTAS, `CREATE VIEW AS`, CTEs, window functions, CASE expressions, and `SELECT *`.
+*   **Column-Level Lineage (New in v0.4.9 🆕)**: Traces how each output column derives from source columns. Shows transformation chain (e.g., `order_timestamp ← orders_raw.order_date via CAST(... AS DATETIME)`).
+*   **SQL Syntax Validation (New in v0.4.9 🆕)**: Detects SQL parse errors and displays structured warnings with line/column references. Shows ⚠️ badge on nodes with syntax issues.
+*   **Batch Hide from Toolbar**: Select multiple nodes → click "Hide" in the selection toolbar to hide them all at once.
+*   **Discovery Mode Fix**: Ghost nodes from Discovery Mode are now hidden when their connected source nodes are hidden, preventing orphan ghost nodes.
 
 ### 🎨 Linear-Inspired UI (New in v0.4.6 ✨)
 *   **Design Token System**: ~80 CSS custom properties for consistent theming across all components.
@@ -114,7 +119,7 @@ Install easily via `pip`:
 pip install sql-dag-flow
 ```
 
-To update to the latest version (**v0.4.6**):
+To update to the latest version (**v0.4.9**):
 
 ```bash
 pip install --upgrade sql-dag-flow

{sql_dag_flow-0.4.7 → sql_dag_flow-0.4.9}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "sql-dag-flow"
-version = "0.4.7"
+version = "0.4.9"
 description = "A sophisticated SQL lineage visualization tool for Medallion Architectures."
 readme = "README.md"
 requires-python = ">=3.8"

{sql_dag_flow-0.4.7 → sql_dag_flow-0.4.9}/src/sql_dag_flow/main.py

@@ -107,15 +107,24 @@ def get_filtered_graph(data: dict = Body(...)):
 def get_path():
     return {"path": CURRENT_DIRECTORY}
 
-@app.get("/export")
-def export_data_dictionary(dialect: str = "bigquery"):
-    """Generates a Markdown data dictionary report."""
+@app.post("/export")
+def export_data_dictionary(data: dict = Body(...)):
+    """Generates a Markdown data dictionary report for visible nodes only."""
+    dialect = data.get("dialect", "bigquery")
+    visible_node_ids = data.get("visible_node_ids", None)  # None = export all
+    
     if not os.path.exists(CURRENT_DIRECTORY):
         raise HTTPException(status_code=400, detail="Directory not found")
     
     tables = parse_sql_files(CURRENT_DIRECTORY, dialect=dialect)
     nodes, edges, cycles = build_graph(tables, discovery_mode=False)
     
+    # Filter to only visible nodes if list provided
+    if visible_node_ids is not None:
+        visible_set = set(visible_node_ids)
+        nodes = [n for n in nodes if n['id'] in visible_set]
+        edges = [e for e in edges if e['source'] in visible_set and e['target'] in visible_set]
+    
     lines = []
     lines.append(f"# Data Dictionary")
     lines.append(f"")
@@ -195,6 +204,17 @@ def export_data_dictionary(dialect: str = "bigquery"):
                 lines.extend(rules_items[:6])
                 lines.append(f"")
             
+            # Schema columns
+            schema = d.get('schema', [])
+            if schema:
+                lines.append(f"**Schema ({len(schema)} columns):**")
+                lines.append(f"")
+                lines.append(f"| Column | Type |")
+                lines.append(f"|--------|------|")
+                for col in schema:
+                    lines.append(f"| `{col.get('name', '?')}` | {col.get('type', 'UNKNOWN')} |")
+                lines.append(f"")
+            
             # Column consumers
             col_consumers = d.get('column_consumers', {})
             if col_consumers:
@@ -207,6 +227,30 @@ def export_data_dictionary(dialect: str = "bigquery"):
                     lines.append(f"| `{col}` | {consumer_labels} |")
                 lines.append(f"")
             
+            # Column lineage
+            col_lineage = d.get('column_lineage', {})
+            if col_lineage:
+                lines.append(f"**Column Lineage:**")
+                lines.append(f"")
+                lines.append(f"| Output Column | Source | Transform |")
+                lines.append(f"|---------------|--------|-----------|")
+                for col, sources in sorted(col_lineage.items()):
+                    for src in sources:
+                        src_ref = f"{src.get('source_table', '')}.{src.get('source_column', '')}" if src.get('source_table') else src.get('source_column', '')
+                        transform = src.get('transform', '') or '—'
+                        lines.append(f"| `{col}` | `{src_ref}` | {transform} |")
+                lines.append(f"")
+            
+            # Syntax warnings
+            syntax_warnings = d.get('syntax_warnings', [])
+            if syntax_warnings:
+                lines.append(f"**⚠️ Syntax Issues ({len(syntax_warnings)}):**")
+                lines.append(f"")
+                for w in syntax_warnings:
+                    loc = f"Line {w.get('line', '?')}, Col {w.get('col', '?')}" if w.get('line') else ""
+                    lines.append(f"- {w.get('description', 'Unknown error')} {f'({loc})' if loc else ''}")
+                lines.append(f"")
+            
             lines.append(f"---")
             lines.append(f"")
     

{sql_dag_flow-0.4.7 → sql_dag_flow-0.4.9}/src/sql_dag_flow/parser.py

@@ -3,8 +3,86 @@ import re
 import time
 import sqlglot
 from sqlglot import exp
+from sqlglot.optimizer.qualify_columns import qualify_columns as sqlglot_qualify_columns
 import networkx as nx
 
+
+def extract_output_columns(parsed, dialect="bigquery"):
+    """
+    Extract output column schema from a parsed SQL AST.
+    Returns a list of {"name": str, "type": str} dicts.
+    
+    Handles:
+    - DDL with explicit column definitions (CREATE TABLE t (col TYPE))
+    - CTAS / CREATE VIEW AS SELECT
+    - WITH...SELECT (final SELECT after CTEs)
+    - Standalone SELECT
+    - Window functions, nested functions, CASE expressions
+    - SELECT * (returns [{"name": "*", "type": "ALL"}])
+    """
+    columns = []
+
+    # Case 1: DDL with explicit column definitions
+    if isinstance(parsed, exp.Create):
+        schema_node = parsed.this
+        if isinstance(schema_node, exp.Schema):
+            for col_def in schema_node.expressions:
+                if isinstance(col_def, exp.ColumnDef):
+                    col_name = col_def.name
+                    col_type_node = col_def.args.get("kind")
+                    type_str = col_type_node.sql(dialect=dialect) if col_type_node else "UNKNOWN"
+                    columns.append({"name": col_name, "type": type_str})
+            if columns:
+                return columns
+
+    # Case 2: Find the final SELECT statement
+    # For CTAS, CREATE VIEW AS, WITH...SELECT, standalone SELECT
+    # We want the outermost SELECT that isn't inside a CTE or subquery
+    select_node = None
+    
+    if isinstance(parsed, exp.Create):
+        # For CTAS / CREATE VIEW AS: get the SELECT inside the CREATE
+        inner = parsed.expression
+        if inner:
+            if isinstance(inner, exp.Select):
+                select_node = inner
+            elif hasattr(inner, 'find'):
+                select_node = inner.find(exp.Select)
+    
+    if not select_node:
+        # For standalone queries: find WITH wrapper or direct SELECT
+        # Walk to find the top-level Select (not nested in subquery)
+        if isinstance(parsed, exp.Select):
+            select_node = parsed
+        else:
+            # Could be a WITH or UNION — find the first select
+            select_node = parsed.find(exp.Select)
+    
+    if select_node:
+        for expr in select_node.expressions:
+            try:
+                if isinstance(expr, exp.Star):
+                    columns.append({"name": "*", "type": "ALL"})
+                elif isinstance(expr, exp.Alias):
+                    alias_name = expr.alias
+                    inner_expr = expr.this
+                    type_str = inner_expr.sql(dialect=dialect, pretty=False)
+                    # Truncate very long expressions for readability
+                    if len(type_str) > 80:
+                        type_str = type_str[:77] + "..."
+                    columns.append({"name": alias_name, "type": type_str})
+                elif isinstance(expr, exp.Column):
+                    columns.append({"name": expr.name, "type": "column"})
+                else:
+                    # Computed expression without alias
+                    expr_sql = expr.sql(dialect=dialect, pretty=False)
+                    name = expr_sql[:40] + "..." if len(expr_sql) > 40 else expr_sql
+                    columns.append({"name": name, "type": "expression"})
+            except Exception:
+                continue
+
+    return columns
+
 def parse_sql_files(directory, allowed_subfolders=None, dialect="bigquery"):
     """
     Recursively scans a directory for .sql files and parses them.
@@ -328,6 +406,9 @@ def parse_sql_files(directory, allowed_subfolders=None, dialect="bigquery"):
                         alias_map[t_name] = t_full
                     
                     # Extract column references with their table qualifier
+                    # Also capture unqualified columns and assign them to source tables
+                    unqualified_columns = set()
+                    
                     for col in parsed.find_all(exp.Column):
                         col_name = col.name
                         col_table = col.table  # The table qualifier (alias or name)
@@ -336,6 +417,36 @@ def parse_sql_files(directory, allowed_subfolders=None, dialect="bigquery"):
                             if source not in column_references:
                                 column_references[source] = set()
                             column_references[source].add(col_name)
+                        elif not col_table:
+                            # Unqualified column — track separately
+                            unqualified_columns.add(col_name)
+                    
+                    # Assign unqualified columns to source tables
+                    # Filter out CTEs and the target table itself (both short and full name)
+                    source_tables = [
+                        v for k, v in alias_map.items() 
+                        if v != target_table_name 
+                        and v.split('.')[-1] != target_table_name
+                        and v not in defined_ctes
+                        and v.split('.')[-1] not in defined_ctes
+                    ]
+                    # Deduplicate (aliases may point to same table)
+                    source_tables = list(set(source_tables))
+                    
+                    if unqualified_columns and source_tables:
+                        if len(source_tables) == 1:
+                            # Single source: assign all unqualified columns to it
+                            src = source_tables[0]
+                            if src not in column_references:
+                                column_references[src] = set()
+                            column_references[src].update(unqualified_columns)
+                        else:
+                            # Multiple sources: assign to ALL sources (best effort)
+                            # The UI will show them as "used" which is better than missing
+                            for src in source_tables:
+                                if src not in column_references:
+                                    column_references[src] = set()
+                                column_references[src].update(unqualified_columns)
                     
                     # Convert sets to sorted lists for JSON serialization
                     column_references = {k: sorted(list(v)) for k, v in column_references.items()}
@@ -389,12 +500,37 @@ def parse_sql_files(directory, allowed_subfolders=None, dialect="bigquery"):
                         "content": sql_content,
                         "ctes": defined_ctes,
                         "cte_deps": cte_deps,
+                        "schema": extract_output_columns(parsed, dialect),
                         "business_rules": business_rules,
                         "complexity": complexity_breakdown,
                         "column_references": column_references,
                         "header_meta": header_meta,
                         "last_modified_days": days_ago
                     }
+                except sqlglot.errors.ParseError as pe:
+                    # Feature 4: Capture structured syntax errors
+                    syntax_warnings = []
+                    for err in pe.errors:
+                        syntax_warnings.append({
+                            "description": err.get("description", str(pe)),
+                            "line": err.get("line"),
+                            "col": err.get("col"),
+                            "highlight": err.get("highlight", "")
+                        })
+                    print(f"Syntax error in {filepath}: {pe}")
+                    tables[filename_base] = {
+                        "id": filename_base,
+                        "label": filename_base,
+                        "layer": layer,
+                        "type": "unknown",
+                        "project": "n/a",
+                        "dataset": "n/a",
+                        "path": filepath,
+                        "dependencies": {},
+                        "error": str(pe),
+                        "syntax_warnings": syntax_warnings,
+                        "content": sql_content
+                    }
                 except Exception as e:
                     print(f"Error parsing {filepath}: {e}")
                     tables[filename_base] = {
@@ -409,6 +545,144 @@ def parse_sql_files(directory, allowed_subfolders=None, dialect="bigquery"):
                         "error": str(e),
                         "content": sql_content
                     }
+    # ===== Second Pass: Qualify Columns & Column Lineage =====
+    # Build a global schema dict for qualify_columns and lineage
+    global_schema = {}  # {dataset: {table: {col: type}}}
+    for tid, tdata in tables.items():
+        schema_cols = tdata.get("schema", [])
+        if not schema_cols:
+            continue
+        ds = tdata.get("dataset", "default")
+        label = tdata.get("label", tid)
+        if ds not in global_schema:
+            global_schema[ds] = {}
+        col_dict = {}
+        for c in schema_cols:
+            col_dict[c["name"]] = c.get("type", "UNKNOWN")
+        global_schema[ds][label] = col_dict
+        # Also add without dataset prefix for simpler lookups
+        if "default" not in global_schema:
+            global_schema["default"] = {}
+        global_schema["default"][label] = col_dict
+    
+    # Re-extract column_references using qualify_columns for precision
+    for tid, tdata in tables.items():
+        if tdata.get("error") or not tdata.get("content"):
+            continue
+        try:
+            parsed = sqlglot.parse_one(tdata["content"], read=dialect)
+            
+            # Try to qualify columns using the global schema
+            try:
+                qualified_ast = sqlglot_qualify_columns(parsed.copy(), schema=global_schema, dialect=dialect)
+            except Exception:
+                qualified_ast = parsed  # Fallback to unqualified
+            
+            # Re-extract column references from qualified AST
+            alias_map = {}
+            defined_ctes = set()
+            for cte in qualified_ast.find_all(exp.CTE):
+                cn = cte.alias_or_name
+                if cn:
+                    defined_ctes.add(cn)
+            
+            for t in qualified_ast.find_all(exp.Table):
+                t_name = t.name
+                t_full = t_name
+                if t.db:
+                    t_full = f"{t.db}.{t_name}"
+                    if t.catalog:
+                        t_full = f"{t.catalog}.{t.db}.{t_name}"
+                if t.alias:
+                    alias_map[t.alias] = t_full
+                alias_map[t_name] = t_full
+            
+            column_references = {}
+            target_name = tdata.get("label", tid)
+            unqualified_columns = set()
+            
+            for col in qualified_ast.find_all(exp.Column):
+                col_name = col.name
+                col_table = col.table
+                if col_table and col_table in alias_map:
+                    source = alias_map[col_table]
+                    if source not in column_references:
+                        column_references[source] = set()
+                    column_references[source].add(col_name)
+                elif not col_table:
+                    unqualified_columns.add(col_name)
+            
+            # Fallback for any remaining unqualified columns
+            if unqualified_columns:
+                source_tables = [
+                    v for k, v in alias_map.items()
+                    if v != target_name
+                    and v.split('.')[-1] != target_name
+                    and v not in defined_ctes
+                    and v.split('.')[-1] not in defined_ctes
+                ]
+                source_tables = list(set(source_tables))
+                if source_tables:
+                    for src in source_tables:
+                        if src not in column_references:
+                            column_references[src] = set()
+                        column_references[src].update(unqualified_columns)
+            
+            column_references = {k: sorted(list(v)) for k, v in column_references.items()}
+            tdata["column_references"] = column_references
+            
+        except Exception:
+            pass  # Keep original column_references
+    
+    # ===== Column-Level Lineage =====
+    # For each table, trace how each output column derives from source columns
+    for tid, tdata in tables.items():
+        if tdata.get("error") or not tdata.get("content"):
+            continue
+        schema_cols = tdata.get("schema", [])
+        if not schema_cols:
+            continue
+        
+        column_lineage = {}  # col_name -> [{source_table, source_column, transform}]
+        sql_content = tdata["content"]
+        
+        for col_info in schema_cols:
+            col_name = col_info["name"]
+            if col_name == "*":
+                continue
+            try:
+                from sqlglot.lineage import lineage as sqlglot_lineage
+                node = sqlglot_lineage(
+                    col_name, sql_content,
+                    schema=global_schema,
+                    dialect=dialect
+                )
+                sources = []
+                for child in node.downstream:
+                    source_col = child.name
+                    # Extract the source table from the expression
+                    source_table = ""
+                    if child.source and isinstance(child.source, exp.Table):
+                        source_table = child.source.name
+                        if child.source.db:
+                            source_table = f"{child.source.db}.{child.source.name}"
+                    
+                    transform = ""
+                    if child.expression and str(child.expression) != source_col:
+                        transform = str(child.expression)
+                    
+                    sources.append({
+                        "source_table": source_table,
+                        "source_column": source_col,
+                        "transform": transform
+                    })
+                if sources:
+                    column_lineage[col_name] = sources
+            except Exception:
+                pass  # Lineage not available for this column
+        
+        if column_lineage:
+            tdata["column_lineage"] = column_lineage
     
     return tables
 
@@ -637,6 +911,9 @@ def build_graph(tables, discovery_mode=False, expanded_nodes=None):
                 source_node_id = lookup.get(source_ref.split(".")[-1])
             
             if source_node_id and source_node_id in all_nodes_data:
+                # Skip self-references (a model shouldn't be its own consumer)
+                if source_node_id == consumer_id:
+                    continue
                 if source_node_id not in column_consumers:
                     column_consumers[source_node_id] = {}
                 for col in columns: