sql-dag-flow 0.4.2__tar.gz → 0.4.7__tar.gz

{sql_dag_flow-0.4.2/src/sql_dag_flow.egg-info → sql_dag_flow-0.4.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sql-dag-flow
-Version: 0.4.2
+Version: 0.4.7
 Summary: A sophisticated SQL lineage visualization tool for Medallion Architectures.
 Author-email: Flavio Sandoval <dsandovalflavio@gmail.com>
 License: MIT
@@ -73,8 +73,11 @@ Specially optimized for the **Medallion Architecture** (Bronze, Silver, Gold) an
     *   **Hide/Show**: Toggle visibility of individual nodes or full branches.
 *   **Advanced Navigation**:
     *   **Sidebar**: Grouped list of nodes with toggle between **By Layer** and **By Project/Dataset** views.
+    *   **Command Palette (Cmd+P)**: Instantly search and navigate to nodes across large projects.
+    *   **Keyboard Arrow Navigation**: Rapidly explore lineage by moving ← (upstream) and → (downstream) between connected nodes.
+    *   **Breadcrumb Trail**: Maintain context while drilling down with a visual history of visited nodes.
     *   **SQL Content Search**: Search inside SQL file content across all nodes — find WHERE clauses, JOINs, or any keyword.
-    *   **Details Panel**: View formatted SQL code, schema preview (DDL, CTAS, Views), and node configuration.
+    *   **Details Panel**: View formatted SQL code, schema preview (DDL, CTAS, Views), node configuration, and add **custom descriptions** to document models.
 
 ### 📝 Notes & Annotations
 *   **Center Placement**: New notes spawn exactly in the center of your view.
@@ -84,12 +87,16 @@ Specially optimized for the **Medallion Architecture** (Bronze, Silver, Gold) an
     *   **Groups**: Create visual containers to group related nodes.
 
 ### 📊 Discovery & Analysis Tools
+*   **Impact Analysis**: Visualize blast radius before making changes. Highlights downstream models, column usage, and risk levels.
+*   **Diff View on Refresh**: Automatically summarizes added, removed, and modified nodes/edges after code changes.
+*   **Column Usage Tracking**: Schema Preview shows which specific columns are used by downstream consumers.
+*   **Staleness Detection**: Automatically flags inactive models (`Last Modified > 90d` = Stale) to help clean up legacy pipelines.
 *   **Business Rule Extraction**: Automatically detects and displays WHERE filters, CASE logic, HAVING clauses, and aggregations from each SQL model.
 *   **Complexity Scoring**: Weighted metric per node (JOINs×3, CTEs×2, Subqueries×3, Filters×1, CASE×2, Aggregations×1, UNIONs×2) with color-coded badges (🟢 Low, 🟡 Medium, 🟠 High, 🔴 Very High). Toggleable via ⚡ button.
-*   **Statistics Panel**: Centered popup with layer distribution bars, edge/source/sink/orphan counts, project/dataset tree, and architecture health validation.
+*   **Statistics Panel**: Centered popup with layer distribution bars, edge/source/sink/orphan counts, project/dataset tree, and architecture health validation (now detects **circular dependencies**).
 *   **Schema Preview**: Extracts columns from DDL, CTAS (`CREATE TABLE AS SELECT`), and `CREATE VIEW` statements.
 
-### 🎨 Linear-Inspired UI (New in v0.4.0 ✨)
+### 🎨 Linear-Inspired UI (New in v0.4.6 ✨)
 *   **Design Token System**: ~80 CSS custom properties for consistent theming across all components.
 *   **Premium Dark Theme**: Deep `#0d0d0d` canvas with warm white text (`#e8e8e6`), never pure white.
 *   **Refined Light Theme**: Warm off-white `#f7f6f3` backgrounds — never harsh pure white.
@@ -99,12 +106,13 @@ Specially optimized for the **Medallion Architecture** (Bronze, Silver, Gold) an
 *   **Custom Scrollbars**: Subtle, styled scrollbars matching the theme.
 *   **Focus Rings**: Accessible focus indicators using the accent color.
 
-### ⚙️ Customization
+### ⚙️ Customization & Export
 *   **Premium UI**:
     *   **Themes**: Toggle between Light and Dark modes.
-    *   **Palettes**: Choose from **Standard**, **Vivid**, **Pastel**, or **Linear** (new — muted, LCH-inspired tones) color schemes.
+    *   **Palettes**: Choose from **Standard**, **Vivid**, **Pastel**, or **Linear** (LCH-inspired tones) color schemes.
     *   **Styles**: Switch between "Full" (colored body) and "Minimal" (colored border) node styles.
-*   **Export**: Save high-resolution **PNG** or vector **SVG** diagrams for documentation.
+*   **Export Dictionary**: Generate and download a comprehensive Markdown Data Dictionary report of your entire DAG.
+*   **Export Graph**: Save high-resolution **PNG** or vector **SVG** diagrams for documentation.
 
 ---
 
@@ -131,7 +139,7 @@ Install easily via `pip`:
 pip install sql-dag-flow
 ```
 
-To update to the latest version (**v0.4.0**):
+To update to the latest version (**v0.4.6**):
 
 ```bash
 pip install --upgrade sql-dag-flow

{sql_dag_flow-0.4.2 → sql_dag_flow-0.4.7}/README.md

@@ -48,8 +48,11 @@ Specially optimized for the **Medallion Architecture** (Bronze, Silver, Gold) an
     *   **Hide/Show**: Toggle visibility of individual nodes or full branches.
 *   **Advanced Navigation**:
     *   **Sidebar**: Grouped list of nodes with toggle between **By Layer** and **By Project/Dataset** views.
+    *   **Command Palette (Cmd+P)**: Instantly search and navigate to nodes across large projects.
+    *   **Keyboard Arrow Navigation**: Rapidly explore lineage by moving ← (upstream) and → (downstream) between connected nodes.
+    *   **Breadcrumb Trail**: Maintain context while drilling down with a visual history of visited nodes.
     *   **SQL Content Search**: Search inside SQL file content across all nodes — find WHERE clauses, JOINs, or any keyword.
-    *   **Details Panel**: View formatted SQL code, schema preview (DDL, CTAS, Views), and node configuration.
+    *   **Details Panel**: View formatted SQL code, schema preview (DDL, CTAS, Views), node configuration, and add **custom descriptions** to document models.
 
 ### 📝 Notes & Annotations
 *   **Center Placement**: New notes spawn exactly in the center of your view.
@@ -59,12 +62,16 @@ Specially optimized for the **Medallion Architecture** (Bronze, Silver, Gold) an
     *   **Groups**: Create visual containers to group related nodes.
 
 ### 📊 Discovery & Analysis Tools
+*   **Impact Analysis**: Visualize blast radius before making changes. Highlights downstream models, column usage, and risk levels.
+*   **Diff View on Refresh**: Automatically summarizes added, removed, and modified nodes/edges after code changes.
+*   **Column Usage Tracking**: Schema Preview shows which specific columns are used by downstream consumers.
+*   **Staleness Detection**: Automatically flags inactive models (`Last Modified > 90d` = Stale) to help clean up legacy pipelines.
 *   **Business Rule Extraction**: Automatically detects and displays WHERE filters, CASE logic, HAVING clauses, and aggregations from each SQL model.
 *   **Complexity Scoring**: Weighted metric per node (JOINs×3, CTEs×2, Subqueries×3, Filters×1, CASE×2, Aggregations×1, UNIONs×2) with color-coded badges (🟢 Low, 🟡 Medium, 🟠 High, 🔴 Very High). Toggleable via ⚡ button.
-*   **Statistics Panel**: Centered popup with layer distribution bars, edge/source/sink/orphan counts, project/dataset tree, and architecture health validation.
+*   **Statistics Panel**: Centered popup with layer distribution bars, edge/source/sink/orphan counts, project/dataset tree, and architecture health validation (now detects **circular dependencies**).
 *   **Schema Preview**: Extracts columns from DDL, CTAS (`CREATE TABLE AS SELECT`), and `CREATE VIEW` statements.
 
-### 🎨 Linear-Inspired UI (New in v0.4.0 ✨)
+### 🎨 Linear-Inspired UI (New in v0.4.6 ✨)
 *   **Design Token System**: ~80 CSS custom properties for consistent theming across all components.
 *   **Premium Dark Theme**: Deep `#0d0d0d` canvas with warm white text (`#e8e8e6`), never pure white.
 *   **Refined Light Theme**: Warm off-white `#f7f6f3` backgrounds — never harsh pure white.
@@ -74,12 +81,13 @@ Specially optimized for the **Medallion Architecture** (Bronze, Silver, Gold) an
 *   **Custom Scrollbars**: Subtle, styled scrollbars matching the theme.
 *   **Focus Rings**: Accessible focus indicators using the accent color.
 
-### ⚙️ Customization
+### ⚙️ Customization & Export
 *   **Premium UI**:
     *   **Themes**: Toggle between Light and Dark modes.
-    *   **Palettes**: Choose from **Standard**, **Vivid**, **Pastel**, or **Linear** (new — muted, LCH-inspired tones) color schemes.
+    *   **Palettes**: Choose from **Standard**, **Vivid**, **Pastel**, or **Linear** (LCH-inspired tones) color schemes.
     *   **Styles**: Switch between "Full" (colored body) and "Minimal" (colored border) node styles.
-*   **Export**: Save high-resolution **PNG** or vector **SVG** diagrams for documentation.
+*   **Export Dictionary**: Generate and download a comprehensive Markdown Data Dictionary report of your entire DAG.
+*   **Export Graph**: Save high-resolution **PNG** or vector **SVG** diagrams for documentation.
 
 ---
 
@@ -106,7 +114,7 @@ Install easily via `pip`:
 pip install sql-dag-flow
 ```
 
-To update to the latest version (**v0.4.0**):
+To update to the latest version (**v0.4.6**):
 
 ```bash
 pip install --upgrade sql-dag-flow

{sql_dag_flow-0.4.2 → sql_dag_flow-0.4.7}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "sql-dag-flow"
-version = "0.4.2"
+version = "0.4.7"
 description = "A sophisticated SQL lineage visualization tool for Medallion Architectures."
 readme = "README.md"
 requires-python = ">=3.8"

{sql_dag_flow-0.4.2 → sql_dag_flow-0.4.7}/src/sql_dag_flow/main.py

@@ -1,4 +1,5 @@
 from fastapi import FastAPI, HTTPException, Body
+from fastapi.responses import FileResponse, Response
 from pydantic import BaseModel
 from fastapi.middleware.cors import CORSMiddleware
 from fastapi.staticfiles import StaticFiles
@@ -10,6 +11,8 @@ import json
 import webbrowser
 import threading
 import time
+import socket
+import argparse
 import shutil
 from .parser import parse_sql_files, build_graph
 
@@ -34,14 +37,15 @@ CURRENT_DIRECTORY = os.getcwd() # Default, updated by start()
 DIAGRAM_FILE = "sql_diagram.json"
 
 @app.get("/graph")
-def get_graph(dialect: str = "bigquery", discovery: bool = False):
+def get_graph(dialect: str = "bigquery", discovery: bool = False, expanded_nodes: str = ""):
     """Parses SQL files in the current directory and returns graph data."""
     if not os.path.exists(CURRENT_DIRECTORY):
-        return {"nodes": [], "edges": [], "error": "Directory not found"}
+        return {"nodes": [], "edges": [], "cycles": [], "error": "Directory not found"}
         
+    exp_nodes_list = [n.strip() for n in expanded_nodes.split(",")] if expanded_nodes else []
     tables = parse_sql_files(CURRENT_DIRECTORY, dialect=dialect)
-    nodes, edges = build_graph(tables, discovery_mode=discovery)
-    return {"nodes": nodes, "edges": edges}
+    nodes, edges, cycles = build_graph(tables, discovery_mode=discovery, expanded_nodes=exp_nodes_list)
+    return {"nodes": nodes, "edges": edges, "cycles": cycles}
 
 @app.post("/config/path")
 def set_path(path_data: dict = Body(...)):
@@ -93,15 +97,126 @@ def get_filtered_graph(data: dict = Body(...)):
     subfolders = data.get("subfolders") # List of strings or None
     dialect = data.get("dialect", "bigquery")
     discovery = data.get("discovery", False)
+    expanded_nodes = data.get("expanded_nodes", [])
     
     tables = parse_sql_files(CURRENT_DIRECTORY, allowed_subfolders=subfolders, dialect=dialect)
-    nodes, edges = build_graph(tables, discovery_mode=discovery)
-    return {"nodes": nodes, "edges": edges}
+    nodes, edges, cycles = build_graph(tables, discovery_mode=discovery, expanded_nodes=expanded_nodes)
+    return {"nodes": nodes, "edges": edges, "cycles": cycles}
 
 @app.get("/config/path")
 def get_path():
     return {"path": CURRENT_DIRECTORY}
 
+@app.get("/export")
+def export_data_dictionary(dialect: str = "bigquery"):
+    """Generates a Markdown data dictionary report."""
+    if not os.path.exists(CURRENT_DIRECTORY):
+        raise HTTPException(status_code=400, detail="Directory not found")
+    
+    tables = parse_sql_files(CURRENT_DIRECTORY, dialect=dialect)
+    nodes, edges, cycles = build_graph(tables, discovery_mode=False)
+    
+    lines = []
+    lines.append(f"# Data Dictionary")
+    lines.append(f"")
+    lines.append(f"**Project Path:** `{CURRENT_DIRECTORY}`  ")
+    lines.append(f"**Total Models:** {len(nodes)}  ")
+    lines.append(f"**Total Dependencies:** {len(edges)}  ")
+    lines.append(f"")
+    
+    if cycles:
+        lines.append(f"## ⚠️ Circular Dependencies ({len(cycles)})")
+        lines.append(f"")
+        for i, cycle in enumerate(cycles, 1):
+            cycle_str = " → ".join([n['label'] for n in cycle])
+            lines.append(f"{i}. {cycle_str} → {cycle[0]['label']}")
+        lines.append(f"")
+    
+    # Group by layer
+    layer_order = ['bronze', 'silver', 'gold', 'external', 'cte', 'other']
+    node_by_layer = {}
+    for n in nodes:
+        layer = n['data'].get('layer', 'other')
+        if layer not in node_by_layer:
+            node_by_layer[layer] = []
+        node_by_layer[layer].append(n)
+    
+    for layer in layer_order:
+        layer_nodes = node_by_layer.get(layer, [])
+        if not layer_nodes:
+            continue
+        lines.append(f"## {layer.capitalize()} Layer ({len(layer_nodes)} models)")
+        lines.append(f"")
+        
+        for n in sorted(layer_nodes, key=lambda x: x['data']['label']):
+            d = n['data'].get('details', {})
+            label = n['data']['label']
+            lines.append(f"### {label}")
+            lines.append(f"")
+            
+            # Metadata table
+            lines.append(f"| Property | Value |")
+            lines.append(f"|----------|-------|")
+            lines.append(f"| **Project** | {d.get('project', '-')} |")
+            lines.append(f"| **Dataset** | {d.get('dataset', '-')} |")
+            lines.append(f"| **Type** | {d.get('type', 'table')} |")
+            lines.append(f"| **Layer** | {layer} |")
+            
+            complexity = d.get('complexity', {})
+            if complexity:
+                lines.append(f"| **Complexity** | {complexity.get('score', 0)} |")
+            
+            header_meta = d.get('header_meta', {})
+            if header_meta.get('description'):
+                lines.append(f"| **Description** | {header_meta['description']} |")
+            if header_meta.get('author'):
+                lines.append(f"| **Author** | {header_meta['author']} |")
+            
+            incoming = n['data'].get('incomingCount', 0)
+            downstream = n['data'].get('downstreamCount', 0)
+            lines.append(f"| **Dependencies** | {incoming} incoming, {downstream} downstream |")
+            lines.append(f"")
+            
+            # Dependencies
+            deps = d.get('dependencies', {})
+            if deps:
+                lines.append(f"**Dependencies:** {', '.join(f'`{k}` ({v})' for k, v in deps.items()) if isinstance(deps, dict) else ', '.join(f'`{x}`' for x in deps)}")
+                lines.append(f"")
+            
+            # Business Rules
+            br = d.get('business_rules', {})
+            rules_items = []
+            for cat, items in br.items():
+                if items:
+                    rules_items.extend([f"- **{cat}**: `{r}`" for r in items[:3]])
+            if rules_items:
+                lines.append(f"**Business Rules:**")
+                lines.append(f"")
+                lines.extend(rules_items[:6])
+                lines.append(f"")
+            
+            # Column consumers
+            col_consumers = d.get('column_consumers', {})
+            if col_consumers:
+                lines.append(f"**Column Usage (downstream):**")
+                lines.append(f"")
+                lines.append(f"| Column | Consumers |")
+                lines.append(f"|--------|-----------|")
+                for col, consumers in sorted(col_consumers.items()):
+                    consumer_labels = ', '.join([c['label'] for c in consumers])
+                    lines.append(f"| `{col}` | {consumer_labels} |")
+                lines.append(f"")
+            
+            lines.append(f"---")
+            lines.append(f"")
+    
+    md_content = "\n".join(lines)
+    return Response(
+        content=md_content,
+        media_type="text/markdown",
+        headers={"Content-Disposition": "attachment; filename=data_dictionary.md"}
+    )
+
 class SaveRequest(BaseModel):
     nodes: list
     edges: list
@@ -257,31 +372,72 @@ if os.path.exists(STATIC_DIR):
         response.headers["Expires"] = "0"
         return response
 
+def _is_port_available(port):
+    """Check if a port is available by attempting to bind to it."""
+    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+        try:
+            s.bind(('127.0.0.1', port))
+            return True
+        except OSError:
+            return False
+
+def _find_available_port(start_port, max_attempts=10):
+    """Find an available port starting from start_port."""
+    for offset in range(max_attempts):
+        port = start_port + offset
+        if _is_port_available(port):
+            return port
+    return None
+
 def start():
     """Entry point for the CLI tool."""
     global CURRENT_DIRECTORY
     
-    # CLI Argument Parsing
-    if len(sys.argv) > 1:
-        path_arg = sys.argv[1]
-        if os.path.exists(path_arg):
-            CURRENT_DIRECTORY = os.path.abspath(path_arg)
+    # CLI Argument Parsing with argparse
+    parser = argparse.ArgumentParser(
+        prog='sql-dag-flow',
+        description='SQL DAG Flow - Medallion Architecture Visualizer'
+    )
+    parser.add_argument('path', nargs='?', default=None, help='Path to SQL project folder')
+    parser.add_argument('--port', '-p', type=int, default=8000, help='Port to run the server on (default: 8000)')
+    
+    # Use parse_known_args to be tolerant of unexpected args
+    args, unknown = parser.parse_known_args()
+    
+    if args.path:
+        # Normalize the path (handles backslashes on Windows, trailing separators, etc.)
+        normalized_path = os.path.normpath(os.path.abspath(args.path))
+        if os.path.exists(normalized_path) and os.path.isdir(normalized_path):
+            CURRENT_DIRECTORY = normalized_path
             print(f"Setting project path from CLI: {CURRENT_DIRECTORY}")
         else:
-            print(f"Warning: Path '{path_arg}' does not exist. Using defaults.")
+            print(f"Warning: Path '{args.path}' does not exist or is not a directory. Using current directory.")
+            CURRENT_DIRECTORY = os.getcwd()
     else:
         CURRENT_DIRECTORY = os.getcwd()
         print(f"Using current directory: {CURRENT_DIRECTORY}")
 
+    # Find an available port
+    requested_port = args.port
+    port = _find_available_port(requested_port)
+    
+    if port is None:
+        print(f"Error: Could not find an available port (tried {requested_port}-{requested_port + 9}).")
+        sys.exit(1)
+    
+    if port != requested_port:
+        print(f"Port {requested_port} is in use. Using port {port} instead.")
+    else:
+        print(f"Starting server on port {port}")
+
     def open_browser():
         time.sleep(1.5)
-        webbrowser.open("http://localhost:8000")
+        webbrowser.open(f"http://localhost:{port}")
 
     threading.Thread(target=open_browser, daemon=True).start()
     
     # Run uvicorn programmatically
-    # Note: When running programmatically, reload=True is not supported easily without other hacks
-    uvicorn.run(app, host="127.0.0.1", port=8000)
+    uvicorn.run(app, host="127.0.0.1", port=port)
 
 if __name__ == "__main__":
     start()

{sql_dag_flow-0.4.2 → sql_dag_flow-0.4.7}/src/sql_dag_flow/parser.py

@@ -1,14 +1,10 @@
 import os
+import re
+import time
 import sqlglot
 from sqlglot import exp
 import networkx as nx
 
-import os
-import sqlglot
-from sqlglot import exp
-import networkx as nx
-import re
-
 def parse_sql_files(directory, allowed_subfolders=None, dialect="bigquery"):
     """
     Recursively scans a directory for .sql files and parses them.
@@ -312,6 +308,74 @@ def parse_sql_files(directory, allowed_subfolders=None, dialect="bigquery"):
                     )
                     
                     complexity_breakdown["score"] = complexity_score
+                    
+                    # ===== Column Reference Extraction =====
+                    # Extract which columns this model references from each source table
+                    # This enables downstream impact analysis
+                    column_references = {}  # source_table -> [columns]
+                    
+                    # Build alias -> table name mapping for the query
+                    alias_map = {}  # alias -> full_table_name
+                    for t in parsed.find_all(exp.Table):
+                        t_name = t.name
+                        t_full = t_name
+                        if t.db:
+                            t_full = f"{t.db}.{t_name}"
+                            if t.catalog:
+                                t_full = f"{t.catalog}.{t.db}.{t_name}"
+                        if t.alias:
+                            alias_map[t.alias] = t_full
+                        alias_map[t_name] = t_full
+                    
+                    # Extract column references with their table qualifier
+                    for col in parsed.find_all(exp.Column):
+                        col_name = col.name
+                        col_table = col.table  # The table qualifier (alias or name)
+                        if col_table and col_table in alias_map:
+                            source = alias_map[col_table]
+                            if source not in column_references:
+                                column_references[source] = set()
+                            column_references[source].add(col_name)
+                    
+                    # Convert sets to sorted lists for JSON serialization
+                    column_references = {k: sorted(list(v)) for k, v in column_references.items()}
+                    
+                    # ===== Header Comment Extraction =====
+                    # Extract metadata from SQL header comments:
+                    #   -- @description: ...
+                    #   -- @author: ...
+                    #   -- @modified: ...
+                    #   -- Description: ... (first block comment)
+                    header_meta = {}
+                    header_lines = []
+                    for line in sql_content.split('\n'):
+                        stripped = line.strip()
+                        if stripped.startswith('--'):
+                            header_lines.append(stripped[2:].strip())
+                        elif stripped == '' and not header_lines:
+                            continue  # skip leading blank lines
+                        else:
+                            break  # stop at first non-comment line
+                    
+                    for hline in header_lines:
+                        # Match @key: value patterns
+                        meta_match = re.match(r'^@(\w+)[:\s]+(.+)$', hline, re.IGNORECASE)
+                        if meta_match:
+                            key = meta_match.group(1).lower()
+                            header_meta[key] = meta_match.group(2).strip()
+                    
+                    # If no @description, use first non-@ comment lines as description
+                    if 'description' not in header_meta:
+                        desc_lines = [l for l in header_lines if not l.startswith('@') and l.strip()]
+                        if desc_lines:
+                            header_meta['description'] = ' '.join(desc_lines[:3])
+                    # ===== File Modification Timestamp =====
+                    try:
+                        mtime = os.path.getmtime(filepath)
+                        days_ago = int((time.time() - mtime) / 86400)
+                    except Exception:
+                        mtime = None
+                        days_ago = None
                              
                     tables[filename_base] = { 
                         "id": filename_base,
@@ -326,7 +390,10 @@ def parse_sql_files(directory, allowed_subfolders=None, dialect="bigquery"):
                         "ctes": defined_ctes,
                         "cte_deps": cte_deps,
                         "business_rules": business_rules,
-                        "complexity": complexity_breakdown
+                        "complexity": complexity_breakdown,
+                        "column_references": column_references,
+                        "header_meta": header_meta,
+                        "last_modified_days": days_ago
                     }
                 except Exception as e:
                     print(f"Error parsing {filepath}: {e}")
@@ -346,11 +413,12 @@ def parse_sql_files(directory, allowed_subfolders=None, dialect="bigquery"):
     return tables
 
 
-def build_graph(tables, discovery_mode=False):
+def build_graph(tables, discovery_mode=False, expanded_nodes=None):
     """
     Constructs nodes and edges for React Flow.
     If discovery_mode is True, creates 'ghost' nodes for dependencies 
     that are not found in the parsed tables.
+    Also creates ghost nodes for any node whose ID is in expanded_nodes list.
     """
     nodes = []
     edges = []
@@ -410,8 +478,8 @@ def build_graph(tables, discovery_mode=False):
                          cte_name = ":".join(parts[2:])
                          cte_internal_deps = tables[source_id].get("cte_deps", {}).get(cte_name, {})
                          
-                         if discovery_mode:
-                             # Discovery Mode: Create CTE ghost node with incoming edges
+                         if discovery_mode or (expanded_nodes and source_id in expanded_nodes):
+                             # Discovery Mode or Expanded: Create CTE ghost node with incoming edges
                              cte_id = dep
                              
                              if cte_id not in missing_nodes:
@@ -504,8 +572,8 @@ def build_graph(tables, discovery_mode=False):
                                          incoming_edges_count[source_id] = incoming_edges_count.get(source_id, 0) + 1
                      continue
 
-                 # Handle missing external nodes (discovery mode only)
-                 if discovery_mode and not target_id:
+                 # Handle missing external nodes (discovery mode or expanded node only)
+                 if (discovery_mode or (expanded_nodes and source_id in expanded_nodes)) and not target_id:
                          # Create a unique ID for the missing node
                          # Use the full dependency name as the ID
                          ghost_id = dep
@@ -555,27 +623,75 @@ def build_graph(tables, discovery_mode=False):
     for edge in edges:
         G.add_edge(edge["source"], edge["target"])
 
+    # ===== Compute Column Downstream Consumers =====
+    # For each node, figure out which downstream models reference its columns
+    # Result: column_consumers[node_id] = { "col_name": [{"node": consumer_id, "label": consumer_label}] }
+    column_consumers = {}
+    
+    for consumer_id, consumer_data in all_nodes_data.items():
+        col_refs = consumer_data.get("column_references", {})
+        for source_ref, columns in col_refs.items():
+            # Resolve source_ref to a node_id using lookup
+            source_node_id = lookup.get(source_ref)
+            if not source_node_id and "." in source_ref:
+                source_node_id = lookup.get(source_ref.split(".")[-1])
+            
+            if source_node_id and source_node_id in all_nodes_data:
+                if source_node_id not in column_consumers:
+                    column_consumers[source_node_id] = {}
+                for col in columns:
+                    if col not in column_consumers[source_node_id]:
+                        column_consumers[source_node_id][col] = []
+                    column_consumers[source_node_id][col].append({
+                        "node": consumer_id,
+                        "label": consumer_data.get("label", consumer_id)
+                    })
+
     for table_name, data in all_nodes_data.items():
         # Calculate nested dependencies (all ancestors in the dependency graph)
         nested_count = 0
         if G.has_node(table_name):
             try:
-                # ancestors() returns all nodes u such that there is a path from u to table_name
                 nested_count = len(nx.ancestors(G, table_name))
             except Exception:
-                pass # distinct graph parts or cycles? cycles shouldn't exist in DAG but safety first
+                pass
+        
+        # Get downstream impact count
+        downstream_count = 0
+        if G.has_node(table_name):
+            try:
+                downstream_count = len(nx.descendants(G, table_name))
+            except Exception:
+                pass
 
         nodes.append({
             "id": table_name,
             "data": {
                 "label": data["label"], 
                 "layer": data["layer"],
-                "details": data,
+                "details": {
+                    **data,
+                    "column_consumers": column_consumers.get(table_name, {})
+                },
                 "incomingCount": incoming_edges_count.get(table_name, 0),
-                "nestedCount": nested_count
+                "nestedCount": nested_count,
+                "downstreamCount": downstream_count
             },
             "position": {"x": 0, "y": 0}, 
             "type": "custom", 
         })
 
-    return nodes, edges
+    # ===== Cycle Detection =====
+    cycles = []
+    try:
+        raw_cycles = list(nx.simple_cycles(G))
+        for cycle in raw_cycles:
+            cycle_labels = []
+            for nid in cycle:
+                label = all_nodes_data.get(nid, {}).get("label", nid)
+                cycle_labels.append({"id": nid, "label": label})
+            cycles.append(cycle_labels)
+    except Exception:
+        pass
+
+    return nodes, edges, cycles