sql-dag-flow 0.7.1__tar.gz → 0.9.0__tar.gz

{sql_dag_flow-0.7.1/src/sql_dag_flow.egg-info → sql_dag_flow-0.9.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sql-dag-flow
-Version: 0.7.1
+Version: 0.9.0
 Summary: A sophisticated SQL lineage visualization tool for Medallion Architectures.
 Author-email: Flavio Sandoval <dsandovalflavio@gmail.com>
 License: MIT
@@ -21,6 +21,8 @@ Requires-Dist: uvicorn
 Requires-Dist: sqlglot
 Requires-Dist: networkx
 Requires-Dist: pydantic
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
 Dynamic: license-file
 
 # SQL DAG Flow
@@ -58,10 +60,13 @@ Specially optimized for the **Medallion Architecture** (Bronze, Silver, Gold) an
 
 ### 🔍 Visualization & Analysis
 *   **Automatic Parsing**: Recursively scans `.sql` files to detect dependencies (`FROM`, `JOIN`, `CTE`s) using `sqlglot`.
+*   **Trustworthy Lineage (New in v0.8.0 🎯)**: Every `.sql` file appears exactly once, even when two folders hold the same filename (`staging/customers.sql` and `marts/customers.sql` no longer overwrite each other — they're addressed by path). Dependency resolution is strict: a reference naming a dataset will never be matched to a model in a *different* dataset, and an ambiguous name draws no edge at all. Anything unresolved, ambiguous or duplicated is reported in a warnings banner instead of failing silently — a missing edge you can see beats a wrong edge you can't.
 *   **Persistent File Cache (New in v0.6.0 ⚡)**: Avoids re-parsing unchanged SQL files across application restarts. Instantly loads your DAG on consecutive days.
 *   **Selective Processing (New in v0.6.0 🚀)**: Dramatically improves performance on large projects (10x-20x) by only analyzing visible nodes for complex features like Qualify Columns and Column-Level Lineage.
 *   **Scoped Views (New in v0.7.0 🎯)**: A saved diagram is now a *scope*. Reopening or refreshing it only re-parses the models actually on your canvas, so parsing stays proportional to your view instead of your whole project — and newly added `.sql` files never flood a curated architecture.
 *   **Scan New Models (New in v0.7.0 🔎)**: A pure filesystem diff (zero SQL parsing, instant on huge projects) that surfaces `.sql` files not yet on your canvas, so you pull them in on demand instead of re-indexing everything.
+*   **Stored Procedures (New in v0.9.0 ⚙️)**: Full support for `CREATE PROCEDURE` files. The body between `BEGIN ... END` is parsed statement by statement, so a procedure shows both what it **reads** (`FROM`/`JOIN`) and what it **writes** (`INSERT`, `MERGE`, `UPDATE`, `DELETE`) — writes become outgoing edges, placing the procedure between its inputs and the tables it produces. `CALL` between procedures is tracked too. Multi-statement scripts benefit as well: every statement is inspected, not just the first.
+*   **20x Faster Column Analysis (New in v0.9.0 ⚡)**: The `qualify_columns` and column-lineage passes used to receive the *entire* project's schema for every model, and re-parse a model once per column. They now get only the tables a model actually reads, and reuse a single parsed AST. On a 300-model project the visible-node analysis went from **457 ms to 23 ms per model** — a full-project run that never finished within two minutes now takes under 7 seconds. The same fix repaired a silent bug: computed columns stored an expression in the type slot, which made `qualify_columns` raise and discard its own work on **every** model. It now runs successfully across the board.
 *   **Medallion Architecture Support**: Automatically categorizes and colors nodes based on folder structure (Bronze, Silver, Gold).
 *   **Advanced Discovery Mode (Improved in v0.6.0 👻)**: Visualize "Ghost Nodes" (missing files or external tables). Includes specific filters to show **Both**, **Only External**, or **Only CTEs**.
 *   **CTE Visualization**: Detects internal Common Table Expressions and displays them as distinct Pink nodes.
@@ -153,7 +158,7 @@ Install easily via `pip`:
 pip install sql-dag-flow
 ```
 
-To update to the latest version (**v0.7.1**):
+To update to the latest version (**v0.9.0**):
 
 ```bash
 pip install --upgrade sql-dag-flow

{sql_dag_flow-0.7.1 → sql_dag_flow-0.9.0}/README.md

@@ -33,10 +33,13 @@ Specially optimized for the **Medallion Architecture** (Bronze, Silver, Gold) an
 
 ### 🔍 Visualization & Analysis
 *   **Automatic Parsing**: Recursively scans `.sql` files to detect dependencies (`FROM`, `JOIN`, `CTE`s) using `sqlglot`.
+*   **Trustworthy Lineage (New in v0.8.0 🎯)**: Every `.sql` file appears exactly once, even when two folders hold the same filename (`staging/customers.sql` and `marts/customers.sql` no longer overwrite each other — they're addressed by path). Dependency resolution is strict: a reference naming a dataset will never be matched to a model in a *different* dataset, and an ambiguous name draws no edge at all. Anything unresolved, ambiguous or duplicated is reported in a warnings banner instead of failing silently — a missing edge you can see beats a wrong edge you can't.
 *   **Persistent File Cache (New in v0.6.0 ⚡)**: Avoids re-parsing unchanged SQL files across application restarts. Instantly loads your DAG on consecutive days.
 *   **Selective Processing (New in v0.6.0 🚀)**: Dramatically improves performance on large projects (10x-20x) by only analyzing visible nodes for complex features like Qualify Columns and Column-Level Lineage.
 *   **Scoped Views (New in v0.7.0 🎯)**: A saved diagram is now a *scope*. Reopening or refreshing it only re-parses the models actually on your canvas, so parsing stays proportional to your view instead of your whole project — and newly added `.sql` files never flood a curated architecture.
 *   **Scan New Models (New in v0.7.0 🔎)**: A pure filesystem diff (zero SQL parsing, instant on huge projects) that surfaces `.sql` files not yet on your canvas, so you pull them in on demand instead of re-indexing everything.
+*   **Stored Procedures (New in v0.9.0 ⚙️)**: Full support for `CREATE PROCEDURE` files. The body between `BEGIN ... END` is parsed statement by statement, so a procedure shows both what it **reads** (`FROM`/`JOIN`) and what it **writes** (`INSERT`, `MERGE`, `UPDATE`, `DELETE`) — writes become outgoing edges, placing the procedure between its inputs and the tables it produces. `CALL` between procedures is tracked too. Multi-statement scripts benefit as well: every statement is inspected, not just the first.
+*   **20x Faster Column Analysis (New in v0.9.0 ⚡)**: The `qualify_columns` and column-lineage passes used to receive the *entire* project's schema for every model, and re-parse a model once per column. They now get only the tables a model actually reads, and reuse a single parsed AST. On a 300-model project the visible-node analysis went from **457 ms to 23 ms per model** — a full-project run that never finished within two minutes now takes under 7 seconds. The same fix repaired a silent bug: computed columns stored an expression in the type slot, which made `qualify_columns` raise and discard its own work on **every** model. It now runs successfully across the board.
 *   **Medallion Architecture Support**: Automatically categorizes and colors nodes based on folder structure (Bronze, Silver, Gold).
 *   **Advanced Discovery Mode (Improved in v0.6.0 👻)**: Visualize "Ghost Nodes" (missing files or external tables). Includes specific filters to show **Both**, **Only External**, or **Only CTEs**.
 *   **CTE Visualization**: Detects internal Common Table Expressions and displays them as distinct Pink nodes.
@@ -128,7 +131,7 @@ Install easily via `pip`:
 pip install sql-dag-flow
 ```
 
-To update to the latest version (**v0.7.1**):
+To update to the latest version (**v0.9.0**):
 
 ```bash
 pip install --upgrade sql-dag-flow

{sql_dag_flow-0.7.1 → sql_dag_flow-0.9.0}/pyproject.toml

@@ -1,42 +1,49 @@
-[build-system]
-requires = ["setuptools>=42", "wheel"]
-build-backend = "setuptools.build_meta"
-
-[project]
-name = "sql-dag-flow"
-version = "0.7.1"
-description = "A sophisticated SQL lineage visualization tool for Medallion Architectures."
-readme = "README.md"
-requires-python = ">=3.8"
-license = {text = "MIT"}
-authors = [
-    {name = "Flavio Sandoval", email = "dsandovalflavio@gmail.com"}
-]
-keywords = ["sql", "lineage", "dag", "visualization", "medallion-architecture", "data-engineering"]
-classifiers = [
-    "Development Status :: 4 - Beta",
-    "Intended Audience :: Developers",
-    "License :: OSI Approved :: MIT License",
-    "Programming Language :: Python :: 3",
-    "Programming Language :: Python :: 3.8",
-    "Programming Language :: Python :: 3.9",
-    "Programming Language :: Python :: 3.10",
-    "Programming Language :: Python :: 3.11",
-]
-dependencies = [
-    "fastapi",
-    "uvicorn",
-    "sqlglot",
-    "networkx",
-    "pydantic"
-]
-
-[project.scripts]
-sql-dag-flow = "sql_dag_flow.main:start"
-
-[tool.setuptools.packages.find]
-where = ["src"]
-include = ["sql_dag_flow*"]
-
-[tool.setuptools.package-data]
+[build-system]
+requires = ["setuptools>=42", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "sql-dag-flow"
+version = "0.9.0"
+description = "A sophisticated SQL lineage visualization tool for Medallion Architectures."
+readme = "README.md"
+requires-python = ">=3.8"
+license = {text = "MIT"}
+authors = [
+    {name = "Flavio Sandoval", email = "dsandovalflavio@gmail.com"}
+]
+keywords = ["sql", "lineage", "dag", "visualization", "medallion-architecture", "data-engineering"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.8",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+]
+dependencies = [
+    "fastapi",
+    "uvicorn",
+    "sqlglot",
+    "networkx",
+    "pydantic"
+]
+
+[project.scripts]
+sql-dag-flow = "sql_dag_flow.main:start"
+
+[project.optional-dependencies]
+dev = ["pytest>=7"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["src", "tests"]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["sql_dag_flow*"]
+
+[tool.setuptools.package-data]
 sql_dag_flow = ["static/**/*"]

{sql_dag_flow-0.7.1 → sql_dag_flow-0.9.0}/src/sql_dag_flow/main.py

@@ -56,33 +56,66 @@ def _cached_parse(directory, subfolders_tuple, dialect, visible_node_ids=None, t
     _parse_cache["time"] = now
     return tables
 
-@app.get("/graph")
-def get_graph(dialect: str = "bigquery", discovery: bool = False, expanded_nodes: str = "", visible_node_ids: str = "", discovery_filter: str = "all"):
-    """Parses SQL files in the current directory and returns graph data."""
+def _normalize_expanded(expanded_nodes):
+    """Accept the dict form, the legacy list form, or the query-string form."""
+    if isinstance(expanded_nodes, dict):
+        return expanded_nodes
+    if isinstance(expanded_nodes, list):
+        return {n: "all" for n in expanded_nodes}
+    result = {}
+    for item in (expanded_nodes or "").split(","):
+        item = item.strip()
+        if not item:
+            continue
+        parts = item.rsplit(":", 1)
+        if len(parts) == 2 and parts[1] in ("all", "external", "cte"):
+            result[parts[0]] = parts[1]
+        else:
+            result[item] = "all"
+    return result
+
+
+def _graph_response(dialect="bigquery", discovery=False, expanded_nodes=None,
+                    discovery_filter="all", subfolders=None,
+                    visible_node_ids=None, target_ids=None):
+    """Single implementation behind every graph route.
+
+    The three routes below differ only in how the request arrives (query
+    string, subfolder filter, explicit scope) — the parse → build → serialize
+    pipeline is identical, so it lives here once.
+    """
     if not os.path.exists(CURRENT_DIRECTORY):
-        return {"nodes": [], "edges": [], "cycles": [], "error": "Directory not found"}
-        
-    # Parse expanded_nodes as dict {node_id: mode}
-    exp_nodes_dict = {}
-    if expanded_nodes:
-        for item in expanded_nodes.split(","):
-            item = item.strip()
-            if not item:
-                continue
-            parts = item.rsplit(":", 1)
-            if len(parts) == 2 and parts[1] in ("all", "external", "cte"):
-                exp_nodes_dict[parts[0]] = parts[1]
-            else:
-                exp_nodes_dict[item] = "all"
-    visible_list = [n.strip() for n in visible_node_ids.split(",")] if visible_node_ids else None
+        return {"nodes": [], "edges": [], "cycles": [], "warnings": [], "error": "Directory not found"}
+
     try:
-        tables = _cached_parse(CURRENT_DIRECTORY, None, dialect, visible_node_ids=visible_list)
-        nodes, edges, cycles = build_graph(tables, discovery_mode=discovery, expanded_nodes=exp_nodes_dict, discovery_filter=discovery_filter)
-        return {"nodes": nodes, "edges": edges, "cycles": cycles}
+        tables = _cached_parse(
+            CURRENT_DIRECTORY,
+            tuple(subfolders) if subfolders else None,
+            dialect,
+            visible_node_ids=visible_node_ids,
+            target_ids=target_ids,
+        )
+        nodes, edges, cycles, warnings = build_graph(
+            tables,
+            discovery_mode=discovery,
+            expanded_nodes=_normalize_expanded(expanded_nodes),
+            discovery_filter=discovery_filter,
+        )
+        return {"nodes": nodes, "edges": edges, "cycles": cycles, "warnings": warnings}
     except Exception as e:
         import traceback
         traceback.print_exc()
-        return {"nodes": [], "edges": [], "error": f"Backend Error: {str(e)}"}
+        return {"nodes": [], "edges": [], "cycles": [], "warnings": [], "error": f"Backend Error: {str(e)}"}
+
+
+@app.get("/graph")
+def get_graph(dialect: str = "bigquery", discovery: bool = False, expanded_nodes: str = "", visible_node_ids: str = "", discovery_filter: str = "all"):
+    """Full-project graph."""
+    visible_list = [n.strip() for n in visible_node_ids.split(",")] if visible_node_ids else None
+    return _graph_response(
+        dialect=dialect, discovery=discovery, expanded_nodes=expanded_nodes,
+        discovery_filter=discovery_filter, visible_node_ids=visible_list,
+    )
 
 @app.post("/config/path")
 def set_path(path_data: dict = Body(...)):
@@ -127,28 +160,15 @@ def scan_folders(path_data: dict = Body(...)):
 
 @app.post("/graph/filtered")
 def get_filtered_graph(data: dict = Body(...)):
-    """Parses SQL files with subfolder filtering."""
-    if not os.path.exists(CURRENT_DIRECTORY):
-        return {"nodes": [], "edges": [], "error": "Directory not found"}
-    
-    subfolders = data.get("subfolders")
-    dialect = data.get("dialect", "bigquery")
-    discovery = data.get("discovery", False)
-    expanded_nodes = data.get("expanded_nodes", {})
-    # Normalize: if frontend sends a list (legacy), convert to dict
-    if isinstance(expanded_nodes, list):
-        expanded_nodes = {n: 'all' for n in expanded_nodes}
-    visible_node_ids = data.get("visible_node_ids", None)
-    discovery_filter = data.get("discovery_filter", "all")
-    
-    try:
-        tables = _cached_parse(CURRENT_DIRECTORY, tuple(subfolders) if subfolders else None, dialect, visible_node_ids=visible_node_ids)
-        nodes, edges, cycles = build_graph(tables, discovery_mode=discovery, expanded_nodes=expanded_nodes, discovery_filter=discovery_filter)
-        return {"nodes": nodes, "edges": edges, "cycles": cycles}
-    except Exception as e:
-        import traceback
-        traceback.print_exc()
-        return {"nodes": [], "edges": [], "error": f"Backend Error: {str(e)}"}
+    """Graph restricted to a set of subfolders."""
+    return _graph_response(
+        dialect=data.get("dialect", "bigquery"),
+        discovery=data.get("discovery", False),
+        expanded_nodes=data.get("expanded_nodes", {}),
+        discovery_filter=data.get("discovery_filter", "all"),
+        subfolders=data.get("subfolders"),
+        visible_node_ids=data.get("visible_node_ids"),
+    )
 
 @app.post("/graph/scoped")
 def get_scoped_graph(data: dict = Body(...)):
@@ -165,29 +185,16 @@ def get_scoped_graph(data: dict = Body(...)):
 
     node_ids = data.get("node_ids") or []
     if not node_ids:
-        return {"nodes": [], "edges": [], "cycles": []}
-
-    dialect = data.get("dialect", "bigquery")
-    discovery = data.get("discovery", False)
-    expanded_nodes = data.get("expanded_nodes", {})
-    if isinstance(expanded_nodes, list):
-        expanded_nodes = {n: 'all' for n in expanded_nodes}
-    discovery_filter = data.get("discovery_filter", "all")
-
-    try:
-        tables = _cached_parse(
-            CURRENT_DIRECTORY, None, dialect,
-            visible_node_ids=node_ids, target_ids=node_ids,
-        )
-        nodes, edges, cycles = build_graph(
-            tables, discovery_mode=discovery,
-            expanded_nodes=expanded_nodes, discovery_filter=discovery_filter,
-        )
-        return {"nodes": nodes, "edges": edges, "cycles": cycles}
-    except Exception as e:
-        import traceback
-        traceback.print_exc()
-        return {"nodes": [], "edges": [], "error": f"Backend Error: {str(e)}"}
+        return {"nodes": [], "edges": [], "cycles": [], "warnings": []}
+
+    return _graph_response(
+        dialect=data.get("dialect", "bigquery"),
+        discovery=data.get("discovery", False),
+        expanded_nodes=data.get("expanded_nodes", {}),
+        discovery_filter=data.get("discovery_filter", "all"),
+        visible_node_ids=node_ids,
+        target_ids=node_ids,
+    )
 
 
 @app.post("/scan/new")
@@ -309,7 +316,7 @@ def export_data_dictionary(data: dict = Body(...)):
         visible_node_ids=visible_node_ids,
         target_ids=visible_node_ids,
     )
-    nodes, edges, cycles = build_graph(tables, discovery_mode=False)
+    nodes, edges, cycles, _warnings = build_graph(tables, discovery_mode=False)
     
     # Filter to only visible nodes if list provided
     if visible_node_ids is not None: