sql-glider 0.1.11__tar.gz → 0.1.13__tar.gz

{sql_glider-0.1.11 → sql_glider-0.1.13}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sql-glider
-Version: 0.1.11
+Version: 0.1.13
 Summary: SQL Utility Toolkit for better understanding, use, and governance of your queries in a native environment.
 Project-URL: Homepage, https://github.com/rycowhi/sql-glider/
 Project-URL: Repository, https://github.com/rycowhi/sql-glider/

sql_glider-0.1.13/plans/2026-01-29-no-star-flag.md

@@ -0,0 +1,47 @@
+# Add `--no-star` Flag
+
+**Status:** Completed
+
+## Overview
+Add `--no-star` flag to `lineage` and `graph build` commands. When set, analysis fails if `SELECT *` or `t.*` cannot be resolved to actual columns.
+
+## Changes
+
+### 1. `src/sqlglider/utils/config.py` — Add to ConfigSettings
+- [x] Add `no_star: Optional[bool] = None`
+
+### 2. `src/sqlglider/lineage/analyzer.py` — Add parameter + enforce
+- [x] Add `no_star: bool = False` to `__init__`, store as `self._no_star`
+- [x] Add `StarResolutionError` exception class (distinct from `ValueError` to avoid being swallowed by skipped-query handler)
+- [x] DML/DDL path: raise `StarResolutionError` before fallback for bare `*` and `t.*`
+- [x] DQL path: add star handling for both bare `*` and `t.*` with same error behavior
+- [x] Re-raise `StarResolutionError` in `analyze_queries` instead of treating as skipped query
+
+### 3. `src/sqlglider/graph/builder.py` — Pass through
+- [x] Add `no_star: bool = False` to `__init__`, store as `self.no_star`
+- [x] Pass to `LineageAnalyzer(sql_content, dialect=file_dialect, no_star=self.no_star)`
+
+### 4. `src/sqlglider/cli.py` — Add CLI options
+- [x] `lineage` command: Add `no_star: bool = typer.Option(False, "--no-star", ...)`
+- [x] Resolve: `no_star = no_star or config.no_star or False`
+- [x] Pass to `LineageAnalyzer(sql, dialect=dialect, no_star=no_star)`
+- [x] `graph_build` command: same option, passed to `GraphBuilder(..., no_star=no_star)`
+
+### 5. `tests/sqlglider/lineage/test_analyzer.py` — Tests
+- [x] Test bare `SELECT *` with `no_star=True` raises `StarResolutionError`
+- [x] Test `SELECT t.*` with `no_star=True` raises `StarResolutionError`
+- [x] Test resolvable star (via CTE) still works with `no_star=True`
+- [x] Test resolvable qualified star (via CTE) still works with `no_star=True`
+- [x] Test default (`no_star=False`) still falls back to `table.*`
+
+## Implementation Notes
+
+### Deviations from original plan
+- Used `StarResolutionError` instead of `ValueError` because `analyze_queries` catches `ValueError` to handle unsupported statement types (skipped queries). A plain `ValueError` would be silently swallowed.
+- Added star handling in the DQL (plain SELECT) code path in addition to the DML/DDL path. The original plan only addressed the DML/DDL path, but plain `SELECT *` queries go through a different branch in `get_output_columns`.
+- Resolvable star tests use CTEs instead of `CREATE TABLE` with explicit columns, since `_extract_schema_from_statement` only handles `CREATE ... AS SELECT`, not DDL with column definitions.
+
+## Verification
+- `uv run pytest` — 597 passed, 1 skipped, coverage 80.48%
+- `uv run basedpyright src/` — 0 errors
+- `uv run ruff check` — all checks passed

sql_glider-0.1.13/plans/2026-01-29-resolve-schema.md

@@ -0,0 +1,49 @@
+# `--resolve-schema` Flag and Catalog Integration
+
+**Status:** Completed
+
+## Overview
+
+Add `--resolve-schema` flag to `graph build` that runs a two-pass process: first extracting schema from all files, then running lineage analysis with the full schema available. Optionally, `--catalog-type` fills schema gaps by pulling DDL from a remote catalog.
+
+## Design Decisions
+
+- **Types are not required** — SQLGlot only needs column names for star expansion; types are stored as `"UNKNOWN"`
+- **Two-pass approach** — Pass 1 extracts schema from all files (order-independent), Pass 2 runs lineage with full schema
+- **Opt-in via `--resolve-schema`** — default behavior unchanged
+- **Catalog fills gaps only** — file-derived schema always wins over catalog-sourced schema
+- **`--catalog-type` requires `--resolve-schema`** — validated at CLI level
+
+## Implementation
+
+- [x] Add `schema` param to `LineageAnalyzer.__init__()` — pre-populates `_file_schema`
+- [x] Add `extract_schema_only()` and `get_extracted_schema()` methods to `LineageAnalyzer`
+- [x] Create `src/sqlglider/utils/schema.py` with `parse_ddl_to_schema()` for DDL column extraction
+- [x] Add `resolve_schema`, `catalog_type`, `catalog_config` to `GraphBuilder`
+- [x] Implement `_extract_schemas()` for pass 1 and `_fill_schema_from_catalog()` for catalog gap-filling
+- [x] Two-pass flow in `add_files()` and `add_manifest()`
+- [x] Add `--resolve-schema` and `--catalog-type` CLI flags to `graph build`
+- [x] Add `resolve_schema` to `ConfigSettings`
+- [x] Tests: 25 new tests (schema parsing, analyzer schema param, cross-file resolution, catalog mocking)
+
+## Files Modified
+
+- `src/sqlglider/lineage/analyzer.py` — schema param, extraction methods
+- `src/sqlglider/graph/builder.py` — two-pass processing, catalog integration
+- `src/sqlglider/cli.py` — CLI flags
+- `src/sqlglider/utils/config.py` — config setting
+- `src/sqlglider/utils/schema.py` — **new** DDL parsing utility
+- `tests/sqlglider/utils/test_schema.py` — **new**
+- `tests/sqlglider/graph/test_builder.py` — resolve schema + catalog tests
+- `tests/sqlglider/lineage/test_analyzer.py` — schema param tests
+
+## Verification
+
+- 617 passed, 1 skipped
+- Coverage: 80.43%
+- basedpyright: 0 errors
+- ruff: all checks passed
+
+## Known Limitations
+
+- Cross-file CTAS chains with `SELECT *` (view B depends on view A via star) may not resolve if both are in separate files and the schema extraction pass processes B before A. This is rare in practice.

{sql_glider-0.1.11 → sql_glider-0.1.13}/src/sqlglider/_version.py

@@ -28,7 +28,7 @@ version_tuple: VERSION_TUPLE
 commit_id: COMMIT_ID
 __commit_id__: COMMIT_ID
 
-__version__ = version = '0.1.11'
-__version_tuple__ = version_tuple = (0, 1, 11)
+__version__ = version = '0.1.13'
+__version_tuple__ = version_tuple = (0, 1, 13)
 
 __commit_id__ = commit_id = None

{sql_glider-0.1.11 → sql_glider-0.1.13}/src/sqlglider/cli.py

@@ -12,7 +12,7 @@ from sqlglot.errors import ParseError
 from typing_extensions import Annotated
 
 from sqlglider.global_models import AnalysisLevel, NodeFormat
-from sqlglider.lineage.analyzer import LineageAnalyzer
+from sqlglider.lineage.analyzer import LineageAnalyzer, SchemaResolutionError
 from sqlglider.lineage.formatters import (
     CsvFormatter,
     JsonFormatter,
@@ -1001,6 +1001,35 @@ def graph_build(
         "--no-star",
         help="Fail if SELECT * cannot be resolved to actual columns",
     ),
+    resolve_schema: bool = typer.Option(
+        False,
+        "--resolve-schema",
+        help="Extract schema from all files before lineage analysis, "
+        "enabling cross-file star resolution",
+    ),
+    catalog_type: Optional[str] = typer.Option(
+        None,
+        "--catalog-type",
+        "-c",
+        help="Catalog provider for pulling DDL of tables not found in files "
+        "(requires --resolve-schema). E.g. 'databricks'",
+    ),
+    dump_schema: Optional[Path] = typer.Option(
+        None,
+        "--dump-schema",
+        help="Dump resolved schema to file (requires --resolve-schema)",
+    ),
+    dump_schema_format: Optional[str] = typer.Option(
+        None,
+        "--dump-schema-format",
+        help="Format for dumped schema: 'text' (default), 'json', or 'csv'",
+    ),
+    strict_schema: bool = typer.Option(
+        False,
+        "--strict-schema",
+        help="Fail if any column's table cannot be identified during schema extraction "
+        "(requires --resolve-schema)",
+    ),
 ) -> None:
     """
     Build a lineage graph from SQL files.
@@ -1036,6 +1065,37 @@ def graph_build(
     dialect = dialect or config.dialect or "spark"
     templater = templater or config.templater  # None means no templating
     no_star = no_star or config.no_star or False
+    resolve_schema = resolve_schema or config.resolve_schema or False
+    strict_schema = strict_schema or config.strict_schema or False
+
+    if strict_schema and not resolve_schema:
+        err_console.print("[red]Error:[/red] --strict-schema requires --resolve-schema")
+        raise typer.Exit(1)
+
+    if catalog_type and not resolve_schema:
+        err_console.print("[red]Error:[/red] --catalog-type requires --resolve-schema")
+        raise typer.Exit(1)
+
+    # Resolve dump_schema options from config
+    dump_schema = dump_schema or (
+        Path(config.dump_schema) if config.dump_schema else None
+    )
+    dump_schema_format = dump_schema_format or config.dump_schema_format or "text"
+
+    if dump_schema and not resolve_schema:
+        err_console.print("[red]Error:[/red] --dump-schema requires --resolve-schema")
+        raise typer.Exit(1)
+
+    if dump_schema_format not in ("text", "json", "csv"):
+        err_console.print(
+            f"[red]Error:[/red] Invalid --dump-schema-format '{dump_schema_format}'. "
+            "Use 'text', 'json', or 'csv'."
+        )
+        raise typer.Exit(1)
+
+    # Only inherit catalog_type from config when resolve_schema is active
+    if resolve_schema and not catalog_type:
+        catalog_type = config.catalog_type
 
     # Validate and convert node format to enum
     try:
@@ -1088,32 +1148,75 @@ def graph_build(
         sql_preprocessor = _preprocess
 
     try:
+        # Build catalog config from config file if available
+        catalog_config_dict = None
+        if catalog_type and config.catalog:
+            provider_config = getattr(config.catalog, catalog_type, None)
+            if provider_config:
+                catalog_config_dict = provider_config.model_dump(exclude_none=True)
+
         builder = GraphBuilder(
             node_format=node_format_enum,
             dialect=dialect,
             sql_preprocessor=sql_preprocessor,
             no_star=no_star,
+            resolve_schema=resolve_schema,
+            catalog_type=catalog_type,
+            catalog_config=catalog_config_dict,
+            strict_schema=strict_schema,
         )
 
-        # Process manifest if provided
-        if manifest:
-            builder.add_manifest(manifest, dialect=dialect)
-
-        # Process paths - collect all files first for progress tracking
+        # Collect file paths for schema extraction
+        path_files: list[Path] = []
         if paths:
-            all_files: list[Path] = []
             for path in paths:
                 if path.is_dir():
                     pattern = f"**/{glob_pattern}" if recursive else glob_pattern
-                    all_files.extend(
+                    path_files.extend(
                         f for f in sorted(path.glob(pattern)) if f.is_file()
                     )
                 elif path.is_file():
-                    all_files.append(path)
+                    path_files.append(path)
                 else:
                     err_console.print(f"[red]Error:[/red] Path not found: {path}")
                     raise typer.Exit(1)
-            builder.add_files(all_files, dialect=dialect)
+
+        manifest_files: list[Path] = []
+        if manifest:
+            from sqlglider.graph.models import Manifest
+
+            manifest_data = Manifest.from_csv(manifest)
+            base_dir = manifest.parent
+            for entry in manifest_data.entries:
+                file_path = Path(entry.file_path)
+                if not file_path.is_absolute():
+                    file_path = (base_dir / entry.file_path).resolve()
+                manifest_files.append(file_path)
+
+        # Extract schema upfront if requested, then dump before graph building
+        all_files = manifest_files + path_files
+        if resolve_schema and all_files:
+            builder.extract_schemas(all_files, dialect=dialect)
+
+            if dump_schema:
+                from sqlglider.graph.formatters import format_schema
+
+                schema_content = format_schema(
+                    builder.resolved_schema, dump_schema_format
+                )
+                dump_schema.write_text(schema_content, encoding="utf-8")
+                console.print(
+                    f"[green]Schema dumped to {dump_schema} "
+                    f"({len(builder.resolved_schema)} table(s))[/green]"
+                )
+
+        # Process manifest if provided
+        if manifest:
+            builder.add_manifest(manifest, dialect=dialect)
+
+        # Process path-based files
+        if path_files:
+            builder.add_files(path_files, dialect=dialect)
 
         # Build and save graph
         graph = builder.build()
@@ -1124,6 +1227,10 @@ def graph_build(
             f"({graph.metadata.total_nodes} nodes, {graph.metadata.total_edges} edges)"
         )
 
+    except SchemaResolutionError as e:
+        err_console.print(f"[red]Error:[/red] {e}")
+        raise typer.Exit(1)
+
     except FileNotFoundError as e:
         err_console.print(f"[red]Error:[/red] {e}")
         raise typer.Exit(1)

{sql_glider-0.1.11 → sql_glider-0.1.13}/src/sqlglider/graph/builder.py

@@ -16,8 +16,9 @@ from sqlglider.graph.models import (
     LineageGraph,
     Manifest,
 )
-from sqlglider.lineage.analyzer import LineageAnalyzer
+from sqlglider.lineage.analyzer import LineageAnalyzer, SchemaResolutionError
 from sqlglider.utils.file_utils import read_sql_file
+from sqlglider.utils.schema import parse_ddl_to_schema
 
 console = Console(stderr=True)
 
@@ -34,6 +35,10 @@ class GraphBuilder:
         dialect: str = "spark",
         sql_preprocessor: Optional[SqlPreprocessor] = None,
         no_star: bool = False,
+        resolve_schema: bool = False,
+        catalog_type: Optional[str] = None,
+        catalog_config: Optional[Dict[str, object]] = None,
+        strict_schema: bool = False,
     ):
         """
         Initialize the graph builder.
@@ -45,16 +50,31 @@ class GraphBuilder:
                              Takes (sql: str, file_path: Path) and returns processed SQL.
                              Useful for templating (e.g., Jinja2 rendering).
             no_star: If True, fail when SELECT * cannot be resolved to columns
+            resolve_schema: If True, run a schema extraction pass across all
+                files before lineage analysis so that schema from any file is
+                available when analyzing every other file.
+            catalog_type: Optional catalog provider name (e.g. "databricks").
+                When set together with resolve_schema, DDL is pulled from the
+                catalog for tables whose schema could not be inferred from files.
+            catalog_config: Optional provider-specific configuration dict
+                passed to the catalog's configure() method.
+            strict_schema: If True, fail during schema extraction when an
+                unqualified column cannot be attributed to a table.
         """
         self.node_format = node_format
         self.dialect = dialect
         self.sql_preprocessor = sql_preprocessor
         self.no_star = no_star
+        self.resolve_schema = resolve_schema
+        self.catalog_type = catalog_type
+        self.catalog_config = catalog_config
+        self.strict_schema = strict_schema
         self.graph: rx.PyDiGraph = rx.PyDiGraph()
         self._node_index_map: Dict[str, int] = {}  # identifier -> rustworkx node index
         self._source_files: Set[str] = set()
         self._edge_set: Set[tuple] = set()  # (source, target) for dedup
         self._skipped_files: List[tuple[str, str]] = []  # (file_path, reason)
+        self._resolved_schema: Dict[str, Dict[str, str]] = {}  # accumulated schema
 
     def add_file(
         self,
@@ -86,7 +106,10 @@ class GraphBuilder:
                 sql_content = self.sql_preprocessor(sql_content, file_path)
 
             analyzer = LineageAnalyzer(
-                sql_content, dialect=file_dialect, no_star=self.no_star
+                sql_content,
+                dialect=file_dialect,
+                no_star=self.no_star,
+                schema=self._resolved_schema if self._resolved_schema else None,
             )
             results = analyzer.analyze_queries(level=AnalysisLevel.COLUMN)
 
@@ -209,23 +232,28 @@ class GraphBuilder:
             entry_dialect = entry.dialect or dialect or self.dialect
             files_with_dialects.append((file_path, entry_dialect))
 
-        # Process with progress
-        if files_with_dialects:
-            total = len(files_with_dialects)
-            with Progress(
-                TextColumn("[progress.description]{task.description}"),
-                BarColumn(),
-                TaskProgressColumn(),
-                console=console,
-                transient=False,
-            ) as progress:
-                task = progress.add_task("Parsing", total=total)
-                for i, (file_path, file_dialect) in enumerate(
-                    files_with_dialects, start=1
-                ):
-                    console.print(f"Parsing file {i}/{total}: {file_path.name}")
-                    self.add_file(file_path, file_dialect)
-                    progress.advance(task)
+        if not files_with_dialects:
+            return self
+
+        # Two-pass schema resolution (skip if already resolved)
+        if self.resolve_schema and not self._resolved_schema:
+            file_paths_only = [fp for fp, _ in files_with_dialects]
+            self.extract_schemas(file_paths_only, dialect)
+
+        total = len(files_with_dialects)
+        description = "Pass 2: Analyzing lineage" if self.resolve_schema else "Parsing"
+        with Progress(
+            TextColumn("[progress.description]{task.description}"),
+            BarColumn(),
+            TaskProgressColumn(),
+            console=console,
+            transient=False,
+        ) as progress:
+            task = progress.add_task(description, total=total)
+            for i, (file_path, file_dialect) in enumerate(files_with_dialects, start=1):
+                console.print(f"Parsing file {i}/{total}: {file_path.name}")
+                self.add_file(file_path, file_dialect)
+                progress.advance(task)
 
         return self
 
@@ -249,8 +277,15 @@ class GraphBuilder:
         if not file_paths:
             return self
 
+        # Two-pass schema resolution (skip if already resolved)
+        if self.resolve_schema and not self._resolved_schema:
+            self.extract_schemas(file_paths, dialect)
+
         if show_progress:
             total = len(file_paths)
+            description = (
+                "Pass 2: Analyzing lineage" if self.resolve_schema else "Parsing"
+            )
             with Progress(
                 TextColumn("[progress.description]{task.description}"),
                 BarColumn(),
@@ -258,7 +293,7 @@ class GraphBuilder:
                 console=console,
                 transient=False,
             ) as progress:
-                task = progress.add_task("Parsing", total=total)
+                task = progress.add_task(description, total=total)
                 for i, file_path in enumerate(file_paths, start=1):
                     console.print(f"Parsing file {i}/{total}: {file_path.name}")
                     self.add_file(file_path, dialect)
@@ -268,6 +303,157 @@ class GraphBuilder:
                 self.add_file(file_path, dialect)
         return self
 
+    def extract_schemas(
+        self,
+        file_paths: List[Path],
+        dialect: Optional[str] = None,
+    ) -> Dict[str, Dict[str, str]]:
+        """Run schema extraction pass and optionally fill from catalog.
+
+        Call this before add_files/add_manifest to resolve schema upfront.
+        The resolved schema is stored internally and also returned.
+
+        Args:
+            file_paths: SQL files to extract schema from
+            dialect: SQL dialect override
+
+        Returns:
+            Resolved schema dict
+        """
+        console.print("[blue]Pass 1: Extracting schema from files[/blue]")
+        self._resolved_schema = self._extract_schemas(file_paths, dialect)
+        if self.catalog_type:
+            self._resolved_schema = self._fill_schema_from_catalog(
+                self._resolved_schema, file_paths, dialect
+            )
+        console.print(
+            f"[blue]Schema resolved for {len(self._resolved_schema)} table(s)[/blue]"
+        )
+        return self._resolved_schema.copy()
+
+    def _extract_schemas(
+        self,
+        file_paths: List[Path],
+        dialect: Optional[str] = None,
+    ) -> Dict[str, Dict[str, str]]:
+        """Run schema extraction pass across all files.
+
+        Parses each file and extracts schema from CREATE TABLE/VIEW
+        statements without performing lineage analysis.
+
+        Args:
+            file_paths: SQL files to extract schema from
+            dialect: SQL dialect override
+
+        Returns:
+            Accumulated schema dict from all files
+        """
+        schema: Dict[str, Dict[str, str]] = {}
+        total = len(file_paths)
+        with Progress(
+            TextColumn("[progress.description]{task.description}"),
+            BarColumn(),
+            TaskProgressColumn(),
+            console=console,
+            transient=False,
+        ) as progress:
+            task = progress.add_task("Pass 1: Extracting schema", total=total)
+            for i, file_path in enumerate(file_paths, start=1):
+                console.print(f"Extracting schema {i}/{total}: {file_path.name}")
+                file_dialect = dialect or self.dialect
+                try:
+                    sql_content = read_sql_file(file_path)
+                    if self.sql_preprocessor:
+                        sql_content = self.sql_preprocessor(sql_content, file_path)
+                    analyzer = LineageAnalyzer(
+                        sql_content,
+                        dialect=file_dialect,
+                        schema=schema,
+                        strict_schema=self.strict_schema,
+                    )
+                    file_schema = analyzer.extract_schema_only()
+                    schema.update(file_schema)
+                except SchemaResolutionError:
+                    raise
+                except Exception:
+                    # Schema extraction failures are non-fatal; the file
+                    # will be reported during the lineage pass if it also fails.
+                    pass
+                progress.advance(task)
+        return schema
+
+    def _fill_schema_from_catalog(
+        self,
+        schema: Dict[str, Dict[str, str]],
+        file_paths: List[Path],
+        dialect: Optional[str] = None,
+    ) -> Dict[str, Dict[str, str]]:
+        """Pull DDL from catalog for tables not yet in schema.
+
+        Extracts all table names referenced across the files, identifies
+        those missing from the schema, and fetches their DDL from the
+        configured catalog provider.
+
+        Args:
+            schema: Schema dict already populated from file extraction
+            file_paths: SQL files to scan for table references
+            dialect: SQL dialect override
+
+        Returns:
+            Updated schema dict with catalog-sourced entries added
+        """
+        from sqlglider.catalog import get_catalog
+
+        catalog = get_catalog(self.catalog_type)  # type: ignore[arg-type]
+        if self.catalog_config:
+            catalog.configure(self.catalog_config)
+
+        # Collect all referenced table names across files
+        all_tables: Set[str] = set()
+        for file_path in file_paths:
+            file_dialect = dialect or self.dialect
+            try:
+                sql_content = read_sql_file(file_path)
+                if self.sql_preprocessor:
+                    sql_content = self.sql_preprocessor(sql_content, file_path)
+                analyzer = LineageAnalyzer(sql_content, dialect=file_dialect)
+                tables_results = analyzer.analyze_tables()
+                for result in tables_results:
+                    for table_info in result.tables:
+                        # Skip CTEs — they don't exist in catalogs
+                        from sqlglider.lineage.analyzer import ObjectType
+
+                        if table_info.object_type != ObjectType.CTE:
+                            all_tables.add(table_info.name)
+            except Exception:
+                pass
+
+        # Find tables missing from schema
+        missing = [t for t in all_tables if t not in schema]
+        if not missing:
+            return schema
+
+        console.print(
+            f"[blue]Pulling DDL from {self.catalog_type} "
+            f"for {len(missing)} table(s)...[/blue]"
+        )
+
+        ddl_results = catalog.get_ddl_batch(missing)
+        file_dialect = dialect or self.dialect
+        for table_name, ddl in ddl_results.items():
+            if ddl.startswith("ERROR:"):
+                console.print(
+                    f"[yellow]Warning:[/yellow] Could not pull DDL "
+                    f"for {table_name}: {ddl}"
+                )
+                continue
+            parsed_schema = parse_ddl_to_schema(ddl, dialect=file_dialect)
+            for name, cols in parsed_schema.items():
+                if name not in schema:
+                    schema[name] = cols
+
+        return schema
+
     def _ensure_node(
         self,
         identifier: str,
@@ -348,6 +534,11 @@ class GraphBuilder:
         """Get mapping from node identifiers to rustworkx indices."""
         return self._node_index_map.copy()
 
+    @property
+    def resolved_schema(self) -> Dict[str, Dict[str, str]]:
+        """Get the resolved schema dictionary from schema extraction pass."""
+        return self._resolved_schema.copy()
+
     @property
     def skipped_files(self) -> List[tuple[str, str]]:
         """Get list of files that were skipped during graph building."""