sql-glider 0.1.12__tar.gz → 0.1.14__tar.gz

{sql_glider-0.1.12 → sql_glider-0.1.14}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sql-glider
-Version: 0.1.12
+Version: 0.1.14
 Summary: SQL Utility Toolkit for better understanding, use, and governance of your queries in a native environment.
 Project-URL: Homepage, https://github.com/rycowhi/sql-glider/
 Project-URL: Repository, https://github.com/rycowhi/sql-glider/

sql_glider-0.1.14/plans/2026-01-29-schema-pruning-optimization.md

@@ -0,0 +1,63 @@
+# Schema Pruning Optimization for `--resolve-schema`
+
+**Status:** Completed
+
+## Overview
+
+Optimize `--resolve-schema` graph build performance by pruning the schema dict to only tables referenced in each query before passing it to `sqlglot.lineage()`. Also moved schema dumping (`--dump-schema`) to occur before graph building (between Pass 1 and Pass 2).
+
+## Problem
+
+`sqlglot.lineage()` performance degrades dramatically with large schema dicts. Benchmarks showed:
+
+| Schema Size | Time (6 columns) |
+|---|---|
+| No schema | 2.6ms |
+| 4 tables | 8.3ms |
+| 204 tables | **1,041ms** |
+
+The full accumulated schema from all files was passed to every `lineage()` call, even though each query only references a handful of tables. For projects with hundreds of files/tables, this made `--resolve-schema` unusable.
+
+## Design Decisions
+
+- **Prune in the analyzer, not the builder** — keeps the optimization localized and benefits all callers of `LineageAnalyzer`, not just graph builds
+- **Prune once per query, not per column** — `_get_query_tables()` is called once before the column loop, and the pruned schema is reused for all columns in that query
+- **Case-insensitive matching** — table names are lowered for comparison to handle mixed-case schemas
+- **Moved schema dump before graph build** — `_resolved_schema` is fully populated after Pass 1, so dumping between passes is safe and gives users earlier feedback. Required exposing `extract_schemas()` as a public method on `GraphBuilder`
+
+## Implementation
+
+- [x] Add schema pruning in `_analyze_column_lineage_internal()` using existing `_get_query_tables()` method
+- [x] Expose `extract_schemas()` as public method on `GraphBuilder`
+- [x] Skip Pass 1 in `add_files()`/`add_manifest()` if `_resolved_schema` is already populated
+- [x] Restructure CLI `graph_build` to call `extract_schemas()` then dump schema before `add_files()`/`add_manifest()`
+- [x] Tests for `extract_schemas()` method and schema pruning correctness
+
+## Files Modified
+
+- `src/sqlglider/lineage/analyzer.py` — schema pruning before `lineage()` calls
+- `src/sqlglider/graph/builder.py` — public `extract_schemas()`, skip Pass 1 when already resolved
+- `src/sqlglider/cli.py` — restructured `graph_build` to dump schema before graph building
+- `tests/sqlglider/lineage/test_analyzer.py` — `TestSchemaPruning` (2 tests)
+- `tests/sqlglider/graph/test_builder.py` — `TestExtractSchemas` (3 tests)
+
+## Benchmark Results (After)
+
+| Schema Size | Time (6 columns) |
+|---|---|
+| No schema | 2.6ms |
+| 4 tables | 8.3ms |
+| 204 tables | **8.3ms** |
+
+Full `analyze_queries` benchmark on complex fixture (analytics_pipeline.sql):
+
+| Scenario | Before | After |
+|---|---|---|
+| No schema | ~392ms | ~392ms |
+| Small schema (4 tables) | ~373ms | ~373ms |
+| Big schema (204 tables) | ~1,400ms+ | **~387ms** |
+
+## Lessons Learned
+
+- The initial assumption was that double-parsing (Pass 1 + Pass 2 both calling `sqlglot.parse()`) was the bottleneck. Benchmarking showed `parse()` costs ~10ms, while `lineage()` with a 200-table schema costs ~1,000ms. Profiling before optimizing avoided wasted effort on AST caching.
+- `sqlglot.lineage()` appears to have O(n) or worse scaling with schema size, even for tables not referenced in the query. Pruning is essential for multi-file workloads.

sql_glider-0.1.14/plans/2026-01-29-tables-scrape-command.md

@@ -0,0 +1,118 @@
+**Status:** Completed
+
+# Plan: `tables scrape` Command
+
+## Overview
+
+Add a `tables scrape` subcommand that performs schema inference (the same logic `graph build --resolve-schema` uses) but outputs the inferred schema directly instead of building a lineage graph. This makes schema inference a standalone, reusable operation.
+
+## Key Changes
+
+### 1. Refactor schema extraction out of GraphBuilder
+
+**File:** `src/sqlglider/graph/builder.py`
+
+Extract `_extract_schemas()` and `_fill_schema_from_catalog()` into a standalone module so both `graph build` and `tables scrape` can use them without instantiating a full `GraphBuilder`.
+
+**New file:** `src/sqlglider/schema/extractor.py`
+- `extract_schemas_from_files(file_paths, dialect, sql_preprocessor, schema, strict_schema, console) -> SchemaDict` — core extraction loop with Rich progress bar
+- `fill_schema_from_catalog(schema, file_paths, dialect, sql_preprocessor, catalog_type, catalog_config, console) -> SchemaDict` — catalog fill logic
+- `extract_and_resolve_schema(file_paths, dialect, sql_preprocessor, strict_schema, catalog_type, catalog_config, console) -> SchemaDict` — high-level orchestrator (extract + optional catalog fill)
+
+**Update GraphBuilder** to delegate to these new functions instead of implementing them inline. `GraphBuilder._extract_schemas` and `_fill_schema_from_catalog` become thin wrappers or are removed, with `extract_schemas()` calling the shared code.
+
+### 2. Add `tables scrape` CLI command
+
+**File:** `src/sqlglider/cli.py`
+
+Add `@tables_app.command("scrape")` with these parameters (mirroring `graph build`):
+
+| Parameter | Source |
+|-----------|--------|
+| `paths` | Same as `graph build` — file(s) or directory(ies) |
+| `--recursive / -r` | Same recursive directory traversal |
+| `--glob / -g` | Same glob pattern (default `*.sql`) |
+| `--manifest / -m` | Same manifest CSV support |
+| `--dialect / -d` | SQL dialect |
+| `--templater / -t` | Templater name |
+| `--var / -v` | Template variables |
+| `--vars-file` | Variables file |
+| `--strict-schema` | Strict schema mode |
+| `--catalog-type / -c` | Catalog provider for remote DDL |
+| `--output-format / -f` | `text` (default), `json`, or `csv` |
+| `--output-file / -o` | Output file path (stdout if omitted) |
+
+**Flow:**
+1. Resolve config defaults (same pattern as `graph build`)
+2. Set up templating preprocessor (same shared code)
+3. Collect files from paths/manifest (same logic as `graph build`)
+4. Call `extract_and_resolve_schema(...)` from the new shared module
+5. Format output using existing `format_schema()` from `src/sqlglider/graph/formatters.py`
+6. Write to file or stdout using `OutputWriter`
+
+### 3. Move schema formatters
+
+The existing formatters in `src/sqlglider/graph/formatters.py` are already generic (they format `SchemaDict`). **Decision: leave them in place** to minimize churn — they work for both `graph build --dump-schema` and `tables scrape`.
+
+### 4. Refactor shared file-collection logic
+
+The file collection code (paths + recursive glob + manifest) is duplicated between `graph build` and the new `tables scrape`. Extract a helper function:
+
+```python
+def _collect_sql_files(
+    paths: Optional[List[Path]],
+    manifest: Optional[Path],
+    recursive: bool,
+    glob_pattern: str,
+) -> tuple[list[Path], list[Path]]:
+    """Returns (manifest_files, path_files)."""
+```
+
+Place this in `cli.py` as a private helper used by both commands.
+
+## Implementation Steps
+
+- [x] Create `src/sqlglider/schema/__init__.py`
+- [x] Create `src/sqlglider/schema/extractor.py` with shared schema extraction logic
+- [x] Update `src/sqlglider/graph/builder.py` to delegate to shared extractor
+- [x] Add `_collect_sql_files` helper to `src/sqlglider/cli.py`
+- [x] Refactor `graph build` to use `_collect_sql_files`
+- [x] Add `tables scrape` command to `src/sqlglider/cli.py`
+- [x] Create `tests/sqlglider/schema/__init__.py`
+- [x] Create `tests/sqlglider/schema/test_extractor.py` (10 tests)
+- [x] Add `TestTablesScrapeCommand` to `tests/sqlglider/test_cli.py` (11 tests)
+- [x] All 672 tests pass, 81.5% coverage, ruff clean
+
+## Files Created/Modified
+
+| File | Action |
+|------|--------|
+| `src/sqlglider/schema/__init__.py` | Created — empty |
+| `src/sqlglider/schema/extractor.py` | Created — shared schema extraction logic |
+| `src/sqlglider/graph/builder.py` | Modified — delegate to shared extractor |
+| `src/sqlglider/cli.py` | Modified — add `tables scrape` command + `_collect_sql_files` helper |
+| `tests/sqlglider/schema/__init__.py` | Created — empty |
+| `tests/sqlglider/schema/test_extractor.py` | Created — tests for shared extractor |
+| `tests/sqlglider/test_cli.py` | Modified — add tests for `tables scrape` command |
+
+## Testing Strategy
+
+1. **Unit tests for `schema/extractor.py`**: Test `extract_schemas_from_files` with CREATE VIEW/TABLE AS SELECT and DQL qualified refs
+2. **CLI tests for `tables scrape`**: Use `CliRunner` to test text/json/csv output, recursive glob, templating, error cases
+3. **Regression**: Full test suite passes (672 tests), coverage at 81.5%
+
+## Verification
+
+```bash
+# Basic usage
+uv run sqlglider tables scrape ./queries/ -r
+
+# With output format
+uv run sqlglider tables scrape ./queries/ -r -f json -o schema.json
+
+# With catalog
+uv run sqlglider tables scrape ./queries/ -r -c databricks -f csv
+
+# Ensure graph build still works
+uv run sqlglider graph build ./queries/ -r --resolve-schema --dump-schema schema.txt -o graph.json
+```

{sql_glider-0.1.12 → sql_glider-0.1.14}/src/sqlglider/_version.py

@@ -28,7 +28,7 @@ version_tuple: VERSION_TUPLE
 commit_id: COMMIT_ID
 __commit_id__: COMMIT_ID
 
-__version__ = version = '0.1.12'
-__version_tuple__ = version_tuple = (0, 1, 12)
+__version__ = version = '0.1.14'
+__version_tuple__ = version_tuple = (0, 1, 14)
 
 __commit_id__ = commit_id = None

{sql_glider-0.1.12 → sql_glider-0.1.14}/src/sqlglider/cli.py

@@ -788,6 +788,274 @@ def tables_pull(
         raise typer.Exit(1)
 
 
+def _collect_sql_files(
+    paths: Optional[List[Path]],
+    manifest: Optional[Path],
+    recursive: bool,
+    glob_pattern: str,
+) -> tuple[list[Path], list[Path]]:
+    """Collect SQL files from paths and/or manifest.
+
+    Args:
+        paths: File or directory paths to scan.
+        manifest: Optional manifest CSV path.
+        recursive: Whether to recurse into directories.
+        glob_pattern: Glob pattern for directory scanning.
+
+    Returns:
+        Tuple of (manifest_files, path_files).
+    """
+    path_files: list[Path] = []
+    if paths:
+        for path in paths:
+            if path.is_dir():
+                pattern = f"**/{glob_pattern}" if recursive else glob_pattern
+                path_files.extend(f for f in sorted(path.glob(pattern)) if f.is_file())
+            elif path.is_file():
+                path_files.append(path)
+            else:
+                err_console.print(f"[red]Error:[/red] Path not found: {path}")
+                raise typer.Exit(1)
+
+    manifest_files: list[Path] = []
+    if manifest:
+        from sqlglider.graph.models import Manifest
+
+        manifest_data = Manifest.from_csv(manifest)
+        base_dir = manifest.parent
+        for entry in manifest_data.entries:
+            file_path = Path(entry.file_path)
+            if not file_path.is_absolute():
+                file_path = (base_dir / entry.file_path).resolve()
+            manifest_files.append(file_path)
+
+    return manifest_files, path_files
+
+
+@tables_app.command("scrape")
+def tables_scrape(
+    paths: List[Path] = typer.Argument(
+        None,
+        help="SQL file(s) or directory path to process",
+    ),
+    recursive: bool = typer.Option(
+        False,
+        "--recursive",
+        "-r",
+        help="Recursively search directories for SQL files",
+    ),
+    glob_pattern: str = typer.Option(
+        "*.sql",
+        "--glob",
+        "-g",
+        help="Glob pattern for matching SQL files in directories",
+    ),
+    manifest: Optional[Path] = typer.Option(
+        None,
+        "--manifest",
+        "-m",
+        exists=True,
+        help="Path to manifest CSV file with file_path and optional dialect columns",
+    ),
+    dialect: Optional[str] = typer.Option(
+        None,
+        "--dialect",
+        "-d",
+        help="SQL dialect (default: spark)",
+    ),
+    templater: Optional[str] = typer.Option(
+        None,
+        "--templater",
+        "-t",
+        help="Templater for SQL preprocessing (e.g., 'jinja', 'none')",
+    ),
+    var: Optional[List[str]] = typer.Option(
+        None,
+        "--var",
+        "-v",
+        help="Template variable in key=value format (repeatable)",
+    ),
+    vars_file: Optional[Path] = typer.Option(
+        None,
+        "--vars-file",
+        exists=True,
+        help="Path to variables file (JSON or YAML)",
+    ),
+    strict_schema: bool = typer.Option(
+        False,
+        "--strict-schema",
+        help="Fail if any column's table cannot be identified during schema extraction",
+    ),
+    catalog_type: Optional[str] = typer.Option(
+        None,
+        "--catalog-type",
+        "-c",
+        help="Catalog provider for pulling DDL of tables not found in files "
+        "(e.g. 'databricks')",
+    ),
+    output_format: Optional[str] = typer.Option(
+        None,
+        "--output-format",
+        "-f",
+        help="Output format: 'text' (default), 'json', or 'csv'",
+    ),
+    output_file: Optional[Path] = typer.Option(
+        None,
+        "--output-file",
+        "-o",
+        help="Output file path (prints to stdout if not provided)",
+    ),
+) -> None:
+    """
+    Scrape schema information from SQL files.
+
+    Infers table and column schemas from DDL statements and DQL column
+    references across one or more SQL files. Supports the same file input
+    modes as `graph build` (paths, directories, manifests).
+
+    Examples:
+
+        # Scrape schema from a directory
+        sqlglider tables scrape ./queries/ -r
+
+        # Output as JSON
+        sqlglider tables scrape ./queries/ -r -f json
+
+        # Save to file
+        sqlglider tables scrape ./queries/ -r -f csv -o schema.csv
+
+        # With Jinja2 templating
+        sqlglider tables scrape ./queries/ -r --templater jinja --var schema=prod
+
+        # With catalog fallback
+        sqlglider tables scrape ./queries/ -r -c databricks
+    """
+    from sqlglider.graph.formatters import format_schema
+    from sqlglider.lineage.analyzer import SchemaResolutionError
+    from sqlglider.schema.extractor import extract_and_resolve_schema
+
+    # Load config for defaults
+    config = load_config()
+    dialect = dialect or config.dialect or "spark"
+    templater = templater or config.templater
+    strict_schema = strict_schema or config.strict_schema or False
+    output_format = output_format or config.output_format or "text"
+
+    if output_format not in ("text", "json", "csv"):
+        err_console.print(
+            f"[red]Error:[/red] Invalid --output-format '{output_format}'. "
+            "Use 'text', 'json', or 'csv'."
+        )
+        raise typer.Exit(1)
+
+    # Only inherit catalog_type from config when not provided via CLI
+    if not catalog_type:
+        catalog_type = config.catalog_type
+
+    # Validate inputs
+    if not paths and not manifest:
+        err_console.print(
+            "[red]Error:[/red] Must provide either file/directory paths or --manifest option."
+        )
+        raise typer.Exit(1)
+
+    # Create SQL preprocessor if templating is enabled
+    sql_preprocessor: Optional[Callable[[str, Path], str]] = None
+    if templater:
+        config_vars_file = None
+        config_vars = None
+        if config.templating:
+            if config.templating.variables_file and not vars_file:
+                config_vars_file = Path(config.templating.variables_file)
+                if not config_vars_file.exists():
+                    err_console.print(
+                        f"[yellow]Warning:[/yellow] Variables file from config "
+                        f"not found: {config_vars_file}"
+                    )
+                    config_vars_file = None
+            config_vars = config.templating.variables
+
+        variables = load_all_variables(
+            cli_vars=var,
+            vars_file=vars_file or config_vars_file,
+            config_vars=config_vars,
+            use_env=True,
+        )
+
+        templater_instance = get_templater(templater)
+
+        def _preprocess(sql: str, file_path: Path) -> str:
+            return templater_instance.render(
+                sql, variables=variables, source_path=file_path
+            )
+
+        sql_preprocessor = _preprocess
+
+    try:
+        # Build catalog config from config file if available
+        catalog_config_dict = None
+        if catalog_type and config.catalog:
+            provider_config = getattr(config.catalog, catalog_type, None)
+            if provider_config:
+                catalog_config_dict = provider_config.model_dump(exclude_none=True)
+
+        # Collect files
+        manifest_files, path_files = _collect_sql_files(
+            paths, manifest, recursive, glob_pattern
+        )
+        all_files = manifest_files + path_files
+
+        if not all_files:
+            err_console.print("[yellow]Warning:[/yellow] No SQL files found.")
+            raise typer.Exit(0)
+
+        # Extract schema
+        schema = extract_and_resolve_schema(
+            all_files,
+            dialect=dialect,
+            sql_preprocessor=sql_preprocessor,
+            strict_schema=strict_schema,
+            catalog_type=catalog_type,
+            catalog_config=catalog_config_dict,
+            console=err_console,
+        )
+
+        if not schema:
+            err_console.print("[yellow]No schema information found.[/yellow]")
+            raise typer.Exit(0)
+
+        # Format and output
+        formatted = format_schema(schema, output_format)
+        if output_file:
+            OutputWriter.write(formatted, output_file)
+            err_console.print(
+                f"[green]Schema written to {output_file} "
+                f"({len(schema)} table(s))[/green]"
+            )
+        else:
+            console.print(formatted, end="")
+
+    except SchemaResolutionError as e:
+        err_console.print(f"[red]Error:[/red] {e}")
+        raise typer.Exit(1)
+
+    except FileNotFoundError as e:
+        err_console.print(f"[red]Error:[/red] {e}")
+        raise typer.Exit(1)
+
+    except TemplaterError as e:
+        err_console.print(f"[red]Error:[/red] {e}")
+        raise typer.Exit(1)
+
+    except ValueError as e:
+        err_console.print(f"[red]Error:[/red] {e}")
+        raise typer.Exit(1)
+
+    except Exception as e:
+        err_console.print(f"[red]Error:[/red] Unexpected error: {e}")
+        raise typer.Exit(1)
+
+
 @app.command()
 def template(
     sql_file: Annotated[
@@ -1166,36 +1434,35 @@ def graph_build(
             strict_schema=strict_schema,
         )
 
+        # Collect file paths for schema extraction
+        manifest_files, path_files = _collect_sql_files(
+            paths, manifest, recursive, glob_pattern
+        )
+
+        # Extract schema upfront if requested, then dump before graph building
+        all_files = manifest_files + path_files
+        if resolve_schema and all_files:
+            builder.extract_schemas(all_files, dialect=dialect)
+
+            if dump_schema:
+                from sqlglider.graph.formatters import format_schema
+
+                schema_content = format_schema(
+                    builder.resolved_schema, dump_schema_format
+                )
+                dump_schema.write_text(schema_content, encoding="utf-8")
+                console.print(
+                    f"[green]Schema dumped to {dump_schema} "
+                    f"({len(builder.resolved_schema)} table(s))[/green]"
+                )
+
         # Process manifest if provided
         if manifest:
             builder.add_manifest(manifest, dialect=dialect)
 
-        # Process paths - collect all files first for progress tracking
-        if paths:
-            all_files: list[Path] = []
-            for path in paths:
-                if path.is_dir():
-                    pattern = f"**/{glob_pattern}" if recursive else glob_pattern
-                    all_files.extend(
-                        f for f in sorted(path.glob(pattern)) if f.is_file()
-                    )
-                elif path.is_file():
-                    all_files.append(path)
-                else:
-                    err_console.print(f"[red]Error:[/red] Path not found: {path}")
-                    raise typer.Exit(1)
-            builder.add_files(all_files, dialect=dialect)
-
-        # Dump resolved schema if requested
-        if dump_schema:
-            from sqlglider.graph.formatters import format_schema
-
-            schema_content = format_schema(builder.resolved_schema, dump_schema_format)
-            dump_schema.write_text(schema_content, encoding="utf-8")
-            console.print(
-                f"[green]Schema dumped to {dump_schema} "
-                f"({len(builder.resolved_schema)} table(s))[/green]"
-            )
+        # Process path-based files
+        if path_files:
+            builder.add_files(path_files, dialect=dialect)
 
         # Build and save graph
         graph = builder.build()