sql-glider 0.1.4__tar.gz → 0.1.6__tar.gz

{sql_glider-0.1.4 → sql_glider-0.1.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sql-glider
-Version: 0.1.4
+Version: 0.1.6
 Summary: SQL Utility Toolkit for better understanding, use, and governance of your queries in a native environment.
 Project-URL: Homepage, https://github.com/rycowhi/sql-glider/
 Project-URL: Repository, https://github.com/rycowhi/sql-glider/

sql_glider-0.1.6/plans/2026-01-26-file-scoped-schema-context.md

@@ -0,0 +1,201 @@
+# Plan: File-Scoped Schema Context for SQL Lineage Analyzer
+
+**Status:** Completed
+
+## Summary
+
+Add file-scoped schema context to the SQL Glider lineage analyzer so that SQLGlot can correctly expand `SELECT *` and trace cross-statement references when a file contains multiple related statements.
+
+## Problem
+
+When analyzing this SQL:
+```sql
+CREATE TEMPORARY VIEW first_view AS (SELECT a, b, c FROM source_table);
+CREATE TEMPORARY VIEW second_view AS
+WITH first_view_cte AS (
+    SELECT *, row_number() OVER (PARTITION BY a ORDER BY b DESC) AS row_num
+    FROM first_view
+)
+SELECT * FROM first_view_cte WHERE c = 1;
+```
+
+**Previous output:** `* -> second_view.*` (useless - no column-level lineage)
+**Expected output:** `first_view.a -> second_view.a`, `first_view.b -> second_view.b`, etc.
+
+## Root Cause
+
+SQLGlot's `lineage()` function accepts a `schema` parameter that provides table/view column definitions. Without this schema context, SQLGlot cannot expand `SELECT *` to actual column names.
+
+## Solution
+
+Build up schema context incrementally as CREATE VIEW/TABLE statements are processed, then pass that schema to subsequent `lineage()` calls.
+
+---
+
+## Implementation Steps
+
+### 1. Add Schema Instance Variable
+
+- [x] Add `_file_schema: Dict[str, Dict[str, str]] = {}` to `LineageAnalyzer.__init__()`
+
+### 2. Add Schema Extraction Methods
+
+- [x] `_extract_schema_from_statement()` - Extract columns from CREATE VIEW/TABLE AS SELECT
+- [x] `_extract_columns_from_select()` - Extract column names from SELECT projections
+- [x] `_resolve_star_columns()` - Resolve SELECT * from file schema or CTEs
+- [x] `_resolve_source_columns()` - Resolve columns from a single source (table, subquery)
+- [x] `_resolve_qualified_star()` - Resolve table-qualified star (e.g., `t.*`)
+- [x] `_extract_subquery_columns()` - Extract columns from subquery's SELECT
+- [x] `_resolve_cte_columns()` - Resolve columns from CTE definitions
+- [x] `_extract_cte_select_columns()` - Extract columns from CTE's SELECT
+
+### 3. Integrate Schema Building into Analysis Loop
+
+- [x] Reset `_file_schema = {}` at start of `analyze_queries()`
+- [x] Call `_extract_schema_from_statement(expr)` in `finally` block AFTER analysis
+- [x] Critical: Schema must be extracted AFTER analysis to avoid confusing SQLGlot
+
+### 4. Pass Schema to lineage() Calls
+
+- [x] Modify `_analyze_column_lineage_internal()` to pass schema:
+```python
+node = lineage(
+    lineage_col,
+    current_query_sql,
+    dialect=self.dialect,
+    schema=self._file_schema if self._file_schema else None,
+)
+```
+
+### 5. Handle SELECT * in get_output_columns()
+
+- [x] Handle `exp.Star` projections by resolving from file schema
+- [x] Handle table-qualified stars (`t.*`) represented as `exp.Column` with `exp.Star` as `this`
+
+---
+
+## Edge Cases Handled
+
+| Case | Implementation |
+|------|----------------|
+| `SELECT *` from unknown table | Returns empty columns, falls back to `*` behavior |
+| Nested `SELECT *` through CTEs | Resolves CTE source from schema first |
+| UNION in CREATE VIEW | Uses first branch's columns |
+| Expressions without aliases | Uses SQL representation as column name |
+| TEMPORARY VIEW | Treated same as regular VIEW |
+| Multiple JOINs | Collects columns from all joined tables |
+| LEFT/RIGHT/FULL OUTER JOIN | Same handling as INNER JOIN |
+| CROSS JOIN | Same handling as INNER JOIN |
+| Subquery in FROM clause | Extracts columns from inner SELECT |
+| Table aliases (`v1 AS x`) | Resolves alias to actual table name |
+| Schema-qualified names | Handles `schema.table` correctly |
+| CTE referencing earlier CTE | Recursive CTE column resolution |
+| `SELECT *, extra_col` | Combines * expansion with extra columns |
+| Table-qualified `t.*` | Handles `v1.*` style syntax |
+| LATERAL VIEW explode | Collects generated columns from `laterals` clause |
+| LATERAL VIEW posexplode | Collects both position and element columns |
+| Multiple LATERAL VIEWs | Collects columns from all LATERAL VIEWs |
+| LATERAL VIEW OUTER | Same handling as regular LATERAL VIEW |
+| LEFT SEMI JOIN | Only includes left table columns (right table excluded) |
+| LEFT ANTI JOIN | Only includes left table columns (right table excluded) |
+
+---
+
+## Files Modified
+
+| File | Changes |
+|------|---------|
+| `src/sqlglider/lineage/analyzer.py` | Added `_file_schema` instance variable; Added 9 schema extraction methods (including `_resolve_lateral_columns`); Modified `analyze_queries()` and `_analyze_column_lineage_internal()` and `get_output_columns()`; Added SEMI/ANTI join handling in `_resolve_star_columns()` |
+| `tests/sqlglider/lineage/test_analyzer.py` | Added `TestFileSchemaExtraction` (9 tests), `TestCrossStatementLineage` (12 tests), `TestLateralViewColumnResolution` (5 tests), and `TestSemiAntiJoinColumnResolution` (3 tests) |
+
+---
+
+## Testing
+
+### Test Classes Added
+
+**TestFileSchemaExtraction (9 tests):**
+- `test_extract_schema_from_create_view`
+- `test_extract_schema_from_create_temporary_view`
+- `test_extract_schema_from_create_table_as`
+- `test_extract_schema_with_aliases`
+- `test_extract_schema_select_star_from_known_table`
+- `test_extract_schema_select_star_from_unknown_table`
+- `test_schema_not_extracted_from_pure_select`
+- `test_schema_not_extracted_from_insert`
+- `test_schema_reset_between_analysis_calls`
+
+**TestCrossStatementLineage (12 tests):**
+- `test_view_referencing_earlier_view`
+- `test_select_star_expansion_through_view`
+- `test_cte_with_select_star_from_view`
+- `test_window_function_with_select_star`
+- `test_insert_from_view_lineage`
+- `test_multi_hop_view_lineage`
+- `test_original_problem_scenario`
+- `test_select_star_from_join`
+- `test_nested_ctes_and_views_with_select_star`
+- `test_select_star_from_subquery`
+- `test_table_qualified_star`
+- `test_table_qualified_star_with_alias`
+
+**TestLateralViewColumnResolution (5 tests):**
+- `test_select_star_with_lateral_view_explode`
+- `test_select_star_with_lateral_view_posexplode`
+- `test_select_star_with_multiple_lateral_views`
+- `test_select_star_with_lateral_view_outer`
+- `test_lateral_view_with_join`
+
+### Verification Commands
+
+```bash
+# Run all tests
+uv run pytest --cov=sqlglider --cov-fail-under=80
+
+# Run schema-related tests
+uv run pytest tests/sqlglider/lineage/test_analyzer.py -k "schema or CrossStatement" -v
+
+# Test the original problem scenario
+uv run sqlglider graph build test_view_window_cte.sql --dialect spark --output graph.json
+```
+
+---
+
+## Implementation Notes
+
+### Critical Timing Issue
+
+Initially, schema extraction was done BEFORE analysis in the loop, which caused SQLGlot to return unqualified column names (e.g., `customer_id` instead of `orders.customer_id`).
+
+**Fix:** Move `_extract_schema_from_statement(expr)` to the `finally` block AFTER analysis completes. This ensures:
+1. The current statement is analyzed without its own schema (correct behavior)
+2. The schema is then extracted for use by subsequent statements
+
+### Table-Qualified Star Handling
+
+Table-qualified stars (`v1.*`) are represented differently than unqualified stars (`*`):
+- `*` is `exp.Star`
+- `v1.*` is `exp.Column` with `this` being `exp.Star` and `table` being `v1`
+
+Both cases needed handling in:
+- `_extract_columns_from_select()` for schema extraction
+- `get_output_columns()` for lineage analysis output
+
+### Subquery Column Resolution
+
+For `SELECT * FROM (SELECT * FROM v1) sub`, the code:
+1. Detects the subquery in `_resolve_source_columns()`
+2. Extracts columns from the inner SELECT via `_extract_subquery_columns()`
+3. Recursively resolves any `SELECT *` in the inner query
+
+---
+
+## Lessons Learned
+
+1. **Timing matters:** Schema context must be built AFTER analyzing a statement, not before, to avoid confusing SQLGlot's lineage tracing.
+
+2. **AST structure varies:** Different SQL constructs have different AST representations (e.g., `*` vs `t.*`), requiring multiple code paths.
+
+3. **Recursive resolution:** CTEs and subqueries can reference other CTEs/views, requiring recursive column resolution.
+
+4. **Edge cases compound:** JOINs + aliases + qualified stars can all combine, requiring careful handling of each case.

{sql_glider-0.1.4 → sql_glider-0.1.6}/src/sqlglider/_version.py

@@ -28,7 +28,7 @@ version_tuple: VERSION_TUPLE
 commit_id: COMMIT_ID
 __commit_id__: COMMIT_ID
 
-__version__ = version = '0.1.4'
-__version_tuple__ = version_tuple = (0, 1, 4)
+__version__ = version = '0.1.6'
+__version_tuple__ = version_tuple = (0, 1, 6)
 
 __commit_id__ = commit_id = None

{sql_glider-0.1.4 → sql_glider-0.1.6}/src/sqlglider/lineage/analyzer.py

@@ -182,20 +182,43 @@ class LineageAnalyzer:
                     # For aliased columns, use the alias as the column name
                     column_name = projection.alias
                     lineage_name = column_name  # SQLGlot lineage uses the alias
-                else:
-                    source_expr = projection
-                    if isinstance(source_expr, exp.Column):
-                        column_name = source_expr.name
-                        lineage_name = column_name
+                    # Qualify with target table
+                    qualified_name = f"{target_table}.{column_name}"
+                    columns.append(qualified_name)
+                    self._column_mapping[qualified_name] = lineage_name
+                elif isinstance(projection, exp.Column):
+                    # Check if this is a table-qualified star (e.g., t.*)
+                    if isinstance(projection.this, exp.Star):
+                        source_table = projection.table
+                        qualified_star_cols: List[str] = []
+                        if source_table and first_select:
+                            qualified_star_cols = self._resolve_qualified_star(
+                                source_table, first_select
+                            )
+                            for col in qualified_star_cols:
+                                qualified_name = f"{target_table}.{col}"
+                                columns.append(qualified_name)
+                                self._column_mapping[qualified_name] = col
+                        if not qualified_star_cols:
+                            # Fallback: can't resolve t.*, use * as column name
+                            qualified_name = f"{target_table}.*"
+                            columns.append(qualified_name)
+                            self._column_mapping[qualified_name] = "*"
                     else:
-                        # For expressions, use the SQL representation
-                        column_name = source_expr.sql(dialect=self.dialect)
+                        column_name = projection.name
                         lineage_name = column_name
-
-                # Qualify with target table
-                qualified_name = f"{target_table}.{column_name}"
-                columns.append(qualified_name)
-                self._column_mapping[qualified_name] = lineage_name
+                        # Qualify with target table
+                        qualified_name = f"{target_table}.{column_name}"
+                        columns.append(qualified_name)
+                        self._column_mapping[qualified_name] = lineage_name
+                else:
+                    # For expressions, use the SQL representation
+                    column_name = projection.sql(dialect=self.dialect)
+                    lineage_name = column_name
+                    # Qualify with target table
+                    qualified_name = f"{target_table}.{column_name}"
+                    columns.append(qualified_name)
+                    self._column_mapping[qualified_name] = lineage_name
 
         else:
             # DQL (pure SELECT): Use the SELECT columns as output
@@ -1342,8 +1365,18 @@ class LineageAnalyzer:
                 # Use the alias name as the column name
                 columns.append(projection.alias)
             elif isinstance(projection, exp.Column):
-                # Use the column name
-                columns.append(projection.name)
+                # Check if this is a table-qualified star (e.g., t.*)
+                if isinstance(projection.this, exp.Star):
+                    # Resolve table-qualified star from known schema
+                    table_name = projection.table
+                    if table_name and first_select:
+                        qualified_star_cols = self._resolve_qualified_star(
+                            table_name, first_select
+                        )
+                        columns.extend(qualified_star_cols)
+                else:
+                    # Use the column name
+                    columns.append(projection.name)
             elif isinstance(projection, exp.Star):
                 # Resolve SELECT * from known schema
                 if first_select:
@@ -1375,6 +1408,64 @@ class LineageAnalyzer:
 
         source = from_clause.this
 
+        # Handle table reference from FROM clause
+        columns.extend(self._resolve_source_columns(source, select_node))
+
+        # Handle JOIN clauses - collect columns from all joined tables
+        # EXCEPT for SEMI and ANTI joins which only return left table columns
+        joins = select_node.args.get("joins")
+        if joins:
+            for join in joins:
+                if isinstance(join, exp.Join):
+                    # SEMI and ANTI joins don't include right table columns in SELECT *
+                    join_kind = join.kind
+                    if join_kind in ("SEMI", "ANTI"):
+                        # Skip right table columns for SEMI/ANTI joins
+                        continue
+                    join_source = join.this
+                    columns.extend(
+                        self._resolve_source_columns(join_source, select_node)
+                    )
+
+        # Handle LATERAL VIEW clauses - collect generated columns
+        laterals = select_node.args.get("laterals")
+        if laterals:
+            for lateral in laterals:
+                if isinstance(lateral, exp.Lateral):
+                    lateral_cols = self._resolve_lateral_columns(lateral)
+                    columns.extend(lateral_cols)
+
+        return columns
+
+    def _resolve_lateral_columns(self, lateral: exp.Lateral) -> List[str]:
+        """
+        Extract generated column names from a LATERAL VIEW clause.
+
+        Args:
+            lateral: The Lateral expression node
+
+        Returns:
+            List of generated column names (e.g., ['elem'] for explode,
+            ['pos', 'elem'] for posexplode)
+        """
+        # Use SQLGlot's built-in property to get alias column names
+        return lateral.alias_column_names or []
+
+    def _resolve_source_columns(
+        self, source: exp.Expression, select_node: exp.Select
+    ) -> List[str]:
+        """
+        Resolve columns from a single source (table, subquery, etc.).
+
+        Args:
+            source: The source expression (Table, Subquery, etc.)
+            select_node: The containing SELECT node for CTE resolution
+
+        Returns:
+            List of column names from the source
+        """
+        columns: List[str] = []
+
         # Handle table reference
         if isinstance(source, exp.Table):
             source_name = self._get_qualified_table_name(source)
@@ -1387,11 +1478,100 @@ class LineageAnalyzer:
                 cte_columns = self._resolve_cte_columns(source_name, select_node)
                 columns.extend(cte_columns)
 
-        # Handle subquery - can't resolve without deeper analysis
-        elif isinstance(source, exp.Subquery) and source.alias:
-            # Check if this subquery alias is in file schema (unlikely)
-            if source.alias in self._file_schema:
+        # Handle subquery with alias
+        elif isinstance(source, exp.Subquery):
+            # First check if this subquery alias is in file schema
+            if source.alias and source.alias in self._file_schema:
                 columns.extend(self._file_schema[source.alias].keys())
+            else:
+                # Extract columns from the subquery's SELECT
+                inner_select = source.this
+                if isinstance(inner_select, exp.Select):
+                    subquery_cols = self._extract_subquery_columns(inner_select)
+                    columns.extend(subquery_cols)
+
+        return columns
+
+    def _resolve_qualified_star(
+        self, table_name: str, select_node: exp.Select
+    ) -> List[str]:
+        """
+        Resolve a table-qualified star (e.g., t.*) to actual column names.
+
+        Args:
+            table_name: The table/alias name qualifying the star
+            select_node: The SELECT node for context
+
+        Returns:
+            List of column names from the specified table
+        """
+        # First check file schema
+        if table_name in self._file_schema:
+            return list(self._file_schema[table_name].keys())
+
+        # Check if it's a CTE reference
+        cte_columns = self._resolve_cte_columns(table_name, select_node)
+        if cte_columns:
+            return cte_columns
+
+        # Check if the table name is an alias - need to resolve the actual table
+        from_clause = select_node.args.get("from")
+        if from_clause and isinstance(from_clause, exp.From):
+            source = from_clause.this
+            if isinstance(source, exp.Table) and source.alias == table_name:
+                actual_name = self._get_qualified_table_name(source)
+                if actual_name in self._file_schema:
+                    return list(self._file_schema[actual_name].keys())
+
+            # Check JOIN clauses for aliased tables
+            joins = select_node.args.get("joins")
+            if joins:
+                for join in joins:
+                    if isinstance(join, exp.Join):
+                        join_source = join.this
+                        if (
+                            isinstance(join_source, exp.Table)
+                            and join_source.alias == table_name
+                        ):
+                            actual_name = self._get_qualified_table_name(join_source)
+                            if actual_name in self._file_schema:
+                                return list(self._file_schema[actual_name].keys())
+
+        return []
+
+    def _extract_subquery_columns(self, subquery_select: exp.Select) -> List[str]:
+        """
+        Extract column names from a subquery's SELECT statement.
+
+        Args:
+            subquery_select: The SELECT expression within the subquery
+
+        Returns:
+            List of column names
+        """
+        columns: List[str] = []
+
+        for projection in subquery_select.expressions:
+            if isinstance(projection, exp.Alias):
+                columns.append(projection.alias)
+            elif isinstance(projection, exp.Column):
+                # Check for table-qualified star (t.*)
+                if isinstance(projection.this, exp.Star):
+                    table_name = projection.table
+                    if table_name:
+                        qualified_cols = self._resolve_qualified_star(
+                            table_name, subquery_select
+                        )
+                        columns.extend(qualified_cols)
+                else:
+                    columns.append(projection.name)
+            elif isinstance(projection, exp.Star):
+                # Resolve SELECT * in subquery
+                star_columns = self._resolve_star_columns(subquery_select)
+                columns.extend(star_columns)
+            else:
+                col_sql = projection.sql(dialect=self.dialect)
+                columns.append(col_sql)
 
         return columns
 

sql_glider-0.1.6/tests/fixtures/original_queries/test_cte_view_star.sql

@@ -0,0 +1,22 @@
+CREATE TEMPORARY VIEW first_view AS
+SELECT
+    v1,
+    v2,
+    v3
+FROM source_db.source_table;
+CREATE TEMPORARY VIEW second_view AS WITH cte AS (
+    SELECT
+        *,
+        row_number() OVER (ORDER BY v1) AS row_num
+    FROM first_view
+)
+
+SELECT * FROM cte
+WHERE row_num = 1;
+
+INSERT INTO target_db.target_table
+SELECT
+    v1,
+    v2,
+    v3
+FROM second_view;