sql-testing-library 0.19.0__tar.gz → 0.21.0__tar.gz

{sql_testing_library-0.19.0 → sql_testing_library-0.21.0}/CHANGELOG.md

@@ -5,6 +5,25 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 0.21.0 (2026-01-03)
+
+### Feat
+
+- enable Redshift nested element access tests (#140)
+- add Redshift support for deeply nested complex types via SUPER (#139)
+- add DuckDB support for deeply nested complex types (#138)
+- add deeply nested complex types support for Athena and Trino (#136)
+
+### Fix
+
+- use GET_ARRAY_LENGTH for Redshift SUPER type arrays
+
+## 0.20.0 (2025-12-07)
+
+### Feat
+
+- add BigQueryMockTable class for explicit three-part naming (#134)
+
 ## 0.19.0 (2025-12-07)
 
 ### Feat

{sql_testing_library-0.19.0 → sql_testing_library-0.21.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sql-testing-library
-Version: 0.19.0
+Version: 0.21.0
 Summary: SQL Testing Framework for Python: Unit test SQL queries with mock data injection for BigQuery, Snowflake, Redshift, Athena, Trino, and DuckDB. Simplify data engineering ETL testing and analytics validation.
 License: MIT
 License-File: LICENSE
@@ -117,7 +117,7 @@ For more details on our journey and the engineering challenges we solved, read t
 - **CTE or Physical Tables**: Automatic fallback for query size limits
 - **Type-Safe Results**: Deserialize results to Pydantic models
 - **Pytest Integration**: Seamless testing with `@sql_test` decorator
-- **SQL Logging**: Comprehensive SQL logging with formatted output, error traces, and temp table queries
+- **SQL Logging**: Comprehensive SQL logging with formatted output, error traces, and temp table queries. Automatically logs SQL on SQL-level failures (syntax errors, table not found). For assertion failures or debugging, use `log_sql=True` or `SQL_TEST_LOG_ALL=true` environment variable
 
 ## Data Types Support
 
@@ -145,17 +145,20 @@ The library supports different data types across database engines. All checkmark
 | **Decimal Array** | `List[Decimal]` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
 | **Optional Array** | `Optional[List[T]]` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
 | **Map/Dict** | `Dict[K, V]` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
-| **Struct/Record** | `dataclass` | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ |
-| **Nested Arrays** | `List[List[T]]` | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
+| **Struct/Record** | `dataclass` | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
+| **Nested Arrays** | `List[List[T]]` | ❌ | ✅ | ✅ | ✅ | 🚧 | ✅ |
+| **Arrays of Structs** | `List[dataclass]` | ✅ | ✅ | ✅ | ✅ | 🚧 | ✅ |
+| **3D Arrays** | `List[List[List[T]]]` | ❌ | ✅ | ✅ | ✅ | 🚧 | ✅ |
+| **Arrays of Arrays of Structs** | `List[List[dataclass]]` | ❌ | ✅ | ✅ | ✅ | 🚧 | ✅ |
 
 ### Database-Specific Notes
 
-- **BigQuery**: NULL arrays become empty arrays `[]`; uses scientific notation for large decimals; dict/map types stored as JSON strings; struct types supported using `STRUCT` syntax with named fields (dataclasses and Pydantic models)
-- **Athena**: 256KB query size limit; supports arrays and maps using `ARRAY[]` and `MAP(ARRAY[], ARRAY[])` syntax; supports struct types using `ROW` with named fields (dataclasses and Pydantic models)
-- **Redshift**: Arrays and maps implemented via SUPER type (JSON parsing); 16MB query size limit; struct types not yet supported
-- **Trino**: Memory catalog for testing; excellent decimal precision; supports arrays, maps, and struct types using `ROW` with named fields (dataclasses and Pydantic models)
-- **Snowflake**: Column names normalized to lowercase; 1MB query size limit; dict/map types implemented via VARIANT type (JSON parsing); struct types not yet supported
-- **DuckDB**: Fast embedded analytics database; excellent SQL standards compliance; supports arrays, maps, and struct types using `STRUCT` syntax with named fields (dataclasses and Pydantic models)
+- **BigQuery**: NULL arrays become empty arrays `[]`; uses scientific notation for large decimals; dict/map types stored as JSON strings; struct types supported using `STRUCT` syntax with named fields (dataclasses and Pydantic models); **does not support nested arrays** (database limitation)
+- **Athena**: 256KB query size limit; supports arrays and maps using `ARRAY[]` and `MAP(ARRAY[], ARRAY[])` syntax; supports struct types using `ROW` with named fields (dataclasses and Pydantic models); **full support for deeply nested types** including nested arrays, arrays of structs, and 3D arrays
+- **Redshift**: Arrays and maps implemented via SUPER type (JSON parsing); 16MB query size limit; **full support for deeply nested types** including struct types using SUPER + JSON_PARSE, nested arrays (2D, 3D+), arrays of structs, and arrays of arrays of structs
+- **Trino**: Memory catalog for testing; excellent decimal precision; supports arrays, maps, and struct types using `ROW` with named fields (dataclasses and Pydantic models); **full support for deeply nested types** including nested arrays, arrays of structs, and 3D arrays
+- **Snowflake**: Column names normalized to lowercase; 1MB query size limit; dict/map types implemented via VARIANT type (JSON parsing); struct types not yet supported (🚧 TODO); nested arrays not yet supported (🚧 TODO)
+- **DuckDB**: Fast embedded analytics database; excellent SQL standards compliance; supports arrays, maps, and struct types using `STRUCT` syntax with named fields (dataclasses and Pydantic models); **full support for deeply nested types** including nested arrays (2D, 3D+), arrays of structs, and arrays of arrays of structs
 
 ## Execution Modes Support
 
@@ -570,8 +573,16 @@ TestCase(
 #### Example Mock Table Implementations:
 
 ```python
-# BigQuery Mock Table
-class UsersMockTable(BaseMockTable):
+# BigQuery Mock Table (Recommended: Use BigQueryMockTable for clearer three-part naming)
+from sql_testing_library import BigQueryMockTable
+
+class UsersMockTable(BigQueryMockTable):
+    project_name = "test-project"
+    dataset_name = "test_dataset"
+    table_name = "users"
+
+# BigQuery Mock Table (Alternative: Use BaseMockTable with combined project.dataset)
+class UsersMockTableAlternative(BaseMockTable):
     def get_database_name(self) -> str:
         return "test-project.test_dataset"  # project.dataset format
 
@@ -1047,6 +1058,7 @@ The adapter_type parameter will use the configuration from the corresponding sec
 - Supports Google Cloud BigQuery service
 - Uses UNION ALL pattern for CTE creation with complex data types
 - Handles authentication via service account or application default credentials
+- **Special Feature**: `BigQueryMockTable` class for explicit three-part naming (project.dataset.table)
 
 #### Athena Adapter
 - Supports Amazon Athena service for querying data in S3
@@ -1085,6 +1097,94 @@ The adapter_type parameter will use the configuration from the corresponding sec
 - No authentication required - perfect for local development and testing
 - Excellent performance for analytical workloads
 
+### BigQuery-Specific: BigQueryMockTable
+
+BigQuery uses a three-part naming scheme (`project.dataset.table`) which doesn't fit naturally into the two-part `database.table` model used by most databases. The `BigQueryMockTable` class provides explicit support for BigQuery's naming convention.
+
+#### The Problem with BaseMockTable
+
+Using `BaseMockTable` for BigQuery requires awkwardly cramming the project and dataset together:
+
+```python
+# Awkward: Combines project.dataset into database_name
+class UsersMockTable(BaseMockTable):
+    def get_database_name(self) -> str:
+        return "test-project.test_dataset"  # Confusing!
+
+    def get_table_name(self) -> str:
+        return "users"
+```
+
+#### The Solution: BigQueryMockTable
+
+`BigQueryMockTable` makes BigQuery's three-part naming explicit and clear:
+
+```python
+from sql_testing_library import BigQueryMockTable
+
+class UsersMockTable(BigQueryMockTable):
+    project_name = "test-project"
+    dataset_name = "test_dataset"
+    table_name = "users"
+```
+
+#### Usage Examples
+
+**Basic Usage:**
+```python
+from sql_testing_library import BigQueryMockTable
+
+class UsersMockTable(BigQueryMockTable):
+    project_name = "my-project"
+    dataset_name = "analytics"
+    table_name = "users"
+
+class OrdersMockTable(BigQueryMockTable):
+    project_name = "my-project"
+    dataset_name = "analytics"
+    table_name = "orders"
+```
+
+**Avoid Repetition with Inheritance:**
+```python
+# Base class for all tables in the same project
+class MyProjectTable(BigQueryMockTable):
+    project_name = "my-project"
+
+# Subclasses only specify dataset and table
+class UsersTable(MyProjectTable):
+    dataset_name = "analytics"
+    table_name = "users"
+
+class OrdersTable(MyProjectTable):
+    dataset_name = "analytics"
+    table_name = "orders"
+```
+
+**Available Methods:**
+```python
+table = UsersMockTable([...])
+
+# BigQuery-specific methods
+table.get_project_name()           # "my-project"
+table.get_dataset_name()           # "analytics"
+table.get_fully_qualified_name()   # "my-project.analytics.users"
+
+# Backwards compatible methods (from BaseMockTable)
+table.get_database_name()          # "my-project.analytics"
+table.get_table_name()             # "users"
+table.get_qualified_name()         # "my-project.analytics.users"
+table.get_cte_alias()              # "my_project_analytics__users"
+```
+
+**Benefits:**
+- ✅ **Clear Semantics**: Each BigQuery component is explicit
+- ✅ **No Confusion**: No more cramming project.dataset together
+- ✅ **Type Safe**: Full type hints and IDE autocomplete
+- ✅ **Backwards Compatible**: Still implements all `BaseMockTable` methods
+- ✅ **Simple**: Just 3 class variables to set
+- ✅ **Flexible**: Use inheritance to share common properties
+
 **Default Behavior:**
 - If adapter_type is not specified in the TestCase or decorator, the library will use the adapter specified in the `[sql_testing]` section's `adapter` setting.
 - If no adapter is specified in the `[sql_testing]` section, it defaults to "bigquery".
@@ -1291,7 +1391,7 @@ The library automatically:
 - Injects mock data via CTEs or temp tables
 - Deserializes results to typed Python objects
 
-For detailed usage and configuration options, see the example files included.
+For detailed usage and configuration options, see the [documentation](https://gurmeetsaran.github.io/sqltesting/).
 
 ## Integration with Mocksmith
 
@@ -1326,19 +1426,36 @@ class Customer:
 customers = [Customer.mock() for _ in range(100)]
 ```
 
-See the [Mocksmith Integration Guide](docs/mocksmith_integration.md) and [examples](examples/mocksmith_integration_example.py) for detailed usage patterns.
+See the [Mocksmith Integration Guide](docs/mocksmith_integration.md) for detailed usage patterns.
 
 ## Known Limitations and TODOs
 
 The library has a few known limitations that are planned to be addressed in future updates:
 
-### Struct Type Support
-- **Redshift**: Struct types are not supported due to lack of native struct/record types (uses SUPER type for JSON)
-- **Snowflake**: Struct types are not supported due to lack of native struct/record types (uses VARIANT type for JSON)
+### Deeply Nested Complex Types Support
+
+**✅ Fully Supported (Athena, Trino, DuckDB & Redshift):**
+- Nested arrays (2D, 3D+): `List[List[int]]`, `List[List[List[int]]]`
+- Arrays of structs: `List[Address]` where Address is a dataclass
+- Arrays of arrays of structs: `List[List[OrderItem]]`
+- Maps with complex values: `Dict[str, str]`, `Dict[str, int]`
+- See `tests/integration/test_deeply_nested_types_integration.py` for comprehensive examples
+- **16 tests passing** across Athena, Trino, DuckDB, and Redshift (both CTE and physical tables modes)
+
+**Implementation Details:**
+- **Athena/Trino**: Use ROW types with named fields
+- **DuckDB**: Use STRUCT types with dictionary-style syntax
+- **Redshift**: Use SUPER type with JSON_PARSE for serialization, recursive JSON parsing for deserialization
 
+**🚧 TODO - Implementation Needed:**
+- **BigQuery**: Does not support nested arrays - this is a **database limitation**, not a library limitation. BigQuery's type system doesn't allow `ARRAY<ARRAY<T>>` constructs
+- **Snowflake**:
+  - Struct type support not implemented (see `Struct type not yet supported for dialect: snowflake`)
+  - Nested arrays not yet supported
+  - TODO: Implement struct/nested type support using VARIANT/OBJECT types
 
 ### Database-Specific Limitations
-- **BigQuery**: Does not support nested arrays (arrays of arrays). This is a BigQuery database limitation, not a library limitation. (See TODO in `test_struct_types_integration.py:test_nested_lists`)
+- **BigQuery**: Does not support nested arrays (arrays of arrays). This is a BigQuery database limitation, not a library limitation.
 
 ### General Improvements
 - Add support for more SQL dialects

{sql_testing_library-0.19.0 → sql_testing_library-0.21.0}/README.md

@@ -57,7 +57,7 @@ For more details on our journey and the engineering challenges we solved, read t
 - **CTE or Physical Tables**: Automatic fallback for query size limits
 - **Type-Safe Results**: Deserialize results to Pydantic models
 - **Pytest Integration**: Seamless testing with `@sql_test` decorator
-- **SQL Logging**: Comprehensive SQL logging with formatted output, error traces, and temp table queries
+- **SQL Logging**: Comprehensive SQL logging with formatted output, error traces, and temp table queries. Automatically logs SQL on SQL-level failures (syntax errors, table not found). For assertion failures or debugging, use `log_sql=True` or `SQL_TEST_LOG_ALL=true` environment variable
 
 ## Data Types Support
 
@@ -85,17 +85,20 @@ The library supports different data types across database engines. All checkmark
 | **Decimal Array** | `List[Decimal]` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
 | **Optional Array** | `Optional[List[T]]` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
 | **Map/Dict** | `Dict[K, V]` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
-| **Struct/Record** | `dataclass` | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ |
-| **Nested Arrays** | `List[List[T]]` | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
+| **Struct/Record** | `dataclass` | ✅ | ✅ | ✅ | ✅ | ❌ | ✅ |
+| **Nested Arrays** | `List[List[T]]` | ❌ | ✅ | ✅ | ✅ | 🚧 | ✅ |
+| **Arrays of Structs** | `List[dataclass]` | ✅ | ✅ | ✅ | ✅ | 🚧 | ✅ |
+| **3D Arrays** | `List[List[List[T]]]` | ❌ | ✅ | ✅ | ✅ | 🚧 | ✅ |
+| **Arrays of Arrays of Structs** | `List[List[dataclass]]` | ❌ | ✅ | ✅ | ✅ | 🚧 | ✅ |
 
 ### Database-Specific Notes
 
-- **BigQuery**: NULL arrays become empty arrays `[]`; uses scientific notation for large decimals; dict/map types stored as JSON strings; struct types supported using `STRUCT` syntax with named fields (dataclasses and Pydantic models)
-- **Athena**: 256KB query size limit; supports arrays and maps using `ARRAY[]` and `MAP(ARRAY[], ARRAY[])` syntax; supports struct types using `ROW` with named fields (dataclasses and Pydantic models)
-- **Redshift**: Arrays and maps implemented via SUPER type (JSON parsing); 16MB query size limit; struct types not yet supported
-- **Trino**: Memory catalog for testing; excellent decimal precision; supports arrays, maps, and struct types using `ROW` with named fields (dataclasses and Pydantic models)
-- **Snowflake**: Column names normalized to lowercase; 1MB query size limit; dict/map types implemented via VARIANT type (JSON parsing); struct types not yet supported
-- **DuckDB**: Fast embedded analytics database; excellent SQL standards compliance; supports arrays, maps, and struct types using `STRUCT` syntax with named fields (dataclasses and Pydantic models)
+- **BigQuery**: NULL arrays become empty arrays `[]`; uses scientific notation for large decimals; dict/map types stored as JSON strings; struct types supported using `STRUCT` syntax with named fields (dataclasses and Pydantic models); **does not support nested arrays** (database limitation)
+- **Athena**: 256KB query size limit; supports arrays and maps using `ARRAY[]` and `MAP(ARRAY[], ARRAY[])` syntax; supports struct types using `ROW` with named fields (dataclasses and Pydantic models); **full support for deeply nested types** including nested arrays, arrays of structs, and 3D arrays
+- **Redshift**: Arrays and maps implemented via SUPER type (JSON parsing); 16MB query size limit; **full support for deeply nested types** including struct types using SUPER + JSON_PARSE, nested arrays (2D, 3D+), arrays of structs, and arrays of arrays of structs
+- **Trino**: Memory catalog for testing; excellent decimal precision; supports arrays, maps, and struct types using `ROW` with named fields (dataclasses and Pydantic models); **full support for deeply nested types** including nested arrays, arrays of structs, and 3D arrays
+- **Snowflake**: Column names normalized to lowercase; 1MB query size limit; dict/map types implemented via VARIANT type (JSON parsing); struct types not yet supported (🚧 TODO); nested arrays not yet supported (🚧 TODO)
+- **DuckDB**: Fast embedded analytics database; excellent SQL standards compliance; supports arrays, maps, and struct types using `STRUCT` syntax with named fields (dataclasses and Pydantic models); **full support for deeply nested types** including nested arrays (2D, 3D+), arrays of structs, and arrays of arrays of structs
 
 ## Execution Modes Support
 
@@ -510,8 +513,16 @@ TestCase(
 #### Example Mock Table Implementations:
 
 ```python
-# BigQuery Mock Table
-class UsersMockTable(BaseMockTable):
+# BigQuery Mock Table (Recommended: Use BigQueryMockTable for clearer three-part naming)
+from sql_testing_library import BigQueryMockTable
+
+class UsersMockTable(BigQueryMockTable):
+    project_name = "test-project"
+    dataset_name = "test_dataset"
+    table_name = "users"
+
+# BigQuery Mock Table (Alternative: Use BaseMockTable with combined project.dataset)
+class UsersMockTableAlternative(BaseMockTable):
     def get_database_name(self) -> str:
         return "test-project.test_dataset"  # project.dataset format
 
@@ -987,6 +998,7 @@ The adapter_type parameter will use the configuration from the corresponding sec
 - Supports Google Cloud BigQuery service
 - Uses UNION ALL pattern for CTE creation with complex data types
 - Handles authentication via service account or application default credentials
+- **Special Feature**: `BigQueryMockTable` class for explicit three-part naming (project.dataset.table)
 
 #### Athena Adapter
 - Supports Amazon Athena service for querying data in S3
@@ -1025,6 +1037,94 @@ The adapter_type parameter will use the configuration from the corresponding sec
 - No authentication required - perfect for local development and testing
 - Excellent performance for analytical workloads
 
+### BigQuery-Specific: BigQueryMockTable
+
+BigQuery uses a three-part naming scheme (`project.dataset.table`) which doesn't fit naturally into the two-part `database.table` model used by most databases. The `BigQueryMockTable` class provides explicit support for BigQuery's naming convention.
+
+#### The Problem with BaseMockTable
+
+Using `BaseMockTable` for BigQuery requires awkwardly cramming the project and dataset together:
+
+```python
+# Awkward: Combines project.dataset into database_name
+class UsersMockTable(BaseMockTable):
+    def get_database_name(self) -> str:
+        return "test-project.test_dataset"  # Confusing!
+
+    def get_table_name(self) -> str:
+        return "users"
+```
+
+#### The Solution: BigQueryMockTable
+
+`BigQueryMockTable` makes BigQuery's three-part naming explicit and clear:
+
+```python
+from sql_testing_library import BigQueryMockTable
+
+class UsersMockTable(BigQueryMockTable):
+    project_name = "test-project"
+    dataset_name = "test_dataset"
+    table_name = "users"
+```
+
+#### Usage Examples
+
+**Basic Usage:**
+```python
+from sql_testing_library import BigQueryMockTable
+
+class UsersMockTable(BigQueryMockTable):
+    project_name = "my-project"
+    dataset_name = "analytics"
+    table_name = "users"
+
+class OrdersMockTable(BigQueryMockTable):
+    project_name = "my-project"
+    dataset_name = "analytics"
+    table_name = "orders"
+```
+
+**Avoid Repetition with Inheritance:**
+```python
+# Base class for all tables in the same project
+class MyProjectTable(BigQueryMockTable):
+    project_name = "my-project"
+
+# Subclasses only specify dataset and table
+class UsersTable(MyProjectTable):
+    dataset_name = "analytics"
+    table_name = "users"
+
+class OrdersTable(MyProjectTable):
+    dataset_name = "analytics"
+    table_name = "orders"
+```
+
+**Available Methods:**
+```python
+table = UsersMockTable([...])
+
+# BigQuery-specific methods
+table.get_project_name()           # "my-project"
+table.get_dataset_name()           # "analytics"
+table.get_fully_qualified_name()   # "my-project.analytics.users"
+
+# Backwards compatible methods (from BaseMockTable)
+table.get_database_name()          # "my-project.analytics"
+table.get_table_name()             # "users"
+table.get_qualified_name()         # "my-project.analytics.users"
+table.get_cte_alias()              # "my_project_analytics__users"
+```
+
+**Benefits:**
+- ✅ **Clear Semantics**: Each BigQuery component is explicit
+- ✅ **No Confusion**: No more cramming project.dataset together
+- ✅ **Type Safe**: Full type hints and IDE autocomplete
+- ✅ **Backwards Compatible**: Still implements all `BaseMockTable` methods
+- ✅ **Simple**: Just 3 class variables to set
+- ✅ **Flexible**: Use inheritance to share common properties
+
 **Default Behavior:**
 - If adapter_type is not specified in the TestCase or decorator, the library will use the adapter specified in the `[sql_testing]` section's `adapter` setting.
 - If no adapter is specified in the `[sql_testing]` section, it defaults to "bigquery".
@@ -1231,7 +1331,7 @@ The library automatically:
 - Injects mock data via CTEs or temp tables
 - Deserializes results to typed Python objects
 
-For detailed usage and configuration options, see the example files included.
+For detailed usage and configuration options, see the [documentation](https://gurmeetsaran.github.io/sqltesting/).
 
 ## Integration with Mocksmith
 
@@ -1266,19 +1366,36 @@ class Customer:
 customers = [Customer.mock() for _ in range(100)]
 ```
 
-See the [Mocksmith Integration Guide](docs/mocksmith_integration.md) and [examples](examples/mocksmith_integration_example.py) for detailed usage patterns.
+See the [Mocksmith Integration Guide](docs/mocksmith_integration.md) for detailed usage patterns.
 
 ## Known Limitations and TODOs
 
 The library has a few known limitations that are planned to be addressed in future updates:
 
-### Struct Type Support
-- **Redshift**: Struct types are not supported due to lack of native struct/record types (uses SUPER type for JSON)
-- **Snowflake**: Struct types are not supported due to lack of native struct/record types (uses VARIANT type for JSON)
+### Deeply Nested Complex Types Support
+
+**✅ Fully Supported (Athena, Trino, DuckDB & Redshift):**
+- Nested arrays (2D, 3D+): `List[List[int]]`, `List[List[List[int]]]`
+- Arrays of structs: `List[Address]` where Address is a dataclass
+- Arrays of arrays of structs: `List[List[OrderItem]]`
+- Maps with complex values: `Dict[str, str]`, `Dict[str, int]`
+- See `tests/integration/test_deeply_nested_types_integration.py` for comprehensive examples
+- **16 tests passing** across Athena, Trino, DuckDB, and Redshift (both CTE and physical tables modes)
+
+**Implementation Details:**
+- **Athena/Trino**: Use ROW types with named fields
+- **DuckDB**: Use STRUCT types with dictionary-style syntax
+- **Redshift**: Use SUPER type with JSON_PARSE for serialization, recursive JSON parsing for deserialization
 
+**🚧 TODO - Implementation Needed:**
+- **BigQuery**: Does not support nested arrays - this is a **database limitation**, not a library limitation. BigQuery's type system doesn't allow `ARRAY<ARRAY<T>>` constructs
+- **Snowflake**:
+  - Struct type support not implemented (see `Struct type not yet supported for dialect: snowflake`)
+  - Nested arrays not yet supported
+  - TODO: Implement struct/nested type support using VARIANT/OBJECT types
 
 ### Database-Specific Limitations
-- **BigQuery**: Does not support nested arrays (arrays of arrays). This is a BigQuery database limitation, not a library limitation. (See TODO in `test_struct_types_integration.py:test_nested_lists`)
+- **BigQuery**: Does not support nested arrays (arrays of arrays). This is a BigQuery database limitation, not a library limitation.
 
 ### General Improvements
 - Add support for more SQL dialects

{sql_testing_library-0.19.0 → sql_testing_library-0.21.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api"
 
 [tool.poetry]
 name = "sql-testing-library"
-version = "0.19.0"
+version = "0.21.0"
 description = "SQL Testing Framework for Python: Unit test SQL queries with mock data injection for BigQuery, Snowflake, Redshift, Athena, Trino, and DuckDB. Simplify data engineering ETL testing and analytics validation."
 authors = ["Gurmeet Saran <gurmeetx@gmail.com>", "Kushal Thakkar <kushal.thakkar@gmail.com>"]
 maintainers = ["Gurmeet Saran <gurmeetx@gmail.com>", "Kushal Thakkar <kushal.thakkar@gmail.com>"]

{sql_testing_library-0.19.0 → sql_testing_library-0.21.0}/src/sql_testing_library/__init__.py

@@ -10,7 +10,7 @@ from ._exceptions import (
     SQLTestingError,  # noqa: F401
     TypeConversionError,  # noqa: F401
 )
-from ._mock_table import BaseMockTable  # noqa: F401
+from ._mock_table import BaseMockTable, BigQueryMockTable  # noqa: F401
 from ._pytest_plugin import sql_test  # noqa: F401
 
 
@@ -27,12 +27,13 @@ try:
 except ImportError:
     __all__ = []
 
-__version__ = "0.19.0"
+__version__ = "0.21.0"
 __all__.extend(
     [
         "SQLTestFramework",
         "TestCase",
         "BaseMockTable",
+        "BigQueryMockTable",
         "DatabaseAdapter",
         "sql_test",
         "SQLTestingError",

{sql_testing_library-0.19.0 → sql_testing_library-0.21.0}/src/sql_testing_library/_adapters/duckdb.py

@@ -34,6 +34,18 @@ except ImportError:
     duckdb = None  # type: ignore
 
 
+# Type mapping from Python types to DuckDB types
+DUCKDB_TYPE_MAPPING = {
+    str: "VARCHAR",
+    int: "BIGINT",
+    float: "DOUBLE",
+    bool: "BOOLEAN",
+    date: "DATE",
+    datetime: "TIMESTAMP",
+    Decimal: "DECIMAL",
+}
+
+
 class DuckDBTypeConverter(BaseTypeConverter):
     """DuckDB-specific type converter."""
 
@@ -146,8 +158,35 @@ class DuckDBAdapter(DatabaseAdapter):
 
             # Use DuckDB's efficient DataFrame insertion
             self.connection.register("temp_df", df)
-            columns = ", ".join(df.columns)
-            insert_sql = f"INSERT INTO {temp_table_name} SELECT {columns} FROM temp_df"
+
+            # Build SELECT with explicit casts for MAP types to avoid STRUCT inference
+            column_selects = []
+            column_types = mock_table.get_column_types()
+            for col_name in df.columns:
+                col_type = column_types.get(col_name, str)
+
+                # Handle Optional types
+                if is_union_type(col_type):
+                    non_none_types = [arg for arg in get_args(col_type) if arg is not type(None)]
+                    if non_none_types:
+                        col_type = non_none_types[0]
+
+                # For MAP types, explicitly cast to avoid STRUCT inference
+                if hasattr(col_type, "__origin__") and col_type.__origin__ is dict:
+                    key_type = get_args(col_type)[0] if get_args(col_type) else str
+                    value_type = get_args(col_type)[1] if len(get_args(col_type)) > 1 else str
+
+                    key_db_type = DUCKDB_TYPE_MAPPING.get(key_type, "VARCHAR")
+                    value_db_type = DUCKDB_TYPE_MAPPING.get(value_type, "VARCHAR")
+
+                    column_selects.append(
+                        f"CAST({col_name} AS MAP({key_db_type}, {value_db_type}))"
+                    )
+                else:
+                    column_selects.append(col_name)
+
+            columns_sql = ", ".join(column_selects)
+            insert_sql = f"INSERT INTO {temp_table_name} SELECT {columns_sql} FROM temp_df"
             self.connection.execute(insert_sql)
             self.connection.unregister("temp_df")
 
@@ -185,12 +224,35 @@ class DuckDBAdapter(DatabaseAdapter):
 
         # Insert data if any
         if not df.empty:
-            df = self._prepare_dataframe_for_duckdb(df, mock_table)
-            self.connection.register("temp_df", df)
-            columns = ", ".join(df.columns)
-            insert_sql = f"INSERT INTO {temp_table_name} SELECT {columns} FROM temp_df"
-            self.connection.execute(insert_sql)
-            self.connection.unregister("temp_df")
+            # Check if table has MAP types - use VALUES approach to avoid STRUCT/MAP confusion
+            column_types = mock_table.get_column_types()
+            has_map_types = any(
+                hasattr(ct, "__origin__")
+                and ct.__origin__ is dict
+                or (
+                    is_union_type(ct)
+                    and any(
+                        hasattr(arg, "__origin__") and arg.__origin__ is dict
+                        for arg in get_args(ct)
+                        if arg is not type(None)
+                    )
+                )
+                for ct in column_types.values()
+            )
+
+            if has_map_types:
+                # Use VALUES approach for tables with MAP types to preserve type information
+                # This avoids DuckDB inferring dicts as STRUCTs from pandas DataFrames
+                insert_sql_actual = f"INSERT INTO {temp_table_name} VALUES\n{values_sql}"
+                self.connection.execute(insert_sql_actual)
+            else:
+                # Use DataFrame approach for better performance when no MAP types
+                df = self._prepare_dataframe_for_duckdb(df, mock_table)
+                self.connection.register("temp_df", df)
+                columns = ", ".join(df.columns)
+                insert_sql_actual = f"INSERT INTO {temp_table_name} SELECT {columns} FROM temp_df"
+                self.connection.execute(insert_sql_actual)
+                self.connection.unregister("temp_df")
 
         return temp_table_name, full_sql
 
@@ -217,126 +279,64 @@ class DuckDBAdapter(DatabaseAdapter):
         # DuckDB doesn't have strict query size limits like cloud databases
         return None
 
-    def _generate_create_table_sql(self, mock_table: BaseMockTable, table_name: str) -> str:
-        """Generate CREATE TABLE SQL for DuckDB."""
+    def _get_column_sql_type(self, col_type: Type) -> str:
+        """Recursively convert Python type to DuckDB SQL type.
+
+        Handles nested types like List[List[int]], List[Struct], etc.
+        """
         from .._types import is_struct_type
 
-        column_types = mock_table.get_column_types()
+        # Handle Optional types
+        if is_union_type(col_type):
+            non_none_types = [arg for arg in get_args(col_type) if arg is not type(None)]
+            if non_none_types:
+                col_type = non_none_types[0]
+
+        # Handle List/Array types (recursive)
+        if hasattr(col_type, "__origin__") and col_type.__origin__ is list:
+            element_type = get_args(col_type)[0] if get_args(col_type) else str
+            element_sql_type = self._get_column_sql_type(element_type)  # Recursive call
+            return f"{element_sql_type}[]"
+
+        # Handle Dict/Map types (recursive for key/value types)
+        elif hasattr(col_type, "__origin__") and col_type.__origin__ is dict:
+            key_type = get_args(col_type)[0] if get_args(col_type) else str
+            value_type = get_args(col_type)[1] if len(get_args(col_type)) > 1 else str
+            key_sql_type = self._get_column_sql_type(key_type)  # Recursive call
+            value_sql_type = self._get_column_sql_type(value_type)  # Recursive call
+            return f"MAP({key_sql_type}, {value_sql_type})"
+
+        # Handle Struct types
+        elif is_struct_type(col_type):
+            struct_def = self._get_struct_definition(col_type)
+            return f"STRUCT{struct_def}"
+
+        # Handle scalar types
+        else:
+            return DUCKDB_TYPE_MAPPING.get(col_type, "VARCHAR")
 
-        # Type mapping from Python types to DuckDB types
-        type_mapping = {
-            str: "VARCHAR",
-            int: "BIGINT",
-            float: "DOUBLE",
-            bool: "BOOLEAN",
-            date: "DATE",
-            datetime: "TIMESTAMP",
-            Decimal: "DECIMAL",
-        }
+    def _generate_create_table_sql(self, mock_table: BaseMockTable, table_name: str) -> str:
+        """Generate CREATE TABLE SQL for DuckDB."""
+        column_types = mock_table.get_column_types()
 
         column_defs = []
         for col_name, col_type in column_types.items():
-            # Handle Optional types (both Optional[X] and X | None)
-            if is_union_type(col_type):
-                # Extract the non-None type from Optional[T] or T | None
-                non_none_types = [arg for arg in get_args(col_type) if arg is not type(None)]
-                if non_none_types:
-                    col_type = non_none_types[0]
-
-            # Handle List/Array types
-            if hasattr(col_type, "__origin__") and col_type.__origin__ is list:
-                # Get the element type from List[T]
-                element_type = get_args(col_type)[0] if get_args(col_type) else str
-
-                # Check if it's a list of structs
-                if is_struct_type(element_type):
-                    # Create struct array type
-                    struct_def = self._get_struct_definition(element_type)
-                    column_defs.append(f"{col_name} STRUCT{struct_def}[]")
-                else:
-                    # Map element type to DuckDB type
-                    element_db_type = type_mapping.get(element_type, "VARCHAR")
-                    column_defs.append(f"{col_name} {element_db_type}[]")
-
-            # Handle Dict/Map types
-            elif hasattr(col_type, "__origin__") and col_type.__origin__ is dict:
-                # DuckDB has native MAP support
-                key_type = get_args(col_type)[0] if get_args(col_type) else str
-                value_type = get_args(col_type)[1] if len(get_args(col_type)) > 1 else str
-
-                key_db_type = type_mapping.get(key_type, "VARCHAR")
-                value_db_type = type_mapping.get(value_type, "VARCHAR")
-                column_defs.append(f"{col_name} MAP({key_db_type}, {value_db_type})")
-
-            # Handle Struct types
-            elif is_struct_type(col_type):
-                struct_def = self._get_struct_definition(col_type)
-                column_defs.append(f"{col_name} STRUCT{struct_def}")
-
-            else:
-                # Handle scalar types
-                db_type = type_mapping.get(col_type, "VARCHAR")
-                column_defs.append(f"{col_name} {db_type}")
+            # Use the recursive type resolver for all types
+            sql_type = self._get_column_sql_type(col_type)
+            column_defs.append(f"{col_name} {sql_type}")
 
         columns_sql = ",\n  ".join(column_defs)
         return f"CREATE TABLE {table_name} (\n  {columns_sql}\n)"
 
     def _get_struct_definition(self, struct_type: Type) -> str:
         """Convert struct type to DuckDB STRUCT definition."""
-        from .._types import is_struct_type
-
-        # Type mapping from Python types to DuckDB types
-        type_mapping = {
-            str: "VARCHAR",
-            int: "BIGINT",
-            float: "DOUBLE",
-            bool: "BOOLEAN",
-            date: "DATE",
-            datetime: "TIMESTAMP",
-            Decimal: "DECIMAL",
-        }
-
         type_hints = get_type_hints(struct_type)
         field_defs = []
 
         for field_name, field_type in type_hints.items():
-            # Handle Optional types (both Optional[X] and X | None)
-            if is_union_type(field_type):
-                # Extract the non-None type from Optional[T] or T | None
-                non_none_types = [arg for arg in get_args(field_type) if arg is not type(None)]
-                if non_none_types:
-                    field_type = non_none_types[0]
-
-            # Handle nested structs
-            if is_struct_type(field_type):
-                nested_struct_def = self._get_struct_definition(field_type)
-                field_defs.append(f"{field_name} STRUCT{nested_struct_def}")
-
-            # Handle List types in structs
-            elif hasattr(field_type, "__origin__") and field_type.__origin__ is list:
-                element_type = get_args(field_type)[0] if get_args(field_type) else str
-                if is_struct_type(element_type):
-                    # List of structs
-                    nested_struct_def = self._get_struct_definition(element_type)
-                    field_defs.append(f"{field_name} STRUCT{nested_struct_def}[]")
-                else:
-                    # List of scalars
-                    element_db_type = type_mapping.get(element_type, "VARCHAR")
-                    field_defs.append(f"{field_name} {element_db_type}[]")
-
-            # Handle Dict types in structs
-            elif hasattr(field_type, "__origin__") and field_type.__origin__ is dict:
-                key_type = get_args(field_type)[0] if get_args(field_type) else str
-                value_type = get_args(field_type)[1] if len(get_args(field_type)) > 1 else str
-
-                key_db_type = type_mapping.get(key_type, "VARCHAR")
-                value_db_type = type_mapping.get(value_type, "VARCHAR")
-                field_defs.append(f"{field_name} MAP({key_db_type}, {value_db_type})")
-
-            else:
-                # Handle scalar types
-                db_type = type_mapping.get(field_type, "VARCHAR")
-                field_defs.append(f"{field_name} {db_type}")
+            # Use the recursive type resolver for each field
+            field_sql_type = self._get_column_sql_type(field_type)
+            field_defs.append(f"{field_name} {field_sql_type}")
 
         return f"({', '.join(field_defs)})"
 

{sql_testing_library-0.19.0 → sql_testing_library-0.21.0}/src/sql_testing_library/_adapters/redshift.py

@@ -29,6 +29,17 @@ except ImportError:
 class RedshiftTypeConverter(BaseTypeConverter):
     """Redshift-specific type converter."""
 
+    def _parse_json_if_string(self, value: Any) -> Any:
+        """Helper to parse JSON strings from SUPER columns."""
+        if isinstance(value, str):
+            import json
+
+            try:
+                return json.loads(value)
+            except json.JSONDecodeError:
+                return value
+        return value
+
     def convert(self, value: Any, target_type: Type) -> Any:
         """Convert Redshift result value to target type."""
         # Handle None/NULL values first
@@ -41,21 +52,32 @@ class RedshiftTypeConverter(BaseTypeConverter):
                 return None
             target_type = self.get_optional_inner_type(target_type)
 
+        # Handle struct types from Redshift SUPER columns
+        from .._types import is_struct_type
+
+        if is_struct_type(target_type):
+            # Parse JSON string if needed
+            parsed_value = self._parse_json_if_string(value)
+            # Delegate to base converter (handles dict → struct)
+            return super().convert(parsed_value, target_type)
+
+        # Handle list/array types from Redshift SUPER columns
+        if hasattr(target_type, "__origin__") and target_type.__origin__ is list:
+            # Parse JSON string if needed
+            parsed_value = self._parse_json_if_string(value)
+
+            if isinstance(parsed_value, list):
+                # Recursively convert each element with proper typing
+                element_type = get_args(target_type)[0] if get_args(target_type) else str
+                return [self.convert(elem, element_type) for elem in parsed_value]
+            else:
+                return []
+
         # Handle dict/map types from Redshift SUPER columns
         if hasattr(target_type, "__origin__") and target_type.__origin__ is dict:
-            # Redshift returns SUPER types as Python dicts already
-            if isinstance(value, dict):
-                return value
-            # If it's a string (shouldn't happen with psycopg2), parse it
-            elif isinstance(value, str):
-                import json
-
-                try:
-                    return json.loads(value)
-                except json.JSONDecodeError:
-                    return {}
-            else:
-                return {}
+            # Parse JSON string if needed
+            parsed_value = self._parse_json_if_string(value)
+            return parsed_value if isinstance(parsed_value, dict) else {}
 
         # Redshift returns proper Python types in most cases, so use base converter
         return super().convert(value, target_type)

{sql_testing_library-0.19.0 → sql_testing_library-0.21.0}/src/sql_testing_library/_mock_table.py

@@ -204,3 +204,94 @@ class BaseMockTable(ABC):
         db_name = self.get_database_name().replace("-", "_").replace(".", "_")
         table_name = self.get_table_name().replace("-", "_").replace(".", "_")
         return f"{db_name}__{table_name}"
+
+
+class BigQueryMockTable(BaseMockTable):
+    """Mock table specifically for BigQuery with three-part naming support.
+
+    BigQuery uses a three-part naming scheme: project.dataset.table
+    This class makes it explicit and provides better semantics than cramming
+    project and dataset into the generic database_name field.
+
+    Two usage patterns are supported:
+
+    Usage - Class variables for table definition:
+        >>> class UsersMockTable(BigQueryMockTable):
+        ...     project_name = "my-project"
+        ...     dataset_name = "analytics"
+        ...     table_name = "users"
+    """
+
+    # Class variables that subclasses must set (mandatory)
+    project_name: str
+    dataset_name: str
+    table_name: str
+
+    def get_bigquery_project(self) -> str:
+        """Return the BigQuery project name from class variable.
+
+        Returns:
+            BigQuery project ID
+
+        Raises:
+            AttributeError: If project_name class variable not set
+        """
+        return self.project_name
+
+    def get_bigquery_dataset(self) -> str:
+        """Return the BigQuery dataset name from class variable.
+
+        Returns:
+            BigQuery dataset name
+
+        Raises:
+            AttributeError: If dataset_name class variable not set
+        """
+        return self.dataset_name
+
+    def get_bigquery_table(self) -> str:
+        """Return the BigQuery table name from class variable.
+
+        Returns:
+            BigQuery table name
+
+        Raises:
+            AttributeError: If table_name class variable not set
+        """
+        return self.table_name
+
+    def get_project_name(self) -> str:
+        """Return the BigQuery project name (alias for get_bigquery_project)."""
+        return self.get_bigquery_project()
+
+    def get_dataset_name(self) -> str:
+        """Return the BigQuery dataset name (alias for get_bigquery_dataset)."""
+        return self.get_bigquery_dataset()
+
+    def get_database_name(self) -> str:
+        """Return database name (for BigQuery, this is project.dataset).
+
+        This implements the BaseMockTable abstract method by combining
+        project and dataset to maintain backwards compatibility with
+        the two-part naming assumption in the base class.
+        """
+        return f"{self.get_bigquery_project()}.{self.get_bigquery_dataset()}"
+
+    def get_table_name(self) -> str:
+        """Return the table name (alias for get_bigquery_table)."""
+        return self.get_bigquery_table()
+
+    def get_fully_qualified_name(self) -> str:
+        """Return the three-part BigQuery table reference.
+
+        Returns:
+            Fully qualified table name in format: project.dataset.table
+
+        Example:
+            >>> table.get_fully_qualified_name()
+            'my-project.analytics.users'
+        """
+        project = self.get_bigquery_project()
+        dataset = self.get_bigquery_dataset()
+        table = self.get_bigquery_table()
+        return f"{project}.{dataset}.{table}"

{sql_testing_library-0.19.0 → sql_testing_library-0.21.0}/src/sql_testing_library/_sql_utils.py

@@ -5,7 +5,44 @@ import json
 from dataclasses import is_dataclass
 from datetime import date, datetime
 from decimal import Decimal
-from typing import Any, Dict, Type, get_type_hints
+from typing import Any, Dict, Type, get_args, get_type_hints
+
+
+def _convert_to_json_serializable(val: Any, val_type: Type) -> Any:
+    """Convert Python value to JSON-serializable format for Redshift SUPER type.
+
+    Recursively handles structs, nested lists, and Decimal types.
+    """
+    from ._types import is_struct_type
+
+    if val is None:
+        return None
+    elif is_struct_type(val_type):
+        # Convert struct to dict
+        if hasattr(val, "__dataclass_fields__"):
+            from dataclasses import asdict
+
+            return asdict(val)
+        elif hasattr(val, "model_dump"):
+            return val.model_dump()
+        elif hasattr(val, "dict"):
+            return val.dict()
+        else:
+            return val
+    elif isinstance(val, Decimal):
+        return float(val)
+    elif isinstance(val, (list, tuple)):
+        # Recursively handle nested lists
+        inner_type = (
+            get_args(val_type)[0]
+            if hasattr(val_type, "__origin__") and get_args(val_type)
+            else type(val[0])
+            if val
+            else str
+        )
+        return [_convert_to_json_serializable(item, inner_type) for item in val]
+    else:
+        return val
 
 
 # SQL type mappings for different dialects
@@ -390,17 +427,11 @@ def format_sql_value(value: Any, column_type: Type, dialect: str = "standard") -
                 formatted_elements.append(formatted_element)
             return f"ARRAY[{', '.join(formatted_elements)}]"
         elif dialect == "redshift":
-            # Redshift uses JSON-like syntax for SUPER arrays
-            # Format elements as JSON (double quotes for strings)
-            # Convert elements to JSON-serializable types
-            json_elements = []
-            for element in value:
-                if isinstance(element, Decimal):
-                    json_elements.append(float(element))
-                else:
-                    json_elements.append(element)
-
-            json_array = json.dumps(json_elements)
+            # Redshift uses SUPER type - convert to JSON-serializable then wrap in JSON_PARSE
+            json_elements = [
+                _convert_to_json_serializable(element, element_type) for element in value
+            ]
+            json_array = json.dumps(json_elements, cls=DecimalEncoder)
             return f"JSON_PARSE('{json_array}')"
         elif dialect == "snowflake":
             # Format each element in the array for Snowflake
@@ -426,6 +457,8 @@ def format_sql_value(value: Any, column_type: Type, dialect: str = "standard") -
             # If it's not a dict, return an empty map
             if dialect in ("athena", "trino"):
                 return "MAP(ARRAY[], ARRAY[])"
+            elif dialect == "duckdb":
+                return "MAP {}"
             else:
                 raise NotImplementedError(f"Map type not yet supported for dialect: {dialect}")
 
@@ -443,6 +476,14 @@ def format_sql_value(value: Any, column_type: Type, dialect: str = "standard") -
                 keys.append(format_sql_value(k, key_type, dialect))
                 values.append(format_sql_value(v, value_type, dialect))
             return f"MAP(ARRAY[{', '.join(keys)}], ARRAY[{', '.join(values)}])"
+        elif dialect == "duckdb":
+            # DuckDB uses MAP {key: value, ...} syntax
+            pairs = []
+            for k, v in value.items():
+                formatted_key = format_sql_value(k, key_type, dialect)
+                formatted_value = format_sql_value(v, value_type, dialect)
+                pairs.append(f"{formatted_key}: {formatted_value}")
+            return f"MAP {{{', '.join(pairs)}}}"
         elif dialect == "redshift":
             # Redshift uses SUPER type with JSON-like syntax for maps
             json_str = json.dumps(value, cls=DecimalEncoder)
@@ -597,6 +638,44 @@ def _format_struct_value(value: Any, struct_type: Type, dialect: str) -> str:
 
         return f"STRUCT({', '.join(field_pairs)})"
 
+    # For DuckDB
+    elif dialect == "duckdb":
+        # Handle NULL struct values
+        if value is None:
+            return "NULL"
+
+        # Get type hints for the struct
+        type_hints = get_type_hints(struct_type)
+
+        # Format non-NULL struct values as DuckDB struct literals
+        field_pairs = []
+        for field_name, field_type in type_hints.items():
+            # Get the field value
+            if is_dataclass(value):
+                field_value = getattr(value, field_name, None)
+            elif is_pydantic_model_class(type(value)):
+                field_value = getattr(value, field_name, None)
+            else:
+                # If it's a dict
+                field_value = value.get(field_name) if isinstance(value, dict) else None
+
+            # Format the field value recursively
+            formatted_value = format_sql_value(field_value, field_type, dialect)
+            field_pairs.append(f"'{field_name}': {formatted_value}")
+
+        return f"{{{', '.join(field_pairs)}}}"
+
+    # For Redshift (using SUPER type with JSON)
+    elif dialect == "redshift":
+        # Handle NULL struct values
+        if value is None:
+            return "NULL"
+
+        # Use helper function to convert struct to JSON-serializable dict
+        json_obj = _convert_to_json_serializable(value, struct_type)
+        json_str = json.dumps(json_obj, cls=DecimalEncoder)
+        return f"JSON_PARSE('{json_str}')"
+
     # For other databases, struct support would need to be implemented
     else:
         raise NotImplementedError(f"Struct type not yet supported for dialect: {dialect}")