sql-testing-library 0.13.0__tar.gz → 0.14.0__tar.gz

{sql_testing_library-0.13.0 → sql_testing_library-0.14.0}/CHANGELOG.md

@@ -5,6 +5,19 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 0.14.0 (2025-06-30)
+
+### Feat
+
+- **bigquery**: add struct support with list fields (#109)
+- **bigquery**: add struct support for big query (#108)
+- add parallel table cleanup for improved performance (#107)
+- add parallel table creation for physical tables mode (#106)
+
+### Fix
+
+- **athena**: handle mixed format structs with lists and maps (#111)
+
 ## 0.13.0 (2025-06-27)
 
 ### Feat

{sql_testing_library-0.13.0 → sql_testing_library-0.14.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: sql-testing-library
-Version: 0.13.0
+Version: 0.14.0
 Summary: A powerful Python framework for unit testing SQL queries across BigQuery, Snowflake, Redshift, Athena, and Trino with mock data
 License: MIT
 Keywords: sql,testing,unit-testing,mock-data,database-testing,bigquery,snowflake,redshift,athena,trino,data-engineering,etl-testing,sql-validation,query-testing
@@ -137,12 +137,12 @@ The library supports different data types across database engines. All checkmark
 | **Decimal Array** | `List[Decimal]` | ✅ | ✅ | ✅ | ✅ | ✅ |
 | **Optional Array** | `Optional[List[T]]` | ✅ | ✅ | ✅ | ✅ | ✅ |
 | **Map/Dict** | `Dict[K, V]` | ✅ | ✅ | ✅ | ✅ | ✅ |
-| **Struct/Record** | `dataclass` | ❌ | ✅ | ❌ | ✅ | ❌ |
+| **Struct/Record** | `dataclass` | ✅ | ✅ | ❌ | ✅ | ❌ |
 | **Nested Arrays** | `List[List[T]]` | ❌ | ❌ | ❌ | ❌ | ❌ |
 
 ### Database-Specific Notes
 
-- **BigQuery**: NULL arrays become empty arrays `[]`; uses scientific notation for large decimals; dict/map types stored as JSON strings; struct types not yet supported
+- **BigQuery**: NULL arrays become empty arrays `[]`; uses scientific notation for large decimals; dict/map types stored as JSON strings; struct types supported using `STRUCT` syntax with named fields (dataclasses and Pydantic models)
 - **Athena**: 256KB query size limit; supports arrays and maps using `ARRAY[]` and `MAP(ARRAY[], ARRAY[])` syntax; supports struct types using `ROW` with named fields (dataclasses and Pydantic models)
 - **Redshift**: Arrays and maps implemented via SUPER type (JSON parsing); 16MB query size limit; struct types not yet supported
 - **Trino**: Memory catalog for testing; excellent decimal precision; supports arrays, maps, and struct types using `ROW` with named fields (dataclasses and Pydantic models)
@@ -301,13 +301,85 @@ def test_physical_tables():
         query="SELECT * FROM table",
         use_physical_tables=True  # Force physical tables
     )
+
+# Physical Tables with Custom Parallel Settings
+@sql_test(
+    mock_tables=[...],
+    result_class=ResultClass,
+    use_physical_tables=True,
+    max_workers=4  # Customize parallel execution
+)
+def test_with_custom_parallelism():
+    return TestCase(query="SELECT * FROM table")
 ```
 
 **Notes:**
 - **CTE Mode**: Default mode, works with all database engines, suitable for most use cases
 - **Physical Tables**: Used automatically when CTE queries exceed database size limits or when explicitly requested
+- **Parallel Table Creation**: When using physical tables with multiple mock tables, they are created in parallel by default for better performance
 - **Snowflake**: Full support for both CTE and physical table modes
 
+### Performance Optimization: Parallel Table Operations
+
+When using `use_physical_tables=True` with multiple mock tables, the library can create and cleanup tables in parallel for better performance.
+
+#### Parallel Table Creation
+
+**Default Behavior:**
+- Parallel creation is **enabled by default** when using physical tables
+- Smart worker allocation based on table count:
+  - 1-2 tables: Same number of workers as tables
+  - 3-5 tables: 3 workers
+  - 6-10 tables: 5 workers
+  - 11+ tables: 8 workers (capped)
+
+**Customization:**
+```python
+# Disable parallel creation
+@sql_test(use_physical_tables=True, parallel_table_creation=False)
+
+# Custom worker count
+@sql_test(use_physical_tables=True, max_workers=2)
+
+# In SQLTestCase directly
+TestCase(
+    query="...",
+    use_physical_tables=True,
+    parallel_table_creation=True,  # Default
+    max_workers=4  # Custom worker limit
+)
+```
+
+#### Parallel Table Cleanup
+
+**Default Behavior:**
+- Parallel cleanup is **enabled by default** when using physical tables
+- Uses the same smart worker allocation as table creation
+- Cleanup errors are logged as warnings (best-effort cleanup)
+
+**Customization:**
+```python
+# Disable parallel cleanup
+@sql_test(use_physical_tables=True, parallel_table_cleanup=False)
+
+# Custom worker count for both creation and cleanup
+@sql_test(use_physical_tables=True, max_workers=2)
+
+# In SQLTestCase directly
+TestCase(
+    query="...",
+    use_physical_tables=True,
+    parallel_table_creation=True,  # Default
+    parallel_table_cleanup=True,   # Default
+    max_workers=4  # Custom worker limit for both operations
+)
+```
+
+**Performance Benefits:**
+- Both table creation and cleanup operations are parallelized when multiple tables are involved
+- Significantly reduces test execution time for tests with many mock tables
+- Particularly beneficial for cloud databases where network latency is a factor
+
 ## Installation
 
 ### For End Users (pip)
@@ -572,9 +644,9 @@ def test_pattern_3():
     )
 ```
 
-### Working with Struct Types (Athena and Trino)
+### Working with Struct Types (Athena, Trino, and BigQuery)
 
-The library supports struct/record types using Python dataclasses or Pydantic models for Athena and Trino:
+The library supports struct/record types using Python dataclasses or Pydantic models for Athena, Trino, and BigQuery:
 
 ```python
 from dataclasses import dataclass
@@ -621,7 +693,7 @@ class EmployeesMockTable(BaseMockTable):
 
 # Test with struct types
 @sql_test(
-    adapter_type="athena",  # or "trino"
+    adapter_type="athena",  # or "trino" or "bigquery"
     mock_tables=[
         EmployeesMockTable([
             Employee(
@@ -667,7 +739,7 @@ def test_struct_with_dot_notation():
 
 # You can also query entire structs
 @sql_test(
-    adapter_type="trino",
+    adapter_type="trino",  # or "athena" or "bigquery"
     mock_tables=[EmployeesMockTable([...])],
     result_class=dict  # Returns full struct as dict
 )
@@ -1172,6 +1244,14 @@ For detailed usage and configuration options, see the example files included.
 
 The library has a few known limitations that are planned to be addressed in future updates:
 
+### Struct Type Support
+- **Redshift**: Struct types are not supported due to lack of native struct/record types (uses SUPER type for JSON)
+- **Snowflake**: Struct types are not supported due to lack of native struct/record types (uses VARIANT type for JSON)
+
+
+### Database-Specific Limitations
+- **BigQuery**: Does not support nested arrays (arrays of arrays). This is a BigQuery database limitation, not a library limitation. (See TODO in `test_struct_types_integration.py:test_nested_lists`)
+
 ### General Improvements
 - Add support for more SQL dialects
 - Improve error handling for malformed SQL

{sql_testing_library-0.13.0 → sql_testing_library-0.14.0}/README.md

@@ -80,12 +80,12 @@ The library supports different data types across database engines. All checkmark
 | **Decimal Array** | `List[Decimal]` | ✅ | ✅ | ✅ | ✅ | ✅ |
 | **Optional Array** | `Optional[List[T]]` | ✅ | ✅ | ✅ | ✅ | ✅ |
 | **Map/Dict** | `Dict[K, V]` | ✅ | ✅ | ✅ | ✅ | ✅ |
-| **Struct/Record** | `dataclass` | ❌ | ✅ | ❌ | ✅ | ❌ |
+| **Struct/Record** | `dataclass` | ✅ | ✅ | ❌ | ✅ | ❌ |
 | **Nested Arrays** | `List[List[T]]` | ❌ | ❌ | ❌ | ❌ | ❌ |
 
 ### Database-Specific Notes
 
-- **BigQuery**: NULL arrays become empty arrays `[]`; uses scientific notation for large decimals; dict/map types stored as JSON strings; struct types not yet supported
+- **BigQuery**: NULL arrays become empty arrays `[]`; uses scientific notation for large decimals; dict/map types stored as JSON strings; struct types supported using `STRUCT` syntax with named fields (dataclasses and Pydantic models)
 - **Athena**: 256KB query size limit; supports arrays and maps using `ARRAY[]` and `MAP(ARRAY[], ARRAY[])` syntax; supports struct types using `ROW` with named fields (dataclasses and Pydantic models)
 - **Redshift**: Arrays and maps implemented via SUPER type (JSON parsing); 16MB query size limit; struct types not yet supported
 - **Trino**: Memory catalog for testing; excellent decimal precision; supports arrays, maps, and struct types using `ROW` with named fields (dataclasses and Pydantic models)
@@ -244,13 +244,85 @@ def test_physical_tables():
         query="SELECT * FROM table",
         use_physical_tables=True  # Force physical tables
     )
+
+# Physical Tables with Custom Parallel Settings
+@sql_test(
+    mock_tables=[...],
+    result_class=ResultClass,
+    use_physical_tables=True,
+    max_workers=4  # Customize parallel execution
+)
+def test_with_custom_parallelism():
+    return TestCase(query="SELECT * FROM table")
 ```
 
 **Notes:**
 - **CTE Mode**: Default mode, works with all database engines, suitable for most use cases
 - **Physical Tables**: Used automatically when CTE queries exceed database size limits or when explicitly requested
+- **Parallel Table Creation**: When using physical tables with multiple mock tables, they are created in parallel by default for better performance
 - **Snowflake**: Full support for both CTE and physical table modes
 
+### Performance Optimization: Parallel Table Operations
+
+When using `use_physical_tables=True` with multiple mock tables, the library can create and cleanup tables in parallel for better performance.
+
+#### Parallel Table Creation
+
+**Default Behavior:**
+- Parallel creation is **enabled by default** when using physical tables
+- Smart worker allocation based on table count:
+  - 1-2 tables: Same number of workers as tables
+  - 3-5 tables: 3 workers
+  - 6-10 tables: 5 workers
+  - 11+ tables: 8 workers (capped)
+
+**Customization:**
+```python
+# Disable parallel creation
+@sql_test(use_physical_tables=True, parallel_table_creation=False)
+
+# Custom worker count
+@sql_test(use_physical_tables=True, max_workers=2)
+
+# In SQLTestCase directly
+TestCase(
+    query="...",
+    use_physical_tables=True,
+    parallel_table_creation=True,  # Default
+    max_workers=4  # Custom worker limit
+)
+```
+
+#### Parallel Table Cleanup
+
+**Default Behavior:**
+- Parallel cleanup is **enabled by default** when using physical tables
+- Uses the same smart worker allocation as table creation
+- Cleanup errors are logged as warnings (best-effort cleanup)
+
+**Customization:**
+```python
+# Disable parallel cleanup
+@sql_test(use_physical_tables=True, parallel_table_cleanup=False)
+
+# Custom worker count for both creation and cleanup
+@sql_test(use_physical_tables=True, max_workers=2)
+
+# In SQLTestCase directly
+TestCase(
+    query="...",
+    use_physical_tables=True,
+    parallel_table_creation=True,  # Default
+    parallel_table_cleanup=True,   # Default
+    max_workers=4  # Custom worker limit for both operations
+)
+```
+
+**Performance Benefits:**
+- Both table creation and cleanup operations are parallelized when multiple tables are involved
+- Significantly reduces test execution time for tests with many mock tables
+- Particularly beneficial for cloud databases where network latency is a factor
+
 ## Installation
 
 ### For End Users (pip)
@@ -515,9 +587,9 @@ def test_pattern_3():
     )
 ```
 
-### Working with Struct Types (Athena and Trino)
+### Working with Struct Types (Athena, Trino, and BigQuery)
 
-The library supports struct/record types using Python dataclasses or Pydantic models for Athena and Trino:
+The library supports struct/record types using Python dataclasses or Pydantic models for Athena, Trino, and BigQuery:
 
 ```python
 from dataclasses import dataclass
@@ -564,7 +636,7 @@ class EmployeesMockTable(BaseMockTable):
 
 # Test with struct types
 @sql_test(
-    adapter_type="athena",  # or "trino"
+    adapter_type="athena",  # or "trino" or "bigquery"
     mock_tables=[
         EmployeesMockTable([
             Employee(
@@ -610,7 +682,7 @@ def test_struct_with_dot_notation():
 
 # You can also query entire structs
 @sql_test(
-    adapter_type="trino",
+    adapter_type="trino",  # or "athena" or "bigquery"
     mock_tables=[EmployeesMockTable([...])],
     result_class=dict  # Returns full struct as dict
 )
@@ -1115,6 +1187,14 @@ For detailed usage and configuration options, see the example files included.
 
 The library has a few known limitations that are planned to be addressed in future updates:
 
+### Struct Type Support
+- **Redshift**: Struct types are not supported due to lack of native struct/record types (uses SUPER type for JSON)
+- **Snowflake**: Struct types are not supported due to lack of native struct/record types (uses VARIANT type for JSON)
+
+
+### Database-Specific Limitations
+- **BigQuery**: Does not support nested arrays (arrays of arrays). This is a BigQuery database limitation, not a library limitation. (See TODO in `test_struct_types_integration.py:test_nested_lists`)
+
 ### General Improvements
 - Add support for more SQL dialects
 - Improve error handling for malformed SQL

{sql_testing_library-0.13.0 → sql_testing_library-0.14.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "poetry.core.masonry.api"
 
 [tool.poetry]
 name = "sql-testing-library"
-version = "0.13.0"
+version = "0.14.0"
 description = "A powerful Python framework for unit testing SQL queries across BigQuery, Snowflake, Redshift, Athena, and Trino with mock data"
 authors = ["Gurmeet Saran <gurmeetx@gmail.com>", "Kushal Thakkar <kushal.thakkar@gmail.com>"]
 maintainers = ["Gurmeet Saran <gurmeetx@gmail.com>", "Kushal Thakkar <kushal.thakkar@gmail.com>"]

{sql_testing_library-0.13.0 → sql_testing_library-0.14.0}/src/sql_testing_library/_adapters/base.py

@@ -1,5 +1,7 @@
 """Base database adapter interface."""
 
+import time
+import uuid
 from abc import ABC, abstractmethod
 from typing import TYPE_CHECKING, Any, List, Optional, Tuple
 
@@ -56,3 +58,17 @@ class DatabaseAdapter(ABC):
     def get_query_size_limit(self) -> Optional[int]:
         """Return query size limit in bytes, or None if no limit."""
         return None
+
+    def get_temp_table_name(self, mock_table: BaseMockTable, prefix: str = "temp") -> str:
+        """Generate a unique temporary table name.
+
+        Args:
+            mock_table: The mock table to generate a name for
+            prefix: The prefix to use (default "temp", Snowflake uses "TEMP")
+
+        Returns:
+            A unique table name with timestamp and UUID
+        """
+        timestamp = int(time.time() * 1000)
+        unique_id = str(uuid.uuid4()).replace("-", "")[:8]
+        return f"{prefix}_{mock_table.get_table_name()}_{timestamp}_{unique_id}"