sql-testing-library 0.13.0__tar.gz → 0.15.0__tar.gz

{sql_testing_library-0.13.0 → sql_testing_library-0.15.0}/CHANGELOG.md

@@ -5,6 +5,32 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## 0.15.0 (2025-07-27)
+
+### Feat
+
+- implement duckdb integration (#117)
+- integrate mocksmith for test data generation and simplify relea… (#112)
+- integrate mocksmith for test data generation and simplify release workflow
+
+### Fix
+
+- added explicit dependency of faker
+- upgrade mocksmith library version
+
+## 0.14.0 (2025-06-30)
+
+### Feat
+
+- **bigquery**: add struct support with list fields (#109)
+- **bigquery**: add struct support for big query (#108)
+- add parallel table cleanup for improved performance (#107)
+- add parallel table creation for physical tables mode (#106)
+
+### Fix
+
+- **athena**: handle mixed format structs with lists and maps (#111)
+
 ## 0.13.0 (2025-06-27)
 
 ### Feat

{sql_testing_library-0.13.0 → sql_testing_library-0.15.0}/PKG-INFO

@@ -1,9 +1,9 @@
 Metadata-Version: 2.3
 Name: sql-testing-library
-Version: 0.13.0
-Summary: A powerful Python framework for unit testing SQL queries across BigQuery, Snowflake, Redshift, Athena, and Trino with mock data
+Version: 0.15.0
+Summary: A powerful Python framework for unit testing SQL queries across BigQuery, Snowflake, Redshift, Athena, Trino, and DuckDB with mock data
 License: MIT
-Keywords: sql,testing,unit-testing,mock-data,database-testing,bigquery,snowflake,redshift,athena,trino,data-engineering,etl-testing,sql-validation,query-testing
+Keywords: sql,testing,unit-testing,mock-data,database-testing,bigquery,snowflake,redshift,athena,trino,duckdb,data-engineering,etl-testing,sql-validation,query-testing
 Author: Gurmeet Saran
 Author-email: gurmeetx@gmail.com
 Maintainer: Gurmeet Saran
@@ -35,6 +35,7 @@ Classifier: Typing :: Typed
 Provides-Extra: all
 Provides-Extra: athena
 Provides-Extra: bigquery
+Provides-Extra: duckdb
 Provides-Extra: redshift
 Provides-Extra: snowflake
 Provides-Extra: trino
@@ -57,7 +58,7 @@ Description-Content-Type: text/markdown
 
 # SQL Testing Library
 
-A powerful Python framework for unit testing SQL queries with mock data injection across BigQuery, Snowflake, Redshift, Athena, and Trino.
+A powerful Python framework for unit testing SQL queries with mock data injection across BigQuery, Snowflake, Redshift, Athena, Trino, and DuckDB.
 
 [![Unit Tests](https://github.com/gurmeetsaran/sqltesting/actions/workflows/tests.yaml/badge.svg)](https://github.com/gurmeetsaran/sqltesting/actions/workflows/tests.yaml)
 [![Athena Integration](https://github.com/gurmeetsaran/sqltesting/actions/workflows/athena-integration.yml/badge.svg)](https://github.com/gurmeetsaran/sqltesting/actions/workflows/athena-integration.yml)
@@ -104,7 +105,7 @@ For more details on our journey and the engineering challenges we solved, read t
 
 ## Features
 
-- **Multi-Database Support**: Test SQL across BigQuery, Athena, Redshift, Trino, and Snowflake
+- **Multi-Database Support**: Test SQL across BigQuery, Athena, Redshift, Trino, Snowflake, and DuckDB
 - **Mock Data Injection**: Use Python dataclasses for type-safe test data
 - **CTE or Physical Tables**: Automatic fallback for query size limits
 - **Type-Safe Results**: Deserialize results to Pydantic models
@@ -117,45 +118,46 @@ The library supports different data types across database engines. All checkmark
 
 ### Primitive Types
 
-| Data Type | Python Type | BigQuery | Athena | Redshift | Trino | Snowflake |
-|-----------|-------------|----------|--------|----------|-------|-----------|
-| **String** | `str` | ✅ | ✅ | ✅ | ✅ | ✅ |
-| **Integer** | `int` | ✅ | ✅ | ✅ | ✅ | ✅ |
-| **Float** | `float` | ✅ | ✅ | ✅ | ✅ | ✅ |
-| **Boolean** | `bool` | ✅ | ✅ | ✅ | ✅ | ✅ |
-| **Date** | `date` | ✅ | ✅ | ✅ | ✅ | ✅ |
-| **Datetime** | `datetime` | ✅ | ✅ | ✅ | ✅ | ✅ |
-| **Decimal** | `Decimal` | ✅ | ✅ | ✅ | ✅ | ✅ |
-| **Optional** | `Optional[T]` | ✅ | ✅ | ✅ | ✅ | ✅ |
+| Data Type | Python Type | BigQuery | Athena | Redshift | Trino | Snowflake | DuckDB |
+|-----------|-------------|----------|--------|----------|-------|-----------|--------|
+| **String** | `str` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| **Integer** | `int` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| **Float** | `float` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| **Boolean** | `bool` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| **Date** | `date` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| **Datetime** | `datetime` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| **Decimal** | `Decimal` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| **Optional** | `Optional[T]` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
 
 ### Complex Types
 
-| Data Type | Python Type | BigQuery | Athena | Redshift | Trino | Snowflake |
-|-----------|-------------|----------|--------|----------|-------|-----------|
-| **String Array** | `List[str]` | ✅ | ✅ | ✅ | ✅ | ✅ |
-| **Integer Array** | `List[int]` | ✅ | ✅ | ✅ | ✅ | ✅ |
-| **Decimal Array** | `List[Decimal]` | ✅ | ✅ | ✅ | ✅ | ✅ |
-| **Optional Array** | `Optional[List[T]]` | ✅ | ✅ | ✅ | ✅ | ✅ |
-| **Map/Dict** | `Dict[K, V]` | ✅ | ✅ | ✅ | ✅ | ✅ |
-| **Struct/Record** | `dataclass` | ❌ | ✅ | ❌ | ✅ | ❌ |
-| **Nested Arrays** | `List[List[T]]` | ❌ | ❌ | ❌ | ❌ | ❌ |
+| Data Type | Python Type | BigQuery | Athena | Redshift | Trino | Snowflake | DuckDB |
+|-----------|-------------|----------|--------|----------|-------|-----------|--------|
+| **String Array** | `List[str]` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| **Integer Array** | `List[int]` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| **Decimal Array** | `List[Decimal]` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| **Optional Array** | `Optional[List[T]]` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| **Map/Dict** | `Dict[K, V]` | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| **Struct/Record** | `dataclass` | ✅ | ✅ | ❌ | ✅ | ❌ | ✅ |
+| **Nested Arrays** | `List[List[T]]` | ❌ | ❌ | ❌ | ❌ | ❌ | ❌ |
 
 ### Database-Specific Notes
 
-- **BigQuery**: NULL arrays become empty arrays `[]`; uses scientific notation for large decimals; dict/map types stored as JSON strings; struct types not yet supported
+- **BigQuery**: NULL arrays become empty arrays `[]`; uses scientific notation for large decimals; dict/map types stored as JSON strings; struct types supported using `STRUCT` syntax with named fields (dataclasses and Pydantic models)
 - **Athena**: 256KB query size limit; supports arrays and maps using `ARRAY[]` and `MAP(ARRAY[], ARRAY[])` syntax; supports struct types using `ROW` with named fields (dataclasses and Pydantic models)
 - **Redshift**: Arrays and maps implemented via SUPER type (JSON parsing); 16MB query size limit; struct types not yet supported
 - **Trino**: Memory catalog for testing; excellent decimal precision; supports arrays, maps, and struct types using `ROW` with named fields (dataclasses and Pydantic models)
 - **Snowflake**: Column names normalized to lowercase; 1MB query size limit; dict/map types implemented via VARIANT type (JSON parsing); struct types not yet supported
+- **DuckDB**: Fast embedded analytics database; excellent SQL standards compliance; supports arrays, maps, and struct types using `STRUCT` syntax with named fields (dataclasses and Pydantic models)
 
 ## Execution Modes Support
 
 The library supports two execution modes for mock data injection. **CTE Mode is the default** and is automatically used unless Physical Tables mode is explicitly requested or required due to query size limits.
 
-| Execution Mode | Description | BigQuery | Athena | Redshift | Trino | Snowflake |
-|----------------|-------------|----------|--------|----------|-------|-----------|
-| **CTE Mode** | Mock data injected as Common Table Expressions | ✅ | ✅ | ✅ | ✅ | ✅ |
-| **Physical Tables** | Mock data created as temporary tables | ✅ | ✅ | ✅ | ✅ | ✅ |
+| Execution Mode | Description | BigQuery | Athena | Redshift | Trino | Snowflake | DuckDB |
+|----------------|-------------|----------|--------|----------|-------|-----------|--------|
+| **CTE Mode** | Mock data injected as Common Table Expressions | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
+| **Physical Tables** | Mock data created as temporary tables | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ |
 
 ### Execution Mode Details
 
@@ -179,14 +181,16 @@ The library supports two execution modes for mock data injection. **CTE Mode is
 | **Redshift** | Temporary tables | Session-specific temp schema | Database automatic | Session end |
 | **Trino** | Memory tables | `memory.default` schema | Library executes `DROP TABLE` | After each test |
 | **Snowflake** | Temporary tables | Session-specific temp schema | Database automatic | Session end |
+| **DuckDB** | Temporary tables | Database-specific temp schema | Library executes `DROP TABLE` | After each test |
 
 #### **Cleanup Behavior Explained**
 
-**Library-Managed Cleanup (BigQuery, Athena, Trino):**
+**Library-Managed Cleanup (BigQuery, Athena, Trino, DuckDB):**
 - The SQL Testing Library explicitly calls cleanup methods after each test
 - **BigQuery**: Creates standard tables in your dataset, then deletes them via `client.delete_table()`
 - **Athena**: Creates external tables backed by S3 data, then drops table metadata via `DROP TABLE IF EXISTS` (⚠️ **S3 data files remain and require separate cleanup**)
 - **Trino**: Creates tables in memory catalog, then drops them via `DROP TABLE IF EXISTS`
+- **DuckDB**: Creates temporary tables in the database, then drops them via `DROP TABLE IF EXISTS`
 
 **Database-Managed Cleanup (Redshift, Snowflake):**
 - These databases have built-in temporary table mechanisms
@@ -211,7 +215,7 @@ A: Trino's memory catalog doesn't automatically clean up tables when sessions en
 A: BigQuery tables created by the library are **standard tables without TTL** - they persist until explicitly deleted. The library immediately calls `client.delete_table()` after each test. If you want to set TTL as a safety net, you can configure it at the dataset level (e.g., 24 hours) to auto-delete any orphaned tables.
 
 **Q: Which databases leave artifacts if tests crash?**
-- **BigQuery, Athena, Trino**: May leave tables if library crashes before cleanup
+- **BigQuery, Athena, Trino, DuckDB**: May leave tables if library crashes before cleanup
 - **Redshift, Snowflake**: No artifacts - temporary tables auto-cleanup on session end
 
 **Q: How to manually clean up orphaned tables?**
@@ -227,6 +231,10 @@ DROP TABLE temp_table_name;
 -- Trino: List and drop tables with temp prefix
 SHOW TABLES FROM memory.default LIKE 'temp_%';
 DROP TABLE memory.default.temp_table_name;
+
+-- DuckDB: List and drop tables with temp prefix
+SHOW TABLES;
+DROP TABLE temp_table_name;
 ```
 
 **Q: How to handle S3 cleanup for Athena tables?**
@@ -277,6 +285,7 @@ aws s3api list-objects-v2 --bucket your-athena-results-bucket --prefix "temp_" \
 | **Redshift** | 16MB | Automatically switches at 16MB |
 | **Trino** | 16MB (estimated) | Large dataset or complex CTEs |
 | **Snowflake** | 1MB | Automatically switches at 1MB |
+| **DuckDB** | 32MB (estimated) | Large dataset or complex CTEs |
 
 ### How to Control Execution Mode
 
@@ -301,13 +310,85 @@ def test_physical_tables():
         query="SELECT * FROM table",
         use_physical_tables=True  # Force physical tables
     )
+
+# Physical Tables with Custom Parallel Settings
+@sql_test(
+    mock_tables=[...],
+    result_class=ResultClass,
+    use_physical_tables=True,
+    max_workers=4  # Customize parallel execution
+)
+def test_with_custom_parallelism():
+    return TestCase(query="SELECT * FROM table")
 ```
 
 **Notes:**
 - **CTE Mode**: Default mode, works with all database engines, suitable for most use cases
 - **Physical Tables**: Used automatically when CTE queries exceed database size limits or when explicitly requested
+- **Parallel Table Creation**: When using physical tables with multiple mock tables, they are created in parallel by default for better performance
 - **Snowflake**: Full support for both CTE and physical table modes
 
+### Performance Optimization: Parallel Table Operations
+
+When using `use_physical_tables=True` with multiple mock tables, the library can create and cleanup tables in parallel for better performance.
+
+#### Parallel Table Creation
+
+**Default Behavior:**
+- Parallel creation is **enabled by default** when using physical tables
+- Smart worker allocation based on table count:
+  - 1-2 tables: Same number of workers as tables
+  - 3-5 tables: 3 workers
+  - 6-10 tables: 5 workers
+  - 11+ tables: 8 workers (capped)
+
+**Customization:**
+```python
+# Disable parallel creation
+@sql_test(use_physical_tables=True, parallel_table_creation=False)
+
+# Custom worker count
+@sql_test(use_physical_tables=True, max_workers=2)
+
+# In SQLTestCase directly
+TestCase(
+    query="...",
+    use_physical_tables=True,
+    parallel_table_creation=True,  # Default
+    max_workers=4  # Custom worker limit
+)
+```
+
+#### Parallel Table Cleanup
+
+**Default Behavior:**
+- Parallel cleanup is **enabled by default** when using physical tables
+- Uses the same smart worker allocation as table creation
+- Cleanup errors are logged as warnings (best-effort cleanup)
+
+**Customization:**
+```python
+# Disable parallel cleanup
+@sql_test(use_physical_tables=True, parallel_table_cleanup=False)
+
+# Custom worker count for both creation and cleanup
+@sql_test(use_physical_tables=True, max_workers=2)
+
+# In SQLTestCase directly
+TestCase(
+    query="...",
+    use_physical_tables=True,
+    parallel_table_creation=True,  # Default
+    parallel_table_cleanup=True,   # Default
+    max_workers=4  # Custom worker limit for both operations
+)
+```
+
+**Performance Benefits:**
+- Both table creation and cleanup operations are parallelized when multiple tables are involved
+- Significantly reduces test execution time for tests with many mock tables
+- Particularly beneficial for cloud databases where network latency is a factor
+
 ## Installation
 
 ### For End Users (pip)
@@ -327,6 +408,9 @@ pip install sql-testing-library[trino]
 # Install with Snowflake support
 pip install sql-testing-library[snowflake]
 
+# Install with DuckDB support
+pip install sql-testing-library[duckdb]
+
 # Or install with all database adapters
 pip install sql-testing-library[all]
 ```
@@ -342,9 +426,10 @@ poetry install --with athena
 poetry install --with redshift
 poetry install --with trino
 poetry install --with snowflake
+poetry install --with duckdb
 
 # Install with all database adapters and dev tools
-poetry install --with bigquery,athena,redshift,trino,snowflake,dev
+poetry install --with bigquery,athena,redshift,trino,snowflake,duckdb,dev
 ```
 
 ## Quick Start
@@ -353,7 +438,7 @@ poetry install --with bigquery,athena,redshift,trino,snowflake,dev
 
 ```ini
 [sql_testing]
-adapter = bigquery  # Use 'bigquery', 'athena', 'redshift', 'trino', or 'snowflake'
+adapter = bigquery  # Use 'bigquery', 'athena', 'redshift', 'trino', 'snowflake', or 'duckdb'
 
 # BigQuery configuration
 [sql_testing.bigquery]
@@ -411,6 +496,10 @@ credentials_path = <path to credentials json>
 #
 # # Option 2: Password authentication (for accounts without MFA)
 # password = <snowflake_password>
+
+# DuckDB configuration
+# [sql_testing.duckdb]
+# database = <path/to/database.duckdb>  # Optional: defaults to in-memory database
 ```
 
 ### Database Context Understanding
@@ -424,6 +513,7 @@ Each database adapter uses a different concept for organizing tables and queries
 | **Redshift** | `{database}` | database only | `"test_db"` | `SELECT * FROM test_db.orders` |
 | **Snowflake** | `{database}.{schema}` | database + schema | `"test_db.public"` | `SELECT * FROM test_db.public.products` |
 | **Trino** | `{catalog}.{schema}` | catalog + schema | `"memory.default"` | `SELECT * FROM memory.default.inventory` |
+| **DuckDB** | `{database}` | database only | `"test_db"` | `SELECT * FROM test_db.analytics` |
 
 #### Key Points:
 
@@ -496,6 +586,14 @@ class ProductsMockTable(BaseMockTable):
 
     def get_table_name(self) -> str:
         return "products"
+
+# DuckDB Mock Table
+class AnalyticsMockTable(BaseMockTable):
+    def get_database_name(self) -> str:
+        return "test_db"  # database only
+
+    def get_table_name(self) -> str:
+        return "analytics"
 ```
 
 2. **Write a test** using one of the flexible patterns:
@@ -572,9 +670,9 @@ def test_pattern_3():
     )
 ```
 
-### Working with Struct Types (Athena and Trino)
+### Working with Struct Types (Athena, Trino, and BigQuery)
 
-The library supports struct/record types using Python dataclasses or Pydantic models for Athena and Trino:
+The library supports struct/record types using Python dataclasses or Pydantic models for Athena, Trino, and BigQuery:
 
 ```python
 from dataclasses import dataclass
@@ -621,7 +719,7 @@ class EmployeesMockTable(BaseMockTable):
 
 # Test with struct types
 @sql_test(
-    adapter_type="athena",  # or "trino"
+    adapter_type="athena",  # or "trino", "bigquery", or "duckdb"
     mock_tables=[
         EmployeesMockTable([
             Employee(
@@ -667,7 +765,7 @@ def test_struct_with_dot_notation():
 
 # You can also query entire structs
 @sql_test(
-    adapter_type="trino",
+    adapter_type="trino",  # or "athena", "bigquery", or "duckdb"
     mock_tables=[EmployeesMockTable([...])],
     result_class=dict  # Returns full struct as dict
 )
@@ -915,9 +1013,21 @@ def test_snowflake_query():
         query="SELECT user_id, name FROM users WHERE user_id = 1",
         default_namespace="test_db"
     )
+
+# Use DuckDB adapter for this test
+@sql_test(
+    adapter_type="duckdb",
+    mock_tables=[...],
+    result_class=UserResult
+)
+def test_duckdb_query():
+    return TestCase(
+        query="SELECT user_id, name FROM users WHERE user_id = 1",
+        default_namespace="test_db"
+    )
 ```
 
-The adapter_type parameter will use the configuration from the corresponding section in pytest.ini, such as `[sql_testing.bigquery]`, `[sql_testing.athena]`, `[sql_testing.redshift]`, `[sql_testing.trino]`, or `[sql_testing.snowflake]`.
+The adapter_type parameter will use the configuration from the corresponding section in pytest.ini, such as `[sql_testing.bigquery]`, `[sql_testing.athena]`, `[sql_testing.redshift]`, `[sql_testing.trino]`, `[sql_testing.snowflake]`, or `[sql_testing.duckdb]`.
 
 **Default Adapter Behavior:**
 - If `adapter_type` is not specified in the test, the library uses the adapter from `[sql_testing]` section's `adapter` setting
@@ -960,6 +1070,14 @@ The adapter_type parameter will use the configuration from the corresponding sec
 - Supports authentication via username and password
 - Optional support for warehouse, role, and schema specification
 
+#### DuckDB Adapter
+- Supports DuckDB embedded analytical database
+- Uses CTAS (CREATE TABLE AS SELECT) for efficient temporary table creation
+- Fast local database with excellent SQL standards compliance
+- Supports both file-based and in-memory databases
+- No authentication required - perfect for local development and testing
+- Excellent performance for analytical workloads
+
 **Default Behavior:**
 - If adapter_type is not specified in the TestCase or decorator, the library will use the adapter specified in the `[sql_testing]` section's `adapter` setting.
 - If no adapter is specified in the `[sql_testing]` section, it defaults to "bigquery".
@@ -1168,10 +1286,53 @@ The library automatically:
 
 For detailed usage and configuration options, see the example files included.
 
+## Integration with Mocksmith
+
+SQL Testing Library works seamlessly with [Mocksmith](https://github.com/gurmeetsaran/mocksmith) for automatic test data generation. Mocksmith can reduce your test setup code by ~70% while providing more realistic test data.
+
+Install mocksmith with: `pip install mocksmith[mock,pydantic]`
+
+### Quick Example
+
+```python
+# Without Mocksmith - Manual data creation
+customers = []
+for i in range(100):
+    customers.append(Customer(
+        id=i + 1,
+        name=f"Customer {i + 1}",
+        email=f"customer{i + 1}@test.com",
+        balance=Decimal(str(random.uniform(0, 10000)))
+    ))
+
+# With Mocksmith - Automatic realistic data
+from mocksmith import mockable, Varchar, Integer, Money
+
+@mockable
+@dataclass
+class Customer:
+    id: Integer()
+    name: Varchar(100)
+    email: Varchar(255)
+    balance: Money()
+
+customers = [Customer.mock() for _ in range(100)]
+```
+
+See the [Mocksmith Integration Guide](docs/mocksmith_integration.md) and [examples](examples/mocksmith_integration_example.py) for detailed usage patterns.
+
 ## Known Limitations and TODOs
 
 The library has a few known limitations that are planned to be addressed in future updates:
 
+### Struct Type Support
+- **Redshift**: Struct types are not supported due to lack of native struct/record types (uses SUPER type for JSON)
+- **Snowflake**: Struct types are not supported due to lack of native struct/record types (uses VARIANT type for JSON)
+
+
+### Database-Specific Limitations
+- **BigQuery**: Does not support nested arrays (arrays of arrays). This is a BigQuery database limitation, not a library limitation. (See TODO in `test_struct_types_integration.py:test_nested_lists`)
+
 ### General Improvements
 - Add support for more SQL dialects
 - Improve error handling for malformed SQL
@@ -1188,4 +1349,5 @@ The library has a few known limitations that are planned to be addressed in futu
   - psycopg2-binary for Redshift
   - trino for Trino
   - snowflake-connector-python for Snowflake
+  - duckdb for DuckDB