sequel-duckdb 0.1.0

data/.kiro/specs/integration-test-database-setup/requirements.md

@@ -0,0 +1,117 @@
+# Requirements Document
+
+## Introduction
+
+This specification addresses database schema and setup issues in integration tests for the sequel-duckdb adapter. Many integration tests are failing because they attempt to perform database operations on tables that don't exist or haven't been properly set up. The test infrastructure needs proper database setup, schema management, and teardown procedures to ensure integration tests run reliably.
+
+## Requirements
+
+### Requirement 1: Automatic Test Database Setup
+
+**User Story:** As a developer running integration tests for the sequel-duckdb adapter, I want test databases to be automatically set up with required schemas, so that tests can focus on functionality rather than database preparation.
+
+#### Acceptance Criteria
+
+1. WHEN integration tests start THEN required test tables SHALL be automatically created
+2. WHEN tests need specific schemas THEN they SHALL be set up before test execution
+3. WHEN tests require sample data THEN it SHALL be inserted during setup
+4. WHEN test setup fails THEN clear error messages SHALL indicate the specific failure
+
+### Requirement 2: Test Table Schema Management
+
+**User Story:** As a developer writing integration tests for the sequel-duckdb adapter, I want standardized test table schemas that cover all data types and scenarios, so that I can test comprehensive functionality without custom setup.
+
+#### Acceptance Criteria
+
+1. WHEN tests need a users table THEN it SHALL include common columns (id, name, email, age, active, created_at)
+2. WHEN tests need type testing THEN tables SHALL include all supported DuckDB data types
+3. WHEN tests need relationship testing THEN foreign key relationships SHALL be properly set up
+4. WHEN tests need constraint testing THEN appropriate constraints SHALL be defined
+
+### Requirement 3: Test Data Fixtures
+
+**User Story:** As a developer running integration tests for the sequel-duckdb adapter, I want consistent test data fixtures, so that tests produce predictable results and can be easily debugged.
+
+#### Acceptance Criteria
+
+1. WHEN tests need sample users THEN standardized user records SHALL be available
+2. WHEN tests need relational data THEN properly linked records SHALL be provided
+3. WHEN tests need edge case data THEN fixtures SHALL include boundary values and special cases
+4. WHEN tests need large datasets THEN performance test fixtures SHALL be available
+
+### Requirement 4: Database Cleanup and Isolation
+
+**User Story:** As a developer running integration tests for the sequel-duckdb adapter, I want each test to run in isolation with a clean database state, so that tests don't interfere with each other and produce consistent results.
+
+#### Acceptance Criteria
+
+1. WHEN each test starts THEN it SHALL have a clean database state
+2. WHEN tests modify data THEN changes SHALL not affect subsequent tests
+3. WHEN tests create temporary tables THEN they SHALL be cleaned up after the test
+4. WHEN tests fail THEN database state SHALL be reset for the next test
+
+### Requirement 5: Schema Introspection Test Support
+
+**User Story:** As a developer testing schema introspection features of the sequel-duckdb adapter, I want test databases with comprehensive schema elements, so that I can verify all introspection functionality works correctly.
+
+#### Acceptance Criteria
+
+1. WHEN testing table listing THEN multiple test tables SHALL exist
+2. WHEN testing column introspection THEN tables SHALL have diverse column types and properties
+3. WHEN testing index introspection THEN various index types SHALL be present
+4. WHEN testing constraint introspection THEN different constraint types SHALL be available
+
+### Requirement 6: Transaction Testing Support
+
+**User Story:** As a developer testing transaction functionality of the sequel-duckdb adapter, I want test scenarios that properly exercise transaction behavior, so that I can verify commit, rollback, and isolation work correctly.
+
+#### Acceptance Criteria
+
+1. WHEN testing transactions THEN test data SHALL support rollback verification
+2. WHEN testing nested transactions THEN appropriate test scenarios SHALL be available
+3. WHEN testing transaction isolation THEN concurrent test scenarios SHALL be supported
+4. WHEN testing transaction errors THEN error conditions SHALL be reproducible
+
+### Requirement 7: Performance Testing Database Setup
+
+**User Story:** As a developer testing performance aspects of the sequel-duckdb adapter, I want test databases with appropriate data volumes, so that I can verify performance optimizations work correctly.
+
+#### Acceptance Criteria
+
+1. WHEN testing bulk operations THEN large datasets SHALL be available
+2. WHEN testing query performance THEN indexed and non-indexed scenarios SHALL be set up
+3. WHEN testing memory usage THEN datasets of various sizes SHALL be available
+4. WHEN testing streaming THEN large result sets SHALL be available for testing
+
+### Requirement 8: Error Condition Testing Setup
+
+**User Story:** As a developer testing error handling in the sequel-duckdb adapter, I want test scenarios that reliably reproduce error conditions, so that I can verify proper error handling and exception mapping.
+
+#### Acceptance Criteria
+
+1. WHEN testing constraint violations THEN tables with constraints SHALL be available
+2. WHEN testing connection errors THEN invalid database scenarios SHALL be reproducible
+3. WHEN testing SQL errors THEN scenarios that trigger DuckDB errors SHALL be available
+4. WHEN testing type errors THEN incompatible data scenarios SHALL be set up
+
+### Requirement 9: Test Database Configuration
+
+**User Story:** As a developer running integration tests for the sequel-duckdb adapter, I want flexible test database configuration, so that tests can run in different environments and scenarios.
+
+#### Acceptance Criteria
+
+1. WHEN tests run in CI THEN database configuration SHALL be automatically appropriate
+2. WHEN tests run locally THEN database configuration SHALL support development workflows
+3. WHEN tests need specific DuckDB settings THEN configuration SHALL be easily adjustable
+4. WHEN tests need different database sizes THEN memory limits SHALL be configurable
+
+### Requirement 10: Test Helper Integration
+
+**User Story:** As a developer writing integration tests for the sequel-duckdb adapter, I want test helpers that work seamlessly with the database setup, so that I can write tests efficiently without boilerplate code.
+
+#### Acceptance Criteria
+
+1. WHEN using test helpers THEN they SHALL work with the established database schema
+2. WHEN creating test data THEN helpers SHALL use the standardized fixtures
+3. WHEN verifying results THEN helpers SHALL understand the test database structure
+4. WHEN cleaning up THEN helpers SHALL properly reset database state

data/.kiro/specs/sequel-duckdb-adapter/design.md

@@ -0,0 +1,542 @@
+# Design Document
+
+## Overview
+
+The Sequel DuckDB adapter will be implemented as a Ruby gem that extends Sequel's database abstraction layer to support DuckDB databases. The design follows Sequel's established adapter architecture patterns, with jeremyevans/sequel as the primary reference, sequel-hexspace as secondary reference for adapter structure, and sequel_impala as tertiary reference for implementation patterns, while leveraging the official ruby-duckdb gem for low-level database connectivity.
+
+The adapter will be structured following the established Sequel extension pattern with the Sequel::DuckDB namespace, ensuring proper integration with Sequel's testing framework and development workflow.
+
+## Architecture
+
+### High-Level Architecture
+
+```mermaid
+graph TB
+    A[Sequel Application] --> B[Sequel Core]
+    B --> C[DuckDB Adapter]
+    C --> D[Ruby-DuckDB Gem]
+    D --> E[DuckDB Engine]
+
+    subgraph "Single Adapter File"
+        F[Database Class]
+        G[Dataset Class]
+        H[Helper Methods]
+    end
+
+    C --> F
+    C --> G
+    C --> H
+```
+
+### File Structure
+
+Following the sequel-hexspace structure pattern:
+
+```text
+lib/
+└── sequel/
+    ├── duckdb.rb                    # Main module entry point
+    ├── duckdb/
+    │   └── version.rb               # Version constant
+    └── adapters/
+        ├── duckdb.rb                # Main adapter file (Database & Dataset classes)
+        └── shared/
+            └── duckdb.rb            # Shared DuckDB-specific functionality
+```
+
+This structure ensures:
+
+- **sequel-hexspace Compatibility**: Follows the exact same structure as sequel-hexspace
+- **Proper Adapter Registration**: Uses Sequel's standard adapter loading mechanism
+- **Shared Functionality**: Common methods separated into shared/duckdb.rb
+- **Mock Database Support**: SQL generation works with Sequel's mock database objects
+- **Maintenance Organization**: Clear separation between main adapter and shared utilities
+
+## Components and Interfaces
+
+### Main Adapter Structure
+
+Following sequel-hexspace pattern:
+
+```ruby
+# lib/sequel/adapters/duckdb.rb
+require 'sequel/adapters/shared/duckdb'
+
+module Sequel
+  module DuckDB
+    class Database < Sequel::Database
+      include Sequel::DuckDB::DatabaseMethods
+      set_adapter_scheme :duckdb
+
+      # Core database methods
+    end
+
+    class Dataset < Sequel::Dataset
+      include Sequel::DuckDB::DatasetMethods
+
+      # SQL generation and query execution
+    end
+  end
+
+  # Register the adapter
+  Database.adapter_scheme :duckdb, DuckDB::Database
+end
+```
+
+```ruby
+# lib/sequel/adapters/shared/duckdb.rb
+require 'duckdb'
+
+module Sequel
+  module DuckDB
+    module DatabaseMethods
+      # Shared database functionality
+    end
+
+    module DatasetMethods
+      # Shared dataset functionality
+    end
+  end
+end
+```
+
+### 1. Database Class
+
+**Purpose:** Main database class that handles connections, transactions, and schema operations, following sequel-hexspace structure.
+
+**Key Methods for Mock Database Compatibility:**
+```ruby
+# lib/sequel/adapters/duckdb.rb
+class Sequel::DuckDB::Database < Sequel::Database
+  include Sequel::DuckDB::DatabaseMethods
+  set_adapter_scheme :duckdb
+
+  # Dataset factory
+  def dataset_class_default
+    Dataset
+  end
+end
+
+# lib/sequel/adapters/shared/duckdb.rb
+module Sequel::DuckDB::DatabaseMethods
+  # Connection management
+  def connect(server)
+    opts = server_opts(server)
+    if opts[:database] == ':memory:'
+      ::DuckDB::Database.new
+    else
+      ::DuckDB::Database.new(opts[:database])
+    end
+  end
+
+  def disconnect_connection(conn)
+    conn.close if conn && !conn.closed?
+  end
+
+  def valid_connection?(conn)
+    conn && !conn.closed?
+  end
+
+  # Schema introspection (works with mock databases)
+  def tables(opts = OPTS)
+    schema_parse_tables(opts)
+  end
+
+  def schema(table, opts = OPTS)
+    schema_parse_table(table, opts)
+  end
+
+  def indexes(table, opts = OPTS)
+    schema_parse_indexes(table, opts)
+  end
+
+  # SQL execution
+  def execute(sql, opts = OPTS, &block)
+    synchronize(opts[:server]) do |conn|
+      return execute_statement(conn, sql, opts, &block)
+    end
+  end
+
+  def execute_insert(sql, opts = OPTS)
+    execute(sql, opts)
+  end
+
+  def execute_update(sql, opts = OPTS)
+    execute(sql, opts)
+  end
+
+  private
+
+  # Schema parsing methods (mockable)
+  def schema_parse_tables(opts)
+    # Query DuckDB system tables or return mock data
+  end
+
+  def schema_parse_table(table_name, opts)
+    # Parse individual table schema
+  end
+
+  def schema_parse_indexes(table_name, opts)
+    # Parse table indexes
+  end
+
+  def execute_statement(conn, sql, opts, &block)
+    # Execute SQL against DuckDB connection
+  end
+end
+```
+
+### 2. Dataset Class
+
+**Purpose:** SQL generation and query execution, fully compatible with Sequel's mock database testing, following sequel-hexspace structure.
+
+**Key Methods for SQL Generation Testing:**
+```ruby
+# lib/sequel/adapters/duckdb.rb
+class Sequel::DuckDB::Dataset < Sequel::Dataset
+  include Sequel::DuckDB::DatasetMethods
+end
+
+# lib/sequel/adapters/shared/duckdb.rb
+module Sequel::DuckDB::DatasetMethods
+  # SQL generation (works with mock databases)
+  def select_sql
+    sql = @opts[:sql]
+    return sql if sql
+
+    columns_sql = select_columns_sql
+    sql = "SELECT #{columns_sql}"
+
+    if supports_select_all_and_offset? && @opts[:offset]
+      sql = select_all_sql(sql)
+    end
+
+    select_from_sql(sql)
+    select_join_sql(sql)
+    select_where_sql(sql)
+    select_group_sql(sql)
+    select_having_sql(sql)
+    select_order_sql(sql)
+    select_limit_sql(sql)
+
+    sql
+  end
+
+  def insert_sql(*values)
+    return static_sql if @opts[:sql]
+
+    columns = insert_columns
+    values = insert_values(values)
+
+    "INSERT INTO #{source_list(@opts[:from])} #{literal(columns)} VALUES #{values.map{|v| literal(v)}.join(', ')}"
+  end
+
+  def update_sql(values = OPTS)
+    return static_sql if @opts[:sql]
+
+    sql = "UPDATE #{source_list(@opts[:from])} SET "
+    sql << update_columns_sql(values)
+    select_where_sql(sql)
+    sql
+  end
+
+  def delete_sql
+    return static_sql if @opts[:sql]
+
+    sql = "DELETE FROM #{source_list(@opts[:from])}"
+    select_where_sql(sql)
+    sql
+  end
+
+  # Query execution
+  def fetch_rows(sql, &block)
+    execute(sql) do |result|
+      result.each(&block)
+    end
+  end
+
+  # DuckDB capabilities
+  def supports_window_functions?
+    true
+  end
+
+  def supports_cte?
+    true
+  end
+
+  def supports_returning?
+    false
+  end
+
+  def supports_select_all_and_offset?
+    true
+  end
+
+  def quote_identifiers_default
+    true
+  end
+
+  private
+
+  # DuckDB-specific SQL generation
+  def select_limit_sql(sql)
+    if limit = @opts[:limit]
+      sql << " LIMIT #{literal(limit)}"
+      if offset = @opts[:offset]
+        sql << " OFFSET #{literal(offset)}"
+      end
+    end
+  end
+
+  def literal_string_append(sql, s)
+    sql << "'" << s.gsub("'", "''") << "'"
+  end
+
+  def literal_date(date)
+    "'#{date}'"
+  end
+
+  def literal_datetime(datetime)
+    "'#{datetime.strftime('%Y-%m-%d %H:%M:%S')}'"
+  end
+
+  def literal_time(time)
+    "'#{time.strftime('%H:%M:%S')}'"
+  end
+
+  def literal_boolean(value)
+    value ? 'TRUE' : 'FALSE'
+  end
+end
+```
+
+## Data Models
+
+### Connection Configuration
+
+```ruby
+# Standard Sequel connection options
+{
+  adapter: 'duckdb',
+  database: '/path/to/database.db',  # or ':memory:' for in-memory
+  # DuckDB-specific options can be passed through
+  readonly: false,
+  config: {
+    threads: 4,
+    memory_limit: '1GB'
+  }
+}
+```
+
+### Schema Information Format
+
+Following Sequel's standard schema format:
+
+```ruby
+# Schema returned by Database#schema
+[
+  [:id, {
+    type: :integer,
+    db_type: 'INTEGER',
+    allow_null: false,
+    default: nil,
+    primary_key: true,
+    auto_increment: true
+  }],
+  [:name, {
+    type: :string,
+    db_type: 'VARCHAR',
+    allow_null: true,
+    default: nil,
+    primary_key: false,
+    auto_increment: false
+  }]
+]
+```
+
+## Error Handling
+
+### Error Mapping Strategy
+
+Map DuckDB errors to appropriate Sequel exceptions:
+
+```ruby
+# Within Database class
+private
+
+def database_error_classes
+  [::DuckDB::Error]
+end
+
+def database_exception_sqlstate(exception, opts)
+  # Extract SQL state from DuckDB error if available
+end
+
+def database_exception_use_sqlstates?
+  true
+end
+```
+
+## Testing Strategy
+
+### Mock Database Support
+
+The single-file structure ensures full compatibility with Sequel's mock database testing:
+
+```ruby
+# Test example
+DB = Sequel.mock(host: 'duckdb')
+DB.extend_datasets(Sequel::DuckDB::Dataset)
+
+# SQL generation tests work without real database
+dataset = DB[:users].where(name: 'John')
+dataset.sql.should == "SELECT * FROM users WHERE (name = 'John')"
+```
+
+## Testing Strategy (CRITICAL COMPONENT)
+
+### Test-Driven Development Approach
+
+**MANDATORY**: All code implementation must follow strict Test-Driven Development (TDD):
+
+1. **Write Tests First**: Before implementing any functionality, comprehensive tests must be written
+2. **Red-Green-Refactor**: Follow the TDD cycle of failing tests, minimal implementation, then refactoring
+3. **100% Coverage**: All implemented functionality must have corresponding test coverage
+4. **Mock and Integration**: Use both mock database tests and real DuckDB integration tests
+
+### Test Structure
+
+Following sequel-hexspace test organization exactly:
+
+```
+test/
+├── all.rb                       # Test runner - loads all test files
+├── spec_helper.rb               # Test configuration, setup, and shared utilities
+├── database_test.rb             # Database connection, transactions, and basic functionality
+├── dataset_test.rb              # Comprehensive SQL generation and query execution tests
+├── schema_test.rb               # Schema operations, introspection, and DDL tests
+├── prepared_statement_test.rb   # Prepared statement functionality and parameter binding
+├── sql_test.rb                  # SQL generation syntax and correctness tests
+└── type_test.rb                 # Data type handling, conversion, and mapping tests
+```
+
+### Test Categories and Requirements
+
+1. **SQL Generation Tests (Unit Tests)**:
+   - Use Sequel's mock database functionality
+   - Test every SQL generation method
+   - Verify correct SQL syntax and structure
+   - Test edge cases and parameter handling
+   - Must be fast and not require database connections
+
+2. **Integration Tests**:
+   - Use real DuckDB in-memory databases
+   - Test actual database operations
+   - Verify data persistence and retrieval
+   - Test connection management and error handling
+   - Test transaction behavior
+
+3. **Schema Tests**:
+   - Test table creation, modification, and deletion
+   - Test index operations
+   - Test schema introspection accuracy
+   - Test constraint handling
+   - Test various DuckDB-specific schema features
+
+4. **Type Conversion Tests**:
+   - Test Ruby ↔ DuckDB type mapping for all supported types
+   - Test edge cases and null handling
+   - Test precision and scale for numeric types
+   - Test date/time handling and timezone considerations
+   - Test binary data and text encoding
+
+5. **Error Handling Tests**:
+   - Test proper Sequel exception mapping
+   - Test connection failure scenarios
+   - Test SQL syntax error handling
+   - Test constraint violation handling
+   - Test timeout and resource limit scenarios
+
+### Test Implementation Requirements
+
+- **Before Any Code**: Tests must be written before implementing functionality
+- **Comprehensive Coverage**: Every public method must have test coverage
+- **Edge Cases**: Tests must cover error conditions and edge cases
+- **Performance**: Tests should include basic performance validation
+- **Documentation**: Tests serve as executable documentation of expected behavior
+
+## Performance Considerations
+
+### Connection Management
+
+- Use DuckDB's connection pooling capabilities
+- Handle connection lifecycle properly
+- Support both file and in-memory databases efficiently
+
+### Query Optimization
+
+- Leverage DuckDB's columnar storage advantages
+- Support DuckDB's parallel query execution
+- Use appropriate data types for optimal performance
+
+### Memory Management
+
+- Handle large result sets efficiently
+- Support streaming results where possible
+- Use DuckDB's memory-mapped file capabilities
+
+## Security Considerations
+
+### SQL Injection Prevention
+
+- Use parameterized queries through Sequel's literal system
+- Proper identifier quoting
+- Input validation and sanitization
+
+### Connection Security
+
+- Support read-only database connections
+- Proper handling of database file permissions
+- Secure connection string parsing
+
+## Deployment and Distribution
+
+### Gem Structure
+
+Following sequel-hexspace structure:
+
+```text
+sequel-duckdb/
+├── lib/
+│   └── sequel/
+│       ├── duckdb.rb                    # Main module entry point
+│       ├── duckdb/
+│       │   └── version.rb               # Version constant
+│       └── adapters/
+│           ├── duckdb.rb                # Main adapter file (Database & Dataset classes)
+│           └── shared/
+│               └── duckdb.rb            # Shared DuckDB-specific functionality
+├── test/                                # Test suite following sequel-hexspace pattern
+├── README.md
+├── CHANGELOG.md
+├── LICENSE
+├── sequel-duckdb.gemspec
+└── Gemfile
+```
+
+### Dependencies
+
+- **sequel**: Core Sequel gem (>= 5.0)
+- **ruby-duckdb**: Official Ruby DuckDB client library
+- **ruby**: Minimum Ruby version 3.1.0
+
+### Compatibility Matrix
+
+| Component   | Version Requirements |
+| ----------- | -------------------- |
+| Ruby        | >= 3.1.0             |
+| Sequel      | >= 5.0               |
+| DuckDB      | >= 0.8.0             |
+| Ruby-DuckDB | >= 0.8.0             |
+
+This design ensures the adapter follows Sequel's established patterns while providing full DuckDB functionality and maintaining compatibility with Sequel's testing framework and mock database support.