sequel-duckdb 0.1.0 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.beads/.beads-credential-key +1 -0
- data/.beads/.gitignore +66 -0
- data/.beads/README.md +85 -0
- data/.beads/config.yaml +56 -0
- data/.beads/hooks/post-checkout +24 -0
- data/.beads/hooks/post-merge +24 -0
- data/.beads/hooks/pre-commit +24 -0
- data/.beads/hooks/pre-push +24 -0
- data/.beads/hooks/prepare-commit-msg +24 -0
- data/.beads/metadata.json +7 -0
- data/.kiro/specs/advanced-sql-features-implementation/design.md +3 -1
- data/.kiro/specs/advanced-sql-features-implementation/requirements.md +1 -1
- data/.kiro/specs/advanced-sql-features-implementation/tasks.md +5 -1
- data/.kiro/specs/duckdb-sql-syntax-compatibility/design.md +15 -1
- data/.kiro/specs/duckdb-sql-syntax-compatibility/requirements.md +1 -1
- data/.kiro/specs/duckdb-sql-syntax-compatibility/tasks.md +13 -0
- data/.kiro/specs/edge-cases-and-validation-fixes/requirements.md +1 -1
- data/.kiro/specs/integration-test-database-setup/requirements.md +1 -1
- data/.kiro/specs/sequel-duckdb-adapter/design.md +8 -1
- data/.kiro/specs/sequel-duckdb-adapter/requirements.md +10 -10
- data/.kiro/specs/sequel-duckdb-adapter/tasks.md +48 -3
- data/.kiro/specs/sql-expression-handling-fix/design.md +34 -1
- data/.kiro/specs/sql-expression-handling-fix/requirements.md +1 -1
- data/.kiro/specs/sql-expression-handling-fix/tasks.md +3 -0
- data/.kiro/specs/test-infrastructure-improvements/requirements.md +1 -1
- data/.kiro/steering/product.md +5 -1
- data/.kiro/steering/structure.md +1 -1
- data/.kiro/steering/tech.md +14 -1
- data/.kiro/steering/testing.md +22 -1
- data/.mdformat.toml +2 -0
- data/.release-please-manifest.json +3 -0
- data/.rubocop.yml +116 -58
- data/.rubocop_todo.yml +323 -0
- data/AGENTS.md +154 -0
- data/API_DOCUMENTATION.md +73 -49
- data/CHANGELOG.md +46 -10
- data/FINAL_STATUS.md +99 -0
- data/LICENSE +1 -1
- data/MIGRATION_EXAMPLES.md +1 -1
- data/PERFORMANCE_OPTIMIZATIONS.md +4 -1
- data/README.md +90 -1
- data/REFACTORING_SUMMARY.md +264 -0
- data/Rakefile +21 -5
- data/TASK_10.2_IMPLEMENTATION_SUMMARY.md +19 -1
- data/docs/DUCKDB_SQL_PATTERNS.md +39 -1
- data/docs/TASK_12_VERIFICATION_SUMMARY.md +14 -1
- data/justfile +52 -0
- data/lib/sequel/adapters/duckdb.rb +137 -108
- data/lib/sequel/adapters/shared/duckdb.rb +292 -1490
- data/lib/sequel/duckdb/helpers/copier.rb +50 -0
- data/lib/sequel/duckdb/helpers/pathifier.rb +141 -0
- data/lib/sequel/duckdb/version.rb +5 -2
- data/plans/date_arithmetic.md +420 -0
- data/plans/engineering/Sequel.md +471 -0
- data/plans/engineering/duckdb.md +712 -0
- data/plans/engineering/sqlite.md +453 -0
- data/plans/mock_connection_bug.md +333 -0
- data/plans/mock_without_driver_gem.md +371 -0
- data/plans/over_engineering_analysis.md +122 -0
- data/plans/schema_management.md +383 -0
- data/release-please-config.json +14 -0
- metadata +49 -27
|
@@ -105,6 +105,7 @@ end
|
|
|
105
105
|
**Purpose:** Main database class that handles connections, transactions, and schema operations, following sequel-hexspace structure.
|
|
106
106
|
|
|
107
107
|
**Key Methods for Mock Database Compatibility:**
|
|
108
|
+
|
|
108
109
|
```ruby
|
|
109
110
|
# lib/sequel/adapters/duckdb.rb
|
|
110
111
|
class Sequel::DuckDB::Database < Sequel::Database
|
|
@@ -191,6 +192,7 @@ end
|
|
|
191
192
|
**Purpose:** SQL generation and query execution, fully compatible with Sequel's mock database testing, following sequel-hexspace structure.
|
|
192
193
|
|
|
193
194
|
**Key Methods for SQL Generation Testing:**
|
|
195
|
+
|
|
194
196
|
```ruby
|
|
195
197
|
# lib/sequel/adapters/duckdb.rb
|
|
196
198
|
class Sequel::DuckDB::Dataset < Sequel::Dataset
|
|
@@ -423,6 +425,7 @@ test/
|
|
|
423
425
|
### Test Categories and Requirements
|
|
424
426
|
|
|
425
427
|
1. **SQL Generation Tests (Unit Tests)**:
|
|
428
|
+
|
|
426
429
|
- Use Sequel's mock database functionality
|
|
427
430
|
- Test every SQL generation method
|
|
428
431
|
- Verify correct SQL syntax and structure
|
|
@@ -430,6 +433,7 @@ test/
|
|
|
430
433
|
- Must be fast and not require database connections
|
|
431
434
|
|
|
432
435
|
2. **Integration Tests**:
|
|
436
|
+
|
|
433
437
|
- Use real DuckDB in-memory databases
|
|
434
438
|
- Test actual database operations
|
|
435
439
|
- Verify data persistence and retrieval
|
|
@@ -437,6 +441,7 @@ test/
|
|
|
437
441
|
- Test transaction behavior
|
|
438
442
|
|
|
439
443
|
3. **Schema Tests**:
|
|
444
|
+
|
|
440
445
|
- Test table creation, modification, and deletion
|
|
441
446
|
- Test index operations
|
|
442
447
|
- Test schema introspection accuracy
|
|
@@ -444,6 +449,7 @@ test/
|
|
|
444
449
|
- Test various DuckDB-specific schema features
|
|
445
450
|
|
|
446
451
|
4. **Type Conversion Tests**:
|
|
452
|
+
|
|
447
453
|
- Test Ruby ↔ DuckDB type mapping for all supported types
|
|
448
454
|
- Test edge cases and null handling
|
|
449
455
|
- Test precision and scale for numeric types
|
|
@@ -451,6 +457,7 @@ test/
|
|
|
451
457
|
- Test binary data and text encoding
|
|
452
458
|
|
|
453
459
|
5. **Error Handling Tests**:
|
|
460
|
+
|
|
454
461
|
- Test proper Sequel exception mapping
|
|
455
462
|
- Test connection failure scenarios
|
|
456
463
|
- Test SQL syntax error handling
|
|
@@ -539,4 +546,4 @@ sequel-duckdb/
|
|
|
539
546
|
| DuckDB | >= 0.8.0 |
|
|
540
547
|
| Ruby-DuckDB | >= 0.8.0 |
|
|
541
548
|
|
|
542
|
-
This design ensures the adapter follows Sequel's established patterns while providing full DuckDB functionality and maintaining compatibility with Sequel's testing framework and mock database support.
|
|
549
|
+
This design ensures the adapter follows Sequel's established patterns while providing full DuckDB functionality and maintaining compatibility with Sequel's testing framework and mock database support.
|
|
@@ -158,15 +158,15 @@ This document outlines the requirements for building a complete Ruby Sequel data
|
|
|
158
158
|
|
|
159
159
|
#### Acceptance Criteria
|
|
160
160
|
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
161
|
+
01. WHEN any code is implemented THEN the system SHALL have tests written BEFORE the implementation (Test-Driven Development)
|
|
162
|
+
02. WHEN SQL generation methods are implemented THEN the system SHALL have unit tests verifying correct SQL output following sequel-hexspace test structure
|
|
163
|
+
03. WHEN database operations are implemented THEN the system SHALL have integration tests using actual DuckDB databases
|
|
164
|
+
04. WHEN data type conversions are implemented THEN the system SHALL have tests covering all supported type mappings
|
|
165
|
+
05. WHEN schema introspection is implemented THEN the system SHALL have tests verifying correct metadata retrieval
|
|
166
|
+
06. WHEN transaction handling is implemented THEN the system SHALL have tests covering commit, rollback, and error scenarios
|
|
167
|
+
07. WHEN error conditions occur THEN the system SHALL have tests verifying proper exception handling
|
|
168
|
+
08. WHEN performance-critical operations are implemented THEN the system SHALL have benchmarks to prevent regressions
|
|
169
|
+
09. WHEN test structure is organized THEN the system SHALL mirror sequel-hexspace test organization with test/all.rb, test/spec_helper.rb, test/database_test.rb, test/dataset_test.rb, test/schema_test.rb, test/prepared_statement_test.rb, test/sql_test.rb, and test/type_test.rb
|
|
170
170
|
10. WHEN mock database testing is needed THEN the system SHALL use Sequel's mock database functionality for SQL generation testing
|
|
171
171
|
11. WHEN integration testing is needed THEN the system SHALL use actual DuckDB in-memory databases for connection and operation testing
|
|
172
172
|
12. WHEN test coverage is measured THEN the system SHALL achieve 100% coverage of all implemented functionality
|
|
@@ -199,4 +199,4 @@ This document outlines the requirements for building a complete Ruby Sequel data
|
|
|
199
199
|
5. WHEN the gem is packaged THEN the system SHALL follow Ruby gem conventions with standard structure (Gemfile, Rakefile, .rubocop.yml)
|
|
200
200
|
6. WHEN dependencies are specified THEN the system SHALL use ruby-duckdb gem exclusively for database connections with appropriate version constraints
|
|
201
201
|
7. WHEN Ruby versions are supported THEN the system SHALL maintain compatibility with Ruby 3.1+ as specified in the technology requirements
|
|
202
|
-
8. WHEN file structure is organized THEN the system SHALL place adapter logic in lib/sequel/adapters/duckdb.rb and shared functionality in lib/sequel/adapters/shared/duckdb.rb following sequel-hexspace conventions exactly
|
|
202
|
+
8. WHEN file structure is organized THEN the system SHALL place adapter logic in lib/sequel/adapters/duckdb.rb and shared functionality in lib/sequel/adapters/shared/duckdb.rb following sequel-hexspace conventions exactly
|
|
@@ -1,19 +1,23 @@
|
|
|
1
1
|
# Implementation Plan
|
|
2
2
|
|
|
3
3
|
- [x] 1. Set up project structure following sequel-hexspace pattern
|
|
4
|
+
|
|
4
5
|
- Create lib/sequel/adapters/duckdb.rb for main Database and Dataset classes
|
|
5
6
|
- Create lib/sequel/adapters/shared/duckdb.rb for DatabaseMethods and DatasetMethods modules
|
|
6
7
|
- Set up proper require structure and module organization
|
|
7
8
|
- _Requirements: 10.3, 13.8_
|
|
8
9
|
|
|
9
10
|
- [x] 2. Implement basic connection management in shared module
|
|
11
|
+
|
|
10
12
|
- [x] 2.1 Create DatabaseMethods module with connection handling
|
|
13
|
+
|
|
11
14
|
- Implement connect method for file and in-memory databases
|
|
12
15
|
- Implement disconnect_connection and valid_connection? methods
|
|
13
16
|
- Add proper error handling for connection failures
|
|
14
17
|
- _Requirements: 1.1, 1.2, 1.4, 1.6_
|
|
15
18
|
|
|
16
19
|
- [x] 2.2 Create Database class with adapter registration
|
|
20
|
+
|
|
17
21
|
- Define Sequel::DuckDB::Database class including DatabaseMethods
|
|
18
22
|
- Set adapter scheme to :duckdb
|
|
19
23
|
- Implement dataset_class_default method
|
|
@@ -21,13 +25,16 @@
|
|
|
21
25
|
- _Requirements: 1.1, 10.2, 13.1_
|
|
22
26
|
|
|
23
27
|
- [x] 2.3 Fix adapter registration (CRITICAL BUG)
|
|
28
|
+
|
|
24
29
|
- Fix incorrect adapter registration in lib/sequel/adapters/duckdb.rb
|
|
25
30
|
- Change from `Sequel::Database.set_shared_adapter_scheme :duckdb, self` to proper registration
|
|
26
31
|
- Use `Database.adapter_scheme :duckdb, DuckDB::Database` pattern
|
|
27
32
|
- _Requirements: 1.1, 13.1_
|
|
28
33
|
|
|
29
34
|
- [x] 3. Set up comprehensive test infrastructure (CRITICAL - TDD REQUIREMENT)
|
|
35
|
+
|
|
30
36
|
- [x] 3.1 Create test infrastructure following sequel-hexspace pattern
|
|
37
|
+
|
|
31
38
|
- Create test/all.rb test runner
|
|
32
39
|
- Create test/spec_helper.rb with test configuration and DuckDB setup
|
|
33
40
|
- Set up test database helpers and utilities for both mock and real DuckDB testing
|
|
@@ -35,6 +42,7 @@
|
|
|
35
42
|
- _Requirements: 11.8, 10.4, 13.1_
|
|
36
43
|
|
|
37
44
|
- [x] 3.2 Create core test files with initial structure
|
|
45
|
+
|
|
38
46
|
- Create test/database_test.rb for connection and basic functionality tests
|
|
39
47
|
- Create test/dataset_test.rb for comprehensive SQL generation testing
|
|
40
48
|
- Create test/schema_test.rb for schema operations and introspection
|
|
@@ -43,7 +51,9 @@
|
|
|
43
51
|
- _Requirements: 11.1, 11.2, 11.4, 11.3_
|
|
44
52
|
|
|
45
53
|
- [x] 4. Implement basic SQL generation in shared module (TDD - TESTS FIRST)
|
|
54
|
+
|
|
46
55
|
- [x] 4.1 Write tests for core SQL generation methods
|
|
56
|
+
|
|
47
57
|
- Write comprehensive tests for select_sql method using mock database
|
|
48
58
|
- Write tests for insert_sql method with various parameter combinations
|
|
49
59
|
- Write tests for update_sql method with WHERE clauses
|
|
@@ -52,6 +62,7 @@
|
|
|
52
62
|
- _Requirements: 2.1, 2.2, 2.3, 2.4, 11.1, 11.2_
|
|
53
63
|
|
|
54
64
|
- [x] 4.2 Implement DatasetMethods module with core SQL generation
|
|
65
|
+
|
|
55
66
|
- Implement select_sql method for basic SELECT statements
|
|
56
67
|
- Implement insert_sql method for INSERT operations
|
|
57
68
|
- Implement update_sql method for UPDATE operations
|
|
@@ -60,19 +71,23 @@
|
|
|
60
71
|
- _Requirements: 2.1, 2.2, 2.3, 2.4_
|
|
61
72
|
|
|
62
73
|
- [x] 4.3 Write tests for Dataset class functionality
|
|
74
|
+
|
|
63
75
|
- Write tests for fetch_rows method using real DuckDB in-memory database
|
|
64
76
|
- Write tests for DuckDB capability flags (window functions, CTE support, etc.)
|
|
65
77
|
- Write integration tests for basic query execution
|
|
66
78
|
- _Requirements: 2.1, 6.1, 6.2, 11.3_
|
|
67
79
|
|
|
68
80
|
- [x] 4.4 Implement Dataset class with shared functionality
|
|
81
|
+
|
|
69
82
|
- Implement fetch_rows method for query execution
|
|
70
83
|
- Add DuckDB capability flags (window functions, CTE support, etc.)
|
|
71
84
|
- Ensure proper integration with DatabaseMethods
|
|
72
85
|
- _Requirements: 2.1, 6.1, 6.2_
|
|
73
86
|
|
|
74
87
|
- [x] 5. Implement data type handling and literal conversion (TDD - TESTS FIRST)
|
|
88
|
+
|
|
75
89
|
- [x] 5.1 Write tests for literal conversion methods
|
|
90
|
+
|
|
76
91
|
- Write tests for literal_string_append with string escaping scenarios
|
|
77
92
|
- Write tests for literal_date, literal_datetime, literal_time methods
|
|
78
93
|
- Write tests for literal_boolean method with true/false values
|
|
@@ -80,6 +95,7 @@
|
|
|
80
95
|
- _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.8, 11.4_
|
|
81
96
|
|
|
82
97
|
- [x] 5.2 Add literal conversion methods to DatasetMethods
|
|
98
|
+
|
|
83
99
|
- Implement literal_string_append for string escaping
|
|
84
100
|
- Implement literal_date, literal_datetime, literal_time methods
|
|
85
101
|
- Implement literal_boolean method for boolean values
|
|
@@ -87,19 +103,23 @@
|
|
|
87
103
|
- _Requirements: 3.1, 3.2, 3.3, 3.4, 3.5, 3.6, 3.8_
|
|
88
104
|
|
|
89
105
|
- [x] 5.3 Write tests for binary data and numeric type support
|
|
106
|
+
|
|
90
107
|
- Write tests for BLOB type mapping for binary data
|
|
91
108
|
- Write tests for integer and float type handling
|
|
92
109
|
- Write tests for Ruby to DuckDB type conversion edge cases
|
|
93
110
|
- _Requirements: 3.2, 3.3, 3.9, 11.4_
|
|
94
111
|
|
|
95
112
|
- [x] 5.4 Add binary data and numeric type support
|
|
113
|
+
|
|
96
114
|
- Implement BLOB type mapping for binary data
|
|
97
115
|
- Add proper integer and float type handling
|
|
98
116
|
- Ensure proper Ruby to DuckDB type conversion
|
|
99
117
|
- _Requirements: 3.2, 3.3, 3.9_
|
|
100
118
|
|
|
101
119
|
- [-] 6. Implement schema introspection in DatabaseMethods (TDD - TESTS FIRST)
|
|
120
|
+
|
|
102
121
|
- [x] 6.1 Write tests for schema introspection methods
|
|
122
|
+
|
|
103
123
|
- Write tests for schema_parse_tables method for table listing
|
|
104
124
|
- Write tests for schema_parse_table method for column information
|
|
105
125
|
- Write tests for schema_parse_indexes method for index introspection
|
|
@@ -107,6 +127,7 @@
|
|
|
107
127
|
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6, 11.5_
|
|
108
128
|
|
|
109
129
|
- [x] 6.2 Add table and schema discovery methods
|
|
130
|
+
|
|
110
131
|
- Implement schema_parse_tables method for table listing
|
|
111
132
|
- Implement schema_parse_table method for column information
|
|
112
133
|
- Implement schema_parse_indexes method for index introspection
|
|
@@ -114,6 +135,7 @@
|
|
|
114
135
|
- _Requirements: 4.1, 4.2, 4.3, 4.4, 4.5, 4.6_
|
|
115
136
|
|
|
116
137
|
- [x] 6.3 Write tests for schema metadata methods
|
|
138
|
+
|
|
117
139
|
- Write tests for tables method using schema_parse_tables
|
|
118
140
|
- Write tests for schema method using schema_parse_table
|
|
119
141
|
- Write tests for indexes method using schema_parse_indexes
|
|
@@ -121,6 +143,7 @@
|
|
|
121
143
|
- _Requirements: 4.7, 4.8, 11.5_
|
|
122
144
|
|
|
123
145
|
- [x] 6.4 Add schema metadata methods
|
|
146
|
+
|
|
124
147
|
- Implement tables method using schema_parse_tables
|
|
125
148
|
- Implement schema method using schema_parse_table
|
|
126
149
|
- Implement indexes method using schema_parse_indexes
|
|
@@ -128,7 +151,9 @@
|
|
|
128
151
|
- _Requirements: 4.7, 4.8_
|
|
129
152
|
|
|
130
153
|
- [-] 7. Implement transaction support in DatabaseMethods
|
|
154
|
+
|
|
131
155
|
- [x] 7.1 Add basic transaction methods
|
|
156
|
+
|
|
132
157
|
- Implement transaction block handling
|
|
133
158
|
- Add automatic commit on successful completion
|
|
134
159
|
- Add automatic rollback on exceptions
|
|
@@ -136,13 +161,16 @@
|
|
|
136
161
|
- _Requirements: 5.1, 5.2, 5.3, 5.4_
|
|
137
162
|
|
|
138
163
|
- [x] 6.2 Add advanced transaction features
|
|
164
|
+
|
|
139
165
|
- Implement savepoint support if available in DuckDB
|
|
140
166
|
- Add transaction isolation level support
|
|
141
167
|
- Implement manual transaction control for autocommit mode
|
|
142
168
|
- _Requirements: 5.5, 5.6, 5.7_
|
|
143
169
|
|
|
144
170
|
- [x] 8. Implement advanced SQL generation features
|
|
171
|
+
|
|
145
172
|
- [x] 8.1 Add complex query support to DatasetMethods
|
|
173
|
+
|
|
146
174
|
- Implement proper WHERE clause generation
|
|
147
175
|
- Add ORDER BY, LIMIT, and OFFSET support
|
|
148
176
|
- Implement GROUP BY and HAVING clause generation
|
|
@@ -150,6 +178,7 @@
|
|
|
150
178
|
- _Requirements: 6.4, 6.5, 6.6, 6.7, 6.8, 6.9_
|
|
151
179
|
|
|
152
180
|
- [x] 8.2 Add DuckDB-specific SQL features
|
|
181
|
+
|
|
153
182
|
- Implement window function support
|
|
154
183
|
- Add Common Table Expression (CTE) support
|
|
155
184
|
- Implement subquery generation
|
|
@@ -157,7 +186,9 @@
|
|
|
157
186
|
- _Requirements: 2.6, 2.7, 2.8_
|
|
158
187
|
|
|
159
188
|
- [x] 9. Implement SQL execution methods in DatabaseMethods
|
|
189
|
+
|
|
160
190
|
- [x] 9.1 Add core SQL execution methods
|
|
191
|
+
|
|
161
192
|
- Implement execute method with connection synchronization
|
|
162
193
|
- Implement execute_insert and execute_update methods
|
|
163
194
|
- Add execute_statement private method for actual SQL execution
|
|
@@ -165,13 +196,16 @@
|
|
|
165
196
|
- _Requirements: 2.1, 2.2, 2.3, 2.4_
|
|
166
197
|
|
|
167
198
|
- [x] 9.2 Add dataset operation support
|
|
199
|
+
|
|
168
200
|
- Implement count, first, and all methods in DatasetMethods
|
|
169
201
|
- Add proper result set handling and conversion
|
|
170
202
|
- Implement streaming result support where possible
|
|
171
203
|
- _Requirements: 6.1, 6.2, 6.3, 9.5_
|
|
172
204
|
|
|
173
205
|
- [-] 10. Implement error handling and logging
|
|
206
|
+
|
|
174
207
|
- [x] 10.1 Add error mapping in DatabaseMethods
|
|
208
|
+
|
|
175
209
|
- Implement database_error_classes method
|
|
176
210
|
- Add database_exception_sqlstate method for SQL state extraction
|
|
177
211
|
- Map DuckDB errors to appropriate Sequel exceptions
|
|
@@ -179,6 +213,7 @@
|
|
|
179
213
|
- _Requirements: 8.1, 8.2, 8.3, 8.7_
|
|
180
214
|
|
|
181
215
|
- [x] 9.2 Add logging and debugging support
|
|
216
|
+
|
|
182
217
|
- Implement SQL query logging using Sequel's logging mechanism
|
|
183
218
|
- Add timing information for slow operations
|
|
184
219
|
- Implement connection pooling error handling
|
|
@@ -186,7 +221,9 @@
|
|
|
186
221
|
- _Requirements: 8.4, 8.5, 8.6, 9.6_
|
|
187
222
|
|
|
188
223
|
- [x] 11. Implement performance optimizations
|
|
224
|
+
|
|
189
225
|
- [x] 11.1 Add efficient result fetching
|
|
226
|
+
|
|
190
227
|
- Optimize fetch_rows method for large result sets
|
|
191
228
|
- Implement prepared statement support if beneficial
|
|
192
229
|
- Add bulk insert optimization methods
|
|
@@ -194,16 +231,17 @@
|
|
|
194
231
|
- _Requirements: 9.1, 9.2, 9.3, 9.4_
|
|
195
232
|
|
|
196
233
|
- [x] 10.2 Add memory and query optimizations
|
|
234
|
+
|
|
197
235
|
- Implement streaming result options for memory efficiency
|
|
198
236
|
- Add index-aware query generation
|
|
199
237
|
- Optimize for DuckDB's columnar storage advantages
|
|
200
238
|
- Implement parallel query execution support
|
|
201
239
|
- _Requirements: 9.5, 9.7_
|
|
202
240
|
|
|
203
|
-
|
|
204
|
-
|
|
205
241
|
- [x] 12. Implement Sequel::Model integration support
|
|
242
|
+
|
|
206
243
|
- [x] 12.1 Add model compatibility methods
|
|
244
|
+
|
|
207
245
|
- Ensure automatic schema introspection works with models
|
|
208
246
|
- Implement proper INSERT statement generation for model creation
|
|
209
247
|
- Add UPDATE statement generation for model updates
|
|
@@ -211,13 +249,16 @@
|
|
|
211
249
|
- _Requirements: 7.1, 7.2, 7.3, 7.4_
|
|
212
250
|
|
|
213
251
|
- [x] 12.2 Add association and validation support
|
|
252
|
+
|
|
214
253
|
- Implement foreign key relationship handling for associations
|
|
215
254
|
- Ensure model validations work correctly with DuckDB constraints
|
|
216
255
|
- Add model callback support during DuckDB operations
|
|
217
256
|
- _Requirements: 7.5, 7.6, 7.7_
|
|
218
257
|
|
|
219
258
|
- [x] 13. Create documentation and examples
|
|
259
|
+
|
|
220
260
|
- [x] 13.1 Write comprehensive README
|
|
261
|
+
|
|
221
262
|
- Create usage examples with connection strings
|
|
222
263
|
- Add sample code for common operations
|
|
223
264
|
- Document DuckDB-specific features and optimizations
|
|
@@ -225,6 +266,7 @@
|
|
|
225
266
|
- _Requirements: 12.1, 12.2, 12.5_
|
|
226
267
|
|
|
227
268
|
- [x] 13.2 Add API documentation and migration examples
|
|
269
|
+
|
|
228
270
|
- Generate complete YARD documentation for all public methods
|
|
229
271
|
- Create Sequel migration examples for DuckDB
|
|
230
272
|
- Document performance tuning techniques
|
|
@@ -232,7 +274,9 @@
|
|
|
232
274
|
- _Requirements: 12.2, 12.3, 12.4, 12.6_
|
|
233
275
|
|
|
234
276
|
- [x] 14. Final integration and compatibility verification
|
|
277
|
+
|
|
235
278
|
- [x] 14.1 Verify Sequel conventions compliance
|
|
279
|
+
|
|
236
280
|
- Ensure adapter follows Sequel's standard exception hierarchy
|
|
237
281
|
- Verify configuration options follow Sequel patterns
|
|
238
282
|
- Test compatibility with Ruby 3.1+ requirements
|
|
@@ -240,8 +284,9 @@
|
|
|
240
284
|
- _Requirements: 13.2, 13.3, 13.4, 13.5, 13.7_
|
|
241
285
|
|
|
242
286
|
- [x] 14.2 Complete end-to-end testing
|
|
287
|
+
|
|
243
288
|
- Run comprehensive test suite with real DuckDB databases
|
|
244
289
|
- Verify all SQL generation produces valid DuckDB syntax
|
|
245
290
|
- Test performance with large datasets
|
|
246
291
|
- Validate memory usage and connection handling
|
|
247
|
-
- _Requirements: 11.7, 9.1, 9.4, 9.5_
|
|
292
|
+
- _Requirements: 11.7, 9.1, 9.4, 9.5_
|
|
@@ -32,6 +32,7 @@ Following Sequel's adapter pattern, the fix will be implemented in the `DatasetM
|
|
|
32
32
|
**Location**: `Sequel::DuckDB::DatasetMethods` module
|
|
33
33
|
|
|
34
34
|
**Interface**:
|
|
35
|
+
|
|
35
36
|
```ruby
|
|
36
37
|
def literal_append(sql, v)
|
|
37
38
|
case v
|
|
@@ -59,18 +60,21 @@ end
|
|
|
59
60
|
### Type Handling Components
|
|
60
61
|
|
|
61
62
|
#### 1. LiteralString Handler
|
|
63
|
+
|
|
62
64
|
- **Purpose**: Process `Sequel.lit()` expressions as raw SQL following Sequel core pattern
|
|
63
65
|
- **Behavior**: Append string content directly without quoting (`sql << v`)
|
|
64
66
|
- **Input**: `Sequel::LiteralString` objects (subclass of `String`)
|
|
65
67
|
- **Output**: Raw SQL string appended to query
|
|
66
68
|
|
|
67
69
|
#### 2. Regular String Handler
|
|
70
|
+
|
|
68
71
|
- **Purpose**: Process regular Ruby strings with appropriate encoding handling
|
|
69
72
|
- **Behavior**: Apply existing binary/text string logic
|
|
70
73
|
- **Input**: Regular `String` objects (excluding `LiteralString`)
|
|
71
74
|
- **Output**: Properly quoted and escaped string literals
|
|
72
75
|
|
|
73
76
|
#### 3. Parent Class Delegation
|
|
77
|
+
|
|
74
78
|
- **Purpose**: Handle all other data types including `SQL::Function`
|
|
75
79
|
- **Behavior**: Delegate to parent class implementation (which already works correctly)
|
|
76
80
|
- **Input**: All other Ruby/Sequel objects including `SQL::Expression` subclasses
|
|
@@ -79,12 +83,15 @@ end
|
|
|
79
83
|
### Integration Points
|
|
80
84
|
|
|
81
85
|
#### Dataset Class Integration
|
|
86
|
+
|
|
82
87
|
The enhanced `literal_append` method integrates with:
|
|
88
|
+
|
|
83
89
|
- **SQL Generation Pipeline**: Core Sequel SQL building process
|
|
84
90
|
- **Query Context Handling**: SELECT, WHERE, ORDER BY, GROUP BY, HAVING clauses
|
|
85
91
|
- **Expression Composition**: Nested and complex expressions
|
|
86
92
|
|
|
87
93
|
#### Sequel Framework Integration
|
|
94
|
+
|
|
88
95
|
- **Parent Class Delegation**: Leverages existing Sequel functionality
|
|
89
96
|
- **Type System Compatibility**: Works with Sequel's expression type hierarchy
|
|
90
97
|
- **Error Handling**: Integrates with Sequel's error reporting system
|
|
@@ -94,6 +101,7 @@ The enhanced `literal_append` method integrates with:
|
|
|
94
101
|
### Input Data Types
|
|
95
102
|
|
|
96
103
|
#### Sequel::LiteralString
|
|
104
|
+
|
|
97
105
|
```ruby
|
|
98
106
|
# Created by: Sequel.lit("YEAR(created_at)")
|
|
99
107
|
# Properties:
|
|
@@ -103,6 +111,7 @@ The enhanced `literal_append` method integrates with:
|
|
|
103
111
|
```
|
|
104
112
|
|
|
105
113
|
#### Sequel::SQL::Function
|
|
114
|
+
|
|
106
115
|
```ruby
|
|
107
116
|
# Created by: Sequel.function(:count, :*)
|
|
108
117
|
# Properties:
|
|
@@ -112,6 +121,7 @@ The enhanced `literal_append` method integrates with:
|
|
|
112
121
|
```
|
|
113
122
|
|
|
114
123
|
#### Regular Data Types
|
|
124
|
+
|
|
115
125
|
```ruby
|
|
116
126
|
# String, Integer, Float, Date, Time, etc.
|
|
117
127
|
# Properties:
|
|
@@ -123,6 +133,7 @@ The enhanced `literal_append` method integrates with:
|
|
|
123
133
|
### Output Data Model
|
|
124
134
|
|
|
125
135
|
#### SQL String Buffer
|
|
136
|
+
|
|
126
137
|
- **Type**: String (mutable)
|
|
127
138
|
- **Content**: Accumulated SQL query text
|
|
128
139
|
- **Modification**: Appended to by `literal_append`
|
|
@@ -132,16 +143,19 @@ The enhanced `literal_append` method integrates with:
|
|
|
132
143
|
### Error Categories
|
|
133
144
|
|
|
134
145
|
#### 1. Unsupported Expression Types
|
|
146
|
+
|
|
135
147
|
- **Scenario**: Unknown SQL expression object encountered
|
|
136
148
|
- **Response**: Clear error message with object type information
|
|
137
149
|
- **Error Type**: `Sequel::Error` or subclass
|
|
138
150
|
|
|
139
151
|
#### 2. Expression Rendering Failures
|
|
152
|
+
|
|
140
153
|
- **Scenario**: Function or expression rendering fails
|
|
141
154
|
- **Response**: Context-aware error with failing expression details
|
|
142
155
|
- **Error Type**: `Sequel::DatabaseError`
|
|
143
156
|
|
|
144
157
|
#### 3. SQL Generation Failures
|
|
158
|
+
|
|
145
159
|
- **Scenario**: Overall SQL generation fails due to expression issues
|
|
146
160
|
- **Response**: Categorized error with query context
|
|
147
161
|
- **Error Type**: `Sequel::DatabaseError`
|
|
@@ -179,18 +193,21 @@ end
|
|
|
179
193
|
### Test Categories
|
|
180
194
|
|
|
181
195
|
#### 1. Unit Tests - SQL Generation
|
|
196
|
+
|
|
182
197
|
- **Framework**: Sequel mock database
|
|
183
198
|
- **Purpose**: Test SQL generation without database connections
|
|
184
199
|
- **Coverage**: All expression types and combinations
|
|
185
200
|
- **Location**: `test/sql_test.rb`
|
|
186
201
|
|
|
187
202
|
#### 2. Integration Tests - Database Operations
|
|
203
|
+
|
|
188
204
|
- **Framework**: Real DuckDB in-memory databases
|
|
189
205
|
- **Purpose**: End-to-end expression functionality
|
|
190
206
|
- **Coverage**: Query execution with expressions
|
|
191
207
|
- **Location**: `test/dataset_test.rb`
|
|
192
208
|
|
|
193
209
|
#### 3. Regression Tests
|
|
210
|
+
|
|
194
211
|
- **Purpose**: Ensure backward compatibility
|
|
195
212
|
- **Coverage**: Existing functionality continues working
|
|
196
213
|
- **Location**: Multiple test files
|
|
@@ -198,12 +215,14 @@ end
|
|
|
198
215
|
### Test Implementation Approach
|
|
199
216
|
|
|
200
217
|
#### Test-Driven Development
|
|
218
|
+
|
|
201
219
|
1. **Write failing tests** for each expression type
|
|
202
220
|
2. **Implement minimal fix** to make tests pass
|
|
203
221
|
3. **Refactor** while maintaining green tests
|
|
204
222
|
4. **Add edge case tests** and handle them
|
|
205
223
|
|
|
206
224
|
#### Test Structure
|
|
225
|
+
|
|
207
226
|
```ruby
|
|
208
227
|
def test_literal_string_handling
|
|
209
228
|
# Test Sequel.lit() expressions
|
|
@@ -225,6 +244,7 @@ end
|
|
|
225
244
|
```
|
|
226
245
|
|
|
227
246
|
### Coverage Requirements
|
|
247
|
+
|
|
228
248
|
- **100% line coverage** for new `literal_append` method
|
|
229
249
|
- **Branch coverage** for all case statements
|
|
230
250
|
- **Edge case coverage** for error conditions
|
|
@@ -233,9 +253,11 @@ end
|
|
|
233
253
|
## Design Decisions and Rationales
|
|
234
254
|
|
|
235
255
|
### Decision 1: Follow Sequel Core Pattern Exactly
|
|
256
|
+
|
|
236
257
|
**Rationale**: Sequel core already has the correct pattern for handling `LiteralString` as a special case of `String`. Following this established pattern ensures compatibility and maintainability.
|
|
237
258
|
|
|
238
259
|
**Evidence**: Sequel core's `literal_append` method in `/sequel/dataset/sql.rb` shows the exact pattern:
|
|
260
|
+
|
|
239
261
|
```ruby
|
|
240
262
|
when String
|
|
241
263
|
case v
|
|
@@ -249,31 +271,39 @@ when String
|
|
|
249
271
|
```
|
|
250
272
|
|
|
251
273
|
### Decision 2: Minimal Modification Approach
|
|
274
|
+
|
|
252
275
|
**Rationale**: The current implementation already works correctly for most types. Only the `LiteralString` handling is missing, so we add the minimal necessary check.
|
|
253
276
|
|
|
254
277
|
**Alternatives Considered**:
|
|
278
|
+
|
|
255
279
|
- Complete rewrite of literal_append (rejected - unnecessary and risky)
|
|
256
280
|
- Separate method for LiteralString (rejected - doesn't follow Sequel patterns)
|
|
257
281
|
|
|
258
282
|
### Decision 3: No Changes to Function Handling
|
|
283
|
+
|
|
259
284
|
**Rationale**: Testing revealed that `SQL::Function` objects already work correctly through parent class delegation. The issue is specifically with `LiteralString` objects being treated as regular strings.
|
|
260
285
|
|
|
261
286
|
**Evidence**:
|
|
287
|
+
|
|
262
288
|
- `db[:test].select(Sequel.function(:count, :*)).sql` produces correct `"SELECT count(*) FROM test"`
|
|
263
289
|
- `db[:test].select(Sequel.lit('YEAR(created_at)')).sql` incorrectly produces `"SELECT 'YEAR(created_at)' FROM test"`
|
|
264
290
|
|
|
265
291
|
### Decision 4: Preserve Existing Time/DateTime/Binary Handling
|
|
292
|
+
|
|
266
293
|
**Rationale**: The current implementation correctly handles `Time`, `DateTime`, and binary string encoding. These should remain unchanged.
|
|
267
294
|
|
|
268
295
|
**Benefits**:
|
|
296
|
+
|
|
269
297
|
- Maintains backward compatibility for existing functionality
|
|
270
298
|
- Focuses fix on the specific problem area
|
|
271
299
|
- Reduces risk of regression in working features
|
|
272
300
|
|
|
273
301
|
### Decision 5: Integration in DatasetMethods Module
|
|
302
|
+
|
|
274
303
|
**Rationale**: Following the established sequel-duckdb pattern where shared functionality is implemented in modules and included in main classes.
|
|
275
304
|
|
|
276
305
|
**Benefits**:
|
|
306
|
+
|
|
277
307
|
- Consistent with existing codebase architecture
|
|
278
308
|
- Allows for easy testing and maintenance
|
|
279
309
|
- Follows Sequel adapter conventions
|
|
@@ -281,18 +311,21 @@ when String
|
|
|
281
311
|
## Implementation Phases
|
|
282
312
|
|
|
283
313
|
### Phase 1: Fix LiteralString Handling
|
|
314
|
+
|
|
284
315
|
- Modify existing `literal_append` method to check for `LiteralString` before regular `String`
|
|
285
316
|
- Add comprehensive unit tests for `LiteralString` handling
|
|
286
317
|
- Verify existing functionality remains intact
|
|
287
318
|
|
|
288
319
|
### Phase 2: Comprehensive Testing
|
|
320
|
+
|
|
289
321
|
- Add integration tests with real database operations
|
|
290
322
|
- Test all SQL clause contexts (SELECT, WHERE, ORDER BY, etc.)
|
|
291
323
|
- Add regression tests to ensure no existing functionality breaks
|
|
292
324
|
|
|
293
325
|
### Phase 3: Edge Cases and Error Handling
|
|
326
|
+
|
|
294
327
|
- Test complex nested expressions
|
|
295
328
|
- Verify error handling for malformed expressions
|
|
296
329
|
- Performance verification with existing benchmarks
|
|
297
330
|
|
|
298
|
-
This design provides a focused, low-risk solution that addresses the core SQL expression handling issues while maintaining full backward compatibility and following established Sequel adapter patterns.
|
|
331
|
+
This design provides a focused, low-risk solution that addresses the core SQL expression handling issues while maintaining full backward compatibility and following established Sequel adapter patterns.
|
|
@@ -83,4 +83,4 @@ This specification addresses critical SQL expression handling issues in the sequ
|
|
|
83
83
|
|
|
84
84
|
1. WHEN an unsupported expression type is encountered THEN a clear error message SHALL be provided
|
|
85
85
|
2. WHEN expression rendering fails THEN the error SHALL include context about the failing expression
|
|
86
|
-
3. WHEN SQL generation fails due to expression issues THEN the error SHALL be properly categorized as a DatabaseError
|
|
86
|
+
3. WHEN SQL generation fails due to expression issues THEN the error SHALL be properly categorized as a DatabaseError
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
# Implementation Plan
|
|
2
2
|
|
|
3
3
|
- [x] 1. Write tests for LiteralString handling
|
|
4
|
+
|
|
4
5
|
- Add test in `test/sql_test.rb` to verify `Sequel.lit()` expressions are not quoted
|
|
5
6
|
- Test LiteralString in SELECT clause: `db[:test].select(Sequel.lit('YEAR(created_at)')).sql`
|
|
6
7
|
- Test LiteralString in WHERE clause: `db[:test].where(Sequel.lit('age > 18')).sql`
|
|
@@ -9,6 +10,7 @@
|
|
|
9
10
|
- _Requirements: 1.1, 1.2, 4.1, 6.1_
|
|
10
11
|
|
|
11
12
|
- [x] 2. Fix literal_append method to handle LiteralString
|
|
13
|
+
|
|
12
14
|
- Modify existing `literal_append` method in `lib/sequel/adapters/shared/duckdb.rb`
|
|
13
15
|
- Add `LiteralString` check as special case of `String` following Sequel core pattern
|
|
14
16
|
- Change `when String` to nested case with `when LiteralString` that does `sql << v`
|
|
@@ -16,6 +18,7 @@
|
|
|
16
18
|
- _Requirements: 1.1, 4.1, 4.2, 4.4, 6.1_
|
|
17
19
|
|
|
18
20
|
- [x] 3. Add integration test with real database
|
|
21
|
+
|
|
19
22
|
- Add test in `test/dataset_test.rb` using real DuckDB database
|
|
20
23
|
- Test that LiteralString expressions actually execute correctly
|
|
21
24
|
- Verify no regression in existing database operations
|
|
@@ -103,4 +103,4 @@ This specification addresses test infrastructure issues in the sequel-duckdb ada
|
|
|
103
103
|
1. WHEN new functionality is added THEN corresponding tests SHALL be required
|
|
104
104
|
2. WHEN tests are removed THEN coverage impact SHALL be evaluated
|
|
105
105
|
3. WHEN test coverage is measured THEN it SHALL include both unit and integration tests
|
|
106
|
-
4. WHEN coverage gaps are identified THEN they SHALL be prioritized for additional testing
|
|
106
|
+
4. WHEN coverage gaps are identified THEN they SHALL be prioritized for additional testing
|
data/.kiro/steering/product.md
CHANGED
|
@@ -3,6 +3,7 @@
|
|
|
3
3
|
sequel-duckdb is a Ruby gem that provides a complete database adapter for the Sequel toolkit to work with DuckDB. This gem enables Ruby applications to connect to and interact with DuckDB databases through Sequel's comprehensive ORM and database abstraction interface.
|
|
4
4
|
|
|
5
5
|
## Key Features
|
|
6
|
+
|
|
6
7
|
- Full DuckDB database adapter for Sequel
|
|
7
8
|
- Ruby 3.1+ compatibility
|
|
8
9
|
- Complete SQL generation and query support
|
|
@@ -10,13 +11,16 @@ sequel-duckdb is a Ruby gem that provides a complete database adapter for the Se
|
|
|
10
11
|
- Comprehensive test coverage for all SQL operations
|
|
11
12
|
|
|
12
13
|
## Reference Projects
|
|
14
|
+
|
|
13
15
|
- **jeremyevans/sequel**: Official Sequel repository for coding conventions
|
|
14
16
|
- **sequel-hexspace**: Secondary reference for adapter structure
|
|
15
17
|
- **sequel_impala**: Tertiary reference for implementation patterns
|
|
16
18
|
|
|
17
19
|
## Target Users
|
|
20
|
+
|
|
18
21
|
- Ruby developers using Sequel ORM
|
|
19
22
|
- Applications requiring DuckDB integration
|
|
20
23
|
|
|
21
24
|
## Implementation Approach
|
|
22
|
-
|
|
25
|
+
|
|
26
|
+
This project follows the proven patterns from existing Sequel adapters, with implementation order guided by git history analysis of reference projects. All development is designed to be AI-agent friendly with thorough documentation and incremental implementation.
|
data/.kiro/steering/structure.md
CHANGED