@leonardovida-md/drizzle-neo-duckdb 1.5.4-15 → 2.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +166 -306
- package/dist/bin/duckdb-introspect-args.d.ts +26 -0
- package/dist/client.d.ts +24 -2
- package/dist/columns.d.ts +49 -13
- package/dist/dialect.d.ts +44 -6
- package/dist/driver.d.ts +21 -4
- package/dist/duckdb-introspect.mjs +3987 -1217
- package/dist/ducklake.d.ts +52 -0
- package/dist/helpers.d.ts +1 -1
- package/dist/helpers.mjs +294 -50
- package/dist/index.d.ts +8 -2
- package/dist/index.mjs +5712 -2346
- package/dist/instance-keys.d.ts +3 -0
- package/dist/introspect.d.ts +14 -0
- package/dist/jev.d.ts +155 -0
- package/dist/motherduck.d.ts +183 -41
- package/dist/olap.d.ts +6 -0
- package/dist/pgduck.d.ts +5 -1
- package/dist/pool.d.ts +32 -5
- package/dist/prepared-statement-cache.d.ts +20 -2
- package/dist/prompt.d.ts +27 -0
- package/dist/select-builder.d.ts +37 -5
- package/dist/session.d.ts +60 -5
- package/dist/sql/array-bounds-text.d.ts +24 -0
- package/dist/sql/ast-transformer.d.ts +17 -5
- package/dist/sql/result-mapper.d.ts +9 -1
- package/dist/sql/selection.d.ts +11 -0
- package/dist/sql/visitors/array-bounds.d.ts +10 -0
- package/dist/sql/visitors/column-qualifier.d.ts +17 -0
- package/dist/sql/visitors/generate-series-alias.d.ts +2 -1
- package/dist/sql/visitors/union-with-hoister.d.ts +9 -0
- package/dist/sql-constants.d.ts +9 -0
- package/dist/time.d.ts +46 -1
- package/dist/value-wrappers.d.ts +21 -2
- package/package.json +5 -12
- package/dist/sql/visitors/array-operators.d.ts +0 -5
- package/dist/utils.d.ts +0 -2
- package/src/array-literals.ts +0 -38
- package/src/bin/duckdb-introspect.ts +0 -268
- package/src/client.ts +0 -901
- package/src/columns.ts +0 -608
- package/src/dialect.ts +0 -221
- package/src/driver.ts +0 -514
- package/src/ducklake.ts +0 -281
- package/src/helpers.ts +0 -18
- package/src/index.ts +0 -19
- package/src/introspect.ts +0 -1089
- package/src/migration-config.ts +0 -9
- package/src/migrator.ts +0 -22
- package/src/motherduck.ts +0 -994
- package/src/olap.ts +0 -315
- package/src/operators.ts +0 -8
- package/src/options.ts +0 -41
- package/src/own-property.ts +0 -17
- package/src/pgduck.ts +0 -57
- package/src/pool.ts +0 -368
- package/src/prepared-statement-cache.ts +0 -142
- package/src/select-builder.ts +0 -86
- package/src/session.ts +0 -562
- package/src/sql/ast-transformer.ts +0 -176
- package/src/sql/result-mapper.ts +0 -364
- package/src/sql/selection.ts +0 -107
- package/src/sql/split-top-level.ts +0 -40
- package/src/sql/visitors/array-operators.ts +0 -306
- package/src/sql/visitors/ast-helpers.ts +0 -39
- package/src/sql/visitors/column-qualifier.ts +0 -550
- package/src/sql/visitors/generate-series-alias.ts +0 -264
- package/src/sql/visitors/union-with-hoister.ts +0 -106
- package/src/time.ts +0 -5
- package/src/utils.ts +0 -2
- package/src/value-wrappers-core.ts +0 -168
- package/src/value-wrappers.ts +0 -165
package/README.md
CHANGED
|
@@ -13,162 +13,122 @@
|
|
|
13
13
|
|
|
14
14
|
<br>
|
|
15
15
|
|
|
16
|
-
**Drizzle DuckDB** brings [Drizzle ORM](https://orm.drizzle.team/) to [DuckDB](https://duckdb.org/),
|
|
16
|
+
**Drizzle DuckDB** brings [Drizzle ORM](https://orm.drizzle.team/) to [DuckDB](https://duckdb.org/), the in-process analytical database. You get Drizzle's type-safe query builder, migrations and TypeScript inference on top of DuckDB. It works with in-memory databases, local files, [MotherDuck](https://motherduck.com/) and Postgres servers running [`pg_duckdb`](https://github.com/duckdb/pg_duckdb).
|
|
17
17
|
|
|
18
|
-
|
|
18
|
+
> **Status:** Experimental. Query building, migrations and type inference are stable. Some DuckDB-specific types and edge cases are still being refined.
|
|
19
19
|
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
> **Note:** The npm package is `@leonardovida-md/drizzle-neo-duckdb`.
|
|
23
|
-
|
|
24
|
-
Docs tip: every docs page has a **Markdown (raw)** button for LLM-friendly source.
|
|
20
|
+
Every docs page has a **Markdown (raw)** button with LLM-friendly source.
|
|
25
21
|
|
|
26
22
|
## Installation
|
|
27
23
|
|
|
28
24
|
```bash
|
|
29
|
-
bun add @leonardovida-md/drizzle-neo-duckdb @duckdb/node-api
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
```bash
|
|
33
|
-
npm install @leonardovida-md/drizzle-neo-duckdb @duckdb/node-api
|
|
34
|
-
```
|
|
35
|
-
|
|
36
|
-
```bash
|
|
37
|
-
pnpm add @leonardovida-md/drizzle-neo-duckdb @duckdb/node-api
|
|
25
|
+
bun add @leonardovida-md/drizzle-neo-duckdb drizzle-orm @duckdb/node-api
|
|
26
|
+
# or: npm install / pnpm add with the same packages
|
|
38
27
|
```
|
|
39
28
|
|
|
40
|
-
Supported
|
|
41
|
-
|
|
29
|
+
| Requirement | Supported versions |
|
|
30
|
+
| ------------------ | ----------------------------------------------------------------------------------- |
|
|
31
|
+
| `drizzle-orm` | 0.40.1 or newer, below 0.46.0 |
|
|
32
|
+
| `@duckdb/node-api` | 1.4.4 or newer, below 1.6.0, including `-r.N` builds such as `1.5.6-r.1` |
|
|
33
|
+
| Runtime | Node.js 18.17 or newer (22 or 24 recommended), or Bun |
|
|
42
34
|
|
|
43
35
|
## Quick Start
|
|
44
36
|
|
|
45
37
|
```typescript
|
|
46
|
-
import { DuckDBInstance } from '@duckdb/node-api';
|
|
47
38
|
import { drizzle } from '@leonardovida-md/drizzle-neo-duckdb';
|
|
48
39
|
import { sql } from 'drizzle-orm';
|
|
49
|
-
import { integer,
|
|
50
|
-
|
|
51
|
-
// Connect to DuckDB
|
|
52
|
-
const instance = await DuckDBInstance.create(':memory:');
|
|
53
|
-
const connection = await instance.connect();
|
|
54
|
-
const db = drizzle(connection);
|
|
40
|
+
import { integer, pgTable, text } from 'drizzle-orm/pg-core';
|
|
55
41
|
|
|
56
|
-
// Define your schema
|
|
57
42
|
const users = pgTable('users', {
|
|
58
43
|
id: integer('id').primaryKey(),
|
|
59
44
|
name: text('name').notNull(),
|
|
60
45
|
email: text('email').notNull(),
|
|
61
46
|
});
|
|
62
47
|
|
|
63
|
-
|
|
48
|
+
const db = await drizzle(':memory:');
|
|
49
|
+
|
|
64
50
|
await db.execute(sql`
|
|
65
|
-
CREATE TABLE
|
|
51
|
+
CREATE TABLE users (
|
|
66
52
|
id INTEGER PRIMARY KEY,
|
|
67
53
|
name TEXT NOT NULL,
|
|
68
54
|
email TEXT NOT NULL
|
|
69
55
|
)
|
|
70
56
|
`);
|
|
71
57
|
|
|
72
|
-
// Insert data
|
|
73
58
|
await db.insert(users).values([
|
|
74
59
|
{ id: 1, name: 'Alice', email: 'alice@example.com' },
|
|
75
60
|
{ id: 2, name: 'Bob', email: 'bob@example.com' },
|
|
76
61
|
]);
|
|
77
62
|
|
|
78
|
-
// Query with full type safety
|
|
79
63
|
const allUsers = await db.select().from(users);
|
|
80
64
|
// ^? { id: number; name: string; email: string }[]
|
|
81
65
|
|
|
82
|
-
|
|
83
|
-
connection.closeSync();
|
|
84
|
-
```
|
|
85
|
-
|
|
86
|
-
## Connecting to DuckDB
|
|
87
|
-
|
|
88
|
-
### In-Memory Database
|
|
89
|
-
|
|
90
|
-
```typescript
|
|
91
|
-
const instance = await DuckDBInstance.create(':memory:');
|
|
92
|
-
const connection = await instance.connect();
|
|
93
|
-
const db = drizzle(connection);
|
|
66
|
+
await db.close();
|
|
94
67
|
```
|
|
95
68
|
|
|
96
|
-
|
|
69
|
+
Schemas use the regular `drizzle-orm/pg-core` builders (`pgTable`, `pgSchema` and so on), and all standard Drizzle query methods work. DuckDB SQL is largely Postgres-compatible, and the dialect adapts queries where it is not.
|
|
97
70
|
|
|
98
|
-
|
|
99
|
-
const instance = await DuckDBInstance.create('./my-database.duckdb');
|
|
100
|
-
const connection = await instance.connect();
|
|
101
|
-
const db = drizzle(connection);
|
|
102
|
-
```
|
|
71
|
+
## Connecting
|
|
103
72
|
|
|
104
|
-
|
|
73
|
+
Pass a path and the driver creates a connection pool for you. `await db.close()` closes it.
|
|
105
74
|
|
|
106
75
|
```typescript
|
|
107
|
-
|
|
108
|
-
motherduck_token: process.env.MOTHERDUCK_TOKEN,
|
|
109
|
-
});
|
|
110
|
-
const connection = await instance.connect();
|
|
111
|
-
const db = drizzle(connection);
|
|
112
|
-
```
|
|
113
|
-
|
|
114
|
-
### pg_duckdb
|
|
76
|
+
import { drizzle } from '@leonardovida-md/drizzle-neo-duckdb';
|
|
115
77
|
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
Postgres wire client such as `pg`:
|
|
78
|
+
// In-memory
|
|
79
|
+
const db = await drizzle(':memory:');
|
|
119
80
|
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
import { sql } from 'drizzle-orm';
|
|
123
|
-
import { drizzle } from '@leonardovida-md/drizzle-neo-duckdb';
|
|
81
|
+
// Local file (created if missing)
|
|
82
|
+
const db = await drizzle('./my-database.duckdb');
|
|
124
83
|
|
|
125
|
-
|
|
126
|
-
|
|
84
|
+
// MotherDuck
|
|
85
|
+
const db = await drizzle({
|
|
86
|
+
connection: {
|
|
87
|
+
path: 'md:my_database',
|
|
88
|
+
options: { motherduck_token: process.env.MOTHERDUCK_TOKEN },
|
|
89
|
+
},
|
|
127
90
|
});
|
|
128
|
-
await client.connect();
|
|
129
91
|
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
92
|
+
// DuckLake catalog
|
|
93
|
+
const db = await drizzle(':memory:', {
|
|
94
|
+
ducklake: {
|
|
95
|
+
catalog: './ducklake.duckdb',
|
|
96
|
+
attachOptions: { dataPath: './ducklake-data' },
|
|
97
|
+
},
|
|
98
|
+
});
|
|
133
99
|
```
|
|
134
100
|
|
|
135
|
-
|
|
101
|
+
Calls with the same file path share one DuckDB instance in a process, so they see each other's writes. `:memory:` is never shared.
|
|
102
|
+
|
|
103
|
+
You can also pass your own connection, pool or `pg_duckdb` client. These calls are synchronous:
|
|
136
104
|
|
|
137
105
|
```typescript
|
|
106
|
+
import { DuckDBInstance } from '@duckdb/node-api';
|
|
138
107
|
import pg from 'pg';
|
|
139
|
-
import {
|
|
140
|
-
createPgDuckConnectionPool,
|
|
141
|
-
drizzle,
|
|
142
|
-
} from '@leonardovida-md/drizzle-neo-duckdb';
|
|
143
108
|
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
109
|
+
// Existing DuckDB connection
|
|
110
|
+
const instance = await DuckDBInstance.create(':memory:');
|
|
111
|
+
const db = drizzle(await instance.connect());
|
|
147
112
|
|
|
148
|
-
|
|
113
|
+
// Postgres server with pg_duckdb. A pg.Client works too.
|
|
114
|
+
const db = drizzle(new pg.Pool({ connectionString: process.env.DATABASE_URL }));
|
|
149
115
|
```
|
|
150
116
|
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
```typescript
|
|
154
|
-
import { DefaultLogger } from 'drizzle-orm';
|
|
155
|
-
|
|
156
|
-
const db = drizzle(connection, {
|
|
157
|
-
logger: new DefaultLogger(),
|
|
158
|
-
});
|
|
159
|
-
```
|
|
117
|
+
A `pg.Pool` is wrapped with `createPgDuckConnectionPool()`, so each transaction runs on one pooled client and `await db.close()` ends the pool.
|
|
160
118
|
|
|
161
|
-
|
|
119
|
+
See [Database Connection](https://leonardovida.github.io/drizzle-duckdb/core/connection) for singleton and serverless patterns.
|
|
162
120
|
|
|
163
121
|
## Connection Pooling
|
|
164
122
|
|
|
165
|
-
DuckDB
|
|
123
|
+
DuckDB runs one query per connection, so the path forms above create a pool (default size 4).
|
|
124
|
+
|
|
125
|
+
- Set the size or a MotherDuck preset: `pool: { size: 8 }`, `pool: 'jumbo'` or `pool: 'giga'`.
|
|
126
|
+
- Tune recycling: `pool: { size: 8, idleTimeoutMs: 60_000, maxLifetimeMs: 10 * 60_000 }`.
|
|
127
|
+
- Disable pooling: `pool: false`.
|
|
128
|
+
- Transactions pin one pooled connection for their whole lifetime.
|
|
129
|
+
- `pool` is ignored when you pass a connection or pool instance.
|
|
166
130
|
|
|
167
|
-
|
|
168
|
-
- Tune timeout and recycling behavior on the auto-created pool: `pool: { size: 8, idleTimeoutMs: 60_000, maxLifetimeMs: 10 * 60_000 }`.
|
|
169
|
-
- Disable pooling for single-connection workloads: `pool: false`.
|
|
170
|
-
- Transactions pin one pooled connection for their entire lifetime; non-transactional queries still use the pool.
|
|
171
|
-
- Create the pool manually when you need the `setup` hook or want to reuse the same pool across multiple `drizzle()` instances:
|
|
131
|
+
Create the pool yourself when you need the `setup` hook or want to share it between `drizzle()` instances:
|
|
172
132
|
|
|
173
133
|
```typescript
|
|
174
134
|
import { DuckDBInstance } from '@duckdb/node-api';
|
|
@@ -185,190 +145,78 @@ const pool = createDuckDBConnectionPool(instance, {
|
|
|
185
145
|
idleTimeoutMs: 60_000,
|
|
186
146
|
});
|
|
187
147
|
const db = drizzle(pool);
|
|
148
|
+
console.log(pool.stats()); // Leases, queue depth, waits, recycling and timeouts
|
|
188
149
|
```
|
|
189
150
|
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
- Use `drizzle-orm/pg-core` for schemas; DuckDB SQL is largely Postgres-compatible.
|
|
193
|
-
- DuckDB-specific helpers: `duckDbList`, `duckDbArray`, `duckDbStruct`, `duckDbMap`, `duckDbJson`, `duckDbBlob`, `duckDbInet`, `duckDbInterval`, `duckDbTimestamp`, `duckDbDate`, `duckDbTime`.
|
|
194
|
-
- `duckDbTimestamp` and `duckDbTime` can preserve DuckDB-specific storage variants such as `TIMESTAMP_NS`, `TIMESTAMP_MS`, `TIMESTAMP_S`, `TIME_NS`, and `TIME WITH TIME ZONE` via the `duckDbType` / `withTimezone` options.
|
|
195
|
-
- Browser-safe imports live under `@leonardovida-md/drizzle-neo-duckdb/helpers` (introspection emits this path).
|
|
196
|
-
|
|
197
|
-
See the [column types](https://leonardovida.github.io/drizzle-duckdb/api/columns) docs for full API.
|
|
151
|
+
`acquireTimeout` limits time in the pool queue. Connection creation and setup run after that wait and have no deadline from this option. Check `pool.stats()` before increasing pool size, since more connections can compete with DuckDB's own query threads.
|
|
198
152
|
|
|
199
|
-
##
|
|
153
|
+
## Configuration
|
|
200
154
|
|
|
201
|
-
|
|
155
|
+
```typescript
|
|
156
|
+
import { DefaultLogger } from 'drizzle-orm';
|
|
202
157
|
|
|
203
|
-
|
|
158
|
+
const db = await drizzle(':memory:', {
|
|
159
|
+
// Log every query
|
|
160
|
+
logger: new DefaultLogger(),
|
|
204
161
|
|
|
205
|
-
|
|
162
|
+
// Pool size, preset, or timeout and recycling options
|
|
163
|
+
pool: { size: 8, idleTimeoutMs: 60_000 },
|
|
206
164
|
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
import {
|
|
210
|
-
mdAccessTokens,
|
|
211
|
-
mdCreateDive,
|
|
212
|
-
mdCreateFlight,
|
|
213
|
-
mdGetDive,
|
|
214
|
-
mdGetFlightRun,
|
|
215
|
-
mdListDives,
|
|
216
|
-
mdListFlights,
|
|
217
|
-
mdRunFlight,
|
|
218
|
-
} from '@leonardovida-md/drizzle-neo-duckdb';
|
|
165
|
+
// Per-connection prepared statement cache (default: disabled)
|
|
166
|
+
prepareCache: { size: 32 },
|
|
219
167
|
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
from ${mdListDives({ limit: 10, includeOrgShares: true })}
|
|
223
|
-
order by updated_at desc
|
|
224
|
-
`);
|
|
168
|
+
// Preserve DECIMAL values as strings in raw, streaming and nested reads
|
|
169
|
+
decimalMode: 'string', // Default: 'number'
|
|
225
170
|
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
from ${mdCreateDive({
|
|
229
|
-
title: 'Revenue Trends',
|
|
230
|
-
content: 'export default function Dive() { return null }',
|
|
231
|
-
description: 'Monthly revenue dashboard',
|
|
232
|
-
})}
|
|
233
|
-
`);
|
|
171
|
+
// Legacy raw SQL join qualification heuristic, opt in only if needed
|
|
172
|
+
qualifyRawJoinColumns: false,
|
|
234
173
|
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
from ${mdGetDive(String(dive.id))}
|
|
238
|
-
`);
|
|
174
|
+
// Throw on Postgres-style array literals like '{1,2,3}' (default: false)
|
|
175
|
+
rejectStringArrayLiterals: false,
|
|
239
176
|
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
from ${mdListFlights({ limit: 25 })}
|
|
243
|
-
`);
|
|
177
|
+
// Map camelCase keys to snake_case column names
|
|
178
|
+
casing: 'snake_case',
|
|
244
179
|
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
180
|
+
// Schema for relational queries
|
|
181
|
+
schema: mySchema,
|
|
182
|
+
});
|
|
183
|
+
```
|
|
249
184
|
|
|
250
|
-
|
|
251
|
-
select flight_id, status
|
|
252
|
-
from ${mdCreateFlight({
|
|
253
|
-
name: 'daily-refresh',
|
|
254
|
-
sourceCode: 'print("hello")',
|
|
255
|
-
scheduleCron: '0 0 * * *',
|
|
256
|
-
maxRuntimeSec: 1_800,
|
|
257
|
-
})}
|
|
258
|
-
`);
|
|
185
|
+
`pool` and `ducklake` apply only to the path and `{ connection }` forms. The other options also work with `drizzle(connection, { ... })`. See [Configuration](https://leonardovida.github.io/drizzle-duckdb/reference/configuration) for the full list.
|
|
259
186
|
|
|
260
|
-
|
|
261
|
-
select run_number, status, created_at
|
|
262
|
-
from ${mdRunFlight(String(flight.flight_id), {
|
|
263
|
-
config: { region: 'eu-west-1' },
|
|
264
|
-
})}
|
|
265
|
-
`);
|
|
187
|
+
## DuckDB Types
|
|
266
188
|
|
|
267
|
-
|
|
268
|
-
select run_number, status, created_at, ended_at
|
|
269
|
-
from ${mdGetFlightRun(String(flight.flight_id), 1)}
|
|
270
|
-
`);
|
|
271
|
-
```
|
|
189
|
+
DuckDB-specific column helpers: `duckDbList`, `duckDbArray`, `duckDbStruct`, `duckDbUnion`, `duckDbMap`, `duckDbJson`, `duckDbBlob`, `duckDbInet`, `duckDbInterval`, `duckDbTimestamp`, `duckDbDate` and `duckDbTime`.
|
|
272
190
|
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
`mdUpdateDiveContent()`, `mdDeleteDive()`, `mdListDiveVersions()`, and
|
|
277
|
-
`mdGetDiveVersion()`. Dive listing and version rows include
|
|
278
|
-
`required_resources` on supported MotherDuck deployments.
|
|
279
|
-
|
|
280
|
-
The older `mdJobs()` helper family remains exported as deprecated compatibility
|
|
281
|
-
aliases. Those helpers call the supported Flight table functions and preserve
|
|
282
|
-
the older `job_*` result column names where the Flight result uses `flight_*`.
|
|
283
|
-
The earlier `mdFlights()`, `mdFlightRuns()`, `mdFlightLogs()`, and
|
|
284
|
-
`mdFlightVersions()` TypeScript helper names also remain as deprecated aliases,
|
|
285
|
-
but new code should use the verb-style Flight helpers.
|
|
286
|
-
`mdGetFlightLogs()` returns one `MotherDuckFlightLogLineRow` per log line with
|
|
287
|
-
`line_number`, `reported_at`, and `line` fields. The old blob-shaped
|
|
288
|
-
`MotherDuckFlightLogsRow` type remains exported for the deprecated
|
|
289
|
-
`mdFlightLogs()` and `mdJobRunLogs()` compatibility views.
|
|
290
|
-
Use `mdGetFlightRun()` to fetch one run by its Flight ID and run number. Pass
|
|
291
|
-
`limit`, `offset`, and `order: 'asc' | 'desc'` to `mdGetFlightLogs()` when you
|
|
292
|
-
only need a page of older or newer log lines.
|
|
293
|
-
For optional Flight fields, `undefined` omits the named parameter and `null`
|
|
294
|
-
emits an explicit SQL `NULL`. MotherDuck treats explicit `NULL` values as clear
|
|
295
|
-
or empty values for nullable Flight options such as `requirementsTxt`, `config`,
|
|
296
|
-
and `flightSecretNames`.
|
|
297
|
-
Use `maxRuntimeSec` on `mdCreateFlight()` or `mdUpdateFlight()` to cap each run
|
|
298
|
-
in seconds. Set it to `0` for no timeout, or omit it to use the plan default.
|
|
299
|
-
|
|
300
|
-
`config` entries are exposed to Flight code as environment variables using the
|
|
301
|
-
config key. Config keys must not be empty, cannot contain `=` or NULL bytes, and
|
|
302
|
-
cannot use reserved runtime names such as `MOTHERDUCK_TOKEN` or
|
|
303
|
-
`MOTHERDUCK_FLIGHTS_RUN`. Config values cannot contain NULL bytes.
|
|
304
|
-
`flightSecretNames` references MotherDuck `TYPE flights` secrets; each secret
|
|
305
|
-
param is exposed as `<SECRET_NAME>_<KEY>`, so a secret named `api_secret` with
|
|
306
|
-
param `API_KEY` becomes `API_SECRET_API_KEY`.
|
|
307
|
-
|
|
308
|
-
## Querying
|
|
309
|
-
|
|
310
|
-
All standard Drizzle query methods work:
|
|
191
|
+
- `duckDbTimestamp` and `duckDbTime` can keep storage variants such as `TIMESTAMP_NS`, `TIMESTAMP_MS`, `TIMESTAMP_S`, `TIME_NS` and `TIME WITH TIME ZONE` through the `duckDbType` and `withTimezone` options.
|
|
192
|
+
- Postgres `json` and `jsonb` columns are rejected. Use `duckDbJson()`.
|
|
193
|
+
- Browser-safe imports live under `@leonardovida-md/drizzle-neo-duckdb/helpers`. Introspection emits this path.
|
|
311
194
|
|
|
312
|
-
|
|
313
|
-
// Select
|
|
314
|
-
const users = await db
|
|
315
|
-
.select()
|
|
316
|
-
.from(usersTable)
|
|
317
|
-
.where(eq(usersTable.active, true));
|
|
318
|
-
|
|
319
|
-
// Insert
|
|
320
|
-
await db
|
|
321
|
-
.insert(usersTable)
|
|
322
|
-
.values({ name: 'Alice', email: 'alice@example.com' });
|
|
323
|
-
|
|
324
|
-
// Insert with returning
|
|
325
|
-
const inserted = await db
|
|
326
|
-
.insert(usersTable)
|
|
327
|
-
.values({ name: 'Bob' })
|
|
328
|
-
.returning({ id: usersTable.id });
|
|
329
|
-
|
|
330
|
-
// Update
|
|
331
|
-
await db
|
|
332
|
-
.update(usersTable)
|
|
333
|
-
.set({ name: 'Updated' })
|
|
334
|
-
.where(eq(usersTable.id, 1));
|
|
335
|
-
|
|
336
|
-
// Delete
|
|
337
|
-
await db.delete(usersTable).where(eq(usersTable.id, 1));
|
|
338
|
-
```
|
|
195
|
+
See [Column Types](https://leonardovida.github.io/drizzle-duckdb/api/columns) for the full API.
|
|
339
196
|
|
|
340
|
-
###
|
|
197
|
+
### Arrays
|
|
341
198
|
|
|
342
|
-
|
|
199
|
+
DuckDB supports `@>`, `<@` and `&&` on lists and fixed-size arrays, so Drizzle's `arrayContains`, `arrayContained` and `arrayOverlaps` work unchanged. The DuckDB helpers emit `array_has_all` and `array_has_any`:
|
|
343
200
|
|
|
344
201
|
```typescript
|
|
345
202
|
import {
|
|
346
|
-
duckDbArrayContains,
|
|
347
203
|
duckDbArrayContained,
|
|
204
|
+
duckDbArrayContains,
|
|
348
205
|
duckDbArrayOverlaps,
|
|
349
206
|
} from '@leonardovida-md/drizzle-neo-duckdb';
|
|
350
207
|
|
|
351
|
-
//
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
//
|
|
358
|
-
|
|
359
|
-
.select()
|
|
360
|
-
.from(products)
|
|
361
|
-
.where(
|
|
362
|
-
duckDbArrayContained(products.tags, ['electronics', 'sale', 'featured'])
|
|
363
|
-
);
|
|
364
|
-
|
|
365
|
-
// Check if arrays overlap
|
|
366
|
-
const results = await db
|
|
367
|
-
.select()
|
|
368
|
-
.from(products)
|
|
369
|
-
.where(duckDbArrayOverlaps(products.tags, ['electronics', 'books']));
|
|
208
|
+
// Tags include every value
|
|
209
|
+
db.select().from(products).where(duckDbArrayContains(products.tags, ['electronics', 'sale']));
|
|
210
|
+
|
|
211
|
+
// Tags are a subset of the values
|
|
212
|
+
db.select().from(products).where(duckDbArrayContained(products.tags, ['electronics', 'sale', 'featured']));
|
|
213
|
+
|
|
214
|
+
// Tags share at least one value
|
|
215
|
+
db.select().from(products).where(duckDbArrayOverlaps(products.tags, ['electronics', 'books']));
|
|
370
216
|
```
|
|
371
217
|
|
|
218
|
+
First-dimension `array_lower(a, 1)` and `array_upper(a, 1)` calls are rewritten to `array_length` expressions, because DuckDB lacks those functions.
|
|
219
|
+
|
|
372
220
|
## Transactions
|
|
373
221
|
|
|
374
222
|
```typescript
|
|
@@ -378,11 +226,14 @@ await db.transaction(async (tx) => {
|
|
|
378
226
|
});
|
|
379
227
|
```
|
|
380
228
|
|
|
381
|
-
|
|
229
|
+
DuckDB differs from Postgres here:
|
|
382
230
|
|
|
383
|
-
|
|
231
|
+
- There is no `SAVEPOINT`. Nested transactions reuse the outer one, so an inner rollback aborts everything.
|
|
232
|
+
- Without a pool, other `db` queries wait while a transaction is open on the shared connection, instead of joining it.
|
|
233
|
+
- Any failed statement aborts the whole transaction. Catching the error inside the callback does not keep earlier writes. `db.transaction()` rolls back and rejects.
|
|
234
|
+
- The `config` argument (`isolationLevel`, `accessMode`, `deferrable`) is deprecated and ignored with a one-time warning, because DuckDB has no `SET TRANSACTION`.
|
|
384
235
|
|
|
385
|
-
|
|
236
|
+
## Migrations
|
|
386
237
|
|
|
387
238
|
```typescript
|
|
388
239
|
import { migrate } from '@leonardovida-md/drizzle-neo-duckdb';
|
|
@@ -390,19 +241,17 @@ import { migrate } from '@leonardovida-md/drizzle-neo-duckdb';
|
|
|
390
241
|
await migrate(db, { migrationsFolder: './drizzle' });
|
|
391
242
|
```
|
|
392
243
|
|
|
393
|
-
|
|
244
|
+
Metadata goes to `drizzle.__drizzle_migrations` by default. Concurrent `migrate()` calls apply each migration once, and DuckLake works as the default catalog. See [Migrations](https://leonardovida.github.io/drizzle-duckdb/features/migrations) for configuration.
|
|
394
245
|
|
|
395
246
|
## Schema Introspection
|
|
396
247
|
|
|
397
|
-
Generate Drizzle schema from an existing
|
|
398
|
-
|
|
399
|
-
### CLI
|
|
248
|
+
Generate a Drizzle schema from an existing database with the CLI:
|
|
400
249
|
|
|
401
250
|
```bash
|
|
402
251
|
bunx duckdb-introspect --url ./my-database.duckdb --out ./drizzle/schema.ts
|
|
403
252
|
```
|
|
404
253
|
|
|
405
|
-
|
|
254
|
+
Or from code:
|
|
406
255
|
|
|
407
256
|
```typescript
|
|
408
257
|
import { introspect } from '@leonardovida-md/drizzle-neo-duckdb';
|
|
@@ -415,80 +264,91 @@ const result = await introspect(db, {
|
|
|
415
264
|
console.log(result.files.schemaTs);
|
|
416
265
|
```
|
|
417
266
|
|
|
418
|
-
See [Introspection
|
|
267
|
+
See [Introspection](https://leonardovida.github.io/drizzle-duckdb/features/introspection) for all options.
|
|
268
|
+
|
|
269
|
+
## Analytics and MotherDuck Helpers
|
|
419
270
|
|
|
420
|
-
|
|
271
|
+
The package ships composable SQL helpers for analytical work:
|
|
272
|
+
|
|
273
|
+
- OLAP helpers such as `sumN`, `percentileCont`, window helpers and the `olap()` grouped measures builder.
|
|
274
|
+
- Lance vector, full-text and hybrid search.
|
|
275
|
+
- MotherDuck table functions for Dives, Flights, file listings and access tokens, plus session identity scalars such as `mdCurrentEmail()` and `mdIsRoleInSession()`.
|
|
276
|
+
- Typed MotherDuck AI builders: `mdPromptJev()` classifies text or images and `mdPrompt()` generates text or structured output.
|
|
277
|
+
- `duckLakeMergeAdjacentFiles()` for DuckLake compaction.
|
|
278
|
+
- Batch and Arrow reads through `db.executeBatches()`, `db.executeBatchesRaw()` and `db.executeArrow()`.
|
|
421
279
|
|
|
422
280
|
```typescript
|
|
423
|
-
|
|
424
|
-
|
|
425
|
-
logger: new DefaultLogger(),
|
|
281
|
+
import { sql } from 'drizzle-orm';
|
|
282
|
+
import { mdCreateFlight, mdListDives, mdRunFlight } from '@leonardovida-md/drizzle-neo-duckdb';
|
|
426
283
|
|
|
427
|
-
|
|
428
|
-
|
|
284
|
+
const dives = await db.execute(sql`
|
|
285
|
+
select id, title, updated_at
|
|
286
|
+
from ${mdListDives({ limit: 10, includeOrgShares: true })}
|
|
287
|
+
order by updated_at desc
|
|
288
|
+
`);
|
|
429
289
|
|
|
430
|
-
|
|
431
|
-
|
|
290
|
+
const [flight] = await db.execute(sql`
|
|
291
|
+
select flight_id, status
|
|
292
|
+
from ${mdCreateFlight({
|
|
293
|
+
name: 'daily-refresh',
|
|
294
|
+
sourceCode: 'print("hello")',
|
|
295
|
+
scheduleCron: '0 0 * * *',
|
|
296
|
+
maxRuntimeSec: 1_800,
|
|
297
|
+
instanceType: 'F16',
|
|
298
|
+
})}
|
|
299
|
+
`);
|
|
432
300
|
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
})
|
|
301
|
+
await db.execute(sql`
|
|
302
|
+
select run_number, status
|
|
303
|
+
from ${mdRunFlight(String(flight.flight_id), { config: { region: 'eu-west-1' } })}
|
|
304
|
+
`);
|
|
436
305
|
```
|
|
437
306
|
|
|
438
|
-
|
|
307
|
+
The older `mdJobs()` and `mdFlights()` helper names remain as deprecated aliases. See [OLAP Helpers](https://leonardovida.github.io/drizzle-duckdb/api/olap-helpers) for the full list, Flight options and log pagination, and [MotherDuck](https://leonardovida.github.io/drizzle-duckdb/integrations/motherduck) for the AI builders.
|
|
439
308
|
|
|
440
309
|
## Known Limitations
|
|
441
310
|
|
|
442
|
-
|
|
443
|
-
|
|
444
|
-
|
|
|
445
|
-
|
|
|
446
|
-
|
|
|
447
|
-
|
|
|
448
|
-
|
|
|
449
|
-
|
|
|
450
|
-
|
|
|
451
|
-
| Streaming results | Chunked reads via `executeBatches()` / `executeArrow()`; no cursor streaming |
|
|
452
|
-
| Concurrent queries | One query per connection; use pooling for parallelism |
|
|
311
|
+
| Feature | Status |
|
|
312
|
+
| --------------------- | ----------------------------------------------------------------------------- |
|
|
313
|
+
| Basic CRUD operations | Full support |
|
|
314
|
+
| Joins and subqueries | Full support |
|
|
315
|
+
| Transactions | No savepoints (nested transactions reuse the outer one). Config is ignored |
|
|
316
|
+
| JSON/JSONB columns | Use `duckDbJson()` instead |
|
|
317
|
+
| Prepared statements | Optional per-connection cache via `prepareCache`, no named statements |
|
|
318
|
+
| Streaming results | Chunked reads via `executeBatches()`, no cursor streaming |
|
|
319
|
+
| Concurrent queries | One query per connection. Use pooling for parallelism |
|
|
453
320
|
|
|
454
|
-
See [Limitations
|
|
321
|
+
See [Limitations](https://leonardovida.github.io/drizzle-duckdb/reference/limitations) for details.
|
|
455
322
|
|
|
456
323
|
## Examples
|
|
457
324
|
|
|
458
|
-
- **[Parquet Analytics](./example/parquet-analytics.ts)**:
|
|
325
|
+
- **[Parquet Analytics](./example/parquet-analytics.ts)**: typed selections and grouped measures over Parquet files without importing tables
|
|
326
|
+
- **[MotherDuck NYC Taxi](./example/motherduck-nyc-taxi.ts)**: query the MotherDuck sample NYC taxi dataset with a connection pool
|
|
327
|
+
- **[Analytics Dashboard](./example/analytics-dashboard.ts)**: local in-memory analytics with DuckDB types and Parquet loading
|
|
328
|
+
- **[DuckLake Local](./example/ducklake-local.ts)** and **[DuckLake on MotherDuck](./example/ducklake-motherduck.ts)**: attach a DuckLake catalog
|
|
459
329
|
|
|
460
|
-
|
|
461
|
-
- **[Analytics Dashboard](./example/analytics-dashboard.ts)**: Local in-memory analytics with DuckDB types and Parquet loading
|
|
462
|
-
|
|
463
|
-
Run examples:
|
|
330
|
+
Run them from the repository root:
|
|
464
331
|
|
|
465
332
|
```bash
|
|
466
|
-
MOTHERDUCK_TOKEN=your_token bun example/motherduck-nyc.ts
|
|
467
333
|
bun example/analytics-dashboard.ts
|
|
334
|
+
MOTHERDUCK_TOKEN=your_token bun example/motherduck-nyc-taxi.ts
|
|
468
335
|
```
|
|
469
336
|
|
|
470
|
-
|
|
337
|
+
See [example/README.md](./example/README.md) for the full list.
|
|
471
338
|
|
|
472
|
-
|
|
339
|
+
## Contributing
|
|
473
340
|
|
|
474
|
-
|
|
475
|
-
2. Note any DuckDB-specific quirks you encounter
|
|
476
|
-
3. Use a clear, imperative commit message
|
|
341
|
+
Contributions are welcome. See [CONTRIBUTING.md](./CONTRIBUTING.md) for the guide and [SECURITY.md](./SECURITY.md) to report a vulnerability.
|
|
477
342
|
|
|
478
343
|
```bash
|
|
479
|
-
#
|
|
480
|
-
bun
|
|
481
|
-
|
|
482
|
-
#
|
|
483
|
-
bun test
|
|
484
|
-
|
|
485
|
-
# Run tests with UI
|
|
486
|
-
bun t
|
|
487
|
-
|
|
488
|
-
# Build
|
|
489
|
-
bun run build
|
|
344
|
+
bun install # install dependencies
|
|
345
|
+
bun run test # run tests (Vitest, not `bun test`)
|
|
346
|
+
bun run t # watch mode with the UI
|
|
347
|
+
bun run build # build dist/
|
|
490
348
|
```
|
|
491
349
|
|
|
350
|
+
Include tests for new features in `test/<feature>.test.ts`, note any DuckDB quirks you hit, and use short imperative commit messages.
|
|
351
|
+
|
|
492
352
|
## License
|
|
493
353
|
|
|
494
354
|
[Apache-2.0](./LICENSE)
|