@mastra/duckdb 1.5.1 → 1.5.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +53 -9
- package/dist/db-Dl0fY488.js +204 -0
- package/dist/db-Dl0fY488.js.map +1 -0
- package/dist/db-TBEcMD49.cjs +215 -0
- package/dist/db-TBEcMD49.cjs.map +1 -0
- package/dist/docs/SKILL.md +1 -1
- package/dist/docs/assets/SOURCE_MAP.json +2 -8
- package/dist/docs/references/reference-storage-duckdb.md +4 -0
- package/dist/index.cjs +708 -805
- package/dist/index.cjs.map +1 -1
- package/dist/index.js +703 -797
- package/dist/index.js.map +1 -1
- package/dist/observability-Dc-V68UX.cjs +3732 -0
- package/dist/observability-Dc-V68UX.cjs.map +1 -0
- package/dist/observability-UnP20Kyg.js +3732 -0
- package/dist/observability-UnP20Kyg.js.map +1 -0
- package/dist/storage/db/index.d.ts +17 -0
- package/dist/storage/db/index.d.ts.map +1 -1
- package/dist/storage/domains/observability/tracing.d.ts.map +1 -1
- package/dist/storage/index.d.ts +13 -0
- package/dist/storage/index.d.ts.map +1 -1
- package/package.json +16 -15
- package/dist/chunk-4LIZE4MC.js +0 -216
- package/dist/chunk-4LIZE4MC.js.map +0 -1
- package/dist/chunk-SMRZJTCI.cjs +0 -219
- package/dist/chunk-SMRZJTCI.cjs.map +0 -1
- package/dist/observability-M35AJUGT.js +0 -3806
- package/dist/observability-M35AJUGT.js.map +0 -1
- package/dist/observability-V2KYD7UF.cjs +0 -3808
- package/dist/observability-V2KYD7UF.cjs.map +0 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,55 @@
|
|
|
1
1
|
# @mastra/duckdb
|
|
2
2
|
|
|
3
|
+
## 1.5.2
|
|
4
|
+
|
|
5
|
+
### Patch Changes
|
|
6
|
+
|
|
7
|
+
- Fixed severe CPU and memory spikes when listing traces from large DuckDB databases. ([#20175](https://github.com/mastra-ai/mastra/pull/20175))
|
|
8
|
+
|
|
9
|
+
Opening the traces page in Studio (or calling the list traces / list branches APIs) against a multi-GB trace database previously decompressed the entire span_events table for every page load, poll, and scroll — pinning all CPU cores and ballooning memory by several GB per query. On multi-GB databases, trace list queries now use roughly 5x less CPU and stay within a bounded memory budget:
|
|
10
|
+
|
|
11
|
+
- Page queries now scan only the time range containing the requested spans instead of the whole table
|
|
12
|
+
- Filtered and custom-ordered queries (status, hasChildError, order by endedAt) now paginate on a narrow column set before reconstructing full span payloads
|
|
13
|
+
- Delta polls now short-circuit when there is no new data
|
|
14
|
+
|
|
15
|
+
Also added `memoryLimit` and `threads` options to `DuckDBStore`. DuckDB previously used its default memory budget of 80% of system RAM, which could push application servers into swap; it now defaults to 2GB. File-backed databases spill larger-than-memory operations to disk. Note that `:memory:` databases cannot spill, so if you run very large queries against an in-memory database, raise `memoryLimit`.
|
|
16
|
+
|
|
17
|
+
```typescript
|
|
18
|
+
const store = new DuckDBStore({
|
|
19
|
+
path: 'mastra.duckdb',
|
|
20
|
+
memoryLimit: '4GB', // default '2GB'
|
|
21
|
+
threads: 2, // default: one per CPU core
|
|
22
|
+
});
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
- Updated dependencies [[`ce93a3c`](https://github.com/mastra-ai/mastra/commit/ce93a3c114ea1cbfbd576f3db41d7c26c9844f5b), [`5718a22`](https://github.com/mastra-ai/mastra/commit/5718a229281dcfd36bcd1f42a242e3717e510a33), [`a211d09`](https://github.com/mastra-ai/mastra/commit/a211d09185dc65a746534914cf38b67f21ee9bac), [`0dca9d0`](https://github.com/mastra-ai/mastra/commit/0dca9d0b1356024a53b72ea6f040db528b126caa), [`6218217`](https://github.com/mastra-ai/mastra/commit/62182171b6cfca0b099f1c6a77a2e65e7639ab86), [`5807d3a`](https://github.com/mastra-ai/mastra/commit/5807d3ae1d259b8b7d6df7e5bf2b485c694af9c8), [`57661af`](https://github.com/mastra-ai/mastra/commit/57661afeca52ff9af4e72675ede2134fa503d5a5), [`05db566`](https://github.com/mastra-ai/mastra/commit/05db566fcbdcbf33d0bffca0c72ec30129e2e3ca), [`57661af`](https://github.com/mastra-ai/mastra/commit/57661afeca52ff9af4e72675ede2134fa503d5a5), [`57661af`](https://github.com/mastra-ai/mastra/commit/57661afeca52ff9af4e72675ede2134fa503d5a5), [`5718a22`](https://github.com/mastra-ai/mastra/commit/5718a229281dcfd36bcd1f42a242e3717e510a33), [`57661af`](https://github.com/mastra-ai/mastra/commit/57661afeca52ff9af4e72675ede2134fa503d5a5), [`d1b7e3a`](https://github.com/mastra-ai/mastra/commit/d1b7e3a978a309a5653eeaa490d2d6c7c53bd093), [`29c584a`](https://github.com/mastra-ai/mastra/commit/29c584a13a88831e5ed1fdeb0ff8e82eae180433), [`c093146`](https://github.com/mastra-ai/mastra/commit/c0931466404d3c521308ea119cb165bb7e695155), [`8124754`](https://github.com/mastra-ai/mastra/commit/8124754ae89fbc69f8136d1df4a91904d0f84c4e), [`d12b2e4`](https://github.com/mastra-ai/mastra/commit/d12b2e4023fd9e3d3e93a9169f5088bcee2a849c)]:
|
|
26
|
+
- @mastra/core@1.54.0
|
|
27
|
+
|
|
28
|
+
## 1.5.2-alpha.0
|
|
29
|
+
|
|
30
|
+
### Patch Changes
|
|
31
|
+
|
|
32
|
+
- Fixed severe CPU and memory spikes when listing traces from large DuckDB databases. ([#20175](https://github.com/mastra-ai/mastra/pull/20175))
|
|
33
|
+
|
|
34
|
+
Opening the traces page in Studio (or calling the list traces / list branches APIs) against a multi-GB trace database previously decompressed the entire span_events table for every page load, poll, and scroll — pinning all CPU cores and ballooning memory by several GB per query. On multi-GB databases, trace list queries now use roughly 5x less CPU and stay within a bounded memory budget:
|
|
35
|
+
|
|
36
|
+
- Page queries now scan only the time range containing the requested spans instead of the whole table
|
|
37
|
+
- Filtered and custom-ordered queries (status, hasChildError, order by endedAt) now paginate on a narrow column set before reconstructing full span payloads
|
|
38
|
+
- Delta polls now short-circuit when there is no new data
|
|
39
|
+
|
|
40
|
+
Also added `memoryLimit` and `threads` options to `DuckDBStore`. DuckDB previously used its default memory budget of 80% of system RAM, which could push application servers into swap; it now defaults to 2GB. File-backed databases spill larger-than-memory operations to disk. Note that `:memory:` databases cannot spill, so if you run very large queries against an in-memory database, raise `memoryLimit`.
|
|
41
|
+
|
|
42
|
+
```typescript
|
|
43
|
+
const store = new DuckDBStore({
|
|
44
|
+
path: 'mastra.duckdb',
|
|
45
|
+
memoryLimit: '4GB', // default '2GB'
|
|
46
|
+
threads: 2, // default: one per CPU core
|
|
47
|
+
});
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
- Updated dependencies [[`ce93a3c`](https://github.com/mastra-ai/mastra/commit/ce93a3c114ea1cbfbd576f3db41d7c26c9844f5b), [`5718a22`](https://github.com/mastra-ai/mastra/commit/5718a229281dcfd36bcd1f42a242e3717e510a33), [`5807d3a`](https://github.com/mastra-ai/mastra/commit/5807d3ae1d259b8b7d6df7e5bf2b485c694af9c8), [`57661af`](https://github.com/mastra-ai/mastra/commit/57661afeca52ff9af4e72675ede2134fa503d5a5), [`57661af`](https://github.com/mastra-ai/mastra/commit/57661afeca52ff9af4e72675ede2134fa503d5a5), [`57661af`](https://github.com/mastra-ai/mastra/commit/57661afeca52ff9af4e72675ede2134fa503d5a5), [`5718a22`](https://github.com/mastra-ai/mastra/commit/5718a229281dcfd36bcd1f42a242e3717e510a33), [`57661af`](https://github.com/mastra-ai/mastra/commit/57661afeca52ff9af4e72675ede2134fa503d5a5), [`d1b7e3a`](https://github.com/mastra-ai/mastra/commit/d1b7e3a978a309a5653eeaa490d2d6c7c53bd093), [`c093146`](https://github.com/mastra-ai/mastra/commit/c0931466404d3c521308ea119cb165bb7e695155)]:
|
|
51
|
+
- @mastra/core@1.54.0-alpha.1
|
|
52
|
+
|
|
3
53
|
## 1.5.1
|
|
4
54
|
|
|
5
55
|
### Patch Changes
|
|
@@ -380,9 +430,7 @@
|
|
|
380
430
|
});
|
|
381
431
|
|
|
382
432
|
export const mastra = new Mastra({
|
|
383
|
-
agents: {
|
|
384
|
-
/* your agents here */
|
|
385
|
-
},
|
|
433
|
+
agents: {/* your agents here */},
|
|
386
434
|
observability: new Observability({
|
|
387
435
|
configs: {
|
|
388
436
|
default: {
|
|
@@ -466,9 +514,7 @@
|
|
|
466
514
|
});
|
|
467
515
|
|
|
468
516
|
export const mastra = new Mastra({
|
|
469
|
-
agents: {
|
|
470
|
-
/* your agents here */
|
|
471
|
-
},
|
|
517
|
+
agents: {/* your agents here */},
|
|
472
518
|
observability: new Observability({
|
|
473
519
|
configs: {
|
|
474
520
|
default: {
|
|
@@ -515,9 +561,7 @@
|
|
|
515
561
|
});
|
|
516
562
|
|
|
517
563
|
export const mastra = new Mastra({
|
|
518
|
-
agents: {
|
|
519
|
-
/* your agents here */
|
|
520
|
-
},
|
|
564
|
+
agents: {/* your agents here */},
|
|
521
565
|
observability: new Observability({
|
|
522
566
|
configs: {
|
|
523
567
|
default: {
|
|
@@ -0,0 +1,204 @@
|
|
|
1
|
+
import { DuckDBInstance, DuckDBTimestampTZValue, DuckDBTimestampValue } from "@duckdb/node-api";
|
|
2
|
+
import { MastraBase } from "@mastra/core/base";
|
|
3
|
+
//#region src/storage/db/index.ts
|
|
4
|
+
/**
|
|
5
|
+
* Bind a single parameter to a prepared statement using explicit typed methods.
|
|
6
|
+
* This avoids the "Cannot create values of type ANY" error that occurs when
|
|
7
|
+
* DuckDB cannot infer parameter types from SQL context (e.g. json_extract_string).
|
|
8
|
+
*/
|
|
9
|
+
function bindParam(stmt, index, value) {
|
|
10
|
+
if (value === null || value === void 0) stmt.bindNull(index);
|
|
11
|
+
else if (typeof value === "string") stmt.bindVarchar(index, value);
|
|
12
|
+
else if (typeof value === "number") if (Number.isInteger(value) && value >= -2147483648 && value <= 2147483647) stmt.bindInteger(index, value);
|
|
13
|
+
else stmt.bindDouble(index, value);
|
|
14
|
+
else if (typeof value === "boolean") stmt.bindBoolean(index, value);
|
|
15
|
+
else if (typeof value === "bigint") stmt.bindBigInt(index, value);
|
|
16
|
+
else if (value instanceof Date) stmt.bindTimestamp(index, new DuckDBTimestampValue(BigInt(value.getTime()) * 1000n));
|
|
17
|
+
else if (value instanceof DuckDBTimestampValue) stmt.bindTimestamp(index, value);
|
|
18
|
+
else if (value instanceof DuckDBTimestampTZValue) stmt.bindTimestampTZ(index, value);
|
|
19
|
+
else stmt.bindVarchar(index, JSON.stringify(value));
|
|
20
|
+
}
|
|
21
|
+
/** Convert DuckDB-specific return types to plain JS types */
|
|
22
|
+
function toJsValue(val) {
|
|
23
|
+
if (val === null || val === void 0) return val;
|
|
24
|
+
if (val instanceof DuckDBTimestampValue) return new Date(Number(val.micros / 1000n));
|
|
25
|
+
if (typeof val === "bigint") return Number(val);
|
|
26
|
+
return val;
|
|
27
|
+
}
|
|
28
|
+
const DEFAULT_MEMORY_LIMIT = "2GB";
|
|
29
|
+
/**
|
|
30
|
+
* Shared DuckDB connection management for Mastra storage.
|
|
31
|
+
* Defaults to a local file (`mastra.duckdb`) when no path is provided.
|
|
32
|
+
* Pass `path: ':memory:'` for an ephemeral in-memory database.
|
|
33
|
+
*/
|
|
34
|
+
var DuckDBConnection = class extends MastraBase {
|
|
35
|
+
instance = null;
|
|
36
|
+
initialized = false;
|
|
37
|
+
initPromise = null;
|
|
38
|
+
path;
|
|
39
|
+
instanceOptions;
|
|
40
|
+
constructor(config = {}) {
|
|
41
|
+
super({
|
|
42
|
+
component: "STORAGE",
|
|
43
|
+
name: "DUCKDB"
|
|
44
|
+
});
|
|
45
|
+
this.path = config.path ?? "mastra.duckdb";
|
|
46
|
+
this.instanceOptions = {
|
|
47
|
+
max_memory: config.memoryLimit ?? DEFAULT_MEMORY_LIMIT,
|
|
48
|
+
...config.threads !== void 0 ? { threads: String(config.threads) } : {}
|
|
49
|
+
};
|
|
50
|
+
}
|
|
51
|
+
async initialize() {
|
|
52
|
+
if (this.initialized && this.instance) return;
|
|
53
|
+
if (this.initPromise) {
|
|
54
|
+
await this.initPromise;
|
|
55
|
+
if (this.instance) return;
|
|
56
|
+
this.initPromise = null;
|
|
57
|
+
this.initialized = false;
|
|
58
|
+
}
|
|
59
|
+
this.initPromise = (async () => {
|
|
60
|
+
try {
|
|
61
|
+
this.instance = await DuckDBInstance.create(this.path, this.instanceOptions);
|
|
62
|
+
this.initialized = true;
|
|
63
|
+
} catch (error) {
|
|
64
|
+
this.instance = null;
|
|
65
|
+
this.initialized = false;
|
|
66
|
+
this.initPromise = null;
|
|
67
|
+
throw error;
|
|
68
|
+
}
|
|
69
|
+
})();
|
|
70
|
+
return this.initPromise;
|
|
71
|
+
}
|
|
72
|
+
/** Create a new connection to the DuckDB instance, initializing if needed. */
|
|
73
|
+
async getConnection() {
|
|
74
|
+
await this.initialize();
|
|
75
|
+
if (!this.instance) throw new Error("DuckDB instance not initialized");
|
|
76
|
+
return this.instance.connect();
|
|
77
|
+
}
|
|
78
|
+
closeConnection(connection) {
|
|
79
|
+
const conn = connection;
|
|
80
|
+
try {
|
|
81
|
+
if (typeof conn?.closeSync === "function") {
|
|
82
|
+
conn.closeSync();
|
|
83
|
+
return;
|
|
84
|
+
}
|
|
85
|
+
if (typeof conn?.disconnectSync === "function") {
|
|
86
|
+
conn.disconnectSync();
|
|
87
|
+
return;
|
|
88
|
+
}
|
|
89
|
+
if (typeof conn?.close === "function") {
|
|
90
|
+
conn.close();
|
|
91
|
+
return;
|
|
92
|
+
}
|
|
93
|
+
if (typeof conn?.disconnect === "function") conn.disconnect();
|
|
94
|
+
} catch {}
|
|
95
|
+
}
|
|
96
|
+
/**
|
|
97
|
+
* Execute a SQL query and return results as objects.
|
|
98
|
+
*/
|
|
99
|
+
async query(sql, params = []) {
|
|
100
|
+
const connection = await this.getConnection();
|
|
101
|
+
try {
|
|
102
|
+
if (params.length === 0) {
|
|
103
|
+
const result = await connection.run(sql);
|
|
104
|
+
const rows = await result.getRows();
|
|
105
|
+
const columns = result.columnNames();
|
|
106
|
+
return rows.map((row) => {
|
|
107
|
+
const obj = {};
|
|
108
|
+
columns.forEach((col, i) => {
|
|
109
|
+
obj[col] = toJsValue(row[i]);
|
|
110
|
+
});
|
|
111
|
+
return obj;
|
|
112
|
+
});
|
|
113
|
+
}
|
|
114
|
+
let paramIndex = 0;
|
|
115
|
+
const preparedSql = sql.replace(/\?/g, () => `$${++paramIndex}`);
|
|
116
|
+
const stmt = await connection.prepare(preparedSql);
|
|
117
|
+
for (let i = 0; i < params.length; i++) bindParam(stmt, i + 1, params[i]);
|
|
118
|
+
const result = await stmt.run();
|
|
119
|
+
const rows = await result.getRows();
|
|
120
|
+
const columns = result.columnNames();
|
|
121
|
+
return rows.map((row) => {
|
|
122
|
+
const obj = {};
|
|
123
|
+
columns.forEach((col, i) => {
|
|
124
|
+
obj[col] = toJsValue(row[i]);
|
|
125
|
+
});
|
|
126
|
+
return obj;
|
|
127
|
+
});
|
|
128
|
+
} finally {
|
|
129
|
+
this.closeConnection(connection);
|
|
130
|
+
}
|
|
131
|
+
}
|
|
132
|
+
/**
|
|
133
|
+
* Execute a SQL statement without returning results.
|
|
134
|
+
*/
|
|
135
|
+
async execute(sql, params = []) {
|
|
136
|
+
const connection = await this.getConnection();
|
|
137
|
+
try {
|
|
138
|
+
if (params.length === 0) {
|
|
139
|
+
await connection.run(sql);
|
|
140
|
+
return;
|
|
141
|
+
}
|
|
142
|
+
let paramIndex = 0;
|
|
143
|
+
const preparedSql = sql.replace(/\?/g, () => `$${++paramIndex}`);
|
|
144
|
+
const stmt = await connection.prepare(preparedSql);
|
|
145
|
+
for (let i = 0; i < params.length; i++) bindParam(stmt, i + 1, params[i]);
|
|
146
|
+
await stmt.run();
|
|
147
|
+
} finally {
|
|
148
|
+
this.closeConnection(connection);
|
|
149
|
+
}
|
|
150
|
+
}
|
|
151
|
+
/**
|
|
152
|
+
* Execute multiple SQL statements in order using a single DuckDB connection.
|
|
153
|
+
*
|
|
154
|
+
* This is intended for schema setup/migrations where statements have no
|
|
155
|
+
* parameters and must remain ordered, but opening a connection per statement
|
|
156
|
+
* would dominate initialization cost. Blank statements are skipped. Like
|
|
157
|
+
* calling execute() repeatedly, this does not wrap statements in a transaction,
|
|
158
|
+
* so prior statements can remain applied if a later statement fails.
|
|
159
|
+
*/
|
|
160
|
+
async executeBatch(sqlStatements) {
|
|
161
|
+
const statements = sqlStatements.map((statement) => statement.trim()).filter(Boolean);
|
|
162
|
+
if (statements.length === 0) return;
|
|
163
|
+
const connection = await this.getConnection();
|
|
164
|
+
try {
|
|
165
|
+
const sql = statements.map((statement, i) => `-- executeBatch statement ${i + 1}\n${statement}`).join("\n;\n") + "\n;";
|
|
166
|
+
await connection.run(sql);
|
|
167
|
+
} finally {
|
|
168
|
+
this.closeConnection(connection);
|
|
169
|
+
}
|
|
170
|
+
}
|
|
171
|
+
/**
|
|
172
|
+
* Escape a value for safe inline SQL use.
|
|
173
|
+
* DuckDB prepared statements can't handle NULL for parameters typed as ANY,
|
|
174
|
+
* so for complex INSERT/UPDATE operations we inline values safely.
|
|
175
|
+
*/
|
|
176
|
+
static sqlValue(value) {
|
|
177
|
+
if (value === null || value === void 0) return "NULL";
|
|
178
|
+
if (typeof value === "number") {
|
|
179
|
+
if (!Number.isFinite(value)) return "NULL";
|
|
180
|
+
return String(value);
|
|
181
|
+
}
|
|
182
|
+
if (typeof value === "boolean") return value ? "TRUE" : "FALSE";
|
|
183
|
+
if (value instanceof Date) return `'${value.toISOString()}'::TIMESTAMP`;
|
|
184
|
+
if (typeof value === "string") return `'${value.replace(/'/g, "''")}'`;
|
|
185
|
+
return `'${JSON.stringify(value).replace(/'/g, "''")}'`;
|
|
186
|
+
}
|
|
187
|
+
/** Release the DuckDB instance, allowing garbage collection. */
|
|
188
|
+
async close() {
|
|
189
|
+
if (this.instance) {
|
|
190
|
+
try {
|
|
191
|
+
const instance = this.instance;
|
|
192
|
+
if (typeof instance.closeSync === "function") instance.closeSync();
|
|
193
|
+
else if (typeof instance.close === "function") instance.close();
|
|
194
|
+
} catch {}
|
|
195
|
+
this.instance = null;
|
|
196
|
+
this.initialized = false;
|
|
197
|
+
this.initPromise = null;
|
|
198
|
+
}
|
|
199
|
+
}
|
|
200
|
+
};
|
|
201
|
+
//#endregion
|
|
202
|
+
export { bindParam as n, DuckDBConnection as t };
|
|
203
|
+
|
|
204
|
+
//# sourceMappingURL=db-Dl0fY488.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"db-Dl0fY488.js","names":[],"sources":["../src/storage/db/index.ts"],"sourcesContent":["import { DuckDBInstance, DuckDBTimestampValue, DuckDBTimestampTZValue } from '@duckdb/node-api';\nimport type { DuckDBPreparedStatement } from '@duckdb/node-api';\nimport { MastraBase } from '@mastra/core/base';\n\n/**\n * Bind a single parameter to a prepared statement using explicit typed methods.\n * This avoids the \"Cannot create values of type ANY\" error that occurs when\n * DuckDB cannot infer parameter types from SQL context (e.g. json_extract_string).\n */\nexport function bindParam(stmt: DuckDBPreparedStatement, index: number, value: unknown): void {\n if (value === null || value === undefined) {\n stmt.bindNull(index);\n } else if (typeof value === 'string') {\n stmt.bindVarchar(index, value);\n } else if (typeof value === 'number') {\n if (Number.isInteger(value) && value >= -2147483648 && value <= 2147483647) {\n stmt.bindInteger(index, value);\n } else {\n stmt.bindDouble(index, value);\n }\n } else if (typeof value === 'boolean') {\n stmt.bindBoolean(index, value);\n } else if (typeof value === 'bigint') {\n stmt.bindBigInt(index, value);\n } else if (value instanceof Date) {\n stmt.bindTimestamp(index, new DuckDBTimestampValue(BigInt(value.getTime()) * 1000n));\n } else if (value instanceof DuckDBTimestampValue) {\n stmt.bindTimestamp(index, value);\n } else if (value instanceof DuckDBTimestampTZValue) {\n stmt.bindTimestampTZ(index, value);\n } else {\n // Fallback: serialize to JSON string\n stmt.bindVarchar(index, JSON.stringify(value));\n }\n}\n\n/** Convert DuckDB-specific return types to plain JS types */\nfunction toJsValue(val: unknown): unknown {\n if (val === null || val === undefined) return val;\n // DuckDBTimestampValue → Date (micros since epoch)\n if (val instanceof DuckDBTimestampValue) {\n return new Date(Number(val.micros / 1000n));\n }\n // BigInt → Number (safe for values we care about)\n if (typeof val === 'bigint') {\n return Number(val);\n }\n return val;\n}\n\n/** Configuration for the DuckDB database connection. */\nexport interface DuckDBStorageConfig {\n /** Path to the DuckDB file. Defaults to 'mastra.duckdb'. Use ':memory:' for ephemeral. */\n path?: string;\n /**\n * Maximum memory DuckDB may use (e.g. '2GB', '512MB').\n * @default '2GB'\n * DuckDB's own default is 80% of system RAM, which is far too aggressive for\n * a store embedded in an application server — a single query against a large\n * database can balloon the process by several GB and push the host into\n * swap. Larger-than-memory operations spill to disk for file-backed\n * databases. Raise this for dedicated analytical workloads.\n */\n memoryLimit?: string;\n /**\n * Number of threads DuckDB may use. Defaults to DuckDB's default (one per\n * CPU core). Lower this to keep queries from monopolizing all cores of a\n * shared application server.\n */\n threads?: number;\n}\n\nconst DEFAULT_MEMORY_LIMIT = '2GB';\n\n/**\n * Shared DuckDB connection management for Mastra storage.\n * Defaults to a local file (`mastra.duckdb`) when no path is provided.\n * Pass `path: ':memory:'` for an ephemeral in-memory database.\n */\nexport class DuckDBConnection extends MastraBase {\n private instance: DuckDBInstance | null = null;\n private initialized = false;\n private initPromise: Promise<void> | null = null;\n private path: string;\n private instanceOptions: Record<string, string>;\n\n constructor(config: DuckDBStorageConfig = {}) {\n super({ component: 'STORAGE', name: 'DUCKDB' });\n this.path = config.path ?? 'mastra.duckdb';\n this.instanceOptions = {\n max_memory: config.memoryLimit ?? DEFAULT_MEMORY_LIMIT,\n ...(config.threads !== undefined ? { threads: String(config.threads) } : {}),\n };\n }\n\n private async initialize(): Promise<void> {\n if (this.initialized && this.instance) return;\n\n if (this.initPromise) {\n await this.initPromise;\n if (this.instance) return;\n this.initPromise = null;\n this.initialized = false;\n }\n\n this.initPromise = (async () => {\n try {\n this.instance = await DuckDBInstance.create(this.path, this.instanceOptions);\n this.initialized = true;\n } catch (error) {\n this.instance = null;\n this.initialized = false;\n this.initPromise = null;\n throw error;\n }\n })();\n\n return this.initPromise;\n }\n\n /** Create a new connection to the DuckDB instance, initializing if needed. */\n async getConnection() {\n await this.initialize();\n if (!this.instance) {\n throw new Error('DuckDB instance not initialized');\n }\n return this.instance.connect();\n }\n\n private closeConnection(connection: unknown): void {\n const conn = connection as {\n closeSync?: () => void;\n disconnectSync?: () => void;\n close?: () => void;\n disconnect?: () => void;\n };\n try {\n if (typeof conn?.closeSync === 'function') {\n conn.closeSync();\n return;\n }\n if (typeof conn?.disconnectSync === 'function') {\n conn.disconnectSync();\n return;\n }\n if (typeof conn?.close === 'function') {\n conn.close();\n return;\n }\n if (typeof conn?.disconnect === 'function') {\n conn.disconnect();\n }\n } catch {\n // Ignore close failures to avoid masking query/execute errors.\n }\n }\n\n /**\n * Execute a SQL query and return results as objects.\n */\n async query<T = Record<string, unknown>>(sql: string, params: unknown[] = []): Promise<T[]> {\n const connection = await this.getConnection();\n try {\n if (params.length === 0) {\n const result = await connection.run(sql);\n const rows = await result.getRows();\n const columns = result.columnNames();\n return rows.map(row => {\n const obj: Record<string, unknown> = {};\n columns.forEach((col, i) => {\n obj[col] = toJsValue(row[i]);\n });\n return obj as T;\n });\n }\n\n let paramIndex = 0;\n const preparedSql = sql.replace(/\\?/g, () => `$${++paramIndex}`);\n const stmt = await connection.prepare(preparedSql);\n for (let i = 0; i < params.length; i++) {\n bindParam(stmt, i + 1, params[i]);\n }\n const result = await stmt.run();\n const rows = await result.getRows();\n const columns = result.columnNames();\n return rows.map(row => {\n const obj: Record<string, unknown> = {};\n columns.forEach((col, i) => {\n obj[col] = toJsValue(row[i]);\n });\n return obj as T;\n });\n } finally {\n this.closeConnection(connection);\n }\n }\n\n /**\n * Execute a SQL statement without returning results.\n */\n async execute(sql: string, params: unknown[] = []): Promise<void> {\n const connection = await this.getConnection();\n try {\n if (params.length === 0) {\n await connection.run(sql);\n return;\n }\n let paramIndex = 0;\n const preparedSql = sql.replace(/\\?/g, () => `$${++paramIndex}`);\n const stmt = await connection.prepare(preparedSql);\n for (let i = 0; i < params.length; i++) {\n bindParam(stmt, i + 1, params[i]);\n }\n await stmt.run();\n } finally {\n this.closeConnection(connection);\n }\n }\n\n /**\n * Execute multiple SQL statements in order using a single DuckDB connection.\n *\n * This is intended for schema setup/migrations where statements have no\n * parameters and must remain ordered, but opening a connection per statement\n * would dominate initialization cost. Blank statements are skipped. Like\n * calling execute() repeatedly, this does not wrap statements in a transaction,\n * so prior statements can remain applied if a later statement fails.\n */\n async executeBatch(sqlStatements: readonly string[]): Promise<void> {\n const statements = sqlStatements.map(statement => statement.trim()).filter(Boolean);\n if (statements.length === 0) return;\n\n const connection = await this.getConnection();\n try {\n const sql =\n statements.map((statement, i) => `-- executeBatch statement ${i + 1}\\n${statement}`).join('\\n;\\n') + '\\n;';\n await connection.run(sql);\n } finally {\n this.closeConnection(connection);\n }\n }\n\n /**\n * Escape a value for safe inline SQL use.\n * DuckDB prepared statements can't handle NULL for parameters typed as ANY,\n * so for complex INSERT/UPDATE operations we inline values safely.\n */\n static sqlValue(value: unknown): string {\n if (value === null || value === undefined) return 'NULL';\n if (typeof value === 'number') {\n if (!Number.isFinite(value)) return 'NULL';\n return String(value);\n }\n if (typeof value === 'boolean') return value ? 'TRUE' : 'FALSE';\n if (value instanceof Date) return `'${value.toISOString()}'::TIMESTAMP`;\n if (typeof value === 'string') return `'${value.replace(/'/g, \"''\")}'`;\n // Objects/arrays → JSON string\n return `'${JSON.stringify(value).replace(/'/g, \"''\")}'`;\n }\n\n /** Release the DuckDB instance, allowing garbage collection. */\n async close(): Promise<void> {\n if (this.instance) {\n try {\n const instance = this.instance as unknown as { closeSync?: () => void; close?: () => void };\n if (typeof instance.closeSync === 'function') {\n instance.closeSync();\n } else if (typeof instance.close === 'function') {\n instance.close();\n }\n } catch {\n // Ignore close failures to allow cleanup of references.\n }\n this.instance = null;\n this.initialized = false;\n this.initPromise = null;\n }\n }\n}\n"],"mappings":";;;;;;;;AASA,SAAgB,UAAU,MAA+B,OAAe,OAAsB;CAC5F,IAAI,UAAU,QAAQ,UAAU,KAAA,GAC9B,KAAK,SAAS,KAAK;MACd,IAAI,OAAO,UAAU,UAC1B,KAAK,YAAY,OAAO,KAAK;MACxB,IAAI,OAAO,UAAU,UAC1B,IAAI,OAAO,UAAU,KAAK,KAAK,SAAS,eAAe,SAAS,YAC9D,KAAK,YAAY,OAAO,KAAK;MAE7B,KAAK,WAAW,OAAO,KAAK;MAEzB,IAAI,OAAO,UAAU,WAC1B,KAAK,YAAY,OAAO,KAAK;MACxB,IAAI,OAAO,UAAU,UAC1B,KAAK,WAAW,OAAO,KAAK;MACvB,IAAI,iBAAiB,MAC1B,KAAK,cAAc,OAAO,IAAI,qBAAqB,OAAO,MAAM,QAAQ,CAAC,IAAI,KAAK,CAAC;MAC9E,IAAI,iBAAiB,sBAC1B,KAAK,cAAc,OAAO,KAAK;MAC1B,IAAI,iBAAiB,wBAC1B,KAAK,gBAAgB,OAAO,KAAK;MAGjC,KAAK,YAAY,OAAO,KAAK,UAAU,KAAK,CAAC;AAEjD;;AAGA,SAAS,UAAU,KAAuB;CACxC,IAAI,QAAQ,QAAQ,QAAQ,KAAA,GAAW,OAAO;CAE9C,IAAI,eAAe,sBACjB,OAAO,IAAI,KAAK,OAAO,IAAI,SAAS,KAAK,CAAC;CAG5C,IAAI,OAAO,QAAQ,UACjB,OAAO,OAAO,GAAG;CAEnB,OAAO;AACT;AAwBA,MAAM,uBAAuB;;;;;;AAO7B,IAAa,mBAAb,cAAsC,WAAW;CAC/C,WAA0C;CAC1C,cAAsB;CACtB,cAA4C;CAC5C;CACA;CAEA,YAAY,SAA8B,CAAC,GAAG;EAC5C,MAAM;GAAE,WAAW;GAAW,MAAM;EAAS,CAAC;EAC9C,KAAK,OAAO,OAAO,QAAQ;EAC3B,KAAK,kBAAkB;GACrB,YAAY,OAAO,eAAe;GAClC,GAAI,OAAO,YAAY,KAAA,IAAY,EAAE,SAAS,OAAO,OAAO,OAAO,EAAE,IAAI,CAAC;EAC5E;CACF;CAEA,MAAc,aAA4B;EACxC,IAAI,KAAK,eAAe,KAAK,UAAU;EAEvC,IAAI,KAAK,aAAa;GACpB,MAAM,KAAK;GACX,IAAI,KAAK,UAAU;GACnB,KAAK,cAAc;GACnB,KAAK,cAAc;EACrB;EAEA,KAAK,eAAe,YAAY;GAC9B,IAAI;IACF,KAAK,WAAW,MAAM,eAAe,OAAO,KAAK,MAAM,KAAK,eAAe;IAC3E,KAAK,cAAc;GACrB,SAAS,OAAO;IACd,KAAK,WAAW;IAChB,KAAK,cAAc;IACnB,KAAK,cAAc;IACnB,MAAM;GACR;EACF,EAAA,CAAG;EAEH,OAAO,KAAK;CACd;;CAGA,MAAM,gBAAgB;EACpB,MAAM,KAAK,WAAW;EACtB,IAAI,CAAC,KAAK,UACR,MAAM,IAAI,MAAM,iCAAiC;EAEnD,OAAO,KAAK,SAAS,QAAQ;CAC/B;CAEA,gBAAwB,YAA2B;EACjD,MAAM,OAAO;EAMb,IAAI;GACF,IAAI,OAAO,MAAM,cAAc,YAAY;IACzC,KAAK,UAAU;IACf;GACF;GACA,IAAI,OAAO,MAAM,mBAAmB,YAAY;IAC9C,KAAK,eAAe;IACpB;GACF;GACA,IAAI,OAAO,MAAM,UAAU,YAAY;IACrC,KAAK,MAAM;IACX;GACF;GACA,IAAI,OAAO,MAAM,eAAe,YAC9B,KAAK,WAAW;EAEpB,QAAQ,CAER;CACF;;;;CAKA,MAAM,MAAmC,KAAa,SAAoB,CAAC,GAAiB;EAC1F,MAAM,aAAa,MAAM,KAAK,cAAc;EAC5C,IAAI;GACF,IAAI,OAAO,WAAW,GAAG;IACvB,MAAM,SAAS,MAAM,WAAW,IAAI,GAAG;IACvC,MAAM,OAAO,MAAM,OAAO,QAAQ;IAClC,MAAM,UAAU,OAAO,YAAY;IACnC,OAAO,KAAK,KAAI,QAAO;KACrB,MAAM,MAA+B,CAAC;KACtC,QAAQ,SAAS,KAAK,MAAM;MAC1B,IAAI,OAAO,UAAU,IAAI,EAAE;KAC7B,CAAC;KACD,OAAO;IACT,CAAC;GACH;GAEA,IAAI,aAAa;GACjB,MAAM,cAAc,IAAI,QAAQ,aAAa,IAAI,EAAE,YAAY;GAC/D,MAAM,OAAO,MAAM,WAAW,QAAQ,WAAW;GACjD,KAAK,IAAI,IAAI,GAAG,IAAI,OAAO,QAAQ,KACjC,UAAU,MAAM,IAAI,GAAG,OAAO,EAAE;GAElC,MAAM,SAAS,MAAM,KAAK,IAAI;GAC9B,MAAM,OAAO,MAAM,OAAO,QAAQ;GAClC,MAAM,UAAU,OAAO,YAAY;GACnC,OAAO,KAAK,KAAI,QAAO;IACrB,MAAM,MAA+B,CAAC;IACtC,QAAQ,SAAS,KAAK,MAAM;KAC1B,IAAI,OAAO,UAAU,IAAI,EAAE;IAC7B,CAAC;IACD,OAAO;GACT,CAAC;EACH,UAAU;GACR,KAAK,gBAAgB,UAAU;EACjC;CACF;;;;CAKA,MAAM,QAAQ,KAAa,SAAoB,CAAC,GAAkB;EAChE,MAAM,aAAa,MAAM,KAAK,cAAc;EAC5C,IAAI;GACF,IAAI,OAAO,WAAW,GAAG;IACvB,MAAM,WAAW,IAAI,GAAG;IACxB;GACF;GACA,IAAI,aAAa;GACjB,MAAM,cAAc,IAAI,QAAQ,aAAa,IAAI,EAAE,YAAY;GAC/D,MAAM,OAAO,MAAM,WAAW,QAAQ,WAAW;GACjD,KAAK,IAAI,IAAI,GAAG,IAAI,OAAO,QAAQ,KACjC,UAAU,MAAM,IAAI,GAAG,OAAO,EAAE;GAElC,MAAM,KAAK,IAAI;EACjB,UAAU;GACR,KAAK,gBAAgB,UAAU;EACjC;CACF;;;;;;;;;;CAWA,MAAM,aAAa,eAAiD;EAClE,MAAM,aAAa,cAAc,KAAI,cAAa,UAAU,KAAK,CAAC,CAAC,CAAC,OAAO,OAAO;EAClF,IAAI,WAAW,WAAW,GAAG;EAE7B,MAAM,aAAa,MAAM,KAAK,cAAc;EAC5C,IAAI;GACF,MAAM,MACJ,WAAW,KAAK,WAAW,MAAM,6BAA6B,IAAI,EAAE,IAAI,WAAW,CAAC,CAAC,KAAK,OAAO,IAAI;GACvG,MAAM,WAAW,IAAI,GAAG;EAC1B,UAAU;GACR,KAAK,gBAAgB,UAAU;EACjC;CACF;;;;;;CAOA,OAAO,SAAS,OAAwB;EACtC,IAAI,UAAU,QAAQ,UAAU,KAAA,GAAW,OAAO;EAClD,IAAI,OAAO,UAAU,UAAU;GAC7B,IAAI,CAAC,OAAO,SAAS,KAAK,GAAG,OAAO;GACpC,OAAO,OAAO,KAAK;EACrB;EACA,IAAI,OAAO,UAAU,WAAW,OAAO,QAAQ,SAAS;EACxD,IAAI,iBAAiB,MAAM,OAAO,IAAI,MAAM,YAAY,EAAE;EAC1D,IAAI,OAAO,UAAU,UAAU,OAAO,IAAI,MAAM,QAAQ,MAAM,IAAI,EAAE;EAEpE,OAAO,IAAI,KAAK,UAAU,KAAK,CAAC,CAAC,QAAQ,MAAM,IAAI,EAAE;CACvD;;CAGA,MAAM,QAAuB;EAC3B,IAAI,KAAK,UAAU;GACjB,IAAI;IACF,MAAM,WAAW,KAAK;IACtB,IAAI,OAAO,SAAS,cAAc,YAChC,SAAS,UAAU;SACd,IAAI,OAAO,SAAS,UAAU,YACnC,SAAS,MAAM;GAEnB,QAAQ,CAER;GACA,KAAK,WAAW;GAChB,KAAK,cAAc;GACnB,KAAK,cAAc;EACrB;CACF;AACF"}
|
|
@@ -0,0 +1,215 @@
|
|
|
1
|
+
let _duckdb_node_api = require("@duckdb/node-api");
|
|
2
|
+
let _mastra_core_base = require("@mastra/core/base");
|
|
3
|
+
//#region src/storage/db/index.ts
|
|
4
|
+
/**
|
|
5
|
+
* Bind a single parameter to a prepared statement using explicit typed methods.
|
|
6
|
+
* This avoids the "Cannot create values of type ANY" error that occurs when
|
|
7
|
+
* DuckDB cannot infer parameter types from SQL context (e.g. json_extract_string).
|
|
8
|
+
*/
|
|
9
|
+
function bindParam(stmt, index, value) {
|
|
10
|
+
if (value === null || value === void 0) stmt.bindNull(index);
|
|
11
|
+
else if (typeof value === "string") stmt.bindVarchar(index, value);
|
|
12
|
+
else if (typeof value === "number") if (Number.isInteger(value) && value >= -2147483648 && value <= 2147483647) stmt.bindInteger(index, value);
|
|
13
|
+
else stmt.bindDouble(index, value);
|
|
14
|
+
else if (typeof value === "boolean") stmt.bindBoolean(index, value);
|
|
15
|
+
else if (typeof value === "bigint") stmt.bindBigInt(index, value);
|
|
16
|
+
else if (value instanceof Date) stmt.bindTimestamp(index, new _duckdb_node_api.DuckDBTimestampValue(BigInt(value.getTime()) * 1000n));
|
|
17
|
+
else if (value instanceof _duckdb_node_api.DuckDBTimestampValue) stmt.bindTimestamp(index, value);
|
|
18
|
+
else if (value instanceof _duckdb_node_api.DuckDBTimestampTZValue) stmt.bindTimestampTZ(index, value);
|
|
19
|
+
else stmt.bindVarchar(index, JSON.stringify(value));
|
|
20
|
+
}
|
|
21
|
+
/** Convert DuckDB-specific return types to plain JS types */
|
|
22
|
+
function toJsValue(val) {
|
|
23
|
+
if (val === null || val === void 0) return val;
|
|
24
|
+
if (val instanceof _duckdb_node_api.DuckDBTimestampValue) return new Date(Number(val.micros / 1000n));
|
|
25
|
+
if (typeof val === "bigint") return Number(val);
|
|
26
|
+
return val;
|
|
27
|
+
}
|
|
28
|
+
const DEFAULT_MEMORY_LIMIT = "2GB";
|
|
29
|
+
/**
|
|
30
|
+
* Shared DuckDB connection management for Mastra storage.
|
|
31
|
+
* Defaults to a local file (`mastra.duckdb`) when no path is provided.
|
|
32
|
+
* Pass `path: ':memory:'` for an ephemeral in-memory database.
|
|
33
|
+
*/
|
|
34
|
+
var DuckDBConnection = class extends _mastra_core_base.MastraBase {
|
|
35
|
+
instance = null;
|
|
36
|
+
initialized = false;
|
|
37
|
+
initPromise = null;
|
|
38
|
+
path;
|
|
39
|
+
instanceOptions;
|
|
40
|
+
constructor(config = {}) {
|
|
41
|
+
super({
|
|
42
|
+
component: "STORAGE",
|
|
43
|
+
name: "DUCKDB"
|
|
44
|
+
});
|
|
45
|
+
this.path = config.path ?? "mastra.duckdb";
|
|
46
|
+
this.instanceOptions = {
|
|
47
|
+
max_memory: config.memoryLimit ?? DEFAULT_MEMORY_LIMIT,
|
|
48
|
+
...config.threads !== void 0 ? { threads: String(config.threads) } : {}
|
|
49
|
+
};
|
|
50
|
+
}
|
|
51
|
+
async initialize() {
|
|
52
|
+
if (this.initialized && this.instance) return;
|
|
53
|
+
if (this.initPromise) {
|
|
54
|
+
await this.initPromise;
|
|
55
|
+
if (this.instance) return;
|
|
56
|
+
this.initPromise = null;
|
|
57
|
+
this.initialized = false;
|
|
58
|
+
}
|
|
59
|
+
this.initPromise = (async () => {
|
|
60
|
+
try {
|
|
61
|
+
this.instance = await _duckdb_node_api.DuckDBInstance.create(this.path, this.instanceOptions);
|
|
62
|
+
this.initialized = true;
|
|
63
|
+
} catch (error) {
|
|
64
|
+
this.instance = null;
|
|
65
|
+
this.initialized = false;
|
|
66
|
+
this.initPromise = null;
|
|
67
|
+
throw error;
|
|
68
|
+
}
|
|
69
|
+
})();
|
|
70
|
+
return this.initPromise;
|
|
71
|
+
}
|
|
72
|
+
/** Create a new connection to the DuckDB instance, initializing if needed. */
|
|
73
|
+
async getConnection() {
|
|
74
|
+
await this.initialize();
|
|
75
|
+
if (!this.instance) throw new Error("DuckDB instance not initialized");
|
|
76
|
+
return this.instance.connect();
|
|
77
|
+
}
|
|
78
|
+
closeConnection(connection) {
|
|
79
|
+
const conn = connection;
|
|
80
|
+
try {
|
|
81
|
+
if (typeof conn?.closeSync === "function") {
|
|
82
|
+
conn.closeSync();
|
|
83
|
+
return;
|
|
84
|
+
}
|
|
85
|
+
if (typeof conn?.disconnectSync === "function") {
|
|
86
|
+
conn.disconnectSync();
|
|
87
|
+
return;
|
|
88
|
+
}
|
|
89
|
+
if (typeof conn?.close === "function") {
|
|
90
|
+
conn.close();
|
|
91
|
+
return;
|
|
92
|
+
}
|
|
93
|
+
if (typeof conn?.disconnect === "function") conn.disconnect();
|
|
94
|
+
} catch {}
|
|
95
|
+
}
|
|
96
|
+
/**
|
|
97
|
+
* Execute a SQL query and return results as objects.
|
|
98
|
+
*/
|
|
99
|
+
async query(sql, params = []) {
|
|
100
|
+
const connection = await this.getConnection();
|
|
101
|
+
try {
|
|
102
|
+
if (params.length === 0) {
|
|
103
|
+
const result = await connection.run(sql);
|
|
104
|
+
const rows = await result.getRows();
|
|
105
|
+
const columns = result.columnNames();
|
|
106
|
+
return rows.map((row) => {
|
|
107
|
+
const obj = {};
|
|
108
|
+
columns.forEach((col, i) => {
|
|
109
|
+
obj[col] = toJsValue(row[i]);
|
|
110
|
+
});
|
|
111
|
+
return obj;
|
|
112
|
+
});
|
|
113
|
+
}
|
|
114
|
+
let paramIndex = 0;
|
|
115
|
+
const preparedSql = sql.replace(/\?/g, () => `$${++paramIndex}`);
|
|
116
|
+
const stmt = await connection.prepare(preparedSql);
|
|
117
|
+
for (let i = 0; i < params.length; i++) bindParam(stmt, i + 1, params[i]);
|
|
118
|
+
const result = await stmt.run();
|
|
119
|
+
const rows = await result.getRows();
|
|
120
|
+
const columns = result.columnNames();
|
|
121
|
+
return rows.map((row) => {
|
|
122
|
+
const obj = {};
|
|
123
|
+
columns.forEach((col, i) => {
|
|
124
|
+
obj[col] = toJsValue(row[i]);
|
|
125
|
+
});
|
|
126
|
+
return obj;
|
|
127
|
+
});
|
|
128
|
+
} finally {
|
|
129
|
+
this.closeConnection(connection);
|
|
130
|
+
}
|
|
131
|
+
}
|
|
132
|
+
/**
|
|
133
|
+
* Execute a SQL statement without returning results.
|
|
134
|
+
*/
|
|
135
|
+
async execute(sql, params = []) {
|
|
136
|
+
const connection = await this.getConnection();
|
|
137
|
+
try {
|
|
138
|
+
if (params.length === 0) {
|
|
139
|
+
await connection.run(sql);
|
|
140
|
+
return;
|
|
141
|
+
}
|
|
142
|
+
let paramIndex = 0;
|
|
143
|
+
const preparedSql = sql.replace(/\?/g, () => `$${++paramIndex}`);
|
|
144
|
+
const stmt = await connection.prepare(preparedSql);
|
|
145
|
+
for (let i = 0; i < params.length; i++) bindParam(stmt, i + 1, params[i]);
|
|
146
|
+
await stmt.run();
|
|
147
|
+
} finally {
|
|
148
|
+
this.closeConnection(connection);
|
|
149
|
+
}
|
|
150
|
+
}
|
|
151
|
+
/**
|
|
152
|
+
* Execute multiple SQL statements in order using a single DuckDB connection.
|
|
153
|
+
*
|
|
154
|
+
* This is intended for schema setup/migrations where statements have no
|
|
155
|
+
* parameters and must remain ordered, but opening a connection per statement
|
|
156
|
+
* would dominate initialization cost. Blank statements are skipped. Like
|
|
157
|
+
* calling execute() repeatedly, this does not wrap statements in a transaction,
|
|
158
|
+
* so prior statements can remain applied if a later statement fails.
|
|
159
|
+
*/
|
|
160
|
+
async executeBatch(sqlStatements) {
|
|
161
|
+
const statements = sqlStatements.map((statement) => statement.trim()).filter(Boolean);
|
|
162
|
+
if (statements.length === 0) return;
|
|
163
|
+
const connection = await this.getConnection();
|
|
164
|
+
try {
|
|
165
|
+
const sql = statements.map((statement, i) => `-- executeBatch statement ${i + 1}\n${statement}`).join("\n;\n") + "\n;";
|
|
166
|
+
await connection.run(sql);
|
|
167
|
+
} finally {
|
|
168
|
+
this.closeConnection(connection);
|
|
169
|
+
}
|
|
170
|
+
}
|
|
171
|
+
/**
|
|
172
|
+
* Escape a value for safe inline SQL use.
|
|
173
|
+
* DuckDB prepared statements can't handle NULL for parameters typed as ANY,
|
|
174
|
+
* so for complex INSERT/UPDATE operations we inline values safely.
|
|
175
|
+
*/
|
|
176
|
+
static sqlValue(value) {
|
|
177
|
+
if (value === null || value === void 0) return "NULL";
|
|
178
|
+
if (typeof value === "number") {
|
|
179
|
+
if (!Number.isFinite(value)) return "NULL";
|
|
180
|
+
return String(value);
|
|
181
|
+
}
|
|
182
|
+
if (typeof value === "boolean") return value ? "TRUE" : "FALSE";
|
|
183
|
+
if (value instanceof Date) return `'${value.toISOString()}'::TIMESTAMP`;
|
|
184
|
+
if (typeof value === "string") return `'${value.replace(/'/g, "''")}'`;
|
|
185
|
+
return `'${JSON.stringify(value).replace(/'/g, "''")}'`;
|
|
186
|
+
}
|
|
187
|
+
/** Release the DuckDB instance, allowing garbage collection. */
|
|
188
|
+
async close() {
|
|
189
|
+
if (this.instance) {
|
|
190
|
+
try {
|
|
191
|
+
const instance = this.instance;
|
|
192
|
+
if (typeof instance.closeSync === "function") instance.closeSync();
|
|
193
|
+
else if (typeof instance.close === "function") instance.close();
|
|
194
|
+
} catch {}
|
|
195
|
+
this.instance = null;
|
|
196
|
+
this.initialized = false;
|
|
197
|
+
this.initPromise = null;
|
|
198
|
+
}
|
|
199
|
+
}
|
|
200
|
+
};
|
|
201
|
+
//#endregion
|
|
202
|
+
Object.defineProperty(exports, "DuckDBConnection", {
|
|
203
|
+
enumerable: true,
|
|
204
|
+
get: function() {
|
|
205
|
+
return DuckDBConnection;
|
|
206
|
+
}
|
|
207
|
+
});
|
|
208
|
+
Object.defineProperty(exports, "bindParam", {
|
|
209
|
+
enumerable: true,
|
|
210
|
+
get: function() {
|
|
211
|
+
return bindParam;
|
|
212
|
+
}
|
|
213
|
+
});
|
|
214
|
+
|
|
215
|
+
//# sourceMappingURL=db-TBEcMD49.cjs.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"db-TBEcMD49.cjs","names":["DuckDBTimestampValue","DuckDBTimestampTZValue","MastraBase","DuckDBInstance"],"sources":["../src/storage/db/index.ts"],"sourcesContent":["import { DuckDBInstance, DuckDBTimestampValue, DuckDBTimestampTZValue } from '@duckdb/node-api';\nimport type { DuckDBPreparedStatement } from '@duckdb/node-api';\nimport { MastraBase } from '@mastra/core/base';\n\n/**\n * Bind a single parameter to a prepared statement using explicit typed methods.\n * This avoids the \"Cannot create values of type ANY\" error that occurs when\n * DuckDB cannot infer parameter types from SQL context (e.g. json_extract_string).\n */\nexport function bindParam(stmt: DuckDBPreparedStatement, index: number, value: unknown): void {\n if (value === null || value === undefined) {\n stmt.bindNull(index);\n } else if (typeof value === 'string') {\n stmt.bindVarchar(index, value);\n } else if (typeof value === 'number') {\n if (Number.isInteger(value) && value >= -2147483648 && value <= 2147483647) {\n stmt.bindInteger(index, value);\n } else {\n stmt.bindDouble(index, value);\n }\n } else if (typeof value === 'boolean') {\n stmt.bindBoolean(index, value);\n } else if (typeof value === 'bigint') {\n stmt.bindBigInt(index, value);\n } else if (value instanceof Date) {\n stmt.bindTimestamp(index, new DuckDBTimestampValue(BigInt(value.getTime()) * 1000n));\n } else if (value instanceof DuckDBTimestampValue) {\n stmt.bindTimestamp(index, value);\n } else if (value instanceof DuckDBTimestampTZValue) {\n stmt.bindTimestampTZ(index, value);\n } else {\n // Fallback: serialize to JSON string\n stmt.bindVarchar(index, JSON.stringify(value));\n }\n}\n\n/** Convert DuckDB-specific return types to plain JS types */\nfunction toJsValue(val: unknown): unknown {\n if (val === null || val === undefined) return val;\n // DuckDBTimestampValue → Date (micros since epoch)\n if (val instanceof DuckDBTimestampValue) {\n return new Date(Number(val.micros / 1000n));\n }\n // BigInt → Number (safe for values we care about)\n if (typeof val === 'bigint') {\n return Number(val);\n }\n return val;\n}\n\n/** Configuration for the DuckDB database connection. */\nexport interface DuckDBStorageConfig {\n /** Path to the DuckDB file. Defaults to 'mastra.duckdb'. Use ':memory:' for ephemeral. */\n path?: string;\n /**\n * Maximum memory DuckDB may use (e.g. '2GB', '512MB').\n * @default '2GB'\n * DuckDB's own default is 80% of system RAM, which is far too aggressive for\n * a store embedded in an application server — a single query against a large\n * database can balloon the process by several GB and push the host into\n * swap. Larger-than-memory operations spill to disk for file-backed\n * databases. Raise this for dedicated analytical workloads.\n */\n memoryLimit?: string;\n /**\n * Number of threads DuckDB may use. Defaults to DuckDB's default (one per\n * CPU core). Lower this to keep queries from monopolizing all cores of a\n * shared application server.\n */\n threads?: number;\n}\n\nconst DEFAULT_MEMORY_LIMIT = '2GB';\n\n/**\n * Shared DuckDB connection management for Mastra storage.\n * Defaults to a local file (`mastra.duckdb`) when no path is provided.\n * Pass `path: ':memory:'` for an ephemeral in-memory database.\n */\nexport class DuckDBConnection extends MastraBase {\n private instance: DuckDBInstance | null = null;\n private initialized = false;\n private initPromise: Promise<void> | null = null;\n private path: string;\n private instanceOptions: Record<string, string>;\n\n constructor(config: DuckDBStorageConfig = {}) {\n super({ component: 'STORAGE', name: 'DUCKDB' });\n this.path = config.path ?? 'mastra.duckdb';\n this.instanceOptions = {\n max_memory: config.memoryLimit ?? DEFAULT_MEMORY_LIMIT,\n ...(config.threads !== undefined ? { threads: String(config.threads) } : {}),\n };\n }\n\n private async initialize(): Promise<void> {\n if (this.initialized && this.instance) return;\n\n if (this.initPromise) {\n await this.initPromise;\n if (this.instance) return;\n this.initPromise = null;\n this.initialized = false;\n }\n\n this.initPromise = (async () => {\n try {\n this.instance = await DuckDBInstance.create(this.path, this.instanceOptions);\n this.initialized = true;\n } catch (error) {\n this.instance = null;\n this.initialized = false;\n this.initPromise = null;\n throw error;\n }\n })();\n\n return this.initPromise;\n }\n\n /** Create a new connection to the DuckDB instance, initializing if needed. */\n async getConnection() {\n await this.initialize();\n if (!this.instance) {\n throw new Error('DuckDB instance not initialized');\n }\n return this.instance.connect();\n }\n\n private closeConnection(connection: unknown): void {\n const conn = connection as {\n closeSync?: () => void;\n disconnectSync?: () => void;\n close?: () => void;\n disconnect?: () => void;\n };\n try {\n if (typeof conn?.closeSync === 'function') {\n conn.closeSync();\n return;\n }\n if (typeof conn?.disconnectSync === 'function') {\n conn.disconnectSync();\n return;\n }\n if (typeof conn?.close === 'function') {\n conn.close();\n return;\n }\n if (typeof conn?.disconnect === 'function') {\n conn.disconnect();\n }\n } catch {\n // Ignore close failures to avoid masking query/execute errors.\n }\n }\n\n /**\n * Execute a SQL query and return results as objects.\n */\n async query<T = Record<string, unknown>>(sql: string, params: unknown[] = []): Promise<T[]> {\n const connection = await this.getConnection();\n try {\n if (params.length === 0) {\n const result = await connection.run(sql);\n const rows = await result.getRows();\n const columns = result.columnNames();\n return rows.map(row => {\n const obj: Record<string, unknown> = {};\n columns.forEach((col, i) => {\n obj[col] = toJsValue(row[i]);\n });\n return obj as T;\n });\n }\n\n let paramIndex = 0;\n const preparedSql = sql.replace(/\\?/g, () => `$${++paramIndex}`);\n const stmt = await connection.prepare(preparedSql);\n for (let i = 0; i < params.length; i++) {\n bindParam(stmt, i + 1, params[i]);\n }\n const result = await stmt.run();\n const rows = await result.getRows();\n const columns = result.columnNames();\n return rows.map(row => {\n const obj: Record<string, unknown> = {};\n columns.forEach((col, i) => {\n obj[col] = toJsValue(row[i]);\n });\n return obj as T;\n });\n } finally {\n this.closeConnection(connection);\n }\n }\n\n /**\n * Execute a SQL statement without returning results.\n */\n async execute(sql: string, params: unknown[] = []): Promise<void> {\n const connection = await this.getConnection();\n try {\n if (params.length === 0) {\n await connection.run(sql);\n return;\n }\n let paramIndex = 0;\n const preparedSql = sql.replace(/\\?/g, () => `$${++paramIndex}`);\n const stmt = await connection.prepare(preparedSql);\n for (let i = 0; i < params.length; i++) {\n bindParam(stmt, i + 1, params[i]);\n }\n await stmt.run();\n } finally {\n this.closeConnection(connection);\n }\n }\n\n /**\n * Execute multiple SQL statements in order using a single DuckDB connection.\n *\n * This is intended for schema setup/migrations where statements have no\n * parameters and must remain ordered, but opening a connection per statement\n * would dominate initialization cost. Blank statements are skipped. Like\n * calling execute() repeatedly, this does not wrap statements in a transaction,\n * so prior statements can remain applied if a later statement fails.\n */\n async executeBatch(sqlStatements: readonly string[]): Promise<void> {\n const statements = sqlStatements.map(statement => statement.trim()).filter(Boolean);\n if (statements.length === 0) return;\n\n const connection = await this.getConnection();\n try {\n const sql =\n statements.map((statement, i) => `-- executeBatch statement ${i + 1}\\n${statement}`).join('\\n;\\n') + '\\n;';\n await connection.run(sql);\n } finally {\n this.closeConnection(connection);\n }\n }\n\n /**\n * Escape a value for safe inline SQL use.\n * DuckDB prepared statements can't handle NULL for parameters typed as ANY,\n * so for complex INSERT/UPDATE operations we inline values safely.\n */\n static sqlValue(value: unknown): string {\n if (value === null || value === undefined) return 'NULL';\n if (typeof value === 'number') {\n if (!Number.isFinite(value)) return 'NULL';\n return String(value);\n }\n if (typeof value === 'boolean') return value ? 'TRUE' : 'FALSE';\n if (value instanceof Date) return `'${value.toISOString()}'::TIMESTAMP`;\n if (typeof value === 'string') return `'${value.replace(/'/g, \"''\")}'`;\n // Objects/arrays → JSON string\n return `'${JSON.stringify(value).replace(/'/g, \"''\")}'`;\n }\n\n /** Release the DuckDB instance, allowing garbage collection. */\n async close(): Promise<void> {\n if (this.instance) {\n try {\n const instance = this.instance as unknown as { closeSync?: () => void; close?: () => void };\n if (typeof instance.closeSync === 'function') {\n instance.closeSync();\n } else if (typeof instance.close === 'function') {\n instance.close();\n }\n } catch {\n // Ignore close failures to allow cleanup of references.\n }\n this.instance = null;\n this.initialized = false;\n this.initPromise = null;\n }\n }\n}\n"],"mappings":";;;;;;;;AASA,SAAgB,UAAU,MAA+B,OAAe,OAAsB;CAC5F,IAAI,UAAU,QAAQ,UAAU,KAAA,GAC9B,KAAK,SAAS,KAAK;MACd,IAAI,OAAO,UAAU,UAC1B,KAAK,YAAY,OAAO,KAAK;MACxB,IAAI,OAAO,UAAU,UAC1B,IAAI,OAAO,UAAU,KAAK,KAAK,SAAS,eAAe,SAAS,YAC9D,KAAK,YAAY,OAAO,KAAK;MAE7B,KAAK,WAAW,OAAO,KAAK;MAEzB,IAAI,OAAO,UAAU,WAC1B,KAAK,YAAY,OAAO,KAAK;MACxB,IAAI,OAAO,UAAU,UAC1B,KAAK,WAAW,OAAO,KAAK;MACvB,IAAI,iBAAiB,MAC1B,KAAK,cAAc,OAAO,IAAIA,iBAAAA,qBAAqB,OAAO,MAAM,QAAQ,CAAC,IAAI,KAAK,CAAC;MAC9E,IAAI,iBAAiBA,iBAAAA,sBAC1B,KAAK,cAAc,OAAO,KAAK;MAC1B,IAAI,iBAAiBC,iBAAAA,wBAC1B,KAAK,gBAAgB,OAAO,KAAK;MAGjC,KAAK,YAAY,OAAO,KAAK,UAAU,KAAK,CAAC;AAEjD;;AAGA,SAAS,UAAU,KAAuB;CACxC,IAAI,QAAQ,QAAQ,QAAQ,KAAA,GAAW,OAAO;CAE9C,IAAI,eAAeD,iBAAAA,sBACjB,OAAO,IAAI,KAAK,OAAO,IAAI,SAAS,KAAK,CAAC;CAG5C,IAAI,OAAO,QAAQ,UACjB,OAAO,OAAO,GAAG;CAEnB,OAAO;AACT;AAwBA,MAAM,uBAAuB;;;;;;AAO7B,IAAa,mBAAb,cAAsCE,kBAAAA,WAAW;CAC/C,WAA0C;CAC1C,cAAsB;CACtB,cAA4C;CAC5C;CACA;CAEA,YAAY,SAA8B,CAAC,GAAG;EAC5C,MAAM;GAAE,WAAW;GAAW,MAAM;EAAS,CAAC;EAC9C,KAAK,OAAO,OAAO,QAAQ;EAC3B,KAAK,kBAAkB;GACrB,YAAY,OAAO,eAAe;GAClC,GAAI,OAAO,YAAY,KAAA,IAAY,EAAE,SAAS,OAAO,OAAO,OAAO,EAAE,IAAI,CAAC;EAC5E;CACF;CAEA,MAAc,aAA4B;EACxC,IAAI,KAAK,eAAe,KAAK,UAAU;EAEvC,IAAI,KAAK,aAAa;GACpB,MAAM,KAAK;GACX,IAAI,KAAK,UAAU;GACnB,KAAK,cAAc;GACnB,KAAK,cAAc;EACrB;EAEA,KAAK,eAAe,YAAY;GAC9B,IAAI;IACF,KAAK,WAAW,MAAMC,iBAAAA,eAAe,OAAO,KAAK,MAAM,KAAK,eAAe;IAC3E,KAAK,cAAc;GACrB,SAAS,OAAO;IACd,KAAK,WAAW;IAChB,KAAK,cAAc;IACnB,KAAK,cAAc;IACnB,MAAM;GACR;EACF,EAAA,CAAG;EAEH,OAAO,KAAK;CACd;;CAGA,MAAM,gBAAgB;EACpB,MAAM,KAAK,WAAW;EACtB,IAAI,CAAC,KAAK,UACR,MAAM,IAAI,MAAM,iCAAiC;EAEnD,OAAO,KAAK,SAAS,QAAQ;CAC/B;CAEA,gBAAwB,YAA2B;EACjD,MAAM,OAAO;EAMb,IAAI;GACF,IAAI,OAAO,MAAM,cAAc,YAAY;IACzC,KAAK,UAAU;IACf;GACF;GACA,IAAI,OAAO,MAAM,mBAAmB,YAAY;IAC9C,KAAK,eAAe;IACpB;GACF;GACA,IAAI,OAAO,MAAM,UAAU,YAAY;IACrC,KAAK,MAAM;IACX;GACF;GACA,IAAI,OAAO,MAAM,eAAe,YAC9B,KAAK,WAAW;EAEpB,QAAQ,CAER;CACF;;;;CAKA,MAAM,MAAmC,KAAa,SAAoB,CAAC,GAAiB;EAC1F,MAAM,aAAa,MAAM,KAAK,cAAc;EAC5C,IAAI;GACF,IAAI,OAAO,WAAW,GAAG;IACvB,MAAM,SAAS,MAAM,WAAW,IAAI,GAAG;IACvC,MAAM,OAAO,MAAM,OAAO,QAAQ;IAClC,MAAM,UAAU,OAAO,YAAY;IACnC,OAAO,KAAK,KAAI,QAAO;KACrB,MAAM,MAA+B,CAAC;KACtC,QAAQ,SAAS,KAAK,MAAM;MAC1B,IAAI,OAAO,UAAU,IAAI,EAAE;KAC7B,CAAC;KACD,OAAO;IACT,CAAC;GACH;GAEA,IAAI,aAAa;GACjB,MAAM,cAAc,IAAI,QAAQ,aAAa,IAAI,EAAE,YAAY;GAC/D,MAAM,OAAO,MAAM,WAAW,QAAQ,WAAW;GACjD,KAAK,IAAI,IAAI,GAAG,IAAI,OAAO,QAAQ,KACjC,UAAU,MAAM,IAAI,GAAG,OAAO,EAAE;GAElC,MAAM,SAAS,MAAM,KAAK,IAAI;GAC9B,MAAM,OAAO,MAAM,OAAO,QAAQ;GAClC,MAAM,UAAU,OAAO,YAAY;GACnC,OAAO,KAAK,KAAI,QAAO;IACrB,MAAM,MAA+B,CAAC;IACtC,QAAQ,SAAS,KAAK,MAAM;KAC1B,IAAI,OAAO,UAAU,IAAI,EAAE;IAC7B,CAAC;IACD,OAAO;GACT,CAAC;EACH,UAAU;GACR,KAAK,gBAAgB,UAAU;EACjC;CACF;;;;CAKA,MAAM,QAAQ,KAAa,SAAoB,CAAC,GAAkB;EAChE,MAAM,aAAa,MAAM,KAAK,cAAc;EAC5C,IAAI;GACF,IAAI,OAAO,WAAW,GAAG;IACvB,MAAM,WAAW,IAAI,GAAG;IACxB;GACF;GACA,IAAI,aAAa;GACjB,MAAM,cAAc,IAAI,QAAQ,aAAa,IAAI,EAAE,YAAY;GAC/D,MAAM,OAAO,MAAM,WAAW,QAAQ,WAAW;GACjD,KAAK,IAAI,IAAI,GAAG,IAAI,OAAO,QAAQ,KACjC,UAAU,MAAM,IAAI,GAAG,OAAO,EAAE;GAElC,MAAM,KAAK,IAAI;EACjB,UAAU;GACR,KAAK,gBAAgB,UAAU;EACjC;CACF;;;;;;;;;;CAWA,MAAM,aAAa,eAAiD;EAClE,MAAM,aAAa,cAAc,KAAI,cAAa,UAAU,KAAK,CAAC,CAAC,CAAC,OAAO,OAAO;EAClF,IAAI,WAAW,WAAW,GAAG;EAE7B,MAAM,aAAa,MAAM,KAAK,cAAc;EAC5C,IAAI;GACF,MAAM,MACJ,WAAW,KAAK,WAAW,MAAM,6BAA6B,IAAI,EAAE,IAAI,WAAW,CAAC,CAAC,KAAK,OAAO,IAAI;GACvG,MAAM,WAAW,IAAI,GAAG;EAC1B,UAAU;GACR,KAAK,gBAAgB,UAAU;EACjC;CACF;;;;;;CAOA,OAAO,SAAS,OAAwB;EACtC,IAAI,UAAU,QAAQ,UAAU,KAAA,GAAW,OAAO;EAClD,IAAI,OAAO,UAAU,UAAU;GAC7B,IAAI,CAAC,OAAO,SAAS,KAAK,GAAG,OAAO;GACpC,OAAO,OAAO,KAAK;EACrB;EACA,IAAI,OAAO,UAAU,WAAW,OAAO,QAAQ,SAAS;EACxD,IAAI,iBAAiB,MAAM,OAAO,IAAI,MAAM,YAAY,EAAE;EAC1D,IAAI,OAAO,UAAU,UAAU,OAAO,IAAI,MAAM,QAAQ,MAAM,IAAI,EAAE;EAEpE,OAAO,IAAI,KAAK,UAAU,KAAK,CAAC,CAAC,QAAQ,MAAM,IAAI,EAAE;CACvD;;CAGA,MAAM,QAAuB;EAC3B,IAAI,KAAK,UAAU;GACjB,IAAI;IACF,MAAM,WAAW,KAAK;IACtB,IAAI,OAAO,SAAS,cAAc,YAChC,SAAS,UAAU;SACd,IAAI,OAAO,SAAS,UAAU,YACnC,SAAS,MAAM;GAEnB,QAAQ,CAER;GACA,KAAK,WAAW;GAChB,KAAK,cAAc;GACnB,KAAK,cAAc;EACrB;CACF;AACF"}
|
package/dist/docs/SKILL.md
CHANGED
|
@@ -1,12 +1,6 @@
|
|
|
1
1
|
{
|
|
2
|
-
"version": "1.5.
|
|
2
|
+
"version": "1.5.2",
|
|
3
3
|
"package": "@mastra/duckdb",
|
|
4
|
-
"exports": {
|
|
5
|
-
"DuckDBConnection": {
|
|
6
|
-
"types": "dist/index.d.ts",
|
|
7
|
-
"implementation": "dist/chunk-4LIZE4MC.js",
|
|
8
|
-
"line": 40
|
|
9
|
-
}
|
|
10
|
-
},
|
|
4
|
+
"exports": {},
|
|
11
5
|
"modules": {}
|
|
12
6
|
}
|
|
@@ -110,6 +110,10 @@ const duckdb = new DuckDBStore({ path: ':memory:' })
|
|
|
110
110
|
|
|
111
111
|
**path** (`string`): Path to the DuckDB database file. Use :memory: for an ephemeral in-memory database. (Default: `'mastra.duckdb'`)
|
|
112
112
|
|
|
113
|
+
**memoryLimit** (`string`): Maximum memory DuckDB may use, such as '2GB' or '512MB'. Larger-than-memory operations spill to disk for file-backed databases. Raise this for dedicated analytical workloads. (Default: `'2GB'`)
|
|
114
|
+
|
|
115
|
+
**threads** (`number`): Number of threads DuckDB may use. Lower this to keep queries from monopolizing all cores of a shared application server. (Default: `one per CPU core`)
|
|
116
|
+
|
|
113
117
|
### Lower-level types
|
|
114
118
|
|
|
115
119
|
`@mastra/duckdb` also exports `DuckDBConnection` for sharing a single underlying database across multiple Mastra storage instances, and the corresponding `DuckDBStorageConfig` type. Most applications won't need these directly.
|