@xeplr/actions 2.2.0 → 2.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +29 -1
- package/lib/builtins/index.js +2 -0
- package/lib/builtins/push-send.js +58 -0
- package/lib/drivers/db/duckdb.js +37 -3
- package/lib/drivers/db/mssql.js +32 -4
- package/lib/drivers/db/mysql.js +67 -5
- package/lib/drivers/db/postgres.js +92 -11
- package/lib/drivers/db/stop.js +89 -0
- package/package.json +23 -19
package/README.md
CHANGED
|
@@ -26,6 +26,7 @@ Everything else is an **optional peer**, required only when an action actually u
|
|
|
26
26
|
| `mailparser` | `email-read` (unless `body: 'none'`), `email-download-attachments` |
|
|
27
27
|
| `nodemailer` (^10.0.13 — up to 10.0.5 has high-severity advisories) | `email-send` with its own connection |
|
|
28
28
|
| `@xeplr/email` | `email-send` with `templateName` |
|
|
29
|
+
| `@xeplr/push` | `push-send` (the host sets it up at boot: `push.init({ knex })`) |
|
|
29
30
|
| `knex` | the knex meta store |
|
|
30
31
|
| `mongodb`, `ssh2-sftp-client`, `exceljs` | declared, but their drivers/sources are not implemented |
|
|
31
32
|
|
|
@@ -121,7 +122,7 @@ The input is checked exactly as a run checks it.
|
|
|
121
122
|
|
|
122
123
|
| previewed (reads, up to 5) | example only (writes or sends; never run) |
|
|
123
124
|
|---|---|
|
|
124
|
-
| `db-fetch` (through the run's own cursor, then stops; nothing spooled), `db-list-tables` / `-views` / `-procedures` / `-columns`, `file-list`, `file-read`, `data-operation` (the run itself: it only computes), `email-read` (the mailbox opened **read-only**, IMAP `EXAMINE`, so nothing is marked read) | `file-move`, `file-delete`, `email-send`, `email-move`, `email-delete`, `email-download-email`, `email-download-attachments`, `export-table`, `export-pages`, `db-procedure` (a procedure may write) |
|
|
125
|
+
| `db-fetch` (through the run's own cursor, then stops; nothing spooled), `db-list-tables` / `-views` / `-procedures` / `-columns`, `file-list`, `file-read`, `data-operation` (the run itself: it only computes), `email-read` (the mailbox opened **read-only**, IMAP `EXAMINE`, so nothing is marked read) | `file-move`, `file-delete`, `email-send`, `push-send`, `email-move`, `email-delete`, `email-download-email`, `email-download-attachments`, `export-table`, `export-pages`, `db-procedure` (a procedure may write) |
|
|
125
126
|
|
|
126
127
|
Neither: `db-push`, `db-move`, `http-request` (an outside system, which may change something), `spawn-program`, `file-upload`; a designer asks for an example by hand. `test/previews.test.js` checks it on a real folder, file and Postgres: five rows read and no file spooled, the real count kept, a deleting step's file still there.
|
|
127
128
|
|
|
@@ -144,6 +145,7 @@ Register the ones you want (`register(actions.builtins.dbFetch)`) or pass a modu
|
|
|
144
145
|
| `dbListColumns` | `db-list-columns` | `{ columns }` — `name`, `dataType`, `udtName` and `maxLength` (characters; `-1` is SQL Server `(max)`, `null` when it has none) for one table. |
|
|
145
146
|
| `fileUpload` | `file-upload` | Read a file (`sourceType`, `sourcePath`), parse it (`format`), load it into `targetTable` (`dbType`, `dbConnection`) as a New table, Append or Replace (`mode`, see [Import modes](#import-modes)). Passes `system.onProgress` (`{ rowsRead, batches, rowsWritten }`, per batch) and `system.signal` (stop) to the uploader, as `db-move` does. |
|
|
146
147
|
| `sendEmail` | `email-send` | Send one message. See [Email](#email). |
|
|
148
|
+
| `pushSend` | `push-send` | A push notification on people's phones and computers, even with the app closed, by `emails` and/or `userIds`: `title` (required), `body`, `url` (a page of the app a tap opens), `tag`. Returns `{ devices, sent, removed, failed }`. Through [`@xeplr/push`](https://www.npmjs.com/package/@xeplr/push), which the host sets up. |
|
|
147
149
|
| `emailRead` | `email-read` | List messages, newest first, with `unread_only`, `limit` (1–1000, default 50), `since`, `from_equals`, `from_contains`, `subject_contains`, `body: 'full' \| 'text' \| 'none'`, optional NDJSON spooling. |
|
|
148
150
|
| `emailMove` | `email-move` | Move a message (`messageId`, `folder`) to `toFolder`. |
|
|
149
151
|
| `emailDelete` | `email-delete` | Flag `\Deleted` **and expunge** — permanent, not a move to Trash. |
|
|
@@ -582,6 +584,32 @@ await driver.query(pool, userSql, params) // reads only
|
|
|
582
584
|
|
|
583
585
|
**Values are bound, never written into SQL.** On MySQL, `stringifyObjects` is always on, so an object or array passed as a value is compared as text instead of being expanded into `key = value` pairs. **Procedure parameter names** are written into the call (`@region = @p0`, `region => $1`) and must be plain names — letters, digits and `_`.
|
|
584
586
|
|
|
587
|
+
## Stopping a query — the stop reaches the database
|
|
588
|
+
|
|
589
|
+
A query nobody is waiting for must not keep running. Every driver's `query` and `fetchStream` take a stop signal and a time limit:
|
|
590
|
+
|
|
591
|
+
```js
|
|
592
|
+
const stop = new AbortController()
|
|
593
|
+
await driver.query(pool, sql, params, { signal: stop.signal, timeoutMs: 120000 })
|
|
594
|
+
for await (const row of driver.fetchStream(pool, { sql, params, signal: stop.signal, timeoutMs: 120000 })) { … }
|
|
595
|
+
stop.abort(new Error('Nobody is waiting for this')) // cancels it IN THE DATABASE
|
|
596
|
+
```
|
|
597
|
+
|
|
598
|
+
- **A stop cancels the query in the database, not just our read.** A `GROUP BY` or `ORDER BY` is worked out in full before the first row comes back, so a stop that only ended our read loop left the database working, holding its CPU and a connection, until it finished.
|
|
599
|
+
- **The two errors:** `code: 'QUERY_STOPPED'`, whose message is the signal's reason; and `code: 'QUERY_TIMEOUT'`, "The database stopped this query after 2 minutes, its time limit…".
|
|
600
|
+
- **A signal that has already fired:** the query is never sent.
|
|
601
|
+
- **Neither option given:** the query runs exactly as before.
|
|
602
|
+
|
|
603
|
+
| Driver | How it is stopped | Time limit |
|
|
604
|
+
|---|---|---|
|
|
605
|
+
| postgres | `pg_cancel_backend(<the query's backend>)`, sent on a connection of its own, so a pool whose connections are all busy (often with the very queries being stopped) cannot block it. The stopped query's connection is destroyed, not reused | `statement_timeout`, enforced by Postgres even if this process dies. For `fetchStream` it applies to each FETCH. On a writable connection it is set for the query and reset after, without adding a transaction |
|
|
606
|
+
| duckdb | `interrupt()` on the query's own connection. A query still waiting for a free connection is let go before it runs | a timer that interrupts the same way |
|
|
607
|
+
| mssql | `request.cancel()`, an attention on the query's own connection | a timer that cancels the same way (SQL Server has no per-statement limit) |
|
|
608
|
+
| mysql | `KILL QUERY <thread id>`, sent on a connection of its own. The stopped query's connection is destroyed | a timer that kills the same way (`max_execution_time` covers SELECTs only, and MariaDB names it differently) |
|
|
609
|
+
| mongo | not yet | — |
|
|
610
|
+
|
|
611
|
+
The tests prove that the database itself let go: Postgres by `pg_stat_activity`, SQL Server by `sys.dm_exec_requests`, MySQL by the processlist, and DuckDB by its one connection answering at once (`test/drivers/stop.*.test.js`). The SQL Server and MySQL ones run when `MSSQL_*` and `MYSQL_*` are set.
|
|
612
|
+
|
|
585
613
|
## Rules the code enforces, and why
|
|
586
614
|
|
|
587
615
|
| rule | why |
|
package/lib/builtins/index.js
CHANGED
|
@@ -22,6 +22,8 @@ module.exports = {
|
|
|
22
22
|
dataOperation: require('./data-operation'),
|
|
23
23
|
spawnProgram: require('./spawn-program'),
|
|
24
24
|
sendEmail: require('./send-email'),
|
|
25
|
+
// A notification on people's devices (Web Push) — peer @xeplr/push.
|
|
26
|
+
pushSend: require('./push-send'),
|
|
25
27
|
|
|
26
28
|
// DB actions — factory-shaped (call fileUpload({ metaKnex }) to get a
|
|
27
29
|
// registerable module) so the Knex handle for the meta store binds at
|
|
@@ -0,0 +1,58 @@
|
|
|
1
|
+
// push-send action — a notification on people's phones and computers, even
|
|
2
|
+
// with the app closed, through @xeplr/push (Web Push).
|
|
3
|
+
//
|
|
4
|
+
// Peer dep: `@xeplr/push`, lazy-required here, so an install that never
|
|
5
|
+
// pushes does not need it. The host app sets it up once at boot —
|
|
6
|
+
// push.init({ knex }) with its VAPID keys — and this step only sends.
|
|
7
|
+
//
|
|
8
|
+
// WHO: account ids (`userIds`) and/or emails (`emails`) — every device each of
|
|
9
|
+
// them switched on, each device once. Nobody to send to is refused, rather
|
|
10
|
+
// than "succeeding" with nothing sent.
|
|
11
|
+
//
|
|
12
|
+
// WHERE A TAP GOES: `url` is a page of the app ("/orders/{params.id}"). A full
|
|
13
|
+
// address to another site is replaced by the app's home by @xeplr/push, so a
|
|
14
|
+
// step can never send people off somewhere else.
|
|
15
|
+
|
|
16
|
+
function asList(v) {
|
|
17
|
+
if (v === undefined || v === null || v === '') return [];
|
|
18
|
+
return (Array.isArray(v) ? v : [v]).map(function(x) { return String(x).trim(); }).filter(Boolean);
|
|
19
|
+
}
|
|
20
|
+
|
|
21
|
+
module.exports = {
|
|
22
|
+
name: 'push-send',
|
|
23
|
+
// It sends, so it is never previewed.
|
|
24
|
+
example: { devices: 2, sent: 2, removed: 0, failed: 0 },
|
|
25
|
+
description: 'Send a push notification to people\'s phones and computers (Web Push, through @xeplr/push) — by account id or email. A tap opens the page given.',
|
|
26
|
+
requires: [], // @xeplr/push checked at runtime
|
|
27
|
+
|
|
28
|
+
inputSchema: [
|
|
29
|
+
{ name: 'emails', type: 'array', order: 1,
|
|
30
|
+
description: 'Who gets it, by email. A bare string is accepted and wrapped.' },
|
|
31
|
+
{ name: 'userIds', type: 'array', order: 2,
|
|
32
|
+
description: 'Who gets it, by account id — instead of, or as well as, emails.' },
|
|
33
|
+
{ name: 'title', type: 'string', required: true, order: 3,
|
|
34
|
+
description: 'The first line, e.g. "RTO: #{params.orderNumber}". At most 120 characters.' },
|
|
35
|
+
{ name: 'body', type: 'string', order: 4,
|
|
36
|
+
description: 'Under it. At most 300 characters.' },
|
|
37
|
+
{ name: 'url', type: 'string', order: 5,
|
|
38
|
+
description: 'The page of the app a tap opens, e.g. /orders/{params.id}. Leave empty for the home page.' },
|
|
39
|
+
{ name: 'tag', type: 'string', order: 6, group: 'Message options',
|
|
40
|
+
description: 'A new notification with the same tag replaces the one shown, so repeats stack as one.' }
|
|
41
|
+
],
|
|
42
|
+
|
|
43
|
+
execute: async function(ctx) {
|
|
44
|
+
var input = ctx.input || {};
|
|
45
|
+
var push;
|
|
46
|
+
try {
|
|
47
|
+
push = require('@xeplr/push');
|
|
48
|
+
} catch (_) {
|
|
49
|
+
throw new Error("push-send: '@xeplr/push' is not installed. Run: npm install @xeplr/push — and set it up at boot with push.init({ knex }).");
|
|
50
|
+
}
|
|
51
|
+
if (!push.enabled || !push.enabled()) {
|
|
52
|
+
throw new Error('push-send: push is not set up on this server — the host must call push.init() with PUSH_VAPID_PUBLIC_KEY / PUSH_VAPID_PRIVATE_KEY / PUSH_VAPID_SUBJECT (npx xeplr-push keys).');
|
|
53
|
+
}
|
|
54
|
+
var target = { emails: asList(input.emails), userIds: asList(input.userIds) };
|
|
55
|
+
if (!target.emails.length && !target.userIds.length) throw new Error('push-send: say who gets it — emails or userIds.');
|
|
56
|
+
return push.send(target, { title: input.title, body: input.body, url: input.url, tag: input.tag });
|
|
57
|
+
}
|
|
58
|
+
};
|
package/lib/drivers/db/duckdb.js
CHANGED
|
@@ -349,6 +349,33 @@ async function acquire(entry) {
|
|
|
349
349
|
return new Promise(function(resolve) { entry.waiters.push(resolve); });
|
|
350
350
|
}
|
|
351
351
|
|
|
352
|
+
// ── STOPPING A QUERY (stop.js) ─────────────────────────────────────────────
|
|
353
|
+
// DuckDB runs in this process, so a stop is conn.interrupt() on the query's
|
|
354
|
+
// own connection, and the time limit is a timer that does the same. A query
|
|
355
|
+
// still waiting for a free connection is let go of before it ever runs.
|
|
356
|
+
var stop = require('./stop');
|
|
357
|
+
|
|
358
|
+
/** acquire(), unless the stop comes first; a connection that arrives too late goes straight back. */
|
|
359
|
+
function acquireUnlessStopped(entry, signal) {
|
|
360
|
+
if (!signal) return acquire(entry);
|
|
361
|
+
stop.throwIfStopped(signal);
|
|
362
|
+
return new Promise(function(resolve, reject) {
|
|
363
|
+
var gaveUp = false;
|
|
364
|
+
function onAbort() { gaveUp = true; reject(stop.stoppedError(signal)); }
|
|
365
|
+
signal.addEventListener('abort', onAbort, { once: true });
|
|
366
|
+
acquire(entry).then(function(conn) {
|
|
367
|
+
signal.removeEventListener('abort', onAbort);
|
|
368
|
+
if (gaveUp) release(entry, conn);
|
|
369
|
+
else resolve(conn);
|
|
370
|
+
}, function(err) { signal.removeEventListener('abort', onAbort); reject(err); });
|
|
371
|
+
});
|
|
372
|
+
}
|
|
373
|
+
|
|
374
|
+
/** Watch one connection's query: interrupt it on a stop or past the limit. */
|
|
375
|
+
function watchConn(conn, opts) {
|
|
376
|
+
return stop.watch(opts && opts.signal, opts && opts.timeoutMs, function() { conn.interrupt(); });
|
|
377
|
+
}
|
|
378
|
+
|
|
352
379
|
function release(entry, conn) {
|
|
353
380
|
var waiter = entry.waiters.shift();
|
|
354
381
|
if (waiter) waiter(conn);
|
|
@@ -421,10 +448,11 @@ function plainDeep(v) {
|
|
|
421
448
|
* The result's OWN return type is what distinguishes the two cases, so that is
|
|
422
449
|
* what this reads. CHANGED_ROWS is DML; QUERY_RESULT is a select.
|
|
423
450
|
*/
|
|
424
|
-
async function query(pool, sql, params) {
|
|
451
|
+
async function query(pool, sql, params, opts) {
|
|
425
452
|
var entry = entryOf(pool);
|
|
426
453
|
refuseIndex(entry, sql);
|
|
427
|
-
var conn = await
|
|
454
|
+
var conn = await acquireUnlessStopped(entry, opts && opts.signal);
|
|
455
|
+
var w = watchConn(conn, opts);
|
|
428
456
|
try {
|
|
429
457
|
var api = require('@duckdb/node-api');
|
|
430
458
|
var reader = await conn.runAndReadAll(sql, params && params.length ? params : undefined);
|
|
@@ -441,8 +469,10 @@ async function query(pool, sql, params) {
|
|
|
441
469
|
});
|
|
442
470
|
return { rows: rows, rowCount: rows.length, columns: columns };
|
|
443
471
|
} catch (err) {
|
|
472
|
+
if (w.why()) throw w.why();
|
|
444
473
|
throw explainLockError(err, entry.key, entry.access, 0);
|
|
445
474
|
} finally {
|
|
475
|
+
w.done();
|
|
446
476
|
release(entry, conn);
|
|
447
477
|
}
|
|
448
478
|
}
|
|
@@ -756,7 +786,8 @@ async function plainValuesPlan(conn, sql) {
|
|
|
756
786
|
*/
|
|
757
787
|
async function* fetchStream(pool, opts) {
|
|
758
788
|
var entry = entryOf(pool);
|
|
759
|
-
var conn = await
|
|
789
|
+
var conn = await acquireUnlessStopped(entry, opts.signal);
|
|
790
|
+
var w = watchConn(conn, opts);
|
|
760
791
|
try {
|
|
761
792
|
var plan = await plainValuesPlan(conn, opts.sql);
|
|
762
793
|
var result = await conn.stream(plan.sql, (opts.params && opts.params.length) ? opts.params : undefined);
|
|
@@ -764,6 +795,7 @@ async function* fetchStream(pool, opts) {
|
|
|
764
795
|
var dates = plan.dateColumns;
|
|
765
796
|
var jsons = plan.jsonColumns;
|
|
766
797
|
while (true) {
|
|
798
|
+
if (w.why()) throw w.why();
|
|
767
799
|
var chunk = await result.fetchChunk();
|
|
768
800
|
if (!chunk || chunk.rowCount === 0) break;
|
|
769
801
|
var rows = chunk.getRowObjects(names);
|
|
@@ -783,8 +815,10 @@ async function* fetchStream(pool, opts) {
|
|
|
783
815
|
}
|
|
784
816
|
}
|
|
785
817
|
} catch (err) {
|
|
818
|
+
if (w.why()) throw w.why();
|
|
786
819
|
throw explainLockError(err, entry.key, entry.access, 0);
|
|
787
820
|
} finally {
|
|
821
|
+
w.done();
|
|
788
822
|
release(entry, conn);
|
|
789
823
|
}
|
|
790
824
|
}
|
package/lib/drivers/db/mssql.js
CHANGED
|
@@ -96,17 +96,34 @@ async function close(pool) {
|
|
|
96
96
|
if (pool) await pool.close();
|
|
97
97
|
}
|
|
98
98
|
|
|
99
|
+
// ── STOPPING A QUERY (stop.js) ─────────────────────────────────────────────
|
|
100
|
+
// A stop is request.cancel(): an ATTENTION sent on the query's own
|
|
101
|
+
// connection, which SQL Server answers by abandoning the batch. SQL Server
|
|
102
|
+
// has no per-statement time limit of its own, so the limit is a timer here
|
|
103
|
+
// that cancels the same way.
|
|
104
|
+
var stop = require('./stop');
|
|
105
|
+
|
|
99
106
|
// Bind params as @p0..@pN and normalize `{ recordset, rowsAffected }` to
|
|
100
107
|
// Postgres shape. NOTE: mssql infers the SQL type from each JS value; a null
|
|
101
108
|
// with no sibling non-null value in the batch falls back to NVARCHAR, which is
|
|
102
109
|
// fine for our INSERTs since the column type is already declared.
|
|
103
|
-
async function query(pool, sql, params) {
|
|
110
|
+
async function query(pool, sql, params, opts) {
|
|
111
|
+
opts = opts || {};
|
|
112
|
+
stop.throwIfStopped(opts.signal);
|
|
104
113
|
var request = pool.request();
|
|
105
114
|
params = params || [];
|
|
106
115
|
for (var i = 0; i < params.length; i++) {
|
|
107
116
|
request.input('p' + i, params[i]);
|
|
108
117
|
}
|
|
109
|
-
var
|
|
118
|
+
var w = stop.watch(opts.signal, opts.timeoutMs, function() { request.cancel(); });
|
|
119
|
+
var r;
|
|
120
|
+
try {
|
|
121
|
+
r = await request.query(sql);
|
|
122
|
+
} catch (err) {
|
|
123
|
+
throw w.why() || err;
|
|
124
|
+
} finally {
|
|
125
|
+
w.done();
|
|
126
|
+
}
|
|
110
127
|
var rows = r.recordset || [];
|
|
111
128
|
var rowCount = Array.isArray(r.rowsAffected)
|
|
112
129
|
? r.rowsAffected.reduce(function(a, b) { return a + b; }, 0)
|
|
@@ -382,8 +399,9 @@ async function ensureUpsertIndex(pool, tableName, primaryKeys) {
|
|
|
382
399
|
// with real backpressure: pause the request once `batchSize` rows are buffered,
|
|
383
400
|
// resume after the consumer drains them.
|
|
384
401
|
//
|
|
385
|
-
// opts: { sql, params?, batchSize? } → async iterable<row>
|
|
402
|
+
// opts: { sql, params?, batchSize?, signal?, timeoutMs? } → async iterable<row>
|
|
386
403
|
async function* fetchStream(pool, opts) {
|
|
404
|
+
stop.throwIfStopped(opts.signal);
|
|
387
405
|
var high = Math.max(1, parseInt(opts.batchSize || 1000, 10));
|
|
388
406
|
var request = pool.request();
|
|
389
407
|
var params = opts.params || [];
|
|
@@ -406,12 +424,21 @@ async function* fetchStream(pool, opts) {
|
|
|
406
424
|
request.on('error', function(err) { error = err; signal(); });
|
|
407
425
|
request.on('done', function() { done = true; signal(); });
|
|
408
426
|
|
|
427
|
+
// A stop cancels the request and wakes the consumer, which may be waiting
|
|
428
|
+
// for a first row the server is still working out.
|
|
429
|
+
var w = stop.watch(opts.signal, opts.timeoutMs, function() {
|
|
430
|
+
try { request.cancel(); } catch (_) {}
|
|
431
|
+
error = error || w.why();
|
|
432
|
+
signal();
|
|
433
|
+
});
|
|
434
|
+
|
|
409
435
|
// Fire the query — with stream=true this drives the events above. We don't
|
|
410
436
|
// await the returned promise; the events are the source of truth.
|
|
411
437
|
request.query(opts.sql).catch(function(err) { error = error || err; signal(); });
|
|
412
438
|
|
|
413
439
|
try {
|
|
414
440
|
while (true) {
|
|
441
|
+
if (w.why()) throw w.why();
|
|
415
442
|
if (buffer.length) {
|
|
416
443
|
var batch = buffer;
|
|
417
444
|
buffer = [];
|
|
@@ -419,11 +446,12 @@ async function* fetchStream(pool, opts) {
|
|
|
419
446
|
for (var j = 0; j < batch.length; j++) yield batch[j];
|
|
420
447
|
continue;
|
|
421
448
|
}
|
|
422
|
-
if (error) throw error;
|
|
449
|
+
if (error) throw w.why() || error;
|
|
423
450
|
if (done) return;
|
|
424
451
|
await new Promise(function(res) { wake = res; });
|
|
425
452
|
}
|
|
426
453
|
} finally {
|
|
454
|
+
w.done();
|
|
427
455
|
// If the consumer breaks out early, stop the underlying request.
|
|
428
456
|
if (!done && !error) { try { request.cancel(); } catch (_) {} }
|
|
429
457
|
}
|
package/lib/drivers/db/mysql.js
CHANGED
|
@@ -106,13 +106,67 @@ async function close(pool) {
|
|
|
106
106
|
if (pool) await pool.end();
|
|
107
107
|
}
|
|
108
108
|
|
|
109
|
+
// ── STOPPING A QUERY (stop.js) ─────────────────────────────────────────────
|
|
110
|
+
// A stop is KILL QUERY <the connection's thread id>, sent on a connection of
|
|
111
|
+
// its own: the pool's may all be busy with the very queries being stopped.
|
|
112
|
+
// The time limit is a timer here that kills the same way — MySQL's own
|
|
113
|
+
// max_execution_time covers SELECTs only, and MariaDB spells it differently.
|
|
114
|
+
var stop = require('./stop');
|
|
115
|
+
|
|
116
|
+
/** Kill whatever thread `threadId` is running, from a connection of its own. */
|
|
117
|
+
async function killQuery(pool, threadId) {
|
|
118
|
+
var mysql = require('mysql2/promise');
|
|
119
|
+
var c = pool.pool.config.connectionConfig;
|
|
120
|
+
var conn = await mysql.createConnection({ host: c.host, port: c.port, user: c.user, password: c.password, connectTimeout: 5000 });
|
|
121
|
+
try {
|
|
122
|
+
await conn.query('KILL QUERY ' + Number(threadId));
|
|
123
|
+
} finally {
|
|
124
|
+
await conn.end().catch(function() {});
|
|
125
|
+
}
|
|
126
|
+
}
|
|
127
|
+
|
|
128
|
+
/** Watch one checked-out connection's query; `killed` is the pending kill. */
|
|
129
|
+
function watchConn(pool, conn, opts) {
|
|
130
|
+
var state = { killed: null };
|
|
131
|
+
state.w = stop.watch(opts.signal, opts.timeoutMs, function() {
|
|
132
|
+
state.killed = killQuery(pool, conn.connection.threadId).catch(function() { /* it finished first, or the server is gone */ });
|
|
133
|
+
});
|
|
134
|
+
return state;
|
|
135
|
+
}
|
|
136
|
+
|
|
137
|
+
/** Give the connection back — destroyed when a query on it was stopped. */
|
|
138
|
+
async function giveBack(state, conn) {
|
|
139
|
+
state.w.done();
|
|
140
|
+
if (state.killed) await state.killed;
|
|
141
|
+
if (state.w.why()) conn.destroy();
|
|
142
|
+
else conn.release();
|
|
143
|
+
}
|
|
144
|
+
|
|
109
145
|
// Normalize mysql2's `[rows|ResultSetHeader, fields]` to Postgres shape.
|
|
110
146
|
//
|
|
111
147
|
// The second element is the field list, which mysql2 returns even for a
|
|
112
148
|
// SELECT that matched nothing — so `columns` describes the result's shape
|
|
113
149
|
// when there is not a single row to infer it from.
|
|
114
|
-
async function query(pool, sql, params) {
|
|
115
|
-
|
|
150
|
+
async function query(pool, sql, params, opts) {
|
|
151
|
+
opts = opts || {};
|
|
152
|
+
stop.throwIfStopped(opts.signal);
|
|
153
|
+
var res;
|
|
154
|
+
if (opts.signal || stop.limitOf(opts.timeoutMs)) {
|
|
155
|
+
var conn = await pool.getConnection();
|
|
156
|
+
var state = watchConn(pool, conn, opts);
|
|
157
|
+
try {
|
|
158
|
+
res = await conn.query(sql, params || []);
|
|
159
|
+
// A killed statement can still come back normally (SLEEP() answers 1
|
|
160
|
+
// when killed): stopped is stopped, whatever it returned.
|
|
161
|
+
if (state.w.why()) throw state.w.why();
|
|
162
|
+
} catch (err) {
|
|
163
|
+
throw state.w.why() || err;
|
|
164
|
+
} finally {
|
|
165
|
+
await giveBack(state, conn);
|
|
166
|
+
}
|
|
167
|
+
} else {
|
|
168
|
+
res = await pool.query(sql, params || []);
|
|
169
|
+
}
|
|
116
170
|
var out = res[0];
|
|
117
171
|
var fields = res[1];
|
|
118
172
|
|
|
@@ -412,10 +466,12 @@ function upsertIndexName(tableName) {
|
|
|
412
466
|
// `for await` over it applies backpressure (highWaterMark bounds the rows
|
|
413
467
|
// buffered in the driver before it pauses reading from the socket).
|
|
414
468
|
//
|
|
415
|
-
// opts: { sql, params?, batchSize? } → async iterable<row>
|
|
469
|
+
// opts: { sql, params?, batchSize?, signal?, timeoutMs? } → async iterable<row>
|
|
416
470
|
async function* fetchStream(pool, opts) {
|
|
471
|
+
stop.throwIfStopped(opts.signal);
|
|
417
472
|
var batchSize = Math.max(1, parseInt(opts.batchSize || 1000, 10));
|
|
418
473
|
var conn = await pool.getConnection();
|
|
474
|
+
var state = watchConn(pool, conn, opts);
|
|
419
475
|
try {
|
|
420
476
|
// .stream() lives on the core (non-promise) connection under .connection.
|
|
421
477
|
var core = conn.connection;
|
|
@@ -423,9 +479,15 @@ async function* fetchStream(pool, opts) {
|
|
|
423
479
|
? { sql: opts.sql, values: opts.params }
|
|
424
480
|
: opts.sql;
|
|
425
481
|
var stream = core.query(q).stream({ highWaterMark: batchSize });
|
|
426
|
-
for await (var row of stream)
|
|
482
|
+
for await (var row of stream) {
|
|
483
|
+
if (state.w.why()) throw state.w.why();
|
|
484
|
+
yield row;
|
|
485
|
+
}
|
|
486
|
+
if (state.w.why()) throw state.w.why();
|
|
487
|
+
} catch (err) {
|
|
488
|
+
throw state.w.why() || err;
|
|
427
489
|
} finally {
|
|
428
|
-
conn
|
|
490
|
+
await giveBack(state, conn);
|
|
429
491
|
}
|
|
430
492
|
}
|
|
431
493
|
|
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
//
|
|
9
9
|
// requires: ['pg']
|
|
10
10
|
// async connect(config) → pool
|
|
11
|
-
// async query(pool, sql, params?)
|
|
11
|
+
// async query(pool, sql, params?, { signal?, timeoutMs }?) → { rows, rowCount, ... }
|
|
12
12
|
// async close(pool) → void
|
|
13
13
|
//
|
|
14
14
|
// buildCreateTableSql(tableName, columns, primaryKeys?) → string
|
|
@@ -82,6 +82,56 @@ async function close(pool) {
|
|
|
82
82
|
if (pool) await pool.end();
|
|
83
83
|
}
|
|
84
84
|
|
|
85
|
+
// ── STOPPING A QUERY (stop.js) ─────────────────────────────────────────────
|
|
86
|
+
// A stop is pg_cancel_backend(<its backend's pid>), sent on a connection of
|
|
87
|
+
// its own: the pool's may all be busy — quite possibly with the very queries
|
|
88
|
+
// being stopped — and a cancel that waits for a free one never arrives. The
|
|
89
|
+
// time limit is statement_timeout, so Postgres enforces it even if this
|
|
90
|
+
// process dies; our own timer, a little later, is only the backstop.
|
|
91
|
+
var stop = require('./stop');
|
|
92
|
+
var QUERY_CANCELED = '57014';
|
|
93
|
+
var BACKSTOP_MS = 2000;
|
|
94
|
+
|
|
95
|
+
/** Cancel whatever `pid` is running, from a connection of its own. */
|
|
96
|
+
async function cancelBackend(pool, pid) {
|
|
97
|
+
var pg = require('pg');
|
|
98
|
+
var o = pool.options || {};
|
|
99
|
+
var c = new pg.Client({ host: o.host, port: o.port, user: o.user, password: o.password, database: o.database, connectionTimeoutMillis: 5000 });
|
|
100
|
+
try {
|
|
101
|
+
await c.connect();
|
|
102
|
+
await c.query('SELECT pg_cancel_backend($1)', [pid]);
|
|
103
|
+
} finally {
|
|
104
|
+
await c.end().catch(function() {});
|
|
105
|
+
}
|
|
106
|
+
}
|
|
107
|
+
|
|
108
|
+
/**
|
|
109
|
+
* Watch one checked-out client's query for `opts` (signal, timeoutMs).
|
|
110
|
+
* `cancelled` is the pending cancel, awaited before the client goes back.
|
|
111
|
+
*/
|
|
112
|
+
function watchClient(pool, client, opts) {
|
|
113
|
+
var limit = stop.limitOf(opts.timeoutMs);
|
|
114
|
+
var state = { cancelled: null, limit: limit };
|
|
115
|
+
state.w = stop.watch(opts.signal, limit ? limit + BACKSTOP_MS : 0, function() {
|
|
116
|
+
state.cancelled = cancelBackend(pool, client.processID).catch(function() { /* it finished first, or the server is gone */ });
|
|
117
|
+
});
|
|
118
|
+
return state;
|
|
119
|
+
}
|
|
120
|
+
|
|
121
|
+
/** The error to reject with: ours when we stopped it, the time limit when Postgres did. */
|
|
122
|
+
function stopReason(state, err) {
|
|
123
|
+
if (state.w.why()) return state.w.why();
|
|
124
|
+
if (err && err.code === QUERY_CANCELED && state.limit) return stop.timeoutError(state.limit);
|
|
125
|
+
return err;
|
|
126
|
+
}
|
|
127
|
+
|
|
128
|
+
/** Give the client back — destroyed when a query on it was stopped, so no half-cancelled state is reused. */
|
|
129
|
+
async function giveBack(state, client) {
|
|
130
|
+
state.w.done();
|
|
131
|
+
if (state.cancelled) await state.cancelled;
|
|
132
|
+
client.release(state.w.why() ? true : undefined);
|
|
133
|
+
}
|
|
134
|
+
|
|
85
135
|
// `columns` is the query's OUTPUT COLUMNS, which the server describes whether
|
|
86
136
|
// or not any row matched. Callers that need the shape of a result — a report
|
|
87
137
|
// listing its columns, say — must not have to infer it from the first row,
|
|
@@ -90,16 +140,33 @@ async function close(pool) {
|
|
|
90
140
|
// Attached to pg's own Result rather than wrapping it: callers already treat
|
|
91
141
|
// this as the canonical shape the other drivers normalize TO, so replacing
|
|
92
142
|
// the object risks dropping a property something else reads.
|
|
93
|
-
async function query(pool, sql, params) {
|
|
143
|
+
async function query(pool, sql, params, opts) {
|
|
144
|
+
opts = opts || {};
|
|
145
|
+
stop.throwIfStopped(opts.signal);
|
|
94
146
|
var result;
|
|
95
|
-
|
|
147
|
+
var limit = stop.limitOf(opts.timeoutMs);
|
|
148
|
+
if (pool.__xeplrReadOnly || opts.signal || limit) {
|
|
149
|
+
var readOnly = Boolean(pool.__xeplrReadOnly);
|
|
96
150
|
var client = await pool.connect();
|
|
151
|
+
var state = watchClient(pool, client, opts);
|
|
97
152
|
try {
|
|
98
|
-
|
|
99
|
-
|
|
153
|
+
// Read-only: inside its own transaction, the limit with it (LOCAL).
|
|
154
|
+
// Otherwise the statement runs as it always did — no transaction added,
|
|
155
|
+
// which some statements refuse — and the limit is on the session until
|
|
156
|
+
// it is reset below.
|
|
157
|
+
if (readOnly) await client.query('BEGIN TRANSACTION READ ONLY');
|
|
158
|
+
if (limit) await client.query((readOnly ? 'SET LOCAL' : 'SET') + ' statement_timeout = ' + limit);
|
|
159
|
+
result = await client.query(readOnly ? { text: sql, values: params || [], queryMode: 'extended' } : sql, readOnly ? undefined : (params || []));
|
|
160
|
+
} catch (err) {
|
|
161
|
+
throw stopReason(state, err);
|
|
100
162
|
} finally {
|
|
101
|
-
|
|
102
|
-
|
|
163
|
+
if (!state.w.why()) {
|
|
164
|
+
try {
|
|
165
|
+
if (readOnly) await client.query('ROLLBACK');
|
|
166
|
+
else if (limit) await client.query('RESET statement_timeout');
|
|
167
|
+
} catch (_) { /* the connection is released either way */ }
|
|
168
|
+
}
|
|
169
|
+
await giveBack(state, client);
|
|
103
170
|
}
|
|
104
171
|
} else {
|
|
105
172
|
result = await pool.query(sql, params || []);
|
|
@@ -342,14 +409,23 @@ async function listProcedures(pool, schema) {
|
|
|
342
409
|
// rows at a time. The async generator naturally applies backpressure: the next
|
|
343
410
|
// FETCH only fires when the consumer asks for more.
|
|
344
411
|
//
|
|
345
|
-
// opts: { sql, params?, batchSize? } → async iterable<row>
|
|
412
|
+
// opts: { sql, params?, batchSize?, signal?, timeoutMs? } → async iterable<row>
|
|
413
|
+
//
|
|
414
|
+
// A GROUP BY or ORDER BY is worked out in full by the FIRST FETCH, which can
|
|
415
|
+
// take minutes — so a stop must cancel that FETCH in Postgres, not wait for
|
|
416
|
+
// it to come back. timeoutMs is per statement (each FETCH), as Postgres
|
|
417
|
+
// counts it.
|
|
346
418
|
async function* fetchStream(pool, opts) {
|
|
419
|
+
stop.throwIfStopped(opts.signal);
|
|
347
420
|
var batchSize = Math.max(1, parseInt(opts.batchSize || 1000, 10));
|
|
421
|
+
var limit = stop.limitOf(opts.timeoutMs);
|
|
348
422
|
var client = await pool.connect();
|
|
423
|
+
var state = watchClient(pool, client, opts);
|
|
349
424
|
var declared = false;
|
|
350
425
|
try {
|
|
351
426
|
var readOnly = Boolean(pool.__xeplrReadOnly);
|
|
352
427
|
await client.query(readOnly ? 'BEGIN TRANSACTION READ ONLY' : 'BEGIN');
|
|
428
|
+
if (limit) await client.query('SET LOCAL statement_timeout = ' + limit);
|
|
353
429
|
// Extended protocol when read-only: one statement, so the caller's SQL
|
|
354
430
|
// cannot close the cursor declaration and carry on with its own.
|
|
355
431
|
await client.query(readOnly
|
|
@@ -357,14 +433,19 @@ async function* fetchStream(pool, opts) {
|
|
|
357
433
|
: 'DECLARE __xeplr_fetch_cur NO SCROLL CURSOR FOR ' + opts.sql, readOnly ? undefined : (opts.params || []));
|
|
358
434
|
declared = true;
|
|
359
435
|
while (true) {
|
|
436
|
+
if (state.w.why()) throw state.w.why();
|
|
360
437
|
var res = await client.query('FETCH FORWARD ' + batchSize + ' FROM __xeplr_fetch_cur');
|
|
361
438
|
if (res.rows.length === 0) break;
|
|
362
439
|
for (var i = 0; i < res.rows.length; i++) yield res.rows[i];
|
|
363
440
|
}
|
|
441
|
+
} catch (err) {
|
|
442
|
+
throw stopReason(state, err);
|
|
364
443
|
} finally {
|
|
365
|
-
|
|
366
|
-
|
|
367
|
-
|
|
444
|
+
if (!state.w.why()) {
|
|
445
|
+
try { if (declared) await client.query('CLOSE __xeplr_fetch_cur'); } catch (_) {}
|
|
446
|
+
try { await client.query(pool.__xeplrReadOnly ? 'ROLLBACK' : 'COMMIT'); } catch (_) {}
|
|
447
|
+
}
|
|
448
|
+
await giveBack(state, client);
|
|
368
449
|
}
|
|
369
450
|
}
|
|
370
451
|
|
|
@@ -0,0 +1,89 @@
|
|
|
1
|
+
// STOPPING A QUERY — the part every driver shares.
|
|
2
|
+
//
|
|
3
|
+
// A query nobody is waiting for must not run (see CLAUDE.md, "No query runs
|
|
4
|
+
// that nobody needs"). Every driver's query() and fetchStream() take
|
|
5
|
+
//
|
|
6
|
+
// signal an AbortSignal: when it fires, the query is CANCELLED IN THE
|
|
7
|
+
// DATABASE (each driver's own way), not just abandoned here —
|
|
8
|
+
// a GROUP BY the database is still working on would otherwise
|
|
9
|
+
// keep its CPU and its connection until it finished
|
|
10
|
+
// timeoutMs a time limit; past it the query is cancelled the same way
|
|
11
|
+
//
|
|
12
|
+
// and reject with one of two errors a caller can tell apart:
|
|
13
|
+
//
|
|
14
|
+
// code QUERY_STOPPED the signal fired; the message is its reason's
|
|
15
|
+
// code QUERY_TIMEOUT the time limit passed
|
|
16
|
+
//
|
|
17
|
+
// Neither is given: the query runs as it always did.
|
|
18
|
+
|
|
19
|
+
/** The error a stopped query rejects with. */
|
|
20
|
+
function stoppedError(signal) {
|
|
21
|
+
var reason = signal && signal.reason;
|
|
22
|
+
// abort() with no reason gives a generic AbortError: say it in our words.
|
|
23
|
+
var said = reason && reason.name !== 'AbortError' && reason.message;
|
|
24
|
+
var err = new Error(said || 'The query was stopped');
|
|
25
|
+
err.code = 'QUERY_STOPPED';
|
|
26
|
+
return err;
|
|
27
|
+
}
|
|
28
|
+
|
|
29
|
+
/** "45 seconds", "2 minutes" — how long a limit is, as a person says it. */
|
|
30
|
+
function durationText(ms) {
|
|
31
|
+
var secs = Math.max(1, Math.round(ms / 1000));
|
|
32
|
+
if (secs < 60) return secs + ' second' + (secs === 1 ? '' : 's');
|
|
33
|
+
var mins = Math.round(secs / 60);
|
|
34
|
+
return mins + ' minute' + (mins === 1 ? '' : 's');
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
/** The error a query past its time limit rejects with. */
|
|
38
|
+
function timeoutError(timeoutMs) {
|
|
39
|
+
var err = new Error('The database stopped this query after ' + durationText(timeoutMs) +
|
|
40
|
+
', its time limit. Narrow it with a filter or a shorter date range, then run it again.');
|
|
41
|
+
err.code = 'QUERY_TIMEOUT';
|
|
42
|
+
return err;
|
|
43
|
+
}
|
|
44
|
+
|
|
45
|
+
/** A positive whole number of milliseconds, or 0 for no limit. */
|
|
46
|
+
function limitOf(timeoutMs) {
|
|
47
|
+
var n = Math.floor(Number(timeoutMs));
|
|
48
|
+
return Number.isFinite(n) && n > 0 ? n : 0;
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
/**
|
|
52
|
+
* Watch one query: call `cancel()` once — when `signal` fires or `timeoutMs`
|
|
53
|
+
* passes, whichever is first — and remember why. `why()` is the error to
|
|
54
|
+
* reject with after the database has given up the query (null: not stopped).
|
|
55
|
+
* `done()` stops watching; call it in a finally.
|
|
56
|
+
*/
|
|
57
|
+
function watch(signal, timeoutMs, cancel) {
|
|
58
|
+
var limit = limitOf(timeoutMs);
|
|
59
|
+
var reason = null;
|
|
60
|
+
var timer = null;
|
|
61
|
+
function fire(err) {
|
|
62
|
+
if (reason) return;
|
|
63
|
+
reason = err;
|
|
64
|
+
try { cancel(); } catch (_) { /* the driver's cancel reports its own failures */ }
|
|
65
|
+
}
|
|
66
|
+
function onAbort() { fire(stoppedError(signal)); }
|
|
67
|
+
if (signal) {
|
|
68
|
+
if (signal.aborted) onAbort();
|
|
69
|
+
else signal.addEventListener('abort', onAbort, { once: true });
|
|
70
|
+
}
|
|
71
|
+
if (limit && !reason) {
|
|
72
|
+
timer = setTimeout(function() { fire(timeoutError(limit)); }, limit);
|
|
73
|
+
if (timer.unref) timer.unref();
|
|
74
|
+
}
|
|
75
|
+
return {
|
|
76
|
+
why: function() { return reason; },
|
|
77
|
+
done: function() {
|
|
78
|
+
if (timer) clearTimeout(timer);
|
|
79
|
+
if (signal) signal.removeEventListener('abort', onAbort);
|
|
80
|
+
}
|
|
81
|
+
};
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
/** Throw now when the signal has already fired: never start a query nobody wants. */
|
|
85
|
+
function throwIfStopped(signal) {
|
|
86
|
+
if (signal && signal.aborted) throw stoppedError(signal);
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
module.exports = { watch: watch, throwIfStopped: throwIfStopped, stoppedError: stoppedError, timeoutError: timeoutError, limitOf: limitOf, durationText: durationText };
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@xeplr/actions",
|
|
3
|
-
"version": "2.2.
|
|
3
|
+
"version": "2.2.2",
|
|
4
4
|
"description": "Named-action registry with typed inputs and safe invocation — plus built-in actions for HTTP, DB, files, email, spawn",
|
|
5
5
|
"main": "index.js",
|
|
6
6
|
"scripts": {
|
|
@@ -42,6 +42,7 @@
|
|
|
42
42
|
"@duckdb/node-api": "1.5.5-r.3",
|
|
43
43
|
"@xeplr/db": "^1.1.0",
|
|
44
44
|
"@xeplr/email": "^1.0.0",
|
|
45
|
+
"@xeplr/push": "^1.0.0",
|
|
45
46
|
"@xeplr/utils": "^1.0.0",
|
|
46
47
|
"csv-parse": "^7.0.2",
|
|
47
48
|
"dejavu-fonts-ttf": "^2.37.3",
|
|
@@ -61,61 +62,64 @@
|
|
|
61
62
|
"unzipper": "^0.12.5"
|
|
62
63
|
},
|
|
63
64
|
"peerDependenciesMeta": {
|
|
64
|
-
"@xeplr/utils": {
|
|
65
|
-
"optional": true
|
|
66
|
-
},
|
|
67
65
|
"@duckdb/node-api": {
|
|
68
66
|
"optional": true
|
|
69
67
|
},
|
|
70
|
-
"
|
|
68
|
+
"@xeplr/email": {
|
|
71
69
|
"optional": true
|
|
72
70
|
},
|
|
73
|
-
"
|
|
71
|
+
"@xeplr/push": {
|
|
74
72
|
"optional": true
|
|
75
73
|
},
|
|
76
|
-
"
|
|
74
|
+
"@xeplr/utils": {
|
|
77
75
|
"optional": true
|
|
78
76
|
},
|
|
79
|
-
"
|
|
77
|
+
"csv-parse": {
|
|
80
78
|
"optional": true
|
|
81
79
|
},
|
|
82
|
-
"
|
|
80
|
+
"dejavu-fonts-ttf": {
|
|
83
81
|
"optional": true
|
|
84
82
|
},
|
|
85
83
|
"exceljs": {
|
|
86
84
|
"optional": true
|
|
87
85
|
},
|
|
88
|
-
"
|
|
86
|
+
"fflate": {
|
|
89
87
|
"optional": true
|
|
90
88
|
},
|
|
91
|
-
"
|
|
89
|
+
"imapflow": {
|
|
92
90
|
"optional": true
|
|
93
91
|
},
|
|
94
|
-
"
|
|
92
|
+
"knex": {
|
|
95
93
|
"optional": true
|
|
96
94
|
},
|
|
97
|
-
"
|
|
95
|
+
"mailparser": {
|
|
98
96
|
"optional": true
|
|
99
97
|
},
|
|
100
|
-
"
|
|
98
|
+
"mongodb": {
|
|
99
|
+
"optional": true
|
|
100
|
+
},
|
|
101
|
+
"mssql": {
|
|
102
|
+
"optional": true
|
|
103
|
+
},
|
|
104
|
+
"mysql2": {
|
|
101
105
|
"optional": true
|
|
102
106
|
},
|
|
103
107
|
"nodemailer": {
|
|
104
108
|
"optional": true
|
|
105
109
|
},
|
|
106
|
-
"
|
|
110
|
+
"pdfkit": {
|
|
107
111
|
"optional": true
|
|
108
112
|
},
|
|
109
|
-
"
|
|
113
|
+
"pg": {
|
|
110
114
|
"optional": true
|
|
111
115
|
},
|
|
112
|
-
"
|
|
116
|
+
"saxes": {
|
|
113
117
|
"optional": true
|
|
114
118
|
},
|
|
115
|
-
"
|
|
119
|
+
"ssh2-sftp-client": {
|
|
116
120
|
"optional": true
|
|
117
121
|
},
|
|
118
|
-
"
|
|
122
|
+
"unzipper": {
|
|
119
123
|
"optional": true
|
|
120
124
|
}
|
|
121
125
|
},
|