@zukhruf/sqlite-queue 0.8.18 → 0.8.20
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +35 -19
- package/dist/contractor.d.ts +6 -0
- package/dist/contractor.d.ts.map +1 -0
- package/dist/contractor.js +70 -0
- package/dist/contractor.js.map +1 -0
- package/dist/index.d.ts +8 -2
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +172 -43
- package/dist/index.js.map +1 -1
- package/dist/job-queue.d.ts +4 -2
- package/dist/job-queue.d.ts.map +1 -1
- package/dist/job-rows.d.ts +1 -1
- package/dist/job-rows.d.ts.map +1 -1
- package/dist/job-rows.js +5 -6
- package/dist/job-rows.js.map +1 -1
- package/dist/job-settler.d.ts +1 -2
- package/dist/job-settler.d.ts.map +1 -1
- package/dist/job-settler.js +12 -8
- package/dist/job-settler.js.map +1 -1
- package/dist/queue-stats-cache.d.ts +14 -0
- package/dist/queue-stats-cache.d.ts.map +1 -0
- package/dist/queue-stats-cache.js +97 -0
- package/dist/queue-stats-cache.js.map +1 -0
- package/dist/release.d.ts +8 -0
- package/dist/release.d.ts.map +1 -0
- package/dist/release.js +27 -0
- package/dist/release.js.map +1 -0
- package/dist/schema.d.ts.map +1 -1
- package/dist/schema.js +4 -2
- package/dist/schema.js.map +1 -1
- package/dist/supervisor.d.ts.map +1 -1
- package/dist/supervisor.js +11 -62
- package/dist/supervisor.js.map +1 -1
- package/package.json +8 -7
package/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# @zukhruf/sqlite-queue
|
|
2
2
|
|
|
3
|
-
A program sends jobs to a queue, and workers do the jobs later, also after the program restarts. [pg-boss](https://github.com/timgit/pg-boss) is a job queue that keeps its jobs in Postgres. A program that has no Postgres server, for example a desktop app or a CLI, cannot use it. This package keeps the jobs in one SQLite file through `node:sqlite`, and gives the API of pg-boss: the same method names, the same options, the same results and the same errors. The API is a declared subset of pg-boss:
|
|
3
|
+
A program sends jobs to a queue, and workers do the jobs later, also after the program restarts. [pg-boss](https://github.com/timgit/pg-boss) is a job queue that keeps its jobs in Postgres. A program that has no Postgres server, for example a desktop app or a CLI, cannot use it. This package keeps the jobs in one SQLite file through `node:sqlite`, and gives the API of pg-boss: the same method names, the same options, the same results and the same errors. The API is a declared subset of pg-boss: 18 methods of `PgBoss`.
|
|
4
4
|
|
|
5
5
|
Zero divergence is the contract. A parity suite runs each scenario on real pg-boss 12.35.1 over PGlite, and on this package. Then it compares what each engine returned, stored and emitted. A test passes only when the two are equal. Thus a program can change from one engine to the other and keep its code.
|
|
6
6
|
|
|
@@ -8,7 +8,7 @@ The words in these documents have one meaning each. See the glossary in [CONTEXT
|
|
|
8
8
|
|
|
9
9
|
## Use it
|
|
10
10
|
|
|
11
|
-
This project is an experiment. It uses `node:sqlite`,
|
|
11
|
+
This project is an experiment. It uses `node:sqlite`, pg-boss for its types and some of its values, and [`@zukhruf/sqlite-notify`](../notify/README.md) for the notify queues.
|
|
12
12
|
|
|
13
13
|
```sh
|
|
14
14
|
npm install @zukhruf/sqlite-queue
|
|
@@ -62,7 +62,19 @@ const queue: JobQueue = new PgBoss(process.env.DATABASE_URL);
|
|
|
62
62
|
- The types of the options, the jobs and the results come from pg-boss. Import them from `pg-boss`, for example `import type { Job, SendOptions } from 'pg-boss'`.
|
|
63
63
|
- Job data is `unknown`. `work()` gives each handler `Job<unknown>`, and `findJobs()` gives `FoundJob<unknown>`. Validate the data that you read.
|
|
64
64
|
- A listener for `error` is necessary. A failure in work that no caller awaits, for example a worker poll, goes to the `error` event. With no listener, `EventEmitter` throws.
|
|
65
|
-
|
|
65
|
+
|
|
66
|
+
## Notify queues
|
|
67
|
+
|
|
68
|
+
A queue with `notify: true` wakes its workers when a job comes, so they do not wait for their poll. pg-boss does this with `LISTEN` and `NOTIFY`. On a queue file, this package does it with [`@zukhruf/sqlite-notify`](../notify/README.md) ([ADR 0007](./docs/adr/0007-a-notify-queue-notifies-each-listener-on-the-file-through-sqlite-notify.md)).
|
|
69
|
+
|
|
70
|
+
- A send to a notify queue notifies when the stored job is due now. The notification is in the transaction of the insert. The sender notifies also when it does not listen, as in pg-boss.
|
|
71
|
+
- The channel is the channel of pg-boss, `pgboss_454650302e472074f72063fd`, and the payload is the queue name.
|
|
72
|
+
- With `useListenNotify`, `start()` listens on the channel. Each `SqliteJobQueue` that listens on the same file wakes the workers of the queue that the payload names, also in another program. The instance that sent the job also hears it.
|
|
73
|
+
- The poll of pg-boss stays. A worker of a notify queue fetches at least every `notifyPollingIntervalSeconds`, 30 by default.
|
|
74
|
+
- When the listen fails, the queue emits the warning of pg-boss, `listen_notify_unavailable`, and the workers poll.
|
|
75
|
+
- On a queue file, `start()` makes the tables of `sqlite-notify`, which start with `zukhruf_notify_`, if they are missing. `stop()` stops the listen.
|
|
76
|
+
- A database in memory has no file that another program can open. A send there wakes only the workers of the same `SqliteJobQueue`, when it listens.
|
|
77
|
+
- A send before `start()`, on a `database` of the caller, does not notify. pg-boss notifies there. The other instances find the job at their poll.
|
|
66
78
|
|
|
67
79
|
## Upgrades
|
|
68
80
|
|
|
@@ -70,7 +82,8 @@ A new release can change the tables. A file that an earlier release made keeps i
|
|
|
70
82
|
|
|
71
83
|
- The schema is a declaration of [`@zukhruf/sqlite-drift`](../drift) with `rows: 'keep'`. `start()` compares the tables and the indexes in the file with the declaration.
|
|
72
84
|
- `start()` makes each missing object. It rebuilds each stale table and keeps its rows, and it replaces each stale index.
|
|
73
|
-
- No release writes a migration, and no
|
|
85
|
+
- No release writes a migration. The file records the version of the release that last applied the schema, and the version comes from the package itself: no person keeps a number ([ADR 0006](./docs/adr/0006-the-file-records-the-release-that-made-its-schema-and-an-older-release-leaves-it-alone.md)).
|
|
86
|
+
- A release that starts on a file of a later release changes nothing and goes on, as pg-boss does on a newer schema.
|
|
74
87
|
- When a kept row does not fit the declared table, `start()` fails with the error of SQLite, and the file stays as it was.
|
|
75
88
|
- When `pgboss_queue` is stale, the rebuild turns foreign keys off around its own transaction. Do not call `start()` in a transaction of your own.
|
|
76
89
|
|
|
@@ -89,21 +102,23 @@ A schedule sends a job to its queue at each time that its cron or rrule rule giv
|
|
|
89
102
|
|
|
90
103
|
## The subset
|
|
91
104
|
|
|
92
|
-
| Method | What it does
|
|
93
|
-
| ------------------------------------------------------------------------------- |
|
|
94
|
-
| `on('error', listener)` | Receives the failures that no caller awaits.
|
|
95
|
-
| `start()`, `stop(options)` | Start and stop the queue. A second call while one runs shares it.
|
|
96
|
-
| `createQueue(name, options)`, `updateQueue(name, options)`, `deleteQueue(name)` | Make, change and remove a queue, with its policy, retries, expiry, retention and dead letter queue.
|
|
97
|
-
| `send(name, data, options)`, `send({ name, data, options })` | Add a job. It gives the id of the job, or `null` when a singleton or throttle refuses the job.
|
|
98
|
-
| `work(name, handler)`, `work(name, options, handler)` | Start workers that fetch jobs and give them to the handler. It gives the id of the work.
|
|
99
|
-
| `offWork(name, options)` | Stop the workers of a queue.
|
|
100
|
-
| `fetch(name, options)` | Claim up to `batchSize` jobs with no worker. It gives the jobs, which are then active.
|
|
101
|
-
| `findJobs(name, options)` | Find jobs by id, key, data or queued state.
|
|
102
|
-
| `complete(name, id, data, options)` | Complete jobs by id or by attempt. With `includeQueued`, also jobs that are not yet active.
|
|
103
|
-
| `
|
|
104
|
-
| `
|
|
105
|
-
|
|
106
|
-
|
|
105
|
+
| Method | What it does |
|
|
106
|
+
| ------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
107
|
+
| `on('error', listener)` | Receives the failures that no caller awaits. |
|
|
108
|
+
| `start()`, `stop(options)` | Start and stop the queue. A second call while one runs shares it. |
|
|
109
|
+
| `createQueue(name, options)`, `updateQueue(name, options)`, `deleteQueue(name)` | Make, change and remove a queue, with its policy, retries, expiry, retention and dead letter queue. |
|
|
110
|
+
| `send(name, data, options)`, `send({ name, data, options })` | Add a job. It gives the id of the job, or `null` when a singleton or throttle refuses the job. |
|
|
111
|
+
| `work(name, handler)`, `work(name, options, handler)` | Start workers that fetch jobs and give them to the handler. It gives the id of the work. |
|
|
112
|
+
| `offWork(name, options)` | Stop the workers of a queue. |
|
|
113
|
+
| `fetch(name, options)` | Claim up to `batchSize` jobs with no worker. It gives the jobs, which are then active. |
|
|
114
|
+
| `findJobs(name, options)` | Find jobs by id, key, data or queued state. |
|
|
115
|
+
| `complete(name, id, data, options)` | Complete jobs by id or by attempt. With `includeQueued`, also jobs that are not yet active. |
|
|
116
|
+
| `touch(name, id)` | Keep the heartbeat of active jobs by id or by attempt. |
|
|
117
|
+
| `getQueueStats(name, options)` | Give the counts of a queue by state, from the cache that the monitor keeps. A read counts again when the cache is older than one hour, or one minute with `force`. |
|
|
118
|
+
| `cancel(name, id)`, `fail(name, id, data)` | Cancel or fail jobs by id or by attempt. |
|
|
119
|
+
| `schedule(name, cron, data, options)`, `unschedule(name, key)` | Send a job on a cron or rrule schedule, and remove the schedule. |
|
|
120
|
+
|
|
121
|
+
Other methods of `PgBoss` are not in the subset, for example `getQueue`, `getJobById`, `retry`, `resume`, `deleteJob`, `sendAfter`, `insert` and `getDb`. A method or an option that needs Postgres itself, for example `getDb` and its SQL, or the `db` option of `fetch`, `complete`, `touch`, `cancel` and `fail`, will not come.
|
|
107
122
|
|
|
108
123
|
## Errors
|
|
109
124
|
|
|
@@ -131,6 +146,7 @@ Code that runs on both engines examines `code`, not the class. On Postgres, the
|
|
|
131
146
|
- [0003 The timekeeper is ported literally](./docs/adr/0003-the-timekeeper-is-ported-literally.md)
|
|
132
147
|
- [0004 The schema is a declaration and keeps the jobs](./docs/adr/0004-the-schema-is-a-declaration-and-keeps-the-jobs.md)
|
|
133
148
|
- [0005 The queue keeps the durability of pg-boss](./docs/adr/0005-the-queue-keeps-the-durability-of-pg-boss.md)
|
|
149
|
+
- [0007 A notify queue notifies each listener on the file through sqlite-notify](./docs/adr/0007-a-notify-queue-notifies-each-listener-on-the-file-through-sqlite-notify.md)
|
|
134
150
|
|
|
135
151
|
## Development
|
|
136
152
|
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"contractor.d.ts","sourceRoot":"","sources":["../src/contractor.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAUhD,qBAAa,UAAU;;IACrB,MAAM,CAAC,KAAK,CAAC,QAAQ,EAAE,YAAY,GAAG,IAAI;CAsE3C"}
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
import { transaction } from '@zukhruf/sqlite-transaction';
|
|
2
|
+
import { Release } from "./release.js";
|
|
3
|
+
import { queueSchema } from "./schema.js";
|
|
4
|
+
// pg-boss's contractor.start() (node_modules/pg-boss/dist/contractor.js:
|
|
5
|
+
// 79-91): a file that a newer release made stays as it is, and any other file
|
|
6
|
+
// gets this release's schema with its rows kept (docs/adr/0004, 0006).
|
|
7
|
+
export class Contractor {
|
|
8
|
+
static start(database) {
|
|
9
|
+
// pg-boss reads isInstalled() and schemaVersion() with no lock
|
|
10
|
+
// (contractor.js:79-91), and locks only in create() and migrate()
|
|
11
|
+
// (plans.js:134, :166). A WAL reader does not wait for a writer, so a
|
|
12
|
+
// file that needs no change starts next to the write of another program.
|
|
13
|
+
if (Contractor.#settled(database))
|
|
14
|
+
return;
|
|
15
|
+
// The rebuild of pgboss_queue needs foreign keys off, which SQLite cannot
|
|
16
|
+
// change inside a transaction, and the read of the release must share the
|
|
17
|
+
// write lock with the apply. In a transaction of the caller, the keys stay
|
|
18
|
+
// as they are, and the declaration fails before it changes the file.
|
|
19
|
+
const keys = !database.isTransaction &&
|
|
20
|
+
Number(database.prepare('PRAGMA foreign_keys').get()?.foreign_keys) === 1;
|
|
21
|
+
if (keys)
|
|
22
|
+
database.exec('PRAGMA foreign_keys = OFF');
|
|
23
|
+
try {
|
|
24
|
+
transaction(database, () => {
|
|
25
|
+
const installed = Contractor.#installed(database);
|
|
26
|
+
if (installed?.isAfter(Release.current))
|
|
27
|
+
return;
|
|
28
|
+
queueSchema.apply(database);
|
|
29
|
+
database
|
|
30
|
+
.prepare(`INSERT INTO pgboss_version (singleton, version) VALUES (1, ?)
|
|
31
|
+
ON CONFLICT (singleton) DO UPDATE SET version = excluded.version`)
|
|
32
|
+
.run(Release.current.version);
|
|
33
|
+
}, { mode: 'immediate' });
|
|
34
|
+
}
|
|
35
|
+
finally {
|
|
36
|
+
if (keys)
|
|
37
|
+
database.exec('PRAGMA foreign_keys = ON');
|
|
38
|
+
}
|
|
39
|
+
}
|
|
40
|
+
// The file records a later release, or this release and fits the
|
|
41
|
+
// declaration. The two reads share one snapshot. The locked transaction
|
|
42
|
+
// reads both again, so a program that changes the file after this read
|
|
43
|
+
// cannot make start() act on it.
|
|
44
|
+
static #settled(database) {
|
|
45
|
+
return transaction(database, () => {
|
|
46
|
+
const installed = Contractor.#installed(database);
|
|
47
|
+
if (installed === undefined)
|
|
48
|
+
return false;
|
|
49
|
+
if (installed.isAfter(Release.current))
|
|
50
|
+
return true;
|
|
51
|
+
return (installed.version === Release.current.version &&
|
|
52
|
+
queueSchema.drift(database) === 'fits');
|
|
53
|
+
}, { mode: 'deferred' });
|
|
54
|
+
}
|
|
55
|
+
// The release that the file records. A file that a release before the
|
|
56
|
+
// version column made records none. Under the write lock, the table cannot
|
|
57
|
+
// change before the apply.
|
|
58
|
+
static #installed(database) {
|
|
59
|
+
const recorded = database
|
|
60
|
+
.prepare(`SELECT 1 FROM pragma_table_info('pgboss_version') WHERE name = 'version'`)
|
|
61
|
+
.get();
|
|
62
|
+
if (recorded === undefined)
|
|
63
|
+
return undefined;
|
|
64
|
+
const row = database.prepare('SELECT version FROM pgboss_version').get();
|
|
65
|
+
return typeof row?.version === 'string'
|
|
66
|
+
? new Release(row.version)
|
|
67
|
+
: undefined;
|
|
68
|
+
}
|
|
69
|
+
}
|
|
70
|
+
//# sourceMappingURL=contractor.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"contractor.js","sourceRoot":"","sources":["../src/contractor.ts"],"names":[],"mappings":"AAEA,OAAO,EAAE,WAAW,EAAE,MAAM,6BAA6B,CAAC;AAE1D,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AACvC,OAAO,EAAE,WAAW,EAAE,MAAM,aAAa,CAAC;AAE1C,yEAAyE;AACzE,8EAA8E;AAC9E,uEAAuE;AACvE,MAAM,OAAO,UAAU;IACrB,MAAM,CAAC,KAAK,CAAC,QAAsB;QACjC,+DAA+D;QAC/D,kEAAkE;QAClE,sEAAsE;QACtE,yEAAyE;QACzE,IAAI,UAAU,CAAC,QAAQ,CAAC,QAAQ,CAAC;YAAE,OAAO;QAC1C,0EAA0E;QAC1E,0EAA0E;QAC1E,2EAA2E;QAC3E,qEAAqE;QACrE,MAAM,IAAI,GACR,CAAC,QAAQ,CAAC,aAAa;YACvB,MAAM,CAAC,QAAQ,CAAC,OAAO,CAAC,qBAAqB,CAAC,CAAC,GAAG,EAAE,EAAE,YAAY,CAAC,KAAK,CAAC,CAAC;QAC5E,IAAI,IAAI;YAAE,QAAQ,CAAC,IAAI,CAAC,2BAA2B,CAAC,CAAC;QACrD,IAAI,CAAC;YACH,WAAW,CACT,QAAQ,EACR,GAAG,EAAE;gBACH,MAAM,SAAS,GAAG,UAAU,CAAC,UAAU,CAAC,QAAQ,CAAC,CAAC;gBAClD,IAAI,SAAS,EAAE,OAAO,CAAC,OAAO,CAAC,OAAO,CAAC;oBAAE,OAAO;gBAChD,WAAW,CAAC,KAAK,CAAC,QAAQ,CAAC,CAAC;gBAC5B,QAAQ;qBACL,OAAO,CACN;+EACiE,CAClE;qBACA,GAAG,CAAC,OAAO,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC;YAClC,CAAC,EACD,EAAE,IAAI,EAAE,WAAW,EAAE,CACtB,CAAC;QACJ,CAAC;gBAAS,CAAC;YACT,IAAI,IAAI;gBAAE,QAAQ,CAAC,IAAI,CAAC,0BAA0B,CAAC,CAAC;QACtD,CAAC;IACH,CAAC;IAED,iEAAiE;IACjE,wEAAwE;IACxE,uEAAuE;IACvE,iCAAiC;IACjC,MAAM,CAAC,QAAQ,CAAC,QAAsB;QACpC,OAAO,WAAW,CAChB,QAAQ,EACR,GAAG,EAAE;YACH,MAAM,SAAS,GAAG,UAAU,CAAC,UAAU,CAAC,QAAQ,CAAC,CAAC;YAClD,IAAI,SAAS,KAAK,SAAS;gBAAE,OAAO,KAAK,CAAC;YAC1C,IAAI,SAAS,CAAC,OAAO,CAAC,OAAO,CAAC,OAAO,CAAC;gBAAE,OAAO,IAAI,CAAC;YACpD,OAAO,CACL,SAAS,CAAC,OAAO,KAAK,OAAO,CAAC,OAAO,CAAC,OAAO;gBAC7C,WAAW,CAAC,KAAK,CAAC,QAAQ,CAAC,KAAK,MAAM,CACvC,CAAC;QACJ,CAAC,EACD,EAAE,IAAI,EAAE,UAAU,EAAE,CACrB,CAAC;IACJ,CAAC;IAED,sEAAsE;IACtE,2EAA2E;IAC3E,2BAA2B;IAC3B,MAAM,CAAC,UAAU,CAAC,QAAsB;QACtC,MAAM,QAAQ,GAAG,QAAQ;aACtB,OAAO,CACN,0EAA0E,CAC3E;aACA,GAAG,EAAE,CAAC;QACT,IAAI,QAAQ,KAAK,SAAS;YAAE,OAAO,SAAS,CAAC;QAC7C,MAAM,GAAG,GAAG,QAAQ,CAAC,OAAO,CAAC,oCAAoC,CAAC,CAAC,GAAG,EAAE,CAAC;QACzE,OAAO,OAAO,GAAG,EAAE,OAAO,KAAK,QAAQ;YACrC,CAAC,CAAC,IAAI,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC;YAC1B,CAAC,CAAC,SAAS,CAAC;IAChB,CAAC;CACF"}
|
package/dist/index.d.ts
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import { EventEmitter } from 'node:events';
|
|
2
2
|
import type { DatabaseSync } from 'node:sqlite';
|
|
3
|
-
import { type Clock, type CompleteOptions, type FetchOptions, type FindJobsOptions, type Job, type JobAttempt, type JobWithMetadata, type MaintenanceOptions, type OffWorkOptions, type PgBossEventMap, type Queue, type Request, type ScheduleOptions, type SchedulingOptions, type SendOptions, type StopOptions, type UpdateQueueOptions, type WorkHandler, type WorkOptions } from 'pg-boss';
|
|
3
|
+
import { type Clock, type CompleteOptions, type FetchOptions, type FindJobsOptions, type Job, type JobAttempt, type JobWithMetadata, type MaintenanceOptions, type OffWorkOptions, type PgBossEventMap, type Queue, type QueueStats, type QueueStatsOptions, type Request, type ScheduleOptions, type SchedulingOptions, type SendOptions, type StopOptions, type UpdateQueueOptions, type WorkHandler, type WorkOptions } from 'pg-boss';
|
|
4
4
|
import type { FoundJob, JobQueue } from './job-queue.ts';
|
|
5
5
|
export { DatabaseError } from './database-error.ts';
|
|
6
6
|
export { UnsupportedStartAfterError } from './start-after.ts';
|
|
@@ -22,6 +22,7 @@ export declare class SqliteJobQueue extends EventEmitter<Pick<PgBossEventMap, 'e
|
|
|
22
22
|
createQueue(name: string, options?: Omit<Queue, 'name'>): Promise<void>;
|
|
23
23
|
updateQueue(name: string, options?: UpdateQueueOptions): Promise<void>;
|
|
24
24
|
deleteQueue(name: string): Promise<void>;
|
|
25
|
+
getQueueStats(name: string, options?: QueueStatsOptions): Promise<QueueStats[]>;
|
|
25
26
|
send(request: Request): Promise<string | null>;
|
|
26
27
|
send<TData extends object>(name: string, data?: TData | null, options?: SendOptions): Promise<string | null>;
|
|
27
28
|
work(name: string, handler: WorkHandler<unknown>): Promise<string>;
|
|
@@ -43,6 +44,12 @@ export declare class SqliteJobQueue extends EventEmitter<Pick<PgBossEventMap, 'e
|
|
|
43
44
|
affected: number;
|
|
44
45
|
settled: string[];
|
|
45
46
|
}>;
|
|
47
|
+
touch(name: string, id: string | string[] | JobAttempt | JobAttempt[]): Promise<{
|
|
48
|
+
jobs: string[];
|
|
49
|
+
requested: number;
|
|
50
|
+
affected: number;
|
|
51
|
+
settled: string[];
|
|
52
|
+
}>;
|
|
46
53
|
fail(name: string, id: string | string[] | JobAttempt | JobAttempt[], data?: object | null): Promise<{
|
|
47
54
|
jobs: string[];
|
|
48
55
|
requested: number;
|
|
@@ -51,7 +58,6 @@ export declare class SqliteJobQueue extends EventEmitter<Pick<PgBossEventMap, 'e
|
|
|
51
58
|
}>;
|
|
52
59
|
schedule<TData extends object>(name: string, cron: string, data?: TData | null, options?: ScheduleOptions): Promise<void>;
|
|
53
60
|
unschedule(name: string, key?: string): Promise<void>;
|
|
54
|
-
private applySchema;
|
|
55
61
|
private mapQueueUpdateOptions;
|
|
56
62
|
private pushOption;
|
|
57
63
|
private pushClearableOption;
|
package/dist/index.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAEA,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAC3C,OAAO,KAAK,EAAE,YAAY,EAAiB,MAAM,aAAa,CAAC;AAG/D,OAAO,EACL,KAAK,KAAK,EAEV,KAAK,eAAe,EACpB,KAAK,YAAY,EACjB,KAAK,eAAe,EACpB,KAAK,GAAG,EACR,KAAK,UAAU,EACf,KAAK,eAAe,EACpB,KAAK,kBAAkB,EACvB,KAAK,cAAc,EACnB,KAAK,cAAc,EACnB,KAAK,KAAK,EACV,KAAK,OAAO,EACZ,KAAK,eAAe,EACpB,KAAK,iBAAiB,EACtB,KAAK,WAAW,EAChB,KAAK,WAAW,EAChB,KAAK,kBAAkB,EAEvB,KAAK,WAAW,EAChB,KAAK,WAAW,EAIjB,MAAM,SAAS,CAAC;
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAEA,OAAO,EAAE,YAAY,EAAE,MAAM,aAAa,CAAC;AAC3C,OAAO,KAAK,EAAE,YAAY,EAAiB,MAAM,aAAa,CAAC;AAG/D,OAAO,EACL,KAAK,KAAK,EAEV,KAAK,eAAe,EACpB,KAAK,YAAY,EACjB,KAAK,eAAe,EACpB,KAAK,GAAG,EACR,KAAK,UAAU,EACf,KAAK,eAAe,EACpB,KAAK,kBAAkB,EACvB,KAAK,cAAc,EACnB,KAAK,cAAc,EACnB,KAAK,KAAK,EACV,KAAK,UAAU,EACf,KAAK,iBAAiB,EACtB,KAAK,OAAO,EACZ,KAAK,eAAe,EACpB,KAAK,iBAAiB,EACtB,KAAK,WAAW,EAChB,KAAK,WAAW,EAChB,KAAK,kBAAkB,EAEvB,KAAK,WAAW,EAChB,KAAK,WAAW,EAIjB,MAAM,SAAS,CAAC;AAejB,OAAO,KAAK,EAAE,QAAQ,EAAE,QAAQ,EAAE,MAAM,gBAAgB,CAAC;AAwBzD,OAAO,EAAE,aAAa,EAAE,MAAM,qBAAqB,CAAC;AACpD,OAAO,EAAE,0BAA0B,EAAE,MAAM,kBAAkB,CAAC;AAE9D,YAAY,EAAE,QAAQ,EAAE,QAAQ,EAAE,MAAM,gBAAgB,CAAC;AA8BzD,MAAM,WAAW,qBACf,SACE,IAAI,CACF,kBAAkB,EAChB,WAAW,GACX,0BAA0B,GAC1B,wBAAwB,GACxB,4BAA4B,GAC5B,2BAA2B,GAC3B,kBAAkB,CACrB,EACD,IAAI,CACF,iBAAiB,EACf,UAAU,GACV,4BAA4B,GAC5B,2BAA2B,GAC3B,6BAA6B,CAChC;IACH,IAAI,CAAC,EAAE,MAAM,CAAC;IACd,QAAQ,CAAC,EAAE,YAAY,CAAC;IACxB,KAAK,CAAC,EAAE,KAAK,CAAC;IACd,eAAe,CAAC,EAAE,OAAO,CAAC;CAC3B;AA4CD,qBAAa,cACX,SAAQ,YAAY,CAClB,IAAI,CAAC,cAAc,EAAE,OAAO,GAAG,SAAS,GAAG,KAAK,GAAG,SAAS,CAAC,CAE/D,YAAW,QAAQ;;IA2CnB,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAqC;gBAEjD,OAAO,CAAC,EAAE,MAAM,GAAG,qBAAqB;IAmDpD,OAAO,KAAK,EAAE,GAMb;IAIK,KAAK,IAAI,OAAO,CAAC,IAAI,CAAC;IAqHtB,IAAI,CAAC,OAAO,GAAE,WAAgB,GAAG,OAAO,CAAC,IAAI,CAAC;IA+M9C,OAAO,CACX,IAAI,EAAE,MAAM,EACZ,OAAO,GAAE,cAA+B,GACvC,OAAO,CAAC,IAAI,CAAC;IAwDV,WAAW,CACf,IAAI,EAAE,MAAM,EACZ,OAAO,GAAE,IAAI,CAAC,KAAK,EAAE,MAAM,CAAM,GAChC,OAAO,CAAC,IAAI,CAAC;IAsDV,WAAW,CACf,IAAI,EAAE,MAAM,EACZ,OAAO,GAAE,kBAAuB,GAC/B,OAAO,CAAC,IAAI,CAAC;IA+CV,WAAW,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAAC,IAAI,CAAC;IA8BxC,aAAa,CACjB,IAAI,EAAE,MAAM,EACZ,OAAO,GAAE,iBAAsB,GAC9B,OAAO,CAAC,UAAU,EAAE,CAAC;IAqDxB,IAAI,CAAC,OAAO,EAAE,OAAO,GAAG,OAAO,CAAC,MAAM,GAAG,IAAI,CAAC;IAC9C,IAAI,CAAC,KAAK,SAAS,MAAM,EACvB,IAAI,EAAE,MAAM,EACZ,IAAI,CAAC,EAAE,KAAK,GAAG,IAAI,EACnB,OAAO,CAAC,EAAE,WAAW,GACpB,OAAO,CAAC,MAAM,GAAG,IAAI,CAAC;IA6LzB,IAAI,CAAC,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,WAAW,CAAC,OAAO,CAAC,GAAG,OAAO,CAAC,MAAM,CAAC;IAClE,IAAI,CACF,IAAI,EAAE,MAAM,EACZ,OAAO,EAAE,WAAW,EACpB,OAAO,EAAE,WAAW,CAAC,OAAO,CAAC,GAC5B,OAAO,CAAC,MAAM,CAAC;IAwLlB,KAAK,CACH,IAAI,EAAE,MAAM,EACZ,OAAO,EAAE,YAAY,GAAG;QAAE,eAAe,EAAE,IAAI,CAAA;KAAE,GAChD,OAAO,CAAC,eAAe,CAAC,OAAO,CAAC,EAAE,CAAC;IACtC,KAAK,CAAC,IAAI,EAAE,MAAM,EAAE,OAAO,CAAC,EAAE,YAAY,GAAG,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,EAAE,CAAC;IAmC9D,QAAQ,CACZ,IAAI,EAAE,MAAM,EACZ,OAAO,GAAE,eAAoB,GAC5B,OAAO,CAAC,QAAQ,CAAC,OAAO,CAAC,EAAE,CAAC;IAwCzB,MAAM,CACV,IAAI,EAAE,MAAM,EACZ,EAAE,EAAE,MAAM,GAAG,MAAM,EAAE,GAAG,UAAU,GAAG,UAAU,EAAE;;;;;;IAa7C,QAAQ,CACZ,IAAI,EAAE,MAAM,EACZ,EAAE,EAAE,MAAM,GAAG,MAAM,EAAE,GAAG,UAAU,GAAG,UAAU,EAAE,EACjD,IAAI,CAAC,EAAE,MAAM,GAAG,IAAI,EACpB,OAAO,GAAE,eAAoB;;;;;;IAqBzB,KAAK,CAAC,IAAI,EAAE,MAAM,EAAE,EAAE,EAAE,MAAM,GAAG,MAAM,EAAE,GAAG,UAAU,GAAG,UAAU,EAAE;;;;;;IAWrE,IAAI,CACR,IAAI,EAAE,MAAM,EACZ,EAAE,EAAE,MAAM,GAAG,MAAM,EAAE,GAAG,UAAU,GAAG,UAAU,EAAE,EACjD,IAAI,CAAC,EAAE,MAAM,GAAG,IAAI;;;;;;IAahB,QAAQ,CAAC,KAAK,SAAS,MAAM,EACjC,IAAI,EAAE,MAAM,EACZ,IAAI,EAAE,MAAM,EACZ,IAAI,CAAC,EAAE,KAAK,GAAG,IAAI,EACnB,OAAO,GAAE,eAAoB,GAC5B,OAAO,CAAC,IAAI,CAAC;IAkEV,UAAU,CAAC,IAAI,EAAE,MAAM,EAAE,GAAG,SAAK,GAAG,OAAO,CAAC,IAAI,CAAC;IASvD,OAAO,CAAC,qBAAqB;IAiC7B,OAAO,CAAC,UAAU;IAYlB,OAAO,CAAC,mBAAmB;IAU3B,OAAO,CAAC,SAAS;IAUjB,OAAO,CAAC,eAAe;IAgCvB,OAAO,CAAC,WAAW;IAYnB,OAAO,CAAC,sBAAsB;YAahB,WAAW;CAqN1B"}
|
package/dist/index.js
CHANGED
|
@@ -1,13 +1,15 @@
|
|
|
1
1
|
import assert from 'node:assert';
|
|
2
|
-
import { randomUUID } from 'node:crypto';
|
|
2
|
+
import { hash, randomUUID } from 'node:crypto';
|
|
3
3
|
import { EventEmitter } from 'node:events';
|
|
4
4
|
import { setImmediate } from 'node:timers/promises';
|
|
5
5
|
import { policies, scheduleKinds, systemClock, } from 'pg-boss';
|
|
6
6
|
import { Latch } from '@zukhruf/async';
|
|
7
|
+
import { Notifications } from '@zukhruf/sqlite-notify';
|
|
7
8
|
import { WriteAheadLog, open } from '@zukhruf/sqlite-open';
|
|
8
9
|
import { transaction } from '@zukhruf/sqlite-transaction';
|
|
9
10
|
import { Activity } from "./activity.js";
|
|
10
11
|
import { Attorney } from "./attorney.js";
|
|
12
|
+
import { Contractor } from "./contractor.js";
|
|
11
13
|
import { DatabaseError } from "./database-error.js";
|
|
12
14
|
import { Database } from "./database.js";
|
|
13
15
|
import { CurrentOperation } from "./in-flight.js";
|
|
@@ -18,9 +20,9 @@ import { Jsonb } from "./jsonb.js";
|
|
|
18
20
|
import { LocalGroups } from "./local-groups.js";
|
|
19
21
|
import { QueueCache } from "./queue-cache.js";
|
|
20
22
|
import { QueueRows } from "./queue-rows.js";
|
|
23
|
+
import { QueueStatsCache } from "./queue-stats-cache.js";
|
|
21
24
|
import { Recurrence } from "./recurrence.js";
|
|
22
25
|
import { Row } from "./rows.js";
|
|
23
|
-
import { queueSchema } from "./schema.js";
|
|
24
26
|
import { SqlValues } from "./sql-values.js";
|
|
25
27
|
import { StartAfter } from "./start-after.js";
|
|
26
28
|
import { Supervisor } from "./supervisor.js";
|
|
@@ -55,6 +57,13 @@ function spreadCopy(error, extra) {
|
|
|
55
57
|
Object.defineProperty(copy, 'stack', { value: error.stack });
|
|
56
58
|
return copy;
|
|
57
59
|
}
|
|
60
|
+
// pg-boss's channel: 'pgboss_' and the first 24 hex digits of the SHA-224 of
|
|
61
|
+
// the schema name (node_modules/pg-boss/dist/plans.js:859-861). The schema is
|
|
62
|
+
// pg-boss's default, 'pgboss' (plans.js:8); this package has no schema option.
|
|
63
|
+
const notifyChannel = `pgboss_${hash('sha224', 'pgboss', 'hex').slice(0, 24)}`;
|
|
64
|
+
// node_modules/pg-boss/dist/manager.js:123, 127
|
|
65
|
+
const QUEUE_STATS_CACHE_TTL_SECONDS = 60 * 60;
|
|
66
|
+
const QUEUE_STATS_FORCE_TTL_SECONDS = 60;
|
|
58
67
|
function normalizeOptions(options) {
|
|
59
68
|
if (typeof options === 'string' || options === undefined) {
|
|
60
69
|
return { path: options ?? ':memory:' };
|
|
@@ -77,11 +86,9 @@ export class SqliteJobQueue extends EventEmitter {
|
|
|
77
86
|
#useListenNotify;
|
|
78
87
|
#warnedFetchOptions = new Set();
|
|
79
88
|
// pg-boss's notifier.available (node_modules/pg-boss/dist/notifier.js:28):
|
|
80
|
-
// this
|
|
81
|
-
// useListenNotify is on. Its wakes stay in this instance: another instance on
|
|
82
|
-
// the same file never hears them, where every pg-boss listener on a schema
|
|
83
|
-
// hears a NOTIFY.
|
|
89
|
+
// this instance listens from start() to stop() when useListenNotify is on.
|
|
84
90
|
#notifier = { kind: 'unavailable' };
|
|
91
|
+
#notifyRoute = { kind: 'stopped' };
|
|
85
92
|
// PgBoss's own #stopped and #started (node_modules/pg-boss/dist/
|
|
86
93
|
// index.js:64, 124, 256): armed before a start runs, so stop() can shut
|
|
87
94
|
// down what a start that threw partway left armed.
|
|
@@ -96,6 +103,7 @@ export class SqliteJobQueue extends EventEmitter {
|
|
|
96
103
|
#wipInterval;
|
|
97
104
|
#queueCache = new QueueCache();
|
|
98
105
|
#queueCacheIntervalSeconds;
|
|
106
|
+
#intervals;
|
|
99
107
|
#queueCacheInterval;
|
|
100
108
|
// The offWork() cleanups a stop waits for, each opened once it settles.
|
|
101
109
|
#pendingCleanups = new Set();
|
|
@@ -120,6 +128,7 @@ export class SqliteJobQueue extends EventEmitter {
|
|
|
120
128
|
const timekeeping = Attorney.applyScheduleConfig(normalized);
|
|
121
129
|
Attorney.validateWarningConfig(normalized);
|
|
122
130
|
this.#queueCacheIntervalSeconds = intervals.queueCacheIntervalSeconds;
|
|
131
|
+
this.#intervals = intervals;
|
|
123
132
|
this.#supervisor = new Supervisor(this.#clock, this.#activity, {
|
|
124
133
|
...intervals,
|
|
125
134
|
// attorney.js:413
|
|
@@ -180,7 +189,7 @@ export class SqliteJobQueue extends EventEmitter {
|
|
|
180
189
|
}),
|
|
181
190
|
};
|
|
182
191
|
}
|
|
183
|
-
this.
|
|
192
|
+
Contractor.start(this.db);
|
|
184
193
|
if (Activity.isAttachable(this.#clock)) {
|
|
185
194
|
this.#attachment = {
|
|
186
195
|
kind: 'attached',
|
|
@@ -195,9 +204,9 @@ export class SqliteJobQueue extends EventEmitter {
|
|
|
195
204
|
}, this.#queueCacheIntervalSeconds * 1000);
|
|
196
205
|
this.#wipInterval = this.#clock.setInterval(() => this.#onWipInterval(), 2000);
|
|
197
206
|
await this.#onCacheQueues(false);
|
|
207
|
+
this.#notifyRoute = this.#openNotifyRoute();
|
|
198
208
|
if (this.#useListenNotify) {
|
|
199
|
-
this.#
|
|
200
|
-
await this.#activity.deliver(() => this.wakeNotifyQueueWorkers());
|
|
209
|
+
await this.#listen(this.#notifyRoute);
|
|
201
210
|
}
|
|
202
211
|
// index.js:159-164
|
|
203
212
|
this.#supervisor.start(this.db);
|
|
@@ -207,6 +216,58 @@ export class SqliteJobQueue extends EventEmitter {
|
|
|
207
216
|
this.#lifecycle = { kind: 'running' };
|
|
208
217
|
return this;
|
|
209
218
|
}
|
|
219
|
+
#openNotifyRoute() {
|
|
220
|
+
if (this.db.location() === null) {
|
|
221
|
+
return { kind: 'memory' };
|
|
222
|
+
}
|
|
223
|
+
const clock = this.#clock;
|
|
224
|
+
const notifications = new Notifications({
|
|
225
|
+
database: this.db,
|
|
226
|
+
clock: {
|
|
227
|
+
setTimeout: (callback, ms) => clock.setTimeout(callback, ms),
|
|
228
|
+
clearTimeout: (timer) => clock.clearTimeout(timer),
|
|
229
|
+
performance: { now: () => clock.now() },
|
|
230
|
+
},
|
|
231
|
+
});
|
|
232
|
+
// The notifier's errors are the instance's (index.js:92).
|
|
233
|
+
notifications.on('error', (error) => this.emit('error', toError(error)));
|
|
234
|
+
return { kind: 'file', notifications };
|
|
235
|
+
}
|
|
236
|
+
// notifier.js:51-64: a listener that cannot start leaves the workers on
|
|
237
|
+
// their poll. Once it listens, each worker of a notify queue fetches, for
|
|
238
|
+
// what was sent before (adapters/pglite.js:144-152, manager.js:1144-1152).
|
|
239
|
+
async #listen(route) {
|
|
240
|
+
if (route.kind === 'file') {
|
|
241
|
+
route.notifications.on('notification', ({ payload }) => this.notifyQueue(payload));
|
|
242
|
+
try {
|
|
243
|
+
await route.notifications.listen(notifyChannel);
|
|
244
|
+
}
|
|
245
|
+
catch (error) {
|
|
246
|
+
this.emit('warning', {
|
|
247
|
+
message: 'Failed to start LISTEN/NOTIFY listener. Continuing with polling only.',
|
|
248
|
+
data: {
|
|
249
|
+
type: 'listen_notify_unavailable',
|
|
250
|
+
error: toError(error).message,
|
|
251
|
+
},
|
|
252
|
+
});
|
|
253
|
+
return;
|
|
254
|
+
}
|
|
255
|
+
}
|
|
256
|
+
this.#notifier = { kind: 'available' };
|
|
257
|
+
await this.#activity.deliver(() => this.wakeNotifyQueueWorkers());
|
|
258
|
+
}
|
|
259
|
+
// The pg_notify(channel, queue name) of pg-boss's insert, in the insert's
|
|
260
|
+
// transaction (plans.js:2335-2350).
|
|
261
|
+
#notify(name) {
|
|
262
|
+
const route = this.#notifyRoute;
|
|
263
|
+
if (route.kind === 'file') {
|
|
264
|
+
route.notifications.notify(notifyChannel, name);
|
|
265
|
+
}
|
|
266
|
+
else if (route.kind === 'memory' && this.#notifier.kind === 'available') {
|
|
267
|
+
// PGlite gives the NOTIFY as the insert runs (src/pglite.ts:980-992).
|
|
268
|
+
this.notifyQueue(name);
|
|
269
|
+
}
|
|
270
|
+
}
|
|
210
271
|
// node_modules/pg-boss/dist/index.js:192-275.
|
|
211
272
|
async stop(options = {}) {
|
|
212
273
|
const starting = this.#starting.inFlight;
|
|
@@ -235,7 +296,7 @@ export class SqliteJobQueue extends EventEmitter {
|
|
|
235
296
|
// fetch in flight lands before the workers stop and their group tracking
|
|
236
297
|
// is cleared.
|
|
237
298
|
if (this.#notifier.kind === 'available') {
|
|
238
|
-
await
|
|
299
|
+
await this.#unlisten(this.#notifyRoute);
|
|
239
300
|
}
|
|
240
301
|
this.#notifier = { kind: 'unavailable' };
|
|
241
302
|
// manager.js:866-880: refuse new work and let every worker wind down.
|
|
@@ -256,6 +317,7 @@ export class SqliteJobQueue extends EventEmitter {
|
|
|
256
317
|
await setImmediate();
|
|
257
318
|
const shutdown = async () => {
|
|
258
319
|
await this.#failWip();
|
|
320
|
+
await this.#closeNotifyRoute();
|
|
259
321
|
const attachment = this.#attachment;
|
|
260
322
|
this.#attachment = { kind: 'detached' };
|
|
261
323
|
if (attachment.kind === 'attached') {
|
|
@@ -286,6 +348,26 @@ export class SqliteJobQueue extends EventEmitter {
|
|
|
286
348
|
}
|
|
287
349
|
await shutdown();
|
|
288
350
|
}
|
|
351
|
+
// notifier.js:70-77: a close that fails goes to the error event.
|
|
352
|
+
async #unlisten(route) {
|
|
353
|
+
if (route.kind !== 'file') {
|
|
354
|
+
await setImmediate();
|
|
355
|
+
return;
|
|
356
|
+
}
|
|
357
|
+
try {
|
|
358
|
+
await route.notifications.unlisten(notifyChannel);
|
|
359
|
+
}
|
|
360
|
+
catch (error) {
|
|
361
|
+
this.emit('error', toError(error));
|
|
362
|
+
}
|
|
363
|
+
}
|
|
364
|
+
async #closeNotifyRoute() {
|
|
365
|
+
const route = this.#notifyRoute;
|
|
366
|
+
this.#notifyRoute = { kind: 'stopped' };
|
|
367
|
+
if (route.kind === 'file') {
|
|
368
|
+
await route.notifications[Symbol.asyncDispose]();
|
|
369
|
+
}
|
|
370
|
+
}
|
|
289
371
|
#closeOwnDatabase() {
|
|
290
372
|
if (this.#ownsDatabase && this.#connection.kind === 'open') {
|
|
291
373
|
this.#connection.database.close();
|
|
@@ -498,6 +580,49 @@ export class SqliteJobQueue extends EventEmitter {
|
|
|
498
580
|
}
|
|
499
581
|
this.#queueCache.evict(name);
|
|
500
582
|
}
|
|
583
|
+
// node_modules/pg-boss/dist/manager.js:2131-2228 with persistQueueStats off,
|
|
584
|
+
// the only mode here: SqliteJobQueueOptions has no persistQueueStats, so the
|
|
585
|
+
// history options are neither read nor checked (2166-2186). pg-boss's
|
|
586
|
+
// vacuum backoff (2211-2213) has no counterpart (see Supervisor).
|
|
587
|
+
async getQueueStats(name, options = {}) {
|
|
588
|
+
Attorney.assertQueueName(name);
|
|
589
|
+
const cached = QueueStatsCache.read(this.db, name);
|
|
590
|
+
if (!cached) {
|
|
591
|
+
throw new Error(`Queue ${name} does not exist`);
|
|
592
|
+
}
|
|
593
|
+
const maxCacheAgeMs = (options.force
|
|
594
|
+
? QUEUE_STATS_FORCE_TTL_SECONDS
|
|
595
|
+
: Math.max(QUEUE_STATS_CACHE_TTL_SECONDS, this.#intervals.monitorIntervalSeconds, this.#intervals.superviseIntervalSeconds)) * 1000;
|
|
596
|
+
// A queue never counted has an infinite cache age (manager.js:2201).
|
|
597
|
+
const { capturedOn } = cached;
|
|
598
|
+
if (capturedOn !== null &&
|
|
599
|
+
this.#clock.now() - capturedOn.getTime() <= maxCacheAgeMs) {
|
|
600
|
+
return [SqliteJobQueue.#toQueueStats(name, { ...cached, capturedOn })];
|
|
601
|
+
}
|
|
602
|
+
// SQLite has one writer, so the refresh cannot lose pg-boss's try-lock to
|
|
603
|
+
// another instance's scan and fall back to the cache (manager.js:
|
|
604
|
+
// 2217-2227): it always counts the queue and returns its row.
|
|
605
|
+
return QueueStatsCache.refresh(this.db, [name], this.#now()).map((row) => SqliteJobQueue.#toQueueStats(name, row));
|
|
606
|
+
}
|
|
607
|
+
// toSnapshot (manager.js:2136-2165) of the cache, which holds gauges only:
|
|
608
|
+
// the deltas stay null with persistQueueStats off.
|
|
609
|
+
static #toQueueStats(name, cached) {
|
|
610
|
+
return {
|
|
611
|
+
name,
|
|
612
|
+
deferredCount: cached.deferredCount,
|
|
613
|
+
queuedCount: cached.queuedCount,
|
|
614
|
+
readyCount: cached.readyCount,
|
|
615
|
+
activeCount: cached.activeCount,
|
|
616
|
+
failedCount: cached.failedCount,
|
|
617
|
+
totalCount: cached.totalCount,
|
|
618
|
+
completedDelta: null,
|
|
619
|
+
failedDelta: null,
|
|
620
|
+
createdDelta: null,
|
|
621
|
+
deltaSeconds: null,
|
|
622
|
+
deltaOn: null,
|
|
623
|
+
capturedOn: cached.capturedOn,
|
|
624
|
+
};
|
|
625
|
+
}
|
|
501
626
|
// node_modules/pg-boss/dist/manager.js:1179-1182, 1232-1258.
|
|
502
627
|
async send(...args) {
|
|
503
628
|
return this.#send(args);
|
|
@@ -517,31 +642,30 @@ export class SqliteJobQueue extends EventEmitter {
|
|
|
517
642
|
// hold refuses the send, whichever field holds it.
|
|
518
643
|
Jsonb.assertEscapes(JobRows.payload(queue.name, data, options));
|
|
519
644
|
const now = this.#now();
|
|
520
|
-
|
|
521
|
-
|
|
522
|
-
|
|
523
|
-
|
|
524
|
-
|
|
525
|
-
|
|
526
|
-
|
|
527
|
-
|
|
528
|
-
|
|
529
|
-
|
|
530
|
-
|
|
531
|
-
|
|
532
|
-
|
|
533
|
-
|
|
534
|
-
|
|
535
|
-
|
|
536
|
-
|
|
537
|
-
|
|
538
|
-
|
|
539
|
-
|
|
540
|
-
|
|
541
|
-
|
|
542
|
-
|
|
543
|
-
}
|
|
544
|
-
return stored.id;
|
|
645
|
+
return transaction(this.db, () => {
|
|
646
|
+
const stored = this.#insertJob(queue.name, data, options, queue, now, {
|
|
647
|
+
startAfter: options.startAfter,
|
|
648
|
+
singletonOn: Throttle.slot(options.singletonSeconds, 0, now),
|
|
649
|
+
}) ??
|
|
650
|
+
// manager.js:1253-1256: a send a unique index refused retries once
|
|
651
|
+
// into the next throttle slot, starting when that slot does.
|
|
652
|
+
(options.singletonNextSlot
|
|
653
|
+
? this.#insertJob(queue.name, data, options, queue, now, {
|
|
654
|
+
startAfter: Throttle.debounceStartAfter(options.singletonSeconds, now),
|
|
655
|
+
singletonOn: Throttle.slot(options.singletonSeconds, options.singletonSeconds, now),
|
|
656
|
+
})
|
|
657
|
+
: null);
|
|
658
|
+
if (stored === null) {
|
|
659
|
+
return null;
|
|
660
|
+
}
|
|
661
|
+
// A notify queue's insert notifies when the stored job is due now,
|
|
662
|
+
// whether or not this instance listens (node_modules/pg-boss/dist/
|
|
663
|
+
// plans.js:2335-2350, manager.js:1132-1134, 1241).
|
|
664
|
+
if (queue.notify && stored.startAfter <= SqlValues.asIso(now)) {
|
|
665
|
+
this.#notify(queue.name);
|
|
666
|
+
}
|
|
667
|
+
return stored.id;
|
|
668
|
+
}, { mode: 'immediate' });
|
|
545
669
|
});
|
|
546
670
|
}
|
|
547
671
|
// manager.js:1360-1440 with __singletonSlots, the cron pass's own insert:
|
|
@@ -552,8 +676,11 @@ export class SqliteJobQueue extends EventEmitter {
|
|
|
552
676
|
const queue = this.getQueueOrThrow(name);
|
|
553
677
|
const now = this.#now();
|
|
554
678
|
transaction(this.db, () => {
|
|
555
|
-
|
|
556
|
-
|
|
679
|
+
const stored = jobs.map((job) => this.#insertJob(queue.name, job.data, { singletonKey: job.singletonKey }, queue, now, { startAfter: undefined, singletonOn: job.singletonOn }));
|
|
680
|
+
// One pg_notify when an inserted job is due now (plans.js:2343-2347).
|
|
681
|
+
if (queue.notify &&
|
|
682
|
+
stored.some((job) => job !== null && job.startAfter <= SqlValues.asIso(now))) {
|
|
683
|
+
this.#notify(queue.name);
|
|
557
684
|
}
|
|
558
685
|
}, { mode: 'immediate' });
|
|
559
686
|
});
|
|
@@ -804,6 +931,13 @@ export class SqliteJobQueue extends EventEmitter {
|
|
|
804
931
|
this.getQueueOrThrow(name);
|
|
805
932
|
return JobSettler.response(ids, JobSettler.complete(this.db, name, ids, attempts, data, Boolean(options.includeQueued), this.#now()));
|
|
806
933
|
}
|
|
934
|
+
// node_modules/pg-boss/dist/manager.js:1994-2002
|
|
935
|
+
async touch(name, id) {
|
|
936
|
+
Attorney.assertQueueName(name);
|
|
937
|
+
const { ids, attempts } = Attorney.mapAttemptArg(id, 'touch');
|
|
938
|
+
this.getQueueOrThrow(name);
|
|
939
|
+
return JobSettler.response(ids, JobSettler.touch(this.db, name, ids, attempts, this.#now()));
|
|
940
|
+
}
|
|
807
941
|
async fail(name, id, data) {
|
|
808
942
|
Attorney.assertQueueName(name);
|
|
809
943
|
const { ids, attempts } = Attorney.mapAttemptArg(id, 'fail');
|
|
@@ -868,11 +1002,6 @@ export class SqliteJobQueue extends EventEmitter {
|
|
|
868
1002
|
WHERE queue_name = ? AND key = ?`)
|
|
869
1003
|
.run(SqlValues.text(name), SqlValues.text(key));
|
|
870
1004
|
}
|
|
871
|
-
applySchema() {
|
|
872
|
-
queueSchema.apply(this.db);
|
|
873
|
-
// The cron claim's one row (plans.js:288-298).
|
|
874
|
-
this.db.exec('INSERT OR IGNORE INTO pgboss_version (singleton) VALUES (1)');
|
|
875
|
-
}
|
|
876
1005
|
mapQueueUpdateOptions(options) {
|
|
877
1006
|
const entries = [];
|
|
878
1007
|
this.pushOption(entries, 'retry_limit', options.retryLimit);
|
|
@@ -1080,7 +1209,7 @@ export class SqliteJobQueue extends EventEmitter {
|
|
|
1080
1209
|
let refreshed;
|
|
1081
1210
|
try {
|
|
1082
1211
|
this.getQueueOrThrow(name);
|
|
1083
|
-
refreshed = JobSettler.touch(this.db, name, held.map(({ row }) => row), this.#now());
|
|
1212
|
+
refreshed = new Set(JobSettler.touch(this.db, name, held.map(({ row }) => row.id), held.map(({ row }) => row.retry_count), this.#now()));
|
|
1084
1213
|
}
|
|
1085
1214
|
catch (error) {
|
|
1086
1215
|
// A heartbeat that could not run says nothing about the claim.
|