cairnq 0.8.0 → 0.10.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +73 -2
- package/dist/_protocol/sql/postgres/installations.sql +33 -0
- package/dist/client.d.ts +19 -4
- package/dist/client.js +20 -5
- package/dist/context.d.ts +29 -2
- package/dist/context.js +44 -7
- package/dist/errors.d.ts +17 -0
- package/dist/errors.js +20 -0
- package/dist/index.d.ts +6 -2
- package/dist/index.js +4 -1
- package/dist/models.js +27 -0
- package/dist/store/base.d.ts +99 -0
- package/dist/store/base.js +87 -11
- package/dist/store/pg-executor.d.ts +78 -0
- package/dist/store/pg-executor.js +27 -0
- package/dist/store/pg-pool.d.ts +16 -0
- package/dist/store/pg-pool.js +148 -0
- package/dist/store/postgres.d.ts +64 -13
- package/dist/store/postgres.js +191 -147
- package/dist/store/sqlite.js +20 -2
- package/dist/worker.d.ts +10 -5
- package/dist/worker.js +16 -13
- package/package.json +6 -4
- package/src/client.ts +33 -6
- package/src/context.ts +46 -5
- package/src/errors.ts +21 -0
- package/src/index.ts +13 -1
- package/src/models.ts +27 -0
- package/src/store/base.ts +150 -0
- package/src/store/pg-executor.ts +91 -0
- package/src/store/pg-pool.ts +157 -0
- package/src/store/postgres.ts +205 -141
- package/src/store/sqlite.ts +23 -2
- package/src/worker.ts +18 -14
package/README.md
CHANGED
|
@@ -1,8 +1,12 @@
|
|
|
1
1
|
# cairnq (TypeScript / Node)
|
|
2
2
|
|
|
3
3
|
SQLite-first, cross-language, storage-centered durable task runtime. The
|
|
4
|
-
TypeScript SDK (Node ≥ 20
|
|
5
|
-
|
|
4
|
+
TypeScript SDK (Node ≥ 20). API and worker processes coordinate only through a
|
|
5
|
+
shared SQLite file.
|
|
6
|
+
|
|
7
|
+
Both drivers are optional peers — install the one you use:
|
|
8
|
+
`npm i better-sqlite3` for the SQLite backend, `npm i pg` for Postgres. Importing
|
|
9
|
+
`cairnq` loads neither, so a Postgres-only deployment builds no native module.
|
|
6
10
|
|
|
7
11
|
```ts
|
|
8
12
|
import { CairnQ, Worker } from "cairnq";
|
|
@@ -98,5 +102,72 @@ worker.task("long.job", async (ctx) => {
|
|
|
98
102
|
Same code, Postgres instead of the file — `CairnQ.postgres(dsn)` /
|
|
99
103
|
`Worker.postgres(dsn)`. Requires the optional `pg` peer dependency (`npm i pg`).
|
|
100
104
|
|
|
105
|
+
### Sharing the application's connection
|
|
106
|
+
|
|
107
|
+
Given a `PgExecutor` instead of a DSN, cairnq runs inside a session the
|
|
108
|
+
application already has — no second driver, no second pool:
|
|
109
|
+
|
|
110
|
+
```ts
|
|
111
|
+
import { CairnQ, type PgExecutor } from "cairnq";
|
|
112
|
+
|
|
113
|
+
const executor: PgExecutor = { /* ~30 lines over your driver */ };
|
|
114
|
+
const tasks = CairnQ.postgres(executor);
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
`schema` puts cairnq's tables in a schema of their own:
|
|
118
|
+
`CairnQ.postgres(dsn, { schema: "cairnq" })` creates it if absent and sets
|
|
119
|
+
`search_path` per connection. The protocol's SQL names no schema, so nothing else
|
|
120
|
+
changes. With your own executor the search_path is yours to set, and `schema`
|
|
121
|
+
becomes an assertion about where it lands.
|
|
122
|
+
|
|
123
|
+
**Every process in a deployment must agree on it.** A queue whose API and worker
|
|
124
|
+
resolve to different schemas is two empty queues, and — because every migration is
|
|
125
|
+
`create table if not exists` — both sides come up healthy, pass their protocol
|
|
126
|
+
version check, and never see each other's tasks. cairnq refuses to connect where
|
|
127
|
+
it can see that about to happen (`SchemaMismatch`); the Python SDK applies the
|
|
128
|
+
same rule.
|
|
129
|
+
|
|
130
|
+
Two things an adapter must get right: `int8` has to come back as a JS number
|
|
131
|
+
(every cairnq `*_ms` is an epoch or a counter, all inside the safe range), and
|
|
132
|
+
`jsonb` as an object rather than a string. An executor cairnq was handed is
|
|
133
|
+
never closed by cairnq.
|
|
134
|
+
|
|
135
|
+
That shared session is also what lets a task's settlement commit together with
|
|
136
|
+
the rows the task produced:
|
|
137
|
+
|
|
138
|
+
```ts
|
|
139
|
+
worker.task("render.document", async (ctx, payload) => {
|
|
140
|
+
const rendered = await render(payload);
|
|
141
|
+
return ctx.succeedIn(async (session) => {
|
|
142
|
+
await db.withSession(session).insert(pages).values(rendered);
|
|
143
|
+
return { pages: rendered.length }; // becomes the task's result
|
|
144
|
+
});
|
|
145
|
+
});
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
Without it the two are separate transactions, and a crash between them leaves
|
|
149
|
+
work durable while the task still reads as running — on retry, recomputed. If
|
|
150
|
+
the lease turns out to be gone, the settlement matches no row and the caller's
|
|
151
|
+
writes roll back with it.
|
|
152
|
+
|
|
153
|
+
## Watching
|
|
154
|
+
|
|
155
|
+
`watch` calls back when the tasks on a queue may have changed — for a dashboard
|
|
156
|
+
that would otherwise poll:
|
|
157
|
+
|
|
158
|
+
```ts
|
|
159
|
+
const stop = tasks.watch({ queues: ["render"] }, async (signal) => {
|
|
160
|
+
if (signal.reason === "done") return refreshOne(signal.taskId!);
|
|
161
|
+
setCounts(await tasks.stats());
|
|
162
|
+
});
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
It is notify-accelerated polling, not an event log. On Postgres an idle watch
|
|
166
|
+
costs nothing and a signal lands within milliseconds; where LISTEN is
|
|
167
|
+
unavailable — a transaction-mode pooler, or SQLite, which has no channel — the
|
|
168
|
+
timer alone still delivers `poll` signals. So the same consumer is correct either
|
|
169
|
+
way, and only its promptness differs. Treat a signal as "re-read now"; the truth
|
|
170
|
+
is in `stats()` / `list()` / `get()`.
|
|
171
|
+
|
|
101
172
|
The protocol (schema + canonical SQL) lives in `../cairnq-protocol` and is shared
|
|
102
173
|
verbatim with the Python SDK. See `../cairnq-protocol/PROTOCOL.md`.
|
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
-- Where cairnq already lives in this database, and where this connection is
|
|
2
|
+
-- pointing. Read-only; runs once per connect, before migrations.
|
|
3
|
+
--
|
|
4
|
+
-- Exists because `search_path` is out-of-band configuration: two processes given
|
|
5
|
+
-- the same DSN can still resolve to different schemas, and because every
|
|
6
|
+
-- migration is `create table if not exists`, the second one to start does not
|
|
7
|
+
-- fail — it quietly builds a parallel, empty installation. Nothing downstream can
|
|
8
|
+
-- tell: protocol_version reads from whichever cairnq_meta the connection sees, so
|
|
9
|
+
-- the version check passes on both sides while the API's tasks are invisible to
|
|
10
|
+
-- the worker forever. The only way to catch that is to look OUTSIDE the
|
|
11
|
+
-- connection's own search_path, which is what this does.
|
|
12
|
+
--
|
|
13
|
+
-- One row per installation, never zero: the LEFT JOIN keeps `current_schema`
|
|
14
|
+
-- readable on a database that holds no cairnq yet, where `schema` is null.
|
|
15
|
+
-- Deliberately NOT an array column — pg_namespace.nspname is `name`, and which
|
|
16
|
+
-- drivers decode a `name[]` (or a text[]) into a list is exactly the kind of
|
|
17
|
+
-- disagreement this protocol keeps out of the SDKs. Scalar columns behave the
|
|
18
|
+
-- same everywhere.
|
|
19
|
+
--
|
|
20
|
+
-- `current_schema()` is null when the search_path names nothing that exists, in
|
|
21
|
+
-- which case the caller cannot conclude anything.
|
|
22
|
+
-- params: (none)
|
|
23
|
+
select
|
|
24
|
+
current_schema()::text as current_schema,
|
|
25
|
+
found.schema
|
|
26
|
+
from (select 1) one
|
|
27
|
+
left join (
|
|
28
|
+
select n.nspname::text as schema
|
|
29
|
+
from pg_class c
|
|
30
|
+
join pg_namespace n on n.oid = c.relnamespace
|
|
31
|
+
where c.relname = 'cairnq_tasks' and c.relkind = 'r'
|
|
32
|
+
) found on true
|
|
33
|
+
order by found.schema;
|
package/dist/client.d.ts
CHANGED
|
@@ -1,7 +1,8 @@
|
|
|
1
1
|
import type { BackpressureOptions } from "./backpressure.js";
|
|
2
2
|
import { type RetentionOptions } from "./retention.js";
|
|
3
3
|
import { type Task, type TaskRef, type TaskStatus } from "./models.js";
|
|
4
|
-
import type {
|
|
4
|
+
import type { PgExecutor } from "./store/pg-executor.js";
|
|
5
|
+
import type { ListInput, PurgeInput, SubmitInput, TaskStore, WatchOptions, WatchSignal } from "./store/base.js";
|
|
5
6
|
import { type TaskDef } from "./task.js";
|
|
6
7
|
import { type PollOptions } from "./wait.js";
|
|
7
8
|
export type SubmitOptions = Omit<SubmitInput, "name" | "payload">;
|
|
@@ -28,10 +29,13 @@ export declare class CairnQ {
|
|
|
28
29
|
static sqlite(path: string, opts?: {
|
|
29
30
|
busyTimeoutMs?: number;
|
|
30
31
|
} & ClientOptions): CairnQ;
|
|
31
|
-
/** Multi-host backend. `
|
|
32
|
-
* optional `pg` package
|
|
33
|
-
|
|
32
|
+
/** Multi-host backend. `source` is a libpq connection string — which requires
|
|
33
|
+
* the optional `pg` package — or a PgExecutor over a driver the application
|
|
34
|
+
* already runs (an ORM's pool, say), which cairnq then shares instead of
|
|
35
|
+
* opening a second one. */
|
|
36
|
+
static postgres(source: string | PgExecutor, opts?: {
|
|
34
37
|
max?: number;
|
|
38
|
+
schema?: string;
|
|
35
39
|
} & ClientOptions): CairnQ;
|
|
36
40
|
get store(): TaskStore;
|
|
37
41
|
connect(): Promise<void>;
|
|
@@ -73,6 +77,17 @@ export declare class CairnQ {
|
|
|
73
77
|
/** Task counts per queue, keyed by status and zero-filled across all statuses
|
|
74
78
|
* — `(await stats()).default.queued` is the backlog of a queue. */
|
|
75
79
|
stats(): Promise<Record<string, Record<TaskStatus, number>>>;
|
|
80
|
+
/**
|
|
81
|
+
* Call `onSignal` when the tasks on `queues` may have changed. Returns an
|
|
82
|
+
* unsubscribe.
|
|
83
|
+
*
|
|
84
|
+
* Notify-accelerated polling, not an event log: a signal means "re-read now",
|
|
85
|
+
* and `stats()` / `list()` / `get()` are where the truth is. On Postgres an
|
|
86
|
+
* idle watch costs nothing and signals land in milliseconds; everywhere else
|
|
87
|
+
* the timer alone still delivers, so the same consumer code is correct either
|
|
88
|
+
* way. See TaskStore.watch for the full contract.
|
|
89
|
+
*/
|
|
90
|
+
watch(opts: WatchOptions, onSignal: (signal: WatchSignal) => void): () => void;
|
|
76
91
|
/** Wait for a task to finish. Resolves with the terminal Task (any status);
|
|
77
92
|
* throws TaskTimeout without stopping the task, so `wait(err.taskId)` picks the
|
|
78
93
|
* same wait back up — from another process, or after a longer deadline. */
|
package/dist/client.js
CHANGED
|
@@ -29,11 +29,13 @@ export class CairnQ {
|
|
|
29
29
|
const { busyTimeoutMs, ...client } = opts;
|
|
30
30
|
return new CairnQ(new SQLiteStore(path, { busyTimeoutMs }), client);
|
|
31
31
|
}
|
|
32
|
-
/** Multi-host backend. `
|
|
33
|
-
* optional `pg` package
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
32
|
+
/** Multi-host backend. `source` is a libpq connection string — which requires
|
|
33
|
+
* the optional `pg` package — or a PgExecutor over a driver the application
|
|
34
|
+
* already runs (an ORM's pool, say), which cairnq then shares instead of
|
|
35
|
+
* opening a second one. */
|
|
36
|
+
static postgres(source, opts = {}) {
|
|
37
|
+
const { max, schema, ...client } = opts;
|
|
38
|
+
return new CairnQ(new PostgresStore(source, { max, schema }), client);
|
|
37
39
|
}
|
|
38
40
|
get store() {
|
|
39
41
|
return this._store;
|
|
@@ -99,6 +101,19 @@ export class CairnQ {
|
|
|
99
101
|
stats() {
|
|
100
102
|
return this._store.stats();
|
|
101
103
|
}
|
|
104
|
+
/**
|
|
105
|
+
* Call `onSignal` when the tasks on `queues` may have changed. Returns an
|
|
106
|
+
* unsubscribe.
|
|
107
|
+
*
|
|
108
|
+
* Notify-accelerated polling, not an event log: a signal means "re-read now",
|
|
109
|
+
* and `stats()` / `list()` / `get()` are where the truth is. On Postgres an
|
|
110
|
+
* idle watch costs nothing and signals land in milliseconds; everywhere else
|
|
111
|
+
* the timer alone still delivers, so the same consumer code is correct either
|
|
112
|
+
* way. See TaskStore.watch for the full contract.
|
|
113
|
+
*/
|
|
114
|
+
watch(opts, onSignal) {
|
|
115
|
+
return this._store.watch(opts, onSignal);
|
|
116
|
+
}
|
|
102
117
|
/** Wait for a task to finish. Resolves with the terminal Task (any status);
|
|
103
118
|
* throws TaskTimeout without stopping the task, so `wait(err.taskId)` picks the
|
|
104
119
|
* same wait back up — from another process, or after a longer deadline. */
|
package/dist/context.d.ts
CHANGED
|
@@ -2,6 +2,7 @@ import { type FailReason } from "./errors.js";
|
|
|
2
2
|
import { type Task } from "./models.js";
|
|
3
3
|
import type { SubmitOptions } from "./client.js";
|
|
4
4
|
import type { TaskStore } from "./store/base.js";
|
|
5
|
+
import type { PgSession } from "./store/pg-executor.js";
|
|
5
6
|
import { type TaskDef } from "./task.js";
|
|
6
7
|
export interface TaskContextOptions {
|
|
7
8
|
retryBackoffMs?: number;
|
|
@@ -18,7 +19,7 @@ export interface TaskContextOptions {
|
|
|
18
19
|
export declare class TaskContext {
|
|
19
20
|
private readonly store;
|
|
20
21
|
private readonly task;
|
|
21
|
-
readonly
|
|
22
|
+
private readonly ownerId;
|
|
22
23
|
private readonly leaseMs;
|
|
23
24
|
private readonly abort;
|
|
24
25
|
private leaseLost;
|
|
@@ -26,8 +27,10 @@ export declare class TaskContext {
|
|
|
26
27
|
private isSettled;
|
|
27
28
|
private readonly backoffMs;
|
|
28
29
|
private readonly backoffMaxMs;
|
|
29
|
-
constructor(store: TaskStore, task: Task,
|
|
30
|
+
constructor(store: TaskStore, task: Task, ownerId: string, leaseMs: number, opts?: TaskContextOptions);
|
|
30
31
|
get taskId(): string;
|
|
32
|
+
/** The worker running this task — what `worker_id` on the row points at. */
|
|
33
|
+
get workerId(): string;
|
|
31
34
|
get name(): string;
|
|
32
35
|
get queue(): string;
|
|
33
36
|
get attempt(): number;
|
|
@@ -74,6 +77,30 @@ export declare class TaskContext {
|
|
|
74
77
|
* this task was already settled.
|
|
75
78
|
*/
|
|
76
79
|
succeed(result?: unknown): Promise<Task | null>;
|
|
80
|
+
/**
|
|
81
|
+
* Finalize this task as succeeded, committing the caller's own writes in the
|
|
82
|
+
* SAME transaction as the settlement. Whatever `write` returns becomes the
|
|
83
|
+
* task's result.
|
|
84
|
+
*
|
|
85
|
+
* await ctx.succeedIn(async (session) => {
|
|
86
|
+
* await db.withSession(session).insert(pages).values(rendered)
|
|
87
|
+
* return { pages: rendered.length }
|
|
88
|
+
* })
|
|
89
|
+
*
|
|
90
|
+
* The alternative — write the rows, then settle — has a window between the two
|
|
91
|
+
* commits where the work is durable but the task still reads as running. A
|
|
92
|
+
* crash there re-runs the whole task, which for a render or an ingest means
|
|
93
|
+
* recomputing it, and for non-idempotent work means doing it twice.
|
|
94
|
+
*
|
|
95
|
+
* `session` is the driver's, so this needs a Postgres store built on a
|
|
96
|
+
* PgExecutor the application shares with its own driver; anything else throws.
|
|
97
|
+
* If the settlement finds the lease gone, `write`'s work is rolled back with
|
|
98
|
+
* it and LostLease is raised. Returns null if this task was already settled.
|
|
99
|
+
*
|
|
100
|
+
* `write` may be replayed if the backend retries the transaction — derive
|
|
101
|
+
* nothing inside it that cannot be derived twice.
|
|
102
|
+
*/
|
|
103
|
+
succeedIn<T>(write: (session: PgSession) => Promise<T>): Promise<Task | null>;
|
|
77
104
|
/**
|
|
78
105
|
* Finalize this task as failed, now. `error` may be a string reason, an Error,
|
|
79
106
|
* a TaskError (which carries its own retryability), or a ready envelope.
|
package/dist/context.js
CHANGED
|
@@ -14,7 +14,7 @@ import { pollWait } from "./wait.js";
|
|
|
14
14
|
export class TaskContext {
|
|
15
15
|
store;
|
|
16
16
|
task;
|
|
17
|
-
|
|
17
|
+
ownerId;
|
|
18
18
|
leaseMs;
|
|
19
19
|
abort = new AbortController();
|
|
20
20
|
leaseLost = false;
|
|
@@ -28,10 +28,10 @@ export class TaskContext {
|
|
|
28
28
|
isSettled = false;
|
|
29
29
|
backoffMs;
|
|
30
30
|
backoffMaxMs;
|
|
31
|
-
constructor(store, task,
|
|
31
|
+
constructor(store, task, ownerId, leaseMs, opts = {}) {
|
|
32
32
|
this.store = store;
|
|
33
33
|
this.task = task;
|
|
34
|
-
this.
|
|
34
|
+
this.ownerId = ownerId;
|
|
35
35
|
this.leaseMs = leaseMs;
|
|
36
36
|
this.backoffMs = opts.retryBackoffMs ?? DEFAULT_RETRY_BACKOFF_MS;
|
|
37
37
|
this.backoffMaxMs = opts.retryBackoffMaxMs ?? DEFAULT_RETRY_BACKOFF_MAX_MS;
|
|
@@ -39,6 +39,10 @@ export class TaskContext {
|
|
|
39
39
|
get taskId() {
|
|
40
40
|
return this.task.id;
|
|
41
41
|
}
|
|
42
|
+
/** The worker running this task — what `worker_id` on the row points at. */
|
|
43
|
+
get workerId() {
|
|
44
|
+
return this.ownerId;
|
|
45
|
+
}
|
|
42
46
|
get name() {
|
|
43
47
|
return this.task.name;
|
|
44
48
|
}
|
|
@@ -138,7 +142,7 @@ export class TaskContext {
|
|
|
138
142
|
async progress(value, message = null) {
|
|
139
143
|
return this.owned(() => this.store.progress({
|
|
140
144
|
taskId: this.task.id,
|
|
141
|
-
workerId: this.
|
|
145
|
+
workerId: this.ownerId,
|
|
142
146
|
progress: value,
|
|
143
147
|
message,
|
|
144
148
|
}));
|
|
@@ -146,7 +150,7 @@ export class TaskContext {
|
|
|
146
150
|
async heartbeat() {
|
|
147
151
|
return this.owned(() => this.store.heartbeat({
|
|
148
152
|
taskId: this.task.id,
|
|
149
|
-
workerId: this.
|
|
153
|
+
workerId: this.ownerId,
|
|
150
154
|
leaseMs: this.leaseMs,
|
|
151
155
|
}));
|
|
152
156
|
}
|
|
@@ -180,7 +184,40 @@ export class TaskContext {
|
|
|
180
184
|
async succeed(result = null) {
|
|
181
185
|
if (this.isSettled)
|
|
182
186
|
return null;
|
|
183
|
-
const task = await this.owned(() => this.store.complete({ taskId: this.task.id, workerId: this.
|
|
187
|
+
const task = await this.owned(() => this.store.complete({ taskId: this.task.id, workerId: this.ownerId, result }));
|
|
188
|
+
this.markSettled();
|
|
189
|
+
return task;
|
|
190
|
+
}
|
|
191
|
+
/**
|
|
192
|
+
* Finalize this task as succeeded, committing the caller's own writes in the
|
|
193
|
+
* SAME transaction as the settlement. Whatever `write` returns becomes the
|
|
194
|
+
* task's result.
|
|
195
|
+
*
|
|
196
|
+
* await ctx.succeedIn(async (session) => {
|
|
197
|
+
* await db.withSession(session).insert(pages).values(rendered)
|
|
198
|
+
* return { pages: rendered.length }
|
|
199
|
+
* })
|
|
200
|
+
*
|
|
201
|
+
* The alternative — write the rows, then settle — has a window between the two
|
|
202
|
+
* commits where the work is durable but the task still reads as running. A
|
|
203
|
+
* crash there re-runs the whole task, which for a render or an ingest means
|
|
204
|
+
* recomputing it, and for non-idempotent work means doing it twice.
|
|
205
|
+
*
|
|
206
|
+
* `session` is the driver's, so this needs a Postgres store built on a
|
|
207
|
+
* PgExecutor the application shares with its own driver; anything else throws.
|
|
208
|
+
* If the settlement finds the lease gone, `write`'s work is rolled back with
|
|
209
|
+
* it and LostLease is raised. Returns null if this task was already settled.
|
|
210
|
+
*
|
|
211
|
+
* `write` may be replayed if the backend retries the transaction — derive
|
|
212
|
+
* nothing inside it that cannot be derived twice.
|
|
213
|
+
*/
|
|
214
|
+
async succeedIn(write) {
|
|
215
|
+
if (this.isSettled)
|
|
216
|
+
return null;
|
|
217
|
+
const task = await this.owned(async () => {
|
|
218
|
+
const { task } = await this.store.completeIn({ taskId: this.task.id, workerId: this.ownerId }, write);
|
|
219
|
+
return task;
|
|
220
|
+
});
|
|
184
221
|
this.markSettled();
|
|
185
222
|
return task;
|
|
186
223
|
}
|
|
@@ -196,7 +233,7 @@ export class TaskContext {
|
|
|
196
233
|
const [envelope, retryable] = asEnvelope(error, opts.retryable ?? true);
|
|
197
234
|
const task = await this.owned(() => this.store.fail({
|
|
198
235
|
taskId: this.task.id,
|
|
199
|
-
workerId: this.
|
|
236
|
+
workerId: this.ownerId,
|
|
200
237
|
error: envelope,
|
|
201
238
|
retryable,
|
|
202
239
|
delayMs: failDelayMs(this.task.attempt, retryable, this.backoffMs, this.backoffMaxMs),
|
package/dist/errors.d.ts
CHANGED
|
@@ -101,6 +101,23 @@ export declare class LostLease extends CairnQError {
|
|
|
101
101
|
export declare class ProtocolVersionMismatch extends CairnQError {
|
|
102
102
|
constructor(message: string);
|
|
103
103
|
}
|
|
104
|
+
/**
|
|
105
|
+
* This connection is not pointed at the cairnq installation the rest of the
|
|
106
|
+
* deployment is using — raised at connect, before any task is written.
|
|
107
|
+
*
|
|
108
|
+
* The schema a Postgres connection resolves to is out-of-band configuration
|
|
109
|
+
* (`search_path`, a `schema` option, an ORM's pool settings), so two processes
|
|
110
|
+
* given the same DSN can still land in different schemas. Every migration is
|
|
111
|
+
* `create table if not exists`, so the odd one out does not fail: it builds a
|
|
112
|
+
* second, empty installation and its protocol version check passes against the
|
|
113
|
+
* cairnq_meta it just created. Left undetected, an API and a worker then agree
|
|
114
|
+
* about everything except WHERE, and no task ever crosses.
|
|
115
|
+
*
|
|
116
|
+
* The Python SDK raises the same named error.
|
|
117
|
+
*/
|
|
118
|
+
export declare class SchemaMismatch extends CairnQError {
|
|
119
|
+
constructor(message: string);
|
|
120
|
+
}
|
|
104
121
|
/** A value could not be encoded for a protocol JSON column (non-finite number,
|
|
105
122
|
* BigInt, circular structure, …). Raised at the boundary — submit rejects with
|
|
106
123
|
* it, and a worker records a handler result that triggers it as a permanent
|
package/dist/errors.js
CHANGED
|
@@ -187,6 +187,26 @@ export class ProtocolVersionMismatch extends CairnQError {
|
|
|
187
187
|
this.name = "ProtocolVersionMismatch";
|
|
188
188
|
}
|
|
189
189
|
}
|
|
190
|
+
/**
|
|
191
|
+
* This connection is not pointed at the cairnq installation the rest of the
|
|
192
|
+
* deployment is using — raised at connect, before any task is written.
|
|
193
|
+
*
|
|
194
|
+
* The schema a Postgres connection resolves to is out-of-band configuration
|
|
195
|
+
* (`search_path`, a `schema` option, an ORM's pool settings), so two processes
|
|
196
|
+
* given the same DSN can still land in different schemas. Every migration is
|
|
197
|
+
* `create table if not exists`, so the odd one out does not fail: it builds a
|
|
198
|
+
* second, empty installation and its protocol version check passes against the
|
|
199
|
+
* cairnq_meta it just created. Left undetected, an API and a worker then agree
|
|
200
|
+
* about everything except WHERE, and no task ever crosses.
|
|
201
|
+
*
|
|
202
|
+
* The Python SDK raises the same named error.
|
|
203
|
+
*/
|
|
204
|
+
export class SchemaMismatch extends CairnQError {
|
|
205
|
+
constructor(message) {
|
|
206
|
+
super(message);
|
|
207
|
+
this.name = "SchemaMismatch";
|
|
208
|
+
}
|
|
209
|
+
}
|
|
190
210
|
/** A value could not be encoded for a protocol JSON column (non-finite number,
|
|
191
211
|
* BigInt, circular structure, …). Raised at the boundary — submit rejects with
|
|
192
212
|
* it, and a worker records a handler result that triggers it as a permanent
|
package/dist/index.d.ts
CHANGED
|
@@ -12,9 +12,13 @@ export { defineTask } from "./task.js";
|
|
|
12
12
|
export type { TaskDef } from "./task.js";
|
|
13
13
|
export { SQLiteStore } from "./store/sqlite.js";
|
|
14
14
|
export { PostgresStore } from "./store/postgres.js";
|
|
15
|
+
export { ListenUnavailable } from "./store/pg-executor.js";
|
|
16
|
+
export type { PgExecutor, PgSession, Row } from "./store/pg-executor.js";
|
|
17
|
+
export { createPoolExecutor } from "./store/pg-pool.js";
|
|
15
18
|
export { TaskStore } from "./store/base.js";
|
|
16
|
-
export type { ListInput, PurgeInput, SubmitInput, Conflict } from "./store/base.js";
|
|
19
|
+
export type { ListInput, PurgeInput, SubmitInput, Conflict, WatchOptions, WatchSignal, } from "./store/base.js";
|
|
20
|
+
export { DEFAULT_WATCH_POLL_MS } from "./store/base.js";
|
|
17
21
|
export type { Task, TaskRef, TaskStatus, TerminalStatus } from "./models.js";
|
|
18
22
|
export { STATUSES, isTerminal, isTerminalStatus, cancelRequested, isQueued, isRunning, isSucceeded, isFailed, isCanceled, } from "./models.js";
|
|
19
|
-
export { CairnQError, AlreadyExists, QueueFull, TaskTimeout, TaskFailed, TaskCanceled, TaskError, LostLease, EventLoopBlocked, ProtocolVersionMismatch, SerializationError, } from "./errors.js";
|
|
23
|
+
export { CairnQError, AlreadyExists, QueueFull, TaskTimeout, TaskFailed, TaskCanceled, TaskError, LostLease, EventLoopBlocked, ProtocolVersionMismatch, SchemaMismatch, SerializationError, } from "./errors.js";
|
|
20
24
|
export type { FailReason } from "./errors.js";
|
package/dist/index.js
CHANGED
|
@@ -6,6 +6,9 @@ export { TaskContext } from "./context.js";
|
|
|
6
6
|
export { defineTask } from "./task.js";
|
|
7
7
|
export { SQLiteStore } from "./store/sqlite.js";
|
|
8
8
|
export { PostgresStore } from "./store/postgres.js";
|
|
9
|
+
export { ListenUnavailable } from "./store/pg-executor.js";
|
|
10
|
+
export { createPoolExecutor } from "./store/pg-pool.js";
|
|
9
11
|
export { TaskStore } from "./store/base.js";
|
|
12
|
+
export { DEFAULT_WATCH_POLL_MS } from "./store/base.js";
|
|
10
13
|
export { STATUSES, isTerminal, isTerminalStatus, cancelRequested, isQueued, isRunning, isSucceeded, isFailed, isCanceled, } from "./models.js";
|
|
11
|
-
export { CairnQError, AlreadyExists, QueueFull, TaskTimeout, TaskFailed, TaskCanceled, TaskError, LostLease, EventLoopBlocked, ProtocolVersionMismatch, SerializationError, } from "./errors.js";
|
|
14
|
+
export { CairnQError, AlreadyExists, QueueFull, TaskTimeout, TaskFailed, TaskCanceled, TaskError, LostLease, EventLoopBlocked, ProtocolVersionMismatch, SchemaMismatch, SerializationError, } from "./errors.js";
|
package/dist/models.js
CHANGED
|
@@ -4,6 +4,19 @@
|
|
|
4
4
|
// conformance suite pins this set against.
|
|
5
5
|
export const STATUSES = ["queued", "running", "succeeded", "failed", "canceled"];
|
|
6
6
|
const JSON_COLUMNS = ["payload", "result", "error", "metadata"];
|
|
7
|
+
// The bigint columns. `attempt` / `max_attempts` / `priority` are int4 and
|
|
8
|
+
// `progress` is double precision, so every driver already gives those as numbers;
|
|
9
|
+
// only int8 has a wire form worth normalizing. Nullability differs per column
|
|
10
|
+
// (completed_at_ms may be null, created_at_ms may not), so the coercion has to
|
|
11
|
+
// preserve null rather than turn it into 0.
|
|
12
|
+
const MS_COLUMNS = [
|
|
13
|
+
"lease_until_ms",
|
|
14
|
+
"run_at_ms",
|
|
15
|
+
"cancel_requested_at_ms",
|
|
16
|
+
"created_at_ms",
|
|
17
|
+
"updated_at_ms",
|
|
18
|
+
"completed_at_ms",
|
|
19
|
+
];
|
|
7
20
|
// As a const tuple so TerminalStatus derives from it — the same declare-once
|
|
8
21
|
// pattern as STATUSES/TaskStatus above.
|
|
9
22
|
export const TERMINAL = ["succeeded", "failed", "canceled"];
|
|
@@ -24,6 +37,20 @@ export function rowToTask(row) {
|
|
|
24
37
|
// already-decoded object. Parse only a string — never assume one backend.
|
|
25
38
|
t[col] = typeof v === "string" ? JSON.parse(v) : (v ?? null);
|
|
26
39
|
}
|
|
40
|
+
for (const col of MS_COLUMNS) {
|
|
41
|
+
const v = row[col];
|
|
42
|
+
// Same argument, for the other column type the drivers disagree about.
|
|
43
|
+
// Postgres sends int8 down the wire as text to protect precision it cannot
|
|
44
|
+
// know is unneeded; `pg` surfaces that as a string, postgres.js does too,
|
|
45
|
+
// asyncpg decodes to int. Normalizing here rather than demanding it of every
|
|
46
|
+
// driver is what keeps an INJECTED executor honest: the alternative is
|
|
47
|
+
// asking an application to change its driver's global int8 handling to suit
|
|
48
|
+
// cairnq, which breaks that application's own bigint columns.
|
|
49
|
+
//
|
|
50
|
+
// Lossless: every one of these is an epoch-ms, and a millisecond timestamp
|
|
51
|
+
// does not reach Number.MAX_SAFE_INTEGER until the year 287396.
|
|
52
|
+
t[col] = v == null ? null : Number(v);
|
|
53
|
+
}
|
|
27
54
|
return t;
|
|
28
55
|
}
|
|
29
56
|
/** Accepts anything carrying a status — a Task or a TaskRef probe. */
|
package/dist/store/base.d.ts
CHANGED
|
@@ -83,6 +83,32 @@ export declare function statementParams(sql: string): readonly string[];
|
|
|
83
83
|
* them in one place is what stops SQLite and Postgres from drifting apart in
|
|
84
84
|
* behavior; the shared SQL already stops them from drifting in wording.
|
|
85
85
|
*/
|
|
86
|
+
/**
|
|
87
|
+
* Why `watch` is calling back.
|
|
88
|
+
*
|
|
89
|
+
* `queued` / `done` come from the store's push channel and name what moved;
|
|
90
|
+
* `poll` is the timer saying the watch cannot rule out a change. None of them
|
|
91
|
+
* carries state — the row is the truth.
|
|
92
|
+
*/
|
|
93
|
+
export interface WatchSignal {
|
|
94
|
+
reason: "queued" | "done" | "poll";
|
|
95
|
+
/** The queue a task was queued on. Only on `queued`. */
|
|
96
|
+
queue?: string;
|
|
97
|
+
/** The task that reached a terminal status. Only on `done`. */
|
|
98
|
+
taskId?: string;
|
|
99
|
+
}
|
|
100
|
+
export interface WatchOptions {
|
|
101
|
+
/** Restrict `queued` signals to these queues. Unset watches every queue. */
|
|
102
|
+
queues?: string[];
|
|
103
|
+
/**
|
|
104
|
+
* How often to signal in the absence of a push channel — and, where there is
|
|
105
|
+
* one, how long a dropped listener can go unnoticed. The default trades a
|
|
106
|
+
* dashboard's idle query rate against how stale it may look.
|
|
107
|
+
*/
|
|
108
|
+
pollMs?: number;
|
|
109
|
+
}
|
|
110
|
+
/** See WatchOptions.pollMs. */
|
|
111
|
+
export declare const DEFAULT_WATCH_POLL_MS = 2000;
|
|
86
112
|
export declare abstract class TaskStore {
|
|
87
113
|
/** Set by useBackpressure; null means submit is ungated. */
|
|
88
114
|
private gate;
|
|
@@ -100,6 +126,34 @@ export declare abstract class TaskStore {
|
|
|
100
126
|
* build ids and payloads before opening the transaction, not within it.
|
|
101
127
|
*/
|
|
102
128
|
protected abstract tx<T>(fn: (fetch: Fetch) => Promise<T>): Promise<T>;
|
|
129
|
+
/**
|
|
130
|
+
* `tx`, but also handing `fn` the driver's own session so the CALLER can run
|
|
131
|
+
* their statements in the same transaction as the protocol's.
|
|
132
|
+
*
|
|
133
|
+
* This is what lets a task's settlement and the rows that task produced commit
|
|
134
|
+
* together. Without it the two are separate transactions and there is a window
|
|
135
|
+
* where the work is durable but the task still reads as unfinished — a crash
|
|
136
|
+
* there costs a full recomputation on retry, and for non-idempotent work costs
|
|
137
|
+
* more than that.
|
|
138
|
+
*
|
|
139
|
+
* Optional, because the session type is the driver's, not the protocol's: a
|
|
140
|
+
* store that has no session worth handing out simply does not implement it and
|
|
141
|
+
* `completeIn` reports that. Postgres implements it; SQLite does not.
|
|
142
|
+
*/
|
|
143
|
+
protected txWithSession?<T>(fn: (fetch: Fetch, session: unknown) => Promise<T>): Promise<T>;
|
|
144
|
+
/**
|
|
145
|
+
* Register for this store's push channel, if it has one; returns an
|
|
146
|
+
* unsubscribe. A store without a push channel does not implement this, and
|
|
147
|
+
* `watch` degrades to its timer alone.
|
|
148
|
+
*/
|
|
149
|
+
protected subscribePush?(onSignal: (signal: WatchSignal) => void): () => void;
|
|
150
|
+
/**
|
|
151
|
+
* Nudge the push channel back up if it has dropped. Called from `watch`'s
|
|
152
|
+
* timer, which is the only thing keeping a client-side subscriber alive: a
|
|
153
|
+
* process that never claims never calls claimWake, so without this a listener
|
|
154
|
+
* that died once would never come back there.
|
|
155
|
+
*/
|
|
156
|
+
protected warmPush?(): void;
|
|
103
157
|
/**
|
|
104
158
|
* Whether it is worth opening the claim transaction at all. SQLite gates its
|
|
105
159
|
* single write lock behind a read-only probe; Postgres readers don't block
|
|
@@ -169,6 +223,28 @@ export declare abstract class TaskStore {
|
|
|
169
223
|
* them.
|
|
170
224
|
*/
|
|
171
225
|
stats(): Promise<Record<string, Record<TaskStatus, number>>>;
|
|
226
|
+
/**
|
|
227
|
+
* Call `onSignal` when the tasks on `queues` may have changed — something was
|
|
228
|
+
* queued, or something finished.
|
|
229
|
+
*
|
|
230
|
+
* This is notify-ACCELERATED POLLING, not an event log, and the difference is
|
|
231
|
+
* the whole contract. Where a push channel is available (Postgres LISTEN) an
|
|
232
|
+
* idle watch costs nothing and a signal arrives within milliseconds of the
|
|
233
|
+
* event. Where it is not — a transaction-mode pooler refuses LISTEN, SQLite has
|
|
234
|
+
* no channel at all — the timer alone still delivers `poll` signals, so a
|
|
235
|
+
* consumer that re-reads on every signal is correct in both cases and merely
|
|
236
|
+
* less prompt in one.
|
|
237
|
+
*
|
|
238
|
+
* What it will NOT do is promise that a signal means something happened, or
|
|
239
|
+
* that every event produces its own signal. Treat a signal as "re-read now"
|
|
240
|
+
* and take the truth from `stats()` / `list()` / `get()`, which is where it
|
|
241
|
+
* lives. `reason` is a hint for reading less: a `done` signal names the task,
|
|
242
|
+
* so a dashboard can refresh that row instead of the list.
|
|
243
|
+
*
|
|
244
|
+
* Returns an unsubscribe. The timer is unref'd — watching does not hold a
|
|
245
|
+
* process open.
|
|
246
|
+
*/
|
|
247
|
+
watch(opts: WatchOptions, onSignal: (signal: WatchSignal) => void): () => void;
|
|
172
248
|
/**
|
|
173
249
|
* How many more tasks fit on `queue` under `maxDepth` — 0 once it is full.
|
|
174
250
|
*
|
|
@@ -262,6 +338,29 @@ export declare abstract class TaskStore {
|
|
|
262
338
|
workerId: string;
|
|
263
339
|
result: unknown;
|
|
264
340
|
}): Promise<Task>;
|
|
341
|
+
/**
|
|
342
|
+
* `complete`, with the caller's own writes committed in the same transaction.
|
|
343
|
+
*
|
|
344
|
+
* `fn` runs first and whatever it returns becomes the task's result; the
|
|
345
|
+
* settlement is the last statement in the transaction. So a lost lease — the
|
|
346
|
+
* settlement matching no row — rolls the caller's writes back with it, and
|
|
347
|
+
* there is no ordering in which the work is recorded but the task is not.
|
|
348
|
+
*
|
|
349
|
+
* The settlement runs LAST rather than checking ownership up front on purpose:
|
|
350
|
+
* the ownership predicate lives in the protocol's complete.sql, and a
|
|
351
|
+
* fail-fast pre-check here would be a second copy of it, free to drift. The
|
|
352
|
+
* cost of that choice is that a doomed attempt does its work before finding
|
|
353
|
+
* out, which is the rare path.
|
|
354
|
+
*
|
|
355
|
+
* `fn` must be replayable for the same reason `tx`'s callback must be.
|
|
356
|
+
*/
|
|
357
|
+
completeIn<S, T>(input: {
|
|
358
|
+
taskId: string;
|
|
359
|
+
workerId: string;
|
|
360
|
+
}, fn: (session: S) => Promise<T>): Promise<{
|
|
361
|
+
task: Task;
|
|
362
|
+
value: T;
|
|
363
|
+
}>;
|
|
265
364
|
fail(input: {
|
|
266
365
|
taskId: string;
|
|
267
366
|
workerId: string;
|