@ultimat3/jobs 27.2.3 → 27.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +6 -6
- package/README.md +12 -2
- package/package.json +8 -7
- package/src/driver-redis-scripts.ts +231 -0
- package/src/driver-redis-steps.ts +64 -0
- package/src/driver-redis.ts +278 -0
- package/src/driver.ts +4 -3
package/CLAUDE.md
CHANGED
|
@@ -64,8 +64,9 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
64
64
|
what a second call would move.**
|
|
65
65
|
- **No read returns a WHOLE row** — `driver-pg-sql.test.ts` scans every production file here
|
|
66
66
|
(discovered, comments stripped) for `select *`/`returning *`.
|
|
67
|
-
- Drivers
|
|
68
|
-
capabilities go behind the interface.
|
|
67
|
+
- Drivers: the six `JobDriver` methods plus optional `introspect`, `backfills`, `leases`; new
|
|
68
|
+
capabilities go behind the interface. **`redisJobDriver`** (`@ultimat3/jobs/redis`, off the
|
|
69
|
+
barrel) is one Lua script per operation: a semantic change lands in memory, pg AND the scripts.
|
|
69
70
|
- Step results are persisted BEFORE the step returns. All time is epoch ms (`nowMs()`, `clock.ts`).
|
|
70
71
|
- **`postgresLeader` is correct only on a DEDICATED connection**; boot uses `postgresLeaseLeader`.
|
|
71
72
|
|
|
@@ -75,8 +76,7 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
75
76
|
`finiteCount()` (a count, caller's minimum); `worker-options.ts` is where `jobWorker` reads them.
|
|
76
77
|
**`bun run finite-bounds` is a floor, never proof**: `concurrencyLimiter`'s five numbers are screened
|
|
77
78
|
with min 0 (zero is a HARD STOP). **A row count is `finiteCount` (min 0)**; `retry.attempts` is one with min 1.
|
|
78
|
-
- **`WorkerOptions.concurrency` is read by OWN key**
|
|
79
|
-
(`worker-slots.test.ts`).
|
|
79
|
+
- **`WorkerOptions.concurrency` is read by OWN key** (`worker-slots.test.ts`).
|
|
80
80
|
- **`job.concurrency` is enforced by `JobDriver.leases`** (one row per held slot in `x_job_leases`);
|
|
81
81
|
`limits.ts` is the per-process fast path. No lease store + a declared `concurrency` makes `start()`
|
|
82
82
|
throw `X_JOB_CONCURRENCY_UNENFORCEABLE`.
|
|
@@ -110,7 +110,7 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
110
110
|
synchronously with its guard.
|
|
111
111
|
- **A fleet slot is taken INSIDE a `try`, released AWAITED, and HELD, not owned**: `false`, or a TTL
|
|
112
112
|
with no renewal landing, CANCELS the run (`X_JOB_SLOT_LOST`); held per job id as a LIST.
|
|
113
|
-
- **The run's signal is a controller this worker owns** (`run-signal.ts`)
|
|
113
|
+
- **The run's signal is a controller this worker owns** (`run-signal.ts`).
|
|
114
114
|
- **A lease is HELD, not owned** (`heartbeat.ts`): one failed renewal warns; a whole window without
|
|
115
115
|
one landing is `jobs.lease.lost`, on this process's clock, asked both sides of the call.
|
|
116
116
|
- **A renewal is decided against `stopped()`** (`renewal-timer.ts`, re-read after every await); the
|
|
@@ -135,7 +135,7 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
135
135
|
- **A lease for a run that never STARTED is `abandon()`ed, never `release()`d** — the rate stamp
|
|
136
136
|
goes with it (`worker-admit.ts`). **A failed pass re-arms at the floor** (`worker-loop.ts`
|
|
137
137
|
catch); `timers` is the test seam.
|
|
138
|
-
- **Every timer body catches before it finalises
|
|
138
|
+
- **Every timer body catches before it finalises.**
|
|
139
139
|
- **Suspension is control flow, and a SHED is not a suspension**: `StepSuspension` →
|
|
140
140
|
`nack({ countsAsAttempt: false, park: true })`; a shed is `countsAsAttempt: false` without `park`,
|
|
141
141
|
stays `ready`, logs `jobs.worker.shed` (no `last_error`). `driver-parity.test.ts`.
|
package/README.md
CHANGED
|
@@ -759,10 +759,20 @@ between them — swapping is `setJobDriver(other)`, and there is **no `jobs.driv
|
|
|
759
759
|
| Driver | Status | Backing | Use |
|
|
760
760
|
|---|---|---|---|
|
|
761
761
|
| `pg` | **default** | `SELECT ... FOR UPDATE SKIP LOCKED`, a partial unique index on `(name, coalesce(tenant_id, ''), idempotency_key)` over live rows (`x_jobs_name_tenant_idempotency_live_idx`), lease-based leader, `x_job_leases` | zero-infra start, most apps |
|
|
762
|
+
| `redis` | complete, `As of 2026-10` | `Bun.redis`; one Lua script per operation, every key in one `{hash tag}` slot | a queue off the database; no `introspect`, `backfills` or `leases` |
|
|
762
763
|
| `memory` | complete | in-process maps | `x dev`, tests |
|
|
763
764
|
|
|
764
|
-
|
|
765
|
-
|
|
765
|
+
**`redisJobDriver({ client?, prefix?, clock?, doneTtlMs? })`** passes `@ultimat3/testing`'s
|
|
766
|
+
`jobDriverConformance`, the suite the other two pass, and `driver-redis.live.test.ts` runs one
|
|
767
|
+
script of operations on it and on the memory driver and compares every answer. Installed with
|
|
768
|
+
`setJobDriver(redisJobDriver())` (or `ServeOptions.runtime.jobs`), imported from its own entry
|
|
769
|
+
`@ultimat3/jobs/redis` so the barrel every role boots carries none of it; `Bun.redis` reads `REDIS_URL`.
|
|
770
|
+
What it does not carry: `introspect` (so `x jobs show`/`retry`/`cancel` and queue pauses answer
|
|
771
|
+
for Postgres only), `backfills`, and `leases` — a job declaring `concurrency` refuses
|
|
772
|
+
`jobWorker().start()` (`X_JOB_CONCURRENCY_UNENFORCEABLE`). A `done` row and its steps expire
|
|
773
|
+
after `doneTtlMs` (a week; `0` keeps them); `dead` and `failed` rows are kept. The outbox still
|
|
774
|
+
stages in Postgres — the relay publishes committed rows onto it with their ids. The NATS stub
|
|
775
|
+
deleted in 25.0.0 stays deleted.
|
|
766
776
|
|
|
767
777
|
The pg SQL lives verbatim in `src/driver-pg-*sql.ts` (`SQL_CLAIM`, `SQL_ENQUEUE`, `SQL_NACK`, …)
|
|
768
778
|
so an agent debugging a stuck queue can read and run the exact statement. The barrel exports
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ultimat3/jobs",
|
|
3
|
-
"version": "27.
|
|
3
|
+
"version": "27.4.0",
|
|
4
4
|
"description": "Durable background work: steps, transactional outbox, cron tasks, one driver interface",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
|
@@ -14,7 +14,8 @@
|
|
|
14
14
|
"provenance": true
|
|
15
15
|
},
|
|
16
16
|
"exports": {
|
|
17
|
-
".": "./src/index.ts"
|
|
17
|
+
".": "./src/index.ts",
|
|
18
|
+
"./redis": "./src/driver-redis.ts"
|
|
18
19
|
},
|
|
19
20
|
"files": [
|
|
20
21
|
"src",
|
|
@@ -32,10 +33,10 @@
|
|
|
32
33
|
"test": "bun test"
|
|
33
34
|
},
|
|
34
35
|
"dependencies": {
|
|
35
|
-
"@ultimat3/core": "27.
|
|
36
|
-
"@ultimat3/db": "27.
|
|
37
|
-
"@ultimat3/entity": "27.
|
|
38
|
-
"@ultimat3/schema": "27.
|
|
39
|
-
"@ultimat3/time": "27.
|
|
36
|
+
"@ultimat3/core": "27.4.0",
|
|
37
|
+
"@ultimat3/db": "27.4.0",
|
|
38
|
+
"@ultimat3/entity": "27.4.0",
|
|
39
|
+
"@ultimat3/schema": "27.4.0",
|
|
40
|
+
"@ultimat3/time": "27.4.0"
|
|
40
41
|
}
|
|
41
42
|
}
|
|
@@ -0,0 +1,231 @@
|
|
|
1
|
+
// The Redis job driver's statements: one Lua script per queue operation, so each runs atomically
|
|
2
|
+
// on the server the way each `driver-pg-sql.ts` statement runs in one transaction. The TypeScript
|
|
3
|
+
// that binds their arguments, and the key layout they share, is `driver-redis.ts`.
|
|
4
|
+
//
|
|
5
|
+
// Every key a script touches carries the namespace's `{hash tag}`, so all of them live in ONE slot
|
|
6
|
+
// and a script is legal on Redis Cluster and Dragonfly alike: `KEYS[1]` is always a namespaced key,
|
|
7
|
+
// which is what routes the call, and the others are built from `ARGV[1]` inside the slot it named.
|
|
8
|
+
|
|
9
|
+
import { LEASE_LAPSED_FINAL_ATTEMPT } from './driver';
|
|
10
|
+
|
|
11
|
+
/**
|
|
12
|
+
* The live states as a Lua set, the four `LIVE_STATES` names — the idempotency namespace holds a
|
|
13
|
+
* key only while one of them does, exactly as `x_jobs_name_tenant_idempotency_live_idx` does.
|
|
14
|
+
*/
|
|
15
|
+
const LIVE = `local LIVE = { ready = true, delayed = true, running = true, suspended = true }`;
|
|
16
|
+
|
|
17
|
+
/**
|
|
18
|
+
* The claim fence `SQL_ACK` carries: the row is `running`, held by this worker, under this claim.
|
|
19
|
+
* `ARGV[1]` namespace, `ARGV[2]` id, `ARGV[3]` worker, `ARGV[4]` claim ordinal.
|
|
20
|
+
*/
|
|
21
|
+
const FENCE = `
|
|
22
|
+
local P = ARGV[1]
|
|
23
|
+
local id = ARGV[2]
|
|
24
|
+
local job = P .. ':job:' .. id
|
|
25
|
+
local held = redis.call('HMGET', job, 'state', 'claimedBy', 'claim', 'queue', 'nk', 'runId')
|
|
26
|
+
if held[1] ~= 'running' or held[2] ~= ARGV[3] or held[3] ~= ARGV[4] then return 0 end
|
|
27
|
+
local queue = held[4]
|
|
28
|
+
redis.call('HDEL', job, 'visibleAt', 'claimedBy')
|
|
29
|
+
redis.call('ZREM', P .. ':run:' .. queue, id)
|
|
30
|
+
`;
|
|
31
|
+
|
|
32
|
+
/**
|
|
33
|
+
* `ARGV`: namespace, id, name, queue, input, idempotency key, runId, maxAttempts, runAt, now,
|
|
34
|
+
* tenant, traceparent, enqueuedBy, namespace field. Answers `{ outcome, id, runId }`: `published`
|
|
35
|
+
* when the caller-allocated id already names a row (`SQL_ENQUEUE`'s `not exists`), `deduped` when
|
|
36
|
+
* a live row holds the key, `created` otherwise.
|
|
37
|
+
*/
|
|
38
|
+
export const REDIS_ENQUEUE = `
|
|
39
|
+
${LIVE}
|
|
40
|
+
local P = ARGV[1]
|
|
41
|
+
local id = ARGV[2]
|
|
42
|
+
local job = P .. ':job:' .. id
|
|
43
|
+
if redis.call('EXISTS', job) == 1 then
|
|
44
|
+
return { 'published', id, redis.call('HGET', job, 'runId') }
|
|
45
|
+
end
|
|
46
|
+
local nk = ARGV[14]
|
|
47
|
+
local holder = redis.call('HGET', P .. ':idem', nk)
|
|
48
|
+
if holder then
|
|
49
|
+
local row = redis.call('HMGET', P .. ':job:' .. holder, 'state', 'runId')
|
|
50
|
+
if row[1] and LIVE[row[1]] then return { 'deduped', holder, row[2] } end
|
|
51
|
+
end
|
|
52
|
+
local now = tonumber(ARGV[10])
|
|
53
|
+
local runAt = tonumber(ARGV[9])
|
|
54
|
+
local state = 'ready'
|
|
55
|
+
if runAt > now then state = 'delayed' end
|
|
56
|
+
redis.call('HSET', job,
|
|
57
|
+
'id', id, 'name', ARGV[3], 'queue', ARGV[4], 'input', ARGV[5], 'idempotencyKey', ARGV[6],
|
|
58
|
+
'runId', ARGV[7], 'attempt', '0', 'maxAttempts', ARGV[8], 'state', state, 'runAt', ARGV[9],
|
|
59
|
+
'createdAt', ARGV[10], 'updatedAt', ARGV[10], 'claim', '0', 'nk', nk)
|
|
60
|
+
if ARGV[11] ~= '' then redis.call('HSET', job, 'tenantId', ARGV[11]) end
|
|
61
|
+
if ARGV[12] ~= '' then redis.call('HSET', job, 'traceparent', ARGV[12]) end
|
|
62
|
+
if ARGV[13] ~= '' then redis.call('HSET', job, 'enqueuedBy', ARGV[13]) end
|
|
63
|
+
redis.call('ZADD', P .. ':wait:' .. ARGV[4], runAt, id)
|
|
64
|
+
redis.call('SADD', P .. ':queues', ARGV[4])
|
|
65
|
+
redis.call('HSET', P .. ':idem', nk, id)
|
|
66
|
+
return { 'created', id, ARGV[7] }
|
|
67
|
+
`.trim();
|
|
68
|
+
|
|
69
|
+
/**
|
|
70
|
+
* `SQL_CLAIM`. `ARGV`: namespace, now, limit, visibility ms, worker, the count of `dropExhausted`
|
|
71
|
+
* names, those names, then the queues. Candidates are every due row of each queue — ready,
|
|
72
|
+
* delayed, suspended, or running past its lease — at most `limit` from each source, ordered by
|
|
73
|
+
* `runAt` and cut at `limit`. A running row on its final attempt is BURIED (`dead`, or `failed`
|
|
74
|
+
* for a dropped name) and never handed out. Answers one `{ 'c' | 'b', HGETALL… }` per row.
|
|
75
|
+
*/
|
|
76
|
+
export const REDIS_CLAIM = `
|
|
77
|
+
local P = ARGV[1]
|
|
78
|
+
local now = tonumber(ARGV[2])
|
|
79
|
+
local limit = tonumber(ARGV[3])
|
|
80
|
+
local visibility = tonumber(ARGV[4])
|
|
81
|
+
local worker = ARGV[5]
|
|
82
|
+
local dropCount = tonumber(ARGV[6])
|
|
83
|
+
local drop = {}
|
|
84
|
+
for i = 1, dropCount do drop[ARGV[6 + i]] = true end
|
|
85
|
+
local candidates = {}
|
|
86
|
+
if limit > 0 then
|
|
87
|
+
for i = 7 + dropCount, #ARGV do
|
|
88
|
+
local queue = ARGV[i]
|
|
89
|
+
for _, source in ipairs({ 'wait', 'park' }) do
|
|
90
|
+
local found = redis.call('ZRANGEBYSCORE', P .. ':' .. source .. ':' .. queue, '-inf', now,
|
|
91
|
+
'WITHSCORES', 'LIMIT', 0, limit)
|
|
92
|
+
for j = 1, #found, 2 do
|
|
93
|
+
table.insert(candidates, { id = found[j], runAt = tonumber(found[j + 1]) })
|
|
94
|
+
end
|
|
95
|
+
end
|
|
96
|
+
local lapsed = redis.call('ZRANGEBYSCORE', P .. ':run:' .. queue, '-inf', now, 'LIMIT', 0, limit)
|
|
97
|
+
for _, id in ipairs(lapsed) do
|
|
98
|
+
local runAt = tonumber(redis.call('HGET', P .. ':job:' .. id, 'runAt'))
|
|
99
|
+
if runAt and runAt <= now then table.insert(candidates, { id = id, runAt = runAt }) end
|
|
100
|
+
end
|
|
101
|
+
end
|
|
102
|
+
end
|
|
103
|
+
table.sort(candidates, function(a, b)
|
|
104
|
+
if a.runAt == b.runAt then return a.id < b.id end
|
|
105
|
+
return a.runAt < b.runAt
|
|
106
|
+
end)
|
|
107
|
+
local out = {}
|
|
108
|
+
for i = 1, math.min(limit, #candidates) do
|
|
109
|
+
local id = candidates[i].id
|
|
110
|
+
local job = P .. ':job:' .. id
|
|
111
|
+
local row = redis.call('HMGET', job, 'state', 'attempt', 'maxAttempts', 'name', 'queue', 'claim', 'nk')
|
|
112
|
+
local state, attempt, maxAttempts, name, queue = row[1], tonumber(row[2]), tonumber(row[3]), row[4], row[5]
|
|
113
|
+
if state == 'running' and attempt >= maxAttempts then
|
|
114
|
+
local final = 'dead'
|
|
115
|
+
if drop[name] then final = 'failed' end
|
|
116
|
+
redis.call('ZREM', P .. ':run:' .. queue, id)
|
|
117
|
+
redis.call('HDEL', job, 'visibleAt', 'claimedBy', 'lastErrorStack')
|
|
118
|
+
redis.call('HSET', job, 'state', final, 'lastError', '${LEASE_LAPSED_FINAL_ATTEMPT}',
|
|
119
|
+
'updatedAt', ARGV[2])
|
|
120
|
+
redis.call('ZADD', P .. ':' .. final .. ':' .. queue, now, id)
|
|
121
|
+
redis.call('HDEL', P .. ':idem', row[7])
|
|
122
|
+
table.insert(out, { 'b', redis.call('HGETALL', job) })
|
|
123
|
+
else
|
|
124
|
+
redis.call('ZREM', P .. ':wait:' .. queue, id)
|
|
125
|
+
redis.call('ZREM', P .. ':park:' .. queue, id)
|
|
126
|
+
local visibleAt = now + visibility
|
|
127
|
+
redis.call('HSET', job, 'state', 'running', 'attempt', attempt + 1,
|
|
128
|
+
'claim', tonumber(row[6]) + 1, 'claimedBy', worker, 'claimedAt', ARGV[2],
|
|
129
|
+
'visibleAt', visibleAt, 'updatedAt', ARGV[2])
|
|
130
|
+
redis.call('ZADD', P .. ':run:' .. queue, visibleAt, id)
|
|
131
|
+
table.insert(out, { 'c', redis.call('HGETALL', job) })
|
|
132
|
+
end
|
|
133
|
+
end
|
|
134
|
+
return out
|
|
135
|
+
`.trim();
|
|
136
|
+
|
|
137
|
+
/** `SQL_ACK`. `ARGV`: the fence's four, then now, then the done row's time to live (0: none). */
|
|
138
|
+
export const REDIS_ACK = `
|
|
139
|
+
${FENCE}
|
|
140
|
+
redis.call('HSET', job, 'state', 'done', 'updatedAt', ARGV[5])
|
|
141
|
+
redis.call('HDEL', P .. ':idem', held[5])
|
|
142
|
+
local ttl = tonumber(ARGV[6])
|
|
143
|
+
if ttl > 0 then
|
|
144
|
+
redis.call('PEXPIRE', job, ttl)
|
|
145
|
+
redis.call('PEXPIRE', P .. ':steps:' .. held[6], ttl)
|
|
146
|
+
end
|
|
147
|
+
return 1
|
|
148
|
+
`.trim();
|
|
149
|
+
|
|
150
|
+
/**
|
|
151
|
+
* `SQL_NACK`. `ARGV`: the fence's four, then now, the state `nackState` chose, runAt, `1` when the
|
|
152
|
+
* attempt counts, `1` when an error is bound, the error, the stack (`''` for none).
|
|
153
|
+
*/
|
|
154
|
+
export const REDIS_NACK = `
|
|
155
|
+
${FENCE}
|
|
156
|
+
local state = ARGV[6]
|
|
157
|
+
local attempt = tonumber(redis.call('HGET', job, 'attempt'))
|
|
158
|
+
if ARGV[8] ~= '1' then attempt = math.max(0, attempt - 1) end
|
|
159
|
+
redis.call('HSET', job, 'state', state, 'runAt', ARGV[7], 'attempt', attempt, 'updatedAt', ARGV[5])
|
|
160
|
+
if ARGV[9] == '1' then
|
|
161
|
+
redis.call('HSET', job, 'lastError', ARGV[10])
|
|
162
|
+
if ARGV[11] == '' then redis.call('HDEL', job, 'lastErrorStack')
|
|
163
|
+
else redis.call('HSET', job, 'lastErrorStack', ARGV[11]) end
|
|
164
|
+
end
|
|
165
|
+
if state == 'ready' then
|
|
166
|
+
redis.call('ZADD', P .. ':wait:' .. queue, ARGV[7], id)
|
|
167
|
+
elseif state == 'suspended' then
|
|
168
|
+
redis.call('ZADD', P .. ':park:' .. queue, ARGV[7], id)
|
|
169
|
+
else
|
|
170
|
+
redis.call('ZADD', P .. ':' .. state .. ':' .. queue, ARGV[5], id)
|
|
171
|
+
redis.call('HDEL', P .. ':idem', held[5])
|
|
172
|
+
end
|
|
173
|
+
return 1
|
|
174
|
+
`.trim();
|
|
175
|
+
|
|
176
|
+
/**
|
|
177
|
+
* `SQL_HEARTBEAT`. `ARGV`: namespace, id, worker (`''`: any), claim (`''`: any), the new lease's
|
|
178
|
+
* end. Answers 1 when the lease moved.
|
|
179
|
+
*/
|
|
180
|
+
export const REDIS_HEARTBEAT = `
|
|
181
|
+
local P = ARGV[1]
|
|
182
|
+
local id = ARGV[2]
|
|
183
|
+
local job = P .. ':job:' .. id
|
|
184
|
+
local held = redis.call('HMGET', job, 'state', 'claimedBy', 'claim', 'queue')
|
|
185
|
+
if held[1] ~= 'running' then return 0 end
|
|
186
|
+
if ARGV[3] ~= '' and held[2] ~= ARGV[3] then return 0 end
|
|
187
|
+
if ARGV[4] ~= '' and held[3] ~= ARGV[4] then return 0 end
|
|
188
|
+
redis.call('HSET', job, 'visibleAt', ARGV[5])
|
|
189
|
+
redis.call('ZADD', P .. ':run:' .. held[4], ARGV[5], id)
|
|
190
|
+
return 1
|
|
191
|
+
`.trim();
|
|
192
|
+
|
|
193
|
+
/**
|
|
194
|
+
* `SQL_STATS`, one consistent read. `ARGV`: namespace, now. Answers one row per queue:
|
|
195
|
+
* `{ queue, ready, delayed, running, suspended, failed, dead, oldest runAt or -1 }` — due is due
|
|
196
|
+
* whatever state the row was WRITTEN in, the split `SQL_STATS` makes.
|
|
197
|
+
*/
|
|
198
|
+
export const REDIS_STATS = `
|
|
199
|
+
local P = ARGV[1]
|
|
200
|
+
local now = tonumber(ARGV[2])
|
|
201
|
+
local out = {}
|
|
202
|
+
for _, queue in ipairs(redis.call('SMEMBERS', P .. ':queues')) do
|
|
203
|
+
local wait = P .. ':wait:' .. queue
|
|
204
|
+
local oldest = redis.call('ZRANGE', wait, 0, 0, 'WITHSCORES')
|
|
205
|
+
local first = -1
|
|
206
|
+
if oldest[2] and tonumber(oldest[2]) <= now then first = oldest[2] end
|
|
207
|
+
table.insert(out, {
|
|
208
|
+
queue,
|
|
209
|
+
redis.call('ZCOUNT', wait, '-inf', now),
|
|
210
|
+
redis.call('ZCOUNT', wait, '(' .. now, '+inf'),
|
|
211
|
+
redis.call('ZCARD', P .. ':run:' .. queue),
|
|
212
|
+
redis.call('ZCARD', P .. ':park:' .. queue),
|
|
213
|
+
redis.call('ZCARD', P .. ':failed:' .. queue),
|
|
214
|
+
redis.call('ZCARD', P .. ':dead:' .. queue),
|
|
215
|
+
first,
|
|
216
|
+
})
|
|
217
|
+
end
|
|
218
|
+
return out
|
|
219
|
+
`.trim();
|
|
220
|
+
|
|
221
|
+
/**
|
|
222
|
+
* `SQL_STEP_PUT` under a claim: the write lands only while that claim still holds the row.
|
|
223
|
+
* `ARGV`: namespace, job id, worker, claim, runId, step name, the record as JSON.
|
|
224
|
+
*/
|
|
225
|
+
export const REDIS_STEP_PUT_FENCED = `
|
|
226
|
+
local P = ARGV[1]
|
|
227
|
+
local held = redis.call('HMGET', P .. ':job:' .. ARGV[2], 'state', 'claimedBy', 'claim')
|
|
228
|
+
if held[1] ~= 'running' or held[2] ~= ARGV[3] or held[3] ~= ARGV[4] then return 0 end
|
|
229
|
+
redis.call('HSET', P .. ':steps:' .. ARGV[5], ARGV[6], ARGV[7])
|
|
230
|
+
return 1
|
|
231
|
+
`.trim();
|
|
@@ -0,0 +1,64 @@
|
|
|
1
|
+
// The Redis driver's `StepStore`: one HASH per run (`steps:<runId>`), step name → the record as
|
|
2
|
+
// JSON. Through JSON exactly as the pg store persists it, so a step replays a `Date` as the same
|
|
3
|
+
// string under every driver. A write made under a claim is fenced in one script on the row that
|
|
4
|
+
// claim must still hold (`SQL_STEP_PUT`'s fence) and is `X_JOB_LEASE_LOST` otherwise.
|
|
5
|
+
|
|
6
|
+
import type { RedisSender } from './driver-redis';
|
|
7
|
+
import { REDIS_STEP_PUT_FENCED } from './driver-redis-scripts';
|
|
8
|
+
import { LeaseLostError } from './errors';
|
|
9
|
+
import type { StepRecord, StepStore } from './steps';
|
|
10
|
+
import { isStepStatus } from './steps';
|
|
11
|
+
|
|
12
|
+
interface StepStoreSeams {
|
|
13
|
+
/** The namespace, `{<prefix>}`. */
|
|
14
|
+
readonly ns: string;
|
|
15
|
+
/** One script, routed to the namespace's slot. */
|
|
16
|
+
readonly run: (script: string, args: readonly (string | number)[]) => Promise<unknown>;
|
|
17
|
+
readonly client: RedisSender;
|
|
18
|
+
}
|
|
19
|
+
|
|
20
|
+
/** A stored step, or nothing — a value that is not one of ours is never cast onto a record. */
|
|
21
|
+
function parseStep(text: unknown): StepRecord | undefined {
|
|
22
|
+
if (typeof text !== 'string') return undefined;
|
|
23
|
+
const record = JSON.parse(text) as StepRecord;
|
|
24
|
+
return isStepStatus(record.status) ? record : undefined;
|
|
25
|
+
}
|
|
26
|
+
|
|
27
|
+
export function redisStepStore({ ns, run, client }: StepStoreSeams): StepStore {
|
|
28
|
+
const key = (runId: string) => `${ns}:steps:${runId}`;
|
|
29
|
+
return {
|
|
30
|
+
async get(runId, name) {
|
|
31
|
+
return parseStep(await client.send('HGET', [key(runId), name]));
|
|
32
|
+
},
|
|
33
|
+
async put(record, by) {
|
|
34
|
+
const text = JSON.stringify({ ...record, output: record.output ?? null });
|
|
35
|
+
if (by === undefined) {
|
|
36
|
+
await client.send('HSET', [key(record.runId), record.name, text]);
|
|
37
|
+
return;
|
|
38
|
+
}
|
|
39
|
+
const landed = await run(REDIS_STEP_PUT_FENCED, [
|
|
40
|
+
by.jobId,
|
|
41
|
+
by.workerId,
|
|
42
|
+
by.claim,
|
|
43
|
+
record.runId,
|
|
44
|
+
record.name,
|
|
45
|
+
text,
|
|
46
|
+
]);
|
|
47
|
+
if (Number(landed) !== 1) throw new LeaseLostError({ job: by.job, jobId: by.jobId });
|
|
48
|
+
},
|
|
49
|
+
async list(runId) {
|
|
50
|
+
const values = await client.send('HVALS', [key(runId)]);
|
|
51
|
+
const records = (Array.isArray(values) ? values : []).flatMap((text: unknown) => {
|
|
52
|
+
const record = parseStep(text);
|
|
53
|
+
return record === undefined ? [] : [record];
|
|
54
|
+
});
|
|
55
|
+
return records.sort((a, b) => a.startedAt - b.startedAt);
|
|
56
|
+
},
|
|
57
|
+
async del(runId, name) {
|
|
58
|
+
await client.send('HDEL', [key(runId), name]);
|
|
59
|
+
},
|
|
60
|
+
async clear(runId) {
|
|
61
|
+
await client.send('DEL', [key(runId)]);
|
|
62
|
+
},
|
|
63
|
+
};
|
|
64
|
+
}
|
|
@@ -0,0 +1,278 @@
|
|
|
1
|
+
// A Redis queue on `Bun.redis` — no client dependency, the runtime ships one. The same contract as
|
|
2
|
+
// the pg driver, held by `@ultimat3/testing`'s `jobDriverConformance`: every operation is one Lua
|
|
3
|
+
// script (`driver-redis-scripts.ts`), so a claim, a settle and a renewal are each atomic on the
|
|
4
|
+
// server the way a `driver-pg-sql.ts` statement is in one transaction. Its own entry,
|
|
5
|
+
// `@ultimat3/jobs/redis`: the barrel is on every role's boot path, and an app on Postgres loads none
|
|
6
|
+
// of this.
|
|
7
|
+
//
|
|
8
|
+
// Key layout, every key inside ONE hash tag `{<prefix>}` (one slot: legal on Redis Cluster):
|
|
9
|
+
// job:<id> HASH the row wait:<queue> ZSET ready/delayed, by runAt
|
|
10
|
+
// park:<queue> ZSET suspended, by runAt run:<queue> ZSET running, by lease end
|
|
11
|
+
// dead:<queue> ZSET dead letters, by time failed:<queue> ZSET failed rows, by time
|
|
12
|
+
// queues SET every queue named idem HASH live key → row id
|
|
13
|
+
// steps:<runId> HASH step name → record JSON
|
|
14
|
+
|
|
15
|
+
import type { Clock } from '@ultimat3/core';
|
|
16
|
+
import { finiteCount, finiteOption, systemClock, uuidV7 } from '@ultimat3/core';
|
|
17
|
+
import { nowMs } from './clock';
|
|
18
|
+
import type {
|
|
19
|
+
ClaimedJob,
|
|
20
|
+
EnqueueRequest,
|
|
21
|
+
EnqueueResult,
|
|
22
|
+
JobDriver,
|
|
23
|
+
JobRecord,
|
|
24
|
+
QueueStats,
|
|
25
|
+
} from './driver';
|
|
26
|
+
import {
|
|
27
|
+
assertClaimBounds,
|
|
28
|
+
assertClaimQueues,
|
|
29
|
+
DEFAULT_QUEUE,
|
|
30
|
+
isJobState,
|
|
31
|
+
nackState,
|
|
32
|
+
} from './driver';
|
|
33
|
+
import {
|
|
34
|
+
REDIS_ACK,
|
|
35
|
+
REDIS_CLAIM,
|
|
36
|
+
REDIS_ENQUEUE,
|
|
37
|
+
REDIS_HEARTBEAT,
|
|
38
|
+
REDIS_NACK,
|
|
39
|
+
REDIS_STATS,
|
|
40
|
+
} from './driver-redis-scripts';
|
|
41
|
+
import { redisStepStore } from './driver-redis-steps';
|
|
42
|
+
import { DriverUnavailableError, JobDuplicateError } from './errors';
|
|
43
|
+
import { MAX_ERROR_STACK_LENGTH } from './introspection';
|
|
44
|
+
|
|
45
|
+
/** The slice of Bun's Redis client this driver uses — `Bun.redis` and `new Bun.RedisClient()`. */
|
|
46
|
+
export interface RedisSender {
|
|
47
|
+
send(command: string, args: string[]): Promise<unknown>;
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
/** A finished row's default life: a week, then Redis reclaims it and its steps. */
|
|
51
|
+
export const DEFAULT_REDIS_DONE_TTL_MS = 7 * 24 * 60 * 60 * 1000;
|
|
52
|
+
|
|
53
|
+
export interface RedisJobDriverOptions {
|
|
54
|
+
/** Injected in tests and for a second server; production reads `Bun.redis` (`REDIS_URL`). */
|
|
55
|
+
readonly client?: RedisSender;
|
|
56
|
+
/** Key namespace, the hash tag every key carries. Default `x:jobs`. */
|
|
57
|
+
readonly prefix?: string;
|
|
58
|
+
readonly clock?: Clock;
|
|
59
|
+
/**
|
|
60
|
+
* How long a `done` row (and its steps) outlives its ack, in ms; `0` keeps it forever. Default
|
|
61
|
+
* `DEFAULT_REDIS_DONE_TTL_MS`: a queue in memory that keeps every finished row is a server that
|
|
62
|
+
* runs out of memory. Dead and failed rows are kept — they are the record of what went wrong.
|
|
63
|
+
*/
|
|
64
|
+
readonly doneTtlMs?: number;
|
|
65
|
+
}
|
|
66
|
+
|
|
67
|
+
function resolveClient(injected: RedisSender | undefined): RedisSender {
|
|
68
|
+
if (injected !== undefined) return injected;
|
|
69
|
+
const ambient = (Bun as unknown as { redis?: RedisSender }).redis;
|
|
70
|
+
if (ambient === undefined || typeof ambient.send !== 'function') {
|
|
71
|
+
throw new DriverUnavailableError({
|
|
72
|
+
driver: 'redis',
|
|
73
|
+
cause: 'Bun.redis is not available, and no client was handed to redisJobDriver()',
|
|
74
|
+
fix: 'REDIS_URL=redis://127.0.0.1:6379 x dev — or hand one in: redisJobDriver({ client: new Bun.RedisClient(url) })',
|
|
75
|
+
});
|
|
76
|
+
}
|
|
77
|
+
return ambient;
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
/** A Lua reply as an array; anything else is the empty one. */
|
|
81
|
+
const list = (reply: unknown): readonly unknown[] => (Array.isArray(reply) ? reply : []);
|
|
82
|
+
|
|
83
|
+
/** `HGETALL`'s flat `[field, value, …]` reply as a map. */
|
|
84
|
+
function fieldsOf(flat: readonly unknown[]): ReadonlyMap<string, string> {
|
|
85
|
+
const fields = new Map<string, string>();
|
|
86
|
+
for (let i = 0; i + 1 < flat.length; i += 2) fields.set(String(flat[i]), String(flat[i + 1]));
|
|
87
|
+
return fields;
|
|
88
|
+
}
|
|
89
|
+
|
|
90
|
+
const num = (fields: ReadonlyMap<string, string>, name: string): number | undefined => {
|
|
91
|
+
const value = fields.get(name);
|
|
92
|
+
return value === undefined ? undefined : Number(value);
|
|
93
|
+
};
|
|
94
|
+
|
|
95
|
+
/** One stored row as the record every driver hands back. Absent fields stay absent. */
|
|
96
|
+
function toRecord(fields: ReadonlyMap<string, string>): JobRecord {
|
|
97
|
+
const state = fields.get('state') ?? '';
|
|
98
|
+
const optional = (name: keyof JobRecord & string) => {
|
|
99
|
+
const value = fields.get(name);
|
|
100
|
+
return value === undefined ? {} : { [name]: value };
|
|
101
|
+
};
|
|
102
|
+
const claim = num(fields, 'claim') ?? 0;
|
|
103
|
+
const visibleAt = num(fields, 'visibleAt');
|
|
104
|
+
return {
|
|
105
|
+
id: fields.get('id') ?? '',
|
|
106
|
+
name: fields.get('name') ?? '',
|
|
107
|
+
queue: fields.get('queue') ?? '',
|
|
108
|
+
input: JSON.parse(fields.get('input') ?? 'null') as unknown,
|
|
109
|
+
idempotencyKey: fields.get('idempotencyKey') ?? '',
|
|
110
|
+
runId: fields.get('runId') ?? '',
|
|
111
|
+
attempt: num(fields, 'attempt') ?? 0,
|
|
112
|
+
maxAttempts: num(fields, 'maxAttempts') ?? 0,
|
|
113
|
+
// A state the list does not know is never cast onto a record: it reads as dead, the one state
|
|
114
|
+
// nothing claims again — the answer `isJobState` gives every driver.
|
|
115
|
+
state: isJobState(state) ? state : 'dead',
|
|
116
|
+
runAt: num(fields, 'runAt') ?? 0,
|
|
117
|
+
createdAt: num(fields, 'createdAt') ?? 0,
|
|
118
|
+
updatedAt: num(fields, 'updatedAt') ?? 0,
|
|
119
|
+
...optional('tenantId'),
|
|
120
|
+
...optional('lastError'),
|
|
121
|
+
...optional('lastErrorStack'),
|
|
122
|
+
...optional('claimedBy'),
|
|
123
|
+
...optional('traceparent'),
|
|
124
|
+
...optional('enqueuedBy'),
|
|
125
|
+
...(claim > 0 ? { claim } : {}),
|
|
126
|
+
...(visibleAt === undefined ? {} : { visibleAt }),
|
|
127
|
+
};
|
|
128
|
+
}
|
|
129
|
+
|
|
130
|
+
/**
|
|
131
|
+
* The idempotency namespace field: name, tenant (`''` for none, as the index's `coalesce`) and
|
|
132
|
+
* key, JSON-joined so no separator inside a value can make two triples one field.
|
|
133
|
+
*/
|
|
134
|
+
const namespaceField = (request: EnqueueRequest): string =>
|
|
135
|
+
JSON.stringify([request.name, request.tenantId ?? '', request.idempotencyKey]);
|
|
136
|
+
|
|
137
|
+
export function redisJobDriver(options: RedisJobDriverOptions = {}): JobDriver {
|
|
138
|
+
const client = resolveClient(options.client);
|
|
139
|
+
const clock = options.clock ?? systemClock;
|
|
140
|
+
const ns = `{${options.prefix ?? 'x:jobs'}}`;
|
|
141
|
+
const doneTtlMs = finiteCount(
|
|
142
|
+
'redisJobDriver',
|
|
143
|
+
'doneTtlMs',
|
|
144
|
+
options.doneTtlMs ?? DEFAULT_REDIS_DONE_TTL_MS,
|
|
145
|
+
);
|
|
146
|
+
// `KEYS[1]` routes the call to the namespace's slot; every key a script builds is in it.
|
|
147
|
+
const run = (script: string, args: readonly (string | number)[]): Promise<unknown> =>
|
|
148
|
+
client.send('EVAL', [script, '1', `${ns}:queues`, ns, ...args.map(String)]);
|
|
149
|
+
|
|
150
|
+
return {
|
|
151
|
+
name: 'redis',
|
|
152
|
+
steps: redisStepStore({ ns, run, client }),
|
|
153
|
+
|
|
154
|
+
async enqueue(request): Promise<EnqueueResult> {
|
|
155
|
+
const at = nowMs(clock);
|
|
156
|
+
const runId = request.runId ?? uuidV7();
|
|
157
|
+
const [outcome, id, heldRunId] = list(
|
|
158
|
+
await run(REDIS_ENQUEUE, [
|
|
159
|
+
request.id ?? uuidV7(),
|
|
160
|
+
request.name,
|
|
161
|
+
request.queue || DEFAULT_QUEUE,
|
|
162
|
+
JSON.stringify(request.input ?? null),
|
|
163
|
+
request.idempotencyKey,
|
|
164
|
+
runId,
|
|
165
|
+
request.maxAttempts,
|
|
166
|
+
request.runAt ?? at,
|
|
167
|
+
at,
|
|
168
|
+
request.tenantId ?? '',
|
|
169
|
+
request.traceparent ?? '',
|
|
170
|
+
request.enqueuedBy ?? '',
|
|
171
|
+
namespaceField(request),
|
|
172
|
+
]),
|
|
173
|
+
).map(String);
|
|
174
|
+
if (outcome === 'deduped' && request.onConflict === 'error') {
|
|
175
|
+
throw new JobDuplicateError({
|
|
176
|
+
job: request.name,
|
|
177
|
+
idempotencyKey: request.idempotencyKey,
|
|
178
|
+
existingId: id ?? '',
|
|
179
|
+
});
|
|
180
|
+
}
|
|
181
|
+
return { id: id ?? '', runId: heldRunId ?? runId, deduped: outcome !== 'created' };
|
|
182
|
+
},
|
|
183
|
+
|
|
184
|
+
async claim(claimOptions): Promise<readonly ClaimedJob[]> {
|
|
185
|
+
assertClaimQueues('redis', claimOptions);
|
|
186
|
+
assertClaimBounds('redis', claimOptions);
|
|
187
|
+
const drop = claimOptions.dropExhausted ?? [];
|
|
188
|
+
const reply = await run(REDIS_CLAIM, [
|
|
189
|
+
nowMs(clock),
|
|
190
|
+
claimOptions.limit,
|
|
191
|
+
claimOptions.visibilityTimeoutMs,
|
|
192
|
+
claimOptions.workerId,
|
|
193
|
+
drop.length,
|
|
194
|
+
...drop,
|
|
195
|
+
...claimOptions.queues,
|
|
196
|
+
]);
|
|
197
|
+
const claimed: ClaimedJob[] = [];
|
|
198
|
+
const buried: JobRecord[] = [];
|
|
199
|
+
for (const entry of list(reply)) {
|
|
200
|
+
const [kind, flat] = list(entry);
|
|
201
|
+
const fields = fieldsOf(list(flat));
|
|
202
|
+
const record = toRecord(fields);
|
|
203
|
+
if (kind === 'b') {
|
|
204
|
+
buried.push(record);
|
|
205
|
+
continue;
|
|
206
|
+
}
|
|
207
|
+
claimed.push({
|
|
208
|
+
...record,
|
|
209
|
+
claimedBy: claimOptions.workerId,
|
|
210
|
+
claim: record.claim ?? 1,
|
|
211
|
+
claimedAt: num(fields, 'claimedAt') ?? 0,
|
|
212
|
+
visibleAt: record.visibleAt ?? 0,
|
|
213
|
+
});
|
|
214
|
+
}
|
|
215
|
+
if (buried.length > 0) claimOptions.onExhausted?.(buried);
|
|
216
|
+
return claimed;
|
|
217
|
+
},
|
|
218
|
+
|
|
219
|
+
async ack(jobId, by): Promise<boolean> {
|
|
220
|
+
const reply = await run(REDIS_ACK, [jobId, by.workerId, by.claim, nowMs(clock), doneTtlMs]);
|
|
221
|
+
return Number(reply) === 1;
|
|
222
|
+
},
|
|
223
|
+
|
|
224
|
+
async nack(jobId, nackOptions): Promise<boolean> {
|
|
225
|
+
finiteOption('the redis driver nack', 'delayMs', nackOptions.delayMs);
|
|
226
|
+
const at = nowMs(clock);
|
|
227
|
+
const error = nackOptions.error;
|
|
228
|
+
const reply = await run(REDIS_NACK, [
|
|
229
|
+
jobId,
|
|
230
|
+
nackOptions.workerId,
|
|
231
|
+
nackOptions.claim,
|
|
232
|
+
at,
|
|
233
|
+
nackState(nackOptions),
|
|
234
|
+
at + nackOptions.delayMs,
|
|
235
|
+
nackOptions.countsAsAttempt === false ? '0' : '1',
|
|
236
|
+
error === undefined ? '0' : '1',
|
|
237
|
+
error ?? '',
|
|
238
|
+
error === undefined ? '' : (nackOptions.stack ?? '').slice(0, MAX_ERROR_STACK_LENGTH),
|
|
239
|
+
]);
|
|
240
|
+
return Number(reply) === 1;
|
|
241
|
+
},
|
|
242
|
+
|
|
243
|
+
async heartbeat(jobId, heartbeatOptions): Promise<boolean> {
|
|
244
|
+
finiteOption(
|
|
245
|
+
'the redis driver heartbeat',
|
|
246
|
+
'visibilityTimeoutMs',
|
|
247
|
+
heartbeatOptions.visibilityTimeoutMs,
|
|
248
|
+
);
|
|
249
|
+
const reply = await run(REDIS_HEARTBEAT, [
|
|
250
|
+
jobId,
|
|
251
|
+
heartbeatOptions.workerId ?? '',
|
|
252
|
+
heartbeatOptions.claim === undefined ? '' : heartbeatOptions.claim,
|
|
253
|
+
nowMs(clock) + heartbeatOptions.visibilityTimeoutMs,
|
|
254
|
+
]);
|
|
255
|
+
return Number(reply) === 1;
|
|
256
|
+
},
|
|
257
|
+
|
|
258
|
+
async stats(): Promise<readonly QueueStats[]> {
|
|
259
|
+
const at = nowMs(clock);
|
|
260
|
+
const rows = list(await run(REDIS_STATS, [at])).map((row): QueueStats => {
|
|
261
|
+
const [queue, ready, delayed, running, suspended, failed, dead, oldest] = list(row);
|
|
262
|
+
const first = Number(oldest);
|
|
263
|
+
return {
|
|
264
|
+
queue: String(queue),
|
|
265
|
+
ready: Number(ready),
|
|
266
|
+
delayed: Number(delayed),
|
|
267
|
+
running: Number(running),
|
|
268
|
+
suspended: Number(suspended),
|
|
269
|
+
failed: Number(failed),
|
|
270
|
+
dead: Number(dead),
|
|
271
|
+
oldestReadyMs: first < 0 ? 0 : at - first,
|
|
272
|
+
};
|
|
273
|
+
});
|
|
274
|
+
// Code units, as `collate "C"` orders `SQL_STATS` — never `localeCompare`.
|
|
275
|
+
return rows.sort((a, b) => (a.queue < b.queue ? -1 : a.queue > b.queue ? 1 : 0));
|
|
276
|
+
},
|
|
277
|
+
};
|
|
278
|
+
}
|
package/src/driver.ts
CHANGED
|
@@ -4,9 +4,10 @@
|
|
|
4
4
|
//
|
|
5
5
|
// There is no config line that picks a backend: boot always builds `postgresJobDriver()`, and a
|
|
6
6
|
// stale `jobs.driver` in `app.config.ts` is refused by `defineConfig` (25.0.0; it was read by
|
|
7
|
-
// nothing since 5.0.0). `postgresJobDriver` and `memoryJobDriver` are the
|
|
8
|
-
// deleted the all-throw `redis` and `nats` stubs
|
|
9
|
-
// ZERO job-code change, and that
|
|
7
|
+
// nothing since 5.0.0). `postgresJobDriver`, `redisJobDriver` and `memoryJobDriver` are the three
|
|
8
|
+
// that exist — 25.0.0 deleted the all-throw `redis` and `nats` stubs, and #710 built the Redis one
|
|
9
|
+
// for real. Swapping the driver is `setJobDriver(other)` and ZERO job-code change, and that
|
|
10
|
+
// is what the interface buys.
|
|
10
11
|
|
|
11
12
|
import { finiteCount, finiteOption } from '@ultimat3/core';
|
|
12
13
|
import type { BackfillLedger } from './backfill-ledger';
|