@ultimat3/jobs 27.2.3 → 27.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CLAUDE.md CHANGED
@@ -64,8 +64,9 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
64
64
  what a second call would move.**
65
65
  - **No read returns a WHOLE row** — `driver-pg-sql.test.ts` scans every production file here
66
66
  (discovered, comments stripped) for `select *`/`returning *`.
67
- - Drivers implement the six `JobDriver` methods plus optional `introspect`, `backfills`, `leases`. New
68
- capabilities go behind the interface. `inspect.ts` returns plain JSON for CLI, `/_x` and MCP.
67
+ - Drivers: the six `JobDriver` methods plus optional `introspect`, `backfills`, `leases`; new
68
+ capabilities go behind the interface. **`redisJobDriver`** (`@ultimat3/jobs/redis`, off the
69
+ barrel) is one Lua script per operation: a semantic change lands in memory, pg AND the scripts.
69
70
  - Step results are persisted BEFORE the step returns. All time is epoch ms (`nowMs()`, `clock.ts`).
70
71
  - **`postgresLeader` is correct only on a DEDICATED connection**; boot uses `postgresLeaseLeader`.
71
72
 
@@ -75,8 +76,7 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
75
76
  `finiteCount()` (a count, caller's minimum); `worker-options.ts` is where `jobWorker` reads them.
76
77
  **`bun run finite-bounds` is a floor, never proof**: `concurrencyLimiter`'s five numbers are screened
77
78
  with min 0 (zero is a HARD STOP). **A row count is `finiteCount` (min 0)**; `retry.attempts` is one with min 1.
78
- - **`WorkerOptions.concurrency` is read by OWN key** — a queue named `constructor`
79
- (`worker-slots.test.ts`).
79
+ - **`WorkerOptions.concurrency` is read by OWN key** (`worker-slots.test.ts`).
80
80
  - **`job.concurrency` is enforced by `JobDriver.leases`** (one row per held slot in `x_job_leases`);
81
81
  `limits.ts` is the per-process fast path. No lease store + a declared `concurrency` makes `start()`
82
82
  throw `X_JOB_CONCURRENCY_UNENFORCEABLE`.
@@ -110,7 +110,7 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
110
110
  synchronously with its guard.
111
111
  - **A fleet slot is taken INSIDE a `try`, released AWAITED, and HELD, not owned**: `false`, or a TTL
112
112
  with no renewal landing, CANCELS the run (`X_JOB_SLOT_LOST`); held per job id as a LIST.
113
- - **The run's signal is a controller this worker owns** (`run-signal.ts`), never `AbortSignal.any`.
113
+ - **The run's signal is a controller this worker owns** (`run-signal.ts`).
114
114
  - **A lease is HELD, not owned** (`heartbeat.ts`): one failed renewal warns; a whole window without
115
115
  one landing is `jobs.lease.lost`, on this process's clock, asked both sides of the call.
116
116
  - **A renewal is decided against `stopped()`** (`renewal-timer.ts`, re-read after every await); the
@@ -135,7 +135,7 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
135
135
  - **A lease for a run that never STARTED is `abandon()`ed, never `release()`d** — the rate stamp
136
136
  goes with it (`worker-admit.ts`). **A failed pass re-arms at the floor** (`worker-loop.ts`
137
137
  catch); `timers` is the test seam.
138
- - **Every timer body catches before it finalises** (`void work().catch(log).finally(...)`).
138
+ - **Every timer body catches before it finalises.**
139
139
  - **Suspension is control flow, and a SHED is not a suspension**: `StepSuspension` →
140
140
  `nack({ countsAsAttempt: false, park: true })`; a shed is `countsAsAttempt: false` without `park`,
141
141
  stays `ready`, logs `jobs.worker.shed` (no `last_error`). `driver-parity.test.ts`.
package/README.md CHANGED
@@ -759,10 +759,20 @@ between them — swapping is `setJobDriver(other)`, and there is **no `jobs.driv
759
759
  | Driver | Status | Backing | Use |
760
760
  |---|---|---|---|
761
761
  | `pg` | **default** | `SELECT ... FOR UPDATE SKIP LOCKED`, a partial unique index on `(name, coalesce(tenant_id, ''), idempotency_key)` over live rows (`x_jobs_name_tenant_idempotency_live_idx`), lease-based leader, `x_job_leases` | zero-infra start, most apps |
762
+ | `redis` | complete, `As of 2026-10` | `Bun.redis`; one Lua script per operation, every key in one `{hash tag}` slot | a queue off the database; no `introspect`, `backfills` or `leases` |
762
763
  | `memory` | complete | in-process maps | `x dev`, tests |
763
764
 
764
- There is **no NATS or Redis jobs driver**: `createNatsDriver` and `createRedisDriver`, all-throw
765
- stubs, were deleted in 25.0.0. A real `redisJobDriver()` would arrive as a minor.
765
+ **`redisJobDriver({ client?, prefix?, clock?, doneTtlMs? })`** passes `@ultimat3/testing`'s
766
+ `jobDriverConformance`, the suite the other two pass, and `driver-redis.live.test.ts` runs one
767
+ script of operations on it and on the memory driver and compares every answer. Installed with
768
+ `setJobDriver(redisJobDriver())` (or `ServeOptions.runtime.jobs`), imported from its own entry
769
+ `@ultimat3/jobs/redis` so the barrel every role boots carries none of it; `Bun.redis` reads `REDIS_URL`.
770
+ What it does not carry: `introspect` (so `x jobs show`/`retry`/`cancel` and queue pauses answer
771
+ for Postgres only), `backfills`, and `leases` — a job declaring `concurrency` refuses
772
+ `jobWorker().start()` (`X_JOB_CONCURRENCY_UNENFORCEABLE`). A `done` row and its steps expire
773
+ after `doneTtlMs` (a week; `0` keeps them); `dead` and `failed` rows are kept. The outbox still
774
+ stages in Postgres — the relay publishes committed rows onto it with their ids. The NATS stub
775
+ deleted in 25.0.0 stays deleted.
766
776
 
767
777
  The pg SQL lives verbatim in `src/driver-pg-*sql.ts` (`SQL_CLAIM`, `SQL_ENQUEUE`, `SQL_NACK`, …)
768
778
  so an agent debugging a stuck queue can read and run the exact statement. The barrel exports
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@ultimat3/jobs",
3
- "version": "27.2.3",
3
+ "version": "27.4.0",
4
4
  "description": "Durable background work: steps, transactional outbox, cron tasks, one driver interface",
5
5
  "license": "MIT",
6
6
  "type": "module",
@@ -14,7 +14,8 @@
14
14
  "provenance": true
15
15
  },
16
16
  "exports": {
17
- ".": "./src/index.ts"
17
+ ".": "./src/index.ts",
18
+ "./redis": "./src/driver-redis.ts"
18
19
  },
19
20
  "files": [
20
21
  "src",
@@ -32,10 +33,10 @@
32
33
  "test": "bun test"
33
34
  },
34
35
  "dependencies": {
35
- "@ultimat3/core": "27.2.3",
36
- "@ultimat3/db": "27.2.3",
37
- "@ultimat3/entity": "27.2.3",
38
- "@ultimat3/schema": "27.2.3",
39
- "@ultimat3/time": "27.2.3"
36
+ "@ultimat3/core": "27.4.0",
37
+ "@ultimat3/db": "27.4.0",
38
+ "@ultimat3/entity": "27.4.0",
39
+ "@ultimat3/schema": "27.4.0",
40
+ "@ultimat3/time": "27.4.0"
40
41
  }
41
42
  }
@@ -0,0 +1,231 @@
1
+ // The Redis job driver's statements: one Lua script per queue operation, so each runs atomically
2
+ // on the server the way each `driver-pg-sql.ts` statement runs in one transaction. The TypeScript
3
+ // that binds their arguments, and the key layout they share, is `driver-redis.ts`.
4
+ //
5
+ // Every key a script touches carries the namespace's `{hash tag}`, so all of them live in ONE slot
6
+ // and a script is legal on Redis Cluster and Dragonfly alike: `KEYS[1]` is always a namespaced key,
7
+ // which is what routes the call, and the others are built from `ARGV[1]` inside the slot it named.
8
+
9
+ import { LEASE_LAPSED_FINAL_ATTEMPT } from './driver';
10
+
11
+ /**
12
+ * The live states as a Lua set, the four `LIVE_STATES` names — the idempotency namespace holds a
13
+ * key only while one of them does, exactly as `x_jobs_name_tenant_idempotency_live_idx` does.
14
+ */
15
+ const LIVE = `local LIVE = { ready = true, delayed = true, running = true, suspended = true }`;
16
+
17
+ /**
18
+ * The claim fence `SQL_ACK` carries: the row is `running`, held by this worker, under this claim.
19
+ * `ARGV[1]` namespace, `ARGV[2]` id, `ARGV[3]` worker, `ARGV[4]` claim ordinal.
20
+ */
21
+ const FENCE = `
22
+ local P = ARGV[1]
23
+ local id = ARGV[2]
24
+ local job = P .. ':job:' .. id
25
+ local held = redis.call('HMGET', job, 'state', 'claimedBy', 'claim', 'queue', 'nk', 'runId')
26
+ if held[1] ~= 'running' or held[2] ~= ARGV[3] or held[3] ~= ARGV[4] then return 0 end
27
+ local queue = held[4]
28
+ redis.call('HDEL', job, 'visibleAt', 'claimedBy')
29
+ redis.call('ZREM', P .. ':run:' .. queue, id)
30
+ `;
31
+
32
+ /**
33
+ * `ARGV`: namespace, id, name, queue, input, idempotency key, runId, maxAttempts, runAt, now,
34
+ * tenant, traceparent, enqueuedBy, namespace field. Answers `{ outcome, id, runId }`: `published`
35
+ * when the caller-allocated id already names a row (`SQL_ENQUEUE`'s `not exists`), `deduped` when
36
+ * a live row holds the key, `created` otherwise.
37
+ */
38
+ export const REDIS_ENQUEUE = `
39
+ ${LIVE}
40
+ local P = ARGV[1]
41
+ local id = ARGV[2]
42
+ local job = P .. ':job:' .. id
43
+ if redis.call('EXISTS', job) == 1 then
44
+ return { 'published', id, redis.call('HGET', job, 'runId') }
45
+ end
46
+ local nk = ARGV[14]
47
+ local holder = redis.call('HGET', P .. ':idem', nk)
48
+ if holder then
49
+ local row = redis.call('HMGET', P .. ':job:' .. holder, 'state', 'runId')
50
+ if row[1] and LIVE[row[1]] then return { 'deduped', holder, row[2] } end
51
+ end
52
+ local now = tonumber(ARGV[10])
53
+ local runAt = tonumber(ARGV[9])
54
+ local state = 'ready'
55
+ if runAt > now then state = 'delayed' end
56
+ redis.call('HSET', job,
57
+ 'id', id, 'name', ARGV[3], 'queue', ARGV[4], 'input', ARGV[5], 'idempotencyKey', ARGV[6],
58
+ 'runId', ARGV[7], 'attempt', '0', 'maxAttempts', ARGV[8], 'state', state, 'runAt', ARGV[9],
59
+ 'createdAt', ARGV[10], 'updatedAt', ARGV[10], 'claim', '0', 'nk', nk)
60
+ if ARGV[11] ~= '' then redis.call('HSET', job, 'tenantId', ARGV[11]) end
61
+ if ARGV[12] ~= '' then redis.call('HSET', job, 'traceparent', ARGV[12]) end
62
+ if ARGV[13] ~= '' then redis.call('HSET', job, 'enqueuedBy', ARGV[13]) end
63
+ redis.call('ZADD', P .. ':wait:' .. ARGV[4], runAt, id)
64
+ redis.call('SADD', P .. ':queues', ARGV[4])
65
+ redis.call('HSET', P .. ':idem', nk, id)
66
+ return { 'created', id, ARGV[7] }
67
+ `.trim();
68
+
69
+ /**
70
+ * `SQL_CLAIM`. `ARGV`: namespace, now, limit, visibility ms, worker, the count of `dropExhausted`
71
+ * names, those names, then the queues. Candidates are every due row of each queue — ready,
72
+ * delayed, suspended, or running past its lease — at most `limit` from each source, ordered by
73
+ * `runAt` and cut at `limit`. A running row on its final attempt is BURIED (`dead`, or `failed`
74
+ * for a dropped name) and never handed out. Answers one `{ 'c' | 'b', HGETALL… }` per row.
75
+ */
76
+ export const REDIS_CLAIM = `
77
+ local P = ARGV[1]
78
+ local now = tonumber(ARGV[2])
79
+ local limit = tonumber(ARGV[3])
80
+ local visibility = tonumber(ARGV[4])
81
+ local worker = ARGV[5]
82
+ local dropCount = tonumber(ARGV[6])
83
+ local drop = {}
84
+ for i = 1, dropCount do drop[ARGV[6 + i]] = true end
85
+ local candidates = {}
86
+ if limit > 0 then
87
+ for i = 7 + dropCount, #ARGV do
88
+ local queue = ARGV[i]
89
+ for _, source in ipairs({ 'wait', 'park' }) do
90
+ local found = redis.call('ZRANGEBYSCORE', P .. ':' .. source .. ':' .. queue, '-inf', now,
91
+ 'WITHSCORES', 'LIMIT', 0, limit)
92
+ for j = 1, #found, 2 do
93
+ table.insert(candidates, { id = found[j], runAt = tonumber(found[j + 1]) })
94
+ end
95
+ end
96
+ local lapsed = redis.call('ZRANGEBYSCORE', P .. ':run:' .. queue, '-inf', now, 'LIMIT', 0, limit)
97
+ for _, id in ipairs(lapsed) do
98
+ local runAt = tonumber(redis.call('HGET', P .. ':job:' .. id, 'runAt'))
99
+ if runAt and runAt <= now then table.insert(candidates, { id = id, runAt = runAt }) end
100
+ end
101
+ end
102
+ end
103
+ table.sort(candidates, function(a, b)
104
+ if a.runAt == b.runAt then return a.id < b.id end
105
+ return a.runAt < b.runAt
106
+ end)
107
+ local out = {}
108
+ for i = 1, math.min(limit, #candidates) do
109
+ local id = candidates[i].id
110
+ local job = P .. ':job:' .. id
111
+ local row = redis.call('HMGET', job, 'state', 'attempt', 'maxAttempts', 'name', 'queue', 'claim', 'nk')
112
+ local state, attempt, maxAttempts, name, queue = row[1], tonumber(row[2]), tonumber(row[3]), row[4], row[5]
113
+ if state == 'running' and attempt >= maxAttempts then
114
+ local final = 'dead'
115
+ if drop[name] then final = 'failed' end
116
+ redis.call('ZREM', P .. ':run:' .. queue, id)
117
+ redis.call('HDEL', job, 'visibleAt', 'claimedBy', 'lastErrorStack')
118
+ redis.call('HSET', job, 'state', final, 'lastError', '${LEASE_LAPSED_FINAL_ATTEMPT}',
119
+ 'updatedAt', ARGV[2])
120
+ redis.call('ZADD', P .. ':' .. final .. ':' .. queue, now, id)
121
+ redis.call('HDEL', P .. ':idem', row[7])
122
+ table.insert(out, { 'b', redis.call('HGETALL', job) })
123
+ else
124
+ redis.call('ZREM', P .. ':wait:' .. queue, id)
125
+ redis.call('ZREM', P .. ':park:' .. queue, id)
126
+ local visibleAt = now + visibility
127
+ redis.call('HSET', job, 'state', 'running', 'attempt', attempt + 1,
128
+ 'claim', tonumber(row[6]) + 1, 'claimedBy', worker, 'claimedAt', ARGV[2],
129
+ 'visibleAt', visibleAt, 'updatedAt', ARGV[2])
130
+ redis.call('ZADD', P .. ':run:' .. queue, visibleAt, id)
131
+ table.insert(out, { 'c', redis.call('HGETALL', job) })
132
+ end
133
+ end
134
+ return out
135
+ `.trim();
136
+
137
+ /** `SQL_ACK`. `ARGV`: the fence's four, then now, then the done row's time to live (0: none). */
138
+ export const REDIS_ACK = `
139
+ ${FENCE}
140
+ redis.call('HSET', job, 'state', 'done', 'updatedAt', ARGV[5])
141
+ redis.call('HDEL', P .. ':idem', held[5])
142
+ local ttl = tonumber(ARGV[6])
143
+ if ttl > 0 then
144
+ redis.call('PEXPIRE', job, ttl)
145
+ redis.call('PEXPIRE', P .. ':steps:' .. held[6], ttl)
146
+ end
147
+ return 1
148
+ `.trim();
149
+
150
+ /**
151
+ * `SQL_NACK`. `ARGV`: the fence's four, then now, the state `nackState` chose, runAt, `1` when the
152
+ * attempt counts, `1` when an error is bound, the error, the stack (`''` for none).
153
+ */
154
+ export const REDIS_NACK = `
155
+ ${FENCE}
156
+ local state = ARGV[6]
157
+ local attempt = tonumber(redis.call('HGET', job, 'attempt'))
158
+ if ARGV[8] ~= '1' then attempt = math.max(0, attempt - 1) end
159
+ redis.call('HSET', job, 'state', state, 'runAt', ARGV[7], 'attempt', attempt, 'updatedAt', ARGV[5])
160
+ if ARGV[9] == '1' then
161
+ redis.call('HSET', job, 'lastError', ARGV[10])
162
+ if ARGV[11] == '' then redis.call('HDEL', job, 'lastErrorStack')
163
+ else redis.call('HSET', job, 'lastErrorStack', ARGV[11]) end
164
+ end
165
+ if state == 'ready' then
166
+ redis.call('ZADD', P .. ':wait:' .. queue, ARGV[7], id)
167
+ elseif state == 'suspended' then
168
+ redis.call('ZADD', P .. ':park:' .. queue, ARGV[7], id)
169
+ else
170
+ redis.call('ZADD', P .. ':' .. state .. ':' .. queue, ARGV[5], id)
171
+ redis.call('HDEL', P .. ':idem', held[5])
172
+ end
173
+ return 1
174
+ `.trim();
175
+
176
+ /**
177
+ * `SQL_HEARTBEAT`. `ARGV`: namespace, id, worker (`''`: any), claim (`''`: any), the new lease's
178
+ * end. Answers 1 when the lease moved.
179
+ */
180
+ export const REDIS_HEARTBEAT = `
181
+ local P = ARGV[1]
182
+ local id = ARGV[2]
183
+ local job = P .. ':job:' .. id
184
+ local held = redis.call('HMGET', job, 'state', 'claimedBy', 'claim', 'queue')
185
+ if held[1] ~= 'running' then return 0 end
186
+ if ARGV[3] ~= '' and held[2] ~= ARGV[3] then return 0 end
187
+ if ARGV[4] ~= '' and held[3] ~= ARGV[4] then return 0 end
188
+ redis.call('HSET', job, 'visibleAt', ARGV[5])
189
+ redis.call('ZADD', P .. ':run:' .. held[4], ARGV[5], id)
190
+ return 1
191
+ `.trim();
192
+
193
+ /**
194
+ * `SQL_STATS`, one consistent read. `ARGV`: namespace, now. Answers one row per queue:
195
+ * `{ queue, ready, delayed, running, suspended, failed, dead, oldest runAt or -1 }` — due is due
196
+ * whatever state the row was WRITTEN in, the split `SQL_STATS` makes.
197
+ */
198
+ export const REDIS_STATS = `
199
+ local P = ARGV[1]
200
+ local now = tonumber(ARGV[2])
201
+ local out = {}
202
+ for _, queue in ipairs(redis.call('SMEMBERS', P .. ':queues')) do
203
+ local wait = P .. ':wait:' .. queue
204
+ local oldest = redis.call('ZRANGE', wait, 0, 0, 'WITHSCORES')
205
+ local first = -1
206
+ if oldest[2] and tonumber(oldest[2]) <= now then first = oldest[2] end
207
+ table.insert(out, {
208
+ queue,
209
+ redis.call('ZCOUNT', wait, '-inf', now),
210
+ redis.call('ZCOUNT', wait, '(' .. now, '+inf'),
211
+ redis.call('ZCARD', P .. ':run:' .. queue),
212
+ redis.call('ZCARD', P .. ':park:' .. queue),
213
+ redis.call('ZCARD', P .. ':failed:' .. queue),
214
+ redis.call('ZCARD', P .. ':dead:' .. queue),
215
+ first,
216
+ })
217
+ end
218
+ return out
219
+ `.trim();
220
+
221
+ /**
222
+ * `SQL_STEP_PUT` under a claim: the write lands only while that claim still holds the row.
223
+ * `ARGV`: namespace, job id, worker, claim, runId, step name, the record as JSON.
224
+ */
225
+ export const REDIS_STEP_PUT_FENCED = `
226
+ local P = ARGV[1]
227
+ local held = redis.call('HMGET', P .. ':job:' .. ARGV[2], 'state', 'claimedBy', 'claim')
228
+ if held[1] ~= 'running' or held[2] ~= ARGV[3] or held[3] ~= ARGV[4] then return 0 end
229
+ redis.call('HSET', P .. ':steps:' .. ARGV[5], ARGV[6], ARGV[7])
230
+ return 1
231
+ `.trim();
@@ -0,0 +1,64 @@
1
+ // The Redis driver's `StepStore`: one HASH per run (`steps:<runId>`), step name → the record as
2
+ // JSON. Through JSON exactly as the pg store persists it, so a step replays a `Date` as the same
3
+ // string under every driver. A write made under a claim is fenced in one script on the row that
4
+ // claim must still hold (`SQL_STEP_PUT`'s fence) and is `X_JOB_LEASE_LOST` otherwise.
5
+
6
+ import type { RedisSender } from './driver-redis';
7
+ import { REDIS_STEP_PUT_FENCED } from './driver-redis-scripts';
8
+ import { LeaseLostError } from './errors';
9
+ import type { StepRecord, StepStore } from './steps';
10
+ import { isStepStatus } from './steps';
11
+
12
+ interface StepStoreSeams {
13
+ /** The namespace, `{<prefix>}`. */
14
+ readonly ns: string;
15
+ /** One script, routed to the namespace's slot. */
16
+ readonly run: (script: string, args: readonly (string | number)[]) => Promise<unknown>;
17
+ readonly client: RedisSender;
18
+ }
19
+
20
+ /** A stored step, or nothing — a value that is not one of ours is never cast onto a record. */
21
+ function parseStep(text: unknown): StepRecord | undefined {
22
+ if (typeof text !== 'string') return undefined;
23
+ const record = JSON.parse(text) as StepRecord;
24
+ return isStepStatus(record.status) ? record : undefined;
25
+ }
26
+
27
+ export function redisStepStore({ ns, run, client }: StepStoreSeams): StepStore {
28
+ const key = (runId: string) => `${ns}:steps:${runId}`;
29
+ return {
30
+ async get(runId, name) {
31
+ return parseStep(await client.send('HGET', [key(runId), name]));
32
+ },
33
+ async put(record, by) {
34
+ const text = JSON.stringify({ ...record, output: record.output ?? null });
35
+ if (by === undefined) {
36
+ await client.send('HSET', [key(record.runId), record.name, text]);
37
+ return;
38
+ }
39
+ const landed = await run(REDIS_STEP_PUT_FENCED, [
40
+ by.jobId,
41
+ by.workerId,
42
+ by.claim,
43
+ record.runId,
44
+ record.name,
45
+ text,
46
+ ]);
47
+ if (Number(landed) !== 1) throw new LeaseLostError({ job: by.job, jobId: by.jobId });
48
+ },
49
+ async list(runId) {
50
+ const values = await client.send('HVALS', [key(runId)]);
51
+ const records = (Array.isArray(values) ? values : []).flatMap((text: unknown) => {
52
+ const record = parseStep(text);
53
+ return record === undefined ? [] : [record];
54
+ });
55
+ return records.sort((a, b) => a.startedAt - b.startedAt);
56
+ },
57
+ async del(runId, name) {
58
+ await client.send('HDEL', [key(runId), name]);
59
+ },
60
+ async clear(runId) {
61
+ await client.send('DEL', [key(runId)]);
62
+ },
63
+ };
64
+ }
@@ -0,0 +1,278 @@
1
+ // A Redis queue on `Bun.redis` — no client dependency, the runtime ships one. The same contract as
2
+ // the pg driver, held by `@ultimat3/testing`'s `jobDriverConformance`: every operation is one Lua
3
+ // script (`driver-redis-scripts.ts`), so a claim, a settle and a renewal are each atomic on the
4
+ // server the way a `driver-pg-sql.ts` statement is in one transaction. Its own entry,
5
+ // `@ultimat3/jobs/redis`: the barrel is on every role's boot path, and an app on Postgres loads none
6
+ // of this.
7
+ //
8
+ // Key layout, every key inside ONE hash tag `{<prefix>}` (one slot: legal on Redis Cluster):
9
+ // job:<id> HASH the row wait:<queue> ZSET ready/delayed, by runAt
10
+ // park:<queue> ZSET suspended, by runAt run:<queue> ZSET running, by lease end
11
+ // dead:<queue> ZSET dead letters, by time failed:<queue> ZSET failed rows, by time
12
+ // queues SET every queue named idem HASH live key → row id
13
+ // steps:<runId> HASH step name → record JSON
14
+
15
+ import type { Clock } from '@ultimat3/core';
16
+ import { finiteCount, finiteOption, systemClock, uuidV7 } from '@ultimat3/core';
17
+ import { nowMs } from './clock';
18
+ import type {
19
+ ClaimedJob,
20
+ EnqueueRequest,
21
+ EnqueueResult,
22
+ JobDriver,
23
+ JobRecord,
24
+ QueueStats,
25
+ } from './driver';
26
+ import {
27
+ assertClaimBounds,
28
+ assertClaimQueues,
29
+ DEFAULT_QUEUE,
30
+ isJobState,
31
+ nackState,
32
+ } from './driver';
33
+ import {
34
+ REDIS_ACK,
35
+ REDIS_CLAIM,
36
+ REDIS_ENQUEUE,
37
+ REDIS_HEARTBEAT,
38
+ REDIS_NACK,
39
+ REDIS_STATS,
40
+ } from './driver-redis-scripts';
41
+ import { redisStepStore } from './driver-redis-steps';
42
+ import { DriverUnavailableError, JobDuplicateError } from './errors';
43
+ import { MAX_ERROR_STACK_LENGTH } from './introspection';
44
+
45
+ /** The slice of Bun's Redis client this driver uses — `Bun.redis` and `new Bun.RedisClient()`. */
46
+ export interface RedisSender {
47
+ send(command: string, args: string[]): Promise<unknown>;
48
+ }
49
+
50
+ /** A finished row's default life: a week, then Redis reclaims it and its steps. */
51
+ export const DEFAULT_REDIS_DONE_TTL_MS = 7 * 24 * 60 * 60 * 1000;
52
+
53
+ export interface RedisJobDriverOptions {
54
+ /** Injected in tests and for a second server; production reads `Bun.redis` (`REDIS_URL`). */
55
+ readonly client?: RedisSender;
56
+ /** Key namespace, the hash tag every key carries. Default `x:jobs`. */
57
+ readonly prefix?: string;
58
+ readonly clock?: Clock;
59
+ /**
60
+ * How long a `done` row (and its steps) outlives its ack, in ms; `0` keeps it forever. Default
61
+ * `DEFAULT_REDIS_DONE_TTL_MS`: a queue in memory that keeps every finished row is a server that
62
+ * runs out of memory. Dead and failed rows are kept — they are the record of what went wrong.
63
+ */
64
+ readonly doneTtlMs?: number;
65
+ }
66
+
67
+ function resolveClient(injected: RedisSender | undefined): RedisSender {
68
+ if (injected !== undefined) return injected;
69
+ const ambient = (Bun as unknown as { redis?: RedisSender }).redis;
70
+ if (ambient === undefined || typeof ambient.send !== 'function') {
71
+ throw new DriverUnavailableError({
72
+ driver: 'redis',
73
+ cause: 'Bun.redis is not available, and no client was handed to redisJobDriver()',
74
+ fix: 'REDIS_URL=redis://127.0.0.1:6379 x dev — or hand one in: redisJobDriver({ client: new Bun.RedisClient(url) })',
75
+ });
76
+ }
77
+ return ambient;
78
+ }
79
+
80
+ /** A Lua reply as an array; anything else is the empty one. */
81
+ const list = (reply: unknown): readonly unknown[] => (Array.isArray(reply) ? reply : []);
82
+
83
+ /** `HGETALL`'s flat `[field, value, …]` reply as a map. */
84
+ function fieldsOf(flat: readonly unknown[]): ReadonlyMap<string, string> {
85
+ const fields = new Map<string, string>();
86
+ for (let i = 0; i + 1 < flat.length; i += 2) fields.set(String(flat[i]), String(flat[i + 1]));
87
+ return fields;
88
+ }
89
+
90
+ const num = (fields: ReadonlyMap<string, string>, name: string): number | undefined => {
91
+ const value = fields.get(name);
92
+ return value === undefined ? undefined : Number(value);
93
+ };
94
+
95
+ /** One stored row as the record every driver hands back. Absent fields stay absent. */
96
+ function toRecord(fields: ReadonlyMap<string, string>): JobRecord {
97
+ const state = fields.get('state') ?? '';
98
+ const optional = (name: keyof JobRecord & string) => {
99
+ const value = fields.get(name);
100
+ return value === undefined ? {} : { [name]: value };
101
+ };
102
+ const claim = num(fields, 'claim') ?? 0;
103
+ const visibleAt = num(fields, 'visibleAt');
104
+ return {
105
+ id: fields.get('id') ?? '',
106
+ name: fields.get('name') ?? '',
107
+ queue: fields.get('queue') ?? '',
108
+ input: JSON.parse(fields.get('input') ?? 'null') as unknown,
109
+ idempotencyKey: fields.get('idempotencyKey') ?? '',
110
+ runId: fields.get('runId') ?? '',
111
+ attempt: num(fields, 'attempt') ?? 0,
112
+ maxAttempts: num(fields, 'maxAttempts') ?? 0,
113
+ // A state the list does not know is never cast onto a record: it reads as dead, the one state
114
+ // nothing claims again — the answer `isJobState` gives every driver.
115
+ state: isJobState(state) ? state : 'dead',
116
+ runAt: num(fields, 'runAt') ?? 0,
117
+ createdAt: num(fields, 'createdAt') ?? 0,
118
+ updatedAt: num(fields, 'updatedAt') ?? 0,
119
+ ...optional('tenantId'),
120
+ ...optional('lastError'),
121
+ ...optional('lastErrorStack'),
122
+ ...optional('claimedBy'),
123
+ ...optional('traceparent'),
124
+ ...optional('enqueuedBy'),
125
+ ...(claim > 0 ? { claim } : {}),
126
+ ...(visibleAt === undefined ? {} : { visibleAt }),
127
+ };
128
+ }
129
+
130
+ /**
131
+ * The idempotency namespace field: name, tenant (`''` for none, as the index's `coalesce`) and
132
+ * key, JSON-joined so no separator inside a value can make two triples one field.
133
+ */
134
+ const namespaceField = (request: EnqueueRequest): string =>
135
+ JSON.stringify([request.name, request.tenantId ?? '', request.idempotencyKey]);
136
+
137
+ export function redisJobDriver(options: RedisJobDriverOptions = {}): JobDriver {
138
+ const client = resolveClient(options.client);
139
+ const clock = options.clock ?? systemClock;
140
+ const ns = `{${options.prefix ?? 'x:jobs'}}`;
141
+ const doneTtlMs = finiteCount(
142
+ 'redisJobDriver',
143
+ 'doneTtlMs',
144
+ options.doneTtlMs ?? DEFAULT_REDIS_DONE_TTL_MS,
145
+ );
146
+ // `KEYS[1]` routes the call to the namespace's slot; every key a script builds is in it.
147
+ const run = (script: string, args: readonly (string | number)[]): Promise<unknown> =>
148
+ client.send('EVAL', [script, '1', `${ns}:queues`, ns, ...args.map(String)]);
149
+
150
+ return {
151
+ name: 'redis',
152
+ steps: redisStepStore({ ns, run, client }),
153
+
154
+ async enqueue(request): Promise<EnqueueResult> {
155
+ const at = nowMs(clock);
156
+ const runId = request.runId ?? uuidV7();
157
+ const [outcome, id, heldRunId] = list(
158
+ await run(REDIS_ENQUEUE, [
159
+ request.id ?? uuidV7(),
160
+ request.name,
161
+ request.queue || DEFAULT_QUEUE,
162
+ JSON.stringify(request.input ?? null),
163
+ request.idempotencyKey,
164
+ runId,
165
+ request.maxAttempts,
166
+ request.runAt ?? at,
167
+ at,
168
+ request.tenantId ?? '',
169
+ request.traceparent ?? '',
170
+ request.enqueuedBy ?? '',
171
+ namespaceField(request),
172
+ ]),
173
+ ).map(String);
174
+ if (outcome === 'deduped' && request.onConflict === 'error') {
175
+ throw new JobDuplicateError({
176
+ job: request.name,
177
+ idempotencyKey: request.idempotencyKey,
178
+ existingId: id ?? '',
179
+ });
180
+ }
181
+ return { id: id ?? '', runId: heldRunId ?? runId, deduped: outcome !== 'created' };
182
+ },
183
+
184
+ async claim(claimOptions): Promise<readonly ClaimedJob[]> {
185
+ assertClaimQueues('redis', claimOptions);
186
+ assertClaimBounds('redis', claimOptions);
187
+ const drop = claimOptions.dropExhausted ?? [];
188
+ const reply = await run(REDIS_CLAIM, [
189
+ nowMs(clock),
190
+ claimOptions.limit,
191
+ claimOptions.visibilityTimeoutMs,
192
+ claimOptions.workerId,
193
+ drop.length,
194
+ ...drop,
195
+ ...claimOptions.queues,
196
+ ]);
197
+ const claimed: ClaimedJob[] = [];
198
+ const buried: JobRecord[] = [];
199
+ for (const entry of list(reply)) {
200
+ const [kind, flat] = list(entry);
201
+ const fields = fieldsOf(list(flat));
202
+ const record = toRecord(fields);
203
+ if (kind === 'b') {
204
+ buried.push(record);
205
+ continue;
206
+ }
207
+ claimed.push({
208
+ ...record,
209
+ claimedBy: claimOptions.workerId,
210
+ claim: record.claim ?? 1,
211
+ claimedAt: num(fields, 'claimedAt') ?? 0,
212
+ visibleAt: record.visibleAt ?? 0,
213
+ });
214
+ }
215
+ if (buried.length > 0) claimOptions.onExhausted?.(buried);
216
+ return claimed;
217
+ },
218
+
219
+ async ack(jobId, by): Promise<boolean> {
220
+ const reply = await run(REDIS_ACK, [jobId, by.workerId, by.claim, nowMs(clock), doneTtlMs]);
221
+ return Number(reply) === 1;
222
+ },
223
+
224
+ async nack(jobId, nackOptions): Promise<boolean> {
225
+ finiteOption('the redis driver nack', 'delayMs', nackOptions.delayMs);
226
+ const at = nowMs(clock);
227
+ const error = nackOptions.error;
228
+ const reply = await run(REDIS_NACK, [
229
+ jobId,
230
+ nackOptions.workerId,
231
+ nackOptions.claim,
232
+ at,
233
+ nackState(nackOptions),
234
+ at + nackOptions.delayMs,
235
+ nackOptions.countsAsAttempt === false ? '0' : '1',
236
+ error === undefined ? '0' : '1',
237
+ error ?? '',
238
+ error === undefined ? '' : (nackOptions.stack ?? '').slice(0, MAX_ERROR_STACK_LENGTH),
239
+ ]);
240
+ return Number(reply) === 1;
241
+ },
242
+
243
+ async heartbeat(jobId, heartbeatOptions): Promise<boolean> {
244
+ finiteOption(
245
+ 'the redis driver heartbeat',
246
+ 'visibilityTimeoutMs',
247
+ heartbeatOptions.visibilityTimeoutMs,
248
+ );
249
+ const reply = await run(REDIS_HEARTBEAT, [
250
+ jobId,
251
+ heartbeatOptions.workerId ?? '',
252
+ heartbeatOptions.claim === undefined ? '' : heartbeatOptions.claim,
253
+ nowMs(clock) + heartbeatOptions.visibilityTimeoutMs,
254
+ ]);
255
+ return Number(reply) === 1;
256
+ },
257
+
258
+ async stats(): Promise<readonly QueueStats[]> {
259
+ const at = nowMs(clock);
260
+ const rows = list(await run(REDIS_STATS, [at])).map((row): QueueStats => {
261
+ const [queue, ready, delayed, running, suspended, failed, dead, oldest] = list(row);
262
+ const first = Number(oldest);
263
+ return {
264
+ queue: String(queue),
265
+ ready: Number(ready),
266
+ delayed: Number(delayed),
267
+ running: Number(running),
268
+ suspended: Number(suspended),
269
+ failed: Number(failed),
270
+ dead: Number(dead),
271
+ oldestReadyMs: first < 0 ? 0 : at - first,
272
+ };
273
+ });
274
+ // Code units, as `collate "C"` orders `SQL_STATS` — never `localeCompare`.
275
+ return rows.sort((a, b) => (a.queue < b.queue ? -1 : a.queue > b.queue ? 1 : 0));
276
+ },
277
+ };
278
+ }
package/src/driver.ts CHANGED
@@ -4,9 +4,10 @@
4
4
  //
5
5
  // There is no config line that picks a backend: boot always builds `postgresJobDriver()`, and a
6
6
  // stale `jobs.driver` in `app.config.ts` is refused by `defineConfig` (25.0.0; it was read by
7
- // nothing since 5.0.0). `postgresJobDriver` and `memoryJobDriver` are the two that exist — 25.0.0
8
- // deleted the all-throw `redis` and `nats` stubs. Swapping the driver is `setJobDriver(other)` and
9
- // ZERO job-code change, and that is what the interface buys.
7
+ // nothing since 5.0.0). `postgresJobDriver`, `redisJobDriver` and `memoryJobDriver` are the three
8
+ // that exist — 25.0.0 deleted the all-throw `redis` and `nats` stubs, and #710 built the Redis one
9
+ // for real. Swapping the driver is `setJobDriver(other)` and ZERO job-code change, and that
10
+ // is what the interface buys.
10
11
 
11
12
  import { finiteCount, finiteOption } from '@ultimat3/core';
12
13
  import type { BackfillLedger } from './backfill-ledger';