@owlmeans/queue 0.1.18-rc.4 → 0.1.18-rc.40

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (71) hide show
  1. package/README.md +285 -8
  2. package/agent-meta/manifest.json +9 -2
  3. package/agent-meta/skills/queue/SKILL.md +241 -12
  4. package/agent-meta/skills/scheduled-jobs/SKILL.md +151 -0
  5. package/build/bridge.d.ts +30 -0
  6. package/build/bridge.d.ts.map +1 -0
  7. package/build/bridge.js +108 -0
  8. package/build/bridge.js.map +1 -0
  9. package/build/config.d.ts +50 -0
  10. package/build/config.d.ts.map +1 -0
  11. package/build/config.js +141 -0
  12. package/build/config.js.map +1 -0
  13. package/build/consts.d.ts +33 -0
  14. package/build/consts.d.ts.map +1 -0
  15. package/build/consts.js +35 -0
  16. package/build/consts.js.map +1 -0
  17. package/build/errors.d.ts +68 -0
  18. package/build/errors.d.ts.map +1 -0
  19. package/build/errors.js +103 -0
  20. package/build/errors.js.map +1 -0
  21. package/build/index.d.ts +10 -0
  22. package/build/index.d.ts.map +1 -1
  23. package/build/index.js +10 -1
  24. package/build/index.js.map +1 -1
  25. package/build/protocol.d.ts +10 -0
  26. package/build/protocol.d.ts.map +1 -0
  27. package/build/protocol.js +56 -0
  28. package/build/protocol.js.map +1 -0
  29. package/build/route.d.ts +9 -0
  30. package/build/route.d.ts.map +1 -0
  31. package/build/route.js +31 -0
  32. package/build/route.js.map +1 -0
  33. package/build/transport.d.ts +15 -0
  34. package/build/transport.d.ts.map +1 -0
  35. package/build/transport.js +59 -0
  36. package/build/transport.js.map +1 -0
  37. package/build/types.d.ts +309 -0
  38. package/build/types.d.ts.map +1 -0
  39. package/build/types.js +2 -0
  40. package/build/types.js.map +1 -0
  41. package/build/types.local.d.ts +3 -0
  42. package/build/types.local.d.ts.map +1 -0
  43. package/build/types.local.js +2 -0
  44. package/build/types.local.js.map +1 -0
  45. package/build/utils.d.ts +4 -0
  46. package/build/utils.d.ts.map +1 -0
  47. package/build/utils.js +4 -0
  48. package/build/utils.js.map +1 -0
  49. package/build/worker.d.ts +27 -0
  50. package/build/worker.d.ts.map +1 -0
  51. package/build/worker.js +59 -0
  52. package/build/worker.js.map +1 -0
  53. package/package.json +13 -3
  54. package/src/bridge.ts +126 -0
  55. package/src/config.ts +171 -0
  56. package/src/consts.ts +38 -0
  57. package/src/errors.ts +121 -0
  58. package/src/index.ts +10 -0
  59. package/src/protocol.ts +74 -0
  60. package/src/route.ts +40 -0
  61. package/src/transport.ts +77 -0
  62. package/src/types.local.ts +3 -0
  63. package/src/types.ts +323 -0
  64. package/src/utils.ts +5 -0
  65. package/src/worker.ts +73 -0
  66. package/tests/bridge.spec.ts +64 -0
  67. package/tests/config.spec.ts +66 -0
  68. package/tests/protocol.spec.ts +58 -0
  69. package/tests/schedule.spec.ts +100 -0
  70. package/tsconfig.json +1 -0
  71. package/build/.gitkeep +0 -0
package/README.md CHANGED
@@ -1,20 +1,297 @@
1
1
  # @owlmeans/queue
2
2
 
3
- Reserved namespace for queue management. Currently a stub with no exports.
3
+ Job queues as resources, plus the route protocol that makes a queued call look like any other call.
4
+ Use it when work must outlive the request. Three cases qualify: the user would otherwise wait more
5
+ than about five seconds, accepted work must survive a process restart, or the work is retried
6
+ against a third party (payment capture, provider sync, bulk mail, report generation, model
7
+ pipelines). A request that finishes in under about two seconds and makes no external call should
8
+ not queue: do it inline. Check for a smaller fix first, such as an index, a batch write or a cached
9
+ aggregate. Recurring work is a declared schedule: the broker produces a job on an interval or a
10
+ cron pattern and a registered processor runs it.
4
11
 
5
- ## Overview
6
-
7
- This package is a placeholder for future queue/task processing functionality within the OwlMeans framework.
12
+ This package is contracts only and carries no broker code. A driver implements them
13
+ (`@owlmeans/redis-queue` does, over BullMQ). Depend on this package from a shared contract package,
14
+ and on the driver only where the application wires itself up.
8
15
 
9
16
  ## Installation
10
17
 
11
18
  ```bash
12
- bun add @owlmeans/queue
19
+ bun add @owlmeans/queue@^0.1.18-rc.40
20
+ ```
21
+
22
+ ## Concepts
23
+
24
+ - **Queue declaration**: `declareQueue(cfg, name, jobs, opts?)` in the shared backend config. It
25
+ names a queue, the job names it accepts, and its worker and default job options. It is an
26
+ address, not a deployment.
27
+ - **Listen**: `listenQueues(cfg, ...names)` in one process's own config says which queues *that
28
+ process* consumes. A process that listens to nothing is a producer only.
29
+ - **Queued protocol**: an immutable backend entrypoint protocol whose route uses `job()` from this
30
+ package (`QUEUE_PROTOCOL`). Callers use `ctx.entrypoint(protocol).call(...)` unchanged,
31
+ and the QUEUE transport carries the call through the broker.
32
+ - **Job record**: `ctx.jobs(queue)` returns a `QueueResource`, the ordinary `Resource<JobRecord>`
33
+ contract plus `wait`, `flow`, `counts` and `close`. `create` enqueues, `list` inspects and `take`
34
+ cancels.
35
+ - **Processor**: a function registered with `QueueWorkerService.process(queue, name, fn)` that gets
36
+ a `JobContext`. It must be safe to run twice and must `touch()` in long loops.
37
+ - **Single-flight id**: a `JobOptions.id` derived from what the job is about (`develop:<storyId>`).
38
+ A duplicate enqueue then returns the existing job instead of a second one.
39
+ - **Schedule**: `declareSchedule(cfg, { id, queue, name, every | pattern, … })` in the shared config.
40
+ Every process that listens to the queue reconciles the broker with the declared schedules when its
41
+ worker starts; each run is an ordinary job, and `JobContext.scheduled` names the schedule.
42
+
43
+ ## Usage
44
+
45
+ ### Declare queues in the shared backend config
46
+
47
+ ```ts
48
+ import { declareQueue } from '@owlmeans/queue'
49
+
50
+ // Split lanes by how long work takes, not by topic: a one-second read queued behind
51
+ // a thirty-minute job is a UI that looks broken.
52
+ declareQueue(cfg, APP_OPS, [appProtocols.files.get.alias, appProtocols.report.build.alias], {
53
+ worker: { concurrency: 16, lockDuration: 30_000 }
54
+ })
55
+
56
+ declareQueue(cfg, APP_BILLING, [appProtocols.invoice.capture.alias], {
57
+ worker: { concurrency: 4, lockDuration: 30_000 },
58
+ defaults: { attempts: 3, backoff: { type: 'exponential', delay: 250 } }
59
+ })
60
+ ```
61
+
62
+ The worker process names what it consumes, in its own config:
63
+
64
+ ```ts
65
+ import { listenQueues } from '@owlmeans/queue'
66
+
67
+ listenQueues(cfg, APP_OPS, APP_BILLING) // the api process omits this and only produces
68
+ ```
69
+
70
+ Both processes build their context through the same backend factory, which wires the driver once:
71
+
72
+ ```ts
73
+ import { appendRedisQueue } from '@owlmeans/redis-queue'
74
+
75
+ appendRedis<C, T>(context)
76
+ appendRedisQueue(context, { hooks: billingQueueHooks(context) })
77
+ ```
78
+
79
+ ### Declare a queued entrypoint
80
+
81
+ ```ts
82
+ import { contract, protocol, typed } from '@owlmeans/entrypoint'
83
+ import { job } from '@owlmeans/queue'
84
+ import { backend, route } from '@owlmeans/route'
85
+
86
+ const aliases = { base: 'app:report', build: 'app:report:build' } as const
87
+ const reportBase = protocol(route(aliases.base, '/reports', backend({ service: APP_WORKER })), contract())
88
+
89
+ export const appProtocols = {
90
+ report: {
91
+ base: reportBase,
92
+ build: protocol(
93
+ route(aliases.build, '/:id/build',
94
+ job({ parent: reportBase, service: APP_WORKER, queue: APP_OPS, timeout: 30_000 })),
95
+ contract.request({ params: typed<ReportParams>(ReportParamsSchema) }, typed<ReportResult>()),
96
+ ),
97
+ },
98
+ } as const
99
+ ```
100
+
101
+ The worker process serves it like any backend entrypoint; the HTTP server skips QUEUE routes. A
102
+ producer calls it with no knowledge of the broker:
103
+
104
+ ```ts
105
+ const result = await context.entrypoint(appProtocols.report.build).call({ params: { id } })
106
+ ```
107
+
108
+ Add `reply: false` to the `job()` options to resolve `Accepted` with `{ id, queue }` as soon as the
109
+ broker has taken the job.
110
+
111
+ ### Enqueue with broker options and wait
112
+
113
+ When a producer needs a job id, a delay or retries, pass the same protocol object to the typed
114
+ helpers. Never pass its alias.
115
+
116
+ ```ts
117
+ import { enqueueProtocol, waitForProtocol } from '@owlmeans/queue'
118
+
119
+ const queued = await enqueueProtocol(context, appProtocols.invoice.capture, {
120
+ body: { invoiceId },
121
+ }, {
122
+ id: `capture:${invoiceId}:${attemptSequence}`, // derived from what the job is about
123
+ delay: 250, attempts: 3, backoff: { type: 'exponential', delay: 250 },
124
+ })
125
+
126
+ const receipt = await waitForProtocol(context, appProtocols.invoice.capture, queued, { timeout: 120_000 })
127
+ ```
128
+
129
+ `queueJobOf(req)` gives a protocol handler `{ id, name, queue, attempt, touch }` when it must compare
130
+ the broker job with a claim it persisted.
131
+
132
+ ### Processors, hooks and job graphs
133
+
134
+ ```ts
135
+ import { DEFAULT_ALIAS } from '@owlmeans/queue'
136
+ import type { QueueWorkerService } from '@owlmeans/queue'
137
+
138
+ const worker = context.service<QueueWorkerService>(DEFAULT_ALIAS)
139
+
140
+ worker.process(APP_OPS, 'app:import:rows', async job => {
141
+ for (const chunk of chunks(job.data.fileId)) {
142
+ await job.touch() // renew the lock on every iteration
143
+ await importChunk(chunk) // idempotent: skips rows already imported
144
+ }
145
+ return { ok: true }
146
+ })
147
+
148
+ worker.hooks({
149
+ onJobDead: async (record, reason) => { // retries exhausted: compensate
150
+ await imports.patch({ id: record.data.importId, status: 'failed', error: reason })
151
+ }
152
+ })
153
+
154
+ // children complete before the parent; siblings run in parallel
155
+ await context.jobs(APP_OPS).flow({
156
+ name: 'app:import:finalize', data: ref, children: [
157
+ { name: 'app:import:validate', data: ref, children: [{ name: 'app:import:rows', data: ref }] }
158
+ ]
159
+ })
160
+ ```
161
+
162
+ ### Recurring jobs
163
+
164
+ ```ts
165
+ import { declareQueue, declareSchedule } from '@owlmeans/queue'
166
+
167
+ // Shared config: the queue first, then its schedules.
168
+ declareQueue(cfg, APP_MAINTENANCE, ['app:maintenance:reconcile', 'app:maintenance:purge'], {
169
+ worker: { concurrency: 1, lockDuration: 60_000 }
170
+ })
171
+ declareSchedule(cfg, {
172
+ id: 'app-nightly-reconcile', queue: APP_MAINTENANCE, name: 'app:maintenance:reconcile',
173
+ pattern: '17 3 * * *', tz: 'UTC', data: {},
174
+ })
175
+ declareSchedule(cfg, {
176
+ id: 'app-purge', queue: APP_MAINTENANCE, name: 'app:maintenance:purge', every: 15 * 60_000,
177
+ })
178
+ ```
179
+
180
+ The process that `listenQueues(cfg, APP_MAINTENANCE)` registers the processors; nothing else is
181
+ wired. A scheduled run carries no caller and no credentials, so it goes to a processor, never to a
182
+ guarded entrypoint:
183
+
184
+ ```ts
185
+ worker.process(APP_MAINTENANCE, 'app:maintenance:reconcile', async job => {
186
+ for await (const page of organizations()) {
187
+ await job.touch()
188
+ await reconcile(page) // safe to run twice: a restart re-runs the whole sweep
189
+ }
190
+ return { scheduled: job.scheduled } // 'app-nightly-reconcile'
191
+ })
13
192
  ```
14
193
 
15
- ## Related Packages
194
+ Renaming a schedule id removes the old scheduler and creates a new one; deleting a declaration
195
+ removes its scheduler at the next worker start.
196
+
197
+ ### Jobs are records
198
+
199
+ ```ts
200
+ import { JobState } from '@owlmeans/queue'
201
+
202
+ const jobs = context.jobs<ImportInput, ImportResult>(APP_OPS)
203
+
204
+ const failed = await jobs.list({ state: JobState.Failed }, { size: 20 })
205
+ const counts = await jobs.counts()
206
+ await jobs.take(jobId) // cancel and return
207
+ ```
208
+
209
+ ## Processor rules
210
+
211
+ - **`touch()` in every long loop.** The broker judges liveness by the lock. Silence longer than
212
+ `lockDuration` looks like a dead worker, so the job is re-run elsewhere while this one is still
213
+ going.
214
+ - **Children report, parents decide.** A child returns `{ ok: false, error }` for a domain failure.
215
+ Throwing means "infrastructure broke, retry me". Parents read `children()` / `failedChildren()`.
216
+ - **Retries default to one attempt** (`DEFAULT_ATTEMPTS`), because a blind retry of model work
217
+ re-spends the tokens that just failed. Raise `attempts` per queue for cheap idempotent work.
218
+ - **`job.signal` is not an interrupt.** An orderly stop closes workers first and waits for jobs in
219
+ flight to return.
220
+
221
+ ## Guards and errors
222
+
223
+ `handleJob` rebuilds the request from the envelope and runs the entrypoint's guards the way the
224
+ HTTP boundary does, attaching the entity. A guarded queued entrypoint therefore needs the
225
+ producer's credentials in the envelope `headers`. `cfg.queue.envelopeTtl` (seconds) bounds how long
226
+ after `enqueuedAt` an envelope is still accepted; past it, the job answers `EnvelopeExpired`.
227
+
228
+ A refusal the caller must recognise is carried in the reply and rebuilt on the producing side as its
229
+ own class, so `instanceof` keeps working across the broker. `QueueTimeout` means the WAIT ended,
230
+ not the job, so read the job back to learn what became of it.
231
+
232
+ ## API
233
+
234
+ | Symbol | Kind | Purpose |
235
+ |---|---|---|
236
+ | `declareQueue(cfg, name, jobs, opts?)` | function | Declare a queue and the job names it accepts; re-declaring replaces |
237
+ | `listenQueues(cfg, ...names)` | function | Name the queues this process consumes |
238
+ | `queueOf(cfg, name)`, `queueOfJob(cfg, job)`, `isListening(cfg, name)` | function | Read declarations back; `queueOf` throws `UnknownQueue` |
239
+ | `declareSchedule(cfg, schedule)` | function | Declare a recurring job after checking it; re-declaring an id replaces |
240
+ | `schedulesOf(cfg, queue?)` | function | The declared schedules, of one queue when named |
241
+ | `assertSchedule(cfg, schedule)`, `assertSchedules(cfg)` | function | Check one schedule, or every declared one plus duplicate ids |
242
+ | `enqueueProtocol(ctx, protocol, request, options?)` | function | Enqueue a QUEUE protocol with `JobOptions`, typed by the protocol |
243
+ | `waitForProtocol(ctx, protocol, job, { timeout }?)` | function | Wait for that job and unwrap the typed reply |
244
+ | `queueJobOf(request)` | function | `QueueJobMeta` of the broker job behind a handler's request, or `null` |
245
+ | `appendQueueTransport(ctx, alias?)`, `makeQueueTransport(alias?)` | function | Register the QUEUE transport so `call()` routes through the broker |
246
+ | `queueWorkerMiddleware(alias?)` | function | Start the worker at the Ready stage, only in a process that listens |
247
+ | `handleJob(ctx, job)`, `entrypointProcessor(ctx)`, `servedJobs(ctx)` | function | The bridge a driver dispatches entrypoint jobs through |
248
+ | `requestOf(envelope)`, `assertFresh(envelope, ttl?)` | function | Rebuild a request from an envelope; enforce `envelopeTtl` |
249
+ | `QueueConfig`, `QueueDeclaration`, `QueueWorkerOptions`, `JobOptions`, `ScheduleDeclaration` | type | Configuration shapes |
250
+ | `QueueResource<D, R>` | type | A queue as a resource, plus `queue`, `wait`, `flow`, `counts`, `close` |
251
+ | `JobRecord<D, R>`, `JobEvent<R>`, `FlowSpec<D>` | type | A job as a record, a lifecycle event, a graph node |
252
+ | `QueueWorkerService` | type | `process`, `start`, `stop`, `listening`, `hooks` |
253
+ | `JobContext<D>`, `JobProcessor<D, R>` | type | What a processor receives (`scheduled` names the schedule of a scheduled run); the processor signature |
254
+ | `QueueHooks` | type | `wrapHandler`, `onJobResult`, `onJobStalled`, `onJobDead` |
255
+ | `QueueJobMeta`, `JobEnvelope`, `JobReply<T>` | type | Broker identity for a handler, the call envelope, the reply shape |
256
+ | `QueueAppend`, `QueueDriver` | type | The `ctx.jobs(queue?)` mixin a driver installs; what a driver supplies |
257
+ | `Config`, `Context`, `QueueTransportRequest`, `QueueTransportResponse` | type | Context, config and transport types |
258
+ | `JobState`, `JobEventType` | enum | Job lifecycle states and event types |
259
+ | `isSettled(state?)` | function | Whether a state is final |
260
+ | `DEFAULT_ALIAS`, `DEFAULT_JOB_TIMEOUT`, `DEFAULT_ATTEMPTS` | const | `'queue'`, `60_000` ms, `1` |
261
+ | `QueueError`, `QueueTimeout`, `UnknownJob`, `UnknownJobName`, `UnknownQueue`, `QueueNotListening`, `JobNotServed`, `EnvelopeExpired`, `ScheduleMisdeclared` | class | Error family |
262
+
263
+ ## Common pitfalls
264
+
265
+ - **Queueing work the request could finish.** It adds a second process to deploy and a place to fail.
266
+ - **Passing alias strings to `enqueueProtocol`/`waitForProtocol`.** They take the protocol object and
267
+ reject strings and non-QUEUE declarations.
268
+ - **Enqueueing a job name the queue did not declare.** It throws `UnknownJobName` rather than
269
+ parking a job nothing can process.
270
+ - **Registering a processor in a process that does not `listen`.** It throws `QueueNotListening`.
271
+ - **Putting `listen` in the shared declaration.** Every deployment of that binary would then
272
+ consume everything.
273
+ - **Random job ids on an admission step.** Derive the id from the subject, or a retried request
274
+ enqueues twice.
275
+ - **Long loops without `touch()`**, or processors that are not safe to run twice.
276
+ - **Throwing from a child for a domain failure.** Return `{ ok: false, error }` instead.
277
+ - **Equal child ids across queues in one flow.** `children()` is keyed by bare id and collapses them.
278
+ - **A self-re-enqueueing job as a cron.** Declare a schedule. `delay` defers a single job once.
279
+ - **A schedule on a queue nothing consumes.** Only a listening worker creates the scheduler, and the
280
+ broker produces each next run when the previous one starts.
281
+ - **Loading different schedule lists in two processes that listen to one queue.** Each start removes
282
+ the schedules its own list does not name.
283
+ - **Declaring a schedule before its queue.** `declareSchedule` checks the queue and the job name and
284
+ throws `UnknownQueue` / `UnknownJobName`.
285
+ - **Calling `ctx.jobs()` with no name once a second queue exists.** It throws `UnknownQueue`.
286
+
287
+ ## Related packages
16
288
 
17
- - [`@owlmeans/redis-resource`](../redis-resource) — Redis Streams provide queue-like semantics today
289
+ - [`@owlmeans/redis-queue`](../redis-queue): the BullMQ driver, and where queue integration tests live
290
+ - [`@owlmeans/route`](../route): transport-neutral route declarations used by the queue-owned `job()` builder
291
+ - [`@owlmeans/entrypoint`](../entrypoint): protocol declarations and the transport seam
292
+ - [`@owlmeans/resource`](../resource): the `Resource` contract jobs are read through
293
+ - [`@owlmeans/server-job`](../server-job) / [`@owlmeans/client-job`](../client-job): listing,
294
+ cancelling and following jobs from an application's UI
18
295
 
19
296
  <!-- owlmeans:agent-guidance:start -->
20
297
  ## Agent guidance
@@ -24,7 +301,7 @@ This package ships embedded agent skills under `agent-meta/`. After installing y
24
301
  your project's skill store (`.agents/skills/`):
25
302
 
26
303
  ```sh
27
- npx @owlmeans/agent-skills
304
+ npx @owlmeans/agent-skills@^0.1.18-rc.48
28
305
  ```
29
306
 
30
307
  The embedded files are version-matched to this package release. Do not edit them
@@ -1,10 +1,17 @@
1
1
  {
2
2
  "schemaVersion": 2,
3
3
  "package": "@owlmeans/queue",
4
- "version": "0.1.18-rc.0",
5
- "generatedAt": "2026-08-16T22:20:50.517Z",
4
+ "version": "0.1.18-rc.40",
5
+ "generatedAt": "2026-10-05T08:32:19.092Z",
6
6
  "canonicalRepo": "https://github.com/owlmeans/common",
7
7
  "entries": [
8
+ {
9
+ "kind": "skill",
10
+ "name": "scheduled-jobs",
11
+ "category": "multi-package",
12
+ "file": "skills/scheduled-jobs/SKILL.md",
13
+ "canonicalPath": ".agents/skills/scheduled-jobs/SKILL.md"
14
+ },
8
15
  {
9
16
  "kind": "skill",
10
17
  "name": "queue",
@@ -1,6 +1,6 @@
1
1
  ---
2
2
  name: queue
3
- description: How to use @owlmeans/queue — abstract job-queue interface (currently a stub) to be implemented by transport-specific queue services. Auto-invoked when importing queue types.
3
+ description: How to use @owlmeans/queue — protocol-object enqueue/wait, job queues as resources, QUEUE declarations, producer/consumer split, single-flight job ids, recurring schedules, lifecycle hooks, and processor rules. Auto-invoked when importing queue types, declaring a job protocol, or declaring a schedule.
4
4
  user-invocable: false
5
5
  ---
6
6
  <!-- AUTO-GENERATED — do not edit. Regenerate via sync-agent-meta. -->
@@ -8,24 +8,253 @@ user-invocable: false
8
8
  # @owlmeans/queue
9
9
 
10
10
  **Layer:** Infra
11
- **Install:** `"@owlmeans/queue": "^0.1.18-rc.0"` in `dependencies`
11
+ **Install:** `"@owlmeans/queue": "^0.1.18-rc.40"` in `dependencies`
12
12
 
13
- ## Key Exports
13
+ Backend transport contracts only. It carries no broker code — a driver package
14
+ (`@owlmeans/redis-queue`) implements them. Depend on this one from a shared backend package;
15
+ generic common and frontend packages must not import it. Depend on the driver only where the
16
+ application wires itself up.
14
17
 
15
- This package is currently a stub that defines the queue interface. Concrete implementations (Redis BullMQ, RabbitMQ, etc.) will register a service that satisfies this interface.
18
+ ## The one idea
16
19
 
17
- | Export | Description |
18
- |--------|-------------|
19
- | Queue types (placeholder) | Abstract queue interface |
20
+ A queued call is an immutable entrypoint protocol whose route names `QUEUE_PROTOCOL`.
21
+ Ordinary calls use `ctx.entrypoint(protocol).call(request)`. Code that needs broker options or a
22
+ job handle uses `enqueueProtocol(ctx, protocol, request, options)` and
23
+ `waitForProtocol(ctx, protocol, job)`. Both derive queue and job name from the protocol object and
24
+ preserve its exact request/reply pair; raw strings and non-queue declarations are rejected.
25
+
26
+ That is what makes it worth having: moving a service-to-service call onto the broker is a change
27
+ to a declaration, not to the places that call it, and the broker holds the work across a restart of
28
+ either side — which an in-process call cannot.
29
+
30
+ ## Declaring
31
+
32
+ Queues and the jobs they accept live in the SHARED backend package, so producers and consumers
33
+ read one list:
34
+
35
+ ```typescript
36
+ declareQueue(cfg, AGENT_WORK, [story.develop.alias], {
37
+ worker: { concurrency: 3, lockDuration: 60_000 }
38
+ })
39
+ ```
40
+
41
+ The job list is enforced, not documentation: enqueueing a name the queue does not declare throws
42
+ `UnknownJobName` instead of parking a job nothing can ever process. Re-declaring a name replaces,
43
+ so a helper that runs once per composed app is safe to call twice.
44
+
45
+ Which queues a process CONSUMES is separate, and lives in that process's own config:
46
+
47
+ ```typescript
48
+ listenQueues(cfg, AGENT_WORK, AGENT_OPS) // absent ⇒ producer only
49
+ ```
50
+
51
+ Keep them apart. A declaration is an address; `listen` is a deployment fact. If a worker bound
52
+ whatever it could serve, every deployment of the same binary would consume everything it imports.
53
+
54
+ A queued entrypoint is an immutable protocol with `job()` in place of `backend()`:
55
+
56
+ ```typescript
57
+ protocol(
58
+ route(agent.story.develop, '/:id/develop',
59
+ job({ parent: agent.story.base, service: AGENT, queue: AGENT_WORK, timeout: 30_000 })),
60
+ contract.request({ params: typed<StoryParams>(StoryParamsSchema) }, typed()),
61
+ )
62
+ ```
63
+
64
+ Import `job`, `QUEUE_PROTOCOL`, `isQueueRoute` and `queueRouteOptions` from `@owlmeans/queue`.
65
+ The generic route package deliberately knows none of the queue option shape.
66
+
67
+ The alias appears in `declareQueue(...jobs)` only because the broker is a string-keyed adapter.
68
+ Application producers and handlers keep the protocol object.
69
+
70
+ When an admission record owns single flight, select the job id atomically first, then enqueue it:
71
+
72
+ ```typescript
73
+ const queued = await enqueueProtocol(context, tokenBalance.reconcile, {
74
+ body: { projectionId },
75
+ }, { id: claimId, delay: 250, attempts: 3, backoff: { type: 'exponential', delay: 250 } })
76
+ const result = await waitForProtocol(context, tokenBalance.reconcile, queued)
77
+ ```
78
+
79
+ Per-call `id`, delay, attempts, backoff and retention belong here; putting them into an alias-based
80
+ helper forfeits protocol inference. `queueJobOf(request)` exposes `{ id, name, queue, attempt }` to
81
+ the rare protocol handler that must compare a broker job with its persisted claim.
82
+
83
+ `reply: false` makes it fire-and-forget: the call resolves `Accepted` with `{ id, queue }` as soon
84
+ as the BROKER has accepted the enqueue — before any worker has picked the job up, so the resolution
85
+ says the job exists and nothing about it having started. That is what a long pipeline wants; watch
86
+ the job by its id for anything further. The default is to wait for the value, for
87
+ `req.timeout ?? route.timeout ?? DEFAULT_JOB_TIMEOUT` (60 s).
88
+
89
+ ## Jobs are records
90
+
91
+ `ctx.jobs<D, R>(queue)` returns a `QueueResource<D, R>` — a `Resource<JobRecord<D, R>>` plus
92
+ pub/sub. So inspecting a queue is the resource contract you already know: `create` enqueues,
93
+ `get`/`load`/`list`/`count` inspect, `delete`/`take` cancel, `purge` bulk-cancels. Only what the
94
+ base contract cannot express is added: `wait(id, { timeout })`, `flow(root)`, `counts()`, and
95
+ `close()` for the connections a producer that waited holds.
96
+
97
+ Called with no queue name it answers the SOLE declared queue and throws `UnknownQueue` once there
98
+ are two — an application that grows a second queue is told to name one rather than quietly served
99
+ the wrong backlog. A queue nothing declared throws the same error at the call site.
20
100
 
21
- ## Usage
101
+ Give `JobOptions.id` a value derived from what the job is ABOUT (`develop:<storyId>`) and a
102
+ duplicate enqueue becomes a no-op returning the job already there — no `RecordExists`. That is
103
+ what makes an admission step safe to retry.
104
+
105
+ ## Writing a processor
22
106
 
23
107
  ```typescript
24
- // Reserve the alias for your queue implementation
25
- import type { Queue } from '@owlmeans/queue'
26
- const queue = ctx.service<Queue>('queue')
108
+ const worker = context.service<QueueWorkerService>(DEFAULT_ALIAS)
109
+
110
+ worker.process(AGENT_WORK, 'agent:story:code', async job => {
111
+ for (const screen of screens) {
112
+ await job.touch() // ← not optional
113
+ await develop(screen)
114
+ }
115
+ return { ok: true }
116
+ })
27
117
  ```
28
118
 
119
+ Registering against a queue this process does not `listen` to throws `QueueNotListening` — the
120
+ question is a deployment one, so it is answered at wiring time rather than when a job arrives.
121
+
122
+ Three rules, each of which has a failure mode attached:
123
+
124
+ - **`touch()` in every long loop.** The broker judges liveness by the lock. Silence for longer
125
+ than `lockDuration` is indistinguishable from a dead worker, so the job is handed to someone
126
+ else while this one is still working — you get two runs, not one. `progress(value)` reports
127
+ without renewing, and the value must be JSON-carryable.
128
+ - **Children report, parents decide.** A child returns `{ ok: false, error }` for a DOMAIN
129
+ failure; it does not throw. Throwing means "infrastructure broke, retry me". A child that
130
+ exhausts its attempts does not fail its parent: the parent runs and reads `children()` /
131
+ `failedChildren()` — both keyed by bare job id — and decides. Give the children of a flow that
132
+ spans queues distinct ids, because that re-keying collapses two ids that are equal across queues.
133
+ - **Retries default to one attempt** (`DEFAULT_ATTEMPTS`). These jobs mostly talk to a model, and
134
+ a blind retry re-spends the tokens that just failed. Raise `attempts` per queue only where jobs
135
+ are cheap and idempotent.
136
+
137
+ `job.signal` is the queue's abort signal, and it is NOT an interrupt. The driver stops by closing
138
+ every worker first — which waits for the jobs in flight to return — and aborts only afterwards, so
139
+ a processor cannot bail out on it and does not need to. An orderly stop drains; an outright kill is
140
+ covered by the rule below.
141
+
142
+ ## Being restartable
143
+
144
+ A worker can die mid-job; the lock expires and the step re-runs. So a processor has to be safe to
145
+ run twice — skip work already recorded, delete the rows a previous attempt created, or accept
146
+ regenerating over a baseline. Decide this per job and say so in a comment; there is no way to make
147
+ it automatic.
148
+
149
+ ## Recurring jobs
150
+
151
+ A recurring sweep is a SCHEDULE on a queue, run by a registered processor:
152
+
153
+ ```typescript
154
+ declareQueue(cfg, APP_MAINTENANCE, ['app:maintenance:reconcile'], {
155
+ worker: { concurrency: 1, lockDuration: 60_000 }
156
+ })
157
+ declareSchedule(cfg, {
158
+ id: 'app-nightly-reconcile', queue: APP_MAINTENANCE, name: 'app:maintenance:reconcile',
159
+ pattern: '17 3 * * *', tz: 'UTC', data: {},
160
+ })
161
+ ```
162
+
163
+ - **Declare it in the shared package, after its queue.** `declareSchedule` runs `assertSchedule`
164
+ first: the queue must be declared (`UnknownQueue`) and must accept the job name
165
+ (`UnknownJobName`), exactly one of `every` (a positive integer of milliseconds) and `pattern`
166
+ (cron), `tz` and `immediately` only with a pattern, never `immediately` with `startDate`, and
167
+ `opts.id` / `opts.delay` refused — the broker names and places every run
168
+ (`ScheduleMisdeclared`, message `<id>:<reason>`). Re-declaring an id replaces it.
169
+ - **The id is the identity.** Changing any other field updates the schedule in place; renaming the
170
+ id is one schedule removed and another created.
171
+ - **Every process that listens to the queue reconciles it at worker start** — the driver creates
172
+ and updates what is declared and removes the schedules it owns that are not. Two listeners of one
173
+ queue must therefore load the same schedule list, which is why it lives in the shared package.
174
+ A producer creates nothing.
175
+ - **A scheduled run is a processor job, never a guarded entrypoint.** Nobody's credentials travel
176
+ with it. `job.scheduled` carries the schedule id; it is absent on a job enqueued any other way.
177
+ - **It needs a consumer.** The broker produces the next run when the previous one starts
178
+ processing, so a schedule on a queue nothing consumes stops producing.
179
+ - **The processor obeys every rule above** — `touch()` through the sweep, and safe to run twice,
180
+ because a restart mid-sweep re-runs it from the start.
181
+
182
+ `schedulesOf(cfg, queue?)` reads the declarations back; `assertSchedules(cfg)` checks all of them,
183
+ including one written into `cfg.queue.schedules` directly, and refuses a duplicate id.
184
+ Which recurring work belongs on a schedule at all is `scheduled-jobs`.
185
+
186
+ ## A queued call is still a guarded call
187
+
188
+ `handleJob` rebuilds the request from the envelope and runs the entrypoint as the HTTP boundary
189
+ would: the declared guards are tried in order, the first that matches authorizes, the entity is
190
+ attached, and only then does the handler run. A queued entrypoint with guards therefore needs the
191
+ producer's credentials on the envelope's `headers`, and an unauthorized job answers
192
+ `AuthorizationError`. An alias this process does not serve answers `JobNotServed` — the job was
193
+ taken by a worker that has no handler for it.
194
+
195
+ Set `cfg.queue.envelopeTtl` (seconds) to bound how long a job envelope stays acceptable. The age is
196
+ measured from the envelope's `enqueuedAt` — when the PRODUCER enqueued it, not when a worker picked
197
+ it up — so a captured envelope stops being replayable, and so does a job that sat behind a backlog
198
+ longer than the window: it answers `EnvelopeExpired` at pickup rather than running late. Size the
199
+ window against the longest backlog you are willing to let run, and leave `envelopeTtl` unset — the
200
+ default — for a queue whose jobs may legitimately wait.
201
+
202
+ ## Errors cross the hop as classes
203
+
204
+ A refusal the caller must recognise — `ContentRefused`, `AgentLocked` — is marshalled into the
205
+ reply and rebuilt on the producer side as its own class, so `catch (e) { if (e instanceof ...) }`
206
+ keeps working across the broker. Only broker- and connection-level faults propagate as throws,
207
+ because those are the only ones a retry can fix.
208
+
209
+ ## What runs where
210
+
211
+ `queueWorkerMiddleware()` starts the worker at the **Ready** stage — processors register while the
212
+ application wires up, and binding earlier would take jobs the process cannot yet run. A process
213
+ that listens to nothing registers no worker and pays nothing.
214
+
215
+ `canServeModule` in `@owlmeans/server-api/utils` excludes QUEUE routes, the same way it excludes
216
+ sockets: the worker takes those jobs off the broker, and mounting them on the HTTP server too would
217
+ answer every call twice. `servedJobs` is the mirror image — the entrypoints this process both serves and
218
+ listens to, grouped by queue, which is what a driver binds.
219
+
220
+ ## Key exports
221
+
222
+ | Export | Description |
223
+ |--------|-------------|
224
+ | `enqueueProtocol` / `waitForProtocol` | Typed protocol-object enqueue and result wait; validates protocol kind and queue configuration. |
225
+ | `queueJobOf(request)` / `QueueJobMeta` | Explicit broker identity boundary for a protocol handler, including `touch()` for long work. |
226
+ | `declareQueue` / `listenQueues` | Configuration — what exists, and what this process consumes |
227
+ | `declareSchedule` / `schedulesOf` / `assertSchedule` / `assertSchedules` | Recurring jobs — declare, read back, check |
228
+ | `queueOf` / `queueOfJob` / `isListening` | Reading it back; `queueOf` throws `UnknownQueue` |
229
+ | `QueueConfig` / `QueueDeclaration` / `QueueWorkerOptions` / `JobOptions` / `ScheduleDeclaration` | The configuration shapes |
230
+ | `QueueResource<D, R>` | A queue addressed as a resource, plus `wait` / `flow` / `counts` / `close` |
231
+ | `JobRecord<D, R>` / `JobState` / `isSettled` / `JobEvent` / `JobEventType` | The record shape and lifecycle |
232
+ | `FlowSpec<D>` | A graph node — `name`, optional `queue`, `data`, `opts`, `children` |
233
+ | `QueueWorkerService` | `process(queue, name, fn)`, `start`, `stop`, `listening`, `hooks` |
234
+ | `JobContext<D>` | What a processor gets: `id`, `name`, `queue`, `attempt`, `data`, `signal`, `scheduled?`, `touch`, `progress`, `children`, `failedChildren` |
235
+ | `QueueHooks` | `wrapHandler`, `onJobResult`, `onJobStalled`, `onJobDead` — the compensation seam |
236
+ | `QueueAppend` | The `ctx.jobs(queue?)` mixin a driver installs; `QueueDriver` is what a driver supplies |
237
+ | `appendQueueTransport` / `makeQueueTransport` | Binds the QUEUE protocol so `call()` routes through the broker |
238
+ | `queueWorkerMiddleware` | Starts the worker at Ready stage |
239
+ | `handleJob` / `servedJobs` / `entrypointProcessor` | The bridge a driver dispatches through |
240
+ | `requestOf` / `assertFresh` / `JobEnvelope` / `JobReply` | The envelope, and rebuilding a request from it |
241
+ | Errors | `QueueError`, `QueueTimeout`, `UnknownJob`, `UnknownJobName`, `UnknownQueue`, `QueueNotListening`, `JobNotServed`, `EnvelopeExpired`, `ScheduleMisdeclared` |
242
+ | Constants | `DEFAULT_ALIAS` (`queue`), `DEFAULT_JOB_TIMEOUT` (60 000 ms), `DEFAULT_ATTEMPTS` (1) |
243
+
29
244
  ## Depends On
30
245
 
31
- - `@owlmeans/context` — service registration
246
+ - `@owlmeans/context` — service registration and the middleware stage
247
+ - `@owlmeans/entrypoint` — the transport seam and the served-entrypoint dispatch
248
+ - `@owlmeans/route` — the generic declaration model that `QUEUE_PROTOCOL` and `job()` extend
249
+ - `@owlmeans/resource` — the `Resource` contract jobs are read through
250
+ - `@owlmeans/auth` / `@owlmeans/auth-common` — guarding a queued call and attaching its entity
251
+ - `@owlmeans/error` — marshalling a refusal across the hop
252
+
253
+ ## Related
254
+
255
+ - `redis-queue` — the BullMQ driver; how schedules are reconciled; integration tests live there
256
+ - `scheduled-jobs` — when recurring work is a schedule, and how to write the sweep it runs
257
+ - `server-job` / `client-job` — exposing a queue's jobs to an application's UI (list, cancel,
258
+ progress over a socket), without touching the contracts here
259
+ - `entrypoint` — protocol declarations, bindings, and the transport lookup
260
+ - `resource` — the criteria and paging semantics `list` follows