@owlmeans/queue 0.1.18-rc.4 → 0.1.18-rc.41

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (110) hide show
  1. package/README.md +286 -8
  2. package/agent-meta/manifest.json +9 -2
  3. package/agent-meta/skills/queue/SKILL.md +243 -12
  4. package/agent-meta/skills/scheduled-jobs/SKILL.md +151 -0
  5. package/build/bridge/types.d.ts +25 -0
  6. package/build/bridge/types.d.ts.map +1 -0
  7. package/build/bridge/types.js +2 -0
  8. package/build/bridge/types.js.map +1 -0
  9. package/build/bridge.d.ts +7 -0
  10. package/build/bridge.d.ts.map +1 -0
  11. package/build/bridge.js +37 -0
  12. package/build/bridge.js.map +1 -0
  13. package/build/config.d.ts +34 -0
  14. package/build/config.d.ts.map +1 -0
  15. package/build/config.js +59 -0
  16. package/build/config.js.map +1 -0
  17. package/build/consts.d.ts +33 -0
  18. package/build/consts.d.ts.map +1 -0
  19. package/build/consts.js +35 -0
  20. package/build/consts.js.map +1 -0
  21. package/build/entrypoint-jobs/types.d.ts +29 -0
  22. package/build/entrypoint-jobs/types.d.ts.map +1 -0
  23. package/build/entrypoint-jobs/types.js +2 -0
  24. package/build/entrypoint-jobs/types.js.map +1 -0
  25. package/build/entrypoint-jobs.d.ts +5 -0
  26. package/build/entrypoint-jobs.d.ts.map +1 -0
  27. package/build/entrypoint-jobs.js +84 -0
  28. package/build/entrypoint-jobs.js.map +1 -0
  29. package/build/errors.d.ts +68 -0
  30. package/build/errors.d.ts.map +1 -0
  31. package/build/errors.js +103 -0
  32. package/build/errors.js.map +1 -0
  33. package/build/index.d.ts +18 -0
  34. package/build/index.d.ts.map +1 -1
  35. package/build/index.js +13 -1
  36. package/build/index.js.map +1 -1
  37. package/build/protocol/types.d.ts +12 -0
  38. package/build/protocol/types.d.ts.map +1 -0
  39. package/build/protocol/types.js +2 -0
  40. package/build/protocol/types.js.map +1 -0
  41. package/build/protocol.d.ts +13 -0
  42. package/build/protocol.d.ts.map +1 -0
  43. package/build/protocol.js +63 -0
  44. package/build/protocol.js.map +1 -0
  45. package/build/queue-config/types.d.ts +29 -0
  46. package/build/queue-config/types.d.ts.map +1 -0
  47. package/build/queue-config/types.js +2 -0
  48. package/build/queue-config/types.js.map +1 -0
  49. package/build/queue-config.d.ts +9 -0
  50. package/build/queue-config.d.ts.map +1 -0
  51. package/build/queue-config.js +83 -0
  52. package/build/queue-config.js.map +1 -0
  53. package/build/queue-route/types.d.ts +14 -0
  54. package/build/queue-route/types.d.ts.map +1 -0
  55. package/build/queue-route/types.js +2 -0
  56. package/build/queue-route/types.js.map +1 -0
  57. package/build/queue-route.d.ts +4 -0
  58. package/build/queue-route.d.ts.map +1 -0
  59. package/build/queue-route.js +24 -0
  60. package/build/queue-route.js.map +1 -0
  61. package/build/route.d.ts +9 -0
  62. package/build/route.d.ts.map +1 -0
  63. package/build/route.js +16 -0
  64. package/build/route.js.map +1 -0
  65. package/build/transport.d.ts +15 -0
  66. package/build/transport.d.ts.map +1 -0
  67. package/build/transport.js +59 -0
  68. package/build/transport.js.map +1 -0
  69. package/build/types.d.ts +312 -0
  70. package/build/types.d.ts.map +1 -0
  71. package/build/types.js +2 -0
  72. package/build/types.js.map +1 -0
  73. package/build/types.local.d.ts +4 -0
  74. package/build/types.local.d.ts.map +1 -0
  75. package/build/types.local.js +2 -0
  76. package/build/types.local.js.map +1 -0
  77. package/build/utils.d.ts +4 -0
  78. package/build/utils.d.ts.map +1 -0
  79. package/build/utils.js +4 -0
  80. package/build/utils.js.map +1 -0
  81. package/build/worker.d.ts +10 -0
  82. package/build/worker.d.ts.map +1 -0
  83. package/build/worker.js +24 -0
  84. package/build/worker.js.map +1 -0
  85. package/package.json +13 -3
  86. package/src/bridge/types.ts +25 -0
  87. package/src/bridge.ts +46 -0
  88. package/src/config.ts +80 -0
  89. package/src/consts.ts +38 -0
  90. package/src/entrypoint-jobs/types.ts +29 -0
  91. package/src/entrypoint-jobs.ts +105 -0
  92. package/src/errors.ts +121 -0
  93. package/src/index.ts +18 -0
  94. package/src/protocol/types.ts +16 -0
  95. package/src/protocol.ts +95 -0
  96. package/src/queue-config/types.ts +29 -0
  97. package/src/queue-config.ts +102 -0
  98. package/src/queue-route/types.ts +14 -0
  99. package/src/queue-route.ts +33 -0
  100. package/src/route.ts +23 -0
  101. package/src/transport.ts +77 -0
  102. package/src/types.ts +326 -0
  103. package/src/utils.ts +5 -0
  104. package/src/worker.ts +27 -0
  105. package/tests/bridge.spec.ts +64 -0
  106. package/tests/config.spec.ts +67 -0
  107. package/tests/protocol.spec.ts +58 -0
  108. package/tests/schedule.spec.ts +99 -0
  109. package/tsconfig.json +1 -0
  110. package/build/.gitkeep +0 -0
package/README.md CHANGED
@@ -1,20 +1,298 @@
1
1
  # @owlmeans/queue
2
2
 
3
- Reserved namespace for queue management. Currently a stub with no exports.
3
+ Job queues as resources, plus the route protocol that makes a queued call look like any other call.
4
+ Use it when work must outlive the request. Three cases qualify: the user would otherwise wait more
5
+ than about five seconds, accepted work must survive a process restart, or the work is retried
6
+ against a third party (payment capture, provider sync, bulk mail, report generation, model
7
+ pipelines). A request that finishes in under about two seconds and makes no external call should
8
+ not queue: do it inline. Check for a smaller fix first, such as an index, a batch write or a cached
9
+ aggregate. Recurring work is a declared schedule: the broker produces a job on an interval or a
10
+ cron pattern and a registered processor runs it.
4
11
 
5
- ## Overview
6
-
7
- This package is a placeholder for future queue/task processing functionality within the OwlMeans framework.
12
+ This package is contracts only and carries no broker code. A driver implements them
13
+ (`@owlmeans/redis-queue` does, over BullMQ). Depend on this package from a shared contract package,
14
+ and on the driver only where the application wires itself up.
8
15
 
9
16
  ## Installation
10
17
 
11
18
  ```bash
12
- bun add @owlmeans/queue
19
+ bun add @owlmeans/queue@^0.1.18-rc.41
20
+ ```
21
+
22
+ ## Concepts
23
+
24
+ - **Queue declaration**: `declareQueue(cfg, name, jobs, opts?)` in the shared backend config. It
25
+ names a queue, the job names it accepts, and its worker and default job options. It is an
26
+ address, not a deployment.
27
+ - **Listen**: `listenQueues(cfg, ...names)` in one process's own config says which queues *that
28
+ process* consumes. A process that listens to nothing is a producer only.
29
+ - **Queued protocol**: an immutable backend entrypoint protocol whose route uses `job()` from this
30
+ package (`QUEUE_PROTOCOL`). Callers use `ctx.entrypoint(protocol).call(...)` unchanged,
31
+ and the QUEUE transport carries the call through the broker.
32
+ - **Job record**: `ctx.jobs(queue)` returns a `QueueResource`, the ordinary `Resource<JobRecord>`
33
+ contract plus `wait`, `flow`, `counts` and `close`. `create` enqueues, `list` inspects and `take`
34
+ cancels.
35
+ - **Processor**: a function registered with `QueueWorkerService.process(queue, name, fn)` that gets
36
+ a `JobContext`. It must be safe to run twice and must `touch()` in long loops.
37
+ - **Single-flight id**: a `JobOptions.id` derived from what the job is about (`develop:<storyId>`).
38
+ A duplicate enqueue then returns the existing job instead of a second one.
39
+ - **Schedule**: `declareSchedule(cfg, { id, queue, name, every | pattern, … })` in the shared config.
40
+ Every process that listens to the queue reconciles the broker with the declared schedules when its
41
+ worker starts; each run is an ordinary job, and `JobContext.scheduled` names the schedule.
42
+
43
+ ## Usage
44
+
45
+ ### Declare queues in the shared backend config
46
+
47
+ ```ts
48
+ import { declareQueue } from '@owlmeans/queue'
49
+
50
+ // Split lanes by how long work takes, not by topic: a one-second read queued behind
51
+ // a thirty-minute job is a UI that looks broken.
52
+ declareQueue(cfg, APP_OPS, [appProtocols.files.get.alias, appProtocols.report.build.alias], {
53
+ worker: { concurrency: 16, lockDuration: 30_000 }
54
+ })
55
+
56
+ declareQueue(cfg, APP_BILLING, [appProtocols.invoice.capture.alias], {
57
+ worker: { concurrency: 4, lockDuration: 30_000 },
58
+ defaults: { attempts: 3, backoff: { type: 'exponential', delay: 250 } }
59
+ })
60
+ ```
61
+
62
+ The worker process names what it consumes, in its own config:
63
+
64
+ ```ts
65
+ import { listenQueues } from '@owlmeans/queue'
66
+
67
+ listenQueues(cfg, APP_OPS, APP_BILLING) // the api process omits this and only produces
68
+ ```
69
+
70
+ Both processes build their context through the same backend factory, which wires the driver once:
71
+
72
+ ```ts
73
+ import { appendRedisQueue } from '@owlmeans/redis-queue'
74
+
75
+ appendRedis<C, T>(context)
76
+ appendRedisQueue(context, { hooks: billingQueueHooks(context) })
77
+ ```
78
+
79
+ ### Declare a queued entrypoint
80
+
81
+ ```ts
82
+ import { contract, protocol, typed } from '@owlmeans/entrypoint'
83
+ import { job } from '@owlmeans/queue'
84
+ import { backend, route } from '@owlmeans/route'
85
+
86
+ const aliases = { base: 'app:report', build: 'app:report:build' } as const
87
+ const reportBase = protocol(route(aliases.base, '/reports', backend({ service: APP_WORKER })), contract())
88
+
89
+ export const appProtocols = {
90
+ report: {
91
+ base: reportBase,
92
+ build: protocol(
93
+ route(aliases.build, '/:id/build',
94
+ job({ parent: reportBase, service: APP_WORKER, queue: APP_OPS, timeout: 30_000 })),
95
+ contract.request({ params: typed<ReportParams>(ReportParamsSchema) }, typed<ReportResult>()),
96
+ ),
97
+ },
98
+ } as const
99
+ ```
100
+
101
+ The worker process serves it like any backend entrypoint; the HTTP server skips QUEUE routes. A
102
+ producer calls it with no knowledge of the broker:
103
+
104
+ ```ts
105
+ const result = await context.entrypoint(appProtocols.report.build).call({ params: { id } })
106
+ ```
107
+
108
+ Add `reply: false` to the `job()` options to resolve `Accepted` with `{ id, queue }` as soon as the
109
+ broker has taken the job.
110
+
111
+ ### Enqueue with broker options and wait
112
+
113
+ When a producer needs a job id, a delay or retries, pass the same protocol object to the typed
114
+ helpers. Never pass its alias.
115
+
116
+ ```ts
117
+ import { queueProtocolOf } from '@owlmeans/queue'
118
+
119
+ const queue = queueProtocolOf(context)
120
+ const queued = await queue.enqueue(appProtocols.invoice.capture, {
121
+ body: { invoiceId },
122
+ }, {
123
+ id: `capture:${invoiceId}:${attemptSequence}`, // derived from what the job is about
124
+ delay: 250, attempts: 3, backoff: { type: 'exponential', delay: 250 },
125
+ })
126
+
127
+ const receipt = await queue.waitFor(appProtocols.invoice.capture, queued, { timeout: 120_000 })
128
+ ```
129
+
130
+ `queueBridgeHelper.queueJobOf(req)` gives a protocol handler `{ id, name, queue, attempt, touch }` when it must compare
131
+ the broker job with a claim it persisted.
132
+
133
+ ### Processors, hooks and job graphs
134
+
135
+ ```ts
136
+ import { DEFAULT_ALIAS } from '@owlmeans/queue'
137
+ import type { QueueWorkerService } from '@owlmeans/queue'
138
+
139
+ const worker = context.service<QueueWorkerService>(DEFAULT_ALIAS)
140
+
141
+ worker.process(APP_OPS, 'app:import:rows', async job => {
142
+ for (const chunk of chunks(job.data.fileId)) {
143
+ await job.touch() // renew the lock on every iteration
144
+ await importChunk(chunk) // idempotent: skips rows already imported
145
+ }
146
+ return { ok: true }
147
+ })
148
+
149
+ worker.hooks({
150
+ onJobDead: async (record, reason) => { // retries exhausted: compensate
151
+ await imports.patch({ id: record.data.importId, status: 'failed', error: reason })
152
+ }
153
+ })
154
+
155
+ // children complete before the parent; siblings run in parallel
156
+ await context.jobs(APP_OPS).flow({
157
+ name: 'app:import:finalize', data: ref, children: [
158
+ { name: 'app:import:validate', data: ref, children: [{ name: 'app:import:rows', data: ref }] }
159
+ ]
160
+ })
161
+ ```
162
+
163
+ ### Recurring jobs
164
+
165
+ ```ts
166
+ import { declareQueue, declareSchedule } from '@owlmeans/queue'
167
+
168
+ // Shared config: the queue first, then its schedules.
169
+ declareQueue(cfg, APP_MAINTENANCE, ['app:maintenance:reconcile', 'app:maintenance:purge'], {
170
+ worker: { concurrency: 1, lockDuration: 60_000 }
171
+ })
172
+ declareSchedule(cfg, {
173
+ id: 'app-nightly-reconcile', queue: APP_MAINTENANCE, name: 'app:maintenance:reconcile',
174
+ pattern: '17 3 * * *', tz: 'UTC', data: {},
175
+ })
176
+ declareSchedule(cfg, {
177
+ id: 'app-purge', queue: APP_MAINTENANCE, name: 'app:maintenance:purge', every: 15 * 60_000,
178
+ })
179
+ ```
180
+
181
+ The process that `listenQueues(cfg, APP_MAINTENANCE)` registers the processors; nothing else is
182
+ wired. A scheduled run carries no caller and no credentials, so it goes to a processor, never to a
183
+ guarded entrypoint:
184
+
185
+ ```ts
186
+ worker.process(APP_MAINTENANCE, 'app:maintenance:reconcile', async job => {
187
+ for await (const page of organizations()) {
188
+ await job.touch()
189
+ await reconcile(page) // safe to run twice: a restart re-runs the whole sweep
190
+ }
191
+ return { scheduled: job.scheduled } // 'app-nightly-reconcile'
192
+ })
13
193
  ```
14
194
 
15
- ## Related Packages
195
+ Renaming a schedule id removes the old scheduler and creates a new one; deleting a declaration
196
+ removes its scheduler at the next worker start.
197
+
198
+ ### Jobs are records
199
+
200
+ ```ts
201
+ import { JobState } from '@owlmeans/queue'
202
+
203
+ const jobs = context.jobs<ImportInput, ImportResult>(APP_OPS)
204
+
205
+ const failed = await jobs.list({ state: JobState.Failed }, { size: 20 })
206
+ const counts = await jobs.counts()
207
+ await jobs.take(jobId) // cancel and return
208
+ ```
209
+
210
+ ## Processor rules
211
+
212
+ - **`touch()` in every long loop.** The broker judges liveness by the lock. Silence longer than
213
+ `lockDuration` looks like a dead worker, so the job is re-run elsewhere while this one is still
214
+ going.
215
+ - **Children report, parents decide.** A child returns `{ ok: false, error }` for a domain failure.
216
+ Throwing means "infrastructure broke, retry me". Parents read `children()` / `failedChildren()`.
217
+ - **Retries default to one attempt** (`DEFAULT_ATTEMPTS`), because a blind retry of model work
218
+ re-spends the tokens that just failed. Raise `attempts` per queue for cheap idempotent work.
219
+ - **`job.signal` is not an interrupt.** An orderly stop closes workers first and waits for jobs in
220
+ flight to return.
221
+
222
+ ## Guards and errors
223
+
224
+ `entrypointJobsOf(ctx).handle` rebuilds the request from the envelope and runs the entrypoint's guards the way the
225
+ HTTP boundary does, attaching the entity. A guarded queued entrypoint therefore needs the
226
+ producer's credentials in the envelope `headers`. `cfg.queue.envelopeTtl` (seconds) bounds how long
227
+ after `enqueuedAt` an envelope is still accepted; past it, the job answers `EnvelopeExpired`.
228
+
229
+ A refusal the caller must recognise is carried in the reply and rebuilt on the producing side as its
230
+ own class, so `instanceof` keeps working across the broker. `QueueTimeout` means the WAIT ended,
231
+ not the job, so read the job back to learn what became of it.
232
+
233
+ ## API
234
+
235
+ | Symbol | Kind | Purpose |
236
+ |---|---|---|
237
+ | `declareQueue(cfg, name, jobs, opts?)` | function | Declare a queue and the job names it accepts; re-declaring replaces |
238
+ | `listenQueues(cfg, ...names)` | function | Name the queues this process consumes |
239
+ | `queueConfigOf(cfg)` — `.queueOf(name)`, `.queueOfJob(job)`, `.isListening(name)` | helper | Read declarations back; `queueOf` throws `UnknownQueue` |
240
+ | `declareSchedule(cfg, schedule)` | function | Declare a recurring job after checking it; re-declaring an id replaces |
241
+ | `queueConfigOf(cfg).schedulesOf(queue?)` | helper | The declared schedules, of one queue when named |
242
+ | `queueConfigOf(cfg)` — `.assertSchedule(schedule)`, `.assertSchedules()` | helper | Check one schedule, or every declared one plus duplicate ids |
243
+ | `queueProtocolOf(ctx).enqueue(protocol, request, options?)` | helper | Enqueue a QUEUE protocol with `JobOptions`, typed by the protocol |
244
+ | `queueProtocolOf(ctx).waitFor(protocol, job, { timeout }?)` | helper | Wait for that job and unwrap the typed reply |
245
+ | `queueBridgeHelper.queueJobOf(request)` | helper | `QueueJobMeta` of the broker job behind a handler's request, or `null` |
246
+ | `appendQueueTransport(ctx, alias?)`, `makeQueueTransport(alias?)` | function | Register the QUEUE transport so `call()` routes through the broker |
247
+ | `queueWorkerMiddleware(alias?)` | function | Start the worker at the Ready stage, only in a process that listens |
248
+ | `entrypointJobsOf(ctx)` — `.handle(job)`, `.processor()`, `.served()` | helper | The bridge a driver dispatches entrypoint jobs through |
249
+ | `queueBridgeHelper` — `.requestOf(envelope, path)`, `.assertFresh(envelope, ttl?)` | helper | Rebuild a request from an envelope; enforce `envelopeTtl` |
250
+ | `QueueConfig`, `QueueDeclaration`, `QueueWorkerOptions`, `JobOptions`, `ScheduleDeclaration` | type | Configuration shapes |
251
+ | `QueueResource<D, R>` | type | A queue as a resource, plus `queue`, `wait`, `flow`, `counts`, `close` |
252
+ | `JobRecord<D, R>`, `JobEvent<R>`, `FlowSpec<D>` | type | A job as a record, a lifecycle event, a graph node |
253
+ | `QueueWorkerService` | type | `process`, `start`, `stop`, `listening`, `hooks` |
254
+ | `JobContext<D>`, `JobProcessor<D, R>` | type | What a processor receives (`scheduled` names the schedule of a scheduled run); the processor signature |
255
+ | `QueueHooks` | type | `wrapHandler`, `onJobResult`, `onJobStalled`, `onJobDead` |
256
+ | `QueueJobMeta`, `JobEnvelope`, `JobReply<T>` | type | Broker identity for a handler, the call envelope, the reply shape |
257
+ | `QueueAppend`, `QueueDriver` | type | The `ctx.jobs(queue?)` mixin a driver installs; what a driver supplies |
258
+ | `Config`, `Context`, `QueueTransportRequest`, `QueueTransportResponse` | type | Context, config and transport types |
259
+ | `JobState`, `JobEventType` | enum | Job lifecycle states and event types |
260
+ | `isSettled(state?)` | function | Whether a state is final |
261
+ | `DEFAULT_ALIAS`, `DEFAULT_JOB_TIMEOUT`, `DEFAULT_ATTEMPTS` | const | `'queue'`, `60_000` ms, `1` |
262
+ | `QueueError`, `QueueTimeout`, `UnknownJob`, `UnknownJobName`, `UnknownQueue`, `QueueNotListening`, `JobNotServed`, `EnvelopeExpired`, `ScheduleMisdeclared` | class | Error family |
263
+
264
+ ## Common pitfalls
265
+
266
+ - **Queueing work the request could finish.** It adds a second process to deploy and a place to fail.
267
+ - **Passing alias strings to `queueProtocolOf(ctx).enqueue`/`.waitFor`.** They take the protocol object and
268
+ reject strings and non-QUEUE declarations.
269
+ - **Enqueueing a job name the queue did not declare.** It throws `UnknownJobName` rather than
270
+ parking a job nothing can process.
271
+ - **Registering a processor in a process that does not `listen`.** It throws `QueueNotListening`.
272
+ - **Putting `listen` in the shared declaration.** Every deployment of that binary would then
273
+ consume everything.
274
+ - **Random job ids on an admission step.** Derive the id from the subject, or a retried request
275
+ enqueues twice.
276
+ - **Long loops without `touch()`**, or processors that are not safe to run twice.
277
+ - **Throwing from a child for a domain failure.** Return `{ ok: false, error }` instead.
278
+ - **Equal child ids across queues in one flow.** `children()` is keyed by bare id and collapses them.
279
+ - **A self-re-enqueueing job as a cron.** Declare a schedule. `delay` defers a single job once.
280
+ - **A schedule on a queue nothing consumes.** Only a listening worker creates the scheduler, and the
281
+ broker produces each next run when the previous one starts.
282
+ - **Loading different schedule lists in two processes that listen to one queue.** Each start removes
283
+ the schedules its own list does not name.
284
+ - **Declaring a schedule before its queue.** `declareSchedule` checks the queue and the job name and
285
+ throws `UnknownQueue` / `UnknownJobName`.
286
+ - **Calling `ctx.jobs()` with no name once a second queue exists.** It throws `UnknownQueue`.
287
+
288
+ ## Related packages
16
289
 
17
- - [`@owlmeans/redis-resource`](../redis-resource) — Redis Streams provide queue-like semantics today
290
+ - [`@owlmeans/redis-queue`](../redis-queue): the BullMQ driver, and where queue integration tests live
291
+ - [`@owlmeans/route`](../route): transport-neutral route declarations used by the queue-owned `job()` builder
292
+ - [`@owlmeans/entrypoint`](../entrypoint): protocol declarations and the transport seam
293
+ - [`@owlmeans/resource`](../resource): the `Resource` contract jobs are read through
294
+ - [`@owlmeans/server-job`](../server-job) / [`@owlmeans/client-job`](../client-job): listing,
295
+ cancelling and following jobs from an application's UI
18
296
 
19
297
  <!-- owlmeans:agent-guidance:start -->
20
298
  ## Agent guidance
@@ -24,7 +302,7 @@ This package ships embedded agent skills under `agent-meta/`. After installing y
24
302
  your project's skill store (`.agents/skills/`):
25
303
 
26
304
  ```sh
27
- npx @owlmeans/agent-skills
305
+ npx @owlmeans/agent-skills@^0.1.18-rc.49
28
306
  ```
29
307
 
30
308
  The embedded files are version-matched to this package release. Do not edit them
@@ -1,10 +1,17 @@
1
1
  {
2
2
  "schemaVersion": 2,
3
3
  "package": "@owlmeans/queue",
4
- "version": "0.1.18-rc.0",
5
- "generatedAt": "2026-08-16T22:20:50.517Z",
4
+ "version": "0.1.18-rc.41",
5
+ "generatedAt": "2026-10-05T21:03:06.369Z",
6
6
  "canonicalRepo": "https://github.com/owlmeans/common",
7
7
  "entries": [
8
+ {
9
+ "kind": "skill",
10
+ "name": "scheduled-jobs",
11
+ "category": "multi-package",
12
+ "file": "skills/scheduled-jobs/SKILL.md",
13
+ "canonicalPath": ".agents/skills/scheduled-jobs/SKILL.md"
14
+ },
8
15
  {
9
16
  "kind": "skill",
10
17
  "name": "queue",
@@ -1,6 +1,6 @@
1
1
  ---
2
2
  name: queue
3
- description: How to use @owlmeans/queue — abstract job-queue interface (currently a stub) to be implemented by transport-specific queue services. Auto-invoked when importing queue types.
3
+ description: How to use @owlmeans/queue — protocol-object enqueue/wait, job queues as resources, QUEUE declarations, producer/consumer split, single-flight job ids, recurring schedules, lifecycle hooks, and processor rules. Auto-invoked when importing queue types, declaring a job protocol, or declaring a schedule.
4
4
  user-invocable: false
5
5
  ---
6
6
  <!-- AUTO-GENERATED — do not edit. Regenerate via sync-agent-meta. -->
@@ -8,24 +8,255 @@ user-invocable: false
8
8
  # @owlmeans/queue
9
9
 
10
10
  **Layer:** Infra
11
- **Install:** `"@owlmeans/queue": "^0.1.18-rc.0"` in `dependencies`
11
+ **Install:** `"@owlmeans/queue": "^0.1.18-rc.41"` in `dependencies`
12
12
 
13
- ## Key Exports
13
+ Backend transport contracts only. It carries no broker code — a driver package
14
+ (`@owlmeans/redis-queue`) implements them. Depend on this one from a shared backend package;
15
+ generic common and frontend packages must not import it. Depend on the driver only where the
16
+ application wires itself up.
14
17
 
15
- This package is currently a stub that defines the queue interface. Concrete implementations (Redis BullMQ, RabbitMQ, etc.) will register a service that satisfies this interface.
18
+ ## The one idea
16
19
 
17
- | Export | Description |
18
- |--------|-------------|
19
- | Queue types (placeholder) | Abstract queue interface |
20
+ A queued call is an immutable entrypoint protocol whose route names `QUEUE_PROTOCOL`.
21
+ Ordinary calls use `ctx.entrypoint(protocol).call(request)`. Code that needs broker options or a
22
+ job handle uses `queueProtocolOf(ctx).enqueue(protocol, request, options)` and
23
+ `queueProtocolOf(ctx).waitFor(protocol, job)`. Both derive queue and job name from the protocol object and
24
+ preserve its exact request/reply pair; raw strings and non-queue declarations are rejected.
25
+
26
+ That is what makes it worth having: moving a service-to-service call onto the broker is a change
27
+ to a declaration, not to the places that call it, and the broker holds the work across a restart of
28
+ either side — which an in-process call cannot.
29
+
30
+ ## Declaring
31
+
32
+ Queues and the jobs they accept live in the SHARED backend package, so producers and consumers
33
+ read one list:
34
+
35
+ ```typescript
36
+ declareQueue(cfg, AGENT_WORK, [story.develop.alias], {
37
+ worker: { concurrency: 3, lockDuration: 60_000 }
38
+ })
39
+ ```
40
+
41
+ The job list is enforced, not documentation: enqueueing a name the queue does not declare throws
42
+ `UnknownJobName` instead of parking a job nothing can ever process. Re-declaring a name replaces,
43
+ so a helper that runs once per composed app is safe to call twice.
44
+
45
+ Which queues a process CONSUMES is separate, and lives in that process's own config:
46
+
47
+ ```typescript
48
+ listenQueues(cfg, AGENT_WORK, AGENT_OPS) // absent ⇒ producer only
49
+ ```
50
+
51
+ Keep them apart. A declaration is an address; `listen` is a deployment fact. If a worker bound
52
+ whatever it could serve, every deployment of the same binary would consume everything it imports.
53
+
54
+ A queued entrypoint is an immutable protocol with `job()` in place of `backend()`:
55
+
56
+ ```typescript
57
+ protocol(
58
+ route(agent.story.develop, '/:id/develop',
59
+ job({ parent: agent.story.base, service: AGENT, queue: AGENT_WORK, timeout: 30_000 })),
60
+ contract.request({ params: typed<StoryParams>(StoryParamsSchema) }, typed()),
61
+ )
62
+ ```
63
+
64
+ Import `job`, `QUEUE_PROTOCOL` and `queueRouteHelper` (`isQueueRoute`, `queueRouteOptions`) from
65
+ `@owlmeans/queue`.
66
+ The generic route package deliberately knows none of the queue option shape.
67
+
68
+ The alias appears in `declareQueue(...jobs)` only because the broker is a string-keyed adapter.
69
+ Application producers and handlers keep the protocol object.
70
+
71
+ When an admission record owns single flight, select the job id atomically first, then enqueue it:
72
+
73
+ ```typescript
74
+ const queue = queueProtocolOf(context)
75
+ const queued = await queue.enqueue(tokenBalance.reconcile, {
76
+ body: { projectionId },
77
+ }, { id: claimId, delay: 250, attempts: 3, backoff: { type: 'exponential', delay: 250 } })
78
+ const result = await queue.waitFor(tokenBalance.reconcile, queued)
79
+ ```
80
+
81
+ Per-call `id`, delay, attempts, backoff and retention belong here; putting them into an alias-based
82
+ helper forfeits protocol inference. `queueBridgeHelper.queueJobOf(request)` exposes `{ id, name,
83
+ queue, attempt }` to the rare protocol handler that must compare a broker job with its persisted claim.
84
+
85
+ `reply: false` makes it fire-and-forget: the call resolves `Accepted` with `{ id, queue }` as soon
86
+ as the BROKER has accepted the enqueue — before any worker has picked the job up, so the resolution
87
+ says the job exists and nothing about it having started. That is what a long pipeline wants; watch
88
+ the job by its id for anything further. The default is to wait for the value, for
89
+ `req.timeout ?? route.timeout ?? DEFAULT_JOB_TIMEOUT` (60 s).
90
+
91
+ ## Jobs are records
92
+
93
+ `ctx.jobs<D, R>(queue)` returns a `QueueResource<D, R>` — a `Resource<JobRecord<D, R>>` plus
94
+ pub/sub. So inspecting a queue is the resource contract you already know: `create` enqueues,
95
+ `get`/`load`/`list`/`count` inspect, `delete`/`take` cancel, `purge` bulk-cancels. Only what the
96
+ base contract cannot express is added: `wait(id, { timeout })`, `flow(root)`, `counts()`, and
97
+ `close()` for the connections a producer that waited holds.
98
+
99
+ Called with no queue name it answers the SOLE declared queue and throws `UnknownQueue` once there
100
+ are two — an application that grows a second queue is told to name one rather than quietly served
101
+ the wrong backlog. A queue nothing declared throws the same error at the call site.
20
102
 
21
- ## Usage
103
+ Give `JobOptions.id` a value derived from what the job is ABOUT (`develop:<storyId>`) and a
104
+ duplicate enqueue becomes a no-op returning the job already there — no `RecordExists`. That is
105
+ what makes an admission step safe to retry.
106
+
107
+ ## Writing a processor
22
108
 
23
109
  ```typescript
24
- // Reserve the alias for your queue implementation
25
- import type { Queue } from '@owlmeans/queue'
26
- const queue = ctx.service<Queue>('queue')
110
+ const worker = context.service<QueueWorkerService>(DEFAULT_ALIAS)
111
+
112
+ worker.process(AGENT_WORK, 'agent:story:code', async job => {
113
+ for (const screen of screens) {
114
+ await job.touch() // ← not optional
115
+ await develop(screen)
116
+ }
117
+ return { ok: true }
118
+ })
27
119
  ```
28
120
 
121
+ Registering against a queue this process does not `listen` to throws `QueueNotListening` — the
122
+ question is a deployment one, so it is answered at wiring time rather than when a job arrives.
123
+
124
+ Three rules, each of which has a failure mode attached:
125
+
126
+ - **`touch()` in every long loop.** The broker judges liveness by the lock. Silence for longer
127
+ than `lockDuration` is indistinguishable from a dead worker, so the job is handed to someone
128
+ else while this one is still working — you get two runs, not one. `progress(value)` reports
129
+ without renewing, and the value must be JSON-carryable.
130
+ - **Children report, parents decide.** A child returns `{ ok: false, error }` for a DOMAIN
131
+ failure; it does not throw. Throwing means "infrastructure broke, retry me". A child that
132
+ exhausts its attempts does not fail its parent: the parent runs and reads `children()` /
133
+ `failedChildren()` — both keyed by bare job id — and decides. Give the children of a flow that
134
+ spans queues distinct ids, because that re-keying collapses two ids that are equal across queues.
135
+ - **Retries default to one attempt** (`DEFAULT_ATTEMPTS`). These jobs mostly talk to a model, and
136
+ a blind retry re-spends the tokens that just failed. Raise `attempts` per queue only where jobs
137
+ are cheap and idempotent.
138
+
139
+ `job.signal` is the queue's abort signal, and it is NOT an interrupt. The driver stops by closing
140
+ every worker first — which waits for the jobs in flight to return — and aborts only afterwards, so
141
+ a processor cannot bail out on it and does not need to. An orderly stop drains; an outright kill is
142
+ covered by the rule below.
143
+
144
+ ## Being restartable
145
+
146
+ A worker can die mid-job; the lock expires and the step re-runs. So a processor has to be safe to
147
+ run twice — skip work already recorded, delete the rows a previous attempt created, or accept
148
+ regenerating over a baseline. Decide this per job and say so in a comment; there is no way to make
149
+ it automatic.
150
+
151
+ ## Recurring jobs
152
+
153
+ A recurring sweep is a SCHEDULE on a queue, run by a registered processor:
154
+
155
+ ```typescript
156
+ declareQueue(cfg, APP_MAINTENANCE, ['app:maintenance:reconcile'], {
157
+ worker: { concurrency: 1, lockDuration: 60_000 }
158
+ })
159
+ declareSchedule(cfg, {
160
+ id: 'app-nightly-reconcile', queue: APP_MAINTENANCE, name: 'app:maintenance:reconcile',
161
+ pattern: '17 3 * * *', tz: 'UTC', data: {},
162
+ })
163
+ ```
164
+
165
+ - **Declare it in the shared package, after its queue.** `declareSchedule` runs
166
+ `queueConfigOf(cfg).assertSchedule` first: the queue must be declared (`UnknownQueue`) and must
167
+ accept the job name (`UnknownJobName`), exactly one of `every` (a positive integer of
168
+ milliseconds) and `pattern` (cron), `tz` and `immediately` only with a pattern, never
169
+ `immediately` with `startDate`, and `opts.id` / `opts.delay` refused — the broker names and places every run
170
+ (`ScheduleMisdeclared`, message `<id>:<reason>`). Re-declaring an id replaces it.
171
+ - **The id is the identity.** Changing any other field updates the schedule in place; renaming the
172
+ id is one schedule removed and another created.
173
+ - **Every process that listens to the queue reconciles it at worker start** — the driver creates
174
+ and updates what is declared and removes the schedules it owns that are not. Two listeners of one
175
+ queue must therefore load the same schedule list, which is why it lives in the shared package.
176
+ A producer creates nothing.
177
+ - **A scheduled run is a processor job, never a guarded entrypoint.** Nobody's credentials travel
178
+ with it. `job.scheduled` carries the schedule id; it is absent on a job enqueued any other way.
179
+ - **It needs a consumer.** The broker produces the next run when the previous one starts
180
+ processing, so a schedule on a queue nothing consumes stops producing.
181
+ - **The processor obeys every rule above** — `touch()` through the sweep, and safe to run twice,
182
+ because a restart mid-sweep re-runs it from the start.
183
+
184
+ `queueConfigOf(cfg).schedulesOf(queue?)` reads the declarations back; `.assertSchedules()` checks
185
+ all of them, including one written into `cfg.queue.schedules` directly, and refuses a duplicate id.
186
+ Which recurring work belongs on a schedule at all is `scheduled-jobs`.
187
+
188
+ ## A queued call is still a guarded call
189
+
190
+ `entrypointJobsOf(ctx).handle(job)` rebuilds the request from the envelope and runs the entrypoint as the HTTP boundary
191
+ would: the declared guards are tried in order, the first that matches authorizes, the entity is
192
+ attached, and only then does the handler run. A queued entrypoint with guards therefore needs the
193
+ producer's credentials on the envelope's `headers`, and an unauthorized job answers
194
+ `AuthorizationError`. An alias this process does not serve answers `JobNotServed` — the job was
195
+ taken by a worker that has no handler for it.
196
+
197
+ Set `cfg.queue.envelopeTtl` (seconds) to bound how long a job envelope stays acceptable. The age is
198
+ measured from the envelope's `enqueuedAt` — when the PRODUCER enqueued it, not when a worker picked
199
+ it up — so a captured envelope stops being replayable, and so does a job that sat behind a backlog
200
+ longer than the window: it answers `EnvelopeExpired` at pickup rather than running late. Size the
201
+ window against the longest backlog you are willing to let run, and leave `envelopeTtl` unset — the
202
+ default — for a queue whose jobs may legitimately wait.
203
+
204
+ ## Errors cross the hop as classes
205
+
206
+ A refusal the caller must recognise — `ContentRefused`, `AgentLocked` — is marshalled into the
207
+ reply and rebuilt on the producer side as its own class, so `catch (e) { if (e instanceof ...) }`
208
+ keeps working across the broker. Only broker- and connection-level faults propagate as throws,
209
+ because those are the only ones a retry can fix.
210
+
211
+ ## What runs where
212
+
213
+ `queueWorkerMiddleware()` starts the worker at the **Ready** stage — processors register while the
214
+ application wires up, and binding earlier would take jobs the process cannot yet run. A process
215
+ that listens to nothing registers no worker and pays nothing.
216
+
217
+ `canServeModule` in `@owlmeans/server-api/utils` excludes QUEUE routes, the same way it excludes
218
+ sockets: the worker takes those jobs off the broker, and mounting them on the HTTP server too would
219
+ answer every call twice. `entrypointJobsOf(ctx).served()` is the mirror image — the entrypoints
220
+ this process both serves and listens to, grouped by queue, which is what a driver binds.
221
+
222
+ ## Key exports
223
+
224
+ | Export | Description |
225
+ |--------|-------------|
226
+ | `queueProtocolOf(ctx)` — `enqueue` / `waitFor` | Typed protocol-object enqueue and result wait; validates protocol kind and queue configuration. |
227
+ | `queueBridgeHelper.queueJobOf(request)` / `QueueJobMeta` | Explicit broker identity boundary for a protocol handler, including `touch()` for long work. |
228
+ | `declareQueue` / `listenQueues` | Configuration — what exists, and what this process consumes |
229
+ | `declareSchedule`; `queueConfigOf(cfg)` — `schedulesOf` / `assertSchedule` / `assertSchedules` | Recurring jobs — declare, read back, check |
230
+ | `queueConfigOf(cfg)` — `queueOf` / `queueOfJob` / `isListening` | Reading it back; `queueOf` throws `UnknownQueue` |
231
+ | `QueueConfig` / `QueueDeclaration` / `QueueWorkerOptions` / `JobOptions` / `ScheduleDeclaration` | The configuration shapes |
232
+ | `QueueResource<D, R>` | A queue addressed as a resource, plus `wait` / `flow` / `counts` / `close` |
233
+ | `JobRecord<D, R>` / `JobState` / `isSettled` / `JobEvent` / `JobEventType` | The record shape and lifecycle |
234
+ | `FlowSpec<D>` | A graph node — `name`, optional `queue`, `data`, `opts`, `children` |
235
+ | `QueueWorkerService` | `process(queue, name, fn)`, `start`, `stop`, `listening`, `hooks` |
236
+ | `JobContext<D>` | What a processor gets: `id`, `name`, `queue`, `attempt`, `data`, `signal`, `scheduled?`, `touch`, `progress`, `children`, `failedChildren` |
237
+ | `QueueHooks` | `wrapHandler`, `onJobResult`, `onJobStalled`, `onJobDead` — the compensation seam |
238
+ | `QueueAppend` | The `ctx.jobs(queue?)` mixin a driver installs; `QueueDriver` is what a driver supplies |
239
+ | `appendQueueTransport` / `makeQueueTransport` | Binds the QUEUE protocol so `call()` routes through the broker |
240
+ | `queueWorkerMiddleware` | Starts the worker at Ready stage |
241
+ | `entrypointJobsOf(ctx)` — `handle` / `served` / `processor` | The bridge a driver dispatches through |
242
+ | `queueBridgeHelper` — `requestOf` / `assertFresh`; `JobEnvelope` / `JobReply` | The envelope, and rebuilding a request from it |
243
+ | Errors | `QueueError`, `QueueTimeout`, `UnknownJob`, `UnknownJobName`, `UnknownQueue`, `QueueNotListening`, `JobNotServed`, `EnvelopeExpired`, `ScheduleMisdeclared` |
244
+ | Constants | `DEFAULT_ALIAS` (`queue`), `DEFAULT_JOB_TIMEOUT` (60 000 ms), `DEFAULT_ATTEMPTS` (1) |
245
+
29
246
  ## Depends On
30
247
 
31
- - `@owlmeans/context` — service registration
248
+ - `@owlmeans/context` — service registration and the middleware stage
249
+ - `@owlmeans/entrypoint` — the transport seam and the served-entrypoint dispatch
250
+ - `@owlmeans/route` — the generic declaration model that `QUEUE_PROTOCOL` and `job()` extend
251
+ - `@owlmeans/resource` — the `Resource` contract jobs are read through
252
+ - `@owlmeans/auth` / `@owlmeans/auth-common` — guarding a queued call and attaching its entity
253
+ - `@owlmeans/error` — marshalling a refusal across the hop
254
+
255
+ ## Related
256
+
257
+ - `redis-queue` — the BullMQ driver; how schedules are reconciled; integration tests live there
258
+ - `scheduled-jobs` — when recurring work is a schedule, and how to write the sweep it runs
259
+ - `server-job` / `client-job` — exposing a queue's jobs to an application's UI (list, cancel,
260
+ progress over a socket), without touching the contracts here
261
+ - `entrypoint` — protocol declarations, bindings, and the transport lookup
262
+ - `resource` — the criteria and paging semantics `list` follows