comq 0.17.1 → 0.18.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,1120 @@
1
+ # Fix comq's poison-message handling
2
+
3
+ > **The design record for how comq handles a message its consumer could not process.**
4
+ > Reviewed from the consumer side in
5
+ > [#272](https://github.com/toa-io/comq/discussions/272) and revised there: the retry queue
6
+ > became shared and delay-named, the delay became configurable, and message provenance moved to
7
+ > where it survives a retry.
8
+ >
9
+ > **Parts 1 and 2 are implemented.** Part 3 — the `Retry` / `Park` verdict API — is decided but
10
+ > not yet scheduled; see the end. Text below is written as a proposal because that is what it
11
+ > was; it describes what the code now does, except where Part 3 says otherwise.
12
+
13
+ When a consumer callback rejects, comq is supposed to retry the message a few times and then
14
+ give up on it. What it actually does is kill the process, and take every other consumer in that
15
+ process with it. Six verified defects, all in [source/channel.js](../source/channel.js) — full
16
+ detail in **Context** at the end.
17
+
18
+ One of the six — the last attempt silently deleting the message — needs a mechanism comq does
19
+ not have today, and the obvious candidate is not one. That comes first, because everything else
20
+ is routine by comparison.
21
+
22
+ - **Part 1** — where a message goes when comq gives up on it. comq parks it in a queue it owns.
23
+ Includes why RabbitMQ's dead-lettering is *not* the mechanism for this, and the one
24
+ compatibility trap to avoid.
25
+ - **Part 2** — the other five defects. No upgrade impact.
26
+ - **Part 3** — the proposed `Retry` / `Park` verdict API, and what it does and does not change
27
+ here (short answer: it adds one branch to Part 2, and one naming decision to Part 1).
28
+
29
+ ---
30
+
31
+ # Part 1 — Where a message goes when comq gives up on it
32
+
33
+ ## The problem
34
+
35
+ The last failed attempt destroys the message. `#discard` ([:406](../source/channel.js#L406)) is:
36
+
37
+ ```js
38
+ this.#channel.nack(message, false, false)
39
+ ```
40
+
41
+ `requeue: false` means "the broker may dead-letter this" — but only if the queue was declared
42
+ with a dead-letter exchange, and **comq never declares one**. `grep` for `x-dead-letter` or
43
+ `arguments:` across `source/` returns nothing; `#assertQueue`
44
+ ([:246-253](../source/channel.js#L246)) passes only `{ durable: true }` or `{ exclusive: true }`.
45
+ So the message is silently deleted. [readme.md:468](../readme.md#L468) tells the reader to
46
+ configure a policy on the broker by hand — a library whose documented failure mode is silent
47
+ deletion.
48
+
49
+ ## The mechanism: comq parks it in a queue it owns
50
+
51
+ On the last attempt, comq publishes the message to `comq.parked.<queue>`, waits for the publisher
52
+ confirm, and only then acks the original.
53
+
54
+ **This is about retention, not delivery.** The message is not going to be processed again — that
55
+ is settled. What is at stake is whether it is still *there* afterwards. A message that has
56
+ exhausted its attempts is evidence that something is wrong: a bad producer, a schema change, a
57
+ bug in the consumer. Today comq deletes that evidence. The requirement is that when comq says it
58
+ has kept the message, that is true and not merely likely.
59
+
60
+ Publishing before acking is what makes it true: the broker holds the parked copy before it
61
+ releases the original, so there is no window in which it holds neither. If comq dies between the
62
+ two, the original is redelivered and parked again — two copies of the same evidence in a queue
63
+ nothing consumes, which is harmless. (Consumers do need to be idempotent, but because of the
64
+ retry path and redelivery generally, not because of this.)
65
+
66
+ > ### Two mechanisms, easily confused
67
+ >
68
+ > | | **Part 2 — retry** | **Part 1 — parking** |
69
+ > |---|---|---|
70
+ > | when | attempts 1–5 | after the 6th failure |
71
+ > | queue | the shared retry queue | `comq.parked.<queue>` |
72
+ > | how it gets there | comq publishes it | comq publishes it |
73
+ > | what happens next | the broker returns it to the source queue after the TTL | nothing — it waits for a human |
74
+ > | purpose | try again later | keep the evidence |
75
+ >
76
+ > Both are new queues comq declares, and **dead-lettering appears in both doing different
77
+ > jobs**: in Part 2 it is the routing trick that carries a message *back* to its source when the
78
+ > TTL fires; in Part 1 it is the rejected way of moving a message to its final resting place.
79
+
80
+ ## Why RabbitMQ's dead-lettering is not the mechanism
81
+
82
+ The obvious alternative is to declare `x-dead-letter-exchange` on the source queue and let the
83
+ broker move the message. It was considered and rejected, for a reason worth recording because
84
+ the readme currently recommends exactly this to users.
85
+
86
+ **Dead-lettering is a routing rule, not a retention guarantee.** It is what the broker does with
87
+ messages it has already given up on, and best-effort is what it was designed to be. RabbitMQ
88
+ republishes to the DLX *internally, without publisher confirms*: a dead-lettered message can be
89
+ lost if the target queue is unavailable, if its length limit is reached under `reject-publish`
90
+ overflow, or across a network partition. If the dead-letter exchange does not exist, messages
91
+ are silently dropped.
92
+
93
+ At-least-once dead-lettering does exist, but requires **quorum queues** plus
94
+ `dead-letter-strategy: at-least-once` plus `overflow: reject-publish`. **Classic queues do not
95
+ support it**, and comq declares classic queues — `#assertQueue` passes `{ durable: true }` with
96
+ no `x-queue-type`.
97
+
98
+ So this was never a choice between two implementations of one thing. We need the message kept;
99
+ dead-lettering does not promise to keep it. Using it would reintroduce silent loss on the exact
100
+ path built to prevent it — narrower than today, but the same class of bug.
101
+
102
+ - [Dead Letter Exchanges](https://www.rabbitmq.com/docs/dlx)
103
+ - [At-Least-Once Dead Lettering](https://www.rabbitmq.com/blog/2022/03/29/at-least-once-dead-lettering)
104
+ - [Quorum Queues](https://www.rabbitmq.com/docs/quorum-queues)
105
+
106
+ > **If comq ever moves to quorum queues** — with `dead-letter-strategy: at-least-once` and
107
+ > `overflow: reject-publish` — dead-lettering becomes lossless and this decision is worth
108
+ > revisiting. It would also close the Part 2 retry queue's caveat. Much larger change; its own
109
+ > decision, someday.
110
+
111
+ ## The compatibility trap this avoids
112
+
113
+ Worth stating explicitly, because it is the reason the rejected alternative is *also* expensive:
114
+ declaring `x-dead-letter-exchange` means adding `arguments` to `#assertQueue`. Every durable
115
+ queue in every comq deployment today was declared **without** `arguments`, and RabbitMQ rejects
116
+ a redeclaration whose arguments differ with `406 PRECONDITION_FAILED`.
117
+
118
+ That failure would be loud rather than silent — the declaration happens inside the
119
+ `failsafe(this, this.#recover, …)` wrapper, `permanent()` ([:463](../source/channel.js#L463))
120
+ classifies `PRECONDITION_FAILED` as unrecoverable, and `io.consume(...)` rejects at wiring
121
+ time — but "the application does not start after the upgrade" is still a migration for every
122
+ user. Parking sidesteps it entirely: `comq.parked.<queue>` is a **new** queue, and the source
123
+ queue's declaration is untouched.
124
+
125
+ A migration would have been acceptable, so this is not what decided it — retention is. But it is a real saving, and it keeps the destination soft: a publish target can
126
+ be renamed or changed at any time, where `arguments` on a durable queue are permanent and every
127
+ later change is another migration.
128
+
129
+ ## What the parked message carries
130
+
131
+ Only what a person doing a post-mortem needs. There is no compatibility requirement — nothing
132
+ reads these but a human — so do not reproduce RabbitMQ's `x-death` format, and do not build
133
+ multi-hop machinery (`x-first-death-*` / `x-last-death-*`): a comq message parks once.
134
+
135
+ **The origin must be captured on the first failure, not at parking time.** This is the one
136
+ non-obvious part, and it was got wrong in an earlier draft. A message that has been retried
137
+ comes back through the *default* exchange, so by the time it is parked:
138
+
139
+ ```json
140
+ "exchange": "",
141
+ "routingKey": "orders..billing",
142
+ "headers": {
143
+ "x-comq-attempt": 1,
144
+ "x-death": [{ "reason": "expired", "queue": "comq.retry.30000",
145
+ "exchange": "comq.retry.30000", "routing-keys": ["orders..billing"] }],
146
+ "x-first-death-exchange": "comq.retry.30000"
147
+ }
148
+ ```
149
+
150
+ `message.fields.exchange` is `''` — it has to be — and `fields.routingKey` is now the source
151
+ queue name. So reading `fields` at parking time would record `''` as the origin of **every**
152
+ parked message that was ever retried, which is all of them except one parked by a first-delivery
153
+ `Park`. The broker does not rescue it either: `x-death[0].exchange` and `x-first-death-exchange`
154
+ both name the *retry* exchange. The original is nowhere unless comq writes it down.
155
+
156
+ So `#retry` writes `x-comq-exchange` / `x-comq-key` on the first failure and later retries
157
+ preserve them (Part 2 §4), and `#dispose` prefers those headers, falling back to `message.fields`
158
+ only for the first-delivery `Park` path where nothing has been written yet:
159
+
160
+ ```js
161
+ const headers = {
162
+ ...message.properties.headers, // x-comq-attempt and, after one retry, the origin
163
+ 'x-comq-queue': queue, // threaded in by Part 2 §3
164
+ 'x-comq-exchange': message.properties.headers?.['x-comq-exchange'] ?? message.fields.exchange,
165
+ 'x-comq-key': message.properties.headers?.['x-comq-key'] ?? message.fields.routingKey,
166
+ 'x-comq-reason': exception?.message,
167
+ 'x-comq-at': Date.now()
168
+ }
169
+ ```
170
+
171
+ **Two attempt counters will be on the message, and the docs must say which is authoritative.**
172
+ RabbitMQ maintains `x-death[0].count`, and it counts *deaths* where `x-comq-attempt` counts
173
+ *deliveries*: on a parked message with the default settings they read `4` and `5`. They will not
174
+ agree, and a move to quorum queues changes `x-death` semantics besides. `x-comq-attempt` is
175
+ authoritative — it is comq's, and it is what the ladder is measured against; `x-death` is the
176
+ broker's own record, useful for its timestamps.
177
+
178
+
179
+ ## The seam
180
+
181
+ Route the terminal branch through one method, so the destination is one function body:
182
+
183
+ ```js
184
+ async #park (queue, message, exception) {
185
+ await this.#dispose(queue, message, exception)
186
+ this.#diagnostics.emit('discard', message, exception)
187
+ }
188
+
189
+ // Where a message that has run out of attempts goes: a queue comq owns, published to
190
+ // before the original is released, so a message the broker holds twice is recoverable
191
+ // and one it no longer holds at all is not.
192
+ async #dispose (queue, message, exception) {
193
+ const headers = { /* as above */ }
194
+ const properties = { ...message.properties, headers, mandatory: true, persistent: true }
195
+
196
+ await this.#publish(DEFAULT, parkedQueueOf(queue), message.content, properties)
197
+ this.#channel.ack(message)
198
+ }
199
+ ```
200
+
201
+ `#dispose` is `async` and every caller sits inside `#failed`'s try/catch (Part 2 §4), so it
202
+ inherits the "cannot reject into amqplib" guarantee for free: if the parking publish fails, the
203
+ message is nacked back to the broker rather than lost.
204
+
205
+ The parking queue is per source queue — depth per consumer is how anyone notices a problem — and
206
+ is declared next to the shared retry topology in `#consume` (Part 2 §3), mirroring the source
207
+ queue's durability. The diagnostic keeps its existing name, `discard`, so nothing downstream
208
+ breaks.
209
+
210
+ ## Parked Requests: what is kept, and what is still not answered
211
+
212
+ A parked Request is preserved; its caller is still not answered. The promise never settles —
213
+ the prefetch deadlock [readme.md:469](../readme.md#L469) already documents — and nothing here
214
+ changes that. Parking makes the request *recoverable*, not *answered*.
215
+
216
+ **The parked copy must be published `persistent: true` regardless of the source topology.**
217
+ `source/topology/request.json` is `persistent: false`, so a parked Request would otherwise be
218
+ delivery-mode 1: sitting in a durable queue, but dropped by a broker restart. Requests are
219
+ non-persistent for latency and the failure path is not the latency path — parking is rare and
220
+ terminal, so the disk write costs nothing. The same applies to the retry copy, where a broker
221
+ restart during the TTL would otherwise lose a retried Request. Both `#retry` and `#dispose`
222
+ therefore set `persistent: true` explicitly rather than inheriting it.
223
+
224
+ (The remaining weakness on that channel is that `confirms: false` means the parking publish gets
225
+ no publisher confirm. Channel command ordering still has the broker process the publish before
226
+ the ack, so this is not a loss race — but there is no positive acknowledgement that the broker
227
+ took it. Fixing that means confirms on the request channel, which is a throughput change well
228
+ outside this work.)
229
+
230
+ **A parked Request carries enough to answer it later.** `#createRequest`
231
+ ([io.js:365](../source/io.js#L365)) sets `correlationId` and `replyTo` in the message properties,
232
+ and both `#retry` and `#dispose` spread `...message.properties`, so both survive into the
233
+ parking queue. Whether an answer would *land* depends on the caller:
234
+
235
+ - **Still running** — and it will be, hanging indefinitely, which is the case that matters. Its
236
+ reply queue is still declared, so a reply published to `replyTo` with that `correlationId`
237
+ would resolve the hung promise.
238
+ - **Restarted** — reply queues are `<queue>..<16 random hex>`
239
+ ([createReplyEmitter.js:33](../source/.io/createReplyEmitter.js#L33)) declared *exclusive*
240
+ (`reply.json` is `durable: false`, so `#assertQueue` uses `EXCLUSIVE`), so it died with the
241
+ old connection. The reply is unroutable, and because `#reply` sets `mandatory: true` the broker
242
+ returns it and the `return` diagnostic fires rather than it vanishing silently.
243
+
244
+ `correlationId` carries a per-process prefix, so it is unique across processes and a replay
245
+ cannot cross wires. None of this is built here — but it is worth recording that the door is
246
+ open, because it turns the deadlock from permanent into merely manual.
247
+
248
+ ## The one thing this does not fix
249
+
250
+ A parked message is out of the broker's way but still needs someone to look at it.
251
+ `comq.parked.*` queues grow without bound unless an operator drains them. That is the right
252
+ default — the alternative is deleting evidence — but the readme must say so, and the `discard`
253
+ diagnostic is the hook for alerting on it.
254
+
255
+ # Part 2 — The remaining fixes
256
+
257
+ None of these change an existing queue declaration, so none of them break an upgrade.
258
+
259
+ ## Design
260
+
261
+ ### The retry loop
262
+
263
+ On failure, comq publishes the message to a **shared retry queue**: a queue with no consumer,
264
+ holding it for a TTL and then dead-lettering it back to the queue it came from. Only after the
265
+ publish is confirmed is the original acked.
266
+
267
+ ```
268
+ exchange comq.retry.30000 (fanout)
269
+ queue comq.retry.30000 bound to it
270
+ x-message-ttl: 30000
271
+ x-dead-letter-exchange: ''
272
+ (no x-dead-letter-routing-key — the original key is preserved)
273
+
274
+ publish → exchange comq.retry.30000, routingKey = <source queue>
275
+ expiry → default exchange, routing key preserved → <source queue>
276
+ ```
277
+
278
+ **How one queue returns messages to many destinations.** The routing key is a property of the
279
+ *message*, not of the queue: it is set at publish time, travels with the message, and RabbitMQ
280
+ reuses it when dead-lettering. Tracing an event that failed in `orders..billing`:
281
+
282
+ | step | exchange | routingKey | lands in |
283
+ |---|---|---|---|
284
+ | comq publishes the retry | `comq.retry.30000` (fanout) | `orders..billing` | `comq.retry.30000` |
285
+ | waits out the TTL | — | `orders..billing` (carried) | — |
286
+ | TTL fires, dead-lettered | `''` (default) | `orders..billing` (**reused**) | `orders..billing` |
287
+
288
+ Three broker behaviours carry it:
289
+
290
+ 1. **A fanout exchange ignores the routing key when routing** — it delivers to whatever is bound,
291
+ which is the single retry queue — but preserves the key on the message. The key is therefore
292
+ free to name the destination rather than being consumed by the routing.
293
+ 2. **No `x-dead-letter-routing-key` on the retry queue.** Set, that argument *overrides* the
294
+ message's key with one fixed value — which is precisely what makes a retry queue per-source.
295
+ Unset, the broker reuses whatever key the message already carries, so each message returns to
296
+ its own source queue.
297
+ 3. **The default exchange has an implicit binding to every queue by name**, so a key of
298
+ `orders..billing` reaches the queue `orders..billing`. Standard AMQP; comq declares nothing
299
+ for it.
300
+
301
+ This is also the mechanical reason the message cannot be published to the retry queue directly:
302
+ through the default exchange its key would be the retry queue's own name, and on expiry it would
303
+ route back to itself — the cycle the broker drops silently at the first TTL.
304
+
305
+ And it is why the origin has to be captured on the first failure: after the round-trip the
306
+ message's `fields.exchange` is `''` and its `fields.routingKey` is the source queue, so the
307
+ exchange it was originally published to is genuinely gone from the message.
308
+
309
+ One queue and one exchange **per distinct delay**, shared by every source queue — not one pair
310
+ per consumed queue. The delay comes from the channel topology, so the count does not depend on
311
+ how many queues are consumed.
312
+
313
+ Worked through, because the rule alone is easy to misread. An application with 100 request
314
+ endpoints and 100 consumed events:
315
+
316
+ | | count | scales with |
317
+ |---|---|---|
318
+ | source queues (unchanged, exist today) | 200 | endpoints + events |
319
+ | **retry queues** | **2** (+ 2 exchanges) | distinct `delay` values — *not* queue count |
320
+ | parking queues | 200 | one per source queue |
321
+
322
+ The 100 endpoints share `comq.retry.5000` and the 100 events share `comq.retry.30000`; add a
323
+ thousand more of each and it is still two. A per-queue retry design would have made it 400 new
324
+ queues instead of 202.
325
+
326
+ Three things do move the retry number, and nothing else does: overriding `delay` (the count is
327
+ distinct values in use), a sharded connection (each shard is its own broker with its own pair),
328
+ and nothing further — more services, replicas and connections all declare the same names, and
329
+ `assertQueue` is idempotent.
330
+
331
+ **The fanout exchange is not optional.** Publishing to the retry queue through the default
332
+ exchange would make the message's own routing key the retry queue's name, so on expiry it would
333
+ dead-letter straight back into itself. RabbitMQ does not loop on that — it detects the cycle
334
+ and, since no rejection occurred in it, drops the message. Verified against a broker: the
335
+ message is gone at the **first** expiry, at exactly one TTL, not after a lap or two. So the
336
+ naive shape fails as "every retried message vanishes silently after one TTL" — no loop to
337
+ notice, no queue growth to alert on. A reader who half-remembers that RabbitMQ detects cycles
338
+ may expect something visible to happen; nothing does.
339
+
340
+ Publishing through a fanout exchange leaves the routing key free to name the destination. A
341
+ direct exchange would need a binding per source queue, which is the proliferation being removed.
342
+
343
+ **Naming: `comq.retry.<delay>`, and the delay is in the name deliberately.** `x-message-ttl` is
344
+ fixed at declare time, so a queue whose name did not carry it could never have that value
345
+ changed — every redeclaration with a different TTL is `406 PRECONDITION_FAILED`, and by the
346
+ placement below that means the application does not start. With the delay in the name, a change
347
+ declares a *new* queue and the old one keeps working: during a rolling deploy, old pods publish
348
+ to the old queue and new pods to the new one, and both dead-letter back to the same unchanged
349
+ source queue. There is no window at all.
350
+
351
+ **Orphans stay, and `x-expires` is not the cleanup.** A retired delay leaves an empty queue
352
+ behind. `x-expires` looks like the answer and is not: a retry queue never has a consumer, so
353
+ "unused" is its permanent state, and the timer would delete it along with whatever is waiting
354
+ inside. An empty queue per retired delay value is the price, and it is a housekeeping item
355
+ rather than a hazard.
356
+
357
+ **Declared at consume time, not on first failure.** Declaring inside the catch block means a
358
+ declaration error surfaces at 3am under load, and a failure there falls through to
359
+ `nack(requeue: true)` → immediate redelivery → immediate re-failure → a hot loop with no delay
360
+ and no attempt counting. Declaring in `#consume` puts it inside the
361
+ `failsafe(this, this.#recover, …)` wrapper, where `io.consume(...)` rejects at wiring time with
362
+ a legible error, and it is re-declared for free on `recover()` (which does `lazy.reset` +
363
+ `recall(this)`).
364
+
365
+ **Sharing centralises the blast radius.** Delete or misconfigure `comq.retry.30000` and every
366
+ retry in the application stops, where a per-queue design would lose one queue's. Re-declaring at
367
+ consume time is the mitigation; the documentation line that already says not to delete these
368
+ queues on a running system covers it.
369
+
370
+ **Durability and exclusive source queues.** The shared queue is always durable, and that is
371
+ always right: every topology that acknowledges is also durable (`request.json` and `event.json`
372
+ are both `durable: true`; `reply.json` is `acknowledgments: false` and has no retry path).
373
+ An *exclusive* source queue — from `io.consume(exchange, callback)` with no group — is the one
374
+ case worth tracing. While that ephemeral subscriber is alive the retry is delivered normally.
375
+ Once its connection is gone there is nobody to deliver to, and the message is dropped at expiry
376
+ because the default exchange finds no such queue. A per-queue retry design loses it too, at
377
+ disconnect rather than at expiry. Neither can do better, so sharing costs nothing here.
378
+
379
+ **Not per-message TTL.** RabbitMQ discards an expired message only once it reaches the head of
380
+ its queue, so a long wait would hold up every shorter one behind it. The wait therefore has to
381
+ be a property of the queue — which is why a backoff ladder is a queue per rung rather than an
382
+ `expiration` per message.
383
+
384
+ **There is no separate attempt count.** `delay` is a **ladder with one rung per retry**, so its
385
+ length decides how many there are and four rungs is five attempts. `MAX_REDELIVERIES`
386
+ ([:481](../source/channel.js#L481)) disappears into it: a retry and its wait are the same event,
387
+ and a count kept beside the ladder would be a second source of truth free to disagree with it.
388
+
389
+ The header is the attempt number, counting from one, and the first delivery does not carry it: a
390
+ consumer reads `headers?.['x-comq-attempt'] ?? 1`.
391
+
392
+ The original code kept a `MAX_REDELIVERIES` named for attempts while counting retries, which made
393
+ `5` mean six deliveries and left [readme.md:466](../readme.md#L466) — "causes exceptions five
394
+ times in a row, it is discarded" — off by one.
395
+
396
+ > **The retry queue inherits the at-most-once caveat described in Part 1.** The return hop is
397
+ > broker dead-lettering, which on classic queues republishes without publisher confirms — so a
398
+ > retried message can be lost if the source queue is unavailable when the TTL fires (target
399
+ > unavailable, length limit under `reject-publish`, or a network partition between the nodes
400
+ > hosting the two queues). Still a large improvement on today's ack-before-publish loss, and the
401
+ > alternative — an in-process `setTimeout` — is worse, since it loses the delay on any restart.
402
+ > A known limitation; it belongs in the docs (§8), not left implicit. Moving comq to quorum
403
+ > queues would close it; see Part 1.
404
+
405
+ ### The backoff ladder is public
406
+
407
+ `delay` is a `Topology` field, defaulted per channel type and overridable per deployment. It is
408
+ a list with **one rung per retry**, so its length is the number of retries and there is no
409
+ separate count to fall out of step with it.
410
+
411
+ | | ladder | attempts | total |
412
+ |---|---|---|---|
413
+ | `event` | 1s, 10s, 30s, 90s | 5 | 131s |
414
+ | `request` | 1s, 3s, 5s, 10s | 5 | 19s |
415
+
416
+ **Events.** A fast first rung catches a momentary blip without making it wait out a long one,
417
+ and the ladder then climbs toward the failures worth retrying — a database primary stepping down,
418
+ a storage reconnect, a third-party blip, and a component mid-rollout. That last one is the most
419
+ common and the easiest to misplace: on the event path "nothing is listening yet" is not a failure
420
+ at all, since the message waits in a durable queue. It is reached *indirectly* — a receiver's
421
+ handler calls a component that is mid-rollout, the call raises, the handler throws, the event is
422
+ retried — and a rollout with `maxUnavailable: 0` across several replicas is minutes rather than
423
+ seconds. A ladder covering only a few seconds parks everything that was merely slow, which makes
424
+ the parking queue mean "something was briefly slow" instead of "something is wrong".
425
+
426
+ **Requests are shorter, and deliberately so.** The failure durations are identical; what differs
427
+ is that a request has someone blocked on it with no timeout. Exhausting the attempts means the
428
+ caller gets **no answer at all**, ever — so giving up early is not a cheaper failure, it is an
429
+ unbounded one. What bounds how much coverage to buy is that a reply arriving long after the
430
+ caller gave up lands in an exclusive reply queue that may no longer exist, and comes back
431
+ unroutable.
432
+
433
+ Different defaults per channel type are free: `event.json` and `request.json` already differ on
434
+ four other settings. The two ladders share the rungs `1000` and `10000`, so between them they
435
+ mean **six** retry queues and six exchanges for a whole application — and that number follows the
436
+ rungs, not the number of queues consumed.
437
+
438
+ And it removes a constraint that should never have set a production number: the feature suite
439
+ can run at 100ms without either default being chosen for it. The original `1000` was picked for
440
+ the suite's wall clock, which is the wrong reason for a production constant.
441
+
442
+
443
+ ### Nothing in the failure path may reject
444
+
445
+ amqplib has nowhere to put a rejection, so every path out of the consumer wrapper must be
446
+ total. Fallback is `nack(message, false, true)` — hand the delivery back to the broker rather
447
+ than lose it — itself wrapped so it can't throw either.
448
+
449
+ `#publish` can reject three ways and all three correctly resolve to "give it back":
450
+ `'Channel closed'` from a dying channel, `INTERRUPTION` ([:479](../source/channel.js#L479)) from
451
+ `#unpaused()` on a failfast/sharded channel, and confirmation rejection during `recover()`
452
+ ([:230](../source/channel.js#L230)).
453
+
454
+ Two extra defects found while designing, fixed in the same pass:
455
+ - [:377](../source/channel.js#L377) `exception.message` throws if a consumer does `throw undefined`
456
+ → use `exception?.message`.
457
+ - [:381](../source/channel.js#L381) `this.#discard(...)` is not awaited; once it becomes the
458
+ dead-letter seam (Part 1), an unawaited rejection there escapes.
459
+
460
+ ### Consequences to accept and document
461
+
462
+ - **Ordering is not preserved across a failure.** A retried message re-enters the queue behind
463
+ messages published while it waited. (Ordering was already not guaranteed — `prefetch: 300`
464
+ means up to 300 concurrent in-flight deliveries.) Retries do not reorder among themselves.
465
+ - **At-least-once, not exactly-once.** Publish-before-ack means a crash between the two leaves
466
+ the broker holding both the retry copy and the unacked original. Consumers must be
467
+ idempotent. This is the deliberate trade against today's at-most-once silent loss.
468
+ - **A retry published to a deleted queue is silently dropped**, and publisher confirms still
469
+ ack it. `mandatory: true` makes the broker return it so the `return` diagnostic fires, but
470
+ asynchronously. Document "do not delete `comq.retry.*` on a running system".
471
+ - **`x-comq-attempt` becomes user-visible** — consumers receive it in `properties.headers` and can
472
+ tell which attempt they're on. Worth documenting as a feature. RabbitMQ also appends
473
+ `x-death` on the way back and rewrites `routingKey` to the source queue name.
474
+
475
+ ## Implementation
476
+
477
+ All in [source/channel.js](../source/channel.js) unless noted.
478
+
479
+ ### 1. Record each queue's declaration options
480
+
481
+ The parking queue must mirror its source queue's lifetime — a durable parking queue behind an
482
+ exclusive `amq.gen-*` source leaks a durable queue nothing will ever read. `#consume` cannot
483
+ re-derive this, because only `#assertBoundQueue` ([:303](../source/channel.js#L303)) passes the
484
+ `{ exclusive: true }` override. Record it where it is known:
485
+
486
+ - new field `#queues = new Map()` next to `#tags` ([:26](../source/channel.js#L26))
487
+ - `this.#queues.clear()` in `create()` next to `this.#tags = []` ([:72](../source/channel.js#L72))
488
+ — a fresh amqplib channel has declared nothing
489
+ - `this.#queues.set(queue, options)` in `#assertQueue` ([:250](../source/channel.js#L250))
490
+
491
+ (The retry queue does not need this: it is shared and always durable, per the Design section.)
492
+
493
+ ### 2. The retry and parking topology
494
+
495
+ The retry exchange and queue are per *delay*, so they are declared once per distinct value
496
+ rather than once per source queue. `#assertRetryQueue` takes no queue argument, which is what
497
+ makes `lazy`'s per-argument memo collapse them:
498
+
499
+ ```js
500
+ async #assertRetryQueue () {
501
+ const delay = this.#topology.delay
502
+ const name = RETRY_PREFIX + delay
503
+
504
+ // fanout, so the routing key stays free to name the destination. Publishing through
505
+ // the default exchange instead would make the message's own routing key this queue's
506
+ // name, and the broker would drop it as a cycle at the first expiry.
507
+ await this.#channel.assertExchange(name, 'fanout', DURABLE)
508
+
509
+ await this.#assertQueue(name, {
510
+ ...DURABLE,
511
+ arguments: {
512
+ 'x-message-ttl': delay,
513
+ 'x-dead-letter-exchange': DEFAULT
514
+ // no x-dead-letter-routing-key: the original key is preserved, and it names
515
+ // the source queue because that is what we published with
516
+ }
517
+ })
518
+
519
+ await this.#channel.bindQueue(name, name, '')
520
+
521
+ return name
522
+ }
523
+
524
+ async #assertParkedQueue (queue) {
525
+ const options = this.#queues.get(queue) ?? (this.#topology.durable ? DURABLE : EXCLUSIVE)
526
+
527
+ return (await this.#assertQueue(parkedQueueOf(queue), options))[0]
528
+ }
529
+ ```
530
+
531
+ Plus `RETRY_PREFIX = 'comq.retry.'`, `PARKED_PREFIX = 'comq.parked.'` and
532
+ `parkedQueueOf = (queue) => PARKED_PREFIX + queue`, where `MAX_REDELIVERIES` sits today
533
+ ([:481](../source/channel.js#L481)) — that constant moves into the topology presets as the
534
+ `RETRY_DELAY` never becomes a constant at all; it is `topology.delay` from the start, and
535
+ `MAX_REDELIVERIES` disappears into the ladder's length.
536
+
537
+ ### 3. `#consume` declares them and threads the queue name
538
+
539
+ The queue name is fully resolved by `#consume` for all three paths — `lazy` splices the
540
+ initializer's return over `args` before the method body runs, so `consume`→`#assertQueue`,
541
+ `subscribe`→`#assertBoundQueue`, `bound`→`#assertKeyedQueue` all deliver the broker-confirmed
542
+ name (including generated `amq.gen-*` names for the exclusive case, `io.js:112`).
543
+
544
+ ```js
545
+ // :352
546
+ if (this.#topology.acknowledgments) {
547
+ await this.#assertRetryQueue()
548
+ await this.#assertParkedQueue(queue)
549
+ consumer = this.#getAcknowledgingConsumer(queue, consumer)
550
+ } else options.noAck = true
551
+ ```
552
+
553
+ `#getAcknowledgingConsumer` becomes `(queue, consumer) => async (message) => {...}`; single call
554
+ site at [:356](../source/channel.js#L356). Declare the source queue **before** the others —
555
+ existing tests at [test/channel.test.js:386](../test/channel.test.js#L386) and `:407` read
556
+ `assertQueue.mock.results[0]`.
557
+
558
+
559
+ ### 4. Rewrite the failure path
560
+
561
+ Replace `#getAcknowledgingConsumer` / `#requeue` ([:370-400](../source/channel.js#L370)) with:
562
+
563
+ ```js
564
+ #getAcknowledgingConsumer = (queue, consumer) =>
565
+ async (message) => {
566
+ try {
567
+ await consumer(message)
568
+ this.#channel.ack(message)
569
+ } catch (exception) {
570
+ if (exception?.message === 'Channel closed') return // the broker requeues it
571
+ await this.#failed(queue, message, exception)
572
+ }
573
+ }
574
+
575
+ // Nothing here may reject: amqplib dispatches a delivery through an event emitter and
576
+ // drops the promise it gets back, so a rejection has no one to catch it and takes the
577
+ // process down.
578
+ async #failed (queue, message, exception) {
579
+ try {
580
+ const attempt = message.properties.headers?.[ATTEMPT_HEADER] ?? 0
581
+
582
+ if (attempt >= this.#topology.attempts) await this.#park(queue, message, exception)
583
+ else await this.#retry(queue, message, attempt, exception)
584
+ } catch {
585
+ this.#nack(message) // hand it back rather than lose it
586
+ }
587
+ }
588
+
589
+ async #retry (queue, message, attempt, exception) {
590
+ const headers = {
591
+ // the origin is recorded on the first failure and carried from then on: after one
592
+ // trip through the retry queue the message comes back through the default exchange,
593
+ // so message.fields no longer describes where it was published
594
+ ...origin(message),
595
+ ...message.properties.headers,
596
+ [ATTEMPT_HEADER]: attempt + 1
597
+ }
598
+
599
+ // persistent regardless of the source topology: a retried Request would otherwise be
600
+ // delivery-mode 1 and would not survive a broker restart during the TTL
601
+ const properties = { ...message.properties, headers, mandatory: true, persistent: true }
602
+
603
+ await this.#publish(RETRY_PREFIX + this.#topology.delay, queue, message.content, properties)
604
+
605
+ this.#channel.ack(message)
606
+ this.#diagnostics.emit('retry', message, exception, attempt + 1)
607
+ }
608
+
609
+ #nack (message) {
610
+ try { this.#channel.nack(message, false, true) } catch { /* nothing left to do */ }
611
+ }
612
+
613
+ // spread *under* the existing headers, so the first failure writes it and later ones
614
+ // leave it alone
615
+ const origin = (message) => ({
616
+ 'x-comq-exchange': message.fields.exchange,
617
+ 'x-comq-key': message.fields.routingKey
618
+ })
619
+ ```
620
+
621
+ Two things worth being explicit about:
622
+
623
+ - **`#retry` copies the properties** rather than mutating them as [:396](../source/channel.js#L396)
624
+ does — the mutated message must not reach the fallback `nack`, and a user consumer may still
625
+ hold a reference.
626
+ - **The publish target is the retry *exchange*, and the routing key is the source queue.** That
627
+ is what survives the expiry and routes the message home.
628
+
629
+ `#park` / `#dispose` are as given in Part 1.
630
+
631
+
632
+ ### 5. Wire the `retry` diagnostic
633
+
634
+ - [source/events.js:7](../source/events.js#L7) — add `'retry'` to `exports.channel`. This single
635
+ switch makes both `source/io.js:271` forward it to `IO.diagnose` **and**
636
+ `source/shards/channel.js:183` `#pipe` forward it with the shard index appended.
637
+ - [types/diagnostic.d.ts:4](../types/diagnostic.d.ts#L4) — add `'retry'` to the `Event` union.
638
+ - [types/io.d.ts:106](../types/io.d.ts#L106) — add the `retry` overload. **Also fix the
639
+ pre-existing bug on that line**: `discard` is declared `(channel, message, index?)` but
640
+ `channel.js:408` has always passed the exception too.
641
+ - [test/io.diagnostics.test.js:31](../test/io.diagnostics.test.js#L31) — add `'retry'` to the
642
+ hardcoded re-emit list.
643
+ - [features/steps/context.js:124](../features/steps/context.js#L124) — add `'retry'` to `EVENTS`.
644
+
645
+ ### 5b. Make the backoff ladder configurable
646
+
647
+ Small and self-contained; the reasoning and the defaults are under *Delay and attempts are both
648
+ public*.
649
+
650
+ - `source/topology/{request,event}.json` — add `"attempts": 5`, and `"delay": 30000` /
651
+ `"delay": 5000`. Leave `reply.json` alone; it is `acknowledgments: false`, so nothing there
652
+ ever counts an attempt or waits a delay.
653
+ - [types/topology.d.ts](../types/topology.d.ts) — add `attempts: number` and `delay: number` to
654
+ `Topology`.
655
+ - [source/connection.js:92](../source/connection.js#L92) — `const topology = presets[type]` hands
656
+ out the shared module-level preset object. Make it merge into a copy,
657
+ `{ ...presets[type], ...this.#overrides?.[type] }`, and never mutate the preset.
658
+ - [source/connect.js](../source/connect.js) — `Connect` is variadic
659
+ (`(...urls: string[]) => Promise<IO>`), so the overrides are an optional trailing argument,
660
+ taken when `typeof last !== 'string'` and typed as `[...urls: string[], options: Options]`.
661
+ Thread it through `create()` → `IO` → `Connection`.
662
+ - [test/presets.test.js](../test/presets.test.js) — all three cases assert with `toStrictEqual`,
663
+ so all three change.
664
+ - New tests: an override reaches the channel; an absent override leaves the preset untouched
665
+ (assert the module object is not mutated across two connections); a consumer configured with
666
+ `attempts: 1` parks on the first delivery, having had its one attempt; two channels with different
667
+ `delay` values declare two differently-named retry queues and neither redeclares the other's.
668
+
669
+ ### 6. Unit tests — new `describe('retry')` in [test/channel.test.js](../test/channel.test.js)
670
+
671
+ `test/amqplib.mock.js` needs no changes. **`test/channel.fixtures.js` randomises `durable`,
672
+ `confirms` and `acknowledgments` with `flip()`** — every new test must set the ones it asserts
673
+ on explicitly, and the file should be run repeatedly.
674
+
675
+ Declaration: the retry **exchange** asserted as `fanout` and the retry queue asserted with the
676
+ delay-derived name, the TTL and **no** `x-dead-letter-routing-key`, and bound to that exchange;
677
+ the parking queue asserted with the right name and no arguments. All asserted *after* the source
678
+ queue; none asserted when `acknowledgments: false`; the parking queue exclusive when the source
679
+ is exclusive (test both values of `durable`) while the retry queue stays durable regardless;
680
+ asserted for the `bound` path; re-asserted after `recover()`. And the one that guards the shared
681
+ design: **two consumers on the same channel declare the retry topology once**, not once each.
682
+
683
+ Failure path — the regression tests that matter:
684
+ - **publishes before acking** —
685
+ `chan.publish.mock.invocationCallOrder[0] < chan.ack.mock.invocationCallOrder[0]`
686
+ - **waits for the confirmation** — `confirms: true`, capture the 5th publish arg, assert `ack`
687
+ not yet called, invoke the callback, assert it then is
688
+ - **does not republish to the original exchange** — build the message with a random
689
+ `fields.exchange`; assert the publish goes to the retry exchange with the **source queue** as
690
+ the routing key, never to `fields.exchange`. *(the fanout re-delivery regression)*
691
+ - **records the origin on the first failure** — `fields.exchange` and `fields.routingKey` are
692
+ written as `x-comq-exchange` / `x-comq-key`; a message that already carries them (a returned
693
+ retry, whose `fields.exchange` is now `''`) keeps the originals rather than overwriting them
694
+ with `''`. *(the wrong-provenance regression his broker run found)*
695
+ - **tolerates a message with no headers** — `properties = {}`. *(the non-comq-publisher
696
+ `TypeError` regression)*
697
+ - **does not throw** — `await expect(callback(message)).resolves.not.toThrow()`. *(the
698
+ process-crash regression)*
699
+ - **does not seal** — `chan.cancel` not called, `seal` spy not called, and a subsequent
700
+ `channel.consume(...)` still works *(proves `#sealed` is still false)*
701
+ - increments `x-comq-attempt`; does not mutate `message.properties`; emits `retry` with the attempt
702
+ - **publishes persistent even from a non-persistent topology** — `topology.persistent = false`
703
+ (the request preset), assert the publish options carry `persistent: true`, for both the retry
704
+ and the parking publish. *(a parked Request would otherwise not survive a broker restart)*
705
+ - **carries `replyTo` and `correlationId` through** — a message with both in its properties is
706
+ parked with both intact, so a hung caller remains answerable
707
+ - **defensive fallback**: publish throws → `nack(message, false, true)`, no `ack`, resolves;
708
+ publish *and* nack both throw → still resolves; confirmation rejected → same as publish throw
709
+ - consumer throwing `'Channel closed'` → no publish, no nack, no ack (complements the existing
710
+ test at [:144](../test/channel.test.js#L144), which covers `ack` throwing)
711
+ - consumer doing `throw undefined` → resolves *(covers the `exception?.message` guard)*
712
+ - **terminal**: `x-comq-attempt: 5` → published to `comq.parked.<queue>`, **not** to the retry queue;
713
+ published before acking, same order assertion as above; carries the `x-comq-*` headers
714
+ including an `x-comq-exchange` that names the original exchange rather than `''`; `nack` not
715
+ called. Restores
716
+ the spirit of the `it.each` requeue test `1df2afe` deleted. The existing `should emit
717
+ 'discard' event` ([:894](../test/channel.test.js#L894)) needs its message fixture extended with
718
+ `fields` — it currently has none, which works only because today's `#discard` never reads
719
+ them.
720
+ - **parking failure**: publish to the parking queue throws → `nack(message, false, true)`, no
721
+ `ack`, resolves. The message survives an unavailable parking queue.
722
+
723
+ ### 7. Feature tests — rewrite [features/events.poison.feature](../features/events.poison.feature)
724
+
725
+ Both scenarios drop `@manual`. Two structural facts shape this:
726
+
727
+ - `features/steps/brokers.js` starts containers in `BeforeAll` and `hooks.js` only disconnects,
728
+ so **durable queues and their contents survive across scenarios**. A message still cycling
729
+ through a retry queue will land in the next scenario's consumer. **Use a distinct exchange
730
+ name per scenario** — do not share `poison`.
731
+ - [features/steps/events.js:41](../features/steps/events.js#L41) builds the throwing consumer.
732
+ Change it to record `properties.headers?.['x-comq-attempt']` into a new `this.attempts` array,
733
+ which turns the whole retry loop into an end-to-end assertion against a real broker.
734
+
735
+ ```gherkin
736
+ Scenario: A poison event is retried and then discarded
737
+ Given that events from the `poison_retried` exchange are causing exceptions
738
+ When an event is emitted to the `poison_retried` exchange
739
+ Then the event is attempted 6 times
740
+ And the message is parked
741
+
742
+ Scenario: A poison event does not stop other consumers
743
+ Given that events from the `poison_isolated` exchange are causing exceptions
744
+ And that `checker` is consuming events from the `numbers_added` exchange
745
+ When an event is emitted to the `poison_isolated` exchange
746
+ And after 1500ms
747
+ And an event is emitted to the `numbers_added` exchange
748
+ Then `checker` receives the event
749
+
750
+ Scenario: A retried event is not re-delivered to the other consumers of its exchange
751
+ Given that events from the `poison_open` exchange are causing exceptions
752
+ And that `witness` is consuming events from the `poison_open` exchange
753
+ When an event is emitted to the `poison_open` exchange
754
+ And after 3000ms
755
+ Then `witness` has received 1 event
756
+ ```
757
+
758
+ Scenario 2 is the direct proof the channel is not sealed — `numbers_added` and
759
+ `poison_isolated` share the one `#events` channel, so under today's code the second
760
+ `emit`/`consume` is dead. Scenario 3 is the fanout regression: exactly once, not once per retry.
761
+
762
+ Steps needed:
763
+ - `Then the event is attempted {int} times` (new, `features/steps/poison.js`) — poll
764
+ `this.attempts` to a deadline, then `assert.deepEqual(this.attempts, [1, 2, 3, 4, 5])`.
765
+ Asserts count, header increment, and that the delay round-trips through the broker. Six
766
+ entries, per the arithmetic above — the first delivery has no `x-comq-attempt` header.
767
+ - `Then the message is discarded` — **already exists** at
768
+ [features/steps/poison.js:7](../features/steps/poison.js#L7) and is currently orphaned (no
769
+ feature file references it). It becomes live; change its `await timeout(300)` to a polling
770
+ wait, via a new `until(predicate, ms)` helper in `test/helpers.js` next to `timeout`.
771
+ - `Then the message is parked` (new) — the real end-to-end assertion, and only possible because
772
+ Part 1 gives comq a queue it owns: `io.process('comq.parked.<queue>', …)` and assert the payload
773
+ arrives, carrying `x-comq-attempt: 5` and the `x-comq-*` headers. Under today's code, and under a
774
+ broker-side DLX, there would be nothing to consume. Prefer this over the diagnostic-flag step
775
+ wherever both would work — it proves the message actually survived, not just that comq
776
+ believes it did.
777
+ - `Then {token} has received {int} event(s)` (new, `features/steps/events.js`) — needs
778
+ per-group counts; today only a single global `eventsConsumedCount` and a last-value-only
779
+ `consumed[group]` exist.
780
+ - `features/steps/context.js` — add `attempts = []` and `counts = {}`, reset on disconnect.
781
+
782
+ ### 8. Docs
783
+
784
+ Replace [readme.md:464-470](../readme.md#L464) (which currently documents the sealed channel and
785
+ the "configure a DLX yourself" workaround) with prose covering: the retry queue and its name;
786
+ `x-message-ttl` + `x-dead-letter-exchange` as the delay mechanism; `x-comq-attempt` visible to
787
+ consumers; **five attempts by default** — the first delivery and four retries — then `discard`,
788
+ which is what [readme.md:466](../readme.md#L466) already claimed, and noting that both
789
+ the ladder is a topology setting (`delay`) with per-channel-type defaults, and its length is
790
+ the number of retries; **the channel keeps consuming**; ordering not
791
+ preserved across a failure; at-least-once / consumers must be idempotent; retries confirmed for
792
+ Events but best-effort for Requests (`confirms: false`, `persistent: false`), and a discarded
793
+ Request is never answered — the caller waits indefinitely, which with a limited prefetch can
794
+ deadlock it; do not delete `comq.retry.*` or `comq.parked.*` on a running system; and **the retry
795
+ queue's return hop is at-most-once on classic queues** — a retry can be lost if the source queue
796
+ is unavailable when the TTL fires.
797
+
798
+ Then the parking half, from Part 1: comq publishes the exhausted message to `comq.parked.<queue>`
799
+ and acks only once the broker confirms it, so it is not deleted and not dependent on a
800
+ broker-side policy; the `x-comq-*` headers it carries; that these queues **grow until someone
801
+ drains them**, which is deliberate — the alternative is deleting evidence — and that `discard`
802
+ is the diagnostic to alert on.
803
+
804
+ **Delete the current recommendation to configure a dead-letter policy**
805
+ ([readme.md:468-470](../readme.md#L468)). It is no longer needed, and the reasoning in Part 1 —
806
+ that broker dead-lettering is at-most-once on classic queues — means it was never the guarantee
807
+ it appeared to be. A DLX on the parking queue itself remains a user's choice; say nothing about
808
+ it either way.
809
+
810
+ The topology section ([readme.md:428](../readme.md#L428)) gains the retry exchange and queue —
811
+ one pair per distinct `delay`, shared by every source queue — and the per-queue parking queue,
812
+ so the queue list an operator sees is documented rather than discovered.
813
+
814
+ Add `retry` to the diagnostics list at [readme.md:556](../readme.md#L556), and to
815
+ `docs/headers.md`: `x-comq-attempt`, the `x-comq-*` parking headers, and a note that a retried message
816
+ also carries the broker's `x-death` — whose `count` is a *second* attempt counter that happens to
817
+ agree with `x-comq-attempt` today. Say which is authoritative (`x-comq-attempt`, comq's own; `x-death` is
818
+ the broker's record and is useful for its timestamps), because two counters on one message is
819
+ otherwise a debugging trap.
820
+
821
+ ## Not affected — checked
822
+
823
+ - `source/shards/channel.js:59` delegates `consume`/`subscribe`/`bound` to per-shard `Channel`s.
824
+ Each shard is a separate broker with its own copy of the queue, declares its own retry
825
+ topology and `comq.parked.*`, and returns retries to its own source. Self-contained —
826
+ though note a parked message lives on the shard that failed it, so a post-mortem means
827
+ looking at every shard. `#retry` uses
828
+ `#publish` directly (not the `failsafe`-wrapped `publish`), so on a paused failfast shard it
829
+ throws `INTERRUPTION` → `#nack` → the message stays on that shard. Acceptable.
830
+ - `IO.seal()` / `IO.close()` and the `#sealed` guards at `:88`, `:101`, `:119` are the
831
+ *explicit* seal and stay exactly as they are. Only the failure path stops calling `seal()`.
832
+ - `source/emitter.js:11` swallows listener exceptions, so a bad `retry` listener can't
833
+ re-introduce the crash.
834
+ - Existing assertions the extra `assertQueue` call could have broken, all verified safe:
835
+ `channel.test.js:185`/`:191` (`send` path, no retry queue), `:358` (`assertExchange` count),
836
+ `:387` (`bindQueue` count — the retry queue is never bound), `:74`/`:363`
837
+ (`toHaveBeenCalledWith`, not counts). Only the `mock.results[0]` reads depend on ordering,
838
+ which the "source queue first" rule preserves.
839
+ - `test/io.*.test.js` use the channel-level `test/connection.mock.js`; unaffected.
840
+
841
+ ## Verification
842
+
843
+ 1. `npm test` (`standard && jest`) — run `npx jest test/channel.test.js` **several times**,
844
+ since `test/channel.fixtures.js` randomises the topology per run.
845
+ 2. `npm run features` (needs Docker; Testcontainers starts two RabbitMQ containers). Time the
846
+ poison feature specifically — 5 attempts in one step at a 1s delay is the tightest new
847
+ timing in the suite, against a 30s per-step cucumber timeout.
848
+ 3. The end-to-end proof is that `features/events.poison.feature` becomes ordinary automatic
849
+ scenarios: a consumer that throws, a message that comes back five times with an incrementing
850
+ `x-comq-attempt` (five deliveries in total), a message that ends up disposed rather than silently
851
+ gone, and — the point of the whole exercise — a *sibling* consumer still alive afterwards.
852
+ 4. Confirm the process no longer exits: today, running the poison scenario ends the node
853
+ process. After the change it should complete the scenario.
854
+ 5. Inspect the broker after a features run — the retry queues empty, `comq.parked.*` holding
855
+ exactly the messages the poison scenarios parked, with `x-comq-exchange` naming the original
856
+ exchange rather than `''`. That last one is the check that the origin is captured on the
857
+ first failure rather than at parking time.
858
+ 6. Confirm the cycle claim directly, once, rather than trusting the document: declare a queue
859
+ with a TTL, `x-dead-letter-exchange: ''` and no routing key, publish to the default exchange
860
+ with that queue as the key, and watch the message vanish at the first expiry. It is the
861
+ reason the retry exchange exists.
862
+ 7. Branch off `dev` (not `release`).
863
+
864
+ ---
865
+
866
+ # Part 3 — A consumer says how it failed (`Retry` / `Park`)
867
+
868
+ A follow-up proposal lets a consumer classify its failure: `throw new Retry()` for a
869
+ dependency that was briefly absent, `throw new Park()` for a message this consumer will never
870
+ process. A bare rejection means `Retry`.
871
+
872
+ ## What it changes in this plan: almost nothing
873
+
874
+ **None of the six defects change.** Every one of them exists whether or not a consumer can
875
+ classify its failure — the verdict is a *routing* decision layered on the failure path, not a
876
+ fix to it. Parts 1 and 2 stand exactly as written, and remain worth landing on their own: a
877
+ `Park()` that reaches today's `#requeue` still seals the channel and still kills the process.
878
+
879
+ **It adds one branch**, in `#failed` (Part 2 §4). The seam already exists:
880
+
881
+ ```js
882
+ const park = verdictOf(exception) === PARK || attempt >= this.#topology.attempts
883
+
884
+ if (park) await this.#park(queue, message, exception)
885
+ else await this.#retry(queue, message, attempt, exception)
886
+ ```
887
+
888
+ `Retry` still respects the ladder — a consumer can say "worth another attempt" without
889
+ being able to say "forever". comq keeps its own count, which is what the proposal's "no
890
+ counts" asks for.
891
+
892
+ **Part 1 already covers it.** Parking is the destination whether the message got there by
893
+ exhausting its attempts or by a consumer saying so on the first delivery. `Park()` does raise
894
+ the stakes — parking stops being a rare terminal event and becomes a routine outcome a consumer
895
+ chooses deliberately — which is a further argument for Part 1's decision, and an argument
896
+ against ever making it opt-in.
897
+
898
+ ## Thrown, not returned — the return channel is already taken
899
+
900
+ [source/io.js:305](../source/io.js#L305): `const reply = await producer(payload)`. For `io.reply`,
901
+ **what the consumer returns is the reply**, and [io.js:308](../source/io.js#L308) then inspects it
902
+ for iterator-ness to choose streaming versus a single reply. A returned symbol or verdict object
903
+ would be encoded and sent to the caller as the answer. `types/io.d.ts:10` says the same:
904
+ `Producer<Input, Output> = (message) => Output | Promise<Output>`.
905
+
906
+ Event consumers ([io.js:345](../source/io.js#L345)) do ignore their return value, so returning would
907
+ work *there* — but a verdict that works for `consume`/`process` and not for `reply` is worse
908
+ than one that works everywhere. Second reason: a thrown verdict can be raised from anywhere in
909
+ the call stack, so a validation helper three frames down can park the message; a return can only
910
+ be made by the top-level function.
911
+
912
+ ## Shape
913
+
914
+ ```js
915
+ // source/verdicts.js — its own module, so source/channel.js can require it
916
+ // without going through the package entry point
917
+ 'use strict'
918
+
919
+ // the global registry, so two copies of comq in one dependency tree still agree
920
+ const VERDICT = Symbol.for('comq.verdict')
921
+
922
+ const RETRY = 'retry'
923
+ const PARK = 'park'
924
+
925
+ class Retry extends Error { [VERDICT] = RETRY }
926
+ class Park extends Error { [VERDICT] = PARK }
927
+
928
+ // a rejection nobody classified is a rejection nobody chose
929
+ const verdictOf = (exception) => exception?.[VERDICT] ?? RETRY
930
+ ```
931
+
932
+ Two deliberate choices:
933
+
934
+ - **`Symbol.for` brand, not `instanceof`.** Required, not defensive: a consumer of comq that
935
+ pins it inside its own binding while the application resolves its own tree makes two copies
936
+ the ordinary resolution rather than the pathological one, and `instanceof` is silently `false`
937
+ across them. A `Park` would then be read as a bare rejection, retried to exhaustion, and
938
+ parked anyway — the right outcome, several retry cycles late, with a spurious `retry`
939
+ diagnostic for each. The global symbol registry is shared across copies; a class identity is not.
940
+ - **`extends Error`**, so a verdict carries a stack, a `message`, and `cause`. A bare
941
+ `new Park()` would throw the cause away, and the `discard` diagnostic's second argument
942
+ ([channel.js:408](../source/channel.js#L408)) would be a marker with nothing about what actually
943
+ went wrong. Use `new Park('…', { cause })` and record both in the parking headers —
944
+ `x-comq-reason` from the verdict's own message, and the cause's message alongside it.
945
+
946
+ ## Consumer code
947
+
948
+ ```js
949
+ const { Retry, Park } = require('comq')
950
+
951
+ // events — fanout
952
+ await io.consume('orders', 'billing', async (order) => {
953
+ if (typeof order.total !== 'number') throw new Park('no total on order')
954
+
955
+ try {
956
+ await billing.charge(order)
957
+ } catch (e) {
958
+ throw new Retry('billing unavailable', { cause: e }) // or just: throw e
959
+ }
960
+ })
961
+
962
+ // tasks
963
+ await io.process('emails', async (job) => {
964
+ await mailer.send(job) // a bare throw means Retry
965
+ })
966
+
967
+ // requests — the return value is the reply, so the verdict must be thrown
968
+ await io.reply('compute', async (input) => {
969
+ if (typeof input.n !== 'number') throw new Park('n must be a number')
970
+
971
+ return { result: input.n * 2 } // ← the return channel is already spoken for
972
+ })
973
+ ```
974
+
975
+ ## `Park` from a Producer is a category error
976
+
977
+ `Park` on an `io.reply` producer would mean the caller never gets a reply and its promise never
978
+ settles — the prefetch deadlock [readme.md:469](../readme.md#L469) already admits. Today that
979
+ happens by accident once the attempts are spent; a `Park` would make it a documented path.
980
+
981
+ **Resolved: reject it.** `consume` and `process` take a `Consumer` — no caller, so somebody has
982
+ to decide what happens to a failed message. `reply` takes a `Producer` — a caller is waiting, and
983
+ what it is owed is a reply. Retry-or-park is not a policy choice there, and comq's own type
984
+ vocabulary already draws the line.
985
+
986
+ The rejection cannot be static — nothing at wiring time knows what a producer will throw — so
987
+ concretely: a `Park` from a producer is treated as a bare rejection, so the message retries and
988
+ eventually parks on the count, and comq emits a diagnostic saying `Park` is not applicable to a
989
+ reply producer. Loud, no new protocol, and the message is not lost.
990
+
991
+ The alternative of sending an error reply was rejected on its own terms: it adds an element to
992
+ the reply protocol that every caller must learn to distinguish, and it collides with a
993
+ distinction callers already make between an error returned as a value and an exception rethrown
994
+ in them. A third kind arriving from the transport would have to be mapped onto one of those.
995
+
996
+ Burning every attempt on a request nobody can process does **not** wedge the consumer, which was
997
+ an earlier worry here: a retry publishes and acks, so the delivery is released immediately and
998
+ the message waits in the retry queue, not in the consumer's unacked set. The cost is a handful
999
+ of pointless invocations.
1000
+
1001
+ ## Sequencing
1002
+
1003
+ Land Parts 1 and 2 first. The crash is a dead process today; the verdict API is additive and
1004
+ non-breaking (bare rejection = `Retry` = the behaviour Part 2 establishes), so it composes
1005
+ cleanly on top as a minor release.
1006
+
1007
+ One thing argues for doing them together: Part 1 names the parking queue and Part 3 names the
1008
+ class, and they should agree — `Park` → `comq.parked.<queue>`, or `Dead` →
1009
+ `comq.parked.<queue>`. Naming it twice means renaming a queue in a later release, which orphans
1010
+ whatever is sitting in the old one. **Pick the noun once**, before Part 1 is implemented; it is
1011
+ the only thing Part 3 needs decided up front.
1012
+
1013
+ ## Surface it adds
1014
+
1015
+ - `source/index.js` currently exports only `connect` and `assert`, and comq has no exported
1016
+ error classes at all — `Retry`/`Park` would be the first. Clean addition, plus
1017
+ `types/index.d.ts` (which re-exports from `./io`; add the two classes there).
1018
+ - `source/verdicts.js` is required directly by `source/channel.js`, not through
1019
+ `source/index.js` — `index.js` pulls in `connect`, and the channel must not depend on it.
1020
+ - Unit tests: `verdictOf` returns `RETRY` for `undefined`, `null`, a string throw, a plain
1021
+ `Error`, and an object with a *different* symbol key; returns `PARK` only for the brand. Then
1022
+ in `test/channel.test.js`, a consumer throwing `Park` publishes to the parking queue on the
1023
+ **first** delivery with no retry publish, and a consumer throwing `Retry` behaves exactly like
1024
+ a bare rejection.
1025
+ - One feature scenario per verdict: a `Park()` on first delivery lands in the parking queue with
1026
+ no retry cycle at all (and completes fast, since it skips five TTL waits — a useful contrast
1027
+ with the full-ladder scenario); a `Retry()` behaves exactly as a bare rejection.
1028
+ - Docs: the verdicts get their own readme section, and the `x-comq-reason` header note in
1029
+ `docs/headers.md` should mention that a `cause` is recorded alongside it.
1030
+
1031
+ ---
1032
+
1033
+ # Context
1034
+
1035
+ ## The six defects
1036
+
1037
+ All introduced by `1df2afe` *refactor: throw exceptions on poison messages*, in
1038
+ `#getAcknowledgingConsumer` ([:370](../source/channel.js#L370)) and `#requeue`
1039
+ ([:393](../source/channel.js#L393)).
1040
+
1041
+ 1. **It acks before it publishes.** [:394](../source/channel.js#L394) acks, then
1042
+ [:399](../source/channel.js#L399) publishes. A crash between the two loses the message.
1043
+ 2. **It republishes to `message.fields.exchange`.** For a queue bound to a fanout exchange that
1044
+ *is* the exchange, so a retry meant for the one subscriber that failed is fanned out to
1045
+ every subscriber again.
1046
+ 3. **It seals the channel** ([:398](../source/channel.js#L398)). `seal()` cancels every consumer tag
1047
+ on the channel; `#sealed` is never cleared (not even by `recover()` at
1048
+ [:213](../source/channel.js#L213)) and `recall.reset` at [:198](../source/channel.js#L198) throws
1049
+ away the callbacks that would let them come back. And it is worse than one subscription:
1050
+ [source/io.js](../source/io.js) routes `consume`, `emit`, `route`, `subscribe`, `process` **and**
1051
+ `enqueue` through a single `#events` channel, so one poison event permanently silences every
1052
+ event consumer *and* every task processor in the process.
1053
+ 4. **It rethrows into amqplib** ([:384](../source/channel.js#L384)), which dispatches deliveries
1054
+ through an EventEmitter listener (`amqplib/lib/channel_model.js:47` →
1055
+ `lib/channel.js:455 return consumer(message)`) and drops the returned promise. Unhandled
1056
+ rejection, and Node ends the process by default. That crash *is* the current retry
1057
+ mechanism: [features/events.poison.feature](../features/events.poison.feature) is entirely
1058
+ `@manual` and instructs a human to "run this scenario five times".
1059
+ 5. **It reads `message.properties.headers[...]` unguarded**
1060
+ ([:379](../source/channel.js#L379), [:396](../source/channel.js#L396)) — see the correction below.
1061
+ [source/shards/channel.js:204](../source/shards/channel.js#L204) already guards the identical
1062
+ pattern with `?.`.
1063
+ 6. **The last attempt destroys the message** — Part 1.
1064
+
1065
+ ## Details worth knowing
1066
+
1067
+ Findings from verifying the above against the code and against amqplib, kept because they
1068
+ correct claims that get made about this area.
1069
+
1070
+ - **The unguarded header read is narrower than it looks.** It is often described as breaking for
1071
+ any task, broadcast or request. It does not, for anything comq published:
1072
+ `amqplib/lib/api_args.js:191` does `Object.create(options.headers || null)` and always assigns
1073
+ `headers` into the publish properties, and `lib/codec.js:65` encodes with
1074
+ `for (const key in val)`, walking the prototype chain. So every comq-published message arrives
1075
+ with a headers table, at minimum `{}`. The `TypeError` is real only for a message published by
1076
+ a **non-comq client** that omits the field table (`defs.js:3438`, `headers: void 0` is the
1077
+ decode default). That is a supported case rather than a hypothetical one — consuming events
1078
+ published by something that is not comq is a thing comq is used for — so the guard is
1079
+ load-bearing, not tidiness. `source/shards/channel.js:204` already guards the identical
1080
+ pattern.
1081
+ - **Publisher confirms cover events only.** `source/topology/request.json` is `confirms: false`
1082
+ and the channel is created with `createChannel`, not `createConfirmChannel`
1083
+ ([channel.js:68](../source/channel.js#L68)) — there is no confirm to await on the request channel.
1084
+ Requests are also `persistent: false`, so a requeued request does not survive a broker restart
1085
+ regardless. This is why Part 2 fixes requests structurally but calls them best-effort.
1086
+ - **`MAX_REDELIVERIES = 5` meant six deliveries, not five.** The counter was absent on the first
1087
+ delivery, so it read `0` and the message was retried; `5 >= 5` first held on the sixth, while
1088
+ [readme.md:466](../readme.md#L466) said "five times in a row". Fixed by counting deliveries
1089
+ rather than retries, which is what the setting's name claims.
1090
+ - **Test coverage for this path collapsed in `1df2afe`.** That commit deleted the requeue unit
1091
+ test and the one automated scenario in `features/rpc.feature`, replaced them with two
1092
+ `@manual` scenarios that assert nothing, and left the `the message is discarded` step in
1093
+ `features/steps/poison.js` referenced by no feature file at all.
1094
+
1095
+ ## Decisions already taken
1096
+
1097
+ - **Retry delay: broker-side delay queue (TTL + DLX).** Survives a process restart and does not
1098
+ hold an unacked delivery against prefetch.
1099
+ - **`seal()` on poison is replaced by a diagnostics event only.** The channel keeps consuming.
1100
+ No circuit breaker.
1101
+ - **Requests get the same structural fixes, best-effort.** `request.json` is not changed; the
1102
+ limitation is documented.
1103
+ - **Parking, not broker dead-lettering** (Part 1). Decided on retention, with the migration cost
1104
+ ruled acceptable and therefore not a criterion: dead-lettering on classic queues can still
1105
+ lose the message, and the point of keeping it is that it is kept.
1106
+
1107
+ ## Still open
1108
+
1109
+ Nothing blocking. The three questions this document opened are resolved:
1110
+
1111
+ - **The noun** — `Park` / `comq.parked.<queue>`. See Part 1.
1112
+ - **`Park` on an `io.reply` producer** — rejected as a category error, treated at runtime as a
1113
+ bare rejection plus a diagnostic. See Part 3.
1114
+ - **Whether Part 3 is built** — **decided, not yet scheduled.** Parts 1 and 2 are the emergency
1115
+ and do not depend on it; a consumer of comq gets retry-then-park by upgrading and writing no
1116
+ code. But the parking queue takes its noun from a verdict class that would not otherwise
1117
+ exist, and without verdicts a message a contract has already refused costs every attempt —
1118
+ two minutes at the event default — and then lands in the parking queue beside the
1119
+ messages that genuinely failed. "Decided, not yet scheduled" rather than "open", because
1120
+ people read this deciding whether to build on it.