comq 0.17.1 → 0.18.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.github/workflows/{dev.yaml → features.yaml} +14 -3
- package/discussions/poison-message-handling.md +1120 -0
- package/docs/headers.md +36 -0
- package/features/events.poison.feature +32 -12
- package/features/steps/context.js +18 -2
- package/features/steps/events.js +5 -1
- package/features/steps/poison.js +65 -4
- package/package.json +1 -1
- package/readme.md +103 -14
- package/source/channel.js +210 -30
- package/source/connect.js +31 -21
- package/source/connection.js +11 -2
- package/source/events.js +1 -1
- package/source/singleton.js +47 -4
- package/source/topology/event.json +7 -1
- package/source/topology/request.json +7 -1
- package/test/assert.test.js +2 -2
- package/test/channel.fixtures.js +2 -1
- package/test/channel.routed.test.js +7 -3
- package/test/channel.test.js +593 -5
- package/test/connect.test.js +31 -2
- package/test/helpers.js +15 -0
- package/test/io.diagnostics.test.js +1 -1
- package/test/presets.test.js +4 -2
- package/test/singleton.test.js +1 -1
- package/types/connection.d.ts +6 -1
- package/types/diagnostic.d.ts +1 -1
- package/types/io.d.ts +3 -1
- package/types/topology.d.ts +16 -0
|
@@ -0,0 +1,1120 @@
|
|
|
1
|
+
# Fix comq's poison-message handling
|
|
2
|
+
|
|
3
|
+
> **The design record for how comq handles a message its consumer could not process.**
|
|
4
|
+
> Reviewed from the consumer side in
|
|
5
|
+
> [#272](https://github.com/toa-io/comq/discussions/272) and revised there: the retry queue
|
|
6
|
+
> became shared and delay-named, the delay became configurable, and message provenance moved to
|
|
7
|
+
> where it survives a retry.
|
|
8
|
+
>
|
|
9
|
+
> **Parts 1 and 2 are implemented.** Part 3 — the `Retry` / `Park` verdict API — is decided but
|
|
10
|
+
> not yet scheduled; see the end. Text below is written as a proposal because that is what it
|
|
11
|
+
> was; it describes what the code now does, except where Part 3 says otherwise.
|
|
12
|
+
|
|
13
|
+
When a consumer callback rejects, comq is supposed to retry the message a few times and then
|
|
14
|
+
give up on it. What it actually does is kill the process, and take every other consumer in that
|
|
15
|
+
process with it. Six verified defects, all in [source/channel.js](../source/channel.js) — full
|
|
16
|
+
detail in **Context** at the end.
|
|
17
|
+
|
|
18
|
+
One of the six — the last attempt silently deleting the message — needs a mechanism comq does
|
|
19
|
+
not have today, and the obvious candidate is not one. That comes first, because everything else
|
|
20
|
+
is routine by comparison.
|
|
21
|
+
|
|
22
|
+
- **Part 1** — where a message goes when comq gives up on it. comq parks it in a queue it owns.
|
|
23
|
+
Includes why RabbitMQ's dead-lettering is *not* the mechanism for this, and the one
|
|
24
|
+
compatibility trap to avoid.
|
|
25
|
+
- **Part 2** — the other five defects. No upgrade impact.
|
|
26
|
+
- **Part 3** — the proposed `Retry` / `Park` verdict API, and what it does and does not change
|
|
27
|
+
here (short answer: it adds one branch to Part 2, and one naming decision to Part 1).
|
|
28
|
+
|
|
29
|
+
---
|
|
30
|
+
|
|
31
|
+
# Part 1 — Where a message goes when comq gives up on it
|
|
32
|
+
|
|
33
|
+
## The problem
|
|
34
|
+
|
|
35
|
+
The last failed attempt destroys the message. `#discard` ([:406](../source/channel.js#L406)) is:
|
|
36
|
+
|
|
37
|
+
```js
|
|
38
|
+
this.#channel.nack(message, false, false)
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
`requeue: false` means "the broker may dead-letter this" — but only if the queue was declared
|
|
42
|
+
with a dead-letter exchange, and **comq never declares one**. `grep` for `x-dead-letter` or
|
|
43
|
+
`arguments:` across `source/` returns nothing; `#assertQueue`
|
|
44
|
+
([:246-253](../source/channel.js#L246)) passes only `{ durable: true }` or `{ exclusive: true }`.
|
|
45
|
+
So the message is silently deleted. [readme.md:468](../readme.md#L468) tells the reader to
|
|
46
|
+
configure a policy on the broker by hand — a library whose documented failure mode is silent
|
|
47
|
+
deletion.
|
|
48
|
+
|
|
49
|
+
## The mechanism: comq parks it in a queue it owns
|
|
50
|
+
|
|
51
|
+
On the last attempt, comq publishes the message to `comq.parked.<queue>`, waits for the publisher
|
|
52
|
+
confirm, and only then acks the original.
|
|
53
|
+
|
|
54
|
+
**This is about retention, not delivery.** The message is not going to be processed again — that
|
|
55
|
+
is settled. What is at stake is whether it is still *there* afterwards. A message that has
|
|
56
|
+
exhausted its attempts is evidence that something is wrong: a bad producer, a schema change, a
|
|
57
|
+
bug in the consumer. Today comq deletes that evidence. The requirement is that when comq says it
|
|
58
|
+
has kept the message, that is true and not merely likely.
|
|
59
|
+
|
|
60
|
+
Publishing before acking is what makes it true: the broker holds the parked copy before it
|
|
61
|
+
releases the original, so there is no window in which it holds neither. If comq dies between the
|
|
62
|
+
two, the original is redelivered and parked again — two copies of the same evidence in a queue
|
|
63
|
+
nothing consumes, which is harmless. (Consumers do need to be idempotent, but because of the
|
|
64
|
+
retry path and redelivery generally, not because of this.)
|
|
65
|
+
|
|
66
|
+
> ### Two mechanisms, easily confused
|
|
67
|
+
>
|
|
68
|
+
> | | **Part 2 — retry** | **Part 1 — parking** |
|
|
69
|
+
> |---|---|---|
|
|
70
|
+
> | when | attempts 1–5 | after the 6th failure |
|
|
71
|
+
> | queue | the shared retry queue | `comq.parked.<queue>` |
|
|
72
|
+
> | how it gets there | comq publishes it | comq publishes it |
|
|
73
|
+
> | what happens next | the broker returns it to the source queue after the TTL | nothing — it waits for a human |
|
|
74
|
+
> | purpose | try again later | keep the evidence |
|
|
75
|
+
>
|
|
76
|
+
> Both are new queues comq declares, and **dead-lettering appears in both doing different
|
|
77
|
+
> jobs**: in Part 2 it is the routing trick that carries a message *back* to its source when the
|
|
78
|
+
> TTL fires; in Part 1 it is the rejected way of moving a message to its final resting place.
|
|
79
|
+
|
|
80
|
+
## Why RabbitMQ's dead-lettering is not the mechanism
|
|
81
|
+
|
|
82
|
+
The obvious alternative is to declare `x-dead-letter-exchange` on the source queue and let the
|
|
83
|
+
broker move the message. It was considered and rejected, for a reason worth recording because
|
|
84
|
+
the readme currently recommends exactly this to users.
|
|
85
|
+
|
|
86
|
+
**Dead-lettering is a routing rule, not a retention guarantee.** It is what the broker does with
|
|
87
|
+
messages it has already given up on, and best-effort is what it was designed to be. RabbitMQ
|
|
88
|
+
republishes to the DLX *internally, without publisher confirms*: a dead-lettered message can be
|
|
89
|
+
lost if the target queue is unavailable, if its length limit is reached under `reject-publish`
|
|
90
|
+
overflow, or across a network partition. If the dead-letter exchange does not exist, messages
|
|
91
|
+
are silently dropped.
|
|
92
|
+
|
|
93
|
+
At-least-once dead-lettering does exist, but requires **quorum queues** plus
|
|
94
|
+
`dead-letter-strategy: at-least-once` plus `overflow: reject-publish`. **Classic queues do not
|
|
95
|
+
support it**, and comq declares classic queues — `#assertQueue` passes `{ durable: true }` with
|
|
96
|
+
no `x-queue-type`.
|
|
97
|
+
|
|
98
|
+
So this was never a choice between two implementations of one thing. We need the message kept;
|
|
99
|
+
dead-lettering does not promise to keep it. Using it would reintroduce silent loss on the exact
|
|
100
|
+
path built to prevent it — narrower than today, but the same class of bug.
|
|
101
|
+
|
|
102
|
+
- [Dead Letter Exchanges](https://www.rabbitmq.com/docs/dlx)
|
|
103
|
+
- [At-Least-Once Dead Lettering](https://www.rabbitmq.com/blog/2022/03/29/at-least-once-dead-lettering)
|
|
104
|
+
- [Quorum Queues](https://www.rabbitmq.com/docs/quorum-queues)
|
|
105
|
+
|
|
106
|
+
> **If comq ever moves to quorum queues** — with `dead-letter-strategy: at-least-once` and
|
|
107
|
+
> `overflow: reject-publish` — dead-lettering becomes lossless and this decision is worth
|
|
108
|
+
> revisiting. It would also close the Part 2 retry queue's caveat. Much larger change; its own
|
|
109
|
+
> decision, someday.
|
|
110
|
+
|
|
111
|
+
## The compatibility trap this avoids
|
|
112
|
+
|
|
113
|
+
Worth stating explicitly, because it is the reason the rejected alternative is *also* expensive:
|
|
114
|
+
declaring `x-dead-letter-exchange` means adding `arguments` to `#assertQueue`. Every durable
|
|
115
|
+
queue in every comq deployment today was declared **without** `arguments`, and RabbitMQ rejects
|
|
116
|
+
a redeclaration whose arguments differ with `406 PRECONDITION_FAILED`.
|
|
117
|
+
|
|
118
|
+
That failure would be loud rather than silent — the declaration happens inside the
|
|
119
|
+
`failsafe(this, this.#recover, …)` wrapper, `permanent()` ([:463](../source/channel.js#L463))
|
|
120
|
+
classifies `PRECONDITION_FAILED` as unrecoverable, and `io.consume(...)` rejects at wiring
|
|
121
|
+
time — but "the application does not start after the upgrade" is still a migration for every
|
|
122
|
+
user. Parking sidesteps it entirely: `comq.parked.<queue>` is a **new** queue, and the source
|
|
123
|
+
queue's declaration is untouched.
|
|
124
|
+
|
|
125
|
+
A migration would have been acceptable, so this is not what decided it — retention is. But it is a real saving, and it keeps the destination soft: a publish target can
|
|
126
|
+
be renamed or changed at any time, where `arguments` on a durable queue are permanent and every
|
|
127
|
+
later change is another migration.
|
|
128
|
+
|
|
129
|
+
## What the parked message carries
|
|
130
|
+
|
|
131
|
+
Only what a person doing a post-mortem needs. There is no compatibility requirement — nothing
|
|
132
|
+
reads these but a human — so do not reproduce RabbitMQ's `x-death` format, and do not build
|
|
133
|
+
multi-hop machinery (`x-first-death-*` / `x-last-death-*`): a comq message parks once.
|
|
134
|
+
|
|
135
|
+
**The origin must be captured on the first failure, not at parking time.** This is the one
|
|
136
|
+
non-obvious part, and it was got wrong in an earlier draft. A message that has been retried
|
|
137
|
+
comes back through the *default* exchange, so by the time it is parked:
|
|
138
|
+
|
|
139
|
+
```json
|
|
140
|
+
"exchange": "",
|
|
141
|
+
"routingKey": "orders..billing",
|
|
142
|
+
"headers": {
|
|
143
|
+
"x-comq-attempt": 1,
|
|
144
|
+
"x-death": [{ "reason": "expired", "queue": "comq.retry.30000",
|
|
145
|
+
"exchange": "comq.retry.30000", "routing-keys": ["orders..billing"] }],
|
|
146
|
+
"x-first-death-exchange": "comq.retry.30000"
|
|
147
|
+
}
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
`message.fields.exchange` is `''` — it has to be — and `fields.routingKey` is now the source
|
|
151
|
+
queue name. So reading `fields` at parking time would record `''` as the origin of **every**
|
|
152
|
+
parked message that was ever retried, which is all of them except one parked by a first-delivery
|
|
153
|
+
`Park`. The broker does not rescue it either: `x-death[0].exchange` and `x-first-death-exchange`
|
|
154
|
+
both name the *retry* exchange. The original is nowhere unless comq writes it down.
|
|
155
|
+
|
|
156
|
+
So `#retry` writes `x-comq-exchange` / `x-comq-key` on the first failure and later retries
|
|
157
|
+
preserve them (Part 2 §4), and `#dispose` prefers those headers, falling back to `message.fields`
|
|
158
|
+
only for the first-delivery `Park` path where nothing has been written yet:
|
|
159
|
+
|
|
160
|
+
```js
|
|
161
|
+
const headers = {
|
|
162
|
+
...message.properties.headers, // x-comq-attempt and, after one retry, the origin
|
|
163
|
+
'x-comq-queue': queue, // threaded in by Part 2 §3
|
|
164
|
+
'x-comq-exchange': message.properties.headers?.['x-comq-exchange'] ?? message.fields.exchange,
|
|
165
|
+
'x-comq-key': message.properties.headers?.['x-comq-key'] ?? message.fields.routingKey,
|
|
166
|
+
'x-comq-reason': exception?.message,
|
|
167
|
+
'x-comq-at': Date.now()
|
|
168
|
+
}
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
**Two attempt counters will be on the message, and the docs must say which is authoritative.**
|
|
172
|
+
RabbitMQ maintains `x-death[0].count`, and it counts *deaths* where `x-comq-attempt` counts
|
|
173
|
+
*deliveries*: on a parked message with the default settings they read `4` and `5`. They will not
|
|
174
|
+
agree, and a move to quorum queues changes `x-death` semantics besides. `x-comq-attempt` is
|
|
175
|
+
authoritative — it is comq's, and it is what the ladder is measured against; `x-death` is the
|
|
176
|
+
broker's own record, useful for its timestamps.
|
|
177
|
+
|
|
178
|
+
|
|
179
|
+
## The seam
|
|
180
|
+
|
|
181
|
+
Route the terminal branch through one method, so the destination is one function body:
|
|
182
|
+
|
|
183
|
+
```js
|
|
184
|
+
async #park (queue, message, exception) {
|
|
185
|
+
await this.#dispose(queue, message, exception)
|
|
186
|
+
this.#diagnostics.emit('discard', message, exception)
|
|
187
|
+
}
|
|
188
|
+
|
|
189
|
+
// Where a message that has run out of attempts goes: a queue comq owns, published to
|
|
190
|
+
// before the original is released, so a message the broker holds twice is recoverable
|
|
191
|
+
// and one it no longer holds at all is not.
|
|
192
|
+
async #dispose (queue, message, exception) {
|
|
193
|
+
const headers = { /* as above */ }
|
|
194
|
+
const properties = { ...message.properties, headers, mandatory: true, persistent: true }
|
|
195
|
+
|
|
196
|
+
await this.#publish(DEFAULT, parkedQueueOf(queue), message.content, properties)
|
|
197
|
+
this.#channel.ack(message)
|
|
198
|
+
}
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
`#dispose` is `async` and every caller sits inside `#failed`'s try/catch (Part 2 §4), so it
|
|
202
|
+
inherits the "cannot reject into amqplib" guarantee for free: if the parking publish fails, the
|
|
203
|
+
message is nacked back to the broker rather than lost.
|
|
204
|
+
|
|
205
|
+
The parking queue is per source queue — depth per consumer is how anyone notices a problem — and
|
|
206
|
+
is declared next to the shared retry topology in `#consume` (Part 2 §3), mirroring the source
|
|
207
|
+
queue's durability. The diagnostic keeps its existing name, `discard`, so nothing downstream
|
|
208
|
+
breaks.
|
|
209
|
+
|
|
210
|
+
## Parked Requests: what is kept, and what is still not answered
|
|
211
|
+
|
|
212
|
+
A parked Request is preserved; its caller is still not answered. The promise never settles —
|
|
213
|
+
the prefetch deadlock [readme.md:469](../readme.md#L469) already documents — and nothing here
|
|
214
|
+
changes that. Parking makes the request *recoverable*, not *answered*.
|
|
215
|
+
|
|
216
|
+
**The parked copy must be published `persistent: true` regardless of the source topology.**
|
|
217
|
+
`source/topology/request.json` is `persistent: false`, so a parked Request would otherwise be
|
|
218
|
+
delivery-mode 1: sitting in a durable queue, but dropped by a broker restart. Requests are
|
|
219
|
+
non-persistent for latency and the failure path is not the latency path — parking is rare and
|
|
220
|
+
terminal, so the disk write costs nothing. The same applies to the retry copy, where a broker
|
|
221
|
+
restart during the TTL would otherwise lose a retried Request. Both `#retry` and `#dispose`
|
|
222
|
+
therefore set `persistent: true` explicitly rather than inheriting it.
|
|
223
|
+
|
|
224
|
+
(The remaining weakness on that channel is that `confirms: false` means the parking publish gets
|
|
225
|
+
no publisher confirm. Channel command ordering still has the broker process the publish before
|
|
226
|
+
the ack, so this is not a loss race — but there is no positive acknowledgement that the broker
|
|
227
|
+
took it. Fixing that means confirms on the request channel, which is a throughput change well
|
|
228
|
+
outside this work.)
|
|
229
|
+
|
|
230
|
+
**A parked Request carries enough to answer it later.** `#createRequest`
|
|
231
|
+
([io.js:365](../source/io.js#L365)) sets `correlationId` and `replyTo` in the message properties,
|
|
232
|
+
and both `#retry` and `#dispose` spread `...message.properties`, so both survive into the
|
|
233
|
+
parking queue. Whether an answer would *land* depends on the caller:
|
|
234
|
+
|
|
235
|
+
- **Still running** — and it will be, hanging indefinitely, which is the case that matters. Its
|
|
236
|
+
reply queue is still declared, so a reply published to `replyTo` with that `correlationId`
|
|
237
|
+
would resolve the hung promise.
|
|
238
|
+
- **Restarted** — reply queues are `<queue>..<16 random hex>`
|
|
239
|
+
([createReplyEmitter.js:33](../source/.io/createReplyEmitter.js#L33)) declared *exclusive*
|
|
240
|
+
(`reply.json` is `durable: false`, so `#assertQueue` uses `EXCLUSIVE`), so it died with the
|
|
241
|
+
old connection. The reply is unroutable, and because `#reply` sets `mandatory: true` the broker
|
|
242
|
+
returns it and the `return` diagnostic fires rather than it vanishing silently.
|
|
243
|
+
|
|
244
|
+
`correlationId` carries a per-process prefix, so it is unique across processes and a replay
|
|
245
|
+
cannot cross wires. None of this is built here — but it is worth recording that the door is
|
|
246
|
+
open, because it turns the deadlock from permanent into merely manual.
|
|
247
|
+
|
|
248
|
+
## The one thing this does not fix
|
|
249
|
+
|
|
250
|
+
A parked message is out of the broker's way but still needs someone to look at it.
|
|
251
|
+
`comq.parked.*` queues grow without bound unless an operator drains them. That is the right
|
|
252
|
+
default — the alternative is deleting evidence — but the readme must say so, and the `discard`
|
|
253
|
+
diagnostic is the hook for alerting on it.
|
|
254
|
+
|
|
255
|
+
# Part 2 — The remaining fixes
|
|
256
|
+
|
|
257
|
+
None of these change an existing queue declaration, so none of them break an upgrade.
|
|
258
|
+
|
|
259
|
+
## Design
|
|
260
|
+
|
|
261
|
+
### The retry loop
|
|
262
|
+
|
|
263
|
+
On failure, comq publishes the message to a **shared retry queue**: a queue with no consumer,
|
|
264
|
+
holding it for a TTL and then dead-lettering it back to the queue it came from. Only after the
|
|
265
|
+
publish is confirmed is the original acked.
|
|
266
|
+
|
|
267
|
+
```
|
|
268
|
+
exchange comq.retry.30000 (fanout)
|
|
269
|
+
queue comq.retry.30000 bound to it
|
|
270
|
+
x-message-ttl: 30000
|
|
271
|
+
x-dead-letter-exchange: ''
|
|
272
|
+
(no x-dead-letter-routing-key — the original key is preserved)
|
|
273
|
+
|
|
274
|
+
publish → exchange comq.retry.30000, routingKey = <source queue>
|
|
275
|
+
expiry → default exchange, routing key preserved → <source queue>
|
|
276
|
+
```
|
|
277
|
+
|
|
278
|
+
**How one queue returns messages to many destinations.** The routing key is a property of the
|
|
279
|
+
*message*, not of the queue: it is set at publish time, travels with the message, and RabbitMQ
|
|
280
|
+
reuses it when dead-lettering. Tracing an event that failed in `orders..billing`:
|
|
281
|
+
|
|
282
|
+
| step | exchange | routingKey | lands in |
|
|
283
|
+
|---|---|---|---|
|
|
284
|
+
| comq publishes the retry | `comq.retry.30000` (fanout) | `orders..billing` | `comq.retry.30000` |
|
|
285
|
+
| waits out the TTL | — | `orders..billing` (carried) | — |
|
|
286
|
+
| TTL fires, dead-lettered | `''` (default) | `orders..billing` (**reused**) | `orders..billing` |
|
|
287
|
+
|
|
288
|
+
Three broker behaviours carry it:
|
|
289
|
+
|
|
290
|
+
1. **A fanout exchange ignores the routing key when routing** — it delivers to whatever is bound,
|
|
291
|
+
which is the single retry queue — but preserves the key on the message. The key is therefore
|
|
292
|
+
free to name the destination rather than being consumed by the routing.
|
|
293
|
+
2. **No `x-dead-letter-routing-key` on the retry queue.** Set, that argument *overrides* the
|
|
294
|
+
message's key with one fixed value — which is precisely what makes a retry queue per-source.
|
|
295
|
+
Unset, the broker reuses whatever key the message already carries, so each message returns to
|
|
296
|
+
its own source queue.
|
|
297
|
+
3. **The default exchange has an implicit binding to every queue by name**, so a key of
|
|
298
|
+
`orders..billing` reaches the queue `orders..billing`. Standard AMQP; comq declares nothing
|
|
299
|
+
for it.
|
|
300
|
+
|
|
301
|
+
This is also the mechanical reason the message cannot be published to the retry queue directly:
|
|
302
|
+
through the default exchange its key would be the retry queue's own name, and on expiry it would
|
|
303
|
+
route back to itself — the cycle the broker drops silently at the first TTL.
|
|
304
|
+
|
|
305
|
+
And it is why the origin has to be captured on the first failure: after the round-trip the
|
|
306
|
+
message's `fields.exchange` is `''` and its `fields.routingKey` is the source queue, so the
|
|
307
|
+
exchange it was originally published to is genuinely gone from the message.
|
|
308
|
+
|
|
309
|
+
One queue and one exchange **per distinct delay**, shared by every source queue — not one pair
|
|
310
|
+
per consumed queue. The delay comes from the channel topology, so the count does not depend on
|
|
311
|
+
how many queues are consumed.
|
|
312
|
+
|
|
313
|
+
Worked through, because the rule alone is easy to misread. An application with 100 request
|
|
314
|
+
endpoints and 100 consumed events:
|
|
315
|
+
|
|
316
|
+
| | count | scales with |
|
|
317
|
+
|---|---|---|
|
|
318
|
+
| source queues (unchanged, exist today) | 200 | endpoints + events |
|
|
319
|
+
| **retry queues** | **2** (+ 2 exchanges) | distinct `delay` values — *not* queue count |
|
|
320
|
+
| parking queues | 200 | one per source queue |
|
|
321
|
+
|
|
322
|
+
The 100 endpoints share `comq.retry.5000` and the 100 events share `comq.retry.30000`; add a
|
|
323
|
+
thousand more of each and it is still two. A per-queue retry design would have made it 400 new
|
|
324
|
+
queues instead of 202.
|
|
325
|
+
|
|
326
|
+
Three things do move the retry number, and nothing else does: overriding `delay` (the count is
|
|
327
|
+
distinct values in use), a sharded connection (each shard is its own broker with its own pair),
|
|
328
|
+
and nothing further — more services, replicas and connections all declare the same names, and
|
|
329
|
+
`assertQueue` is idempotent.
|
|
330
|
+
|
|
331
|
+
**The fanout exchange is not optional.** Publishing to the retry queue through the default
|
|
332
|
+
exchange would make the message's own routing key the retry queue's name, so on expiry it would
|
|
333
|
+
dead-letter straight back into itself. RabbitMQ does not loop on that — it detects the cycle
|
|
334
|
+
and, since no rejection occurred in it, drops the message. Verified against a broker: the
|
|
335
|
+
message is gone at the **first** expiry, at exactly one TTL, not after a lap or two. So the
|
|
336
|
+
naive shape fails as "every retried message vanishes silently after one TTL" — no loop to
|
|
337
|
+
notice, no queue growth to alert on. A reader who half-remembers that RabbitMQ detects cycles
|
|
338
|
+
may expect something visible to happen; nothing does.
|
|
339
|
+
|
|
340
|
+
Publishing through a fanout exchange leaves the routing key free to name the destination. A
|
|
341
|
+
direct exchange would need a binding per source queue, which is the proliferation being removed.
|
|
342
|
+
|
|
343
|
+
**Naming: `comq.retry.<delay>`, and the delay is in the name deliberately.** `x-message-ttl` is
|
|
344
|
+
fixed at declare time, so a queue whose name did not carry it could never have that value
|
|
345
|
+
changed — every redeclaration with a different TTL is `406 PRECONDITION_FAILED`, and by the
|
|
346
|
+
placement below that means the application does not start. With the delay in the name, a change
|
|
347
|
+
declares a *new* queue and the old one keeps working: during a rolling deploy, old pods publish
|
|
348
|
+
to the old queue and new pods to the new one, and both dead-letter back to the same unchanged
|
|
349
|
+
source queue. There is no window at all.
|
|
350
|
+
|
|
351
|
+
**Orphans stay, and `x-expires` is not the cleanup.** A retired delay leaves an empty queue
|
|
352
|
+
behind. `x-expires` looks like the answer and is not: a retry queue never has a consumer, so
|
|
353
|
+
"unused" is its permanent state, and the timer would delete it along with whatever is waiting
|
|
354
|
+
inside. An empty queue per retired delay value is the price, and it is a housekeeping item
|
|
355
|
+
rather than a hazard.
|
|
356
|
+
|
|
357
|
+
**Declared at consume time, not on first failure.** Declaring inside the catch block means a
|
|
358
|
+
declaration error surfaces at 3am under load, and a failure there falls through to
|
|
359
|
+
`nack(requeue: true)` → immediate redelivery → immediate re-failure → a hot loop with no delay
|
|
360
|
+
and no attempt counting. Declaring in `#consume` puts it inside the
|
|
361
|
+
`failsafe(this, this.#recover, …)` wrapper, where `io.consume(...)` rejects at wiring time with
|
|
362
|
+
a legible error, and it is re-declared for free on `recover()` (which does `lazy.reset` +
|
|
363
|
+
`recall(this)`).
|
|
364
|
+
|
|
365
|
+
**Sharing centralises the blast radius.** Delete or misconfigure `comq.retry.30000` and every
|
|
366
|
+
retry in the application stops, where a per-queue design would lose one queue's. Re-declaring at
|
|
367
|
+
consume time is the mitigation; the documentation line that already says not to delete these
|
|
368
|
+
queues on a running system covers it.
|
|
369
|
+
|
|
370
|
+
**Durability and exclusive source queues.** The shared queue is always durable, and that is
|
|
371
|
+
always right: every topology that acknowledges is also durable (`request.json` and `event.json`
|
|
372
|
+
are both `durable: true`; `reply.json` is `acknowledgments: false` and has no retry path).
|
|
373
|
+
An *exclusive* source queue — from `io.consume(exchange, callback)` with no group — is the one
|
|
374
|
+
case worth tracing. While that ephemeral subscriber is alive the retry is delivered normally.
|
|
375
|
+
Once its connection is gone there is nobody to deliver to, and the message is dropped at expiry
|
|
376
|
+
because the default exchange finds no such queue. A per-queue retry design loses it too, at
|
|
377
|
+
disconnect rather than at expiry. Neither can do better, so sharing costs nothing here.
|
|
378
|
+
|
|
379
|
+
**Not per-message TTL.** RabbitMQ discards an expired message only once it reaches the head of
|
|
380
|
+
its queue, so a long wait would hold up every shorter one behind it. The wait therefore has to
|
|
381
|
+
be a property of the queue — which is why a backoff ladder is a queue per rung rather than an
|
|
382
|
+
`expiration` per message.
|
|
383
|
+
|
|
384
|
+
**There is no separate attempt count.** `delay` is a **ladder with one rung per retry**, so its
|
|
385
|
+
length decides how many there are and four rungs is five attempts. `MAX_REDELIVERIES`
|
|
386
|
+
([:481](../source/channel.js#L481)) disappears into it: a retry and its wait are the same event,
|
|
387
|
+
and a count kept beside the ladder would be a second source of truth free to disagree with it.
|
|
388
|
+
|
|
389
|
+
The header is the attempt number, counting from one, and the first delivery does not carry it: a
|
|
390
|
+
consumer reads `headers?.['x-comq-attempt'] ?? 1`.
|
|
391
|
+
|
|
392
|
+
The original code kept a `MAX_REDELIVERIES` named for attempts while counting retries, which made
|
|
393
|
+
`5` mean six deliveries and left [readme.md:466](../readme.md#L466) — "causes exceptions five
|
|
394
|
+
times in a row, it is discarded" — off by one.
|
|
395
|
+
|
|
396
|
+
> **The retry queue inherits the at-most-once caveat described in Part 1.** The return hop is
|
|
397
|
+
> broker dead-lettering, which on classic queues republishes without publisher confirms — so a
|
|
398
|
+
> retried message can be lost if the source queue is unavailable when the TTL fires (target
|
|
399
|
+
> unavailable, length limit under `reject-publish`, or a network partition between the nodes
|
|
400
|
+
> hosting the two queues). Still a large improvement on today's ack-before-publish loss, and the
|
|
401
|
+
> alternative — an in-process `setTimeout` — is worse, since it loses the delay on any restart.
|
|
402
|
+
> A known limitation; it belongs in the docs (§8), not left implicit. Moving comq to quorum
|
|
403
|
+
> queues would close it; see Part 1.
|
|
404
|
+
|
|
405
|
+
### The backoff ladder is public
|
|
406
|
+
|
|
407
|
+
`delay` is a `Topology` field, defaulted per channel type and overridable per deployment. It is
|
|
408
|
+
a list with **one rung per retry**, so its length is the number of retries and there is no
|
|
409
|
+
separate count to fall out of step with it.
|
|
410
|
+
|
|
411
|
+
| | ladder | attempts | total |
|
|
412
|
+
|---|---|---|---|
|
|
413
|
+
| `event` | 1s, 10s, 30s, 90s | 5 | 131s |
|
|
414
|
+
| `request` | 1s, 3s, 5s, 10s | 5 | 19s |
|
|
415
|
+
|
|
416
|
+
**Events.** A fast first rung catches a momentary blip without making it wait out a long one,
|
|
417
|
+
and the ladder then climbs toward the failures worth retrying — a database primary stepping down,
|
|
418
|
+
a storage reconnect, a third-party blip, and a component mid-rollout. That last one is the most
|
|
419
|
+
common and the easiest to misplace: on the event path "nothing is listening yet" is not a failure
|
|
420
|
+
at all, since the message waits in a durable queue. It is reached *indirectly* — a receiver's
|
|
421
|
+
handler calls a component that is mid-rollout, the call raises, the handler throws, the event is
|
|
422
|
+
retried — and a rollout with `maxUnavailable: 0` across several replicas is minutes rather than
|
|
423
|
+
seconds. A ladder covering only a few seconds parks everything that was merely slow, which makes
|
|
424
|
+
the parking queue mean "something was briefly slow" instead of "something is wrong".
|
|
425
|
+
|
|
426
|
+
**Requests are shorter, and deliberately so.** The failure durations are identical; what differs
|
|
427
|
+
is that a request has someone blocked on it with no timeout. Exhausting the attempts means the
|
|
428
|
+
caller gets **no answer at all**, ever — so giving up early is not a cheaper failure, it is an
|
|
429
|
+
unbounded one. What bounds how much coverage to buy is that a reply arriving long after the
|
|
430
|
+
caller gave up lands in an exclusive reply queue that may no longer exist, and comes back
|
|
431
|
+
unroutable.
|
|
432
|
+
|
|
433
|
+
Different defaults per channel type are free: `event.json` and `request.json` already differ on
|
|
434
|
+
four other settings. The two ladders share the rungs `1000` and `10000`, so between them they
|
|
435
|
+
mean **six** retry queues and six exchanges for a whole application — and that number follows the
|
|
436
|
+
rungs, not the number of queues consumed.
|
|
437
|
+
|
|
438
|
+
And it removes a constraint that should never have set a production number: the feature suite
|
|
439
|
+
can run at 100ms without either default being chosen for it. The original `1000` was picked for
|
|
440
|
+
the suite's wall clock, which is the wrong reason for a production constant.
|
|
441
|
+
|
|
442
|
+
|
|
443
|
+
### Nothing in the failure path may reject
|
|
444
|
+
|
|
445
|
+
amqplib has nowhere to put a rejection, so every path out of the consumer wrapper must be
|
|
446
|
+
total. Fallback is `nack(message, false, true)` — hand the delivery back to the broker rather
|
|
447
|
+
than lose it — itself wrapped so it can't throw either.
|
|
448
|
+
|
|
449
|
+
`#publish` can reject three ways and all three correctly resolve to "give it back":
|
|
450
|
+
`'Channel closed'` from a dying channel, `INTERRUPTION` ([:479](../source/channel.js#L479)) from
|
|
451
|
+
`#unpaused()` on a failfast/sharded channel, and confirmation rejection during `recover()`
|
|
452
|
+
([:230](../source/channel.js#L230)).
|
|
453
|
+
|
|
454
|
+
Two extra defects found while designing, fixed in the same pass:
|
|
455
|
+
- [:377](../source/channel.js#L377) `exception.message` throws if a consumer does `throw undefined`
|
|
456
|
+
→ use `exception?.message`.
|
|
457
|
+
- [:381](../source/channel.js#L381) `this.#discard(...)` is not awaited; once it becomes the
|
|
458
|
+
dead-letter seam (Part 1), an unawaited rejection there escapes.
|
|
459
|
+
|
|
460
|
+
### Consequences to accept and document
|
|
461
|
+
|
|
462
|
+
- **Ordering is not preserved across a failure.** A retried message re-enters the queue behind
|
|
463
|
+
messages published while it waited. (Ordering was already not guaranteed — `prefetch: 300`
|
|
464
|
+
means up to 300 concurrent in-flight deliveries.) Retries do not reorder among themselves.
|
|
465
|
+
- **At-least-once, not exactly-once.** Publish-before-ack means a crash between the two leaves
|
|
466
|
+
the broker holding both the retry copy and the unacked original. Consumers must be
|
|
467
|
+
idempotent. This is the deliberate trade against today's at-most-once silent loss.
|
|
468
|
+
- **A retry published to a deleted queue is silently dropped**, and publisher confirms still
|
|
469
|
+
ack it. `mandatory: true` makes the broker return it so the `return` diagnostic fires, but
|
|
470
|
+
asynchronously. Document "do not delete `comq.retry.*` on a running system".
|
|
471
|
+
- **`x-comq-attempt` becomes user-visible** — consumers receive it in `properties.headers` and can
|
|
472
|
+
tell which attempt they're on. Worth documenting as a feature. RabbitMQ also appends
|
|
473
|
+
`x-death` on the way back and rewrites `routingKey` to the source queue name.
|
|
474
|
+
|
|
475
|
+
## Implementation
|
|
476
|
+
|
|
477
|
+
All in [source/channel.js](../source/channel.js) unless noted.
|
|
478
|
+
|
|
479
|
+
### 1. Record each queue's declaration options
|
|
480
|
+
|
|
481
|
+
The parking queue must mirror its source queue's lifetime — a durable parking queue behind an
|
|
482
|
+
exclusive `amq.gen-*` source leaks a durable queue nothing will ever read. `#consume` cannot
|
|
483
|
+
re-derive this, because only `#assertBoundQueue` ([:303](../source/channel.js#L303)) passes the
|
|
484
|
+
`{ exclusive: true }` override. Record it where it is known:
|
|
485
|
+
|
|
486
|
+
- new field `#queues = new Map()` next to `#tags` ([:26](../source/channel.js#L26))
|
|
487
|
+
- `this.#queues.clear()` in `create()` next to `this.#tags = []` ([:72](../source/channel.js#L72))
|
|
488
|
+
— a fresh amqplib channel has declared nothing
|
|
489
|
+
- `this.#queues.set(queue, options)` in `#assertQueue` ([:250](../source/channel.js#L250))
|
|
490
|
+
|
|
491
|
+
(The retry queue does not need this: it is shared and always durable, per the Design section.)
|
|
492
|
+
|
|
493
|
+
### 2. The retry and parking topology
|
|
494
|
+
|
|
495
|
+
The retry exchange and queue are per *delay*, so they are declared once per distinct value
|
|
496
|
+
rather than once per source queue. `#assertRetryQueue` takes no queue argument, which is what
|
|
497
|
+
makes `lazy`'s per-argument memo collapse them:
|
|
498
|
+
|
|
499
|
+
```js
|
|
500
|
+
async #assertRetryQueue () {
|
|
501
|
+
const delay = this.#topology.delay
|
|
502
|
+
const name = RETRY_PREFIX + delay
|
|
503
|
+
|
|
504
|
+
// fanout, so the routing key stays free to name the destination. Publishing through
|
|
505
|
+
// the default exchange instead would make the message's own routing key this queue's
|
|
506
|
+
// name, and the broker would drop it as a cycle at the first expiry.
|
|
507
|
+
await this.#channel.assertExchange(name, 'fanout', DURABLE)
|
|
508
|
+
|
|
509
|
+
await this.#assertQueue(name, {
|
|
510
|
+
...DURABLE,
|
|
511
|
+
arguments: {
|
|
512
|
+
'x-message-ttl': delay,
|
|
513
|
+
'x-dead-letter-exchange': DEFAULT
|
|
514
|
+
// no x-dead-letter-routing-key: the original key is preserved, and it names
|
|
515
|
+
// the source queue because that is what we published with
|
|
516
|
+
}
|
|
517
|
+
})
|
|
518
|
+
|
|
519
|
+
await this.#channel.bindQueue(name, name, '')
|
|
520
|
+
|
|
521
|
+
return name
|
|
522
|
+
}
|
|
523
|
+
|
|
524
|
+
async #assertParkedQueue (queue) {
|
|
525
|
+
const options = this.#queues.get(queue) ?? (this.#topology.durable ? DURABLE : EXCLUSIVE)
|
|
526
|
+
|
|
527
|
+
return (await this.#assertQueue(parkedQueueOf(queue), options))[0]
|
|
528
|
+
}
|
|
529
|
+
```
|
|
530
|
+
|
|
531
|
+
Plus `RETRY_PREFIX = 'comq.retry.'`, `PARKED_PREFIX = 'comq.parked.'` and
|
|
532
|
+
`parkedQueueOf = (queue) => PARKED_PREFIX + queue`, where `MAX_REDELIVERIES` sits today
|
|
533
|
+
([:481](../source/channel.js#L481)) — that constant moves into the topology presets as the
|
|
534
|
+
`RETRY_DELAY` never becomes a constant at all; it is `topology.delay` from the start, and
|
|
535
|
+
`MAX_REDELIVERIES` disappears into the ladder's length.
|
|
536
|
+
|
|
537
|
+
### 3. `#consume` declares them and threads the queue name
|
|
538
|
+
|
|
539
|
+
The queue name is fully resolved by `#consume` for all three paths — `lazy` splices the
|
|
540
|
+
initializer's return over `args` before the method body runs, so `consume`→`#assertQueue`,
|
|
541
|
+
`subscribe`→`#assertBoundQueue`, `bound`→`#assertKeyedQueue` all deliver the broker-confirmed
|
|
542
|
+
name (including generated `amq.gen-*` names for the exclusive case, `io.js:112`).
|
|
543
|
+
|
|
544
|
+
```js
|
|
545
|
+
// :352
|
|
546
|
+
if (this.#topology.acknowledgments) {
|
|
547
|
+
await this.#assertRetryQueue()
|
|
548
|
+
await this.#assertParkedQueue(queue)
|
|
549
|
+
consumer = this.#getAcknowledgingConsumer(queue, consumer)
|
|
550
|
+
} else options.noAck = true
|
|
551
|
+
```
|
|
552
|
+
|
|
553
|
+
`#getAcknowledgingConsumer` becomes `(queue, consumer) => async (message) => {...}`; single call
|
|
554
|
+
site at [:356](../source/channel.js#L356). Declare the source queue **before** the others —
|
|
555
|
+
existing tests at [test/channel.test.js:386](../test/channel.test.js#L386) and `:407` read
|
|
556
|
+
`assertQueue.mock.results[0]`.
|
|
557
|
+
|
|
558
|
+
|
|
559
|
+
### 4. Rewrite the failure path
|
|
560
|
+
|
|
561
|
+
Replace `#getAcknowledgingConsumer` / `#requeue` ([:370-400](../source/channel.js#L370)) with:
|
|
562
|
+
|
|
563
|
+
```js
|
|
564
|
+
#getAcknowledgingConsumer = (queue, consumer) =>
|
|
565
|
+
async (message) => {
|
|
566
|
+
try {
|
|
567
|
+
await consumer(message)
|
|
568
|
+
this.#channel.ack(message)
|
|
569
|
+
} catch (exception) {
|
|
570
|
+
if (exception?.message === 'Channel closed') return // the broker requeues it
|
|
571
|
+
await this.#failed(queue, message, exception)
|
|
572
|
+
}
|
|
573
|
+
}
|
|
574
|
+
|
|
575
|
+
// Nothing here may reject: amqplib dispatches a delivery through an event emitter and
|
|
576
|
+
// drops the promise it gets back, so a rejection has no one to catch it and takes the
|
|
577
|
+
// process down.
|
|
578
|
+
async #failed (queue, message, exception) {
|
|
579
|
+
try {
|
|
580
|
+
const attempt = message.properties.headers?.[ATTEMPT_HEADER] ?? 0
|
|
581
|
+
|
|
582
|
+
if (attempt >= this.#topology.attempts) await this.#park(queue, message, exception)
|
|
583
|
+
else await this.#retry(queue, message, attempt, exception)
|
|
584
|
+
} catch {
|
|
585
|
+
this.#nack(message) // hand it back rather than lose it
|
|
586
|
+
}
|
|
587
|
+
}
|
|
588
|
+
|
|
589
|
+
async #retry (queue, message, attempt, exception) {
|
|
590
|
+
const headers = {
|
|
591
|
+
// the origin is recorded on the first failure and carried from then on: after one
|
|
592
|
+
// trip through the retry queue the message comes back through the default exchange,
|
|
593
|
+
// so message.fields no longer describes where it was published
|
|
594
|
+
...origin(message),
|
|
595
|
+
...message.properties.headers,
|
|
596
|
+
[ATTEMPT_HEADER]: attempt + 1
|
|
597
|
+
}
|
|
598
|
+
|
|
599
|
+
// persistent regardless of the source topology: a retried Request would otherwise be
|
|
600
|
+
// delivery-mode 1 and would not survive a broker restart during the TTL
|
|
601
|
+
const properties = { ...message.properties, headers, mandatory: true, persistent: true }
|
|
602
|
+
|
|
603
|
+
await this.#publish(RETRY_PREFIX + this.#topology.delay, queue, message.content, properties)
|
|
604
|
+
|
|
605
|
+
this.#channel.ack(message)
|
|
606
|
+
this.#diagnostics.emit('retry', message, exception, attempt + 1)
|
|
607
|
+
}
|
|
608
|
+
|
|
609
|
+
#nack (message) {
|
|
610
|
+
try { this.#channel.nack(message, false, true) } catch { /* nothing left to do */ }
|
|
611
|
+
}
|
|
612
|
+
|
|
613
|
+
// spread *under* the existing headers, so the first failure writes it and later ones
|
|
614
|
+
// leave it alone
|
|
615
|
+
const origin = (message) => ({
|
|
616
|
+
'x-comq-exchange': message.fields.exchange,
|
|
617
|
+
'x-comq-key': message.fields.routingKey
|
|
618
|
+
})
|
|
619
|
+
```
|
|
620
|
+
|
|
621
|
+
Two things worth being explicit about:
|
|
622
|
+
|
|
623
|
+
- **`#retry` copies the properties** rather than mutating them as [:396](../source/channel.js#L396)
|
|
624
|
+
does — the mutated message must not reach the fallback `nack`, and a user consumer may still
|
|
625
|
+
hold a reference.
|
|
626
|
+
- **The publish target is the retry *exchange*, and the routing key is the source queue.** That
|
|
627
|
+
is what survives the expiry and routes the message home.
|
|
628
|
+
|
|
629
|
+
`#park` / `#dispose` are as given in Part 1.
|
|
630
|
+
|
|
631
|
+
|
|
632
|
+
### 5. Wire the `retry` diagnostic
|
|
633
|
+
|
|
634
|
+
- [source/events.js:7](../source/events.js#L7) — add `'retry'` to `exports.channel`. This single
|
|
635
|
+
switch makes both `source/io.js:271` forward it to `IO.diagnose` **and**
|
|
636
|
+
`source/shards/channel.js:183` `#pipe` forward it with the shard index appended.
|
|
637
|
+
- [types/diagnostic.d.ts:4](../types/diagnostic.d.ts#L4) — add `'retry'` to the `Event` union.
|
|
638
|
+
- [types/io.d.ts:106](../types/io.d.ts#L106) — add the `retry` overload. **Also fix the
|
|
639
|
+
pre-existing bug on that line**: `discard` is declared `(channel, message, index?)` but
|
|
640
|
+
`channel.js:408` has always passed the exception too.
|
|
641
|
+
- [test/io.diagnostics.test.js:31](../test/io.diagnostics.test.js#L31) — add `'retry'` to the
|
|
642
|
+
hardcoded re-emit list.
|
|
643
|
+
- [features/steps/context.js:124](../features/steps/context.js#L124) — add `'retry'` to `EVENTS`.
|
|
644
|
+
|
|
645
|
+
### 5b. Make the backoff ladder configurable
|
|
646
|
+
|
|
647
|
+
Small and self-contained; the reasoning and the defaults are under *Delay and attempts are both
|
|
648
|
+
public*.
|
|
649
|
+
|
|
650
|
+
- `source/topology/{request,event}.json` — add `"attempts": 5`, and `"delay": 30000` /
|
|
651
|
+
`"delay": 5000`. Leave `reply.json` alone; it is `acknowledgments: false`, so nothing there
|
|
652
|
+
ever counts an attempt or waits a delay.
|
|
653
|
+
- [types/topology.d.ts](../types/topology.d.ts) — add `attempts: number` and `delay: number` to
|
|
654
|
+
`Topology`.
|
|
655
|
+
- [source/connection.js:92](../source/connection.js#L92) — `const topology = presets[type]` hands
|
|
656
|
+
out the shared module-level preset object. Make it merge into a copy,
|
|
657
|
+
`{ ...presets[type], ...this.#overrides?.[type] }`, and never mutate the preset.
|
|
658
|
+
- [source/connect.js](../source/connect.js) — `Connect` is variadic
|
|
659
|
+
(`(...urls: string[]) => Promise<IO>`), so the overrides are an optional trailing argument,
|
|
660
|
+
taken when `typeof last !== 'string'` and typed as `[...urls: string[], options: Options]`.
|
|
661
|
+
Thread it through `create()` → `IO` → `Connection`.
|
|
662
|
+
- [test/presets.test.js](../test/presets.test.js) — all three cases assert with `toStrictEqual`,
|
|
663
|
+
so all three change.
|
|
664
|
+
- New tests: an override reaches the channel; an absent override leaves the preset untouched
|
|
665
|
+
(assert the module object is not mutated across two connections); a consumer configured with
|
|
666
|
+
`attempts: 1` parks on the first delivery, having had its one attempt; two channels with different
|
|
667
|
+
`delay` values declare two differently-named retry queues and neither redeclares the other's.
|
|
668
|
+
|
|
669
|
+
### 6. Unit tests — new `describe('retry')` in [test/channel.test.js](../test/channel.test.js)
|
|
670
|
+
|
|
671
|
+
`test/amqplib.mock.js` needs no changes. **`test/channel.fixtures.js` randomises `durable`,
|
|
672
|
+
`confirms` and `acknowledgments` with `flip()`** — every new test must set the ones it asserts
|
|
673
|
+
on explicitly, and the file should be run repeatedly.
|
|
674
|
+
|
|
675
|
+
Declaration: the retry **exchange** asserted as `fanout` and the retry queue asserted with the
|
|
676
|
+
delay-derived name, the TTL and **no** `x-dead-letter-routing-key`, and bound to that exchange;
|
|
677
|
+
the parking queue asserted with the right name and no arguments. All asserted *after* the source
|
|
678
|
+
queue; none asserted when `acknowledgments: false`; the parking queue exclusive when the source
|
|
679
|
+
is exclusive (test both values of `durable`) while the retry queue stays durable regardless;
|
|
680
|
+
asserted for the `bound` path; re-asserted after `recover()`. And the one that guards the shared
|
|
681
|
+
design: **two consumers on the same channel declare the retry topology once**, not once each.
|
|
682
|
+
|
|
683
|
+
Failure path — the regression tests that matter:
|
|
684
|
+
- **publishes before acking** —
|
|
685
|
+
`chan.publish.mock.invocationCallOrder[0] < chan.ack.mock.invocationCallOrder[0]`
|
|
686
|
+
- **waits for the confirmation** — `confirms: true`, capture the 5th publish arg, assert `ack`
|
|
687
|
+
not yet called, invoke the callback, assert it then is
|
|
688
|
+
- **does not republish to the original exchange** — build the message with a random
|
|
689
|
+
`fields.exchange`; assert the publish goes to the retry exchange with the **source queue** as
|
|
690
|
+
the routing key, never to `fields.exchange`. *(the fanout re-delivery regression)*
|
|
691
|
+
- **records the origin on the first failure** — `fields.exchange` and `fields.routingKey` are
|
|
692
|
+
written as `x-comq-exchange` / `x-comq-key`; a message that already carries them (a returned
|
|
693
|
+
retry, whose `fields.exchange` is now `''`) keeps the originals rather than overwriting them
|
|
694
|
+
with `''`. *(the wrong-provenance regression his broker run found)*
|
|
695
|
+
- **tolerates a message with no headers** — `properties = {}`. *(the non-comq-publisher
|
|
696
|
+
`TypeError` regression)*
|
|
697
|
+
- **does not throw** — `await expect(callback(message)).resolves.not.toThrow()`. *(the
|
|
698
|
+
process-crash regression)*
|
|
699
|
+
- **does not seal** — `chan.cancel` not called, `seal` spy not called, and a subsequent
|
|
700
|
+
`channel.consume(...)` still works *(proves `#sealed` is still false)*
|
|
701
|
+
- increments `x-comq-attempt`; does not mutate `message.properties`; emits `retry` with the attempt
|
|
702
|
+
- **publishes persistent even from a non-persistent topology** — `topology.persistent = false`
|
|
703
|
+
(the request preset), assert the publish options carry `persistent: true`, for both the retry
|
|
704
|
+
and the parking publish. *(a parked Request would otherwise not survive a broker restart)*
|
|
705
|
+
- **carries `replyTo` and `correlationId` through** — a message with both in its properties is
|
|
706
|
+
parked with both intact, so a hung caller remains answerable
|
|
707
|
+
- **defensive fallback**: publish throws → `nack(message, false, true)`, no `ack`, resolves;
|
|
708
|
+
publish *and* nack both throw → still resolves; confirmation rejected → same as publish throw
|
|
709
|
+
- consumer throwing `'Channel closed'` → no publish, no nack, no ack (complements the existing
|
|
710
|
+
test at [:144](../test/channel.test.js#L144), which covers `ack` throwing)
|
|
711
|
+
- consumer doing `throw undefined` → resolves *(covers the `exception?.message` guard)*
|
|
712
|
+
- **terminal**: `x-comq-attempt: 5` → published to `comq.parked.<queue>`, **not** to the retry queue;
|
|
713
|
+
published before acking, same order assertion as above; carries the `x-comq-*` headers
|
|
714
|
+
including an `x-comq-exchange` that names the original exchange rather than `''`; `nack` not
|
|
715
|
+
called. Restores
|
|
716
|
+
the spirit of the `it.each` requeue test `1df2afe` deleted. The existing `should emit
|
|
717
|
+
'discard' event` ([:894](../test/channel.test.js#L894)) needs its message fixture extended with
|
|
718
|
+
`fields` — it currently has none, which works only because today's `#discard` never reads
|
|
719
|
+
them.
|
|
720
|
+
- **parking failure**: publish to the parking queue throws → `nack(message, false, true)`, no
|
|
721
|
+
`ack`, resolves. The message survives an unavailable parking queue.
|
|
722
|
+
|
|
723
|
+
### 7. Feature tests — rewrite [features/events.poison.feature](../features/events.poison.feature)
|
|
724
|
+
|
|
725
|
+
Both scenarios drop `@manual`. Two structural facts shape this:
|
|
726
|
+
|
|
727
|
+
- `features/steps/brokers.js` starts containers in `BeforeAll` and `hooks.js` only disconnects,
|
|
728
|
+
so **durable queues and their contents survive across scenarios**. A message still cycling
|
|
729
|
+
through a retry queue will land in the next scenario's consumer. **Use a distinct exchange
|
|
730
|
+
name per scenario** — do not share `poison`.
|
|
731
|
+
- [features/steps/events.js:41](../features/steps/events.js#L41) builds the throwing consumer.
|
|
732
|
+
Change it to record `properties.headers?.['x-comq-attempt']` into a new `this.attempts` array,
|
|
733
|
+
which turns the whole retry loop into an end-to-end assertion against a real broker.
|
|
734
|
+
|
|
735
|
+
```gherkin
|
|
736
|
+
Scenario: A poison event is retried and then discarded
|
|
737
|
+
Given that events from the `poison_retried` exchange are causing exceptions
|
|
738
|
+
When an event is emitted to the `poison_retried` exchange
|
|
739
|
+
Then the event is attempted 6 times
|
|
740
|
+
And the message is parked
|
|
741
|
+
|
|
742
|
+
Scenario: A poison event does not stop other consumers
|
|
743
|
+
Given that events from the `poison_isolated` exchange are causing exceptions
|
|
744
|
+
And that `checker` is consuming events from the `numbers_added` exchange
|
|
745
|
+
When an event is emitted to the `poison_isolated` exchange
|
|
746
|
+
And after 1500ms
|
|
747
|
+
And an event is emitted to the `numbers_added` exchange
|
|
748
|
+
Then `checker` receives the event
|
|
749
|
+
|
|
750
|
+
Scenario: A retried event is not re-delivered to the other consumers of its exchange
|
|
751
|
+
Given that events from the `poison_open` exchange are causing exceptions
|
|
752
|
+
And that `witness` is consuming events from the `poison_open` exchange
|
|
753
|
+
When an event is emitted to the `poison_open` exchange
|
|
754
|
+
And after 3000ms
|
|
755
|
+
Then `witness` has received 1 event
|
|
756
|
+
```
|
|
757
|
+
|
|
758
|
+
Scenario 2 is the direct proof the channel is not sealed — `numbers_added` and
|
|
759
|
+
`poison_isolated` share the one `#events` channel, so under today's code the second
|
|
760
|
+
`emit`/`consume` is dead. Scenario 3 is the fanout regression: exactly once, not once per retry.
|
|
761
|
+
|
|
762
|
+
Steps needed:
|
|
763
|
+
- `Then the event is attempted {int} times` (new, `features/steps/poison.js`) — poll
|
|
764
|
+
`this.attempts` to a deadline, then `assert.deepEqual(this.attempts, [1, 2, 3, 4, 5])`.
|
|
765
|
+
Asserts count, header increment, and that the delay round-trips through the broker. Six
|
|
766
|
+
entries, per the arithmetic above — the first delivery has no `x-comq-attempt` header.
|
|
767
|
+
- `Then the message is discarded` — **already exists** at
|
|
768
|
+
[features/steps/poison.js:7](../features/steps/poison.js#L7) and is currently orphaned (no
|
|
769
|
+
feature file references it). It becomes live; change its `await timeout(300)` to a polling
|
|
770
|
+
wait, via a new `until(predicate, ms)` helper in `test/helpers.js` next to `timeout`.
|
|
771
|
+
- `Then the message is parked` (new) — the real end-to-end assertion, and only possible because
|
|
772
|
+
Part 1 gives comq a queue it owns: `io.process('comq.parked.<queue>', …)` and assert the payload
|
|
773
|
+
arrives, carrying `x-comq-attempt: 5` and the `x-comq-*` headers. Under today's code, and under a
|
|
774
|
+
broker-side DLX, there would be nothing to consume. Prefer this over the diagnostic-flag step
|
|
775
|
+
wherever both would work — it proves the message actually survived, not just that comq
|
|
776
|
+
believes it did.
|
|
777
|
+
- `Then {token} has received {int} event(s)` (new, `features/steps/events.js`) — needs
|
|
778
|
+
per-group counts; today only a single global `eventsConsumedCount` and a last-value-only
|
|
779
|
+
`consumed[group]` exist.
|
|
780
|
+
- `features/steps/context.js` — add `attempts = []` and `counts = {}`, reset on disconnect.
|
|
781
|
+
|
|
782
|
+
### 8. Docs
|
|
783
|
+
|
|
784
|
+
Replace [readme.md:464-470](../readme.md#L464) (which currently documents the sealed channel and
|
|
785
|
+
the "configure a DLX yourself" workaround) with prose covering: the retry queue and its name;
|
|
786
|
+
`x-message-ttl` + `x-dead-letter-exchange` as the delay mechanism; `x-comq-attempt` visible to
|
|
787
|
+
consumers; **five attempts by default** — the first delivery and four retries — then `discard`,
|
|
788
|
+
which is what [readme.md:466](../readme.md#L466) already claimed, and noting that both
|
|
789
|
+
the ladder is a topology setting (`delay`) with per-channel-type defaults, and its length is
|
|
790
|
+
the number of retries; **the channel keeps consuming**; ordering not
|
|
791
|
+
preserved across a failure; at-least-once / consumers must be idempotent; retries confirmed for
|
|
792
|
+
Events but best-effort for Requests (`confirms: false`, `persistent: false`), and a discarded
|
|
793
|
+
Request is never answered — the caller waits indefinitely, which with a limited prefetch can
|
|
794
|
+
deadlock it; do not delete `comq.retry.*` or `comq.parked.*` on a running system; and **the retry
|
|
795
|
+
queue's return hop is at-most-once on classic queues** — a retry can be lost if the source queue
|
|
796
|
+
is unavailable when the TTL fires.
|
|
797
|
+
|
|
798
|
+
Then the parking half, from Part 1: comq publishes the exhausted message to `comq.parked.<queue>`
|
|
799
|
+
and acks only once the broker confirms it, so it is not deleted and not dependent on a
|
|
800
|
+
broker-side policy; the `x-comq-*` headers it carries; that these queues **grow until someone
|
|
801
|
+
drains them**, which is deliberate — the alternative is deleting evidence — and that `discard`
|
|
802
|
+
is the diagnostic to alert on.
|
|
803
|
+
|
|
804
|
+
**Delete the current recommendation to configure a dead-letter policy**
|
|
805
|
+
([readme.md:468-470](../readme.md#L468)). It is no longer needed, and the reasoning in Part 1 —
|
|
806
|
+
that broker dead-lettering is at-most-once on classic queues — means it was never the guarantee
|
|
807
|
+
it appeared to be. A DLX on the parking queue itself remains a user's choice; say nothing about
|
|
808
|
+
it either way.
|
|
809
|
+
|
|
810
|
+
The topology section ([readme.md:428](../readme.md#L428)) gains the retry exchange and queue —
|
|
811
|
+
one pair per distinct `delay`, shared by every source queue — and the per-queue parking queue,
|
|
812
|
+
so the queue list an operator sees is documented rather than discovered.
|
|
813
|
+
|
|
814
|
+
Add `retry` to the diagnostics list at [readme.md:556](../readme.md#L556), and to
|
|
815
|
+
`docs/headers.md`: `x-comq-attempt`, the `x-comq-*` parking headers, and a note that a retried message
|
|
816
|
+
also carries the broker's `x-death` — whose `count` is a *second* attempt counter that happens to
|
|
817
|
+
agree with `x-comq-attempt` today. Say which is authoritative (`x-comq-attempt`, comq's own; `x-death` is
|
|
818
|
+
the broker's record and is useful for its timestamps), because two counters on one message is
|
|
819
|
+
otherwise a debugging trap.
|
|
820
|
+
|
|
821
|
+
## Not affected — checked
|
|
822
|
+
|
|
823
|
+
- `source/shards/channel.js:59` delegates `consume`/`subscribe`/`bound` to per-shard `Channel`s.
|
|
824
|
+
Each shard is a separate broker with its own copy of the queue, declares its own retry
|
|
825
|
+
topology and `comq.parked.*`, and returns retries to its own source. Self-contained —
|
|
826
|
+
though note a parked message lives on the shard that failed it, so a post-mortem means
|
|
827
|
+
looking at every shard. `#retry` uses
|
|
828
|
+
`#publish` directly (not the `failsafe`-wrapped `publish`), so on a paused failfast shard it
|
|
829
|
+
throws `INTERRUPTION` → `#nack` → the message stays on that shard. Acceptable.
|
|
830
|
+
- `IO.seal()` / `IO.close()` and the `#sealed` guards at `:88`, `:101`, `:119` are the
|
|
831
|
+
*explicit* seal and stay exactly as they are. Only the failure path stops calling `seal()`.
|
|
832
|
+
- `source/emitter.js:11` swallows listener exceptions, so a bad `retry` listener can't
|
|
833
|
+
re-introduce the crash.
|
|
834
|
+
- Existing assertions the extra `assertQueue` call could have broken, all verified safe:
|
|
835
|
+
`channel.test.js:185`/`:191` (`send` path, no retry queue), `:358` (`assertExchange` count),
|
|
836
|
+
`:387` (`bindQueue` count — the retry queue is never bound), `:74`/`:363`
|
|
837
|
+
(`toHaveBeenCalledWith`, not counts). Only the `mock.results[0]` reads depend on ordering,
|
|
838
|
+
which the "source queue first" rule preserves.
|
|
839
|
+
- `test/io.*.test.js` use the channel-level `test/connection.mock.js`; unaffected.
|
|
840
|
+
|
|
841
|
+
## Verification
|
|
842
|
+
|
|
843
|
+
1. `npm test` (`standard && jest`) — run `npx jest test/channel.test.js` **several times**,
|
|
844
|
+
since `test/channel.fixtures.js` randomises the topology per run.
|
|
845
|
+
2. `npm run features` (needs Docker; Testcontainers starts two RabbitMQ containers). Time the
|
|
846
|
+
poison feature specifically — 5 attempts in one step at a 1s delay is the tightest new
|
|
847
|
+
timing in the suite, against a 30s per-step cucumber timeout.
|
|
848
|
+
3. The end-to-end proof is that `features/events.poison.feature` becomes ordinary automatic
|
|
849
|
+
scenarios: a consumer that throws, a message that comes back five times with an incrementing
|
|
850
|
+
`x-comq-attempt` (five deliveries in total), a message that ends up disposed rather than silently
|
|
851
|
+
gone, and — the point of the whole exercise — a *sibling* consumer still alive afterwards.
|
|
852
|
+
4. Confirm the process no longer exits: today, running the poison scenario ends the node
|
|
853
|
+
process. After the change it should complete the scenario.
|
|
854
|
+
5. Inspect the broker after a features run — the retry queues empty, `comq.parked.*` holding
|
|
855
|
+
exactly the messages the poison scenarios parked, with `x-comq-exchange` naming the original
|
|
856
|
+
exchange rather than `''`. That last one is the check that the origin is captured on the
|
|
857
|
+
first failure rather than at parking time.
|
|
858
|
+
6. Confirm the cycle claim directly, once, rather than trusting the document: declare a queue
|
|
859
|
+
with a TTL, `x-dead-letter-exchange: ''` and no routing key, publish to the default exchange
|
|
860
|
+
with that queue as the key, and watch the message vanish at the first expiry. It is the
|
|
861
|
+
reason the retry exchange exists.
|
|
862
|
+
7. Branch off `dev` (not `release`).
|
|
863
|
+
|
|
864
|
+
---
|
|
865
|
+
|
|
866
|
+
# Part 3 — A consumer says how it failed (`Retry` / `Park`)
|
|
867
|
+
|
|
868
|
+
A follow-up proposal lets a consumer classify its failure: `throw new Retry()` for a
|
|
869
|
+
dependency that was briefly absent, `throw new Park()` for a message this consumer will never
|
|
870
|
+
process. A bare rejection means `Retry`.
|
|
871
|
+
|
|
872
|
+
## What it changes in this plan: almost nothing
|
|
873
|
+
|
|
874
|
+
**None of the six defects change.** Every one of them exists whether or not a consumer can
|
|
875
|
+
classify its failure — the verdict is a *routing* decision layered on the failure path, not a
|
|
876
|
+
fix to it. Parts 1 and 2 stand exactly as written, and remain worth landing on their own: a
|
|
877
|
+
`Park()` that reaches today's `#requeue` still seals the channel and still kills the process.
|
|
878
|
+
|
|
879
|
+
**It adds one branch**, in `#failed` (Part 2 §4). The seam already exists:
|
|
880
|
+
|
|
881
|
+
```js
|
|
882
|
+
const park = verdictOf(exception) === PARK || attempt >= this.#topology.attempts
|
|
883
|
+
|
|
884
|
+
if (park) await this.#park(queue, message, exception)
|
|
885
|
+
else await this.#retry(queue, message, attempt, exception)
|
|
886
|
+
```
|
|
887
|
+
|
|
888
|
+
`Retry` still respects the ladder — a consumer can say "worth another attempt" without
|
|
889
|
+
being able to say "forever". comq keeps its own count, which is what the proposal's "no
|
|
890
|
+
counts" asks for.
|
|
891
|
+
|
|
892
|
+
**Part 1 already covers it.** Parking is the destination whether the message got there by
|
|
893
|
+
exhausting its attempts or by a consumer saying so on the first delivery. `Park()` does raise
|
|
894
|
+
the stakes — parking stops being a rare terminal event and becomes a routine outcome a consumer
|
|
895
|
+
chooses deliberately — which is a further argument for Part 1's decision, and an argument
|
|
896
|
+
against ever making it opt-in.
|
|
897
|
+
|
|
898
|
+
## Thrown, not returned — the return channel is already taken
|
|
899
|
+
|
|
900
|
+
[source/io.js:305](../source/io.js#L305): `const reply = await producer(payload)`. For `io.reply`,
|
|
901
|
+
**what the consumer returns is the reply**, and [io.js:308](../source/io.js#L308) then inspects it
|
|
902
|
+
for iterator-ness to choose streaming versus a single reply. A returned symbol or verdict object
|
|
903
|
+
would be encoded and sent to the caller as the answer. `types/io.d.ts:10` says the same:
|
|
904
|
+
`Producer<Input, Output> = (message) => Output | Promise<Output>`.
|
|
905
|
+
|
|
906
|
+
Event consumers ([io.js:345](../source/io.js#L345)) do ignore their return value, so returning would
|
|
907
|
+
work *there* — but a verdict that works for `consume`/`process` and not for `reply` is worse
|
|
908
|
+
than one that works everywhere. Second reason: a thrown verdict can be raised from anywhere in
|
|
909
|
+
the call stack, so a validation helper three frames down can park the message; a return can only
|
|
910
|
+
be made by the top-level function.
|
|
911
|
+
|
|
912
|
+
## Shape
|
|
913
|
+
|
|
914
|
+
```js
|
|
915
|
+
// source/verdicts.js — its own module, so source/channel.js can require it
|
|
916
|
+
// without going through the package entry point
|
|
917
|
+
'use strict'
|
|
918
|
+
|
|
919
|
+
// the global registry, so two copies of comq in one dependency tree still agree
|
|
920
|
+
const VERDICT = Symbol.for('comq.verdict')
|
|
921
|
+
|
|
922
|
+
const RETRY = 'retry'
|
|
923
|
+
const PARK = 'park'
|
|
924
|
+
|
|
925
|
+
class Retry extends Error { [VERDICT] = RETRY }
|
|
926
|
+
class Park extends Error { [VERDICT] = PARK }
|
|
927
|
+
|
|
928
|
+
// a rejection nobody classified is a rejection nobody chose
|
|
929
|
+
const verdictOf = (exception) => exception?.[VERDICT] ?? RETRY
|
|
930
|
+
```
|
|
931
|
+
|
|
932
|
+
Two deliberate choices:
|
|
933
|
+
|
|
934
|
+
- **`Symbol.for` brand, not `instanceof`.** Required, not defensive: a consumer of comq that
|
|
935
|
+
pins it inside its own binding while the application resolves its own tree makes two copies
|
|
936
|
+
the ordinary resolution rather than the pathological one, and `instanceof` is silently `false`
|
|
937
|
+
across them. A `Park` would then be read as a bare rejection, retried to exhaustion, and
|
|
938
|
+
parked anyway — the right outcome, several retry cycles late, with a spurious `retry`
|
|
939
|
+
diagnostic for each. The global symbol registry is shared across copies; a class identity is not.
|
|
940
|
+
- **`extends Error`**, so a verdict carries a stack, a `message`, and `cause`. A bare
|
|
941
|
+
`new Park()` would throw the cause away, and the `discard` diagnostic's second argument
|
|
942
|
+
([channel.js:408](../source/channel.js#L408)) would be a marker with nothing about what actually
|
|
943
|
+
went wrong. Use `new Park('…', { cause })` and record both in the parking headers —
|
|
944
|
+
`x-comq-reason` from the verdict's own message, and the cause's message alongside it.
|
|
945
|
+
|
|
946
|
+
## Consumer code
|
|
947
|
+
|
|
948
|
+
```js
|
|
949
|
+
const { Retry, Park } = require('comq')
|
|
950
|
+
|
|
951
|
+
// events — fanout
|
|
952
|
+
await io.consume('orders', 'billing', async (order) => {
|
|
953
|
+
if (typeof order.total !== 'number') throw new Park('no total on order')
|
|
954
|
+
|
|
955
|
+
try {
|
|
956
|
+
await billing.charge(order)
|
|
957
|
+
} catch (e) {
|
|
958
|
+
throw new Retry('billing unavailable', { cause: e }) // or just: throw e
|
|
959
|
+
}
|
|
960
|
+
})
|
|
961
|
+
|
|
962
|
+
// tasks
|
|
963
|
+
await io.process('emails', async (job) => {
|
|
964
|
+
await mailer.send(job) // a bare throw means Retry
|
|
965
|
+
})
|
|
966
|
+
|
|
967
|
+
// requests — the return value is the reply, so the verdict must be thrown
|
|
968
|
+
await io.reply('compute', async (input) => {
|
|
969
|
+
if (typeof input.n !== 'number') throw new Park('n must be a number')
|
|
970
|
+
|
|
971
|
+
return { result: input.n * 2 } // ← the return channel is already spoken for
|
|
972
|
+
})
|
|
973
|
+
```
|
|
974
|
+
|
|
975
|
+
## `Park` from a Producer is a category error
|
|
976
|
+
|
|
977
|
+
`Park` on an `io.reply` producer would mean the caller never gets a reply and its promise never
|
|
978
|
+
settles — the prefetch deadlock [readme.md:469](../readme.md#L469) already admits. Today that
|
|
979
|
+
happens by accident once the attempts are spent; a `Park` would make it a documented path.
|
|
980
|
+
|
|
981
|
+
**Resolved: reject it.** `consume` and `process` take a `Consumer` — no caller, so somebody has
|
|
982
|
+
to decide what happens to a failed message. `reply` takes a `Producer` — a caller is waiting, and
|
|
983
|
+
what it is owed is a reply. Retry-or-park is not a policy choice there, and comq's own type
|
|
984
|
+
vocabulary already draws the line.
|
|
985
|
+
|
|
986
|
+
The rejection cannot be static — nothing at wiring time knows what a producer will throw — so
|
|
987
|
+
concretely: a `Park` from a producer is treated as a bare rejection, so the message retries and
|
|
988
|
+
eventually parks on the count, and comq emits a diagnostic saying `Park` is not applicable to a
|
|
989
|
+
reply producer. Loud, no new protocol, and the message is not lost.
|
|
990
|
+
|
|
991
|
+
The alternative of sending an error reply was rejected on its own terms: it adds an element to
|
|
992
|
+
the reply protocol that every caller must learn to distinguish, and it collides with a
|
|
993
|
+
distinction callers already make between an error returned as a value and an exception rethrown
|
|
994
|
+
in them. A third kind arriving from the transport would have to be mapped onto one of those.
|
|
995
|
+
|
|
996
|
+
Burning every attempt on a request nobody can process does **not** wedge the consumer, which was
|
|
997
|
+
an earlier worry here: a retry publishes and acks, so the delivery is released immediately and
|
|
998
|
+
the message waits in the retry queue, not in the consumer's unacked set. The cost is a handful
|
|
999
|
+
of pointless invocations.
|
|
1000
|
+
|
|
1001
|
+
## Sequencing
|
|
1002
|
+
|
|
1003
|
+
Land Parts 1 and 2 first. The crash is a dead process today; the verdict API is additive and
|
|
1004
|
+
non-breaking (bare rejection = `Retry` = the behaviour Part 2 establishes), so it composes
|
|
1005
|
+
cleanly on top as a minor release.
|
|
1006
|
+
|
|
1007
|
+
One thing argues for doing them together: Part 1 names the parking queue and Part 3 names the
|
|
1008
|
+
class, and they should agree — `Park` → `comq.parked.<queue>`, or `Dead` →
|
|
1009
|
+
`comq.parked.<queue>`. Naming it twice means renaming a queue in a later release, which orphans
|
|
1010
|
+
whatever is sitting in the old one. **Pick the noun once**, before Part 1 is implemented; it is
|
|
1011
|
+
the only thing Part 3 needs decided up front.
|
|
1012
|
+
|
|
1013
|
+
## Surface it adds
|
|
1014
|
+
|
|
1015
|
+
- `source/index.js` currently exports only `connect` and `assert`, and comq has no exported
|
|
1016
|
+
error classes at all — `Retry`/`Park` would be the first. Clean addition, plus
|
|
1017
|
+
`types/index.d.ts` (which re-exports from `./io`; add the two classes there).
|
|
1018
|
+
- `source/verdicts.js` is required directly by `source/channel.js`, not through
|
|
1019
|
+
`source/index.js` — `index.js` pulls in `connect`, and the channel must not depend on it.
|
|
1020
|
+
- Unit tests: `verdictOf` returns `RETRY` for `undefined`, `null`, a string throw, a plain
|
|
1021
|
+
`Error`, and an object with a *different* symbol key; returns `PARK` only for the brand. Then
|
|
1022
|
+
in `test/channel.test.js`, a consumer throwing `Park` publishes to the parking queue on the
|
|
1023
|
+
**first** delivery with no retry publish, and a consumer throwing `Retry` behaves exactly like
|
|
1024
|
+
a bare rejection.
|
|
1025
|
+
- One feature scenario per verdict: a `Park()` on first delivery lands in the parking queue with
|
|
1026
|
+
no retry cycle at all (and completes fast, since it skips five TTL waits — a useful contrast
|
|
1027
|
+
with the full-ladder scenario); a `Retry()` behaves exactly as a bare rejection.
|
|
1028
|
+
- Docs: the verdicts get their own readme section, and the `x-comq-reason` header note in
|
|
1029
|
+
`docs/headers.md` should mention that a `cause` is recorded alongside it.
|
|
1030
|
+
|
|
1031
|
+
---
|
|
1032
|
+
|
|
1033
|
+
# Context
|
|
1034
|
+
|
|
1035
|
+
## The six defects
|
|
1036
|
+
|
|
1037
|
+
All introduced by `1df2afe` *refactor: throw exceptions on poison messages*, in
|
|
1038
|
+
`#getAcknowledgingConsumer` ([:370](../source/channel.js#L370)) and `#requeue`
|
|
1039
|
+
([:393](../source/channel.js#L393)).
|
|
1040
|
+
|
|
1041
|
+
1. **It acks before it publishes.** [:394](../source/channel.js#L394) acks, then
|
|
1042
|
+
[:399](../source/channel.js#L399) publishes. A crash between the two loses the message.
|
|
1043
|
+
2. **It republishes to `message.fields.exchange`.** For a queue bound to a fanout exchange that
|
|
1044
|
+
*is* the exchange, so a retry meant for the one subscriber that failed is fanned out to
|
|
1045
|
+
every subscriber again.
|
|
1046
|
+
3. **It seals the channel** ([:398](../source/channel.js#L398)). `seal()` cancels every consumer tag
|
|
1047
|
+
on the channel; `#sealed` is never cleared (not even by `recover()` at
|
|
1048
|
+
[:213](../source/channel.js#L213)) and `recall.reset` at [:198](../source/channel.js#L198) throws
|
|
1049
|
+
away the callbacks that would let them come back. And it is worse than one subscription:
|
|
1050
|
+
[source/io.js](../source/io.js) routes `consume`, `emit`, `route`, `subscribe`, `process` **and**
|
|
1051
|
+
`enqueue` through a single `#events` channel, so one poison event permanently silences every
|
|
1052
|
+
event consumer *and* every task processor in the process.
|
|
1053
|
+
4. **It rethrows into amqplib** ([:384](../source/channel.js#L384)), which dispatches deliveries
|
|
1054
|
+
through an EventEmitter listener (`amqplib/lib/channel_model.js:47` →
|
|
1055
|
+
`lib/channel.js:455 return consumer(message)`) and drops the returned promise. Unhandled
|
|
1056
|
+
rejection, and Node ends the process by default. That crash *is* the current retry
|
|
1057
|
+
mechanism: [features/events.poison.feature](../features/events.poison.feature) is entirely
|
|
1058
|
+
`@manual` and instructs a human to "run this scenario five times".
|
|
1059
|
+
5. **It reads `message.properties.headers[...]` unguarded**
|
|
1060
|
+
([:379](../source/channel.js#L379), [:396](../source/channel.js#L396)) — see the correction below.
|
|
1061
|
+
[source/shards/channel.js:204](../source/shards/channel.js#L204) already guards the identical
|
|
1062
|
+
pattern with `?.`.
|
|
1063
|
+
6. **The last attempt destroys the message** — Part 1.
|
|
1064
|
+
|
|
1065
|
+
## Details worth knowing
|
|
1066
|
+
|
|
1067
|
+
Findings from verifying the above against the code and against amqplib, kept because they
|
|
1068
|
+
correct claims that get made about this area.
|
|
1069
|
+
|
|
1070
|
+
- **The unguarded header read is narrower than it looks.** It is often described as breaking for
|
|
1071
|
+
any task, broadcast or request. It does not, for anything comq published:
|
|
1072
|
+
`amqplib/lib/api_args.js:191` does `Object.create(options.headers || null)` and always assigns
|
|
1073
|
+
`headers` into the publish properties, and `lib/codec.js:65` encodes with
|
|
1074
|
+
`for (const key in val)`, walking the prototype chain. So every comq-published message arrives
|
|
1075
|
+
with a headers table, at minimum `{}`. The `TypeError` is real only for a message published by
|
|
1076
|
+
a **non-comq client** that omits the field table (`defs.js:3438`, `headers: void 0` is the
|
|
1077
|
+
decode default). That is a supported case rather than a hypothetical one — consuming events
|
|
1078
|
+
published by something that is not comq is a thing comq is used for — so the guard is
|
|
1079
|
+
load-bearing, not tidiness. `source/shards/channel.js:204` already guards the identical
|
|
1080
|
+
pattern.
|
|
1081
|
+
- **Publisher confirms cover events only.** `source/topology/request.json` is `confirms: false`
|
|
1082
|
+
and the channel is created with `createChannel`, not `createConfirmChannel`
|
|
1083
|
+
([channel.js:68](../source/channel.js#L68)) — there is no confirm to await on the request channel.
|
|
1084
|
+
Requests are also `persistent: false`, so a requeued request does not survive a broker restart
|
|
1085
|
+
regardless. This is why Part 2 fixes requests structurally but calls them best-effort.
|
|
1086
|
+
- **`MAX_REDELIVERIES = 5` meant six deliveries, not five.** The counter was absent on the first
|
|
1087
|
+
delivery, so it read `0` and the message was retried; `5 >= 5` first held on the sixth, while
|
|
1088
|
+
[readme.md:466](../readme.md#L466) said "five times in a row". Fixed by counting deliveries
|
|
1089
|
+
rather than retries, which is what the setting's name claims.
|
|
1090
|
+
- **Test coverage for this path collapsed in `1df2afe`.** That commit deleted the requeue unit
|
|
1091
|
+
test and the one automated scenario in `features/rpc.feature`, replaced them with two
|
|
1092
|
+
`@manual` scenarios that assert nothing, and left the `the message is discarded` step in
|
|
1093
|
+
`features/steps/poison.js` referenced by no feature file at all.
|
|
1094
|
+
|
|
1095
|
+
## Decisions already taken
|
|
1096
|
+
|
|
1097
|
+
- **Retry delay: broker-side delay queue (TTL + DLX).** Survives a process restart and does not
|
|
1098
|
+
hold an unacked delivery against prefetch.
|
|
1099
|
+
- **`seal()` on poison is replaced by a diagnostics event only.** The channel keeps consuming.
|
|
1100
|
+
No circuit breaker.
|
|
1101
|
+
- **Requests get the same structural fixes, best-effort.** `request.json` is not changed; the
|
|
1102
|
+
limitation is documented.
|
|
1103
|
+
- **Parking, not broker dead-lettering** (Part 1). Decided on retention, with the migration cost
|
|
1104
|
+
ruled acceptable and therefore not a criterion: dead-lettering on classic queues can still
|
|
1105
|
+
lose the message, and the point of keeping it is that it is kept.
|
|
1106
|
+
|
|
1107
|
+
## Still open
|
|
1108
|
+
|
|
1109
|
+
Nothing blocking. The three questions this document opened are resolved:
|
|
1110
|
+
|
|
1111
|
+
- **The noun** — `Park` / `comq.parked.<queue>`. See Part 1.
|
|
1112
|
+
- **`Park` on an `io.reply` producer** — rejected as a category error, treated at runtime as a
|
|
1113
|
+
bare rejection plus a diagnostic. See Part 3.
|
|
1114
|
+
- **Whether Part 3 is built** — **decided, not yet scheduled.** Parts 1 and 2 are the emergency
|
|
1115
|
+
and do not depend on it; a consumer of comq gets retry-then-park by upgrading and writing no
|
|
1116
|
+
code. But the parking queue takes its noun from a verdict class that would not otherwise
|
|
1117
|
+
exist, and without verdicts a message a contract has already refused costs every attempt —
|
|
1118
|
+
two minutes at the event default — and then lands in the parking queue beside the
|
|
1119
|
+
messages that genuinely failed. "Decided, not yet scheduled" rather than "open", because
|
|
1120
|
+
people read this deciding whether to build on it.
|