comq 0.22.0 → 0.23.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/discussions/one-parking-queue.md +63 -0
- package/discussions/one-reply-queue.md +44 -0
- package/discussions/poison-message-handling.md +2 -1
- package/docs/headers.md +1 -1
- package/features/parking.feature +32 -0
- package/features/rpc.replies.feature +26 -0
- package/features/steps/brokers.js +25 -1
- package/features/steps/parked.js +7 -18
- package/features/steps/parking.js +113 -0
- package/features/steps/poison.js +17 -19
- package/features/steps/replies.js +35 -0
- package/package.json +1 -1
- package/readme.md +17 -33
- package/source/.io/ReplyPipe.js +16 -1
- package/source/channel.js +19 -27
- package/source/index.js +0 -2
- package/source/io.js +32 -49
- package/test/ReplyPipe.test.js +40 -0
- package/test/channel.test.js +29 -18
- package/test/io.request.test.js +13 -3
- package/types/index.d.ts +0 -6
- package/types/io.d.ts +0 -6
- package/features/closing.feature +0 -32
- package/features/steps/closing.js +0 -109
- package/source/abandoned.js +0 -15
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
# One parking queue
|
|
2
|
+
|
|
3
|
+
> Supersedes the parking queue's naming and lifetime in
|
|
4
|
+
> [poison message handling](./poison-message-handling.md), Part 1. Everything else about
|
|
5
|
+
> parking — when a message is parked, what it carries, what is published before what — is
|
|
6
|
+
> unchanged.
|
|
7
|
+
|
|
8
|
+
Everything a consumer could not process is kept in one queue, `comq.parked`, whatever it was
|
|
9
|
+
consumed from. It is durable, it is declared once per channel beside the retry queues, and comq
|
|
10
|
+
never removes it or what is in it.
|
|
11
|
+
|
|
12
|
+
A parked message already says where it came from. `x-comq-queue` is the queue it was consumed
|
|
13
|
+
from, `x-comq-exchange` and `x-comq-key` where it was originally published, and all three are
|
|
14
|
+
recorded on the first failure rather than read at parking time. Reading one source's parked
|
|
15
|
+
messages is a match on `x-comq-queue`.
|
|
16
|
+
|
|
17
|
+
## What this changes
|
|
18
|
+
|
|
19
|
+
| | before | after |
|
|
20
|
+
|---|---|---|
|
|
21
|
+
| parking queues | one per consumed queue | one |
|
|
22
|
+
| its lifetime | the lifetime of the queue it served | durable, always |
|
|
23
|
+
| what a groupless subscriber parks | went with its connection | kept |
|
|
24
|
+
| scales with | how many queues the connection consumes | nothing |
|
|
25
|
+
|
|
26
|
+
## Why the queue per source goes
|
|
27
|
+
|
|
28
|
+
**It was measured.** The cost of this design was worked through when it was made, at 200 source
|
|
29
|
+
queues and 200 parking queues, and accepted. In a deployment of about forty processes the
|
|
30
|
+
parking queues came to 1318 of 2731 queues on the broker — 48 % — and every one of them was
|
|
31
|
+
empty, with no consumer and no message it had ever held. At about 70 KB of broker memory per
|
|
32
|
+
queue, that is the largest single item on a broker that had been blocking publishers on a memory
|
|
33
|
+
alarm.
|
|
34
|
+
|
|
35
|
+
The count is what makes it grow: one per consumed queue means the broker pays for how widely an
|
|
36
|
+
application is decomposed, forever, whether or not anything ever fails.
|
|
37
|
+
|
|
38
|
+
**Depth per source survives.** The reason recorded for the queue per source was that depth per
|
|
39
|
+
consumer is how anyone notices a problem. What notices a problem is the `discard` diagnostic,
|
|
40
|
+
which fires per parked message and names the queue it came from. Depth in `comq.parked` still
|
|
41
|
+
says how much is waiting, and `x-comq-queue` still says what for.
|
|
42
|
+
|
|
43
|
+
**The lifetime rule goes with it.** The parking queue mirrored its source's durability so that a
|
|
44
|
+
durable parking queue would not be left behind an exclusive `amq.gen-*` source — one leaked per
|
|
45
|
+
restart, forever. With a fixed name there is nothing to leak, and the rule that prevented the
|
|
46
|
+
leak was throwing away the evidence it was meant to keep: a groupless subscriber's failures
|
|
47
|
+
vanished with its connection, at the moment there was nobody left to tell.
|
|
48
|
+
|
|
49
|
+
## Upgrading
|
|
50
|
+
|
|
51
|
+
`comq.parked.<queue>` queues from an earlier version keep whatever they already hold. comq stops
|
|
52
|
+
declaring them and stops publishing to them, so they are drained and deleted once, by hand.
|
|
53
|
+
|
|
54
|
+
Nothing is redeclared with different arguments, so no assertion answers `PRECONDITION_FAILED`
|
|
55
|
+
and no broker has to be emptied. During a rollout the processes still on the old version go on
|
|
56
|
+
parking per source and the new ones park in `comq.parked`; both are readable, and both say
|
|
57
|
+
which queue a message came from.
|
|
58
|
+
|
|
59
|
+
## What comq says about it
|
|
60
|
+
|
|
61
|
+
That the queue is there, that a message in it names its source, and that comq removes neither.
|
|
62
|
+
What to do with a queue on a broker — a `max-length`, a TTL, an alert on depth — is read off
|
|
63
|
+
those facts by whoever can see the broker.
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
# One reply queue
|
|
2
|
+
|
|
3
|
+
An `IO` receives every reply on one queue, `comq.reply..<id>`, whatever it sent the request to.
|
|
4
|
+
|
|
5
|
+
A reply is found by its `correlationId`. `ReplyEmitter.next` builds one from eight random bytes
|
|
6
|
+
and a counter, and it is unique across processes — it has to be, because a producer's control
|
|
7
|
+
queue sees the requests of every consumer and tells them apart by nothing else. Routing a reply
|
|
8
|
+
reads that identifier and nothing else: the queue it arrived on was never consulted.
|
|
9
|
+
|
|
10
|
+
## What this changes
|
|
11
|
+
|
|
12
|
+
| | before | after |
|
|
13
|
+
|---|---|---|
|
|
14
|
+
| reply queues | one per distinct queue or exchange requested | one |
|
|
15
|
+
| its name | `<target>..<id>` | `comq.reply..<id>` |
|
|
16
|
+
| scales with | how widely the caller fans out | nothing |
|
|
17
|
+
|
|
18
|
+
Both are exclusive, so both go with the connection. A sharded connection holds one per shard, as
|
|
19
|
+
it held one per target per shard before.
|
|
20
|
+
|
|
21
|
+
## Why the queue per target goes
|
|
22
|
+
|
|
23
|
+
The count was the caller's to set and the broker's to pay: a connection that requests fifty
|
|
24
|
+
queues held fifty queues and fifty consumers for as long as it lived, whether or not it ever
|
|
25
|
+
requested them again. Nothing bounded it but how widely the application was decomposed.
|
|
26
|
+
|
|
27
|
+
The name was the argument for it — a queue named after what is being called says, to somebody
|
|
28
|
+
reading the broker, who is waiting on what. It says it of a queue that carries no messages
|
|
29
|
+
between requests, and the same thing is on every message that does pass through: `correlationId`
|
|
30
|
+
identifies the request, and the reply-side handler knows which one it belongs to.
|
|
31
|
+
|
|
32
|
+
## What is given up
|
|
33
|
+
|
|
34
|
+
Reading the broker no longer tells you which operations a connection calls. A reply queue names
|
|
35
|
+
its connection instead, and what it is waiting for is in flight rather than on the broker.
|
|
36
|
+
|
|
37
|
+
## Upgrading
|
|
38
|
+
|
|
39
|
+
Reply queues are exclusive and are asserted on the first request, so a process on the old
|
|
40
|
+
version and one on the new hold different queues and neither declares the other's. Nothing is
|
|
41
|
+
redeclared, and a rollout needs no order.
|
|
42
|
+
|
|
43
|
+
A parked request keeps the `replyTo` it was sent with, so one parked by an old process still
|
|
44
|
+
names the queue that process was listening on — which, as before, is gone once that process is.
|
|
@@ -7,7 +7,8 @@
|
|
|
7
7
|
> where it survives a retry.
|
|
8
8
|
>
|
|
9
9
|
> **All three parts are implemented.** Text below is written as a proposal because that is what
|
|
10
|
-
> it was; it describes what the code now does
|
|
10
|
+
> it was; it describes what the code now does — except for the parking queue's name and
|
|
11
|
+
> lifetime, which [one parking queue](./one-parking-queue.md) supersedes.
|
|
11
12
|
|
|
12
13
|
When a consumer callback rejects, comq is supposed to retry the message a few times and then
|
|
13
14
|
give up on it. What it actually does is kill the process, and take every other consumer in that
|
package/docs/headers.md
CHANGED
|
@@ -26,7 +26,7 @@ the message properties.
|
|
|
26
26
|
| `x-comq-attempt` | a retried message | Which attempt this delivery is, counting from one. Absent on the first, so read it as `headers?.['x-comq-attempt'] ?? 1`. The last attempt is one more than the number of rungs in the [backoff ladder](../readme.md#retries). |
|
|
27
27
|
| `x-comq-exchange` | a retried or parked message | The exchange it was originally published to. |
|
|
28
28
|
| `x-comq-key` | a retried or parked message | The routing key it was originally published with. |
|
|
29
|
-
| `x-comq-queue` | a parked message | The queue it was consumed from. |
|
|
29
|
+
| `x-comq-queue` | a parked message | The queue it was consumed from. One `comq.parked` queue holds what every source parked, so this is what tells them apart. |
|
|
30
30
|
| `x-comq-reason` | a parked message | The `message` of the exception that ended its attempts. |
|
|
31
31
|
| `x-comq-cause` | a parked message | The `message` of that exception's `cause`, when it has one — a verdict is usually thrown with the failure that prompted it. |
|
|
32
32
|
| `x-comq-at` | a parked message | When it was parked, as `Date.now()`. |
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
Feature: The parking queue
|
|
2
|
+
|
|
3
|
+
What a consumer could not process is kept in one queue, `comq.parked`, whatever it was
|
|
4
|
+
consumed from. The message says where it came from, so the queue does not have to.
|
|
5
|
+
|
|
6
|
+
Background:
|
|
7
|
+
Given an active connection to the broker
|
|
8
|
+
|
|
9
|
+
Scenario: Messages parked from two queues are kept in one
|
|
10
|
+
|
|
11
|
+
A queue per source would repeat what every parked message already carries, and
|
|
12
|
+
would stay on the broker for as long as the broker does.
|
|
13
|
+
|
|
14
|
+
Given that events from the `parking_left` exchange are causing exceptions
|
|
15
|
+
And that events from the `parking_right` exchange are causing exceptions
|
|
16
|
+
When an event is emitted to the `parking_left` exchange
|
|
17
|
+
And an event is emitted to the `parking_right` exchange
|
|
18
|
+
Then a message parked from the `parking_left` exchange is in the parking queue
|
|
19
|
+
And a message parked from the `parking_right` exchange is in the parking queue
|
|
20
|
+
|
|
21
|
+
Scenario: A message parked from a groupless subscription outlives its connection
|
|
22
|
+
|
|
23
|
+
A groupless subscriber consumes a queue of its own that the broker removes with the
|
|
24
|
+
connection. What it could not process is not ephemeral: it is waiting for a person,
|
|
25
|
+
and there is nobody left to tell.
|
|
26
|
+
|
|
27
|
+
Given that groupless events from the `parking_ephemeral` exchange are causing exceptions
|
|
28
|
+
When an event is emitted to the `parking_ephemeral` exchange
|
|
29
|
+
And the message is parked
|
|
30
|
+
And the connection is closed
|
|
31
|
+
And an active connection to the broker
|
|
32
|
+
Then a message parked from the `parking_ephemeral` exchange is in the parking queue
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
Feature: Reply queues
|
|
2
|
+
|
|
3
|
+
A connection consumes its replies from one queue, whatever it sends requests to. The
|
|
4
|
+
reply is found by its correlation identifier, which is unique across processes, so the
|
|
5
|
+
queue does not have to name what was called.
|
|
6
|
+
|
|
7
|
+
Scenario: One reply queue however many queues are requested
|
|
8
|
+
|
|
9
|
+
Given an active connection to the broker
|
|
10
|
+
And a producer replying `replies_alpha` queue
|
|
11
|
+
And a producer replying `replies_beta` queue
|
|
12
|
+
When the consumer sends a request to the `replies_alpha` queue
|
|
13
|
+
Then the consumer receives the reply
|
|
14
|
+
When the consumer sends a request to the `replies_beta` queue
|
|
15
|
+
Then the consumer receives the reply
|
|
16
|
+
And the broker holds 1 reply queue
|
|
17
|
+
|
|
18
|
+
Scenario: One reply queue per shard
|
|
19
|
+
|
|
20
|
+
A shard is a broker of its own, and a request published to one is answered on it.
|
|
21
|
+
|
|
22
|
+
Given an active sharded connection
|
|
23
|
+
And a producer replying `replies_sharded` queue
|
|
24
|
+
When the consumer sends 10 requests to the `replies_sharded` queue
|
|
25
|
+
Then all replies have been received
|
|
26
|
+
And each broker holds 1 reply queue
|
|
@@ -12,8 +12,13 @@ const IMAGE = 'rabbitmq:4.3-management'
|
|
|
12
12
|
const USER = 'developer'
|
|
13
13
|
const PASSWORD = 'secret'
|
|
14
14
|
const AMQP_PORT = 5672
|
|
15
|
+
const MANAGEMENT_PORT = 15672
|
|
15
16
|
const BROKERS_AMOUNT = 2
|
|
16
17
|
const HOST_PORTS = [5673, 5674]
|
|
18
|
+
|
|
19
|
+
// what a scenario reads the topology from: a queue nothing holds open is
|
|
20
|
+
// invisible over AMQP, and its absence is what some of them are about
|
|
21
|
+
const MANAGEMENT_HOST_PORTS = [15673, 15674]
|
|
17
22
|
const HEALTHCHECK_INTERVAL = global.COMQ_TESTING_HEALTHCHECK_INTERVAL ?? 1000
|
|
18
23
|
|
|
19
24
|
/** @type {import('@testcontainers/rabbitmq').StartedRabbitMQContainer[]} */
|
|
@@ -33,7 +38,9 @@ async function startBrokers () {
|
|
|
33
38
|
})
|
|
34
39
|
.withAutoRemove(false)
|
|
35
40
|
.withName(`comq-rmq-${n}`)
|
|
36
|
-
.withExposedPorts(
|
|
41
|
+
.withExposedPorts(
|
|
42
|
+
{ container: AMQP_PORT, host: hostPort },
|
|
43
|
+
{ container: MANAGEMENT_PORT, host: MANAGEMENT_HOST_PORTS[n] })
|
|
37
44
|
.withHealthCheck({
|
|
38
45
|
test: ['CMD', 'rabbitmq-diagnostics', '-q', 'ping'],
|
|
39
46
|
interval: 5000,
|
|
@@ -88,6 +95,22 @@ function getAddress (n = 0) {
|
|
|
88
95
|
return `localhost:${HOST_PORTS[n]}`
|
|
89
96
|
}
|
|
90
97
|
|
|
98
|
+
/**
|
|
99
|
+
* The queues on a broker, by name.
|
|
100
|
+
*
|
|
101
|
+
* @param {number} [n]
|
|
102
|
+
* @returns {Promise<string[]>}
|
|
103
|
+
*/
|
|
104
|
+
async function queues (n = 0) {
|
|
105
|
+
const authorization = 'Basic ' + Buffer.from(`${USER}:${PASSWORD}`).toString('base64')
|
|
106
|
+
const url = `http://localhost:${MANAGEMENT_HOST_PORTS[n]}/api/queues/%2F?columns=name`
|
|
107
|
+
const response = await fetch(url, { headers: { authorization } })
|
|
108
|
+
|
|
109
|
+
if (!response.ok) throw new Error(`Cannot read the queues of broker ${n}: ${response.status}`)
|
|
110
|
+
|
|
111
|
+
return (await response.json()).map((queue) => queue.name)
|
|
112
|
+
}
|
|
113
|
+
|
|
91
114
|
/**
|
|
92
115
|
* @param {number} [n]
|
|
93
116
|
*/
|
|
@@ -129,5 +152,6 @@ module.exports = {
|
|
|
129
152
|
startBrokers,
|
|
130
153
|
stopBrokers,
|
|
131
154
|
getAddress,
|
|
155
|
+
queues,
|
|
132
156
|
actions
|
|
133
157
|
}
|
package/features/steps/parked.js
CHANGED
|
@@ -1,11 +1,10 @@
|
|
|
1
1
|
'use strict'
|
|
2
2
|
|
|
3
3
|
const assert = require('node:assert')
|
|
4
|
-
const amqplib = require('amqplib')
|
|
5
4
|
const { Given, When, Then } = require('@cucumber/cucumber')
|
|
6
5
|
|
|
7
|
-
const { getAddress, USER, PASSWORD } = require('./brokers')
|
|
8
6
|
const { until, timeout } = require('../../test/helpers')
|
|
7
|
+
const { reading, pick } = require('./parking')
|
|
9
8
|
|
|
10
9
|
Given('a producer failing every request to the {token} queue',
|
|
11
10
|
/**
|
|
@@ -52,18 +51,9 @@ When('the parked request from the {token} queue is answered by hand',
|
|
|
52
51
|
* @this {comq.features.Context}
|
|
53
52
|
*/
|
|
54
53
|
async function (queue) {
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
try {
|
|
59
|
-
let parked
|
|
60
|
-
|
|
61
|
-
await channel.consume('comq.parked.' + queue, (message) => {
|
|
62
|
-
parked = message
|
|
63
|
-
channel.ack(message)
|
|
64
|
-
})
|
|
65
|
-
|
|
66
|
-
await until(() => parked !== undefined)
|
|
54
|
+
await reading(async (channel) => {
|
|
55
|
+
const parked = await pick(channel,
|
|
56
|
+
(one) => one.properties.headers?.['x-comq-queue'] === queue)
|
|
67
57
|
|
|
68
58
|
assert.notEqual(parked, undefined, `Nothing was parked from ${queue}`)
|
|
69
59
|
|
|
@@ -74,15 +64,14 @@ When('the parked request from the {token} queue is answered by hand',
|
|
|
74
64
|
|
|
75
65
|
this.answer = { answered: true, was: JSON.parse(parked.content.toString()) }
|
|
76
66
|
|
|
67
|
+
channel.ack(parked)
|
|
68
|
+
|
|
77
69
|
channel.sendToQueue(replyTo, Buffer.from(JSON.stringify(this.answer)),
|
|
78
70
|
{ correlationId, contentType })
|
|
79
71
|
|
|
80
72
|
// let the frame reach the broker before the connection goes
|
|
81
73
|
await timeout(100)
|
|
82
|
-
}
|
|
83
|
-
await channel.close()
|
|
84
|
-
await connection.close()
|
|
85
|
-
}
|
|
74
|
+
})
|
|
86
75
|
})
|
|
87
76
|
|
|
88
77
|
Then('the caller receives the reply',
|
|
@@ -0,0 +1,113 @@
|
|
|
1
|
+
'use strict'
|
|
2
|
+
|
|
3
|
+
const assert = require('node:assert')
|
|
4
|
+
const amqplib = require('amqplib')
|
|
5
|
+
const { Given, When, Then } = require('@cucumber/cucumber')
|
|
6
|
+
|
|
7
|
+
const { getAddress, USER, PASSWORD } = require('./brokers')
|
|
8
|
+
const { timeout } = require('../../test/helpers')
|
|
9
|
+
|
|
10
|
+
Given('(that )groupless events from the {token} exchange are causing exceptions',
|
|
11
|
+
/**
|
|
12
|
+
* @param {string} exchange
|
|
13
|
+
* @this {comq.features.Context}
|
|
14
|
+
*/
|
|
15
|
+
async function (exchange) {
|
|
16
|
+
await this.io.consume(exchange, (payload, properties) => {
|
|
17
|
+
this.attempts.push(properties.headers?.['x-comq-attempt'] ?? 1)
|
|
18
|
+
|
|
19
|
+
throw new Error('Expected exception')
|
|
20
|
+
})
|
|
21
|
+
})
|
|
22
|
+
|
|
23
|
+
When('the connection is closed',
|
|
24
|
+
/**
|
|
25
|
+
* @this {comq.features.Context}
|
|
26
|
+
*/
|
|
27
|
+
async function () {
|
|
28
|
+
await this.disconnect()
|
|
29
|
+
})
|
|
30
|
+
|
|
31
|
+
Then('a message parked from the {token} exchange is in the parking queue',
|
|
32
|
+
/**
|
|
33
|
+
* @param {string} exchange
|
|
34
|
+
* @this {comq.features.Context}
|
|
35
|
+
*/
|
|
36
|
+
async function (exchange) {
|
|
37
|
+
await reading(async (channel) => {
|
|
38
|
+
const message = await pick(channel,
|
|
39
|
+
(one) => one.properties.headers?.['x-comq-exchange'] === exchange)
|
|
40
|
+
|
|
41
|
+
assert.notEqual(message, undefined,
|
|
42
|
+
`Nothing in '${PARKED}' came from the '${exchange}' exchange`)
|
|
43
|
+
|
|
44
|
+
const headers = message.properties.headers
|
|
45
|
+
|
|
46
|
+
// a groupless subscriber consumes a name the broker chose, so what is asserted
|
|
47
|
+
// here is that the message says which queue, not which name that is
|
|
48
|
+
assert.notEqual(headers['x-comq-queue'], undefined,
|
|
49
|
+
'The parked message does not name the queue it came from')
|
|
50
|
+
|
|
51
|
+
assert.notEqual(headers['x-comq-reason'], undefined, 'The parked message has no reason')
|
|
52
|
+
|
|
53
|
+
channel.nack(message, false, true)
|
|
54
|
+
})
|
|
55
|
+
})
|
|
56
|
+
|
|
57
|
+
/**
|
|
58
|
+
* Runs `read` against a connection of its own, so that what it does to the parking
|
|
59
|
+
* queue is not what comq is doing to it.
|
|
60
|
+
*
|
|
61
|
+
* @param {(channel: import('amqplib').Channel) => Promise<void>} read
|
|
62
|
+
*/
|
|
63
|
+
async function reading (read) {
|
|
64
|
+
const connection = await amqplib.connect(`amqp://${USER}:${PASSWORD}@${getAddress(0)}`)
|
|
65
|
+
const channel = await connection.createChannel()
|
|
66
|
+
|
|
67
|
+
try {
|
|
68
|
+
await read(channel)
|
|
69
|
+
} finally {
|
|
70
|
+
await channel.close().catch(noop)
|
|
71
|
+
await connection.close().catch(noop)
|
|
72
|
+
}
|
|
73
|
+
}
|
|
74
|
+
|
|
75
|
+
/**
|
|
76
|
+
* The first message in the parking queue that `matches`, waited for. One queue holds what
|
|
77
|
+
* every scenario before this one left in it, so everything that does not match is given
|
|
78
|
+
* back; the match is left unacknowledged, for the caller to take or to return.
|
|
79
|
+
*
|
|
80
|
+
* @param {import('amqplib').Channel} channel
|
|
81
|
+
* @param {(message: import('amqplib').GetMessage) => boolean} matches
|
|
82
|
+
* @returns {Promise<import('amqplib').GetMessage | undefined>}
|
|
83
|
+
*/
|
|
84
|
+
async function pick (channel, matches) {
|
|
85
|
+
const deadline = Date.now() + DEADLINE
|
|
86
|
+
|
|
87
|
+
do {
|
|
88
|
+
const taken = []
|
|
89
|
+
|
|
90
|
+
let delivery
|
|
91
|
+
|
|
92
|
+
while ((delivery = await channel.get(PARKED, { noAck: false })) !== false) taken.push(delivery)
|
|
93
|
+
|
|
94
|
+
const match = taken.find(matches)
|
|
95
|
+
|
|
96
|
+
for (const one of taken) if (one !== match) channel.nack(one, false, true)
|
|
97
|
+
|
|
98
|
+
if (match !== undefined) return match
|
|
99
|
+
|
|
100
|
+
await timeout(50)
|
|
101
|
+
} while (Date.now() < deadline)
|
|
102
|
+
}
|
|
103
|
+
|
|
104
|
+
const PARKED = 'comq.parked'
|
|
105
|
+
|
|
106
|
+
/** The ladder the suite runs on is half a second; this is room for it and the broker. */
|
|
107
|
+
const DEADLINE = 5000
|
|
108
|
+
|
|
109
|
+
function noop () {}
|
|
110
|
+
|
|
111
|
+
exports.PARKED = PARKED
|
|
112
|
+
exports.reading = reading
|
|
113
|
+
exports.pick = pick
|
package/features/steps/poison.js
CHANGED
|
@@ -3,6 +3,7 @@
|
|
|
3
3
|
const assert = require('node:assert')
|
|
4
4
|
const { until } = require('../../test/helpers')
|
|
5
5
|
const { Then } = require('@cucumber/cucumber')
|
|
6
|
+
const { PARKED, reading, pick } = require('./parking')
|
|
6
7
|
|
|
7
8
|
Then('the event is attempted {int} time(s)',
|
|
8
9
|
/**
|
|
@@ -35,32 +36,29 @@ Then('the parked message is kept, and says it came from the {token} exchange',
|
|
|
35
36
|
*/
|
|
36
37
|
async function (exchange) {
|
|
37
38
|
const queue = exchange + '..exceptions' // the group the throwing consumer uses
|
|
38
|
-
const parked = 'comq.parked.' + queue
|
|
39
39
|
|
|
40
|
-
await
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
await until(() => this.parked[queue] !== undefined)
|
|
40
|
+
await reading(async (channel) => {
|
|
41
|
+
const message = await pick(channel,
|
|
42
|
+
(one) => one.properties.headers?.['x-comq-queue'] === queue)
|
|
45
43
|
|
|
46
|
-
|
|
44
|
+
assert.notEqual(message, undefined, `Nothing in '${PARKED}' came from ${queue}`)
|
|
47
45
|
|
|
48
|
-
|
|
46
|
+
const headers = message.properties.headers
|
|
49
47
|
|
|
50
|
-
|
|
48
|
+
assert.notEqual(headers['x-comq-reason'], undefined, 'The parked message has no reason')
|
|
51
49
|
|
|
52
|
-
|
|
53
|
-
|
|
50
|
+
// the message came back through the default exchange on every retry, so this
|
|
51
|
+
// is only right if the origin was recorded on the first failure
|
|
52
|
+
assert.equal(headers['x-comq-exchange'], exchange,
|
|
53
|
+
`The parked message says it came from '${headers['x-comq-exchange']}'`)
|
|
54
54
|
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
55
|
+
// absent on a message parked from its first delivery, exactly as a consumer reads it;
|
|
56
|
+
// derived rather than stated, so it cannot drift from the configured ladder
|
|
57
|
+
assert.equal(headers['x-comq-attempt'] ?? 1, this.attempts.length,
|
|
58
|
+
'The parked message did not have as many attempts as the consumer saw')
|
|
59
59
|
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
assert.equal(headers['x-comq-attempt'] ?? 1, this.attempts.length,
|
|
63
|
-
'The parked message did not have as many attempts as the consumer saw')
|
|
60
|
+
channel.ack(message)
|
|
61
|
+
})
|
|
64
62
|
})
|
|
65
63
|
|
|
66
64
|
Then('{token} has received {int} event(s)',
|
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
'use strict'
|
|
2
|
+
|
|
3
|
+
const assert = require('node:assert')
|
|
4
|
+
const { Then } = require('@cucumber/cucumber')
|
|
5
|
+
|
|
6
|
+
const { queues, BROKERS_AMOUNT } = require('./brokers')
|
|
7
|
+
|
|
8
|
+
Then('the broker holds {int} reply queue(s)',
|
|
9
|
+
/**
|
|
10
|
+
* @param {number} count
|
|
11
|
+
*/
|
|
12
|
+
async function (count) {
|
|
13
|
+
await held(0, count)
|
|
14
|
+
})
|
|
15
|
+
|
|
16
|
+
Then('each broker holds {int} reply queue(s)',
|
|
17
|
+
/**
|
|
18
|
+
* @param {number} count
|
|
19
|
+
*/
|
|
20
|
+
async function (count) {
|
|
21
|
+
for (let n = 0; n < BROKERS_AMOUNT; n++) await held(n, count)
|
|
22
|
+
})
|
|
23
|
+
|
|
24
|
+
/**
|
|
25
|
+
* @param {number} broker
|
|
26
|
+
* @param {number} count
|
|
27
|
+
*/
|
|
28
|
+
async function held (broker, count) {
|
|
29
|
+
const replies = (await queues(broker)).filter((name) => name.startsWith(PREFIX))
|
|
30
|
+
|
|
31
|
+
assert.equal(replies.length, count,
|
|
32
|
+
`Broker ${broker} holds ${replies.length} reply queue(s): ${replies.join(', ')}`)
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
const PREFIX = 'comq.reply'
|
package/package.json
CHANGED
package/readme.md
CHANGED
|
@@ -113,7 +113,8 @@ Send encoded Request message with `replyTo` and `correlationId` properties set a
|
|
|
113
113
|
return decoded Reply content. The promise stays pending until the Reply arrives, or until the
|
|
114
114
|
[timeout](#timeout) passes.
|
|
115
115
|
|
|
116
|
-
On the initial call,
|
|
116
|
+
On the initial call, the queue for Requests is asserted, and so is the one this `IO` receives
|
|
117
|
+
every Reply on.
|
|
117
118
|
|
|
118
119
|
`options` is the encoding, or an object:
|
|
119
120
|
|
|
@@ -550,7 +551,9 @@ requests and are expecting replies.
|
|
|
550
551
|
- An exchange is asserted as _fanout_ for [Emission](#emission)
|
|
551
552
|
and [Consumption](#consumption), and as _direct_ for [Routing](#routing). One name is one or
|
|
552
553
|
the other: asserting it as both is what the broker refuses.
|
|
553
|
-
-
|
|
554
|
+
- One queue takes every Reply an `IO` receives, `comq.reply..<id>`, _exclusive_ and gone with the
|
|
555
|
+
connection. A Reply is matched by its `correlationId`, which is unique across processes, so the
|
|
556
|
+
queue names neither what was called nor what is waiting.
|
|
554
557
|
- A queue [`back`](#addressed-requests) holds is _exclusive_, bound under its Key to a _direct_
|
|
555
558
|
exchange.
|
|
556
559
|
|
|
@@ -558,7 +561,8 @@ comq declares two kinds of queue of its own, for [failed messages](#retries):
|
|
|
558
561
|
|
|
559
562
|
- `comq.retry.<delay>`, with a fanout exchange of the same name, one pair per distinct rung of
|
|
560
563
|
the [backoff ladder](#retries), shared by every queue that uses it.
|
|
561
|
-
- `comq.parked
|
|
564
|
+
- `comq.parked`, one queue for everything parked, whatever it was consumed from, always
|
|
565
|
+
_durable_.
|
|
562
566
|
|
|
563
567
|
See [queue assertion options](https://amqp-node.github.io/amqplib/channel_api.html#channel_assertQueue).
|
|
564
568
|
|
|
@@ -638,8 +642,10 @@ verdict's own message.
|
|
|
638
642
|
|
|
639
643
|
#### Parked messages
|
|
640
644
|
|
|
641
|
-
A message that has run out of attempts is published to `comq.parked
|
|
642
|
-
only once the broker confirms it.
|
|
645
|
+
A message that has run out of attempts is published to `comq.parked` and acknowledged
|
|
646
|
+
only once the broker confirms it. There is one such queue, and it holds what every queue this
|
|
647
|
+
connection consumes could not process — including a groupless subscription's, which the broker
|
|
648
|
+
removes with the connection while what it could not process stays. It is not deleted, and it does not depend on a broker-side
|
|
643
649
|
policy. The [`discard`](#diagnostics) diagnostic event is emitted when it happens, and
|
|
644
650
|
[`retry`](#diagnostics) on every attempt before it.
|
|
645
651
|
|
|
@@ -648,9 +654,12 @@ name where it was originally published, `x-comq-queue` the queue it was consumed
|
|
|
648
654
|
`x-comq-reason` the exception's message, and `x-comq-at` when it was parked. Its original
|
|
649
655
|
properties are kept as they were.
|
|
650
656
|
|
|
651
|
-
|
|
652
|
-
|
|
653
|
-
|
|
657
|
+
A message is found by the queue it names rather than by the queue it is in: `comq.parked` holds
|
|
658
|
+
messages from every source, so reading one source's means matching on `x-comq-queue`.
|
|
659
|
+
|
|
660
|
+
The queue grows until somebody drains it, which is deliberate — the alternative is deleting
|
|
661
|
+
evidence. Nothing comq does removes it or what it holds. Alert on [`discard`](#diagnostics), and
|
|
662
|
+
do not delete `comq.retry.*` or `comq.parked` on a running system.
|
|
654
663
|
|
|
655
664
|
> **A parked Request is never answered.** A Consumer awaiting its Reply waits until its
|
|
656
665
|
> [timeout](#timeout), or indefinitely without one, and with a limited prefetch that can deadlock
|
|
@@ -734,28 +743,6 @@ Sending Requests, receiving Replies, and emitting Events will still be available
|
|
|
734
743
|
Keys held by [`back`](#addressed-requests) are withdrawn first, so a call published from then on is
|
|
735
744
|
refused.
|
|
736
745
|
|
|
737
|
-
### Abandoning
|
|
738
|
-
|
|
739
|
-
`async IO.abandon(): void`
|
|
740
|
-
|
|
741
|
-
Stop waiting for Replies.
|
|
742
|
-
|
|
743
|
-
A [close](#disconnection) waits for every `producer` and `consumer` call to return, and such a
|
|
744
|
-
call may itself be waiting on a Request of its own. Where the answer to that Request is the very
|
|
745
|
-
thing going away — the process it would come from is shutting down alongside this one — the close
|
|
746
|
-
waits for as long as the answer takes.
|
|
747
|
-
|
|
748
|
-
`IO.abandon()` ends that wait:
|
|
749
|
-
|
|
750
|
-
- every outstanding Reply is rejected with `Abandoned`, so a caller inside a `producer` or
|
|
751
|
-
`consumer` is told at once rather than held;
|
|
752
|
-
- a Request made from then on is refused with `Abandoned`, so a callback cannot start another
|
|
753
|
-
wait on the way out.
|
|
754
|
-
|
|
755
|
-
The Requests those Replies belonged to were sent and may well be processed; what is given up is
|
|
756
|
-
this side's interest in the answers. It may be called before a close or while one is already under
|
|
757
|
-
way, which is how a close that has begun is told to stop waiting.
|
|
758
|
-
|
|
759
746
|
### Disconnection
|
|
760
747
|
|
|
761
748
|
`async IO.close(): void`
|
|
@@ -764,9 +751,6 @@ way, which is how a close that has begun is told to stop waiting.
|
|
|
764
751
|
2. Wait for any outstanding messages to be processed[^2] and acknowledged.
|
|
765
752
|
3. Close the connection.
|
|
766
753
|
|
|
767
|
-
Step 2 still runs where the Replies have been [abandoned](#abandoning): each message is processed
|
|
768
|
-
to a verdict and acknowledged, and what was given up is the waiting rather than the drain.
|
|
769
|
-
|
|
770
754
|
[^2]: Therefore, if the underlying connection is lost, `.close()` will only be completed once the
|
|
771
755
|
connection is [recovered](#connection-tolerance).
|
|
772
756
|
|
package/source/.io/ReplyPipe.js
CHANGED
|
@@ -60,12 +60,25 @@ class ReplyPipe extends EventEmitter {
|
|
|
60
60
|
ok: { correlationId, replyTo: feedback.queue, ...CONTROL, headers: FLOW }
|
|
61
61
|
}
|
|
62
62
|
|
|
63
|
+
// The source is not read until the confirmation is published, and a blocked broker holds the
|
|
64
|
+
// publish for as long as it stays blocked. A source failing in the meantime — a reply stream
|
|
65
|
+
// being relayed times out — would raise an `error` nobody listens to. How it failed is not
|
|
66
|
+
// reported to the consumer anyway: `#pump` ends the stream once it finds the source destroyed.
|
|
67
|
+
stream.on('error', noop)
|
|
68
|
+
|
|
63
69
|
channel.diagnose('return', this.#onReturn)
|
|
64
70
|
feedback.on(correlationId, this.#control)
|
|
65
71
|
}
|
|
66
72
|
|
|
67
73
|
async pipe () {
|
|
68
|
-
|
|
74
|
+
try {
|
|
75
|
+
await this.#transmit(control.ok, this.#properties.ok)
|
|
76
|
+
} catch (exception) {
|
|
77
|
+
// nobody is going to read the source now
|
|
78
|
+
this.#interrupt()
|
|
79
|
+
|
|
80
|
+
throw exception
|
|
81
|
+
}
|
|
69
82
|
|
|
70
83
|
if (this.#closed) return
|
|
71
84
|
|
|
@@ -198,4 +211,6 @@ const CONTROL = { type: 'control', mandatory: true }
|
|
|
198
211
|
|
|
199
212
|
const FLOW = { [FLOW_HEADER]: true }
|
|
200
213
|
|
|
214
|
+
function noop () {}
|
|
215
|
+
|
|
201
216
|
exports.ReplyPipe = ReplyPipe
|