comq 0.17.1 → 0.19.0

package/docs/headers.md

@@ -15,3 +15,40 @@ to [amqplib.publish](https://amqp-node.github.io/amqplib/channel_api.html#channe
 
 `consumer` function's signature
 is `async? (payload: any, [properties: comq.amqp.Properties]): void`
+
+## Headers set by comq
+
+A message that failed at least once carries headers comq wrote, and a consumer sees them among
+the message properties.
+
+| Header | On | Meaning |
+|---|---|---|
+| `x-comq-attempt` | a retried message | Which attempt this delivery is, counting from one. Absent on the first, so read it as `headers?.['x-comq-attempt'] ?? 1`. The last attempt is one more than the number of rungs in the [backoff ladder](../readme.md#retries). |
+| `x-comq-exchange` | a retried or parked message | The exchange it was originally published to. |
+| `x-comq-key` | a retried or parked message | The routing key it was originally published with. |
+| `x-comq-queue` | a parked message | The queue it was consumed from. |
+| `x-comq-reason` | a parked message | The `message` of the exception that ended its attempts. |
+| `x-comq-cause` | a parked message | The `message` of that exception's `cause`, when it has one — a verdict is usually thrown with the failure that prompted it. |
+| `x-comq-at` | a parked message | When it was parked, as `Date.now()`. |
+
+`x-comq-exchange` and `x-comq-key` are recorded on the first failure rather than read at parking
+time: a message returning from the retry queue arrives through the default exchange, so by then
+its own delivery fields describe that hop rather than where it was published.
+
+AMQP defines no retry counter, so `x-comq-attempt` is comq's own rather than a convention —
+implementations that roll their own commonly use `x-retry-count` or `x-retries`, and none of
+those are standard either. It is prefixed for the same reason as the rest: an unprefixed name
+would be free to collide with the application's own headers, or with another library's.
+
+> Before this, the header was named `x-attempt`. A message already in flight under the old name
+> is read as a first delivery and gets a full ladder of attempts rather than the remainder of
+> one.
+
+A retried message also carries RabbitMQ's own [`x-death`](https://www.rabbitmq.com/docs/dlx),
+whose `x-death[0].count` looks like a second attempt counter. It is not the same number: it
+counts the times the message expired out of a retry queue, where `x-comq-attempt` counts the
+times it was delivered, so on a parked message with the default four-rung ladder they read `4`
+and `5`.
+
+**`x-comq-attempt` is the authoritative one** — it is comq's, and it is what the `attempts`
+setting is compared against. `x-death` is the broker's record, useful for its timestamps.

package/features/events.poison.feature

@@ -1,22 +1,53 @@
 Feature: Poison events
 
+  A message whose consumer throws is retried a few times and then kept, and the
+  consumers around it go on working.
+
   Background:
     Given an active connection to the broker
 
-  @manual
-  Scenario: Sending a poison event
+  Scenario: A poison event is retried and then parked
+
+    Four attempts, the default: the first delivery and three retries, each waiting
+    longer than the last. The counter is the `x-comq-attempt` header, which the first
+    delivery does not carry.
+
+    Given that events from the `poison_retried` exchange are causing exceptions
+    When an event is emitted to the `poison_retried` exchange
+    Then the event is attempted 4 times
+    And the message is parked
+    And the parked message is kept, and says it came from the `poison_retried` exchange
+
+  Scenario: A poison event does not stop the other consumers
+
+    The events channel is shared by every consumption in the process, so a
+    failure that sealed it would take this second consumer down with it.
+
+    Given that events from the `poison_isolated` exchange are causing exceptions
+    And that `witness` is consuming events from the `poison_witness` exchange
+    When an event is emitted to the `poison_isolated` exchange
+    And after 300ms
+    And an event is emitted to the `poison_witness` exchange
+    Then `witness` receives the event
+
+  Scenario: A retried event is not redelivered to the other consumers of its exchange
 
-    This scenario will cause an exception, run it once.
+    The exchange is a fanout: republishing the retry to it rather than to the
+    retry queue would hand the message to every subscriber again, once per attempt.
 
-    Given that events from the `poison` exchange are causing exceptions
-    Then an event is emitted to the `poison` exchange
-    And after 100ms
+    Given that events from the `poison_open` exchange are causing exceptions
+    And that `bystander` is consuming events from the `poison_open` exchange
+    When an event is emitted to the `poison_open` exchange
+    And after 1000ms
+    Then `bystander` has received 1 event
 
-  @manual
-  Scenario: Consuming poison events
+  Scenario: A consumer refusing a message parks it at once
 
-    After the poison event is sent, run this scenario five times
-    and verify that the poison event was discarded.
+    A `Park` says the consumer will never process this message, so climbing the
+    ladder would only delay the inevitable.
 
-    Given that events from the `poison` exchange are causing exceptions
-    And after 100ms
+    Given that events from the `poison_refused` exchange are refused as poison
+    When an event is emitted to the `poison_refused` exchange
+    Then the event is attempted 1 time
+    And the message is parked
+    And the parked message is kept, and says it came from the `poison_refused` exchange

package/features/steps/context.js

@@ -18,6 +18,9 @@ class Context extends World {
   eventsPublishedCount = 0
   eventsConsumedCount = 0
   events = {}
+  attempts = []
+  counts = {}
+  parked = {}
   processed
   enqueued
   tasksProcessedCount = 0
@@ -63,6 +66,9 @@ class Context extends World {
     this.io = undefined
     this.connected = false
     this.events = {}
+    this.attempts = []
+    this.counts = {}
+    this.parked = {}
   }
 
   /**
@@ -73,7 +79,9 @@ class Context extends World {
   async #connect (urls, method = connect) {
     if (this.io !== undefined) await this.disconnect()
 
-    this.io = await method(...urls)
+    // the retry delay is a topology setting, so the suite need not wait out a
+    // production one to see the mechanism work
+    this.io = await method(...urls, TOPOLOGY)
     this.connected = true
 
     for (const event of EVENTS) this.io.diagnose(event, () => (this.events[event] = true))
@@ -121,6 +129,14 @@ class Context extends World {
 const PROTOCOL = 'amqp://'
 
 /** @type {comq.diagnostics.Event[]} */
-const EVENTS = ['open', 'close', 'flow', 'discard', 'pause', 'resume', 'exhausted']
+const EVENTS = ['open', 'close', 'flow', 'discard', 'retry', 'pause', 'resume', 'exhausted']
+
+/** @type {comq.topology.Overrides} */
+// as many rungs as the presets have, so the suite exercises the shipped attempt count,
+// at a wall clock it can wait out
+const LADDER = [50, 100, 150, 200]
+
+/** @type {comq.topology.Overrides} */
+const TOPOLOGY = { event: { delay: LADDER }, request: { delay: LADDER } }
 
 exports.Context = Context

package/features/steps/events.js

@@ -38,13 +38,31 @@ Given('(that )events are exclusively consumed from the {token} exchange',
     this.consumptionPromise = consume.call(this, undefined, exchange)
   })
 
+Given('(that )events from the {token} exchange are refused as poison',
+  /**
+   * @param {string} exchange
+   * @this {comq.features.Context}
+   */
+  async function (exchange) {
+    const { Park } = require('../../')
+
+    await this.io.consume(exchange, 'exceptions', (payload, properties) => {
+      this.attempts.push(properties.headers?.['x-comq-attempt'] ?? 1)
+
+      throw new Park('Expected refusal')
+    })
+  })
+
 Given('(that )events from the {token} exchange are causing exceptions',
   /**
    * @param {string} exchange
    * @this {comq.features.Context}
    */
   async function (exchange) {
-    await this.io.consume(exchange, 'exceptions', () => {
+    await this.io.consume(exchange, 'exceptions', (payload, properties) => {
+      // absent on the first delivery, which is attempt one
+      this.attempts.push(properties.headers?.['x-comq-attempt'] ?? 1)
+
       throw new Error('Expected exception')
     })
   })
@@ -181,6 +199,7 @@ async function consume (group, exchange) {
   const consumer = async (payload, properties) => {
     this.consumed[group] = { payload, properties }
     this.eventsConsumedCount++
+    this.counts[group] = (this.counts[group] ?? 0) + 1
   }
 
   return group === undefined

package/features/steps/poison.js

@@ -1,15 +1,77 @@
 'use strict'
 
 const assert = require('node:assert')
-const { timeout } = require('../../test/helpers')
+const { until } = require('../../test/helpers')
 const { Then } = require('@cucumber/cucumber')
 
-Then('the message is discarded',
+Then('the event is attempted {int} time(s)',
+  /**
+   * @param {number} times
+   * @this {comq.features.Context}
+   */
+  async function (times) {
+    await until(() => this.attempts.length >= times)
+
+    const expected = Array.from({ length: times }, (_, index) => index + 1)
+
+    assert.deepEqual(this.attempts, expected,
+      `Expected attempts ${expected.join(', ')} but saw ${this.attempts.join(', ')}`)
+  })
+
+Then('the message is parked',
   /**
    * @this {comq.features.Context}
    */
   async function () {
-    if (this.events.discard !== true) await timeout(300)
+    await until(() => this.events.discard === true)
+
+    assert.equal(this.events.discard, true, 'The message was not parked')
+  })
+
+Then('the parked message is kept, and says it came from the {token} exchange',
+  /**
+   * @param {string} exchange
+   * @this {comq.features.Context}
+   */
+  async function (exchange) {
+    const queue = exchange + '..exceptions' // the group the throwing consumer uses
+    const parked = 'comq.parked.' + queue
+
+    await this.io.process(parked, (payload, properties) => {
+      this.parked[queue] = { payload, properties }
+    })
+
+    await until(() => this.parked[queue] !== undefined)
+
+    const message = this.parked[queue]
+
+    assert.notEqual(message, undefined, `Nothing was parked in ${parked}`)
+
+    const headers = message.properties.headers
+
+    assert.equal(headers['x-comq-queue'], queue, 'The parked message does not name its queue')
+    assert.notEqual(headers['x-comq-reason'], undefined, 'The parked message has no reason')
+
+    // the message came back through the default exchange on every retry, so this
+    // is only right if the origin was recorded on the first failure
+    assert.equal(headers['x-comq-exchange'], exchange,
+      `The parked message says it came from '${headers['x-comq-exchange']}'`)
+
+    // absent on a message parked from its first delivery, exactly as a consumer reads it;
+    // derived rather than stated, so it cannot drift from the configured ladder
+    assert.equal(headers['x-comq-attempt'] ?? 1, this.attempts.length,
+      'The parked message did not have as many attempts as the consumer saw')
+  })
+
+Then('{token} has received {int} event(s)',
+  /**
+   * @param {string} group
+   * @param {number} count
+   * @this {comq.features.Context}
+   */
+  async function (group, count) {
+    await until(() => this.counts[group] >= count)
 
-    assert.equal(this.events.discard, true, 'The {message} was not discarded')
+    assert.equal(this.counts[group] ?? 0, count,
+      `Expected ${group} to receive ${count} event(s), got ${this.counts[group] ?? 0}`)
   })

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "comq",
-  "version": "0.17.1",
+  "version": "0.19.0",
   "description": "ComQ",
   "author": {
     "name": "Artem Gurtovoi",

package/readme.md

@@ -425,6 +425,24 @@ dynamic, such as those that depend on runtime data like incoming messages, makin
 impossible or hard to maintain. The tradeoff of potentially encountering runtime topology
 declaration exceptions, which are more likely to happen during development, is deemed acceptable.
 
+### Settings
+
+Each channel type has a preset, and the trailing argument of `connect` overrides it:
+
+```javascript
+const io = await comq.connect(url, {
+  event: { delay: [5000, 60000] },  // two retries
+  request: { delay: 1000 }          // one
+})
+```
+
+`delay` governs [retries](#retries); the rest of [the settings](./types/topology.d.ts) are not
+meant to be changed.
+
+> Changing `delay` declares new retry queues rather than redeclaring the existing ones, so a
+> rolling deploy that changes it has no window in which either version fails. The queues left
+> behind are empty and can be removed once nothing is publishing to them.
+
 ### Channels
 
 `IO` lazy creates individual channels for Requests, Replies, and Events.
@@ -450,6 +468,12 @@ requests and are expecting replies.
   the other: asserting it as both is what the broker refuses.
 - Queues for Replies are _exclusive_ and _auto deleted_.
 
+comq declares two kinds of queue of its own, for [failed messages](#retries):
+
+- `comq.retry.<delay>`, with a fanout exchange of the same name, one pair per distinct rung of
+  the [backoff ladder](#retries), shared by every queue that uses it.
+- `comq.parked.<queue>`, one per consumed queue, declared to live as long as it does.
+
 See [queue assertion options](https://amqp-node.github.io/amqplib/channel_api.html#channel_assertQueue).
 
 ### Messages
@@ -461,27 +485,125 @@ See [queue assertion options](https://amqp-node.github.io/amqplib/channel_api.ht
   manual [acknowledgment mode](https://www.rabbitmq.com/confirms.html#acknowledgment-modes),
   and Replies are consumed using automatic mode.
 
-If an incoming message causes an exception, the corresponding channel is sealed, the message is republished, and the
-exception is thrown.
-If the message causes exceptions five times in a row, it is discarded.
+#### Retries
+
+If an incoming message causes an exception, comq publishes it to a *retry queue* and only then
+acknowledges the original. The retry queue has no consumer: it holds the message for
+`delay` milliseconds and then returns it to the queue it came from, so the wait is the broker's
+and outlives a restart of this process without holding a delivery against the
+[prefetch limit](#channels).
+
+Each attempt increments the [`x-comq-attempt`](./docs/headers.md) header, which the consumer
+receives among the message properties. `delay` is a backoff ladder with **one rung per retry**,
+so its length decides how many there are: the four rungs of the default are five attempts, and
+once a message has climbed it there is nowhere left to wait and it is *parked*.
+
+The channel keeps consuming throughout. A message one consumer cannot handle stops neither the
+other consumers nor that consumer's next message; only the message that failed is delayed.
+
+`delay` is a [topology](#topology) setting:
+
+| | ladder | attempts | total |
+|---|---|---|---|
+| Event | 1s, 10s, 30s, 90s | 5 | 131s |
+| Request | 1s, 3s, 5s, 10s | 5 | 19s |
+
+Requests are shorter because a caller is blocked on one with no timeout, and a Reply arriving
+long after it gave up has nowhere useful to land. Nobody waits on an Event.
+
+One retry queue and one exchange are declared per distinct wait and shared by every queue that
+uses them, so their number grows with the length of the ladder rather than with the number of
+queues.
+
+#### Saying how it failed
+
+A consumer can classify its failure, which decides whether the message is worth another
+attempt at all:
+
+```javascript
+const { Retry, Park } = require('comq')
+
+await io.consume('orders', 'billing', async (order) => {
+  if (typeof order.total !== 'number') throw new Park('no total on order')
+
+  try {
+    await billing.charge(order)
+  } catch (e) {
+    throw new Retry('billing unavailable', { cause: e })
+  }
+})
+```
+
+- `Park` says this consumer will never process this message, however many times it is handed
+  over. It is [parked](#parked-messages) at once, without climbing the ladder.
+- `Retry` says whatever was missing may be back shortly. This is what an unclassified rejection
+  already means, so throwing it changes nothing but the reader's certainty.
+
+**A bare rejection means `Retry`** — a failure nobody classified is a failure nobody chose — so
+consumers written before this behave exactly as they did.
+
+Both are `Error`s and both take a `cause`, which is recorded on the parked message alongside the
+verdict's own message.
+
+> **`Park` is not applicable to a `Producer` given to [`IO.reply`](#reply).** A Request
+> has a caller awaiting a reply, so what happens to a failed one is not a policy choice. A `Park`
+> thrown there is treated as an ordinary failure — retried, then parked on the count — and the
+> parked message says so as its reason.
+
+#### Parked messages
+
+A message that has run out of attempts is published to `comq.parked.<queue>` and acknowledged
+only once the broker confirms it. It is not deleted, and it does not depend on a broker-side
+policy. The [`discard`](#diagnostics) diagnostic event is emitted when it happens, and
+[`retry`](#diagnostics) on every attempt before it.
+
+A parked message carries what a person looking at it needs: `x-comq-exchange` and `x-comq-key`
+name where it was originally published, `x-comq-queue` the queue it was consumed from,
+`x-comq-reason` the exception's message, and `x-comq-at` when it was parked. Its original
+properties are kept as they were.
+
+Parked queues grow until somebody drains them, which is deliberate — the alternative is deleting
+evidence. Alert on [`discard`](#diagnostics), and do not delete `comq.retry.*` or `comq.parked.*`
+queues on a running system.
+
+> **A parked Request is never answered.** A Consumer awaiting its Reply waits indefinitely, and
+> with a limited prefetch that can deadlock it. Parking keeps the Request rather than deleting
+> it — and it keeps `replyTo` and `correlationId`, so a Reply can still be produced from it by
+> hand while the caller is alive — but comq itself sends no Reply and reports no error to the
+> caller. Give a Request a timeout on the calling side if you cannot tolerate that.
+
+#### What is guaranteed
+
+The copy is published *before* the original is acknowledged, so a process that dies between the
+two leaves the broker holding both and the message is handled twice: **consumers must be
+idempotent**. That is the deliberate trade — a message the broker holds twice can be recovered,
+one it no longer holds at all cannot.
+
+A retried message re-enters its queue behind the messages published while it waited, so
+**ordering is not preserved across a failure**.
 
-> It is highly recommended to set up a dead letter exchange policy to analyze messages that caused
-> exceptions. Note that in some cases, if the problematic message is a Request, a Consumer will
-> never receive a Reply, and this can result in a prefetch deadlock of a Consumer.
+Retries and parked messages are published *persistent* whatever the channel is, so they survive a
+restart of the broker even on the Request channel. Ordinary publishing is untouched: Requests and
+Replies stay [delivery mode 1](#messages), and only a message that has already failed is written
+to disk. Two weaker points remain on that channel: it
+does not use publisher confirms, so comq has no positive acknowledgement that the broker took the
+copy; and the return hop of a retry is performed by the broker's dead-lettering, which on classic
+queues is at-most-once and can lose the message if the source queue is unavailable when the delay
+expires.
 
 See:
 
 - [Consumer Acknowledgments and Publisher Confirms](https://www.rabbitmq.com/confirms.html)
-- [Negative Acknowledgment and Requeuing of Deliveries](https://www.rabbitmq.com/confirms.html#consumer-nacks-requeue)
 - [Dead Letter Exchanges](https://www.rabbitmq.com/dlx.html)
+- [At-Least-Once Dead Lettering](https://www.rabbitmq.com/blog/2022/03/29/at-least-once-dead-lettering)
 
 ### Cheatsheet
 
-| Message | Prefetch  | Confirms | Queue     | Acknowledgment | Persistent |
-|---------|-----------|----------|-----------|----------------|------------|
-| Request | limited   | no       | durable   | manual         | no         |
-| Reply   | unlimited | no       | exclusive | automatic      | no         |
-| Event   | limited   | yes      | durable   | manual         | yes        |
+| Message | Prefetch  | Confirms | Queue     | Acknowledgment | Persistent | Retries          |
+|---------|-----------|----------|-----------|----------------|------------|------------------|
+| Request | limited   | no       | durable   | manual         | no         | 1s, 3s, 5s, 10s  |
+| Reply   | unlimited | no       | exclusive | automatic      | no         | —                |
+| Event   | limited   | yes      | durable   | manual         | yes        | 1s, 10s, 30s, 90s |
 
 ## Graceful shutdown
 
@@ -553,10 +675,12 @@ Subscribe to one of the diagnostic events:
 - `lost`: a shard has lost its connection, hence the requests awaiting their replies on it are
   re-sent. Channel type is passed.
 - `recover`: channel's topology is recovered. Channel type is passed.
-- `discard`: message is [discarded](#messages) as it repeatedly caused
-  exceptions. Channel type,
+- `discard`: message is [parked](#parked-messages), having run out of attempts. Channel type,
   raw [amqp message object](https://amqp-node.github.io/amqplib/channel_api.html#channel_consume)
   and the exception are passed as arguments.
+- `retry`: message caused an exception and has been published to its
+  [retry queue](#retries). Channel type, the raw amqp message object, the exception and the
+  attempt number are passed as arguments.
 - `return`: message is returned by the broker as unroutable. Channel type and the raw
   [amqp message object](https://amqp-node.github.io/amqplib/channel_api.html#channel_publish) are
   passed as arguments. In the case of a [sharded connection](#sharded-connection), the message is