comq 0.23.0 → 0.24.1

package/discussions/amqp-1.0.md

@@ -0,0 +1,127 @@
+# AMQP 1.0
+
+**comq stays on AMQP 0-9-1 over [amqplib](https://github.com/amqp-node/amqplib).**
+
+RabbitMQ 4 speaks AMQP 1.0 natively and keeps 0-9-1 as a first-class protocol; both are
+supported out of the box and neither is on its way out. The choice is therefore comq's to make on
+what each protocol and each client gives it, and the answer today is that 1.0 costs a rewrite of
+the transport layer, changes comq's public API, breaks the wire between comq versions, and
+returns nothing comq uses.
+
+Two separate questions hide in "switch to the official client", and they have different answers.
+The client is unusable for comq today. The protocol is usable and in places cleaner, and still
+does not pay for itself.
+
+Measured on 2026-09-20 against RabbitMQ 4.3.6, with `rabbitmq-amqp-js-client` 1.0.0, `rhea` 3.0.5
+and `amqplib` 2.0.1.
+
+## The client
+
+[`rabbitmq-amqp-js-client`](https://github.com/coders51/rabbitmq-amqp-js-client) is listed among
+RabbitMQ's AMQP 1.0 client libraries and is maintained by coders51 rather than by the RabbitMQ
+team, unlike the Java, .NET, Go and Python clients. It is a thin wrapper over
+[rhea](https://github.com/amqp/rhea), which does the protocol.
+
+What it offers an application is a connection, a publisher, a consumer and a management link.
+What comq needs from a client, it does not reach:
+
+| comq | the client |
+|---|---|
+| 16 [diagnostic events](../readme.md#diagnostics) | connection, publisher and consumer state changes are unobservable ([#40](https://github.com/coders51/rabbitmq-amqp-js-client/issues/40)) |
+| [`seal`](../readme.md#sealing) stops consuming and waits | `Consumer.close` is synchronous and returns nothing ([#34](https://github.com/coders51/rabbitmq-amqp-js-client/issues/34)) |
+| topology recovered after a reconnect | the links come back, the topology does not ([#42](https://github.com/coders51/rabbitmq-amqp-js-client/issues/42)) |
+| prefetch 300, per channel | rhea's default credit window of 1000, fixed |
+| Requests and Replies are published unconfirmed | `publish` always awaits settlement |
+| `amqp://user:pass@host/vhost`, heartbeat 15s | host, port, username, password |
+
+Three defects it has today would each stop comq on its own:
+
+- **A publish to a queue that does not exist never settles.** The broker refuses the attach with
+  `amqp:not-found`; `createPublisher` resolves regardless, and the `publish` promise stays
+  pending forever.
+- **Attaching to another connection's exclusive queue kills the process.** `createConsumer`
+  resolves, the broker ends the session, and rhea emits an unhandled `error` on a container the
+  client keeps private. That is exactly what [`back`](../readme.md#addressed-requests) does on
+  every key already held.
+- **Every body goes out as an `amqp-value` section**, Buffers included, so a 0-9-1 consumer
+  receives the section bytes in front of the payload and `type: 'amqp-1.0'` among the properties.
+  A process on 0-9-1 and a process on the client cannot exchange a message, which rules out a
+  rolling migration. rhea with an explicit `data_section` interoperates cleanly, so this is the
+  client's message model rather than the protocol.
+
+Queue `arguments` are typed `Record<string, string>` while the broker requires integers for
+`x-message-ttl` (`expected integer, got longstr`) and accepts the numbers the typing forbids. The
+dependency on rhea is `github:amqp/rhea` — a git reference that resolves to `git+ssh` in a
+lockfile and builds on install, which would land in comq's dependency tree and in every CI that
+installs comq.
+
+Adopted with those gaps filled locally, "the official client" becomes "rhea, plus a management
+link we wrote" — which is the layer comq already is, over a client with fifteen years behind it.
+
+## The protocol
+
+Everything comq does is expressible over AMQP 1.0, and three mechanisms fit it better than 0-9-1
+does:
+
+- **An unroutable publish comes back as a `RELEASED` outcome on the send itself**, where 0-9-1
+  reports it asynchronously through `mandatory` and `basic.return` and comq has to correlate it.
+- **The `ttl` header is honoured per message**, so a [`call`](../readme.md#timeout)'s expiration
+  carries unchanged.
+- **Credit is granted per link**, which is what comq's [channel segregation](../readme.md#channels)
+  approximates with a prefetch count per channel.
+
+The rest holds: an exclusive queue declared over the management link is refused to a second
+connection (`400 cannot obtain exclusive access`), so `back` still holds a Key; numeric queue
+arguments are accepted, so the [retry ladder](../readme.md#retries) and `comq.parked` are
+unchanged. Streams with offsets and server-side filters are the one capability 0-9-1 cannot match,
+and comq consumes no streams.
+
+### What it costs
+
+**comq's public API is shaped by 0-9-1.** `Properties` is in the consumer signature, `emit`,
+`enqueue` and `route` take publish options, and `discard`, `retry` and `return` hand listeners a
+raw amqplib message. Every one of those changes, and so does what reads them — `@toa.io/bindings.amqp`
+reads `message.properties.correlationId` and `properties.headers`.
+
+**Message properties land elsewhere.** A 0-9-1 header prefixed `x-` arrives over 1.0 as a message
+annotation rather than an application property, beside the broker's own `x-exchange` and
+`x-routing-key`; `replyTo: 'rq'` arrives as `/queues/rq`. comq's [`x-comq-*` headers](../docs/headers.md)
+move with them.
+
+**The management UI and API stop reporting message rates** for AMQP 1.0 connections; queue depth
+survives, `message_stats` is null. Frame-intercepting plugins and transactions are 0-9-1 only,
+and comq uses neither.
+
+**The work is the transport layer and the unit suite.** `source/connection.js` and
+`source/channel.js` are 1,256 lines written against amqplib's channel API, `source/io.js` another
+797 that speak its vocabulary, and `test/` is built on `test/amqplib.mock.js` — some 6,400 lines
+that go with the thing they mock. The 88 scenarios in `features/` are behaviour-level and would
+carry over, which makes them the whole safety net for the change.
+
+### What it does not buy
+
+Throughput is level and the latency comq optimises for is worse. Same broker over loopback,
+20,000 messages of ~180 bytes, three runs, `TCP_NODELAY` on both:
+
+| | amqplib | rhea |
+|---|---|---|
+| publish, awaiting confirmation or settlement | 77–88k msg/s | 79–103k msg/s |
+| consume and acknowledge | 121–143k msg/s | 110–128k msg/s |
+
+Sequential request and reply, 2,000 round trips, one in flight:
+
+| | p50 | p99 |
+|---|---|---|
+| amqplib | 0.20 ms | 0.36–0.41 ms |
+| rhea | 0.26–0.29 ms | 0.51–0.57 ms |
+
+rhea enables `TCP_NODELAY` when a receiver is attached and leaves it to the `tcp_no_delay` option
+otherwise, so a connection that only publishes is left to Nagle — the same delay comq took out of
+amqplib in 0.20.1, and worth 22 ms per round trip when it is missed.
+
+## What would reverse this
+
+- comq needs RabbitMQ streams with server-side filtering.
+- The client exposes connection and link state, a consumer close that waits, a credit setting and
+  a vhost — the four things its own issues already name.
+- RabbitMQ announces an end for AMQP 0-9-1.

package/features/heartbeat.feature

@@ -0,0 +1,16 @@
+@heavy
+Feature: Idle connection tolerance
+
+  A connection that says nothing but heartbeats is as healthy as a busy one, and is kept for as
+  long as the broker goes on answering them, however long that is.
+
+  Background:
+    Given 1s AMQP heartbeat
+
+  Scenario: Keeping an idle connection
+    Given an active connection to the broker
+    Then the connection is not lost for 20 seconds
+
+  Scenario: Keeping an idle sharded connection
+    Given an active sharded connection
+    Then the connection is not lost for 20 seconds

package/features/rpc.call.feature

@@ -65,7 +65,29 @@ Feature: Calls under a key
   Scenario: A caller stops waiting for a holder that crashed
     Given a holder never answering `void` under the `a` key
     When the consumer calls `void` under the `a` key with a 1000ms timeout
-    And after 100ms
+    And the silent holder has taken the call
     And the silent holder crashes
     Then the consumer stops waiting
     And a call to `void` under the `a` key is refused within 10 seconds
+
+  Scenario: A caller stops waiting at its timeout
+    Given a holder on another connection answering `slow` under the `a` key in 1000ms
+    When the consumer calls `slow` under the `a` key with a 200ms timeout
+    Then the consumer stops waiting
+
+  Scenario: A call its holder has not taken in time is never processed
+    Given a holder not taking `later` under the `a` key
+    When the consumer calls `later` under the `a` key with a 200ms timeout
+    Then the consumer stops waiting
+    When after 300ms
+    And the idle holder starts taking
+    Then the idle holder takes nothing
+
+  Scenario: A call abandoned by its signal stays in its queue
+    Given a holder not taking `stray` under the `a` key
+    When the consumer calls `stray` under the `a` key with a signal
+    And after 100ms
+    And the signal aborts
+    Then the consumer stops waiting for the aborted call
+    When the idle holder starts taking
+    Then the idle holder takes the call

package/features/rpc.call.recovery.feature

@@ -17,14 +17,14 @@ Feature: Calls under a key across failures
     it the key. The holder finds the key taken until the broker lets that connection go, and
     claims it then.
 
-    Given watchdog interval is set to 2000ms with 60s AMQP heartbeat
+    Given 1s AMQP heartbeat
     And a network that can go silent
     And an active connection to the broker
     And a holder answering `echo` under the `a` key:
       """
       (payload) => payload
       """
-    When the network goes silent
+    When the network goes silent, with the broker still hearing heartbeats
     Then the connection is lost within 10 seconds
     And the holder finds its key taken while the broker holds the silent connection
     When the silent connection is let go
@@ -39,13 +39,6 @@ Feature: Calls under a key across failures
     And the broker is up
     Then the call ends
 
-  Scenario: A caller stops waiting for a request at its timeout while the broker is down
-    Given an active connection to the broker
-    And a producer replying `echo` queue
-    When the broker has crashed
-    And the consumer sends a request to the `echo` queue with a 1000ms timeout
-    Then the consumer stops waiting within 5 seconds
-
   Scenario: A caller stops waiting for a call at its timeout while the broker is down
     Given an active connection to the broker
     And a holder on another connection answering `echo` under the `a` key
@@ -53,20 +46,6 @@ Feature: Calls under a key across failures
     And the consumer calls `echo` under the `a` key with a 1000ms timeout
     Then the consumer stops waiting within 5 seconds
 
-  Scenario: A request re-sent after the broker restarts expires at its caller's deadline
-
-    The request is lost with the broker and sent again once the connection is restored, carrying
-    the time its caller has left.
-
-    Given an active connection to the broker
-    When the consumer sends a request to the `resent` queue with a 30000ms timeout
-    And after 200ms
-    And the broker has crashed
-    And the broker is up
-    Then the consumer stops waiting within 40 seconds
-    When a producer counting requests to the `resent` queue
-    Then the producer receives nothing
-
   Scenario: A call reaches its holder while one of the brokers is down
     Given an active sharded connection
     And a holder on another connection answering `echo` under the `a` key

package/features/rpc.parked.feature

@@ -5,13 +5,13 @@ Feature: Parked requests
 
   Scenario: A parked request can still be answered
 
-    A caller that sets no timeout is certain of an eventual answer for as long as it is
-    alive, and parking is what keeps that true when no consumer can produce one: the
-    request is kept, along with who was asking. Somebody else can answer it, and the
-    caller — still waiting, still holding an exclusive reply queue — receives it.
+    A caller is certain of an eventual answer for as long as it is alive, and parking
+    is what keeps that true when no consumer can produce one: the request is kept, along
+    with who was asking. Somebody else can answer it, and the caller — still waiting,
+    still holding an exclusive reply queue — receives it.
 
-    A caller that sets a timeout trades that for an end to the wait: once it stops
-    waiting, a parked request is addressed to a caller that has gone.
+    This is why a request has no timeout and cannot be withdrawn: either would leave the
+    parked message addressed to a caller that is no longer listening.
 
     Given a producer failing every request to the `unanswerable` queue
     When a request is sent to the `unanswerable` queue

package/features/rpc.shards.retransmission.feature

@@ -0,0 +1,46 @@
+@heavy
+Feature: Re-sending the requests a lost shard held
+
+  A Request is re-sent when the shard it went through is lost, since its Reply may be lost with it.
+  One that went through a shard that stays is answered through that shard, and is never re-sent
+  because another one was lost: a Request is not assumed to be idempotent.
+
+  Each scenario first takes one broker away from the consumer, so that the Requests it sends go
+  through the other one, and gives the broker back before losing one of them again.
+
+  Background:
+    Given 1s AMQP heartbeat
+    And a network that can go silent
+    And an active sharded connection
+    And a producer replying `echo` queue
+    # the consumer's channels exist on both brokers before one of them is taken away
+    And the consumer sends a request to the `echo` queue
+    And the consumer receives the reply
+    And a producer on another connection counting the requests to the `numbered` queue, answering when told
+
+  Scenario: Requests through the broker that stays are not re-sent when the other one is lost
+    Given the consumer's connection to broker 1 is cut
+    When the consumer sends 10 numbered requests to the `numbered` queue
+    And the consumer's connection to broker 1 is restored
+    And the consumer's connection to broker 1 is cut
+    And the producer answers
+    Then every numbered request is answered within 10 seconds
+    And every numbered request has been executed once
+
+  Scenario: Replies to the requests through the broker that stays are not lost with the other one
+    Given the consumer's connection to broker 1 is cut
+    When the consumer sends 10 numbered requests to the `numbered` queue
+    And the consumer's connection to broker 1 is restored
+    And the consumer's network to broker 1 goes silent
+    And the producer answers
+    Then every numbered request is answered within 15 seconds
+    And every numbered request has been executed once
+
+  Scenario: Requests through the broker that is lost are re-sent and answered
+    Given the consumer's connection to broker 0 is cut
+    When the consumer sends 10 numbered requests to the `numbered` queue
+    And the consumer's connection to broker 0 is restored
+    And the consumer's network to broker 1 goes silent
+    And the producer answers
+    Then every numbered request is answered within 15 seconds
+    And every numbered request has been executed twice

package/features/shards.feature

@@ -49,7 +49,7 @@ Feature: Sharded Connection
     Then the broker is up
 
   Scenario: Shard stops responding while requests are sent
-    Given watchdog interval is set to 3000ms with 1s AMQP heartbeat
+    Given 1s AMQP heartbeat
     And an active sharded connection
     And a producer replying `stalled` queue
     When the consumer sends 10 requests to the `stalled` queue as a broker freezes

package/features/silence.feature

@@ -2,10 +2,11 @@ Feature: Silent Connection Tolerance
 
   A connection that has gone silent is what a machine that woke from sleep is
   left with: the socket is open, neither the broker nor the operating system
-  reports anything, and nothing but the watchdog tells it from an idle connection.
+  reports anything, and nothing but the missing heartbeats tell it from an idle
+  connection.
 
   Background:
-    Given watchdog interval is set to 2000ms with 60s AMQP heartbeat
+    Given 1s AMQP heartbeat
     And a network that can go silent
 
   Scenario: Restoring a connection that has gone silent

package/features/steps/.types/context.d.ts

@@ -49,7 +49,11 @@ declare namespace comq.features {
     readonly address: string
 
     open(): Promise<void>
-    silence(): void
+    silence(heard?: boolean): void
+    refuse(): void
+    admit(): void
+    release(): void
+    cut(): void
     close(): Promise<void>
   }
 
@@ -57,6 +61,7 @@ declare namespace comq.features {
     client: Socket
     upstream: Socket
     silent: boolean
+    heartbeat?: NodeJS.Timeout
   }
 
 }

package/features/steps/brokers.js

@@ -139,7 +139,7 @@ const actions = {
     await docker('docker', ['kill', `comq-rmq-${n}`])
   },
   // a frozen broker keeps its connections open while answering nothing, which is
-  // what a publisher cannot tell from a working one until the watchdog fires
+  // what a publisher cannot tell from a working one until its heartbeats go missing
   frozen: async (n = 0) => {
     await docker('docker', ['pause', `comq-rmq-${n}`])
   }

package/features/steps/call.js

@@ -1,7 +1,7 @@
 'use strict'
 
 const assert = require('node:assert')
-const amqplib = require('amqplib')
+const amqplib = require('@toa.io/amqplib')
 const { Given, When, Then, After } = require('@cucumber/cucumber')
 
 const { connect, Unroutable } = require('../../')
@@ -84,11 +84,76 @@ Given('a holder never answering {token} under the {token} key',
     await channel.assertExchange(exchange, 'direct', { durable: true })
     await channel.assertQueue(queue, { exclusive: true })
     await channel.bindQueue(queue, exchange, key)
-    await channel.consume(queue, () => undefined)
+    await channel.consume(queue, () => (this.handed = true))
 
     this.silent = connection
   })
 
+Then('the silent holder has taken the call',
+  /**
+   * The publication is awaited nowhere, and what the crash has to find already queued is what
+   * the holder has been handed.
+   *
+   * @this {comq.features.Context}
+   */
+  async function () {
+    assert.equal(await until(() => this.handed === true, 10_000), true,
+      'The silent holder was handed nothing')
+  })
+
+Given('a holder not taking {token} under the {token} key',
+  /**
+   * Holds the key as comq does, and takes nothing until told to, so that a call waits in its
+   * queue. Each scenario holds a key of its own: the broker releases an exclusive queue once it
+   * has noticed the connection close, and a scenario that declared it again before then would be
+   * refused it.
+   *
+   * @param {string} exchange
+   * @param {string} key
+   * @this {comq.features.Context}
+   */
+  async function (exchange, key) {
+    const connection = await amqplib.connect(url(0))
+    const channel = await connection.createChannel()
+    const queue = exchange + '.' + key
+
+    await channel.assertExchange(exchange, 'direct', { durable: true })
+    await channel.assertQueue(queue, { exclusive: true })
+    await channel.bindQueue(queue, exchange, key)
+
+    this.idle = { connection, channel, queue, taken: 0 }
+  })
+
+When('the idle holder starts taking',
+  /**
+   * @this {comq.features.Context}
+   */
+  async function () {
+    const idle = this.idle
+
+    await idle.channel.consume(idle.queue, () => idle.taken++, { noAck: true })
+  })
+
+When('the consumer calls {token} under the {token} key with a signal',
+  /**
+   * @param {string} exchange
+   * @param {string} key
+   * @this {comq.features.Context}
+   */
+  async function (exchange, key) {
+    this.controller = new AbortController()
+
+    call(this, this.io.call(exchange, key, { key }, { signal: this.controller.signal }))
+  })
+
+When('the signal aborts',
+  /**
+   * @this {comq.features.Context}
+   */
+  function () {
+    this.controller.abort()
+  })
+
 When('the silent holder crashes',
   /**
    * Its socket is destroyed without a word, as a process that dies leaves it.
@@ -149,16 +214,6 @@ When('the consumer calls {token} under the {token} key {number} times',
     this.replies = await Promise.all(calls)
   })
 
-When('the consumer sends a request to the {token} queue with a {number}ms timeout',
-  /**
-   * @param {string} queue
-   * @param {number} ms
-   * @this {comq.features.Context}
-   */
-  async function (queue, ms) {
-    call(this, this.io.request(queue, { queue }, { timeout: ms }))
-  })
-
 When('the holder is sealed',
   /**
    * @this {comq.features.Context}
@@ -192,19 +247,32 @@ When('the first holder disconnects',
     await this.holder.close()
   })
 
-When('a producer counting requests to the {token} queue',
+Then('the idle holder takes nothing',
   /**
-   * @param {string} queue
    * @this {comq.features.Context}
    */
-  async function (queue) {
-    this.produced = 0
+  async function () {
+    await timeout(500)
 
-    await this.io.reply(queue, () => {
-      this.produced++
+    assert.equal(this.idle.taken, 0, 'An expired call was taken')
+  })
 
-      return null
-    })
+Then('the idle holder takes the call',
+  /**
+   * @this {comq.features.Context}
+   */
+  async function () {
+    await until(() => this.idle.taken > 0, 5000)
+
+    assert.equal(this.idle.taken, 1)
+  })
+
+Then('the consumer stops waiting for the aborted call',
+  /**
+   * @this {comq.features.Context}
+   */
+  async function () {
+    await assert.rejects(this.reply, (error) => error.name === 'AbortError')
   })
 
 Then('the call is refused as unroutable',
@@ -373,16 +441,6 @@ Then('every call is answered',
     assert.deepEqual(this.replies, this.replies.map((_, n) => n))
   })
 
-Then('the producer receives nothing',
-  /**
-   * @this {comq.features.Context}
-   */
-  async function () {
-    await timeout(500)
-
-    assert.equal(this.produced, 0, 'An expired request was processed')
-  })
-
 After(
   /**
    * @this {comq.features.Context}
@@ -390,9 +448,11 @@ After(
   async function () {
     await Promise.all((this.holders ?? []).map((io) => io.close()))
     await this.silent?.close().catch(() => undefined)
+    await this.idle?.connection.close().catch(() => undefined)
 
     this.holders = []
     this.silent = undefined
+    this.idle = undefined
   })
 
 /**

package/features/steps/connection.js

@@ -5,6 +5,16 @@ const { timeout } = require('../../test/helpers')
 
 const { Given, When, Then } = require('@cucumber/cucumber')
 
+Given('{number}s AMQP heartbeat',
+  /**
+   * A connection that has received nothing for two heartbeats is taken for lost.
+   *
+   * @param {number} heartbeat
+   */
+  function (heartbeat) {
+    global.COMQ_TESTING_AMQP_HEARTBEAT = heartbeat
+  })
+
 Given('an active connection to the broker',
   /**
    * @this {comq.features.Context}
@@ -176,6 +186,19 @@ Then('the connection is lost within {number} second(s)',
     await emitted.call(this, 'close', seconds, 'connection was not lost')
   })
 
+Then('the connection is not lost for {number} seconds',
+  /**
+   * @param {number} seconds
+   * @this {comq.features.Context}
+   */
+  async function (seconds) {
+    delete this.events.close
+
+    await timeout(seconds * 1000)
+
+    assert.equal(this.events.close, undefined, 'The connection was lost')
+  })
+
 Then('publishing is {paused-event} within {number} second(s)',
   /**
    * @param {'paused' | 'resumed'} key

package/features/steps/context.js

@@ -114,8 +114,7 @@ class Context extends World {
     const url = PROTOCOL + user + ':' + password + '@' + address
     const query = []
 
-    // the watchdog measures silence, so it may only be shortened along with the
-    // interval at which a healthy broker is expected to say something
+    // a connection gone silent is told by its missing heartbeats, so a short one tells it soon
     if (global.COMQ_TESTING_AMQP_HEARTBEAT !== undefined) {
       query.push('heartbeat=' + global.COMQ_TESTING_AMQP_HEARTBEAT)
     }

package/features/steps/events.js

@@ -225,7 +225,7 @@ async function consumed (group) {
 /**
  * @param {string} exchange
  * @param {any} message
- * @param {import('amqplib').Options.Publish} [properties]
+ * @param {import('@toa.io/amqplib').Options.Publish} [properties]
  * @return {Promise<void>}
  * @this {comq.features.Context}
  */

package/features/steps/hooks.js

@@ -12,7 +12,6 @@ After(
    * @this {comq.features.Context}
    */
   async function () {
-    delete global.COMQ_TESTING_WATCHDOG_INTERVAL
     delete global.COMQ_TESTING_AMQP_HEARTBEAT
     delete global.COMQ_TESTING_AMQP_CHANNEL_MAX
 

package/features/steps/network.js

@@ -26,6 +26,14 @@ When('the network goes silent',
     silence.call(this)
   })
 
+When('the network goes silent, with the broker still hearing heartbeats',
+  /**
+   * @this {comq.features.Context}
+   */
+  function () {
+    silence.call(this, true)
+  })
+
 When('the network goes silent and refuses new connections',
   /**
    * @this {comq.features.Context}
@@ -66,11 +74,12 @@ When('the consumer sends a request to the {token} queue as the network goes sile
   })
 
 /**
+ * @param {boolean} [heard]
  * @this {comq.features.Context}
  */
-function silence () {
+function silence (heard) {
   // only a connection lost from now on counts as lost
   delete this.events.close
 
-  for (const network of this.networks) network.silence()
+  for (const network of this.networks) network.silence(heard)
 }