comq 0.25.1 → 0.26.0

package/readme.md

@@ -18,6 +18,7 @@ for distributed, eventually consistent systems running on Node.js.
 - [Poison message handling](#messages)
 - [Connection tolerance](#connection-tolerance) and broker restart resilience
 - [Sharded connection](#sharded-connection) :rocket:
+- [Virtual shards](#virtual-shards) that follow DNS
 - [Singleton connection](#singleton-connection)
 - [Graceful shutdown](#graceful-shutdown)
 
@@ -81,10 +82,10 @@ Event published to a fanout exchange carries none.
 
 `async IO.reply(queue: string, producer): void`
 
-`producer` function's signature is `async? (message: any): any`
+`producer` function's signature is `async? (message: any, properties?): any`
 
 Assert a `queue` and start consuming Requests. Received messages are decoded and the resulting
-content is passed to the `producer`. The result returned by the `producer` is then encoded and sent
+content is passed to the `producer`, followed by the [properties](docs/headers.md) of the Request. The result returned by the `producer` is then encoded and sent
 back to the queue specified in the `replyTo` property of the Request, along with a `correlationId`
 that has the same value as in the Request.
 
@@ -109,8 +110,11 @@ await io.reply('add_numbers', ({ a, b }) => (a + b))
 
 `async IO.request(queue: string, payload: any, encoding?: string): any`
 
+`async IO.request(queue: string, payload: any, properties?): any`
+
 Send encoded Request message with `replyTo` and `correlationId` properties set and
-return decoded Reply content. The promise stays pending until the Reply arrives, however long
+return decoded Reply content. A Request may carry [properties](docs/headers.md) of its own, such
+as `headers`. The promise stays pending until the Reply arrives, however long
 that takes: a Request has no timeout and cannot be withdrawn. One no Producer can answer is
 [parked](#parked-messages), and can still be answered while its caller waits.
 
@@ -508,6 +512,89 @@ const io = await connect(shard0, shard1)
 await io.close()
 ```
 
+### Virtual shards
+
+The URLs of a sharded connection are *names*, and several of them may stand for one broker. A
+connection is made to each broker rather than to each name: thirty-two names resolving to two
+addresses are two connections.
+
+```javascript
+const io = await connect('amqp://rmq0.example.com', 'amqp://rmq1.example.com', /* ... */)
+```
+
+A range in a host is as many URLs. Its end is exclusive, so a range that starts at zero names as
+many hosts as its end says, and a start written with leading zeros pads every number to its
+width:
+
+```javascript
+await connect('amqp://developer:secret@rmq[0..32].example.com') // rmq0 … rmq31
+await connect('amqp://developer:secret@rmq[00..32].example.com') // rmq00 … rmq31
+```
+
+This is what lets the number of brokers change without anyone being reconfigured: the names stay,
+and what they resolve to moves.
+
+A broker is told by its address, port, virtual host and credentials. A name with several
+addresses is still one broker, connected through one of them: two connections to one broker
+would compete for the queues that are exclusive to a connection. Different addresses are taken
+for different brokers.
+
+#### Following the names
+
+The names are resolved again every `interval`. What they resolve to is seldom seen changing
+whole: records are updated one after another, and until their TTL has run out caches answer
+with the old address and the new one in turn. So a change is acted upon only once the answer has
+stayed the same for the `settle` time, and anything else observed meanwhile starts that time
+anew. A name that fails to resolve keeps its address, which is reported as `error`.
+
+Then the brokers are brought in line with the names, making before breaking:
+
+1. **Join.** A connection is made to each broker that is newly named. Whatever is consumed is
+   consumed from it as well, and then it joins the pool. Until every named broker has joined,
+   nothing below happens: a name pointing at a broker that is not there yet takes nothing away.
+2. **Retire.** A broker no name stands for is taken out of the pool: nothing new is published to
+   it. It goes on being consumed from, and a Reply still goes back through it.
+3. **Leave.** A retired broker is disconnected once, for the whole `linger` time, it has had
+   nothing to do: the queues consumed on it are empty, no message is with its consumer or
+   waiting to be [retried](#retries), no publication awaits its confirmation, and no Request
+   sent through it awaits its Reply.
+
+Everyone connected follows the names on their own, and not at the same moment. The `linger` time
+is what covers the difference: what is published to a retired broker by those who have not seen
+the names move yet is consumed by those who have. There is no deadline, since leaving a broker
+that is still published to would strand the messages. A retired broker that is named again is
+back in the pool, and one that stays out of reach for the `linger` time is given up on.
+
+A broker that is out of reach is not waited for: what its names resolve to next is taken at
+once, as it is when a single connection is restored by its name.
+
+The [guarantees](#what-is-guaranteed) are the ones a sharded connection has. A publication is
+confirmed by whichever broker took it, a consumer does not leave a queue that holds messages,
+and a Request is either answered through the broker it was sent to or re-sent as it is when a
+shard is lost.
+
+#### Timing
+
+The trailing argument of `connect` takes it next to the [topology settings](#settings), in
+milliseconds:
+
+```javascript
+await connect(...urls, { resolution: { interval: 10_000, settle: 60_000, linger: 60_000 } })
+```
+
+- `interval`: how often the names are resolved, 10 seconds by default.
+- `settle`: how long an answer is to hold before it is acted upon.
+- `linger`: how long a retired broker is to stay idle before it is left.
+
+Unless given, `settle` and `linger` are the longest TTL the names have been answered with, and
+no less than 30 seconds: a TTL bounds how long a cache may answer with what is no longer true,
+and says nothing of records being updated one by one. A name that is not found in DNS itself (one
+from the hosts file, or one completed by the resolver's search list) has no TTL, and makes them
+60 seconds.
+
+A single URL is one connection made by its name, as it always was, and nothing here applies to
+it.
+
 ## Singleton connection
 
 `async assert(url: string): IO`
@@ -821,6 +908,12 @@ Subscribe to one of the diagnostic events:
   reported only once every shard has rejected it.
 - `taken`: a Key [`back`](#addressed-requests) claims is held by another connection on this broker, and
   is claimed again. Channel type and the queue name are passed.
+- `join`: a broker the names of the [virtual shards](#virtual-shards) stand for has joined the
+  pool, or a retired one is back in it. The shard number and the address are passed.
+- `retire`: no name stands for a broker any longer, and nothing new is published to it. The
+  shard number and the address are passed.
+- `leave`: a retired broker has had nothing more to give, and is disconnected. The shard number
+  and the address are passed.
 - `pause`: channel is paused. Channel type is passed.
   In the case of a [sharded connection](#sharded-connection), it means that there is no shard left
   to publish to, be it because every one of them has rejected a publish or lost its connection.
@@ -830,7 +923,9 @@ In the case of a [sharded connection](#sharded-connection), an additional argume
 shard number will be passed to listeners.
 This is applicable except for the `pause` and `resume` events,
 which are emitted when the associated channels are paused or resumed across all shards.
-The shard number corresponds to the position of the argument used in the `connect` function call.
+A shard is a broker, and is numbered in the order the brokers are first named: with a URL per
+broker that is the position of the argument used in the `connect` function call. A broker that
+[joins](#virtual-shards) later takes the next number, and a number is not given twice.
 
 [^3]: As the [`connect`](#connect) function returns an instance of `IO` *after* the connection has been
 established, there is no way to capture the initial `open` event.
@@ -847,6 +942,9 @@ io.diagnose('close', (error, shard) => console.log('AMQP connection closed', { m
 io.diagnose('error', (error, shard) => console.log('AMQP connection failed', { message: error.message, shard }))
 io.diagnose('lost', (type, shard) => console.log('AMQP shard lost', { type, shard }))
 io.diagnose('recover', (type, shard) => console.log('AMQP channel recovered', { type, shard }))
+io.diagnose('join', (shard, address) => console.log('AMQP broker joined', { shard, address }))
+io.diagnose('retire', (shard, address) => console.log('AMQP broker retired', { shard, address }))
+io.diagnose('leave', (shard, address) => console.log('AMQP broker left', { shard, address }))
 ```
 
 # Gratitude

package/source/channel.js

@@ -52,6 +52,21 @@ class Channel {
   /** @type {Set<Promise<void>>} failed messages on their way to a retry or the parking queue */
   #settling = new Set()
 
+  /** @type {Set<string>} the queues being consumed */
+  #queues = new Set()
+
+  /** @type {number} how many deliveries are with their consumers */
+  #handling = 0
+
+  /** @type {Promex | null} resolved once no delivery is with its consumer */
+  #rest = null
+
+  /** @type {number} when a message was last published, or a delivery was last done with */
+  #active = 0
+
+  /** @type {number} when the last of the messages waiting to be retried is delivered again */
+  #awaited = 0
+
   #diagnostics = emitter.create()
 
   #closed = false
@@ -87,6 +102,7 @@ class Channel {
 
     // a fresh channel has bound nothing
     this.#held.clear()
+    this.#queues.clear()
 
     await this.#channel.prefetch(this.#topology.prefetch)
 
@@ -267,6 +283,44 @@ class Channel {
     return this.#closed
   }
 
+  /**
+   * Resolves once the deliveries that are with their consumers are done with, and so is what
+   * was published. It is waited for between sealing and closing by whoever means to lose none
+   * of them: a channel that is closed takes back what it has delivered.
+   *
+   * @return {Promise<void>}
+   */
+  async settled () {
+    while (this.#handling > 0) await (this.#rest ??= new Promex())
+
+    await Promise.allSettled([...this.#confirmations, ...this.#settling])
+  }
+
+  /**
+   * Since when the channel has had nothing to do, which is now while it has: a delivery with
+   * its consumer, a publication awaiting its confirmation, a failed message that is yet to come
+   * back, or a message in a queue it consumes. It is what a broker that is being left is
+   * asked, since whatever is left behind on it is lost to this connection.
+   *
+   * @return {Promise<number>}
+   */
+  async quiet () {
+    const now = Date.now()
+
+    const busy = this.#handling > 0 || this.#confirmations.size > 0 ||
+      this.#settling.size > 0 || this.#paused !== null || this.#awaited > now
+
+    if (busy) return now
+
+    try {
+      if (await this.#depth() > 0) return now
+    } catch {
+      return now // what cannot be told is not taken for nothing
+    }
+
+    return Math.max(this.#active, this.#awaited)
+  }
+
   async recover (connection) {
     this.#connection = connection
 
@@ -372,6 +426,45 @@ class Channel {
     return true
   }
 
+  /**
+   * @return {Promise<number>} how many messages wait in the queues being consumed
+   */
+  async #depth () {
+    let depth = 0
+
+    for (const queue of this.#queues) depth += await this.#count(queue)
+
+    return depth
+  }
+
+  /**
+   * Asked on a channel of its own, for the reason `#declare` has one: a queue that is gone is
+   * answered by closing the channel that asked.
+   *
+   * @param {string} queue
+   * @return {Promise<number>}
+   */
+  async #count (queue) {
+    const probe = await this.#connection.createChannel()
+
+    /** @type {Error & { code?: number } | undefined} */
+    let refusal
+
+    probe.on('error', (error) => { refusal = error })
+
+    try {
+      const { messageCount } = await probe.checkQueue(queue)
+
+      await probe.close().catch(noop)
+
+      return messageCount
+    } catch (exception) {
+      if (refusal?.code === NOT_FOUND) return 0
+
+      throw refusal ?? exception
+    }
+  }
+
   // region initializers
 
   /**
@@ -532,6 +625,8 @@ class Channel {
 
     options = Object.assign({ persistent: this.#topology.persistent }, options)
 
+    this.#active = Date.now()
+
     const confirmation = this.#topology.confirms ? this.#confirmation() : undefined
     const resume = this.#channel.publish(exchange, queue, buffer, options, confirmation?.callback)
 
@@ -570,16 +665,58 @@ class Channel {
       /** @type {comq.amqp.options.Consume} */
       const options = {}
 
-      if (this.#topology.acknowledgments) consumer = this.#getAcknowledgingConsumer(queue, consumer)
-      else options.noAck = true
+      if (this.#topology.acknowledgments) {
+        consumer = this.#getAcknowledgingConsumer(queue, consumer)
+      } else {
+        consumer = this.#attended(consumer)
+        options.noAck = true
+      }
 
       const response = await this.#channel.consume(queue, consumer, options)
 
       this.#tags.push(response.consumerTag)
+      this.#queues.add(queue)
 
       return response.consumerTag
     })
 
+  /**
+   * Counts the deliveries that are with a consumer nothing is acknowledged for, without a
+   * promise of its own for the ones that are done with as they arrive.
+   *
+   * @param {comq.channels.Consumer} consumer
+   * @returns {comq.channels.Consumer}
+   */
+  #attended = (consumer) =>
+    (message) => {
+      this.#handling++
+
+      let result
+
+      try {
+        result = consumer(message)
+      } catch (exception) {
+        this.#attendedTo()
+
+        throw exception
+      }
+
+      if (typeof result?.then === 'function') result.then(this.#attendedTo, this.#attendedTo)
+      else this.#attendedTo()
+
+      return result
+    }
+
+  #attendedTo = () => {
+    this.#handling--
+    this.#active = Date.now()
+
+    if (this.#handling === 0 && this.#rest !== null) {
+      this.#rest.resolve()
+      this.#rest = null
+    }
+  }
+
   /**
    * @param {string} queue the queue being consumed
    * @param {comq.channels.Consumer} consumer
@@ -587,6 +724,8 @@ class Channel {
    */
   #getAcknowledgingConsumer = (queue, consumer) =>
     async (message) => {
+      this.#handling++
+
       try {
         await consumer(message)
 
@@ -601,6 +740,8 @@ class Channel {
         await settling
 
         this.#settling.delete(settling)
+      } finally {
+        this.#attendedTo()
       }
     }
 
@@ -647,6 +788,7 @@ class Channel {
     await this.#publish(this.#retryQueueOf(attempt), queue, message.content, properties)
 
     this.#channel.ack(message)
+    this.#awaited = Math.max(this.#awaited, Date.now() + this.#delays[attempt - 1])
 
     // the attempt that just failed, rather than the one it is about to get
     this.#diagnostics.emit('retry', message, exception, attempt)
@@ -781,6 +923,9 @@ const INTERRUPTION = /** @type {Error} */ Symbol('internal interruption')
 // the AMQP reply code a broker closes a channel with when another connection holds the queue
 const RESOURCE_LOCKED = 405
 
+// and the one it closes a channel with when asked about a queue it does not have
+const NOT_FOUND = 404
+
 const RETRY_PREFIX = 'comq.retry.'
 
 /** Where every message that has run out of attempts is kept, whatever it was consumed from. */

package/source/connect.js

@@ -16,29 +16,29 @@ const assert = async (...args) => {
 }
 
 /**
- * The urls are variadic, so the overrides are the trailing argument when there is one
- * that is not a url.
+ * The urls are variadic, so the options are the trailing argument when there is one
+ * that is not a url: the topology overrides, and how the names are followed.
  *
- * @param {(string | comq.topology.Overrides)[]} args
- * @return {[string[], comq.topology.Overrides]}
+ * @param {(string | comq.Options)[]} args
+ * @return {[string[], comq.topology.Overrides, comq.shards.Timing | undefined]}
  */
 const split = (args) => {
   const last = args[args.length - 1]
 
   const urls = /** @type {string[]} */ (typeof last === 'object' ? args.slice(0, -1) : args)
-  const overrides = /** @type {comq.topology.Overrides} */ (typeof last === 'object' ? last : {})
+  const { resolution, ...overrides } = /** @type {comq.Options} */ (typeof last === 'object' ? last : {})
 
-  return [urls, overrides]
+  return [urls.flatMap(shards.expand), overrides, resolution]
 }
 
 /**
- * @param {(string | comq.topology.Overrides)[]} args
- * @param {new (url: string, overrides: comq.topology.Overrides) => comq.Connection} ConnectionClass
+ * @param {(string | comq.Options)[]} args
+ * @param {new (url: string, overrides: comq.topology.Overrides, servername?: string) => comq.Connection} ConnectionClass
  * @return {Promise<IO>}
  */
 const create = async (args, ConnectionClass) => {
-  const [urls, overrides] = split(args)
-  const connection = connectionOf(urls, overrides, ConnectionClass)
+  const [urls, overrides, resolution] = split(args)
+  const connection = connectionOf(urls, overrides, resolution, ConnectionClass)
 
   await connection.open()
 
@@ -46,17 +46,20 @@ const create = async (args, ConnectionClass) => {
 }
 
 /**
+ * Several URLs are the names of the shards, which may be fewer than the names: a connection
+ * is made to each broker they stand for. A single URL is connected by its name, as it is.
+ *
  * @param {string[]} urls
  * @param {comq.topology.Overrides} overrides
+ * @param {comq.shards.Timing | undefined} resolution
  * @param {Function} ConnectionClass
  * @return {comq.Connection}
  */
-const connectionOf = (urls, overrides, ConnectionClass) => {
-  const connections = urls.map((url) => new ConnectionClass(url, overrides))
-
-  if (connections.length === 1) return connections[0]
+const connectionOf = (urls, overrides, resolution, ConnectionClass) => {
+  if (urls.length === 1) return new ConnectionClass(urls[0], overrides)
 
-  return new shards.Connection(connections)
+  return new shards.Connection(urls,
+    (url, servername) => new ConnectionClass(url, overrides, servername), resolution)
 }
 
 exports.connect = connect

package/source/connection.js

@@ -38,18 +38,21 @@ class Connection {
   /** @type {comq.topology.Overrides} */
   #overrides
 
+  /** @type {object} */
+  #options
+
   #diagnostics = emitter.create()
 
-  /**
-   * @param {string} url
-   */
   /**
    * @param {string} url
    * @param {comq.topology.Overrides} [overrides] per channel type, merged over the presets
+   * @param {string} [servername] the name TLS verifies the broker by, when the URL has its
+   * address in place of the name
    */
-  constructor (url, overrides = {}) {
+  constructor (url, overrides = {}, servername = undefined) {
     this.#url = heartbeaten(url)
     this.#overrides = overrides
+    this.#options = servername === undefined ? SOCKET_OPTIONS : { ...SOCKET_OPTIONS, servername }
   }
 
   get connected () {
@@ -255,7 +258,7 @@ class Connection {
    * @return {Promise<comq.amqp.Connection>}
    */
   async #connect () {
-    const connecting = amqp.connect(this.#url, SOCKET_OPTIONS)
+    const connecting = amqp.connect(this.#url, this.#options)
 
     let timer
 

package/source/events.js

@@ -3,5 +3,12 @@
 /** @type {comq.diagnostics.Event[]} */
 exports.connection = ['open', 'close', 'error', 'reconnect', 'exhausted']
 
+/**
+ * What a sharded connection tells of the brokers its names stand for.
+ *
+ * @type {comq.diagnostics.Event[]}
+ */
+exports.shards = ['join', 'retire', 'leave']
+
 /** @type {comq.diagnostics.Event[]} */
 exports.channel = ['flow', 'drain', 'recover', 'discard', 'retry', 'pause', 'resume', 'return', 'lost', 'remove', 'taken']

package/source/io.js

@@ -63,7 +63,7 @@ class IO {
   constructor (connection) {
     this.#connection = connection
 
-    for (const event of events.connection) {
+    for (const event of [...events.connection, ...events.shards]) {
       const forwarder = (...args) => this.#diagnostics.emit(event, ...args)
 
       this.#connection.diagnose(event, forwarder)
@@ -89,16 +89,22 @@ class IO {
    *
    * @param {string} queue
    * @param {any | Readable} payload
-   * @param {comq.Encoding} [encoding]
+   * @param {comq.Encoding | comq.amqp.Properties} [encoding]
    * @returns {Promise<any | Readable>}
    */
   async request (queue, payload, encoding) {
+    if (typeof encoding !== 'object' || encoding === null) {
+      return await this.#request(queue, payload, { encoding })
+    }
+
     // a caller of the version that took a timeout here would otherwise wait on in silence
-    if (typeof encoding === 'object' && encoding !== null) {
-      throw new TypeError('A Request takes an encoding and waits for its Reply')
+    if ('timeout' in encoding || 'signal' in encoding) {
+      throw new TypeError('A Request takes an encoding or properties and waits for its Reply')
     }
 
-    return await this.#request(queue, payload, { encoding })
+    const { contentType, ...properties } = encoding
+
+    return await this.#request(queue, payload, { encoding: contentType, properties })
   }
 
   /**
@@ -294,6 +300,9 @@ class IO {
 
     this.#setupRetransmission()
 
+    // a shard that is being retired is not left while it owes a Reply
+    this.#requests.occupy?.((index) => this.#awaiting(index))
+
     // on a sharded connection, only once every shard has returned it
     this.#requests.diagnose('return', this.#returned)
   }
@@ -393,7 +402,7 @@ class IO {
        */
       async (request) => {
         const payload = decode(request)
-        const reply = await produce(producer, payload)
+        const reply = await produce(producer, payload, request.properties)
 
         if (request.properties.replyTo === undefined) return
 
@@ -455,7 +464,9 @@ class IO {
 
     const [buffer, contentType] = this.#encode(payload, terms.encoding)
     const request = this.#createRequest(contentType, signal)
-    const properties = { ...request.properties }
+
+    // what makes it a Request is comq's to set, whatever it was sent with
+    const properties = { ...terms.properties, ...request.properties }
 
     if (expires !== undefined) {
       const left = Math.ceil(expires - Date.now())
@@ -745,6 +756,16 @@ class IO {
     }
   }
 
+  /**
+   * @param {number} index
+   * @return {boolean} whether a Request sent through a shard is yet to be answered
+   */
+  #awaiting (index) {
+    for (const request of this.#pendingReplies.values()) if (request.shard === index) return true
+
+    return false
+  }
+
   /**
    * @param {comq.Request} request
    * @param {Promex} reply
@@ -856,10 +877,11 @@ async function abortable (promise, signal) {
  *
  * @param {comq.Producer} producer
  * @param {any} payload
+ * @param {comq.amqp.Properties} properties
  */
-async function produce (producer, payload) {
+async function produce (producer, payload, properties) {
   try {
-    return await producer(payload)
+    return await producer(payload, properties)
   } catch (exception) {
     if (verdictOf(exception) !== PARK) throw exception