comq 0.25.1 → 0.27.0

package/docs/headers.md

@@ -1,6 +1,7 @@
 # Message Properties
 
-Events may be emitted and consumed with custom properties using an optional `properties` argument.
+Events and Requests may be sent and received with custom properties using an optional `properties`
+argument.
 
 ## Emission
 
@@ -16,6 +17,20 @@ to [amqplib.publish](https://amqp-node.github.io/amqplib/channel_api.html#channe
 `consumer` function's signature
 is `async? (payload: any, [properties: comq.amqp.Properties]): void`
 
+## Request
+
+`async IO.request(queue: string, payload: any, [properties: comq.amqp.Properties]): any`
+
+`replyTo` and `correlationId` are set by comq whatever is passed, and `contentType` is the
+encoding of the Request. A Request takes no `timeout` and no `signal`.
+
+## Reply
+
+`async IO.reply(queue: string, producer): void`
+
+`producer` function's signature
+is `async? (payload: any, [properties: comq.amqp.Properties]): any`
+
 ## Headers set by comq
 
 A message that failed at least once carries headers comq wrote, and a consumer sees them among

package/features/properties.feature

@@ -27,3 +27,42 @@ Feature: Message properties
       """yaml
       appId: test
       """
+
+  Scenario: Sending a request with headers
+    Given function replying `greet` queue:
+      """
+      ({ name }, properties) => properties.headers.greeting + ' ' + name
+      """
+    When the consumer sends the following request to the `greet` queue with properties:
+      """yaml
+      payload:
+        name: world
+      properties:
+        headers:
+          greeting: hello
+      """
+    Then the consumer receives the reply:
+      """yaml
+      hello world
+      """
+
+  Scenario: A retried request keeps its headers
+    Given function replying `greet` queue:
+      """
+      (_, properties) => {
+        if (properties.headers['x-comq-attempt'] === undefined) throw new Error('once')
+
+        return properties.headers.greeting
+      }
+      """
+    When the consumer sends the following request to the `greet` queue with properties:
+      """yaml
+      payload: {}
+      properties:
+        headers:
+          greeting: hello
+      """
+    Then the consumer receives the reply:
+      """yaml
+      hello
+      """

package/features/shards.virtual.feature

@@ -0,0 +1,84 @@
+Feature: Virtual shards
+
+  The URLs of a sharded connection are names, of which several may stand for one broker.
+  A connection is made to each broker, and the brokers are followed as the names move.
+
+  Scenario: Names sharing a broker share a connection
+    Given 4 names, half of them standing for each broker
+    When I connect to the names
+    Then 1 connection is open to each broker
+
+  Scenario: Requests over the names of one broker
+    Given 4 names standing for the first broker
+    And an active connection to the names
+    And a producer replying `virtual_one` queue
+    When the consumer sends 20 requests to the `virtual_one` queue
+    Then all replies have been received
+    And 1 connection is open to the first broker
+    And 0 connections are open to the second broker
+
+  Scenario: A name moves to another broker
+    Given 4 names standing for the first broker
+    And an active connection to the names
+    And a producer replying `virtual_moved` queue
+    And tasks from the `virtual_moved_tasks` queue are being processed
+    When half of the names move to the second broker
+    Then the second broker has joined
+    And 1 connection is open to each broker
+    When the consumer sends 50 requests to the `virtual_moved` queue
+    Then all replies have been received
+    When a task is enqueued to the `virtual_moved_tasks` queue of the second broker by another connection
+    Then the task is processed
+
+  Scenario: A broker no name stands for is drained before it is left
+    Given 4 names standing for the first broker
+    And a broker lingers for 300ms
+    And an active connection to the names
+    And tasks from the `virtual_left_behind` queue are being processed
+    And a producer replying `virtual_slow` queue in 2000ms
+    When the consumer sends 5 requests to the `virtual_slow` queue
+    And the names move to the second broker
+    Then the second broker has joined
+    And the first broker is retired
+    When a task is enqueued to the `virtual_left_behind` queue of the first broker by another connection
+    Then the task is processed
+    When after 1000ms
+    Then the first broker is not left
+    And all replies have been received
+    And the first broker is left
+    And 0 connections are open to the first broker
+    When the consumer sends 20 requests to the `virtual_slow` queue
+    Then all replies have been received
+
+  Scenario: Names observed while they are moving are not followed
+    Given 4 names standing for the first broker
+    And an active connection to the names
+    When the names keep moving between the brokers for 1500ms
+    And after 500ms
+    Then no broker has joined or retired
+    And 0 connections are open to the second broker
+
+  Scenario: A broker that is out of reach is followed at once
+    Given 4 names standing for the first broker
+    And the names settle in 60000ms
+    And an active connection to the names
+    And a producer replying `virtual_followed` queue
+    When the first broker is out of reach
+    And the names move to the second broker
+    Then the second broker has joined
+    When the consumer sends 20 requests to the `virtual_followed` queue
+    Then all replies have been received
+
+  Scenario: A retiring broker that is named again is back
+    Given 4 names standing for the first broker
+    And a broker lingers for 5000ms
+    And an active connection to the names
+    And a producer replying `virtual_back` queue
+    When the names move to the second broker
+    Then the first broker is retired
+    When the names move to the first broker
+    Then the first broker has joined
+    And the second broker is retired
+    And the first broker is not left
+    When the consumer sends 20 requests to the `virtual_back` queue
+    Then all replies have been received

package/features/steps/context.js

@@ -42,6 +42,15 @@ class Context extends World {
   /** @type {comq.features.Network[]} */
   networks = []
 
+  /** @type {Record<string, string>} the address each name of a shard resolves to */
+  names = {}
+
+  /** @type {comq.shards.Timing} */
+  resolution = { interval: 50, settle: 300, linger: 600 }
+
+  /** @type {Record<'join' | 'retire' | 'leave', string[]>} the addresses of the brokers that did */
+  brokers = { join: [], retire: [], leave: [] }
+
   async connect (user, password) {
     const urls = this.#urls(user, password)
 
@@ -54,6 +63,23 @@ class Context extends World {
     await this.#connect(urls, assert)
   }
 
+  /**
+   * Connects to the names of the shards, which a broker is found behind at whatever port the
+   * names are given: see `names.js`.
+   *
+   * @param {string[]} hosts
+   * @param {number} port
+   */
+  async connectNames (hosts, port) {
+    const urls = hosts.map((host) => PROTOCOL + USER + ':' + PASSWORD + '@' + host + ':' + port)
+
+    await this.#connect(urls, connect, { ...TOPOLOGY, resolution: this.resolution })
+
+    for (const event of Object.keys(this.brokers)) {
+      this.io.diagnose(event, (_index, address) => this.brokers[event].push(address))
+    }
+  }
+
   async unplug () {
     await Promise.all(this.networks.map((network) => network.close()))
 
@@ -78,14 +104,15 @@ class Context extends World {
   /**
    * @param {string[]} urls
    * @param {comq.Connect} [method]
+   * @param {comq.Options} [options]
    * @return {Promise<void>}
    */
-  async #connect (urls, method = connect) {
+  async #connect (urls, method = connect, options = TOPOLOGY) {
     if (this.io !== undefined) await this.disconnect()
 
     // the retry delay is a topology setting, so the suite need not wait out a
     // production one to see the mechanism work
-    this.io = await method(...urls, TOPOLOGY)
+    this.io = await method(...urls, options)
     this.connected = true
 
     for (const event of EVENTS) this.io.diagnose(event, () => (this.events[event] = true))

package/features/steps/hooks.js

@@ -14,6 +14,7 @@ After(
   async function () {
     delete global.COMQ_TESTING_AMQP_HEARTBEAT
     delete global.COMQ_TESTING_AMQP_CHANNEL_MAX
+    delete global.COMQ_TESTING_LOOKUP
 
     await Promise.all(
       Array.from({ length: BROKERS_AMOUNT }, (_, n) => actions.up(n))

package/features/steps/names.js

@@ -0,0 +1,235 @@
+'use strict'
+
+const assert = require('node:assert')
+const { Given, When, Then, defineParameterType } = require('@cucumber/cucumber')
+const { connect } = require('../../')
+const { timeout, until } = require('../../test/helpers')
+const { BROKERS_AMOUNT, getAddress, USER, PASSWORD } = require('./brokers')
+const { Network } = require('./networks')
+
+defineParameterType({
+  name: 'broker',
+  regexp: /(first|second)/,
+  transformer: (value) => ORDINALS.indexOf(value)
+})
+
+Given('{number} names standing for the {broker} broker',
+  /**
+   * @param {number} amount
+   * @param {number} n
+   * @this {comq.features.Context}
+   */
+  async function (amount, n) {
+    await expose.call(this)
+
+    for (let i = 0; i < amount; i++) this.names[name(i)] = ADDRESSES[n]
+  })
+
+Given('{number} names, half of them standing for each broker',
+  /**
+   * @param {number} amount
+   * @this {comq.features.Context}
+   */
+  async function (amount) {
+    await expose.call(this)
+
+    for (let i = 0; i < amount; i++) this.names[name(i)] = ADDRESSES[i < amount / 2 ? 0 : 1]
+  })
+
+Given('the names settle in {number}ms',
+  /**
+   * @param {number} settle
+   * @this {comq.features.Context}
+   */
+  function (settle) {
+    this.resolution.settle = settle
+  })
+
+Given('a broker lingers for {number}ms',
+  /**
+   * @param {number} linger
+   * @this {comq.features.Context}
+   */
+  function (linger) {
+    this.resolution.linger = linger
+  })
+
+Given('an active connection to the names', connectNames)
+
+When('I connect to the names', connectNames)
+
+When('half of the names move to the {broker} broker',
+  /**
+   * @param {number} n
+   * @this {comq.features.Context}
+   */
+  function (n) {
+    const names = Object.keys(this.names)
+
+    for (const name of names.slice(names.length / 2)) this.names[name] = ADDRESSES[n]
+  })
+
+When('the names move to the {broker} broker', move)
+
+When('the names keep moving between the brokers for {number}ms',
+  /**
+   * @param {number} duration
+   * @this {comq.features.Context}
+   */
+  async function (duration) {
+    const flips = Math.floor(duration / FLIP_MS)
+
+    // an odd number of moves would leave the names moved
+    for (let i = 0; i < flips + flips % 2; i++) {
+      move.call(this, (i + 1) % 2)
+
+      await timeout(FLIP_MS)
+    }
+  })
+
+When('the {broker} broker is out of reach',
+  /**
+   * @param {number} n
+   * @this {comq.features.Context}
+   */
+  function (n) {
+    this.networks[n].cut()
+  })
+
+When('a task is enqueued to the {token} queue of the {broker} broker by another connection',
+  /**
+   * @param {string} queue
+   * @param {number} n
+   * @this {comq.features.Context}
+   */
+  async function (queue, n) {
+    const io = await connect('amqp://' + USER + ':' + PASSWORD + '@' + getAddress(n))
+
+    this.enqueued = 'left behind'
+
+    await io.enqueue(queue, this.enqueued)
+    await io.close()
+  })
+
+Then('the task is processed',
+  /**
+   * @this {comq.features.Context}
+   */
+  async function () {
+    const processed = await until(() => this.processed === this.enqueued)
+
+    assert.equal(processed, true, 'the task was not processed')
+  })
+
+Then('the {broker} broker has joined', heard('join'))
+
+Then('the {broker} broker is retired', heard('retire'))
+
+Then('the {broker} broker is left', heard('leave'))
+
+Then('the {broker} broker is not left',
+  /**
+   * @param {number} n
+   * @this {comq.features.Context}
+   */
+  function (n) {
+    assert.equal(this.brokers.leave.includes(ADDRESSES[n]), false, 'the broker is left')
+  })
+
+Then('no broker has joined or retired',
+  /**
+   * @this {comq.features.Context}
+   */
+  function () {
+    assert.deepEqual(this.brokers, { join: [], retire: [], leave: [] })
+  })
+
+Then('{number} connection(s) is/are open to the {broker} broker', open)
+
+Then('{number} connection is open to each broker',
+  /**
+   * @param {number} amount
+   * @this {comq.features.Context}
+   */
+  async function (amount) {
+    for (let n = 0; n < BROKERS_AMOUNT; n++) await open.call(this, amount, n)
+  })
+
+/**
+ * Each broker is given an address of its own, at one port: a name resolves to an address and
+ * says nothing of a port, and the brokers of the suite differ in nothing but theirs.
+ *
+ * @this {comq.features.Context}
+ */
+async function expose () {
+  if (this.networks.length > 0) return
+
+  for (let n = 0; n < BROKERS_AMOUNT; n++) {
+    const network = new Network(n, ADDRESSES[n], PORT)
+
+    await network.open()
+
+    this.networks[n] = network
+  }
+
+  global.COMQ_TESTING_LOOKUP = async (host) => ({ addresses: [this.names[host]], ttl: 0 })
+}
+
+/**
+ * @this {comq.features.Context}
+ */
+async function connectNames () {
+  await this.connectNames(Object.keys(this.names), PORT)
+}
+
+/**
+ * @param {number} n
+ * @this {comq.features.Context}
+ */
+function move (n) {
+  for (const name of Object.keys(this.names)) this.names[name] = ADDRESSES[n]
+}
+
+/**
+ * @param {number} amount
+ * @param {number} n
+ * @this {comq.features.Context}
+ */
+async function open (amount, n) {
+  await until(() => this.networks[n].connections === amount)
+
+  assert.equal(this.networks[n].connections, amount)
+}
+
+/**
+ * @param {'join' | 'retire' | 'leave'} event
+ */
+function heard (event) {
+  /**
+   * @param {number} n
+   * @this {comq.features.Context}
+   */
+  return async function (n) {
+    const happened = await until(() => this.brokers[event].includes(ADDRESSES[n]), 10_000)
+
+    assert.equal(happened, true, `the broker did not ${event}`)
+  }
+}
+
+/**
+ * @param {number | string} i
+ * @return {string}
+ */
+function name (i) {
+  return `shard-${i}.test`
+}
+
+const ORDINALS = ['first', 'second']
+
+/** Loopback addresses, which every one of 127.0.0.0/8 is. */
+const ADDRESSES = ['127.0.0.2', '127.0.0.3']
+
+const PORT = 56790
+
+/** Sooner than the names settle. */
+const FLIP_MS = 100

package/features/steps/networks.js

@@ -25,24 +25,42 @@ class Network {
 
   #refusing = false
 
+  /** @type {string} */
+  #listenHost
+
+  /** @type {number} */
+  #listenPort
+
   /**
    * @param {number} [n] broker index
+   * @param {string} [host] the address the broker is found at through this network
+   * @param {number} [port] any that is free unless given
    */
-  constructor (n = 0) {
-    const [host, port] = getAddress(n).split(':')
+  constructor (n = 0, host = '127.0.0.1', port = 0) {
+    const [brokerHost, brokerPort] = getAddress(n).split(':')
 
-    this.#host = host
-    this.#port = Number(port)
+    this.#host = brokerHost
+    this.#port = Number(brokerPort)
+    this.#listenHost = host
+    this.#listenPort = port
   }
 
   get address () {
     return 'localhost:' + this.#server.address().port
   }
 
+  /** How many connections are open through this network. */
+  get connections () {
+    return this.#tunnels.size
+  }
+
   async open () {
     this.#server = net.createServer(this.#tunnel)
 
-    await new Promise((resolve) => this.#server.listen(0, '127.0.0.1', resolve))
+    await new Promise((resolve, reject) => {
+      this.#server.once('error', reject)
+      this.#server.listen(this.#listenPort, this.#listenHost, resolve)
+    })
   }
 
   /**

package/features/steps/rpc.js

@@ -144,6 +144,18 @@ When('the consumer sends the following request to the {token} queue:',
     await send.call(this, queue, payload)
   })
 
+When('the consumer sends the following request to the {token} queue with properties:',
+  /**
+   * @param {string} queue
+   * @param {string} yaml
+   * @this {comq.features.Context}
+   */
+  async function (queue, yaml) {
+    const { payload, properties } = parse(yaml)
+
+    await send.call(this, queue, payload, properties)
+  })
+
 When('the consumer sends a request to the {token} queue',
   /**
    * @param {string} queue
@@ -567,10 +579,10 @@ function sendMany (amount, queue) {
   }
 }
 
-async function send (queue, payload) {
+async function send (queue, payload, properties) {
   if (this.expected) await this.expected
 
-  this.reply = this.io.request(queue, payload)
+  this.reply = this.io.request(queue, payload, properties)
 }
 
 /**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "comq",
-  "version": "0.25.1",
+  "version": "0.27.0",
   "description": "ComQ",
   "author": {
     "name": "Artem Gurtovoi",