comq 0.27.0 → 0.28.0

package/.github/workflows/features.yaml

@@ -37,3 +37,28 @@ jobs:
 
       - name: Run fast feature tests
         run: npm run features:fast
+
+  # a minor or patch bump by Dependabot is merged once its features pass
+  automerge:
+    needs: features-fast
+    if: github.actor == 'dependabot[bot]' && startsWith(github.ref_name, 'dependabot/')
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    permissions:
+      contents: write
+      pull-requests: write
+
+    steps:
+      - name: Merge
+        env:
+          GH_TOKEN: ${{ github.token }}
+          MESSAGE: ${{ github.event.head_commit.message }}
+        run: |
+          types=$(grep -oP 'update-type: version-update:semver-\K\w+' <<< "$MESSAGE" || true)
+
+          if [ -z "$types" ] || grep -qvx -e minor -e patch <<< "$types"; then
+            echo "Not a minor or patch bump, left for a review"
+            exit 0
+          fi
+
+          gh pr merge "$GITHUB_REF_NAME" --repo "$GITHUB_REPOSITORY" --merge --match-head-commit "$GITHUB_SHA"

package/docs/shards.md

@@ -0,0 +1,72 @@
+# Changing the number of brokers
+
+How brokers are added to and removed from under a connection that follows
+[names](../readme.md#virtual-shards), with everyone connected staying as they are.
+
+## Names
+
+- **As many names as there may ever be brokers.** A broker is in the pool while at least one name
+  stands for it, so the number of names is the most brokers there can be. Names cost nothing:
+  thirty-two of them over two brokers are two connections.
+- **How many names a broker has does not matter.** A publication goes to a broker chosen at random
+  among those in the pool, not to a name: a broker with one name out of thirty-two takes as much as
+  one with the rest.
+- **One address per name.** A name with several addresses is still one broker, connected through
+  one of them. Records that rotate between brokers spread nothing.
+- **A short TTL.** Unless [given](../readme.md#timing), the `settle` and `linger` times are the
+  longest TTL of the names and no less than 15 minutes, so the TTL is what a change takes.
+- **The same names for everyone.** One who connects by addresses, or by names of their own, follows
+  nothing, and is cut off once a broker is removed.
+
+## Adding a broker
+
+1. **Start the broker and give it what the others have**: the virtual hosts, the users and their
+   permissions.
+2. **Copy the exchanges, the queues and the bindings between them** from a broker in the pool.
+   Everyone follows the names on their own, so the one who emits an Event may join the new broker
+   before the one who consumes it has: an exchange with no queue bound to it yet drops what is
+   published to it. A Request needs none of this, its queue being declared by the one who sends it.
+
+   Queues exclusive to a connection, and their bindings, are not to be copied: they are declared
+   by the connection that owns them when it joins.
+3. **Point names at it.** Which ones and how many is of no consequence, so move as few as it
+   takes for the names to be spread evenly: these are the ones to move back once it is removed.
+4. **See the connections arrive.** A `settle` time after the records have changed, everyone who
+   follows the names is connected to the new broker as well as to the others.
+
+Nothing is taken away while a broker is being added: a name pointing at a broker that is not there
+yet, or not reachable, retires no one.
+
+## Removing a broker
+
+1. **Point its names at the others.** A `settle` time later nothing new is published to it. It is
+   still consumed from, and what was sent through it is still answered through it.
+2. **Wait until it has no connections.** Each one leaves by itself once, for the `linger` time,
+   it has had nothing to do there. There is no deadline: a connection with a message still in
+   work, or a Request still unanswered, stays.
+3. **Look at what is left** before the broker is stopped:
+   - *a connection that stays* is of one who does not follow the names, or of one whose work
+     there does not end;
+   - *a message in a queue nobody consumes* had no consumer before the broker was retired
+     either, and goes with the broker.
+4. **Stop the broker.**
+
+Until it is stopped, the removal is undone by pointing a name back at the broker: it is in the
+pool again once the answer has settled.
+
+## Replacing a broker
+
+Both of the above in one change of the records: the names of the old broker are pointed at the new
+one. The new broker is joined before the old one is retired, so the order need not be kept by hand.
+The new broker is prepared as [one being added](#adding-a-broker) is, and the old one is stopped as
+[one being removed](#removing-a-broker) is.
+
+## What to expect
+
+- A change of the records may be made in any order and need not be made at once: nothing is acted
+  upon until the answers have stayed the same for the `settle` time.
+- A broker that is in the pool and goes out of reach is a [lost shard](../readme.md#sharded-connection),
+  whatever the names say: what was sent through it and is not answered is sent again through the
+  others. Retiring a broker first is what makes its removal lose nothing.
+- No one is restarted or reconfigured. One who starts while the names are moving connects to what
+  they resolve to then, and follows them from there.

package/features/steps/call.js

@@ -6,7 +6,7 @@ const { Given, When, Then, After } = require('@cucumber/cucumber')
 
 const { connect, Unroutable } = require('../../')
 const { parse } = require('./yaml')
-const { getAddress, USER, PASSWORD } = require('./brokers')
+const { getAddress, USER, PASSWORD, BROKERS_AMOUNT } = require('./brokers')
 const { timeout, until } = require('../../test/helpers')
 
 Given('a holder answering {token} under the {token} key:',
@@ -35,6 +35,8 @@ Given('a holder on another connection answering {token} under the {token} key',
     this.holder = await another(this)
 
     await this.holder.back(exchange, key, echo)
+
+    if (this.sharded) await everywhere(exchange, key)
   })
 
 Given('a holder on another connection answering {token} under the {token} key in {number}ms',
@@ -494,6 +496,28 @@ function url (broker) {
   return `amqp://${USER}:${PASSWORD}@${getAddress(broker)}`
 }
 
+/**
+ * Waits for a key to be held on every broker. It is held once any shard holds it, and on a
+ * broker where the holder of the scenario before still has it, it is claimed for a while yet:
+ * a scenario that takes the other broker down by then has nobody to call.
+ *
+ * @param {string} exchange
+ * @param {string} key
+ */
+async function everywhere (exchange, key) {
+  for (let broker = 0; broker < BROKERS_AMOUNT; broker++) {
+    const caller = await connect(url(broker))
+
+    try {
+      const held = await eventually(() => caller.call(exchange, key, { key }, { timeout: 1000 }), 30)
+
+      assert.equal(held, true, `The key is not held on broker ${broker}`)
+    } finally {
+      await caller.close()
+    }
+  }
+}
+
 /**
  * Makes an attempt until it succeeds, or until the time is up.
  *

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "comq",
-  "version": "0.27.0",
+  "version": "0.28.0",
   "description": "ComQ",
   "author": {
     "name": "Artem Gurtovoi",

package/readme.md

@@ -514,7 +514,7 @@ await io.close()
 
 ### Virtual shards
 
-The URLs of a sharded connection are *names*, and several of them may stand for one broker. A
+Where the URLs of a sharded connection are *names*, several of them may stand for one broker. A
 connection is made to each broker rather than to each name: thirty-two names resolving to two
 addresses are two connections.
 
@@ -523,7 +523,7 @@ const io = await connect('amqp://rmq0.example.com', 'amqp://rmq1.example.com', /
 ```
 
 This is what lets the number of brokers change without anyone being reconfigured: the names stay,
-and what they resolve to moves.
+and what they resolve to moves. See [how a broker is added, removed or replaced](docs/shards.md).
 
 A broker is told by its address, port, virtual host and credentials. A name with several
 addresses is still one broker, connected through one of them: two connections to one broker
@@ -534,9 +534,12 @@ for different brokers.
 
 The names are resolved again every `interval`. What they resolve to is seldom seen changing
 whole: records are updated one after another, and until their TTL has run out caches answer
-with the old address and the new one in turn. So a change is acted upon only once the answer has
-stayed the same for the `settle` time, and anything else observed meanwhile starts that time
-anew. A name that fails to resolve keeps its address, which is reported as `error`.
+with the old address and the new one in turn. So a change of a name is acted upon only once its
+answer has stayed the same for the `settle` time, and anything else observed meanwhile starts
+that time anew. A name that fails to resolve keeps its address, which is reported as `error`.
+
+A name is resolved once per process, however many connections stand on it: those made with the
+same timing share what it resolves to, and each is told when it has changed.
 
 Then the brokers are brought in line with the names, making before breaking:
 
@@ -570,18 +573,18 @@ The trailing argument of `connect` takes it next to the [topology settings](#set
 milliseconds:
 
 ```javascript
-await connect(...urls, { resolution: { interval: 10_000, settle: 60_000, linger: 60_000 } })
+await connect(...urls, { resolution: { interval: 300_000, settle: 900_000, linger: 900_000 } })
 ```
 
-- `interval`: how often the names are resolved, 10 seconds by default.
+- `interval`: how often the names are resolved, 5 minutes by default.
 - `settle`: how long an answer is to hold before it is acted upon.
 - `linger`: how long a retired broker is to stay idle before it is left.
 
-Unless given, `settle` and `linger` are the longest TTL the names have been answered with, and
-no less than 30 seconds: a TTL bounds how long a cache may answer with what is no longer true,
-and says nothing of records being updated one by one. A name that is not found in DNS itself (one
-from the hosts file, or one completed by the resolver's search list) has no TTL, and makes them
-60 seconds.
+Unless given, `settle` is the longest TTL a name has been answered with, `linger` the longest
+of the names, and both no less than 15 minutes: a TTL bounds how long a cache may answer with what
+is no longer true, and says nothing of records being updated one by one. A name that is not found
+in DNS itself (one from the hosts file, or one completed by the resolver's search list) has no
+TTL, and makes them 30 minutes.
 
 ## Singleton connection
 

package/source/shards/connection.js

@@ -3,7 +3,7 @@
 const events = require('../events')
 const channel = require('./channel')
 const emitter = require('../emitter')
-const { Watch } = require('./watch')
+const { Name, INTERVAL_MS, FLOOR_MS } = require('./name')
 const { hostOf, locate } = require('./resolve')
 
 /**
@@ -23,8 +23,17 @@ class Connection {
   /** @type {(url: string, servername?: string) => comq.Connection} */
   #create
 
-  /** @type {Watch} */
-  #watch
+  /** @type {string[]} the names the URLs are of */
+  #hosts
+
+  /** @type {comq.shards.Timing} */
+  #timing
+
+  /** @type {Name[]} the names followed while open */
+  #names = []
+
+  /** @type {Array<() => void>} what stops following each of them */
+  #leaving = []
 
   /** @type {Map<string, string>} the address of each name */
   #addresses = new Map()
@@ -38,6 +47,9 @@ class Connection {
   /** @type {number} the number the next shard gets: one is never given twice */
   #next = 0
 
+  /** @type {Set<string>} the brokers that failed to join, asked again only once an interval is up */
+  #refused = new Set()
+
   /** @type {Set<comq.Channel>} */
   #channels = new Set()
 
@@ -58,17 +70,14 @@ class Connection {
     this.#urls = urls
     this.#create = create
 
-    const hosts = urls.map(hostOf).filter((host) => host !== undefined)
-
-    this.#watch = new Watch(hosts, timing, this.#urgent)
-    this.#watch.on('change', (addresses) => this.#follow(addresses))
-    this.#watch.on('error', (exception) => this.#diagnostics.emit('error', exception))
+    this.#hosts = Array.from(new Set(urls.map(hostOf).filter((host) => host !== undefined)))
+    this.#timing = timing ?? {}
   }
 
   async open () {
     this.#closed = false
 
-    const addresses = await this.#watch.start()
+    const addresses = await this.#start()
 
     if (this.#closed) return
 
@@ -95,7 +104,7 @@ class Connection {
   async close () {
     this.#closed = true
     this.#opened = false
-    this.#watch.stop()
+    this.#unfollow()
 
     clearTimeout(this.#inspection)
 
@@ -191,7 +200,7 @@ class Connection {
     }
 
     // a connection that is lost may be so because its broker has moved
-    const lost = (error) => { if (error !== undefined) this.#watch.poke() }
+    const lost = (error) => { if (error !== undefined) this.#poke(shard.address) }
 
     connection.diagnose('close', lost)
     shard.listeners.push(['close', lost])
@@ -241,14 +250,74 @@ class Connection {
   }
 
   /**
-   * @param {Map<string, string>} addresses
+   * Follows the names, each of which is shared with whoever else in the process follows it.
+   *
+   * @return {Promise<Map<string, string>>} the address of each name
    */
-  #follow (addresses) {
-    this.#addresses = addresses
-    this.#wanted = this.#locations(addresses)
+  async #start () {
+    this.#unfollow()
+
+    const follower = { change: this.#follow, error: this.#error, urgent: this.#urgent }
+
+    this.#names = this.#hosts.map((host) => Name.of(host, this.#timing))
+    this.#leaving = this.#names.map((name) => name.follow(follower))
+
+    await Promise.all(this.#names.map((name) => name.start()))
+
+    // a name followed by others may have moved while the rest were being resolved
+    return new Map(this.#names.map((name) => [name.host, /** @type {string} */ (name.address)]))
+  }
+
+  #unfollow () {
+    for (const leave of this.#leaving) leave()
+
+    this.#names = []
+    this.#leaving = []
+  }
+
+  /**
+   * Looks now at the names that stand for a broker.
+   *
+   * @param {string | undefined} address
+   */
+  #poke (address) {
+    for (const name of this.#names) {
+      if (this.#addresses.get(name.host) === address) name.poke()
+    }
+  }
+
+  /**
+   * @param {string} host
+   * @param {string} address
+   */
+  #follow = (host, address) => {
+    if (this.#addresses.get(host) === address) return
+
+    this.#addresses.set(host, address)
+    this.#wanted = this.#locations(this.#addresses)
     this.#reconcile()
   }
 
+  /**
+   * @param {Error} exception
+   */
+  #error = (exception) => {
+    this.#diagnostics.emit('error', exception)
+  }
+
+  /** How often the shards are looked at again. */
+  get #interval () {
+    return this.#timing.interval ?? INTERVAL_MS
+  }
+
+  /**
+   * How long a retired broker is to stay idle before it is left: as long as any of the names
+   * may be cached for, unless told.
+   */
+  get #linger () {
+    return this.#timing.linger ?? Math.max(FLOOR_MS, ...this.#names.map((name) => name.period))
+  }
+
   /**
    * Brings the shards in line with the brokers the names stand for, making before breaking:
    * nothing is retired until every broker that is named has joined, so a name that points at
@@ -257,13 +326,16 @@ class Connection {
   #reconcile () {
     if (!this.#opened || this.#closed) return
 
+    let made = true
+
     for (const [url, location] of this.#wanted) {
+      if (this.#shardOf(url) !== undefined) continue
+
       // one that is being left is joined anew once it has been
-      if (this.#shardOf(url) === undefined) this.#join(location).catch(noop)
+      if (this.#refused.has(url)) made = false
+      else this.#join(location).catch(noop)
     }
 
-    let made = true
-
     for (const shard of this.#shards.values()) {
       const named = this.#wanted.has(shard.url)
 
@@ -295,7 +367,7 @@ class Connection {
       this.#diagnostics.emit('error', exception, shard.index)
       this.#dismiss(shard)
 
-      return this.#later(() => this.#reconcile())
+      return this.#defer(location.url)
     }
 
     if (shard.state !== JOINING) return
@@ -369,10 +441,25 @@ class Connection {
     // one that never joined is not said to have left
     if (retired) this.#diagnostics.emit(LEAVE, shard.index, shard.address)
 
-    if (failed) this.#later(() => this.#reconcile())
+    if (failed) this.#defer(shard.url)
     else this.#reconcile()
   }
 
+  /**
+   * Asks a broker that failed to join again once an interval is up, rather than whenever
+   * another name moves meanwhile.
+   *
+   * @param {string} url
+   */
+  #defer (url) {
+    this.#refused.add(url)
+
+    this.#later(() => {
+      this.#refused.delete(url)
+      this.#reconcile()
+    })
+  }
+
   /**
    * Looks at the retiring shards for as long as there are any.
    */
@@ -403,7 +490,7 @@ class Connection {
    */
   async #idle (shard) {
     const now = Date.now()
-    const linger = this.#watch.linger
+    const linger = this.#linger
 
     if (shard.connection.connected === false) {
       shard.down ??= now
@@ -424,7 +511,7 @@ class Connection {
    * @return {ReturnType<setTimeout>}
    */
   #later (fn) {
-    const timer = setTimeout(() => Promise.resolve().then(fn).catch(noop), this.#watch.interval)
+    const timer = setTimeout(() => Promise.resolve().then(fn).catch(noop), this.#interval)
 
     timer.unref()
 
@@ -437,7 +524,7 @@ class Connection {
    * @param {Error} exception
    */
   async #cancel (openings, shards, exception) {
-    this.#watch.stop()
+    this.#unfollow()
 
     const settled = await Promise.allSettled(openings)
     const opened = shards.filter((_, index) => settled[index].status === 'fulfilled')