pmtiles-swarm 0.14.2 → 0.15.0

package/CHANGELOG.md

@@ -7,6 +7,43 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.15.0
+### ✨ Features and improvements
+- **The DHT routing table is remembered between runs**, which is the difference between publishing
+  reliably and gambling on each start. Field measurements on a domestic connection: a fresh socket
+  usually found one node and never recovered, while roughly one start in seven found sixteen within
+  five seconds — and no amount of retrying rescued a bad one.
+
+  This is what libtorrent does, and why the libtorrent engine's DHT works on hosts where a fresh
+  bittorrent-dht socket does not: it saves its table and reloads it rather than bootstrapping cold
+  every time. Saved to `dht-nodes.json` in the data directory (`mutable.statePath`), written every
+  five minutes and on shutdown, and never overwritten with an empty table — a bad run must not
+  replace a good table with its own nothing.
+
+  The bootstrap list also now matches libtorrent's rather than the library default, adding
+  `dht.libtorrent.org:25401` and `router.bitcomet.com:6881`. Remembered nodes are tried first and
+  the hostnames stay behind them, since a saved table can be entirely stale.
+- **A DHT socket that cannot reach the network is replaced rather than retried.** Retrying on a bad
+  socket failed at 30s, 60s, 2m and 4m in the field while a restart succeeded, so after two futile
+  cycles the publisher now opens a new socket itself instead of waiting for someone to restart the
+  service. A socket that is finding peers is never replaced, whatever its puts are doing.
+
+## 0.14.3
+### 🐞 Bug fixes
+- **The publisher waits for a usable routing table rather than a single node.** A freshly
+  bootstrapped table holds one entry — the bootstrap host, which stores nothing — so publishing
+  against it failed once per category with "No nodes to query" before any retry could help. It now
+  waits for eight, for up to two minutes, and says what it is waiting for as it goes.
+
+  Two minutes rather than longer because this turned out to be bimodal rather than slow: a socket
+  that can reach the DHT fills its table in a few seconds, and one that cannot is still empty ten
+  minutes later. Waiting past that buys nothing and delays saying so.
+
+  Worth knowing where that comes from, since the log looks like a swarm problem and is not: on a
+  multi-WAN router each new UDP socket is assigned a gateway by the load balancer and then keeps
+  it, so a socket that lands on a WAN with no working return path never recovers — which is why
+  retrying could not rescue a bad run and only a restart changed the outcome.
+
 ## 0.14.2
 ### 🐞 Bug fixes
 - **Waiting for the DHT's `ready` event was not enough.** It fires when the bootstrap lookup

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.14.2",
+  "version": "0.15.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/config.js

@@ -438,6 +438,15 @@ const DEFAULTS = {
      * own, and two sockets cannot hold one port.
      */
     dhtPort: 0,
+    /**
+     * Where to remember the DHT routing table between runs.
+     *
+     * Defaults to `dht-nodes.json` in the data directory. Bootstrapping from
+     * hostnames alone is unreliable enough that a node doing it on every start
+     * is gambling each time; libtorrent saves its table for the same reason,
+     * which is why its DHT works on hosts where a fresh socket does not.
+     */
+    statePath: undefined,
   },
   /**
    * What this node has served, at `GET /api/stats`.

package/src/dht-state.js

@@ -0,0 +1,76 @@
+/**
+ * Keeping a DHT routing table across restarts.
+ *
+ * Bootstrapping is not reliably quick. Measured on a domestic connection, a
+ * fresh socket usually found one node and never recovered, while roughly one
+ * attempt in seven found sixteen within five seconds — and no amount of
+ * retrying rescued a bad one. A node that has to bootstrap from nothing on
+ * every start is therefore gambling on each start.
+ *
+ * libtorrent does not gamble: it saves its routing table and reloads it, which
+ * is why the libtorrent engine's DHT works on hosts where a fresh
+ * bittorrent-dht socket does not. This does the same thing — a table that
+ * worked once is remembered, so later starts begin with peers that answered
+ * recently rather than with three hostnames.
+ */
+
+import fs from 'node:fs/promises';
+
+/**
+ * Bootstrap hosts, as libtorrent uses rather than the shorter default.
+ *
+ * More names is more chances that one answers, and they cost nothing when the
+ * saved table already works. `dht.libtorrent.org` in particular is absent from
+ * bittorrent-dht's defaults.
+ */
+export const BOOTSTRAP = [
+  'router.bittorrent.com:6881',
+  'dht.transmissionbt.com:6881',
+  'router.utorrent.com:6881',
+  'dht.libtorrent.org:25401',
+  'router.bitcomet.com:6881',
+];
+
+/**
+ * Nodes to start from: whatever was saved, plus the bootstrap hosts.
+ * @param {string} [path] - Where the table was saved.
+ * @returns {Promise<Array>} - Addresses for the `bootstrap` option.
+ */
+export async function loadNodes(path) {
+  if (!path) return [...BOOTSTRAP];
+  try {
+    const saved = JSON.parse(await fs.readFile(path, 'utf8'));
+    const nodes = (saved.nodes ?? [])
+      .filter((node) => node?.host && node?.port)
+      .map((node) => `${node.host}:${node.port}`);
+    // The hostnames stay in the list. A saved table can be entirely stale —
+    // a laptop that moved networks, a node that was off for a week — and
+    // falling back to bootstrapping is better than starting nowhere.
+    return [...nodes, ...BOOTSTRAP];
+  } catch {
+    // No file yet, or an unreadable one. Neither is worth reporting: the
+    // bootstrap hosts are a complete answer on their own.
+    return [...BOOTSTRAP];
+  }
+}
+
+/**
+ * Saves the current routing table.
+ * @param {string} [path] - Where to write it.
+ * @param {object} dht - The DHT to read.
+ * @returns {Promise<number>} - How many nodes were saved.
+ */
+export async function saveNodes(path, dht) {
+  if (!path || !dht?.toJSON) return 0;
+  const nodes = dht.toJSON().nodes ?? [];
+  // An empty table is not worth writing, and writing it would replace a good
+  // table from a previous run with the results of a bad one.
+  if (nodes.length === 0) return 0;
+
+  const body = JSON.stringify({ savedAt: new Date().toISOString(), nodes });
+  // Written then renamed, so a crash mid-write cannot leave a truncated file
+  // that the next start has to fail on.
+  await fs.writeFile(`${path}.tmp`, body);
+  await fs.rename(`${path}.tmp`, path);
+  return nodes.length;
+}

package/src/index.js

@@ -1,5 +1,6 @@
 #!/usr/bin/env node
 import fs from 'node:fs/promises';
+import path from 'node:path';
 import { parseArgs } from 'node:util';
 import { createApp } from './api.js';
 import { assertSafeToListen, createAuth } from './auth.js';
@@ -272,29 +273,60 @@ PMTILES_SWARM_PUBLIC_URL
         import('bittorrent-dht').then((m) => m.default),
       ]);
       const pem = await fs.readFile(config.mutable.keyPath, 'utf8');
-      const dht = new DHT();
-      // Bound explicitly rather than left to bind implicitly on first send,
-      // so the port is predictable and can be forwarded if you want this to
-      // be a reachable DHT node. 0 takes an ephemeral one, which is all
-      // publishing needs.
-      await new Promise((resolve, reject) => {
-        dht.once('error', reject);
-        dht.listen(config.mutable.dhtPort ?? 0, resolve);
-      });
-      console.log(`[mutable] DHT on UDP ${dht.address().port}`);
+      // A factory rather than an instance, so the publisher can replace a
+      // socket that proves unable to reach the DHT. Bound explicitly rather
+      // than left to bind implicitly on first send, so the port is
+      // predictable and can be forwarded; 0 takes an ephemeral one, which is
+      // all publishing needs — and is what lets a replacement differ from
+      // what it replaced.
+      const { loadNodes, saveNodes } = await import('./dht-state.js');
+      const statePath =
+        config.mutable.statePath ?? path.join(config.dataDir, 'dht-nodes.json');
+
+      const openDht = async () => {
+        // Started from a table that worked before where there is one, the way
+        // libtorrent does. Bootstrapping from hostnames alone is unreliable
+        // enough that a node doing it on every start is gambling each time.
+        const bootstrap = await loadNodes(statePath);
+        const dht = new DHT({ bootstrap });
+        await new Promise((resolve, reject) => {
+          dht.once('error', reject);
+          dht.listen(config.mutable.dhtPort ?? 0, resolve);
+        });
+        console.log(
+          `[mutable] DHT on UDP ${dht.address().port} ` +
+            `(${bootstrap.length} bootstrap addresses)`,
+        );
+        return dht;
+      };
+
       publisher = new MutablePublisher({
         catalog,
-        dht,
+        createDht: openDht,
         key: publisherKeyFromPem(pem),
         intervalMs: (config.mutable.republishSeconds ?? 1800) * 1000,
       });
+      // Saved periodically rather than only on the way out, because the run
+      // that finds a good table is often the one that is later killed rather
+      // than stopped, and a table nobody wrote down is a table nobody keeps.
+      const saveTable = setInterval(() => {
+        publisher
+          .saveTable((dht) => saveNodes(statePath, dht))
+          .catch(() => {});
+      }, 5 * 60_000);
+      saveTable.unref?.();
+
       stoppers.unshift({
         label: 'mutable publisher',
-        stop: () => {
+        stop: async () => {
+          clearInterval(saveTable);
+          const saved = await publisher
+            .saveTable((dht) => saveNodes(statePath, dht))
+            .catch(() => 0);
+          if (saved) console.log(`[mutable] remembered ${saved} DHT nodes`);
           publisher.stop();
-          dht.destroy();
         },
-        ms: 2000,
+        ms: 3000,
       });
       publisher.start();
     } catch (error) {

package/src/publisher.js

@@ -50,12 +50,48 @@ import { mutableMagnet, publishInfoHash } from './mutable.js';
 /** Republish well inside the ~2h a DHT keeps an item. */
 const DEFAULT_INTERVAL_MS = 30 * 60 * 1000;
 
-/** How long to wait for the DHT to bootstrap before publishing anyway. */
-const DEFAULT_READY_MS = 60_000;
+/**
+ * How long to wait for the DHT to find peers before publishing anyway.
+ *
+ * Minutes rather than seconds, because bootstrapping is not uniformly fast and
+ * publishing early does not fail gracefully -- it fails once per category with
+ * a message that reads like something is broken.
+ *
+ * Not longer, because in practice this is bimodal rather than slow: a socket
+ * that can reach the DHT fills its table in a few seconds, and one that cannot
+ * is still empty ten minutes later. Observed on a multi-WAN router, where each
+ * new socket gets a gateway assigned by the load balancer and is then stuck
+ * with it -- so retrying never rescued a bad socket, and only a restart
+ * re-rolled it. Waiting past a couple of minutes buys nothing and delays
+ * saying so.
+ */
+const DEFAULT_READY_MS = 2 * 60_000;
+
+/** How often to say that it is still waiting, so a slow start is legible. */
+const WAITING_LOG_MS = 60_000;
 
 /** First retry delay after an attempt where nothing was published. */
 const RETRY_MS = 30_000;
 
+/**
+ * Failed cycles before opening a new DHT socket.
+ *
+ * A socket that cannot reach the DHT never recovers: retrying on it failed at
+ * 30s, 60s, 2m and 4m in the field, while a restart -- a new socket, a new
+ * source port, a new NAT state -- worked roughly one attempt in seven. So when
+ * retrying is demonstrably futile, re-roll rather than wait for a human.
+ */
+const REROLL_AFTER = 2;
+
+/**
+ * Nodes wanted before a first publish is worth attempting.
+ *
+ * One is what a freshly bootstrapped table holds, and a put against it fails
+ * with "No nodes to query" -- the entry is the bootstrap host, not a peer that
+ * will store anything.
+ */
+const MINIMUM_NODES = 8;
+
 export class MutablePublisher {
   #catalog;
   #dht;
@@ -64,6 +100,10 @@ export class MutablePublisher {
   #timer = null;
   #retryTimer = null;
   #stopped = false;
+  #createDht = null;
+  #ownsDht = false;
+  /** Consecutive cycles that published nothing, for deciding to re-roll. */
+  #failedCycles = 0;
   #log;
   /** Last published infohash per category, so an unchanged build stays quiet. */
   #published = new Map();
@@ -71,14 +111,19 @@ export class MutablePublisher {
   /**
    * @param {object} options - Wiring.
    * @param {object} options.catalog - Catalog to read categories from.
-   * @param {object} options.dht - A bittorrent-dht instance.
+   * @param {object} [options.dht] - A bittorrent-dht instance to use as-is.
+   * @param {Function} [options.createDht] - Opens a fresh one, already listening.
+   *   Given this, the publisher owns the socket and replaces it when it proves
+   *   unable to reach the DHT.
    * @param {object} options.key - Keypair from `publisherKeyFromPem`.
    * @param {number} [options.intervalMs] - Republish interval.
    * @param {Function} [options.log] - Where to report.
    */
-  constructor({ catalog, dht, key, intervalMs, log = console.log }) {
+  constructor({ catalog, dht, createDht, key, intervalMs, log = console.log }) {
     this.#catalog = catalog;
-    this.#dht = dht;
+    this.#dht = dht ?? null;
+    this.#createDht = createDht ?? null;
+    this.#ownsDht = Boolean(createDht) && !dht;
     this.#key = key;
     this.#intervalMs = intervalMs ?? DEFAULT_INTERVAL_MS;
     this.#log = log;
@@ -149,7 +194,10 @@ export class MutablePublisher {
       }
     }
     if (done.length === 0 && this.#current().size > 0) {
+      this.#failedCycles += 1;
       this.#scheduleRetry(options.retryDelayMs ?? RETRY_MS);
+    } else if (done.length > 0) {
+      this.#failedCycles = 0;
     }
     return done;
   }
@@ -175,6 +223,7 @@ export class MutablePublisher {
    * Starts publishing, and keeps republishing before the records expire.
    * @param {object} [options] - Timing.
    * @param {number} [options.readyMs] - Grace before the first publish.
+   * @param {number} [options.retryMs] - First retry delay, for tests.
    * @returns {void}
    */
   start(options = {}) {
@@ -183,7 +232,7 @@ export class MutablePublisher {
     // bet on how long bootstrapping takes -- one this lost in the field, where
     // fifteen seconds was not enough and every category failed on the first
     // attempt.
-    this.#firstPublish(options.readyMs ?? DEFAULT_READY_MS).catch((error) =>
+    this.#firstPublish(options.readyMs ?? DEFAULT_READY_MS, options.retryMs).catch((error) =>
       this.#log(`[mutable] first publish failed: ${error.message}`),
     );
 
@@ -205,23 +254,24 @@ export class MutablePublisher {
    * @param {number} readyMs - How long to wait before going ahead regardless.
    * @returns {Promise<void>} - Resolves once the first attempt is done.
    */
-  async #firstPublish(readyMs) {
+  async #firstPublish(readyMs, retryMs) {
+    if (!this.#dht && this.#createDht) this.#dht = await this.#createDht();
     const nodes = await this.#whenDhtReady(readyMs);
     if (this.#stopped) return;
-    if (nodes === 0) {
+    if (nodes !== null && nodes < MINIMUM_NODES) {
       // Named rather than left to be inferred. A routing table that is still
       // empty after a minute means the bootstrap queries are not being
       // answered, and every "No nodes to query" after this is that same fact
       // reported once per category.
       this.#log(
-        `[mutable] the DHT found no peers in ${Math.round(readyMs / 1000)}s. Publishing ` +
-          'will fail until it does — check that outbound UDP is not blocked, and that ' +
-          'the bootstrap hosts resolve',
+        `[mutable] the DHT found only ${nodes} nodes in ${Math.round(readyMs / 60000)} ` +
+          'minutes. Publishing will keep retrying — if it never succeeds, check that ' +
+          'outbound UDP is not blocked and that the bootstrap hosts resolve',
       );
     } else if (nodes) {
       this.#log(`[mutable] DHT ready with ${nodes} nodes`);
     }
-    await this.publishAll({ force: true });
+    await this.publishAll({ force: true, retryDelayMs: retryMs });
   }
 
   /**
@@ -248,10 +298,25 @@ export class MutablePublisher {
       });
     }
 
+    // Waited out rather than given up on. A table with a single bootstrap
+    // entry is not enough to store a record -- the put needs somewhere to put
+    // it -- so this holds until there are a few, saying so as it goes.
     let count = this.#nodeCount();
-    while (count === 0 && Date.now() < deadline && !this.#stopped) {
+    let lastSaid = Date.now();
+    while (count !== null && count < MINIMUM_NODES && Date.now() < deadline && !this.#stopped) {
+      if (Date.now() - lastSaid >= WAITING_LOG_MS) {
+        lastSaid = Date.now();
+        this.#log(
+          `[mutable] waiting for the DHT (${count} nodes so far). ` +
+            'Bootstrapping can take several minutes on a home connection',
+        );
+      }
+      // Never past the deadline: sleeping a flat two seconds overshoots a
+      // short wait by the whole interval, which makes the wait unpredictable
+      // and the behaviour hard to test.
+      const remaining = deadline - Date.now();
       await new Promise((resolve) => {
-        const timer = setTimeout(resolve, 1000);
+        const timer = setTimeout(resolve, Math.max(10, Math.min(2000, remaining)));
         timer.unref?.();
       });
       count = this.#nodeCount();
@@ -281,17 +346,64 @@ export class MutablePublisher {
    */
   #scheduleRetry(delayMs) {
     if (this.#stopped || this.#retryTimer) return;
-    const timer = setTimeout(() => {
+    const timer = setTimeout(async () => {
       this.#retryTimer = null;
       if (this.#stopped) return;
-      this.publishAll({ retryDelayMs: Math.min(delayMs * 2, this.#intervalMs) }).catch(
-        (error) => this.#log(`[mutable] retry failed: ${error.message}`),
-      );
+      try {
+        await this.#rerollIfHopeless();
+        await this.publishAll({
+          retryDelayMs: Math.min(delayMs * 2, this.#intervalMs),
+        });
+      } catch (error) {
+        this.#log(`[mutable] retry failed: ${error.message}`);
+      }
     }, delayMs);
     timer.unref?.();
     this.#retryTimer = timer;
   }
 
+  /**
+   * Lets a caller read the current DHT, whichever socket that now is.
+   *
+   * Needed because this may replace the socket it was given, so a caller
+   * holding the original would save a table belonging to a socket that has
+   * been closed.
+   * @param {Function} save - Receives the live DHT.
+   * @returns {Promise<*>} - Whatever `save` returns, or 0 with no socket.
+   */
+  async saveTable(save) {
+    if (!this.#dht) return 0;
+    return save(this.#dht);
+  }
+
+  /**
+   * Replaces the DHT socket when retrying on it is demonstrably futile.
+   *
+   * Only when this publisher opened the socket, and only while its table is
+   * still unusable — a socket that is finding peers is working, whatever the
+   * puts are doing, and swapping it would throw away a good one.
+   * @returns {Promise<void>} - Resolves once any replacement is ready.
+   */
+  async #rerollIfHopeless() {
+    if (!this.#ownsDht || this.#failedCycles < REROLL_AFTER) return;
+    const count = this.#nodeCount();
+    if (count === null || count >= MINIMUM_NODES) return;
+
+    this.#log(
+      `[mutable] this DHT socket has found ${count} nodes and is not recovering — ` +
+        'opening a new one',
+    );
+    try {
+      this.#dht?.destroy?.();
+    } catch {
+      // A socket being replaced is not worth reporting if it objects to closing.
+    }
+    this.#dht = await this.#createDht();
+    this.#failedCycles = 0;
+    const found = await this.#whenDhtReady(DEFAULT_READY_MS);
+    if (found !== null) this.#log(`[mutable] new DHT socket has ${found} nodes`);
+  }
+
   /**
    * Stops republishing.
    * @returns {void}
@@ -302,5 +414,13 @@ export class MutablePublisher {
     if (this.#retryTimer) clearTimeout(this.#retryTimer);
     this.#timer = null;
     this.#retryTimer = null;
+    // Only what this publisher opened: a caller that supplied a socket owns it.
+    if (this.#ownsDht) {
+      try {
+        this.#dht?.destroy?.();
+      } catch {
+        // Shutting down; nothing useful to do about a socket that will not close.
+      }
+    }
   }
 }