pmtiles-swarm 0.11.0 → 0.12.0

package/CHANGELOG.md

@@ -7,6 +7,45 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.12.0
+### ✨ Features and improvements
+- **A category can now be addressed without a server at all.** A node that builds can publish a
+  signed DHT record (BEP 46) naming whichever archive is currently newest in each category, so a
+  style can point at a magnet that never goes stale:
+
+  ```
+  magnet:?xs=urn:btpk:<public key>&s=openmaptiles&dn=…&ws=…
+  ```
+
+  No infohash in it, which is the whole point — an infohash is what goes stale on the next build,
+  and it is why the fragment convention added in 0.11.0 could not be used for `/latest/` URLs. The
+  salt is the category name, so **one keypair addresses every category** rather than needing one
+  each.
+
+  Turn it on with `mutable.publish` and a key from the new **`pmtiles-swarm publisher-key`**
+  command. Off by default.
+
+  **Only the node that builds needs the key.** Serving nodes receive the public half on the catalog
+  entry, through the same sync that already carries `magnet` and `webSeeds`, and assemble the
+  identical magnet from it — there is nothing secret in one. Ten nodes behind a balancer hand out
+  the same string and none of them can publish. Run exactly one publisher: two under one key would
+  fight over the sequence number.
+
+  It is a **signing key rather than a credential**. Whoever holds it can tell your subscribers that
+  any archive is the current build, signed, and clients will believe it.
+
+  Records expire from the DHT after roughly two hours, so the node republishes on a timer
+  (`republishSeconds`, default 1800). That timer is the feature, not an optimisation — without it
+  a record published once works all afternoon and quietly stops resolving by evening. A category
+  whose put fails does not stop the others.
+
+  **`bittorrent-dht` is now a direct dependency** rather than reached for through webtorrent's
+  client, so publishing works on a node running the libtorrent engine alone.
+- **The TileJSON's `torrent.mutable` block carries the magnet**, built from the public key, so no
+  consumer has to know how to assemble one. `mutableMagnet()` also accepts a hex key now — which
+  is all a serving node has — and carries `ws=` web seeds, so a client with no peers can still
+  range-read the archive over HTTP.
+
 ## 0.11.0
 ### ✨ Features and improvements
 - **The magnet can travel in the TileJSON URL's fragment**, and the console will build that string

package/docs/serving-tiles.md

@@ -226,6 +226,54 @@ nothing publishes those records yet.
 For an immutable `/archives/<infohash>/tiles.json` URL the question does not
 arise: both halves name the same fixed archive.
 
+### A fragment that survives a rebuild
+
+The caveat above — a pinned infohash going stale — is what BEP 46 fixes. A node
+that publishes signs a DHT record naming whichever infohash is current, and the
+magnet then names the **category** rather than a build:
+
+```
+magnet:?xs=urn:btpk:<public key>&s=openmaptiles&dn=…&ws=…
+```
+
+No infohash anywhere, so nothing to go stale. A client resolves the record over
+the DHT and joins whatever is current.
+
+Turn it on with a key on the node that builds:
+
+```sh
+pmtiles-swarm publisher-key > /etc/pmtiles-swarm/publisher.pem
+chmod 600 /etc/pmtiles-swarm/publisher.pem
+```
+
+```json
+{ "mutable": { "publish": true, "keyPath": "/etc/pmtiles-swarm/publisher.pem" } }
+```
+
+The magnet then appears in every TileJSON as `torrent.mutable.magnet`, and the
+console's copy button uses it for category URLs.
+
+**Only the node that builds needs the key.** Serving nodes receive the public
+half on the catalog entry, through the same subscription sync that carries
+`magnet` and `webSeeds`, and assemble the identical magnet from it — there is
+nothing secret in one. Ten nodes behind a balancer hand out the same string and
+none of them can publish.
+
+**Run exactly one publisher.** Two nodes publishing under one key would fight
+over the sequence number, each overwriting the other's claim about what is
+current.
+
+**It is a signing key, not a credential.** Whoever holds it can tell your
+subscribers that any archive is the current build, signed, and they will believe
+it. Treat it the way you would a code-signing key: lose it and every style
+pointing at that public key breaks permanently.
+
+**Records expire after roughly two hours**, so the node republishes on a timer
+(`republishSeconds`, default 1800). That timer is not an optimisation — without
+it a record published once works all afternoon and stops resolving by evening.
+If the publisher is offline longer than that, the DHT path goes quiet until it
+returns; the HTTP TileJSON URL is unaffected.
+
 ## The `torrent` block
 
 TileJSON documents from this server carry a non-standard `torrent` member:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.11.0",
+  "version": "0.12.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",
@@ -32,6 +32,7 @@
   "dependencies": {
     "@maplibre/maplibre-gl-inspect": "^1.8.2",
     "bencode": "^4.0.1",
+    "bittorrent-dht": "^11.0.12",
     "chokidar": "^5.0.0",
     "create-torrent": "^6.1.0",
     "express": "^5.2.1",

package/src/config.js

@@ -397,6 +397,35 @@ const DEFAULTS = {
    * instead sidesteps the question entirely.
    */
   trustProxy: false,
+  /**
+   * Announcing the current build of each category over the DHT (BEP 46).
+   *
+   * A category is the only stable handle this system has — every archive is
+   * addressed by its infohash, so a style pointing at one goes stale on the
+   * next build. `/latest/<category>/` fixes that with a server; this fixes it
+   * without one, as a signed DHT record naming whichever infohash is current.
+   *
+   * **Only the node that builds needs this.** Serving nodes carry the public
+   * half on the catalog entry and hand it out in the TileJSON; publishing is
+   * the only thing the secret is used for. Two nodes publishing under one key
+   * would fight over the sequence number, so run exactly one publisher.
+   *
+   * The key is a signing key, not a credential: whoever holds it can tell your
+   * subscribers that any archive is the current build, signed. Treat it the way
+   * you would a code-signing key. Generate one with
+   * `pmtiles-swarm publisher-key`.
+   *
+   * Records expire from the DHT after roughly two hours, so this republishes on
+   * a timer. Nothing else keeps them alive.
+   */
+  mutable: {
+    /** Publish records. Off unless a key is configured and this is set. */
+    publish: false,
+    /** PEM file holding the ed25519 keypair. Never leaves the publisher. */
+    keyPath: undefined,
+    /** How often to republish, in seconds. */
+    republishSeconds: 1800,
+  },
   /**
    * What this node has served, at `GET /api/stats`.
    *

package/src/index.js

@@ -109,6 +109,7 @@ async function main() {
 Usage:
   pmtiles-swarm [--config FILE]          start the node
   pmtiles-swarm status [--config FILE]   ask a running node what it is doing
+  pmtiles-swarm publisher-key            print a new BEP 46 signing key
 
   --config, -c   path to a JSON config file
   --port,   -p   override the listen port
@@ -135,6 +136,20 @@ PMTILES_SWARM_PUBLIC_URL
     return;
   }
 
+  if (positionals[0] === 'publisher-key') {
+    const { generatePublisherKey, publisherKeyToPem } = await import('./mutable.js');
+    const key = generatePublisherKey();
+    // The PEM on stdout so it can be redirected to a file; everything else on
+    // stderr so that redirect stays clean.
+    process.stdout.write(publisherKeyToPem(key));
+    console.error('');
+    console.error(`public key: ${Buffer.from(key.publicKey).toString('hex')}`);
+    console.error('Save the PEM where only this node can read it, and point');
+    console.error('mutable.keyPath at it. It signs what your subscribers');
+    console.error('believe is the current build, so treat it as a signing key.');
+    return;
+  }
+
   if (positionals.length > 0) {
     console.error(`unknown command: ${positionals[0]}`);
     console.error('try: pmtiles-swarm status');
@@ -245,6 +260,42 @@ PMTILES_SWARM_PUBLIC_URL
       ? null
       : new TileStats({ recent: config.tileStats?.recent });
 
+  // Announcing the current build of each category over the DHT. Only ever on
+  // the node that builds: the key signs what subscribers believe is current,
+  // and two publishers under one key would fight over the sequence number.
+  let publisher;
+  if (config.mutable?.publish && config.mutable?.keyPath) {
+    try {
+      const [{ publisherKeyFromPem }, { MutablePublisher }, DHT] = await Promise.all([
+        import('./mutable.js'),
+        import('./publisher.js'),
+        import('bittorrent-dht').then((m) => m.default),
+      ]);
+      const pem = await fs.readFile(config.mutable.keyPath, 'utf8');
+      const dht = new DHT();
+      publisher = new MutablePublisher({
+        catalog,
+        dht,
+        key: publisherKeyFromPem(pem),
+        intervalMs: (config.mutable.republishSeconds ?? 1800) * 1000,
+      });
+      stoppers.unshift({
+        label: 'mutable publisher',
+        stop: () => {
+          publisher.stop();
+          dht.destroy();
+        },
+        ms: 2000,
+      });
+      publisher.start();
+    } catch (error) {
+      // Never fatal: a node that cannot publish should still serve. Loudly
+      // reported, because the failure is otherwise invisible until a
+      // subscriber's style quietly stops resolving.
+      console.error(`[mutable] not publishing: ${error.message}`);
+    }
+  }
+
   const tiles = new TileStore({ catalog, engine, config });
   library.attachTiles(tiles);
   const warm = new WarmRunner(tiles);

package/src/mutable.js

@@ -82,21 +82,35 @@ function rawPublicKey(publicKey) {
  *
  * Note this carries no infohash: `xs=urn:btpk:` names the public key, and the
  * client resolves it through the DHT to whatever infohash is current.
- * @param {Uint8Array} publicKey - Raw 32-byte public key.
+ * @param {Uint8Array | string} publicKey - Raw 32-byte public key, or its hex form.
+ *   A serving node has only the hex, off the catalog entry, and must be able to
+ *   build this string without ever seeing the raw key or the private half.
  * @param {object} [options] - Extra magnet parameters.
  * @param {string} [options.name] - Display name for the archive.
  * @param {string[]} [options.trackers] - Tracker announce URLs.
+ * @param {string[]} [options.webSeeds] - BEP 19 web seeds.
  * @param {string} [options.salt] - Salt, when one key publishes several archives.
  * @returns {string} - A BEP 46 magnet URI.
  */
 export function mutableMagnet(publicKey, options = {}) {
-  const hex = Buffer.from(publicKey).toString('hex');
+  // Buffer.from(string) would read hex as UTF-8 and produce a 64-byte key, so
+  // the two forms have to be told apart rather than coerced.
+  const hex =
+    typeof publicKey === 'string'
+      ? publicKey.toLowerCase()
+      : Buffer.from(publicKey).toString('hex');
   const parts = [`magnet:?xs=urn:btpk:${hex}`];
   if (options.name) parts.push(`dn=${encodeURIComponent(options.name)}`);
   if (options.salt) parts.push(`s=${encodeURIComponent(options.salt)}`);
   for (const tracker of options.trackers ?? []) {
     parts.push(`tr=${encodeURIComponent(tracker)}`);
   }
+  // Carried because it is what makes a magnet useful with no peers at all: a
+  // client can range-read the archive over HTTP and still be correct, which is
+  // the difference between a slow first paint and a blank map.
+  for (const seed of options.webSeeds ?? []) {
+    parts.push(`ws=${encodeURIComponent(seed)}`);
+  }
   return parts.join('&');
 }
 

package/src/publisher.js

@@ -0,0 +1,178 @@
+/**
+ * Announcing the current build of a category over the DHT (BEP 46).
+ *
+ * A category is the only stable handle this system has. Every archive is
+ * addressed by its infohash, which is what makes a tile immutable and leaves a
+ * style with nothing to point at that survives a rebuild. `/latest/<category>/`
+ * solves that with a server; this solves it without one — a signed DHT record,
+ * addressed by public key, naming whichever infohash is current.
+ *
+ * Only the node that builds needs the private key. Serving nodes carry the
+ * public half on the catalog entry and hand it out in the TileJSON, and
+ * publishing is the only operation the secret is used for. Two nodes
+ * publishing under one key would fight over `seq`, so this is deliberately a
+ * single-publisher design.
+ *
+ * The salt is the category name, so one keypair addresses every category
+ * rather than needing one each.
+ *
+ * The part that rots quietly: DHT nodes drop mutable items after roughly two
+ * hours. A record published once works all afternoon and is gone by evening,
+ * so republishing on a timer is not an optimisation here — it is the feature.
+ */
+
+import { mutableMagnet, publishInfoHash } from './mutable.js';
+
+/** Republish well inside the ~2h a DHT keeps an item. */
+const DEFAULT_INTERVAL_MS = 30 * 60 * 1000;
+
+/** Grace period before the first publish, for the DHT to find peers. */
+const DEFAULT_READY_MS = 15_000;
+
+export class MutablePublisher {
+  #catalog;
+  #dht;
+  #key;
+  #intervalMs;
+  #timer = null;
+  #stopped = false;
+  #log;
+  /** Last published infohash per category, so an unchanged build stays quiet. */
+  #published = new Map();
+
+  /**
+   * @param {object} options - Wiring.
+   * @param {object} options.catalog - Catalog to read categories from.
+   * @param {object} options.dht - A bittorrent-dht instance.
+   * @param {object} options.key - Keypair from `publisherKeyFromPem`.
+   * @param {number} [options.intervalMs] - Republish interval.
+   * @param {Function} [options.log] - Where to report.
+   */
+  constructor({ catalog, dht, key, intervalMs, log = console.log }) {
+    this.#catalog = catalog;
+    this.#dht = dht;
+    this.#key = key;
+    this.#intervalMs = intervalMs ?? DEFAULT_INTERVAL_MS;
+    this.#log = log;
+  }
+
+  /** @returns {string} - The public key, hex, which is safe to publish. */
+  get publicKeyHex() {
+    return Buffer.from(this.#key.publicKey).toString('hex');
+  }
+
+  /**
+   * The newest archive in each category — what a record points at.
+   * @returns {Map<string, object>} - Category to catalog entry.
+   */
+  #current() {
+    const newest = new Map();
+    for (const category of this.#catalog.categories()) {
+      // byCategory() is newest first, the same rule /latest/<category>/ uses,
+      // so the record and the endpoint cannot disagree about what is current.
+      const [entry] = this.#catalog.byCategory(category);
+      if (entry) newest.set(category, entry);
+    }
+    return newest;
+  }
+
+  /**
+   * Publishes every category once.
+   * @param {object} [options] - Behaviour.
+   * @param {boolean} [options.force] - Log even when nothing changed.
+   * @returns {Promise<object[]>} - What was published.
+   */
+  async publishAll(options = {}) {
+    const done = [];
+    for (const [category, entry] of this.#current()) {
+      if (this.#stopped) break;
+      // Republished even when unchanged: the record expires whether or not the
+      // build has moved. `force` only decides whether it is worth saying.
+      const changed = this.#published.get(category) !== entry.infoHash;
+      try {
+        const result = await publishInfoHash(this.#dht, this.#key, entry.infoHash, {
+          salt: category,
+        });
+        this.#published.set(category, entry.infoHash);
+        done.push({ category, infoHash: entry.infoHash, ...result });
+
+        // Stamped on the entry so the TileJSON's torrent block carries the
+        // identity without any endpoint needing to know a publisher exists.
+        await this.#catalog.put({
+          infoHash: entry.infoHash,
+          mutable: { publicKey: this.publicKeyHex, salt: category, seq: result.seq },
+        });
+
+        if (changed || options.force) {
+          this.#log(
+            `[mutable] ${category} -> ${entry.infoHash.slice(0, 12)} ` +
+              `(seq ${result.seq}, ${result.nodes} nodes)`,
+          );
+        }
+      } catch (error) {
+        // One category failing must not stop the rest.
+        this.#log(`[mutable] ${category} failed: ${error.message}`);
+      }
+    }
+    return done;
+  }
+
+  /**
+   * The magnet a style should point at for a category.
+   *
+   * Public key only — there is nothing secret in it, which is why every
+   * serving node can hand it out and none of them can publish.
+   * @param {string} category - Which category.
+   * @param {object} [entry] - Newest entry, for the name and web seeds.
+   * @returns {string} - A BEP 46 magnet URI.
+   */
+  magnetFor(category, entry) {
+    return mutableMagnet(this.#key.publicKey, {
+      salt: category,
+      name: entry?.name,
+      webSeeds: entry?.webSeeds,
+    });
+  }
+
+  /**
+   * Starts publishing, and keeps republishing before the records expire.
+   * @param {object} [options] - Timing.
+   * @param {number} [options.readyMs] - Grace before the first publish.
+   * @returns {void}
+   */
+  start(options = {}) {
+    // A put into a DHT that has not found peers yet reaches nobody, and
+    // bootstrapping is the first thing a fresh node does. Waiting costs one
+    // interval of staleness at worst and makes the first publish mean
+    // something.
+    const first = setTimeout(() => {
+      if (this.#stopped) return;
+      this.publishAll({ force: true }).catch((error) =>
+        this.#log(`[mutable] first publish failed: ${error.message}`),
+      );
+    }, options.readyMs ?? DEFAULT_READY_MS);
+    first.unref?.();
+
+    this.#timer = setInterval(() => {
+      this.publishAll().catch((error) =>
+        this.#log(`[mutable] republish failed: ${error.message}`),
+      );
+    }, this.#intervalMs);
+    this.#timer.unref?.();
+
+    this.#log(
+      `[mutable] publishing ${this.#catalog.categories().length} categories as ` +
+        `${this.publicKeyHex.slice(0, 16)}… every ${Math.round(this.#intervalMs / 60000)}m`,
+    );
+  }
+
+  /**
+   * Stops republishing.
+   * @returns {void}
+   */
+  stop() {
+    this.#stopped = true;
+    if (this.#timer) clearInterval(this.#timer);
+    this.#timer = null;
+  }
+}

package/src/tilejson.js

@@ -1,3 +1,4 @@
+import { mutableMagnet } from './mutable.js';
 import { TileStore } from './tiles.js';
 
 /**
@@ -95,6 +96,13 @@ function buildTorrentBlock(entry, root) {
       publicKey: entry.mutable.publicKey,
       salt: entry.mutable.salt,
       seq: entry.mutable.seq,
+      // Built here so a consumer does not have to know how to assemble one,
+      // and buildable by any node, because it contains only the public half.
+      magnet: mutableMagnet(entry.mutable.publicKey, {
+        salt: entry.mutable.salt,
+        name: entry.name,
+        webSeeds: entry.webSeeds,
+      }),
     };
   }
   return block;