pmtiles-swarm 0.13.0 → 0.14.0

package/CHANGELOG.md

@@ -7,6 +7,60 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.14.0
+### ✨ Features and improvements
+- **The Categories page hands you the URL a style should actually use.** A new **For a style** row
+  gives the category's TileJSON URL with a magnet in its fragment, which is the string worth
+  copying — the plain TileJSON row is still there for anything that only speaks HTTP.
+
+  Where a publisher is announcing the category over the DHT, the fragment carries the **mutable**
+  magnet (`xs=urn:btpk:…&s=<category>`), which is the form a category needs: it names the category
+  rather than a build, so it does not go stale on the next one while the URL keeps following it.
+  Without a publisher it falls back to the newest build's own magnet, which pins that build but
+  still beats a blank map on the only occasion it is read at all.
+
+  Also on `GET /api/categories` as `endpoints.styleUrl`, and `null` for a category whose newest
+  archive is not PMTiles — the same rule the tile endpoints already follow.
+
+  The magnet was already available on an individual archive, but a category URL is what a style
+  points at, and that is the page where it was missing.
+
+### 🐞 Bug fixes
+- **CI and the release workflow install with `--ignore-scripts`.** The 0.14.0 release failed in
+  `npm ci` because `node-datachannel` — WebTorrent's WebRTC binary, four levels down the tree —
+  found no prebuilt binary for the runner and crashed trying to build from source. The same
+  lockfile had published minutes earlier, so it was the download rather than an incompatibility.
+
+  Nothing in CI needs that binary: the suite passes without it, and `npm publish` ships source
+  rather than `node_modules`, so install scripts have no bearing on what is published. A
+  third-party binary download should not be able to block a release. Whether WebRTC works on a
+  given machine is still checked where it matters, after installing on the host.
+
+## 0.13.1
+### 🐞 Bug fixes
+- **A node with a secondary engine no longer takes a quarter of an hour to start listening.**
+  Handing an archive to a second seeding client makes it hash every byte before it will serve any
+  — minutes for tens of gigabytes — and that hand-over was **awaited** inside `add()`. On startup,
+  where the library is restored one archive at a time, the cost landed end to end before the node
+  would bind its port. A seventeen-archive library sat silent for about fifteen minutes.
+
+  The hand-over is now queued and the caller carries on. Nothing depended on waiting for it: the
+  periodic sweep that already exists to catch archives finishing later is the same mechanism, and
+  a failed hand-over already un-marks itself so that sweep retries it.
+
+  Queued through one chain rather than fired off freely, so hand-overs still run one at a time —
+  seventeen archives hashing at once on a spinning disk is slower than seventeen in turn, and much
+  harder to reason about.
+- **Restoring a large library reports progress.** It said nothing at all until it had finished, so
+  a node that was working and a node that was stuck looked identical for as long as it took — long
+  enough, on a real library, to go looking for a debugger:
+
+  ```
+  [restore] 6 of 17 after 15s
+  ```
+
+  On a timer rather than per archive, so a small library stays quiet.
+
 ## 0.13.0
 ### ✨ Features and improvements
 - **The publisher's DHT socket is bound explicitly, on a configurable port.** It was left to bind

package/docs/architecture-diagram.md

@@ -224,7 +224,7 @@ New data means a new infohash, which is what makes cache invalidation free.
 %%{init: {"flowchart": {"rankSpacing": 60}}}%%
 graph LR
     NEW["new planet.pmtiles<br/><small>weekly build</small>"] --> HASH["primary re-hashes<br/>→ new infohash"]
-    HASH --> BEP["BEP 46 mutable entry<br/><small>same public key, seq+1</small>"]
+    HASH --> BEP["BEP 46 mutable entry<br/><small>same public key, new seq</small>"]
     HASH --> FEED["new RSS item"]
 
     BEP --> FOLLOW["clients following the key<br/>see the new version"]
@@ -239,8 +239,36 @@ graph LR
 **Key points:** tile URLs are content-addressed, so they never need invalidating.
 The old infohash stays valid and servable for as long as anyone still holds it —
 useful for clients pinned to a known-good build — while new requests move to the
-new one as soon as they re-read `tiles.json`. The only mutable thing in the whole
-system is that one document.
+new one as soon as they re-read `tiles.json`.
+
+**There are exactly two mutable things**, and they say the same thing by
+different means: the `/latest/<category>/` documents, which need this server, and
+the BEP 46 record, which does not. A style pointing at the public key resolves
+the current build over the DHT with nothing of ours running at all. Only the node
+that builds signs those records — see
+[security.md](security.md#the-publisher-key-is-not-a-credential), because the key
+behaves unlike every other secret in the system.
+
+The sequence number is derived from the clock rather than incremented, so a
+publisher that is rebuilt or restored from backup carries on without needing to
+remember where it had got to.
+
+### Bootstrapping without the server
+
+A torrent-aware client still has to *learn* the magnet from somewhere, and until
+it does, the swarm — the part that depends on no server — is unreachable
+precisely when the server is down. The fix is that the magnet travels in the
+**fragment** of the TileJSON URL a style already carries:
+
+```
+https://swarm.example.org/latest/openmaptiles/tiles.json#magnet:?xs=urn:btpk:…&s=openmaptiles&ws=…
+```
+
+A fragment is never sent in an HTTP request, so ordinary clients fetch the
+TileJSON and ignore it while a swarm-aware one reads the magnet before making any
+call. With a BEP 46 key in it (`xs=urn:btpk:`) rather than an infohash, that
+string does not go stale on the next build either. See
+[serving-tiles.md](serving-tiles.md#a-fragment-that-survives-a-rebuild).
 
 ---
 

package/docs/security.md

@@ -188,6 +188,56 @@ Three ways past it, in order of preference:
 2. Bind to `127.0.0.1` and reach it through a reverse proxy that authenticates.
 3. Set `allowUnauthenticated: true`, if the network really is trusted.
 
+## The publisher key is not a credential
+
+Everything above guards *this node*: who may read the API, who may change
+settings, who may add an archive. The publisher key is a different kind of
+secret, and the difference is worth stating plainly because the habits do not
+carry over.
+
+`mutable.keyPath` holds an ed25519 private key used to sign BEP 46 records — the
+DHT entries that say which archive is the current build of a category. It is a
+**signing key**, closer to a code-signing certificate than to an API token.
+
+| | An access credential (`apiKey`, a token, `password`) | The publisher key |
+| --- | --- | --- |
+| What it grants | access to one node | authority over what your subscribers believe |
+| If it leaks | that node is compromised | anyone can sign a record pointing your subscribers at any archive |
+| If you lose it | mint another | every style pointing at that public key breaks, permanently |
+| Rotation | revoke and reissue | there is none |
+
+That third row is the one people are unprepared for. A public key **is** the
+identity: there is no registry to update and no way to tell a subscriber that a
+new key is also you. Losing the file ends that identity.
+
+And the second row is why it does not belong on a serving tier. A leaked API key
+lets someone read a node. A leaked publisher key lets someone tell every
+subscriber that an archive they control is your latest planet build — signed,
+verifying correctly, and indistinguishable from a real announcement.
+
+**So it lives on exactly one machine.** Only the node that builds needs it;
+serving nodes carry the public half on the catalog entry and hand it out in the
+TileJSON, which is why a serving tier can be compromised without anyone being
+able to publish. Running two publishers under one key is separately a mistake:
+they fight over the sequence number.
+
+Practically:
+
+- `chmod 400`, owned by the service account. Nothing in the product writes it
+  back, so it does not need to be writable and should not be.
+- Back it up **off the machine**, and treat the backup as seriously as the
+  original.
+- Do not put it in an image, a config-sync set, or anywhere a standby will
+  receive it. Under HA sync the configuration replicates and the key does not,
+  which is the intended asymmetry rather than an oversight — the standby logs
+  `not publishing: ENOENT` and serves normally.
+- It is unrelated to `auth`. Turning authentication off does not expose it, and
+  rotating the API key does not affect it.
+
+Generate one with `pmtiles-swarm publisher-key`. Setup detail is in
+[running-as-a-service.md](running-as-a-service.md#the-publisher-key); what it is
+*for* is in [serving-tiles.md](serving-tiles.md#a-fragment-that-survives-a-rebuild).
+
 ## What this is not
 
 **It is not a defence against someone on your network.** Sessions are bearer

package/docs/serving-tiles.md

@@ -291,7 +291,13 @@ TileJSON documents from this server carry a non-standard `torrent` member:
     "torrent": "https://swarm.example.org/archives/913d…/archive.torrent",
     "name": "planet.pmtiles",
     "size": 77242531840,
-    "webseeds": ["https://maps.example.org/planet.pmtiles"]
+    "webseeds": ["https://maps.example.org/planet.pmtiles"],
+    "mutable": {
+      "publicKey": "7680dc95248eb807…",
+      "salt": "openmaptiles",
+      "seq": 1786108931,
+      "magnet": "magnet:?xs=urn:btpk:7680dc95248eb807…&s=openmaptiles&ws=…"
+    }
   }
 }
 ```
@@ -309,10 +315,29 @@ load it. That also means a torrent-aware client gets a working map immediately
 over HTTP while the swarm is still finding peers, rather than staring at an empty
 canvas for the 90 to 240 seconds a cold magnet can take to resolve metadata.
 
-For an archive published as a mutable torrent, the block also carries
-`mutable.publicKey`, so a client that understands BEP 46 can follow updates
-rather than pinning to the version the document was generated from. See
-[publishing](publishing.md).
+### The `mutable` sub-block
+
+Present only when a publisher is announcing this archive's category over the
+DHT. It is the difference between a document describing *this build* and one
+describing *the current build*:
+
+| | |
+| --- | --- |
+| `publicKey` | The identity to resolve against. Public — there is nothing secret in the block |
+| `salt` | The category, so one key can address several |
+| `seq` | The sequence of the record this document was generated beside |
+| `magnet` | Assembled from the above, ready to paste into a style's URL fragment |
+
+`magnet` is built rather than left to the consumer because every node can build
+it — it contains only the public half — so a fleet behind a balancer hands out
+one identical string and none of them can publish. A BEP 46 client resolves it
+and follows updates rather than pinning to the build this document happened to
+describe.
+
+See [a fragment that survives a rebuild](#a-fragment-that-survives-a-rebuild)
+above for how it reaches a style, and
+[security.md](security.md#the-publisher-key-is-not-a-credential) for why the
+private half lives on exactly one machine.
 
 ## Health checks
 
@@ -564,6 +589,13 @@ first tile can be seconds away.
 **Raster archives get the raster.** There is nothing to inspect in an image, so
 the panel says so and the map is for checking coverage.
 
+**What it has actually served** is on the archive's detail, as `served`, and
+across the node at `GET /api/stats` — requests, bytes, a breakdown by zoom and
+status, and which client addresses asked. Worth reading beside `reading`: an
+archive being read through the swarm while serving thousands of tiles is a
+different situation from one doing neither. See the
+[README](../README.md#seeing-what-a-node-is-actually-serving).
+
 No symbol layers are drawn and no glyphs are configured, deliberately: an
 archive carries tiles, not fonts, and a preview that needed a font server to
 render would not be a preview of the archive.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.13.0",
+  "version": "0.14.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -14,6 +14,7 @@ import {
   isPublicSurface,
 } from './auth.js';
 import { normalizeCategories } from './catalog.js';
+import { mutableMagnet } from './mutable.js';
 import { guessKind } from './library.js';
 import { QBittorrentEngine } from './engines/qbittorrent.js';
 import { RESTART_REQUIRED, redactConfig, saveConfig } from './config.js';
@@ -60,6 +61,25 @@ function route(handler) {
  * @param {object} deps.config - Resolved configuration.
  * @returns {import('express').Express} - The configured app.
  */
+/**
+ * A TileJSON URL carrying a magnet in its fragment.
+ * @param {string} category - Which category.
+ * @param {object} newest - Its newest entry.
+ * @param {string} base - Public base URL.
+ * @returns {string} - The URL a style should use.
+ */
+function styleUrlFor(category, newest, base) {
+  const url = `${base}/latest/${category}/tiles.json`;
+  const magnet = newest?.mutable?.publicKey
+    ? mutableMagnet(newest.mutable.publicKey, {
+        salt: newest.mutable.salt ?? category,
+        name: newest.name,
+        webSeeds: newest.webSeeds,
+      })
+    : newest?.magnet;
+  return magnet ? `${url}#${magnet}` : url;
+}
+
 export function createApp({
   library,
   catalog,
@@ -1703,6 +1723,18 @@ export function createApp({
             servable,
             endpoints: {
               tileJson: servable ? `${base}/latest/${category}/tiles.json` : null,
+              // The same URL with a magnet in the fragment, which is what a
+              // style should carry. A fragment is never sent in a request, so
+              // an ordinary client fetches the TileJSON and ignores it, while
+              // a swarm-aware one has the magnet before it makes any call --
+              // and can therefore still start when this server cannot answer.
+              //
+              // The mutable magnet where there is one, because a category is
+              // precisely where an infohash goes stale: it names the category
+              // and resolves the current build over the DHT. Otherwise the
+              // newest build's own magnet, which pins that build but still
+              // beats a blank map when the fallback is needed at all.
+              styleUrl: servable ? styleUrlFor(category, newest, base) : null,
               torrent: `${base}/latest/${category}/archive.torrent`,
               magnet: `${base}/latest/${category}/magnet`,
               feed: `${base}/feed/${category}.xml`,

package/src/engines/composite.js

@@ -36,6 +36,8 @@ export class CompositeEngine {
   #timer;
   /** Infohashes handed to secondaries, so they are not handed over twice. */
   #shared = new Set();
+  /** Serialises background hand-overs, so they never run all at once. */
+  #shareChain = Promise.resolve();
   /** Set once a stop has begun, so late timers do nothing. */
   #stopping = false;
 
@@ -133,12 +135,54 @@ export class CompositeEngine {
     const infoHash = await this.#primary.add(request);
 
     if (this.#shareable(request) && (await this.#primaryHasItAll(infoHash))) {
-      await this.#shareOne(infoHash, request);
+      // Queued rather than awaited. Handing an archive to a seeding client it
+      // has not seen before makes it hash every byte first, which is minutes
+      // for tens of gigabytes -- so awaiting it here put that cost inside the
+      // caller. On startup, where the library is restored one archive at a
+      // time, that meant a node sat silent for a quarter of an hour before it
+      // would listen, doing work the periodic sweep exists to do anyway.
+      //
+      // Serialised through one chain rather than fired off freely: seventeen
+      // archives hashing at once on a spinning disk is slower than seventeen
+      // in turn, and far harder to reason about.
+      this.#queueShare(infoHash, request);
     }
 
     return infoHash;
   }
 
+  /**
+   * Resolves once every queued hand-over has finished.
+   *
+   * Nothing in normal operation waits for this — that is the point of the
+   * queue — but a caller that wants to observe the result, a test above all,
+   * needs somewhere to wait rather than a guess about microtask order.
+   * @returns {Promise<void>} - Resolves when the queue is empty.
+   */
+  async whenShared() {
+    // Awaited twice: the first await settles what is queued now, and anything
+    // that queued more while it ran is picked up by the second.
+    await this.#shareChain;
+    await this.#shareChain;
+  }
+
+  /**
+   * Runs a share after any already queued, without making the caller wait.
+   * @param {string} infoHash - The archive.
+   * @param {object} request - The original add request.
+   * @returns {void}
+   */
+  #queueShare(infoHash, request) {
+    this.#shareChain = this.#shareChain
+      .then(() => (this.#stopping ? undefined : this.#shareOne(infoHash, request)))
+      // #shareOne already swallows an engine refusing the archive; this is the
+      // last resort, so one failure cannot break the chain for everything
+      // queued behind it.
+      .catch((error) =>
+        console.warn(`[engine] could not hand over ${infoHash}: ${error.message}`),
+      );
+  }
+
   /**
    * Whether the primary actually holds the whole archive.
    *

package/src/library.js

@@ -1463,9 +1463,41 @@ export class Library {
    */
   async restore() {
     const entries = this.#catalog.list();
-    let restored = 0;
-    let failed = 0;
+    const tally = { restored: 0, failed: 0 };
+
+    // Restoring a large library is slow, and until now it said nothing until
+    // it had finished -- so a node that was working and a node that was stuck
+    // looked identical for as long as it took, which on a real library was
+    // long enough to reach for a debugger. Reported on a timer rather than per
+    // archive, so a small library stays quiet and a large one stops being a
+    // mystery.
+    const startedAt = Date.now();
+    const progress = setInterval(() => {
+      const seconds = Math.round((Date.now() - startedAt) / 1000);
+      console.log(
+        `[restore] ${tally.restored + tally.failed} of ${entries.length} ` +
+          `after ${seconds}s`,
+      );
+    }, 15_000);
+    progress.unref?.();
 
+    try {
+      return await this.#restoreEach(entries, tally);
+    } finally {
+      clearInterval(progress);
+    }
+  }
+
+  /**
+   * The restore loop itself.
+   *
+   * Counts into the caller's tally rather than its own, so the progress timer
+   * above has something to read while this is still running.
+   * @param {object[]} entries - Catalog entries, newest first.
+   * @param {{restored: number, failed: number}} tally - Mutated as it goes.
+   * @returns {Promise<{restored: number, failed: number}>} - That tally.
+   */
+  async #restoreEach(entries, tally) {
     for (const entry of entries) {
       // An engine that cannot open its port will fail every one of these, each
       // after its own timeout. Stopping at the first is the difference between
@@ -1473,13 +1505,13 @@ export class Library {
       // minutes per archive.
       if (this.#engine.fatalError) {
         console.error(`[restore] stopping: ${this.#engine.fatalError.message}`);
-        failed += entries.length - restored;
+        tally.failed += entries.length - tally.restored;
         break;
       }
 
       try {
         if (!entry.torrentPath && !entry.magnet) {
-          failed++;
+          tally.failed++;
           continue;
         }
 
@@ -1495,7 +1527,7 @@ export class Library {
                 `usable (${error.code ?? error.message}), so it cannot start. ` +
                 'Move it with Set location, or make that path reachable again.',
             );
-            failed++;
+            tally.failed++;
             continue;
           }
         }
@@ -1505,14 +1537,14 @@ export class Library {
         // own add and skip that, which is why an archive that could not find a
         // peer stayed unable to find one across every restart.
         await this.#readd(entry);
-        restored++;
+        tally.restored++;
       } catch (error) {
-        failed++;
+        tally.failed++;
         console.error(`[restore] ${entry.name}: ${error.message}`);
       }
     }
 
-    return { restored, failed };
+    return tally;
   }
 
   /**

package/src/web/index.html

@@ -2356,6 +2356,7 @@
             const newest = entry.newest;
             const rows = [
               ['TileJSON', entry.endpoints.tileJson],
+              ['For a style', entry.endpoints.styleUrl],
               ['Torrent', entry.endpoints.torrent],
               ['Magnet', entry.endpoints.magnet],
               ['Feed', entry.endpoints.feed],
@@ -2384,7 +2385,20 @@
                             <td style="width:8rem">${label}</td>
                             <td>${
                               url
-                                ? `<code>${escapeHtml(url)}</code>`
+                                ? `<code>${escapeHtml(
+                                    url.length > 96 ? `${url.slice(0, 96)}…` : url,
+                                  )}</code>${
+                                    label === 'For a style'
+                                      ? `<div class="sub">the magnet rides in the fragment, which is
+                                         never sent to the server — an ordinary client fetches the
+                                         TileJSON and ignores it, a swarm-aware one reads it before
+                                         making any call${
+                                           url.includes('xs=urn:btpk')
+                                             ? ' and follows the category rather than this build'
+                                             : ''
+                                         }</div>`
+                                      : ''
+                                  }`
                                 : '<span class="sub">not a PMTiles archive, so there is no tile endpoint</span>'
                             }</td>
                             <td style="width:11rem">${
@@ -2406,6 +2420,7 @@
           const [index, row] = button.dataset.copy.split('-').map(Number);
           const url = [
             categories[index].endpoints.tileJson,
+            categories[index].endpoints.styleUrl,
             categories[index].endpoints.torrent,
             categories[index].endpoints.magnet,
             categories[index].endpoints.feed,