pmtiles-swarm 0.12.0 → 0.13.0

package/CHANGELOG.md

@@ -7,6 +7,40 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.13.0
+### ✨ Features and improvements
+- **The publisher's DHT socket is bound explicitly, on a configurable port.** It was left to bind
+  implicitly on its first send, which works but takes an unpredictable ephemeral port and reports
+  nothing — so there was no way to forward it, and no way to tell which one it had. `mutable.dhtPort`
+  now sets it (`0`, ephemeral, by default) and the port is logged at startup.
+
+  Publishing needs no forward either way: a put is outbound, and the replies come back on the same
+  socket the way any UDP client's do. Pinning and forwarding one makes this a *reachable* DHT node
+  instead, which earns a better routing table and contributes back — worth having on a node that
+  runs continuously.
+
+  It must not collide with an engine's port. Each seeding engine runs a DHT of its own, so a node
+  with both has three UDP participants and only this one is placed here; two sockets cannot hold
+  one port and the node would fail to start.
+
+### 📚 Documentation
+- **[docs/running-as-a-service.md](docs/running-as-a-service.md) covers the publisher key**:
+  generating it as the service account so ownership is right, `chmod 400` because nothing ever
+  writes it back, and why it must be backed up off the machine — losing it breaks every style
+  pointing at that public key, permanently, with no reissue. Also what happens under HA config
+  sync: the configuration replicates to the standby and the key does not, so the standby logs
+  `not publishing: ENOENT` and serves on, which is the intended outcome rather than a fault. And
+  how to confirm it works, including that `nodes: 0` in the log means nobody stored the record
+  however healthy the rest of the line looks.
+- **The ports table lists the DHT port**, which had only been described in the publisher section —
+  not where anyone looks when deciding what to forward.
+- **Why publishing does not reuse an engine's DHT** is recorded in
+  [src/publisher.js](src/publisher.js), since it will be asked again. libtorrent's is unreachable:
+  the 2.x Python bindings expose neither `dht_put_item` nor `dht_get_item`, though the alerts are
+  bound, so the C++ side supports BEP 44 and there is simply no method to call. WebTorrent's *is*
+  `bittorrent-dht` and could be reused to save a socket; that is a deliberate choice rather than an
+  oversight, taken to keep one code path that behaves the same whichever engine is configured.
+
 ## 0.12.0
 ### ✨ Features and improvements
 - **A category can now be addressed without a server at all.** A node that builds can publish a

package/docs/running-as-a-service.md

@@ -354,6 +354,116 @@ What the running service actually has:
 systemctl show -p ReadWritePaths -p UMask -p SupplementaryGroups pmtiles-swarm
 ```
 
+## The publisher key
+
+Only needed if this node publishes BEP 46 records — the signed DHT entries that
+let a style point at a category rather than at a build that goes stale. Skip
+this section entirely if it does not build archives.
+
+**Generate it as the service account**, so the file is owned by the user that
+has to read it:
+
+```sh
+sudo -u pmtiles-swarm -H /var/lib/pmtiles-swarm/node_modules/.bin/pmtiles-swarm   publisher-key > /etc/pmtiles-swarm/publisher.pem
+sudo chown pmtiles-swarm:pmtiles-swarm /etc/pmtiles-swarm/publisher.pem
+sudo chmod 400 /etc/pmtiles-swarm/publisher.pem
+```
+
+The PEM goes to stdout and the public key to stderr, so the redirect above
+captures only the key material and you still see the public half on the
+terminal. Write that public key down — it is what subscribers point at, and it
+is the one part you will want later.
+
+`400` rather than `600`: the service only ever reads this. Nothing in the
+product writes it back, so removing write permission costs nothing and means a
+compromised process cannot quietly replace it.
+
+Then in `/etc/pmtiles-swarm/swarm.config.json`:
+
+```json
+{
+  "mutable": {
+    "publish": true,
+    "keyPath": "/etc/pmtiles-swarm/publisher.pem"
+  }
+}
+```
+
+`/etc/pmtiles-swarm` is already in `ReadWritePaths` because the console rewrites
+the configuration there, so the unit needs no change.
+
+### Back it up, off this machine
+
+Losing this file breaks **every style pointing at that public key, permanently**
+— there is no recovery, no reissue, and no way to prove to a subscriber that a
+new key is you. It is not like an API key you can rotate.
+
+Back it up somewhere that is not this disk, and treat the backup as seriously as
+the original: whoever holds it can publish a signed record telling your
+subscribers that any archive is the current build, and clients will believe it
+because the signature checks out.
+
+### Exactly one publisher
+
+Two nodes publishing under one key fight over the sequence number, each
+overwriting the other's claim about what is current. So the PEM belongs on the
+build node and nowhere else.
+
+**This matters if you run HA config sync.** The configuration will replicate to
+the standby, `publish: true` and all — but the PEM will not, because you are
+not going to copy it. The standby then logs
+
+```
+[mutable] not publishing: ENOENT: no such file or directory
+```
+
+on every start and carries on serving normally. That is the intended outcome
+rather than a fault: the config syncing is harmless, and the key not syncing is
+the point. If the noise bothers you, set `mutable.publish` to `false` in the
+standby's config after the sync.
+
+### It does not need a port forwarded
+
+The DHT socket is separate from the seeding engine's, and by default takes an
+ephemeral UDP port (`mutable.dhtPort: 0`). That is enough to publish: a put is
+outbound — find nodes, then send — and the replies come back on the same
+socket the way any UDP client's do, which NAT handles without help.
+
+Setting a fixed port and forwarding it makes this a *reachable* DHT node, which
+means better lookups and contributing back to the network. Worth doing if this
+node is long-lived, but nothing here requires it.
+
+**Do not reuse the libtorrent engine's port.** That engine runs a DHT of its own
+on `libtorrent.listen`, and two sockets cannot hold one port — the node would
+fail to start.
+
+The port in use is reported at startup:
+
+```
+[mutable] DHT on UDP 63213
+```
+
+### Confirming it works
+
+```sh
+journalctl -u pmtiles-swarm | grep mutable
+```
+
+A healthy publisher says which categories it is announcing and under which key
+at startup, then one line per category whenever a build moves:
+
+```
+[mutable] publishing 3 categories as 7680dc95248eb807… every 30m
+[mutable] openmaptiles -> 4813a0e68e4b (seq 1786108931, 8 nodes)
+```
+
+`nodes` is how many DHT peers stored the record. **Zero means nobody did**, and
+the record does not exist however healthy the log looks otherwise — check that
+UDP is not blocked and that the DHT is reachable.
+
+The first publish waits about fifteen seconds after start, because a put into a
+DHT that has not finished bootstrapping reaches nobody.
+
 ## The sidecar
 
 The libtorrent engine runs Python as a child process, so the service user needs
@@ -373,16 +483,33 @@ rather than exiting. Name the interpreter explicitly when the service user's
 
 ## Ports
 
-Four listeners, and only the peer ports want a firewall rule. See
+Five listeners, and only the peer ports want a firewall rule. See
 [ports and reachability](engines.md#ports-and-reachability) for the detail.
 
 | | |
 | --- | --- |
 | `libtorrent.listen` — 6881, TCP and UDP | forward it |
 | `webtorrent.clientOptions.torrentPort` — pin it, or it changes every start | forward it |
+| `mutable.dhtPort` — UDP, ephemeral by default | optional; see below |
 | `port` — 8090 | your proxy or CDN |
 | `adminPort` — 8091, bound to `127.0.0.1` | nothing; that is the point |
 
+`mutable.dhtPort` is the odd one. Publishing works without any forward, because
+a put is outbound and the replies come back on the same socket the way any UDP
+client's do. Pin it and forward it only if you want this to be a *reachable*
+DHT node — which earns a better routing table and contributes back, and is
+worth having on a node that runs continuously.
+
+Pin it to something free: **not** `libtorrent.listen`, which is a DHT of its
+own, and not `torrentPort`. `6883` sits clear of both.
+
+```json
+{ "mutable": { "dhtPort": 6883 } }
+```
+
+Note that each engine runs its own DHT as well, so a node with both engines has
+three UDP participants. Only this one is yours to place.
+
 ## Updating
 
 ```sh

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.12.0",
+  "version": "0.13.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/config.js

@@ -425,6 +425,19 @@ const DEFAULTS = {
     keyPath: undefined,
     /** How often to republish, in seconds. */
     republishSeconds: 1800,
+    /**
+     * UDP port for this node's DHT socket. 0 takes an ephemeral one.
+     *
+     * Publishing does not need a forwarded port: a put is outbound, and the
+     * replies come back on the same socket the way any UDP client's do, which
+     * NAT handles. Setting a fixed port and forwarding it makes this a
+     * reachable DHT node, which gives better lookups and contributes back —
+     * but nothing here requires it.
+     *
+     * Do not reuse the libtorrent engine's port. That engine runs a DHT of its
+     * own, and two sockets cannot hold one port.
+     */
+    dhtPort: 0,
   },
   /**
    * What this node has served, at `GET /api/stats`.

package/src/index.js

@@ -273,6 +273,15 @@ PMTILES_SWARM_PUBLIC_URL
       ]);
       const pem = await fs.readFile(config.mutable.keyPath, 'utf8');
       const dht = new DHT();
+      // Bound explicitly rather than left to bind implicitly on first send,
+      // so the port is predictable and can be forwarded if you want this to
+      // be a reachable DHT node. 0 takes an ephemeral one, which is all
+      // publishing needs.
+      await new Promise((resolve, reject) => {
+        dht.once('error', reject);
+        dht.listen(config.mutable.dhtPort ?? 0, resolve);
+      });
+      console.log(`[mutable] DHT on UDP ${dht.address().port}`);
       publisher = new MutablePublisher({
         catalog,
         dht,

package/src/publisher.js

@@ -21,6 +21,30 @@
  * so republishing on a timer is not an optimisation here — it is the feature.
  */
 
+/*
+ * Why this uses bittorrent-dht rather than an engine's own DHT.
+ *
+ * Both seeding engines run a DHT already, so a third one looks redundant.
+ * It is not, for two different reasons:
+ *
+ *   libtorrent   - its DHT lives in the Python sidecar, and the 2.x Python
+ *                  bindings do not expose dht_put_item or dht_get_item at all.
+ *                  The alerts are bound (dht_mutable_item_alert, dht_put_alert)
+ *                  so the C++ side supports BEP 44, but there is no method to
+ *                  start one. Checked against 2.0.13; worth re-checking if a
+ *                  later binding adds them, because a sidecar op would then be
+ *                  the tidier answer for a libtorrent-only node.
+ *
+ *   webtorrent   - its DHT *is* bittorrent-dht, reachable as `client.dht`, so
+ *                  reusing it is possible and would save a socket. Not done,
+ *                  to keep one code path that behaves the same whichever
+ *                  engine is configured.
+ *
+ * The dependency itself costs nothing: webtorrent already depends on the same
+ * version, and npm dedupes them to one install. Declaring it directly only
+ * removes the reliance on a transitive.
+ */
+
 import { mutableMagnet, publishInfoHash } from './mutable.js';
 
 /** Republish well inside the ~2h a DHT keeps an item. */