pmtiles-swarm 0.3.0 → 0.3.1

package/CHANGELOG.md

@@ -7,6 +7,54 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.3.1
+
+Depends on pmtiles-torrent 0.3.0, which is what carries the resume-data fix below to an installed
+copy — 0.3.0 of this package shipped against a sidecar that could not find its own resume data.
+
+### ✨ Features and improvements
+- **The sample configuration ships with the package**, so an installed copy has one to copy from
+  rather than only the repository — which is the one place someone installing from npm has not got.
+- **The `allowScripts` warning from npm 11.17 is explained.** npm is moving dependency install
+  scripts behind an allowlist; today it warns and still runs them, so an install that prints it is
+  fine. One of those scripts matters — `node-datachannel` downloads the WebRTC binary WebTorrent
+  needs, which is not in the published tarball — so the documentation gives a one-line check that
+  it landed, and says to leave `--strict-allow-scripts` alone, since in testing it blocked approved
+  scripts as well as unapproved ones.
+- **The service documentation installs into the account's own directory, not globally.**
+  `sudo npm install -g` fails on some machines: a WebTorrent dependency runs `npx only-allow pnpm`
+  as a preinstall step, and under `sudo` that npx cannot write root's cache. Installing as the
+  service account avoids root's cache entirely, keeps the version pinned per service, and makes
+  upgrading one command. `--ignore-scripts` is explicitly not the answer — `node-datachannel`
+  fetches its prebuilt binary in an install script, and without it WebTorrent cannot do WebRTC,
+  which is the only reason to run it alongside libtorrent.
+- **Running as a systemd service is documented**, with a unit file, and now with the account setup:
+  creating a dedicated `pmtiles-swarm` system user and group, the two directories, and where Node
+  and the package go. Both directories have to be writable by the service, including the one under
+  `/etc` — minting a token or pressing Save rewrites the configuration, so a root-owned file the
+  service can only read loses tokens on restart. Two things in it are not
+  preferences: `Restart=always` is required rather than optional, because the console's *Save &
+  Restart* detects the supervisor and exits 0 expecting to be brought back — under
+  `Restart=on-failure` the first use of it stops the node and leaves a unit reporting success. And
+  the `ExecStop=/bin/kill -15 $MAINPID` line commonly copied between units should be omitted, since
+  systemd already sends SIGTERM and the node installs its handlers before it begins work.
+
+### 🐞 Bug fixes
+- **A restart no longer re-hashes every archive.** Resume data was being written and never found —
+  the lookup used an infohash no caller supplied, and `add_torrent_params.info_hashes` reads as
+  forty zeros for a torrent added from a `.torrent` file, so keying it off the torrent alone would
+  not have helped either. Fixed in pmtiles-torrent 0.3.0; on a 512 MiB archive the difference
+  measured 1.21s with a full re-hash against 0.02s with none, and it scales with the archive.
+  Resume data is also written every `resumeSaveIntervalSeconds` (five minutes by default) rather
+  than only at shutdown, so a kill or a power cut costs the last few minutes instead of everything.
+- **Every path in the configuration resolves against the configuration file.** `dataDir`,
+  `savePath`, `cacheSavePath` and watched folders already did; `locations[].path` and
+  `libtorrent.resumeDir` were left relative, which means relative to the working directory. Started
+  by hand from the repository the two agree, so it never showed — but a service does not run from
+  the directory its config lives in, and under systemd the working directory defaults to `/`, so
+  `./data/resume` became `/data/resume`: somewhere the unit almost certainly cannot write, for a
+  reason nothing in the config hints at.
+
 ## 0.3.0
 ### ✨ Features and improvements
 - **The peers column distinguishes who is connected from what the swarm holds.** `0 / 2` on a

package/README.md

@@ -31,6 +31,8 @@ node src/index.js --config swarm.config.json
 - **[docs/security.md](docs/security.md)** — what is public, what is guarded, named tokens with
   roles, console sign-in, and why an unauthenticated node refuses to listen on a reachable
   address.
+- **[docs/running-as-a-service.md](docs/running-as-a-service.md)** — a systemd unit, why
+  `Restart=always` is required rather than optional, and which ports want a firewall rule.
 - **[docs/architecture-diagram.md](docs/architecture-diagram.md)** — how a publishing node, a
   serving tier, the swarm and both kinds of client fit together.
 
@@ -131,12 +133,22 @@ poll a stranger's server before you have read it.
 `swarm.config.json` itself is gitignored — along with every other
 `swarm.config*.json` and `*.bak` — because it holds an API key.
 
+**`adminPort` is unset by default**, and one listener then serves everything.
+Setting it splits the surface in two: `port` keeps the tiles, TileJSON,
+`.torrent` files and feeds — everything a stranger or a peer is meant to reach
+— while `adminPort` takes the console and the rest of `/api/`. Bind that to
+`127.0.0.1` and the part that can rewrite this file is unreachable rather than
+merely guarded. See [Two ports](#two-ports).
+
 ```json
 {
   "port": 8090,
+  "adminPort": 8091,
+  "adminHost": "127.0.0.1",
   "dataDir": "./data",
-  "engine": "qbittorrent",
-  "qbittorrent": { "url": "http://127.0.0.1:8080", "username": "admin", "password": "…" },
+  "engine": "libtorrent",
+  "secondaryEngines": ["webtorrent"],
+  "libtorrent": { "python": "python3", "listen": "0.0.0.0:6881" },
   "savePath": "./data/torrents-data",
   "incompleteSuffix": ".incomplete",
   "pieceLength": 4194304,
@@ -147,7 +159,7 @@ poll a stranger's server before you have read it.
   "watch": [
     {
       "path": "/mnt/maps/incoming",
-      "category": "basemaps",
+      "categories": ["basemaps"],
       "publishDir": "/var/www/pmtiles",
       "webSeedBase": "https://maps.example.org/files"
     }

package/docs/running-as-a-service.md

@@ -0,0 +1,268 @@
+# Running as a systemd service
+
+Setting up the account, then the unit. Two lines in that unit are not optional,
+and one line most people copy from elsewhere should be deleted — the rest is
+ordinary.
+
+## An account of its own
+
+A system account: no password, no login, and nothing on the machine belongs to
+it except the archives.
+
+```sh
+sudo groupadd --system pmtiles-swarm
+sudo useradd --system --gid pmtiles-swarm \
+  --home-dir /var/lib/pmtiles-swarm --create-home \
+  --shell /usr/sbin/nologin \
+  --comment "pmtiles-swarm service" pmtiles-swarm
+```
+
+`--system` keeps it out of the human UID range. Home is the data directory, so
+npm and Python per-user state land with the archives.
+
+Then the two directories it needs:
+
+```sh
+sudo install -d -o pmtiles-swarm -g pmtiles-swarm -m 0750 /var/lib/pmtiles-swarm
+sudo install -d -o pmtiles-swarm -g pmtiles-swarm -m 0750 /etc/pmtiles-swarm
+```
+
+**Both have to be writable by the service, including the one under `/etc`** —
+minting a token or pressing Save in the console rewrites `swarm.config.json`.
+Root-owned, tokens vanish on restart.
+
+The file itself holds an API key, so nobody else needs to read it:
+
+```sh
+SAMPLE=/var/lib/pmtiles-swarm/node_modules/pmtiles-swarm/swarm.config.json.sample
+sudo install -o pmtiles-swarm -g pmtiles-swarm -m 0600 "$SAMPLE" \
+  /etc/pmtiles-swarm/swarm.config.json
+sudoedit /etc/pmtiles-swarm/swarm.config.json
+```
+
+Generate the key rather than inventing one:
+
+```sh
+node -e "console.log(require('crypto').randomBytes(32).toString('base64url'))"
+```
+
+## Node, and the package
+
+Node from your distribution or NodeSource — the package needs `^22.13.0 || 24`:
+
+```sh
+curl -fsSL https://deb.nodesource.com/setup_24.x | sudo -E bash -
+sudo apt-get install -y nodejs
+```
+
+Then install into the service account's own directory, as that account:
+
+```sh
+sudo -u pmtiles-swarm -H npm install --prefix /var/lib/pmtiles-swarm pmtiles-swarm
+sudo -u pmtiles-swarm -H /var/lib/pmtiles-swarm/node_modules/.bin/pmtiles-swarm --help
+```
+
+That path is what goes in `ExecStart`. Upgrading is the same command again.
+
+Installing as the account rather than into a root-owned tree also sidesteps a
+failure `sudo npm install -g` can hit: `ip-set`, a dependency of WebTorrent,
+runs `npx only-allow pnpm` before installing, and under `sudo` that npx may not
+be able to write root's cache.
+
+```
+npm error command sh -c npx only-allow pnpm
+npm error enoent Could not read package.json: ENOENT:
+  open '/root/.npm/_npx/0b83cd9ca5e1325c/package.json'
+```
+
+If you would rather install as root and hand the tree over — the usual pattern
+— give npm a cache it can write and chown afterwards:
+
+```sh
+sudo env npm_config_cache=/var/cache/npm \
+  npm install --prefix /var/lib/pmtiles-swarm pmtiles-swarm
+sudo chown -R pmtiles-swarm:pmtiles-swarm /var/lib/pmtiles-swarm
+```
+
+**nvm does not affect either.** `sudo` resets `PATH` to `secure_path`, so
+`sudo npm` is `/usr/bin/npm` whatever `nvm use` last selected. Only the
+absolute path in `ExecStart` decides what the service runs.
+
+### The allowScripts warning
+
+npm 11.17 and newer print this, and it is not an error:
+
+```
+npm warn allow-scripts 5 packages have install scripts not yet covered by allowScripts:
+npm warn allow-scripts   node-datachannel@0.32.3 (install: prebuild-install -r napi || …)
+npm warn allow-scripts   ip-set@3.0.0 (preinstall: npx only-allow pnpm)
+```
+
+npm is moving dependency install scripts behind an allowlist. Today it warns
+and still runs them. One of those scripts matters: `node-datachannel` downloads
+the prebuilt binary WebTorrent needs for WebRTC, and it is not in the published
+tarball. Check it landed:
+
+```sh
+sudo -u pmtiles-swarm -H node -e \
+  "import('/var/lib/pmtiles-swarm/node_modules/webtorrent').then(() => console.log('ok'))"
+```
+
+`npm approve-scripts node-datachannel` records the approval in `package.json`
+and silences the warning for that package:
+
+```json
+{ "allowScripts": { "node-datachannel@0.32.3": true } }
+```
+
+Leave `--strict-allow-scripts` alone for now. It blocks unapproved scripts
+outright, and in testing it also blocked approved ones — the resulting install
+has no WebRTC binary and WebTorrent will not load.
+
+The libtorrent engine also needs `python3` with the bindings, checked as the
+service account:
+
+```sh
+sudo apt-get install -y python3-libtorrent
+sudo -u pmtiles-swarm python3 -c "import libtorrent; print(libtorrent.__version__)"
+```
+
+## The unit
+
+```ini
+[Unit]
+Description=pmtiles-swarm
+Documentation=https://github.com/TechIdiots-LLC/pmtiles-swarm
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User=pmtiles-swarm
+Group=pmtiles-swarm
+
+WorkingDirectory=/var/lib/pmtiles-swarm
+
+# Absolute: systemd reads no shell profile.
+ExecStart=/var/lib/pmtiles-swarm/node_modules/.bin/pmtiles-swarm \
+  --config /etc/pmtiles-swarm/swarm.config.json
+
+# Required. The console's Save & Restart exits and expects to be brought back;
+# see below.
+Restart=always
+RestartSec=5
+
+# Stopping announces "stopped" to every tracker, releases the data directory
+# lock and cancels downloads in flight. Worst case is about 20 seconds.
+TimeoutStopSec=45
+
+# A seeding node holds a socket per peer, and the tile reader holds file
+# descriptors of its own.
+LimitNOFILE=65535
+
+# The archives and the sidecar are the only things it needs to touch.
+ProtectSystem=strict
+ProtectHome=read-only
+PrivateTmp=true
+NoNewPrivileges=true
+# Both, since the console rewrites the configuration when a token is minted.
+ReadWritePaths=/var/lib/pmtiles-swarm /etc/pmtiles-swarm
+
+[Install]
+WantedBy=multi-user.target
+```
+
+## The two lines that matter
+
+**`Restart=always`, not `on-failure`.** The console's *Save & Restart* applies
+settings a running process cannot take — the port, the data directory, the
+torrent client. Under a supervisor the node does not relaunch itself: it shuts
+down and **exits 0**, expecting to be brought back.
+
+`Restart=on-failure` ignores an exit 0, so the first use of that button would
+stop the node and leave the unit reporting success.
+
+**No `ExecStop=`.** systemd already sends `SIGTERM`, and the node handles it
+from the moment it starts. `ExecStop=/bin/kill -15 $MAINPID` is redundant, and
+becomes wrong if the unit ever uses `KillMode=process` — it would stop the node
+while the Python sidecar kept running and kept the data directory locked.
+
+Leave `KillMode` at its default, so the sidecar goes with its parent.
+
+## Paths
+
+Every path in the configuration resolves **relative to the configuration
+file**, not to the working directory:
+
+```
+/etc/pmtiles-swarm/swarm.config.json     with "dataDir": "./data"
+  -> /etc/pmtiles-swarm/data
+```
+
+That is usually not what you want for a service. Use absolute paths for
+anything that holds data:
+
+```json
+{
+  "dataDir": "/var/lib/pmtiles-swarm",
+  "savePath": "/var/lib/pmtiles-swarm/archives",
+  "libtorrent": { "resumeDir": "/var/lib/pmtiles-swarm/resume" }
+}
+```
+
+`ProtectSystem=strict` makes the whole filesystem read-only apart from what
+`ReadWritePaths` names, so every one of those has to be listed. An archive
+directory on another mount needs its own entry.
+
+## The sidecar
+
+The libtorrent engine runs Python as a child process, so the service user needs
+`python3` with libtorrent importable — not your login shell's Python:
+
+```sh
+sudo -u pmtiles-swarm python3 -c "import libtorrent; print(libtorrent.__version__)"
+```
+
+If that fails, the node still starts and reports the engine as unavailable
+rather than exiting. Name the interpreter explicitly when the service user's
+`PATH` is not what you tested with, which under systemd it usually is not:
+
+```json
+{ "libtorrent": { "python": "/usr/bin/python3" } }
+```
+
+## Ports
+
+Four listeners, and only the peer ports want a firewall rule. See
+[ports and reachability](engines.md#ports-and-reachability) for the detail.
+
+| | |
+| --- | --- |
+| `libtorrent.listen` — 6881, TCP and UDP | forward it |
+| `webtorrent.clientOptions.torrentPort` — pin it, or it changes every start | forward it |
+| `port` — 8090 | your proxy or CDN |
+| `adminPort` — 8091, bound to `127.0.0.1` | nothing; that is the point |
+
+## Checking it
+
+```sh
+sudo systemctl daemon-reload
+sudo systemctl enable --now pmtiles-swarm
+journalctl -u pmtiles-swarm -f
+```
+
+A healthy start says which engine came up, how many archives were handed back
+to it, and which ports it is listening on. Two lines are worth reading for:
+
+* `could not listen on … port` — something else holds it, most often a previous
+  run that has not finished stopping.
+* `data directory is already in use by pid …` — one node per data directory,
+  enforced with a lock file. Under `Restart=always` this usually means the old
+  process outlived `TimeoutStopSec`; raise it rather than removing the lock.
+
+Then check it is actually serving:
+
+```sh
+curl -fsS localhost:8090/feed.xml >/dev/null && echo "public surface ok"
+curl -fsS localhost:8091/api/status | head -c 200
+```

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.3.0",
+  "version": "0.3.1",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",
@@ -10,6 +10,7 @@
   "files": [
     "src",
     "docs",
+    "swarm.config.json.sample",
     "NOTICE.md",
     "CHANGELOG.md"
   ],
@@ -38,7 +39,7 @@
     "maplibre-gl": "^6.2.0",
     "parse-torrent": "^11.0.24",
     "pmtiles": "^4.4.1",
-    "pmtiles-torrent": "^0.2.0",
+    "pmtiles-torrent": "^0.3.0",
     "webtorrent": "^3.0.21"
   },
   "engines": {

package/src/config.js

@@ -570,6 +570,15 @@ const DEFAULTS = {
    * because splicing two builds together produces a torrent for bytes that
    * never existed.
    */
+  /**
+   * How often to write resume data, in seconds. Zero disables it.
+   *
+   * Resume data is what lets a restart skip re-hashing the store — on an
+   * 800 GB archive, the difference between instant and half an hour. A clean
+   * stop always writes it; this is for the stops that are not clean, where
+   * everything since the last write has to be checked again.
+   */
+  resumeSaveIntervalSeconds: 300,
   fetchAttempts: 10,
   /** How long to wait before resuming a download that stopped. */
   fetchRetrySeconds: 5,
@@ -779,6 +788,22 @@ export async function loadConfig(configPath) {
     path: path.resolve(base, entry.path),
   }));
 
+  // Every remaining path, resolved the same way as the rest.
+  //
+  // These two were left relative, which means relative to the working
+  // directory rather than to the file they are written in. Started by hand
+  // from the repository those agree, so it never showed; under systemd the
+  // working directory defaults to `/`, so `./data/resume` becomes
+  // `/data/resume` — somewhere the service almost certainly cannot write, for
+  // a reason nothing in the config hints at.
+  config.locations = (config.locations ?? []).map((entry) => ({
+    ...entry,
+    path: entry.path ? path.resolve(base, entry.path) : entry.path,
+  }));
+  if (config.libtorrent?.resumeDir) {
+    config.libtorrent.resumeDir = path.resolve(base, config.libtorrent.resumeDir);
+  }
+
   return config;
 }
 

package/src/index.js

@@ -321,6 +321,20 @@ PMTILES_SWARM_PUBLIC_URL
   sources.start();
   seeding.start();
   speed.start();
+
+  // Resume data on a timer as well as at shutdown. A clean stop writes it; a
+  // kill, a crash or a power cut does not, and whatever is lost is re-hashed
+  // on the way back up.
+  let resumeTimer;
+  const resumeSeconds = config.resumeSaveIntervalSeconds ?? 300;
+  if (resumeSeconds > 0 && engine.saveResume) {
+    resumeTimer = setInterval(() => {
+      engine
+        .saveResume()
+        .catch((error) => console.warn(`[resume] could not save: ${error.message}`));
+    }, resumeSeconds * 1000);
+    resumeTimer.unref?.();
+  }
   hooks.start();
   completion.start();
 
@@ -352,6 +366,7 @@ PMTILES_SWARM_PUBLIC_URL
   }
 
   stoppers.unshift(
+    { label: 'resume timer', stop: () => resumeTimer && clearInterval(resumeTimer), ms: 500 },
     { label: 'origin checks', stop: () => originTimer && clearInterval(originTimer), ms: 500 },
     { label: 'schedulers', stop: () => {
       sources.stop();

package/swarm.config.json.sample

@@ -0,0 +1,86 @@
+{
+  "port": 8090,
+  "adminPort": 8091,
+  "adminHost": "127.0.0.1",
+  "publicUrl": "https://maps.example.org",
+
+  "dataDir": "./data",
+  "savePath": "./data/torrents-data",
+  "savePathLayout": "infohash",
+  "locations": [
+    { "name": "default", "path": "./data/torrents-data" },
+    { "name": "bulk storage", "path": "/EDIT-ME/bulk/archives" }
+  ],
+
+  "engine": "libtorrent",
+  "secondaryEngines": ["webtorrent"],
+  "libtorrent": {
+    "python": "python3",
+    "resumeDir": "./data/resume",
+    "listen": "0.0.0.0:6881",
+    "upnp": true,
+    "natpmp": true
+  },
+  "webtorrent": {
+    "clientOptions": { "torrentPort": 6882 }
+  },
+
+  "torrentFormat": "hybrid",
+  "pieceLength": 4194304,
+  "maxConnections": 100,
+  "trackers": [
+    "udp://tracker.opentrackr.org:1337/announce",
+    "udp://tracker.torrent.eu.org:451/announce",
+    "wss://tracker.openwebtorrent.com"
+  ],
+
+  "speed": {
+    "uploadLimit": 0,
+    "downloadLimit": 0,
+    "alternative": { "uploadLimit": 2097152, "downloadLimit": 20971520 },
+    "schedule": { "enabled": false, "from": "08:00", "to": "23:00", "days": "everyday" }
+  },
+  "seeding": { "ratio": null, "minutes": null, "then": "stop" },
+
+  "feedTitle": "Example map archives",
+  "feedCopyright": "© OpenStreetMap contributors",
+  "feedMaxItems": 50,
+
+  "watch": [
+    {
+      "path": "/EDIT-ME/maps/generated",
+      "categories": ["basemaps"],
+      "webSeedBase": "https://maps.example.org/files",
+      "stabilitySeconds": 30,
+      "pollSeconds": 0
+    }
+  ],
+
+  "sources": [
+    {
+      "name": "daily-planet",
+      "url": "https://build.example.org/{YYYYMMDD}.pmtiles",
+      "filename": "planet-{YYYYMMDD}.pmtiles",
+      "categories": ["planet"],
+      "offsetDays": -1,
+      "lookbackDays": 0,
+      "at": "10:00",
+      "keep": 2,
+      "webSeed": true,
+      "latestLink": "planet-latest.pmtiles"
+    }
+  ],
+
+  "subscriptions": [
+    { "url": "https://peer.example.org/api/catalog", "protocol": "api", "mode": "cache" }
+  ],
+
+  "tiles": { "sparse": null },
+
+  "auth": {
+    "apiKey": "REPLACE-WITH-A-LONG-RANDOM-STRING",
+    "username": "admin",
+    "password": "REPLACE-ME",
+    "tokens": []
+  }
+}