pmtiles-swarm 0.92.0 → 0.95.0

package/CHANGELOG.md

@@ -2,11 +2,94 @@
 
 ## master
 ### ✨ Features and improvements
-- _...Add new stuff here..._
+- **`init --systemd` regenerates the unit without touching the configuration.** The unit is derived
+  from the configuration and from how many archives the library holds, and both move — a watched
+  folder added later belongs in `ReadWritePaths`, and a grown library needs longer to write its
+  resume data than the unit allows. Running it again used to be refused, which sent people to
+  `--force`, which replaces `swarm.config.json` outright: tokens, stacks, feeds and all, to
+  regenerate a file beside it. It now reads the configuration, writes only the unit, and says so —
+  along with the `diff` to run before replacing what is installed, since `ReadWritePaths` is derived
+  and a path added to the installed unit by hand will not be in the new one.
 
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.95.0
+### ✨ Features and improvements
+- **The defaults now assume a library of hundreds, not a handful.** `tiles.maxOpenArchives` goes
+  from 16 to 128 and `tiles.directoryCacheEntries` from 200 to 2000. Sixteen was set for the most
+  expensive kind of open archive and then applied to every kind, which is wrong for the node this is
+  increasingly used to build: a stack assembled from a provider's file index names several hundred
+  sources, and a bake walks every one of them. At sixteen such a run spent most of its time
+  reopening archives it had just closed, and each reopen re-reads a header and a directory.
+
+  So the limit is now three limits, because the handles cost different things. A complete archive is
+  a file descriptor and the unit allows 65535 of them. A **cache-mode** archive carries a piece cache
+  sized from the torrent's piece length — at 16 MiB pieces a hundred of them is gigabytes — so
+  `tiles.maxOpenSwarmArchives` keeps the old ceiling of 16 and is counted separately. A **URL or
+  bucket** archive holds an HTTP reader and nothing else, and costs a network round trip to reopen,
+  so `tiles.maxOpenRemoteArchives` is 64. A node serving only cache-mode archives behaves as before.
+
+- **The unit's stop timeout is derived from the library it was written for.** Stopping writes resume
+  data for every archive and the node allows two seconds apiece, so a fixed five minutes covered a
+  library of 145 and no more — silently, because outgrowing it produces no error, just a library
+  that comes back at 0% and re-hashes for hours. `init --systemd` now counts the catalog and sizes
+  `TimeoutStopSec` to fit, and a node whose library has outgrown a default unit says so at startup.
+
+
+### 🐞 Bug fixes
+- **A sidecar left running by a previous start is now killed rather than fought.** This is the
+  restart that has to be done two or three times before it takes. The sidecar exits when its pipe
+  closes, which covers an orderly stop — but not one in the middle of hashing, since libtorrent does
+  not hand control back until the check finishes, and not a node killed outright. What is left holds
+  the listen port and the resume directory, the next start fails against it, and since a failed
+  start leaves its own sidecar they accumulate.
+
+  Stopping now insists: the shutdown request goes first so resume data is still saved, and a sidecar
+  that has not gone within a few seconds is killed rather than left behind. And a start reaps what a
+  previous run left before spawning anything — by pid, but only where `/proc` confirms that pid is
+  really a sidecar, since pids are reused and killing whatever inherited one would be worse than the
+  problem.
+
+## 0.94.0
+### 🐞 Bug fixes
+- **Saving settings rewrote a proxy list nobody had touched.** A trusted-proxy list may be stored as
+  a string — `"loopback, 10.0.0.0/8"` is what the documentation shows — and the box rendered that as
+  one line while reading it back as an array. The two never compared equal, so the field was sent on
+  every Save whether or not anybody had looked at it, and before the previous release that rewrite
+  was the thing that stopped the node from starting. Opening the settings page and pressing Save was
+  enough to do it.
+
+  The box now shows one entry per line whichever way the config wrote them, and records what a save
+  will read back rather than what the config holds — so an untouched field is untouched.
+
+## 0.93.0
+### 🐞 Bug fixes
+- **A trusted-proxy list typed with commas stopped the node from starting.** The settings field
+  split what was typed on newlines only, so one line reading `172.16.1.2, 172.16.1.3` was saved as
+  an array holding both addresses in one string. Express splits a comma list when it is handed a
+  bare string and never inside an array, so proxy-addr was given `172.16.1.2, 172.16.1.3` as a
+  single address and threw -- while the app was being built, before the listener binds. The node
+  would not start, could not be reached, and could not be corrected from the console that had
+  written the value; on the node that found this it was 155 restarts.
+
+  Three things were wrong and all three are fixed. The field now splits on commas as well as
+  newlines. Every shape the setting can be written in -- a string, an array, commas, spaces,
+  newlines -- is flattened to what Express wants. And an entry that is not an address is ignored
+  and logged rather than thrown: this setting is not worth a node that will not boot, and trusting
+  nobody is the safe end of being wrong about it.
+
+- **A failed restore took the whole node down, console included.** Handing the library back to the
+  engine at startup already tolerates a failure per archive; the call coming apart as a whole was
+  unguarded, and it happens before the listener binds — so under `Restart=always` the result is a
+  crash loop with no console to look at and no way to see why. It is now reported and the node
+  starts anyway, where every archive shows as **not loaded** until it is fixed.
+
+- **Nothing tested that the node starts at all.** Every other test builds the pieces `src/index.js`
+  wires together and never runs the wiring, so an import cycle or a step that throws before the
+  listener binds was a failure only a real start could find. There is now a boot test that runs the
+  entry point the way the service does and asks it for a page.
+
 ## 0.92.0
 ### 🐞 Bug fixes
 - **A stack's TileJSON published the addresses its sources are read from.** A URL source is named by

package/docs/configuration.md

@@ -82,8 +82,17 @@ Takes effect on the next request; no restart. See
 ### `trustProxy`
 
 Takes anything Express accepts: `true`, a hop count, or a subnet list such as
-`"loopback, 10.0.0.0/8"`. Off by default, because trusting these headers from an
-untrusted client lets it claim any protocol or address it likes.
+`"loopback, 10.0.0.0/8"`. A list may be written as one string or as an array,
+and either may separate its entries with commas, spaces or newlines — all four
+shapes mean the same thing here. Off by default, because trusting these headers
+from an untrusted client lets it claim any protocol or address it likes.
+
+An entry that is not an address, a subnet, or one of `loopback`, `linklocal`
+and `uniquelocal` is **ignored and logged**, and a value that cannot be read at
+all leaves the node trusting nobody. Deliberately not fatal: Express compiles
+this the moment it is set, which is before the listener binds, so a value it
+refuses used to be a node that would not start, could not be reached, and could
+not be corrected from the console that wrote it.
 
 Set it when a proxy terminates TLS, or the TileJSON will advertise `http://` tile
 URLs that browsers block as mixed content. Setting `publicUrl` instead sidesteps
@@ -715,21 +724,43 @@ through the swarm, pulling only the pieces a requested tile lives in — which i
 what lets a machine with 10 GiB free serve a 700 GiB planet. See
 [serving-tiles.md](serving-tiles.md).
 
-| setting                       | default  |                                                                  |
-| ----------------------------- | -------- | ---------------------------------------------------------------- |
-| `tiles.maxOpenArchives`       | `16`     | each holds a descriptor or a torrent reader plus its piece cache |
-| `tiles.directoryCacheEntries` | `200`    | header and directory cache, shared across archives               |
-| `tiles.pieceCacheBytes`       | unset    | sized from the torrent's piece length when unset                 |
-| `tiles.hydrateIdleMs`         | unset    | idle time before background hydration resumes                    |
-| `tiles.pieceTimeoutMs`        | `120000` | how long to wait for one piece                                   |
-| `tiles.readyTimeoutMs`        | `60000`  | how long to wait for torrent metadata                            |
-| `tiles.metadataTimeoutMs`     | `120000` | how long a background metadata read may take                     |
-| `tiles.headerTimeoutMs`       | `12000`  | how long a TileJSON request waits for a header                   |
-| `tiles.sparse`                | unset    | `true` for 404 on a missing tile, `false` for 204                |
+| setting                       | default  |                                                              |
+| ----------------------------- | -------- | ------------------------------------------------------------ |
+| `tiles.maxOpenArchives`       | `128`    | complete archives kept open; each is a file descriptor       |
+| `tiles.maxOpenSwarmArchives`  | `16`     | cache-mode readers, counted apart: each holds a piece cache  |
+| `tiles.maxOpenRemoteArchives` | `64`     | readers for a URL or a bucket; cheap to hold, dear to reopen |
+| `tiles.directoryCacheEntries` | `2000`   | header and directory cache, shared across archives           |
+| `tiles.pieceCacheBytes`       | unset    | sized from the torrent's piece length when unset             |
+| `tiles.hydrateIdleMs`         | unset    | idle time before background hydration resumes                |
+| `tiles.pieceTimeoutMs`        | `120000` | how long to wait for one piece                               |
+| `tiles.readyTimeoutMs`        | `60000`  | how long to wait for torrent metadata                        |
+| `tiles.metadataTimeoutMs`     | `120000` | how long a background metadata read may take                 |
+| `tiles.headerTimeoutMs`       | `12000`  | how long a TileJSON request waits for a header               |
+| `tiles.sparse`                | unset    | `true` for 404 on a missing tile, `false` for 204            |
 
 Leave `pieceCacheBytes` unset unless you have a reason. A fixed budget is a trap
 with 16 MiB pieces, since 64 MiB holds only four.
 
+### Three limits, because the handles cost different things
+
+They were one limit, set at sixteen for the most expensive kind and applied to
+every kind. That is wrong for the node this is increasingly used to build: a
+stack assembled from a provider's file index names several hundred sources, and
+a bake walks every one of them. At sixteen such a run spends most of its time
+reopening archives it has just closed, and each reopen re-reads a header and a
+directory.
+
+- **A complete archive** is a file descriptor and its share of the directory
+  cache. The unit allows 65535 descriptors, so hundreds of these are nothing.
+- **A cache-mode archive** carries a piece cache sized from the torrent's piece
+  length. At 16 MiB pieces a hundred of them is gigabytes, which is why this
+  one keeps the old ceiling and is counted separately.
+- **A URL or bucket archive** holds an HTTP reader and the summary it has
+  already read. Cheap to keep, and expensive to reopen: a reopen costs a header
+  and a directory fetch over the network rather than off a disk.
+
+A node serving nothing but cache-mode archives behaves exactly as it did.
+
 `headerTimeoutMs` is shorter than the others on purpose: somebody is waiting on
 it. A cache-mode archive with no web seed and no peers has nothing to read a
 header from, and taking a full minute to say so looks like a hang rather than

package/docs/internals.md

@@ -25,6 +25,7 @@ Operator-facing documentation is elsewhere — see [publishing](publishing.md),
 - [Scheduled sources](#scheduled-sources)
 - [Running two engines at once](#running-two-engines-at-once)
 - [Why libtorrent runs as a sidecar](#why-libtorrent-runs-as-a-sidecar)
+  - [A sidecar that outlives the node](#a-sidecar-that-outlives-the-node)
 - [Making a torrent out of a map](#making-a-torrent-out-of-a-map)
 - [Resuming a partial download](#resuming-a-partial-download)
 - [Retiring and pruning a subscription](#retiring-and-pruning-a-subscription)
@@ -579,6 +580,40 @@ under its final name until it finishes. Do not point a web server at a libtorren
 save path that is also serving web seeds. Completion is still recorded correctly
 — the watcher finds no marked file and notes that the archive is whole.
 
+### A sidecar that outlives the node
+
+The one failure mode of running libtorrent in another process, and the reason a
+service can need starting two or three times before it takes.
+
+The sidecar exits when its pipe closes, which covers an orderly stop and most
+crashes. It does not cover a sidecar in the middle of **hashing**: libtorrent
+does not hand control back until a check finishes, so the process notices
+neither the closed pipe nor a signal until it does — minutes on a large
+archive, hours on a planet. Nor does it cover a node killed outright, which
+takes no part in stopping anything.
+
+What is left behind still holds the listen port and the resume directory. The
+next start fails against it, and since a failed start leaves its own sidecar,
+they accumulate. In the journal it reads as:
+
+```
+pmtiles-swarm.service: Unit process 102573 (python) remains running after unit stopped.
+pmtiles-swarm.service: Failed to kill control group ...: Invalid argument
+```
+
+Two things prevent it. Stopping **insists**: the shutdown request goes first,
+so resume data is saved either way, and a sidecar that has not gone within a
+few seconds is killed rather than left. And a start **reaps** what a previous
+run left, before it spawns anything — the sidecar's pid is recorded beside its
+resume data, and a recorded pid is killed only when `/proc` says that process
+is really a sidecar. Pids are reused, and killing whatever inherited one would
+be worse than the problem.
+
+Neither is a substitute for `KillMode=mixed` in the unit. That is what stops
+systemd signalling the sidecar at the same moment as the node, which kills it
+before it can write anything down — see
+[running-as-a-service.md](running-as-a-service.md).
+
 ## Making a torrent out of a map
 
 Two choices here are specific to distributing maps rather than generic files.

package/docs/running-as-a-service.md

@@ -189,6 +189,50 @@ The unit below is what it generates, with your paths in it. Worth reading
 either way — a generated file you do not understand is a hand-written one you
 have not written yet.
 
+## Regenerating the unit on a node that already runs
+
+The unit is derived from two things that move: the configuration, and how many
+archives the library holds. A watched folder added six months ago belongs in
+`ReadWritePaths`, and a library that has grown needs longer to write its resume
+data than the unit allows. So running this again is ordinary:
+
+```
+sudo pmtiles-swarm init --systemd --config /etc/pmtiles-swarm/swarm.config.json
+```
+
+**It reads the configuration and writes only the unit**, next to that
+configuration — never into `/etc/systemd/system`, and never over
+`swarm.config.json`. Installing it is your step, and worth a look first:
+
+```
+diff /etc/systemd/system/pmtiles-swarm.service \
+     /etc/pmtiles-swarm/pmtiles-swarm.service
+sudo cp /etc/pmtiles-swarm/pmtiles-swarm.service /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemctl restart pmtiles-swarm
+```
+
+**`--force` is a different command.** It replaces `swarm.config.json` with a
+fresh one — tokens, stacks, feeds and all — and it is for starting over, not
+for regenerating a unit. Nothing about writing the unit needs it.
+
+**Run it as root, not as the service account.** It writes into `/etc`, which
+the service account cannot do and should not be able to do. What decides the
+`User=` line is `--user`, which defaults to `pmtiles-swarm`; pass it if the
+account is called something else:
+
+```
+sudo pmtiles-swarm init --systemd \
+  --config /etc/pmtiles-swarm/swarm.config.json \
+  --user maps
+```
+
+`ReadWritePaths` is derived from the configuration, so **a path added to the
+installed unit by hand is not in the generated one**. That is what the diff is
+for. The durable fix is to move such a path into the configuration — as a save
+location, a watched folder, or a cache directory — after which it is derived
+like the rest.
+
 ## The unit
 
 Annotated, because every directive here has cost somebody a diagnosis.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.92.0",
+  "version": "0.95.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -22,7 +22,12 @@ import {
 import { mutableMagnet, trackersFromMagnet } from './mutable.js';
 import { guessKind } from './library.js';
 import { QBittorrentEngine } from './engines/qbittorrent.js';
-import { RESTART_REQUIRED, redactConfig, saveConfig } from './config.js';
+import {
+  RESTART_REQUIRED,
+  redactConfig,
+  saveConfig,
+  trustProxyFor,
+} from './config.js';
 import { freeSpace, listLocations } from './locations.js';
 import { restart, restartMode } from './restart.js';
 import { parseFeed, renderFeed } from './feed.js';
@@ -260,7 +265,26 @@ export function createApp({
   // the TileJSON advertises http:// tile URLs. A browser that loaded the map
   // over https then blocks every one of them as mixed content, which looks
   // like an empty map rather than like a configuration mistake.
-  if (config.trustProxy) app.set('trust proxy', config.trustProxy);
+  //
+  // Normalised first, and then guarded anyway. Express compiles this value
+  // the moment it is set, which is before the listener binds -- so a value it
+  // cannot compile is not a setting that fails to apply, it is a node that
+  // will not start, cannot be reached, and cannot be corrected from the
+  // console that wrote it. Nothing about which addresses to trust is worth
+  // that: an unparseable one is reported and the node comes up trusting
+  // nobody, which is the safe end of being wrong.
+  const trustProxy = trustProxyFor(config.trustProxy);
+  if (trustProxy !== false) {
+    try {
+      app.set('trust proxy', trustProxy);
+    } catch (error) {
+      console.error(
+        `[config] trustProxy could not be applied (${error.message}). ` +
+          'X-Forwarded-* headers are being ignored; the node is starting ' +
+          'anyway so this can be corrected from Settings.',
+      );
+    }
+  }
   app.use(express.json({ limit: '1mb' }));
 
   // Tiles, TileJSON and the feed stay public — serving them is the point.

package/src/config.js

@@ -1,4 +1,5 @@
 import fs from 'node:fs/promises';
+import net from 'node:net';
 import { hashPassword } from './auth.js';
 import path from 'node:path';
 
@@ -10,6 +11,79 @@ import path from 'node:path';
  * Every setting is documented in docs/configuration.md. Comments here say only
  * what a value is; why it is what it is belongs in the document.
  */
+/** What proxy-addr accepts as a name for a group of addresses. */
+const TRUST_KEYWORDS = new Set(['loopback', 'linklocal', 'uniquelocal']);
+
+/**
+ * Whether one entry of a trust list is something proxy-addr can compile.
+ * @param {string} entry - One address, subnet or keyword.
+ * @returns {boolean} - True if it is usable.
+ */
+function isTrustEntry(entry) {
+  if (TRUST_KEYWORDS.has(entry)) return true;
+  const [address, mask, ...rest] = entry.split('/');
+  if (rest.length > 0) return false;
+  if (!net.isIP(address)) return false;
+  if (mask === undefined) return true;
+  if (net.isIP(mask)) return true;
+  const bits = Number(mask);
+  if (!Number.isInteger(bits) || bits < 0) return false;
+  return bits <= (net.isIP(address) === 6 ? 128 : 32);
+}
+
+/**
+ * What to hand Express as `trust proxy`, from what the config says.
+ *
+ * Express takes four different things here and means something different by
+ * each, and it splits a comma-separated list only when it is handed a bare
+ * string -- never inside an array. So `["10.0.0.1, 10.0.0.2"]`, which is what
+ * a settings field split on newlines alone produces from one line with a
+ * comma in it, reaches proxy-addr as a single address, and proxy-addr throws.
+ *
+ * That throw happened while the app was being built, before the listener
+ * bound: a node that could not start, could not be reached, and could not be
+ * corrected from the console that wrote the value. So every shape is
+ * flattened here, and anything left that is not an address is dropped with a
+ * warning rather than carried into Express.
+ * @param {boolean|number|string|string[]} value - `config.trustProxy`.
+ * @returns {boolean|number|string[]} - Something Express can compile.
+ */
+export function trustProxyFor(value) {
+  if (
+    value === true ||
+    value === false ||
+    value === undefined ||
+    value === null
+  ) {
+    return value === true;
+  }
+  if (typeof value === 'number') return Number.isFinite(value) ? value : false;
+
+  const entries = (Array.isArray(value) ? value : [value])
+    .flatMap((one) => String(one).split(/[\s,]+/))
+    .map((one) => one.trim())
+    .filter(Boolean);
+  if (!entries.length) return false;
+
+  // A lone number is a hop count, and a lone `true` trusts everybody. Both
+  // are things somebody may type into a field that mostly takes addresses.
+  if (entries.length === 1) {
+    if (entries[0] === 'true') return true;
+    if (entries[0] === 'false') return false;
+    if (!Number.isNaN(Number(entries[0]))) return Number(entries[0]);
+  }
+
+  const usable = entries.filter((entry) => isTrustEntry(entry));
+  for (const entry of entries) {
+    if (usable.includes(entry)) continue;
+    console.warn(
+      `[config] trustProxy: ignoring "${entry}", which is not an address, ` +
+        'a subnet, or one of loopback, linklocal, uniquelocal.',
+    );
+  }
+  return usable.length ? usable : false;
+}
+
 const DEFAULTS = {
   port: 8090,
   host: '0.0.0.0',
@@ -290,12 +364,47 @@ const DEFAULTS = {
    */
   tiles: {
     /**
-     * Open archives kept alive at once. Each holds a file descriptor or a
-     * torrent reader plus its piece cache, so this bounds both.
+     * Open archives kept alive at once.
+     *
+     * Sized for a library of hundreds, because that is what a node that
+     * merges layers holds: a stack built from a provider's file index names
+     * four hundred sources and a bake walks every one of them. At sixteen,
+     * such a run spent most of its time reopening archives it had just
+     * closed, and each reopen re-reads a header and a directory.
+     *
+     * A complete archive open is a file descriptor and its share of the
+     * directory cache, which is why this can be large -- the unit allows
+     * 65535 of them. An archive read through the swarm is not: it carries a
+     * piece cache, and those are bounded separately below.
+     */
+    maxOpenArchives: 128,
+    /**
+     * Open archives read through the swarm, which is the expensive kind.
+     *
+     * Counted apart from the limit above and kept where that limit used to
+     * be. A cache-mode reader holds a piece cache sized from the torrent's
+     * piece length, so with 16 MiB pieces a hundred of them is gigabytes --
+     * the reason the single old limit could not simply be raised.
+     */
+    maxOpenSwarmArchives: 16,
+    /**
+     * Open archives read straight from a URL or a bucket.
+     *
+     * Cheap in memory -- an HTTP reader and the summary it has already read
+     * -- and expensive to reopen, since a reopen costs a header and a
+     * directory fetch over the network. So this sits well above the swarm
+     * limit and below the local one.
+     */
+    maxOpenRemoteArchives: 64,
+    /**
+     * Header and directory cache entries, shared across every archive.
+     *
+     * One archive contributes several: a header, a root directory, and a leaf
+     * per region being read. Two hundred was a handful of archives' worth, so
+     * a stack over hundreds of sources evicted its own directories between
+     * one tile and the next.
      */
-    maxOpenArchives: 16,
-    /** Header and directory cache entries, shared across every archive. */
-    directoryCacheEntries: 200,
+    directoryCacheEntries: 2000,
     /**
      * Byte budget for the piece cache of one swarm-read archive. Unset sizes
      * it from the torrent's piece length, which is safer than a fixed budget.

package/src/engines/libtorrent.js

@@ -1,4 +1,5 @@
 import { spawn } from 'node:child_process';
+import fs from 'node:fs/promises';
 import { createRequire } from 'node:module';
 import path from 'node:path';
 
@@ -29,6 +30,122 @@ function resolveSidecar() {
   }
 }
 
+/**
+ * Ends a child process, politely and then not.
+ * @param {object} child - The process.
+ * @param {number} graceMs - How long the polite request gets.
+ * @returns {Promise<void>} - When it has gone.
+ */
+export async function stopChild(child, graceMs) {
+  if (!child || child.exitCode !== null || child.signalCode !== null) return;
+  const gone = new Promise((resolve) => child.once('exit', resolve));
+  child.kill();
+  const settled = await Promise.race([
+    gone.then(() => true),
+    new Promise((resolve) => {
+      const timer = setTimeout(() => resolve(false), graceMs);
+      timer.unref?.();
+    }),
+  ]);
+  if (settled) return;
+  console.warn(
+    '[libtorrent] the sidecar did not stop when asked -- it is usually ' +
+      'hashing, which cannot be interrupted -- so it is being killed. Its ' +
+      'resume data was already saved.',
+  );
+  child.kill('SIGKILL');
+  await gone;
+}
+
+/**
+ * Where the pid of the running sidecar is written.
+ * @param {string} [resumeDir] - Where resume data is kept.
+ * @returns {string|null} - The path, or null when there is nowhere to put it.
+ */
+function sidecarPidPath(resumeDir) {
+  return resumeDir ? path.join(resumeDir, 'sidecar.pid') : null;
+}
+
+/**
+ * Records which process is the sidecar, so a later start can recognise it.
+ * @param {string} [resumeDir] - Where resume data is kept.
+ * @param {number} pid - The sidecar.
+ * @returns {Promise<void>} - When written.
+ */
+async function rememberSidecarPid(resumeDir, pid) {
+  const file = sidecarPidPath(resumeDir);
+  if (!file) return;
+  await fs.writeFile(file, String(pid)).catch(() => {});
+}
+
+/**
+ * Forgets the recorded pid.
+ * @param {string} [resumeDir] - Where resume data is kept.
+ * @returns {Promise<void>} - When removed.
+ */
+async function forgetSidecarPid(resumeDir) {
+  const file = sidecarPidPath(resumeDir);
+  if (!file) return;
+  await fs.rm(file, { force: true }).catch(() => {});
+}
+
+/**
+ * Kills a sidecar left behind by a previous run.
+ *
+ * A node killed outright -- SIGKILL, an OOM, a power cut -- takes no part in
+ * stopping its sidecar, and a sidecar mid-hash does not notice its pipe close.
+ * It goes on holding the listen port and the resume directory, and the next
+ * start fails against it. Reaped here rather than lived with, because the
+ * alternative is the operator restarting the service until it takes.
+ *
+ * Identified by its command line, not by the pid alone: pids are reused, and
+ * killing whatever inherited one would be far worse than the problem.
+ * @param {string} [resumeDir] - Where resume data is kept.
+ * @returns {Promise<boolean>} - Whether one was killed.
+ */
+export async function reapStaleSidecar(resumeDir) {
+  const file = sidecarPidPath(resumeDir);
+  if (!file) return false;
+  let pid;
+  try {
+    pid = Number(await fs.readFile(file, 'utf8'));
+  } catch {
+    return false;
+  }
+  if (!Number.isInteger(pid) || pid <= 1 || pid === process.pid) {
+    await forgetSidecarPid(resumeDir);
+    return false;
+  }
+
+  // Only where the process table can be read as files, which is where this
+  // runs as a service. Elsewhere a stale pid is left alone: guessing is worse.
+  let cmdline;
+  try {
+    cmdline = await fs.readFile(`/proc/${pid}/cmdline`, 'utf8');
+  } catch {
+    await forgetSidecarPid(resumeDir);
+    return false;
+  }
+  if (!cmdline.includes('libtorrent_sidecar')) {
+    await forgetSidecarPid(resumeDir);
+    return false;
+  }
+
+  console.warn(
+    `[libtorrent] a sidecar from a previous run is still running (pid ${pid}). ` +
+      'It holds the listen port and the resume directory this one needs, so ' +
+      'it is being killed. This is what a start that has to be repeated ' +
+      'two or three times looks like.',
+  );
+  try {
+    process.kill(pid, 'SIGKILL');
+  } catch {
+    // Gone between reading and killing, which is the good outcome.
+  }
+  await forgetSidecarPid(resumeDir);
+  return true;
+}
+
 /**
  * A SeedEngine backed by libtorrent, through a sidecar process.
  *
@@ -117,6 +234,9 @@ export class LibtorrentEngine {
 
     this.#ready = new Promise((resolve, reject) => {
       const script = this.#options.script ?? resolveSidecar();
+      // Before the spawn, not after: a sidecar from a previous run holds the
+      // port this one is about to ask for.
+      const reaped = reapStaleSidecar(this.#options.resumeDir);
 
       const child = spawn(this.#options.python, [script], {
         stdio: ['pipe', 'pipe', 'pipe'],
@@ -140,6 +260,7 @@ export class LibtorrentEngine {
         },
       });
       this.#child = child;
+      reaped.then(() => rememberSidecarPid(this.#options.resumeDir, child.pid));
 
       // Nothing of the last sidecar's is carried into this one. Being killed
       // does not wait for a newline, so a sidecar that died partway through a
@@ -802,9 +923,19 @@ export class LibtorrentEngine {
     await this.#call('shutdown', {}, options.timeoutMs ?? 15000).catch(
       () => {},
     );
-    this.#child?.kill();
+
+    // Asked, then insisted. A sidecar in the middle of hashing is not reading
+    // its pipe and does not act on a signal until libtorrent hands control
+    // back, which on a large archive is minutes -- so a plain SIGTERM left it
+    // running after the node had gone. systemd then reports a unit process
+    // that remains after the unit stopped, the next start finds the old one
+    // still holding the listen port, and the library comes back holding
+    // nothing. That is the restart that has to be done two or three times.
+    const child = this.#child;
     this.#child = null;
     this.#ready = null;
+    await stopChild(child, options.killGraceMs ?? 5000);
+    await forgetSidecarPid(this.#options.resumeDir);
   }
 
   /**

package/src/index.js

@@ -28,6 +28,7 @@ import {
   installSignalHandlers,
   runStoppers,
 } from './shutdown.js';
+import { TIMEOUT_STOP_SECONDS } from './systemd.js';
 import { ScheduledSourceManager } from './sources.js';
 import { StackExportScheduler } from './stack-exports.js';
 import { StackFeedSubscriber } from './stack-feed.js';
@@ -338,13 +339,47 @@ PMTILES_SWARM_PUBLIC_URL
     );
 
   const catalogued = catalog.list().length;
-  if (catalogued > 0) {
-    const { restored, failed } = await library.restore();
-    console.log(
-      `[restore] ${restored} of ${catalogued} archives handed back to the engine` +
-        (failed > 0 ? ` (${failed} could not be)` : ''),
+  // Said once at startup, because the way this fails is silent. Stopping
+  // writes resume data for every archive and the engine is given two seconds
+  // apiece; a unit that allows less than that is killed partway through, and
+  // what comes back re-hashes every archive whose resume data never landed.
+  // There is no error at that moment -- only a library at 0% and hours of
+  // disk. The generated unit derives its allowance from the library it was
+  // written for, so this is really "your library has grown since then".
+  const stopNeeds = Math.ceil(engineStopMs(catalogued) / 1000);
+  if (stopNeeds > TIMEOUT_STOP_SECONDS) {
+    console.warn(
+      `[shutdown] stopping this library needs about ${stopNeeds}s to write ` +
+        `resume data, which is more than the ${TIMEOUT_STOP_SECONDS}s ` +
+        'a default systemd unit allows. Re-run `pmtiles-swarm init --systemd` ' +
+        'to regenerate the unit, or raise TimeoutStopSec by hand -- a stop cut ' +
+        'short re-hashes every archive it had not saved.',
     );
   }
+  if (catalogued > 0) {
+    // Reported, never fatal. Restore already tolerates a failure per archive;
+    // what this catches is the whole call coming apart -- and the node it
+    // takes down with it is the one that could have said so. Under
+    // `Restart=always` that is a crash loop with no console to look at, which
+    // is a worse failure than a library that is not being seeded: the console
+    // marks an archive the engine has no record of as `not loaded`, so a node
+    // that comes up says exactly what went wrong here.
+    try {
+      const { restored, failed } = await library.restore();
+      console.log(
+        `[restore] ${restored} of ${catalogued} archives handed back to the engine` +
+          (failed > 0 ? ` (${failed} could not be)` : ''),
+      );
+    } catch (error) {
+      console.error(
+        `[restore] could not hand the library back to the engine: ` +
+          `${error.stack ?? error.message}
+` +
+          '[restore] the node is starting anyway; every archive will show as ' +
+          'not loaded until this is fixed and it is restarted.',
+      );
+    }
+  }
 
   // And again if the engine loses its backing process and starts another. A
   // replacement holds nothing, so without this the node would come back

package/src/init-command.js

@@ -3,7 +3,7 @@ import fs from 'node:fs/promises';
 import os from 'node:os';
 import path from 'node:path';
 import { hashPassword } from './auth.js';
-import { writablePaths } from './config.js';
+import { loadConfig, writablePaths } from './config.js';
 import { directoryCommands, unitFor } from './systemd.js';
 
 /**
@@ -78,6 +78,25 @@ function firstConfig({ dataDir, savePath, passwordHash }) {
   };
 }
 
+/**
+ * How many archives the catalog holds, if there is one to read.
+ *
+ * Best effort by design: this runs before the node does, on a machine that
+ * may have no data directory at all, and a missing or unreadable catalog is
+ * an install rather than an error.
+ * @param {object} config - The resolved configuration.
+ * @returns {Promise<number>} - The count, or zero.
+ */
+async function countCatalog(config) {
+  try {
+    const file = path.join(config.dataDir, 'catalog.json');
+    const held = JSON.parse(await fs.readFile(file, 'utf8'));
+    return Array.isArray(held.entries) ? held.entries.length : 0;
+  } catch {
+    return 0;
+  }
+}
+
 /**
  * Writes a first configuration file.
  *
@@ -128,10 +147,49 @@ export async function runInit(options = {}, write = console.log) {
     .access(configPath)
     .then(() => true)
     .catch(() => false);
+
+  // A node that already has a configuration and wants the unit is the
+  // ordinary reason to run this a second time: the unit is derived from the
+  // configuration and from how many archives the library holds, and both move.
+  // Refusing sent people to --force, which replaces the configuration --
+  // tokens, stacks, feeds and all -- to regenerate a file beside it.
+  if (exists && options.systemd && !options.force) {
+    const held = await loadConfig(configPath);
+    const unitPath = path.join(
+      path.dirname(configPath),
+      'pmtiles-swarm.service',
+    );
+    await fs.writeFile(
+      unitPath,
+      unitFor({
+        config: held,
+        configPath,
+        user: options.user,
+        execStart: options.execStart,
+        archives: await countCatalog(held),
+      }),
+    );
+    write(`Wrote ${unitPath}`);
+    write('');
+    write(`  ${configPath} was read, not written.`);
+    write('');
+    write('  Compare it with the one that is installed before replacing it:');
+    write('');
+    write(`    diff /etc/systemd/system/pmtiles-swarm.service ${unitPath}`);
+    write('');
+    write('  ReadWritePaths is derived from the configuration, so a path');
+    write('  added to the installed unit by hand is not in this one. Move it');
+    write('  into the configuration — as a save location, a watched folder or');
+    write('  a cache path — and it will be derived from now on.');
+    return 0;
+  }
+
   if (exists && !options.force) {
     write(
       `${configPath} already exists. Pass --force to replace it — and take a ` +
-        'copy first, since the tokens in it are not recoverable.',
+        'copy first, since the tokens in it are not recoverable. To write ' +
+        'just the unit from the configuration that is there, pass --systemd ' +
+        'without --force.',
     );
     return 1;
   }
@@ -164,6 +222,10 @@ export async function runInit(options = {}, write = console.log) {
         configPath,
         user: options.user,
         execStart: options.execStart,
+        // So the stop timeout fits the library this node already holds. On a
+        // fresh install there is none and the default stands; re-running this
+        // after the library has grown is what keeps the two together.
+        archives: await countCatalog(config),
       }),
     );
     write(`Wrote ${unitPath}`);

package/src/systemd.js

@@ -1,5 +1,6 @@
 import os from 'node:os';
 import path from 'node:path';
+import { engineStopMs } from './shutdown.js';
 import { writablePaths } from './config.js';
 
 /**
@@ -26,7 +27,29 @@ import { writablePaths } from './config.js';
  */
 
 /** What the resume save can want, before the library has grown into it. */
-const TIMEOUT_STOP_SECONDS = 300;
+export const TIMEOUT_STOP_SECONDS = 300;
+
+/**
+ * How long systemd should allow a stop, for a library of this size.
+ *
+ * The node gives its engine two seconds an archive to write resume data, and
+ * systemd's allowance has to outlast that or the save is cut off partway --
+ * which is exactly the library that returns at 0% and re-hashes for hours.
+ * A fixed five minutes covered a library of 145 and no more, silently, so
+ * this is derived the same way `ReadWritePaths` and the thread pool are.
+ *
+ * Rounded up to whole minutes, with a wide margin over the node's own budget:
+ * being generous here costs nothing, since systemd stops waiting the moment
+ * the process is gone.
+ * @param {object} config - The resolved configuration.
+ * @param {number} [archives] - How many the catalog holds.
+ * @returns {number} - Seconds for `TimeoutStopSec`.
+ */
+export function stopTimeoutFor(config, archives = 0) {
+  const budget = engineStopMs(archives) / 1000;
+  const wanted = Math.ceil((budget * 1.5 + 60) / 60) * 60;
+  return Math.max(TIMEOUT_STOP_SECONDS, wanted);
+}
 
 /** libuv's own, which is what a unit that says nothing gets. */
 const DEFAULT_THREADPOOL = 4;
@@ -68,11 +91,13 @@ export function unitFor({
   user = 'pmtiles-swarm',
   execStart,
   workingDirectory,
+  archives = 0,
 }) {
   const home = workingDirectory ?? `/var/lib/${user}`;
   const binary = execStart ?? `${home}/node_modules/.bin/pmtiles-swarm`;
   const paths = writablePaths(config, configPath);
   const threadPool = threadPoolFor(config);
+  const stopSeconds = stopTimeoutFor(config, archives);
 
   // Wrapped the way systemd's own examples are, because this list grows with
   // every watched folder and a single line of them is unreadable in a diff.
@@ -128,9 +153,14 @@ RestartSec=5
 # Stopping writes resume data for every archive, and the node allows its engine
 # two seconds per torrent to do it. This has to outlast that: killed mid-save,
 # every archive that had not been written re-hashes its whole store on the way
-# back up, which for a 700 GiB archive is hours. Raise it as the library grows
-# — tools/resume-doctor.py prints the arithmetic for yours.
-TimeoutStopSec=${TIMEOUT_STOP_SECONDS}
+# back up, which for a 700 GiB archive is hours.
+#
+# Derived from the ${archives} archive(s) this node holds, with room to grow.
+# A fixed five minutes covered a library of 145 and no more, and covered it
+# silently — the symptom of outgrowing it is not an error but a library that
+# comes back at 0%. Re-running init --systemd recomputes this;
+# tools/resume-doctor.py prints the arithmetic for yours.
+TimeoutStopSec=${stopSeconds}
 
 # The node stops the sidecar itself, and needs it alive to do so. The default,
 # control-group, signals both at once and the sidecar dies before it can write

package/src/tiles.js

@@ -76,7 +76,7 @@ export class TileStore {
     this.#engine = engine;
     this.#config = config;
     this.#directoryCache = new SharedPromiseCache(
-      config.tiles?.directoryCacheEntries ?? 200,
+      config.tiles?.directoryCacheEntries ?? 2000,
     );
   }
 
@@ -240,7 +240,7 @@ export class TileStore {
     };
     this.#remote.set(url, handle);
 
-    const limit = this.#config.tiles?.maxOpenRemoteArchives ?? 16;
+    const limit = this.#config.tiles?.maxOpenRemoteArchives ?? 64;
     while (this.#remote.size > limit) {
       const [oldest] = this.#remote.keys();
       this.#remote.delete(oldest);
@@ -431,13 +431,38 @@ export class TileStore {
     const handle = await this.#openArchive(entry);
     this.#open.set(entry.infoHash, handle);
 
-    const limit = this.#config.tiles?.maxOpenArchives ?? 16;
-    while (this.#open.size > limit) {
-      const [oldest, victim] = this.#open.entries().next().value;
-      this.#open.delete(oldest);
+    // Two budgets, because the two kinds of handle cost quite different
+    // things. A complete archive is a file descriptor; one read through the
+    // swarm carries a piece cache sized from the torrent's piece length,
+    // which with 16 MiB pieces is tens of megabytes each. A single limit had
+    // to be set for the expensive kind, which left a library of hundreds of
+    // local archives reopening files it had just closed.
+    const tiles = this.#config.tiles ?? {};
+    await this.#evict(() => true, tiles.maxOpenArchives ?? 128);
+    await this.#evict(
+      (one) => one.mode === 'swarm',
+      tiles.maxOpenSwarmArchives ?? 16,
+    );
+    return handle;
+  }
+
+  /**
+   * Closes the least recently used handles until a budget is met.
+   *
+   * Least recently used first, which a Map gives for nothing: reading one
+   * moves it to the end, so iteration order is oldest first.
+   * @param {Function} counts - Whether a handle counts against this budget.
+   * @param {number} limit - How many of them to keep.
+   * @returns {Promise<void>} - When the closing is done.
+   */
+  async #evict(counts, limit) {
+    if (!Number.isFinite(limit) || limit < 1) return;
+    const held = [...this.#open].filter(([, one]) => counts(one));
+    while (held.length > limit) {
+      const [key, victim] = held.shift();
+      this.#open.delete(key);
       await this.#release(victim);
     }
-    return handle;
   }
 
   /**

package/src/web/index.html

@@ -5279,17 +5279,42 @@ Every piece is hashed against the ` +
               key: 'tiles.maxOpenArchives',
               label: 'Archives kept open',
               type: 'number',
+              placeholder: '128',
+              help:
+                'A complete archive open is a file descriptor and its share ' +
+                'of the directory cache, so this can be large — the unit ' +
+                'allows 65535 of them. Sized for a node that merges layers, ' +
+                'where one stack names hundreds of sources and a bake walks ' +
+                'every one. Past the number of archives this node serves it ' +
+                'stops doing anything.',
+            },
+            {
+              key: 'tiles.maxOpenSwarmArchives',
+              label: 'Cache-mode archives kept open',
+              type: 'number',
               placeholder: '16',
               help:
-                'Each holds a file descriptor, and in cache mode a piece ' +
-                'cache as well. Past the number of archives this node serves ' +
-                'it stops doing anything.',
+                'Counted apart from the limit above, because this is the ' +
+                'expensive kind: a cache-mode reader holds a piece cache ' +
+                'sized from the torrent’s piece length, so with 16 MiB pieces ' +
+                'a hundred of them is gigabytes. That is why the one limit ' +
+                'above could not simply be raised.',
+            },
+            {
+              key: 'tiles.maxOpenRemoteArchives',
+              label: 'URL and bucket archives kept open',
+              type: 'number',
+              placeholder: '64',
+              help:
+                'Cheap to hold — an HTTP reader and the summary it has ' +
+                'already read — and expensive to reopen, since a reopen costs ' +
+                'a header and a directory fetch over the network.',
             },
             {
               key: 'tiles.directoryCacheEntries',
               label: 'Directory cache entries',
               type: 'number',
-              placeholder: '200',
+              placeholder: '2000',
               restart: true,
               help:
                 'PMTiles finds a tile through a root directory and then a ' +
@@ -5579,11 +5604,15 @@ Every piece is hashed against the ` +
               type: 'addresses',
               placeholder: 'off — or one address or subnet per line',
               help:
-                'One per line: 172.16.1.49, or 172.16.1.0/24, or several. A ' +
-                'lone number is read as a hop count instead, and a lone `true` ' +
-                'trusts any caller at all — worth avoiding, since this header ' +
-                'is what decides which address the node believes a request ' +
-                'came from.',
+                'An address or a subnet — <code>172.16.1.49</code>, ' +
+                '<code>172.16.1.0/24</code>, <code>loopback</code> — one ' +
+                'per line, or separated by commas. A lone number is read ' +
+                'as a hop count instead, and a lone <code>true</code> ' +
+                'trusts any caller at all, which is worth avoiding: this ' +
+                'header is what decides which address the node believes a ' +
+                'request came from. Anything that is not an address is ' +
+                'ignored and said so in the log, rather than stopping the ' +
+                'node from starting.',
             },
             {
               key: 'publicIndex',
@@ -6307,7 +6336,14 @@ Every piece is hashed against the ` +
               field.type === 'secret'
                 ? `<input type="password" autocomplete="new-password" data-setting="${field.key}"${locked} data-type="secret" data-initial="&quot;&quot;" value="" placeholder="${escapeHtml(field.placeholder ?? 'unchanged')}" />`
                 : field.type === 'addresses'
-                ? `<textarea data-setting="${field.key}"${locked} data-type="addresses"${initial} rows="3" placeholder="${escapeHtml(field.placeholder ?? '')}">${escapeHtml(Array.isArray(value) ? value.join('\n') : value === undefined || value === null ? '' : String(value))}</textarea>`
+                ? // Its own initial, because this is the one field whose
+                    // box holds a different shape from the config. A
+                    // stored string never compares equal to the array
+                    // the box parses to, so sharing `initial` meant
+                    // every Save rewrote a setting nobody had touched
+                    // -- which is how a value that worked became one
+                    // the node would not start with.
+                    `<textarea data-setting="${field.key}"${locked} data-type="addresses" data-initial="${escapeHtml(JSON.stringify(parseAddresses(addressText(value))))}" rows="3" placeholder="${escapeHtml(field.placeholder ?? '')}">${escapeHtml(addressText(value))}</textarea>`
                 : field.type === 'list'
                 ? `<textarea data-setting="${field.key}"${locked} data-type="list"${initial} rows="${Math.min(8, Math.max(3, (Array.isArray(value) ? value.length : 0) + 1))}" placeholder="${escapeHtml(field.placeholder ?? '')}">${escapeHtml((Array.isArray(value) ? value : []).join('\n'))}</textarea>`
                 : field.type === 'boolean'
@@ -6337,6 +6373,56 @@ Every piece is hashed against the ` +
         return panel;
       }
 
+      /**
+       * The entries an address field holds, however the config wrote them.
+       *
+       * A list may be stored as an array or as one comma-separated string --
+       * Express accepts both, and the documented example is a string. Shown
+       * one per line either way, so that what a save reads back is the list
+       * that was rendered rather than a different shape of the same thing.
+       * @param {null|boolean|number|string|string[]} value - What the config holds.
+       * @returns {string} - The textarea's contents.
+       */
+      const addressText = (value) => {
+        if (value === undefined || value === null) return '';
+        const entries = Array.isArray(value) ? value : [String(value)];
+        return entries
+          .flatMap((entry) => String(entry).split(/[\n,]/))
+          .map((entry) => entry.trim())
+          .filter(Boolean)
+          .join('\n');
+      };
+
+      /**
+       * What an address field means by what it holds.
+       *
+       * Express takes four things here and means something different by each:
+       * `true` trusts any caller, a number is a hop count, and a string or an
+       * array of them names the proxies. What was typed decides which.
+       *
+       * Commas separate as well as newlines. Express splits a comma list only
+       * when it is handed a bare string and never inside an array, so one line
+       * reading `10.0.0.1, 10.0.0.2` was saved as a one-element array holding
+       * both -- which is not an address, and which Express rejects while the
+       * app is being built, before the listener binds. The node then would not
+       * start, and could not be corrected from the console that wrote it.
+       * @param {string} text - What the box holds.
+       * @returns {null|boolean|number|string[]} - What to save.
+       */
+      const parseAddresses = (text) => {
+        const lines = String(text)
+          .split(/[\n,]/)
+          .map((line) => line.trim())
+          .filter(Boolean);
+        if (!lines.length) return null;
+        if (lines.length === 1) {
+          if (lines[0] === 'true') return true;
+          if (lines[0] === 'false') return false;
+          if (!Number.isNaN(Number(lines[0]))) return Number(lines[0]);
+        }
+        return lines;
+      };
+
       /**
        * Collects every schema control into an update, grouped by top-level key.
        *
@@ -6368,22 +6454,8 @@ Every piece is hashed against the ` +
             continue;
           }
           if (type === 'boolean') value = element.checked;
-          else if (type === 'addresses') {
-            // Express takes four things here and means something different by
-            // each: `true` trusts any caller, a number is a hop count, and a
-            // string or an array of them names the proxies. One per line, and
-            // what was typed decides which of the four it is.
-            const lines = String(element.value)
-              .split('\n')
-              .map((line) => line.trim())
-              .filter(Boolean);
-            if (!lines.length) value = null;
-            else if (lines.length === 1 && (lines[0] === 'true' || lines[0] === 'false')) {
-              value = lines[0] === 'true';
-            } else if (lines.length === 1 && !Number.isNaN(Number(lines[0]))) {
-              value = Number(lines[0]);
-            } else value = lines;
-          } else if (type === 'list') {
+          else if (type === 'addresses') value = parseAddresses(element.value);
+          else if (type === 'list') {
             // One per line, which is how a person reads a list of trackers —
             // and empty means an empty list rather than "unset", because a
             // node with no trackers is a real thing to want and JSON would