pmtiles-swarm 0.94.0 → 0.96.0

package/CHANGELOG.md

@@ -7,6 +7,75 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.96.0
+### ✨ Features and improvements
+- **The swarm-read limit is memory now, not a count.** `tiles.maxOpenSwarmArchives: 16` becomes
+  `tiles.swarmCacheBytes: 1 GiB`, because a count was the wrong unit for it. What is expensive about
+  a cache-mode reader is its piece cache, which is **RAM** — a map of whole pieces in the node's own
+  heap — and how big it is depends on the torrent: `max(64 MiB, 8 × pieceLength)`. Sixteen readers
+  is a gigabyte against the 4 MiB pieces this project creates and two against the 16 MiB pieces a
+  planet torrent usually has, which is not a limit anybody chose.
+
+  Each reader is asked what its cache costs rather than assumed, so the count now follows from the
+  memory: halve `pieceCacheBytes` and twice as many archives stay open for the same gigabyte. The
+  last reader is never closed, whatever the budget says — the archive just opened is the one being
+  read. `maxOpenSwarmArchives` remains for a node that wants a hard count as well; unset by default.
+
+  Complete archives are unaffected: they hold a file descriptor and no piece cache, and are bounded
+  by `maxOpenArchives`.
+- **`init --systemd` regenerates the unit without touching the configuration.** The unit is derived
+  from the configuration and from how many archives the library holds, and both move — a watched
+  folder added later belongs in `ReadWritePaths`, and a grown library needs longer to write its
+  resume data than the unit allows. Running it again used to be refused, which sent people to
+  `--force`, which replaces `swarm.config.json` outright: tokens, stacks, feeds and all, to
+  regenerate a file beside it. It now reads the configuration, writes only the unit, and says so —
+  along with the `diff` to run before replacing what is installed, since `ReadWritePaths` is derived
+  and a path added to the installed unit by hand will not be in the new one.
+
+### 🐞 Bug fixes
+- **The unit's `ReadWritePaths` left out where an export writes.** `stackExports[].savePath` and
+  `.publishDir`, and a watched folder's `publishDir`, were not among the directories derived from
+  the configuration — so a nightly bake to a directory named nowhere else ran for an hour and was
+  refused the write at the end, with that directory's permissions perfect. Which is the exact
+  failure the derived list exists to prevent. Re-run `init --systemd` to pick them up.
+
+## 0.95.0
+### ✨ Features and improvements
+- **The defaults now assume a library of hundreds, not a handful.** `tiles.maxOpenArchives` goes
+  from 16 to 128 and `tiles.directoryCacheEntries` from 200 to 2000. Sixteen was set for the most
+  expensive kind of open archive and then applied to every kind, which is wrong for the node this is
+  increasingly used to build: a stack assembled from a provider's file index names several hundred
+  sources, and a bake walks every one of them. At sixteen such a run spent most of its time
+  reopening archives it had just closed, and each reopen re-reads a header and a directory.
+
+  So the limit is now three limits, because the handles cost different things. A complete archive is
+  a file descriptor and the unit allows 65535 of them. A **cache-mode** archive carries a piece cache
+  sized from the torrent's piece length — at 16 MiB pieces a hundred of them is gigabytes — so
+  `tiles.maxOpenSwarmArchives` keeps the old ceiling of 16 and is counted separately. A **URL or
+  bucket** archive holds an HTTP reader and nothing else, and costs a network round trip to reopen,
+  so `tiles.maxOpenRemoteArchives` is 64. A node serving only cache-mode archives behaves as before.
+
+- **The unit's stop timeout is derived from the library it was written for.** Stopping writes resume
+  data for every archive and the node allows two seconds apiece, so a fixed five minutes covered a
+  library of 145 and no more — silently, because outgrowing it produces no error, just a library
+  that comes back at 0% and re-hashes for hours. `init --systemd` now counts the catalog and sizes
+  `TimeoutStopSec` to fit, and a node whose library has outgrown a default unit says so at startup.
+
+
+### 🐞 Bug fixes
+- **A sidecar left running by a previous start is now killed rather than fought.** This is the
+  restart that has to be done two or three times before it takes. The sidecar exits when its pipe
+  closes, which covers an orderly stop — but not one in the middle of hashing, since libtorrent does
+  not hand control back until the check finishes, and not a node killed outright. What is left holds
+  the listen port and the resume directory, the next start fails against it, and since a failed
+  start leaves its own sidecar they accumulate.
+
+  Stopping now insists: the shutdown request goes first so resume data is still saved, and a sidecar
+  that has not gone within a few seconds is killed rather than left behind. And a start reaps what a
+  previous run left before spawning anything — by pid, but only where `/proc` confirms that pid is
+  really a sidecar, since pids are reused and killing whatever inherited one would be worse than the
+  problem.
+
 ## 0.94.0
 ### 🐞 Bug fixes
 - **Saving settings rewrote a proxy list nobody had touched.** A trusted-proxy list may be stored as

package/docs/configuration.md

@@ -724,20 +724,70 @@ through the swarm, pulling only the pieces a requested tile lives in — which i
 what lets a machine with 10 GiB free serve a 700 GiB planet. See
 [serving-tiles.md](serving-tiles.md).
 
-| setting                       | default  |                                                                  |
-| ----------------------------- | -------- | ---------------------------------------------------------------- |
-| `tiles.maxOpenArchives`       | `16`     | each holds a descriptor or a torrent reader plus its piece cache |
-| `tiles.directoryCacheEntries` | `200`    | header and directory cache, shared across archives               |
-| `tiles.pieceCacheBytes`       | unset    | sized from the torrent's piece length when unset                 |
-| `tiles.hydrateIdleMs`         | unset    | idle time before background hydration resumes                    |
-| `tiles.pieceTimeoutMs`        | `120000` | how long to wait for one piece                                   |
-| `tiles.readyTimeoutMs`        | `60000`  | how long to wait for torrent metadata                            |
-| `tiles.metadataTimeoutMs`     | `120000` | how long a background metadata read may take                     |
-| `tiles.headerTimeoutMs`       | `12000`  | how long a TileJSON request waits for a header                   |
-| `tiles.sparse`                | unset    | `true` for 404 on a missing tile, `false` for 204                |
-
-Leave `pieceCacheBytes` unset unless you have a reason. A fixed budget is a trap
-with 16 MiB pieces, since 64 MiB holds only four.
+| setting                       | default  |                                                              |
+| ----------------------------- | -------- | ------------------------------------------------------------ |
+| `tiles.maxOpenArchives`       | `128`    | complete archives kept open; each is a file descriptor       |
+| `tiles.swarmCacheBytes`       | `1 GiB`  | **memory** the piece caches of swarm reads share             |
+| `tiles.maxOpenSwarmArchives`  | unset    | a hard count of swarm readers as well, if you want one       |
+| `tiles.maxOpenRemoteArchives` | `64`     | readers for a URL or a bucket; cheap to hold, dear to reopen |
+| `tiles.directoryCacheEntries` | `2000`   | header and directory cache, shared across archives           |
+| `tiles.pieceCacheBytes`       | unset    | sized from the torrent's piece length when unset             |
+| `tiles.hydrateIdleMs`         | unset    | idle time before background hydration resumes                |
+| `tiles.pieceTimeoutMs`        | `120000` | how long to wait for one piece                               |
+| `tiles.readyTimeoutMs`        | `60000`  | how long to wait for torrent metadata                        |
+| `tiles.metadataTimeoutMs`     | `120000` | how long a background metadata read may take                 |
+| `tiles.headerTimeoutMs`       | `12000`  | how long a TileJSON request waits for a header               |
+| `tiles.sparse`                | unset    | `true` for 404 on a missing tile, `false` for 204            |
+
+### Why a swarm read caches pieces in memory at all
+
+The swarm's unit is a **piece** — 4 MiB for archives this project creates,
+commonly 16 MiB for a planet torrent. A tile is a few kilobytes inside one. So
+fetching a piece to serve one tile and throwing it away means the next tile
+pays for the whole piece again, and PMTiles guarantees there will be a next
+tile in the same piece: its tiles are laid out in Hilbert order, so tiles that
+are neighbours on the map are neighbours in the file. A tile lookup also reads
+the header and a directory before the tile itself, and those live in pieces
+that every single request would otherwise re-fetch.
+
+Not on disk, because a disk copy of the pieces is what cache mode exists to
+avoid: the point is a machine with 10 GiB free serving a 700 GiB planet. The
+engine keeps whatever it keeps; this is the layer above it, and evicting from
+it costs nothing.
+
+**Trading cache size for open archives.** The two settings multiply. Halving
+`pieceCacheBytes` doubles how many archives stay open within the same
+`swarmCacheBytes`, at the cost of a lower hit rate on each. `16 MiB × 64
+archives` and `64 MiB × 16 archives` are the same gigabyte spent differently:
+the first suits a node reading a tile here and a tile there across many
+archives, the second a node serving a region of a few.
+
+Leave `pieceCacheBytes` unset unless you have such a reason. A fixed budget is
+a trap with 16 MiB pieces, since 64 MiB holds only four.
+
+### Three limits, because the handles cost different things
+
+They were one limit, set at sixteen for the most expensive kind and applied to
+every kind. That is wrong for the node this is increasingly used to build: a
+stack assembled from a provider's file index names several hundred sources, and
+a bake walks every one of them. At sixteen such a run spends most of its time
+reopening archives it has just closed, and each reopen re-reads a header and a
+directory.
+
+- **A complete archive** is a file descriptor and its share of the directory
+  cache. The unit allows 65535 descriptors, so hundreds of these are nothing.
+- **A cache-mode archive** carries a piece cache, and it is **memory** — a map
+  of whole pieces in the node's own heap, not a directory on disk. Bounded by
+  `swarmCacheBytes` rather than by a count, because a count is the wrong unit:
+  a reader allows itself `max(64 MiB, 8 × pieceLength)`, so sixteen of them is
+  1 GiB against the 4 MiB pieces this project creates and 2 GiB against the
+  16 MiB pieces a planet torrent usually has. Stated as memory, the number
+  means what an operator cares about and the count follows from it.
+- **A URL or bucket archive** holds an HTTP reader and the summary it has
+  already read. Cheap to keep, and expensive to reopen: a reopen costs a header
+  and a directory fetch over the network rather than off a disk.
+
+A node serving nothing but cache-mode archives behaves exactly as it did.
 
 `headerTimeoutMs` is shorter than the others on purpose: somebody is waiting on
 it. A cache-mode archive with no web seed and no peers has nothing to read a

package/docs/internals.md

@@ -25,6 +25,7 @@ Operator-facing documentation is elsewhere — see [publishing](publishing.md),
 - [Scheduled sources](#scheduled-sources)
 - [Running two engines at once](#running-two-engines-at-once)
 - [Why libtorrent runs as a sidecar](#why-libtorrent-runs-as-a-sidecar)
+  - [A sidecar that outlives the node](#a-sidecar-that-outlives-the-node)
 - [Making a torrent out of a map](#making-a-torrent-out-of-a-map)
 - [Resuming a partial download](#resuming-a-partial-download)
 - [Retiring and pruning a subscription](#retiring-and-pruning-a-subscription)
@@ -579,6 +580,40 @@ under its final name until it finishes. Do not point a web server at a libtorren
 save path that is also serving web seeds. Completion is still recorded correctly
 — the watcher finds no marked file and notes that the archive is whole.
 
+### A sidecar that outlives the node
+
+The one failure mode of running libtorrent in another process, and the reason a
+service can need starting two or three times before it takes.
+
+The sidecar exits when its pipe closes, which covers an orderly stop and most
+crashes. It does not cover a sidecar in the middle of **hashing**: libtorrent
+does not hand control back until a check finishes, so the process notices
+neither the closed pipe nor a signal until it does — minutes on a large
+archive, hours on a planet. Nor does it cover a node killed outright, which
+takes no part in stopping anything.
+
+What is left behind still holds the listen port and the resume directory. The
+next start fails against it, and since a failed start leaves its own sidecar,
+they accumulate. In the journal it reads as:
+
+```
+pmtiles-swarm.service: Unit process 102573 (python) remains running after unit stopped.
+pmtiles-swarm.service: Failed to kill control group ...: Invalid argument
+```
+
+Two things prevent it. Stopping **insists**: the shutdown request goes first,
+so resume data is saved either way, and a sidecar that has not gone within a
+few seconds is killed rather than left. And a start **reaps** what a previous
+run left, before it spawns anything — the sidecar's pid is recorded beside its
+resume data, and a recorded pid is killed only when `/proc` says that process
+is really a sidecar. Pids are reused, and killing whatever inherited one would
+be worse than the problem.
+
+Neither is a substitute for `KillMode=mixed` in the unit. That is what stops
+systemd signalling the sidecar at the same moment as the node, which kills it
+before it can write anything down — see
+[running-as-a-service.md](running-as-a-service.md).
+
 ## Making a torrent out of a map
 
 Two choices here are specific to distributing maps rather than generic files.

package/docs/running-as-a-service.md

@@ -189,6 +189,50 @@ The unit below is what it generates, with your paths in it. Worth reading
 either way — a generated file you do not understand is a hand-written one you
 have not written yet.
 
+## Regenerating the unit on a node that already runs
+
+The unit is derived from two things that move: the configuration, and how many
+archives the library holds. A watched folder added six months ago belongs in
+`ReadWritePaths`, and a library that has grown needs longer to write its resume
+data than the unit allows. So running this again is ordinary:
+
+```
+sudo pmtiles-swarm init --systemd --config /etc/pmtiles-swarm/swarm.config.json
+```
+
+**It reads the configuration and writes only the unit**, next to that
+configuration — never into `/etc/systemd/system`, and never over
+`swarm.config.json`. Installing it is your step, and worth a look first:
+
+```
+diff /etc/systemd/system/pmtiles-swarm.service \
+     /etc/pmtiles-swarm/pmtiles-swarm.service
+sudo cp /etc/pmtiles-swarm/pmtiles-swarm.service /etc/systemd/system/
+sudo systemctl daemon-reload
+sudo systemctl restart pmtiles-swarm
+```
+
+**`--force` is a different command.** It replaces `swarm.config.json` with a
+fresh one — tokens, stacks, feeds and all — and it is for starting over, not
+for regenerating a unit. Nothing about writing the unit needs it.
+
+**Run it as root, not as the service account.** It writes into `/etc`, which
+the service account cannot do and should not be able to do. What decides the
+`User=` line is `--user`, which defaults to `pmtiles-swarm`; pass it if the
+account is called something else:
+
+```
+sudo pmtiles-swarm init --systemd \
+  --config /etc/pmtiles-swarm/swarm.config.json \
+  --user maps
+```
+
+`ReadWritePaths` is derived from the configuration, so **a path added to the
+installed unit by hand is not in the generated one**. That is what the diff is
+for. The durable fix is to move such a path into the configuration — as a save
+location, a watched folder, or a cache directory — after which it is derived
+like the rest.
+
 ## The unit
 
 Annotated, because every directive here has cost somebody a diagnosis.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.94.0",
+  "version": "0.96.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/config.js

@@ -364,12 +364,62 @@ const DEFAULTS = {
    */
   tiles: {
     /**
-     * Open archives kept alive at once. Each holds a file descriptor or a
-     * torrent reader plus its piece cache, so this bounds both.
+     * Open archives kept alive at once.
+     *
+     * Sized for a library of hundreds, because that is what a node that
+     * merges layers holds: a stack built from a provider's file index names
+     * four hundred sources and a bake walks every one of them. At sixteen,
+     * such a run spent most of its time reopening archives it had just
+     * closed, and each reopen re-reads a header and a directory.
+     *
+     * A complete archive open is a file descriptor and its share of the
+     * directory cache, which is why this can be large -- the unit allows
+     * 65535 of them. An archive read through the swarm is not: it carries a
+     * piece cache, and those are bounded separately below.
+     */
+    maxOpenArchives: 128,
+    /**
+     * Memory the piece caches of swarm-read archives may hold between them.
+     *
+     * A count would be the wrong unit. What is expensive about a cache-mode
+     * reader is its piece cache, and how big that is depends on the torrent:
+     * `max(64 MiB, 8 x pieceLength)`, so 64 MiB for the 4 MiB pieces this
+     * project creates and 128 MiB for the 16 MiB pieces a planet torrent
+     * usually has. Sixteen readers is a gigabyte in one case and two in the
+     * other, which is not a limit anybody chose.
+     *
+     * Stated as memory, the number means what an operator actually cares
+     * about, and the count follows from it: lower `pieceCacheBytes` and more
+     * archives stay open, at the same ceiling. A budget rather than an
+     * allocation -- an archive that has served three tiles holds three
+     * pieces.
+     */
+    swarmCacheBytes: 1024 * 1024 * 1024,
+    /**
+     * A hard count of swarm-read archives, for a node that wants one.
+     *
+     * Unset by default: the budget above is the better limit, because it is
+     * the thing that runs out. Set this as well and the smaller wins.
+     */
+    maxOpenSwarmArchives: undefined,
+    /**
+     * Open archives read straight from a URL or a bucket.
+     *
+     * Cheap in memory -- an HTTP reader and the summary it has already read
+     * -- and expensive to reopen, since a reopen costs a header and a
+     * directory fetch over the network. So this sits well above the swarm
+     * limit and below the local one.
+     */
+    maxOpenRemoteArchives: 64,
+    /**
+     * Header and directory cache entries, shared across every archive.
+     *
+     * One archive contributes several: a header, a root directory, and a leaf
+     * per region being read. Two hundred was a handful of archives' worth, so
+     * a stack over hundreds of sources evicted its own directories between
+     * one tile and the next.
      */
-    maxOpenArchives: 16,
-    /** Header and directory cache entries, shared across every archive. */
-    directoryCacheEntries: 200,
+    directoryCacheEntries: 2000,
     /**
      * Byte budget for the piece cache of one swarm-read archive. Unset sizes
      * it from the torrent's piece length, which is safer than a fixed budget.
@@ -729,8 +779,16 @@ export function writablePaths(config, configPath) {
     // directory holding it is written to as surely as any of the above.
     configPath ? path.dirname(path.resolve(configPath)) : undefined,
     ...(config?.watch ?? []).map((entry) => entry?.path),
+    ...(config?.watch ?? []).map((entry) => entry?.publishDir),
     ...(config?.locations ?? []).map((entry) => entry?.path),
     ...(config?.subscriptions ?? []).map((entry) => entry?.savePath),
+    // Where a scheduled export puts the archive it just built, and where it
+    // publishes a copy of it. Missed for as long as this list has existed:
+    // a nightly bake to a directory named nowhere else ran for an hour and
+    // was refused the write at the end, with the directory's permissions
+    // perfect -- which is the exact failure this list exists to prevent.
+    ...(config?.stackExports ?? []).map((entry) => entry?.savePath),
+    ...(config?.stackExports ?? []).map((entry) => entry?.publishDir),
   ].filter((value) => typeof value === 'string' && value);
 
   // Deduplicated but deliberately not collapsed into common ancestors. A

package/src/engines/libtorrent.js

@@ -1,4 +1,5 @@
 import { spawn } from 'node:child_process';
+import fs from 'node:fs/promises';
 import { createRequire } from 'node:module';
 import path from 'node:path';
 
@@ -29,6 +30,122 @@ function resolveSidecar() {
   }
 }
 
+/**
+ * Ends a child process, politely and then not.
+ * @param {object} child - The process.
+ * @param {number} graceMs - How long the polite request gets.
+ * @returns {Promise<void>} - When it has gone.
+ */
+export async function stopChild(child, graceMs) {
+  if (!child || child.exitCode !== null || child.signalCode !== null) return;
+  const gone = new Promise((resolve) => child.once('exit', resolve));
+  child.kill();
+  const settled = await Promise.race([
+    gone.then(() => true),
+    new Promise((resolve) => {
+      const timer = setTimeout(() => resolve(false), graceMs);
+      timer.unref?.();
+    }),
+  ]);
+  if (settled) return;
+  console.warn(
+    '[libtorrent] the sidecar did not stop when asked -- it is usually ' +
+      'hashing, which cannot be interrupted -- so it is being killed. Its ' +
+      'resume data was already saved.',
+  );
+  child.kill('SIGKILL');
+  await gone;
+}
+
+/**
+ * Where the pid of the running sidecar is written.
+ * @param {string} [resumeDir] - Where resume data is kept.
+ * @returns {string|null} - The path, or null when there is nowhere to put it.
+ */
+function sidecarPidPath(resumeDir) {
+  return resumeDir ? path.join(resumeDir, 'sidecar.pid') : null;
+}
+
+/**
+ * Records which process is the sidecar, so a later start can recognise it.
+ * @param {string} [resumeDir] - Where resume data is kept.
+ * @param {number} pid - The sidecar.
+ * @returns {Promise<void>} - When written.
+ */
+async function rememberSidecarPid(resumeDir, pid) {
+  const file = sidecarPidPath(resumeDir);
+  if (!file) return;
+  await fs.writeFile(file, String(pid)).catch(() => {});
+}
+
+/**
+ * Forgets the recorded pid.
+ * @param {string} [resumeDir] - Where resume data is kept.
+ * @returns {Promise<void>} - When removed.
+ */
+async function forgetSidecarPid(resumeDir) {
+  const file = sidecarPidPath(resumeDir);
+  if (!file) return;
+  await fs.rm(file, { force: true }).catch(() => {});
+}
+
+/**
+ * Kills a sidecar left behind by a previous run.
+ *
+ * A node killed outright -- SIGKILL, an OOM, a power cut -- takes no part in
+ * stopping its sidecar, and a sidecar mid-hash does not notice its pipe close.
+ * It goes on holding the listen port and the resume directory, and the next
+ * start fails against it. Reaped here rather than lived with, because the
+ * alternative is the operator restarting the service until it takes.
+ *
+ * Identified by its command line, not by the pid alone: pids are reused, and
+ * killing whatever inherited one would be far worse than the problem.
+ * @param {string} [resumeDir] - Where resume data is kept.
+ * @returns {Promise<boolean>} - Whether one was killed.
+ */
+export async function reapStaleSidecar(resumeDir) {
+  const file = sidecarPidPath(resumeDir);
+  if (!file) return false;
+  let pid;
+  try {
+    pid = Number(await fs.readFile(file, 'utf8'));
+  } catch {
+    return false;
+  }
+  if (!Number.isInteger(pid) || pid <= 1 || pid === process.pid) {
+    await forgetSidecarPid(resumeDir);
+    return false;
+  }
+
+  // Only where the process table can be read as files, which is where this
+  // runs as a service. Elsewhere a stale pid is left alone: guessing is worse.
+  let cmdline;
+  try {
+    cmdline = await fs.readFile(`/proc/${pid}/cmdline`, 'utf8');
+  } catch {
+    await forgetSidecarPid(resumeDir);
+    return false;
+  }
+  if (!cmdline.includes('libtorrent_sidecar')) {
+    await forgetSidecarPid(resumeDir);
+    return false;
+  }
+
+  console.warn(
+    `[libtorrent] a sidecar from a previous run is still running (pid ${pid}). ` +
+      'It holds the listen port and the resume directory this one needs, so ' +
+      'it is being killed. This is what a start that has to be repeated ' +
+      'two or three times looks like.',
+  );
+  try {
+    process.kill(pid, 'SIGKILL');
+  } catch {
+    // Gone between reading and killing, which is the good outcome.
+  }
+  await forgetSidecarPid(resumeDir);
+  return true;
+}
+
 /**
  * A SeedEngine backed by libtorrent, through a sidecar process.
  *
@@ -117,6 +234,9 @@ export class LibtorrentEngine {
 
     this.#ready = new Promise((resolve, reject) => {
       const script = this.#options.script ?? resolveSidecar();
+      // Before the spawn, not after: a sidecar from a previous run holds the
+      // port this one is about to ask for.
+      const reaped = reapStaleSidecar(this.#options.resumeDir);
 
       const child = spawn(this.#options.python, [script], {
         stdio: ['pipe', 'pipe', 'pipe'],
@@ -140,6 +260,7 @@ export class LibtorrentEngine {
         },
       });
       this.#child = child;
+      reaped.then(() => rememberSidecarPid(this.#options.resumeDir, child.pid));
 
       // Nothing of the last sidecar's is carried into this one. Being killed
       // does not wait for a newline, so a sidecar that died partway through a
@@ -802,9 +923,19 @@ export class LibtorrentEngine {
     await this.#call('shutdown', {}, options.timeoutMs ?? 15000).catch(
       () => {},
     );
-    this.#child?.kill();
+
+    // Asked, then insisted. A sidecar in the middle of hashing is not reading
+    // its pipe and does not act on a signal until libtorrent hands control
+    // back, which on a large archive is minutes -- so a plain SIGTERM left it
+    // running after the node had gone. systemd then reports a unit process
+    // that remains after the unit stopped, the next start finds the old one
+    // still holding the listen port, and the library comes back holding
+    // nothing. That is the restart that has to be done two or three times.
+    const child = this.#child;
     this.#child = null;
     this.#ready = null;
+    await stopChild(child, options.killGraceMs ?? 5000);
+    await forgetSidecarPid(this.#options.resumeDir);
   }
 
   /**

package/src/index.js

@@ -28,6 +28,7 @@ import {
   installSignalHandlers,
   runStoppers,
 } from './shutdown.js';
+import { TIMEOUT_STOP_SECONDS } from './systemd.js';
 import { ScheduledSourceManager } from './sources.js';
 import { StackExportScheduler } from './stack-exports.js';
 import { StackFeedSubscriber } from './stack-feed.js';
@@ -338,6 +339,23 @@ PMTILES_SWARM_PUBLIC_URL
     );
 
   const catalogued = catalog.list().length;
+  // Said once at startup, because the way this fails is silent. Stopping
+  // writes resume data for every archive and the engine is given two seconds
+  // apiece; a unit that allows less than that is killed partway through, and
+  // what comes back re-hashes every archive whose resume data never landed.
+  // There is no error at that moment -- only a library at 0% and hours of
+  // disk. The generated unit derives its allowance from the library it was
+  // written for, so this is really "your library has grown since then".
+  const stopNeeds = Math.ceil(engineStopMs(catalogued) / 1000);
+  if (stopNeeds > TIMEOUT_STOP_SECONDS) {
+    console.warn(
+      `[shutdown] stopping this library needs about ${stopNeeds}s to write ` +
+        `resume data, which is more than the ${TIMEOUT_STOP_SECONDS}s ` +
+        'a default systemd unit allows. Re-run `pmtiles-swarm init --systemd` ' +
+        'to regenerate the unit, or raise TimeoutStopSec by hand -- a stop cut ' +
+        'short re-hashes every archive it had not saved.',
+    );
+  }
   if (catalogued > 0) {
     // Reported, never fatal. Restore already tolerates a failure per archive;
     // what this catches is the whole call coming apart -- and the node it

package/src/init-command.js

@@ -3,7 +3,7 @@ import fs from 'node:fs/promises';
 import os from 'node:os';
 import path from 'node:path';
 import { hashPassword } from './auth.js';
-import { writablePaths } from './config.js';
+import { loadConfig, writablePaths } from './config.js';
 import { directoryCommands, unitFor } from './systemd.js';
 
 /**
@@ -78,6 +78,25 @@ function firstConfig({ dataDir, savePath, passwordHash }) {
   };
 }
 
+/**
+ * How many archives the catalog holds, if there is one to read.
+ *
+ * Best effort by design: this runs before the node does, on a machine that
+ * may have no data directory at all, and a missing or unreadable catalog is
+ * an install rather than an error.
+ * @param {object} config - The resolved configuration.
+ * @returns {Promise<number>} - The count, or zero.
+ */
+async function countCatalog(config) {
+  try {
+    const file = path.join(config.dataDir, 'catalog.json');
+    const held = JSON.parse(await fs.readFile(file, 'utf8'));
+    return Array.isArray(held.entries) ? held.entries.length : 0;
+  } catch {
+    return 0;
+  }
+}
+
 /**
  * Writes a first configuration file.
  *
@@ -128,10 +147,49 @@ export async function runInit(options = {}, write = console.log) {
     .access(configPath)
     .then(() => true)
     .catch(() => false);
+
+  // A node that already has a configuration and wants the unit is the
+  // ordinary reason to run this a second time: the unit is derived from the
+  // configuration and from how many archives the library holds, and both move.
+  // Refusing sent people to --force, which replaces the configuration --
+  // tokens, stacks, feeds and all -- to regenerate a file beside it.
+  if (exists && options.systemd && !options.force) {
+    const held = await loadConfig(configPath);
+    const unitPath = path.join(
+      path.dirname(configPath),
+      'pmtiles-swarm.service',
+    );
+    await fs.writeFile(
+      unitPath,
+      unitFor({
+        config: held,
+        configPath,
+        user: options.user,
+        execStart: options.execStart,
+        archives: await countCatalog(held),
+      }),
+    );
+    write(`Wrote ${unitPath}`);
+    write('');
+    write(`  ${configPath} was read, not written.`);
+    write('');
+    write('  Compare it with the one that is installed before replacing it:');
+    write('');
+    write(`    diff /etc/systemd/system/pmtiles-swarm.service ${unitPath}`);
+    write('');
+    write('  ReadWritePaths is derived from the configuration, so a path');
+    write('  added to the installed unit by hand is not in this one. Move it');
+    write('  into the configuration — as a save location, a watched folder or');
+    write('  a cache path — and it will be derived from now on.');
+    return 0;
+  }
+
   if (exists && !options.force) {
     write(
       `${configPath} already exists. Pass --force to replace it — and take a ` +
-        'copy first, since the tokens in it are not recoverable.',
+        'copy first, since the tokens in it are not recoverable. To write ' +
+        'just the unit from the configuration that is there, pass --systemd ' +
+        'without --force.',
     );
     return 1;
   }
@@ -164,6 +222,10 @@ export async function runInit(options = {}, write = console.log) {
         configPath,
         user: options.user,
         execStart: options.execStart,
+        // So the stop timeout fits the library this node already holds. On a
+        // fresh install there is none and the default stands; re-running this
+        // after the library has grown is what keeps the two together.
+        archives: await countCatalog(config),
       }),
     );
     write(`Wrote ${unitPath}`);

package/src/systemd.js

@@ -1,5 +1,6 @@
 import os from 'node:os';
 import path from 'node:path';
+import { engineStopMs } from './shutdown.js';
 import { writablePaths } from './config.js';
 
 /**
@@ -26,7 +27,29 @@ import { writablePaths } from './config.js';
  */
 
 /** What the resume save can want, before the library has grown into it. */
-const TIMEOUT_STOP_SECONDS = 300;
+export const TIMEOUT_STOP_SECONDS = 300;
+
+/**
+ * How long systemd should allow a stop, for a library of this size.
+ *
+ * The node gives its engine two seconds an archive to write resume data, and
+ * systemd's allowance has to outlast that or the save is cut off partway --
+ * which is exactly the library that returns at 0% and re-hashes for hours.
+ * A fixed five minutes covered a library of 145 and no more, silently, so
+ * this is derived the same way `ReadWritePaths` and the thread pool are.
+ *
+ * Rounded up to whole minutes, with a wide margin over the node's own budget:
+ * being generous here costs nothing, since systemd stops waiting the moment
+ * the process is gone.
+ * @param {object} config - The resolved configuration.
+ * @param {number} [archives] - How many the catalog holds.
+ * @returns {number} - Seconds for `TimeoutStopSec`.
+ */
+export function stopTimeoutFor(config, archives = 0) {
+  const budget = engineStopMs(archives) / 1000;
+  const wanted = Math.ceil((budget * 1.5 + 60) / 60) * 60;
+  return Math.max(TIMEOUT_STOP_SECONDS, wanted);
+}
 
 /** libuv's own, which is what a unit that says nothing gets. */
 const DEFAULT_THREADPOOL = 4;
@@ -68,11 +91,13 @@ export function unitFor({
   user = 'pmtiles-swarm',
   execStart,
   workingDirectory,
+  archives = 0,
 }) {
   const home = workingDirectory ?? `/var/lib/${user}`;
   const binary = execStart ?? `${home}/node_modules/.bin/pmtiles-swarm`;
   const paths = writablePaths(config, configPath);
   const threadPool = threadPoolFor(config);
+  const stopSeconds = stopTimeoutFor(config, archives);
 
   // Wrapped the way systemd's own examples are, because this list grows with
   // every watched folder and a single line of them is unreadable in a diff.
@@ -128,9 +153,14 @@ RestartSec=5
 # Stopping writes resume data for every archive, and the node allows its engine
 # two seconds per torrent to do it. This has to outlast that: killed mid-save,
 # every archive that had not been written re-hashes its whole store on the way
-# back up, which for a 700 GiB archive is hours. Raise it as the library grows
-# — tools/resume-doctor.py prints the arithmetic for yours.
-TimeoutStopSec=${TIMEOUT_STOP_SECONDS}
+# back up, which for a 700 GiB archive is hours.
+#
+# Derived from the ${archives} archive(s) this node holds, with room to grow.
+# A fixed five minutes covered a library of 145 and no more, and covered it
+# silently — the symptom of outgrowing it is not an error but a library that
+# comes back at 0%. Re-running init --systemd recomputes this;
+# tools/resume-doctor.py prints the arithmetic for yours.
+TimeoutStopSec=${stopSeconds}
 
 # The node stops the sidecar itself, and needs it alive to do so. The default,
 # control-group, signals both at once and the sidecar dies before it can write

package/src/tiles.js

@@ -49,6 +49,42 @@ export class TileReadError extends Error {
   }
 }
 
+/** What a reader's piece cache holds before it has seen the torrent. */
+const STARTING_CACHE_BYTES = 64 * 1024 * 1024;
+
+/**
+ * Which swarm-read archives to close for their piece caches to fit a budget.
+ *
+ * Each reader is asked what its cache costs rather than assumed, because only
+ * it knows: the budget it sets for itself is `max(64 MiB, 8 x pieceLength)`,
+ * and the piece length is the torrent's. So a node reading 4 MiB-piece
+ * archives keeps four times as many open as one reading 16 MiB-piece
+ * archives, for the same memory -- which is the arithmetic an operator would
+ * otherwise have to do by hand with every piece length in front of them.
+ *
+ * Least recently used first, and never the last one: the archive just opened
+ * is the one being read, and closing it to meet a budget it alone exceeds
+ * would serve no tile at all.
+ * @param {Array} held - `[key, handle]` pairs, least recently used first.
+ * @param {number} budget - Bytes the piece caches may hold between them.
+ * @returns {Array} - The pairs to close, in the order to close them.
+ */
+export function overSwarmBudget(held, budget) {
+  if (!Number.isFinite(budget) || budget <= 0) return [];
+  const swarm = held.filter(([, one]) => one?.mode === 'swarm');
+  const cost = (one) => one.source?.stats?.cacheBudget ?? STARTING_CACHE_BYTES;
+
+  let total = swarm.reduce((sum, [, one]) => sum + cost(one), 0);
+  const closing = [];
+  for (const pair of swarm) {
+    if (total <= budget) break;
+    if (swarm.length - closing.length <= 1) break;
+    closing.push(pair);
+    total -= cost(pair[1]);
+  }
+  return closing;
+}
+
 /**
  * Opens archives on demand and reads tiles out of them.
  */
@@ -76,7 +112,7 @@ export class TileStore {
     this.#engine = engine;
     this.#config = config;
     this.#directoryCache = new SharedPromiseCache(
-      config.tiles?.directoryCacheEntries ?? 200,
+      config.tiles?.directoryCacheEntries ?? 2000,
     );
   }
 
@@ -240,7 +276,7 @@ export class TileStore {
     };
     this.#remote.set(url, handle);
 
-    const limit = this.#config.tiles?.maxOpenRemoteArchives ?? 16;
+    const limit = this.#config.tiles?.maxOpenRemoteArchives ?? 64;
     while (this.#remote.size > limit) {
       const [oldest] = this.#remote.keys();
       this.#remote.delete(oldest);
@@ -431,15 +467,56 @@ export class TileStore {
     const handle = await this.#openArchive(entry);
     this.#open.set(entry.infoHash, handle);
 
-    const limit = this.#config.tiles?.maxOpenArchives ?? 16;
-    while (this.#open.size > limit) {
-      const [oldest, victim] = this.#open.entries().next().value;
-      this.#open.delete(oldest);
-      await this.#release(victim);
+    // Two budgets, because the two kinds of handle cost quite different
+    // things. A complete archive is a file descriptor; one read through the
+    // swarm carries a piece cache sized from the torrent's piece length,
+    // which with 16 MiB pieces is tens of megabytes each. A single limit had
+    // to be set for the expensive kind, which left a library of hundreds of
+    // local archives reopening files it had just closed.
+    const tiles = this.#config.tiles ?? {};
+    await this.#evict(() => true, tiles.maxOpenArchives ?? 128);
+    if (tiles.maxOpenSwarmArchives !== undefined) {
+      await this.#evict(
+        (one) => one.mode === 'swarm',
+        tiles.maxOpenSwarmArchives,
+      );
     }
+    await this.#evictSwarmMemory(tiles.swarmCacheBytes ?? 1024 * 1024 * 1024);
     return handle;
   }
 
+  /**
+   * Closes the least recently used handles until a budget is met.
+   *
+   * Least recently used first, which a Map gives for nothing: reading one
+   * moves it to the end, so iteration order is oldest first.
+   * @param {Function} counts - Whether a handle counts against this budget.
+   * @param {number} limit - How many of them to keep.
+   * @returns {Promise<void>} - When the closing is done.
+   */
+  async #evict(counts, limit) {
+    if (!Number.isFinite(limit) || limit < 1) return;
+    const held = [...this.#open].filter(([, one]) => counts(one));
+    while (held.length > limit) {
+      const [key, victim] = held.shift();
+      this.#open.delete(key);
+      await this.#release(victim);
+    }
+  }
+
+  /**
+   * Closes swarm-read archives until their piece caches fit a byte budget.
+   * @param {number} budget - Bytes the piece caches may hold between them.
+   * @returns {Promise<void>} - When the closing is done.
+   */
+  async #evictSwarmMemory(budget) {
+    for (const [key] of overSwarmBudget([...this.#open], budget)) {
+      const victim = this.#open.get(key);
+      this.#open.delete(key);
+      await this.#release(victim);
+    }
+  }
+
   /**
    * Opens an archive, choosing between the local file and the swarm.
    * @param {object} entry - Catalog entry.

package/src/web/index.html

@@ -5279,17 +5279,47 @@ Every piece is hashed against the ` +
               key: 'tiles.maxOpenArchives',
               label: 'Archives kept open',
               type: 'number',
-              placeholder: '16',
+              placeholder: '128',
               help:
-                'Each holds a file descriptor, and in cache mode a piece ' +
-                'cache as well. Past the number of archives this node serves ' +
-                'it stops doing anything.',
+                'A complete archive open is a file descriptor and its share ' +
+                'of the directory cache, so this can be large — the unit ' +
+                'allows 65535 of them. Sized for a node that merges layers, ' +
+                'where one stack names hundreds of sources and a bake walks ' +
+                'every one. Past the number of archives this node serves it ' +
+                'stops doing anything.',
+            },
+            {
+              key: 'tiles.swarmCacheBytes',
+              label: 'Piece cache memory, all swarm reads',
+              type: 'number',
+              placeholder: '1073741824',
+              help:
+                'RAM, not disk. An archive read through the swarm keeps whole ' +
+                'pieces in memory, because the swarm hands out pieces and a ' +
+                'tile is a few kilobytes of one — without it, every tile pays ' +
+                'for a 4 or 16 MiB fetch. Each reader allows itself ' +
+                '<code>max(64 MiB, 8 × piece length)</code>, so this is the ' +
+                'number that decides how many stay open: lower ' +
+                '<b>Piece cache bytes</b> below and more of them fit. A ' +
+                'budget, not an allocation — a reader that has served three ' +
+                'tiles holds three pieces. Complete archives are unaffected: ' +
+                'they hold a file descriptor and no cache.',
+            },
+            {
+              key: 'tiles.maxOpenRemoteArchives',
+              label: 'URL and bucket archives kept open',
+              type: 'number',
+              placeholder: '64',
+              help:
+                'Cheap to hold — an HTTP reader and the summary it has ' +
+                'already read — and expensive to reopen, since a reopen costs ' +
+                'a header and a directory fetch over the network.',
             },
             {
               key: 'tiles.directoryCacheEntries',
               label: 'Directory cache entries',
               type: 'number',
-              placeholder: '200',
+              placeholder: '2000',
               restart: true,
               help:
                 'PMTiles finds a tile through a root directory and then a ' +