pmtiles-swarm 0.58.1 → 0.59.0

package/CHANGELOG.md

@@ -7,6 +7,75 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.59.0
+### ✨ Features and improvements
+- **Requires pmtiles-torrent 0.10.2**, which is what actually ends the re-checking: a
+  `seedOnly` add now discards resume data that would cancel the claim, and the periodic save
+  leaves a hashing torrent alone. Everything below only stops the node making more of it.
+- **`pmtiles-swarm init` writes a first configuration, and optionally the unit to run it.**
+  Every path it writes is absolute, which is the one mistake it exists to make impossible: a
+  relative path resolves against the config file, the documented layout puts that file in
+  `/etc`, and so `./data` there means a catalog, a resume directory and potentially a 700 GiB
+  archive on the configuration partition. State under `/etc` is refused rather than warned
+  about — at the moment a config is written there is nothing to migrate, and the same mistake
+  found later costs a stopped service and a careful move.
+
+  `--systemd` also writes `pmtiles-swarm.service` beside it, with `ReadWritePaths` **derived
+  from the configuration it just wrote**. That derivation is the point. A unit and a config
+  that disagree is every systemd failure this project has diagnosed, and none of them look
+  like what they are: a `savePath` missing from that line is refused inside the unit's
+  namespace before any permission bit is read, so the directory's owner and mode are both
+  perfect and the write still fails. Two files generated from one source cannot drift.
+
+  It installs nothing. The unit is written next to the configuration and the `cp` into
+  `/etc/systemd/system` is one of the commands it prints, alongside an `install -d` for every
+  directory involved with the right owner and mode.
+
+  `--password` is hashed before it is written, and left out entirely when none is given.
+  There is deliberately no placeholder: `auth.password` accepts plaintext, so `REPLACE-ME` in
+  that field is a working password until somebody notices — a credential that looks set and
+  is not.
+
+- **Two scripts for diagnosing a library that re-checks on every start**, in `tools/`.
+  `resume-doctor.py` reads a node's real configuration, catalog, stored `.torrent` files and
+  resume directory and says, for each archive, what libtorrent will do on the next start and
+  why — applying libtorrent's own rules and citing the file and line each came from. It also
+  reads the unit through `systemctl show`, does the arithmetic on every deadline that can cut
+  a resume save short, and hashes pieces rather than trusting the catalog. `resume-experiment.py`
+  proves the five behaviours involved on whatever libtorrent is actually installed, because
+  1.2, 2.0 and 2.1 differ enough that a claim verified on one is not a claim about the other.
+
+### 🐞 Bug fixes
+- **A stop no longer abandons the sidecar mid-write, which is where resume data was going.**
+  Three separate bounds decided how long the engine step had, and the smallest won: eight
+  seconds for the step, fifteen for the whole shutdown, fifteen for the shutdown RPC. The
+  sidecar allows each torrent two seconds of its resume-save budget, so past four archives the
+  node gave up first and every torrent it had not persisted re-hashed its whole store on the
+  way back up. That is the state a library gets stuck in: checking, on every start, for hours.
+
+  The engine step is now worked out from the catalog — two seconds a torrent, over a floor —
+  and the shutdown watchdog is derived from the steps it is meant to contain rather than being
+  a second deadline kept in agreement with them by hand. It was not in agreement.
+
+  The same fixed 60s applied to the periodic save, so past thirty archives the call gave up
+  before the sidecar finished, and the written/asked counts never came back — which is why the
+  shortfall this reports could not be seen from outside. **`TimeoutStopSec` in the documented
+  unit rises from 45 to 300 seconds**, and existing installs need it raised by hand.
+
+- **A `.torrent` that moved with `dataDir` is found again instead of silently becoming a
+  magnet.** `torrentPath` is recorded absolute, so moving state out of `/etc` — which
+  `docs/running-as-a-service.md` tells you to do — left every catalog entry naming a directory
+  that no longer existed. Nothing repointed them and nothing complained, because an unreadable
+  `.torrent` was treated as "use the magnet instead".
+
+  That fallback is the damage rather than a graceful degradation. A magnet carries no metadata
+  and neither does resume data, so the archive waits on BEP 9 for a file list that only a peer
+  can supply — and for an archive this node originated there is nobody to ask. Seen in the
+  field: twenty archives at 0% in `downloading_metadata`, indefinitely, after one documented
+  migration. The current `dataDir` is now tried second, the recorded path is corrected in
+  place so the warning is printed once rather than for ever, and falling back to a magnet at
+  all now says so.
+
 ## 0.58.1
 ### 🐞 Bug fixes
 - **The sample configuration put every piece of state under the config file.** `"dataDir": "./data"`,

package/docs/running-as-a-service.md

@@ -151,8 +151,50 @@ sudo apt-get install -y python3-libtorrent
 sudo -u pmtiles-swarm python3 -c "import libtorrent; print(libtorrent.__version__)"
 ```
 
+## The configuration, and a unit for it
+
+`init` writes both, and the unit's `ReadWritePaths` is derived from the
+configuration it just wrote. That derivation is the reason to use it: a unit and
+a configuration that disagree is the failure this whole document is arranged
+around, and two files generated from one source cannot.
+
+```sh
+sudo -u pmtiles-swarm -H /var/lib/pmtiles-swarm/node_modules/.bin/pmtiles-swarm   init --config /etc/pmtiles-swarm/swarm.config.json   --data-dir /var/lib/pmtiles-swarm/data   --save-path /mnt/store/torrent-data   --systemd --password 'the console password'
+```
+
+That writes the configuration, generates `pmtiles-swarm.service` beside it, and
+prints a command for every directory involved — each with the right owner and
+mode. Nothing privileged happens and nothing is installed: the unit is written
+next to the configuration, and copying it into `/etc/systemd/system` is one of
+the lines it hands you.
+
+Two refusals worth knowing before you type it. State under `/etc` is rejected
+rather than warned about, because at the moment a configuration is written there
+is nothing to migrate — the same mistake found six months later costs a stopped
+service and a careful move. And a configuration that already exists is left
+alone unless `--force` says otherwise, since the credentials in it are not
+recoverable.
+
+`--password` is hashed before it is written. Leave it off and there is no
+console password at all: the generated API key is the way in until you set one.
+There is deliberately no placeholder, because `auth.password` accepts plaintext
+and a placeholder in that field is a working password until somebody notices.
+
+**Re-run it after adding a watched folder**, or a subscription with a save path
+of its own. Both are directories the node writes to, and neither reaches
+`ReadWritePaths` by itself. `tools/resume-doctor.py` checks a running node's
+unit against its live configuration and says which directories are missing.
+
+The unit below is what it generates, with your paths in it. Worth reading
+either way — a generated file you do not understand is a hand-written one you
+have not written yet.
+
 ## The unit
 
+Annotated, because every directive here has cost somebody a diagnosis.
+`init --systemd` writes this file with your paths already in it; what
+follows is what those lines are for.
+
 ```ini
 [Unit]
 Description=pmtiles-swarm
@@ -177,8 +219,12 @@ Restart=always
 RestartSec=5
 
 # Stopping announces "stopped" to every tracker, releases the data directory
-# lock and cancels downloads in flight. Worst case is about 20 seconds.
-TimeoutStopSec=45
+# lock, cancels downloads in flight and writes resume data for every archive.
+# That last part is what scales: the node allows its engine two seconds per
+# torrent, so a library of fifty wants well over a minute and systemd has to
+# outlast it. Too low here and the process is killed mid-save, which costs a
+# re-hash of everything unwritten on the way back up.
+TimeoutStopSec=300
 
 # The node stops the sidecar itself, and needs it alive to do so. The default
 # signals both at once, which kills the sidecar before it can write its resume
@@ -233,6 +279,17 @@ itself, in order, and waits for the resume data to be written. Nothing is left
 running: anything still alive when `TimeoutStopSec` expires is killed, sidecar
 included.
 
+**`TimeoutStopSec`, and why it is not a round number you pick once.** Stopping
+writes resume data for every archive, and the node allows its engine two seconds
+per torrent to do it — so the figure that matters grows with the library. Three
+hundred seconds covers a hundred and forty archives; a larger node needs more.
+Set it too low and systemd kills the process mid-save, and every archive that
+had not been written re-hashes its whole store on the way back up, which for a
+700 GiB archive is hours.
+
+`tools/resume-doctor.py` prints the arithmetic for your library and says whether
+the unit's value covers it.
+
 **No `ExecStop=`.** systemd already sends `SIGTERM`, and the node handles it
 from the moment it starts. `ExecStop=/bin/kill -15 $MAINPID` is redundant, and
 becomes wrong under `KillMode=process` — that one leaves the rest of the unit
@@ -779,3 +836,111 @@ once rather than diagnosing later:
   `journalctl -u pmtiles-swarm | grep -i onComplete` — and a hook redirecting its
   own output to a file will have nothing for the journal to show, which is not
   the same as not having run.
+
+## When archives re-check on every start
+
+The symptom is always the same shape — a restart, and then a library sitting at
+0% or grinding through a re-hash of archives that are whole on the disk. The
+causes are not the same shape at all, so guessing between them is expensive:
+re-hashing a 700 GiB archive is half an hour during which it serves nobody.
+
+Two scripts in `tools/` answer it without guessing. Neither starts anything,
+neither writes anything, and both are safe against a running node.
+
+```sh
+sudo -u pmtiles-swarm python3 tools/resume-doctor.py \
+  -c /etc/pmtiles-swarm/swarm.config.json
+```
+
+**`resume-doctor.py`** reads your configuration, catalog, stored `.torrent`
+files and resume directory, and says for each archive what libtorrent will do on
+the next start and why — seed at once, stat its files, or re-hash the store. It
+applies libtorrent's own rules rather than a guess at them, and cites the file
+and line each one came from. It also reads the unit through `systemctl show`,
+does the arithmetic on every deadline that can cut a resume save short, and
+scans the journal for the failures that leave a trace.
+
+```sh
+sudo -u pmtiles-swarm python3 tools/resume-experiment.py
+```
+
+**`resume-experiment.py`** proves the behaviour on the libtorrent you actually
+have, rather than the one the documentation was written against. It builds a
+real torrent over a temporary file, saves real resume data, adds it again, and
+reports what happened. 2.0 and 2.1 differ enough to be worth the thirty seconds.
+
+Three things they exist to tell apart, because the first two look identical from
+outside and want opposite fixes:
+
+- **No resume data is not the problem.** An archive recorded complete is added
+  with `seed_mode`, and that claim stands on its own: libtorrent stats the files
+  and seeds, without hashing a byte. A missing resume file costs nothing.
+- **Stale resume data is the problem, and it is worse than none.** A resume file
+  holding a _partial_ bitfield cancels the `seed_mode` claim — libtorrent drops
+  it if a single piece is unset. The archive then comes back as a downloader
+  with every byte already on disk, and fetches again what it has. Deleting that
+  one file is the repair; with none, it seeds at once.
+- **A short or missing file is a genuine re-check.** `mismatching_file_size` is
+  the only thing here that really does mean re-hashing, and it is about the data
+  on disk rather than about resume data at all. libtorrent 2.x records no
+  mtimes, so nothing in this depends on a timestamp.
+
+Which archives carry stale resume data is not random, and the pattern is worth
+knowing before it looks like a second bug. An archive from a watched folder was
+read end to end before it was ever registered: it is complete from its first
+moment and no partial bitfield is ever written for it. An archive joined from a
+feed, a magnet or a peer is genuinely partial for hours, with partial resume
+data written for it every `resumeSaveIntervalSeconds` throughout. Only the
+second kind has anything to leave behind, so trouble concentrating there is this
+failure's ordinary shape rather than something the import path introduced.
+
+Two more things the doctor looks at, because they are the ones a snapshot alone
+gets wrong.
+
+**Whether the catalog still points at its own `.torrent` files.** Each entry
+records `torrentPath` as an absolute path, and restore reads that rather than
+recomputing it — so moving `dataDir` out of `/etc`, which this guide tells you
+to do, leaves every recorded path behind. Nothing warns, because a missing
+`.torrent` is not treated as an error: restore quietly falls back to the magnet.
+That fallback is the damage. A magnet carries no metadata, resume data does not
+carry it either, and so the archive waits on BEP 9 for a file list that only a
+peer can supply — from a swarm where this node is the origin. It sits at 0% in
+`downloading_metadata` indefinitely, which no amount of re-checking will fix.
+
+**Whether the periodic save gets through the whole library.** Every torrent is
+asked to save in the same cycle, so a healthy `resumeDir` has all its files
+within one interval of each other. Ages spread wider than that mean some cycles
+are finishing early, and the ages alone cannot say whether the same archives
+lose every time. `--watch` measures it rather than inferring it:
+
+```sh
+sudo -u pmtiles-swarm python3 tools/resume-doctor.py \
+  -c /etc/pmtiles-swarm/swarm.config.json --watch 660
+```
+
+It reports each save cycle as it happens, how many of them covered every
+archive, and which archives were never written at all.
+
+**A poisoned archive may heal itself, so run this twice.** The commonest way a
+whole archive acquires a partial bitfield is a save taken while it was
+re-checking: `write_resume_data` truncates `have_pieces` to the pieces checked
+so far, and the five-minute timer lands inside that window every time a large
+archive checks. That shortened bitfield is what cancels `seed_mode` on the next
+start — so the archive comes back partial, checks again, and writes another one.
+The loop needs no restart to keep itself going.
+
+The doctor tells the two apart by length: a bitfield shorter than the torrent
+was written mid-check, and one at full length is what a finished check
+concluded. A check that completes rewrites its own resume file correctly, so an
+archive poisoned in one run can read `seeds at once` in the next with nothing
+done to it. **Only the archives that stay poisoned across two runs, several
+minutes apart, are worth deleting a file for.**
+
+Two things it deliberately does not treat as faults. An archive whose data sits
+outside the `savePathLayout` shape is normal — one adopted from a file this node
+already holds keeps that file, and a node may keep archives beside whatever
+produced them. What matters is whether the data is where the entry says, which
+is checked directly. And every directory an archive actually lives in has to
+appear in `ReadWritePaths`, not just the `savePath` in the configuration; the
+doctor checks each one in use, because a deliberately-placed archive is exactly
+the case that gets left out of the unit.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.58.1",
+  "version": "0.59.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",
@@ -46,7 +46,7 @@
     "maplibre-gl": "^6.2.0",
     "parse-torrent": "^11.0.24",
     "pmtiles": "^4.4.1",
-    "pmtiles-torrent": "^0.9.1",
+    "pmtiles-torrent": "^0.10.2",
     "webtorrent": "^3.0.21"
   },
   "engines": {

package/src/config.js

@@ -507,12 +507,52 @@ function clone(value) {
   return value;
 }
 
+/**
+ * Every directory this configuration means to write to.
+ *
+ * Which is exactly what `ReadWritePaths` has to name. Under
+ * `ProtectSystem=strict` a directory missing from that line is refused inside
+ * the unit's namespace, before any permission bit is consulted — so it fails
+ * with ownership and mode both perfect, which is a hard thing to go looking
+ * for. Deriving the list from the config rather than writing it out by hand is
+ * the only way the two cannot drift.
+ *
+ * See docs/running-as-a-service.md — "Where it writes".
+ * @param {object} config - A config whose paths have been resolved.
+ * @param {string} [configPath] - The config file, whose directory is written to.
+ * @returns {string[]} - Absolute directories, deduplicated and shortest-first.
+ */
+export function writablePaths(config, configPath) {
+  const found = [
+    config?.dataDir,
+    config?.savePath,
+    config?.cacheSavePath,
+    config?.libtorrent?.resumeDir,
+    config?.torrentDropDir,
+    // The console rewrites the configuration when a token is minted, so the
+    // directory holding it is written to as surely as any of the above.
+    configPath ? path.dirname(path.resolve(configPath)) : undefined,
+    ...(config?.watch ?? []).map((entry) => entry?.path),
+    ...(config?.locations ?? []).map((entry) => entry?.path),
+    ...(config?.subscriptions ?? []).map((entry) => entry?.savePath),
+  ].filter((value) => typeof value === 'string' && value);
+
+  // Deduplicated but deliberately not collapsed into common ancestors. A
+  // shorter list would grant the same access — `ReadWritePaths` covers a
+  // directory and everything under it — but the two callers want different
+  // things from it, and only one of them would be served. Creating the
+  // directories needs each of them named; and collapsing quietly widens the
+  // grant to whatever ancestor happens to be shared, which for a config beside
+  // its data is the whole tree above both.
+  return [...new Set(found.map((value) => path.resolve(value)))].sort();
+}
+
 /**
  * Complains about state that has landed in /etc.
  *
  * Nobody chooses this. The documented service layout puts the config file in
- * /etc, every path resolves relative to that file, and the sample reads
- * "./data" — so the catalog and the resume directory end up on the partition
+ * /etc, every path resolves relative to that file, and the sample read
+ * "./data" — so the catalog and the resume directory ended up on the partition
  * meant for configuration. Warned rather than corrected: it is a real path
  * that works, and moving a running node's data would be worse than saying so.
  *

package/src/engines/composite.js

@@ -433,7 +433,7 @@ export class CompositeEngine {
    * @returns {Promise<{written: number, asked: number}>} - Totals across every
    *   engine that keeps resume data.
    */
-  async saveResume(infoHash) {
+  async saveResume(infoHash, options = {}) {
     // Summed rather than dropped, so a caller can tell the difference between
     // "every torrent wrote" and "half of them will be re-hashed on the next
     // start". An engine that fails outright contributes nothing to either
@@ -444,7 +444,7 @@ export class CompositeEngine {
       // WebTorrent keeps none, and says so by not offering the method.
       if (!engine.saveResume) continue;
       try {
-        const result = (await engine.saveResume(infoHash)) ?? {};
+        const result = (await engine.saveResume(infoHash, options)) ?? {};
         written += Number(result.written) || 0;
         asked += Number(result.asked) || 0;
       } catch (error) {
@@ -653,19 +653,21 @@ export class CompositeEngine {
    * Shuts every engine down.
    * @returns {Promise<void>} - Resolves once all are stopped.
    */
-  async destroy() {
+  async destroy(options = {}) {
     this.#stopping = true;
     if (this.#timer) clearInterval(this.#timer);
     this.#timer = undefined;
 
     for (const engine of this.#secondaries) {
       await engine
-        .destroy()
+        .destroy(options)
         .catch((error) =>
           console.error(`[engine] ${engine.name}: ${error.message}`),
         );
     }
-    await this.#primary.destroy();
+    // Forwarded, because the primary is the one holding resume data and the
+    // budget was worked out from how much of it there is to write.
+    await this.#primary.destroy(options);
   }
 }
 

package/src/engines/libtorrent.js

@@ -763,25 +763,40 @@ export class LibtorrentEngine {
    *   were told to write resume data, and how many actually did before the
    *   deadline.
    */
-  async saveResume(infoHash) {
+  async saveResume(infoHash, options = {}) {
     // Returned rather than discarded. The sidecar reports both numbers, and
     // the gap between them is the thing worth knowing: a torrent that did not
     // write is one that gets re-hashed on the next start, which for a 700 GiB
     // archive is the difference between seeding in seconds and seeding in half
     // an hour. That answer was being thrown away here.
-    return this.#call('save_resume', { infoHash });
+    //
+    // The default #call timeout is 60s and the sidecar's own budget is two
+    // seconds per torrent, so past thirty archives the call gave up first —
+    // and then the counts never came back, which is why the shortfall this
+    // reports could not be seen from outside.
+    return this.#call('save_resume', { infoHash }, options.timeoutMs);
   }
 
   /**
    * Saves resume data and stops the sidecar.
+   *
+   * The timeout is the caller's to set, because only the caller knows how many
+   * torrents are about to be written down and the sidecar spends two seconds
+   * per torrent. Fifteen seconds was the fixed value, which meant every
+   * library past seven archives had its resume save cut off — and each torrent
+   * that missed was re-hashed in full on the way back up.
+   * @param {object} [options] - Overrides.
+   * @param {number} [options.timeoutMs] - How long the sidecar gets to finish.
    * @returns {Promise<void>} - Resolves once stopped.
    */
-  async destroy() {
+  async destroy(options = {}) {
     // Set before anything else, so the exit this is about to cause is
     // recognised as intended by the handler that sees it.
     this.#stopping = true;
     if (!this.#child) return;
-    await this.#call('shutdown', {}, 15000).catch(() => {});
+    await this.#call('shutdown', {}, options.timeoutMs ?? 15000).catch(
+      () => {},
+    );
     this.#child?.kill();
     this.#child = null;
     this.#ready = null;

package/src/index.js

@@ -16,7 +16,12 @@ import { assertPortsFree, claimDataDir } from './lock.js';
 import { ProgramHooks } from './hooks.js';
 import { SpeedLimits } from './rate-limits.js';
 import { SeedingLimits } from './seeding.js';
-import { closeServer, installSignalHandlers, runStoppers } from './shutdown.js';
+import {
+  closeServer,
+  engineStopMs,
+  installSignalHandlers,
+  runStoppers,
+} from './shutdown.js';
 import { ScheduledSourceManager } from './sources.js';
 import { SubscriptionManager } from './subscriptions.js';
 import { TileStats } from './tile-stats.js';
@@ -100,6 +105,12 @@ async function main() {
       port: { type: 'string', short: 'p' },
       help: { type: 'boolean', short: 'h' },
       json: { type: 'boolean' },
+      'data-dir': { type: 'string' },
+      'save-path': { type: 'string' },
+      force: { type: 'boolean' },
+      systemd: { type: 'boolean' },
+      user: { type: 'string' },
+      password: { type: 'string' },
     },
     allowPositionals: true,
   });
@@ -109,10 +120,17 @@ async function main() {
 
 Usage:
   pmtiles-swarm [--config FILE]          start the node
+  pmtiles-swarm init [--config FILE]     write a first configuration
   pmtiles-swarm status [--config FILE]   ask a running node what it is doing
   pmtiles-swarm publisher-key            print a new BEP 46 signing key
 
   --config, -c   path to a JSON config file
+  --data-dir     where state goes, for init. Absolute, and never under /etc
+  --save-path    where archive data goes, for init
+  --systemd      also write a unit file, for init
+  --user         service account the unit runs as. Default pmtiles-swarm
+  --password     console password, stored hashed. For init
+  --force        let init replace a config that already exists
   --port,   -p   override the listen port
   --json         machine-readable output, for the status command
   --help,   -h   this message
@@ -124,6 +142,21 @@ PMTILES_SWARM_PUBLIC_URL
     return;
   }
 
+  // Ahead of loadConfig, which is the point: there is no config to load yet.
+  if (positionals[0] === 'init') {
+    const { runInit } = await import('./init-command.js');
+    process.exitCode = await runInit({
+      config: values.config,
+      dataDir: values['data-dir'],
+      savePath: values['save-path'],
+      systemd: values.systemd,
+      user: values.user,
+      password: values.password,
+      force: values.force,
+    });
+    return;
+  }
+
   const config = await loadConfig(values.config);
   if (values.port) config.port = Number(values.port);
 
@@ -211,7 +244,19 @@ PMTILES_SWARM_PUBLIC_URL
   const engine = createEngine(config);
   // The slowest, because it announces "stopped" to every tracker, and an
   // unreachable one costs a timeout each. Registered first so it stops last.
-  stoppers.unshift({ label: 'engine', stop: () => engine.destroy(), ms: 8000 });
+  //
+  // Scaled to the library, because what dominates this step is writing resume
+  // data and the sidecar gives each torrent two seconds of that budget. A flat
+  // eight seconds was under it for any node past four archives, so the node
+  // abandoned the sidecar mid-write on every stop and re-hashed on the way
+  // back up whatever had not been persisted — which is how a library ends up
+  // permanently checking rather than seeding.
+  const resumeStopMs = engineStopMs(catalog.list().length);
+  stoppers.unshift({
+    label: 'engine',
+    stop: () => engine.destroy({ timeoutMs: resumeStopMs }),
+    ms: resumeStopMs + 2000,
+  });
   try {
     await engine.connect();
     console.log(`[engine] ${engine.name} ready`);
@@ -528,7 +573,12 @@ PMTILES_SWARM_PUBLIC_URL
   if (resumeSeconds > 0 && engine.saveResume) {
     resumeTimer = setInterval(() => {
       engine
-        .saveResume()
+        // Recomputed each tick rather than captured, because the library grows:
+        // a node that started with four archives and watched forty in would
+        // otherwise still be allowing the budget for four.
+        .saveResume(undefined, {
+          timeoutMs: engineStopMs(catalog.list().length),
+        })
         .then((result) => {
           // Said out loud, because the shortfall is the thing that costs.
           // A torrent that did not write inside the deadline is one that gets

package/src/init-command.js

@@ -0,0 +1,231 @@
+import crypto from 'node:crypto';
+import fs from 'node:fs/promises';
+import os from 'node:os';
+import path from 'node:path';
+import { hashPassword } from './auth.js';
+import { writablePaths } from './config.js';
+import { directoryCommands, unitFor } from './systemd.js';
+
+/**
+ * Where state goes when nobody said.
+ *
+ * Never beside the config file, which is the whole point of this command: a
+ * path relative to a config in /etc puts a catalog and an archive on the
+ * partition meant for configuration. See docs/running-as-a-service.md.
+ * @returns {string} - An absolute directory.
+ */
+function defaultDataDir() {
+  const service =
+    process.platform !== 'win32' &&
+    typeof process.getuid === 'function' &&
+    process.getuid() === 0;
+  return service
+    ? '/var/lib/pmtiles-swarm/data'
+    : path.join(process.cwd(), 'data');
+}
+
+/**
+ * The configuration a new node starts from.
+ * @param {object} paths - Resolved dataDir, savePath and any password hash.
+ * @returns {object} - The config to write.
+ */
+function firstConfig({ dataDir, savePath, passwordHash }) {
+  return {
+    port: 8090,
+    adminPort: 8091,
+    adminHost: '127.0.0.1',
+
+    dataDir,
+    savePath,
+    savePathLayout: 'infohash',
+
+    engine: 'libtorrent',
+    secondaryEngines: ['webtorrent'],
+    libtorrent: {
+      python: 'python3',
+      resumeDir: path.join(dataDir, 'resume'),
+      listen: '0.0.0.0:6881',
+      upnp: true,
+      natpmp: true,
+    },
+    webtorrent: { clientOptions: { torrentPort: 6882 } },
+
+    torrentFormat: 'hybrid',
+    pieceLength: 4194304,
+    maxConnections: 100,
+    trackers: [
+      'udp://tracker.opentrackr.org:1337/announce',
+      'udp://tracker.torrent.eu.org:451/announce',
+      'udp://tracker-udp.gbitt.info:80/announce',
+      'wss://tracker.openwebtorrent.com',
+      'wss://tracker.webtorrent.dev',
+    ],
+
+    auth: {
+      username: 'admin',
+      // Generated rather than left as a placeholder. A sample saying
+      // REPLACE-ME is a sample that ships unreplaced.
+      apiKey: crypto.randomBytes(32).toString('hex'),
+      // Hashed here, never stored as the plaintext that was typed. A password
+      // key is written only when one was given: the same reasoning as the API
+      // key, one step further on. A placeholder would be a credential that
+      // looks set and is not, and `auth.password` accepts plaintext — so
+      // "REPLACE-ME" in that field is a working password until somebody
+      // notices.
+      ...(passwordHash ? { passwordHash } : {}),
+      tokens: [],
+    },
+  };
+}
+
+/**
+ * Writes a first configuration file.
+ *
+ * Everything it writes is absolute. A path that resolves against the config
+ * file is the one mistake this command exists to make impossible: the
+ * documented layout puts that file in /etc, so `./data` there means a catalog,
+ * a resume directory and potentially a 700 GiB archive on the configuration
+ * partition — and the person who followed both documents did nothing wrong.
+ *
+ * State under /etc is refused rather than warned about. At the moment a config
+ * is written there is nothing to preserve and nothing to migrate, so refusing
+ * costs a retyped flag; the same mistake found later costs a stopped service
+ * and a careful move.
+ * @param {object} [options] - Flags from the command line.
+ * @param {Function} [write] - Where to report, for tests.
+ * @returns {Promise<number>} - Exit code.
+ */
+export async function runInit(options = {}, write = console.log) {
+  const configPath = path.resolve(
+    options.config ?? path.join(process.cwd(), 'swarm.config.json'),
+  );
+  const dataDir = path.resolve(options.dataDir ?? defaultDataDir());
+  const savePath = path.resolve(
+    options.savePath ?? path.join(dataDir, 'torrents-data'),
+  );
+
+  // Both what was typed and what it resolved to: on Windows the first is the
+  // only one that can be recognised, and on POSIX the second catches a
+  // relative path that lands there anyway.
+  for (const [name, ...values] of [
+    ['--data-dir', options.dataDir, dataDir],
+    ['--save-path', options.savePath, savePath],
+  ]) {
+    const value = values.find(
+      (candidate) =>
+        typeof candidate === 'string' && candidate.startsWith('/etc/'),
+    );
+    if (value) {
+      write(
+        `${name} is ${value}. /etc is for configuration; state belongs under ` +
+          '/var/lib or a data disk. Nothing has been written.',
+      );
+      return 1;
+    }
+  }
+
+  const exists = await fs
+    .access(configPath)
+    .then(() => true)
+    .catch(() => false);
+  if (exists && !options.force) {
+    write(
+      `${configPath} already exists. Pass --force to replace it — and take a ` +
+        'copy first, since the tokens in it are not recoverable.',
+    );
+    return 1;
+  }
+
+  await fs.mkdir(path.dirname(configPath), { recursive: true });
+  const config = firstConfig({
+    dataDir,
+    savePath,
+    passwordHash: options.password ? hashPassword(options.password) : undefined,
+  });
+  await fs.writeFile(configPath, `${JSON.stringify(config, null, 2)}${os.EOL}`);
+
+  write(`Wrote ${configPath}`);
+  write('');
+  write(`  dataDir   ${dataDir}`);
+  write(`  savePath  ${savePath}`);
+  write(`  resumeDir ${config.libtorrent.resumeDir}`);
+  write('');
+
+  const unitPath = path.join(path.dirname(configPath), 'pmtiles-swarm.service');
+  if (options.systemd) {
+    // Written beside the configuration rather than into /etc/systemd/system.
+    // Installing a unit is a privileged, system-changing act and this command
+    // may be run by anyone; the copy is one line and it is the caller's to
+    // make.
+    await fs.writeFile(
+      unitPath,
+      unitFor({
+        config,
+        configPath,
+        user: options.user,
+        execStart: options.execStart,
+      }),
+    );
+    write(`Wrote ${unitPath}`);
+    write('');
+    write('  Its ReadWritePaths was derived from the configuration above, so');
+    write('  the two cannot disagree. Adding a watched folder later means');
+    write('  re-running this, or adding the folder to that line by hand.');
+    if (path.sep !== '/') {
+      // Said rather than refused: writing the unit is still the fastest way to
+      // see its shape, and somebody may be preparing a config to carry across.
+      // But every path in it is this machine's, and systemd will not read them.
+      write('');
+      write('  Written on a platform that is not the one this runs on, so the');
+      write('  paths in it are this machine’s. Re-run init on the server, or');
+      write('  rewrite every path in the unit before installing it.');
+    }
+    write('');
+  }
+
+  write('Next:');
+  write('');
+  for (const command of directoryCommands({
+    config,
+    configPath,
+    user: options.user,
+  })) {
+    write(`  ${command}`);
+  }
+  write('');
+
+  if (options.systemd) {
+    write(`  sudo cp ${unitPath} /etc/systemd/system/`);
+    write('  sudo systemctl daemon-reload');
+    write('  sudo systemctl enable --now pmtiles-swarm');
+    write('');
+  }
+
+  if (!config.auth.passwordHash) {
+    write('  There is no console password. The API key in the file is the way');
+    write('  in until you set one: re-run with --password, or add a plaintext');
+    write('  auth.password, which is hashed the first time it is read.');
+    write('');
+  }
+
+  write('  Point a watch folder at where your archives are built, or add one');
+  write('  from the console. A watched folder is a directory this writes to,');
+  write(
+    options.systemd
+      ? '  so it has to go in ReadWritePaths as well.'
+      : '  so it has to go in the unit ReadWritePaths as well.',
+  );
+
+  if (!options.systemd) {
+    // Still printed for anyone writing the unit by hand, which is the case
+    // this was for before there was a generator. Derived from the same place
+    // the generated one is, so the advice cannot be worse than the file.
+    write('');
+    write('Under systemd every one of those has to be in ReadWritePaths:');
+    write('');
+    write(`  ReadWritePaths=${writablePaths(config, configPath).join(' ')}`);
+    write('');
+    write('Re-run with --systemd to have that written for you, as a unit.');
+  }
+  return 0;
+}

package/src/library.js

@@ -667,16 +667,77 @@ export class Library {
    * @param {object} entry - Catalog entry.
    * @returns {Promise<void>} - Resolves once added.
    */
-  async #readd(entry) {
-    const torrentFile = entry.torrentPath
-      ? await fs
-          .readFile(entry.torrentPath)
-          .then((buffer) => new Uint8Array(buffer))
-          .catch(() => null)
+  /**
+   * An entry's metainfo, from wherever it actually is.
+   *
+   * `torrentPath` is recorded absolute, so moving `dataDir` — which the service
+   * guide tells you to do, out of `/etc` — leaves every entry in the catalog
+   * naming a directory that no longer exists. Nothing repoints them, and
+   * nothing complained, because an unreadable `.torrent` was simply treated as
+   * "use the magnet instead".
+   *
+   * So the recorded path is tried first and the current `dataDir` second. The
+   * file is stored under the infohash either way, which is what makes the
+   * second lookup possible at all.
+   * @param {object} entry - The catalog entry.
+   * @returns {Promise<Uint8Array | null>} - The metainfo, or null.
+   */
+  async #metainfoFor(entry) {
+    const read = (file) =>
+      file
+        ? fs
+            .readFile(file)
+            .then((buffer) => new Uint8Array(buffer))
+            .catch(() => null)
+        : Promise.resolve(null);
+
+    const recorded = await read(entry.torrentPath);
+    if (recorded) return recorded;
+
+    const beside = entry.infoHash
+      ? path.join(this.torrentDir, `${entry.infoHash}.torrent`)
       : null;
+    if (!beside || beside === entry.torrentPath) return null;
+
+    const found = await read(beside);
+    if (!found) return null;
+
+    console.warn(
+      `[restore] ${entry.name}: the catalog names ${entry.torrentPath}, which ` +
+        `is not there; using ${beside} instead. Its recorded path is being ` +
+        'corrected.',
+    );
+    // Corrected in place, so the warning is printed once rather than on every
+    // start for the rest of the node's life.
+    await this.#catalog
+      .put({ infoHash: entry.infoHash, torrentPath: beside })
+      .catch(() => null);
+    return found;
+  }
+
+  async #readd(entry) {
+    const torrentFile = await this.#metainfoFor(entry);
     const magnet = torrentFile ? undefined : this.#withTrackers(entry);
     if (!torrentFile && !magnet) return;
 
+    // Falling back to the magnet is not a quiet equivalent, and it used to
+    // happen silently. A magnet carries no metadata and neither does resume
+    // data, so the archive waits on BEP 9 for a file list — which no peer can
+    // supply for an archive this node originated. It sits at 0% in
+    // downloading_metadata indefinitely, and no re-check will move it.
+    //
+    // Only where the entry claims a metainfo it cannot produce. An archive
+    // that never had one was joined by magnet and is waiting on
+    // captureMetadata in the ordinary way; saying this about it would be a
+    // warning on the normal path.
+    if (!torrentFile && entry.torrentPath) {
+      console.warn(
+        `[restore] ${entry.name}: no .torrent could be read, so it is being ` +
+          'added as a magnet and must fetch its metadata from a peer. If this ' +
+          'node is the only seeder, it will not finish.',
+      );
+    }
+
     await this.#engine.add({
       torrentFile: torrentFile ?? undefined,
       magnet,

package/src/shutdown.js

@@ -18,8 +18,53 @@
  * every step is bounded, and the whole sequence is bounded again behind that.
  */
 
-/** How long the whole shutdown may take before it gives up on itself. */
-const WATCHDOG_MS = 15000;
+/** Slack over the sum of the steps, for the shutdown's own bookkeeping. */
+const WATCHDOG_SLACK_MS = 5000;
+
+/** What a step is allowed when it does not say. */
+const STEP_MS = 5000;
+
+/**
+ * How long the whole shutdown may take before it gives up on itself.
+ *
+ * Derived from the steps rather than fixed, because a fixed bound is a second
+ * deadline that has to be kept in agreement with them by hand — and was not.
+ * Fifteen seconds sat under an engine step that legitimately wants two seconds
+ * per torrent, so on any library past seven archives the watchdog fired first
+ * and exited the process while the sidecar was still writing resume data. What
+ * that costs is a re-hash of everything that had not been written yet, on the
+ * way back up.
+ *
+ * Computed when the signal arrives, so a step registered later is still
+ * covered.
+ * @param {Array<{ms?: number}>} stoppers - The steps to be run.
+ * @returns {number} - The bound, in milliseconds.
+ */
+export function watchdogFor(stoppers) {
+  const total = stoppers.reduce(
+    (sum, stopper) => sum + (stopper.ms ?? STEP_MS),
+    0,
+  );
+  return total + WATCHDOG_SLACK_MS;
+}
+
+/**
+ * How long to let the engine stop, given how much it has to write down.
+ *
+ * The sidecar allows each torrent two seconds of its resume-save budget, so
+ * the only bound that does not eventually cut a library short is one that
+ * counts them. The floor covers a node with nothing in it, where what remains
+ * is announcing "stopped" to trackers that may not answer.
+ *
+ * Whatever this returns has to be under the unit's `TimeoutStopSec`, or
+ * systemd kills the process while it is still working. See
+ * docs/running-as-a-service.md.
+ * @param {number} archives - How many archives the catalog holds.
+ * @returns {number} - Milliseconds.
+ */
+export function engineStopMs(archives) {
+  return Math.max(15000, archives * 2000 + 10000);
+}
 
 /**
  * Runs one shutdown step, giving up on it rather than waiting for ever.
@@ -125,11 +170,14 @@ export function installSignalHandlers(stoppers, options = {}) {
     stopping = true;
     console.log(`\n[shutdown] ${signal}`);
 
-    // A last resort, in case a step ignores its own bound.
+    // A last resort, in case a step ignores its own bound — never a bound in
+    // its own right, which is what it silently became when it was shorter than
+    // the steps it was meant to outlast.
+    const limitMs = watchdogFor(stoppers);
     const watchdog = setTimeout(() => {
-      console.warn('[shutdown] took too long; exiting anyway');
+      console.warn(`[shutdown] took longer than ${limitMs}ms; exiting anyway`);
       exit(1);
-    }, WATCHDOG_MS);
+    }, limitMs);
 
     await runStoppers(stoppers);
 

package/src/systemd.js

@@ -0,0 +1,158 @@
+import path from 'node:path';
+import { writablePaths } from './config.js';
+
+/**
+ * A unit file written from the configuration it will run.
+ *
+ * Every failure this project has diagnosed under systemd came from the unit
+ * and the configuration disagreeing, and none of them looked like what they
+ * were:
+ *
+ *   - a `savePath` missing from `ReadWritePaths` fails inside the unit's
+ *     namespace, before any permission bit is read, so the directory's owner
+ *     and mode are both perfect and the write is still refused
+ *   - the default `KillMode` signals the Python sidecar at the same instant as
+ *     the node, so the sidecar dies before it can write resume data and every
+ *     archive returns at 0% to be re-checked
+ *   - a `TimeoutStopSec` shorter than the resume save kills the process
+ *     mid-write, which costs the same re-check by a different route
+ *
+ * None of that is discoverable from the symptom. All of it is decidable from
+ * the configuration, which is why this is generated rather than documented.
+ */
+
+/** What the resume save can want, before the library has grown into it. */
+const TIMEOUT_STOP_SECONDS = 300;
+
+/**
+ * The unit file for a configuration.
+ * @param {object} options - What to write.
+ * @param {object} options.config - The resolved configuration.
+ * @param {string} options.configPath - Where that configuration lives.
+ * @param {string} [options.user] - Service account. Default 'pmtiles-swarm'.
+ * @param {string} [options.execStart] - The binary. Default the documented install.
+ * @param {string} [options.workingDirectory] - Default the binary's install root.
+ * @returns {string} - The unit, ready to install.
+ */
+export function unitFor({
+  config,
+  configPath,
+  user = 'pmtiles-swarm',
+  execStart,
+  workingDirectory,
+}) {
+  const home = workingDirectory ?? `/var/lib/${user}`;
+  const binary = execStart ?? `${home}/node_modules/.bin/pmtiles-swarm`;
+  const paths = writablePaths(config, configPath);
+
+  // Wrapped the way systemd's own examples are, because this list grows with
+  // every watched folder and a single line of them is unreadable in a diff.
+  const readWrite = paths.join(' \\\n  ');
+
+  return `# pmtiles-swarm, generated by \`pmtiles-swarm init --systemd\`.
+#
+# Install it, then check what systemd actually merged — which is the only
+# account of it worth trusting:
+#
+#   sudo cp pmtiles-swarm.service /etc/systemd/system/
+#   sudo systemctl daemon-reload
+#   sudo systemctl enable --now pmtiles-swarm
+#   systemctl show -p ReadWritePaths -p KillMode -p TimeoutStopUSec pmtiles-swarm
+#
+# ReadWritePaths below was derived from ${path.basename(configPath)}. Adding a
+# watched folder, a subscription with its own save path, or a cache directory
+# to that file means adding it here too — or the write is refused with the
+# directory's permissions perfect. Re-running init --systemd rewrites this.
+
+[Unit]
+Description=pmtiles-swarm
+Documentation=https://github.com/TechIdiots-LLC/pmtiles-swarm
+After=network-online.target
+Wants=network-online.target
+
+[Service]
+Type=simple
+User=${user}
+Group=${user}
+
+WorkingDirectory=${home}
+
+# Absolute: systemd reads no shell profile.
+ExecStart=${binary} \\
+  --config ${path.resolve(configPath)}
+
+# Required, not a preference. The console's Save & Restart applies settings a
+# running process cannot take, so it exits 0 and expects to be brought back;
+# Restart=on-failure ignores an exit 0 and would leave the node stopped with
+# the unit reporting success.
+Restart=always
+RestartSec=5
+
+# Stopping writes resume data for every archive, and the node allows its engine
+# two seconds per torrent to do it. This has to outlast that: killed mid-save,
+# every archive that had not been written re-hashes its whole store on the way
+# back up, which for a 700 GiB archive is hours. Raise it as the library grows
+# — tools/resume-doctor.py prints the arithmetic for yours.
+TimeoutStopSec=${TIMEOUT_STOP_SECONDS}
+
+# The node stops the sidecar itself, and needs it alive to do so. The default,
+# control-group, signals both at once and the sidecar dies before it can write
+# anything. That is what a library returning at 0% after every restart is.
+KillMode=mixed
+
+# A seeding node holds a socket per peer, and the tile reader holds file
+# descriptors of its own.
+LimitNOFILE=65535
+
+# Everything not named below is read-only inside this unit's namespace.
+ProtectSystem=strict
+ProtectHome=read-only
+PrivateTmp=true
+NoNewPrivileges=true
+
+ReadWritePaths=${readWrite}
+
+# Group-writable, so what this service creates in a folder shared with whatever
+# builds the archives can still be modified by it.
+UMask=0002
+
+[Install]
+WantedBy=multi-user.target
+`;
+}
+
+/**
+ * The commands that make the directories this unit needs, owned correctly.
+ *
+ * Printed rather than run. init may be running as root or as a person, and a
+ * directory created by the wrong one is the same refused write as a missing
+ * `ReadWritePaths` entry — with even less to go on, because the unit is right.
+ * @param {object} options - What to write.
+ * @param {object} options.config - The resolved configuration.
+ * @param {string} options.configPath - Where that configuration lives.
+ * @param {string} [options.user] - Service account.
+ * @returns {string[]} - Shell commands, in the order they should be run.
+ */
+export function directoryCommands({
+  config,
+  configPath,
+  user = 'pmtiles-swarm',
+}) {
+  // The config's own directory is excluded: it exists already, since the file
+  // was just written into it, and it is usually /etc where this would be wrong.
+  const configDir = path.dirname(path.resolve(configPath));
+  const wanted = writablePaths(config, configPath).filter(
+    (value) => value !== configDir,
+  );
+
+  return [
+    `sudo useradd --system --home-dir /var/lib/${user} --shell /usr/sbin/nologin ${user}`,
+    ...wanted.map(
+      (value) => `sudo install -d -o ${user} -g ${user} -m 0775 ${value}`,
+    ),
+    // The directory as well as the file. The console rewrites the
+    // configuration by writing a temp file beside it and renaming, which needs
+    // write permission on the directory rather than on the file.
+    `sudo chown ${user} ${configDir} ${path.resolve(configPath)}`,
+  ];
+}