pmtiles-swarm 0.4.3 → 0.4.5

package/CHANGELOG.md

@@ -7,6 +7,39 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.4.5
+### 🐞 Bug fixes
+- **A hook whose command could not be started is tried again.** Completion is recorded before
+  the command runs, so that a six-hour build is not started six times over — but a command that
+  never launched has not started anything, and keeping the record meant fixing the path and
+  still never seeing it run. The archive was permanently, silently done. A failure to spawn now
+  hands the record back; a command that ran and failed keeps it, because retrying that every
+  minute is how a broken build becomes a broken loop. A spawn failure also raised two accounts
+  of itself on some platforms — the real error, then a nonsense exit code — and only the first
+  stands now.
+
+### 📚 Documentation
+- **The service guide is organised around the thing that actually costs an afternoon.**
+  Permissions were spread across three sections and `ReadWritePaths` was explained twice, in
+  neither place completely. There is now one **Where it writes** section built on the fact that
+  three separate things decide whether a write succeeds — the filesystem bits, the group the
+  process actually holds, and `ReadWritePaths` — that each refuse on their own and all fail
+  identically. It also covers creating the archive directory, which was never mentioned even
+  though `savePath` is the entry most often missing from `ReadWritePaths`; why `chmod -R` is
+  the wrong tool, since on a directory the execute bit is the search bit; `SupplementaryGroups=`
+  for when a group will not appear; and that `PrivateTmp=true` hides a hook's lock and log.
+
+## 0.4.4
+### 🐞 Bug fixes
+- **The console no longer claims an `.incomplete` file that is not there.** The marker was a
+  literal in the page, drawn beside every unfinished archive, with a tooltip naming the file it
+  was supposedly on disk as. libtorrent renames nothing — the rename would have to happen in the
+  sidecar — so on the engine most people run, that named a file which did not exist, next to one
+  sitting under its final name at 25% downloaded. Each engine now says whether it marks
+  incomplete files, the composite answers for its primary since that is the engine writing the
+  bytes, and `/api/status` combines that with `incompleteSuffix` — which can also be empty — to
+  report the marker actually in use, or none. The console draws only what it is told.
+
 ## 0.4.3
 ### 🐞 Bug fixes
 - **The lock file keeps the optional native builds `ws` asks for.** `bufferutil` and

package/docs/running-as-a-service.md

@@ -1,8 +1,14 @@
 # Running as a systemd service
 
-Setting up the account, then the unit. Two lines in that unit are not optional,
-and one line most people copy from elsewhere should be deleted — the rest is
-ordinary.
+Setting up the account, then the unit, then the directories it writes to.
+
+Two lines in that unit are not optional and one line most people copy from
+elsewhere should be deleted, but neither is what costs the afternoon. That is
+**permission to write**, which here means three separate things that all have to
+agree: the filesystem bits, the group the process actually holds, and
+`ReadWritePaths`. Any one of them says no on its own, and the failure looks the
+same each time — so [Where it writes](#where-it-writes) is worth reading before
+the first archive rather than after.
 
 ## An account of its own
 
@@ -46,6 +52,24 @@ Generate the key rather than inventing one:
 node -e "console.log(require('crypto').randomBytes(32).toString('base64url'))"
 ```
 
+### Where the archives go
+
+Wherever `savePath` points, and it is rarely under `/var/lib` — archives are
+measured in hundreds of gigabytes and usually live on their own mount. Make it
+before the first download, owned by the service:
+
+```sh
+sudo install -d -o pmtiles-swarm -g pmtiles-swarm -m 0755 /mnt/store/torrent-data
+```
+
+`0755` rather than `0750` because it leaves the option of serving those files
+over HTTP later; `0750` if you would rather they stay private. Nothing in
+pmtiles-swarm depends on which you choose.
+
+A directory this account owns outright needs nothing further. One that another
+service also writes to is a different job — see
+[a folder shared with another service](#a-folder-shared-with-another-service).
+
 ## Node, and the package
 
 Node from your distribution or NodeSource — the package needs `^22.13.0 || 24`:
@@ -160,18 +184,27 @@ TimeoutStopSec=45
 # descriptors of its own.
 LimitNOFILE=65535
 
-# The archives and the sidecar are the only things it needs to touch.
+# Everything else is read-only inside this unit's namespace.
 ProtectSystem=strict
 ProtectHome=read-only
 PrivateTmp=true
 NoNewPrivileges=true
-# Both, since the console rewrites the configuration when a token is minted.
+
+# The starting pair: /var/lib for the data directory, /etc because the console
+# rewrites the configuration when a token is minted. Anywhere else the
+# configuration points — savePath above all — has to be added here or the write
+# is refused whatever its permissions say. See "Where it writes".
 ReadWritePaths=/var/lib/pmtiles-swarm /etc/pmtiles-swarm
 
 [Install]
 WantedBy=multi-user.target
 ```
 
+`PrivateTmp=true` is worth one more note, because it surprises people writing
+hooks: the service gets its own `/tmp`, so a script that keeps a lock or a log
+there is invisible from a normal shell, and a run started by hand cannot see the
+lock a hook run holds. Have hooks log somewhere real.
+
 ## The two lines that matter
 
 **`Restart=always`, not `on-failure`.** The console's *Save & Restart* applies
@@ -189,61 +222,95 @@ while the Python sidecar kept running and kept the data directory locked.
 
 Leave `KillMode` at its default, so the sidecar goes with its parent.
 
-## Paths
+## Where it writes
 
-Every path in the configuration resolves **relative to the configuration
-file**, not to the working directory:
+Three separate things decide whether the service can write to a directory, and
+**each can refuse on its own**. A permission problem, a group problem and a
+namespace problem all produce the same symptom — an operation that silently does
+nothing, or a hook that exits 1 with no output — so it is worth confirming all
+three rather than guessing between them.
+
+### 1. The paths in the configuration
+
+Every path resolves **relative to the configuration file**, not to the working
+directory:
 
 ```
 /etc/pmtiles-swarm/swarm.config.json     with "dataDir": "./data"
   -> /etc/pmtiles-swarm/data
 ```
 
-That is usually not what you want for a service. Use absolute paths for
-anything that holds data:
+That is rarely what you want for a service. Use absolute paths for anything
+holding data:
 
 ```json
 {
   "dataDir": "/var/lib/pmtiles-swarm",
-  "savePath": "/var/lib/pmtiles-swarm/archives",
+  "savePath": "/mnt/store/torrent-data",
   "libtorrent": { "resumeDir": "/var/lib/pmtiles-swarm/resume" }
 }
 ```
 
-`ProtectSystem=strict` makes the whole filesystem read-only apart from what
-`ReadWritePaths` names, so every one of those has to be listed. An archive
-directory on another mount needs its own entry.
+### 2. Every one of them in `ReadWritePaths`
+
+`ProtectSystem=strict` presents the whole filesystem as read-only inside the
+unit's namespace. The refusal happens **there, before any permission bit is
+consulted** — so a directory whose ownership and mode are perfect still fails if
+it is not named here.
+
+List every directory the configuration points at, plus anywhere a hook writes:
+
+```ini
+ReadWritePaths=/var/lib/pmtiles-swarm /etc/pmtiles-swarm   /mnt/store /mnt/work/planetiler
+```
+
+`ReadWritePaths=` is a list: repeated assignments **merge** rather than replace,
+whether in the unit itself or in a drop-in from `systemctl edit pmtiles-swarm`.
+So a drop-in adds to what the unit already names. An empty assignment on its own
+line is the only thing that resets it.
+
+The one to forget is `savePath`, because it is usually on another mount and
+nothing complains until a download starts — at which point a torrent that cannot
+write fails in a way that reads like a network problem.
+
+### 3. Permission on the directory itself
 
-## Sharing a folder with another service
+A directory this account owns needs nothing beyond
+[the setup above](#where-the-archives-go). A shared one does.
+
+### A folder shared with another service
 
 A folder produced by something else — a generation script, or a directory a
-torrent client already owns — needs three things, and group membership is only
-the first of them.
+torrent client already owns — takes three steps, and group membership is only
+the first:
 
 ```sh
 # 1. Put the service account in the owning group.
 sudo usermod -aG qbittorrent-nox pmtiles-swarm
 
 # 2. Give that group write, and setgid so new entries inherit it.
-sudo find /mnt/hd-16TB/store/generated -type d -exec chmod 2775 {} +
-sudo find /mnt/hd-16TB/store/generated -type f -exec chmod 664 {} +
+sudo find /mnt/store/generated -type d -exec chmod 2775 {} +
+sudo find /mnt/store/generated -type f -exec chmod 664 {} +
 
-# 3. Make what the service creates group-writable too.
-sudo systemctl edit pmtiles-swarm
-sudo systemctl restart pmtiles-swarm
+# 3. Make what this service creates group-writable too, then restart.
+sudo systemctl edit pmtiles-swarm     # [Service] / UMask=0002
+sudo systemctl daemon-reload && sudo systemctl restart pmtiles-swarm
 ```
 
-Step 3 opens an override; the two lines to put in it are:
+**A folder at 0755 gives the group `r-x`.** Membership alone buys read access and
+nothing else — enough to hash and seed an archive, not enough for anything that
+writes. So this looks like it worked right up until the first thing that does.
 
-```ini
-[Service]
-UMask=0002
-```
+The `2` in `2775` is setgid, and it is what stops this drifting: without it a
+file the service creates belongs to group `pmtiles-swarm`, the other service
+cannot touch it, and you are back here in a month. `UMask=0002` is the same
+thought for the mode — without it a new file is `0644` and the other account can
+delete it but not modify it.
 
-**A folder at 0755 gives the group `r-x`.** Membership alone buys read access
-and nothing else, which is enough to hash and seed an archive and not enough to
-do anything else with the folder — so this looks like it worked until the first
-thing that writes.
+Split by type rather than using `chmod -R`. On a **directory** the execute bit
+is the search bit: it permits resolving a path *through* the directory, so
+removing it leaves a folder whose contents you can list and not one of which you
+can open. Files should lose it; directories must not.
 
 Three features want write, and it is worth knowing which, because a read-only
 folder is a perfectly reasonable way to run:
@@ -254,33 +321,38 @@ folder is a perfectly reasonable way to run:
 | `keep`, `keepDays` | Deletes retired builds |
 | `onComplete` | Whatever the script does, since it runs as this account |
 
-Renaming and deleting need write on the **directory**, not on the file, which is
-why the directory bits are the ones that matter. `UMask=0002` matters for the
-other direction: without it a file the service creates is `0644`, and the other
-service can delete it but not modify it.
+Renaming and deleting need write on the **directory**, not on the file — which is
+why the directory bits are the ones that matter, and why a build written under a
+temporary name and renamed into place works with directory write alone.
+
+### Two checks that lie
+
+**`id pmtiles-swarm`** reads `/etc/group` and shows the new group the instant
+`usermod` returns, whether or not the running process has it. Supplementary
+groups are read when a process starts, so the restart is not optional — and this
+is what proves it:
+
+```sh
+grep -E '^(Uid|Gid|Groups)'   /proc/$(systemctl show -p MainPID --value pmtiles-swarm)/status
+getent group qbittorrent-nox      # is that GID in the Groups line above?
+```
 
-**Group membership is read when a process starts**, so the restart is not
-optional. Neither is `ReadWritePaths`: `ProtectSystem=strict` presents the rest
-of the filesystem as read-only inside the unit's namespace, and the write is
-refused there before the permission bits are consulted. Every folder outside
-`/var/lib/pmtiles-swarm` has to be named, in a drop-in from
-`systemctl edit pmtiles-swarm`:
+If it is missing even after a restart, name it outright rather than relying on
+how systemd resolves groups when `User=` and `Group=` are both set:
 
 ```ini
-[Service]
-ReadWritePaths=/mnt/store/generated /mnt/work/planetiler
-UMask=0002
+SupplementaryGroups=qbittorrent-nox
 ```
 
-`ReadWritePaths=` accumulates, so a drop-in adds to what the unit already lists
-rather than replacing it.
+**`sudo -u pmtiles-swarm touch …`** runs outside the unit's namespace, so it
+succeeds on permission bits alone while the service is still being refused by
+`ProtectSystem`. It can prove a permission problem; it cannot clear one.
 
-Two things that look like checks and are not. `id pmtiles-swarm` reads
-`/etc/group` and shows the new group the instant `usermod` returns, whether or
-not the running process has it — read `/proc/$(systemctl show -p MainPID --value
-pmtiles-swarm)/status` instead. And `sudo -u pmtiles-swarm touch …` runs outside
-the unit's namespace, so it succeeds on permissions alone while the service is
-still being refused.
+What the running service actually has:
+
+```sh
+systemctl show -p ReadWritePaths -p UMask -p SupplementaryGroups pmtiles-swarm
+```
 
 ## The sidecar
 
@@ -371,3 +443,21 @@ Then check it is actually serving:
 curl -fsS localhost:8090/feed.xml >/dev/null && echo "public surface ok"
 curl -fsS localhost:8091/api/status | head -c 200
 ```
+
+And that it can write where it is supposed to, which nothing above proves:
+
+```sh
+systemctl show -p ReadWritePaths -p UMask -p SupplementaryGroups pmtiles-swarm
+grep -E '^Groups' /proc/$(systemctl show -p MainPID --value pmtiles-swarm)/status
+```
+
+Two things that go wrong quietly rather than loudly, and are worth confirming
+once rather than diagnosing later:
+
+* An archive that re-hashes its whole store on every start means resume data is
+  not being written. There should be one file per torrent in `resumeDir` within
+  `resumeSaveIntervalSeconds` of a start.
+* A hook that never seems to run. It logs what it launched and why it stopped —
+  `journalctl -u pmtiles-swarm | grep -i onComplete` — and a hook redirecting its
+  own output to a file will have nothing for the journal to show, which is not
+  the same as not having run.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.4.3",
+  "version": "0.4.5",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -408,6 +408,14 @@ export function createApp({
       }
       res.json({
         version: VERSION,
+        // What an unfinished archive is actually called on disk, or null when
+        // nothing renames it. Both halves matter: the setting can be empty,
+        // and an engine can ignore it entirely. The console showed the marker
+        // for every unfinished archive regardless, which on libtorrent was a
+        // filename that did not exist.
+        incompleteMarker: engine.marksIncomplete
+          ? config.incompleteSuffix || null
+          : null,
         engine: { name: engine.name, ok: engineOk, error: engineError },
         archives: catalog.list().length,
         categories: catalog.categories(),

package/src/engines/composite.js

@@ -58,6 +58,9 @@ export class CompositeEngine {
     this.shareIntervalMs = Math.max(5, shareIntervalSeconds) * 1000;
     this.shareTimeoutMs = Math.max(60, shareTimeoutSeconds) * 1000;
     this.name = [primary.name, ...this.#secondaries.map((e) => e.name)].join('+');
+    // The primary is the one that writes the file. A secondary only ever
+    // receives an archive that is already whole, so it never marks anything.
+    this.marksIncomplete = primary.marksIncomplete ?? false;
   }
 
   /** The engine that owns the data. @returns {object} - The primary. */

package/src/engines/libtorrent.js

@@ -78,6 +78,12 @@ export class LibtorrentEngine {
       throw new Error('libtorrent engine requires a savePath');
     }
     this.name = 'libtorrent';
+    // Whether `incompleteSuffix` means anything here: it does not. The rename
+    // would have to happen in the sidecar, so a partial archive sits under its
+    // final name from the first byte — which is why a web server must never be
+    // pointed at this engine's save path, and why the console must not claim
+    // a marker that is not there.
+    this.marksIncomplete = false;
     if (options.listen !== undefined && typeof options.listen !== 'string') {
       // Caught here because the alternative is a C++ converter error four
       // frames into a Python traceback, which says nothing about which setting

package/src/engines/webtorrent.js

@@ -54,6 +54,9 @@ export class WebTorrentSeedEngine {
       throw new Error('WebTorrent engine requires a savePath');
     }
     this.name = 'webtorrent';
+    // Honoured, by replacing the store — the only thing that decides where
+    // bytes land.
+    this.marksIncomplete = true;
     this.#options = { readyTimeoutMs: 300000, ...options };
   }
 

package/src/hooks.js

@@ -169,9 +169,25 @@ export class ProgramHooks {
 
       this.#running.add(entry.infoHash);
       fired.push(entry);
-      this.#fire('onComplete', entry).finally(() =>
-        this.#running.delete(entry.infoHash),
-      );
+      this.#fire('onComplete', entry)
+        .then(async (result) => {
+          // Recorded before running so a six-hour build is not started six
+          // times over — but a command that never launched has not started
+          // anything, and keeping the stamp would mean fixing the path and
+          // still never seeing it run. That one case is given back.
+          if (result?.started === false) {
+            await this.#library.catalog.put({
+              infoHash: entry.infoHash,
+              completedAt: null,
+            });
+            console.warn(
+              `[hook] ${entry.name}: the command never started, so this will ` +
+                'be tried again on the next sweep',
+            );
+          }
+        })
+        .catch(() => {})
+        .finally(() => this.#running.delete(entry.infoHash));
     }
 
     return fired;
@@ -237,7 +253,13 @@ export class ProgramHooks {
         stream?.on('data', collect);
       }
 
-      const report = (problem) => {
+      // A failure to spawn raises 'error' and then, on some platforms, 'close'
+      // with a nonsense exit code — so the first account of what happened is
+      // the true one and the second is noise.
+      let reported = false;
+      const report = (problem, started = true) => {
+        if (reported) return;
+        reported = true;
         if (problem) {
           console.error(`[${label}] ${entry.name}: ${problem}`);
         } else {
@@ -248,12 +270,14 @@ export class ProgramHooks {
         for (const line of tail) {
           if (line.trim()) console.log(`[${label}]   ${line}`);
         }
-        resolve();
+        resolve({ started });
       };
 
       // A command that could not be started at all — no such file, not
-      // executable — never reaches 'close'.
-      child.on('error', (error) => report(error.message));
+      // executable, a working directory that is not there — never reaches
+      // 'close'. Reported as not started, which is what lets the caller try
+      // again: this is a configuration to fix, not a job that ran and failed.
+      child.on('error', (error) => report(error.message, false));
       child.on('close', (code, signal) => {
         if (signal) return report(`killed by ${signal}`);
         report(code === 0 ? undefined : `exited with code ${code}`);

package/src/web/index.html

@@ -725,6 +725,9 @@
       // ── Archives ──────────────────────────────────────────────────────────
       let archives = [];
       let selected = null;
+      // Null until /api/status says otherwise, so nothing claims a marker
+      // before the node has said whether its engine applies one.
+      let incompleteMarker = null;
       let activeTab = 'general';
 
       // What the header switch shows, so a click knows what it is toggling to.
@@ -882,6 +885,9 @@
           renderSpeed(speed);
           renderFetching(adds?.running);
           if (status.version) $('version').textContent = `v${status.version}`;
+          // What an unfinished archive is really called on disk, or null when
+          // nothing renames it. The engine decides, not the setting alone.
+          incompleteMarker = status.incompleteMarker ?? null;
           const engine = status.engine;
           $('status').innerHTML =
             `engine <b>${engine.name}</b> ${engine.ok ? 'ready' : 'unavailable'}` +
@@ -994,8 +1000,8 @@
             <td class="sub">${
               entry.paused ? 'paused' : escapeHtml(s.state ?? '—')
             }${
-              entry.complete === false && progress < 1
-                ? `<div class="sub" title="on disk as ${escapeHtml(entry.name)}.incomplete until it is whole">.incomplete</div>`
+              entry.complete === false && progress < 1 && incompleteMarker
+                ? `<div class="sub" title="on disk as ${escapeHtml(entry.name)}${escapeHtml(incompleteMarker)} until it is whole">${escapeHtml(incompleteMarker)}</div>`
                 : ''
             }</td>`;
           tr.onclick = () => {