pmtiles-swarm 0.4.1 → 0.4.2

package/CHANGELOG.md

@@ -7,6 +7,31 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.4.2
+### 🐞 Bug fixes
+- **Resume data is saved on a node running more than one engine.** The periodic save is only
+  scheduled if the engine offers `saveResume`, and the composite engine — the one in use
+  whenever `secondaryEngines` is set — did not, so it was never scheduled at all. The only
+  writes left were at shutdown, and those hit the second half of this: the sidecar asked
+  `need_save_resume_data()` first, which answers "has anything changed since the last save"
+  rather than "does a resume file exist". An archive that had been seeding since it was added
+  answers no, so nothing was written for it and it re-hashed its whole store on every start —
+  half an hour of disk, for 800 GB, before it serves anything. Both halves are fixed; the
+  sidecar half ships in `pmtiles-torrent`.
+- **A hook is no longer killed for being talkative.** Its output was collected whole into a
+  buffer, and past that buffer's size the child is killed — so a hook that generates a planet
+  could die hours in for the offence of saying too much, and the output that would have
+  explained it was the thing that overflowed. Output is streamed now and only the last twenty
+  lines are kept, so how much a hook says cannot decide whether it survives.
+
+### 📚 Documentation
+- **Sharing a folder with another service**, in the service guide: group membership is only the
+  first of three steps, and a folder at 0755 gives that group `r-x` — enough to hash and seed an
+  archive and not enough for `latestLink`, retention or a hook, so it looks like it worked until
+  the first thing that writes.
+- **The read-only hooks panel says to restart.** Setting `allowHooksFromApi` in the config file
+  unlocks nothing until the node reads it, which it does once, at startup.
+
 ## 0.4.1
 ### 🐞 Bug fixes
 - **A feed no longer walks backwards through its own history.** An item already taken was

package/docs/running-as-a-service.md

@@ -214,6 +214,74 @@ anything that holds data:
 `ReadWritePaths` names, so every one of those has to be listed. An archive
 directory on another mount needs its own entry.
 
+## Sharing a folder with another service
+
+A folder produced by something else — a generation script, or a directory a
+torrent client already owns — needs three things, and group membership is only
+the first of them.
+
+```sh
+# 1. Put the service account in the owning group.
+sudo usermod -aG qbittorrent-nox pmtiles-swarm
+
+# 2. Give that group write, and setgid so new entries inherit it.
+sudo find /mnt/hd-16TB/store/generated -type d -exec chmod 2775 {} +
+sudo find /mnt/hd-16TB/store/generated -type f -exec chmod 664 {} +
+
+# 3. Make what the service creates group-writable too.
+sudo systemctl edit pmtiles-swarm
+sudo systemctl restart pmtiles-swarm
+```
+
+Step 3 opens an override; the two lines to put in it are:
+
+```ini
+[Service]
+UMask=0002
+```
+
+**A folder at 0755 gives the group `r-x`.** Membership alone buys read access
+and nothing else, which is enough to hash and seed an archive and not enough to
+do anything else with the folder — so this looks like it worked until the first
+thing that writes.
+
+Three features want write, and it is worth knowing which, because a read-only
+folder is a perfectly reasonable way to run:
+
+| | |
+| --- | --- |
+| `latestLink` | Creates and replaces a name in the folder |
+| `keep`, `keepDays` | Deletes retired builds |
+| `onComplete` | Whatever the script does, since it runs as this account |
+
+Renaming and deleting need write on the **directory**, not on the file, which is
+why the directory bits are the ones that matter. `UMask=0002` matters for the
+other direction: without it a file the service creates is `0644`, and the other
+service can delete it but not modify it.
+
+**Group membership is read when a process starts**, so the restart is not
+optional. Neither is `ReadWritePaths`: `ProtectSystem=strict` presents the rest
+of the filesystem as read-only inside the unit's namespace, and the write is
+refused there before the permission bits are consulted. Every folder outside
+`/var/lib/pmtiles-swarm` has to be named, in a drop-in from
+`systemctl edit pmtiles-swarm`:
+
+```ini
+[Service]
+ReadWritePaths=/mnt/store/generated /mnt/work/planetiler
+UMask=0002
+```
+
+`ReadWritePaths=` accumulates, so a drop-in adds to what the unit already lists
+rather than replacing it.
+
+Two things that look like checks and are not. `id pmtiles-swarm` reads
+`/etc/group` and shows the new group the instant `usermod` returns, whether or
+not the running process has it — read `/proc/$(systemctl show -p MainPID --value
+pmtiles-swarm)/status` instead. And `sudo -u pmtiles-swarm touch …` runs outside
+the unit's namespace, so it succeeds on permissions alone while the service is
+still being refused.
+
 ## The sidecar
 
 The libtorrent engine runs Python as a child process, so the service user needs

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.4.1",
+  "version": "0.4.2",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/engines/composite.js

@@ -340,6 +340,32 @@ export class CompositeEngine {
     }
   }
 
+  /**
+   * Persists resume data on every engine that keeps any.
+   *
+   * Missing entirely until now, and the caller checks for it before setting
+   * its timer — so on any node with a secondary engine the periodic save was
+   * never scheduled, and the only resume data ever written was whatever the
+   * shutdown path managed. An archive that had been seeding since it was added
+   * therefore re-hashed its whole store on every start, which for 800 GB is
+   * half an hour of disk before it serves anything.
+   * @param {string} [infoHash] - One archive, or all of them when omitted.
+   * @returns {Promise<void>} - Resolves once every engine has been asked.
+   */
+  async saveResume(infoHash) {
+    for (const engine of [this.#primary, ...this.#secondaries]) {
+      // WebTorrent keeps none, and says so by not offering the method.
+      if (!engine.saveResume) continue;
+      try {
+        await engine.saveResume(infoHash);
+      } catch (error) {
+        console.warn(
+          `[composite] ${engine.name} could not save resume data: ${error.message}`,
+        );
+      }
+    }
+  }
+
   /**
    * Peers from every engine, labelled with which one found them.
    * @param {string} infoHash - The archive.

package/src/hooks.js

@@ -1,6 +1,9 @@
-import { execFile } from 'node:child_process';
+import { spawn } from 'node:child_process';
 import path from 'node:path';
 
+/** How much of a hook's output to repeat into this node's own log. */
+const TAIL_LINES = 20;
+
 /**
  * Running something when an archive arrives, and when it finishes.
  *
@@ -200,32 +203,61 @@ export class ProgramHooks {
     console.log(`[${label}] ${entry.name}: ${command} ${filled.join(' ')}`);
 
     return new Promise((resolve) => {
-      execFile(
-        command,
-        filled,
-        {
-          // A tile build runs for hours. Nothing here should assume otherwise,
-          // so the default is no timeout at all.
-          timeout: (timeoutSeconds ?? 0) * 1000,
-          maxBuffer: 4 * 1024 * 1024,
-          cwd: hook.cwd,
-          env: { ...process.env, ...(hook.env ?? {}) },
-        },
-        (error, stdout, stderr) => {
-          if (error) {
-            console.error(`[${label}] ${entry.name}: ${error.message}`);
-          } else {
-            console.log(`[${label}] ${entry.name}: finished`);
-          }
-          const output = `${stdout ?? ''}${stderr ?? ''}`.trim();
-          if (output) {
-            for (const line of output.split('\n').slice(-20)) {
-              console.log(`[${label}]   ${line}`);
-            }
-          }
-          resolve();
-        },
-      );
+      const timeout = (timeoutSeconds ?? 0) * 1000;
+      const child = spawn(command, filled, {
+        // A tile build runs for hours. Nothing here should assume otherwise,
+        // so the default is no timeout at all.
+        ...(timeout > 0 ? { timeout } : {}),
+        cwd: hook.cwd,
+        env: { ...process.env, ...(hook.env ?? {}) },
+      });
+
+      // Streamed, and only the tail is kept.
+      //
+      // This used to collect the whole of stdout and stderr into a buffer, and
+      // a buffer has a size: past it, execFile kills the child. A hook that
+      // generates a planet says far more than any buffer worth holding, so a
+      // build could be killed hours in for the offence of being talkative —
+      // and the output that would have explained it was the thing that
+      // overflowed. Nothing is held now but the last few lines, so how much a
+      // hook says cannot decide whether it survives.
+      const tail = [];
+      let partial = '';
+      const collect = (chunk) => {
+        partial += chunk;
+        const lines = partial.split('\n');
+        partial = lines.pop() ?? '';
+        for (const line of lines) {
+          tail.push(line);
+          if (tail.length > TAIL_LINES) tail.shift();
+        }
+      };
+      for (const stream of [child.stdout, child.stderr]) {
+        stream?.setEncoding('utf8');
+        stream?.on('data', collect);
+      }
+
+      const report = (problem) => {
+        if (problem) {
+          console.error(`[${label}] ${entry.name}: ${problem}`);
+        } else {
+          console.log(`[${label}] ${entry.name}: finished`);
+        }
+        // A last line with no newline after it is still a line.
+        if (partial) collect('\n');
+        for (const line of tail) {
+          if (line.trim()) console.log(`[${label}]   ${line}`);
+        }
+        resolve();
+      };
+
+      // A command that could not be started at all — no such file, not
+      // executable — never reaches 'close'.
+      child.on('error', (error) => report(error.message));
+      child.on('close', (code, signal) => {
+        if (signal) return report(`killed by ${signal}`);
+        report(code === 0 ? undefined : `exited with code ${code}`);
+      });
     });
   }
 }

package/src/web/index.html

@@ -2602,7 +2602,11 @@
                      token that manages maps — it would be one that runs code.
                      Edit them in the config file, or set
                      <code>"allowHooksFromApi": true</code> there to take that
-                     trade deliberately and unlock this panel.
+                     trade deliberately and unlock this panel —
+                     <b>then restart</b>, since the config file is read once at
+                     startup and nothing re-reads it. This panel is still
+                     read-only because the running node has not seen that
+                     setting, whatever the file now says.
                    </p>`
             }
           </div>`;