pmtiles-swarm 0.4.0 → 0.4.2

package/CHANGELOG.md

@@ -7,6 +7,50 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.4.2
+### 🐞 Bug fixes
+- **Resume data is saved on a node running more than one engine.** The periodic save is only
+  scheduled if the engine offers `saveResume`, and the composite engine — the one in use
+  whenever `secondaryEngines` is set — did not, so it was never scheduled at all. The only
+  writes left were at shutdown, and those hit the second half of this: the sidecar asked
+  `need_save_resume_data()` first, which answers "has anything changed since the last save"
+  rather than "does a resume file exist". An archive that had been seeding since it was added
+  answers no, so nothing was written for it and it re-hashed its whole store on every start —
+  half an hour of disk, for 800 GB, before it serves anything. Both halves are fixed; the
+  sidecar half ships in `pmtiles-torrent`.
+- **A hook is no longer killed for being talkative.** Its output was collected whole into a
+  buffer, and past that buffer's size the child is killed — so a hook that generates a planet
+  could die hours in for the offence of saying too much, and the output that would have
+  explained it was the thing that overflowed. Output is streamed now and only the last twenty
+  lines are kept, so how much a hook says cannot decide whether it survives.
+
+### 📚 Documentation
+- **Sharing a folder with another service**, in the service guide: group membership is only the
+  first of three steps, and a folder at 0755 gives that group `r-x` — enough to hash and seed an
+  archive and not enough for `latestLink`, retention or a hook, so it looks like it worked until
+  the first thing that writes.
+- **The read-only hooks panel says to restart.** Setting `allowHooksFromApi` in the config file
+  unlocks nothing until the node reads it, which it does once, at startup.
+
+## 0.4.1
+### 🐞 Bug fixes
+- **A feed no longer walks backwards through its own history.** An item already taken was
+  skipped and the loop carried on to the older one below it, and the cap counts what was
+  *added* — so every poll took exactly one archive and every poll took a different one, until
+  the whole backlog was on disk. Against planet.openstreetmap.org that is five 88 GiB dumps
+  arriving a quarter of an hour apart from a subscription asking for one. Items run newest
+  first, so reaching one already held now stops the pass: everything after it is older than
+  something already on disk. A build that could not be fetched stops it too — one bad fetch is
+  a reason to retry shortly, not to take last week's instead.
+- **A subscription's `mode` had no effect.** It reached the add as `paused`, which nothing
+  reads — not the library and not the engine — so every item a feed brought in arrived as a
+  cache whatever the subscription said, and a `"mode": "mirror"` feed quietly fetched nothing.
+  A cache subscription only looked correct because cache is the default. It is now passed as
+  `mode`, the name the library actually reads.
+- **The console says what an empty availability bar means.** It counts connected peers and not
+  this node, so an archive only this node holds shows nothing — the truth about the swarm rather
+  than about the file, but worth saying beside a Downloaded bar that is full.
+
 ## 0.4.0
 ### ✨ Features and improvements
 - **A watched folder can set the torrent comment**, which is where attribution and licence belong —

package/docs/running-as-a-service.md

@@ -214,6 +214,74 @@ anything that holds data:
 `ReadWritePaths` names, so every one of those has to be listed. An archive
 directory on another mount needs its own entry.
 
+## Sharing a folder with another service
+
+A folder produced by something else — a generation script, or a directory a
+torrent client already owns — needs three things, and group membership is only
+the first of them.
+
+```sh
+# 1. Put the service account in the owning group.
+sudo usermod -aG qbittorrent-nox pmtiles-swarm
+
+# 2. Give that group write, and setgid so new entries inherit it.
+sudo find /mnt/hd-16TB/store/generated -type d -exec chmod 2775 {} +
+sudo find /mnt/hd-16TB/store/generated -type f -exec chmod 664 {} +
+
+# 3. Make what the service creates group-writable too.
+sudo systemctl edit pmtiles-swarm
+sudo systemctl restart pmtiles-swarm
+```
+
+Step 3 opens an override; the two lines to put in it are:
+
+```ini
+[Service]
+UMask=0002
+```
+
+**A folder at 0755 gives the group `r-x`.** Membership alone buys read access
+and nothing else, which is enough to hash and seed an archive and not enough to
+do anything else with the folder — so this looks like it worked until the first
+thing that writes.
+
+Three features want write, and it is worth knowing which, because a read-only
+folder is a perfectly reasonable way to run:
+
+| | |
+| --- | --- |
+| `latestLink` | Creates and replaces a name in the folder |
+| `keep`, `keepDays` | Deletes retired builds |
+| `onComplete` | Whatever the script does, since it runs as this account |
+
+Renaming and deleting need write on the **directory**, not on the file, which is
+why the directory bits are the ones that matter. `UMask=0002` matters for the
+other direction: without it a file the service creates is `0644`, and the other
+service can delete it but not modify it.
+
+**Group membership is read when a process starts**, so the restart is not
+optional. Neither is `ReadWritePaths`: `ProtectSystem=strict` presents the rest
+of the filesystem as read-only inside the unit's namespace, and the write is
+refused there before the permission bits are consulted. Every folder outside
+`/var/lib/pmtiles-swarm` has to be named, in a drop-in from
+`systemctl edit pmtiles-swarm`:
+
+```ini
+[Service]
+ReadWritePaths=/mnt/store/generated /mnt/work/planetiler
+UMask=0002
+```
+
+`ReadWritePaths=` accumulates, so a drop-in adds to what the unit already lists
+rather than replacing it.
+
+Two things that look like checks and are not. `id pmtiles-swarm` reads
+`/etc/group` and shows the new group the instant `usermod` returns, whether or
+not the running process has it — read `/proc/$(systemctl show -p MainPID --value
+pmtiles-swarm)/status` instead. And `sudo -u pmtiles-swarm touch …` runs outside
+the unit's namespace, so it succeeds on permissions alone while the service is
+still being refused.
+
 ## The sidecar
 
 The libtorrent engine runs Python as a child process, so the service user needs

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.4.0",
+  "version": "0.4.2",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/engines/composite.js

@@ -340,6 +340,32 @@ export class CompositeEngine {
     }
   }
 
+  /**
+   * Persists resume data on every engine that keeps any.
+   *
+   * Missing entirely until now, and the caller checks for it before setting
+   * its timer — so on any node with a secondary engine the periodic save was
+   * never scheduled, and the only resume data ever written was whatever the
+   * shutdown path managed. An archive that had been seeding since it was added
+   * therefore re-hashed its whole store on every start, which for 800 GB is
+   * half an hour of disk before it serves anything.
+   * @param {string} [infoHash] - One archive, or all of them when omitted.
+   * @returns {Promise<void>} - Resolves once every engine has been asked.
+   */
+  async saveResume(infoHash) {
+    for (const engine of [this.#primary, ...this.#secondaries]) {
+      // WebTorrent keeps none, and says so by not offering the method.
+      if (!engine.saveResume) continue;
+      try {
+        await engine.saveResume(infoHash);
+      } catch (error) {
+        console.warn(
+          `[composite] ${engine.name} could not save resume data: ${error.message}`,
+        );
+      }
+    }
+  }
+
   /**
    * Peers from every engine, labelled with which one found them.
    * @param {string} infoHash - The archive.

package/src/hooks.js

@@ -1,6 +1,9 @@
-import { execFile } from 'node:child_process';
+import { spawn } from 'node:child_process';
 import path from 'node:path';
 
+/** How much of a hook's output to repeat into this node's own log. */
+const TAIL_LINES = 20;
+
 /**
  * Running something when an archive arrives, and when it finishes.
  *
@@ -200,32 +203,61 @@ export class ProgramHooks {
     console.log(`[${label}] ${entry.name}: ${command} ${filled.join(' ')}`);
 
     return new Promise((resolve) => {
-      execFile(
-        command,
-        filled,
-        {
-          // A tile build runs for hours. Nothing here should assume otherwise,
-          // so the default is no timeout at all.
-          timeout: (timeoutSeconds ?? 0) * 1000,
-          maxBuffer: 4 * 1024 * 1024,
-          cwd: hook.cwd,
-          env: { ...process.env, ...(hook.env ?? {}) },
-        },
-        (error, stdout, stderr) => {
-          if (error) {
-            console.error(`[${label}] ${entry.name}: ${error.message}`);
-          } else {
-            console.log(`[${label}] ${entry.name}: finished`);
-          }
-          const output = `${stdout ?? ''}${stderr ?? ''}`.trim();
-          if (output) {
-            for (const line of output.split('\n').slice(-20)) {
-              console.log(`[${label}]   ${line}`);
-            }
-          }
-          resolve();
-        },
-      );
+      const timeout = (timeoutSeconds ?? 0) * 1000;
+      const child = spawn(command, filled, {
+        // A tile build runs for hours. Nothing here should assume otherwise,
+        // so the default is no timeout at all.
+        ...(timeout > 0 ? { timeout } : {}),
+        cwd: hook.cwd,
+        env: { ...process.env, ...(hook.env ?? {}) },
+      });
+
+      // Streamed, and only the tail is kept.
+      //
+      // This used to collect the whole of stdout and stderr into a buffer, and
+      // a buffer has a size: past it, execFile kills the child. A hook that
+      // generates a planet says far more than any buffer worth holding, so a
+      // build could be killed hours in for the offence of being talkative —
+      // and the output that would have explained it was the thing that
+      // overflowed. Nothing is held now but the last few lines, so how much a
+      // hook says cannot decide whether it survives.
+      const tail = [];
+      let partial = '';
+      const collect = (chunk) => {
+        partial += chunk;
+        const lines = partial.split('\n');
+        partial = lines.pop() ?? '';
+        for (const line of lines) {
+          tail.push(line);
+          if (tail.length > TAIL_LINES) tail.shift();
+        }
+      };
+      for (const stream of [child.stdout, child.stderr]) {
+        stream?.setEncoding('utf8');
+        stream?.on('data', collect);
+      }
+
+      const report = (problem) => {
+        if (problem) {
+          console.error(`[${label}] ${entry.name}: ${problem}`);
+        } else {
+          console.log(`[${label}] ${entry.name}: finished`);
+        }
+        // A last line with no newline after it is still a line.
+        if (partial) collect('\n');
+        for (const line of tail) {
+          if (line.trim()) console.log(`[${label}]   ${line}`);
+        }
+        resolve();
+      };
+
+      // A command that could not be started at all — no such file, not
+      // executable — never reaches 'close'.
+      child.on('error', (error) => report(error.message));
+      child.on('close', (code, signal) => {
+        if (signal) return report(`killed by ${signal}`);
+        report(code === 0 ? undefined : `exited with code ${code}`);
+      });
     });
   }
 }

package/src/subscriptions.js

@@ -141,12 +141,26 @@ export class SubscriptionManager {
         continue;
       }
       const marker = item.infoHash ?? item.magnet ?? item.torrentUrl;
-      if (this.#seen.has(marker)) continue;
-      this.#seen.add(marker);
+
+      // Stop, rather than skip past it to the next one.
+      //
+      // Items run newest first, so reaching one already taken means
+      // everything after it is older than something on disk. Skipping to the
+      // next instead takes one more archive every poll and walks backwards
+      // through the feed until the whole backlog is held — which for the
+      // planet feed is five 88 GiB dumps nobody asked for, arriving one a
+      // quarter of an hour apart. The cap counts what was added, so it does
+      // not stop this: each poll adds exactly one, and each poll adds a
+      // different one.
+      if (this.#seen.has(marker)) break;
 
       try {
         const entry = await this.#add(item, subscription);
         if (entry) {
+          // Marked only once it worked. A .torrent that 404s for an hour
+          // should be retried on the next poll rather than passed over for
+          // good on the strength of one bad fetch.
+          this.#seen.add(marker);
           added.push(entry);
           console.log(
             `[feed] ${subscription.mode ?? 'cache'} ${entry.name} from ${subscription.url}`,
@@ -154,6 +168,10 @@ export class SubscriptionManager {
         }
       } catch (error) {
         console.error(`[feed] could not add "${item.title}": ${error.message}`);
+        // And stop here too. The newest build being briefly unreachable is a
+        // reason to try again shortly, not a reason to fetch last week's
+        // instead — which is the same backwards walk by another route.
+        break;
       }
     }
     return added;
@@ -340,8 +358,15 @@ export class SubscriptionManager {
       // from one built here or added by hand, and would happily delete both.
       subscriptionUrl: subscription.url,
       // Cache mode joins the swarm without pulling the whole archive; the
-      // pieces it does hold are still served to other peers.
-      paused: (subscription.mode ?? 'cache') === 'cache',
+      // pieces it does hold are still served to other peers. Mirror commits to
+      // a whole copy.
+      //
+      // Passed as `mode`, which is the name the library reads. It had been
+      // passed as `paused`, which nothing reads — so a feed's mode did not
+      // reach the add at all, and every item arrived as cache whatever the
+      // subscription said. Defaulted here as well as there, so the two cannot
+      // drift apart into disagreeing about what an unset mode means.
+      mode: subscription.mode ?? 'cache',
     };
 
     // The .torrent is preferred where there is one: it carries the trackers

package/src/web/index.html

@@ -1190,7 +1190,9 @@
                 shown as held when <i>all</i> of them are. Availability is the
                 <i>rarest</i> piece in each column: one piece nobody has is the
                 answer to "can this still be completed", however well supplied
-                its neighbours are.
+                its neighbours are. It counts <i>connected peers</i> and not
+                this node, so an archive only this node holds shows an empty
+                bar — which is the truth about the swarm, not about the file.
               </div>
               ${
                 (info.peers ?? []).length > 0
@@ -2600,7 +2602,11 @@
                      token that manages maps — it would be one that runs code.
                      Edit them in the config file, or set
                      <code>"allowHooksFromApi": true</code> there to take that
-                     trade deliberately and unlock this panel.
+                     trade deliberately and unlock this panel —
+                     <b>then restart</b>, since the config file is read once at
+                     startup and nothing re-reads it. This panel is still
+                     read-only because the running node has not seen that
+                     setting, whatever the file now says.
                    </p>`
             }
           </div>`;