pmtiles-swarm 0.41.0 → 0.41.2

package/CHANGELOG.md

@@ -7,6 +7,52 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.41.2
+### ✨ Features and improvements
+
+### 🐞 Bug fixes
+- **An archive that finished downloading during an unclean stop no longer comes back at 0% for ever.**
+  A downloaded archive is written under a marker — `planet.pmtiles.incomplete` — and renamed the
+  instant it is whole. The rename and the catalog entry recording it are two steps, so a stop between
+  them leaves an archive that is finished on disk and unfinished in the record.
+
+  That disagreement was permanent, and worse than it looks. Restore re-added the entry with the marker
+  attached, so the engine opened a filename nothing was at any more, found no data, and began
+  downloading an archive the node already held. The sweep that would have noticed takes the engine's
+  word over the disk's whenever the engine has one — and the engine's word was now 0%. Rechecking did
+  not help either, since it hashes the marked name, which is the wrong file to look at. No number of
+  restarts recovered it.
+
+  Restore now checks the disk before handing over anything recorded as unfinished: marked file gone
+  and the real one whole means the rename happened and the record of it did not, so the entry is
+  corrected and the archive handed over as the complete thing it is. A genuine partial download is
+  left exactly as it was — claiming otherwise would offer peers an archive this node cannot serve.
+
+  This is why the archives that arrive from a feed or a URL were the ones that sat at 0%: an archive
+  built here is recorded complete the moment it is registered, having just been read end to end, so it
+  never carries a marker to disagree about.
+
+## 0.41.1
+### ✨ Features and improvements
+
+### 🐞 Bug fixes
+- **`KillMode=mixed` in the documented unit, and it should have been there from the start.** systemd's
+  default is `control-group`, which sends `SIGTERM` to every process in the unit at the same instant —
+  the node and the Python sidecar together. The node's shutdown then asks a sidecar that is already
+  dying to write its resume data, into a pipe that is closing, and the answer never comes.
+
+  Resume data is how an archive comes back knowing what it holds. Without it, archives return at 0%
+  after a restart and only a recheck discovers they were complete all along. `mixed` signals the main
+  process only; the node stops the sidecar itself and waits for the write. Nothing is left running —
+  anything still alive at `TimeoutStopSec` is killed, sidecar included. **Existing installs need this
+  added by hand**, followed by `systemctl daemon-reload`.
+- **The check added in 0.40.0 asked its first question only of archives recorded as complete**, and so
+  missed the case that prompted it. An archive interrupted mid-download comes back recorded as
+  incomplete, so a restore that failed to hand it over left it absent from the engine and unreported
+  by the very check meant to notice — which is what a row reading 0% with no state at all is. Whether
+  the engine is holding an archive is now asked of everything restore handed over; whether the
+  `seedOnly` claim held is still asked only of the archives that make one.
+
 ## 0.41.0
 ### ✨ Features and improvements
 - **An archive's details now open directly under the row you clicked**, instead of below the whole

package/docs/running-as-a-service.md

@@ -180,6 +180,11 @@ RestartSec=5
 # lock and cancels downloads in flight. Worst case is about 20 seconds.
 TimeoutStopSec=45
 
+# The node stops the sidecar itself, and needs it alive to do so. The default
+# signals both at once, which kills the sidecar before it can write its resume
+# data — see "The two lines that matter".
+KillMode=mixed
+
 # A seeding node holds a socket per peer, and the tile reader holds file
 # descriptors of its own.
 LimitNOFILE=65535
@@ -215,12 +220,23 @@ down and **exits 0**, expecting to be brought back.
 `Restart=on-failure` ignores an exit 0, so the first use of that button would
 stop the node and leave the unit reporting success.
 
+**`KillMode=mixed`, not the default.** The default is `control-group`, which
+sends `SIGTERM` to _every_ process in the unit at the same instant — the node
+and the Python sidecar together. The node's shutdown then asks a sidecar that is
+already dying to write its resume data, into a pipe that is closing, and the
+answer never comes. Resume data is how an archive comes back knowing what it
+holds; without it, archives return at 0% after a restart and have to be
+rechecked to find out they were complete all along.
+
+`mixed` sends the signal to the main process only. The node stops the sidecar
+itself, in order, and waits for the resume data to be written. Nothing is left
+running: anything still alive when `TimeoutStopSec` expires is killed, sidecar
+included.
+
 **No `ExecStop=`.** systemd already sends `SIGTERM`, and the node handles it
 from the moment it starts. `ExecStop=/bin/kill -15 $MAINPID` is redundant, and
-becomes wrong if the unit ever uses `KillMode=process` — it would stop the node
-while the Python sidecar kept running and kept the data directory locked.
-
-Leave `KillMode` at its default, so the sidecar goes with its parent.
+becomes wrong under `KillMode=process` — that one leaves the rest of the unit
+running indefinitely, so the sidecar would keep the data directory locked.
 
 ## Where it writes
 
@@ -587,8 +603,10 @@ and lives as long as it does, so a new sidecar sits on disk doing nothing until
 the service is restarted. Most of what changes between releases is in there.
 
 Nothing under `/etc/pmtiles-swarm` is touched, and restarting does not re-check
-the archives: a clean stop writes resume data, and `TimeoutStopSec` above
-leaves room for it.
+the archives: a clean stop writes resume data, and `TimeoutStopSec` above leaves
+room for it. That depends on `KillMode=mixed` — without it the sidecar is
+signalled at the same moment as the node and dies before it can write anything,
+which is what a library that comes back at 0% after every restart looks like.
 
 Confirm both halves moved, since the sidecar has its own version:
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.41.0",
+  "version": "0.41.2",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/library.js

@@ -1819,6 +1819,72 @@ export class Library {
     return queued;
   }
 
+  /**
+   * Reconciles what the catalog says about an archive with what is on disk.
+   *
+   * An archive downloaded here is written under a marker — `planet.pmtiles`
+   * becomes `planet.pmtiles.incomplete` — and renamed the instant it is whole.
+   * The rename and the catalog entry that records it are two steps, so a stop
+   * between them leaves an archive that is finished on disk and unfinished in
+   * the record.
+   *
+   * That disagreement does not settle itself, and it is worse than it looks.
+   * Restore re-adds an entry it believes incomplete with the marker attached,
+   * so the engine opens `planet.pmtiles.incomplete` — a name nothing is at any
+   * more — finds no data, and begins downloading a 128 GiB archive this node
+   * already holds. The sweep that would notice takes the engine's word over
+   * the disk's whenever the engine has one, and the engine's word is now 0%.
+   * Rechecking does not help either: it hashes the marked name, which is the
+   * wrong file to be looking at. Nothing recovers, through any number of
+   * restarts.
+   *
+   * Only archives that were downloaded can be in this state — one built here
+   * is recorded complete the moment it is registered, having just been read
+   * end to end — which is why the archives that come from a feed or a URL are
+   * the ones that sit at 0%.
+   * @param {object} entry - The catalog entry restore is about to hand over.
+   * @returns {Promise<object>} - That entry, or the corrected one.
+   */
+  async #reconcileMarker(entry) {
+    if (entry.complete !== false || !entry.savePath || !entry.name) {
+      return entry;
+    }
+
+    const marked = onDiskPath(entry, this.#config);
+    const real = path.join(entry.savePath, entry.name);
+    // Markers turned off, so there is no second name to disagree with.
+    if (!marked || marked === real) return entry;
+
+    // Still under the marker, which is what an unfinished download should look
+    // like. Nothing to correct, whatever its progress.
+    const partial = await fs.stat(marked).then(
+      () => true,
+      () => false,
+    );
+    if (partial) return entry;
+
+    // The marked name is gone and the real one is whole: the rename happened
+    // and the record of it did not.
+    const whole = await alreadyComplete({
+      savePath: entry.savePath,
+      name: entry.name,
+      size: entry.size,
+    });
+    if (!whole) return entry;
+
+    console.warn(
+      `[complete] ${entry.name} is whole under its own name but was recorded ` +
+        'as unfinished — a stop between the rename and the record. Handing it ' +
+        'to the engine as the complete archive it is.',
+    );
+    return (
+      (await this.#catalog.put({
+        infoHash: entry.infoHash,
+        complete: true,
+      })) ?? { ...entry, complete: true }
+    );
+  }
+
   /**
    * Checks that what restore just claimed is actually being seeded.
    *
@@ -1840,10 +1906,7 @@ export class Library {
    * @returns {Promise<void>} - Resolves once every claim has been checked.
    */
   async #verifySeeding(entries) {
-    const claimed = entries.filter(
-      (entry) => entry.complete !== false && entry.mode !== 'cache',
-    );
-    if (claimed.length === 0) return;
+    if (entries.length === 0) return;
 
     // One listing rather than a status call each: this runs over the whole
     // library on every start, and a round trip per archive is a cost paid by
@@ -1854,25 +1917,37 @@ export class Library {
     }
 
     let wrong = 0;
-    for (const entry of claimed) {
+    for (const entry of entries) {
       const status = held.get(entry.infoHash);
-
-      // Checking is the engine doing the right thing already, and progress
-      // during it is the fraction hashed rather than the fraction held.
-      if (status && (status.progress >= 1 || status.state === 'checking')) {
-        continue;
-      }
-      wrong += 1;
       const label = `[seeding] ${entry.name}`;
 
+      // Held at all is a different question from held whole, and it is asked
+      // of everything restore handed over. Asking it only of complete archives
+      // was the first version of this check, and it missed the case that
+      // prompted it: an archive interrupted mid-download comes back recorded
+      // as incomplete, so a restore that silently failed to hand it over left
+      // it absent from the engine and unreported by the very check meant to
+      // notice. Absent is absent — it is neither seeding nor downloading.
       if (!status) {
+        wrong += 1;
         console.error(
-          `${label}: recorded as complete, but the engine is not holding it ` +
-            'at all — restore counted it and nothing is seeding it.',
+          `${label}: restore handed this to the engine and the engine is not ` +
+            'holding it. It is neither seeding nor downloading, and nothing ' +
+            'will start it before the next restart.',
         );
         continue;
       }
 
+      // Everything below is about the `seedOnly` claim, and only a complete
+      // archive makes one. An incomplete one is supposed to read as a partial
+      // download, so its progress says nothing about whether anything is wrong.
+      if (entry.complete === false || entry.mode === 'cache') continue;
+
+      // Checking is the engine doing the right thing already, and progress
+      // during it is the fraction hashed rather than the fraction held.
+      if (status.progress >= 1 || status.state === 'checking') continue;
+      wrong += 1;
+
       const file = entry.savePath
         ? path.join(entry.savePath, entry.name)
         : null;
@@ -1921,8 +1996,8 @@ export class Library {
 
     if (wrong > 0) {
       console.error(
-        `[seeding] ${wrong} of ${claimed.length} archives recorded as ` +
-          'complete are not being seeded. The lines above say which and why.',
+        `[seeding] ${wrong} of ${entries.length} restored archives are not ` +
+          'in the state the catalog describes. The lines above say which and why.',
       );
     }
   }
@@ -1951,8 +2026,9 @@ export class Library {
     }, 15_000);
     progress.unref?.();
 
+    const handed = [];
     try {
-      await this.#restoreEach(entries, tally);
+      await this.#restoreEach(entries, tally, handed);
     } finally {
       clearInterval(progress);
     }
@@ -1962,7 +2038,7 @@ export class Library {
     // reports on a library half restored. Never allowed to fail the restore —
     // this is a report about seeding, and a node that could not produce it is
     // still a node that restored what it could.
-    await this.#verifySeeding(entries).catch((error) =>
+    await this.#verifySeeding(handed).catch((error) =>
       console.warn(
         `[seeding] could not check what is being seeded: ${error.message}`,
       ),
@@ -1980,7 +2056,7 @@ export class Library {
    * @param {{restored: number, failed: number}} tally - Mutated as it goes.
    * @returns {Promise<{restored: number, failed: number}>} - That tally.
    */
-  async #restoreEach(entries, tally) {
+  async #restoreEach(entries, tally, handed) {
     for (const entry of entries) {
       // An engine that cannot open its port will fail every one of these, each
       // after its own timeout. Stopping at the first is the difference between
@@ -2015,11 +2091,18 @@ export class Library {
           }
         }
 
+        // Before the add, because the add is what acts on the disagreement:
+        // an entry wrongly recorded as unfinished is handed to the engine
+        // under a filename nothing is at, and the engine starts downloading
+        // an archive that is already here.
+        const settled = await this.#reconcileMarker(entry);
+
         // Through the same path as every other re-add, so an archive stored
         // with no trackers is repaired here too. Restoring used to build its
         // own add and skip that, which is why an archive that could not find a
         // peer stayed unable to find one across every restart.
-        await this.#readd(entry);
+        await this.#readd(settled);
+        handed.push(settled);
         tally.restored++;
       } catch (error) {
         tally.failed++;