pmtiles-swarm 0.98.3 → 0.99.0

package/CHANGELOG.md

@@ -7,6 +7,35 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.99.0
+### ✨ Features and improvements
+- **A sidecar crash now names the frame it happened in.** `pmtiles-torrent` 0.11.0 arms
+  `faulthandler` in the sidecar, so a segfault writes a Python traceback to stderr on its way out
+  and this node forwards it into the log line by line, beside the `sidecar killed by SIGSEGV` that
+  used to be the whole story. Nothing to configure; it costs nothing until the process faults.
+
+  Worth having because that signal is otherwise the only evidence. On the node this came from it
+  appeared several times per start for eleven days, and each crash takes the archives handed over
+  before it — which is what leaves a library reading `not loaded` after a restart.
+
+### 🐞 Bug fixes
+- **An archive the engine took and then did not keep is handed back, rather than only reported.**
+  The seeding check has been able to spot this for a while and its own message admitted the rest:
+  "nothing will start it before the next restart". It was right, which is why the cure was always
+  another restart. The usual cause is a sidecar that dies partway through a restore — the
+  replacement holds nothing, so the archives handed over before it died are absent while the ones
+  after it are fine, and `add` resolving is no evidence that anything was kept.
+
+  Those are now handed back once and re-checked, and the log says whether it took. Once, not in a
+  loop: an engine that refuses twice will not be talked round by a third try. And only when the
+  engine is holding *some* of the library — one holding none of it, or one that could not be listed
+  at all, is not suffering a per-archive fault, and re-adding everything on the strength of that
+  answer is how a node spends its start hashing what it already had.
+- **A pinned archive in a stack was listed by its infohash.** Forty characters of hex, where the
+  picker that offered it had shown a filename. The row leads with the archive's name now and shows
+  the infohash as what it resolves to, which for a pinned source is exactly what it means: this
+  build and no later one.
+
 ## 0.98.3
 ### ✨ Features and improvements
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.98.3",
+  "version": "0.99.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",
@@ -49,7 +49,7 @@
     "maplibre-gl": "^6.2.0",
     "parse-torrent": "^11.0.24",
     "pmtiles": "^4.4.1",
-    "pmtiles-torrent": "^0.10.2",
+    "pmtiles-torrent": "^0.11.0",
     "webtorrent": "^3.0.21"
   },
   "engines": {

package/src/library.js

@@ -2154,16 +2154,46 @@ export class Library {
    * @param {object[]} entries - The entries restore worked through.
    * @returns {Promise<void>} - Resolves once every claim has been checked.
    */
+  /**
+   * Whether the engine is holding this archive right now.
+   *
+   * Asked of one archive rather than by re-listing the library: this runs only
+   * for an archive that has just been handed back, which on a healthy node is
+   * never.
+   * @param {string} infoHash - The archive.
+   * @returns {Promise<boolean>} - Whether the engine has it.
+   */
+  async #engineHolds(infoHash) {
+    const status = await this.#engine.get?.(infoHash).catch(() => null);
+    return Boolean(status);
+  }
+
   async #verifySeeding(entries) {
     if (entries.length === 0) return;
 
     // One listing rather than a status call each: this runs over the whole
     // library on every start, and a round trip per archive is a cost paid by
     // every node to catch a fault most of them do not have.
-    const held = new Map();
-    for (const status of await this.#engine.list().catch(() => [])) {
-      held.set(status.infoHash, status);
-    }
+    // Whether the listing worked is a separate fact from what it contained,
+    // and collapsing the two into an empty map is what made this only ever
+    // able to report. A repair has to know the difference: an engine that
+    // could not be asked is not an engine holding nothing.
+    const listing = await this.#engine.list().then(
+      (all) => ({ answered: true, all }),
+      () => ({ answered: false, all: [] }),
+    );
+    const held = new Map(
+      listing.all.map((status) => [status.infoHash, status]),
+    );
+
+    // Handing archives back one at a time only makes sense against an engine
+    // that is holding some of them. One holding none of what it was just given
+    // is not suffering a per-archive fault -- it is a replacement that came up
+    // empty, or a listing that cannot be trusted -- and re-adding the whole
+    // library on the strength of a bad answer is how a node spends its start
+    // hashing everything it already had. That case is the reconnect handler's,
+    // and it is reported here rather than acted on.
+    const repairable = listing.answered && held.size > 0;
 
     let wrong = 0;
     for (const entry of entries) {
@@ -2178,11 +2208,50 @@ export class Library {
       // it absent from the engine and unreported by the very check meant to
       // notice. Absent is absent — it is neither seeding nor downloading.
       if (!status) {
+        // Handed back rather than only reported. This is the one fault here
+        // with an obvious remedy -- the archive was restorable a moment ago,
+        // since restore did it without complaint -- and the usual cause is a
+        // sidecar that died partway through and was replaced by one holding
+        // nothing. The replacement is up by the time this runs, so the second
+        // attempt is against a working engine.
+        //
+        // Once, not in a loop. An engine that refuses twice is not going to be
+        // talked round by a third try, and a restore that retried for ever
+        // would keep a node busy instead of letting it say what is wrong.
+        if (!repairable) {
+          wrong += 1;
+          console.error(
+            `${label}: restore handed this to the engine and the engine is ` +
+              'not holding it. It is neither seeding nor downloading, and ' +
+              'nothing will start it before the next restart.',
+          );
+          continue;
+        }
+
+        const again = await this.#readd(entry).then(
+          () => true,
+          (error) => {
+            console.error(
+              `${label}: could not be handed back: ${error.message}`,
+            );
+            return false;
+          },
+        );
+        const recovered = again && (await this.#engineHolds(entry.infoHash));
+        if (recovered) {
+          console.warn(
+            `${label}: the engine was not holding this after restore, so it ` +
+              'was handed back. It is loaded now.',
+          );
+          continue;
+        }
+
         wrong += 1;
         console.error(
-          `${label}: restore handed this to the engine and the engine is not ` +
-            'holding it. It is neither seeding nor downloading, and nothing ' +
-            'will start it before the next restart.',
+          `${label}: restore handed this to the engine, the engine is not ` +
+            'holding it, and handing it back did not take either. It is ' +
+            'neither seeding nor downloading. The log above this says what ' +
+            'the engine has been doing; a restart is the next thing to try.',
         );
         continue;
       }

package/src/web/index.html

@@ -7963,6 +7963,17 @@ Every piece is hashed against the ` +
           ? source.bounds.map((n) => Number(n).toFixed(2)).join(', ')
           : null;
 
+        // A pinned archive is named in the recipe by its infohash, which is
+        // forty characters of hex and tells nobody which archive it is. The
+        // filename is what the picker offered and what an operator recognises,
+        // so lead with that and let the infohash be what it resolves to --
+        // which for a pinned source is exactly what it means: this build, and
+        // no later one. Falls back to the hash for an archive that has been
+        // removed, where the name is all that is gone.
+        const pinnedArchive = kind === 'archive';
+        const label =
+          pinnedArchive && source.archiveName ? source.archiveName : source.name;
+
         const notes = [];
         if (index === all.length - 1 && all.length > 1) {
           notes.push('<span class="pill">wins</span>');
@@ -7975,7 +7986,7 @@ Every piece is hashed against the ` +
         return `
           <tr${source.resolved ? '' : ' class="bad"'}>
             <td class="muted">${index}</td>
-            <td><code>${escapeHtml(source.name)}</code>
+            <td><code>${escapeHtml(label)}</code>
                 <span class="muted">${kind}</span></td>
             <td>${
               source.resolved
@@ -7985,7 +7996,13 @@ Every piece is hashed against the ` +
                     ? `<span class="muted" title="Another recipe, evaluated for this tile and merged as one layer. It follows every later change to that stack.">recipe of ${
                         source.nested ?? 0
                       } source${source.nested === 1 ? '' : 's'}</span>`
-                    : `<code class="muted">${escapeHtml(source.archiveName ?? '')}</code>`
+                    : pinnedArchive
+                      ? `<code class="muted" title="${escapeHtml(
+                          source.infohash ?? source.name,
+                        )}">${escapeHtml(
+                          (source.infohash ?? source.name).slice(0, 12),
+                        )}…</code>`
+                      : `<code class="muted">${escapeHtml(source.archiveName ?? '')}</code>`
                 : '<span class="bad">does not resolve</span>'
             }</td>
             <td>${zooms}${