pmtiles-swarm 0.98.2 → 0.99.0

package/CHANGELOG.md

@@ -7,6 +7,51 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.99.0
+### ✨ Features and improvements
+- **A sidecar crash now names the frame it happened in.** `pmtiles-torrent` 0.11.0 arms
+  `faulthandler` in the sidecar, so a segfault writes a Python traceback to stderr on its way out
+  and this node forwards it into the log line by line, beside the `sidecar killed by SIGSEGV` that
+  used to be the whole story. Nothing to configure; it costs nothing until the process faults.
+
+  Worth having because that signal is otherwise the only evidence. On the node this came from it
+  appeared several times per start for eleven days, and each crash takes the archives handed over
+  before it — which is what leaves a library reading `not loaded` after a restart.
+
+### 🐞 Bug fixes
+- **An archive the engine took and then did not keep is handed back, rather than only reported.**
+  The seeding check has been able to spot this for a while and its own message admitted the rest:
+  "nothing will start it before the next restart". It was right, which is why the cure was always
+  another restart. The usual cause is a sidecar that dies partway through a restore — the
+  replacement holds nothing, so the archives handed over before it died are absent while the ones
+  after it are fine, and `add` resolving is no evidence that anything was kept.
+
+  Those are now handed back once and re-checked, and the log says whether it took. Once, not in a
+  loop: an engine that refuses twice will not be talked round by a third try. And only when the
+  engine is holding *some* of the library — one holding none of it, or one that could not be listed
+  at all, is not suffering a per-archive fault, and re-adding everything on the strength of that
+  answer is how a node spends its start hashing what it already had.
+- **A pinned archive in a stack was listed by its infohash.** Forty characters of hex, where the
+  picker that offered it had shown a filename. The row leads with the archive's name now and shows
+  the infohash as what it resolves to, which for a pinned source is exactly what it means: this
+  build and no later one.
+
+## 0.98.3
+### ✨ Features and improvements
+
+### 🐞 Bug fixes
+- **A stack could not be pointed at a completed MBTiles archive.** The source picker offered only
+  PMTiles, so an MBTiles the node was perfectly able to serve tiles from was simply absent from the
+  menu. It offers a complete one now, by the same rule the tile route enforces — and still hides one
+  that is arriving, since a stack naming that would answer 503 for every tile until the download
+  finished.
+- **A planet dump was offered as a stack source.** The same filter read an archive with no recorded
+  kind as PMTiles, which is not a harmless default: a mirrored `.osm.pbf` appeared in the menu, and
+  naming it would have produced a recipe that failed at its first tile. The kind now falls back to
+  the filename before anything is assumed, which is how `prewarm.js` resolves it after the identical
+  mistake sent every mirrored dump round the header-read backoff for ever. Such an archive is also
+  labelled `unknown · not servable` in the list, where it previously said nothing at all.
+
 ## 0.98.2
 ### ✨ Features and improvements
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.98.2",
+  "version": "0.99.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",
@@ -49,7 +49,7 @@
     "maplibre-gl": "^6.2.0",
     "parse-torrent": "^11.0.24",
     "pmtiles": "^4.4.1",
-    "pmtiles-torrent": "^0.10.2",
+    "pmtiles-torrent": "^0.11.0",
     "webtorrent": "^3.0.21"
   },
   "engines": {

package/src/library.js

@@ -2154,16 +2154,46 @@ export class Library {
    * @param {object[]} entries - The entries restore worked through.
    * @returns {Promise<void>} - Resolves once every claim has been checked.
    */
+  /**
+   * Whether the engine is holding this archive right now.
+   *
+   * Asked of one archive rather than by re-listing the library: this runs only
+   * for an archive that has just been handed back, which on a healthy node is
+   * never.
+   * @param {string} infoHash - The archive.
+   * @returns {Promise<boolean>} - Whether the engine has it.
+   */
+  async #engineHolds(infoHash) {
+    const status = await this.#engine.get?.(infoHash).catch(() => null);
+    return Boolean(status);
+  }
+
   async #verifySeeding(entries) {
     if (entries.length === 0) return;
 
     // One listing rather than a status call each: this runs over the whole
     // library on every start, and a round trip per archive is a cost paid by
     // every node to catch a fault most of them do not have.
-    const held = new Map();
-    for (const status of await this.#engine.list().catch(() => [])) {
-      held.set(status.infoHash, status);
-    }
+    // Whether the listing worked is a separate fact from what it contained,
+    // and collapsing the two into an empty map is what made this only ever
+    // able to report. A repair has to know the difference: an engine that
+    // could not be asked is not an engine holding nothing.
+    const listing = await this.#engine.list().then(
+      (all) => ({ answered: true, all }),
+      () => ({ answered: false, all: [] }),
+    );
+    const held = new Map(
+      listing.all.map((status) => [status.infoHash, status]),
+    );
+
+    // Handing archives back one at a time only makes sense against an engine
+    // that is holding some of them. One holding none of what it was just given
+    // is not suffering a per-archive fault -- it is a replacement that came up
+    // empty, or a listing that cannot be trusted -- and re-adding the whole
+    // library on the strength of a bad answer is how a node spends its start
+    // hashing everything it already had. That case is the reconnect handler's,
+    // and it is reported here rather than acted on.
+    const repairable = listing.answered && held.size > 0;
 
     let wrong = 0;
     for (const entry of entries) {
@@ -2178,11 +2208,50 @@ export class Library {
       // it absent from the engine and unreported by the very check meant to
       // notice. Absent is absent — it is neither seeding nor downloading.
       if (!status) {
+        // Handed back rather than only reported. This is the one fault here
+        // with an obvious remedy -- the archive was restorable a moment ago,
+        // since restore did it without complaint -- and the usual cause is a
+        // sidecar that died partway through and was replaced by one holding
+        // nothing. The replacement is up by the time this runs, so the second
+        // attempt is against a working engine.
+        //
+        // Once, not in a loop. An engine that refuses twice is not going to be
+        // talked round by a third try, and a restore that retried for ever
+        // would keep a node busy instead of letting it say what is wrong.
+        if (!repairable) {
+          wrong += 1;
+          console.error(
+            `${label}: restore handed this to the engine and the engine is ` +
+              'not holding it. It is neither seeding nor downloading, and ' +
+              'nothing will start it before the next restart.',
+          );
+          continue;
+        }
+
+        const again = await this.#readd(entry).then(
+          () => true,
+          (error) => {
+            console.error(
+              `${label}: could not be handed back: ${error.message}`,
+            );
+            return false;
+          },
+        );
+        const recovered = again && (await this.#engineHolds(entry.infoHash));
+        if (recovered) {
+          console.warn(
+            `${label}: the engine was not holding this after restore, so it ` +
+              'was handed back. It is loaded now.',
+          );
+          continue;
+        }
+
         wrong += 1;
         console.error(
-          `${label}: restore handed this to the engine and the engine is not ` +
-            'holding it. It is neither seeding nor downloading, and nothing ' +
-            'will start it before the next restart.',
+          `${label}: restore handed this to the engine, the engine is not ` +
+            'holding it, and handing it back did not take either. It is ' +
+            'neither seeding nor downloading. The log above this says what ' +
+            'the engine has been doing; a restart is the next thing to try.',
         );
         continue;
       }

package/src/web/index.html

@@ -1836,11 +1836,31 @@
        * @returns {boolean} - Whether to offer the tile endpoints.
        */
       const isServable = (entry) => {
-        const kind = entry.kind ?? 'pmtiles';
+        const kind = kindOf(entry);
         if (kind === 'pmtiles') return true;
         return kind === 'mbtiles' && entry.complete === true;
       };
 
+      /**
+       * What kind of archive this is, from the record or failing that the name.
+       *
+       * An entry from before the kind was recorded carries none, and reading
+       * that as PMTiles is not a harmless default: it is how a planet dump ends
+       * up labelled servable and offered as a stack source, where naming it
+       * would produce a recipe that fails at its first tile. `prewarm.js`
+       * resolves it exactly this way, after the same mistake sent every
+       * mirrored .osm.pbf round the header-read backoff for ever.
+       * @param {object} entry - A catalog entry.
+       * @returns {string} - 'pmtiles', 'mbtiles', or 'unknown'.
+       */
+      const kindOf = (entry) => {
+        if (entry.kind) return entry.kind;
+        const name = entry.name ?? '';
+        if (/\.pmtiles$/i.test(name)) return 'pmtiles';
+        if (/\.mbtiles$/i.test(name)) return 'mbtiles';
+        return 'unknown';
+      };
+
       /**
        * What to say under the size about an archive that is not PMTiles.
        *
@@ -1852,7 +1872,7 @@
        * @returns {string} - The note, or empty for a PMTiles archive.
        */
       const kindNote = (entry) => {
-        const kind = entry.kind ?? 'pmtiles';
+        const kind = kindOf(entry);
         if (kind === 'pmtiles') return '';
         if (kind !== 'mbtiles') return `${escapeHtml(kind)} · not servable`;
         return entry.complete === true
@@ -2563,7 +2583,7 @@
         // anything for an archive read out of the swarm. An MBTiles is read
         // whole or not at all — once it is here there is nothing left to
         // fetch, and before that there is no tile to warm.
-        const warmable = (entry.kind ?? 'pmtiles') === 'pmtiles';
+        const warmable = kindOf(entry) === 'pmtiles';
         const torrentUrl = `${base}/archives/${entry.infoHash}/archive.torrent`;
         const summary = entry.pmtiles ?? {};
         const mode = entry.mode ?? 'mirror';
@@ -2648,7 +2668,7 @@
           ${
             servable
               ? `<div class="sub" style="margin-bottom:1rem"><code>${escapeHtml(tileJson)}</code></div>`
-              : (entry.kind ?? 'pmtiles') === 'mbtiles'
+              : kindOf(entry) === 'mbtiles'
                 ? `<div class="sub" style="margin-bottom:1rem">
                      This MBTiles archive is still arriving, so it has no tile endpoint yet. It is
                      SQLite, which cannot be read a byte range at a time out of the swarm the way
@@ -7943,6 +7963,17 @@ Every piece is hashed against the ` +
           ? source.bounds.map((n) => Number(n).toFixed(2)).join(', ')
           : null;
 
+        // A pinned archive is named in the recipe by its infohash, which is
+        // forty characters of hex and tells nobody which archive it is. The
+        // filename is what the picker offered and what an operator recognises,
+        // so lead with that and let the infohash be what it resolves to --
+        // which for a pinned source is exactly what it means: this build, and
+        // no later one. Falls back to the hash for an archive that has been
+        // removed, where the name is all that is gone.
+        const pinnedArchive = kind === 'archive';
+        const label =
+          pinnedArchive && source.archiveName ? source.archiveName : source.name;
+
         const notes = [];
         if (index === all.length - 1 && all.length > 1) {
           notes.push('<span class="pill">wins</span>');
@@ -7955,7 +7986,7 @@ Every piece is hashed against the ` +
         return `
           <tr${source.resolved ? '' : ' class="bad"'}>
             <td class="muted">${index}</td>
-            <td><code>${escapeHtml(source.name)}</code>
+            <td><code>${escapeHtml(label)}</code>
                 <span class="muted">${kind}</span></td>
             <td>${
               source.resolved
@@ -7965,7 +7996,13 @@ Every piece is hashed against the ` +
                     ? `<span class="muted" title="Another recipe, evaluated for this tile and merged as one layer. It follows every later change to that stack.">recipe of ${
                         source.nested ?? 0
                       } source${source.nested === 1 ? '' : 's'}</span>`
-                    : `<code class="muted">${escapeHtml(source.archiveName ?? '')}</code>`
+                    : pinnedArchive
+                      ? `<code class="muted" title="${escapeHtml(
+                          source.infohash ?? source.name,
+                        )}">${escapeHtml(
+                          (source.infohash ?? source.name).slice(0, 12),
+                        )}…</code>`
+                      : `<code class="muted">${escapeHtml(source.archiveName ?? '')}</code>`
                 : '<span class="bad">does not resolve</span>'
             }</td>
             <td>${zooms}${
@@ -8175,10 +8212,12 @@ Every piece is hashed against the ` +
           // missing from a menu the operator is looking at. This is the same
           // endpoint the Archives view reads, and it answers a bare array.
           const held = await api('/api/torrents');
+          // The same rule the tile route enforces, because a stack source is
+          // read through exactly that path: a complete MBTiles belongs here,
+          // and one still arriving does not -- a stack naming it would answer
+          // 503 for every tile until the download finished.
           const free = (held ?? []).filter(
-            (entry) =>
-              !taken.has(entry.infoHash) &&
-              (entry.kind ?? 'pmtiles') === 'pmtiles',
+            (entry) => !taken.has(entry.infoHash) && isServable(entry),
           );
           if (free.length) {
             groups.push(