pmtiles-swarm 0.35.0 → 0.35.1

package/CHANGELOG.md

@@ -7,6 +7,38 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.35.1
+### ✨ Features and improvements
+
+### 🐞 Bug fixes
+- **A sidecar that dies is started again, instead of taking the node down with it until somebody
+  notices.** It was given up on for good: the readiness promise stayed resolved and the process
+  handle stayed null, so every call from then on threw `libtorrent sidecar is not running` — once a
+  second, indefinitely. One crash and the node stopped seeding its whole library while whatever
+  download was in front of it carried on reporting progress, which is what made it look fine.
+
+  A replacement holds nothing, so the catalogue is handed back to it as well. Coming back empty
+  would be the worse failure of the two: `list` answers, so the node reads as healthy while seeding
+  none of its archives. Only a sidecar that reached ready at least once is restarted — one that has
+  never started is a missing python or a missing binding, and retrying that per call is a spawn
+  storm against a fault no amount of retrying fixes.
+- **The sidecar's death is now in the log.** The exit code went only into the error handed to calls
+  that happened to be in flight, so a sidecar that died with nothing pending died silently. With no
+  stderr behind it — which is what being killed rather than failing looks like, the OOM killer being
+  the usual reason on a node hashing or downloading something large — there was nothing in the log
+  to say it had happened at all, only the consequences.
+- **A source that could not be read is no longer called the wrong format.** Every transport fault
+  — a refused connection, a dropped body, a 404 — came back from `identifyUrl` as `unknown`, and so
+  was reported as “this does not look like a map archive”. Seen in the field on a scheduled source
+  that had answered a HEAD seconds earlier, with `fetch failed` on the line below it in the same
+  log, which was the truth for both. Worse on a node with `allowUnknownArchives` set: an
+  unreachable URL passed the format check and the add went ahead on sixteen bytes nothing had
+  managed to read. It now says `could not read <url>: <why>`, and `allowUnknown` no longer applies
+  to it, being about format.
+- **A failed add no longer haunts `/api/adds`.** An add is registered as running before its source
+  is read, and nothing removed it if identification failed — so the console drew a download that
+  was not happening, with a cancel button that cancelled nothing, until the process restarted.
+
 ## 0.35.0
 ### ✨ Features and improvements
 - **Requires pmtiles-torrent 0.7.0, which stops the libtorrent sidecar going deaf while it works.**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.35.0",
+  "version": "0.35.1",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/engines/composite.js

@@ -63,6 +63,22 @@ export class CompositeEngine {
     return this.#secondaries;
   }
 
+  /**
+   * Passes a reconnect handler to whichever engines can lose their backing
+   * process and start another.
+   *
+   * Only libtorrent has one today. Offered to every engine that will take it
+   * rather than reached for by name, so a second one gains this by
+   * implementing the method.
+   * @param {Function} handler - Called once a replacement is ready.
+   * @returns {void}
+   */
+  onReconnect(handler) {
+    for (const engine of [this.#primary, ...this.#secondaries]) {
+      engine.onReconnect?.(handler);
+    }
+  }
+
   /**
    * The underlying WebTorrent client, where one of the engines is WebTorrent.
    *

package/src/engines/libtorrent.js

@@ -44,6 +44,12 @@ function resolveSidecar() {
 export class LibtorrentEngine {
   #options;
   #child = null;
+  /** Whether the sidecar has ever announced itself, which decides if a death is worth retrying. */
+  #everReady = false;
+  /** Set when a working sidecar died, so the next one knows it is a replacement. */
+  #lost = false;
+  /** Called once a replacement sidecar is ready, to give it the library back. */
+  #onReconnect = null;
   #pending = new Map();
   #nextId = 1;
   #buffer = '';
@@ -90,6 +96,18 @@ export class LibtorrentEngine {
     return this.#version;
   }
 
+  /**
+   * Registers what to do once a replacement sidecar is ready.
+   *
+   * The engine can restart the process but cannot repopulate it — it does not
+   * know the catalogue. Whoever does hands this a way to put it back.
+   * @param {Function} handler - Called with no arguments; may return a promise.
+   * @returns {void}
+   */
+  onReconnect(handler) {
+    this.#onReconnect = handler;
+  }
+
   /**
    * Starts the sidecar and waits for it to report readiness.
    * @returns {Promise<void>} - Resolves once usable.
@@ -170,7 +188,33 @@ export class LibtorrentEngine {
           if (message.event === 'ready') {
             clearTimeout(timer);
             this.#version = message.libtorrent;
+            // Remembered so a later death can be told from a start that never
+            // worked. One is worth trying again, the other is a broken install
+            // and retrying it per call is a spawn storm. See the exit handler.
+            this.#everReady = true;
             resolve();
+
+            // A replacement sidecar holds nothing. Coming back empty is worse
+            // than staying down was: `list` answers, so the node looks healthy
+            // while seeding none of its library. Whoever owns the catalogue is
+            // told to hand it back.
+            //
+            // Not awaited, and deferred out of this turn: the respawn happens
+            // inside somebody's call, and restoring re-enters the engine.
+            if (this.#lost) {
+              this.#lost = false;
+              const restore = this.#onReconnect;
+              if (restore) {
+                setImmediate(() => {
+                  Promise.resolve(restore()).catch((error) =>
+                    console.error(
+                      `[libtorrent] could not hand the library back to the ` +
+                        `replacement sidecar: ${error.message}`,
+                    ),
+                  );
+                });
+              }
+            }
             continue;
           }
           this.#settle(message);
@@ -219,9 +263,39 @@ export class LibtorrentEngine {
           return;
         }
 
+        // Said out loud, always. The exit code used to go only into the error
+        // handed to whatever calls happened to be in flight, so a sidecar that
+        // died with nothing pending died silently — and a death with no stderr
+        // behind it, which is what being killed by the OOM killer looks like,
+        // left nothing in the log at all. What the operator saw instead was
+        // "libtorrent sidecar is not running" once a second, for ever, with no
+        // line anywhere saying when it stopped or why.
+        console.error(
+          `[libtorrent] sidecar exited (code ${code}). No stderr above this ` +
+            'means it was killed rather than failing — the OOM killer is the ' +
+            'usual reason on a node hashing or downloading a large archive.',
+        );
+
         const error = new Error(`libtorrent sidecar exited (code ${code})`);
         for (const { reject: fail } of this.#pending.values()) fail(error);
         this.#pending.clear();
+
+        // A sidecar that worked and then died is worth starting again. It was
+        // not: #ready stayed resolved and #child stayed null, so every call
+        // from then on threw "libtorrent sidecar is not running" until the
+        // whole service was restarted by hand — one crash and the node stopped
+        // seeding its entire library, with the download in front of it still
+        // running and reporting progress as though nothing had happened.
+        //
+        // Only when it had reached ready at least once. A sidecar that has
+        // never started is a missing python or a missing binding, and
+        // respawning that on every call is a spawn storm against a fault no
+        // amount of retrying fixes.
+        if (this.#everReady) {
+          this.#ready = null;
+          this.#lost = true;
+          return;
+        }
         reject(error);
       });
     });

package/src/identify.js

@@ -92,12 +92,33 @@ export async function identifyFile(filePath) {
  * @returns {Promise<ArchiveKind>} - What it is.
  */
 export async function identifyUrl(url, options = {}) {
+  // "I could not read this" and "this is not an archive" are different
+  // answers, and returning UNKNOWN for both made every network fault report
+  // itself as a format one. Seen in the field as a source that had answered a
+  // HEAD seconds earlier being refused with "this does not look like a map
+  // archive", while the line under it in the same log said "fetch failed" —
+  // which was the truth for both of them.
+  //
+  // Worse than a confusing message on a node with allowUnknownArchives set:
+  // an unreachable URL passed assertPublishable as an unknown format and the
+  // add went ahead, on sixteen bytes nothing had managed to read.
+  let response;
   try {
-    const response = await fetch(url, {
+    response = await fetch(url, {
       headers: { range: 'bytes=0-15' },
       signal: options.signal,
     });
-    if (!response.ok && response.status !== 206) return UNKNOWN;
+  } catch (error) {
+    // An abort is the caller's own doing and already means something to it.
+    if (error?.name === 'AbortError') throw error;
+    throw unreachable(url, error?.message ?? String(error), error);
+  }
+
+  if (!response.ok && response.status !== 206) {
+    throw unreachable(url, `${response.status} ${response.statusText}`.trim());
+  }
+
+  try {
     if (!response.body) return UNKNOWN;
 
     // Read the first bytes off the stream and then stop, rather than
@@ -121,11 +142,33 @@ export async function identifyUrl(url, options = {}) {
     }
 
     return identifyBytes(head.subarray(0, filled));
-  } catch {
-    return UNKNOWN;
+  } catch (error) {
+    // The body died partway through the sixteen bytes. Still a transport
+    // failure rather than a verdict on the format.
+    if (error?.name === 'AbortError') throw error;
+    throw unreachable(url, error?.message ?? String(error), error);
   }
 }
 
+/**
+ * The error for a source that could not be read at all.
+ * @param {string} url - What was being read.
+ * @param {string} why - The transport's own account of it.
+ * @param {Error} [cause] - The original failure, kept for a stack.
+ * @returns {Error} - Ready to throw.
+ */
+function unreachable(url, why, cause) {
+  const error = new Error(
+    `could not read ${url}: ${why}. Nothing was published — this says the ` +
+      'source could not be reached, not that it is the wrong format, so ' +
+      'allowUnknown does not apply.',
+    cause ? { cause } : undefined,
+  );
+  // A source this node could not reach is not the caller's bad request.
+  error.status = 502;
+  return error;
+}
+
 /**
  * Checks a file is something this node is willing to publish.
  * @param {ArchiveKind} identified - The result of identifying it.

package/src/index.js

@@ -272,6 +272,20 @@ PMTILES_SWARM_PUBLIC_URL
     );
   }
 
+  // And again if the engine loses its backing process and starts another. A
+  // replacement holds nothing, so without this the node would come back
+  // answering every call and seeding none of its library — which reads as
+  // healthy, and is the worse of the two failures.
+  engine.onReconnect?.(async () => {
+    const held = catalog.list().length;
+    if (held === 0) return;
+    const { restored, failed } = await library.restore();
+    console.log(
+      `[restore] ${restored} of ${held} archives handed back to the replacement` +
+        (failed > 0 ? ` (${failed} could not be)` : ''),
+    );
+  });
+
   // What this node has served. In memory and bounded, so it costs the same
   // after a billion tiles as after ten; `tileStats.recent: 0` keeps the
   // counters and drops the per-request ring, and `false` turns it off.

package/src/library.js

@@ -749,10 +749,22 @@ export class Library {
     // Probing reads only the header and directory, so this is cheap even
     // against a multi-gigabyte archive — worth doing before committing to a
     // download that may take hours.
-    const identified = await identifyUrl(url, { signal: controller.signal });
-    assertPublishable(identified, {
-      allowUnknown: options.allowUnknown ?? this.#config.allowUnknownArchives,
-    });
+    //
+    // Cleaned up on the way out, because this is the one stretch that can fail
+    // with the add already listed as running. It left an entry behind that
+    // nothing would ever remove: /api/adds reported a download that was not
+    // happening, the console drew it under "being added" with a cancel button
+    // that cancelled nothing, and it stayed until the process restarted.
+    let identified;
+    try {
+      identified = await identifyUrl(url, { signal: controller.signal });
+      assertPublishable(identified, {
+        allowUnknown: options.allowUnknown ?? this.#config.allowUnknownArchives,
+      });
+    } catch (error) {
+      this.#running.delete(url);
+      throw error;
+    }
 
     // Everything a caller can do something about has now been checked: the URL
     // answers, it is an archive of a kind this will publish, and it is not a