pmtiles-swarm 0.35.0 → 0.35.2

package/CHANGELOG.md

@@ -7,6 +7,57 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.35.2
+### ✨ Features and improvements
+
+### 🐞 Bug fixes
+- **A killed sidecar is reported with the signal that killed it.** `child.on('exit')` gives a null
+  code when a process dies by signal and puts the name in the *second* argument, which 0.35.1 did
+  not read — so the one case where the code carries no information printed `exited (code null)`
+  and withheld the word that does. It now says `killed by SIGKILL` or `killed by SIGTERM`, and
+  only mentions the OOM killer for `SIGKILL`, that being the one it cannot ask for politely. The
+  0.35.1 wording guessed at memory whatever had happened, which on a box with 122 GiB free sent
+  the reader somewhere there was nothing to find.
+- **The in-process hashing fallback says what it costs.** When libtorrent cannot build a torrent,
+  creation falls back to hashing in the node process — correct, since a torrent matters more than
+  its format, but not a smaller version of the same thing: it reads the whole archive in the
+  process that also serves tiles and the console. For a 698 GiB archive that is a console which
+  has apparently locked up, with nothing in the log joining it to the sidecar that died some time
+  earlier. The warning now names the size, says the hash is happening here, and says that fixing
+  libtorrent is worth more than waiting for it.
+
+## 0.35.1
+### ✨ Features and improvements
+
+### 🐞 Bug fixes
+- **A sidecar that dies is started again, instead of taking the node down with it until somebody
+  notices.** It was given up on for good: the readiness promise stayed resolved and the process
+  handle stayed null, so every call from then on threw `libtorrent sidecar is not running` — once a
+  second, indefinitely. One crash and the node stopped seeding its whole library while whatever
+  download was in front of it carried on reporting progress, which is what made it look fine.
+
+  A replacement holds nothing, so the catalogue is handed back to it as well. Coming back empty
+  would be the worse failure of the two: `list` answers, so the node reads as healthy while seeding
+  none of its archives. Only a sidecar that reached ready at least once is restarted — one that has
+  never started is a missing python or a missing binding, and retrying that per call is a spawn
+  storm against a fault no amount of retrying fixes.
+- **The sidecar's death is now in the log.** The exit code went only into the error handed to calls
+  that happened to be in flight, so a sidecar that died with nothing pending died silently. With no
+  stderr behind it — which is what being killed rather than failing looks like, the OOM killer being
+  the usual reason on a node hashing or downloading something large — there was nothing in the log
+  to say it had happened at all, only the consequences.
+- **A source that could not be read is no longer called the wrong format.** Every transport fault
+  — a refused connection, a dropped body, a 404 — came back from `identifyUrl` as `unknown`, and so
+  was reported as “this does not look like a map archive”. Seen in the field on a scheduled source
+  that had answered a HEAD seconds earlier, with `fetch failed` on the line below it in the same
+  log, which was the truth for both. Worse on a node with `allowUnknownArchives` set: an
+  unreachable URL passed the format check and the add went ahead on sixteen bytes nothing had
+  managed to read. It now says `could not read <url>: <why>`, and `allowUnknown` no longer applies
+  to it, being about format.
+- **A failed add no longer haunts `/api/adds`.** An add is registered as running before its source
+  is read, and nothing removed it if identification failed — so the console drew a download that
+  was not happening, with a cancel button that cancelled nothing, until the process restarted.
+
 ## 0.35.0
 ### ✨ Features and improvements
 - **Requires pmtiles-torrent 0.7.0, which stops the libtorrent sidecar going deaf while it works.**

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.35.0",
+  "version": "0.35.2",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/engines/composite.js

@@ -63,6 +63,22 @@ export class CompositeEngine {
     return this.#secondaries;
   }
 
+  /**
+   * Passes a reconnect handler to whichever engines can lose their backing
+   * process and start another.
+   *
+   * Only libtorrent has one today. Offered to every engine that will take it
+   * rather than reached for by name, so a second one gains this by
+   * implementing the method.
+   * @param {Function} handler - Called once a replacement is ready.
+   * @returns {void}
+   */
+  onReconnect(handler) {
+    for (const engine of [this.#primary, ...this.#secondaries]) {
+      engine.onReconnect?.(handler);
+    }
+  }
+
   /**
    * The underlying WebTorrent client, where one of the engines is WebTorrent.
    *

package/src/engines/libtorrent.js

@@ -44,6 +44,12 @@ function resolveSidecar() {
 export class LibtorrentEngine {
   #options;
   #child = null;
+  /** Whether the sidecar has ever announced itself, which decides if a death is worth retrying. */
+  #everReady = false;
+  /** Set when a working sidecar died, so the next one knows it is a replacement. */
+  #lost = false;
+  /** Called once a replacement sidecar is ready, to give it the library back. */
+  #onReconnect = null;
   #pending = new Map();
   #nextId = 1;
   #buffer = '';
@@ -90,6 +96,18 @@ export class LibtorrentEngine {
     return this.#version;
   }
 
+  /**
+   * Registers what to do once a replacement sidecar is ready.
+   *
+   * The engine can restart the process but cannot repopulate it — it does not
+   * know the catalogue. Whoever does hands this a way to put it back.
+   * @param {Function} handler - Called with no arguments; may return a promise.
+   * @returns {void}
+   */
+  onReconnect(handler) {
+    this.#onReconnect = handler;
+  }
+
   /**
    * Starts the sidecar and waits for it to report readiness.
    * @returns {Promise<void>} - Resolves once usable.
@@ -170,7 +188,33 @@ export class LibtorrentEngine {
           if (message.event === 'ready') {
             clearTimeout(timer);
             this.#version = message.libtorrent;
+            // Remembered so a later death can be told from a start that never
+            // worked. One is worth trying again, the other is a broken install
+            // and retrying it per call is a spawn storm. See the exit handler.
+            this.#everReady = true;
             resolve();
+
+            // A replacement sidecar holds nothing. Coming back empty is worse
+            // than staying down was: `list` answers, so the node looks healthy
+            // while seeding none of its library. Whoever owns the catalogue is
+            // told to hand it back.
+            //
+            // Not awaited, and deferred out of this turn: the respawn happens
+            // inside somebody's call, and restoring re-enters the engine.
+            if (this.#lost) {
+              this.#lost = false;
+              const restore = this.#onReconnect;
+              if (restore) {
+                setImmediate(() => {
+                  Promise.resolve(restore()).catch((error) =>
+                    console.error(
+                      `[libtorrent] could not hand the library back to the ` +
+                        `replacement sidecar: ${error.message}`,
+                    ),
+                  );
+                });
+              }
+            }
             continue;
           }
           this.#settle(message);
@@ -203,7 +247,7 @@ export class LibtorrentEngine {
         );
       });
 
-      child.on('exit', (code) => {
+      child.on('exit', (code, signal) => {
         clearTimeout(timer);
         this.#child = null;
 
@@ -219,9 +263,50 @@ export class LibtorrentEngine {
           return;
         }
 
-        const error = new Error(`libtorrent sidecar exited (code ${code})`);
+        // Said out loud, always. The exit code used to go only into the error
+        // handed to whatever calls happened to be in flight, so a sidecar that
+        // died with nothing pending died silently — and a death with no stderr
+        // behind it, which is what being killed by the OOM killer looks like,
+        // left nothing in the log at all. What the operator saw instead was
+        // "libtorrent sidecar is not running" once a second, for ever, with no
+        // line anywhere saying when it stopped or why.
+        // The signal, not just the code. A process killed by one exits with a
+        // null code, so reporting the code alone said "exited (code null)" —
+        // which names the one case where the code carries no information and
+        // withholds the word that does. SIGKILL is the OOM killer or someone
+        // with a big hammer; SIGTERM is something asking politely, which for a
+        // sidecar nobody meant to stop is usually the service manager taking
+        // the whole cgroup down.
+        const how = signal ? `killed by ${signal}` : `exited with code ${code}`;
+        const why = signal
+          ? signal === 'SIGKILL'
+            ? ' Nothing asks for SIGKILL politely: on a node hashing or ' +
+              'downloading a large archive this is the OOM killer, and ' +
+              '`dmesg -T | grep -i oom` will say so outright.'
+            : ''
+          : ' No stderr above this means it failed without saying why.';
+        console.error(`[libtorrent] sidecar ${how}.${why}`);
+
+        const error = new Error(`libtorrent sidecar ${how}`);
         for (const { reject: fail } of this.#pending.values()) fail(error);
         this.#pending.clear();
+
+        // A sidecar that worked and then died is worth starting again. It was
+        // not: #ready stayed resolved and #child stayed null, so every call
+        // from then on threw "libtorrent sidecar is not running" until the
+        // whole service was restarted by hand — one crash and the node stopped
+        // seeding its entire library, with the download in front of it still
+        // running and reporting progress as though nothing had happened.
+        //
+        // Only when it had reached ready at least once. A sidecar that has
+        // never started is a missing python or a missing binding, and
+        // respawning that on every call is a spawn storm against a fault no
+        // amount of retrying fixes.
+        if (this.#everReady) {
+          this.#ready = null;
+          this.#lost = true;
+          return;
+        }
         reject(error);
       });
     });

package/src/identify.js

@@ -92,12 +92,33 @@ export async function identifyFile(filePath) {
  * @returns {Promise<ArchiveKind>} - What it is.
  */
 export async function identifyUrl(url, options = {}) {
+  // "I could not read this" and "this is not an archive" are different
+  // answers, and returning UNKNOWN for both made every network fault report
+  // itself as a format one. Seen in the field as a source that had answered a
+  // HEAD seconds earlier being refused with "this does not look like a map
+  // archive", while the line under it in the same log said "fetch failed" —
+  // which was the truth for both of them.
+  //
+  // Worse than a confusing message on a node with allowUnknownArchives set:
+  // an unreachable URL passed assertPublishable as an unknown format and the
+  // add went ahead, on sixteen bytes nothing had managed to read.
+  let response;
   try {
-    const response = await fetch(url, {
+    response = await fetch(url, {
       headers: { range: 'bytes=0-15' },
       signal: options.signal,
     });
-    if (!response.ok && response.status !== 206) return UNKNOWN;
+  } catch (error) {
+    // An abort is the caller's own doing and already means something to it.
+    if (error?.name === 'AbortError') throw error;
+    throw unreachable(url, error?.message ?? String(error), error);
+  }
+
+  if (!response.ok && response.status !== 206) {
+    throw unreachable(url, `${response.status} ${response.statusText}`.trim());
+  }
+
+  try {
     if (!response.body) return UNKNOWN;
 
     // Read the first bytes off the stream and then stop, rather than
@@ -121,11 +142,33 @@ export async function identifyUrl(url, options = {}) {
     }
 
     return identifyBytes(head.subarray(0, filled));
-  } catch {
-    return UNKNOWN;
+  } catch (error) {
+    // The body died partway through the sixteen bytes. Still a transport
+    // failure rather than a verdict on the format.
+    if (error?.name === 'AbortError') throw error;
+    throw unreachable(url, error?.message ?? String(error), error);
   }
 }
 
+/**
+ * The error for a source that could not be read at all.
+ * @param {string} url - What was being read.
+ * @param {string} why - The transport's own account of it.
+ * @param {Error} [cause] - The original failure, kept for a stack.
+ * @returns {Error} - Ready to throw.
+ */
+function unreachable(url, why, cause) {
+  const error = new Error(
+    `could not read ${url}: ${why}. Nothing was published — this says the ` +
+      'source could not be reached, not that it is the wrong format, so ' +
+      'allowUnknown does not apply.',
+    cause ? { cause } : undefined,
+  );
+  // A source this node could not reach is not the caller's bad request.
+  error.status = 502;
+  return error;
+}
+
 /**
  * Checks a file is something this node is willing to publish.
  * @param {ArchiveKind} identified - The result of identifying it.

package/src/index.js

@@ -272,6 +272,20 @@ PMTILES_SWARM_PUBLIC_URL
     );
   }
 
+  // And again if the engine loses its backing process and starts another. A
+  // replacement holds nothing, so without this the node would come back
+  // answering every call and seeding none of its library — which reads as
+  // healthy, and is the worse of the two failures.
+  engine.onReconnect?.(async () => {
+    const held = catalog.list().length;
+    if (held === 0) return;
+    const { restored, failed } = await library.restore();
+    console.log(
+      `[restore] ${restored} of ${held} archives handed back to the replacement` +
+        (failed > 0 ? ` (${failed} could not be)` : ''),
+    );
+  });
+
   // What this node has served. In memory and bounded, so it costs the same
   // after a billion tiles as after ten; `tileStats.recent: 0` keeps the
   // counters and drops the per-request ring, and `false` turns it off.

package/src/library.js

@@ -749,10 +749,22 @@ export class Library {
     // Probing reads only the header and directory, so this is cheap even
     // against a multi-gigabyte archive — worth doing before committing to a
     // download that may take hours.
-    const identified = await identifyUrl(url, { signal: controller.signal });
-    assertPublishable(identified, {
-      allowUnknown: options.allowUnknown ?? this.#config.allowUnknownArchives,
-    });
+    //
+    // Cleaned up on the way out, because this is the one stretch that can fail
+    // with the add already listed as running. It left an entry behind that
+    // nothing would ever remove: /api/adds reported a download that was not
+    // happening, the console drew it under "being added" with a cancel button
+    // that cancelled nothing, and it stayed until the process restarted.
+    let identified;
+    try {
+      identified = await identifyUrl(url, { signal: controller.signal });
+      assertPublishable(identified, {
+        allowUnknown: options.allowUnknown ?? this.#config.allowUnknownArchives,
+      });
+    } catch (error) {
+      this.#running.delete(url);
+      throw error;
+    }
 
     // Everything a caller can do something about has now been checked: the URL
     // answers, it is an archive of a kind this will publish, and it is not a

package/src/torrent-create.js

@@ -671,7 +671,21 @@ async function buildTorrent(input, name, size, options) {
       };
     } catch (error) {
       // A torrent is more important than the format of a torrent.
-      console.warn(`[create] ${error.message}; falling back to a v1 torrent`);
+      //
+      // Said with what it costs, though, because the fallback is not a smaller
+      // version of the same thing. libtorrent hashes in its own process; this
+      // hashes in *this* one, so an archive large enough to be worth handing
+      // to libtorrent is now being read end to end by the process also serving
+      // tiles and the console — which is how a sidecar dying mid-create turns
+      // into a console that has apparently locked up, with nothing in the log
+      // connecting the two.
+      const gib = size ? ` (${(size / 1024 ** 3).toFixed(1)} GiB)` : '';
+      console.warn(
+        `[create] ${error.message}; hashing${gib} in this process instead, ` +
+          'which is slower, holds no hybrid v2 layers, and competes with ' +
+          'everything else this node is doing. Fixing whatever stopped ' +
+          'libtorrent is worth more than waiting for this.',
+      );
     }
   }