pmtiles-swarm 0.71.0 → 0.72.0

package/CHANGELOG.md

@@ -7,6 +7,45 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.72.0
+### ✨ Features and improvements
+- **An unfinished export is picked up when the node starts.** The checkpoint was always there;
+  finding it again meant somebody remembering to press the button. That is fine for an export
+  stopped on purpose and wrong for one a crash took, which is the case that costs the most and
+  gives the least warning.
+
+  The checkpoint now records what the job was - what the archive is called, what the file is
+  called, where it was going, what it is filed under - because none of that can be worked out from
+  the tiles on disk, and a resumed export has to be the one somebody asked for rather than a new
+  one with today's date on it.
+
+  Only where the recipe still resolves to what it did. `bakeRevision` covers what each source
+  became, so a rebuilt source means the checkpoint holds half of a map that no longer exists;
+  that is left alone and reported, to be discarded when somebody exports again deliberately.
+  `stacks.resumeExports` turns it off for a node where hours of merging should never begin
+  without being asked for.
+
+### 🐞 Bug fixes
+- **The node was exiting when a peer wire outlived its torrent.** Twice in one evening on a real
+  node, both times killing an export that was hours in:
+
+      torrent.js:2092  this.client._debugId    -> reading '_debugId' of null
+      peer.js:201      this.swarm.client.dht   -> reading 'dht' of null
+
+  WebTorrent nulls `torrent.client` when a torrent is destroyed and does not always tear down that
+  torrent's peer wires with it. One fired a keep-alive timeout afterwards and the other finished a
+  handshake, and both reached through the dead reference. They happen inside a timer or a socket
+  callback, so there is no promise to reject and no call of ours to wrap - it is an uncaught
+  exception, and Node's answer to that is to exit. Under `Restart=always` that reads as a
+  mysterious restart rather than as a crash.
+
+  `src/crash-guard.js` survives exactly that shape and nothing else: a TypeError, about one of a
+  named list of properties, raised from a frame inside the torrent libraries. Every other uncaught
+  exception still stops the process, because one that survives everything is one that lies about
+  its own state. The two stacks from the journal are in the tests verbatim, alongside the near
+  misses that must still be fatal - the same error from our own code, a different error from
+  theirs.
+
 ## 0.71.0
 ### ✨ Features and improvements
 - **The preview comes back to where it was opened from.** The link out of a preview said

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.71.0",
+  "version": "0.72.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/bake-jobs.js

@@ -1,3 +1,4 @@
+import fs from 'node:fs/promises';
 import path from 'node:path';
 import {
   assertBakeable,
@@ -108,6 +109,77 @@ export class BakeManager {
     return true;
   }
 
+  /**
+   * Picks up exports a previous run did not finish.
+   *
+   * A checkpoint is the hours already spent, and finding one again used to
+   * mean somebody remembering to press the button. That is fine for an export
+   * stopped on purpose and wrong for one a crash took -- which is the case
+   * that costs the most and gives the least warning.
+   *
+   * Only where the recipe still resolves to what it did. `bakeRevision` covers
+   * what each source became, so a rebuilt source means the checkpoint holds
+   * half of a map that no longer exists; that is left alone rather than
+   * continued, and discarded when somebody exports again deliberately.
+   * @param {Function} resolve - `(stackId) => resolved stack | null`.
+   * @returns {Promise<object[]>} - The jobs started.
+   */
+  async resumeAll(resolve) {
+    const started = [];
+
+    for (const root of this.#workRoots()) {
+      const directory = path.join(root, WORK_DIR);
+      const found = await fs.readdir(directory).catch(() => []);
+
+      for (const stackId of found) {
+        if (this.#jobs.has(stackId)) continue;
+        const state = await fs
+          .readFile(path.join(directory, stackId, 'bake-state.json'), 'utf8')
+          .then((raw) => JSON.parse(raw))
+          .catch(() => null);
+        // Written by a version that did not record what the job was. There is
+        // nothing to reproduce it from, so it waits for a person.
+        if (!state?.describe) continue;
+
+        const resolved = resolve(stackId);
+        if (!resolved || bakeRevision(resolved) !== state.revision) continue;
+
+        try {
+          const job = await this.start({ resolved, ...state.describe });
+          started.push(job);
+          console.log(
+            `[bake] picking up ${stackId} where it stopped: ` +
+              `${state.written ?? 0} tiles already merged`,
+          );
+        } catch (error) {
+          // A stack that cannot be baked now -- no codec, sources gone -- is
+          // said out loud and left. Its checkpoint is still there.
+          console.warn(`[bake] could not resume ${stackId}: ${error.message}`);
+        }
+      }
+    }
+
+    return started;
+  }
+
+  /**
+   * Everywhere an export might have left work.
+   *
+   * The working directory follows the destination, and a destination is
+   * whatever was chosen at the time -- so this is every place one could have
+   * been: the named locations, the default save path, and the data directory,
+   * where exports worked before the working directory moved.
+   * @returns {string[]} - Roots to look under, without repeats.
+   */
+  #workRoots() {
+    const roots = [
+      ...(this.#config.locations ?? []).map((one) => one?.path),
+      this.#config.savePath,
+      this.#config.dataDir,
+    ].filter(Boolean);
+    return [...new Set(roots.map((one) => path.resolve(one)))];
+  }
+
   /**
    * Stops every running bake and waits for each to write its checkpoint.
    *
@@ -341,6 +413,17 @@ export class BakeManager {
       header: { format },
       pauseMs: this.#config.stacks?.bakePauseMs ?? 0,
       concurrency: this.#concurrency(),
+      // Written into the checkpoint in the shape `start` takes, so picking one
+      // up is handing it back rather than reconstructing it. Nothing here can
+      // be worked out from the tiles on disk: what the archive is called, what
+      // the file is called, where it was going, what it is filed under.
+      describe: {
+        name: job.archiveName,
+        filename: job.name,
+        publishDir: job.publishDir ?? null,
+        description: options.description ?? null,
+        categories: options.categories ?? null,
+      },
       metadata: {
         name: job.archiveName,
         // Only what was asked for. Falling back to the recipe's own

package/src/bake.js

@@ -468,6 +468,11 @@ export async function bakeStack(options) {
     checkpointSeconds = DEFAULT_CHECKPOINT_SECONDS,
     pauseMs = 0,
     concurrency = DEFAULT_CONCURRENCY,
+    // Written into the checkpoint and handed back by `readCheckpoint`. A
+    // resumed export has to reproduce the one somebody asked for -- its name,
+    // where it was going, what it was filed under -- and none of that can be
+    // worked out from the tiles on disk.
+    describe,
   } = options;
   const batchSize = Math.max(1, Math.floor(concurrency));
 
@@ -516,6 +521,7 @@ export async function bakeStack(options) {
         dataBytes: writer.dataBytes,
         addressed: writer.addressedTiles,
         clustered: writer.clustered,
+        ...(describe ? { describe } : {}),
       },
       writer.entries,
       persisted,

package/src/config.js

@@ -388,6 +388,19 @@ const DEFAULTS = {
      * not clustered is bad at the one thing PMTiles is for.
      */
     bakeConcurrency: 4,
+    /**
+     * Whether an unfinished export is picked up when the node starts.
+     *
+     * A checkpoint is the hours already spent, and the case that costs most is
+     * the one nobody chose: a crash, or a restart in the middle of a run. On
+     * by default for that reason. Off for a node where hours of merging should
+     * never begin without somebody asking for it.
+     *
+     * Only where the recipe still resolves to what it did. A rebuilt source
+     * means the checkpoint holds half of a map that no longer exists, and that
+     * is left alone either way.
+     */
+    resumeExports: true,
   },
   /**
    * Folders scanned for new archives. Each entry is `{ path, categories,

package/src/crash-guard.js

@@ -0,0 +1,94 @@
+/**
+ * Surviving the one crash a torrent client raises from a timer.
+ *
+ * WebTorrent nulls `torrent.client` when a torrent is destroyed, and does not
+ * always tear down that torrent's peer wires with it. A wire that outlives its
+ * torrent then fires — a keep-alive timeout, or a handshake finishing — and
+ * reaches through the dead reference:
+ *
+ *     torrent.js:2092  this.client._debugId    → reading '_debugId' of null
+ *     peer.js:201      this.swarm.client.dht   → reading 'dht' of null
+ *
+ * Both happen inside a `setTimeout` or a socket callback, so there is no
+ * promise to reject and no call of ours to wrap: it is an uncaught exception,
+ * and Node's answer to that is to exit. Under `Restart=always` the service
+ * comes straight back, which is why this reads as a mysterious restart rather
+ * than as a crash — and anything the node was in the middle of, an export
+ * above all, is simply gone.
+ *
+ * So this is deliberately narrow. It survives exactly the shape above and
+ * nothing else: a TypeError, about a null property this list names, raised
+ * from inside the torrent libraries. Every other uncaught exception keeps
+ * Node's behaviour, because a process that survives everything is a process
+ * that lies about its own state.
+ */
+
+/** Packages whose internals may raise this. Ours is not among them. */
+const LIBRARIES = [
+  '/webtorrent/',
+  '/bittorrent-protocol/',
+  '/torrent-discovery/',
+];
+
+/** The properties a wire reaches for on a client that has gone. */
+const REACHING_FOR = ['_debugId', 'dht', 'client', 'swarm', 'torrent', 'wires'];
+
+/**
+ * Whether an uncaught exception is a peer wire touching a destroyed torrent.
+ * @param {unknown} error - What was thrown.
+ * @returns {boolean} - True when it is safe to carry on.
+ */
+export function isDeadTorrentWire(error) {
+  if (!(error instanceof TypeError)) return false;
+
+  const message = String(error.message ?? '');
+  const reading =
+    /Cannot read propert(?:y|ies) of (?:null|undefined) \(reading '([^']+)'\)/.exec(
+      message,
+    );
+  // Older phrasings say it the other way round.
+  const older = /Cannot read property '([^']+)' of (?:null|undefined)/.exec(
+    message,
+  );
+  const property = reading?.[1] ?? older?.[1];
+  if (!property || !REACHING_FOR.includes(property)) return false;
+
+  // The frame that threw has to be theirs. An error of ours that happens to
+  // mention the same property is a bug worth dying on.
+  const stack = String(error.stack ?? '').replaceAll('\\', '/');
+  return LIBRARIES.some((library) => stack.includes(library));
+}
+
+/**
+ * Installs the guard.
+ *
+ * Returns what it registered so a test can take it off again, and so nothing
+ * has to reach into `process` to find out what is installed.
+ * @param {object} [options] - `onSurvived`, and `exit` for testing.
+ * @returns {Function} - The handler, for `process.off`.
+ */
+export function installCrashGuard(options = {}) {
+  const survived =
+    options.onSurvived ??
+    ((error) => {
+      console.warn(
+        `[torrent] a peer wire outlived its torrent and threw: ${error.message}. ` +
+          'Carried on — this is a fault inside webtorrent, not a state this ' +
+          'node cannot continue from.',
+      );
+    });
+  const exit = options.exit ?? ((code) => process.exit(code));
+
+  const handler = (error) => {
+    if (isDeadTorrentWire(error)) {
+      survived(error);
+      return;
+    }
+    // Node's own behaviour for everything else: say what happened, and stop.
+    console.error(error);
+    exit(1);
+  };
+
+  process.on('uncaughtException', handler);
+  return handler;
+}

package/src/index.js

@@ -5,8 +5,9 @@ import { parseArgs } from 'node:util';
 import { createApp } from './api.js';
 import { assertSafeToListen, createAuth } from './auth.js';
 import { Catalog } from './catalog.js';
-import { StackStore } from './stacks.js';
+import { StackStore, resolveStack } from './stacks.js';
 import { StackCache } from './stack-cache.js';
+import { installCrashGuard } from './crash-guard.js';
 import { CutlineStore } from './cutlines.js';
 import { BakeManager } from './bake-jobs.js';
 import { loadCodec } from './codec.js';
@@ -212,6 +213,11 @@ PMTILES_SWARM_PUBLIC_URL
   /** @type {Array<{label: string, stop: () => unknown, ms?: number}>} */
   const stoppers = [];
   installSignalHandlers(stoppers);
+  // Before anything opens a socket. A peer wire that outlives its torrent
+  // throws from a timer, which is an uncaught exception, which is an exit --
+  // and under Restart=always that reads as a mysterious restart rather than a
+  // crash, taking whatever was running with it.
+  installCrashGuard();
 
   // Before anything is created or any port is bound: an unauthenticated node
   // on a reachable address fails silently, working perfectly right up until
@@ -474,6 +480,25 @@ PMTILES_SWARM_PUBLIC_URL
     cutlines,
   });
 
+  // Anything a previous run did not finish. Deliberately after the stacks and
+  // the catalog are loaded, since a checkpoint is only worth picking up where
+  // the recipe still resolves to what it did when the work was done.
+  //
+  // Not awaited: an export is hours, and the node should be answering requests
+  // while it runs rather than after it.
+  if (config.stacks?.resumeExports !== false) {
+    bakes
+      .resumeAll((stackId) => {
+        const stack = stacks?.list().find((one) => one.id === stackId);
+        if (!stack) return null;
+        return resolveStack(stack, {
+          archive: (hash) => catalog.get(hash),
+          category: (name) => catalog.byCategory(name)[0] ?? null,
+        });
+      })
+      .catch((error) => console.warn(`[bake] resume failed: ${error.message}`));
+  }
+
   // Early in the sequence, so an export is told to stop before the pieces it
   // reads through are taken away. Its checkpoint is the hours already spent.
   stoppers.unshift({

package/src/web/index.html

@@ -5228,6 +5228,16 @@ Every piece is hashed against the ` +
                 'restores the old behaviour. They are written in order ' +
                 'whatever it is set to.',
             },
+            {
+              key: 'stacks.resumeExports',
+              label: 'Pick up an unfinished export when the node starts',
+              type: 'boolean',
+              restart: true,
+              help:
+                'A checkpoint is the hours already spent, and the case that ' +
+                'costs most is the one nobody chose - a crash, or a restart ' +
+                'mid-run. Only where the recipe still resolves to what it did.',
+            },
             {
               key: 'stacks.bakePauseMs',
               label: 'Pause between batches while exporting',