pmtiles-swarm 0.71.0 → 0.72.1

package/CHANGELOG.md

@@ -7,6 +7,68 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.72.1
+### 🐞 Bug fixes
+- **A settings save could set something that was not a setting.** `saveConfig` rejected a key it
+  did not recognise by asking `key in DEFAULTS`, and `in` walks the prototype chain: every object
+  inherits `__proto__`, `constructor` and `toString`, so all three passed as known settings. The
+  assignment underneath then did what those names mean rather than what a setting means -
+  `config.__proto__ = {...}` replaces the running config's prototype - and the result was written
+  to the config file. `Object.hasOwn` asks the question that was meant.
+
+  Reachable only by an admin token, which is a token that can change any setting anyway, so this
+  is a gate that was not doing its job rather than a way in. Found by reading through what
+  `security/detect-object-injection` had to say, which is what it is switched on for.
+- **A date placeholder that was nearly right took exponentially long to reject.** `expandTemplate`
+  read a `{...}` group as a date pattern by testing it against `^[YMDymd]+([-_./ ]?[YMDymd]+)*$`.
+  With the separator optional, a run of field letters can be divided between the group and the `+`
+  in front of it in every possible way, so a group that is all field letters and one character that
+  cannot match has to try all of them before saying no: 24 characters took 100ms, and each two
+  after that doubled it.
+
+  Requiring the separator inside the group accepts exactly the same set of patterns - a run can
+  only be matched one way now - and rejects the near miss immediately. The template is config
+  rather than anything a stranger sends, so this was a source of surprise rather than a way in.
+
+## 0.72.0
+### ✨ Features and improvements
+- **An unfinished export is picked up when the node starts.** The checkpoint was always there;
+  finding it again meant somebody remembering to press the button. That is fine for an export
+  stopped on purpose and wrong for one a crash took, which is the case that costs the most and
+  gives the least warning.
+
+  The checkpoint now records what the job was - what the archive is called, what the file is
+  called, where it was going, what it is filed under - because none of that can be worked out from
+  the tiles on disk, and a resumed export has to be the one somebody asked for rather than a new
+  one with today's date on it.
+
+  Only where the recipe still resolves to what it did. `bakeRevision` covers what each source
+  became, so a rebuilt source means the checkpoint holds half of a map that no longer exists;
+  that is left alone and reported, to be discarded when somebody exports again deliberately.
+  `stacks.resumeExports` turns it off for a node where hours of merging should never begin
+  without being asked for.
+
+### 🐞 Bug fixes
+- **The node was exiting when a peer wire outlived its torrent.** Twice in one evening on a real
+  node, both times killing an export that was hours in:
+
+      torrent.js:2092  this.client._debugId    -> reading '_debugId' of null
+      peer.js:201      this.swarm.client.dht   -> reading 'dht' of null
+
+  WebTorrent nulls `torrent.client` when a torrent is destroyed and does not always tear down that
+  torrent's peer wires with it. One fired a keep-alive timeout afterwards and the other finished a
+  handshake, and both reached through the dead reference. They happen inside a timer or a socket
+  callback, so there is no promise to reject and no call of ours to wrap - it is an uncaught
+  exception, and Node's answer to that is to exit. Under `Restart=always` that reads as a
+  mysterious restart rather than as a crash.
+
+  `src/crash-guard.js` survives exactly that shape and nothing else: a TypeError, about one of a
+  named list of properties, raised from a frame inside the torrent libraries. Every other uncaught
+  exception still stops the process, because one that survives everything is one that lies about
+  its own state. The two stacks from the journal are in the tests verbatim, alongside the near
+  misses that must still be fatal - the same error from our own code, a different error from
+  theirs.
+
 ## 0.71.0
 ### ✨ Features and improvements
 - **The preview comes back to where it was opened from.** The link out of a preview said

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.71.0",
+  "version": "0.72.1",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -84,10 +84,11 @@ function route(handler) {
  * Progress has its own route already: `runningAdds()` feeds `/api/adds`, the
  * console polls it, and `DELETE /api/adds` cancels the ones that can be.
  * @param {object} res - The response to answer.
- * @param {Function} start - Called with `{onValidated}`; returns the add's promise.
- * @param {object} accepted - Fields describing the source, for the 202 body.
- * @param {string} message - What the 202 tells the caller is now happening.
- * @param {string} what - Prefixed log tag and source, for a failure nobody is waiting on.
+ * @param {object} options - The add itself, and what to say about it.
+ * @param {Function} options.start - Called with `{onValidated}`; returns the add's promise.
+ * @param {object} options.accepted - Fields describing the source, for the 202 body.
+ * @param {string} options.message - What the 202 tells the caller is now happening.
+ * @param {string} options.what - Prefixed log tag and source, for a failure nobody is waiting on.
  * @returns {Promise<void>} - Resolves once the response has been sent.
  */
 async function acceptAdd(res, { start, accepted, message, what }) {
@@ -154,7 +155,6 @@ async function acceptAdd(res, { start, accepted, message, what }) {
  * worse, because it makes the shape of the fragment depend on what happened to
  * be available, so every reader has to handle both anyway. One shape,
  * `URLSearchParams` reads it, and a magnet survives the round trip exactly.
- *
  * @param {string} url - The URL a style points at.
  * @param {object} handles - `{torrent, magnet}`, either of which may be absent.
  * @returns {string} - The URL, with a fragment if there is anything to put in one.
@@ -205,6 +205,11 @@ function sourceUrlFor(category, newest, base) {
   return withSwarmHandles(url, { torrent, magnet });
 }
 
+/**
+ * Builds the HTTP surface over everything the node has already started.
+ * @param {object} parts - The node's live pieces.
+ * @returns {object} - The Express app.
+ */
 export function createApp({
   library,
   catalog,

package/src/auth.js

@@ -338,7 +338,10 @@ export function createAuth(config) {
     /** Whether any credential is configured. */
     enabled,
 
-    /** Whether a password login is possible, as opposed to only a token. */
+    /**
+     * Whether a password login is possible, as opposed to only a token.
+     * @returns {boolean} - True when a password or a hash is set.
+     */
     get passwordLoginEnabled() {
       return Boolean(settings().password || settings().passwordHash);
     },

package/src/bake-jobs.js

@@ -1,3 +1,4 @@
+import fs from 'node:fs/promises';
 import path from 'node:path';
 import {
   assertBakeable,
@@ -57,6 +58,7 @@ export function workDirFor(job, config = {}) {
 /** A bake that has finished is kept this long, so the console can report it. */
 const KEEP_FINISHED_MS = 10 * 60 * 1000;
 
+/** The exports a node is running, and the ones it has just finished. */
 export class BakeManager {
   #library;
   #tiles;
@@ -108,6 +110,77 @@ export class BakeManager {
     return true;
   }
 
+  /**
+   * Picks up exports a previous run did not finish.
+   *
+   * A checkpoint is the hours already spent, and finding one again used to
+   * mean somebody remembering to press the button. That is fine for an export
+   * stopped on purpose and wrong for one a crash took -- which is the case
+   * that costs the most and gives the least warning.
+   *
+   * Only where the recipe still resolves to what it did. `bakeRevision` covers
+   * what each source became, so a rebuilt source means the checkpoint holds
+   * half of a map that no longer exists; that is left alone rather than
+   * continued, and discarded when somebody exports again deliberately.
+   * @param {Function} resolve - `(stackId) => resolved stack | null`.
+   * @returns {Promise<object[]>} - The jobs started.
+   */
+  async resumeAll(resolve) {
+    const started = [];
+
+    for (const root of this.#workRoots()) {
+      const directory = path.join(root, WORK_DIR);
+      const found = await fs.readdir(directory).catch(() => []);
+
+      for (const stackId of found) {
+        if (this.#jobs.has(stackId)) continue;
+        const state = await fs
+          .readFile(path.join(directory, stackId, 'bake-state.json'), 'utf8')
+          .then((raw) => JSON.parse(raw))
+          .catch(() => null);
+        // Written by a version that did not record what the job was. There is
+        // nothing to reproduce it from, so it waits for a person.
+        if (!state?.describe) continue;
+
+        const resolved = resolve(stackId);
+        if (!resolved || bakeRevision(resolved) !== state.revision) continue;
+
+        try {
+          const job = await this.start({ resolved, ...state.describe });
+          started.push(job);
+          console.log(
+            `[bake] picking up ${stackId} where it stopped: ` +
+              `${state.written ?? 0} tiles already merged`,
+          );
+        } catch (error) {
+          // A stack that cannot be baked now -- no codec, sources gone -- is
+          // said out loud and left. Its checkpoint is still there.
+          console.warn(`[bake] could not resume ${stackId}: ${error.message}`);
+        }
+      }
+    }
+
+    return started;
+  }
+
+  /**
+   * Everywhere an export might have left work.
+   *
+   * The working directory follows the destination, and a destination is
+   * whatever was chosen at the time -- so this is every place one could have
+   * been: the named locations, the default save path, and the data directory,
+   * where exports worked before the working directory moved.
+   * @returns {string[]} - Roots to look under, without repeats.
+   */
+  #workRoots() {
+    const roots = [
+      ...(this.#config.locations ?? []).map((one) => one?.path),
+      this.#config.savePath,
+      this.#config.dataDir,
+    ].filter(Boolean);
+    return [...new Set(roots.map((one) => path.resolve(one)))];
+  }
+
   /**
    * Stops every running bake and waits for each to write its checkpoint.
    *
@@ -285,6 +358,7 @@ export class BakeManager {
    * @param {string} workDir - Where the unfinished work lives.
    * @param {string} destination - Where the archive goes.
    * @param {string} format - The output format.
+   * @param {object} options - `signal` to stop early, and how often to checkpoint.
    * @returns {Promise<void>} - Resolves when the file is written.
    */
   async #merge(
@@ -341,6 +415,17 @@ export class BakeManager {
       header: { format },
       pauseMs: this.#config.stacks?.bakePauseMs ?? 0,
       concurrency: this.#concurrency(),
+      // Written into the checkpoint in the shape `start` takes, so picking one
+      // up is handing it back rather than reconstructing it. Nothing here can
+      // be worked out from the tiles on disk: what the archive is called, what
+      // the file is called, where it was going, what it is filed under.
+      describe: {
+        name: job.archiveName,
+        filename: job.name,
+        publishDir: job.publishDir ?? null,
+        description: options.description ?? null,
+        categories: options.categories ?? null,
+      },
       metadata: {
         name: job.archiveName,
         // Only what was asked for. Falling back to the recipe's own

package/src/bake.js

@@ -468,6 +468,11 @@ export async function bakeStack(options) {
     checkpointSeconds = DEFAULT_CHECKPOINT_SECONDS,
     pauseMs = 0,
     concurrency = DEFAULT_CONCURRENCY,
+    // Written into the checkpoint and handed back by `readCheckpoint`. A
+    // resumed export has to reproduce the one somebody asked for -- its name,
+    // where it was going, what it was filed under -- and none of that can be
+    // worked out from the tiles on disk.
+    describe,
   } = options;
   const batchSize = Math.max(1, Math.floor(concurrency));
 
@@ -516,6 +521,7 @@ export async function bakeStack(options) {
         dataBytes: writer.dataBytes,
         addressed: writer.addressedTiles,
         clustered: writer.clustered,
+        ...(describe ? { describe } : {}),
       },
       writer.entries,
       persisted,

package/src/catalog.js

@@ -8,7 +8,6 @@ import path from 'node:path';
  * dozens, the whole thing is human-readable and hand-editable, and it avoids a
  * native dependency in a project that already asks a lot of the install.
  * Writes go through a temp file and a rename, so a crash cannot truncate it.
- *
  * @typedef {object} CatalogEntry
  * @property {string} infoHash - Hex v1 infohash. The catalog's primary key.
  * @property {string} name - Archive filename.
@@ -99,6 +98,7 @@ export function newerFirst(a, b) {
   return String(b.createdAt ?? '').localeCompare(String(a.createdAt ?? ''));
 }
 
+/** What the node knows about every archive it holds. */
 export class Catalog {
   #file;
   #entries = new Map();

package/src/config.js

@@ -388,6 +388,19 @@ const DEFAULTS = {
      * not clustered is bad at the one thing PMTiles is for.
      */
     bakeConcurrency: 4,
+    /**
+     * Whether an unfinished export is picked up when the node starts.
+     *
+     * A checkpoint is the hours already spent, and the case that costs most is
+     * the one nobody chose: a crash, or a restart in the middle of a run. On
+     * by default for that reason. Off for a node where hours of merging should
+     * never begin without somebody asking for it.
+     *
+     * Only where the recipe still resolves to what it did. A rebuilt source
+     * means the checkpoint holds half of a map that no longer exists, and that
+     * is left alone either way.
+     */
+    resumeExports: true,
   },
   /**
    * Folders scanned for new archives. Each entry is `{ path, categories,
@@ -525,16 +538,13 @@ function merge(base, override) {
   // `libtorrent`, which would otherwise alter the defaults themselves.
   const out = {};
   for (const [key, value] of Object.entries(base ?? {})) {
-    // eslint-disable-next-line security/detect-object-injection -- keys come from DEFAULTS
     out[key] = clone(value);
   }
   for (const [key, value] of Object.entries(override ?? {})) {
     if (value === undefined) continue;
-    // eslint-disable-next-line security/detect-object-injection -- keys come from a config file the operator controls
     out[key] =
       value && typeof value === 'object' && !Array.isArray(value)
-        ? // eslint-disable-next-line security/detect-object-injection -- as above
-          merge(base[key] ?? {}, value)
+        ? merge(base[key] ?? {}, value)
         : value;
   }
   return out;
@@ -913,7 +923,10 @@ export async function saveConfig(config, updates, configPath) {
   );
 
   for (const [key] of changing) {
-    if (!(key in DEFAULTS)) {
+    // `in` would walk the prototype chain, which says yes to `__proto__` and
+    // to `constructor` — and the assignment below then swaps the config's
+    // prototype rather than setting a setting.
+    if (!Object.hasOwn(DEFAULTS, key)) {
       throw new Error(`unknown setting: ${key}`);
     }
     if (guarded.has(key)) {

package/src/crash-guard.js

@@ -0,0 +1,94 @@
+/**
+ * Surviving the one crash a torrent client raises from a timer.
+ *
+ * WebTorrent nulls `torrent.client` when a torrent is destroyed, and does not
+ * always tear down that torrent's peer wires with it. A wire that outlives its
+ * torrent then fires — a keep-alive timeout, or a handshake finishing — and
+ * reaches through the dead reference:
+ *
+ *     torrent.js:2092  this.client._debugId    → reading '_debugId' of null
+ *     peer.js:201      this.swarm.client.dht   → reading 'dht' of null
+ *
+ * Both happen inside a `setTimeout` or a socket callback, so there is no
+ * promise to reject and no call of ours to wrap: it is an uncaught exception,
+ * and Node's answer to that is to exit. Under `Restart=always` the service
+ * comes straight back, which is why this reads as a mysterious restart rather
+ * than as a crash — and anything the node was in the middle of, an export
+ * above all, is simply gone.
+ *
+ * So this is deliberately narrow. It survives exactly the shape above and
+ * nothing else: a TypeError, about a null property this list names, raised
+ * from inside the torrent libraries. Every other uncaught exception keeps
+ * Node's behaviour, because a process that survives everything is a process
+ * that lies about its own state.
+ */
+
+/** Packages whose internals may raise this. Ours is not among them. */
+const LIBRARIES = [
+  '/webtorrent/',
+  '/bittorrent-protocol/',
+  '/torrent-discovery/',
+];
+
+/** The properties a wire reaches for on a client that has gone. */
+const REACHING_FOR = ['_debugId', 'dht', 'client', 'swarm', 'torrent', 'wires'];
+
+/**
+ * Whether an uncaught exception is a peer wire touching a destroyed torrent.
+ * @param {unknown} error - What was thrown.
+ * @returns {boolean} - True when it is safe to carry on.
+ */
+export function isDeadTorrentWire(error) {
+  if (!(error instanceof TypeError)) return false;
+
+  const message = String(error.message ?? '');
+  const reading =
+    /Cannot read propert(?:y|ies) of (?:null|undefined) \(reading '([^']+)'\)/.exec(
+      message,
+    );
+  // Older phrasings say it the other way round.
+  const older = /Cannot read property '([^']+)' of (?:null|undefined)/.exec(
+    message,
+  );
+  const property = reading?.[1] ?? older?.[1];
+  if (!property || !REACHING_FOR.includes(property)) return false;
+
+  // The frame that threw has to be theirs. An error of ours that happens to
+  // mention the same property is a bug worth dying on.
+  const stack = String(error.stack ?? '').replaceAll('\\', '/');
+  return LIBRARIES.some((library) => stack.includes(library));
+}
+
+/**
+ * Installs the guard.
+ *
+ * Returns what it registered so a test can take it off again, and so nothing
+ * has to reach into `process` to find out what is installed.
+ * @param {object} [options] - `onSurvived`, and `exit` for testing.
+ * @returns {Function} - The handler, for `process.off`.
+ */
+export function installCrashGuard(options = {}) {
+  const survived =
+    options.onSurvived ??
+    ((error) => {
+      console.warn(
+        `[torrent] a peer wire outlived its torrent and threw: ${error.message}. ` +
+          'Carried on — this is a fault inside webtorrent, not a state this ' +
+          'node cannot continue from.',
+      );
+    });
+  const exit = options.exit ?? ((code) => process.exit(code));
+
+  const handler = (error) => {
+    if (isDeadTorrentWire(error)) {
+      survived(error);
+      return;
+    }
+    // Node's own behaviour for everything else: say what happened, and stop.
+    console.error(error);
+    exit(1);
+  };
+
+  process.on('uncaughtException', handler);
+  return handler;
+}

package/src/cutlines.js

@@ -14,6 +14,7 @@ import { fromBounds, fromGeoJSON } from './cutline.js';
 /** Where they live under the data directory. */
 const DIRECTORY = 'cutlines';
 
+/** Loads the cutline files once and hands out prepared shapes. */
 export class CutlineStore {
   #dir;
   #shapes = new Map();
@@ -26,7 +27,10 @@ export class CutlineStore {
     this.#dir = path.join(dataDir ?? './data', DIRECTORY);
   }
 
-  /** Where cutlines are read from. @returns {string} - The directory. */
+  /**
+   * Where cutlines are read from.
+   * @returns {string} - The directory.
+   */
   get directory() {
     return this.#dir;
   }
@@ -56,7 +60,10 @@ export class CutlineStore {
     }
   }
 
-  /** Every cutline that loaded. @returns {string[]} - Their names. */
+  /**
+   * Every cutline that loaded.
+   * @returns {string[]} - Their names.
+   */
   list() {
     return [...this.#shapes.keys()].sort();
   }

package/src/engines/composite.js

@@ -53,12 +53,18 @@ export class CompositeEngine {
     this.marksIncomplete = primary.marksIncomplete ?? false;
   }
 
-  /** The engine that owns the data. @returns {object} - The primary. */
+  /**
+   * The engine that owns the data.
+   * @returns {object} - The primary.
+   */
   get primary() {
     return this.#primary;
   }
 
-  /** The engines that only seed. @returns {object[]} - The secondaries. */
+  /**
+   * The engines that only seed.
+   * @returns {object[]} - The secondaries.
+   */
   get secondaries() {
     return this.#secondaries;
   }
@@ -342,6 +348,10 @@ export class CompositeEngine {
     return { ...(engines[0] ?? { state: 'unknown' }), engines };
   }
 
+  /**
+   * Every torrent, gathered from every engine.
+   * @returns {Promise<object[]>} - Normalised torrents.
+   */
   async list() {
     if (this.#stopping) return [];
     const primary = await this.#primary.list();
@@ -490,6 +500,7 @@ export class CompositeEngine {
    * present at all, since it is the only one that can produce a hybrid v1+v2
    * torrent — and a hybrid serves v1 and v2 clients alike, so having it seed
    * rather than lead is no reason to make a lesser torrent.
+   * @param {string} filePath - What to build it from.
    * @param {object} options - Creation options.
    * @returns {Promise<object>} - The created torrent.
    */

package/src/engines/libtorrent.js

@@ -477,6 +477,10 @@ export class LibtorrentEngine {
     }
   }
 
+  /**
+   * Every torrent the sidecar holds.
+   * @returns {Promise<object[]>} - Normalised torrents.
+   */
   async list() {
     // A node that is shutting down still has a console polling it and a sweep
     // or two in flight. Answering "the sidecar exited" to each of them fills
@@ -759,6 +763,7 @@ export class LibtorrentEngine {
   /**
    * Persists resume data, so the next start skips re-hashing the store.
    * @param {string} [infoHash] - One torrent, or all when omitted.
+   * @param {object} [options] - `timeoutMs` to override the default deadline.
    * @returns {Promise<{written: number, asked: number}>} - How many torrents
    *   were told to write resume data, and how many actually did before the
    *   deadline.
@@ -807,7 +812,7 @@ export class LibtorrentEngine {
    * @param {string} op - Operation name.
    * @param {object} params - Operation parameters.
    * @param {number} [timeoutMs] - How long to wait. Default 60s.
-   * @returns {Promise<any>} - The result.
+   * @returns {Promise<unknown>} - Whatever the sidecar answered.
    */
   async #call(op, params, timeoutMs = 60000) {
     if (op !== 'shutdown') await this.connect();

package/src/engines/webtorrent.js

@@ -357,6 +357,10 @@ export class WebTorrentSeedEngine {
     };
   }
 
+  /**
+   * Every torrent the client holds.
+   * @returns {Promise<object[]>} - Normalised torrents.
+   */
   async list() {
     if (!this.#client) return [];
     return this.#client.torrents.map((torrent) => this.#normalise(torrent));

package/src/feed.js

@@ -212,7 +212,6 @@ export function formatBytes(bytes) {
  * has none to give: planetiler writes that section after every tile, so on a
  * planet archive it is the very end of the file. An entry summarised from a
  * feed therefore stays due for warming, which is what fills the rest in.
- *
  * @param {string} block - The item XML.
  * @returns {object | undefined} - The summary, or undefined if absent.
  */
@@ -344,6 +343,7 @@ function isoDate(value) {
  * @returns {string | undefined} - Decoded text, if found.
  */
 function tag(block, name) {
+  // eslint-disable-next-line security/detect-non-literal-regexp -- name is one of this module's own element names
   const pattern = new RegExp(`<${name}\\b[^>]*>([\\s\\S]*?)</${name}>`, 'i');
   const match = pattern.exec(block);
   if (!match) return undefined;

package/src/hooks.js

@@ -33,7 +33,6 @@ const TAIL_LINES = 20;
  *   %N name          %I infohash      %F content path
  *   %L category      %G categories    %D save path
  *   %Z size          %C file count
- *
  * @param {string} argument - An argument possibly containing placeholders.
  * @param {object} entry - The catalog entry that finished.
  * @returns {string} - The argument with placeholders replaced.
@@ -96,7 +95,10 @@ export class ProgramHooks {
     this.#config = config;
   }
 
-  /** Whether either command is configured. @returns {boolean} - True when armed. */
+  /**
+   * Whether either command is configured.
+   * @returns {boolean} - True when armed.
+   */
   get enabled() {
     return Boolean(
       this.#config.onComplete?.command || this.#config.onAdded?.command,
@@ -129,7 +131,10 @@ export class ProgramHooks {
     }
   }
 
-  /** Stops watching. @returns {void} */
+  /**
+   * Stops watching.
+   * @returns {void}
+   */
   stop() {
     if (this.#timer) clearInterval(this.#timer);
     this.#timer = undefined;

package/src/incomplete.js

@@ -147,7 +147,10 @@ export class CompletionWatcher {
     this.#config = config;
   }
 
-  /** Whether marking is switched on at all. @returns {boolean} - True when armed. */
+  /**
+   * Whether marking is switched on at all.
+   * @returns {boolean} - True when armed.
+   */
   get enabled() {
     return suffixFor(this.#config) !== '';
   }
@@ -168,7 +171,10 @@ export class CompletionWatcher {
     this.#timer.unref?.();
   }
 
-  /** Stops watching. @returns {void} */
+  /**
+   * Stops watching.
+   * @returns {void}
+   */
   stop() {
     if (this.#timer) clearInterval(this.#timer);
     this.#timer = undefined;

package/src/index.js

@@ -5,8 +5,9 @@ import { parseArgs } from 'node:util';
 import { createApp } from './api.js';
 import { assertSafeToListen, createAuth } from './auth.js';
 import { Catalog } from './catalog.js';
-import { StackStore } from './stacks.js';
+import { StackStore, resolveStack } from './stacks.js';
 import { StackCache } from './stack-cache.js';
+import { installCrashGuard } from './crash-guard.js';
 import { CutlineStore } from './cutlines.js';
 import { BakeManager } from './bake-jobs.js';
 import { loadCodec } from './codec.js';
@@ -212,6 +213,11 @@ PMTILES_SWARM_PUBLIC_URL
   /** @type {Array<{label: string, stop: () => unknown, ms?: number}>} */
   const stoppers = [];
   installSignalHandlers(stoppers);
+  // Before anything opens a socket. A peer wire that outlives its torrent
+  // throws from a timer, which is an uncaught exception, which is an exit --
+  // and under Restart=always that reads as a mysterious restart rather than a
+  // crash, taking whatever was running with it.
+  installCrashGuard();
 
   // Before anything is created or any port is bound: an unauthenticated node
   // on a reachable address fails silently, working perfectly right up until
@@ -474,6 +480,25 @@ PMTILES_SWARM_PUBLIC_URL
     cutlines,
   });
 
+  // Anything a previous run did not finish. Deliberately after the stacks and
+  // the catalog are loaded, since a checkpoint is only worth picking up where
+  // the recipe still resolves to what it did when the work was done.
+  //
+  // Not awaited: an export is hours, and the node should be answering requests
+  // while it runs rather than after it.
+  if (config.stacks?.resumeExports !== false) {
+    bakes
+      .resumeAll((stackId) => {
+        const stack = stacks?.list().find((one) => one.id === stackId);
+        if (!stack) return null;
+        return resolveStack(stack, {
+          archive: (hash) => catalog.get(hash),
+          category: (name) => catalog.byCategory(name)[0] ?? null,
+        });
+      })
+      .catch((error) => console.warn(`[bake] resume failed: ${error.message}`));
+  }
+
   // Early in the sequence, so an export is told to stop before the pieces it
   // reads through are taken away. Its checkpoint is the hours already spent.
   stoppers.unshift({

package/src/library.js

@@ -102,6 +102,7 @@ async function newestMtime(dir) {
   return newest;
 }
 
+/** The archives this node holds, and the engine seeding them. */
 export class Library {
   #catalog;
   #engine;
@@ -1587,6 +1588,7 @@ export class Library {
    * @param {string} magnet - Its magnet URI.
    * @param {object} engine - The engine it came from.
    * @param {object} options - Adopt options.
+   * @param {boolean} [readable] - Whether the file can be opened from here.
    * @returns {Promise<object>} - The catalog entry.
    */
   async #adoptInPlace(torrent, magnet, engine, options, readable = true) {
@@ -2285,6 +2287,7 @@ export class Library {
    * above has something to read while this is still running.
    * @param {object[]} entries - Catalog entries, newest first.
    * @param {{restored: number, failed: number}} tally - Mutated as it goes.
+   * @param {object} handed - What the caller was handed, for the seeding check.
    * @returns {Promise<{restored: number, failed: number}>} - That tally.
    */
   async #restoreEach(entries, tally, handed) {

package/src/mutable.js

@@ -105,6 +105,12 @@ export function trackersFromMagnet(magnet) {
   return new URLSearchParams(magnet.slice('magnet:?'.length)).getAll('tr');
 }
 
+/**
+ * A magnet URI for a mutable series.
+ * @param {string|Uint8Array} publicKey - The series key, hex or bytes.
+ * @param {object} [options] - `infoHash` for the current build, `trackers` to announce.
+ * @returns {string} - The magnet URI.
+ */
 export function mutableMagnet(publicKey, options = {}) {
   // Buffer.from(string) would read hex as UTF-8 and produce a 64-byte key, so
   // the two forms have to be told apart rather than coerced.

package/src/origin.js

@@ -190,7 +190,6 @@ async function contentLooksDifferent(entry) {
 
     const fields = ['tileCount', 'minZoom', 'maxZoom', 'format', 'specVersion'];
     for (const field of fields) {
-      // eslint-disable-next-line security/detect-object-injection -- field comes from the constant list above
       if (stored[field] !== undefined && stored[field] !== fresh[field]) {
         return true;
       }

package/src/pixels.js

@@ -17,6 +17,7 @@ const here = path.dirname(fileURLToPath(import.meta.url));
 /** How long to wait for a worker to answer before giving up on it. */
 const REPLY_TIMEOUT_MS = 120000;
 
+/** A pool of threads for the pixel work, so a merge never blocks the server. */
 export class PixelWorker {
   #workers = [];
   #pending = new Map();
@@ -75,7 +76,10 @@ export class PixelWorker {
     }
   }
 
-  /** How many threads this pool holds. @returns {number} - The count. */
+  /**
+   * How many threads this pool holds.
+   * @returns {number} - The count.
+   */
   get size() {
     return this.#workers.length;
   }

package/src/pmtiles-probe.js

@@ -108,6 +108,12 @@ export function customEncodingFactors(metadata) {
  */
 export const SUMMARY_VERSION = 3;
 
+/**
+ * What the catalog keeps about an archive, from its header and metadata.
+ * @param {object} header - A parsed PMTiles v3 header.
+ * @param {object} [metadata] - The archive's own metadata document.
+ * @returns {object} - The summary, at `SUMMARY_VERSION`.
+ */
 export function summarize(header, metadata = {}) {
   const type = TILE_TYPES[header.tileType] ?? TILE_TYPES[0];
 

package/src/pmtiles-scan.js

@@ -125,7 +125,7 @@ async function openArchive(archive) {
  * per-tile yield across a planet is the slowest part of the loop.
  * @param {string|object} archive - Path to the `.pmtiles`, or a byte source.
  * @param {object} [options] - `signal` to stop early.
- * @returns {AsyncGenerator<number[]>} - Chunks of ascending tile ids.
+ * @yields {number[]} - Chunks of ascending tile ids.
  */
 export async function* scanTileIds(archive, options = {}) {
   const { source, header, root, owned } = await openArchive(archive);
@@ -190,7 +190,7 @@ export async function* scanTileIds(archive, options = {}) {
  * holding every tile id of every source in memory at once.
  * @param {Array<string|object>} archives - Paths, or byte sources.
  * @param {object} [options] - `signal` to stop early.
- * @returns {AsyncGenerator<number>} - Ascending tile ids, each once.
+ * @yields {number} - Ascending tile ids, each once.
  */
 export async function* unionOfTileIds(archives, options = {}) {
   const readers = archives.map((archive) => ({

package/src/pmtiles-write.js

@@ -274,12 +274,18 @@ export class PMTilesWriter {
     return new PMTilesWriter({ ...options, tempPath, handle });
   }
 
-  /** How many tiles have been offered, including duplicates. @returns {number} - The count. */
+  /**
+   * How many tiles have been offered, including duplicates.
+   * @returns {number} - The count.
+   */
   get addressedTiles() {
     return this.#addressed;
   }
 
-  /** How many entries the directories will hold. @returns {number} - The count. */
+  /**
+   * How many entries the directories will hold.
+   * @returns {number} - The count.
+   */
   get tileEntries() {
     return this.#entries.length;
   }
@@ -296,12 +302,18 @@ export class PMTilesWriter {
     return this.#entries;
   }
 
-  /** Tile bytes buffered so far. @returns {number} - The byte count. */
+  /**
+   * Tile bytes buffered so far.
+   * @returns {number} - The byte count.
+   */
   get dataBytes() {
     return this.#offset;
   }
 
-  /** Whether the tiles have arrived in order. @returns {boolean} - True while they have. */
+  /**
+   * Whether the tiles have arrived in order.
+   * @returns {boolean} - True while they have.
+   */
   get clustered() {
     return this.#clustered;
   }

package/src/prewarm.js

@@ -71,6 +71,7 @@ function tooEarly(error) {
   return /metadata has not arrived|not held here/i.test(error?.message ?? '');
 }
 
+/** Pulls each archive's header and root directory before anything asks for a tile. */
 export class HeadWarmer {
   #tiles;
   #catalog;
@@ -411,7 +412,10 @@ export class HeadWarmer {
     this.#timer.unref?.();
   }
 
-  /** Stops warming. @returns {void} */
+  /**
+   * Stops warming.
+   * @returns {void}
+   */
   stop() {
     if (this.#timer) clearInterval(this.#timer);
     this.#timer = undefined;

package/src/publisher.js

@@ -68,6 +68,7 @@ const REROLL_AFTER = 2;
  */
 const MINIMUM_NODES = 8;
 
+/** Publishes the catalog under a BEP 46 key, and keeps it published. */
 export class MutablePublisher {
   #catalog;
   #dht;
@@ -250,6 +251,7 @@ export class MutablePublisher {
   /**
    * Waits for the DHT to bootstrap, then publishes.
    * @param {number} readyMs - How long to wait before going ahead regardless.
+   * @param {number} retryMs - How long to wait before trying again.
    * @returns {Promise<void>} - Resolves once the first attempt is done.
    */
   async #firstPublish(readyMs, retryMs) {
@@ -375,7 +377,7 @@ export class MutablePublisher {
    * holding the original would save a table belonging to a socket that has
    * been closed.
    * @param {Function} save - Receives the live DHT.
-   * @returns {Promise<*>} - Whatever `save` returns, or 0 with no socket.
+   * @returns {Promise<unknown>} - Whatever `save` returns, or 0 with no socket.
    */
   async saveTable(save) {
     if (!this.#dht) return 0;

package/src/rate-limits.js

@@ -204,7 +204,10 @@ export class SpeedLimits {
     this.#timer.unref?.();
   }
 
-  /** Stops watching. @returns {void} */
+  /**
+   * Stops watching.
+   * @returns {void}
+   */
   stop() {
     clearInterval(this.#timer);
     this.#timer = undefined;

package/src/savepath.js

@@ -11,6 +11,7 @@
 const MAX_SEGMENT = 120;
 
 /** Reserved on Windows whatever the extension. */
+// eslint-disable-next-line security/detect-unsafe-regex -- the alternatives share no prefix, so there is nothing to backtrack over
 const RESERVED = /^(con|prn|aux|nul|com[1-9]|lpt[1-9])(\..*)?$/i;
 
 /** Separators, control characters, and the ones Windows refuses. */

package/src/seeding.js

@@ -198,7 +198,10 @@ export class SeedingLimits {
     this.#timer.unref?.();
   }
 
-  /** Stops the sweep. @returns {void} */
+  /**
+   * Stops the sweep.
+   * @returns {void}
+   */
   stop() {
     if (this.#timer) clearInterval(this.#timer);
     this.#timer = undefined;

package/src/sources.js

@@ -32,8 +32,13 @@ export function expandTemplate(template, date) {
 
   return String(template).replace(/\{([^{}]*)\}/g, (whole, body) => {
     // Separators an upstream might put between the fields. Anything else means
-    // this is not a date at all.
-    if (!/^[YMDymd]+([-_./ ]?[YMDymd]+)*$/.test(body)) return whole;
+    // this is not a date at all. The separator is required inside the group
+    // rather than optional: optional, a run of field letters can be divided
+    // between the group and the `+` in front of it in exponentially many ways,
+    // and something that is nearly a date but does not match takes that long
+    // to rule out.
+    // eslint-disable-next-line security/detect-unsafe-regex -- a run of field letters can only be divided one way now the separator is required
+    if (!/^[YMDymd]+(?:[-_./ ][YMDymd]+)*$/.test(body)) return whole;
 
     return body.replace(/([YMDymd])\1*|[-_./ ]/g, (run) => {
       const field = run[0].toLowerCase();
@@ -220,15 +225,16 @@ export class ScheduledSourceManager {
   #running = false;
   #lastRun = new Map();
 
+  /** Reads the clock. Injectable so a long import can be simulated. */
+  #now;
+
   /**
    * Creates the manager.
    * @param {import('./library.js').Library} library - Where imports go.
    * @param {import('./catalog.js').Catalog} catalog - Used to skip what we already have.
    * @param {object} config - Resolved configuration.
+   * @param {object} [options] - `now` reads the clock.
    */
-  /** Reads the clock. Injectable so a long import can be simulated. */
-  #now;
-
   constructor(library, catalog, config, { now = () => new Date() } = {}) {
     this.#library = library;
     this.#catalog = catalog;
@@ -558,6 +564,7 @@ export class ScheduledSourceManager {
     let pattern = ARCHIVE_PATTERN;
     if (source.match) {
       try {
+        // eslint-disable-next-line security/detect-non-literal-regexp -- the pattern is the operator's own source config
         pattern = new RegExp(source.match, 'i');
       } catch (error) {
         throw new Error(
@@ -629,7 +636,6 @@ export class ScheduledSourceManager {
    *
    * The dated file stays the real one either way, so it remains seedable under
    * its own torrent while consumers reference a fixed path.
-   *
    * @param {object} source - The source definition.
    * @param {object} entry - The freshly imported entry.
    * @returns {Promise<void>} - Resolves once linked, or logs and continues.

package/src/stack-cache.js

@@ -28,6 +28,7 @@ import path from 'node:path';
  */
 const SHARD = 2;
 
+/** Merged stack tiles kept on disk, bounded by total size. */
 export class StackCache {
   #dir;
   #maxBytes;
@@ -177,7 +178,7 @@ export class StackCache {
    * would issue its own reads to every source underneath.
    * @param {string} key - What is being produced.
    * @param {Function} work - Produces the tile.
-   * @returns {Promise<any>} - What `work` returned.
+   * @returns {Promise<unknown>} - What `work` returned.
    */
   async once(key, work) {
     const running = this.#inFlight.get(key);

package/src/stacks.js

@@ -21,7 +21,6 @@ import { BLEND_MODES, isBlendMode } from './rgba.js';
  * catalog entries, and serving a tile by handing back the bytes of whichever
  * source has it. No pixel is decoded, which is what lets this ship before the
  * codec question is settled.
- *
  * @typedef {object} StackSource
  * @property {string} [category] - Resolve to the newest build in this category.
  * @property {string} [archive] - Or pin one infohash. Exactly one of the two.
@@ -39,7 +38,6 @@ import { BLEND_MODES, isBlendMode } from './rgba.js';
  * @property {number} [heightAdjustment] - Metres, added after masking.
  * @property {number} [opacity] - 0-1, scales source alpha. RGBA only.
  * @property {string} [blend] - Blend operator. RGBA only.
- *
  * @typedef {object} Stack
  * @property {string} id - URL segment.
  * @property {string} [title] - Shown in the console and in TileJSON.
@@ -304,7 +302,6 @@ export function resolveStack(stack, resolvers) {
  *                      stopped.
  *   category           An older build in a category with others. Removing it
  *                      changes nothing the stack can see.
- *
  * @param {Stack[]} stacks - Every stack.
  * @param {object} entry - The catalog entry about to go.
  * @param {Function} categoryInfo - Name to `{ count, newest }` infohash.

package/src/subscriptions.js

@@ -141,6 +141,7 @@ export class SubscriptionManager {
       // appetites, e.g. only taking Europe extracts.
       if (
         subscription.filter &&
+        // eslint-disable-next-line security/detect-non-literal-regexp -- the filter is this node's own subscription config
         !new RegExp(subscription.filter, 'i').test(item.title)
       ) {
         continue;
@@ -233,6 +234,7 @@ export class SubscriptionManager {
     const archives = (document.archives ?? []).filter(
       (archive) =>
         !subscription.filter ||
+        // eslint-disable-next-line security/detect-non-literal-regexp -- as above
         new RegExp(subscription.filter, 'i').test(archive.name ?? ''),
     );
 

package/src/tilejson.js

@@ -9,7 +9,6 @@ import { TileStore } from './tiles.js';
  * just a saving: a node in cache mode holds almost none of the archive, so
  * reading the header to answer a TileJSON request would mean pulling pieces out
  * of the swarm before a map has asked for a single tile.
- *
  * @see https://github.com/mapbox/tilejson-spec/tree/master/3.0.0
  */
 

package/src/torrent-create.js

@@ -247,7 +247,6 @@ export async function createTorrentFromUrl(url, options = {}) {
  * Used to decide whether a file already under its final name is the archive
  * being asked for. A HEAD is enough and costs nothing next to the alternative,
  * which is transferring the whole thing a second time to find out.
- *
  * @param {string} url - The archive's URL.
  * @param {AbortSignal} [signal] - Cancels the probe.
  * @returns {Promise<number>} - Length, or 0.

package/src/traffic-stats.js

@@ -37,7 +37,6 @@ const PRUNE_EVERY_MS = 10 * 60 * 1000;
  * A week of 15-second samples is 40,000 points and a chart a thousand pixels
  * wide; sending all of them wastes the transfer and the browser's time to draw
  * something no eye can resolve. Averaging into buckets keeps the shape.
- *
  * @param {number} from - Start of the window, unix seconds.
  * @param {number} to - End of the window, unix seconds.
  * @param {number} buckets - How many points are wanted.
@@ -116,7 +115,6 @@ export class TrafficStats {
    * Zero rows are written as readily as busy ones: a gap in the series would
    * be indistinguishable from the node having been switched off, and "this
    * archive did nothing all week" is a real answer somebody wants.
-   *
    * @returns {Promise<number>} - How many rows were written.
    */
   async sample() {
@@ -163,7 +161,6 @@ export class TrafficStats {
    * Averaged into buckets rather than returned raw — see bucketSeconds. The
    * timestamp of a bucket is its start, so a caller plotting them gets evenly
    * spaced points without having to know the sample interval.
-   *
    * @param {object} [options] - Options.
    * @param {string} [options.infoHash] - One archive, or every one summed.
    * @param {number} [options.hours] - How far back to read.
@@ -226,7 +223,6 @@ export class TrafficStats {
    * sampled over. Approximate by construction -- it assumes each sample held
    * until the next -- and the right shape of approximate: it cannot drift from
    * the graph above it, because it is the same numbers.
-   *
    * @param {object} [options] - Options.
    * @param {number} [options.hours] - How far back to read.
    * @returns {object[]} - `{infoHash, down, up, samples}`, busiest first.
@@ -254,7 +250,10 @@ export class TrafficStats {
       }));
   }
 
-  /** Starts sampling and pruning. @returns {void} */
+  /**
+   * Starts sampling and pruning.
+   * @returns {void}
+   */
   start() {
     if (this.#timer) return;
     const tick = () =>
@@ -276,7 +275,10 @@ export class TrafficStats {
     this.#pruneTimer.unref?.();
   }
 
-  /** Stops sampling. @returns {void} */
+  /**
+   * Stops sampling.
+   * @returns {void}
+   */
   stop() {
     if (this.#timer) clearInterval(this.#timer);
     if (this.#pruneTimer) clearInterval(this.#pruneTimer);
@@ -292,7 +294,6 @@ export class TrafficStats {
  * resume data, the DHT node cache. The config directory is the operator's:
  * hand-edited, diffed, copied between nodes, and the thing you reach for when a
  * node will not start. A database that grows on its own does not belong there.
- *
  * @param {object} config - Resolved configuration.
  * @returns {Promise<object>} - An open DatabaseSync.
  */

package/src/watch.js

@@ -22,7 +22,6 @@ import { retains, retire } from './retention.js';
  * `monthly-*.pmtiles` is what they mean. Every other character is escaped, so
  * a name containing regex punctuation matches itself rather than becoming a
  * pattern by accident.
- *
  * @param {string} pattern - A glob, matched against the basename.
  * @returns {RegExp} - Anchored, case-insensitive.
  */
@@ -35,9 +34,11 @@ export function globToRegExp(pattern) {
       return character.replace(/[.+^${}()|[\]\\]/g, '\\$&');
     })
     .join('');
+  // eslint-disable-next-line security/detect-non-literal-regexp -- body is the escaped glob built just above
   return new RegExp(`^${body}$`, 'i');
 }
 
+/** Watches folders and imports the archives that appear in them. */
 export class WatchManager {
   #library;
   #watchers = [];

package/src/web/index.html

@@ -5228,6 +5228,16 @@ Every piece is hashed against the ` +
                 'restores the old behaviour. They are written in order ' +
                 'whatever it is set to.',
             },
+            {
+              key: 'stacks.resumeExports',
+              label: 'Pick up an unfinished export when the node starts',
+              type: 'boolean',
+              restart: true,
+              help:
+                'A checkpoint is the hours already spent, and the case that ' +
+                'costs most is the one nobody chose - a crash, or a restart ' +
+                'mid-run. Only where the recipe still resolves to what it did.',
+            },
             {
               key: 'stacks.bakePauseMs',
               label: 'Pause between batches while exporting',

package/tools/tile-bench.mjs

@@ -305,7 +305,12 @@ function twoModes(values) {
   };
 }
 
-/** A compact picture of where the latencies actually fell. */
+/**
+ * A compact picture of where the latencies actually fell.
+ * @param {number[]} values - Latencies, in milliseconds.
+ * @param {number} [width] - Widest bar, in characters.
+ * @returns {string[]} - One line per bucket.
+ */
 function histogram(values, width = 34) {
   if (values.length === 0) return [];
   const top = Math.max(...values);
@@ -323,7 +328,12 @@ function histogram(values, width = 34) {
   });
 }
 
-/** The value at a percentile of a sorted copy. */
+/**
+ * The value at a percentile of a sorted copy.
+ * @param {number[]} values - Latencies, in milliseconds.
+ * @param {number} fraction - Where to look, 0 to 1.
+ * @returns {number} - The value there, or 0 when there are none.
+ */
 function percentile(values, fraction) {
   if (values.length === 0) return 0;
   const sorted = [...values].sort((one, two) => one - two);
@@ -427,7 +437,12 @@ function report(sides) {
   void width;
 }
 
-/** How much slower the second is than the first, as a readable ratio. */
+/**
+ * How much slower the second is than the first, as a readable ratio.
+ * @param {number} one - A's median, in milliseconds.
+ * @param {number} two - B's median, in milliseconds.
+ * @returns {string} - A phrase for the summary, or nothing when either is missing.
+ */
 function ratio(one, two) {
   if (!one || !two) return '';
   const factor = two / one;