pmtiles-swarm 0.70.0 → 0.72.0

package/CHANGELOG.md

@@ -7,6 +7,72 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.72.0
+### ✨ Features and improvements
+- **An unfinished export is picked up when the node starts.** The checkpoint was always there;
+  finding it again meant somebody remembering to press the button. That is fine for an export
+  stopped on purpose and wrong for one a crash took, which is the case that costs the most and
+  gives the least warning.
+
+  The checkpoint now records what the job was - what the archive is called, what the file is
+  called, where it was going, what it is filed under - because none of that can be worked out from
+  the tiles on disk, and a resumed export has to be the one somebody asked for rather than a new
+  one with today's date on it.
+
+  Only where the recipe still resolves to what it did. `bakeRevision` covers what each source
+  became, so a rebuilt source means the checkpoint holds half of a map that no longer exists;
+  that is left alone and reported, to be discarded when somebody exports again deliberately.
+  `stacks.resumeExports` turns it off for a node where hours of merging should never begin
+  without being asked for.
+
+### 🐞 Bug fixes
+- **The node was exiting when a peer wire outlived its torrent.** Twice in one evening on a real
+  node, both times killing an export that was hours in:
+
+      torrent.js:2092  this.client._debugId    -> reading '_debugId' of null
+      peer.js:201      this.swarm.client.dht   -> reading 'dht' of null
+
+  WebTorrent nulls `torrent.client` when a torrent is destroyed and does not always tear down that
+  torrent's peer wires with it. One fired a keep-alive timeout afterwards and the other finished a
+  handshake, and both reached through the dead reference. They happen inside a timer or a socket
+  callback, so there is no promise to reject and no call of ours to wrap - it is an uncaught
+  exception, and Node's answer to that is to exit. Under `Restart=always` that reads as a
+  mysterious restart rather than as a crash.
+
+  `src/crash-guard.js` survives exactly that shape and nothing else: a TypeError, about one of a
+  named list of properties, raised from a frame inside the torrent libraries. Every other uncaught
+  exception still stops the process, because one that survives everything is one that lies about
+  its own state. The two stacks from the journal are in the tests verbatim, alongside the near
+  misses that must still be fatal - the same error from our own code, a different error from
+  theirs.
+
+## 0.71.0
+### ✨ Features and improvements
+- **The preview comes back to where it was opened from.** The link out of a preview said
+  "console" and went to the archive list, whatever had been previewed - so previewing a stack and
+  coming back meant finding the Stacks tab again. The console now names its view in the address
+  and honours one it is given, and a preview links to the view it belongs to.
+
+- **A category offers terrain where its newest build is terrain.** Archives and stacks got the
+  second button; categories did not, on either page. Same rule and the same reason: the raster
+  keeps the plain name because it is the view that shows a hole.
+
+### 🐞 Bug fixes
+- **A stopped export resumed; a restarted service did not.** Both were called "stopping keeps the
+  work", and only one of them did. **Stop export** cancels the job, which writes a checkpoint. A
+  service restart tells it nothing at all: the process is torn down, and whatever had been merged
+  since the last checkpoint was merged again.
+
+  Two things were wrong. Nothing asked a running export to stop when the node did, so exports are
+  now a shutdown step of their own - cancelled early in the sequence, before the pieces they read
+  through are taken away, and waited for so each writes its checkpoint.
+
+  And a checkpoint fired on a tile count alone, every five thousand. That is the wrong measure for
+  a bake merging slowly, which can run for an hour without reaching it - an export that had done a
+  few hundred tiles had never checkpointed at all, so a restart lost everything it had done. There
+  is a clock now as well, thirty seconds, whichever comes first. A checkpoint costs about eleven
+  milliseconds.
+
 ## 0.70.0
 ### ✨ Features and improvements
 - **An export works on the disk the archive is going to.** It worked under the data directory and

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.70.0",
+  "version": "0.72.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/bake-jobs.js

@@ -1,3 +1,4 @@
+import fs from 'node:fs/promises';
 import path from 'node:path';
 import {
   assertBakeable,
@@ -108,6 +109,109 @@ export class BakeManager {
     return true;
   }
 
+  /**
+   * Picks up exports a previous run did not finish.
+   *
+   * A checkpoint is the hours already spent, and finding one again used to
+   * mean somebody remembering to press the button. That is fine for an export
+   * stopped on purpose and wrong for one a crash took -- which is the case
+   * that costs the most and gives the least warning.
+   *
+   * Only where the recipe still resolves to what it did. `bakeRevision` covers
+   * what each source became, so a rebuilt source means the checkpoint holds
+   * half of a map that no longer exists; that is left alone rather than
+   * continued, and discarded when somebody exports again deliberately.
+   * @param {Function} resolve - `(stackId) => resolved stack | null`.
+   * @returns {Promise<object[]>} - The jobs started.
+   */
+  async resumeAll(resolve) {
+    const started = [];
+
+    for (const root of this.#workRoots()) {
+      const directory = path.join(root, WORK_DIR);
+      const found = await fs.readdir(directory).catch(() => []);
+
+      for (const stackId of found) {
+        if (this.#jobs.has(stackId)) continue;
+        const state = await fs
+          .readFile(path.join(directory, stackId, 'bake-state.json'), 'utf8')
+          .then((raw) => JSON.parse(raw))
+          .catch(() => null);
+        // Written by a version that did not record what the job was. There is
+        // nothing to reproduce it from, so it waits for a person.
+        if (!state?.describe) continue;
+
+        const resolved = resolve(stackId);
+        if (!resolved || bakeRevision(resolved) !== state.revision) continue;
+
+        try {
+          const job = await this.start({ resolved, ...state.describe });
+          started.push(job);
+          console.log(
+            `[bake] picking up ${stackId} where it stopped: ` +
+              `${state.written ?? 0} tiles already merged`,
+          );
+        } catch (error) {
+          // A stack that cannot be baked now -- no codec, sources gone -- is
+          // said out loud and left. Its checkpoint is still there.
+          console.warn(`[bake] could not resume ${stackId}: ${error.message}`);
+        }
+      }
+    }
+
+    return started;
+  }
+
+  /**
+   * Everywhere an export might have left work.
+   *
+   * The working directory follows the destination, and a destination is
+   * whatever was chosen at the time -- so this is every place one could have
+   * been: the named locations, the default save path, and the data directory,
+   * where exports worked before the working directory moved.
+   * @returns {string[]} - Roots to look under, without repeats.
+   */
+  #workRoots() {
+    const roots = [
+      ...(this.#config.locations ?? []).map((one) => one?.path),
+      this.#config.savePath,
+      this.#config.dataDir,
+    ].filter(Boolean);
+    return [...new Set(roots.map((one) => path.resolve(one)))];
+  }
+
+  /**
+   * Stops every running bake and waits for each to write its checkpoint.
+   *
+   * For a service stopping. Without this a restart kills a bake where it
+   * stands: the merging half is cancellable and keeps its work, but only if
+   * something tells it to stop, and a process being torn down does not. What
+   * was merged since the last checkpoint would be merged again.
+   * @param {object} [options] - `timeoutMs` to stop waiting.
+   * @returns {Promise<number>} - How many were stopped.
+   */
+  async stopAll(options = {}) {
+    const running = [...this.#jobs.values()].filter((job) => !job.finishedAt);
+    if (running.length === 0) return 0;
+
+    for (const job of running) {
+      job.cancelling = true;
+      job.controller.abort();
+    }
+
+    // Bounded, because a shutdown that waits for ever is a shutdown that gets
+    // killed harder. A checkpoint is milliseconds; what takes time is the tile
+    // in flight noticing it has been abandoned.
+    const timeout = new Promise((resolve) =>
+      setTimeout(resolve, options.timeoutMs ?? 10000).unref?.(),
+    );
+    await Promise.race([
+      Promise.all(running.map((job) => job.promise?.catch(() => {}))),
+      timeout,
+    ]);
+    return running.length;
+  }
+
   /**
    * Starts a bake, and answers as soon as it is running rather than when it
    * finishes.
@@ -309,6 +413,17 @@ export class BakeManager {
       header: { format },
       pauseMs: this.#config.stacks?.bakePauseMs ?? 0,
       concurrency: this.#concurrency(),
+      // Written into the checkpoint in the shape `start` takes, so picking one
+      // up is handing it back rather than reconstructing it. Nothing here can
+      // be worked out from the tiles on disk: what the archive is called, what
+      // the file is called, where it was going, what it is filed under.
+      describe: {
+        name: job.archiveName,
+        filename: job.name,
+        publishDir: job.publishDir ?? null,
+        description: options.description ?? null,
+        categories: options.categories ?? null,
+      },
       metadata: {
         name: job.archiveName,
         // Only what was asked for. Falling back to the recipe's own

package/src/bake.js

@@ -25,6 +25,17 @@ import { Compression, PMTilesWriter, TileType } from './pmtiles-write.js';
 /** Tiles merged between checkpoints. */
 const DEFAULT_CHECKPOINT_EVERY = 5000;
 
+/**
+ * Seconds between checkpoints, however few tiles have been done.
+ *
+ * A count alone is the wrong measure. A bake merging slowly -- a big tile, a
+ * cache-mode source, a stack with several layers -- can run for an hour
+ * without reaching five thousand, and a process killed before its first
+ * checkpoint has nothing to resume from at all. A checkpoint costs about
+ * eleven milliseconds, so doing one on the clock is close to free.
+ */
+const DEFAULT_CHECKPOINT_SECONDS = 30;
+
 /** Tiles merged at once. */
 const DEFAULT_CONCURRENCY = 4;
 
@@ -454,8 +465,14 @@ export async function bakeStack(options) {
     header = {},
     deduplicate = true,
     checkpointEvery = DEFAULT_CHECKPOINT_EVERY,
+    checkpointSeconds = DEFAULT_CHECKPOINT_SECONDS,
     pauseMs = 0,
     concurrency = DEFAULT_CONCURRENCY,
+    // Written into the checkpoint and handed back by `readCheckpoint`. A
+    // resumed export has to reproduce the one somebody asked for -- its name,
+    // where it was going, what it was filed under -- and none of that can be
+    // worked out from the tiles on disk.
+    describe,
   } = options;
   const batchSize = Math.max(1, Math.floor(concurrency));
 
@@ -486,6 +503,7 @@ export async function bakeStack(options) {
   let skipped = found?.skipped ?? 0;
   let lastTileId = found?.lastTileId ?? -1;
   let sinceCheckpoint = 0;
+  let checkpointedAt = Date.now();
   let persisted = found?.entries.length ?? 0;
 
   /**
@@ -503,11 +521,13 @@ export async function bakeStack(options) {
         dataBytes: writer.dataBytes,
         addressed: writer.addressedTiles,
         clustered: writer.clustered,
+        ...(describe ? { describe } : {}),
       },
       writer.entries,
       persisted,
     );
     sinceCheckpoint = 0;
+    checkpointedAt = Date.now();
   };
 
   try {
@@ -563,7 +583,13 @@ export async function bakeStack(options) {
       }
 
       batch = [];
-      if (sinceCheckpoint >= checkpointEvery) await checkpoint();
+      // Whichever comes first. The count keeps a fast bake from checkpointing
+      // constantly; the clock keeps a slow one from never checkpointing at
+      // all.
+      const due =
+        sinceCheckpoint >= checkpointEvery ||
+        Date.now() - checkpointedAt >= checkpointSeconds * 1000;
+      if (due) await checkpoint();
       signal?.throwIfAborted();
 
       // Handing time back, where the operator asked for that. A bake on a node

package/src/config.js

@@ -388,6 +388,19 @@ const DEFAULTS = {
      * not clustered is bad at the one thing PMTiles is for.
      */
     bakeConcurrency: 4,
+    /**
+     * Whether an unfinished export is picked up when the node starts.
+     *
+     * A checkpoint is the hours already spent, and the case that costs most is
+     * the one nobody chose: a crash, or a restart in the middle of a run. On
+     * by default for that reason. Off for a node where hours of merging should
+     * never begin without somebody asking for it.
+     *
+     * Only where the recipe still resolves to what it did. A rebuilt source
+     * means the checkpoint holds half of a map that no longer exists, and that
+     * is left alone either way.
+     */
+    resumeExports: true,
   },
   /**
    * Folders scanned for new archives. Each entry is `{ path, categories,

package/src/crash-guard.js

@@ -0,0 +1,94 @@
+/**
+ * Surviving the one crash a torrent client raises from a timer.
+ *
+ * WebTorrent nulls `torrent.client` when a torrent is destroyed, and does not
+ * always tear down that torrent's peer wires with it. A wire that outlives its
+ * torrent then fires — a keep-alive timeout, or a handshake finishing — and
+ * reaches through the dead reference:
+ *
+ *     torrent.js:2092  this.client._debugId    → reading '_debugId' of null
+ *     peer.js:201      this.swarm.client.dht   → reading 'dht' of null
+ *
+ * Both happen inside a `setTimeout` or a socket callback, so there is no
+ * promise to reject and no call of ours to wrap: it is an uncaught exception,
+ * and Node's answer to that is to exit. Under `Restart=always` the service
+ * comes straight back, which is why this reads as a mysterious restart rather
+ * than as a crash — and anything the node was in the middle of, an export
+ * above all, is simply gone.
+ *
+ * So this is deliberately narrow. It survives exactly the shape above and
+ * nothing else: a TypeError, about a null property this list names, raised
+ * from inside the torrent libraries. Every other uncaught exception keeps
+ * Node's behaviour, because a process that survives everything is a process
+ * that lies about its own state.
+ */
+
+/** Packages whose internals may raise this. Ours is not among them. */
+const LIBRARIES = [
+  '/webtorrent/',
+  '/bittorrent-protocol/',
+  '/torrent-discovery/',
+];
+
+/** The properties a wire reaches for on a client that has gone. */
+const REACHING_FOR = ['_debugId', 'dht', 'client', 'swarm', 'torrent', 'wires'];
+
+/**
+ * Whether an uncaught exception is a peer wire touching a destroyed torrent.
+ * @param {unknown} error - What was thrown.
+ * @returns {boolean} - True when it is safe to carry on.
+ */
+export function isDeadTorrentWire(error) {
+  if (!(error instanceof TypeError)) return false;
+
+  const message = String(error.message ?? '');
+  const reading =
+    /Cannot read propert(?:y|ies) of (?:null|undefined) \(reading '([^']+)'\)/.exec(
+      message,
+    );
+  // Older phrasings say it the other way round.
+  const older = /Cannot read property '([^']+)' of (?:null|undefined)/.exec(
+    message,
+  );
+  const property = reading?.[1] ?? older?.[1];
+  if (!property || !REACHING_FOR.includes(property)) return false;
+
+  // The frame that threw has to be theirs. An error of ours that happens to
+  // mention the same property is a bug worth dying on.
+  const stack = String(error.stack ?? '').replaceAll('\\', '/');
+  return LIBRARIES.some((library) => stack.includes(library));
+}
+
+/**
+ * Installs the guard.
+ *
+ * Returns what it registered so a test can take it off again, and so nothing
+ * has to reach into `process` to find out what is installed.
+ * @param {object} [options] - `onSurvived`, and `exit` for testing.
+ * @returns {Function} - The handler, for `process.off`.
+ */
+export function installCrashGuard(options = {}) {
+  const survived =
+    options.onSurvived ??
+    ((error) => {
+      console.warn(
+        `[torrent] a peer wire outlived its torrent and threw: ${error.message}. ` +
+          'Carried on — this is a fault inside webtorrent, not a state this ' +
+          'node cannot continue from.',
+      );
+    });
+  const exit = options.exit ?? ((code) => process.exit(code));
+
+  const handler = (error) => {
+    if (isDeadTorrentWire(error)) {
+      survived(error);
+      return;
+    }
+    // Node's own behaviour for everything else: say what happened, and stop.
+    console.error(error);
+    exit(1);
+  };
+
+  process.on('uncaughtException', handler);
+  return handler;
+}

package/src/index.js

@@ -5,8 +5,9 @@ import { parseArgs } from 'node:util';
 import { createApp } from './api.js';
 import { assertSafeToListen, createAuth } from './auth.js';
 import { Catalog } from './catalog.js';
-import { StackStore } from './stacks.js';
+import { StackStore, resolveStack } from './stacks.js';
 import { StackCache } from './stack-cache.js';
+import { installCrashGuard } from './crash-guard.js';
 import { CutlineStore } from './cutlines.js';
 import { BakeManager } from './bake-jobs.js';
 import { loadCodec } from './codec.js';
@@ -212,6 +213,11 @@ PMTILES_SWARM_PUBLIC_URL
   /** @type {Array<{label: string, stop: () => unknown, ms?: number}>} */
   const stoppers = [];
   installSignalHandlers(stoppers);
+  // Before anything opens a socket. A peer wire that outlives its torrent
+  // throws from a timer, which is an uncaught exception, which is an exit --
+  // and under Restart=always that reads as a mysterious restart rather than a
+  // crash, taking whatever was running with it.
+  installCrashGuard();
 
   // Before anything is created or any port is bound: an unauthenticated node
   // on a reachable address fails silently, working perfectly right up until
@@ -474,6 +480,41 @@ PMTILES_SWARM_PUBLIC_URL
     cutlines,
   });
 
+  // Anything a previous run did not finish. Deliberately after the stacks and
+  // the catalog are loaded, since a checkpoint is only worth picking up where
+  // the recipe still resolves to what it did when the work was done.
+  //
+  // Not awaited: an export is hours, and the node should be answering requests
+  // while it runs rather than after it.
+  if (config.stacks?.resumeExports !== false) {
+    bakes
+      .resumeAll((stackId) => {
+        const stack = stacks?.list().find((one) => one.id === stackId);
+        if (!stack) return null;
+        return resolveStack(stack, {
+          archive: (hash) => catalog.get(hash),
+          category: (name) => catalog.byCategory(name)[0] ?? null,
+        });
+      })
+      .catch((error) => console.warn(`[bake] resume failed: ${error.message}`));
+  }
+
+  // Early in the sequence, so an export is told to stop before the pieces it
+  // reads through are taken away. Its checkpoint is the hours already spent.
+  stoppers.unshift({
+    label: 'stack exports',
+    stop: async () => {
+      const stopped = await bakes.stopAll();
+      if (stopped > 0) {
+        console.log(
+          `[shutdown] stopped ${stopped} stack export(s); each kept its ` +
+            'checkpoint, so exporting again carries on from there',
+        );
+      }
+    },
+    ms: 12000,
+  });
+
   // Reads the head of anything joined but not yet understood — the header,
   // then the root directory and metadata it points at. Without this an archive
   // being mirrored is unservable until the download happens to reach byte

package/src/web/index.html

@@ -4019,7 +4019,12 @@ Every piece is hashed against the ` +
                     ? `<div class="links">
                          ${copyable(ends.tileJson, 'TileJSON')}
                          ${copyable(ends.xyz, 'XYZ')}
-                         ${link(ends.preview, 'preview')}
+                         ${
+                           drawsAsTerrain(newest)
+                             ? `${link(`${ends.preview}?raw=1`, 'preview')}
+                                ${link(ends.preview, 'terrain')}`
+                             : link(ends.preview, 'preview')
+                         }
                          ${link(ends.torrent, '.torrent', true)}
                          ${copyable(ends.magnet, 'magnet', true)}
                          ${copyable(ends.feed, 'RSS')}
@@ -5223,6 +5228,16 @@ Every piece is hashed against the ` +
                 'restores the old behaviour. They are written in order ' +
                 'whatever it is set to.',
             },
+            {
+              key: 'stacks.resumeExports',
+              label: 'Pick up an unfinished export when the node starts',
+              type: 'boolean',
+              restart: true,
+              help:
+                'A checkpoint is the hours already spent, and the case that ' +
+                'costs most is the one nobody chose - a crash, or a restart ' +
+                'mid-run. Only where the recipe still resolves to what it did.',
+            },
             {
               key: 'stacks.bakePauseMs',
               label: 'Pause between batches while exporting',
@@ -6995,17 +7010,19 @@ Every piece is hashed against the ` +
       }
 
       // ── Tabs ──────────────────────────────────────────────────────────────
+      /** Views a URL may name, so a link can come back to the right one. */
+      const TABS = ['archives', 'categories', 'stacks', 'traffic', 'settings'];
+
       const showTab = (name) => {
-        for (const tab of [
-          'archives',
-          'categories',
-          'stacks',
-          'traffic',
-          'settings',
-        ]) {
+        for (const tab of TABS) {
           $(`view-${tab}`).hidden = tab !== name;
           $(`tab-${tab}`).classList.toggle('on', tab === name);
         }
+        // Written into the address, so a page that sends somebody here can say
+        // where to land -- a preview of a stack belongs back at Stacks, not at
+        // whatever the console opens on. `replaceState` rather than assigning
+        // the hash, which would scroll and add a history entry per tab.
+        history.replaceState(null, '', name === 'archives' ? '#' : `#${name}`);
         if (name === 'settings') loadSettings().catch((e) => toast(e.message));
         if (name === 'categories') loadCategories().catch((e) => toast(e.message));
         if (name === 'stacks') loadStacks().catch((e) => toast(e.message));
@@ -7014,6 +7031,13 @@ Every piece is hashed against the ` +
           loadSwarmTraffic();
         }
       };
+      // A view named in the address wins over the one this opens on. Ignored
+      // when it names nothing, so an old bookmark is not a blank console.
+      const tabFromUrl = () => {
+        const named = location.hash.replace('#', '');
+        return TABS.includes(named) ? named : null;
+      };
+
       // The footer's year, set from the clock rather than typed into a file
       // nobody will remember to edit. The version beside it arrives with the
       // first status, and reads "…" until then.
@@ -8312,6 +8336,10 @@ Every piece is hashed against the ` +
           // An unreachable server is reported by refresh() in the status line.
         }
         refresh();
+        // After the archives have been asked for, so the tab that opens is the
+        // one asked for and not the one this happens to start on.
+        const named = tabFromUrl();
+        if (named && named !== 'archives') showTab(named);
       }
 
       start();

package/src/web/preview.html

@@ -69,7 +69,7 @@
       <code id="endpoint"></code>
       <span class="links">
         <a id="mode" href="#" hidden></a>
-        <a href="/">← console</a>
+        <a id="back" href="/">← console</a>
       </span>
     </header>
     <div id="note" hidden></div>
@@ -96,6 +96,18 @@
       );
       const $ = (id) => document.getElementById(id);
 
+      // Back to the view this was opened from rather than to whatever the
+      // console opens on. The path already says which: a stack was reached
+      // from Stacks, a category from Categories, an archive from the list.
+      const back = $('back');
+      if (back) {
+        back.href = location.pathname.startsWith('/stacks/')
+          ? '/#stacks'
+          : location.pathname.startsWith('/latest/')
+            ? '/#categories'
+            : '/';
+      }
+
       const fail = (message) => {
         document.body.innerHTML = `<div class="banner">${message}</div>`;
       };