pmtiles-swarm 0.69.0 → 0.71.0

package/CHANGELOG.md

@@ -7,6 +7,60 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.71.0
+### ✨ Features and improvements
+- **The preview comes back to where it was opened from.** The link out of a preview said
+  "console" and went to the archive list, whatever had been previewed - so previewing a stack and
+  coming back meant finding the Stacks tab again. The console now names its view in the address
+  and honours one it is given, and a preview links to the view it belongs to.
+
+- **A category offers terrain where its newest build is terrain.** Archives and stacks got the
+  second button; categories did not, on either page. Same rule and the same reason: the raster
+  keeps the plain name because it is the view that shows a hole.
+
+### 🐞 Bug fixes
+- **A stopped export resumed; a restarted service did not.** Both were called "stopping keeps the
+  work", and only one of them did. **Stop export** cancels the job, which writes a checkpoint. A
+  service restart tells it nothing at all: the process is torn down, and whatever had been merged
+  since the last checkpoint was merged again.
+
+  Two things were wrong. Nothing asked a running export to stop when the node did, so exports are
+  now a shutdown step of their own - cancelled early in the sequence, before the pieces they read
+  through are taken away, and waited for so each writes its checkpoint.
+
+  And a checkpoint fired on a tile count alone, every five thousand. That is the wrong measure for
+  a bake merging slowly, which can run for an hour without reaching it - an export that had done a
+  few hundred tiles had never checkpointed at all, so a restart lost everything it had done. There
+  is a clock now as well, thirty seconds, whichever comes first. A checkpoint costs about eleven
+  milliseconds.
+
+## 0.70.0
+### ✨ Features and improvements
+- **An export works on the disk the archive is going to.** It worked under the data directory and
+  moved the finished file afterwards, which is the wrong disk twice over. The bytes have to go
+  where there is room for them, and a 700 GiB archive is not something a data directory is sized
+  for - while the disk chosen to hold the finished archive is, by definition.
+
+  The second problem was the move. `publish` renames within a filesystem and copies across one, so
+  a bake that worked elsewhere ended with a full copy of the whole archive - and until that
+  finished, the buffered tile data and the finished file both sat on the data directory's disk.
+  Working at the destination makes the last step a rename and removes the second full-size write.
+
+  Nothing appears at the destination until the end either way: the archive is assembled in one
+  pass at finalize, so until then `<destination>/bakes/<stack>/` holds the buffered tiles and the
+  checkpoint and no `.pmtiles` at all.
+
+### 🐞 Bug fixes
+- **A stack said "without sharp installed, its tiles answer 501" whether or not sharp was
+  installed.** It is a conditional the console can resolve and the reader cannot: nothing on the
+  page said which way it fell, so the line told somebody with a working stack that it might not
+  be, and somebody with a broken one that it might be fine.
+
+  `/api/stacks` has always reported the codec. Now the stack row uses it. With one installed there
+  is nothing to say, so nothing is said - and the "needs a codec" badge is gone too, because a
+  warning on something that works teaches people to ignore warnings. Without one it says the tiles
+  answer 501 *here*, names the field responsible, and says what to install.
+
 ## 0.69.0
 ### ✨ Features and improvements
 - **An export merges several tiles at once.** It did one at a time, start to finish, before
@@ -40,9 +94,6 @@
   while it ran was never looked at again, and the export completed as though nobody had asked it
   to stop. Checked around each batch as well now.
 
-### 🐞 Bug fixes
-- _...Add new stuff here..._
-
 ## 0.68.3
 ### 🐞 Bug fixes
 - **Exporting a stack failed on its first merged tile.** "Cannot transfer object of unsupported

package/docs/tile-stacks.md

@@ -789,6 +789,20 @@ One bake per stack at a time. Two runs of one recipe write the same checkpoint
 files over each other, and the second would resume the first's work believing it
 were its own.
 
+The work happens **on the filesystem the archive is going to** — under
+`<destination>/bakes/<stack>/` — not under the data directory. The bytes have
+to go where there is room for them, and a 700 GiB archive is not something a
+data directory is sized for, while the disk chosen to hold the finished archive
+is by definition. It also turns the last step of a long job from a copy of the
+whole archive into a rename, and means the buffered tile data and the finished
+archive never both have to fit on the data directory's disk.
+
+Nothing appears at the destination itself until the end. The archive is
+assembled in one pass at finalize, so until then the directory holds the
+buffered tiles and the checkpoint and no `.pmtiles` at all. Choosing a different
+location for a resumed bake looks for the checkpoint in the new place and finds
+none, so it starts over.
+
 ### What exists now
 
 `src/pmtiles-write.js` writes archives, `src/pmtiles-scan.js` reads back what one

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.69.0",
+  "version": "0.71.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/bake-jobs.js

@@ -24,9 +24,36 @@ import { outputFormat } from './stack-tile.js';
  * first's work as if it were its own.
  */
 
-/** Where unfinished bakes live, under the data directory. */
+/** What the working directory is called, wherever it sits. */
 const WORK_DIR = 'bakes';
 
+/**
+ * Where a bake does its work.
+ *
+ * On the filesystem the archive is going to, not under the data directory.
+ * Two reasons, and the first is the one that decides it: the bytes have to go
+ * somewhere there is room for them, and a 700 GiB archive is not something a
+ * data directory is sized for -- while the disk chosen to hold the finished
+ * archive is, by definition.
+ *
+ * The second falls out of the first. `publish` renames the finished file into
+ * place where it can and copies it where it cannot, so working on the
+ * destination's filesystem turns the last step of a long job from a full copy
+ * of the whole archive into a rename. That also removes the second full-size
+ * write, since the buffered tile data and the finished archive no longer have
+ * to exist on the data directory's disk at the same time.
+ *
+ * The directory is removed when the bake finishes. A cancelled one keeps it,
+ * which is the point of it.
+ * @param {object} job - The job, carrying where it will publish to.
+ * @param {object} config - The node's configuration.
+ * @returns {string} - The working directory.
+ */
+export function workDirFor(job, config = {}) {
+  const root = job.publishDir ?? config.savePath ?? config.dataDir ?? './data';
+  return path.join(root, WORK_DIR, job.stackId);
+}
+
 /** A bake that has finished is kept this long, so the console can report it. */
 const KEEP_FINISHED_MS = 10 * 60 * 1000;
 
@@ -81,6 +108,38 @@ export class BakeManager {
     return true;
   }
 
+  /**
+   * Stops every running bake and waits for each to write its checkpoint.
+   *
+   * For a service stopping. Without this a restart kills a bake where it
+   * stands: the merging half is cancellable and keeps its work, but only if
+   * something tells it to stop, and a process being torn down does not. What
+   * was merged since the last checkpoint would be merged again.
+   * @param {object} [options] - `timeoutMs` to stop waiting.
+   * @returns {Promise<number>} - How many were stopped.
+   */
+  async stopAll(options = {}) {
+    const running = [...this.#jobs.values()].filter((job) => !job.finishedAt);
+    if (running.length === 0) return 0;
+
+    for (const job of running) {
+      job.cancelling = true;
+      job.controller.abort();
+    }
+
+    // Bounded, because a shutdown that waits for ever is a shutdown that gets
+    // killed harder. A checkpoint is milliseconds; what takes time is the tile
+    // in flight noticing it has been abandoned.
+    const timeout = new Promise((resolve) =>
+      setTimeout(resolve, options.timeoutMs ?? 10000).unref?.(),
+    );
+    await Promise.race([
+      Promise.all(running.map((job) => job.promise?.catch(() => {}))),
+      timeout,
+    ]);
+    return running.length;
+  }
+
   /**
    * Starts a bake, and answers as soon as it is running rather than when it
    * finishes.
@@ -173,11 +232,7 @@ export class BakeManager {
    * @returns {Promise<void>} - Resolves when the archive is in the catalog.
    */
   async #run(job, resolved, codec, options) {
-    const workDir = path.join(
-      this.#config.dataDir ?? './data',
-      WORK_DIR,
-      job.stackId,
-    );
+    const workDir = workDirFor(job, this.#config);
     const destination = path.join(workDir, job.name);
     const format = outputFormat(resolved);
 

package/src/bake.js

@@ -25,6 +25,17 @@ import { Compression, PMTilesWriter, TileType } from './pmtiles-write.js';
 /** Tiles merged between checkpoints. */
 const DEFAULT_CHECKPOINT_EVERY = 5000;
 
+/**
+ * Seconds between checkpoints, however few tiles have been done.
+ *
+ * A count alone is the wrong measure. A bake merging slowly -- a big tile, a
+ * cache-mode source, a stack with several layers -- can run for an hour
+ * without reaching five thousand, and a process killed before its first
+ * checkpoint has nothing to resume from at all. A checkpoint costs about
+ * eleven milliseconds, so doing one on the clock is close to free.
+ */
+const DEFAULT_CHECKPOINT_SECONDS = 30;
+
 /** Tiles merged at once. */
 const DEFAULT_CONCURRENCY = 4;
 
@@ -454,6 +465,7 @@ export async function bakeStack(options) {
     header = {},
     deduplicate = true,
     checkpointEvery = DEFAULT_CHECKPOINT_EVERY,
+    checkpointSeconds = DEFAULT_CHECKPOINT_SECONDS,
     pauseMs = 0,
     concurrency = DEFAULT_CONCURRENCY,
   } = options;
@@ -486,6 +498,7 @@ export async function bakeStack(options) {
   let skipped = found?.skipped ?? 0;
   let lastTileId = found?.lastTileId ?? -1;
   let sinceCheckpoint = 0;
+  let checkpointedAt = Date.now();
   let persisted = found?.entries.length ?? 0;
 
   /**
@@ -508,6 +521,7 @@ export async function bakeStack(options) {
       persisted,
     );
     sinceCheckpoint = 0;
+    checkpointedAt = Date.now();
   };
 
   try {
@@ -563,7 +577,13 @@ export async function bakeStack(options) {
       }
 
       batch = [];
-      if (sinceCheckpoint >= checkpointEvery) await checkpoint();
+      // Whichever comes first. The count keeps a fast bake from checkpointing
+      // constantly; the clock keeps a slow one from never checkpointing at
+      // all.
+      const due =
+        sinceCheckpoint >= checkpointEvery ||
+        Date.now() - checkpointedAt >= checkpointSeconds * 1000;
+      if (due) await checkpoint();
       signal?.throwIfAborted();
 
       // Handing time back, where the operator asked for that. A bake on a node

package/src/index.js

@@ -474,6 +474,22 @@ PMTILES_SWARM_PUBLIC_URL
     cutlines,
   });
 
+  // Early in the sequence, so an export is told to stop before the pieces it
+  // reads through are taken away. Its checkpoint is the hours already spent.
+  stoppers.unshift({
+    label: 'stack exports',
+    stop: async () => {
+      const stopped = await bakes.stopAll();
+      if (stopped > 0) {
+        console.log(
+          `[shutdown] stopped ${stopped} stack export(s); each kept its ` +
+            'checkpoint, so exporting again carries on from there',
+        );
+      }
+    },
+    ms: 12000,
+  });
+
   // Reads the head of anything joined but not yet understood — the header,
   // then the root directory and metadata it points at. Without this an archive
   // being mirrored is unservable until the download happens to reach byte

package/src/web/index.html

@@ -4019,7 +4019,12 @@ Every piece is hashed against the ` +
                     ? `<div class="links">
                          ${copyable(ends.tileJson, 'TileJSON')}
                          ${copyable(ends.xyz, 'XYZ')}
-                         ${link(ends.preview, 'preview')}
+                         ${
+                           drawsAsTerrain(newest)
+                             ? `${link(`${ends.preview}?raw=1`, 'preview')}
+                                ${link(ends.preview, 'terrain')}`
+                             : link(ends.preview, 'preview')
+                         }
                          ${link(ends.torrent, '.torrent', true)}
                          ${copyable(ends.magnet, 'magnet', true)}
                          ${copyable(ends.feed, 'RSS')}
@@ -5225,7 +5230,7 @@ Every piece is hashed against the ` +
             },
             {
               key: 'stacks.bakePauseMs',
-              label: 'Pause between tiles while exporting',
+              label: 'Pause between batches while exporting',
               type: 'number',
               placeholder: '0',
               unit: 'ms',
@@ -6995,17 +7000,19 @@ Every piece is hashed against the ` +
       }
 
       // ── Tabs ──────────────────────────────────────────────────────────────
+      /** Views a URL may name, so a link can come back to the right one. */
+      const TABS = ['archives', 'categories', 'stacks', 'traffic', 'settings'];
+
       const showTab = (name) => {
-        for (const tab of [
-          'archives',
-          'categories',
-          'stacks',
-          'traffic',
-          'settings',
-        ]) {
+        for (const tab of TABS) {
           $(`view-${tab}`).hidden = tab !== name;
           $(`tab-${tab}`).classList.toggle('on', tab === name);
         }
+        // Written into the address, so a page that sends somebody here can say
+        // where to land -- a preview of a stack belongs back at Stacks, not at
+        // whatever the console opens on. `replaceState` rather than assigning
+        // the hash, which would scroll and add a history entry per tab.
+        history.replaceState(null, '', name === 'archives' ? '#' : `#${name}`);
         if (name === 'settings') loadSettings().catch((e) => toast(e.message));
         if (name === 'categories') loadCategories().catch((e) => toast(e.message));
         if (name === 'stacks') loadStacks().catch((e) => toast(e.message));
@@ -7014,6 +7021,13 @@ Every piece is hashed against the ` +
           loadSwarmTraffic();
         }
       };
+      // A view named in the address wins over the one this opens on. Ignored
+      // when it names nothing, so an old bookmark is not a blank console.
+      const tabFromUrl = () => {
+        const named = location.hash.replace('#', '');
+        return TABS.includes(named) ? named : null;
+      };
+
       // The footer's year, set from the clock rather than typed into a file
       // nobody will remember to edit. The version beside it arrives with the
       // first status, and reads "…" until then.
@@ -7042,11 +7056,15 @@ Every piece is hashed against the ` +
       // The shapes this node has, so a source can be clipped to one by name
       // rather than by remembering a filename.
       let stackCutlines = [];
+      // Which codec is installed, or null. A stack row needs it to say whether
+      // its pixel work is a problem here or merely a fact about the recipe.
+      let stackCodec = null;
 
       const loadStacks = async () => {
         const body = await api('/api/stacks');
         const { stacks, codec } = body;
         stackCutlines = body.cutlines ?? [];
+        stackCodec = codec ?? null;
         // A category resolving to a new build changes this, which is the
         // reason to poll at all -- but most polls change nothing, and
         // repainting a list somebody is about to click is worse than not.
@@ -7108,8 +7126,12 @@ Every piece is hashed against the ` +
         if (unresolved.length) {
           badges.push('<span class="pill warn">sources missing</span>');
         }
-        if (stack.needsCodec) {
-          badges.push('<span class="pill warn">needs a codec</span>');
+        // Only where it is actually a problem. With a codec installed the
+        // recipe asking for pixel work is a fact about the recipe, not
+        // something wrong with it, and a warning badge on a working stack
+        // teaches people to ignore warning badges.
+        if (stack.needsCodec && !stackCodec) {
+          badges.push('<span class="pill warn">no codec for this</span>');
         }
         if (stack.pinned) badges.push('<span class="pill">pinned</span>');
 
@@ -7191,10 +7213,10 @@ Every piece is hashed against the ` +
             }
             ${bakeLine}
             ${
-              stack.needsCodec
-                ? `<p class="muted">Serving this means decoding pixels
-                   (<code>${escapeHtml(stack.needsCodec)}</code>). Without
-                   <code>sharp</code> installed, its tiles answer 501.</p>`
+              stack.needsCodec && !stackCodec
+                ? `<p class="bad">Its tiles answer 501 here: serving this means
+                   decoding pixels (<code>${escapeHtml(stack.needsCodec)}</code>)
+                   and this node has no codec. <code>npm install sharp</code>.</p>`
                 : ''
             }
             <table class="tight">
@@ -8304,6 +8326,10 @@ Every piece is hashed against the ` +
           // An unreachable server is reported by refresh() in the status line.
         }
         refresh();
+        // After the archives have been asked for, so the tab that opens is the
+        // one asked for and not the one this happens to start on.
+        const named = tabFromUrl();
+        if (named && named !== 'archives') showTab(named);
       }
 
       start();

package/src/web/preview.html

@@ -69,7 +69,7 @@
       <code id="endpoint"></code>
       <span class="links">
         <a id="mode" href="#" hidden></a>
-        <a href="/">← console</a>
+        <a id="back" href="/">← console</a>
       </span>
     </header>
     <div id="note" hidden></div>
@@ -96,6 +96,18 @@
       );
       const $ = (id) => document.getElementById(id);
 
+      // Back to the view this was opened from rather than to whatever the
+      // console opens on. The path already says which: a stack was reached
+      // from Stacks, a category from Categories, an archive from the list.
+      const back = $('back');
+      if (back) {
+        back.href = location.pathname.startsWith('/stacks/')
+          ? '/#stacks'
+          : location.pathname.startsWith('/latest/')
+            ? '/#categories'
+            : '/';
+      }
+
       const fail = (message) => {
         document.body.innerHTML = `<div class="banner">${message}</div>`;
       };