pmtiles-swarm 0.64.0 → 0.66.0

package/CHANGELOG.md

@@ -4,6 +4,62 @@
 ### ✨ Features and improvements
 - _...Add new stuff here..._
 
+### 🐞 Bug fixes
+- _...Add new stuff here..._
+
+## 0.66.0
+### ✨ Features and improvements
+- **An export asks where it should go, and what to call it.** **Export to archive** now opens a
+  dialog instead of a confirmation, because there is something to decide. The save location uses
+  the same picker every other destination in this console uses, so a location named once under
+  Settings is offered here too — a baked planet is hundreds of gigabytes and which disk it lands on
+  is not a detail. The name and description default to the stack's and can be changed.
+
+  The filename is shown as the name is typed. A name is not a filename — separators and the
+  characters Windows refuses become dashes, and the date is appended so successive builds do not
+  collide — and a preview promising one thing while the server writes another would be worse than
+  no preview at all. A test asserts the browser's rule and `safeSegment` agree.
+
+### 🐞 Bug fixes
+- **A save location the node did not know was silently ignored.** The export resolved it after the
+  merge and swallowed the error, so naming a location that did not exist, or one that could not be
+  written, meant an hour of merging answered with a shrug and the archive on the default disk. It
+  is resolved before anything starts and the error is allowed through, which is what lets the
+  dialog put it beside the field that caused it.
+
+
+
+### 🐞 Bug fixes
+- _...Add new stuff here..._
+
+## 0.65.0
+### ✨ Features and improvements
+- **A bake stays out of the way of the node it runs on.** It runs in the main process — unlike
+  hashing, there is no sidecar to send it to — and the merge's pixel maths is entirely
+  synchronous, so every millisecond of it is a millisecond the node is not answering requests.
+  Three changes, all of them from measuring rather than assuming.
+
+  **The pixel maths moved to a worker.** `src/pixels.js` runs `elevation.js` and `rgba.js`
+  unchanged on another thread. Against a request arriving every 5 ms while a bake runs, the delay
+  that request sees at the 99th percentile falls from 10.9 ms to 6.5 ms with two sources, and from
+  18.6 ms to 10.6 ms with four. The bake is 2–17% slower for it. Rasters are handed over rather
+  than copied — a decoded tile is most of a megabyte per source — so a merge takes ownership of
+  what it is given, which is safe because nothing reads a contribution afterwards and is asserted
+  rather than assumed. Serving does not use it: one tile is a few milliseconds nobody notices.
+
+  **The checkpoint appends instead of rewriting.** It went through `serializeDirectory`, which was
+  elegant reuse and the wrong tool — that is a distribution format, and producing it costs a varint
+  pass over every entry. Re-encoding all of them every time also made the total work quadratic in
+  the length of the job. Fixed 24-byte records can be appended, and only the last one can change
+  once written, so a checkpoint costs the work since the last one: **446 ms at four million entries
+  becomes 11 ms, and stays 11 ms**.
+
+  **And `stacks.bakePauseMs`**, how long a bake waits between tiles. Zero by default, which is
+  right for a node baking and doing nothing else. On a node that is also serving maps it is the
+  direct trade between how long the bake takes and how much of the machine it takes while running.
+
+
+
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 

package/docs/tile-stacks.md

@@ -45,6 +45,7 @@ of its parts.
   - [What to iterate](#what-to-iterate)
   - [Running it](#running-it)
   - [What a baked archive says about itself](#what-a-baked-archive-says-about-itself)
+  - [Staying out of the way of the node it runs on](#staying-out-of-the-way-of-the-node-it-runs-on)
   - [What identifies a bake](#what-identifies-a-bake)
   - [Starting one, and watching it](#starting-one-and-watching-it)
   - [What exists now](#what-exists-now)
@@ -636,6 +637,59 @@ reason the live stack answers 404.
 - **Cancellable**, and stopping keeps the work: realising it is the wrong recipe
   should not mean waiting it out, and it should not mean starting over either.
 
+### Staying out of the way of the node it runs on
+
+A bake runs in the main process, and unlike hashing there is no sidecar to send
+it to. That matters because `elevation.js` and `rgba.js` are entirely
+synchronous — decoding heights, masking, resampling and painting are loops over
+typed arrays — so every millisecond of it is a millisecond the node is not
+answering requests. Three things follow from having measured that rather than
+assumed it.
+
+**The pixel maths goes to a worker.** `src/pixels.js` and `src/pixel-worker.js`
+run the same functions, unchanged, on another thread. Measured against a request
+arriving every 5 ms while a bake runs, the delay that request sees at the 99th
+percentile:
+
+| workload                  | on the main thread | in a worker |
+| ------------------------- | ------------------ | ----------- |
+| 2 sources, 512px          | 10.9 ms            | 6.5 ms      |
+| 4 sources, 512px          | 18.6 ms            | 10.6 ms     |
+| 8 sources, 512px, blurred | 22.3 ms            | 17.1 ms     |
+
+The bake itself is 2–17% slower for it, which is the trade. Rasters are handed
+to the worker rather than copied — a decoded tile is most of a megabyte per
+source, and copying each one is work on the very thread this exists to keep
+free. The cost of that is the caller gives them up, which is safe because
+nothing reads a contribution after its merge, and is asserted rather than left
+as a comment.
+
+Serving does not use it. One tile's merge is a few milliseconds nobody notices,
+and a thread per request would cost more than it saved.
+
+**The checkpoint appends instead of rewriting.** The first version wrote entries
+through `serializeDirectory`, which was elegant reuse and the wrong tool: that
+is a _distribution_ format, and producing it costs a varint pass over every
+entry — 264 ms at a million entries, of which only 12 ms is the compression.
+Re-encoding all of them every checkpoint also made the total work quadratic in
+the length of the job.
+
+A checkpoint is read once, by this process, on the machine that wrote it. Fixed
+24-byte records cost nothing to produce and can be appended, and only the last
+one can change after it is written — a run of identical tiles extends it. So a
+checkpoint now costs the work since the last one:
+
+| entries   | rewriting everything | appending |
+| --------- | -------------------- | --------- |
+| 100,000   | 36 ms                | 11 ms     |
+| 1,000,000 | 191 ms               | 11 ms     |
+| 4,000,000 | 446 ms               | 11 ms     |
+
+**And there is a knob.** `stacks.bakePauseMs` is how long a bake waits between
+tiles. Zero is as fast as it can go, which is right for a node baking and doing
+nothing else. On a node that is also serving maps it is the direct trade: how
+long the bake takes against how much of the machine it takes while it runs.
+
 ### What identifies a bake
 
 `bakeRevision` is the recipe's revision and what each source resolved to,
@@ -676,6 +730,20 @@ worth writing at all.
 planet bake is hours, and a request that waited for the archive is a request
 nothing could hold open. `DELETE` on the same address stops it.
 
+The button opens a dialog rather than a confirmation, because there is
+something to decide. **Where it lands** uses the same picker every other
+destination in this console uses, so a location named once under Settings is
+offered here too — and a baked planet is hundreds of gigabytes, so which disk it
+goes on is not a detail. The location is resolved _before_ anything is merged:
+a name this node does not know, or a path it cannot write, is the caller's
+mistake and they can fix it, but only if they are told now rather than an hour
+later.
+
+**What it is called** defaults to the stack's title and can be changed, as can
+the description. The dialog shows the filename as the name is typed, because a
+name is not a filename — the same sanitising the server does, and a preview that
+promised something else would be worse than no preview.
+
 A bake has two halves and they are watched in two places, deliberately. Merging
 is about a stack, so it is reported on the stack — tiles written, tiles skipped,
 the zoom it is working through. What happens afterwards is an archive being

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.64.0",
+  "version": "0.66.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -3181,6 +3181,8 @@ export function createApp({
           categories: req.body?.categories,
           location: req.body?.location,
           savePath: req.body?.savePath,
+          name: req.body?.name,
+          description: req.body?.description,
         });
         return res.status(202).json({ bake: job });
       } catch (error) {

package/src/bake-jobs.js

@@ -6,6 +6,7 @@ import {
   bakedName,
   mergeTileFor,
 } from './bake.js';
+import { PixelWorker } from './pixels.js';
 import { outputFormat } from './stack-tile.js';
 
 /**
@@ -112,9 +113,21 @@ export class BakeManager {
       throw error;
     }
 
+    // Resolved before anything starts, and allowed to throw. A location that
+    // does not exist or cannot be written is the caller's mistake and they can
+    // fix it -- but only if they are told now. Checked at the end instead, it
+    // is an hour of merging answered with a shrug and the default disk.
+    const publishDir =
+      options.publishDir ?? (await this.#savePath(options)) ?? undefined;
+
+    // What the archive will be called, which the caller may have chosen. The
+    // file follows it; the date is what keeps successive builds apart.
+    const archiveName = options.name?.trim() || resolved.stack.title || stackId;
+
     const job = {
       stackId,
       title: resolved.stack.title ?? stackId,
+      archiveName,
       phase: 'merging',
       startedAt: new Date().toISOString(),
       written: 0,
@@ -125,7 +138,8 @@ export class BakeManager {
       finishedAt: null,
       error: null,
       infoHash: null,
-      name: bakedName(resolved, { when: new Date() }),
+      name: bakedName(resolved, { when: new Date(), name: archiveName }),
+      publishDir,
     };
     this.#jobs.set(stackId, job);
 
@@ -158,6 +172,63 @@ export class BakeManager {
     const destination = path.join(workDir, job.name);
     const format = outputFormat(resolved);
 
+    // Only where there is pixel work to move. A passthrough bake hands bytes
+    // straight through and would pay for a thread it never uses.
+    const pixels = codec ? new PixelWorker() : null;
+    try {
+      await this.#merge(
+        job,
+        resolved,
+        codec,
+        pixels,
+        workDir,
+        destination,
+        format,
+        options,
+      );
+    } finally {
+      await pixels?.close().catch(() => {});
+    }
+
+    // The second half. `addLocalArchive` registers the add in the library's own
+    // in-progress list, which is what the archives view already draws -- so the
+    // hashing shows up there without this having to report it twice.
+    job.phase = 'importing';
+    const entry = await this.#library.addLocalArchive(destination, {
+      categories: options.categories ?? resolved.stack.categories,
+      // Moved out of the working directory as it is taken on, so a finished
+      // archive does not live among the checkpoint files of the job that made
+      // it.
+      publishDir: job.publishDir,
+      mode: 'mirror',
+    });
+
+    job.infoHash = entry.infoHash;
+    job.phase = 'done';
+    job.finishedAt = new Date().toISOString();
+  }
+
+  /**
+   * The merging half, which is where the time goes.
+   * @param {object} job - The job to update as it goes.
+   * @param {object} resolved - The resolved stack.
+   * @param {object|null} codec - The codec, where the recipe needs one.
+   * @param {object|null} pixels - Somewhere to do the pixel maths.
+   * @param {string} workDir - Where the unfinished work lives.
+   * @param {string} destination - Where the archive goes.
+   * @param {string} format - The output format.
+   * @returns {Promise<void>} - Resolves when the file is written.
+   */
+  async #merge(
+    job,
+    resolved,
+    codec,
+    pixels,
+    workDir,
+    destination,
+    format,
+    options,
+  ) {
     // Read through the tile store rather than off the disk, so a cache-mode
     // source is scanned the same way it is served: its directories come out of
     // the swarm, and the store holds them to its own byte budget.
@@ -194,13 +265,15 @@ export class BakeManager {
         resolved,
         tiles: this.#tiles,
         codec,
+        pixels,
         signal: job.controller.signal,
         format,
       }),
       header: { format },
+      pauseMs: this.#config.stacks?.bakePauseMs ?? 0,
       metadata: {
-        name: resolved.stack.title ?? job.stackId,
-        description: resolved.stack.description,
+        name: job.archiveName,
+        description: options.description ?? resolved.stack.description,
         attribution: resolved.stack.attribution,
         encoding: resolved.stack.output?.encoding,
         encodingFactors: resolved.stack.output,
@@ -215,23 +288,6 @@ export class BakeManager {
     });
 
     job.tiles = result.written;
-
-    // The second half. `addLocalArchive` registers the add in the library's own
-    // in-progress list, which is what the archives view already draws -- so the
-    // hashing shows up there without this having to report it twice.
-    job.phase = 'importing';
-    const entry = await this.#library.addLocalArchive(destination, {
-      categories: options.categories ?? resolved.stack.categories,
-      // Moved out of the working directory as it is taken on, so a finished
-      // archive does not live among the checkpoint files of the job that made
-      // it.
-      publishDir: options.publishDir ?? (await this.#savePath(options)),
-      mode: 'mirror',
-    });
-
-    job.infoHash = entry.infoHash;
-    job.phase = 'done';
-    job.finishedAt = new Date().toISOString();
   }
 
   /**
@@ -240,9 +296,10 @@ export class BakeManager {
    * @returns {Promise<string|undefined>} - A path, or undefined for the default.
    */
   async #savePath(options) {
-    const explicit = await this.#library
-      .resolveSavePath(options)
-      .catch(() => undefined);
+    // Not caught. `resolveSavePath` refuses a name this node does not know and
+    // a path it cannot write, and both are worth refusing rather than quietly
+    // putting hundreds of gigabytes somewhere else.
+    const explicit = await this.#library.resolveSavePath(options);
     return explicit ?? this.#config.savePath ?? undefined;
   }
 
@@ -262,6 +319,8 @@ export class BakeManager {
       skipped: job.skipped,
       zoom: job.zoom,
       name: job.name,
+      archiveName: job.archiveName,
+      publishDir: job.publishDir ?? null,
       infoHash: job.infoHash,
       error: job.error,
     };

package/src/bake.js

@@ -1,18 +1,12 @@
 import fs from 'node:fs/promises';
 import path from 'node:path';
-import zlib from 'node:zlib';
 import { tileIdToZxy } from 'pmtiles';
-import { deserializeDirectory, unionOfTileIds } from './pmtiles-scan.js';
+import { unionOfTileIds } from './pmtiles-scan.js';
 import crypto from 'node:crypto';
 import { safeSegment } from './savepath.js';
 import { answerStackTile, outputFormat, outputSize } from './stack-tile.js';
 import { needsCodec, stackRevision } from './stacks.js';
-import {
-  Compression,
-  PMTilesWriter,
-  TileType,
-  serializeDirectory,
-} from './pmtiles-write.js';
+import { Compression, PMTilesWriter, TileType } from './pmtiles-write.js';
 
 /**
  * Running a stack over its sources' coverage and writing a real archive.
@@ -51,6 +45,62 @@ export function checkpointPaths(workDir) {
   };
 }
 
+/**
+ * One entry, as a checkpoint stores it.
+ *
+ * A fixed record rather than `serializeDirectory`, which was the first thing
+ * tried here and is the wrong tool. That is a *distribution* format: compact on
+ * disk, and it costs a varint pass over every entry to produce -- 264 ms at a
+ * million entries, of which only 12 ms is the compression. Re-encoding all of
+ * them every checkpoint also makes the total work quadratic in the length of
+ * the job.
+ *
+ * A checkpoint wants none of that. It is read once, by this process, on a
+ * machine that just wrote it. Fixed-width records cost nothing to produce and
+ * can be appended, which is what turns the checkpoint from a stall that grows
+ * into one that does not.
+ */
+const RECORD_BYTES = 24;
+
+/**
+ * Packs entries into fixed records.
+ * @param {object[]} entries - The entries to pack.
+ * @returns {Buffer} - `RECORD_BYTES` per entry.
+ */
+function packEntries(entries) {
+  const buffer = Buffer.alloc(entries.length * RECORD_BYTES);
+  for (const [index, entry] of entries.entries()) {
+    const at = index * RECORD_BYTES;
+    // Doubles for the two that can be large: a tile id past z26 and an offset
+    // past 4 GiB both exceed what 32 bits hold, and both are safe integers.
+    buffer.writeDoubleLE(entry.tileId, at);
+    buffer.writeDoubleLE(entry.offset, at + 8);
+    buffer.writeUInt32LE(entry.length, at + 16);
+    buffer.writeUInt32LE(entry.runLength, at + 20);
+  }
+  return buffer;
+}
+
+/**
+ * Reads packed records back into entries.
+ * @param {Buffer} buffer - What `packEntries` wrote.
+ * @returns {object[]} - The entries.
+ */
+function unpackEntries(buffer) {
+  const count = Math.floor(buffer.length / RECORD_BYTES);
+  const entries = new Array(count);
+  for (let index = 0; index < count; index += 1) {
+    const at = index * RECORD_BYTES;
+    entries[index] = {
+      tileId: buffer.readDoubleLE(at),
+      offset: buffer.readDoubleLE(at + 8),
+      length: buffer.readUInt32LE(at + 16),
+      runLength: buffer.readUInt32LE(at + 20),
+    };
+  }
+  return entries;
+}
+
 /**
  * Reads a checkpoint, if there is one worth resuming from.
  *
@@ -83,10 +133,10 @@ export async function readCheckpoint(workDir, revision) {
   // shorter than the entries say, some of what they point at is not there.
   if (buffered.size < state.dataBytes) return null;
 
-  let entries;
-  try {
-    entries = deserializeDirectory(zlib.gunzipSync(stored));
-  } catch {
+  const entries = unpackEntries(stored);
+  // A record file shorter than the state claims means the two disagree about
+  // what was written, and the entries are the half that can be believed.
+  if (state.entryCount !== undefined && entries.length !== state.entryCount) {
     return null;
   }
 
@@ -94,25 +144,41 @@ export async function readCheckpoint(workDir, revision) {
 }
 
 /**
- * Writes a checkpoint, replacing whatever was there.
+ * Writes a checkpoint, appending what is new rather than rewriting it all.
  *
- * Entries go through `serializeDirectory`, which is the compact form this
- * project already writes and reads. Nothing here invents a second encoding for
- * the same array.
+ * Only the last entry can change once written -- a run of identical tiles
+ * extends it -- so everything before that is already on disk and correct. The
+ * write is therefore the size of the work since the last checkpoint, not the
+ * size of the job so far.
  * @param {string} workDir - The working directory.
- * @param {object} state - Scalars worth keeping.
+ * @param {object} state - Scalars worth keeping, including `entryCount`.
  * @param {object[]} entries - The entries as they stand.
- * @returns {Promise<void>} - Resolves once both files are on disk.
+ * @param {number} [persisted] - How many are already on disk.
+ * @returns {Promise<number>} - How many are on disk now.
  */
-export async function writeCheckpoint(workDir, state, entries) {
+export async function writeCheckpoint(workDir, state, entries, persisted = 0) {
   const paths = checkpointPaths(workDir);
+
+  // Back up one, because the last record written may have grown a longer run
+  // since. Everything before it is settled.
+  const from = Math.max(0, Math.min(persisted, entries.length) - 1);
+  const handle = await fs.open(paths.entries, persisted > 0 ? 'r+' : 'w');
+  try {
+    const tail = packEntries(entries.slice(from));
+    if (tail.length > 0)
+      await handle.write(tail, 0, tail.length, from * RECORD_BYTES);
+    await handle.truncate(entries.length * RECORD_BYTES);
+  } finally {
+    await handle.close();
+  }
+
   // Entries first. A state file naming more progress than the entries hold
   // would resume into an archive missing tiles it believes it wrote.
   await fs.writeFile(
-    paths.entries,
-    entries.length > 0 ? serializeDirectory(entries) : Buffer.alloc(0),
+    paths.state,
+    JSON.stringify({ ...state, entryCount: entries.length }),
   );
-  await fs.writeFile(paths.state, JSON.stringify(state));
+  return entries.length;
 }
 
 /**
@@ -166,12 +232,16 @@ function stamp(when = new Date()) {
  * map and two files cannot share a path. The archive's *name* is not dated --
  * see `bakedMetadata` -- so a rebuild keeps its identity the way every other
  * rebuild here does.
+ *
+ * Follows the name it is given, so renaming an export in the dialog renames the
+ * file too. A name and a filename that disagree is a small thing that costs an
+ * afternoon the first time somebody goes looking on disk.
  * @param {object} resolved - The resolved stack.
- * @param {object} [options] - `when`, and `suffix` to break a tie.
+ * @param {object} [options] - `name` to override the stack's, `when`, `suffix`.
  * @returns {string} - A filename.
  */
 export function bakedName(resolved, options = {}) {
-  const title = resolved.stack.title ?? resolved.stack.id;
+  const title = options.name ?? resolved.stack.title ?? resolved.stack.id;
   const slug = safeSegment(title) || 'stack';
   const tail = options.suffix ? `-${options.suffix}` : '';
   return `${slug}-${stamp(options.when)}${tail}.pmtiles`;
@@ -297,7 +367,7 @@ export function tileTypeFor(format) {
  * @returns {Function} - `(z, x, y) => Promise<Buffer|null>`.
  */
 export function mergeTileFor(options) {
-  const { resolved, tiles, codec, signal } = options;
+  const { resolved, tiles, codec, signal, pixels } = options;
   const format = options.format ?? outputFormat(resolved);
   const size = options.size ?? outputSize(resolved.stack);
 
@@ -313,6 +383,7 @@ export function mergeTileFor(options) {
       signal,
       size,
       format,
+      pixels,
     });
 
     // A required source that cannot be read stops the job. Baking around it
@@ -357,6 +428,7 @@ export async function bakeStack(options) {
     header = {},
     deduplicate = true,
     checkpointEvery = DEFAULT_CHECKPOINT_EVERY,
+    pauseMs = 0,
   } = options;
 
   if (!sources?.length) throw new Error('a bake needs at least one source');
@@ -386,13 +458,14 @@ export async function bakeStack(options) {
   let skipped = found?.skipped ?? 0;
   let lastTileId = found?.lastTileId ?? -1;
   let sinceCheckpoint = 0;
+  let persisted = found?.entries.length ?? 0;
 
   /**
    * Writes down where the job has got to.
    * @returns {Promise<void>} - Resolves once it is durable.
    */
   const checkpoint = async () => {
-    await writeCheckpoint(
+    persisted = await writeCheckpoint(
       workDir,
       {
         revision,
@@ -404,6 +477,7 @@ export async function bakeStack(options) {
         clustered: writer.clustered,
       },
       writer.entries,
+      persisted,
     );
     sinceCheckpoint = 0;
   };
@@ -429,6 +503,16 @@ export async function bakeStack(options) {
       sinceCheckpoint += 1;
       onProgress?.({ written, skipped, tileId, z, x, y });
       if (sinceCheckpoint >= checkpointEvery) await checkpoint();
+
+      // Handing time back, where the operator asked for that. A bake on a node
+      // that is also serving tiles holds the main thread for a few milliseconds
+      // per tile -- the pixel maths is synchronous -- and this is the knob that
+      // trades how long the bake takes for how much of the machine it takes
+      // while it runs. Off by default, because a node baking nothing else pays
+      // for this and gets nothing.
+      if (pauseMs > 0) {
+        await new Promise((resolve) => setTimeout(resolve, pauseMs));
+      }
     }
   } catch (error) {
     // A cancelled bake keeps its work. Deleting it would make stopping and

package/src/config.js

@@ -364,6 +364,17 @@ const DEFAULTS = {
      * be thrown away, and a backup should be able to tell the difference.
      */
     cacheDir: undefined,
+    /**
+     * Milliseconds a bake waits between tiles. Zero is as fast as it can go.
+     *
+     * The pixel maths a merge does is synchronous, so a bake holds the main
+     * thread for a few milliseconds per tile -- and on a node that is also
+     * serving tiles, that is every request waiting behind it. This is the knob
+     * that trades how long the bake takes for how much of the machine it takes
+     * while it runs. Off by default: a node baking nothing pays for this and
+     * gets nothing.
+     */
+    bakePauseMs: 0,
   },
   /**
    * Folders scanned for new archives. Each entry is `{ path, categories,

package/src/pixel-worker.js

@@ -0,0 +1,70 @@
+import { parentPort } from 'node:worker_threads';
+import { encodeHeights, fillNodata, mergeElevation } from './elevation.js';
+import { compositeRgba, toRaster } from './rgba.js';
+
+/**
+ * The pixel half of a merge, off the main thread.
+ *
+ * `elevation.js` and `rgba.js` are entirely synchronous — decoding heights,
+ * masking, resampling and painting are all loops over typed arrays — so on the
+ * main thread a merge blocks everything else for as long as it takes. Serving
+ * one tile that way is a few milliseconds nobody notices. A bake is that, over
+ * and over, for hours, on a node that is also answering requests.
+ *
+ * This runs the same functions, unchanged, in a worker. Nothing here decides
+ * anything; the caller has already worked out what to merge and how.
+ */
+
+/**
+ * Merges one tile's contributions and encodes the result.
+ * @param {object} job - `{space, contributions, options, output}`.
+ * @returns {object|null} - A raster, or null where nothing covered the tile.
+ */
+function run(job) {
+  const { space, contributions, options, output = {} } = job;
+
+  if (space === 'rgba') {
+    const composited = compositeRgba(contributions, options);
+    if (!composited) return null;
+    // Alpha is kept unless the recipe asks for a flat tile.
+    return toRaster(composited, output.alpha !== false);
+  }
+
+  const merged = mergeElevation(contributions, options);
+  if (!merged) return null;
+  fillNodata(merged, output.nodata);
+  return encodeHeights(merged, {
+    ...output,
+    width: options.size,
+    height: options.size,
+    encoding: job.encoding,
+  });
+}
+
+parentPort?.on('message', (message) => {
+  try {
+    const raster = run(message.job);
+    if (!raster) {
+      parentPort.postMessage({ id: message.id, raster: null });
+      return;
+    }
+    // Copied into a buffer of its own and transferred, rather than handed over
+    // as it stands. A Buffer can be a view into a shared pool, and transferring
+    // that pool would take memory this worker is still using along with it.
+    const bytes = Uint8Array.prototype.slice.call(raster.data);
+    parentPort.postMessage(
+      {
+        id: message.id,
+        raster: {
+          data: bytes,
+          width: raster.width,
+          height: raster.height,
+          channels: raster.channels,
+        },
+      },
+      [bytes.buffer],
+    );
+  } catch (error) {
+    parentPort.postMessage({ id: message.id, error: error.message });
+  }
+});

package/src/pixels.js

@@ -0,0 +1,129 @@
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { Worker } from 'node:worker_threads';
+
+/**
+ * Somewhere other than the main thread to do a merge's pixel maths.
+ *
+ * Opt-in and handed to `answerStackTile`, rather than switched on for
+ * everything. Serving one tile does a few milliseconds of synchronous work and
+ * nobody notices; a bake does that for hours beside a node that is also
+ * answering requests, and there it is the difference between a slow bake and a
+ * slow node. See docs/tile-stacks.md — "Running it".
+ */
+
+const here = path.dirname(fileURLToPath(import.meta.url));
+
+/** How long to wait for a worker to answer before giving up on it. */
+const REPLY_TIMEOUT_MS = 120000;
+
+export class PixelWorker {
+  #worker;
+  #pending = new Map();
+  #next = 1;
+  #closed = false;
+
+  /**
+   * Starts a worker.
+   * @param {object} [options] - `timeoutMs` for a reply.
+   */
+  constructor(options = {}) {
+    this.#worker = new Worker(path.join(here, 'pixel-worker.js'));
+    this.timeoutMs = options.timeoutMs ?? REPLY_TIMEOUT_MS;
+    // The thread does not hold the process open. A bake keeps it busy; nothing
+    // else should keep it alive.
+    this.#worker.unref();
+
+    this.#worker.on('message', (message) => {
+      const waiting = this.#pending.get(message.id);
+      if (!waiting) return;
+      this.#pending.delete(message.id);
+      clearTimeout(waiting.timer);
+      if (message.error) waiting.reject(new Error(message.error));
+      else waiting.resolve(message.raster);
+    });
+
+    // A worker that dies takes every request in flight with it, and saying so
+    // is better than leaving them pending for ever.
+    const fail = (error) => {
+      this.#closed = true;
+      for (const waiting of this.#pending.values()) {
+        clearTimeout(waiting.timer);
+        waiting.reject(error);
+      }
+      this.#pending.clear();
+    };
+    this.#worker.on('error', fail);
+    this.#worker.on('exit', (code) => {
+      if (!this.#closed)
+        fail(new Error(`the pixel worker exited with ${code}`));
+    });
+  }
+
+  /**
+   * Merges one tile's contributions, off this thread.
+   * @param {object} job - `{space, contributions, options, output, encoding}`.
+   * @returns {Promise<object|null>} - A raster, or null where nothing covered it.
+   */
+  async merge(job) {
+    // Async so that everything below rejects rather than throws. A caller
+    // reaching for `.catch` on a job it handed over should not be handed an
+    // exception instead, and a malformed raster is otherwise thrown from the
+    // copy below before there is a promise to reject.
+    if (this.#closed) throw new Error('the pixel worker is closed');
+
+    const id = this.#next;
+    this.#next += 1;
+
+    // Rasters are handed over rather than cloned, and where possible without
+    // being copied first either. A decoded tile is most of a megabyte per
+    // source, and copying every one of them is work on the very thread this
+    // exists to keep free -- enough of it to cost more than it saves.
+    //
+    // Copied only when the buffer does not own its memory. Node pools
+    // allocations under 4 KiB, and transferring a pooled buffer would hand
+    // over the whole pool with everything else in it; a raster is far larger
+    // than that and so is always its own allocation, but the check is what
+    // makes that a fact rather than an assumption.
+    //
+    // Either way the caller gives up these rasters. Nothing reads them after a
+    // merge, which is what makes that safe.
+    const transfers = [];
+    const owned = (bytes) =>
+      bytes.byteOffset === 0 && bytes.buffer.byteLength === bytes.byteLength;
+    const contributions = job.contributions.map((contribution) => {
+      if (!contribution?.raster?.data) return contribution;
+      const source = contribution.raster.data;
+      const bytes = owned(source)
+        ? source
+        : Uint8Array.prototype.slice.call(source);
+      transfers.push(bytes.buffer);
+      return {
+        ...contribution,
+        raster: { ...contribution.raster, data: bytes },
+      };
+    });
+
+    return new Promise((resolve, reject) => {
+      const timer = setTimeout(() => {
+        this.#pending.delete(id);
+        reject(new Error('the pixel worker did not answer'));
+      }, this.timeoutMs);
+      timer.unref?.();
+      this.#pending.set(id, { resolve, reject, timer });
+      this.#worker.postMessage(
+        { id, job: { ...job, contributions } },
+        transfers,
+      );
+    });
+  }
+
+  /**
+   * Stops the worker.
+   * @returns {Promise<void>} - Resolves once it is gone.
+   */
+  async close() {
+    this.#closed = true;
+    await this.#worker.terminate();
+  }
+}

package/src/stack-tile.js

@@ -294,6 +294,7 @@ async function merge({
   rgba,
   format,
   gathered,
+  pixels,
 }) {
   const { contributors, contributions } = gathered;
   const first = contributions.find(Boolean);
@@ -306,6 +307,34 @@ async function merge({
     Math.max(...contributions.filter(Boolean).map((c) => c.raster.width));
   const output = resolved.stack.output ?? {};
 
+  const merging = {
+    z,
+    x,
+    y,
+    size: grid,
+    resampling: resolved.stack.resampling,
+    gaussianBlurSigma: resolved.stack.gaussianBlurSigma,
+  };
+
+  // Off this thread, where somebody has provided somewhere to put it. The same
+  // functions either way -- a worker runs `elevation.js` and `rgba.js`
+  // unchanged -- so this is about where the work happens and not what it is.
+  if (pixels) {
+    const off = await pixels.merge({
+      space: rgba ? 'rgba' : 'elevation',
+      contributions,
+      options: merging,
+      output,
+      encoding: output.encoding ?? first.source?.encoding,
+    });
+    if (!off) return { contributors, format, empty: true };
+    const body = await codec.encode(off, {
+      format,
+      lossless: rgba ? output.lossless === true : true,
+    });
+    return { contributors, format, body };
+  }
+
   // The two spaces differ only here. Everything around this -- reading, the
   // parent fallback, the cache, the headers -- is the same either way, which is
   // why they are one path rather than two.
@@ -370,7 +399,8 @@ async function merge({
  * @returns {Promise<StackAnswer>} - What to serve, or why there is nothing.
  */
 export async function answerStackTile(options) {
-  const { resolved, z, x, y, tiles, codec, stackCache, signal, size } = options;
+  const { resolved, z, x, y, tiles, codec, stackCache, signal, size, pixels } =
+    options;
   const format = options.format ?? outputFormat(resolved);
   const rgba = resolved.stack.space === 'rgba';
 
@@ -426,6 +456,7 @@ export async function answerStackTile(options) {
       rgba,
       format,
       gathered,
+      pixels,
     });
 
     // Awaited, though it is tempting not to be. The write is a local disk write

package/src/web/index.html

@@ -644,6 +644,59 @@
       </form>
     </dialog>
 
+    <dialog id="bake-dialog">
+      <form method="dialog" id="bake-form">
+        <h2>Export to archive</h2>
+        <div class="sub" style="margin-bottom:0.8rem">
+          Runs the recipe over its sources and writes the result as a real
+          archive, with its own infohash, seeded like any other. It reads every
+          tile the sources hold, so it can take hours — and it can be stopped,
+          which keeps the work for the next run to carry on from.
+        </div>
+
+        <div class="field">
+          <label>Stack</label>
+          <div class="sub"><code id="bake-stack"></code></div>
+        </div>
+
+        <div class="field">
+          <label for="bake-name">Name</label>
+          <input id="bake-name" placeholder="Terrain" />
+          <div class="sub">
+            What the archive is called, and what a rebuild of it keeps — so
+            leave the date out of it. The file gets one:
+            <code id="bake-filename"></code>
+          </div>
+        </div>
+
+        <div class="field">
+          <label for="bake-description">Description</label>
+          <input id="bake-description" placeholder="Bathymetry under terrain" />
+          <div class="sub">
+            Optional. The date this was baked is added to whatever is here.
+          </div>
+        </div>
+
+        <div id="bake-location"></div>
+
+        <div class="field">
+          <label for="bake-categories">Categories</label>
+          <input id="bake-categories" placeholder="terrain, weekly" />
+          <div class="sub">
+            Comma separated. What the archive is filed under once it is added,
+            which is what <code>/latest/&lt;category&gt;/</code> follows. Left
+            empty, it takes the stack's own.
+          </div>
+        </div>
+
+        <div class="sub" id="bake-error" style="color: var(--danger)"></div>
+        <menu>
+          <button value="cancel">Cancel</button>
+          <button id="bake-go" value="default">Export</button>
+        </menu>
+      </form>
+    </dialog>
+
     <dialog id="adopt-dialog">
       <form method="dialog" id="adopt-form">
         <h2>Adopt existing torrents</h2>
@@ -5149,6 +5202,18 @@ Every piece is hashed against the ` +
                 'and caching them would put a second copy of the archive ' +
                 'beside the first. Evicted least-recently-used.',
             },
+            {
+              key: 'stacks.bakePauseMs',
+              label: 'Pause between tiles while exporting',
+              type: 'number',
+              placeholder: '0',
+              unit: 'ms',
+              help:
+                'Zero is as fast as it can go. A merge does its pixel maths ' +
+                'on the main thread, so an export holds it for a few ' +
+                'milliseconds per tile and every request waits behind that. ' +
+                'Set this on a node that is also serving maps.',
+            },
             {
               key: 'stacks.cacheDir',
               label: 'Where they go',
@@ -7666,6 +7731,87 @@ Every piece is hashed against the ` +
       $('stacks-add').onclick = () =>
         openStackDialog().catch((e) => toast(e.message));
 
+      /**
+       * Asks where an export should land before starting it.
+       *
+       * A dialog rather than a confirm, because there is something to decide:
+       * a baked planet is hundreds of gigabytes and the disk it goes on is not
+       * a detail. The same picker every other destination in this console
+       * uses, so a location named once under Settings is offered here too.
+       * @param {string} id - The stack to export.
+       * @returns {Promise<void>} - Resolves once it is asked for or dismissed.
+       */
+      async function openBakeDialog(id, stack) {
+        $('bake-stack').textContent = id;
+        $('bake-error').textContent = '';
+        $('bake-name').value = stack?.title ?? id;
+        $('bake-description').value = stack?.description ?? '';
+        $('bake-categories').value = (stack?.categories ?? []).join(', ');
+        showBakeFilename();
+        $('bake-location').innerHTML = locationPicker('bake-loc');
+        await fillLocations('bake-loc');
+        $('bake-dialog').showModal();
+      }
+
+      /**
+       * Shows what the file will be called, as the name is typed.
+       *
+       * The same rules the server applies, because a name is not a filename:
+       * separators and the characters Windows refuses become dashes, and the
+       * date is appended so successive builds do not collide. Seeing that
+       * before pressing the button beats finding it on disk afterwards.
+       * @returns {void}
+       */
+      function showBakeFilename() {
+        const typed = $('bake-name').value.trim() || $('bake-stack').textContent;
+        const slug =
+          typed
+            .replace(/[/\\<>:"|?*\u0000-\u001f\u007f -]/g, '-')
+            .replace(/-{2,}/g, '-')
+            .replace(/^[.\s-]+/, '')
+            .replace(/[.\s-]+$/, '')
+            .slice(0, 120) || 'stack';
+        const day = new Date().toISOString().slice(0, 10).replaceAll('-', '');
+        $('bake-filename').textContent = `${slug}-${day}.pmtiles`;
+      }
+
+      $('bake-name').addEventListener('input', showBakeFilename);
+
+      $('bake-form').addEventListener('submit', async (event) => {
+        if (event.submitter?.value === 'cancel') return;
+        event.preventDefault();
+
+        const id = $('bake-stack').textContent;
+        const categories = $('bake-categories')
+          .value.split(',')
+          .map((one) => one.trim())
+          .filter(Boolean);
+        const name = $('bake-name').value.trim();
+        const description = $('bake-description').value.trim();
+
+        try {
+          await api(`/api/stacks/${encodeURIComponent(id)}/bake`, {
+            method: 'POST',
+            body: {
+              ...chosenLocation('bake-loc'),
+              ...(categories.length > 0 ? { categories } : {}),
+              ...(name ? { name } : {}),
+              ...(description ? { description } : {}),
+            },
+          });
+        } catch (error) {
+          // Kept open, because the answer is usually something to change here
+          // -- a location that does not exist, or a codec that is not
+          // installed -- rather than something to go away and think about.
+          $('bake-error').textContent = error.message;
+          return;
+        }
+
+        $('bake-dialog').close();
+        toast(`Exporting ${id}`);
+        loadStacks().catch((e) => toast(e.message));
+      });
+
       // Editing and deleting hang off the list, which is redrawn on every
       // change, so they are delegated the same way the source rows are.
       $('stacks-list').addEventListener('click', async (event) => {
@@ -7674,27 +7820,11 @@ Every piece is hashed against the ` +
         const bake = event.target.dataset.stackBake;
         const stopBake = event.target.dataset.stackBakeStop;
         if (bake !== undefined) {
-          // Confirmed, because it is not a small thing: it reads every tile
-          // every source holds, writes a whole archive, and then hashes it.
-          if (
-            !confirm(
-              `Run "${bake}" over its sources and write the result as a new ` +
-                'archive?\n\nThis reads every tile the sources hold and can ' +
-                'take hours. It can be stopped, and stopping keeps the work.',
-            )
-          ) {
-            return;
-          }
-          try {
-            await api(`/api/stacks/${encodeURIComponent(bake)}/bake`, {
-              method: 'POST',
-              body: {},
-            });
-            toast(`Exporting ${bake}`);
-          } catch (error) {
-            toast(error.message);
-          }
-          loadStacks().catch((e) => toast(e.message));
+          const { stacks } = await api('/api/stacks');
+          await openBakeDialog(
+            bake,
+            stacks.find((one) => one.id === bake),
+          ).catch((e) => toast(e.message));
         } else if (stopBake !== undefined) {
           try {
             await api(`/api/stacks/${encodeURIComponent(stopBake)}/bake`, {