pmtiles-swarm 0.74.0 → 0.76.0

package/CHANGELOG.md

@@ -7,8 +7,73 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
-## 0.74.0
+## 0.76.0
 ### ✨ Features and improvements
+- **A stopped export stays stopped.** Stopping one left a checkpoint, and a checkpoint says what was
+  in progress but not why it stopped - so the next restart could not tell somebody pressing Stop
+  from a crash, and picked it up again. Stopping now writes a marker beside the checkpoint, and a
+  restart reports what it is holding rather than resuming it. Starting or resuming clears the marker.
+
+  And the work can be thrown away, which it could not be before: `DELETE /api/stacks/:id/bake/work`
+  removes the working directory, which for an abandoned export is hundreds of gigabytes of buffered
+  tiles that previously had to be found by hand. Refused while a merge is running rather than pulled
+  out from under it.
+
+### 🐞 Bug fixes
+- **An edit inside one clock tick went unnoticed.** `stacks.json` is re-read when its modification
+  time changes, and a filesystem's clock is coarser than an edit: on NTFS the tick is about 15 ms,
+  measured here at 36 rapid rewrites in 40 landing on the same timestamp. Two edits that close
+  together left the mtime alone and the second was never read.
+
+  The size is compared as well now, which catches the ones that changed the file's length - most of
+  them - and costs nothing, since the stat was already being made.
+
+  This is also what made the reload test flake, roughly one run in three: it wrote the file twice in
+  quick succession and then depended on how those two writes fell against the clock. It stamps both
+  writes to the same instant now and asserts they really are the same, so it tests whether a change
+  is noticed rather than whether the clock happened to tick.
+- **A stack with a shallow global source served holes above z14.** How far the merge would climb
+  for a source with no tile at this zoom was a fixed six levels. GEBCO is z0-8 and the sea floor has
+  no more detail to give, so a stack serving z16 has to upscale that z8 tile eight levels - and at
+  z15 the climb stopped one short of the only tile that existed. Over open water, where the other
+  source was sparse and had nothing either, no source contributed at all and the stack correctly
+  answered no-tile. A rectangular hole, one tile wide, in the middle of the sea.
+
+  It is derived now rather than fixed: the deepest zoom the stack serves, less the shallowest source
+  under it. Nothing to set and nothing to get wrong - the right answer is computable, and a smaller
+  one would only punch holes. Somebody who wants the merge to stop climbing says so with `maxzoom`,
+  which stops the stack serving that deep at all: the same wish, said where it also stops the work.
+  Never below the old six, so no stack reaches less far than it did.
+
+## 0.75.0
+### ✨ Features and improvements
+- **`maskRange`, because nodata is a band and not a number.** `maskValues` and `maskColors` both
+  compare exactly, and cubic resampling overshoots at every edge it crosses - so a sea authored as
+  `0` arrives scattered across `-0.9` to `0`. The exact matches were masked and everything between
+  them was left standing.
+
+  Measured on a real merge, a terrain source over GEBCO at z13. The recipe masked `#018696` and
+  `#0186a0`, which decode to -1.0 m and 0.0 m; the nine colours between them, -0.9 m to -0.1 m,
+  were never masked. 1,227 pixels of one tile stood clear of all eight neighbours, the worst 25.1 m
+  above the bathymetry underneath - a scattering of 25 m pillars over open water, which is what
+  stipples a hillshade.
+
+  ```json
+  { "archive": "planet", "maskRange": [-1, 0] }
+  ```
+
+  A band rather than a width either side of a value, because nodata is rarely symmetric about
+  anything: sea is everything up to zero and nothing above it, and a width reaching a metre down
+  reaches a metre up as well, into ground that is really there. It is also what a recipe naming two
+  colours was already reaching for - they are the ends of one. A list of bands is accepted for a
+  source with a sentinel as well as a range, and the edges are inclusive, compared in thousandths
+  so `[-0.2, 0]` includes the -0.2 a Float32Array stores as -0.20000000298.
+
+  Worth recording that no average shows the problem this solves. That merged tile measures 0.2 m of
+  roughness at the median and is smooth by every summary statistic, because half a per cent of
+  pixels never move the middle of a distribution. `tools/terrain-probe.mjs` reports the tail and
+  counts pixels standing clear of their neighbours, which is the shape it makes.
+
 - **The holes a mask leaves are feathered now, not just a cutline's edge.** `feather` faded a source
   in at its `cutline` or `bounds` and did nothing about `maskValues` or `maskColors` - which is
   where most of these recipes actually stop, and why the field looked like it did nothing.

package/docs/tile-stacks.md

@@ -241,6 +241,7 @@ it differs from the snake_case rio-rgbify-merge uses.
 | `encoding`             | `mapbox` or `terrarium`. Elevation space only.                                                             |
 | `baseVal` / `interval` | Mapbox decode offset and step. Default `-10000` and `0.1`.                                                 |
 | `maskValues`           | Decoded heights meaning "no data here". Elevation space only.                                              |
+| `maskRange`            | `[low, high]` in metres, or a list of them. Everything inside is nodata. Elevation space only.             |
 | `heightAdjustment`     | Metres, added **after** masking. Elevation space only.                                                     |
 | `feather`              | Pixels to fade in over at the edge of the source's shape. Needs a `cutline` or `bounds`. Max 64.           |
 | `opacity`              | `0`–`1`, scales the source alpha. RGBA space only.                                                         |
@@ -1014,6 +1015,74 @@ missing. Serving a source unclipped because its cutline could not be found would
 put back exactly the data somebody asked to remove, which is the one failure a
 clip must not have.
 
+## What a mask has to match
+
+`maskValues` and `maskColors` both compare exactly, and an archive that was
+resampled on its way to being built does not hold the number it was authored
+with. Cubic overshoots at every edge it crosses, so a sea authored as exactly
+`0` arrives as a field of `0` with `-0.4` and `0.1` scattered through it near
+the coast.
+
+Masking by colour makes this sharper rather than softer, because a colour is one
+exact number and the resampled ground either side of it is several others. A
+recipe masking `#018696` and `#0186a0` is masking **-1.0 m** and **0.0 m** —
+and leaving `#018697` through `#01869f`, which are -0.9 m through -0.1 m,
+entirely alone.
+
+Measured on a real merge — a planet DEM over GEBCO bathymetry, one tile at z13:
+
+|                                               |               |
+| --------------------------------------------- | ------------- |
+| planet source, pixels at exactly `0`          | 109,441       |
+| its lowest value                              | -0.4 m        |
+| merged tile, pixels at exactly `0`            | 13            |
+| merged tile, pixels clear of all 8 neighbours | 1,227 (0.47%) |
+| the worst of them                             | 25.1 m        |
+
+`maskValues: [0]` took the 109,441 zeroes and left the rest standing. With
+bathymetry underneath, each survivor became a spike as tall as the difference
+between the two sources — a scattering of 25 m pillars over open water, which
+is what stipples a hillshade.
+
+`maskRange` says the band outright:
+
+```json
+{ "archive": "planet", "maskRange": [-1, 0] }
+```
+
+A band rather than a width either side of a value, because nodata is rarely
+symmetric about anything. Sea is everything up to zero and nothing above it,
+and a width reaching a metre down reaches a metre up as well, into ground that
+is really there. It is also what the recipe above was already reaching for: its
+two colours are the ends of one.
+
+Several, where a source has a sentinel as well as a band:
+
+```json
+{
+  "archive": "planet",
+  "maskRange": [
+    [-1, 0],
+    [-10001, -9999]
+  ]
+}
+```
+
+The edges are inclusive and compared in thousandths, because a `Float32Array`
+holds -0.2 as -0.20000000298 and an edge that does not include the number
+written on it leaves a row of pixels behind.
+
+It is a trade rather than a free win: a band wide enough to catch the
+resampling's overshoot also eats genuine ground inside it. Reaching down to the
+lowest value the sea arrives at is usually the right price; reaching up above
+zero is not, which is the asymmetry a band can express and a width cannot.
+
+Worth knowing that neither a median nor a colour ramp shows this. The merged
+tile above measures 0.2 m of roughness at the median and looks smooth by every
+summary statistic — half a per cent of pixels never move the middle of a
+distribution. `tools/terrain-probe.mjs` reports the tail and counts pixels
+standing clear of their neighbours, which is the shape this makes.
+
 ## Feathering a seam
 
 _Built for a cutline and for `bounds`. A mask edge is not feathered yet, and

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.74.0",
+  "version": "0.76.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -3303,6 +3303,31 @@ export function createApp({
     }),
   );
 
+  /**
+   * Throws away what a stopped export had done.
+   *
+   * Separate from stopping, and deliberately a second decision. Stopping keeps
+   * the work because an export may be hours in and somebody may want it back;
+   * this is for when they do not, and until now the only way to be rid of
+   * hundreds of gigabytes of buffered tiles was to find the directory by hand.
+   */
+  app.delete(
+    '/api/stacks/:id/bake/work',
+    route(async (req, res) => {
+      if (!bakes) {
+        return res.status(501).json({ error: 'this node does not bake' });
+      }
+      const discarded = await bakes.discard(req.params.id);
+      if (!discarded) {
+        return res.status(409).json({
+          error:
+            'there is no stopped export for that stack, or one is still running',
+        });
+      }
+      return res.json({ discarded: true });
+    }),
+  );
+
   /**
    * Creates or replaces a stack.
    *

package/src/bake-jobs.js

@@ -7,7 +7,11 @@ import {
   bakeStack,
   bakedArchiveName,
   bakedName,
+  clearStopped,
+  discardCheckpoint,
+  markStopped,
   mergeTileFor,
+  wasStopped,
 } from './bake.js';
 import { PixelWorker } from './pixels.js';
 import { outputFormat } from './stack-tile.js';
@@ -124,6 +128,10 @@ export class BakeManager {
   #loadCodec;
   #cutlines;
   #jobs = new Map();
+  // Exports somebody stopped, whose work is still on disk. Kept so the console
+  // can offer to resume or discard them: after a restart there is no job in
+  // memory, only a directory nobody would think to look for.
+  #held = new Map();
 
   /**
    * @param {object} deps - The library, the tile store, the config and the codec probe.
@@ -152,11 +160,36 @@ export class BakeManager {
    */
   get(stackId) {
     const job = this.#jobs.get(stackId);
-    return job ? this.#describe(job) : null;
+    if (job) return this.#describe(job);
+
+    // Nothing running, but work somebody stopped may still be there. Reported
+    // in the same shape so the console has one thing to read.
+    const stopped = this.#held.get(stackId);
+    if (!stopped) return null;
+    return {
+      stackId,
+      phase: 'stopped',
+      written: stopped.written,
+      resumable: true,
+      ...stopped.describe,
+    };
+  }
+
+  /**
+   * Every stack with work waiting that nobody has decided about.
+   * @returns {string[]} - Their ids.
+   */
+  heldStacks() {
+    return [...this.#held.keys()];
   }
 
   /**
-   * Stops a bake, leaving its work where the next run can pick it up.
+   * Stops a bake, leaving its work for somebody to pick up or throw away.
+   *
+   * Marked as stopped on purpose, so the next start leaves it alone. The work
+   * is still there and `resume` takes it up again -- what changes is who
+   * decides, which for a job that may be hours from finishing should be a
+   * person rather than a restart.
    * @param {string} stackId - Which stack.
    * @returns {boolean} - True if there was one to stop.
    */
@@ -164,10 +197,46 @@ export class BakeManager {
     const job = this.#jobs.get(stackId);
     if (!job || job.finishedAt) return false;
     job.cancelling = true;
+    job.stoppedOnPurpose = true;
     job.controller.abort();
+    // Not awaited: the abort has to reach the merge now, and a mark written a
+    // moment later is still written long before anything restarts.
+    markStopped(workDirFor(job, this.#config)).catch(() => {});
     return true;
   }
 
+  /**
+   * Throws away what a stopped export had done.
+   *
+   * The counterpart to stopping. An export that will not be finished leaves
+   * hundreds of gigabytes of buffered tiles behind, and until now the only way
+   * to be rid of them was to find the directory by hand.
+   * @param {string} stackId - Which stack.
+   * @returns {Promise<boolean>} - True if there was work to discard.
+   */
+  async discard(stackId) {
+    const running = this.#jobs.get(stackId);
+    // Refused rather than raced. Removing the directory under a running merge
+    // would have it fail on its next write, reporting a disk problem for
+    // something somebody chose.
+    if (running && !running.finishedAt) return false;
+
+    let found = false;
+    for (const root of this.#workRoots()) {
+      const directory = path.join(root, WORK_DIR, stackId);
+      const there = await fs
+        .access(directory)
+        .then(() => true)
+        .catch(() => false);
+      if (!there) continue;
+      await discardCheckpoint(directory);
+      found = true;
+    }
+    this.#jobs.delete(stackId);
+    this.#held.delete(stackId);
+    return found;
+  }
+
   /**
    * Picks up exports a previous run did not finish.
    *
@@ -203,6 +272,21 @@ export class BakeManager {
         const resolved = resolve(stackId);
         if (!resolved || bakeRevision(resolved) !== state.revision) continue;
 
+        // Somebody stopped this one. It stays where it is until they say
+        // otherwise -- an export begun again by a restart is the opposite of
+        // what pressing Stop meant.
+        if (await wasStopped(path.join(directory, stackId))) {
+          this.#held.set(stackId, {
+            written: state.written ?? 0,
+            describe: state.describe,
+          });
+          console.log(
+            `[bake] ${stackId} was stopped on purpose; leaving its ` +
+              `${state.written ?? 0} tiles for you to resume or discard`,
+          );
+          continue;
+        }
+
         try {
           const job = await this.start({ resolved, ...state.describe });
           started.push(job);
@@ -367,6 +451,12 @@ export class BakeManager {
     const destination = path.join(workDir, job.name);
     const format = outputFormat(resolved);
 
+    // Running again is the answer to having been stopped, so the mark goes.
+    // Left behind, an export somebody restarted by hand would be passed over
+    // by the next restart, which is the same surprise the other way round.
+    await clearStopped(workDir);
+    this.#held.delete(job.stackId);
+
     // Sized with the batch, so every merge in flight has a thread to do its
     // arithmetic on rather than queueing behind one. Only where there is pixel
     // work to move: a passthrough bake hands bytes straight through and would

package/src/bake.js

@@ -42,6 +42,7 @@ const DEFAULT_CONCURRENCY = 4;
 /** What the working directory holds while a bake is in progress. */
 const FILES = Object.freeze({
   state: 'bake-state.json',
+  stopped: 'bake-stopped.json',
   entries: 'bake-entries.bin',
   tiles: 'bake-tiles.bin',
 });
@@ -59,6 +60,64 @@ export function checkpointPaths(workDir) {
   };
 }
 
+/**
+ * Records that an export was stopped on purpose.
+ *
+ * A checkpoint says what was in progress, not why it stopped, so a crash and
+ * somebody pressing Stop look identical on disk -- and the node picks both up
+ * on the next start. That is right for the crash and wrong for the person, who
+ * stopped it and then watched it begin again.
+ *
+ * Written beside the checkpoint rather than into it, so it cannot be lost to a
+ * half-finished write of the state the export is still making.
+ * @param {string} workDir - The bake's working directory.
+ * @returns {Promise<void>} - Resolves once it is written.
+ */
+export async function markStopped(workDir) {
+  await fs
+    .writeFile(
+      path.join(workDir, FILES.stopped),
+      JSON.stringify({ at: new Date().toISOString() }),
+    )
+    .catch(() => {});
+}
+
+/**
+ * Forgets that an export was stopped, because it is running again.
+ * @param {string} workDir - The bake's working directory.
+ * @returns {Promise<void>} - Resolves once it is gone.
+ */
+export async function clearStopped(workDir) {
+  await fs
+    .rm(path.join(workDir, FILES.stopped), { force: true })
+    .catch(() => {});
+}
+
+/**
+ * Whether an export was stopped on purpose rather than interrupted.
+ * @param {string} workDir - The bake's working directory.
+ * @returns {Promise<boolean>} - True when somebody stopped it.
+ */
+export async function wasStopped(workDir) {
+  return fs
+    .access(path.join(workDir, FILES.stopped))
+    .then(() => true)
+    .catch(() => false);
+}
+
+/**
+ * Forgets an export's unfinished work.
+ *
+ * The whole directory, because a checkpoint is only meaningful with the tiles
+ * it names -- leaving either behind is leaving something that will be picked
+ * up and found wanting.
+ * @param {string} workDir - The bake's working directory.
+ * @returns {Promise<void>} - Resolves once it is gone.
+ */
+export async function discardCheckpoint(workDir) {
+  await fs.rm(workDir, { recursive: true, force: true });
+}
+
 /**
  * One entry, as a checkpoint stores it.
  *

package/src/elevation.js

@@ -162,6 +162,77 @@ export function maskHeights(heights, maskValues) {
   return heights;
 }
 
+/**
+ * Reads `maskRange` into a list of low-high pairs.
+ *
+ * One pair or several, because both read naturally: `[-1, 0]` is the common
+ * case and `[[-1, 0], [-10001, -9999]]` is a source with a sentinel as well as
+ * a band. A pair the wrong way round is taken as the same band rather than
+ * refused -- the recipe validation says so, and a merge that has got this far
+ * should not silently mask nothing.
+ * @param {Array<number[]>|number[]|undefined} maskRange - What the recipe said.
+ * @returns {Array<number[]>} - Pairs, low first.
+ */
+export function rangesOf(maskRange) {
+  if (!Array.isArray(maskRange) || maskRange.length === 0) return [];
+  const pairs = Array.isArray(maskRange[0]) ? maskRange : [maskRange];
+  const out = [];
+  for (const pair of pairs) {
+    if (!Array.isArray(pair) || pair.length < 2) continue;
+    const low = Number(pair[0]);
+    const high = Number(pair[1]);
+    if (!Number.isFinite(low) || !Number.isFinite(high)) continue;
+    out.push(low <= high ? [low, high] : [high, low]);
+  }
+  return out;
+}
+
+/**
+ * Blanks every height inside a band.
+ *
+ * The shape nodata actually has. An archive resampled on its way to being
+ * built does not hold the number it was authored with -- cubic overshoots at
+ * every edge it crosses, so a sea authored as exactly 0 arrives as a field of
+ * 0 with -0.4 and 0.1 scattered through it near the coast. Masking the exact
+ * values takes some of them and leaves the rest standing, and where another
+ * source is underneath each survivor becomes a spike as tall as the difference
+ * between them: measured at 25 m on a real merge, on half a per cent of the
+ * tile, which is what stipples a hillshade.
+ *
+ * A band rather than a width either side of a value, because nodata is rarely
+ * symmetric about anything. Sea is everything from the deepest sentinel up to
+ * zero and nothing above it, and a width wide enough to reach the bottom of
+ * that reaches the same distance into real ground. See docs/tile-stacks.md --
+ * "What a mask has to match".
+ * @param {Float32Array} heights - Metres, modified in place.
+ * @param {Array<number[]>|number[]} [maskRange] - A `[low, high]` pair, or a list of them.
+ * @returns {Float32Array} - The same array.
+ */
+export function maskRanges(heights, maskRange) {
+  const bands = rangesOf(maskRange).map(([low, high]) => [
+    Math.round(low * 1000),
+    Math.round(high * 1000),
+  ]);
+  if (bands.length === 0) return heights;
+
+  // Compared in thousandths, the same way maskHeights compares. A Float32Array
+  // holds -0.2 as -0.20000000298, which falls outside a band written as
+  // [-0.2, 0] -- and an edge that does not include the number written on it is
+  // a mask that quietly leaves a row of pixels behind.
+  for (let i = 0; i < heights.length; i += 1) {
+    const height = heights[i];
+    if (Number.isNaN(height)) continue;
+    const at = Math.round(height * 1000);
+    for (const [low, high] of bands) {
+      if (at >= low && at <= high) {
+        heights[i] = Number.NaN;
+        break;
+      }
+    }
+  }
+  return heights;
+}
+
 /**
  * Reads one colour from a recipe into a packed 24-bit value.
  *
@@ -638,8 +709,12 @@ export function mergeElevation(contributions, options) {
     // Both masks say the same thing -- nothing here -- and both have to run
     // before the height adjustment, which would otherwise shift the values
     // out from under the comparison.
+    //
     maskHeights(heights, source.maskValues);
     maskColors(heights, contribution.raster, source.maskColors);
+    // A band as well as, not instead of: a source may have a sentinel it names
+    // exactly and a range of ground it does not want either.
+    maskRanges(heights, source.maskRange);
     if (source.heightAdjustment) {
       for (let i = 0; i < heights.length; i += 1) {
         heights[i] += source.heightAdjustment;

package/src/stack-tile.js

@@ -5,6 +5,7 @@ import {
   fillNodata,
   maskColors,
   maskHeights,
+  maskRanges,
   mergeElevation,
 } from './elevation.js';
 import { paddedKnown, parentsFor } from './mask-edge.js';
@@ -35,9 +36,42 @@ import { TileReadError } from './tiles.js';
  * a tile in an archive or a hole where one is not needed.
  */
 
-/** How far up the pyramid a merge will climb for a source with no tile here. */
+/**
+ * How far up the pyramid a merge will climb for a source with no tile here,
+ * where the stack does not say and nothing can be worked out.
+ */
 const PARENT_LIMIT = 6;
 
+/**
+ * How far this stack has to climb for its shallowest source to keep working.
+ *
+ * A global source is shallow on purpose -- GEBCO is z0-8 and the sea floor has
+ * no more detail to give -- so serving a stack to z16 means upscaling that z8
+ * tile eight levels. A fixed limit truncates exactly the arrangement this
+ * feature is for: at z15 the climb stopped one level short of the only tile
+ * that exists, no source contributed, and the stack answered no-tile over open
+ * water.
+ *
+ * So it is derived from what the stack spans rather than assumed or set: the
+ * right answer is computable, and a recipe naming a smaller one would only
+ * punch holes in itself. Somebody who wants the merge to stop climbing says so
+ * with `maxzoom`, which stops the stack serving that deep at all -- the same
+ * wish, said where it also stops the work.
+ *
+ * Never below the old fixed limit, so no stack reaches less far than it did.
+ * @param {object} resolved - The resolved stack.
+ * @returns {number} - Levels a source may climb.
+ */
+export function parentLimitFor(resolved) {
+  const { maxzoom } = stackCoverage(resolved);
+  const shallowest = resolved.sources
+    .map((source) => source.entry?.pmtiles?.maxZoom)
+    .filter((zoom) => Number.isFinite(zoom));
+  if (!shallowest.length || !Number.isFinite(maxzoom)) return PARENT_LIMIT;
+
+  return Math.max(PARENT_LIMIT, maxzoom - Math.min(...shallowest));
+}
+
 /** The deepest zoom a tile id is defined for. */
 const MAX_ZOOM = 26;
 
@@ -140,8 +174,8 @@ export function clipsFor(resolved, cutlines, z, x, y, size = 256) {
  * @param {object} options - Source, coordinates, the tile store and whether to climb.
  * @returns {Promise<object|null>} - The tile and the zoom it came from.
  */
-async function readFrom({ source, z, x, y, tiles, climb, signal }) {
-  const floor = climb ? Math.max(0, z - PARENT_LIMIT) : z;
+async function readFrom({ source, z, x, y, tiles, climb, signal, limit }) {
+  const floor = climb ? Math.max(0, z - (limit ?? PARENT_LIMIT)) : z;
   for (let at = z; at >= floor; at -= 1) {
     const shift = z - at;
     const tile = await tiles.getTile(
@@ -321,6 +355,7 @@ export function passThroughRead({
  * @returns {Promise<object>} - `{contributors, contributions}` or `{error}`.
  */
 async function readAll({ resolved, z, x, y, tiles, signal, clips }) {
+  const limit = parentLimitFor(resolved);
   return Promise.all(
     resolved.sources.map(async (source, index) => {
       if (!source.entry) return { source, found: null };
@@ -339,6 +374,7 @@ async function readAll({ resolved, z, x, y, tiles, signal, clips }) {
             tiles,
             climb: true,
             signal,
+            limit,
           }),
         };
       } catch (error) {
@@ -450,7 +486,11 @@ async function gather({ reads, z, x, y, tiles, codec, signal, size, rgba }) {
  * @returns {boolean} - True when it masks anything.
  */
 function masksAnything(recipe) {
-  return Boolean(recipe?.maskValues?.length || recipe?.maskColors?.length);
+  return Boolean(
+    recipe?.maskValues?.length ||
+    recipe?.maskColors?.length ||
+    recipe?.maskRange?.length,
+  );
 }
 
 /**
@@ -506,12 +546,13 @@ async function readMaskEdges({
             .catch(() => null);
           if (!raster) return;
 
-          // The same two masks the merge applies, asked of the parent. A ramp
+          // The same masking the merge applies, asked of the parent. A ramp
           // measured against a different idea of where the holes are would fade
           // toward ground that is not a hole.
           const heights = decodeHeights(raster, recipe);
           maskHeights(heights, recipe.maskValues);
           maskColors(heights, raster, recipe.maskColors);
+          maskRanges(heights, recipe.maskRange);
 
           const flags = new Uint8Array(heights.length);
           for (let i = 0; i < flags.length; i += 1) {

package/src/stacks.js

@@ -35,6 +35,8 @@ import { BLEND_MODES, isBlendMode } from './rgba.js';
  * @property {number} [baseVal] - Mapbox decode offset.
  * @property {number} [interval] - Mapbox decode step.
  * @property {number[]} [maskValues] - Heights meaning "no data here".
+ * @property {Array<number[]>|number[]} [maskRange] - A `[low, high]` band of
+ *   heights meaning "no data here", or a list of them.
  * @property {Array<string|number[]>} [maskColors] - Pixel colours meaning the
  *   same, as "#rrggbb" or [r, g, b]. Exact, where a height mask has to round.
  * @property {number} [heightAdjustment] - Metres, added after masking.
@@ -127,6 +129,28 @@ export function validateStack(stack) {
     if (source?.maskValues !== undefined && !Array.isArray(source.maskValues)) {
       problems.push(`sources[${index}].maskValues must be a list`);
     }
+    if (source?.maskRange !== undefined) {
+      // One pair or a list of them. Anything else is a range nobody can read,
+      // and a mask that silently matches nothing is the failure this whole
+      // feature is most prone to.
+      const given = Array.isArray(source.maskRange) ? source.maskRange : null;
+      const pairs =
+        given && given.length && Array.isArray(given[0]) ? given : [given];
+      const usable =
+        given &&
+        given.length > 0 &&
+        pairs.every(
+          (pair) =>
+            Array.isArray(pair) &&
+            pair.length === 2 &&
+            pair.every((edge) => Number.isFinite(Number(edge))),
+        );
+      if (!usable) {
+        problems.push(
+          `sources[${index}].maskRange must be [low, high], or a list of them`,
+        );
+      }
+    }
     if (source?.feather !== undefined) {
       const feather = Number(source.feather);
       if (!Number.isFinite(feather) || feather < 0 || feather > MAX_FEATHER) {
@@ -504,6 +528,7 @@ export class StackStore {
   #stacks = new Map();
   #problems = new Map();
   #mtime = null;
+  #size = null;
   #checkedAt = 0;
 
   /**
@@ -528,10 +553,13 @@ export class StackStore {
     let raw;
     try {
       raw = JSON.parse(await fs.readFile(this.#file, 'utf8'));
-      this.#mtime = (await fs.stat(this.#file)).mtimeMs;
+      const stat = await fs.stat(this.#file);
+      this.#mtime = stat.mtimeMs;
+      this.#size = stat.size;
     } catch (error) {
       if (error.code === 'ENOENT') {
         this.#mtime = null;
+        this.#size = null;
         return;
       }
       throw error;
@@ -564,9 +592,15 @@ export class StackStore {
     if (now - this.#checkedAt < 1000) return false;
     this.#checkedAt = now;
 
+    // Size as well as the timestamp. A filesystem's clock is coarser than an
+    // edit: on NTFS the tick is about 15 ms, so two writes in quick succession
+    // land on the same mtime and the second would never be seen. Comparing the
+    // length as well catches the ones that changed it, which is most of them --
+    // and it costs nothing, since the stat was made anyway.
     const stat = await fs.stat(this.#file).catch(() => null);
     const mtime = stat?.mtimeMs ?? null;
-    if (mtime === this.#mtime) return false;
+    const size = stat?.size ?? null;
+    if (mtime === this.#mtime && size === this.#size) return false;
     await this.load();
     return true;
   }
@@ -652,6 +686,8 @@ export class StackStore {
 `,
     );
     await fs.rename(temp, this.#file);
-    this.#mtime = (await fs.stat(this.#file)).mtimeMs;
+    const written = await fs.stat(this.#file);
+    this.#mtime = written.mtimeMs;
+    this.#size = written.size;
   }
 }

package/src/web/index.html

@@ -1904,6 +1904,24 @@
           ({ '&': '&amp;', '<': '&lt;', '>': '&gt;', '"': '&quot;', "'": '&#39;' })[c],
         );
 
+      /**
+       * One end of a mask range, for the box that shows it.
+       *
+       * A recipe may hold several bands where the editor offers one. The first
+       * is what the boxes show, and the row says how many more there are --
+       * silently dropping them would be worse than saying the editor cannot
+       * draw them.
+       * @param {*} range - What the recipe says.
+       * @param {number} which - 0 for the low end, 1 for the high.
+       * @returns {string} - What to put in the box.
+       */
+      const rangeEdge = (range, which) => {
+        if (!Array.isArray(range) || range.length === 0) return '';
+        const first = Array.isArray(range[0]) ? range[0] : range;
+        const edge = first[which];
+        return Number.isFinite(Number(edge)) ? String(edge) : '';
+      };
+
       /**
        * Redraws the open detail pane, when it is one that changes by itself.
        *
@@ -7618,6 +7636,25 @@ Every piece is hashed against the ` +
                <input style="width:12rem" placeholder="-10000, 0, -1"
                       data-stack-field="maskValues" data-stack-index="${index}"
                       value="${escapeHtml((source.maskValues ?? []).join(', '))}" />
+             </label>
+             <label class="choice"
+                    title="Mask every height between these two, inclusive. Separate from Mask heights, which takes values one at a time — nodata is often a band rather than a number, because an archive resampled on its way to being built does not hold what it was authored with. A sea authored as 0 arrives scattered across -0.9 m to 0, and masking the two ends of that leaves everything between, standing proud of whatever is underneath. Leave both empty for no band.">
+               Mask between
+               <input type="number" step="any" style="width:5.5rem" placeholder="low"
+                      data-stack-field="maskRangeLow" data-stack-index="${index}"
+                      value="${rangeEdge(source.maskRange, 0)}" />
+               and
+               <input type="number" step="any" style="width:5.5rem" placeholder="high"
+                      data-stack-field="maskRangeHigh" data-stack-index="${index}"
+                      value="${rangeEdge(source.maskRange, 1)}" /> m
+               ${
+                 Array.isArray(source.maskRange) &&
+                 Array.isArray(source.maskRange[0]) &&
+                 source.maskRange.length > 1
+                   ? `<span class="sub">and ${source.maskRange.length - 1} more,
+                      which editing here replaces</span>`
+                   : ''
+               }
              </label>`;
 
         return `
@@ -7739,6 +7776,22 @@ Every piece is hashed against the ` +
             .filter(Boolean);
         } else if (field === 'required') {
           source.required = event.target.checked;
+        } else if (field === 'maskRangeLow' || field === 'maskRangeHigh') {
+          // A band needs both ends, so one on its own is a half-typed thought
+          // rather than a range -- read them together, and write nothing until
+          // there are two numbers to write.
+          const row = event.target.closest('.card');
+          const edge = (name) => {
+            const box = row?.querySelector(`[data-stack-field="${name}"]`);
+            return box && box.value.trim() !== '' ? Number(box.value) : null;
+          };
+          const low = edge('maskRangeLow');
+          const high = edge('maskRangeHigh');
+          if (Number.isFinite(low) && Number.isFinite(high)) {
+            source.maskRange = low <= high ? [low, high] : [high, low];
+          } else if (low === null && high === null) {
+            delete source.maskRange;
+          }
         } else if (field === 'feather') {
           const pixels = Number(event.target.value);
           if (event.target.value === '' || !(pixels > 0)) delete source.feather;

package/tools/terrain-probe.mjs

@@ -0,0 +1,250 @@
+#!/usr/bin/env node
+/**
+ * What is actually in a terrain tile, in metres.
+ *
+ * A colour ramp and a hillshade disagree about what looks wrong, and both of
+ * them lie about scale: quantisation of a tenth of a metre dithers a ramp's
+ * band edge into a checkerboard, and a hillshade turns the same tenth of a
+ * metre into a field of bumps if the ground is flat enough. Neither tells you
+ * which you are looking at. This prints the numbers.
+ *
+ *   node tools/terrain-probe.mjs <tile-url> [--encoding mapbox] [--interval 0.1]
+ *
+ * where <tile-url> is one tile, for instance
+ *
+ *   http://127.0.0.1:8090/archives/<infohash>/10/512/380.webp
+ *   http://127.0.0.1:8090/stacks/<id>/10/512/380.webp
+ *
+ * The two lines worth reading are the step histogram and the flat-ground
+ * roughness. If the steps are all one quantum, what you are seeing is the
+ * encoding and the renderer, not the data.
+ */
+
+import { loadCodec } from '../src/codec.js';
+import { decodeHeights } from '../src/elevation.js';
+
+const args = process.argv.slice(2);
+const url = args.find((a) => !a.startsWith('--'));
+if (!url) {
+  console.error(
+    'usage: node tools/terrain-probe.mjs <tile-url> [--encoding mapbox] [--interval 0.1] [--baseVal -10000]',
+  );
+  process.exit(2);
+}
+
+/**
+ * One named option off the command line.
+ * @param {string} name - Its name, without the dashes.
+ * @param {string} fallback - What it is when absent.
+ * @returns {string} - The value.
+ */
+const option = (name, fallback) => {
+  const at = args.indexOf(`--${name}`);
+  return at >= 0 && args[at + 1] ? args[at + 1] : fallback;
+};
+
+const source = {
+  encoding: option('encoding', 'mapbox'),
+  interval: Number(option('interval', '0.1')),
+  baseVal: Number(option('baseVal', '-10000')),
+};
+
+const codec = await loadCodec();
+if (!codec) {
+  console.error('this needs sharp: npm install sharp');
+  process.exit(1);
+}
+
+const response = await fetch(url);
+if (!response.ok) {
+  console.error(`${url} answered ${response.status}`);
+  process.exit(1);
+}
+const raster = await codec.decode(Buffer.from(await response.arrayBuffer()), {
+  channels: 3,
+});
+const heights = decodeHeights(raster, source);
+const { width } = raster;
+// Off the URL, so roughness can be reported against the ground a pixel covers
+// rather than in the abstract. A metre between neighbours means one thing at
+// z8 and another at z16.
+const coordinates = url.match(/\/(\d+)\/(\d+)\/(\d+)\.[a-z]+$/);
+const zoom = Number(coordinates?.[1] ?? 0);
+const tileRow = Number(coordinates?.[3] ?? 0);
+
+// Walked rather than spread into Math.min: a 512px tile is 262,144 numbers and
+// that many arguments overflows the call stack.
+let low = Infinity;
+let high = -Infinity;
+let known = 0;
+let zeroes = 0;
+for (const value of heights) {
+  if (!Number.isFinite(value)) continue;
+  known += 1;
+  if (value === 0) zeroes += 1;
+  if (value < low) low = value;
+  if (value > high) high = value;
+}
+if (known === 0) {
+  console.log(`${url}
+  every pixel is nodata`);
+  process.exit(0);
+}
+
+// Every step between neighbouring pixels, left to right and top to bottom.
+const steps = [];
+for (let row = 0; row < width; row += 1) {
+  for (let column = 1; column < width; column += 1) {
+    steps.push(
+      Math.abs(
+        heights[row * width + column] - heights[row * width + column - 1],
+      ),
+    );
+  }
+}
+for (let row = 1; row < width; row += 1) {
+  for (let column = 0; column < width; column += 1) {
+    steps.push(
+      Math.abs(
+        heights[row * width + column] - heights[(row - 1) * width + column],
+      ),
+    );
+  }
+}
+
+const quantum = source.encoding === 'terrarium' ? 1 / 256 : source.interval;
+const inQuanta = steps.map((s) => Math.round(s / quantum));
+const counts = new Map();
+for (const q of inQuanta) counts.set(q, (counts.get(q) ?? 0) + 1);
+
+console.log(`${url}`);
+console.log(
+  `  ${width}x${width}, ${source.encoding}, one quantum = ${quantum} m`,
+);
+console.log(`  heights ${low.toFixed(2)} m to ${high.toFixed(2)} m`);
+console.log(`  exactly zero: ${zeroes} of ${known} pixels`);
+
+console.log('\n  step between neighbouring pixels:');
+const ordered = [...counts.entries()].sort((a, b) => a[0] - b[0]).slice(0, 8);
+for (const [q, n] of ordered) {
+  const share = n / steps.length;
+  console.log(
+    `   ${String(q).padStart(4)} quanta (${(q * quantum).toFixed(2)} m) ` +
+      `${'#'.repeat(Math.round(share * 40)).padEnd(40)} ${(share * 100).toFixed(1)}%`,
+  );
+}
+
+// A median cannot see sparse spikes, and sparse spikes are what stipple a
+// hillshade: a scattering of pixels standing well clear of their neighbours,
+// too few to move the middle of any distribution. So the tail is what gets
+// reported, and separately the count of pixels that disagree with everything
+// around them.
+const flat = [];
+let spikes = 0;
+let worstSpike = 0;
+for (let row = 1; row < width - 1; row += 1) {
+  for (let column = 1; column < width - 1; column += 1) {
+    const at = row * width + column;
+    const here = heights[at];
+    if (!Number.isFinite(here)) continue;
+
+    const across = heights[at + 1] - 2 * here + heights[at - 1];
+    const down = heights[at + width] - 2 * here + heights[at - width];
+    if (Number.isFinite(across) && Number.isFinite(down)) {
+      flat.push(Math.abs(across) + Math.abs(down));
+    }
+
+    // Standing clear of every neighbour, in the same direction. Terrain does
+    // not do this; a pixel that survived a mask its neighbours did not, or one
+    // the encoding placed a quantum out, does.
+    let above = 0;
+    let below = 0;
+    let seen = 0;
+    let nearest = Infinity;
+    for (const [dy, dx] of [
+      [-1, -1],
+      [-1, 0],
+      [-1, 1],
+      [0, -1],
+      [0, 1],
+      [1, -1],
+      [1, 0],
+      [1, 1],
+    ]) {
+      const neighbour = heights[(row + dy) * width + column + dx];
+      if (!Number.isFinite(neighbour)) continue;
+      seen += 1;
+      if (here > neighbour) above += 1;
+      else if (here < neighbour) below += 1;
+      nearest = Math.min(nearest, Math.abs(here - neighbour));
+    }
+    if (seen === 8 && (above === 8 || below === 8) && nearest > quantum * 1.5) {
+      spikes += 1;
+      worstSpike = Math.max(worstSpike, nearest);
+    }
+  }
+}
+
+flat.sort((one, two) => one - two);
+/**
+ * A value at a percentile of the sorted roughness.
+ * @param {number} share - Where to look, 0 to 1.
+ * @returns {number} - Metres.
+ */
+const at = (share) =>
+  flat[Math.min(flat.length - 1, Math.floor(flat.length * share))] ?? 0;
+
+// How much ground one pixel covers, so roughness can be read as a slope.
+// Web mercator shrinks with latitude, and the tile's row is where that comes
+// from -- a metre between neighbours means one thing at z8 and another at z16.
+const rows = 2 ** zoom;
+// Clamped: a row outside the pyramid would put the latitude past the pole and
+// report no ground at all, which reads as a broken tile rather than a typo.
+const row = Math.min(Math.max(tileRow, 0), rows - 1);
+const latitude = Math.atan(Math.sinh(Math.PI * (1 - (2 * row) / rows)));
+const groundMetres = (40075016.686 * Math.cos(latitude)) / rows / width;
+
+console.log('\n  roughness, as the second difference between neighbours:');
+for (const [label, share] of [
+  ['half of them are under', 0.5],
+  ['nine in ten under', 0.9],
+  ['ninety-nine in a hundred under', 0.99],
+  ['the worst', 1],
+]) {
+  console.log(
+    `   ${label.padEnd(32)} ${at(share).toFixed(2).padStart(9)} m` +
+      ` (${(at(share) / quantum).toFixed(0)} quanta)`,
+  );
+}
+
+console.log(
+  `\n  relief across this tile: ${(high - low).toFixed(1)} m` +
+    `, about ${groundMetres.toFixed(0)} m of ground per pixel`,
+);
+console.log(
+  `  pixels standing clear of all eight neighbours: ${spikes}` +
+    ` (${((spikes / known) * 100).toFixed(2)}%)` +
+    (spikes ? `, worst ${worstSpike.toFixed(1)} m` : ''),
+);
+
+console.log('');
+if (spikes / known > 0.002 && worstSpike > quantum * 10) {
+  console.log(
+    '  -> a scattering of pixels stands well clear of everything around it.\n' +
+      '     That is what stipples a hillshade, and terrain does not do it. Look\n' +
+      '     for something applied per pixel: a mask that matched some pixels and\n' +
+      '     not their neighbours, or two sources meeting one pixel at a time.',
+  );
+} else if (at(0.99) <= quantum * 2.5) {
+  console.log(
+    "  -> smooth to the encoding's own resolution, in the tail as well as the\n" +
+      '     middle. A checkerboard here is the colour ramp dithering across a\n' +
+      '     band edge, and a hillshade exaggerating a tenth of a metre.',
+  );
+} else {
+  console.log(
+    `  -> rough, but evenly so, against ${(high - low).toFixed(0)} m of relief across\n` +
+      '     the tile. That is what terrain looks like; compare a tile of flat\n' +
+      '     ground before calling it noise.',
+  );
+}