pmtiles-swarm 0.73.0 → 0.74.0

package/CHANGELOG.md

@@ -7,6 +7,52 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.74.0
+### ✨ Features and improvements
+- **The holes a mask leaves are feathered now, not just a cutline's edge.** `feather` faded a source
+  in at its `cutline` or `bounds` and did nothing about `maskValues` or `maskColors` - which is
+  where most of these recipes actually stop, and why the field looked like it did nothing.
+
+  A tile cannot measure that edge alone: the hole may continue past its border and nothing inside it
+  says so. Carrying the edge outward, which is the obvious answer, changes the seam by nothing at
+  all - measured across four coastline geometries, to the metre - because it only adds holes where
+  the edge pixel is already one and the distance is already zero. The tile that gets it wrong is
+  the tile with no hole in it, and no rule applied to its own pixels can invent one.
+
+  So the mask is read from the source's parent, which covers this tile and its three siblings and
+  sees past every one of their borders. Four parents cover any tile's surroundings, each is shared
+  by four children, and they are read only for a source that both masks and fades. The parent
+  supplies the border; the middle is the tile's own pixels, because a ramp measured entirely against
+  half-resolution pixels climbs in two-pixel steps, which is the terracing the fade exists to
+  remove. A 1000 m drop over a 16 px fade, stepping along the edge two tiles share:
+
+      coastline crossing at 45°        1000 m unfeathered  ->   63 m
+      crossing steeply                 1000 m              ->    0 m
+      parallel, just past the seam        0 m              ->  125 m
+      parallel, right on the seam         0 m              ->  125 m
+
+  What is left is bounded by the parent's resolution rather than by the height difference - two
+  steps of the ramp, or `2 / feather` of the drop - and shrinks as the feather widens.
+
+  The field is on every source in the editor now rather than appearing only once a clip is set,
+  with a note where a source neither masks nor clips and it would do nothing.
+
+## 0.73.1
+### 🐞 Bug fixes
+- **The console could not start at all in 0.73.0.** A string literal in the export dialog's output
+  summary lost its escape, so the console's module failed to parse - and a module that does not
+  parse does not run one line of itself. Every page rendered its static markup and then sat on
+  "connecting…" for ever, with one `SyntaxError` in the browser console and nothing anywhere else.
+
+  Nothing was in a position to catch it. eslint does not read HTML, prettier reports the file clean
+  whether or not the script inside it is valid, and the tests that do read that script match
+  patterns in its text rather than running it. So 320 KB of JavaScript shipped without anything
+  having asked whether it parses.
+
+  `test/web-parse.test.js` now hands every inline script in every page to `node --check` - the same
+  engine that will run it, so there is no second grammar to disagree with. Verified against this
+  exact fault: the test fails on it and names the line, where prettier and eslint both pass.
+
 ## 0.73.0
 ### ✨ Features and improvements
 - **A source can fade in at the edge of its shape instead of stopping dead.** Where a

package/docs/tile-stacks.md

@@ -1114,29 +1114,64 @@ The offline merge still has this. `scipy.ndimage.gaussian_filter` defaults to
 `mode='reflect'` and is handed one tile's array, so an archive built with a
 sigma carries the seam.
 
-### Where the margin comes from for the feather
-
-The border above solves the upscaled case, which is the one the blur needed.
-Feathering has two more.
-
-**A missing tile costs nothing.** Whether a neighbour exists is a directory
-lookup, not a decode. A source that vanishes at a tile edge can ramp down to 0
-approaching that edge using only knowledge of which neighbours are absent.
-
-**A mask hole at native resolution is the expensive one.** There the edge is in
-the source's own pixels at full resolution, with no parent to widen the window
-into, so the margin genuinely means reading the neighbouring tiles — up to eight
-reads and eight decodes for that source, on that tile. That is the case to
-bound, to make opt-in, and to measure before believing.
+### Where the margin comes from for a mask edge
+
+A cutline is known in full, so the ramp beside a tile costs a few extra
+rasterised rows. A mask edge is not: it lives in the source's own decoded
+pixels, and a tile cannot see whether the hole continues past its own border.
+
+That is not a boundary condition to be chosen well, which is the first thing
+worth writing down because it looks like one. Carrying the edge outward — the
+obvious fix — adds holes only where the edge pixel is already a hole, and there
+the distance is already zero. Measured against four coastline geometries it
+changes the seam by nothing at all, to the metre. The tile that gets it wrong is
+the tile with **no hole in it**, and no rule applied to its own pixels can
+invent one.
+
+So the mask is read from the source's **parent**, which covers this tile and its
+three siblings and therefore sees past every one of their borders. Four parents
+cover any tile's surroundings, each is shared by four children, and they are
+read only for a source that both masks and fades.
+
+The parent supplies the border only. Its pixels are half this tile's, so a ramp
+measured entirely against them climbs in two-pixel steps — terracing, which is
+the artefact the fade exists to remove — and the middle is overwritten with the
+tile's own pixels, which are exact.
+
+A 1000 m drop faded over 16 px, measuring the step along the edge two tiles
+share:
+
+| coastline at the seam        | no fade | tile alone | with parents |
+| ---------------------------- | ------- | ---------- | ------------ |
+| crossing at 45°              | 1000 m  | 1000 m     | 63 m         |
+| crossing steeply             | 1000 m  | 938 m      | 0 m          |
+| parallel, just past the seam | 0 m     | 438 m      | 125 m        |
+| parallel, on the seam        | 0 m     | 938 m      | 125 m        |
+
+"tile alone" is the version that measures inside the tile and is why it is not
+the version that shipped: it moves the wall off the coastline, where it is at
+least geographically meaningful, and onto the tile grid.
+
+What is left is bounded by the parent's resolution rather than by the height
+difference — two steps of the ramp, or `2 / feather` of the drop, and it shrinks
+as the feather widens. Reading the eight neighbours at native resolution instead
+would make it exact, at nine times the reads and decodes for that source; the
+extra precision is finer than a sixteen-pixel ramp can show.
+
+**A source that vanishes at a tile edge is still open.** A sparse archive has no
+tile outside its extent, so the seam lands exactly on a tile boundary. It is the
+cheapest of the three to detect — whether a neighbour exists is a directory
+lookup rather than a decode — and it is not done.
 
 ### What it does
 
 `feather` on a source, in pixels, `0` and absent meaning the edge is a switch as
-before. It fades that source in over that many pixels measured inward from its
-cutline or its `bounds`:
+before. It fades that source in over that many pixels measured inward from
+wherever it stops — the holes `maskValues` and `maskColors` leave, and the edge
+of a `cutline` or `bounds`:
 
 ```json
-{ "archive": "swissalti", "cutline": "switzerland", "feather": 16 }
+{ "archive": "swissalti", "maskValues": [0], "feather": 16 }
 ```
 
 The step left at the seam is the height difference divided by the feather, which

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.73.0",
+  "version": "0.74.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/elevation.js

@@ -1,3 +1,5 @@
+import { cropMask, featherMask } from './cutline.js';
+
 /**
  * Merging terrain tiles, in metres rather than in pixels.
  *
@@ -691,6 +693,46 @@ export function mergeElevation(contributions, options) {
         if (contribution.coverage[i] <= 0) heights[i] = Number.NaN;
       }
     }
+
+    // The edge a mask left, which is the one most of these recipes have.
+    // Measured on a grid that reaches past this tile, because the pixels that
+    // decide how far a hole is may be in the next one -- `gather` reads the
+    // source's parents for exactly this. Without them there is nothing to
+    // measure against and the edge stays as it was, rather than being faded
+    // against a hole this tile cannot see.
+    if (contribution.neighbourhood) {
+      const { known, margin } = contribution.neighbourhood;
+      const side = size + margin * 2;
+
+      // The parent is half this tile's resolution, so a ramp measured entirely
+      // against it climbs in two-pixel steps -- terracing, which is the
+      // artefact this whole feature exists to remove. So the parent supplies
+      // the border only, and the middle is overwritten with this tile's own
+      // pixels, which are exact. Distances inside the tile are what the ramp
+      // is mostly made of; only the last few pixels of its reach are
+      // approximated, and there a pixel of error is a pixel of a sixteen-pixel
+      // fade.
+      const grid = Uint8Array.from(known);
+      for (let row = 0; row < size; row += 1) {
+        const from = row * size;
+        const to = (row + margin) * side + margin;
+        for (let column = 0; column < size; column += 1) {
+          grid[to + column] = Number.isNaN(heights[from + column]) ? 0 : 1;
+        }
+      }
+
+      const ramp = cropMask(featherMask(grid, side, margin), size, margin);
+      // The narrower of the two where a cutline has ramped as well: each is a
+      // separate statement about how much of this source belongs here, and
+      // both have to hold.
+      if (contribution.coverage) {
+        for (let i = 0; i < ramp.length; i += 1) {
+          ramp[i] = Math.min(ramp[i], contribution.coverage[i]);
+        }
+      }
+      contribution.coverage = ramp;
+    }
+
     return heights;
   });
 

package/src/mask-edge.js

@@ -0,0 +1,118 @@
+/**
+ * The edge a mask leaves, measured across the tile boundary.
+ *
+ * `maskValues` and `maskColors` turn a source's nodata into holes, and where a
+ * high-resolution source ends the merge steps straight down to whatever is
+ * underneath. Feathering that edge means knowing how far each pixel is from
+ * the hole — and a tile cannot answer that on its own, because the hole may
+ * continue past its border and nothing inside the tile says so.
+ *
+ * That is not a boundary condition to be chosen well. Carrying the edge
+ * outward adds holes only where the edge pixel is already one, where the
+ * distance is already zero, so it changes nothing — measured, to the metre.
+ * The tile that gets it wrong is the tile with no hole in it at all, and no
+ * rule applied to its own pixels can invent one.
+ *
+ * So the mask is read from the source's **parent**, which covers this tile and
+ * its three siblings and therefore sees past every one of their borders. Four
+ * parents cover any tile's surroundings, and each is shared by four children.
+ *
+ * The parent supplies the border only. Its pixels are half this tile's, so a
+ * ramp measured entirely against them climbs in two-pixel steps -- terracing,
+ * which is the artefact the fade exists to remove -- and the middle is
+ * overwritten with the tile's own. What is left is a seam bounded by the
+ * parent's resolution rather than by the height difference: two steps of the
+ * ramp, against the whole drop unfeathered. See docs/tile-stacks.md --
+ * "Feathering a seam".
+ */
+
+/**
+ * Which parent tiles cover this tile and the border around it.
+ *
+ * At most four: the tile occupies one quadrant of its parent, so a border
+ * around it reaches past two of that parent's sides and into the two
+ * neighbours and the one diagonal.
+ * @param {object} tile - z, x, y of the tile being built.
+ * @param {number} size - Pixels per side of that tile.
+ * @param {number} feather - The border wanted, in those pixels.
+ * @returns {object|null} - `{z, tiles, west, north, per}`, or null at z0.
+ */
+export function parentsFor(tile, size, feather) {
+  const z = tile.z - 1;
+  if (z < 0 || !(feather > 0)) return null;
+
+  // In units of one parent tile: this tile is half of one a side, and the
+  // border is a fraction of that again.
+  const per = feather / (2 * size);
+  const west = tile.x / 2 - per;
+  const east = (tile.x + 1) / 2 + per;
+  const north = tile.y / 2 - per;
+  const south = (tile.y + 1) / 2 + per;
+
+  const span = 2 ** z;
+  const tiles = [];
+  for (let row = Math.floor(north); row <= Math.floor(south); row += 1) {
+    for (
+      let column = Math.floor(west);
+      column <= Math.floor(east);
+      column += 1
+    ) {
+      // Off the top or bottom of the world there is no tile and never was.
+      // Off the side there is: the map wraps, and a source that covers the
+      // antimeridian covers both halves of it.
+      if (row < 0 || row >= span) continue;
+      tiles.push({
+        z,
+        x: ((column % span) + span) % span,
+        y: row,
+        column,
+        row,
+      });
+    }
+  }
+  return { z, tiles, west, north, per };
+}
+
+/**
+ * A padded grid saying where the source has data, at the output's resolution.
+ *
+ * Sampled nearest from the parents rather than interpolated: this is a yes or
+ * no about each pixel, and a half-yes between a hole and the ground beside it
+ * is not a smaller hole, it is a wrong answer about which one the pixel is.
+ * @param {object} layout - What `parentsFor` returned.
+ * @param {Map} known - Parent key `"x,y"` to a Uint8Array, 1 where it has data.
+ * @param {number} size - Pixels per side of the tile being built.
+ * @param {number} feather - The border, in those pixels.
+ * @param {number} parentSize - Pixels per side of a parent.
+ * @returns {Uint8Array} - `size + 2 * feather` a side, 1 where there is data.
+ */
+export function paddedKnown(layout, known, size, feather, parentSize) {
+  const side = size + feather * 2;
+  const out = new Uint8Array(side * side);
+  // One output pixel is this much of a parent tile.
+  const step = 1 / (2 * size);
+
+  for (let row = 0; row < side; row += 1) {
+    // The middle of the pixel, in parent-tile units.
+    const y = layout.north + (row + 0.5) * step;
+    const parentRow = Math.floor(y);
+    for (let column = 0; column < side; column += 1) {
+      const x = layout.west + (column + 0.5) * step;
+      const parentColumn = Math.floor(x);
+      const grid = known.get(`${parentColumn},${parentRow}`);
+      // A parent nothing could be read from is ground this source does not
+      // cover, which is what a hole means anyway.
+      if (!grid) continue;
+      const px = Math.min(
+        parentSize - 1,
+        Math.floor((x - parentColumn) * parentSize),
+      );
+      const py = Math.min(
+        parentSize - 1,
+        Math.floor((y - parentRow) * parentSize),
+      );
+      out[row * side + column] = grid[py * parentSize + px];
+    }
+  }
+  return out;
+}

package/src/stack-tile.js

@@ -1,9 +1,13 @@
 import {
   assembleChildren,
+  decodeHeights,
   encodeHeights,
   fillNodata,
+  maskColors,
+  maskHeights,
   mergeElevation,
 } from './elevation.js';
+import { paddedKnown, parentsFor } from './mask-edge.js';
 import {
   INSIDE,
   OUTSIDE,
@@ -428,14 +432,105 @@ async function gather({ reads, z, x, y, tiles, codec, signal, size, rgba }) {
 
     contributions.push({
       source: read.source.source,
+      // Carried so a feathered source can go back for its parents, which is
+      // the one thing the merge needs that is not already in this raster.
+      infoHash: read.source.entry?.infoHash,
       parentZ: read.found.parentZ,
       raster,
     });
   }
 
+  await readMaskEdges({ contributions, z, x, y, size, tiles, codec, signal });
   return { contributors, contributions };
 }
 
+/**
+ * Whether a source's recipe leaves holes a feather would have to fade.
+ * @param {object} recipe - One source out of a recipe.
+ * @returns {boolean} - True when it masks anything.
+ */
+function masksAnything(recipe) {
+  return Boolean(recipe?.maskValues?.length || recipe?.maskColors?.length);
+}
+
+/**
+ * Reads the parents a feathered source needs to see past its own tile.
+ *
+ * Only for a source that both fades and masks. A cutline's ramp is measured
+ * from a shape this node holds in full and needs nothing read; a mask's is
+ * measured from pixels, and the pixels that decide it are partly in the
+ * neighbouring tiles. See src/mask-edge.js for why the parent rather than
+ * those neighbours.
+ *
+ * A parent that cannot be read is left out rather than failing the tile. It
+ * means the ramp is measured against less than it could be, which is a slightly
+ * wrong edge -- and a tile that will not draw at all is worse than one whose
+ * coastline fades over fifteen pixels instead of sixteen.
+ * @param {object} options - The contributions, where they are, and what to read with.
+ * @returns {Promise<void>} - Resolves once every feathered source has its grid.
+ */
+async function readMaskEdges({
+  contributions,
+  z,
+  x,
+  y,
+  size,
+  tiles,
+  codec,
+  signal,
+}) {
+  if (!codec) return;
+  await Promise.all(
+    contributions.map(async (contribution) => {
+      if (!contribution) return;
+      const recipe = contribution.source ?? {};
+      const feather = featherFor(recipe);
+      if (!feather || !masksAnything(recipe)) return;
+
+      const grid = contribution.raster?.width ?? size;
+      const layout = parentsFor({ z, x, y }, grid, feather);
+      if (!layout) return;
+
+      const known = new Map();
+      let parentSize = grid;
+      await Promise.all(
+        layout.tiles.map(async (parent) => {
+          const tile = await tiles
+            .getTile(contribution.infoHash, parent.z, parent.x, parent.y, {
+              signal,
+            })
+            .catch(() => null);
+          if (!tile?.data) return;
+          const raster = await codec
+            .decode(Buffer.from(tile.data), { channels: 3 })
+            .catch(() => null);
+          if (!raster) return;
+
+          // The same two masks the merge applies, asked of the parent. A ramp
+          // measured against a different idea of where the holes are would fade
+          // toward ground that is not a hole.
+          const heights = decodeHeights(raster, recipe);
+          maskHeights(heights, recipe.maskValues);
+          maskColors(heights, raster, recipe.maskColors);
+
+          const flags = new Uint8Array(heights.length);
+          for (let i = 0; i < flags.length; i += 1) {
+            flags[i] = Number.isNaN(heights[i]) ? 0 : 1;
+          }
+          parentSize = raster.width;
+          known.set(`${parent.column},${parent.row}`, flags);
+        }),
+      );
+
+      if (known.size === 0) return;
+      contribution.neighbourhood = {
+        known: paddedKnown(layout, known, grid, feather, parentSize),
+        margin: feather,
+      };
+    }),
+  );
+}
+
 /**
  * Merges what the sources gave into one encoded tile.
  * @param {object} options - Everything the merge needs.

package/src/stacks.js

@@ -137,9 +137,16 @@ export function validateStack(stack) {
     }
     // A fade with no edge to fade at. Refused rather than ignored: it reads as
     // a source that blends into what is under it, and it does nothing at all.
-    if (source?.feather && !source.cutline && !source.bounds) {
+    // A mask is an edge as much as a cutline is -- the hole it leaves is where
+    // most of these recipes actually stop.
+    const fades =
+      source?.cutline ||
+      source?.bounds ||
+      source?.maskValues?.length ||
+      source?.maskColors?.length;
+    if (source?.feather && !fades) {
       problems.push(
-        `sources[${index}].feather needs a cutline or bounds to fade at`,
+        `sources[${index}].feather needs a cutline, bounds, or a mask to fade at`,
       );
     }
     if (source?.encoding === 'custom' && !hasUsableEncoding(source)) {

package/src/web/index.html

@@ -7672,19 +7672,23 @@ Every piece is hashed against the ` +
                      </label>`
                   : ''
               }
-              ${
-                source.cutline || source.bounds
-                  ? `<label class="choice"
-                            title="Fade this source in over this many pixels at the edge of its shape, instead of stopping dead. Where a high-resolution source meets a coarser one the two disagree — by their survey and by their vertical datum — and the step between them reads as a wall under a hillshade. The drop is divided by this: 40 m across a 8 px feather is 5 m a pixel. 0 is a hard edge.">
-                       Fade in
-                       <input type="number" min="0" max="64" step="1" style="width:4.5rem"
-                              placeholder="0"
-                              data-stack-field="feather" data-stack-index="${index}"
-                              value="${source.feather ?? ''}" />
-                       <span class="sub">px</span>
-                     </label>`
-                  : ''
-              }
+              <label class="choice"
+                     title="Fade this source in over this many pixels wherever it stops — the holes a mask leaves, and the edge of a cutline or box. Where a high-resolution source meets a coarser one the two disagree, by their survey and by their vertical datum, and the step between them reads as a wall under a hillshade. The drop is divided by this: 40 m across an 8 px fade is 5 m a pixel. 0 is a hard edge.">
+                Fade in
+                <input type="number" min="0" max="64" step="1" style="width:4.5rem"
+                       placeholder="0"
+                       data-stack-field="feather" data-stack-index="${index}"
+                       value="${source.feather ?? ''}" />
+                <span class="sub">px${
+                  source.feather &&
+                  !source.cutline &&
+                  !source.bounds &&
+                  !(source.maskValues ?? []).length &&
+                  !(source.maskColors ?? []).length
+                    ? ' — nothing to fade at yet: mask something, or clip it'
+                    : ''
+                }</span>
+              </label>
               ${
                 source.cutline && !(stackCutlines ?? []).includes(source.cutline)
                   ? `<span class="bad">no cutline called
@@ -7767,10 +7771,6 @@ Every piece is hashed against the ` +
           if (source) {
             delete source.cutline;
             delete source.bounds;
-            // A fade needs an edge to fade at, and the recipe validation says
-            // so -- leaving it behind would make the source unsaveable with
-            // nothing on screen to explain why.
-            if (!event.target.value) delete source.feather;
             if (event.target.value === '__bounds') {
               source.bounds = [-180, -85, 180, 85];
             } else if (event.target.value) {
@@ -7946,9 +7946,9 @@ Every piece is hashed against the ` +
         const output = stack?.output ?? {};
         const rgba = stack?.space === 'rgba';
         const parts = [
-          rgba ? 'rgba' : (output.encoding ?? 'the sources' encoding'),
+          rgba ? 'rgba' : (output.encoding ?? "the sources' encoding"),
           output.format ?? "the sources' format",
-          output.tileSize ? `${output.tileSize} px` : 'the sources' tile size',
+          output.tileSize ? `${output.tileSize} px` : "the sources' tile size",
         ];
         if (!rgba) {
           parts.push(`${stack?.resampling ?? 'bilinear'} resampling`);