pmtiles-swarm 0.100.0 → 0.102.0

package/CHANGELOG.md

@@ -7,6 +7,114 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.102.0
+### ✨ Features and improvements
+- **Hillshade and contours can be switched off in the preview.** Two boxes in the header, shown only
+  for the layers that page actually drew. Not only a cosmetic switch for the contours: MapLibre
+  stops fetching from a source once no visible layer draws from it, so unticking the box really does
+  stop the nine tile merges each contour tile costs. 3D stays where it was, on MapLibre's own
+  terrain button.
+
+### 🐞 Bug fixes
+- **The contour preview asked for a tile at the literal coordinates `{z}`.** Its source URL was
+  built with `new URL()`, which percent-encodes braces — so the template went out as
+  `contours/%7Bz%7D/%7Bx%7D/%7By%7D.pbf`, MapLibre never substituted it, and every request came back
+  `400: z, x and y must be integers`. Joined to the origin instead, since the path was already
+  absolute and there was nothing to resolve.
+
+## 0.101.0
+### ✨ Features and improvements
+- **Contours, traced from what a stack merges.** `GET /stacks/<id>/contours/{z}/{x}/{y}.pbf` returns
+  a gzipped vector tile of contour lines. The interval comes from the recipe's `contours.thresholds`
+  or from `?interval=` on the request — contours are a view of a stack rather than a property of one,
+  and the same terrain is wanted at 10 m on a walking map and 100 m on an atlas.
+
+  Drawn here rather than from an archive for one reason. A tool reading archives has to answer "what
+  is the elevation?" for ground no archive covers, and an encoded terrain tile cannot say
+  "nothing" — every triple of bytes is a height. So it invents one, and a constant beside real
+  terrain is a cliff, which a tracer renders as lines packed tight along the seam. A stack has `NaN`
+  and hands its holes over unfilled, so a line stops at the edge of the data instead.
+
+  A level may name several intervals. `[100, 500]` draws every hundred metres and marks every fifth,
+  and each feature carries `level` — how many intervals its height divides by — so a style draws the
+  major lines thicker and labels only those, from one layer. That is `maplibre-contour`'s convention,
+  so a style written for its tiles works against these.
+
+  Worth knowing what it costs: a contour tile is traced from its own tile **plus its eight
+  neighbours**, because a line crossing an edge has to be traced from the ground on both sides or it
+  will not meet the line next door. Roughly nine merged terrain tiles each. See
+  [tile-stacks.md](docs/tile-stacks.md) — "Contours from a stack".
+
+  A **Contours** button sits beside Preview and Terrain on any stack that draws as terrain, and the
+  preview draws the lines over the hillshade rather than instead of it — a contour on its own says
+  nothing about which side of it is uphill. Major lines draw themselves thicker from `level`, with
+  no second layer and no filter. There are no labels: text needs a glyph server, and a preview that
+  fetched fonts from elsewhere would be a preview of that as much as of the archive. `ele` is on
+  every feature for a style that has fonts to spend.
+
+  **An export can write them too.** The export dialog's *What to make* list has its second entry,
+  and `POST /api/stacks/<id>/bake` takes `kind: "contours"` with an optional `thresholds`. The
+  archive is written as gzipped MVT with the right header, and the run narrows itself to the zooms
+  its thresholds actually draw at — without that it walks every tile the sources hold at z0–z8 to
+  trace nothing, which on a planet is hours spent producing silence.
+
+  The interval is part of the export's revision, so a checkpoint cannot be resumed across a change
+  to it: a 20 m run continued into a 100 m one would splice two sets of lines into an archive
+  nothing downstream could tell apart.
+
+  **Merged heights are cached in memory**, under `stacks.heightsCacheBytes` (64 MiB, zero to turn
+  it off). A different cache from `stacks.cacheBytes` and for a different shape of reuse: that one
+  holds encoded tiles so a second request for one tile is free, this holds the numbers a tile was
+  merged from, because several *different* tiles are built out of the same neighbours. An N×N
+  contour run needs (N+2)² merges and asks for 9N² — measured at 4× fewer source reads and twice
+  the speed on a 4×4 block, approaching 9× on a large one. A feathered source benefits more mildly,
+  since four sibling tiles share the parents its ramp is measured against.
+
+  Each contour export gets its own, which goes when the job does. A bake deliberately does not
+  touch the disk cache — a planet export would evict the serving node's entire cache with tiles
+  nobody will ask for again — and that is exactly why it can have one of its own.
+
+  **Contours from an archive or a category too**, at `/archives/<infohash>/contours/{z}/{x}/{y}.pbf`
+  and `/latest/<category>/contours/{z}/{x}/{y}.pbf`. An archive is already terrain — its pixels are
+  a packed height and it states the packing in its own metadata — so these decode the tile and do
+  nothing else: no recipe to resolve, no sources to merge, no masks or clips, because there is no
+  recipe saying to. Nine reads a tile against the stack path's nine merges.
+
+  They also do not climb to a parent where the archive has no tile at that zoom, which the stack
+  path would: for contours that is the wrong favour, since a line traced from an upscaled parent is
+  the parent's line drawn twice as thick rather than detail the zoom has. An infohash is immutable
+  and cached as such; a category revalidates, and is tagged by the build it resolved to so a
+  rebuild lands as a new tag rather than the same one with different lines behind it.
+
+  The **Contours** button now appears anywhere terrain does — a stack's row, an archive's panel, and
+  the public catalogue — rather than on stacks alone. The preview rewrites its own path into the
+  contour endpoint, which is the same rewrite for all three.
+- **An export can write part of a stack.** A zoom range, an area, or both, from the export dialog or
+  as `minzoom` / `maxzoom` / `bounds` on `POST /api/stacks/<id>/bake`. Absent still means all of it.
+  An export reads every tile its sources hold, which for a planet is hours and a file nobody wanted
+  all of.
+
+  The two narrow differently. PMTiles orders tile ids by zoom and then along a Hilbert curve, so a
+  zoom range is a contiguous run of ids and the scan **ends** at the deepest zoom asked for rather
+  than filtering past it — which matters, because not enumerating unwanted zooms is why the export
+  iterates coverage in the first place. A box is not contiguous, so it is a test per tile; it runs
+  before the merge, so what it saves is the merge.
+
+  A tile is written when it **overlaps** the box, not when it sits inside it, so an archive reaches
+  its own edge instead of stopping up to a tile short. The dialog can also fill the box in from a
+  `z/x/y` tile, which is how a planet is usually split: regions that tile evenly, never overlap, and
+  have a name to agree on.
+
+  Three things move with the selection, each a silent fault otherwise. The **revision**, so a
+  checkpoint taken under one selection is not resumed under another — the stream of ids is different
+  and resuming would skip whatever the new selection adds below the mark. Both **names**, since two
+  exports of one recipe over different ground are two archives and a date does not tell them apart.
+  And the header's **bounds**, which the writer defaults to the whole world: an archive claiming a
+  planet and holding one country is one a client keeps asking for tiles that were never written. The
+  zooms are still read off the tiles actually written, which is more honest than the request.
+
+### 🐞 Bug fixes
+
 ## 0.100.0
 ### ✨ Features and improvements
 - **A nested stack takes `maskColors`, and fades into what is under it.** Two things a stack could

package/docs/tile-stacks.md

@@ -63,6 +63,7 @@ of its parts.
 - [Syncing a stack to another node](#syncing-a-stack-to-another-node)
 - [What the offline merge got wrong](#what-the-offline-merge-got-wrong)
 - [The stack editor](#the-stack-editor)
+- [Contours from a stack](#contours-from-a-stack)
 - [Merging vector sources](#merging-vector-sources)
 - [Staging](#staging)
 - [Open questions](#open-questions)
@@ -2320,6 +2321,94 @@ Debounced, and it should not follow the map past the stack's `maxzoom`: above
 that the client overzooms and there is nothing new to see, but the tiles are
 still requested and still composited.
 
+## Contours from a stack
+
+A contour tile is drawn from the heights a stack merges, not from an archive of
+terrain. That distinction is the whole reason to do it here.
+
+A tool that reads archives has to answer "what is the elevation here?" for
+ground no archive covers, and an encoded terrain tile has no way to say
+"nothing" — every triple of bytes is some height. So it invents one. A constant
+beside real terrain is a **cliff**, and a cliff under a contour tracer comes out
+as lines packed arbitrarily tight along the seam. Filling with `-10000` rather
+than `0` only moves the cliff and makes it taller. That is not a fault in the
+tool; it is the shape of its input.
+
+A stack has `NaN`, and `stackHeights` deliberately leaves its holes unfilled —
+the same property that lets one stack show through another. `maplibre-contour`
+already understands it: `HeightTile.fromRawDem` maps anything invalid to `NaN`,
+the tracer skips it, and `combineNeighbors` answers `NaN` for a neighbour that
+is missing altogether rather than throwing. So a tile the stack covers none of
+is passed as `undefined` and reads as no-data. Nothing is invented anywhere, and
+a line stops at the edge of the data instead of diving off a cliff.
+
+Where you want contours to run on past the coast, put a base under them — the
+merge is what makes them continuous, and `featherMetres` is what stops the seam
+between two sources becoming its own little cliff. For contours that feather is
+load-bearing rather than cosmetic.
+
+### Nine tiles per tile
+
+A contour crossing a tile edge has to be traced from the ground on both sides,
+or it will not meet the line in the next tile. So a contour tile is drawn from
+its own tile plus its eight neighbours: **roughly nine merged terrain tiles
+each**, asked for together rather than in turn.
+
+Every one of those is a tile some neighbouring contour tile also wants, so a
+cache in front of the merge is what decides whether a run of these is affordable
+— not an optimisation to add later. Baking deliberately bypasses the merged-tile
+cache, which is right for terrain, where each tile is written once and never
+wanted again. It is not right here.
+
+### How far apart the lines go
+
+Per zoom, because one interval is wrong at both ends of a map: at z8 a 20 m
+contour is a band of ink, and at z15 a 500 m one is a blank tile through most of
+the world. A recipe says either a number, meaning that interval wherever
+contours are drawn at all, or a table of zoom to intervals. Saying nothing gets
+a built-in table.
+
+A level may name more than one interval. `[100, 500]` draws a line every hundred
+metres and marks every fifth, and each feature carries `level` — how many of the
+intervals its height divides by, so 500 outranks 100. A style reads that to draw
+the major lines thicker and label only those, from one layer rather than two
+passes. The convention is `maplibre-contour`'s, so a style written against its
+tiles works against these.
+
+A zoom the table skips reads as the entry above it: a table naming 12 and 14
+means 12 and 13 share a setting. Below the shallowest entry nothing is drawn at
+all, and that is checked _before_ the nine merges — at z5 a tile is most of a
+continent, and nine merges is an expensive way to answer nothing.
+
+### What the endpoint claims
+
+The zoom range is the thresholds', not the stack's. A stack serving z0–z16 draws
+no contours at z2, and a client told otherwise fetches empty tiles all the way
+down. It is never deeper than the stack has ground for either: a contour traced
+from an upscaled parent is the parent's line drawn twice as thick, not new
+detail.
+
+### Why the encoding is written here
+
+`maplibre-contour` has a vector tile encoder and does not export it, and
+`vt-pbf` — the obvious dependency — is built against `pbf` 3 while this tree
+resolves `pbf` 5, whose `Pbf` default export no longer exists. So `contour-mvt.js`
+writes the tile directly against `pbf` 5. It is narrow enough to be worth
+owning: one layer of line strings with two numeric properties, where a general
+encoder carries polygons, points, mixed property types and many layers.
+
+Heights are written in ascending order, so two runs over the same contours
+produce the same bytes whatever order the tracer closed its fragments in — which
+is what lets an export be resumed and a tile be keyed by content. A tile no
+contour crossed is no tile at all rather than an empty layer, which a client
+would pay to fetch and draw nothing from.
+
+`maplibre-contour` itself is loaded through `createRequire`: the published 0.1.0
+declares no `import` condition, so `import 'maplibre-contour'` fails outright.
+The fix is in upstream main and unreleased. A git dependency would push that
+requirement onto everyone installing this package, so the `require` path is
+taken instead, and becomes an ordinary import when a release carries the fix.
+
 ## Merging vector sources
 
 **Not built.** This is the design, written down before anything is started so

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.100.0",
+  "version": "0.102.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",
@@ -46,8 +46,10 @@
     "create-torrent": "^6.1.0",
     "express": "^5.2.1",
     "fs-chunk-store": "^5.0.1",
+    "maplibre-contour": "^0.1.0",
     "maplibre-gl": "^6.2.0",
     "parse-torrent": "^11.0.24",
+    "pbf": "^5.1.2",
     "pmtiles": "^4.4.1",
     "pmtiles-torrent": "^0.11.1",
     "webtorrent": "^3.0.21"

package/src/api.js

@@ -57,6 +57,13 @@ import {
   stacksAffectedBy,
 } from './stacks.js';
 import { bakeRevision } from './bake.js';
+import zlib from 'node:zlib';
+import {
+  contourTile,
+  heightsFromArchive,
+  heightsFromStack,
+} from './contour-tile.js';
+import { contourProblems } from './contour-options.js';
 
 const here = path.dirname(fileURLToPath(import.meta.url));
 
@@ -251,6 +258,7 @@ export function createApp({
   tiles,
   stacks,
   stackCache,
+  heightsCache,
   cutlines,
   bakes,
   warm,
@@ -3528,6 +3536,15 @@ export function createApp({
           publicDownload: req.body?.publicDownload,
           keep: req.body?.keep,
           keepDays: req.body?.keepDays,
+          // Which part of the stack to write. Absent is all of it, which is
+          // what an export has always meant. See src/bake-selection.js.
+          minzoom: req.body?.minzoom,
+          maxzoom: req.body?.maxzoom,
+          bounds: req.body?.bounds,
+          // And what to make of it. Terrain unless asked otherwise, so
+          // nothing already written has to start saying so.
+          kind: req.body?.kind,
+          thresholds: req.body?.thresholds,
         });
         return res.status(202).json({ bake: job });
       } catch (error) {
@@ -3952,6 +3969,56 @@ export function createApp({
    * @param {number} y - Row.
    * @returns {void}
    */
+  /**
+   * Gzips a vector tile, the way the archive tile route does.
+   * @param {Buffer} data - The tile.
+   * @returns {Promise<Buffer>} - The compressed tile.
+   */
+  const gzip = (data) =>
+    new Promise((resolve, reject) =>
+      zlib.gzip(data, (error, out) => (error ? reject(error) : resolve(out))),
+    );
+
+  /**
+   * What contour intervals to draw a stack at.
+   *
+   * The recipe's, unless the request asked for something else. A query
+   * parameter rather than only a recipe field because contours are a view of a
+   * stack rather than a property of one -- the same terrain is wanted at 10 m
+   * on a walking map and 100 m on an atlas, and neither is the stack's
+   * business.
+   * @param {object} stack - The stack definition.
+   * @param {object} [query] - The request's query parameters.
+   * @returns {number|object|undefined} - What the recipe or the caller named.
+   */
+  const contourThresholdsFor = (stack, query = {}) => {
+    const asked = query.interval ?? query.thresholds;
+    if (asked !== undefined && asked !== '') {
+      const flat = Number(asked);
+      if (Number.isFinite(flat)) return flat;
+      try {
+        return JSON.parse(asked);
+      } catch {
+        // Left to `contourProblems` to refuse and name, rather than answering
+        // a different question than the one that was asked.
+        return asked;
+      }
+    }
+    return stack.contours?.thresholds;
+  };
+
+  /**
+   * A short, stable tag for one set of thresholds.
+   * @param {number|object} [thresholds] - What is being drawn.
+   * @returns {string} - Twelve hex characters.
+   */
+  const contourTag = (thresholds) =>
+    crypto
+      .createHash('sha1')
+      .update(JSON.stringify(thresholds ?? null))
+      .digest('hex')
+      .slice(0, 12);
+
   const stackHeaders = (res, resolved, contributors, z, x, y) => {
     res.setHeader('access-control-allow-origin', '*');
     res.setHeader('x-stack-sources', contributors.join(', '));
@@ -4099,6 +4166,228 @@ export function createApp({
     return res.send(answer.body);
   });
 
+  /**
+   * Contour lines traced from what a stack merges.
+   *
+   * Its own endpoint rather than a format the tile route answers in, because
+   * it is a different map: vector where that is raster, drawn at the zooms the
+   * thresholds name rather than the ones the stack has ground for, and costing
+   * nine merges a tile rather than one. Sharing a route would mean one set of
+   * headers describing two things.
+   *
+   * See docs/tile-stacks.md -- "Contours from a stack".
+   */
+  /**
+   * Answers one contour tile, for whatever produced the resolution.
+   *
+   * A stack, one archive, or whichever build a category currently points at.
+   * The three differ only in where the heights come from and what identifies
+   * them, so both arrive as functions and everything else -- the coordinate
+   * checks, the codec refusal, the abort, the headers -- is written once.
+   * @param {import('express').Request} req - The request.
+   * @param {import('express').Response} res - The response.
+   * @param {object} options - `heightsAt`, `tag`, `thresholds`, `immutable`.
+   * @returns {Promise<void>} - Resolves once answered.
+   */
+  const answerContours = async (req, res, options = {}) => {
+    // Always: contours are traced from heights, so every source has to be
+    // decoded whatever would otherwise have been passed through.
+    const codec = await loadCodec();
+    if (!codec) {
+      return res.status(501).json({
+        error:
+          'tracing contours means decoding pixels, and this node has no codec',
+        hint: 'npm install sharp',
+      });
+    }
+
+    const z = Number(req.params.z);
+    const x = Number(req.params.x);
+    const y = Number(req.params.y);
+    if (![z, x, y].every(Number.isInteger)) {
+      return res.status(400).json({ error: 'z, x and y must be integers' });
+    }
+    const limit = 2 ** z;
+    if (z < 0 || z > 26 || x < 0 || y < 0 || x >= limit || y >= limit) {
+      return res.status(400).json({ error: 'tile coordinates out of range' });
+    }
+
+    const thresholds = options.thresholds;
+    const wrong = contourProblems({ thresholds });
+    if (wrong.length) {
+      return res.status(400).json({ error: wrong.join('; ') });
+    }
+
+    const controller = new AbortController();
+    // Nine merges deep, so an abandoned request here is nine abandoned reads
+    // per source. A panning map does that constantly.
+    res.on('close', () => {
+      if (!res.writableEnded) controller.abort();
+    });
+
+    let tile;
+    try {
+      tile = await contourTile({
+        heightsAt: options.heightsAt(codec, controller.signal),
+        z,
+        x,
+        y,
+        thresholds,
+      });
+    } catch (error) {
+      if (error?.name === 'AbortError') return;
+      throw error;
+    }
+
+    res.setHeader('access-control-allow-origin', '*');
+    res.setHeader('x-stack-sources', 'contours');
+    // The thresholds decide the bytes as much as the ground does, so they go
+    // in the tag. Without that a style asking for 50 m lines is answered from
+    // a cache holding 100 m ones.
+    res.setHeader(
+      'etag',
+      `W/"${options.tag(z, x, y)}-${contourTag(thresholds)}"`,
+    );
+    // Pinned content can never change; a category moves when its newest build
+    // does, and a stack when its recipe does. The same split the tile routes
+    // make, for the same reason.
+    res.setHeader(
+      'cache-control',
+      options.immutable
+        ? 'public, max-age=31536000, immutable'
+        : 'public, max-age=300, must-revalidate',
+    );
+    // Nothing crossed a threshold here. 404 rather than an empty tile, which
+    // a client pays to fetch and draws nothing from.
+    if (!tile) return res.status(404).end();
+
+    res.type('application/x-protobuf');
+    res.setHeader('content-encoding', 'gzip');
+    return res.send(await gzip(tile));
+  };
+
+  /**
+   * Whether an archive's tiles are heights at all.
+   *
+   * The same test the console uses to decide whether to offer a terrain
+   * preview: an archive naming no encoding is a picture, and tracing a
+   * picture's channels as though they were a number produces lines that mean
+   * nothing rather than an error.
+   * @param {object} entry - A catalog entry.
+   * @returns {boolean} - Whether contours can be traced from it.
+   */
+  const tracesAsTerrain = (entry) =>
+    ['terrarium', 'mapbox', 'custom'].includes(entry?.pmtiles?.encoding);
+
+  /**
+   * Contour lines traced from what a stack merges.
+   *
+   * Its own endpoint rather than a format the tile route answers in, because
+   * it is a different map: vector where that is raster, drawn at the zooms the
+   * thresholds name rather than the ones the stack has ground for, and costing
+   * nine merges a tile rather than one. Sharing a route would mean one set of
+   * headers describing two things.
+   *
+   * See docs/tile-stacks.md -- "Contours from a stack".
+   */
+  const serveStackContours = route(async (req, res) => {
+    await stacks?.refresh();
+    const resolved = stackOr404(req, res);
+    if (!resolved) return;
+    if (resolved.stack.space === 'rgba') {
+      return res.status(400).json({
+        error: 'this stack is imagery, and a colour has no contours to trace',
+      });
+    }
+    return answerContours(req, res, {
+      thresholds: contourThresholdsFor(resolved.stack, req.query),
+      immutable: isPinned(resolved),
+      tag: (z, x, y) => stackEtag(resolved, z, x, y).slice(1, -1),
+      heightsAt: (codec, signal) =>
+        heightsFromStack({
+          resolved,
+          tiles,
+          codec,
+          cutlines,
+          signal,
+          heightsCache,
+        }),
+    });
+  });
+
+  /** Contours traced from one archive, which is a stack of one source. */
+  const serveArchiveContours = route(async (req, res) => {
+    const entry = catalog.get(req.params.infoHash);
+    if (!entry) return res.status(404).json({ error: 'unknown archive' });
+    if (!tracesAsTerrain(entry)) {
+      return res.status(400).json({
+        error:
+          'this archive does not say its pixels are heights, so there is ' +
+          'nothing to trace contours from',
+        hint: 'an archive states that as `encoding` in its metadata',
+      });
+    }
+    return answerContours(req, res, {
+      thresholds: contourThresholdsFor({}, req.query),
+      // An infohash names one build and can never come to mean another.
+      immutable: true,
+      tag: (z, x, y) => `${entry.infoHash.slice(0, 20)}-${z}/${x}/${y}`,
+      heightsAt: (codec, signal) =>
+        heightsFromArchive({ entry, tiles, codec, signal, heightsCache }),
+    });
+  });
+
+  /** The same, for whichever build a category currently points at. */
+  const serveCategoryContours = route(async (req, res) => {
+    const entry = newestIn(req.params.category, req);
+    if (!entry) {
+      return res.status(404).json({ error: 'nothing is in that category' });
+    }
+    if (!tracesAsTerrain(entry)) {
+      return res.status(400).json({
+        error:
+          "this category's newest build does not say its pixels are heights",
+      });
+    }
+    return answerContours(req, res, {
+      thresholds: contourThresholdsFor({}, req.query),
+      // A category moves when a newer build lands, so it revalidates.
+      immutable: false,
+      // Tagged by the build it resolved to rather than by the category, so a
+      // rebuild lands as a new tag rather than as the same one with different
+      // lines behind it.
+      tag: (z, x, y) => `${entry.infoHash.slice(0, 20)}-${z}/${x}/${y}`,
+      heightsAt: (codec, signal) =>
+        heightsFromArchive({ entry, tiles, codec, signal, heightsCache }),
+    });
+  });
+
+  /**
+   * The three ways in. A contour tile is a vector tile whatever produced it,
+   * so the extension is checked once and in one place.
+   * @param {Function} handler - What answers it.
+   * @returns {Function} - An express handler.
+   */
+  const contourRoute = (handler) => (req, res, next) => {
+    if (!['pbf', 'mvt'].includes(req.params.ext)) {
+      return res.status(400).json({ error: 'contours are served as pbf' });
+    }
+    return handler(req, res, next);
+  };
+
+  app.get(
+    '/stacks/:id/contours/:z/:x/:y.:ext',
+    contourRoute(serveStackContours),
+  );
+  app.get(
+    '/archives/:infoHash/contours/:z/:x/:y.:ext',
+    contourRoute(serveArchiveContours),
+  );
+  app.get(
+    '/latest/:category/contours/:z/:x/:y.:ext',
+    contourRoute(serveCategoryContours),
+  );
+
   app.get('/stacks/:id/:size/:z/:x/:y.:ext', (req, res, next) => {
     // Anything that is not a size this serves is not a size at all -- and
     // must not fall through to the shorter shape, where it would be read as a