pmtiles-swarm 0.67.0 → 0.68.1

package/CHANGELOG.md

@@ -4,6 +4,99 @@
 ### ✨ Features and improvements
 - _...Add new stuff here..._
 
+### 🐞 Bug fixes
+- _...Add new stuff here..._
+
+## 0.68.1
+### ✨ Features and improvements
+- **A cutline may have holes in it, which is to say it may be a real boundary.** 0.68.0 refused
+  any shape with interior rings, on the reasoning that the even-odd fill would treat a hole as
+  solid. It does not — a ray to a point inside a hole crosses the outer ring and then the inner
+  one, which is two crossings, which is outside — so the refusal protected against nothing and
+  turned away almost every boundary worth clipping to. A country is islands and enclaves and
+  lakes; the German cutline this was written for is ninety-three rings.
+
+  Tested rather than reasoned about, because getting it wrong fills in exactly the ground somebody
+  cut out.
+
+  The documentation now also says how a shapefile becomes a cutline, since that is where every
+  real one starts: one `ogr2ogr -t_srs EPSG:4326`, and the filename is the name the recipe uses.
+  With a note to simplify — a boundary drawn for surveying carries far more detail than a tile can
+  show, and every point of it is a segment to index and test.
+
+
+
+### 🐞 Bug fixes
+- _...Add new stuff here..._
+
+## 0.68.0
+### ✨ Features and improvements
+- **A source can be clipped to a shape.** `cutline` names a polygon kept under `data/cutlines/`;
+  `bounds` is a rectangle written straight into the recipe. Both are chosen in the stack editor,
+  which offers the shapes this node actually has rather than asking anybody to remember a
+  filename.
+
+  Most of the time this is already solved a step earlier, and the design says so: a build running
+  `gdalwarp -cutline … -dstnodata` writes the boundary into the archive, and `maskValues` removes
+  exactly those pixels. What that cannot reach is a source **this node did not build** — an
+  archive is content-addressed, so re-clipping one means republishing it. That is the case this is
+  for, and it is the federated case this project exists for.
+
+  **A rectangle is a cutline with four corners**, built through the same code rather than beside
+  it, so there is no second implementation to disagree with the first. A test asserts a `bounds`
+  and the same shape drawn as GeoJSON classify identically across a whole zoom level.
+
+  The cost is avoided rather than paid. Every tile is first classified **outside**, **inside** or
+  **partial**: outside skips the read entirely — no swarm round trip, no decode, no merge — inside
+  skips the mask entirely, and only a tile the edge actually crosses is rasterised, by scanline,
+  over segments found through a grid index built once when the cutline loads. For a country
+  boundary that is a band one tile wide; everything else is settled without touching a pixel.
+
+  It composes with the per-tile short-circuit: a clipped source still hands its bytes through
+  untouched where the tile is wholly inside, because there the clip provably changes nothing.
+
+  **A cutline a recipe names and this node has not got refuses the source.** Not "serve it
+  unclipped" — that would put back exactly the data somebody asked to remove, which is the one
+  failure a clip must not have. The stack reports it beside its other problems and the editor says
+  so on the row.
+
+  Also fixed while building it: horizontal edges were being dropped when a shape was prepared, on
+  the grounds that they contribute nothing to the even-odd rule. True, and they are still edges —
+  dropping them meant nothing noticed a rectangle's north and south sides crossing a tile, which
+  read as `inside` for a tile half of which was outside, and the mask was then never applied.
+
+
+- **A stack that merges somewhere no longer merges everywhere.** Passthrough was decided per
+  recipe: one source with a mask made the whole stack a merging one, and every tile was decoded
+  and re-encoded — including the great majority where a single untouched source covers the ground
+  and its stored bytes were already the answer. `passThroughRead` now decides that per tile,
+  before anything is decoded, which is the point: checked afterwards it would save the encode and
+  not the decode.
+
+  Its own function with seventeen tests, one per condition, because a short-circuit that fires
+  when it should not does not fail — it serves the wrong pixels quietly, and an archive baked from
+  them is wrong the same way.
+
+  Masks are the subtle half, and the reason it refuses any source carrying one. A tile having a
+  single contributor does not make that contributor cover the tile: a mask turns pixels into
+  nodata, the merge fills those, and handing the stored bytes over instead would show the ground
+  the mask was there to remove. With masks refused, nodata has nowhere else to come from —
+  `decodeHeights` is arithmetic over bytes, and the only other sources of it are the parent
+  resample and a resize, both refused as well.
+
+- **The catalogue page lists stacks.** A stack has no infohash and appears in no feed, so nothing
+  about it was discoverable: the only way to know a node served one was to be told its id.
+  `GET /stacks/` answers the list on the public listener, beside the tiles and TileJSON it
+  describes, and the page renders each with its TileJSON, XYZ template and preview — terrain
+  first where the stack is terrain.
+
+  Not the console's list. That one names what each source resolved to and what is missing, which
+  is the operator's view and names infohashes a visitor was never offered. The public one also
+  leaves out any stack it cannot serve — a recipe with a problem, or one wanting a codec this node
+  has not got — because a link that answers 501 is worse than no link.
+
+
+
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 

package/docs/tile-stacks.md

@@ -49,6 +49,8 @@ of its parts.
   - [What identifies a bake](#what-identifies-a-bake)
   - [Starting one, and watching it](#starting-one-and-watching-it)
   - [What exists now](#what-exists-now)
+- [Clipping a source to a shape](#clipping-a-source-to-a-shape)
+- [Finding a stack](#finding-a-stack)
 - [Syncing a stack to another node](#syncing-a-stack-to-another-node)
 - [What the offline merge got wrong](#what-the-offline-merge-got-wrong)
 - [The stack editor](#the-stack-editor)
@@ -297,6 +299,25 @@ For a request `GET /stacks/<id>/{z}/{x}/{y}.webp`:
    its bytes through untouched. No decode, no encode. This is what a stack
    degenerates to over most of the world when the top layer is dense, and it
    costs a comparison to detect.
+
+   `passThroughRead` in `src/stack-tile.js`, checked _before_ anything is
+   decoded — checked afterwards it would save the encode and not the decode.
+   Its own function with its own tests, because a short-circuit that fires when
+   it should not does not fail: it serves the wrong pixels quietly, and an
+   archive baked from them is wrong the same way.
+
+   The masks are the subtle half. A tile having one contributor does not make
+   that contributor cover the tile: a mask turns pixels into nodata, the merge
+   fills those, and passing the stored bytes through instead would show the
+   ground the mask was there to remove. Refusing any source carrying a mask is
+   what makes the rest safe — nodata has nowhere else to come from, since
+   `decodeHeights` is arithmetic over bytes and the only other sources of it
+   are the parent resample and a resize, both refused as well.
+
+   An explicit output size is refused for a different reason: the source's
+   pixel width is not known until it is decoded, so a resize cannot be ruled
+   out from here.
+
 6. **Decode** each contributing tile in a worker.
 7. **Mask** to a coverage mask, then apply `heightAdjustment` (elevation) or
    `opacity` (rgba). Masking comes first — the mask values are exact decoded
@@ -793,6 +814,154 @@ means re-hashing everything already buffered, and the cost of starting it empty
 is that a tile identical to one from before the interruption is stored twice.
 The archive is correct either way; it is a little larger.
 
+## Clipping a source to a shape
+
+A source is often only meant to apply inside a boundary — a national DEM inside
+its country, a survey inside its extent — and everywhere else the layer beneath
+it should show through.
+
+**Most of the time this is already solved, one step earlier.** A build that runs
+`gdalwarp -cutline … -dstnodata` writes the boundary into the archive: outside
+it, every pixel is the nodata value. `maskValues` then removes exactly those,
+which is what the offline merge configs do and what a stack recipe does with the
+same field. Nothing more is needed, and a cutline that duplicated it would be
+slower and no more correct.
+
+What that cannot reach is a source **this node did not build**. An archive is
+content-addressed, so re-clipping one means republishing it — impossible for
+somebody else's, and expensive for a large one whose boundary has changed. That
+is the case this is for, and it is squarely the federated case this project
+exists for.
+
+Two shapes, and one of them is nearly free:
+
+```json
+{ "category": "opendtm-de", "cutline": "germany" }
+{ "category": "massgis", "bounds": [-73.6, 41.1, -69.8, 42.9] }
+```
+
+`bounds` is a rectangle in WGS84, written the way every other bounds in this
+project is. `cutline` names a polygon. They are the same question asked of
+different shapes, so they are the same code: a rectangle is a cutline with four
+corners, and treating it as one means there is no second implementation to
+disagree with the first. What differs is only that a rectangle needs no file, no
+index and no rasterising worth the name — which is why it is worth having even
+though the polygon subsumes it.
+
+### Geometry by reference
+
+A recipe names a cutline rather than carrying it. `"cutline": "germany"`
+resolves to `data/cutlines/germany.geojson`, in WGS84. A recipe stays a small
+document that a person can read and a peer could one day be sent; a megabyte of
+coordinates inlined in one is neither. A rectangle is small enough to sit in the
+recipe itself, so `bounds` does.
+
+WGS84 on purpose. Web Mercator is arithmetic from there — no projection library,
+no dependency, no CRS handling beyond refusing what is not WGS84.
+
+### Getting a shapefile in
+
+A cutline usually starts life as a shapefile in a projected CRS, because that is
+what `gdalwarp -cutline` wants. One `ogr2ogr` converts it, and it is the same
+tool that produced the shapefile in the first place:
+
+```sh
+ogr2ogr -t_srs EPSG:4326 \
+  data/cutlines/germany.geojson \
+  datasets/OpenDTM_DE/cutline/germany_cutline_25832.shp
+```
+
+The name of the file is the name the recipe uses, so that one becomes
+`"cutline": "germany"`. Nothing has to restart: cutlines are read when the node
+starts and the console lists whatever is there.
+
+**Simplify it.** A national boundary drawn for surveying carries far more detail
+than a tile can show — Germany's is ninety-three rings and sixty-five thousand
+points in its first record alone — and every one of them is a segment to index
+and to test. What matters is being right to about a pixel at the deepest zoom
+served, which at z14 is roughly five metres:
+
+```sh
+ogr2ogr -t_srs EPSG:4326 -simplify 0.0001 out.geojson in.shp
+```
+
+Holes and islands are kept, and want to be: a country is islands and enclaves
+and lakes. Under the even-odd rule an interior ring needs no special handling
+and gets none — a ray to a point inside one crosses the outer ring and then the
+inner ring, which is two crossings, which is outside.
+
+Edges are treated as straight lines in Mercator after their endpoints are
+projected. That is what every rasteriser does, GDAL's included, and the error
+over a tile's span is far below a pixel except at zooms where a tile spans a
+continent — where a cutline is not the thing deciding the answer anyway.
+
+### Three answers, and only one of them costs anything
+
+The cost of a clip is per pixel, and paying it per tile would make a continental
+boundary unusable. It is avoided by asking a cheaper question first:
+
+- **Outside.** No part of the tile is within the shape. The source contributes
+  nothing — and this is decided _before the tile is read_, so it costs no swarm
+  read, no decode and no merge.
+- **Inside.** The tile is wholly within the shape. The clip cannot change any
+  pixel, so it is not applied at all.
+- **Partial.** Only here is a mask rasterised, and only for the tile in hand.
+
+For a country boundary at any useful zoom, almost every tile is one of the first
+two. The third is a band one tile wide along the border.
+
+Classification is a bounding-box test, then a look at whichever edges could
+reach the tile. "Whichever" is the important word: a national boundary is tens
+of thousands of segments, and testing all of them per tile would cost more than
+the rasterising it is meant to avoid. The segments are indexed into a coarse
+grid once, when the cutline is loaded, and a tile only ever looks at the buckets
+it overlaps.
+
+With no edge near the tile, one point decides it: a tile that no boundary
+crosses is entirely on one side of it.
+
+### Where it applies
+
+As a coverage mask on the decoded heights, in the same place and for the same
+reason as `maskValues` — before any height adjustment, because an adjustment
+would shift values out from under a comparison, and because the answer is the
+same either way: nothing here.
+
+It applies at the geometry of the tile that was **asked for**, not of the tile
+that answered. A source falling back to a parent still gets clipped to the
+square being served, which is the ground the pixels will end up covering.
+
+### What it costs the short-circuit
+
+A clipped source cannot take the passthrough in
+[Evaluating one tile](#evaluating-one-tile) — with one exception that is worth
+having, because it is the common one. Where the tile is classified **inside**,
+the clip provably changes nothing, and the bytes may go through untouched
+exactly as if none were named. Outside, there is no contribution at all, so the
+question does not arise. Only a partial tile is genuinely disqualified.
+
+### Refusing what cannot be honoured
+
+A cutline named by a recipe and not present on disk is a stack problem, reported
+the way an unresolvable source is: the stack is listed, and it says what is
+missing. Serving a source unclipped because its cutline could not be found would
+put back exactly the data somebody asked to remove, which is the one failure a
+clip must not have.
+
+## Finding a stack
+
+A stack has no infohash and appears in no feed, so nothing about it is
+discoverable the way an archive is. `GET /stacks/` answers the list, on the
+public listener beside the tiles and the TileJSON it describes, and the
+catalogue page renders it.
+
+Not the console's list. That one reports what each source resolved to, what is
+missing and what cannot be served — the operator's view, naming infohashes a
+visitor was never offered. The public one says what a visitor can point a map
+at, and says nothing at all about a stack they cannot: a recipe with a problem,
+or one asking for pixel work this node has no codec for, is left out rather than
+advertised as a link that answers 501.
+
 ## Syncing a stack to another node
 
 The question is whether a stack can travel between nodes the way an archive
@@ -903,7 +1072,9 @@ state says what a stack is and how to add one instead. It is not conditioned on
 having a codec either — a passthrough stack needs none, and a node without
 `sharp` can still build and serve one.
 
-What follows is the editing half, which does not exist yet.
+What follows is the editing half, which the console now does: a stack can be
+added, changed and removed there, and the source rows are edited in the order
+the file holds them.
 
 ### The list is shown in the file's order
 
@@ -1103,7 +1274,7 @@ handling whatsoever.
     level up, and `resampleFromParent` crops the right sub-square.
   - **Source smaller than the output** (256px source, 512px stack) needs the
     other direction: fetch the four children at `z+1` and assemble them into
-    one grid before merging. That is the piece that does not exist yet.
+    one grid before merging.
 
   **Both are implemented.** A size may be asked for in the URL —
   `/stacks/<id>/512/{z}/{x}/{y}.webp`, the way tileserver-gl takes one — or set

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.67.0",
+  "version": "0.68.1",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -36,7 +36,7 @@ import { buildTileJson, extensionMatches, tileExtension } from './tilejson.js';
 import { SUMMARY_VERSION } from './pmtiles-probe.js';
 import { TileReadError } from './tiles.js';
 import { loadCodec } from './codec.js';
-import { answerStackTile, outputSize } from './stack-tile.js';
+import { answerStackTile, outputFormat, outputSize } from './stack-tile.js';
 import {
   isPinned,
   needsCodec,
@@ -215,6 +215,7 @@ export function createApp({
   tiles,
   stacks,
   stackCache,
+  cutlines,
   bakes,
   warm,
   config,
@@ -2264,6 +2265,66 @@ export function createApp({
     }),
   );
 
+  // The same idea one prefix over. Everything under /stacks/ is already public
+  // -- the TileJSON, the tiles, the preview -- so the index of what is on
+  // offer belongs beside them rather than behind the console's door.
+  //
+  // Not the console's list. That one reports what each source resolved to,
+  // what is missing and what cannot be served, which is the operator's
+  // business and names infohashes a visitor was not offered. This says what a
+  // visitor can point a map at, and says nothing about a stack they cannot.
+  app.get(
+    '/stacks/',
+    route(async (req, res) => {
+      await stacks?.refresh();
+      res.setHeader('access-control-allow-origin', '*');
+
+      const base = baseUrl(req);
+      const listed = [];
+      for (const stack of stacks?.list() ?? []) {
+        const resolved = resolveFor(stack, req);
+        // A recipe with a problem, or one asking for pixel work this node
+        // cannot do, is not something to advertise: a link that answers 501 is
+        // worse than no link.
+        if (stacks.problems(stack.id).length > 0) continue;
+        if (needsCodec(stack) && !(await loadCodec())) continue;
+        if (!resolved.sources.some((source) => source.entry)) continue;
+
+        const coverage = stackCoverage(resolved);
+        const format = outputFormat(resolved);
+        const id = encodeURIComponent(stack.id);
+        listed.push({
+          id: stack.id,
+          title: stack.title ?? stack.id,
+          space: stack.space ?? 'elevation',
+          minzoom: coverage.minzoom,
+          maxzoom: coverage.maxzoom,
+          bounds: coverage.bounds,
+          format,
+          encoding: stack.output?.encoding ?? null,
+          sparse: stack.sparse ?? true,
+          sources: resolved.sources.length,
+          endpoints: {
+            tileJson: `${base}/stacks/${id}/tiles.json`,
+            xyz: `${base}/stacks/${id}/{z}/{x}/{y}.${tileExtension(format)}`,
+            preview: `${base}/stacks/${id}/preview`,
+          },
+        });
+      }
+
+      res.setHeader(
+        'etag',
+        `"${crypto.createHash('sha1').update(JSON.stringify(listed)).digest('hex')}"`,
+      );
+      res.setHeader('cache-control', 'public, max-age=60, must-revalidate');
+      res.json({
+        format: 'pmtiles-swarm-stacks/1',
+        generatedAt: new Date().toISOString(),
+        stacks: listed,
+      });
+    }),
+  );
+
   // A stable handle for "the current one". Every archive is addressed by
   // infohash, which is right — it is what makes a tile immutable — but it
   // leaves nothing for a style to point at that survives a rebuild. A category
@@ -3083,6 +3144,22 @@ export function createApp({
    * @param {import('express').Request} req - The request, for its credential.
    * @returns {object} - The resolution.
    */
+  /**
+   * Cutlines a recipe names that this node does not have.
+   *
+   * Serving a source unclipped because its shape could not be found would put
+   * back exactly the data somebody asked to remove, so the stack says so and
+   * the tile path refuses the source.
+   * @param {object} stack - The recipe.
+   * @returns {string[]} - One line per missing cutline.
+   */
+  const cutlineProblems = (stack) =>
+    (stack.sources ?? []).flatMap((source, index) => {
+      if (!source?.cutline) return [];
+      const why = cutlines?.problem(source.cutline);
+      return why ? [`sources[${index}].cutline: ${why}`] : [];
+    });
+
   const resolveFor = (stack, req) =>
     resolveStack(stack, {
       archive: (hash) => catalog.get(hash) ?? null,
@@ -3108,7 +3185,11 @@ export function createApp({
           id: stack.id,
           title: stack.title ?? stack.id,
           space: stack.space ?? 'elevation',
-          problems: stacks.problems(stack.id),
+          // The store's own problems, plus any cutline a source names that
+          // this node has not got. Both stop the stack being servable and
+          // both belong in the same list -- a stack listed as fine that
+          // refuses every tile is worse than one that says why.
+          problems: [...stacks.problems(stack.id), ...cutlineProblems(stack)],
           // Named separately from `problems` because it is not a mistake: a
           // recipe asking for pixel work is perfectly valid and simply cannot
           // be served yet. See docs/tile-stacks.md — "The codec problem".
@@ -3154,7 +3235,13 @@ export function createApp({
       // that need it, rather than leaving a 501 to be discovered at the first
       // tile. Null is a fact about this node, not about the recipes.
       const codec = await loadCodec();
-      res.json({ stacks: list, codec: codec?.name ?? null });
+      res.json({
+        stacks: list,
+        codec: codec?.name ?? null,
+        // So the editor can offer the shapes this node actually has rather
+        // than asking somebody to remember a filename.
+        cutlines: cutlines?.list() ?? [],
+      });
     }),
   );
 
@@ -3541,6 +3628,7 @@ export function createApp({
         signal: controller.signal,
         size,
         format,
+        cutlines,
       });
     } catch (error) {
       // The client went away mid-merge. Nothing to answer and nobody to answer

package/src/bake-jobs.js

@@ -35,16 +35,18 @@ export class BakeManager {
   #tiles;
   #config;
   #loadCodec;
+  #cutlines;
   #jobs = new Map();
 
   /**
    * @param {object} deps - The library, the tile store, the config and the codec probe.
    */
-  constructor({ library, tiles, config, loadCodec }) {
+  constructor({ library, tiles, config, loadCodec, cutlines }) {
     this.#library = library;
     this.#tiles = tiles;
     this.#config = config;
     this.#loadCodec = loadCodec;
+    this.#cutlines = cutlines;
   }
 
   /**
@@ -273,6 +275,7 @@ export class BakeManager {
         tiles: this.#tiles,
         codec,
         pixels,
+        cutlines: this.#cutlines,
         signal: job.controller.signal,
         format,
       }),

package/src/bake.js

@@ -389,7 +389,7 @@ export function tileTypeFor(format) {
  * @returns {Function} - `(z, x, y) => Promise<Buffer|null>`.
  */
 export function mergeTileFor(options) {
-  const { resolved, tiles, codec, signal, pixels } = options;
+  const { resolved, tiles, codec, signal, pixels, cutlines } = options;
   const format = options.format ?? outputFormat(resolved);
   const size = options.size ?? outputSize(resolved.stack);
 
@@ -406,6 +406,7 @@ export function mergeTileFor(options) {
       size,
       format,
       pixels,
+      cutlines,
     });
 
     // A required source that cannot be read stops the job. Baking around it