pmtiles-swarm 0.91.0 → 0.92.0

package/CHANGELOG.md

@@ -7,6 +7,27 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.92.0
+### 🐞 Bug fixes
+- **A stack's TileJSON published the addresses its sources are read from.** A URL source is named by
+  its URL and an S3 source by its bucket and key, and the document naming them is served to anybody
+  who can load the map — while the archives at the other end are read with credentials nobody else
+  has. The sources are gone from it entirely, which was the right answer anyway: they are the
+  ingredients of one endpoint, not a list for a client to join, and listing them invited fetching
+  those instead of the tiles. What is left is a count and a `revision` fingerprint covering the
+  recipe and what every source resolved to — the one thing the list was good for, in twenty bytes
+  rather than tens of kilobytes. The document for a 459-source stack is now 336 bytes.
+
+- **Signing in threw away the view you asked for.** A link to `#stacks` on a guarded node asked for a
+  password and then showed the archives, with the address still reading `#stacks` — which is what
+  made it look broken rather than like a redirect. Both ways into the console now land where the
+  address says.
+
+- **An archive the engine has no record of was drawn as 0%.** That is a different fact from "none of
+  it is here", and it is the more alarming one to get wrong: a library the engine failed to take back
+  after a restart read as a library that had lost its data. The row now says **not loaded**, and says
+  where the reason is logged and that nothing on disk has been touched.
+
 ## 0.91.0
 ### ✨ Features and improvements
 - **A source at a URL can be added by hand, not only imported.** **Add source → an address you

package/docs/tile-stacks.md

@@ -60,6 +60,7 @@ of its parts.
 - [Syncing a stack to another node](#syncing-a-stack-to-another-node)
 - [What the offline merge got wrong](#what-the-offline-merge-got-wrong)
 - [The stack editor](#the-stack-editor)
+- [Merging vector sources](#merging-vector-sources)
 - [Staging](#staging)
 - [Open questions](#open-questions)
 
@@ -541,21 +542,42 @@ Derived from the resolved sources rather than from any one archive:
 - `attribution` — every source's, concatenated.
 - `tiles` — `/stacks/<id>/{z}/{x}/{y}.<ext>`.
 
-Plus a non-standard `stack` block naming what it resolved to, mirroring the
-`latest` block on `/latest/<category>/tiles.json`, so a consumer can tell one
-resolution from the next:
+Plus a non-standard `stack` block, mirroring the `latest` block on
+`/latest/<category>/tiles.json`, so a consumer can tell one resolution from the
+next:
 
 ```jsonc
 "stack": {
   "id": "planet-terrain",
-  "revision": 7,
-  "sources": [
-    { "category": "gebco", "infohash": "a074186d…", "name": "GEBCO_2026_…" },
-    { "category": "planet-bathymetry", "infohash": "…", "name": "…" }
-  ]
+  "space": "elevation",
+  "sources": 2,
+  "revision": "9f2c1a77b3e04d16"
 }
 ```
 
+`revision` covers the recipe and what every source resolved to, so it moves
+when a category resolves to a new build — which is the whole reason the block
+exists.
+
+### The sources themselves are not in it
+
+They were once, and it was wrong twice over.
+
+A stack's sources are not a client's to join. They are the ingredients of one
+endpoint, and this document exists to point at that endpoint: a map reads
+`tiles`, not the archives behind it. Listing them invites somebody to fetch
+those instead, which is the one thing a stack is there to stop them having to
+do.
+
+And a source may be an **address**. A URL, or a bucket and a key, published in
+a document that is served to anybody who can load the map — while the archive
+at the other end of it is read with credentials nobody else has. The tiles are
+public on purpose; where they come from is not.
+
+What is left says the same thing the list said, in twenty bytes rather than
+tens of kilobytes: how many sources there are, and a fingerprint that moves
+when any of them does.
+
 ## When a source will not answer
 
 A cache-mode source with no reachable peers, a category that resolves to an
@@ -2199,6 +2221,135 @@ Debounced, and it should not follow the map past the stack's `maxzoom`: above
 that the client overzooms and there is nothing new to see, but the tiles are
 still requested and still composited.
 
+## Merging vector sources
+
+**Not built.** This is the design, written down before anything is started so
+that the hard part is on record rather than discovered halfway through it.
+
+Most of the machinery is indifferent to what a tile contains. Source
+resolution, per-source `minzoom`, `maxzoom` and `bounds`, painting order,
+nested stacks, the cache, the ETag, `unionOfTileIds` and the bake all treat a
+tile as an opaque thing, and the byte-for-byte passthrough already serves a
+vector archive today. What a vector space adds is a merge, and the merge is
+where the analogy with the two pixel spaces breaks.
+
+### What has no vector analogue
+
+`opacity`, `blend`, `feather`, `featherMetres`, `gaussianBlurSigma`,
+`output.encoding`, `output.tileSize`, `maskValues`, `maskRange` and
+`maskColors`. Every one of them is an operation on a pixel, and there are no
+pixels. A recipe naming any of them under a vector space should be **refused by
+validation**, not accepted and ignored: a stack that says `feather: 50` and
+draws a hard edge is a worse answer than one that will not save.
+
+The idea of a mask survives in three other forms — clip to a **shape**, keep or
+drop whole **layers** by name, and filter **features** by a property. Those
+three are the vector mask, and none of them is a mask in the sense the
+elevation space means.
+
+### Overlap is the whole problem
+
+The pixel spaces work because the unit is a pixel, every source has one
+everywhere, and "the topmost covered source wins" is a complete rule. Vector's
+unit is a feature. Two sources that both carry a `water` layer over the same
+ground hold **two polygons of the same lake**, and emitting both overwrites
+neither. The tile carries both, the style draws both, and it shows as doubled
+outlines, darker semi-transparent fills, duplicated labels, and z-fighting
+between them.
+
+So the first thing a vector space has to define is what winning means. Three
+answers, and the third is not a default:
+
+**Per-layer replacement.** For each layer _name_, the topmost source carrying it
+supplies it whole, and the same-named layer of every source below is dropped for
+that tile. No geometry work at all: read the layer list, choose, re-emit. It is
+the right answer for a base plus an overlay of layers the base does not have —
+an OSM base under your own `buildings`, or under contours generated from the
+terrain these stacks already merge — and it is correct for a base plus a patch
+wherever the tile falls wholly inside the patch.
+
+**Clip and union.** Each source is clipped to its own coverage, and the source
+below is additionally clipped _against_ the coverage of the one above it, so the
+two cannot both carry the same lake. This is the true counterpart of the pixel
+merge, and the only thing that works on a tile the boundary of a patch runs
+through. A **rectangle** clip in tile coordinates is cheap and exact —
+Sutherland-Hodgman for polygons, Liang-Barsky for lines — and a rectangle is
+what an imported file list states for every patch in it. An arbitrary
+**cutline** is a different matter and wants a real polygon-clipping library.
+
+**Renaming.** The patch's `water` becomes `patch_water`, and the style decides
+what to do with it. No conflict and no geometry work, at the cost of a style
+that has to know. Worth having as a way out; not worth having as the rule.
+
+The recommendation is per-layer replacement as the default, rectangle clipping
+for the base-and-patch case, and a **recipe warning** wherever two sources carry
+the same layer with no clip between them: _sources 0 and 1 both provide `water`,
+and tiles where they meet will carry both_. Silent doubling is the failure that
+gets found six months later by somebody squinting at a coastline.
+
+### A seam cannot be feathered
+
+Where two sources disagree geometrically at a boundary, a road jogs and a
+coastline steps, and there is nothing to fade. [Feathering a
+seam](#feathering-a-seam) works because a pixel can be a mixture of two values.
+A feature cannot be a mixture of two features.
+
+Vector merging is therefore clean only where the sources were cut to agree — the
+same condition that makes a provider's raster patches merge cleanly, minus the
+fade that hides the cases where it is not quite true. That belongs in the
+console rather than on the map.
+
+### Overzooming a parent
+
+Possible, and needed: a base to z12 under a patch to z14 has to climb the base
+at z13, exactly as [resampling from a parent](#resampling-from-a-parent-tile)
+does for pixels. Scale the parent's coordinates by the zoom difference, subtract
+the quadrant's offset, clip to the extent plus the buffer. Exact for geometry,
+and it invents nothing: the generalisation and the label placement were chosen
+for the parent's zoom, so an overzoomed tile is thinner than a real one at that
+zoom and looks it.
+
+### Extent, buffer and schema
+
+Sources at different **extents** — 4096 and 512 are both common — have to be
+normalised to one before anything is combined. A power-of-two ratio is exact;
+anything else quantises coordinates.
+
+**Buffers** have to be re-clipped after any coordinate change, or features leak
+past the edge of the tile and draw twice at the join.
+
+**Schemas** are the quiet one. Two sources whose `water` layers use `class` and
+`kind` merge into a layer the style half understands, and nothing in the merge
+can reconcile them. Worth comparing key sets on the first merged tile and
+reporting the disagreement as a problem on the stack, the way an unresolved
+source is reported.
+
+**Feature ids** collide across sources. Renumbering settles the collision and
+breaks any client-side feature state keyed on them, so it is a choice to make
+out loud rather than a detail to settle inside the encoder.
+
+### The libraries
+
+`@mapbox/vector-tile` and `pbf` to read, `vt-pbf` to write. Small, pure
+JavaScript, and ordinary dependencies rather than a probe: the optional
+treatment [the codec gets](#the-codec-problem) exists because a native build
+genuinely fails to install on some platforms, and none of that applies here.
+
+Not written here either, unlike the PMTiles writer and the S3 signer. A protobuf
+round-trip has more edge cases than either — geometry command encoding, unknown
+value types, key and value pools shared across features — and getting one wrong
+produces a tile that parses and is subtly wrong.
+
+### Staging, when it is started
+
+1. **Layer operations only.** `space: "vector"`, with keep, drop and rename by
+   layer name, over the per-source zoom and bounds skipping that already exists.
+   Covers base-plus-overlay, and needs no geometry code whatsoever.
+2. **Overzoom and rectangle clipping.** Covers base-plus-patch, which is the
+   shape [importing a list of URLs](#importing-a-list-of-urls) makes easy to
+   build.
+3. **Cutline clipping and property filters.**
+
 ## Staging
 
 1. ~~**Recipe and resolution.**~~ Done. `data/stacks.json`, load and validate,
@@ -2230,10 +2381,11 @@ handling whatsoever.
 
 ## Open questions
 
-- **Vector tiles.** Merging MVT layers from two archives is a real want and a
-  completely different operation — decode protobuf, merge layer by layer,
-  re-encode, with feature ID collisions to settle. It should be its own feature
-  and should not be smuggled in under `space`.
+- **Vector tiles.** Designed rather than open now: see [merging vector
+  sources](#merging-vector-sources). It is its own space and its own merge,
+  and what stays open is whether the overlap rule proposed there — layer
+  replacement by default, clipping for a patch — holds up against a real
+  pair of archives.
 - **Does the recipe travel?** A stack is a small JSON document and the feed
   already distributes documents. Publishing recipes so a subscriber gets the
   stack along with its sources is attractive, and raises an obvious question

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.91.0",
+  "version": "0.92.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -50,15 +50,7 @@ import {
   stackCoverage,
   stackEtag,
 } from './stacks.js';
-
-/**
- * How many sources a stack's TileJSON names before it starts counting.
- *
- * Generous for a hand-written recipe -- those run to a handful -- and a hard
- * stop for one imported from a provider's index, where the list is hundreds
- * long and every entry is an address no client reads.
- */
-const TILEJSON_SOURCE_LIMIT = 25;
+import { bakeRevision } from './bake.js';
 
 const here = path.dirname(fileURLToPath(import.meta.url));
 
@@ -3800,34 +3792,27 @@ export function createApp({
         minzoom: coverage.minzoom,
         maxzoom: coverage.maxzoom,
         bounds: coverage.bounds,
-        // Names what it resolved to, so a consumer can tell one resolution
-        // from the next without diffing tile URLs — the same reason the
-        // `/latest/` document carries a `latest` block.
+        // Says that it resolved, and whether the resolution has moved, so a
+        // consumer can tell one from the next without diffing tile URLs --
+        // the same reason the `/latest/` document carries a `latest` block.
+        //
+        // A count and a fingerprint rather than the sources themselves. They
+        // were listed here once, and it was wrong twice over. A stack's
+        // sources are not a client's to join: they are the ingredients of one
+        // endpoint, and a document listing them invites somebody to fetch
+        // those instead of the tiles. And a source may be an address -- a URL,
+        // or a bucket and a key -- which this document has no business
+        // publishing: the tiles are public where the archives behind them are
+        // read with credentials nobody else has.
         //
-        // Capped. A stack imported from a provider's file index has hundreds
-        // of sources — Mapterhorn's is 458 — and listing every one puts tens
-        // of kilobytes of addresses in a document every map load fetches, to
-        // say something no client does anything with. What the block is for is
-        // telling one resolution from the next, and what moves between
-        // resolutions is the infohashes, so those are what it keeps.
+        // The fingerprint covers the recipe and what every source resolved
+        // to, so it moves when a category resolves to a new build. That was
+        // the only thing the list was good for.
         stack: {
           id: resolved.stack.id,
           space: resolved.stack.space ?? 'elevation',
-          sources: resolved.sources
-            .slice(0, TILEJSON_SOURCE_LIMIT)
-            .map((source) => ({
-              name: source.name,
-              infohash: source.entry?.infoHash ?? null,
-              archive: source.entry?.name ?? null,
-            })),
-          // Said rather than silently truncated: a consumer comparing two of
-          // these has to know it is looking at part of a list.
-          ...(resolved.sources.length > TILEJSON_SOURCE_LIMIT
-            ? {
-                more: resolved.sources.length - TILEJSON_SOURCE_LIMIT,
-                total: resolved.sources.length,
-              }
-            : {}),
+          sources: resolved.sources.length,
+          revision: bakeRevision(resolved),
         },
       };
       // A stack says how its tiles are encoded, so a style pointing at it does

package/src/web/index.html

@@ -1875,6 +1875,11 @@
         for (const entry of shown) {
           const s = entry.status ?? {};
           const progress = s.progress ?? 0;
+          // The engine has never mentioned this archive. That is not the same
+          // fact as "none of it is here", and drawing it as 0% said the second
+          // -- so a library the engine failed to take back after a restart
+          // read as a library that had lost its data.
+          const known = Boolean(entry.status);
           const mode = entry.mode ?? 'mirror';
           const tr = document.createElement('tr');
           if (entry.infoHash === selected) tr.className = 'on';
@@ -1893,12 +1898,14 @@
             <td>${added(entry.createdAt)}</td>
             <td><span class="pill ${mode}">${mode}</span></td>
             <td>${originCell(entry)}</td>
-            <td>
-              <div class="track">
-                <i class="${progress >= 1 ? 'done' : ''}" style="width:${Math.min(100, progress * 100)}%"></i>
-              </div>
-              <div class="sub">${pct(progress)}</div>
-            </td>
+            <td>${
+              known
+                ? `<div class="track">
+                     <i class="${progress >= 1 ? 'done' : ''}" style="width:${Math.min(100, progress * 100)}%"></i>
+                   </div>
+                   <div class="sub">${pct(progress)}</div>`
+                : `<div class="sub bad" title="The engine has no record of this archive, so nothing here is being seeded or downloaded. It usually means the restore after a restart could not hand it back — the log says why, under [restore]. Nothing has been deleted; the data on disk is untouched.">not loaded</div>`
+            }</td>
             <td title="${peersTitle(s)}">${s.peers ?? 0} / ${s.seeds ?? 0}${swarmSuffix(s)}</td>
             <td>${rate(s.downloadSpeed)}</td>
             <td>${rate(s.uploadSpeed)}</td>
@@ -7506,6 +7513,22 @@ Every piece is hashed against the ` +
         return TABS.includes(named) ? named : null;
       };
 
+      /**
+       * Lands on the view the address names, once there is a console to land
+       * in.
+       *
+       * Two ways in, and both have to honour it: opening the page already
+       * signed in, and signing in on arrival. Only the first did, so following
+       * a link to `#stacks` on a guarded node asked for a password and then
+       * showed the archives -- the address still saying `#stacks`, which is
+       * the part that makes it read as broken rather than as a redirect.
+       * @returns {void}
+       */
+      const landFromUrl = () => {
+        const named = tabFromUrl();
+        if (named && named !== 'archives') showTab(named);
+      };
+
       // The footer's year, set from the clock rather than typed into a file
       // nobody will remember to edit. The version beside it arrives with the
       // first status, and reads "…" until then.
@@ -9451,6 +9474,7 @@ Every piece is hashed against the ` +
           $('login').hidden = true;
           $('login-pass').value = '';
           refresh();
+          landFromUrl();
         } catch (error) {
           $('login-error').textContent = error.message;
         }
@@ -9476,8 +9500,7 @@ Every piece is hashed against the ` +
         refresh();
         // After the archives have been asked for, so the tab that opens is the
         // one asked for and not the one this happens to start on.
-        const named = tabFromUrl();
-        if (named && named !== 'archives') showTab(named);
+        landFromUrl();
       }
 
       start();