pmtiles-swarm 0.54.2 → 0.55.1

package/CHANGELOG.md

@@ -7,6 +7,53 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.55.1
+### 🐞 Bug fixes
+- **0.55.0 reached new archives only.** `encoding` sits in the metadata of archives some nodes have
+  been serving for months, the prober learned to read it, and nothing changed — because a summary is
+  written into the catalog once and never questioned. Every path that re-read one was gated on the
+  summary being *absent*, and these summaries were present, merely old.
+
+  The summary now carries a `summaryVersion`, and one written by an older prober is re-read once, in
+  the background, on the next TileJSON request for that archive. Rate-limited to once a minute per
+  archive, as the vector-layer backfill beside it already was, and written back on any read that
+  produced something newer — the old early return threw away everything except vector layers, which
+  would have discarded the encoding it went to fetch.
+
+  This is general, not a fix for one field: raise `SUMMARY_VERSION` whenever the prober learns to
+  read something new, and every archive already in the catalog picks it up on its own. Nothing is
+  needed on an upgrade beyond asking for the archive's TileJSON, which anything using it does anyway.
+
+## 0.55.0
+### ✨ Features and improvements
+- **The TileJSON now carries a `raster-dem` archive's `encoding`**, read out of the archive's own
+  metadata the same way `sparse` already is. Nothing in a PMTiles header carries this — the header
+  knows the tile is WebP, not what its three channels mean — so the metadata is the only place it can
+  come from, and without it a consumer falls back to a default that is wrong for exactly the archives
+  that most need to speak up: a terrarium-packed DEM read as `mapbox` decodes every mountain into
+  noise, silently, with a plausible-looking map on screen.
+
+  Same key and meaning as tileserver-gl, which reads `tileJSON.encoding` and accepts `terrarium` or
+  `mapbox`. The difference is where it comes from — there it is configuration, set per layer beside
+  the server; here it travels with the archive, so a mirror reads the same answer without being
+  configured again and a style no longer has to restate what the archive already knows. Restating it
+  is how a style and its data drift into disagreeing. MapLibre applies TileJSON members to the source
+  after construction, so this overrides an encoding written in the style, which is the intended
+  direction.
+
+  `custom` brings `redFactor`, `greenFactor`, `blueFactor` and `baseShift` with it, and only when all
+  four are present: `custom` means "the channels mean what these numbers say", so the word without
+  the numbers is an archive nobody can read. Anything outside the three values the style
+  specification defines is dropped rather than passed on — a client handed an encoding it does not
+  recognise is worse off than one handed nothing.
+
+  Carried through the feed as `<pmtiles:encoding>` too, so a subscriber serves elevation correctly
+  from the moment it joins rather than reading noise until it has probed the header itself. The
+  custom factors deliberately are not: an archive needing four numbers to be legible is one a mirror
+  should read for itself rather than take on trust from somebody else's document.
+
+### 🐞 Bug fixes
+
 ## 0.54.2
 ### ✨ Features and improvements
 - **Written down and tested: a range read is cached exactly as a tile read is**, because it is the

package/docs/tilejson.md

@@ -16,6 +16,7 @@ page is what they mean.
   - [One string, both kinds of client](#one-string-both-kinds-of-client)
   - [Where this magnet shows up](#where-this-magnet-shows-up)
 - [`sparse`](#sparse)
+- [`encoding`](#encoding)
 - [A complete example](#a-complete-example)
 - [What a plain client sees](#what-a-plain-client-sees)
 
@@ -145,6 +146,48 @@ reads the same answer rather than falling back to guessing from the tile format.
 See
 [internals.md](internals.md#answering-for-a-tile-that-is-not-there).
 
+## `encoding`
+
+How a `raster-dem` archive packs elevation into pixels: `terrarium`, `mapbox` or
+`custom`. Present only when the archive itself said so in its metadata.
+
+Nothing in a PMTiles header carries this — the header knows the tile is WebP,
+not what its three channels mean — so the metadata is the only place it can come
+from, and the PMTiles specification explicitly allows metadata beyond the keys
+it names. Without it a consumer falls back to a default, and the default is
+wrong for exactly the archives that most need to speak up: a terrarium-packed
+DEM read as `mapbox` decodes every mountain into noise, silently, with a
+plausible-looking map on screen and nothing to point at.
+
+Same key and same meaning as tileserver-gl, which reads `tileJSON.encoding` and
+accepts `terrarium` or `mapbox`. The difference is where it comes from: there it
+is configuration, set per layer in the server's own config, so the answer lives
+beside the server rather than beside the data. Here it travels with the archive,
+which is the point — a node mirroring it reads the same answer without being
+configured again, and a style pointing at this TileJSON no longer has to restate
+an encoding the archive already knows. Restating it is how a style and its data
+drift into disagreeing.
+
+MapLibre applies every TileJSON member to the source after the source is
+constructed, so an `encoding` here **overrides** one written in the style. That
+is the intended direction: the archive is the thing that knows.
+
+Anything other than the three values the style specification defines is dropped
+rather than passed on. A client handed an encoding it does not recognise is
+worse off than one handed nothing, because nothing at least leaves it free to
+use its own default.
+
+### `redFactor`, `greenFactor`, `blueFactor`, `baseShift`
+
+Carried only alongside `encoding: "custom"`, and only when all four are present.
+`custom` means "the channels mean what these numbers say", so publishing the
+word without the numbers publishes an archive nobody can read — and three
+numbers with one missing is no better, so the whole claim is dropped instead.
+
+The feed carries `<pmtiles:encoding>` but not these: an archive that needs four
+numbers to be legible is one a mirror should read for itself rather than take on
+trust from somebody else's document.
+
 ## A complete example
 
 ```json

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.54.2",
+  "version": "0.55.1",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -33,6 +33,7 @@ import {
 } from './sources.js';
 import { limitFor, remaining } from './seeding.js';
 import { buildTileJson, extensionMatches } from './tilejson.js';
+import { SUMMARY_VERSION } from './pmtiles-probe.js';
 import { TileReadError } from './tiles.js';
 
 const here = path.dirname(fileURLToPath(import.meta.url));
@@ -363,9 +364,18 @@ export function createApp({
    * @returns {boolean} - True to attempt a read now.
    */
   const metadataRetries = new Map();
-  const needsVectorLayers = (summary, infoHash) => {
-    if (!summary || summary.format !== 'pbf' || summary.vectorLayers)
-      return false;
+  const needsReread = (summary, infoHash) => {
+    if (!summary) return false;
+    // Two reasons to go back to the archive. The first is the one this began
+    // as: a vector archive whose layers have not been read yet. The second is
+    // the general case it turned out to be — a summary written by an older
+    // prober, missing whatever that prober did not know to look for. Without
+    // it, adding a field to the summary reaches new archives only, and every
+    // archive already in the catalog keeps its hole until somebody removes and
+    // re-adds it by hand.
+    const stale = summary.summaryVersion !== SUMMARY_VERSION;
+    const missingLayers = summary.format === 'pbf' && !summary.vectorLayers;
+    if (!stale && !missingLayers) return false;
     const last = metadataRetries.get(infoHash) ?? 0;
     if (Date.now() - last < 60000) return false;
     metadataRetries.set(infoHash, Date.now());
@@ -388,13 +398,18 @@ export function createApp({
     // .torrent endpoints work without one. Enriching a summary is the least
     // important thing here and must never be what takes a reply down.
     if (typeof tiles?.summarize !== 'function') return;
-    if (!needsVectorLayers(entry.pmtiles, entry.infoHash)) return;
+    if (!needsReread(entry.pmtiles, entry.infoHash)) return;
 
     const timeoutMs = config.tiles?.metadataTimeoutMs ?? 120000;
     tiles
       .summarize(entry.infoHash, { timeoutMs })
       .then(async (summary) => {
-        if (!summary.vectorLayers) return;
+        // Written back whenever the read produced something newer than what
+        // was stored. Returning early unless vector layers turned up was right
+        // when layers were the only thing this looked for; it would now throw
+        // away the encoding it went to fetch.
+        const fresh = summary.summaryVersion !== entry.pmtiles?.summaryVersion;
+        if (!summary.vectorLayers && !fresh) return;
         await catalog.put({
           infoHash: entry.infoHash,
           pmtiles: { ...entry.pmtiles, ...summary },

package/src/feed.js

@@ -18,6 +18,7 @@
  */
 
 import { mutableMagnet } from './mutable.js';
+import { metadataEncoding } from './pmtiles-probe.js';
 
 const PMTILES_NS = 'https://github.com/TechIdiots-LLC/pmtiles-swarm/ns/1.0';
 
@@ -119,6 +120,14 @@ function renderItem(entry, baseUrl) {
         map.attribution
           ? `      <pmtiles:attribution>${xml(map.attribution)}</pmtiles:attribution>`
           : '',
+        // So a subscriber can serve elevation correctly from the moment it
+        // joins, rather than reading noise until it has probed the header
+        // itself. The custom factors are deliberately not carried: an archive
+        // that needs four numbers to be legible is one a mirror should read
+        // for itself rather than take on trust from a feed.
+        map.encoding
+          ? `      <pmtiles:encoding>${xml(map.encoding)}</pmtiles:encoding>`
+          : '',
       ]
         .filter(Boolean)
         .join('\n')
@@ -233,6 +242,11 @@ function mapSummary(block) {
     bounds: bounds.length === 4 ? bounds : undefined,
     tileCount: number('pmtiles:tiles'),
     attribution: tag(block, 'pmtiles:attribution'),
+    // Carried because a subscriber serves tiles from this summary before it
+    // has read a byte of the archive, and for a raster-dem archive the
+    // encoding is the difference between elevation and noise. Validated on the
+    // way in, since a feed is somebody else's document.
+    encoding: metadataEncoding(tag(block, 'pmtiles:encoding')),
     // Says where this came from, because a summary taken on trust from another
     // node is not the same fact as one read off the archive's own header, and
     // the difference matters when the two disagree.

package/src/pmtiles-probe.js

@@ -58,6 +58,49 @@ export function metadataFlag(value) {
   return undefined;
 }
 
+/**
+ * What a `raster-dem` archive says about how its elevations are packed.
+ *
+ * Nothing in a PMTiles header says this — the header knows the tile is WebP,
+ * not what the three channels mean — so the only place it can come from is the
+ * archive's own metadata. Without it a consumer falls back to a default, and
+ * the default is wrong for exactly the archives that most need to say
+ * something: a terrarium-packed DEM read as `mapbox` decodes every mountain
+ * into noise, silently, with a plausible-looking map on screen.
+ *
+ * The values are the ones the style specification defines, and anything else
+ * is dropped rather than passed on. A tile client handed an encoding it does
+ * not recognise is worse off than one handed nothing, because nothing at least
+ * leaves it free to use its own default.
+ * @param {unknown} value - Whatever the metadata held.
+ * @returns {string | undefined} - A known encoding, or undefined.
+ */
+export function metadataEncoding(value) {
+  if (typeof value !== 'string') return undefined;
+  const text = value.trim().toLowerCase();
+  return ['terrarium', 'mapbox', 'custom'].includes(text) ? text : undefined;
+}
+
+/**
+ * The numbers a `custom` encoding is meaningless without.
+ *
+ * `encoding: "custom"` says "the channels mean what these four factors say",
+ * so carrying the word and not the factors publishes an archive nobody can
+ * read. They travel together or not at all.
+ * @param {object} metadata - The archive's metadata.
+ * @returns {object | undefined} - The four factors, or undefined.
+ */
+export function customEncodingFactors(metadata) {
+  const named = ['redFactor', 'greenFactor', 'blueFactor', 'baseShift'];
+  const out = {};
+  for (const name of named) {
+    const value = Number(metadata[name]);
+    if (!Number.isFinite(value)) return undefined;
+    out[name] = value;
+  }
+  return out;
+}
+
 /**
  * Reads an archive's header and metadata.
  *
@@ -67,6 +110,24 @@ export function metadataFlag(value) {
  * @param {string} location - Local path or http(s) URL.
  * @returns {Promise<PMTilesSummary>} - The summary.
  */
+/**
+ * What this prober reads, as a number that goes up when that changes.
+ *
+ * A summary is stored in the catalog and never read again, which is right --
+ * re-reading a header out of the swarm is not free, and the answer does not
+ * change for a given infohash. It goes wrong the moment the prober learns to
+ * read something new: every archive probed before that keeps a summary with a
+ * hole in it, for ever, and the only way out was to remove and re-add it.
+ *
+ * `encoding` was the case that made this obvious. It sat in the metadata of
+ * archives this node had been serving for months, and adding the code to read
+ * it changed nothing at all, because nothing ever asked again.
+ *
+ * So: stamp what the prober knew at the time, and re-read once when that is
+ * behind. Raise this whenever a field is added to the summary below.
+ */
+export const SUMMARY_VERSION = 2;
+
 export function summarize(header, metadata = {}) {
   const type = TILE_TYPES[header.tileType] ?? TILE_TYPES[0];
 
@@ -80,6 +141,7 @@ export function summarize(header, metadata = {}) {
   );
 
   return {
+    summaryVersion: SUMMARY_VERSION,
     specVersion: header.specVersion,
     format: type.format,
     contentType: type.contentType,
@@ -103,6 +165,16 @@ export function summarize(header, metadata = {}) {
     // the same key, so an archive built to be served there carries the answer
     // with it and does not have to be configured again here.
     sparse: metadataFlag(metadata.sparse),
+    // How a raster-dem archive packs elevation into pixels: terrarium, mapbox
+    // or custom. Read from the archive rather than configured here, for the
+    // same reason `sparse` is — the archive knows, and a mirror of it should
+    // not have to be told again.
+    encoding: metadataEncoding(metadata.encoding),
+    // Only where the encoding is custom, since they mean nothing otherwise.
+    encodingFactors:
+      metadataEncoding(metadata.encoding) === 'custom'
+        ? customEncodingFactors(metadata)
+        : undefined,
   };
 }
 

package/src/tilejson.js

@@ -55,6 +55,16 @@ export function buildTileJson(entry, baseUrl) {
   // Passed on so the next node to mirror this archive reads the same answer we
   // did, rather than falling back to a guess from the tile format.
   if (summary.sparse !== undefined) doc.sparse = summary.sparse;
+  // How to read the pixels of a raster-dem archive. tileserver-gl reads the
+  // same key, so an archive built to be served there carries the answer with
+  // it — and a style pointing at this TileJSON no longer has to restate an
+  // encoding that the archive already knows, which is how a style and its data
+  // drift into disagreeing.
+  if (summary.encoding) doc.encoding = summary.encoding;
+  // `custom` is unreadable without them, so they go wherever it does.
+  if (summary.encoding === 'custom' && summary.encodingFactors) {
+    Object.assign(doc, summary.encodingFactors);
+  }
   if (summary.format === 'pbf') doc.format = 'pbf';
   else if (summary.format) doc.format = summary.format;