pmtiles-swarm 0.97.0 → 0.98.0

package/CHANGELOG.md

@@ -7,6 +7,35 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.98.0
+### ✨ Features and improvements
+- **A finished MBTiles archive works as a stack layer, and now says what it holds.** It always read:
+  the tile store opens one from a complete local copy, and a stack asks the store for tiles like
+  anything else. What it could not do was describe itself. Only PMTiles was ever probed for a
+  summary — right when only PMTiles could be served, and never revisited once MBTiles became
+  servable — so those entries carried none, and a summary is what `stackCoverage` reads. A stack
+  naming one advertised the fallbacks in its TileJSON, z0–z14 over the whole world, however narrow
+  the archive actually was. Nothing refused it and nothing warned: the recipe was valid, the tiles
+  were right, and only the coverage was a fiction.
+
+  Both formats that can be served are read for a summary now, through the same summariser —
+  `probeMbtiles` off the adapter's `getHeader` and `getMetadata`, which is what that adapter exists
+  for. Most of the MBTiles spec is optional, so where an archive states nothing the reader derives
+  what it can: the zoom range comes from the tiles table when `minzoom` and `maxzoom` are absent.
+
+  The metadata reader also carries `encoding` through, with the four factors a custom packing is
+  unreadable without. An elevation stack decodes the pixels rather than passing them through, so an
+  archive that lost its encoding was not a tile that failed but a tile of wrong heights.
+
+  Archives already in the catalog are summarised by the head warmer on its next pass, once complete
+  — an MBTiles has no head to pull out of a swarm, so there is nothing to be due until the file is
+  whole, and then the read is local and instant. Unchanged: `/latest/<category>` still offers no
+  tile endpoint for an MBTiles, because a category is a promise to every node and this one is only
+  keepable on a node holding the whole file. See [tile-stacks.md](docs/tile-stacks.md) — "An
+  MBTiles archive as a source".
+
+### 🐞 Bug fixes
+
 ## 0.97.0
 ### ✨ Features and improvements
 - **A stack can be told to forget what it has merged, and forgets it by itself when its recipe

package/docs/tile-stacks.md

@@ -29,6 +29,7 @@ of its parts.
 - [Painting order](#painting-order)
 - [The two pixel spaces](#the-two-pixel-spaces)
 - [Naming a source](#naming-a-source)
+  - [An MBTiles archive as a source](#an-mbtiles-archive-as-a-source)
 - [The config file](#the-config-file)
 - [Translating a rio-rgbify-merge config](#translating-a-rio-rgbify-merge-config)
 - [Evaluating one tile](#evaluating-one-tile)
@@ -174,6 +175,35 @@ something else.
 A stack may mix the two. Whether _any_ source is category-resolved decides the
 whole stack's caching headers, below.
 
+### An MBTiles archive as a source
+
+Either form may name an MBTiles archive, with one condition: **this node must
+hold the complete file.** MBTiles is SQLite, whose pages are scattered rather
+than spatially clustered, so it cannot be read a byte range at a time out of a
+swarm the way PMTiles can — a source still arriving answers 503, and no amount
+of waiting for the right pieces changes that. A stack with an MBTiles source is
+therefore only servable on nodes that have finished downloading it.
+
+Everything else is the same. It resolves, merges, clips, feathers and bakes
+like any other source, because the tile store presents it through the same
+three methods a PMTiles archive answers.
+
+What it covers comes from the metadata table, read at import by `probeMbtiles`
+and summarised by the same `summarize` the PMTiles path uses. Most of the
+MBTiles spec is optional, so where an archive states nothing the reader derives
+what it can — the zoom range comes from the tiles table itself when `minzoom`
+and `maxzoom` are absent, and bounds default to the whole world.
+
+Two keys outside the spec are read, both because this is where every tool that
+needs them puts them: `sparse`, which tileserver-gl reads the same way, and
+`encoding` — with `redFactor`, `greenFactor`, `blueFactor` and `baseShift` for
+a custom packing. An elevation stack decodes the pixels rather than passing
+them through, so an archive that loses its encoding is not a tile that fails but
+a tile of wrong heights.
+
+An archive already in the catalog from before any of this was probed is
+summarised by the head warmer on its next pass, once it is complete.
+
 ## The config file
 
 Stacks live in `data/stacks.json`, beside `data/catalog.json` and for the same

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.97.0",
+  "version": "0.98.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/library.js

@@ -14,6 +14,7 @@ import {
 import { publishingBase, publishingFor, reachability } from './catalog.js';
 import { linkLatest } from './latest-link.js';
 import { checkOrigin, fingerprintOrigin } from './origin.js';
+import { probeMbtiles } from './mbtiles.js';
 import { probePMTiles } from './pmtiles-probe.js';
 import { archiveDirName } from './savepath.js';
 import {
@@ -303,11 +304,12 @@ export class Library {
     });
 
     try {
-      // Only PMTiles can have its tiles served, so only PMTiles gets probed.
-      const summary =
-        identified.kind === 'pmtiles'
-          ? await probePMTiles(absolute).catch(() => undefined)
-          : undefined;
+      // Both formats that can be served get read for one. The kind comes from
+      // the content rather than the extension, which is what identify exists
+      // for.
+      const summary = await probeArchive(absolute, identified.kind).catch(
+        () => undefined,
+      );
 
       const created = await createTorrentFromFile(absolute, {
         creator: this.#creator(),
@@ -967,6 +969,11 @@ export class Library {
     // progress through runningAdds() and says "watch the log" as it closes.
     options.onValidated?.({ url, kind: identified.kind });
 
+    // PMTiles only, and not an oversight: this reads the archive over range
+    // requests before anything has been downloaded, and there is no partial
+    // read of a SQLite file that answers anything. An MBTiles mirrored from a
+    // URL is summarised by the warmer once the download finishes, which is the
+    // first moment it can be.
     const summary =
       identified.kind === 'pmtiles'
         ? await probePMTiles(url).catch(() => undefined)
@@ -1605,8 +1612,9 @@ export class Library {
     // facts, and probing on the second one's behalf would hang.
     let summary;
     if (readable && torrent.progress === 1 && torrent.savePath) {
-      summary = await probePMTiles(
+      summary = await probeArchive(
         path.join(torrent.savePath, torrent.name),
+        guessKind(torrent.name ?? ''),
       ).catch(() => undefined);
     }
 
@@ -3383,6 +3391,30 @@ export function guessKind(name) {
   return undefined;
 }
 
+/**
+ * Reads the summary out of whichever kind of archive this is.
+ *
+ * Only PMTiles was ever probed, which was right when only PMTiles could be
+ * served. MBTiles became servable from a complete local copy and this did not
+ * follow, so those entries carried no summary at all -- and a summary is what
+ * everything downstream reads coverage from. A stack naming one advertised the
+ * fallbacks in its TileJSON, z0-z14 over the whole world, rather than the zoom
+ * range and bounds the archive states about itself.
+ *
+ * Anything else is left alone rather than guessed at. A .osm.pbf being mirrored
+ * has no summary to read and asking for one is how it ends up reported as a
+ * broken map instead of a file being distributed.
+ * @param {string} location - Local path, or a URL for PMTiles.
+ * @param {string} [kind] - What it was identified as; the name decides if not.
+ * @returns {Promise<object|undefined>} - The summary, where there is one.
+ */
+export async function probeArchive(location, kind) {
+  const format = kind ?? guessKind(location);
+  if (format === 'mbtiles') return probeMbtiles(location);
+  if (format === 'pmtiles') return probePMTiles(location);
+  return undefined;
+}
+
 /**
  * Moves a file, whether or not the two paths share a filesystem.
  *

package/src/mbtiles.js

@@ -1,4 +1,5 @@
 import path from 'node:path';
+import { summarize } from './pmtiles-probe.js';
 
 /**
  * Reading tiles out of a completed MBTiles archive.
@@ -199,6 +200,17 @@ export class MbtilesArchive {
       // Same key tileserver-gl reads, carried through so an MBTiles archive
       // gets the same treatment a PMTiles one does.
       sparse: meta.sparse,
+      // How heights are packed into the pixels, and the four factors a custom
+      // packing is unreadable without. Not in the MBTiles spec -- neither is
+      // `sparse` -- but this is where every tool that needs it puts it, and a
+      // terrain archive that loses it is decoded as whatever the reader
+      // assumes. That is not a tile that fails; it is a tile of wrong heights,
+      // which is worse.
+      encoding: meta.encoding,
+      redFactor: meta.redFactor,
+      greenFactor: meta.greenFactor,
+      blueFactor: meta.blueFactor,
+      baseShift: meta.baseShift,
     };
   }
 
@@ -254,3 +266,32 @@ export class MbtilesArchive {
     }
   }
 }
+
+/**
+ * Reads an MBTiles archive's summary, the way probePMTiles reads a PMTiles one.
+ *
+ * The same summariser, off the same two methods -- which is the whole point of
+ * the adapter above: MBTiles keeps in a metadata table what PMTiles keeps in a
+ * fixed header, and everything upstream of `summarize` should not have to know
+ * which it was reading. `getHeader` even derives the zoom range from the tiles
+ * table where the metadata omits it, so a summary comes back for an archive
+ * that declares almost nothing about itself.
+ *
+ * Local paths only, and not by oversight. probePMTiles takes a URL because
+ * PMTiles is read a byte range at a time, and SQLite is not: there is no
+ * partial read of an MBTiles that answers anything, which is the same reason
+ * it is served only from a complete local copy.
+ * @param {string} file - Path to the .mbtiles file.
+ * @returns {Promise<object>} - The summary, in the shape the catalog keeps.
+ */
+export async function probeMbtiles(file) {
+  const archive = await MbtilesArchive.open(file);
+  try {
+    return summarize(
+      await archive.getHeader(),
+      (await archive.getMetadata()) ?? {},
+    );
+  } finally {
+    archive.close();
+  }
+}

package/src/prewarm.js

@@ -156,7 +156,14 @@ export class HeadWarmer {
     // not, since an archive joined by magnet has no kind until its metadata
     // arrives.
     const kind = entry.kind ?? guessKind(entry.name ?? '');
-    if (kind !== 'pmtiles') return false;
+    if (kind !== 'pmtiles' && kind !== 'mbtiles') return false;
+
+    // MBTiles has no head to pull out of the swarm -- it is SQLite, read whole
+    // or not at all -- so there is nothing here to be due until the file is.
+    // Once it is, the read is local and instant, and it is the only thing that
+    // gives an archive already in the catalog the summary this did not use to
+    // record for it.
+    if (kind === 'mbtiles' && entry.complete !== true) return false;
 
     // A summary is only an answer about *this disk* if a header on this disk
     // produced it. `format` was standing in for that and does not mean it.
@@ -289,6 +296,11 @@ export class HeadWarmer {
           `[warm] ${entry.name}: header read; its metadata is at the far end ` +
             'of the archive and has not arrived yet',
         );
+      } else if ((entry.kind ?? guessKind(entry.name ?? '')) === 'mbtiles') {
+        // Not a header: MBTiles keeps in a metadata table what PMTiles keeps
+        // in a fixed header, and a log saying otherwise sends whoever reads it
+        // looking for the wrong thing.
+        console.log(`[warm] ${entry.name}: metadata read`);
       } else {
         console.log(`[warm] ${entry.name}: header read`);
       }