pmtiles-swarm 0.97.0 → 0.98.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -7,6 +7,55 @@
7
7
  ### 🐞 Bug fixes
8
8
  - _...Add new stuff here..._
9
9
 
10
+ ## 0.98.1
11
+ ### ✨ Features and improvements
12
+ - **The two archive readers no longer import each other.** Summarising an MBTiles archive reached
13
+ into `pmtiles-probe.js` for `summarize`, which put one format's reader inside the other's module
14
+ and dragged the `pmtiles` package in behind it. The half neither format owns — the summary shape,
15
+ its version, the metadata readers and the tile-type table — is now `archive-summary.js`, which
16
+ imports nothing and opens nothing: it takes a header and a metadata document and returns a
17
+ summary. `pmtiles-probe.js` keeps the PMTiles half and `mbtiles.js` the SQLite half, and neither
18
+ reaches the other.
19
+
20
+ Nothing about what the node serves changes; this is where the code lives.
21
+
22
+ ### 🐞 Bug fixes
23
+ - **An MBTiles archive holding MLT tiles was summarised as an unknown format.** The table mapping a
24
+ format name onto a tile type had been written twice, in opposite directions, and the copy beside
25
+ the MBTiles reader was written by hand and had no entry for `mlt` — so an archive declaring it
26
+ came back as `unknown`, which is what the catalog then reported and what a stack over it read.
27
+ There is one table now, with the reverse lookup derived from it, so the two halves cannot drift
28
+ apart again.
29
+
30
+ ## 0.98.0
31
+ ### ✨ Features and improvements
32
+ - **A finished MBTiles archive works as a stack layer, and now says what it holds.** It always read:
33
+ the tile store opens one from a complete local copy, and a stack asks the store for tiles like
34
+ anything else. What it could not do was describe itself. Only PMTiles was ever probed for a
35
+ summary — right when only PMTiles could be served, and never revisited once MBTiles became
36
+ servable — so those entries carried none, and a summary is what `stackCoverage` reads. A stack
37
+ naming one advertised the fallbacks in its TileJSON, z0–z14 over the whole world, however narrow
38
+ the archive actually was. Nothing refused it and nothing warned: the recipe was valid, the tiles
39
+ were right, and only the coverage was a fiction.
40
+
41
+ Both formats that can be served are read for a summary now, through the same summariser —
42
+ `probeMbtiles` off the adapter's `getHeader` and `getMetadata`, which is what that adapter exists
43
+ for. Most of the MBTiles spec is optional, so where an archive states nothing the reader derives
44
+ what it can: the zoom range comes from the tiles table when `minzoom` and `maxzoom` are absent.
45
+
46
+ The metadata reader also carries `encoding` through, with the four factors a custom packing is
47
+ unreadable without. An elevation stack decodes the pixels rather than passing them through, so an
48
+ archive that lost its encoding was not a tile that failed but a tile of wrong heights.
49
+
50
+ Archives already in the catalog are summarised by the head warmer on its next pass, once complete
51
+ — an MBTiles has no head to pull out of a swarm, so there is nothing to be due until the file is
52
+ whole, and then the read is local and instant. Unchanged: `/latest/<category>` still offers no
53
+ tile endpoint for an MBTiles, because a category is a promise to every node and this one is only
54
+ keepable on a node holding the whole file. See [tile-stacks.md](docs/tile-stacks.md) — "An
55
+ MBTiles archive as a source".
56
+
57
+ ### 🐞 Bug fixes
58
+
10
59
  ## 0.97.0
11
60
  ### ✨ Features and improvements
12
61
  - **A stack can be told to forget what it has merged, and forgets it by itself when its recipe
@@ -29,6 +29,7 @@ of its parts.
29
29
  - [Painting order](#painting-order)
30
30
  - [The two pixel spaces](#the-two-pixel-spaces)
31
31
  - [Naming a source](#naming-a-source)
32
+ - [An MBTiles archive as a source](#an-mbtiles-archive-as-a-source)
32
33
  - [The config file](#the-config-file)
33
34
  - [Translating a rio-rgbify-merge config](#translating-a-rio-rgbify-merge-config)
34
35
  - [Evaluating one tile](#evaluating-one-tile)
@@ -174,6 +175,35 @@ something else.
174
175
  A stack may mix the two. Whether _any_ source is category-resolved decides the
175
176
  whole stack's caching headers, below.
176
177
 
178
+ ### An MBTiles archive as a source
179
+
180
+ Either form may name an MBTiles archive, with one condition: **this node must
181
+ hold the complete file.** MBTiles is SQLite, whose pages are scattered rather
182
+ than spatially clustered, so it cannot be read a byte range at a time out of a
183
+ swarm the way PMTiles can — a source still arriving answers 503, and no amount
184
+ of waiting for the right pieces changes that. A stack with an MBTiles source is
185
+ therefore only servable on nodes that have finished downloading it.
186
+
187
+ Everything else is the same. It resolves, merges, clips, feathers and bakes
188
+ like any other source, because the tile store presents it through the same
189
+ three methods a PMTiles archive answers.
190
+
191
+ What it covers comes from the metadata table, read at import by `probeMbtiles`
192
+ and summarised by the same `summarize` the PMTiles path uses. Most of the
193
+ MBTiles spec is optional, so where an archive states nothing the reader derives
194
+ what it can — the zoom range comes from the tiles table itself when `minzoom`
195
+ and `maxzoom` are absent, and bounds default to the whole world.
196
+
197
+ Two keys outside the spec are read, both because this is where every tool that
198
+ needs them puts them: `sparse`, which tileserver-gl reads the same way, and
199
+ `encoding` — with `redFactor`, `greenFactor`, `blueFactor` and `baseShift` for
200
+ a custom packing. An elevation stack decodes the pixels rather than passing
201
+ them through, so an archive that loses its encoding is not a tile that fails but
202
+ a tile of wrong heights.
203
+
204
+ An archive already in the catalog from before any of this was probed is
205
+ summarised by the head warmer on its next pass, once it is complete.
206
+
177
207
  ## The config file
178
208
 
179
209
  Stacks live in `data/stacks.json`, beside `data/catalog.json` and for the same
@@ -439,9 +469,11 @@ This is the one thing that stops a stack being a small feature, and it is
439
469
  narrower than "the node cannot handle tiles". It already does, in most of the
440
470
  ways a stack needs:
441
471
 
442
- - `identify.js` reads an archive's magic bytes; the prober reads PMTiles
443
- headers, directories and metadata; `mbtiles.js` queries tile rows out of
444
- SQLite.
472
+ - `identify.js` reads an archive's magic bytes; `pmtiles-probe.js` reads PMTiles
473
+ headers, directories and metadata; `mbtiles.js` queries tile rows and the
474
+ metadata table out of SQLite. Neither format's reader imports the other's:
475
+ what they share is `archive-summary.js`, which turns a header and a metadata
476
+ document into the summary the catalog keeps and opens nothing itself.
445
477
  - `TileStore.getTile` resolves an archive, reads a tile through the local file
446
478
  or the swarm, and knows its format from the header rather than by guessing.
447
479
  - The tile route already gzips vector tiles through `node:zlib`, abandons a
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pmtiles-swarm",
3
- "version": "0.97.0",
3
+ "version": "0.98.1",
4
4
  "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
5
5
  "type": "module",
6
6
  "main": "src/index.js",
package/src/api.js CHANGED
@@ -38,7 +38,7 @@ import {
38
38
  } from './sources.js';
39
39
  import { limitFor, remaining } from './seeding.js';
40
40
  import { buildTileJson, extensionMatches, tileExtension } from './tilejson.js';
41
- import { SUMMARY_VERSION } from './pmtiles-probe.js';
41
+ import { SUMMARY_VERSION } from './archive-summary.js';
42
42
  import { TileReadError } from './tiles.js';
43
43
  import { loadCodec } from './codec.js';
44
44
  import { answerStackTile, outputFormat, outputSize } from './stack-tile.js';
@@ -0,0 +1,183 @@
1
+ /**
2
+ * The summary this project keeps about a map archive, and the vocabulary for
3
+ * building one.
4
+ *
5
+ * What a subscriber needs in order to decide whether they want an archive:
6
+ * coverage, zoom range, tile type, size. This is what makes a map RSS feed more
7
+ * useful than a generic torrent feed — an item can say "raster webp, z0-14,
8
+ * covering Switzerland, 36 GiB" instead of just a filename, so a consumer can
9
+ * filter before committing to a download.
10
+ *
11
+ * Here rather than beside either reader because both formats produce one, and
12
+ * neither should have to import the other to do it. Nothing in this file opens
13
+ * a file or a socket: it takes a header and a metadata document and returns the
14
+ * summary, which is why it can be shared without dragging one format's
15
+ * dependencies into the other's module.
16
+ */
17
+
18
+ /**
19
+ * Maps tile type numbers onto names and content types.
20
+ *
21
+ * The numbering is PMTiles', because a number was needed and that one is
22
+ * written down. An MBTiles archive names its format in a metadata row instead,
23
+ * and `tileTypeFor` is how it says the same thing in the same terms.
24
+ */
25
+ const TILE_TYPES = {
26
+ 0: { format: 'unknown', contentType: 'application/octet-stream' },
27
+ 1: { format: 'pbf', contentType: 'application/x-protobuf' },
28
+ 2: { format: 'png', contentType: 'image/png' },
29
+ 3: { format: 'jpeg', contentType: 'image/jpeg' },
30
+ 4: { format: 'webp', contentType: 'image/webp' },
31
+ 5: { format: 'avif', contentType: 'image/avif' },
32
+ // No MIME is registered for MLT, and nothing selects a tile by content type.
33
+ 6: { format: 'mlt', contentType: 'application/octet-stream' },
34
+ };
35
+
36
+ /**
37
+ * The tile type number for a format an archive names in words.
38
+ *
39
+ * The inverse of the table above, kept beside it so the two halves cannot
40
+ * drift. `mvt` and `jpg` are spellings the same formats appear under in the
41
+ * wild; anything unrecognised is 0, which reads back as "unknown" rather than
42
+ * as a format nothing can serve.
43
+ * @param {string} [format] - What the archive calls its tiles.
44
+ * @returns {number} - The tile type number.
45
+ */
46
+ export function tileTypeFor(format) {
47
+ const named = String(format ?? '').toLowerCase();
48
+ const found = Object.entries(TILE_TYPES).find(
49
+ ([, type]) => type.format === named,
50
+ );
51
+ if (found) return Number(found[0]);
52
+ return { mvt: 1, jpg: 3 }[named] ?? 0;
53
+ }
54
+
55
+ /**
56
+ * Summary of an archive, as stored in the catalog and published in the feed.
57
+ * @typedef {object} ArchiveSummary
58
+ * @property {number} specVersion - PMTiles spec version.
59
+ * @property {string} format - Tile format: pbf, png, jpeg, webp, avif.
60
+ * @property {string} contentType - Matching content type.
61
+ * @property {number} minZoom - Lowest zoom present.
62
+ * @property {number} maxZoom - Highest zoom present.
63
+ * @property {number[]} bounds - [minLon, minLat, maxLon, maxLat].
64
+ * @property {number[]} center - [lon, lat, zoom].
65
+ * @property {number} tileCount - Addressed tile count.
66
+ * @property {boolean} clustered - Whether tiles are stored in Hilbert order.
67
+ * @property {string} [name] - Name from the archive metadata.
68
+ * @property {string} [description] - Description from the archive metadata.
69
+ * @property {string} [attribution] - Attribution from the archive metadata.
70
+ * @property {object[]} [vectorLayers] - Vector layer definitions, for pbf archives.
71
+ * @property {boolean} [sparse] - What the archive says about missing tiles, if it says anything.
72
+ */
73
+
74
+ /**
75
+ * Reads a flag out of archive metadata, which is not reliably typed.
76
+ *
77
+ * A PMTiles JSON blob carries a real boolean, but the same metadata routinely
78
+ * arrives having been round-tripped through MBTiles, where every value is TEXT
79
+ * — so the honest reading of `"false"` is false, not "a non-empty string".
80
+ * @param {unknown} value - Whatever the metadata held.
81
+ * @returns {boolean | undefined} - The flag, or undefined if it said nothing.
82
+ */
83
+ export function metadataFlag(value) {
84
+ if (value === undefined || value === null || value === '') return undefined;
85
+ if (typeof value === 'boolean') return value;
86
+ if (typeof value === 'number') return value !== 0;
87
+ const text = String(value).trim().toLowerCase();
88
+ if (['true', '1', 'yes'].includes(text)) return true;
89
+ if (['false', '0', 'no'].includes(text)) return false;
90
+ return undefined;
91
+ }
92
+
93
+ /**
94
+ * The tile encoding an archive declares, if it is one MapLibre understands.
95
+ *
96
+ * See docs/tilejson.md — `encoding`.
97
+ * @param {unknown} value - Whatever the metadata held.
98
+ * @returns {string | undefined} - A known encoding, or undefined.
99
+ */
100
+ export function metadataEncoding(value) {
101
+ if (typeof value !== 'string') return undefined;
102
+ const text = value.trim().toLowerCase();
103
+ // An unrecognised value is dropped: it would cost a client its own default.
104
+ return ['terrarium', 'mapbox', 'custom', 'mlt'].includes(text)
105
+ ? text
106
+ : undefined;
107
+ }
108
+
109
+ /**
110
+ * The four factors a `custom` encoding is unreadable without. All or nothing.
111
+ * @param {object} metadata - The archive's metadata.
112
+ * @returns {object | undefined} - The four factors, or undefined.
113
+ */
114
+ export function customEncodingFactors(metadata) {
115
+ const named = ['redFactor', 'greenFactor', 'blueFactor', 'baseShift'];
116
+ const out = {};
117
+ for (const name of named) {
118
+ const value = Number(metadata[name]);
119
+ if (!Number.isFinite(value)) return undefined;
120
+ out[name] = value;
121
+ }
122
+ return out;
123
+ }
124
+
125
+ /**
126
+ * What this prober reads. Raise it whenever a field is added to the summary,
127
+ * and archives probed by an older build are re-read once. See
128
+ * docs/internals.md — "Re-reading a summary an older prober wrote".
129
+ */
130
+ export const SUMMARY_VERSION = 3;
131
+
132
+ /**
133
+ * What the catalog keeps about an archive, from its header and metadata.
134
+ * @param {object} header - A parsed PMTiles v3 header.
135
+ * @param {object} [metadata] - The archive's own metadata document.
136
+ * @returns {object} - The summary, at `SUMMARY_VERSION`.
137
+ */
138
+ export function summarize(header, metadata = {}) {
139
+ const type = TILE_TYPES[header.tileType] ?? TILE_TYPES[0];
140
+
141
+ // An archive with no bounds set reports all zeroes; treat that as global
142
+ // rather than as a point at null island.
143
+ const hasBounds = !(
144
+ header.minLon === 0 &&
145
+ header.minLat === 0 &&
146
+ header.maxLon === 0 &&
147
+ header.maxLat === 0
148
+ );
149
+
150
+ return {
151
+ summaryVersion: SUMMARY_VERSION,
152
+ specVersion: header.specVersion,
153
+ format: type.format,
154
+ contentType: type.contentType,
155
+ minZoom: header.minZoom,
156
+ maxZoom: header.maxZoom,
157
+ bounds: hasBounds
158
+ ? [header.minLon, header.minLat, header.maxLon, header.maxLat]
159
+ : [-180, -85.051129, 180, 85.051129],
160
+ center: [
161
+ header.centerLon,
162
+ header.centerLat,
163
+ header.centerZoom || Math.round(header.maxZoom / 2),
164
+ ],
165
+ tileCount: header.numAddressedTiles,
166
+ clustered: header.clustered,
167
+ name: metadata.name,
168
+ description: metadata.description,
169
+ attribution: metadata.attribution,
170
+ vectorLayers: metadata.vector_layers,
171
+ // What the archive says about its own missing tiles. tileserver-gl reads
172
+ // the same key, so an archive built to be served there carries the answer
173
+ // with it and does not have to be configured again here.
174
+ sparse: metadataFlag(metadata.sparse),
175
+ // The header settles MLT; only the metadata can settle elevation packing.
176
+ encoding:
177
+ type.format === 'mlt' ? 'mlt' : metadataEncoding(metadata.encoding),
178
+ encodingFactors:
179
+ metadataEncoding(metadata.encoding) === 'custom'
180
+ ? customEncodingFactors(metadata)
181
+ : undefined,
182
+ };
183
+ }
package/src/feed.js CHANGED
@@ -18,7 +18,7 @@
18
18
  */
19
19
 
20
20
  import { mutableMagnet } from './mutable.js';
21
- import { metadataEncoding } from './pmtiles-probe.js';
21
+ import { metadataEncoding } from './archive-summary.js';
22
22
 
23
23
  const PMTILES_NS = 'https://github.com/TechIdiots-LLC/pmtiles-swarm/ns/1.0';
24
24
 
package/src/library.js CHANGED
@@ -14,6 +14,7 @@ import {
14
14
  import { publishingBase, publishingFor, reachability } from './catalog.js';
15
15
  import { linkLatest } from './latest-link.js';
16
16
  import { checkOrigin, fingerprintOrigin } from './origin.js';
17
+ import { probeMbtiles } from './mbtiles.js';
17
18
  import { probePMTiles } from './pmtiles-probe.js';
18
19
  import { archiveDirName } from './savepath.js';
19
20
  import {
@@ -303,11 +304,12 @@ export class Library {
303
304
  });
304
305
 
305
306
  try {
306
- // Only PMTiles can have its tiles served, so only PMTiles gets probed.
307
- const summary =
308
- identified.kind === 'pmtiles'
309
- ? await probePMTiles(absolute).catch(() => undefined)
310
- : undefined;
307
+ // Both formats that can be served get read for one. The kind comes from
308
+ // the content rather than the extension, which is what identify exists
309
+ // for.
310
+ const summary = await probeArchive(absolute, identified.kind).catch(
311
+ () => undefined,
312
+ );
311
313
 
312
314
  const created = await createTorrentFromFile(absolute, {
313
315
  creator: this.#creator(),
@@ -967,6 +969,11 @@ export class Library {
967
969
  // progress through runningAdds() and says "watch the log" as it closes.
968
970
  options.onValidated?.({ url, kind: identified.kind });
969
971
 
972
+ // PMTiles only, and not an oversight: this reads the archive over range
973
+ // requests before anything has been downloaded, and there is no partial
974
+ // read of a SQLite file that answers anything. An MBTiles mirrored from a
975
+ // URL is summarised by the warmer once the download finishes, which is the
976
+ // first moment it can be.
970
977
  const summary =
971
978
  identified.kind === 'pmtiles'
972
979
  ? await probePMTiles(url).catch(() => undefined)
@@ -1605,8 +1612,9 @@ export class Library {
1605
1612
  // facts, and probing on the second one's behalf would hang.
1606
1613
  let summary;
1607
1614
  if (readable && torrent.progress === 1 && torrent.savePath) {
1608
- summary = await probePMTiles(
1615
+ summary = await probeArchive(
1609
1616
  path.join(torrent.savePath, torrent.name),
1617
+ guessKind(torrent.name ?? ''),
1610
1618
  ).catch(() => undefined);
1611
1619
  }
1612
1620
 
@@ -3383,6 +3391,30 @@ export function guessKind(name) {
3383
3391
  return undefined;
3384
3392
  }
3385
3393
 
3394
+ /**
3395
+ * Reads the summary out of whichever kind of archive this is.
3396
+ *
3397
+ * Only PMTiles was ever probed, which was right when only PMTiles could be
3398
+ * served. MBTiles became servable from a complete local copy and this did not
3399
+ * follow, so those entries carried no summary at all -- and a summary is what
3400
+ * everything downstream reads coverage from. A stack naming one advertised the
3401
+ * fallbacks in its TileJSON, z0-z14 over the whole world, rather than the zoom
3402
+ * range and bounds the archive states about itself.
3403
+ *
3404
+ * Anything else is left alone rather than guessed at. A .osm.pbf being mirrored
3405
+ * has no summary to read and asking for one is how it ends up reported as a
3406
+ * broken map instead of a file being distributed.
3407
+ * @param {string} location - Local path, or a URL for PMTiles.
3408
+ * @param {string} [kind] - What it was identified as; the name decides if not.
3409
+ * @returns {Promise<object|undefined>} - The summary, where there is one.
3410
+ */
3411
+ export async function probeArchive(location, kind) {
3412
+ const format = kind ?? guessKind(location);
3413
+ if (format === 'mbtiles') return probeMbtiles(location);
3414
+ if (format === 'pmtiles') return probePMTiles(location);
3415
+ return undefined;
3416
+ }
3417
+
3386
3418
  /**
3387
3419
  * Moves a file, whether or not the two paths share a filesystem.
3388
3420
  *
package/src/mbtiles.js CHANGED
@@ -1,4 +1,5 @@
1
1
  import path from 'node:path';
2
+ import { summarize, tileTypeFor } from './archive-summary.js';
2
3
 
3
4
  /**
4
5
  * Reading tiles out of a completed MBTiles archive.
@@ -20,17 +21,6 @@ import path from 'node:path';
20
21
  * MBTiles 1.3: https://github.com/mapbox/mbtiles-spec/blob/master/1.3/spec.md
21
22
  */
22
23
 
23
- /** Maps an MBTiles `format` onto the PMTiles tile-type number summarize expects. */
24
- const TILE_TYPES = {
25
- pbf: 1,
26
- mvt: 1,
27
- png: 2,
28
- jpg: 3,
29
- jpeg: 3,
30
- webp: 4,
31
- avif: 5,
32
- };
33
-
34
24
  /** The whole world, for an archive that declares no bounds. */
35
25
  const WHOLE_WORLD = [-180, -85.051129, 180, 85.051129];
36
26
 
@@ -165,7 +155,7 @@ export class MbtilesArchive {
165
155
  return {
166
156
  // MBTiles has no equivalent, and nothing reads this except to report it.
167
157
  specVersion: 3,
168
- tileType: TILE_TYPES[format] ?? 0,
158
+ tileType: tileTypeFor(format),
169
159
  minZoom,
170
160
  maxZoom,
171
161
  minLon: bounds[0],
@@ -199,6 +189,17 @@ export class MbtilesArchive {
199
189
  // Same key tileserver-gl reads, carried through so an MBTiles archive
200
190
  // gets the same treatment a PMTiles one does.
201
191
  sparse: meta.sparse,
192
+ // How heights are packed into the pixels, and the four factors a custom
193
+ // packing is unreadable without. Not in the MBTiles spec -- neither is
194
+ // `sparse` -- but this is where every tool that needs it puts it, and a
195
+ // terrain archive that loses it is decoded as whatever the reader
196
+ // assumes. That is not a tile that fails; it is a tile of wrong heights,
197
+ // which is worse.
198
+ encoding: meta.encoding,
199
+ redFactor: meta.redFactor,
200
+ greenFactor: meta.greenFactor,
201
+ blueFactor: meta.blueFactor,
202
+ baseShift: meta.baseShift,
202
203
  };
203
204
  }
204
205
 
@@ -254,3 +255,32 @@ export class MbtilesArchive {
254
255
  }
255
256
  }
256
257
  }
258
+
259
+ /**
260
+ * Reads an MBTiles archive's summary, the way probePMTiles reads a PMTiles one.
261
+ *
262
+ * The same summariser, off the same two methods -- which is the whole point of
263
+ * the adapter above: MBTiles keeps in a metadata table what PMTiles keeps in a
264
+ * fixed header, and everything upstream of `summarize` should not have to know
265
+ * which it was reading. `getHeader` even derives the zoom range from the tiles
266
+ * table where the metadata omits it, so a summary comes back for an archive
267
+ * that declares almost nothing about itself.
268
+ *
269
+ * Local paths only, and not by oversight. probePMTiles takes a URL because
270
+ * PMTiles is read a byte range at a time, and SQLite is not: there is no
271
+ * partial read of an MBTiles that answers anything, which is the same reason
272
+ * it is served only from a complete local copy.
273
+ * @param {string} file - Path to the .mbtiles file.
274
+ * @returns {Promise<object>} - The summary, in the shape the catalog keeps.
275
+ */
276
+ export async function probeMbtiles(file) {
277
+ const archive = await MbtilesArchive.open(file);
278
+ try {
279
+ return summarize(
280
+ await archive.getHeader(),
281
+ (await archive.getMetadata()) ?? {},
282
+ );
283
+ } finally {
284
+ archive.close();
285
+ }
286
+ }
@@ -1,97 +1,15 @@
1
1
  import { FetchSource, PMTiles } from 'pmtiles';
2
+ import { summarize } from './archive-summary.js';
2
3
  import { NodeFileSource } from './file-source.js';
3
4
 
4
5
  /**
5
- * Reads the facts about an archive that a subscriber needs in order to decide
6
- * whether they want it: coverage, zoom range, tile type, size.
6
+ * Reading a PMTiles archive's summary, from a local path or over HTTP.
7
7
  *
8
- * This is what makes a map RSS feed more useful than a generic torrent feed —
9
- * an item can say "raster webp, z0-14, covering Switzerland, 36 GiB" instead of
10
- * just a filename, so a consumer can filter before committing to a download.
8
+ * The summary itself is built in archive-summary.js, which knows nothing about
9
+ * either format. What is here is the half that is PMTiles: opening one, and
10
+ * where its header and metadata are kept.
11
11
  */
12
12
 
13
- /** Maps PMTiles tile type numbers onto names and content types. */
14
- const TILE_TYPES = {
15
- 0: { format: 'unknown', contentType: 'application/octet-stream' },
16
- 1: { format: 'pbf', contentType: 'application/x-protobuf' },
17
- 2: { format: 'png', contentType: 'image/png' },
18
- 3: { format: 'jpeg', contentType: 'image/jpeg' },
19
- 4: { format: 'webp', contentType: 'image/webp' },
20
- 5: { format: 'avif', contentType: 'image/avif' },
21
- // No MIME is registered for MLT, and nothing selects a tile by content type.
22
- 6: { format: 'mlt', contentType: 'application/octet-stream' },
23
- };
24
-
25
- /**
26
- * Summary of an archive, as stored in the catalog and published in the feed.
27
- * @typedef {object} PMTilesSummary
28
- * @property {number} specVersion - PMTiles spec version.
29
- * @property {string} format - Tile format: pbf, png, jpeg, webp, avif.
30
- * @property {string} contentType - Matching content type.
31
- * @property {number} minZoom - Lowest zoom present.
32
- * @property {number} maxZoom - Highest zoom present.
33
- * @property {number[]} bounds - [minLon, minLat, maxLon, maxLat].
34
- * @property {number[]} center - [lon, lat, zoom].
35
- * @property {number} tileCount - Addressed tile count.
36
- * @property {boolean} clustered - Whether tiles are stored in Hilbert order.
37
- * @property {string} [name] - Name from the archive metadata.
38
- * @property {string} [description] - Description from the archive metadata.
39
- * @property {string} [attribution] - Attribution from the archive metadata.
40
- * @property {object[]} [vectorLayers] - Vector layer definitions, for pbf archives.
41
- * @property {boolean} [sparse] - What the archive says about missing tiles, if it says anything.
42
- */
43
-
44
- /**
45
- * Reads a flag out of archive metadata, which is not reliably typed.
46
- *
47
- * A PMTiles JSON blob carries a real boolean, but the same metadata routinely
48
- * arrives having been round-tripped through MBTiles, where every value is TEXT
49
- * — so the honest reading of `"false"` is false, not "a non-empty string".
50
- * @param {unknown} value - Whatever the metadata held.
51
- * @returns {boolean | undefined} - The flag, or undefined if it said nothing.
52
- */
53
- export function metadataFlag(value) {
54
- if (value === undefined || value === null || value === '') return undefined;
55
- if (typeof value === 'boolean') return value;
56
- if (typeof value === 'number') return value !== 0;
57
- const text = String(value).trim().toLowerCase();
58
- if (['true', '1', 'yes'].includes(text)) return true;
59
- if (['false', '0', 'no'].includes(text)) return false;
60
- return undefined;
61
- }
62
-
63
- /**
64
- * The tile encoding an archive declares, if it is one MapLibre understands.
65
- *
66
- * See docs/tilejson.md — `encoding`.
67
- * @param {unknown} value - Whatever the metadata held.
68
- * @returns {string | undefined} - A known encoding, or undefined.
69
- */
70
- export function metadataEncoding(value) {
71
- if (typeof value !== 'string') return undefined;
72
- const text = value.trim().toLowerCase();
73
- // An unrecognised value is dropped: it would cost a client its own default.
74
- return ['terrarium', 'mapbox', 'custom', 'mlt'].includes(text)
75
- ? text
76
- : undefined;
77
- }
78
-
79
- /**
80
- * The four factors a `custom` encoding is unreadable without. All or nothing.
81
- * @param {object} metadata - The archive's metadata.
82
- * @returns {object | undefined} - The four factors, or undefined.
83
- */
84
- export function customEncodingFactors(metadata) {
85
- const named = ['redFactor', 'greenFactor', 'blueFactor', 'baseShift'];
86
- const out = {};
87
- for (const name of named) {
88
- const value = Number(metadata[name]);
89
- if (!Number.isFinite(value)) return undefined;
90
- out[name] = value;
91
- }
92
- return out;
93
- }
94
-
95
13
  /**
96
14
  * Reads an archive's header and metadata.
97
15
  *
@@ -99,72 +17,7 @@ export function customEncodingFactors(metadata) {
99
17
  * even for a multi-terabyte archive — and it works against an HTTP URL without
100
18
  * downloading it.
101
19
  * @param {string} location - Local path or http(s) URL.
102
- * @returns {Promise<PMTilesSummary>} - The summary.
103
- */
104
- /**
105
- * What this prober reads. Raise it whenever a field is added to the summary,
106
- * and archives probed by an older build are re-read once. See
107
- * docs/internals.md — "Re-reading a summary an older prober wrote".
108
- */
109
- export const SUMMARY_VERSION = 3;
110
-
111
- /**
112
- * What the catalog keeps about an archive, from its header and metadata.
113
- * @param {object} header - A parsed PMTiles v3 header.
114
- * @param {object} [metadata] - The archive's own metadata document.
115
- * @returns {object} - The summary, at `SUMMARY_VERSION`.
116
- */
117
- export function summarize(header, metadata = {}) {
118
- const type = TILE_TYPES[header.tileType] ?? TILE_TYPES[0];
119
-
120
- // An archive with no bounds set reports all zeroes; treat that as global
121
- // rather than as a point at null island.
122
- const hasBounds = !(
123
- header.minLon === 0 &&
124
- header.minLat === 0 &&
125
- header.maxLon === 0 &&
126
- header.maxLat === 0
127
- );
128
-
129
- return {
130
- summaryVersion: SUMMARY_VERSION,
131
- specVersion: header.specVersion,
132
- format: type.format,
133
- contentType: type.contentType,
134
- minZoom: header.minZoom,
135
- maxZoom: header.maxZoom,
136
- bounds: hasBounds
137
- ? [header.minLon, header.minLat, header.maxLon, header.maxLat]
138
- : [-180, -85.051129, 180, 85.051129],
139
- center: [
140
- header.centerLon,
141
- header.centerLat,
142
- header.centerZoom || Math.round(header.maxZoom / 2),
143
- ],
144
- tileCount: header.numAddressedTiles,
145
- clustered: header.clustered,
146
- name: metadata.name,
147
- description: metadata.description,
148
- attribution: metadata.attribution,
149
- vectorLayers: metadata.vector_layers,
150
- // What the archive says about its own missing tiles. tileserver-gl reads
151
- // the same key, so an archive built to be served there carries the answer
152
- // with it and does not have to be configured again here.
153
- sparse: metadataFlag(metadata.sparse),
154
- // The header settles MLT; only the metadata can settle elevation packing.
155
- encoding:
156
- type.format === 'mlt' ? 'mlt' : metadataEncoding(metadata.encoding),
157
- encodingFactors:
158
- metadataEncoding(metadata.encoding) === 'custom'
159
- ? customEncodingFactors(metadata)
160
- : undefined,
161
- };
162
- }
163
-
164
- /**
165
- * Reads an archive's header and metadata.
166
- * @param {string} location - Local path or http(s) URL.
167
- * @returns {Promise<PMTilesSummary>} - The summary.
20
+ * @returns {Promise<ArchiveSummary>} - The summary.
168
21
  */
169
22
  export async function probePMTiles(location) {
170
23
  const isHttp = /^https?:\/\//i.test(location);
package/src/prewarm.js CHANGED
@@ -156,7 +156,14 @@ export class HeadWarmer {
156
156
  // not, since an archive joined by magnet has no kind until its metadata
157
157
  // arrives.
158
158
  const kind = entry.kind ?? guessKind(entry.name ?? '');
159
- if (kind !== 'pmtiles') return false;
159
+ if (kind !== 'pmtiles' && kind !== 'mbtiles') return false;
160
+
161
+ // MBTiles has no head to pull out of the swarm -- it is SQLite, read whole
162
+ // or not at all -- so there is nothing here to be due until the file is.
163
+ // Once it is, the read is local and instant, and it is the only thing that
164
+ // gives an archive already in the catalog the summary this did not use to
165
+ // record for it.
166
+ if (kind === 'mbtiles' && entry.complete !== true) return false;
160
167
 
161
168
  // A summary is only an answer about *this disk* if a header on this disk
162
169
  // produced it. `format` was standing in for that and does not mean it.
@@ -289,6 +296,11 @@ export class HeadWarmer {
289
296
  `[warm] ${entry.name}: header read; its metadata is at the far end ` +
290
297
  'of the archive and has not arrived yet',
291
298
  );
299
+ } else if ((entry.kind ?? guessKind(entry.name ?? '')) === 'mbtiles') {
300
+ // Not a header: MBTiles keeps in a metadata table what PMTiles keeps
301
+ // in a fixed header, and a log saying otherwise sends whoever reads it
302
+ // looking for the wrong thing.
303
+ console.log(`[warm] ${entry.name}: metadata read`);
292
304
  } else {
293
305
  console.log(`[warm] ${entry.name}: header read`);
294
306
  }
package/src/tiles.js CHANGED
@@ -7,7 +7,7 @@ import { TorrentSource } from 'pmtiles-torrent';
7
7
  import { NodeFileSource } from './file-source.js';
8
8
  import { onDiskPath } from './incomplete.js';
9
9
  import { MbtilesArchive, isMbtiles } from './mbtiles.js';
10
- import { summarize } from './pmtiles-probe.js';
10
+ import { summarize } from './archive-summary.js';
11
11
  import { LibtorrentReadEngine } from './read-engine.js';
12
12
 
13
13
  /**