pmtiles-swarm 0.97.0 → 0.98.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +49 -0
- package/docs/tile-stacks.md +35 -3
- package/package.json +1 -1
- package/src/api.js +1 -1
- package/src/archive-summary.js +183 -0
- package/src/feed.js +1 -1
- package/src/library.js +38 -6
- package/src/mbtiles.js +42 -12
- package/src/pmtiles-probe.js +6 -153
- package/src/prewarm.js +13 -1
- package/src/tiles.js +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,55 @@
|
|
|
7
7
|
### 🐞 Bug fixes
|
|
8
8
|
- _...Add new stuff here..._
|
|
9
9
|
|
|
10
|
+
## 0.98.1
|
|
11
|
+
### ✨ Features and improvements
|
|
12
|
+
- **The two archive readers no longer import each other.** Summarising an MBTiles archive reached
|
|
13
|
+
into `pmtiles-probe.js` for `summarize`, which put one format's reader inside the other's module
|
|
14
|
+
and dragged the `pmtiles` package in behind it. The half neither format owns — the summary shape,
|
|
15
|
+
its version, the metadata readers and the tile-type table — is now `archive-summary.js`, which
|
|
16
|
+
imports nothing and opens nothing: it takes a header and a metadata document and returns a
|
|
17
|
+
summary. `pmtiles-probe.js` keeps the PMTiles half and `mbtiles.js` the SQLite half, and neither
|
|
18
|
+
reaches the other.
|
|
19
|
+
|
|
20
|
+
Nothing about what the node serves changes; this is where the code lives.
|
|
21
|
+
|
|
22
|
+
### 🐞 Bug fixes
|
|
23
|
+
- **An MBTiles archive holding MLT tiles was summarised as an unknown format.** The table mapping a
|
|
24
|
+
format name onto a tile type had been written twice, in opposite directions, and the copy beside
|
|
25
|
+
the MBTiles reader was written by hand and had no entry for `mlt` — so an archive declaring it
|
|
26
|
+
came back as `unknown`, which is what the catalog then reported and what a stack over it read.
|
|
27
|
+
There is one table now, with the reverse lookup derived from it, so the two halves cannot drift
|
|
28
|
+
apart again.
|
|
29
|
+
|
|
30
|
+
## 0.98.0
|
|
31
|
+
### ✨ Features and improvements
|
|
32
|
+
- **A finished MBTiles archive works as a stack layer, and now says what it holds.** It always read:
|
|
33
|
+
the tile store opens one from a complete local copy, and a stack asks the store for tiles like
|
|
34
|
+
anything else. What it could not do was describe itself. Only PMTiles was ever probed for a
|
|
35
|
+
summary — right when only PMTiles could be served, and never revisited once MBTiles became
|
|
36
|
+
servable — so those entries carried none, and a summary is what `stackCoverage` reads. A stack
|
|
37
|
+
naming one advertised the fallbacks in its TileJSON, z0–z14 over the whole world, however narrow
|
|
38
|
+
the archive actually was. Nothing refused it and nothing warned: the recipe was valid, the tiles
|
|
39
|
+
were right, and only the coverage was a fiction.
|
|
40
|
+
|
|
41
|
+
Both formats that can be served are read for a summary now, through the same summariser —
|
|
42
|
+
`probeMbtiles` off the adapter's `getHeader` and `getMetadata`, which is what that adapter exists
|
|
43
|
+
for. Most of the MBTiles spec is optional, so where an archive states nothing the reader derives
|
|
44
|
+
what it can: the zoom range comes from the tiles table when `minzoom` and `maxzoom` are absent.
|
|
45
|
+
|
|
46
|
+
The metadata reader also carries `encoding` through, with the four factors a custom packing is
|
|
47
|
+
unreadable without. An elevation stack decodes the pixels rather than passing them through, so an
|
|
48
|
+
archive that lost its encoding was not a tile that failed but a tile of wrong heights.
|
|
49
|
+
|
|
50
|
+
Archives already in the catalog are summarised by the head warmer on its next pass, once complete
|
|
51
|
+
— an MBTiles has no head to pull out of a swarm, so there is nothing to be due until the file is
|
|
52
|
+
whole, and then the read is local and instant. Unchanged: `/latest/<category>` still offers no
|
|
53
|
+
tile endpoint for an MBTiles, because a category is a promise to every node and this one is only
|
|
54
|
+
keepable on a node holding the whole file. See [tile-stacks.md](docs/tile-stacks.md) — "An
|
|
55
|
+
MBTiles archive as a source".
|
|
56
|
+
|
|
57
|
+
### 🐞 Bug fixes
|
|
58
|
+
|
|
10
59
|
## 0.97.0
|
|
11
60
|
### ✨ Features and improvements
|
|
12
61
|
- **A stack can be told to forget what it has merged, and forgets it by itself when its recipe
|
package/docs/tile-stacks.md
CHANGED
|
@@ -29,6 +29,7 @@ of its parts.
|
|
|
29
29
|
- [Painting order](#painting-order)
|
|
30
30
|
- [The two pixel spaces](#the-two-pixel-spaces)
|
|
31
31
|
- [Naming a source](#naming-a-source)
|
|
32
|
+
- [An MBTiles archive as a source](#an-mbtiles-archive-as-a-source)
|
|
32
33
|
- [The config file](#the-config-file)
|
|
33
34
|
- [Translating a rio-rgbify-merge config](#translating-a-rio-rgbify-merge-config)
|
|
34
35
|
- [Evaluating one tile](#evaluating-one-tile)
|
|
@@ -174,6 +175,35 @@ something else.
|
|
|
174
175
|
A stack may mix the two. Whether _any_ source is category-resolved decides the
|
|
175
176
|
whole stack's caching headers, below.
|
|
176
177
|
|
|
178
|
+
### An MBTiles archive as a source
|
|
179
|
+
|
|
180
|
+
Either form may name an MBTiles archive, with one condition: **this node must
|
|
181
|
+
hold the complete file.** MBTiles is SQLite, whose pages are scattered rather
|
|
182
|
+
than spatially clustered, so it cannot be read a byte range at a time out of a
|
|
183
|
+
swarm the way PMTiles can — a source still arriving answers 503, and no amount
|
|
184
|
+
of waiting for the right pieces changes that. A stack with an MBTiles source is
|
|
185
|
+
therefore only servable on nodes that have finished downloading it.
|
|
186
|
+
|
|
187
|
+
Everything else is the same. It resolves, merges, clips, feathers and bakes
|
|
188
|
+
like any other source, because the tile store presents it through the same
|
|
189
|
+
three methods a PMTiles archive answers.
|
|
190
|
+
|
|
191
|
+
What it covers comes from the metadata table, read at import by `probeMbtiles`
|
|
192
|
+
and summarised by the same `summarize` the PMTiles path uses. Most of the
|
|
193
|
+
MBTiles spec is optional, so where an archive states nothing the reader derives
|
|
194
|
+
what it can — the zoom range comes from the tiles table itself when `minzoom`
|
|
195
|
+
and `maxzoom` are absent, and bounds default to the whole world.
|
|
196
|
+
|
|
197
|
+
Two keys outside the spec are read, both because this is where every tool that
|
|
198
|
+
needs them puts them: `sparse`, which tileserver-gl reads the same way, and
|
|
199
|
+
`encoding` — with `redFactor`, `greenFactor`, `blueFactor` and `baseShift` for
|
|
200
|
+
a custom packing. An elevation stack decodes the pixels rather than passing
|
|
201
|
+
them through, so an archive that loses its encoding is not a tile that fails but
|
|
202
|
+
a tile of wrong heights.
|
|
203
|
+
|
|
204
|
+
An archive already in the catalog from before any of this was probed is
|
|
205
|
+
summarised by the head warmer on its next pass, once it is complete.
|
|
206
|
+
|
|
177
207
|
## The config file
|
|
178
208
|
|
|
179
209
|
Stacks live in `data/stacks.json`, beside `data/catalog.json` and for the same
|
|
@@ -439,9 +469,11 @@ This is the one thing that stops a stack being a small feature, and it is
|
|
|
439
469
|
narrower than "the node cannot handle tiles". It already does, in most of the
|
|
440
470
|
ways a stack needs:
|
|
441
471
|
|
|
442
|
-
- `identify.js` reads an archive's magic bytes;
|
|
443
|
-
headers, directories and metadata; `mbtiles.js` queries tile rows
|
|
444
|
-
SQLite.
|
|
472
|
+
- `identify.js` reads an archive's magic bytes; `pmtiles-probe.js` reads PMTiles
|
|
473
|
+
headers, directories and metadata; `mbtiles.js` queries tile rows and the
|
|
474
|
+
metadata table out of SQLite. Neither format's reader imports the other's:
|
|
475
|
+
what they share is `archive-summary.js`, which turns a header and a metadata
|
|
476
|
+
document into the summary the catalog keeps and opens nothing itself.
|
|
445
477
|
- `TileStore.getTile` resolves an archive, reads a tile through the local file
|
|
446
478
|
or the swarm, and knows its format from the header rather than by guessing.
|
|
447
479
|
- The tile route already gzips vector tiles through `node:zlib`, abandons a
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pmtiles-swarm",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.98.1",
|
|
4
4
|
"description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "src/index.js",
|
package/src/api.js
CHANGED
|
@@ -38,7 +38,7 @@ import {
|
|
|
38
38
|
} from './sources.js';
|
|
39
39
|
import { limitFor, remaining } from './seeding.js';
|
|
40
40
|
import { buildTileJson, extensionMatches, tileExtension } from './tilejson.js';
|
|
41
|
-
import { SUMMARY_VERSION } from './
|
|
41
|
+
import { SUMMARY_VERSION } from './archive-summary.js';
|
|
42
42
|
import { TileReadError } from './tiles.js';
|
|
43
43
|
import { loadCodec } from './codec.js';
|
|
44
44
|
import { answerStackTile, outputFormat, outputSize } from './stack-tile.js';
|
|
@@ -0,0 +1,183 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* The summary this project keeps about a map archive, and the vocabulary for
|
|
3
|
+
* building one.
|
|
4
|
+
*
|
|
5
|
+
* What a subscriber needs in order to decide whether they want an archive:
|
|
6
|
+
* coverage, zoom range, tile type, size. This is what makes a map RSS feed more
|
|
7
|
+
* useful than a generic torrent feed — an item can say "raster webp, z0-14,
|
|
8
|
+
* covering Switzerland, 36 GiB" instead of just a filename, so a consumer can
|
|
9
|
+
* filter before committing to a download.
|
|
10
|
+
*
|
|
11
|
+
* Here rather than beside either reader because both formats produce one, and
|
|
12
|
+
* neither should have to import the other to do it. Nothing in this file opens
|
|
13
|
+
* a file or a socket: it takes a header and a metadata document and returns the
|
|
14
|
+
* summary, which is why it can be shared without dragging one format's
|
|
15
|
+
* dependencies into the other's module.
|
|
16
|
+
*/
|
|
17
|
+
|
|
18
|
+
/**
|
|
19
|
+
* Maps tile type numbers onto names and content types.
|
|
20
|
+
*
|
|
21
|
+
* The numbering is PMTiles', because a number was needed and that one is
|
|
22
|
+
* written down. An MBTiles archive names its format in a metadata row instead,
|
|
23
|
+
* and `tileTypeFor` is how it says the same thing in the same terms.
|
|
24
|
+
*/
|
|
25
|
+
const TILE_TYPES = {
|
|
26
|
+
0: { format: 'unknown', contentType: 'application/octet-stream' },
|
|
27
|
+
1: { format: 'pbf', contentType: 'application/x-protobuf' },
|
|
28
|
+
2: { format: 'png', contentType: 'image/png' },
|
|
29
|
+
3: { format: 'jpeg', contentType: 'image/jpeg' },
|
|
30
|
+
4: { format: 'webp', contentType: 'image/webp' },
|
|
31
|
+
5: { format: 'avif', contentType: 'image/avif' },
|
|
32
|
+
// No MIME is registered for MLT, and nothing selects a tile by content type.
|
|
33
|
+
6: { format: 'mlt', contentType: 'application/octet-stream' },
|
|
34
|
+
};
|
|
35
|
+
|
|
36
|
+
/**
|
|
37
|
+
* The tile type number for a format an archive names in words.
|
|
38
|
+
*
|
|
39
|
+
* The inverse of the table above, kept beside it so the two halves cannot
|
|
40
|
+
* drift. `mvt` and `jpg` are spellings the same formats appear under in the
|
|
41
|
+
* wild; anything unrecognised is 0, which reads back as "unknown" rather than
|
|
42
|
+
* as a format nothing can serve.
|
|
43
|
+
* @param {string} [format] - What the archive calls its tiles.
|
|
44
|
+
* @returns {number} - The tile type number.
|
|
45
|
+
*/
|
|
46
|
+
export function tileTypeFor(format) {
|
|
47
|
+
const named = String(format ?? '').toLowerCase();
|
|
48
|
+
const found = Object.entries(TILE_TYPES).find(
|
|
49
|
+
([, type]) => type.format === named,
|
|
50
|
+
);
|
|
51
|
+
if (found) return Number(found[0]);
|
|
52
|
+
return { mvt: 1, jpg: 3 }[named] ?? 0;
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
/**
|
|
56
|
+
* Summary of an archive, as stored in the catalog and published in the feed.
|
|
57
|
+
* @typedef {object} ArchiveSummary
|
|
58
|
+
* @property {number} specVersion - PMTiles spec version.
|
|
59
|
+
* @property {string} format - Tile format: pbf, png, jpeg, webp, avif.
|
|
60
|
+
* @property {string} contentType - Matching content type.
|
|
61
|
+
* @property {number} minZoom - Lowest zoom present.
|
|
62
|
+
* @property {number} maxZoom - Highest zoom present.
|
|
63
|
+
* @property {number[]} bounds - [minLon, minLat, maxLon, maxLat].
|
|
64
|
+
* @property {number[]} center - [lon, lat, zoom].
|
|
65
|
+
* @property {number} tileCount - Addressed tile count.
|
|
66
|
+
* @property {boolean} clustered - Whether tiles are stored in Hilbert order.
|
|
67
|
+
* @property {string} [name] - Name from the archive metadata.
|
|
68
|
+
* @property {string} [description] - Description from the archive metadata.
|
|
69
|
+
* @property {string} [attribution] - Attribution from the archive metadata.
|
|
70
|
+
* @property {object[]} [vectorLayers] - Vector layer definitions, for pbf archives.
|
|
71
|
+
* @property {boolean} [sparse] - What the archive says about missing tiles, if it says anything.
|
|
72
|
+
*/
|
|
73
|
+
|
|
74
|
+
/**
|
|
75
|
+
* Reads a flag out of archive metadata, which is not reliably typed.
|
|
76
|
+
*
|
|
77
|
+
* A PMTiles JSON blob carries a real boolean, but the same metadata routinely
|
|
78
|
+
* arrives having been round-tripped through MBTiles, where every value is TEXT
|
|
79
|
+
* — so the honest reading of `"false"` is false, not "a non-empty string".
|
|
80
|
+
* @param {unknown} value - Whatever the metadata held.
|
|
81
|
+
* @returns {boolean | undefined} - The flag, or undefined if it said nothing.
|
|
82
|
+
*/
|
|
83
|
+
export function metadataFlag(value) {
|
|
84
|
+
if (value === undefined || value === null || value === '') return undefined;
|
|
85
|
+
if (typeof value === 'boolean') return value;
|
|
86
|
+
if (typeof value === 'number') return value !== 0;
|
|
87
|
+
const text = String(value).trim().toLowerCase();
|
|
88
|
+
if (['true', '1', 'yes'].includes(text)) return true;
|
|
89
|
+
if (['false', '0', 'no'].includes(text)) return false;
|
|
90
|
+
return undefined;
|
|
91
|
+
}
|
|
92
|
+
|
|
93
|
+
/**
|
|
94
|
+
* The tile encoding an archive declares, if it is one MapLibre understands.
|
|
95
|
+
*
|
|
96
|
+
* See docs/tilejson.md — `encoding`.
|
|
97
|
+
* @param {unknown} value - Whatever the metadata held.
|
|
98
|
+
* @returns {string | undefined} - A known encoding, or undefined.
|
|
99
|
+
*/
|
|
100
|
+
export function metadataEncoding(value) {
|
|
101
|
+
if (typeof value !== 'string') return undefined;
|
|
102
|
+
const text = value.trim().toLowerCase();
|
|
103
|
+
// An unrecognised value is dropped: it would cost a client its own default.
|
|
104
|
+
return ['terrarium', 'mapbox', 'custom', 'mlt'].includes(text)
|
|
105
|
+
? text
|
|
106
|
+
: undefined;
|
|
107
|
+
}
|
|
108
|
+
|
|
109
|
+
/**
|
|
110
|
+
* The four factors a `custom` encoding is unreadable without. All or nothing.
|
|
111
|
+
* @param {object} metadata - The archive's metadata.
|
|
112
|
+
* @returns {object | undefined} - The four factors, or undefined.
|
|
113
|
+
*/
|
|
114
|
+
export function customEncodingFactors(metadata) {
|
|
115
|
+
const named = ['redFactor', 'greenFactor', 'blueFactor', 'baseShift'];
|
|
116
|
+
const out = {};
|
|
117
|
+
for (const name of named) {
|
|
118
|
+
const value = Number(metadata[name]);
|
|
119
|
+
if (!Number.isFinite(value)) return undefined;
|
|
120
|
+
out[name] = value;
|
|
121
|
+
}
|
|
122
|
+
return out;
|
|
123
|
+
}
|
|
124
|
+
|
|
125
|
+
/**
|
|
126
|
+
* What this prober reads. Raise it whenever a field is added to the summary,
|
|
127
|
+
* and archives probed by an older build are re-read once. See
|
|
128
|
+
* docs/internals.md — "Re-reading a summary an older prober wrote".
|
|
129
|
+
*/
|
|
130
|
+
export const SUMMARY_VERSION = 3;
|
|
131
|
+
|
|
132
|
+
/**
|
|
133
|
+
* What the catalog keeps about an archive, from its header and metadata.
|
|
134
|
+
* @param {object} header - A parsed PMTiles v3 header.
|
|
135
|
+
* @param {object} [metadata] - The archive's own metadata document.
|
|
136
|
+
* @returns {object} - The summary, at `SUMMARY_VERSION`.
|
|
137
|
+
*/
|
|
138
|
+
export function summarize(header, metadata = {}) {
|
|
139
|
+
const type = TILE_TYPES[header.tileType] ?? TILE_TYPES[0];
|
|
140
|
+
|
|
141
|
+
// An archive with no bounds set reports all zeroes; treat that as global
|
|
142
|
+
// rather than as a point at null island.
|
|
143
|
+
const hasBounds = !(
|
|
144
|
+
header.minLon === 0 &&
|
|
145
|
+
header.minLat === 0 &&
|
|
146
|
+
header.maxLon === 0 &&
|
|
147
|
+
header.maxLat === 0
|
|
148
|
+
);
|
|
149
|
+
|
|
150
|
+
return {
|
|
151
|
+
summaryVersion: SUMMARY_VERSION,
|
|
152
|
+
specVersion: header.specVersion,
|
|
153
|
+
format: type.format,
|
|
154
|
+
contentType: type.contentType,
|
|
155
|
+
minZoom: header.minZoom,
|
|
156
|
+
maxZoom: header.maxZoom,
|
|
157
|
+
bounds: hasBounds
|
|
158
|
+
? [header.minLon, header.minLat, header.maxLon, header.maxLat]
|
|
159
|
+
: [-180, -85.051129, 180, 85.051129],
|
|
160
|
+
center: [
|
|
161
|
+
header.centerLon,
|
|
162
|
+
header.centerLat,
|
|
163
|
+
header.centerZoom || Math.round(header.maxZoom / 2),
|
|
164
|
+
],
|
|
165
|
+
tileCount: header.numAddressedTiles,
|
|
166
|
+
clustered: header.clustered,
|
|
167
|
+
name: metadata.name,
|
|
168
|
+
description: metadata.description,
|
|
169
|
+
attribution: metadata.attribution,
|
|
170
|
+
vectorLayers: metadata.vector_layers,
|
|
171
|
+
// What the archive says about its own missing tiles. tileserver-gl reads
|
|
172
|
+
// the same key, so an archive built to be served there carries the answer
|
|
173
|
+
// with it and does not have to be configured again here.
|
|
174
|
+
sparse: metadataFlag(metadata.sparse),
|
|
175
|
+
// The header settles MLT; only the metadata can settle elevation packing.
|
|
176
|
+
encoding:
|
|
177
|
+
type.format === 'mlt' ? 'mlt' : metadataEncoding(metadata.encoding),
|
|
178
|
+
encodingFactors:
|
|
179
|
+
metadataEncoding(metadata.encoding) === 'custom'
|
|
180
|
+
? customEncodingFactors(metadata)
|
|
181
|
+
: undefined,
|
|
182
|
+
};
|
|
183
|
+
}
|
package/src/feed.js
CHANGED
|
@@ -18,7 +18,7 @@
|
|
|
18
18
|
*/
|
|
19
19
|
|
|
20
20
|
import { mutableMagnet } from './mutable.js';
|
|
21
|
-
import { metadataEncoding } from './
|
|
21
|
+
import { metadataEncoding } from './archive-summary.js';
|
|
22
22
|
|
|
23
23
|
const PMTILES_NS = 'https://github.com/TechIdiots-LLC/pmtiles-swarm/ns/1.0';
|
|
24
24
|
|
package/src/library.js
CHANGED
|
@@ -14,6 +14,7 @@ import {
|
|
|
14
14
|
import { publishingBase, publishingFor, reachability } from './catalog.js';
|
|
15
15
|
import { linkLatest } from './latest-link.js';
|
|
16
16
|
import { checkOrigin, fingerprintOrigin } from './origin.js';
|
|
17
|
+
import { probeMbtiles } from './mbtiles.js';
|
|
17
18
|
import { probePMTiles } from './pmtiles-probe.js';
|
|
18
19
|
import { archiveDirName } from './savepath.js';
|
|
19
20
|
import {
|
|
@@ -303,11 +304,12 @@ export class Library {
|
|
|
303
304
|
});
|
|
304
305
|
|
|
305
306
|
try {
|
|
306
|
-
//
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
307
|
+
// Both formats that can be served get read for one. The kind comes from
|
|
308
|
+
// the content rather than the extension, which is what identify exists
|
|
309
|
+
// for.
|
|
310
|
+
const summary = await probeArchive(absolute, identified.kind).catch(
|
|
311
|
+
() => undefined,
|
|
312
|
+
);
|
|
311
313
|
|
|
312
314
|
const created = await createTorrentFromFile(absolute, {
|
|
313
315
|
creator: this.#creator(),
|
|
@@ -967,6 +969,11 @@ export class Library {
|
|
|
967
969
|
// progress through runningAdds() and says "watch the log" as it closes.
|
|
968
970
|
options.onValidated?.({ url, kind: identified.kind });
|
|
969
971
|
|
|
972
|
+
// PMTiles only, and not an oversight: this reads the archive over range
|
|
973
|
+
// requests before anything has been downloaded, and there is no partial
|
|
974
|
+
// read of a SQLite file that answers anything. An MBTiles mirrored from a
|
|
975
|
+
// URL is summarised by the warmer once the download finishes, which is the
|
|
976
|
+
// first moment it can be.
|
|
970
977
|
const summary =
|
|
971
978
|
identified.kind === 'pmtiles'
|
|
972
979
|
? await probePMTiles(url).catch(() => undefined)
|
|
@@ -1605,8 +1612,9 @@ export class Library {
|
|
|
1605
1612
|
// facts, and probing on the second one's behalf would hang.
|
|
1606
1613
|
let summary;
|
|
1607
1614
|
if (readable && torrent.progress === 1 && torrent.savePath) {
|
|
1608
|
-
summary = await
|
|
1615
|
+
summary = await probeArchive(
|
|
1609
1616
|
path.join(torrent.savePath, torrent.name),
|
|
1617
|
+
guessKind(torrent.name ?? ''),
|
|
1610
1618
|
).catch(() => undefined);
|
|
1611
1619
|
}
|
|
1612
1620
|
|
|
@@ -3383,6 +3391,30 @@ export function guessKind(name) {
|
|
|
3383
3391
|
return undefined;
|
|
3384
3392
|
}
|
|
3385
3393
|
|
|
3394
|
+
/**
|
|
3395
|
+
* Reads the summary out of whichever kind of archive this is.
|
|
3396
|
+
*
|
|
3397
|
+
* Only PMTiles was ever probed, which was right when only PMTiles could be
|
|
3398
|
+
* served. MBTiles became servable from a complete local copy and this did not
|
|
3399
|
+
* follow, so those entries carried no summary at all -- and a summary is what
|
|
3400
|
+
* everything downstream reads coverage from. A stack naming one advertised the
|
|
3401
|
+
* fallbacks in its TileJSON, z0-z14 over the whole world, rather than the zoom
|
|
3402
|
+
* range and bounds the archive states about itself.
|
|
3403
|
+
*
|
|
3404
|
+
* Anything else is left alone rather than guessed at. A .osm.pbf being mirrored
|
|
3405
|
+
* has no summary to read and asking for one is how it ends up reported as a
|
|
3406
|
+
* broken map instead of a file being distributed.
|
|
3407
|
+
* @param {string} location - Local path, or a URL for PMTiles.
|
|
3408
|
+
* @param {string} [kind] - What it was identified as; the name decides if not.
|
|
3409
|
+
* @returns {Promise<object|undefined>} - The summary, where there is one.
|
|
3410
|
+
*/
|
|
3411
|
+
export async function probeArchive(location, kind) {
|
|
3412
|
+
const format = kind ?? guessKind(location);
|
|
3413
|
+
if (format === 'mbtiles') return probeMbtiles(location);
|
|
3414
|
+
if (format === 'pmtiles') return probePMTiles(location);
|
|
3415
|
+
return undefined;
|
|
3416
|
+
}
|
|
3417
|
+
|
|
3386
3418
|
/**
|
|
3387
3419
|
* Moves a file, whether or not the two paths share a filesystem.
|
|
3388
3420
|
*
|
package/src/mbtiles.js
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import path from 'node:path';
|
|
2
|
+
import { summarize, tileTypeFor } from './archive-summary.js';
|
|
2
3
|
|
|
3
4
|
/**
|
|
4
5
|
* Reading tiles out of a completed MBTiles archive.
|
|
@@ -20,17 +21,6 @@ import path from 'node:path';
|
|
|
20
21
|
* MBTiles 1.3: https://github.com/mapbox/mbtiles-spec/blob/master/1.3/spec.md
|
|
21
22
|
*/
|
|
22
23
|
|
|
23
|
-
/** Maps an MBTiles `format` onto the PMTiles tile-type number summarize expects. */
|
|
24
|
-
const TILE_TYPES = {
|
|
25
|
-
pbf: 1,
|
|
26
|
-
mvt: 1,
|
|
27
|
-
png: 2,
|
|
28
|
-
jpg: 3,
|
|
29
|
-
jpeg: 3,
|
|
30
|
-
webp: 4,
|
|
31
|
-
avif: 5,
|
|
32
|
-
};
|
|
33
|
-
|
|
34
24
|
/** The whole world, for an archive that declares no bounds. */
|
|
35
25
|
const WHOLE_WORLD = [-180, -85.051129, 180, 85.051129];
|
|
36
26
|
|
|
@@ -165,7 +155,7 @@ export class MbtilesArchive {
|
|
|
165
155
|
return {
|
|
166
156
|
// MBTiles has no equivalent, and nothing reads this except to report it.
|
|
167
157
|
specVersion: 3,
|
|
168
|
-
tileType:
|
|
158
|
+
tileType: tileTypeFor(format),
|
|
169
159
|
minZoom,
|
|
170
160
|
maxZoom,
|
|
171
161
|
minLon: bounds[0],
|
|
@@ -199,6 +189,17 @@ export class MbtilesArchive {
|
|
|
199
189
|
// Same key tileserver-gl reads, carried through so an MBTiles archive
|
|
200
190
|
// gets the same treatment a PMTiles one does.
|
|
201
191
|
sparse: meta.sparse,
|
|
192
|
+
// How heights are packed into the pixels, and the four factors a custom
|
|
193
|
+
// packing is unreadable without. Not in the MBTiles spec -- neither is
|
|
194
|
+
// `sparse` -- but this is where every tool that needs it puts it, and a
|
|
195
|
+
// terrain archive that loses it is decoded as whatever the reader
|
|
196
|
+
// assumes. That is not a tile that fails; it is a tile of wrong heights,
|
|
197
|
+
// which is worse.
|
|
198
|
+
encoding: meta.encoding,
|
|
199
|
+
redFactor: meta.redFactor,
|
|
200
|
+
greenFactor: meta.greenFactor,
|
|
201
|
+
blueFactor: meta.blueFactor,
|
|
202
|
+
baseShift: meta.baseShift,
|
|
202
203
|
};
|
|
203
204
|
}
|
|
204
205
|
|
|
@@ -254,3 +255,32 @@ export class MbtilesArchive {
|
|
|
254
255
|
}
|
|
255
256
|
}
|
|
256
257
|
}
|
|
258
|
+
|
|
259
|
+
/**
|
|
260
|
+
* Reads an MBTiles archive's summary, the way probePMTiles reads a PMTiles one.
|
|
261
|
+
*
|
|
262
|
+
* The same summariser, off the same two methods -- which is the whole point of
|
|
263
|
+
* the adapter above: MBTiles keeps in a metadata table what PMTiles keeps in a
|
|
264
|
+
* fixed header, and everything upstream of `summarize` should not have to know
|
|
265
|
+
* which it was reading. `getHeader` even derives the zoom range from the tiles
|
|
266
|
+
* table where the metadata omits it, so a summary comes back for an archive
|
|
267
|
+
* that declares almost nothing about itself.
|
|
268
|
+
*
|
|
269
|
+
* Local paths only, and not by oversight. probePMTiles takes a URL because
|
|
270
|
+
* PMTiles is read a byte range at a time, and SQLite is not: there is no
|
|
271
|
+
* partial read of an MBTiles that answers anything, which is the same reason
|
|
272
|
+
* it is served only from a complete local copy.
|
|
273
|
+
* @param {string} file - Path to the .mbtiles file.
|
|
274
|
+
* @returns {Promise<object>} - The summary, in the shape the catalog keeps.
|
|
275
|
+
*/
|
|
276
|
+
export async function probeMbtiles(file) {
|
|
277
|
+
const archive = await MbtilesArchive.open(file);
|
|
278
|
+
try {
|
|
279
|
+
return summarize(
|
|
280
|
+
await archive.getHeader(),
|
|
281
|
+
(await archive.getMetadata()) ?? {},
|
|
282
|
+
);
|
|
283
|
+
} finally {
|
|
284
|
+
archive.close();
|
|
285
|
+
}
|
|
286
|
+
}
|
package/src/pmtiles-probe.js
CHANGED
|
@@ -1,97 +1,15 @@
|
|
|
1
1
|
import { FetchSource, PMTiles } from 'pmtiles';
|
|
2
|
+
import { summarize } from './archive-summary.js';
|
|
2
3
|
import { NodeFileSource } from './file-source.js';
|
|
3
4
|
|
|
4
5
|
/**
|
|
5
|
-
*
|
|
6
|
-
* whether they want it: coverage, zoom range, tile type, size.
|
|
6
|
+
* Reading a PMTiles archive's summary, from a local path or over HTTP.
|
|
7
7
|
*
|
|
8
|
-
*
|
|
9
|
-
*
|
|
10
|
-
*
|
|
8
|
+
* The summary itself is built in archive-summary.js, which knows nothing about
|
|
9
|
+
* either format. What is here is the half that is PMTiles: opening one, and
|
|
10
|
+
* where its header and metadata are kept.
|
|
11
11
|
*/
|
|
12
12
|
|
|
13
|
-
/** Maps PMTiles tile type numbers onto names and content types. */
|
|
14
|
-
const TILE_TYPES = {
|
|
15
|
-
0: { format: 'unknown', contentType: 'application/octet-stream' },
|
|
16
|
-
1: { format: 'pbf', contentType: 'application/x-protobuf' },
|
|
17
|
-
2: { format: 'png', contentType: 'image/png' },
|
|
18
|
-
3: { format: 'jpeg', contentType: 'image/jpeg' },
|
|
19
|
-
4: { format: 'webp', contentType: 'image/webp' },
|
|
20
|
-
5: { format: 'avif', contentType: 'image/avif' },
|
|
21
|
-
// No MIME is registered for MLT, and nothing selects a tile by content type.
|
|
22
|
-
6: { format: 'mlt', contentType: 'application/octet-stream' },
|
|
23
|
-
};
|
|
24
|
-
|
|
25
|
-
/**
|
|
26
|
-
* Summary of an archive, as stored in the catalog and published in the feed.
|
|
27
|
-
* @typedef {object} PMTilesSummary
|
|
28
|
-
* @property {number} specVersion - PMTiles spec version.
|
|
29
|
-
* @property {string} format - Tile format: pbf, png, jpeg, webp, avif.
|
|
30
|
-
* @property {string} contentType - Matching content type.
|
|
31
|
-
* @property {number} minZoom - Lowest zoom present.
|
|
32
|
-
* @property {number} maxZoom - Highest zoom present.
|
|
33
|
-
* @property {number[]} bounds - [minLon, minLat, maxLon, maxLat].
|
|
34
|
-
* @property {number[]} center - [lon, lat, zoom].
|
|
35
|
-
* @property {number} tileCount - Addressed tile count.
|
|
36
|
-
* @property {boolean} clustered - Whether tiles are stored in Hilbert order.
|
|
37
|
-
* @property {string} [name] - Name from the archive metadata.
|
|
38
|
-
* @property {string} [description] - Description from the archive metadata.
|
|
39
|
-
* @property {string} [attribution] - Attribution from the archive metadata.
|
|
40
|
-
* @property {object[]} [vectorLayers] - Vector layer definitions, for pbf archives.
|
|
41
|
-
* @property {boolean} [sparse] - What the archive says about missing tiles, if it says anything.
|
|
42
|
-
*/
|
|
43
|
-
|
|
44
|
-
/**
|
|
45
|
-
* Reads a flag out of archive metadata, which is not reliably typed.
|
|
46
|
-
*
|
|
47
|
-
* A PMTiles JSON blob carries a real boolean, but the same metadata routinely
|
|
48
|
-
* arrives having been round-tripped through MBTiles, where every value is TEXT
|
|
49
|
-
* — so the honest reading of `"false"` is false, not "a non-empty string".
|
|
50
|
-
* @param {unknown} value - Whatever the metadata held.
|
|
51
|
-
* @returns {boolean | undefined} - The flag, or undefined if it said nothing.
|
|
52
|
-
*/
|
|
53
|
-
export function metadataFlag(value) {
|
|
54
|
-
if (value === undefined || value === null || value === '') return undefined;
|
|
55
|
-
if (typeof value === 'boolean') return value;
|
|
56
|
-
if (typeof value === 'number') return value !== 0;
|
|
57
|
-
const text = String(value).trim().toLowerCase();
|
|
58
|
-
if (['true', '1', 'yes'].includes(text)) return true;
|
|
59
|
-
if (['false', '0', 'no'].includes(text)) return false;
|
|
60
|
-
return undefined;
|
|
61
|
-
}
|
|
62
|
-
|
|
63
|
-
/**
|
|
64
|
-
* The tile encoding an archive declares, if it is one MapLibre understands.
|
|
65
|
-
*
|
|
66
|
-
* See docs/tilejson.md — `encoding`.
|
|
67
|
-
* @param {unknown} value - Whatever the metadata held.
|
|
68
|
-
* @returns {string | undefined} - A known encoding, or undefined.
|
|
69
|
-
*/
|
|
70
|
-
export function metadataEncoding(value) {
|
|
71
|
-
if (typeof value !== 'string') return undefined;
|
|
72
|
-
const text = value.trim().toLowerCase();
|
|
73
|
-
// An unrecognised value is dropped: it would cost a client its own default.
|
|
74
|
-
return ['terrarium', 'mapbox', 'custom', 'mlt'].includes(text)
|
|
75
|
-
? text
|
|
76
|
-
: undefined;
|
|
77
|
-
}
|
|
78
|
-
|
|
79
|
-
/**
|
|
80
|
-
* The four factors a `custom` encoding is unreadable without. All or nothing.
|
|
81
|
-
* @param {object} metadata - The archive's metadata.
|
|
82
|
-
* @returns {object | undefined} - The four factors, or undefined.
|
|
83
|
-
*/
|
|
84
|
-
export function customEncodingFactors(metadata) {
|
|
85
|
-
const named = ['redFactor', 'greenFactor', 'blueFactor', 'baseShift'];
|
|
86
|
-
const out = {};
|
|
87
|
-
for (const name of named) {
|
|
88
|
-
const value = Number(metadata[name]);
|
|
89
|
-
if (!Number.isFinite(value)) return undefined;
|
|
90
|
-
out[name] = value;
|
|
91
|
-
}
|
|
92
|
-
return out;
|
|
93
|
-
}
|
|
94
|
-
|
|
95
13
|
/**
|
|
96
14
|
* Reads an archive's header and metadata.
|
|
97
15
|
*
|
|
@@ -99,72 +17,7 @@ export function customEncodingFactors(metadata) {
|
|
|
99
17
|
* even for a multi-terabyte archive — and it works against an HTTP URL without
|
|
100
18
|
* downloading it.
|
|
101
19
|
* @param {string} location - Local path or http(s) URL.
|
|
102
|
-
* @returns {Promise<
|
|
103
|
-
*/
|
|
104
|
-
/**
|
|
105
|
-
* What this prober reads. Raise it whenever a field is added to the summary,
|
|
106
|
-
* and archives probed by an older build are re-read once. See
|
|
107
|
-
* docs/internals.md — "Re-reading a summary an older prober wrote".
|
|
108
|
-
*/
|
|
109
|
-
export const SUMMARY_VERSION = 3;
|
|
110
|
-
|
|
111
|
-
/**
|
|
112
|
-
* What the catalog keeps about an archive, from its header and metadata.
|
|
113
|
-
* @param {object} header - A parsed PMTiles v3 header.
|
|
114
|
-
* @param {object} [metadata] - The archive's own metadata document.
|
|
115
|
-
* @returns {object} - The summary, at `SUMMARY_VERSION`.
|
|
116
|
-
*/
|
|
117
|
-
export function summarize(header, metadata = {}) {
|
|
118
|
-
const type = TILE_TYPES[header.tileType] ?? TILE_TYPES[0];
|
|
119
|
-
|
|
120
|
-
// An archive with no bounds set reports all zeroes; treat that as global
|
|
121
|
-
// rather than as a point at null island.
|
|
122
|
-
const hasBounds = !(
|
|
123
|
-
header.minLon === 0 &&
|
|
124
|
-
header.minLat === 0 &&
|
|
125
|
-
header.maxLon === 0 &&
|
|
126
|
-
header.maxLat === 0
|
|
127
|
-
);
|
|
128
|
-
|
|
129
|
-
return {
|
|
130
|
-
summaryVersion: SUMMARY_VERSION,
|
|
131
|
-
specVersion: header.specVersion,
|
|
132
|
-
format: type.format,
|
|
133
|
-
contentType: type.contentType,
|
|
134
|
-
minZoom: header.minZoom,
|
|
135
|
-
maxZoom: header.maxZoom,
|
|
136
|
-
bounds: hasBounds
|
|
137
|
-
? [header.minLon, header.minLat, header.maxLon, header.maxLat]
|
|
138
|
-
: [-180, -85.051129, 180, 85.051129],
|
|
139
|
-
center: [
|
|
140
|
-
header.centerLon,
|
|
141
|
-
header.centerLat,
|
|
142
|
-
header.centerZoom || Math.round(header.maxZoom / 2),
|
|
143
|
-
],
|
|
144
|
-
tileCount: header.numAddressedTiles,
|
|
145
|
-
clustered: header.clustered,
|
|
146
|
-
name: metadata.name,
|
|
147
|
-
description: metadata.description,
|
|
148
|
-
attribution: metadata.attribution,
|
|
149
|
-
vectorLayers: metadata.vector_layers,
|
|
150
|
-
// What the archive says about its own missing tiles. tileserver-gl reads
|
|
151
|
-
// the same key, so an archive built to be served there carries the answer
|
|
152
|
-
// with it and does not have to be configured again here.
|
|
153
|
-
sparse: metadataFlag(metadata.sparse),
|
|
154
|
-
// The header settles MLT; only the metadata can settle elevation packing.
|
|
155
|
-
encoding:
|
|
156
|
-
type.format === 'mlt' ? 'mlt' : metadataEncoding(metadata.encoding),
|
|
157
|
-
encodingFactors:
|
|
158
|
-
metadataEncoding(metadata.encoding) === 'custom'
|
|
159
|
-
? customEncodingFactors(metadata)
|
|
160
|
-
: undefined,
|
|
161
|
-
};
|
|
162
|
-
}
|
|
163
|
-
|
|
164
|
-
/**
|
|
165
|
-
* Reads an archive's header and metadata.
|
|
166
|
-
* @param {string} location - Local path or http(s) URL.
|
|
167
|
-
* @returns {Promise<PMTilesSummary>} - The summary.
|
|
20
|
+
* @returns {Promise<ArchiveSummary>} - The summary.
|
|
168
21
|
*/
|
|
169
22
|
export async function probePMTiles(location) {
|
|
170
23
|
const isHttp = /^https?:\/\//i.test(location);
|
package/src/prewarm.js
CHANGED
|
@@ -156,7 +156,14 @@ export class HeadWarmer {
|
|
|
156
156
|
// not, since an archive joined by magnet has no kind until its metadata
|
|
157
157
|
// arrives.
|
|
158
158
|
const kind = entry.kind ?? guessKind(entry.name ?? '');
|
|
159
|
-
if (kind !== 'pmtiles') return false;
|
|
159
|
+
if (kind !== 'pmtiles' && kind !== 'mbtiles') return false;
|
|
160
|
+
|
|
161
|
+
// MBTiles has no head to pull out of the swarm -- it is SQLite, read whole
|
|
162
|
+
// or not at all -- so there is nothing here to be due until the file is.
|
|
163
|
+
// Once it is, the read is local and instant, and it is the only thing that
|
|
164
|
+
// gives an archive already in the catalog the summary this did not use to
|
|
165
|
+
// record for it.
|
|
166
|
+
if (kind === 'mbtiles' && entry.complete !== true) return false;
|
|
160
167
|
|
|
161
168
|
// A summary is only an answer about *this disk* if a header on this disk
|
|
162
169
|
// produced it. `format` was standing in for that and does not mean it.
|
|
@@ -289,6 +296,11 @@ export class HeadWarmer {
|
|
|
289
296
|
`[warm] ${entry.name}: header read; its metadata is at the far end ` +
|
|
290
297
|
'of the archive and has not arrived yet',
|
|
291
298
|
);
|
|
299
|
+
} else if ((entry.kind ?? guessKind(entry.name ?? '')) === 'mbtiles') {
|
|
300
|
+
// Not a header: MBTiles keeps in a metadata table what PMTiles keeps
|
|
301
|
+
// in a fixed header, and a log saying otherwise sends whoever reads it
|
|
302
|
+
// looking for the wrong thing.
|
|
303
|
+
console.log(`[warm] ${entry.name}: metadata read`);
|
|
292
304
|
} else {
|
|
293
305
|
console.log(`[warm] ${entry.name}: header read`);
|
|
294
306
|
}
|
package/src/tiles.js
CHANGED
|
@@ -7,7 +7,7 @@ import { TorrentSource } from 'pmtiles-torrent';
|
|
|
7
7
|
import { NodeFileSource } from './file-source.js';
|
|
8
8
|
import { onDiskPath } from './incomplete.js';
|
|
9
9
|
import { MbtilesArchive, isMbtiles } from './mbtiles.js';
|
|
10
|
-
import { summarize } from './
|
|
10
|
+
import { summarize } from './archive-summary.js';
|
|
11
11
|
import { LibtorrentReadEngine } from './read-engine.js';
|
|
12
12
|
|
|
13
13
|
/**
|