pmtiles-swarm 0.63.0 → 0.65.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +190 -0
- package/NOTICE.md +23 -0
- package/README.md +7 -5
- package/docs/tile-stacks.md +298 -11
- package/package.json +1 -1
- package/src/api.js +119 -323
- package/src/bake-jobs.js +315 -0
- package/src/bake.js +546 -0
- package/src/config.js +11 -0
- package/src/index.js +9 -0
- package/src/pixel-worker.js +70 -0
- package/src/pixels.js +129 -0
- package/src/pmtiles-scan.js +254 -0
- package/src/pmtiles-write.js +517 -0
- package/src/stack-tile.js +474 -0
- package/src/web/index.html +137 -6
- package/src/web/public.html +28 -2
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,196 @@
|
|
|
4
4
|
### ✨ Features and improvements
|
|
5
5
|
- _...Add new stuff here..._
|
|
6
6
|
|
|
7
|
+
### 🐞 Bug fixes
|
|
8
|
+
- _...Add new stuff here..._
|
|
9
|
+
|
|
10
|
+
## 0.65.0
|
|
11
|
+
### ✨ Features and improvements
|
|
12
|
+
- **A bake stays out of the way of the node it runs on.** It runs in the main process — unlike
|
|
13
|
+
hashing, there is no sidecar to send it to — and the merge's pixel maths is entirely
|
|
14
|
+
synchronous, so every millisecond of it is a millisecond the node is not answering requests.
|
|
15
|
+
Three changes, all of them from measuring rather than assuming.
|
|
16
|
+
|
|
17
|
+
**The pixel maths moved to a worker.** `src/pixels.js` runs `elevation.js` and `rgba.js`
|
|
18
|
+
unchanged on another thread. Against a request arriving every 5 ms while a bake runs, the delay
|
|
19
|
+
that request sees at the 99th percentile falls from 10.9 ms to 6.5 ms with two sources, and from
|
|
20
|
+
18.6 ms to 10.6 ms with four. The bake is 2–17% slower for it. Rasters are handed over rather
|
|
21
|
+
than copied — a decoded tile is most of a megabyte per source — so a merge takes ownership of
|
|
22
|
+
what it is given, which is safe because nothing reads a contribution afterwards and is asserted
|
|
23
|
+
rather than assumed. Serving does not use it: one tile is a few milliseconds nobody notices.
|
|
24
|
+
|
|
25
|
+
**The checkpoint appends instead of rewriting.** It went through `serializeDirectory`, which was
|
|
26
|
+
elegant reuse and the wrong tool — that is a distribution format, and producing it costs a varint
|
|
27
|
+
pass over every entry. Re-encoding all of them every time also made the total work quadratic in
|
|
28
|
+
the length of the job. Fixed 24-byte records can be appended, and only the last one can change
|
|
29
|
+
once written, so a checkpoint costs the work since the last one: **446 ms at four million entries
|
|
30
|
+
becomes 11 ms, and stays 11 ms**.
|
|
31
|
+
|
|
32
|
+
**And `stacks.bakePauseMs`**, how long a bake waits between tiles. Zero by default, which is
|
|
33
|
+
right for a node baking and doing nothing else. On a node that is also serving maps it is the
|
|
34
|
+
direct trade between how long the bake takes and how much of the machine it takes while running.
|
|
35
|
+
|
|
36
|
+
|
|
37
|
+
|
|
38
|
+
### 🐞 Bug fixes
|
|
39
|
+
- _...Add new stuff here..._
|
|
40
|
+
|
|
41
|
+
## 0.64.0
|
|
42
|
+
### ✨ Features and improvements
|
|
43
|
+
- **Export a stack to an archive, from the console.** **Export to archive** sits beside Edit and
|
|
44
|
+
Delete on a stack. It runs the recipe over its sources, writes a real `.pmtiles`, and hands the
|
|
45
|
+
file to the library — which hashes it, makes a torrent and puts it in the catalog, so what comes
|
|
46
|
+
out has an infohash and is seeded like anything else here. That is what makes the live endpoint
|
|
47
|
+
the preview of something rather than an endpoint on its own.
|
|
48
|
+
|
|
49
|
+
**Two halves, watched in two places.** Merging is about a stack, so tiles written, tiles skipped
|
|
50
|
+
and the zoom it is working through are reported on the stack. What follows is an archive being
|
|
51
|
+
added, which this node already reports on the archives view through the library's in-progress
|
|
52
|
+
list — so the second half is handed over rather than drawn twice, and the line on the stack says
|
|
53
|
+
where to look.
|
|
54
|
+
|
|
55
|
+
Stopping keeps the work. The checkpoint is the hours already spent, so **Stop export** leaves it
|
|
56
|
+
where the next run picks it up rather than throwing it away. One bake per stack at a time: two
|
|
57
|
+
runs of one recipe write the same checkpoint files over each other, and the second would resume
|
|
58
|
+
the first's work believing it were its own.
|
|
59
|
+
|
|
60
|
+
A bake is identified by the recipe's revision **and what each source resolved to**. The recipe
|
|
61
|
+
alone cannot see the difference: a stack naming a category resolves to whichever build is
|
|
62
|
+
current, so the same recipe over a rebuilt source is a different bake, and a checkpoint that
|
|
63
|
+
could not tell would resume across the change and produce an archive half of one map and half of
|
|
64
|
+
another.
|
|
65
|
+
|
|
66
|
+
The file is dated — `Terrain-20260822.pmtiles` — and the archive's **name** is not. A rebuild
|
|
67
|
+
here keeps its name and mints a new infohash, which is what lets `/latest/<category>/` follow it;
|
|
68
|
+
dating the name would make every build a different map. The date goes in `description`. `name` is
|
|
69
|
+
always written, because these get converted to mbtiles by other tools and a nameless metadata
|
|
70
|
+
block is not valid there.
|
|
71
|
+
|
|
72
|
+
The codec is required for a recipe that asks for pixel work, and refused when the button is
|
|
73
|
+
pressed rather than an hour in. **A cache-mode source is allowed**, which reverses what the
|
|
74
|
+
design first said: reading one pulls pieces through the swarm, and the tile store already holds
|
|
75
|
+
those to a byte budget and drops what it stops using. Slow is the operator's call to make. The
|
|
76
|
+
sources are scanned through the store for the same reason, so a cache-mode archive's directories
|
|
77
|
+
come out of the swarm the way its tiles do.
|
|
78
|
+
|
|
79
|
+
- **The per-tile merge is one function, shared by the route and the bake.** `src/stack-tile.js`.
|
|
80
|
+
The tile route was 380 lines of Express with the whole merge inline — reading each source,
|
|
81
|
+
climbing to a parent, stitching children where tile sizes differ, both pixel spaces, the cache —
|
|
82
|
+
and none of it callable from anywhere else. A bake had to produce exactly what a request would,
|
|
83
|
+
and two implementations of that disagree eventually.
|
|
84
|
+
|
|
85
|
+
What is left in the route is what is genuinely about HTTP: parsing, status codes, the
|
|
86
|
+
abort-on-close wiring, the stats hook. Behaviour is unchanged and the existing stack suites say
|
|
87
|
+
so.
|
|
88
|
+
|
|
89
|
+
- **The bake driver, with checkpointing and cancellation.** `src/bake.js` runs a stack over its
|
|
90
|
+
sources and writes the result as a real archive: union the sources' coverage, merge in tile-id
|
|
91
|
+
order, write, checkpoint, stop when told. `src/pmtiles-scan.js` is the other new piece — it
|
|
92
|
+
reads back *which* tiles an archive holds, which the `pmtiles` package does not offer because
|
|
93
|
+
a tile server never needs to ask.
|
|
94
|
+
|
|
95
|
+
It iterates coverage rather than a zoom range, and that is what decides whether the job is
|
|
96
|
+
possible: a planet at z16 is 5.7 billion coordinates, and asking each one whether a source
|
|
97
|
+
covers it does not finish. The union of what the sources actually hold skips most of the
|
|
98
|
+
pyramid without a single decode.
|
|
99
|
+
|
|
100
|
+
**Sparseness is a consequence rather than a setting.** A tile no source covered is never
|
|
101
|
+
written, so the archive is sparse by construction — and the baked metadata says so, under the
|
|
102
|
+
key tileserver-gl reads and this project's prober reads. `encoding` travels the same way, with
|
|
103
|
+
its four factors when it is `custom`, because a terrain archive that does not say how to read
|
|
104
|
+
its pixels is an image of nothing in particular.
|
|
105
|
+
|
|
106
|
+
**A cancelled bake keeps its work.** Deleting it would make stopping and failing the same
|
|
107
|
+
thing, and this is a job somebody may have been running since yesterday. The checkpoint is the
|
|
108
|
+
buffered tile data, the entries as `serializeDirectory` writes them, and a small JSON state —
|
|
109
|
+
the entries reuse the archive's own serialization rather than inventing a second format for
|
|
110
|
+
the same array. A checkpoint belongs to one revision of one recipe; a checkpoint for anything
|
|
111
|
+
else is discarded rather than continued, because resuming a changed recipe produces an archive
|
|
112
|
+
that is half one map and half another and nothing downstream could tell.
|
|
113
|
+
|
|
114
|
+
The merge is handed in rather than reached for. That keeps the design's own constraint honest
|
|
115
|
+
— one per-tile answer serves both a request and a bake — and it is also what is left to do:
|
|
116
|
+
the orchestration still lives inline in the stack tile route, and extracting it is the next
|
|
117
|
+
step.
|
|
118
|
+
|
|
119
|
+
- **This node can write a PMTiles file.** `src/pmtiles-write.js`: varints, directories, the
|
|
120
|
+
fixed header and a writer that takes tiles one at a time and finalises an archive. The
|
|
121
|
+
`pmtiles` package everything here reads with is read-only, so this is the other half of it,
|
|
122
|
+
ported from the reference implementation in protomaps/PMTiles.
|
|
123
|
+
|
|
124
|
+
It exists for stage 8 of [tile stacks](docs/tile-stacks.md) — baking a stack into a real
|
|
125
|
+
archive with a real infohash — but it is a module on its own and tested as one. Every test
|
|
126
|
+
reads the result back through the same `pmtiles` library that answers served tiles, and one
|
|
127
|
+
reads it through this project's own prober, so an archive written here is one the node can
|
|
128
|
+
take into its catalog.
|
|
129
|
+
|
|
130
|
+
Three deliberate departures from the reference, all of them written down in
|
|
131
|
+
[docs/tile-stacks.md](docs/tile-stacks.md#writing-a-pmtiles-file):
|
|
132
|
+
|
|
133
|
+
Deduplication hashes with a full-length digest rather than the reference's 64 bits. A
|
|
134
|
+
collision does not raise anything — it points one tile at another tile's bytes, in a file
|
|
135
|
+
that is then hashed, torrented and served to other people. At a billion distinct tiles a
|
|
136
|
+
64-bit hash collides about 2.7% of the time, and at five billion about half the time.
|
|
137
|
+
|
|
138
|
+
Varints are built by division rather than by shifting. JavaScript's bitwise operators are
|
|
139
|
+
32-bit, so `value >>= 7` mangles any offset past 4 GiB — which in an archive worth baking
|
|
140
|
+
arrives early. There is a test at 6 GiB.
|
|
141
|
+
|
|
142
|
+
Deduplication can be turned off. It is the one part whose memory grows with the archive, one
|
|
143
|
+
entry per distinct tile, and go-pmtiles makes it a flag for the same reason. Run-length
|
|
144
|
+
encoding still applies without it, and for terrain — long runs of identical ocean and
|
|
145
|
+
identical nodata — that is most of the saving.
|
|
146
|
+
|
|
147
|
+
`clustered` is reported honestly rather than assumed. Tiles written out of tile-id order
|
|
148
|
+
still produce a valid archive, and the header says it is unclustered, because an unclustered
|
|
149
|
+
archive cannot answer a range read in one seek — which is the reason this project serves
|
|
150
|
+
PMTiles rather than MBTiles at all.
|
|
151
|
+
|
|
152
|
+
- **The bake and the sync are designed.** [docs/tile-stacks.md](docs/tile-stacks.md) gained
|
|
153
|
+
three sections: what a bake iterates and why it is the sources' own coverage rather than the
|
|
154
|
+
zoom range (a planet at z16 is 5.7 billion tiles, and asking each one whether a source covers
|
|
155
|
+
it is the difference between a job that finishes and one that does not), what running one
|
|
156
|
+
requires, and what syncing a stack between nodes would have to answer.
|
|
157
|
+
|
|
158
|
+
The sync section is the useful half. A stack is a mutable document whose sources are named by
|
|
159
|
+
category or infohash, so the same recipe means different things on different nodes — sent
|
|
160
|
+
somewhere missing a source it either breaks outright or silently resolves to a different map.
|
|
161
|
+
A feed is the wrong shape for that. Baking sidesteps it: a baked stack is an ordinary archive
|
|
162
|
+
with an infohash, and archives already sync.
|
|
163
|
+
|
|
164
|
+
The stage list also said the console reads stacks but does not edit them, which stopped being
|
|
165
|
+
true when the editor shipped.
|
|
166
|
+
|
|
167
|
+
- **A terrain archive says so before you open it.** The console detail, the stacks list and
|
|
168
|
+
the public catalogue now offer two preview buttons where the encoding is one the preview can
|
|
169
|
+
draw as a DEM: the raster keeps its name, and **Terrain** sits beside it. Before this the
|
|
170
|
+
terrain view existed but nothing pointed at it — you had to open the preview and notice a
|
|
171
|
+
link in the header — and nothing on a listing said which archives were terrain at all.
|
|
172
|
+
|
|
173
|
+
The raster stays first and keeps the plain name deliberately. It is the view that shows a
|
|
174
|
+
hole, because a missing DEM tile hillshades as flat ground rather than as missing.
|
|
175
|
+
|
|
176
|
+
`/api/stacks` and the categories payload now carry `encoding` for the same reason, so a page
|
|
177
|
+
can decide without fetching a TileJSON per row.
|
|
178
|
+
|
|
179
|
+
### 🐞 Bug fixes
|
|
180
|
+
- **A catalogue helper declared in the wrong scope.** The public page renders archives and
|
|
181
|
+
categories from two separate functions, and the terrain check went inside the first — a
|
|
182
|
+
`ReferenceError` the moment a terrain category was drawn. `node --check` accepts it and so
|
|
183
|
+
does a scope check, because the name exists; it is only wrong from the other function.
|
|
184
|
+
|
|
185
|
+
The scope checks that exist for exactly this were only ever run against the console. They now
|
|
186
|
+
run against the catalogue and the preview too, and the helper both renderers share is
|
|
187
|
+
asserted to be declared where both can reach it.
|
|
188
|
+
|
|
189
|
+
- **Three copies of one rule, none of them checked against each other.** The preview decides
|
|
190
|
+
what to draw as terrain, and the console and the catalogue decide whether to offer the
|
|
191
|
+
button. A button offered on something that does not render as terrain is worse than no
|
|
192
|
+
button, so the three are now lifted out of their pages and asserted to agree across eight
|
|
193
|
+
cases — `mlt`, an unknown encoding, and `custom` with none, half and all four of its factors
|
|
194
|
+
among them.
|
|
195
|
+
|
|
196
|
+
|
|
7
197
|
### 🐞 Bug fixes
|
|
8
198
|
- _...Add new stuff here..._
|
|
9
199
|
|
package/NOTICE.md
CHANGED
|
@@ -57,6 +57,29 @@ The prebuilt `@img/sharp-*` packages it resolves carry libvips (LGPL-3.0-or-late
|
|
|
57
57
|
dependencies, each under their own terms; sharp links to libvips dynamically and ships it
|
|
58
58
|
unmodified.
|
|
59
59
|
|
|
60
|
+
## rio-rgbify — MIT
|
|
61
|
+
|
|
62
|
+
> Copyright (c) 2016 Mapbox
|
|
63
|
+
> https://github.com/mapbox/rio-rgbify
|
|
64
|
+
|
|
65
|
+
The original terrain-RGB encoder: it is what established the `base` and `interval` packing that
|
|
66
|
+
`src/elevation.js` decodes and encodes, and the defaults there — base `-10000`, interval `0.1` —
|
|
67
|
+
are Mapbox's Terrain-RGB format rather than a choice made here.
|
|
68
|
+
|
|
69
|
+
[rio-rgbify-merge](https://github.com/TechIdiots-LLC/rio-rgbify-merge) is a fork of it by
|
|
70
|
+
TechIdiots LLC, under the same MIT license and carrying the same copyright line. Tile stacks are
|
|
71
|
+
the on-the-fly counterpart to that fork's offline merge, and the pixel maths is deliberately kept
|
|
72
|
+
the same so a stack previewed live and a stack baked to a file do not disagree — see
|
|
73
|
+
[docs/tile-stacks.md](docs/tile-stacks.md).
|
|
74
|
+
|
|
75
|
+
Which half is whose is worth stating, because the two are credited differently. The encoder traces
|
|
76
|
+
to Mapbox in 2016. The **merge** behaviour this project follows — layer priority, masking by height
|
|
77
|
+
and by colour, sparse tiles — was added in the fork in 2026 and is TechIdiots LLC's own work, not
|
|
78
|
+
Mapbox's.
|
|
79
|
+
|
|
80
|
+
No code is copied either way. `src/elevation.js` and `src/rgba.js` are JavaScript written against
|
|
81
|
+
the same rules, and `docs/tile-stacks.md` records where those rules differ from the fork's and why.
|
|
82
|
+
|
|
60
83
|
## qBittorrent — GPL-2.0-or-later
|
|
61
84
|
|
|
62
85
|
> https://github.com/qbittorrent/qBittorrent
|
package/README.md
CHANGED
|
@@ -804,11 +804,13 @@ which the endpoint answers 501.
|
|
|
804
804
|
| `GET` | `/api/stacks` | Every stack, with what each source resolved to and why one cannot be served yet |
|
|
805
805
|
| `GET` | `/api/torrents/:infoHash/stacks` | Which stacks would break if this archive were removed, and how |
|
|
806
806
|
| `GET`, `PUT`, `DELETE` | `/api/stacks/:id/raw`, `/api/stacks/:id` | Read a recipe as written, save one, or remove it. `/raw` is the recipe; `/api/stacks` is what it resolved to |
|
|
807
|
-
|
|
808
|
-
| `
|
|
809
|
-
| `GET`
|
|
810
|
-
| `GET`
|
|
811
|
-
| `GET`
|
|
807
|
+
|
|
808
|
+
| `POST`, `DELETE` | `/api/stacks/:id/bake` | Run the stack over its sources and write the result as an archive, or stop one that is running. Answers as soon as the job starts; watch it on `/api/stacks` |
|
|
809
|
+
| `GET` | `/stacks/:id/preview` | A map of a stack, for looking at it — **public** |
|
|
810
|
+
| `GET` | `/stacks/:id/tiles.json` | TileJSON for a stack — **public**. `maxzoom` is the maximum over its sources, not the minimum |
|
|
811
|
+
| `GET` | `/stacks/:id/:size/:z/:x/:y.:ext` | The same tile at 256 or 512 px. A tile's coordinates are an extent rather than a pixel count, so this is the same ground on a finer or coarser grid |
|
|
812
|
+
| `GET` | `/stacks/:id/:z/:x/:y.:ext` | One tile of a stack — **public**. Answered by the topmost source holding it; `X-Stack-Sources` says which were asked and what each said |
|
|
813
|
+
| `GET` | `/feed.xml`, `/feed/:category.xml`, `/latest/:category.xml` | RSS — **public** |
|
|
812
814
|
|
|
813
815
|
Everything under `/api/` is guarded once a credential is configured; tiles, TileJSON and the feeds
|
|
814
816
|
never are. A `peer` token may read but not change, and may be narrowed to some categories. See
|
package/docs/tile-stacks.md
CHANGED
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
# Tile stacks
|
|
2
2
|
|
|
3
|
-
**Status:
|
|
3
|
+
**Status: all eight stages are implemented.** Elevation stacks work: a recipe is
|
|
4
4
|
defined, resolved and served, and a source is masked, shifted, resampled from a
|
|
5
5
|
parent and painted in order. Merged tiles are cached on disk against a
|
|
6
6
|
byte budget, and image stacks composite with opacity and blend modes. The
|
|
7
|
-
console lists stacks
|
|
8
|
-
still serves the passthrough case and answers 501 for the rest.
|
|
7
|
+
console lists stacks, diagnoses them and edits them. A node with no codec
|
|
8
|
+
installed still serves the passthrough case and answers 501 for the rest.
|
|
9
9
|
|
|
10
10
|
A stack is a recipe for combining several archives into one tile endpoint,
|
|
11
11
|
evaluated per request rather than baked into a file. Where
|
|
@@ -15,7 +15,9 @@ regional lidar patch over a global DEM.
|
|
|
15
15
|
|
|
16
16
|
This is the on-the-fly counterpart to the offline merge in
|
|
17
17
|
[rio-rgbify-merge](https://github.com/TechIdiots-LLC/rio-rgbify-merge)
|
|
18
|
-
(`pip install rio-rgbify-merge`)
|
|
18
|
+
(`pip install rio-rgbify-merge`), a fork of
|
|
19
|
+
[mapbox/rio-rgbify](https://github.com/mapbox/rio-rgbify) which added the merge
|
|
20
|
+
the encoder never had. The pixel maths is the same and is
|
|
19
21
|
deliberately kept the same, so a stack can be previewed live and then baked into
|
|
20
22
|
a real archive without the two disagreeing. What the swarm adds is that a source is named by
|
|
21
23
|
_category_ rather than by path, so a stack keeps working across a rebuild of any
|
|
@@ -39,6 +41,15 @@ of its parts.
|
|
|
39
41
|
- [TileJSON for a stack](#tilejson-for-a-stack)
|
|
40
42
|
- [When a source will not answer](#when-a-source-will-not-answer)
|
|
41
43
|
- [Baking a stack into an archive](#baking-a-stack-into-an-archive)
|
|
44
|
+
- [Writing a PMTiles file](#writing-a-pmtiles-file)
|
|
45
|
+
- [What to iterate](#what-to-iterate)
|
|
46
|
+
- [Running it](#running-it)
|
|
47
|
+
- [What a baked archive says about itself](#what-a-baked-archive-says-about-itself)
|
|
48
|
+
- [Staying out of the way of the node it runs on](#staying-out-of-the-way-of-the-node-it-runs-on)
|
|
49
|
+
- [What identifies a bake](#what-identifies-a-bake)
|
|
50
|
+
- [Starting one, and watching it](#starting-one-and-watching-it)
|
|
51
|
+
- [What exists now](#what-exists-now)
|
|
52
|
+
- [Syncing a stack to another node](#syncing-a-stack-to-another-node)
|
|
42
53
|
- [What the offline merge got wrong](#what-the-offline-merge-got-wrong)
|
|
43
54
|
- [The stack editor](#the-stack-editor)
|
|
44
55
|
- [Staging](#staging)
|
|
@@ -523,9 +534,280 @@ and the merge code is shared. It also inverts the cost argument completely: the
|
|
|
523
534
|
swarm-read latency that makes on-the-fly stacking painful is irrelevant to a
|
|
524
535
|
batch job that runs overnight.
|
|
525
536
|
|
|
526
|
-
|
|
527
|
-
|
|
528
|
-
|
|
537
|
+
That constraint was honoured: `mergeElevation` and the RGBA equivalent are pure
|
|
538
|
+
functions of their contributions, so the bake is a different driver over the
|
|
539
|
+
same core rather than a second implementation of it.
|
|
540
|
+
|
|
541
|
+
Most of what a bake needs is already here. `stackCoverage` says what zooms and
|
|
542
|
+
bounds a stack reaches. The merge path answers a tile. `#running` in the library
|
|
543
|
+
already registers a long job with an `AbortController`, a name, a start time and
|
|
544
|
+
a received/total pair, and the console already draws that list, so a bake is
|
|
545
|
+
another entry in it rather than new machinery. `settleFromStaging` exists for
|
|
546
|
+
precisely this shape: work into a staging directory, hash, then move into place
|
|
547
|
+
once the infohash exists, because an infohash cannot be known before the bytes
|
|
548
|
+
do. The last step is `createTorrentFromFile`, which is what every other archive
|
|
549
|
+
built here goes through.
|
|
550
|
+
|
|
551
|
+
One piece is genuinely missing: nothing in this project writes a PMTiles file.
|
|
552
|
+
|
|
553
|
+
### Writing a PMTiles file
|
|
554
|
+
|
|
555
|
+
The `pmtiles` npm package is read-only. It exports `bytesToHeader`, `findTile`,
|
|
556
|
+
`readVarint` and `zxyToTileId`, and nothing that serialises. So
|
|
557
|
+
`src/pmtiles-write.js` is new, but it is a port of a small, well-understood
|
|
558
|
+
reference rather than a design problem. The Python implementation in
|
|
559
|
+
protomaps/PMTiles is 124 lines for the writer, plus about 70 for
|
|
560
|
+
`serialize_directory` and `serialize_header`; the Hilbert maths does not need
|
|
561
|
+
porting at all, because `zxyToTileId` already ships.
|
|
562
|
+
|
|
563
|
+
The reader shipping in the same package is what makes this cheap to trust: a
|
|
564
|
+
round trip is write it, read it back with `bytesToHeader` and `findTile`, and
|
|
565
|
+
probe it with `pmtiles-probe.js` - the same prober every other archive here goes
|
|
566
|
+
through.
|
|
567
|
+
|
|
568
|
+
Three things the Python reference does that a port must not copy.
|
|
569
|
+
|
|
570
|
+
**It deduplicates on a 64-bit hash.** go-pmtiles uses FNV-128a for the same job,
|
|
571
|
+
and at the scale a bake works at that is not a stylistic difference. A collision
|
|
572
|
+
does not raise anything: it points one tile at another tile's bytes, in a file
|
|
573
|
+
that is then hashed, torrented and served to other people.
|
|
574
|
+
|
|
575
|
+
| distinct tiles | 64-bit | 128-bit |
|
|
576
|
+
| -------------- | -------- | ------- |
|
|
577
|
+
| 10^7 | 3e-06 | ~0 |
|
|
578
|
+
| 10^8 | 3e-04 | ~0 |
|
|
579
|
+
| 10^9 | **2.7%** | ~0 |
|
|
580
|
+
| 5 x 10^9 | **49%** | ~0 |
|
|
581
|
+
|
|
582
|
+
Use a 128-bit digest, and compare the bytes on a hit rather than trusting the
|
|
583
|
+
digest alone. The cost is a comparison against a tile already in hand; the
|
|
584
|
+
failure it prevents is silent and permanent.
|
|
585
|
+
|
|
586
|
+
**It buffers tile data in a temporary file and copies it in at the end**,
|
|
587
|
+
because `tile_data_offset` is not known until the directories have been sized.
|
|
588
|
+
Peak disk is then twice the tile bytes, which for a planet bake is a terabyte of
|
|
589
|
+
transient space. The format does not require it: `tile_data_offset` may be
|
|
590
|
+
anything, so tile data can be written straight into the output at a generous
|
|
591
|
+
fixed offset, leaving a small hole rather than making a second copy.
|
|
592
|
+
|
|
593
|
+
**It detects `clustered` rather than requiring it.** Writing tiles in ascending
|
|
594
|
+
tile-id order is what makes an archive answer a range read in one seek, and
|
|
595
|
+
range reads over HTTP are the reason this project serves PMTiles rather than
|
|
596
|
+
MBTiles at all. An unclustered bake is valid and bad at the only thing it exists
|
|
597
|
+
for, so ordering is a requirement of the bake here, not an outcome to be
|
|
598
|
+
reported.
|
|
599
|
+
|
|
600
|
+
The run-length encoding in `write_tile` is worth keeping exactly as it is:
|
|
601
|
+
consecutive tile ids sharing an offset collapse into one entry, and for terrain,
|
|
602
|
+
with its long runs of identical ocean and identical nodata, that is most of the
|
|
603
|
+
saving.
|
|
604
|
+
|
|
605
|
+
### What to iterate
|
|
606
|
+
|
|
607
|
+
Not the zoom range. A planet at z0-z14 is around 350 million tiles and at z16
|
|
608
|
+
around 5.7 billion, and enumerating that to ask each one whether any source
|
|
609
|
+
covers it is the difference between a job that finishes and one that does not.
|
|
610
|
+
|
|
611
|
+
Iterate the sources' own coverage instead. Every source is a PMTiles archive
|
|
612
|
+
whose directories already say which tiles it has; the union of those, in tile-id
|
|
613
|
+
order, is exactly the set a stack can answer for. For terrain over ocean that
|
|
614
|
+
skips most of the pyramid without a single decode.
|
|
615
|
+
|
|
616
|
+
`sparse` follows from the same fact and needs no separate decision: a tile no
|
|
617
|
+
source covered is not written, and the baked archive is sparse for the same
|
|
618
|
+
reason the live stack answers 404.
|
|
619
|
+
|
|
620
|
+
### Running it
|
|
621
|
+
|
|
622
|
+
- **The codec is required**, and refused up front. A bake that is not pure
|
|
623
|
+
passthrough decodes and re-encodes every tile, so `sharp` stops being optional
|
|
624
|
+
for it — and a node without one should find out when it presses the button,
|
|
625
|
+
not an hour in.
|
|
626
|
+
- **A cache-mode source is allowed.** An earlier draft here said to refuse one.
|
|
627
|
+
That was wrong: reading a cache-mode archive pulls pieces through the swarm,
|
|
628
|
+
and the tile store already holds those to a byte budget and drops what it
|
|
629
|
+
stops using. A long bake against a cached source is therefore slow, not
|
|
630
|
+
unbounded — and slow is the operator's call to make, not this document's. The
|
|
631
|
+
sources are scanned through the store for the same reason, so a cache-mode
|
|
632
|
+
archive's directories come out of the swarm the way its tiles do.
|
|
633
|
+
- **Checkpointing.** Hours to days means the process will be interrupted, and
|
|
634
|
+
resume state is the part of this project that has already gone subtly wrong
|
|
635
|
+
once - see `tools/resume-doctor.py` and what it exists to diagnose. Design it
|
|
636
|
+
rather than discovering it.
|
|
637
|
+
- **Cancellable**, and stopping keeps the work: realising it is the wrong recipe
|
|
638
|
+
should not mean waiting it out, and it should not mean starting over either.
|
|
639
|
+
|
|
640
|
+
### Staying out of the way of the node it runs on
|
|
641
|
+
|
|
642
|
+
A bake runs in the main process, and unlike hashing there is no sidecar to send
|
|
643
|
+
it to. That matters because `elevation.js` and `rgba.js` are entirely
|
|
644
|
+
synchronous — decoding heights, masking, resampling and painting are loops over
|
|
645
|
+
typed arrays — so every millisecond of it is a millisecond the node is not
|
|
646
|
+
answering requests. Three things follow from having measured that rather than
|
|
647
|
+
assumed it.
|
|
648
|
+
|
|
649
|
+
**The pixel maths goes to a worker.** `src/pixels.js` and `src/pixel-worker.js`
|
|
650
|
+
run the same functions, unchanged, on another thread. Measured against a request
|
|
651
|
+
arriving every 5 ms while a bake runs, the delay that request sees at the 99th
|
|
652
|
+
percentile:
|
|
653
|
+
|
|
654
|
+
| workload | on the main thread | in a worker |
|
|
655
|
+
| ------------------------- | ------------------ | ----------- |
|
|
656
|
+
| 2 sources, 512px | 10.9 ms | 6.5 ms |
|
|
657
|
+
| 4 sources, 512px | 18.6 ms | 10.6 ms |
|
|
658
|
+
| 8 sources, 512px, blurred | 22.3 ms | 17.1 ms |
|
|
659
|
+
|
|
660
|
+
The bake itself is 2–17% slower for it, which is the trade. Rasters are handed
|
|
661
|
+
to the worker rather than copied — a decoded tile is most of a megabyte per
|
|
662
|
+
source, and copying each one is work on the very thread this exists to keep
|
|
663
|
+
free. The cost of that is the caller gives them up, which is safe because
|
|
664
|
+
nothing reads a contribution after its merge, and is asserted rather than left
|
|
665
|
+
as a comment.
|
|
666
|
+
|
|
667
|
+
Serving does not use it. One tile's merge is a few milliseconds nobody notices,
|
|
668
|
+
and a thread per request would cost more than it saved.
|
|
669
|
+
|
|
670
|
+
**The checkpoint appends instead of rewriting.** The first version wrote entries
|
|
671
|
+
through `serializeDirectory`, which was elegant reuse and the wrong tool: that
|
|
672
|
+
is a _distribution_ format, and producing it costs a varint pass over every
|
|
673
|
+
entry — 264 ms at a million entries, of which only 12 ms is the compression.
|
|
674
|
+
Re-encoding all of them every checkpoint also made the total work quadratic in
|
|
675
|
+
the length of the job.
|
|
676
|
+
|
|
677
|
+
A checkpoint is read once, by this process, on the machine that wrote it. Fixed
|
|
678
|
+
24-byte records cost nothing to produce and can be appended, and only the last
|
|
679
|
+
one can change after it is written — a run of identical tiles extends it. So a
|
|
680
|
+
checkpoint now costs the work since the last one:
|
|
681
|
+
|
|
682
|
+
| entries | rewriting everything | appending |
|
|
683
|
+
| --------- | -------------------- | --------- |
|
|
684
|
+
| 100,000 | 36 ms | 11 ms |
|
|
685
|
+
| 1,000,000 | 191 ms | 11 ms |
|
|
686
|
+
| 4,000,000 | 446 ms | 11 ms |
|
|
687
|
+
|
|
688
|
+
**And there is a knob.** `stacks.bakePauseMs` is how long a bake waits between
|
|
689
|
+
tiles. Zero is as fast as it can go, which is right for a node baking and doing
|
|
690
|
+
nothing else. On a node that is also serving maps it is the direct trade: how
|
|
691
|
+
long the bake takes against how much of the machine it takes while it runs.
|
|
692
|
+
|
|
693
|
+
### What identifies a bake
|
|
694
|
+
|
|
695
|
+
`bakeRevision` is the recipe's revision and what each source resolved to,
|
|
696
|
+
hashed together. The recipe alone is not enough. A stack naming a category
|
|
697
|
+
resolves to whichever build is current, so the same recipe over a rebuilt source
|
|
698
|
+
is a different bake — and a checkpoint that could not tell would resume across
|
|
699
|
+
the change and produce an archive that is half one map and half another.
|
|
700
|
+
|
|
701
|
+
The file is dated: `Terrain-20260822.pmtiles`. The archive's **name** is not.
|
|
702
|
+
A rebuild here keeps its name and mints a new infohash, which is what lets
|
|
703
|
+
`/latest/<category>/` follow it; dating the name would make every build a
|
|
704
|
+
different map. The date goes in `description` instead, so an archive read out of
|
|
705
|
+
context still says what produced it and when.
|
|
706
|
+
|
|
707
|
+
`name` is always written, because these archives get converted to mbtiles by
|
|
708
|
+
other tools and a nameless metadata block is not valid there.
|
|
709
|
+
|
|
710
|
+
### What a baked archive says about itself
|
|
711
|
+
|
|
712
|
+
Two metadata keys beyond the name, and both are the keys tileserver-gl reads and
|
|
713
|
+
this project's own prober reads, so a baked archive is understood wherever it
|
|
714
|
+
lands.
|
|
715
|
+
|
|
716
|
+
`sparse` is true unless the recipe says otherwise. For a bake this is not a
|
|
717
|
+
claim, it is a description: a tile no source covered is never written, so the
|
|
718
|
+
archive is sparse by construction. The flag is what makes a client overzoom the
|
|
719
|
+
parent rather than draw nothing, and without it a terrain map is full of holes
|
|
720
|
+
that render as sea.
|
|
721
|
+
|
|
722
|
+
`encoding` says how to read the pixels. A terrain-RGB archive without it is an
|
|
723
|
+
image of nothing in particular, and `custom` carries its four factors or is not
|
|
724
|
+
worth writing at all.
|
|
725
|
+
|
|
726
|
+
### Starting one, and watching it
|
|
727
|
+
|
|
728
|
+
**Export to archive**, on the stack, beside Edit and Delete. `POST
|
|
729
|
+
/api/stacks/<id>/bake` starts it and answers as soon as the job is running: a
|
|
730
|
+
planet bake is hours, and a request that waited for the archive is a request
|
|
731
|
+
nothing could hold open. `DELETE` on the same address stops it.
|
|
732
|
+
|
|
733
|
+
A bake has two halves and they are watched in two places, deliberately. Merging
|
|
734
|
+
is about a stack, so it is reported on the stack — tiles written, tiles skipped,
|
|
735
|
+
the zoom it is working through. What happens afterwards is an archive being
|
|
736
|
+
added, and this node already reports that on the archives view through the
|
|
737
|
+
library's own in-progress list, so the second half is handed over rather than
|
|
738
|
+
drawn twice. The line on the stack says where to look.
|
|
739
|
+
|
|
740
|
+
One bake per stack at a time. Two runs of one recipe write the same checkpoint
|
|
741
|
+
files over each other, and the second would resume the first's work believing it
|
|
742
|
+
were its own.
|
|
743
|
+
|
|
744
|
+
### What exists now
|
|
745
|
+
|
|
746
|
+
`src/pmtiles-write.js` writes archives, `src/pmtiles-scan.js` reads back what one
|
|
747
|
+
holds, and `src/bake.js` is the driver: union the sources' coverage, merge in
|
|
748
|
+
tile-id order, write, checkpoint, stop when told.
|
|
749
|
+
|
|
750
|
+
The merge itself is handed in as a function, and `mergeTileFor` is what supplies
|
|
751
|
+
it: the same `answerStackTile` the tile route calls, so a baked tile and a served
|
|
752
|
+
one come out of one implementation and cannot drift. The cache is bypassed there
|
|
753
|
+
— it is sized for tiles people ask for twice, and a whole-pyramid run would evict
|
|
754
|
+
all of those in favour of tiles nobody will ask for again.
|
|
755
|
+
|
|
756
|
+
The checkpoint is three files in a working directory: the buffered tile data,
|
|
757
|
+
the entries as `serializeDirectory` writes them, and a small JSON state. Entries
|
|
758
|
+
go through the same serialization the archive itself uses rather than inventing
|
|
759
|
+
a second format for the same array. A checkpoint belongs to one revision of one
|
|
760
|
+
recipe; anything else is discarded rather than continued, because resuming a
|
|
761
|
+
changed recipe produces an archive that is half one map and half another and
|
|
762
|
+
nothing downstream could tell.
|
|
763
|
+
|
|
764
|
+
The deduplication map is deliberately not part of a checkpoint. Rebuilding it
|
|
765
|
+
means re-hashing everything already buffered, and the cost of starting it empty
|
|
766
|
+
is that a tile identical to one from before the interruption is stored twice.
|
|
767
|
+
The archive is correct either way; it is a little larger.
|
|
768
|
+
|
|
769
|
+
## Syncing a stack to another node
|
|
770
|
+
|
|
771
|
+
The question is whether a stack can travel between nodes the way an archive
|
|
772
|
+
does, and the answer is that it is a different kind of thing.
|
|
773
|
+
|
|
774
|
+
An archive is immutable and content-addressed: an infohash either matches or it
|
|
775
|
+
does not, and two nodes converge because they are fetching the same bytes. A
|
|
776
|
+
stack is a mutable document that gets edited, so syncing it is a question about
|
|
777
|
+
conflicts and clobbering rather than about missing pieces. A feed is the wrong
|
|
778
|
+
shape for it.
|
|
779
|
+
|
|
780
|
+
There is a sharper problem underneath. A recipe names its sources by category or
|
|
781
|
+
by infohash, so **the same recipe means different things on different nodes**.
|
|
782
|
+
Sent to a node missing one source, an infohash-pinned stack is permanently
|
|
783
|
+
broken; a category-named one silently resolves to that node's newest build of
|
|
784
|
+
that category, which may be a different map entirely. Either way the recipe
|
|
785
|
+
travels and the meaning does not.
|
|
786
|
+
|
|
787
|
+
Three shapes, in the order they are worth considering:
|
|
788
|
+
|
|
789
|
+
**Pull into a namespace the receiver owns.** `/api/stacks/<id>/raw` already
|
|
790
|
+
returns the recipe as written. A subscribing node polls a peer's stack list the
|
|
791
|
+
way it already polls for archives, and adopts what it finds under `<peer>:<id>`,
|
|
792
|
+
so an adopted stack can never overwrite a local one and it is obvious on the
|
|
793
|
+
screen which node a recipe came from. This reuses the remote-node machinery
|
|
794
|
+
rather than adding a feed to it. A source that does not resolve locally is
|
|
795
|
+
reported through `problems`, which the console already separates from an invalid
|
|
796
|
+
recipe and from one needing a codec; an adopted stack with a missing source is a
|
|
797
|
+
normal thing to look at, not an error to refuse.
|
|
798
|
+
|
|
799
|
+
**Export a bundle.** The recipe plus the resolved infohashes of its sources, as
|
|
800
|
+
one file. Unambiguous and reproducible, and the right answer when what is wanted
|
|
801
|
+
is _this exact map_. It pins, so it does not follow a rebuild, which is either
|
|
802
|
+
the point or the problem depending on why it was sent.
|
|
803
|
+
|
|
804
|
+
**Nothing automatic.** Copy the recipe and let the operator resolve the sources.
|
|
805
|
+
Honest, and possibly right for as long as stacks are few.
|
|
806
|
+
|
|
807
|
+
Baking sidesteps the question rather than answering it. A baked stack is an
|
|
808
|
+
ordinary archive with an infohash, and archives already sync, so where what is
|
|
809
|
+
wanted is the _output_ on another node rather than the _recipe_, that is the
|
|
810
|
+
mechanism, and it carries no ambiguity about what the sources resolved to.
|
|
529
811
|
|
|
530
812
|
## What the offline merge got wrong
|
|
531
813
|
|
|
@@ -743,10 +1025,15 @@ still requested and still composited.
|
|
|
743
1025
|
with a byte budget, LRU eviction and single-flight.
|
|
744
1026
|
6. ~~**RGBA space.**~~ Done. `src/rgba.js`: opacity, the separable blend
|
|
745
1027
|
modes, colour masking and alpha-correct resampling.
|
|
746
|
-
7.
|
|
747
|
-
resolved to and why one cannot be served
|
|
748
|
-
|
|
749
|
-
8.
|
|
1028
|
+
7. ~~**Console.**~~ Done. A Stacks view listing every stack with what each
|
|
1029
|
+
source resolved to and why one cannot be served, and an editor that adds,
|
|
1030
|
+
changes and removes them.
|
|
1031
|
+
8. ~~**Bake.**~~ Done. A coverage-driven run to a new `.pmtiles`, hashed and
|
|
1032
|
+
registered like any other archive. `src/pmtiles-write.js` writes archives,
|
|
1033
|
+
`src/pmtiles-scan.js` says what one holds, `src/stack-tile.js` is the
|
|
1034
|
+
per-tile merge shared with the tile route, `src/bake.js` drives the run with
|
|
1035
|
+
checkpointing and cancellation, and `src/bake-jobs.js` is the job the console
|
|
1036
|
+
starts and watches.
|
|
750
1037
|
|
|
751
1038
|
Stages 1 and 2 are worth doing on their own even if the rest waits: a
|
|
752
1039
|
category-resolved passthrough endpoint that picks whichever source has the tile
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pmtiles-swarm",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.65.0",
|
|
4
4
|
"description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "src/index.js",
|