pmtiles-swarm 0.90.0 → 0.92.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +77 -0
- package/docs/configuration.md +91 -0
- package/docs/tile-stacks.md +188 -12
- package/package.json +1 -1
- package/src/api.js +76 -72
- package/src/bake-jobs.js +22 -9
- package/src/config.js +23 -0
- package/src/index.js +4 -0
- package/src/pmtiles-write.js +8 -3
- package/src/s3-source.js +302 -0
- package/src/stacks.js +26 -4
- package/src/tiles.js +55 -1
- package/src/web/index.html +246 -34
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,83 @@
|
|
|
7
7
|
### 🐞 Bug fixes
|
|
8
8
|
- _...Add new stuff here..._
|
|
9
9
|
|
|
10
|
+
## 0.92.0
|
|
11
|
+
### 🐞 Bug fixes
|
|
12
|
+
- **A stack's TileJSON published the addresses its sources are read from.** A URL source is named by
|
|
13
|
+
its URL and an S3 source by its bucket and key, and the document naming them is served to anybody
|
|
14
|
+
who can load the map — while the archives at the other end are read with credentials nobody else
|
|
15
|
+
has. The sources are gone from it entirely, which was the right answer anyway: they are the
|
|
16
|
+
ingredients of one endpoint, not a list for a client to join, and listing them invited fetching
|
|
17
|
+
those instead of the tiles. What is left is a count and a `revision` fingerprint covering the
|
|
18
|
+
recipe and what every source resolved to — the one thing the list was good for, in twenty bytes
|
|
19
|
+
rather than tens of kilobytes. The document for a 459-source stack is now 336 bytes.
|
|
20
|
+
|
|
21
|
+
- **Signing in threw away the view you asked for.** A link to `#stacks` on a guarded node asked for a
|
|
22
|
+
password and then showed the archives, with the address still reading `#stacks` — which is what
|
|
23
|
+
made it look broken rather than like a redirect. Both ways into the console now land where the
|
|
24
|
+
address says.
|
|
25
|
+
|
|
26
|
+
- **An archive the engine has no record of was drawn as 0%.** That is a different fact from "none of
|
|
27
|
+
it is here", and it is the more alarming one to get wrong: a library the engine failed to take back
|
|
28
|
+
after a restart read as a library that had lost its data. The row now says **not loaded**, and says
|
|
29
|
+
where the reason is logged and that nothing on disk has been touched.
|
|
30
|
+
|
|
31
|
+
## 0.91.0
|
|
32
|
+
### ✨ Features and improvements
|
|
33
|
+
- **A source at a URL can be added by hand, not only imported.** **Add source → an address you
|
|
34
|
+
type…** in the stack editor, writing the same source the importer does. The card asks for the
|
|
35
|
+
address and for the zoom range the archive holds — a catalog archive states its range in its own
|
|
36
|
+
header and this has no header anybody has read, so leaving it empty means every tile asks it.
|
|
37
|
+
Anything published as a plain HTTPS download works, an S3 object or presigned URL included, and so
|
|
38
|
+
does `s3://bucket/key.pmtiles` for a private bucket — see below.
|
|
39
|
+
|
|
40
|
+
- **A stack's source list folds up past five sources.** An imported list is several hundred rows,
|
|
41
|
+
which buried every other stack on the page under one of them. The fold says how many there are and
|
|
42
|
+
names the base; a stack of a base and a layer or two stays open, since folding that hides nothing
|
|
43
|
+
worth a click.
|
|
44
|
+
|
|
45
|
+
|
|
46
|
+
- **A stack source may be an object in a private S3 bucket.**
|
|
47
|
+
`s3://bucket/terrain.pmtiles`, read with a signed request per byte range. Only for a bucket that is
|
|
48
|
+
not public — a public object or a presigned URL is an ordinary HTTPS address and always worked.
|
|
49
|
+
`endpoint` is what makes it S3-compatible rather than S3: MinIO, Ceph, Garage, R2, Wasabi and B2
|
|
50
|
+
all answer the same protocol at their own address, and path-style addressing is the default because
|
|
51
|
+
it is what they speak. Credentials go under **Settings → Feeds → S3 buckets**, per bucket or once
|
|
52
|
+
for an account; with none configured the standard `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`,
|
|
53
|
+
`AWS_REGION` and `AWS_S3_ENDPOINT` variables are read, so a machine already set up to reach a
|
|
54
|
+
bucket needs nothing typed in.
|
|
55
|
+
|
|
56
|
+
Signed here rather than by an SDK: SigV4 for a GET is a hash, four HMACs and a string, all of which
|
|
57
|
+
`node:crypto` has. An AWS client would be a large dependency for one signature, and an optional one
|
|
58
|
+
would make reading a bucket work on some installs and not on others for no reason the operator
|
|
59
|
+
could see — so it is neither optional nor a dependency. The signer is checked against AWS's own
|
|
60
|
+
published test vectors rather than against itself.
|
|
61
|
+
|
|
62
|
+
|
|
63
|
+
### 🐞 Bug fixes
|
|
64
|
+
- **A stack of remote sources could be served but not exported.** The bake refused any stack whose
|
|
65
|
+
sources resolved to no archive on this node, which is every stack read from URLs — refusing exactly
|
|
66
|
+
the export worth having, since baking is how terrain that lives somewhere else becomes something
|
|
67
|
+
this node holds and can seed. It also had no way to walk a remote archive's directories, which is
|
|
68
|
+
how a bake finds which tiles exist at all.
|
|
69
|
+
|
|
70
|
+
- **An archive whose deepest tiles repeat said it stopped a zoom short.** Identical tiles collapse
|
|
71
|
+
into one directory entry covering a range of ids, so the last entry's own id can sit at a shallower
|
|
72
|
+
zoom than the tiles it addresses — and the header was read off that id. A client asks for nothing
|
|
73
|
+
past a header's `maxZoom`, so the deepest zoom of such an archive was unreachable. Found baking a
|
|
74
|
+
stack over flat terrain, where every tile of the last zoom was the same tile.
|
|
75
|
+
|
|
76
|
+
- **A stack used as a source of another stack was reported as unresolved.** It resolves to a recipe
|
|
77
|
+
rather than to bytes, and the console's reader looked only for a catalog entry or an address — so a
|
|
78
|
+
working nested stack showed "does not resolve", with no zoom range and no box, and put a "sources
|
|
79
|
+
missing" badge on the stack above it. It now says what it stands for and how deep it reaches,
|
|
80
|
+
worked out from the sources that stack would serve.
|
|
81
|
+
|
|
82
|
+
- **A stack's advertised box ignored any source that stated none.** The union was taken over
|
|
83
|
+
whichever sources had a box, so a stack of a global base plus regional patches — an imported list,
|
|
84
|
+
exactly — advertised the patches as its extent and left the base out of it. A source with no box
|
|
85
|
+
is not one covering nothing, so it is now every box or none, and none means the world.
|
|
86
|
+
|
|
10
87
|
## 0.90.0
|
|
11
88
|
### ✨ Features and improvements
|
|
12
89
|
- **A stack source may name a URL instead of a category, archive or stack.** `{ "url": "https://…" }`,
|
package/docs/configuration.md
CHANGED
|
@@ -20,6 +20,7 @@ Some settings only take effect on restart. Those are marked **restart**.
|
|
|
20
20
|
- [Scheduled sources](#scheduled-sources)
|
|
21
21
|
- [Subscriptions](#subscriptions)
|
|
22
22
|
- [Serving tiles](#serving-tiles)
|
|
23
|
+
- [S3 buckets](#s3-buckets)
|
|
23
24
|
- [Mutable publishing](#mutable-publishing)
|
|
24
25
|
- [Authentication](#authentication)
|
|
25
26
|
- [Statistics](#statistics)
|
|
@@ -776,6 +777,96 @@ connection, a piece already in flight. Ten attempts later it is one piece at the
|
|
|
776
777
|
far end of an archive nobody has finished, and asking every fifteen seconds
|
|
777
778
|
achieves nothing but log lines.
|
|
778
779
|
|
|
780
|
+
## S3 buckets
|
|
781
|
+
|
|
782
|
+
A stack source may be an address rather than an archive this node holds, and
|
|
783
|
+
that address may name an object in a bucket:
|
|
784
|
+
|
|
785
|
+
```json
|
|
786
|
+
{
|
|
787
|
+
"sources": [{ "url": "s3://terrain/planet.pmtiles", "encoding": "terrarium" }]
|
|
788
|
+
}
|
|
789
|
+
```
|
|
790
|
+
|
|
791
|
+
Only for a bucket that is **not public**. A public object, or a presigned URL,
|
|
792
|
+
is an ordinary HTTPS address: write it as one and none of this applies.
|
|
793
|
+
|
|
794
|
+
```json
|
|
795
|
+
{
|
|
796
|
+
"s3": [
|
|
797
|
+
{
|
|
798
|
+
"bucket": "terrain",
|
|
799
|
+
"endpoint": "https://minio.lan:9000",
|
|
800
|
+
"region": "us-east-1",
|
|
801
|
+
"accessKeyId": "…",
|
|
802
|
+
"secretAccessKey": "…"
|
|
803
|
+
}
|
|
804
|
+
]
|
|
805
|
+
}
|
|
806
|
+
```
|
|
807
|
+
|
|
808
|
+
| field | default | |
|
|
809
|
+
| ----------------- | ------------------ | ---------------------------------------------------- |
|
|
810
|
+
| `bucket` | any | which bucket this row is for; unset matches all |
|
|
811
|
+
| `endpoint` | AWS for the region | what makes it S3-compatible rather than S3 |
|
|
812
|
+
| `region` | `us-east-1` | signed into every request, whether or not it is used |
|
|
813
|
+
| `accessKeyId` | — | |
|
|
814
|
+
| `secretAccessKey` | — | |
|
|
815
|
+
| `sessionToken` | unset | for temporary credentials |
|
|
816
|
+
| `pathStyle` | path, unless AWS | `false` puts the bucket in the hostname |
|
|
817
|
+
|
|
818
|
+
A source is read through the row naming its bucket, or through whichever row
|
|
819
|
+
names none: one set of credentials for a whole account is the ordinary case,
|
|
820
|
+
and writing the same keys once per bucket is busywork with a copy-paste
|
|
821
|
+
mistake in it.
|
|
822
|
+
|
|
823
|
+
### The endpoint is the whole point
|
|
824
|
+
|
|
825
|
+
MinIO, Ceph, Garage, Cloudflare R2, Wasabi and Backblaze B2 all answer the
|
|
826
|
+
same protocol at their own address, so naming that address is the difference
|
|
827
|
+
between "S3" and "S3-compatible". Path-style addressing is what they speak and
|
|
828
|
+
is the default here; AWS withdrew it for buckets made after September 2020, so
|
|
829
|
+
an `amazonaws.com` endpoint puts the bucket in the hostname instead. Set
|
|
830
|
+
`pathStyle` only for a server that insists on the other one.
|
|
831
|
+
|
|
832
|
+
### Credentials from the environment
|
|
833
|
+
|
|
834
|
+
With no rows configured at all, the standard variables are read — the same
|
|
835
|
+
ones the AWS CLI, rclone, go-pmtiles and tileserver-gl use, so a machine
|
|
836
|
+
already set up to reach a bucket needs nothing written here:
|
|
837
|
+
|
|
838
|
+
| variable | |
|
|
839
|
+
| -------------------------------------------- | ------------------------------- |
|
|
840
|
+
| `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` | required for any of it to apply |
|
|
841
|
+
| `AWS_SESSION_TOKEN` | temporary credentials |
|
|
842
|
+
| `AWS_REGION`, `AWS_DEFAULT_REGION` | region |
|
|
843
|
+
| `AWS_S3_ENDPOINT`, `AWS_ENDPOINT_URL_S3` | endpoint |
|
|
844
|
+
| `AWS_S3_FORCE_PATH_STYLE` | addressing |
|
|
845
|
+
|
|
846
|
+
### Signed here, not by an SDK
|
|
847
|
+
|
|
848
|
+
SigV4 for a GET is a hash, four HMACs and a string, all of which `node:crypto`
|
|
849
|
+
already has. An AWS client would be a large dependency for one signature, and
|
|
850
|
+
an optional one would make reading a bucket work on some installs and not on
|
|
851
|
+
others for no reason the operator could see — so it is neither optional nor a
|
|
852
|
+
dependency. The signer is checked against AWS's own published test vectors
|
|
853
|
+
rather than against itself.
|
|
854
|
+
|
|
855
|
+
Every request is signed as it goes out, including the range: the header, each
|
|
856
|
+
directory and each tile. Nothing is presigned and nothing is cached but the
|
|
857
|
+
bytes, so revoking a key stops reads at the next request rather than at the
|
|
858
|
+
next restart. Changing the settings drops the open readers for the same
|
|
859
|
+
reason.
|
|
860
|
+
|
|
861
|
+
### What a bucket source is, and is not
|
|
862
|
+
|
|
863
|
+
It is a URL source in every other respect — see
|
|
864
|
+
[tile-stacks.md](tile-stacks.md), "A source read straight from a URL". It
|
|
865
|
+
carries its own `minzoom`, `maxzoom` and `bounds`, it is never seeded or
|
|
866
|
+
retired, and it can be baked: a stack over private buckets exports to an
|
|
867
|
+
ordinary archive with an infohash, which is the path from "terrain we hold in
|
|
868
|
+
a bucket" to "terrain we publish".
|
|
869
|
+
|
|
779
870
|
## Mutable publishing
|
|
780
871
|
|
|
781
872
|
Announcing the current build of each category over the DHT (BEP 46).
|
package/docs/tile-stacks.md
CHANGED
|
@@ -60,6 +60,7 @@ of its parts.
|
|
|
60
60
|
- [Syncing a stack to another node](#syncing-a-stack-to-another-node)
|
|
61
61
|
- [What the offline merge got wrong](#what-the-offline-merge-got-wrong)
|
|
62
62
|
- [The stack editor](#the-stack-editor)
|
|
63
|
+
- [Merging vector sources](#merging-vector-sources)
|
|
63
64
|
- [Staging](#staging)
|
|
64
65
|
- [Open questions](#open-questions)
|
|
65
66
|
|
|
@@ -541,21 +542,42 @@ Derived from the resolved sources rather than from any one archive:
|
|
|
541
542
|
- `attribution` — every source's, concatenated.
|
|
542
543
|
- `tiles` — `/stacks/<id>/{z}/{x}/{y}.<ext>`.
|
|
543
544
|
|
|
544
|
-
Plus a non-standard `stack` block
|
|
545
|
-
|
|
546
|
-
|
|
545
|
+
Plus a non-standard `stack` block, mirroring the `latest` block on
|
|
546
|
+
`/latest/<category>/tiles.json`, so a consumer can tell one resolution from the
|
|
547
|
+
next:
|
|
547
548
|
|
|
548
549
|
```jsonc
|
|
549
550
|
"stack": {
|
|
550
551
|
"id": "planet-terrain",
|
|
551
|
-
"
|
|
552
|
-
"sources":
|
|
553
|
-
|
|
554
|
-
{ "category": "planet-bathymetry", "infohash": "…", "name": "…" }
|
|
555
|
-
]
|
|
552
|
+
"space": "elevation",
|
|
553
|
+
"sources": 2,
|
|
554
|
+
"revision": "9f2c1a77b3e04d16"
|
|
556
555
|
}
|
|
557
556
|
```
|
|
558
557
|
|
|
558
|
+
`revision` covers the recipe and what every source resolved to, so it moves
|
|
559
|
+
when a category resolves to a new build — which is the whole reason the block
|
|
560
|
+
exists.
|
|
561
|
+
|
|
562
|
+
### The sources themselves are not in it
|
|
563
|
+
|
|
564
|
+
They were once, and it was wrong twice over.
|
|
565
|
+
|
|
566
|
+
A stack's sources are not a client's to join. They are the ingredients of one
|
|
567
|
+
endpoint, and this document exists to point at that endpoint: a map reads
|
|
568
|
+
`tiles`, not the archives behind it. Listing them invites somebody to fetch
|
|
569
|
+
those instead, which is the one thing a stack is there to stop them having to
|
|
570
|
+
do.
|
|
571
|
+
|
|
572
|
+
And a source may be an **address**. A URL, or a bucket and a key, published in
|
|
573
|
+
a document that is served to anybody who can load the map — while the archive
|
|
574
|
+
at the other end of it is read with credentials nobody else has. The tiles are
|
|
575
|
+
public on purpose; where they come from is not.
|
|
576
|
+
|
|
577
|
+
What is left says the same thing the list said, in twenty bytes rather than
|
|
578
|
+
tens of kilobytes: how many sources there are, and a fingerprint that moves
|
|
579
|
+
when any of them does.
|
|
580
|
+
|
|
559
581
|
## When a source will not answer
|
|
560
582
|
|
|
561
583
|
A cache-mode source with no reachable peers, a category that resolves to an
|
|
@@ -1681,6 +1703,22 @@ archive — Mapterhorn's own base stops at z12 — is upscaled for a deeper
|
|
|
1681
1703
|
request exactly as GEBCO is, through the same code, because reading one is now
|
|
1682
1704
|
a question of which store method answers and nothing else.
|
|
1683
1705
|
|
|
1706
|
+
### Adding one by hand
|
|
1707
|
+
|
|
1708
|
+
**Add source → an address you type…** in the stack editor, which is the same
|
|
1709
|
+
source the importer writes and edits the same way. The card asks for two
|
|
1710
|
+
things nothing else can know: the address, and the zoom range the archive
|
|
1711
|
+
holds. A catalog archive states its range in its own header and a URL source
|
|
1712
|
+
has no header this node has read, so an unstated range means every tile asks
|
|
1713
|
+
it — correct, and the thing worth avoiding once there are several.
|
|
1714
|
+
|
|
1715
|
+
Anything published as a plain HTTPS download works, which includes an S3
|
|
1716
|
+
bucket that serves range requests: an object URL, or a presigned one, is an
|
|
1717
|
+
address like any other, and this asks for byte ranges the way any PMTiles
|
|
1718
|
+
reader does. What it cannot do is a private bucket named `s3://…`, which is
|
|
1719
|
+
not a URL a browser or a fetch can follow — that needs a signed request and
|
|
1720
|
+
somewhere to keep the credentials, and neither exists here yet.
|
|
1721
|
+
|
|
1684
1722
|
### Cheap to have hundreds of
|
|
1685
1723
|
|
|
1686
1724
|
The one thing that makes hundreds of these practical rather than merely
|
|
@@ -1830,6 +1868,14 @@ that is a map changing at 4am with nothing to say why. A hand-written source
|
|
|
1830
1868
|
carries no `importedFrom`, is never replaced, and is the place for anything the
|
|
1831
1869
|
index could not have told us.
|
|
1832
1870
|
|
|
1871
|
+
### Seeing them in the console
|
|
1872
|
+
|
|
1873
|
+
A stack's source list collapses behind a summary once there are more than
|
|
1874
|
+
five, with the count and the base named on the fold. A stack imported from a
|
|
1875
|
+
provider's list is several hundred rows and drawing them flat buries every
|
|
1876
|
+
other stack on the page under one of them; a stack of a base and a layer or
|
|
1877
|
+
two stays open, since folding that hides nothing worth a click.
|
|
1878
|
+
|
|
1833
1879
|
## Finding a stack
|
|
1834
1880
|
|
|
1835
1881
|
A stack has no infohash and appears in no feed, so nothing about it is
|
|
@@ -2175,6 +2221,135 @@ Debounced, and it should not follow the map past the stack's `maxzoom`: above
|
|
|
2175
2221
|
that the client overzooms and there is nothing new to see, but the tiles are
|
|
2176
2222
|
still requested and still composited.
|
|
2177
2223
|
|
|
2224
|
+
## Merging vector sources
|
|
2225
|
+
|
|
2226
|
+
**Not built.** This is the design, written down before anything is started so
|
|
2227
|
+
that the hard part is on record rather than discovered halfway through it.
|
|
2228
|
+
|
|
2229
|
+
Most of the machinery is indifferent to what a tile contains. Source
|
|
2230
|
+
resolution, per-source `minzoom`, `maxzoom` and `bounds`, painting order,
|
|
2231
|
+
nested stacks, the cache, the ETag, `unionOfTileIds` and the bake all treat a
|
|
2232
|
+
tile as an opaque thing, and the byte-for-byte passthrough already serves a
|
|
2233
|
+
vector archive today. What a vector space adds is a merge, and the merge is
|
|
2234
|
+
where the analogy with the two pixel spaces breaks.
|
|
2235
|
+
|
|
2236
|
+
### What has no vector analogue
|
|
2237
|
+
|
|
2238
|
+
`opacity`, `blend`, `feather`, `featherMetres`, `gaussianBlurSigma`,
|
|
2239
|
+
`output.encoding`, `output.tileSize`, `maskValues`, `maskRange` and
|
|
2240
|
+
`maskColors`. Every one of them is an operation on a pixel, and there are no
|
|
2241
|
+
pixels. A recipe naming any of them under a vector space should be **refused by
|
|
2242
|
+
validation**, not accepted and ignored: a stack that says `feather: 50` and
|
|
2243
|
+
draws a hard edge is a worse answer than one that will not save.
|
|
2244
|
+
|
|
2245
|
+
The idea of a mask survives in three other forms — clip to a **shape**, keep or
|
|
2246
|
+
drop whole **layers** by name, and filter **features** by a property. Those
|
|
2247
|
+
three are the vector mask, and none of them is a mask in the sense the
|
|
2248
|
+
elevation space means.
|
|
2249
|
+
|
|
2250
|
+
### Overlap is the whole problem
|
|
2251
|
+
|
|
2252
|
+
The pixel spaces work because the unit is a pixel, every source has one
|
|
2253
|
+
everywhere, and "the topmost covered source wins" is a complete rule. Vector's
|
|
2254
|
+
unit is a feature. Two sources that both carry a `water` layer over the same
|
|
2255
|
+
ground hold **two polygons of the same lake**, and emitting both overwrites
|
|
2256
|
+
neither. The tile carries both, the style draws both, and it shows as doubled
|
|
2257
|
+
outlines, darker semi-transparent fills, duplicated labels, and z-fighting
|
|
2258
|
+
between them.
|
|
2259
|
+
|
|
2260
|
+
So the first thing a vector space has to define is what winning means. Three
|
|
2261
|
+
answers, and the third is not a default:
|
|
2262
|
+
|
|
2263
|
+
**Per-layer replacement.** For each layer _name_, the topmost source carrying it
|
|
2264
|
+
supplies it whole, and the same-named layer of every source below is dropped for
|
|
2265
|
+
that tile. No geometry work at all: read the layer list, choose, re-emit. It is
|
|
2266
|
+
the right answer for a base plus an overlay of layers the base does not have —
|
|
2267
|
+
an OSM base under your own `buildings`, or under contours generated from the
|
|
2268
|
+
terrain these stacks already merge — and it is correct for a base plus a patch
|
|
2269
|
+
wherever the tile falls wholly inside the patch.
|
|
2270
|
+
|
|
2271
|
+
**Clip and union.** Each source is clipped to its own coverage, and the source
|
|
2272
|
+
below is additionally clipped _against_ the coverage of the one above it, so the
|
|
2273
|
+
two cannot both carry the same lake. This is the true counterpart of the pixel
|
|
2274
|
+
merge, and the only thing that works on a tile the boundary of a patch runs
|
|
2275
|
+
through. A **rectangle** clip in tile coordinates is cheap and exact —
|
|
2276
|
+
Sutherland-Hodgman for polygons, Liang-Barsky for lines — and a rectangle is
|
|
2277
|
+
what an imported file list states for every patch in it. An arbitrary
|
|
2278
|
+
**cutline** is a different matter and wants a real polygon-clipping library.
|
|
2279
|
+
|
|
2280
|
+
**Renaming.** The patch's `water` becomes `patch_water`, and the style decides
|
|
2281
|
+
what to do with it. No conflict and no geometry work, at the cost of a style
|
|
2282
|
+
that has to know. Worth having as a way out; not worth having as the rule.
|
|
2283
|
+
|
|
2284
|
+
The recommendation is per-layer replacement as the default, rectangle clipping
|
|
2285
|
+
for the base-and-patch case, and a **recipe warning** wherever two sources carry
|
|
2286
|
+
the same layer with no clip between them: _sources 0 and 1 both provide `water`,
|
|
2287
|
+
and tiles where they meet will carry both_. Silent doubling is the failure that
|
|
2288
|
+
gets found six months later by somebody squinting at a coastline.
|
|
2289
|
+
|
|
2290
|
+
### A seam cannot be feathered
|
|
2291
|
+
|
|
2292
|
+
Where two sources disagree geometrically at a boundary, a road jogs and a
|
|
2293
|
+
coastline steps, and there is nothing to fade. [Feathering a
|
|
2294
|
+
seam](#feathering-a-seam) works because a pixel can be a mixture of two values.
|
|
2295
|
+
A feature cannot be a mixture of two features.
|
|
2296
|
+
|
|
2297
|
+
Vector merging is therefore clean only where the sources were cut to agree — the
|
|
2298
|
+
same condition that makes a provider's raster patches merge cleanly, minus the
|
|
2299
|
+
fade that hides the cases where it is not quite true. That belongs in the
|
|
2300
|
+
console rather than on the map.
|
|
2301
|
+
|
|
2302
|
+
### Overzooming a parent
|
|
2303
|
+
|
|
2304
|
+
Possible, and needed: a base to z12 under a patch to z14 has to climb the base
|
|
2305
|
+
at z13, exactly as [resampling from a parent](#resampling-from-a-parent-tile)
|
|
2306
|
+
does for pixels. Scale the parent's coordinates by the zoom difference, subtract
|
|
2307
|
+
the quadrant's offset, clip to the extent plus the buffer. Exact for geometry,
|
|
2308
|
+
and it invents nothing: the generalisation and the label placement were chosen
|
|
2309
|
+
for the parent's zoom, so an overzoomed tile is thinner than a real one at that
|
|
2310
|
+
zoom and looks it.
|
|
2311
|
+
|
|
2312
|
+
### Extent, buffer and schema
|
|
2313
|
+
|
|
2314
|
+
Sources at different **extents** — 4096 and 512 are both common — have to be
|
|
2315
|
+
normalised to one before anything is combined. A power-of-two ratio is exact;
|
|
2316
|
+
anything else quantises coordinates.
|
|
2317
|
+
|
|
2318
|
+
**Buffers** have to be re-clipped after any coordinate change, or features leak
|
|
2319
|
+
past the edge of the tile and draw twice at the join.
|
|
2320
|
+
|
|
2321
|
+
**Schemas** are the quiet one. Two sources whose `water` layers use `class` and
|
|
2322
|
+
`kind` merge into a layer the style half understands, and nothing in the merge
|
|
2323
|
+
can reconcile them. Worth comparing key sets on the first merged tile and
|
|
2324
|
+
reporting the disagreement as a problem on the stack, the way an unresolved
|
|
2325
|
+
source is reported.
|
|
2326
|
+
|
|
2327
|
+
**Feature ids** collide across sources. Renumbering settles the collision and
|
|
2328
|
+
breaks any client-side feature state keyed on them, so it is a choice to make
|
|
2329
|
+
out loud rather than a detail to settle inside the encoder.
|
|
2330
|
+
|
|
2331
|
+
### The libraries
|
|
2332
|
+
|
|
2333
|
+
`@mapbox/vector-tile` and `pbf` to read, `vt-pbf` to write. Small, pure
|
|
2334
|
+
JavaScript, and ordinary dependencies rather than a probe: the optional
|
|
2335
|
+
treatment [the codec gets](#the-codec-problem) exists because a native build
|
|
2336
|
+
genuinely fails to install on some platforms, and none of that applies here.
|
|
2337
|
+
|
|
2338
|
+
Not written here either, unlike the PMTiles writer and the S3 signer. A protobuf
|
|
2339
|
+
round-trip has more edge cases than either — geometry command encoding, unknown
|
|
2340
|
+
value types, key and value pools shared across features — and getting one wrong
|
|
2341
|
+
produces a tile that parses and is subtly wrong.
|
|
2342
|
+
|
|
2343
|
+
### Staging, when it is started
|
|
2344
|
+
|
|
2345
|
+
1. **Layer operations only.** `space: "vector"`, with keep, drop and rename by
|
|
2346
|
+
layer name, over the per-source zoom and bounds skipping that already exists.
|
|
2347
|
+
Covers base-plus-overlay, and needs no geometry code whatsoever.
|
|
2348
|
+
2. **Overzoom and rectangle clipping.** Covers base-plus-patch, which is the
|
|
2349
|
+
shape [importing a list of URLs](#importing-a-list-of-urls) makes easy to
|
|
2350
|
+
build.
|
|
2351
|
+
3. **Cutline clipping and property filters.**
|
|
2352
|
+
|
|
2178
2353
|
## Staging
|
|
2179
2354
|
|
|
2180
2355
|
1. ~~**Recipe and resolution.**~~ Done. `data/stacks.json`, load and validate,
|
|
@@ -2206,10 +2381,11 @@ handling whatsoever.
|
|
|
2206
2381
|
|
|
2207
2382
|
## Open questions
|
|
2208
2383
|
|
|
2209
|
-
- **Vector tiles.**
|
|
2210
|
-
|
|
2211
|
-
|
|
2212
|
-
|
|
2384
|
+
- **Vector tiles.** Designed rather than open now: see [merging vector
|
|
2385
|
+
sources](#merging-vector-sources). It is its own space and its own merge,
|
|
2386
|
+
and what stays open is whether the overlap rule proposed there — layer
|
|
2387
|
+
replacement by default, clipping for a patch — holds up against a real
|
|
2388
|
+
pair of archives.
|
|
2213
2389
|
- **Does the recipe travel?** A stack is a small JSON document and the feed
|
|
2214
2390
|
already distributes documents. Publishing recipes so a subscriber gets the
|
|
2215
2391
|
stack along with its sources is attractive, and raises an obvious question
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pmtiles-swarm",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.92.0",
|
|
4
4
|
"description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "src/index.js",
|
package/src/api.js
CHANGED
|
@@ -50,15 +50,7 @@ import {
|
|
|
50
50
|
stackCoverage,
|
|
51
51
|
stackEtag,
|
|
52
52
|
} from './stacks.js';
|
|
53
|
-
|
|
54
|
-
/**
|
|
55
|
-
* How many sources a stack's TileJSON names before it starts counting.
|
|
56
|
-
*
|
|
57
|
-
* Generous for a hand-written recipe -- those run to a handful -- and a hard
|
|
58
|
-
* stop for one imported from a provider's index, where the list is hundreds
|
|
59
|
-
* long and every entry is an address no client reads.
|
|
60
|
-
*/
|
|
61
|
-
const TILEJSON_SOURCE_LIMIT = 25;
|
|
53
|
+
import { bakeRevision } from './bake.js';
|
|
62
54
|
|
|
63
55
|
const here = path.dirname(fileURLToPath(import.meta.url));
|
|
64
56
|
|
|
@@ -3400,45 +3392,64 @@ export function createApp({
|
|
|
3400
3392
|
// The order the recipe lists them: lowest priority first, last
|
|
3401
3393
|
// wins. The console shows this order as it stands rather than
|
|
3402
3394
|
// inverting it, so the screen and the file never disagree.
|
|
3403
|
-
sources: resolved.sources.map((source) =>
|
|
3404
|
-
|
|
3405
|
-
|
|
3406
|
-
|
|
3407
|
-
|
|
3408
|
-
|
|
3409
|
-
|
|
3410
|
-
|
|
3411
|
-
|
|
3412
|
-
|
|
3413
|
-
|
|
3414
|
-
|
|
3415
|
-
|
|
3416
|
-
|
|
3417
|
-
|
|
3418
|
-
|
|
3419
|
-
|
|
3420
|
-
|
|
3421
|
-
|
|
3422
|
-
|
|
3423
|
-
|
|
3424
|
-
|
|
3425
|
-
|
|
3426
|
-
|
|
3427
|
-
|
|
3428
|
-
|
|
3429
|
-
|
|
3430
|
-
|
|
3431
|
-
|
|
3432
|
-
|
|
3433
|
-
|
|
3434
|
-
|
|
3435
|
-
|
|
3436
|
-
source.
|
|
3437
|
-
|
|
3438
|
-
|
|
3439
|
-
|
|
3440
|
-
|
|
3441
|
-
|
|
3395
|
+
sources: resolved.sources.map((source) => {
|
|
3396
|
+
// A nested stack answers for the ground its own sources cover,
|
|
3397
|
+
// and knows it without opening anything -- so the row can say
|
|
3398
|
+
// what it reaches rather than a dash.
|
|
3399
|
+
const inner = source.nested ? stackCoverage(source.nested) : null;
|
|
3400
|
+
return {
|
|
3401
|
+
index: source.index,
|
|
3402
|
+
name: source.name,
|
|
3403
|
+
pinned: source.pinned,
|
|
3404
|
+
required: source.required,
|
|
3405
|
+
// What kind of source this is, said outright rather than left to
|
|
3406
|
+
// be inferred from which fields happen to be null. A URL source
|
|
3407
|
+
// has no infohash and no catalog entry, so a reader guessing from
|
|
3408
|
+
// those calls it an unresolved category -- which is two wrong
|
|
3409
|
+
// things about a source that is working perfectly well.
|
|
3410
|
+
kind: source.remote
|
|
3411
|
+
? 'url'
|
|
3412
|
+
: source.nested
|
|
3413
|
+
? 'stack'
|
|
3414
|
+
: source.pinned
|
|
3415
|
+
? 'archive'
|
|
3416
|
+
: 'category',
|
|
3417
|
+
// A URL source resolves by definition: there is nothing to look
|
|
3418
|
+
// up, the address is the answer. Whether it can actually be read
|
|
3419
|
+
// is a question for the first tile, the same as for an archive
|
|
3420
|
+
// this node holds but cannot open.
|
|
3421
|
+
resolved:
|
|
3422
|
+
Boolean(source.entry) ||
|
|
3423
|
+
Boolean(source.remote) ||
|
|
3424
|
+
Boolean(source.nested),
|
|
3425
|
+
// How many sources the nested recipe has, which is the only
|
|
3426
|
+
// useful thing to say in a column that names an archive file for
|
|
3427
|
+
// every other kind: a stack resolves to a recipe, not to bytes.
|
|
3428
|
+
nested: source.nested ? source.nested.sources.length : null,
|
|
3429
|
+
url: source.remote ?? null,
|
|
3430
|
+
infohash: source.entry?.infoHash ?? null,
|
|
3431
|
+
archiveName: source.entry?.name ?? null,
|
|
3432
|
+
// The recipe's own, where it states them. For a URL source that is
|
|
3433
|
+
// the only place they are known without opening the file, and it
|
|
3434
|
+
// is what the merge itself uses to decide whether to.
|
|
3435
|
+
minzoom:
|
|
3436
|
+
source.entry?.pmtiles?.minZoom ??
|
|
3437
|
+
source.source?.minzoom ??
|
|
3438
|
+
inner?.minzoom ??
|
|
3439
|
+
null,
|
|
3440
|
+
maxzoom:
|
|
3441
|
+
source.entry?.pmtiles?.maxZoom ??
|
|
3442
|
+
source.source?.maxzoom ??
|
|
3443
|
+
inner?.maxzoom ??
|
|
3444
|
+
null,
|
|
3445
|
+
bounds:
|
|
3446
|
+
source.entry?.pmtiles?.bounds ??
|
|
3447
|
+
(Array.isArray(source.source?.bounds)
|
|
3448
|
+
? source.source.bounds
|
|
3449
|
+
: (inner?.bounds ?? null)),
|
|
3450
|
+
format: source.entry?.pmtiles?.format ?? null,
|
|
3451
|
+
};
|
|
3452
|
+
}),
|
|
3442
3453
|
};
|
|
3443
3454
|
});
|
|
3444
3455
|
// Reported so the console can say "install sharp" beside the stacks
|
|
@@ -3781,34 +3792,27 @@ export function createApp({
|
|
|
3781
3792
|
minzoom: coverage.minzoom,
|
|
3782
3793
|
maxzoom: coverage.maxzoom,
|
|
3783
3794
|
bounds: coverage.bounds,
|
|
3784
|
-
//
|
|
3785
|
-
// from the next without diffing tile URLs
|
|
3786
|
-
// `/latest/` document carries a `latest` block.
|
|
3795
|
+
// Says that it resolved, and whether the resolution has moved, so a
|
|
3796
|
+
// consumer can tell one from the next without diffing tile URLs --
|
|
3797
|
+
// the same reason the `/latest/` document carries a `latest` block.
|
|
3787
3798
|
//
|
|
3788
|
-
//
|
|
3789
|
-
//
|
|
3790
|
-
//
|
|
3791
|
-
//
|
|
3792
|
-
//
|
|
3793
|
-
//
|
|
3799
|
+
// A count and a fingerprint rather than the sources themselves. They
|
|
3800
|
+
// were listed here once, and it was wrong twice over. A stack's
|
|
3801
|
+
// sources are not a client's to join: they are the ingredients of one
|
|
3802
|
+
// endpoint, and a document listing them invites somebody to fetch
|
|
3803
|
+
// those instead of the tiles. And a source may be an address -- a URL,
|
|
3804
|
+
// or a bucket and a key -- which this document has no business
|
|
3805
|
+
// publishing: the tiles are public where the archives behind them are
|
|
3806
|
+
// read with credentials nobody else has.
|
|
3807
|
+
//
|
|
3808
|
+
// The fingerprint covers the recipe and what every source resolved
|
|
3809
|
+
// to, so it moves when a category resolves to a new build. That was
|
|
3810
|
+
// the only thing the list was good for.
|
|
3794
3811
|
stack: {
|
|
3795
3812
|
id: resolved.stack.id,
|
|
3796
3813
|
space: resolved.stack.space ?? 'elevation',
|
|
3797
|
-
sources: resolved.sources
|
|
3798
|
-
|
|
3799
|
-
.map((source) => ({
|
|
3800
|
-
name: source.name,
|
|
3801
|
-
infohash: source.entry?.infoHash ?? null,
|
|
3802
|
-
archive: source.entry?.name ?? null,
|
|
3803
|
-
})),
|
|
3804
|
-
// Said rather than silently truncated: a consumer comparing two of
|
|
3805
|
-
// these has to know it is looking at part of a list.
|
|
3806
|
-
...(resolved.sources.length > TILEJSON_SOURCE_LIMIT
|
|
3807
|
-
? {
|
|
3808
|
-
more: resolved.sources.length - TILEJSON_SOURCE_LIMIT,
|
|
3809
|
-
total: resolved.sources.length,
|
|
3810
|
-
}
|
|
3811
|
-
: {}),
|
|
3814
|
+
sources: resolved.sources.length,
|
|
3815
|
+
revision: bakeRevision(resolved),
|
|
3812
3816
|
},
|
|
3813
3817
|
};
|
|
3814
3818
|
// A stack says how its tiles are encoded, so a style pointing at it does
|