pmtiles-swarm 0.66.0 → 0.68.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +102 -0
- package/docs/tile-stacks.md +166 -12
- package/package.json +1 -1
- package/src/api.js +91 -3
- package/src/bake-jobs.js +19 -6
- package/src/bake.js +36 -13
- package/src/cutline.js +368 -0
- package/src/cutlines.js +108 -0
- package/src/elevation.js +9 -0
- package/src/index.js +14 -1
- package/src/rgba.js +9 -0
- package/src/stack-tile.js +212 -9
- package/src/stacks.js +26 -0
- package/src/web/index.html +122 -25
- package/src/web/public.html +127 -28
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,108 @@
|
|
|
4
4
|
### ✨ Features and improvements
|
|
5
5
|
- _...Add new stuff here..._
|
|
6
6
|
|
|
7
|
+
### 🐞 Bug fixes
|
|
8
|
+
- _...Add new stuff here..._
|
|
9
|
+
|
|
10
|
+
## 0.68.0
|
|
11
|
+
### ✨ Features and improvements
|
|
12
|
+
- **A source can be clipped to a shape.** `cutline` names a polygon kept under `data/cutlines/`;
|
|
13
|
+
`bounds` is a rectangle written straight into the recipe. Both are chosen in the stack editor,
|
|
14
|
+
which offers the shapes this node actually has rather than asking anybody to remember a
|
|
15
|
+
filename.
|
|
16
|
+
|
|
17
|
+
Most of the time this is already solved a step earlier, and the design says so: a build running
|
|
18
|
+
`gdalwarp -cutline … -dstnodata` writes the boundary into the archive, and `maskValues` removes
|
|
19
|
+
exactly those pixels. What that cannot reach is a source **this node did not build** — an
|
|
20
|
+
archive is content-addressed, so re-clipping one means republishing it. That is the case this is
|
|
21
|
+
for, and it is the federated case this project exists for.
|
|
22
|
+
|
|
23
|
+
**A rectangle is a cutline with four corners**, built through the same code rather than beside
|
|
24
|
+
it, so there is no second implementation to disagree with the first. A test asserts a `bounds`
|
|
25
|
+
and the same shape drawn as GeoJSON classify identically across a whole zoom level.
|
|
26
|
+
|
|
27
|
+
The cost is avoided rather than paid. Every tile is first classified **outside**, **inside** or
|
|
28
|
+
**partial**: outside skips the read entirely — no swarm round trip, no decode, no merge — inside
|
|
29
|
+
skips the mask entirely, and only a tile the edge actually crosses is rasterised, by scanline,
|
|
30
|
+
over segments found through a grid index built once when the cutline loads. For a country
|
|
31
|
+
boundary that is a band one tile wide; everything else is settled without touching a pixel.
|
|
32
|
+
|
|
33
|
+
It composes with the per-tile short-circuit: a clipped source still hands its bytes through
|
|
34
|
+
untouched where the tile is wholly inside, because there the clip provably changes nothing.
|
|
35
|
+
|
|
36
|
+
**A cutline a recipe names and this node has not got refuses the source.** Not "serve it
|
|
37
|
+
unclipped" — that would put back exactly the data somebody asked to remove, which is the one
|
|
38
|
+
failure a clip must not have. The stack reports it beside its other problems and the editor says
|
|
39
|
+
so on the row.
|
|
40
|
+
|
|
41
|
+
Also fixed while building it: horizontal edges were being dropped when a shape was prepared, on
|
|
42
|
+
the grounds that they contribute nothing to the even-odd rule. True, and they are still edges —
|
|
43
|
+
dropping them meant nothing noticed a rectangle's north and south sides crossing a tile, which
|
|
44
|
+
read as `inside` for a tile half of which was outside, and the mask was then never applied.
|
|
45
|
+
|
|
46
|
+
|
|
47
|
+
- **A stack that merges somewhere no longer merges everywhere.** Passthrough was decided per
|
|
48
|
+
recipe: one source with a mask made the whole stack a merging one, and every tile was decoded
|
|
49
|
+
and re-encoded — including the great majority where a single untouched source covers the ground
|
|
50
|
+
and its stored bytes were already the answer. `passThroughRead` now decides that per tile,
|
|
51
|
+
before anything is decoded, which is the point: checked afterwards it would save the encode and
|
|
52
|
+
not the decode.
|
|
53
|
+
|
|
54
|
+
Its own function with seventeen tests, one per condition, because a short-circuit that fires
|
|
55
|
+
when it should not does not fail — it serves the wrong pixels quietly, and an archive baked from
|
|
56
|
+
them is wrong the same way.
|
|
57
|
+
|
|
58
|
+
Masks are the subtle half, and the reason it refuses any source carrying one. A tile having a
|
|
59
|
+
single contributor does not make that contributor cover the tile: a mask turns pixels into
|
|
60
|
+
nodata, the merge fills those, and handing the stored bytes over instead would show the ground
|
|
61
|
+
the mask was there to remove. With masks refused, nodata has nowhere else to come from —
|
|
62
|
+
`decodeHeights` is arithmetic over bytes, and the only other sources of it are the parent
|
|
63
|
+
resample and a resize, both refused as well.
|
|
64
|
+
|
|
65
|
+
- **The catalogue page lists stacks.** A stack has no infohash and appears in no feed, so nothing
|
|
66
|
+
about it was discoverable: the only way to know a node served one was to be told its id.
|
|
67
|
+
`GET /stacks/` answers the list on the public listener, beside the tiles and TileJSON it
|
|
68
|
+
describes, and the page renders each with its TileJSON, XYZ template and preview — terrain
|
|
69
|
+
first where the stack is terrain.
|
|
70
|
+
|
|
71
|
+
Not the console's list. That one names what each source resolved to and what is missing, which
|
|
72
|
+
is the operator's view and names infohashes a visitor was never offered. The public one also
|
|
73
|
+
leaves out any stack it cannot serve — a recipe with a problem, or one wanting a codec this node
|
|
74
|
+
has not got — because a link that answers 501 is worse than no link.
|
|
75
|
+
|
|
76
|
+
|
|
77
|
+
|
|
78
|
+
### 🐞 Bug fixes
|
|
79
|
+
- _...Add new stuff here..._
|
|
80
|
+
|
|
81
|
+
## 0.67.0
|
|
82
|
+
### ✨ Features and improvements
|
|
83
|
+
- **An exported archive is dated, and the filename is its own field.** Both the archive's name and
|
|
84
|
+
the file it lands in now carry the date by default, and both can be changed — separately. They
|
|
85
|
+
answer different questions: `Terrain-20260822.pmtiles` is what somebody finds on disk,
|
|
86
|
+
`Terrain 20260822` is what a map client shows, and tying one to the other only guarantees that
|
|
87
|
+
one of them is wrong whenever they should differ.
|
|
88
|
+
|
|
89
|
+
A filename given by hand is reduced to a single path segment before it is used, because it is
|
|
90
|
+
joined to a save path and a filename is exactly the kind of field somebody puts a slash in.
|
|
91
|
+
`../../etc/passwd` is tested.
|
|
92
|
+
|
|
93
|
+
The description starts empty and stays empty unless something is typed. It used to be
|
|
94
|
+
prefilled from the recipe, which is a different thing — a recipe describes how tiles are
|
|
95
|
+
combined, an archive describes what it is, and only the person exporting it knows that. The
|
|
96
|
+
server no longer falls back to the recipe either: filling in a field the dialog showed as
|
|
97
|
+
blank is a worse surprise than having no description. The date is recorded there regardless,
|
|
98
|
+
because a name can be changed to anything and then nothing else says when the archive was
|
|
99
|
+
made.
|
|
100
|
+
|
|
101
|
+
This corrects something 0.64.0 asserted and this project does not do. The name was left undated
|
|
102
|
+
on the reasoning that `/latest/<category>/` follows a rebuild by name. It does not: it resolves a
|
|
103
|
+
category and takes the newest by date, and nothing here looks an archive up by name at all — the
|
|
104
|
+
only name comparison in the codebase refuses two archives the same *file* path. The
|
|
105
|
+
documentation said so as well, and now says what is true.
|
|
106
|
+
|
|
107
|
+
|
|
108
|
+
|
|
7
109
|
### 🐞 Bug fixes
|
|
8
110
|
- _...Add new stuff here..._
|
|
9
111
|
|
package/docs/tile-stacks.md
CHANGED
|
@@ -49,6 +49,8 @@ of its parts.
|
|
|
49
49
|
- [What identifies a bake](#what-identifies-a-bake)
|
|
50
50
|
- [Starting one, and watching it](#starting-one-and-watching-it)
|
|
51
51
|
- [What exists now](#what-exists-now)
|
|
52
|
+
- [Clipping a source to a shape](#clipping-a-source-to-a-shape)
|
|
53
|
+
- [Finding a stack](#finding-a-stack)
|
|
52
54
|
- [Syncing a stack to another node](#syncing-a-stack-to-another-node)
|
|
53
55
|
- [What the offline merge got wrong](#what-the-offline-merge-got-wrong)
|
|
54
56
|
- [The stack editor](#the-stack-editor)
|
|
@@ -297,6 +299,25 @@ For a request `GET /stacks/<id>/{z}/{x}/{y}.webp`:
|
|
|
297
299
|
its bytes through untouched. No decode, no encode. This is what a stack
|
|
298
300
|
degenerates to over most of the world when the top layer is dense, and it
|
|
299
301
|
costs a comparison to detect.
|
|
302
|
+
|
|
303
|
+
`passThroughRead` in `src/stack-tile.js`, checked _before_ anything is
|
|
304
|
+
decoded — checked afterwards it would save the encode and not the decode.
|
|
305
|
+
Its own function with its own tests, because a short-circuit that fires when
|
|
306
|
+
it should not does not fail: it serves the wrong pixels quietly, and an
|
|
307
|
+
archive baked from them is wrong the same way.
|
|
308
|
+
|
|
309
|
+
The masks are the subtle half. A tile having one contributor does not make
|
|
310
|
+
that contributor cover the tile: a mask turns pixels into nodata, the merge
|
|
311
|
+
fills those, and passing the stored bytes through instead would show the
|
|
312
|
+
ground the mask was there to remove. Refusing any source carrying a mask is
|
|
313
|
+
what makes the rest safe — nodata has nowhere else to come from, since
|
|
314
|
+
`decodeHeights` is arithmetic over bytes and the only other sources of it
|
|
315
|
+
are the parent resample and a resize, both refused as well.
|
|
316
|
+
|
|
317
|
+
An explicit output size is refused for a different reason: the source's
|
|
318
|
+
pixel width is not known until it is decoded, so a resize cannot be ruled
|
|
319
|
+
out from here.
|
|
320
|
+
|
|
300
321
|
6. **Decode** each contributing tile in a worker.
|
|
301
322
|
7. **Mask** to a coverage mask, then apply `heightAdjustment` (elevation) or
|
|
302
323
|
`opacity` (rgba). Masking comes first — the mask values are exact decoded
|
|
@@ -698,14 +719,26 @@ resolves to whichever build is current, so the same recipe over a rebuilt source
|
|
|
698
719
|
is a different bake — and a checkpoint that could not tell would resume across
|
|
699
720
|
the change and produce an archive that is half one map and half another.
|
|
700
721
|
|
|
701
|
-
|
|
702
|
-
|
|
703
|
-
|
|
704
|
-
|
|
705
|
-
|
|
722
|
+
Both the file and the archive's name are dated, and both can be changed —
|
|
723
|
+
separately, because they answer different questions. `Terrain-20260822.pmtiles`
|
|
724
|
+
is what somebody finds on disk; `Terrain 20260822` is what a map client shows.
|
|
725
|
+
Tying one to the other only guarantees that one of them is wrong whenever they
|
|
726
|
+
should differ.
|
|
727
|
+
|
|
728
|
+
An earlier version of this document said the name had to stay undated so
|
|
729
|
+
`/latest/<category>/` could follow a rebuild. That was wrong. `/latest/`
|
|
730
|
+
resolves a category and takes the newest by date; nothing in this project looks
|
|
731
|
+
an archive up by name at all, and the only name comparison there is refuses two
|
|
732
|
+
archives the same _file_ path. A dated name is free, and it answers the question
|
|
733
|
+
somebody holding two builds actually has.
|
|
706
734
|
|
|
707
735
|
`name` is always written, because these archives get converted to mbtiles by
|
|
708
|
-
other tools and a nameless metadata block is not valid there.
|
|
736
|
+
other tools and a nameless metadata block is not valid there. The date also goes
|
|
737
|
+
in `description`, so an archive read out of context says what produced it.
|
|
738
|
+
|
|
739
|
+
A chosen filename is reduced to one path segment before it is used. That is not
|
|
740
|
+
politeness — the name is joined to a save path, and a filename is exactly the
|
|
741
|
+
kind of field somebody puts a slash in.
|
|
709
742
|
|
|
710
743
|
### What a baked archive says about itself
|
|
711
744
|
|
|
@@ -739,10 +772,11 @@ a name this node does not know, or a path it cannot write, is the caller's
|
|
|
739
772
|
mistake and they can fix it, but only if they are told now rather than an hour
|
|
740
773
|
later.
|
|
741
774
|
|
|
742
|
-
**What it is called**
|
|
743
|
-
the
|
|
744
|
-
|
|
745
|
-
|
|
775
|
+
**What it is called** is two fields, both dated by default and both editable:
|
|
776
|
+
the archive's name, and the filename. The dialog says what a filename will
|
|
777
|
+
actually become where sanitising would change it, using the same rule the server
|
|
778
|
+
applies — a field that shows one filename while the server writes another is
|
|
779
|
+
worse than a field that shows nothing.
|
|
746
780
|
|
|
747
781
|
A bake has two halves and they are watched in two places, deliberately. Merging
|
|
748
782
|
is about a stack, so it is reported on the stack — tiles written, tiles skipped,
|
|
@@ -780,6 +814,124 @@ means re-hashing everything already buffered, and the cost of starting it empty
|
|
|
780
814
|
is that a tile identical to one from before the interruption is stored twice.
|
|
781
815
|
The archive is correct either way; it is a little larger.
|
|
782
816
|
|
|
817
|
+
## Clipping a source to a shape
|
|
818
|
+
|
|
819
|
+
A source is often only meant to apply inside a boundary — a national DEM inside
|
|
820
|
+
its country, a survey inside its extent — and everywhere else the layer beneath
|
|
821
|
+
it should show through.
|
|
822
|
+
|
|
823
|
+
**Most of the time this is already solved, one step earlier.** A build that runs
|
|
824
|
+
`gdalwarp -cutline … -dstnodata` writes the boundary into the archive: outside
|
|
825
|
+
it, every pixel is the nodata value. `maskValues` then removes exactly those,
|
|
826
|
+
which is what the offline merge configs do and what a stack recipe does with the
|
|
827
|
+
same field. Nothing more is needed, and a cutline that duplicated it would be
|
|
828
|
+
slower and no more correct.
|
|
829
|
+
|
|
830
|
+
What that cannot reach is a source **this node did not build**. An archive is
|
|
831
|
+
content-addressed, so re-clipping one means republishing it — impossible for
|
|
832
|
+
somebody else's, and expensive for a large one whose boundary has changed. That
|
|
833
|
+
is the case this is for, and it is squarely the federated case this project
|
|
834
|
+
exists for.
|
|
835
|
+
|
|
836
|
+
Two shapes, and one of them is nearly free:
|
|
837
|
+
|
|
838
|
+
```json
|
|
839
|
+
{ "category": "opendtm-de", "cutline": "germany" }
|
|
840
|
+
{ "category": "massgis", "bounds": [-73.6, 41.1, -69.8, 42.9] }
|
|
841
|
+
```
|
|
842
|
+
|
|
843
|
+
`bounds` is a rectangle in WGS84, written the way every other bounds in this
|
|
844
|
+
project is. `cutline` names a polygon. They are the same question asked of
|
|
845
|
+
different shapes, so they are the same code: a rectangle is a cutline with four
|
|
846
|
+
corners, and treating it as one means there is no second implementation to
|
|
847
|
+
disagree with the first. What differs is only that a rectangle needs no file, no
|
|
848
|
+
index and no rasterising worth the name — which is why it is worth having even
|
|
849
|
+
though the polygon subsumes it.
|
|
850
|
+
|
|
851
|
+
### Geometry by reference
|
|
852
|
+
|
|
853
|
+
A recipe names a cutline rather than carrying it. `"cutline": "germany"`
|
|
854
|
+
resolves to `data/cutlines/germany.geojson`, in WGS84. A recipe stays a small
|
|
855
|
+
document that a person can read and a peer could one day be sent; a megabyte of
|
|
856
|
+
coordinates inlined in one is neither. A rectangle is small enough to sit in the
|
|
857
|
+
recipe itself, so `bounds` does.
|
|
858
|
+
|
|
859
|
+
WGS84 on purpose. Web Mercator is arithmetic from there — no projection library,
|
|
860
|
+
no dependency, no CRS handling beyond refusing what is not WGS84. Anything else
|
|
861
|
+
converts once with the `ogr2ogr` that produced the shapefile in the first place.
|
|
862
|
+
|
|
863
|
+
Edges are treated as straight lines in Mercator after their endpoints are
|
|
864
|
+
projected. That is what every rasteriser does, GDAL's included, and the error
|
|
865
|
+
over a tile's span is far below a pixel except at zooms where a tile spans a
|
|
866
|
+
continent — where a cutline is not the thing deciding the answer anyway.
|
|
867
|
+
|
|
868
|
+
### Three answers, and only one of them costs anything
|
|
869
|
+
|
|
870
|
+
The cost of a clip is per pixel, and paying it per tile would make a continental
|
|
871
|
+
boundary unusable. It is avoided by asking a cheaper question first:
|
|
872
|
+
|
|
873
|
+
- **Outside.** No part of the tile is within the shape. The source contributes
|
|
874
|
+
nothing — and this is decided _before the tile is read_, so it costs no swarm
|
|
875
|
+
read, no decode and no merge.
|
|
876
|
+
- **Inside.** The tile is wholly within the shape. The clip cannot change any
|
|
877
|
+
pixel, so it is not applied at all.
|
|
878
|
+
- **Partial.** Only here is a mask rasterised, and only for the tile in hand.
|
|
879
|
+
|
|
880
|
+
For a country boundary at any useful zoom, almost every tile is one of the first
|
|
881
|
+
two. The third is a band one tile wide along the border.
|
|
882
|
+
|
|
883
|
+
Classification is a bounding-box test, then a look at whichever edges could
|
|
884
|
+
reach the tile. "Whichever" is the important word: a national boundary is tens
|
|
885
|
+
of thousands of segments, and testing all of them per tile would cost more than
|
|
886
|
+
the rasterising it is meant to avoid. The segments are indexed into a coarse
|
|
887
|
+
grid once, when the cutline is loaded, and a tile only ever looks at the buckets
|
|
888
|
+
it overlaps.
|
|
889
|
+
|
|
890
|
+
With no edge near the tile, one point decides it: a tile that no boundary
|
|
891
|
+
crosses is entirely on one side of it.
|
|
892
|
+
|
|
893
|
+
### Where it applies
|
|
894
|
+
|
|
895
|
+
As a coverage mask on the decoded heights, in the same place and for the same
|
|
896
|
+
reason as `maskValues` — before any height adjustment, because an adjustment
|
|
897
|
+
would shift values out from under a comparison, and because the answer is the
|
|
898
|
+
same either way: nothing here.
|
|
899
|
+
|
|
900
|
+
It applies at the geometry of the tile that was **asked for**, not of the tile
|
|
901
|
+
that answered. A source falling back to a parent still gets clipped to the
|
|
902
|
+
square being served, which is the ground the pixels will end up covering.
|
|
903
|
+
|
|
904
|
+
### What it costs the short-circuit
|
|
905
|
+
|
|
906
|
+
A clipped source cannot take the passthrough in
|
|
907
|
+
[Evaluating one tile](#evaluating-one-tile) — with one exception that is worth
|
|
908
|
+
having, because it is the common one. Where the tile is classified **inside**,
|
|
909
|
+
the clip provably changes nothing, and the bytes may go through untouched
|
|
910
|
+
exactly as if none were named. Outside, there is no contribution at all, so the
|
|
911
|
+
question does not arise. Only a partial tile is genuinely disqualified.
|
|
912
|
+
|
|
913
|
+
### Refusing what cannot be honoured
|
|
914
|
+
|
|
915
|
+
A cutline named by a recipe and not present on disk is a stack problem, reported
|
|
916
|
+
the way an unresolvable source is: the stack is listed, and it says what is
|
|
917
|
+
missing. Serving a source unclipped because its cutline could not be found would
|
|
918
|
+
put back exactly the data somebody asked to remove, which is the one failure a
|
|
919
|
+
clip must not have.
|
|
920
|
+
|
|
921
|
+
## Finding a stack
|
|
922
|
+
|
|
923
|
+
A stack has no infohash and appears in no feed, so nothing about it is
|
|
924
|
+
discoverable the way an archive is. `GET /stacks/` answers the list, on the
|
|
925
|
+
public listener beside the tiles and the TileJSON it describes, and the
|
|
926
|
+
catalogue page renders it.
|
|
927
|
+
|
|
928
|
+
Not the console's list. That one reports what each source resolved to, what is
|
|
929
|
+
missing and what cannot be served — the operator's view, naming infohashes a
|
|
930
|
+
visitor was never offered. The public one says what a visitor can point a map
|
|
931
|
+
at, and says nothing at all about a stack they cannot: a recipe with a problem,
|
|
932
|
+
or one asking for pixel work this node has no codec for, is left out rather than
|
|
933
|
+
advertised as a link that answers 501.
|
|
934
|
+
|
|
783
935
|
## Syncing a stack to another node
|
|
784
936
|
|
|
785
937
|
The question is whether a stack can travel between nodes the way an archive
|
|
@@ -890,7 +1042,9 @@ state says what a stack is and how to add one instead. It is not conditioned on
|
|
|
890
1042
|
having a codec either — a passthrough stack needs none, and a node without
|
|
891
1043
|
`sharp` can still build and serve one.
|
|
892
1044
|
|
|
893
|
-
What follows is the editing half, which does
|
|
1045
|
+
What follows is the editing half, which the console now does: a stack can be
|
|
1046
|
+
added, changed and removed there, and the source rows are edited in the order
|
|
1047
|
+
the file holds them.
|
|
894
1048
|
|
|
895
1049
|
### The list is shown in the file's order
|
|
896
1050
|
|
|
@@ -1090,7 +1244,7 @@ handling whatsoever.
|
|
|
1090
1244
|
level up, and `resampleFromParent` crops the right sub-square.
|
|
1091
1245
|
- **Source smaller than the output** (256px source, 512px stack) needs the
|
|
1092
1246
|
other direction: fetch the four children at `z+1` and assemble them into
|
|
1093
|
-
one grid before merging.
|
|
1247
|
+
one grid before merging.
|
|
1094
1248
|
|
|
1095
1249
|
**Both are implemented.** A size may be asked for in the URL —
|
|
1096
1250
|
`/stacks/<id>/512/{z}/{x}/{y}.webp`, the way tileserver-gl takes one — or set
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pmtiles-swarm",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.68.0",
|
|
4
4
|
"description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "src/index.js",
|
package/src/api.js
CHANGED
|
@@ -36,7 +36,7 @@ import { buildTileJson, extensionMatches, tileExtension } from './tilejson.js';
|
|
|
36
36
|
import { SUMMARY_VERSION } from './pmtiles-probe.js';
|
|
37
37
|
import { TileReadError } from './tiles.js';
|
|
38
38
|
import { loadCodec } from './codec.js';
|
|
39
|
-
import { answerStackTile, outputSize } from './stack-tile.js';
|
|
39
|
+
import { answerStackTile, outputFormat, outputSize } from './stack-tile.js';
|
|
40
40
|
import {
|
|
41
41
|
isPinned,
|
|
42
42
|
needsCodec,
|
|
@@ -215,6 +215,7 @@ export function createApp({
|
|
|
215
215
|
tiles,
|
|
216
216
|
stacks,
|
|
217
217
|
stackCache,
|
|
218
|
+
cutlines,
|
|
218
219
|
bakes,
|
|
219
220
|
warm,
|
|
220
221
|
config,
|
|
@@ -2264,6 +2265,66 @@ export function createApp({
|
|
|
2264
2265
|
}),
|
|
2265
2266
|
);
|
|
2266
2267
|
|
|
2268
|
+
// The same idea one prefix over. Everything under /stacks/ is already public
|
|
2269
|
+
// -- the TileJSON, the tiles, the preview -- so the index of what is on
|
|
2270
|
+
// offer belongs beside them rather than behind the console's door.
|
|
2271
|
+
//
|
|
2272
|
+
// Not the console's list. That one reports what each source resolved to,
|
|
2273
|
+
// what is missing and what cannot be served, which is the operator's
|
|
2274
|
+
// business and names infohashes a visitor was not offered. This says what a
|
|
2275
|
+
// visitor can point a map at, and says nothing about a stack they cannot.
|
|
2276
|
+
app.get(
|
|
2277
|
+
'/stacks/',
|
|
2278
|
+
route(async (req, res) => {
|
|
2279
|
+
await stacks?.refresh();
|
|
2280
|
+
res.setHeader('access-control-allow-origin', '*');
|
|
2281
|
+
|
|
2282
|
+
const base = baseUrl(req);
|
|
2283
|
+
const listed = [];
|
|
2284
|
+
for (const stack of stacks?.list() ?? []) {
|
|
2285
|
+
const resolved = resolveFor(stack, req);
|
|
2286
|
+
// A recipe with a problem, or one asking for pixel work this node
|
|
2287
|
+
// cannot do, is not something to advertise: a link that answers 501 is
|
|
2288
|
+
// worse than no link.
|
|
2289
|
+
if (stacks.problems(stack.id).length > 0) continue;
|
|
2290
|
+
if (needsCodec(stack) && !(await loadCodec())) continue;
|
|
2291
|
+
if (!resolved.sources.some((source) => source.entry)) continue;
|
|
2292
|
+
|
|
2293
|
+
const coverage = stackCoverage(resolved);
|
|
2294
|
+
const format = outputFormat(resolved);
|
|
2295
|
+
const id = encodeURIComponent(stack.id);
|
|
2296
|
+
listed.push({
|
|
2297
|
+
id: stack.id,
|
|
2298
|
+
title: stack.title ?? stack.id,
|
|
2299
|
+
space: stack.space ?? 'elevation',
|
|
2300
|
+
minzoom: coverage.minzoom,
|
|
2301
|
+
maxzoom: coverage.maxzoom,
|
|
2302
|
+
bounds: coverage.bounds,
|
|
2303
|
+
format,
|
|
2304
|
+
encoding: stack.output?.encoding ?? null,
|
|
2305
|
+
sparse: stack.sparse ?? true,
|
|
2306
|
+
sources: resolved.sources.length,
|
|
2307
|
+
endpoints: {
|
|
2308
|
+
tileJson: `${base}/stacks/${id}/tiles.json`,
|
|
2309
|
+
xyz: `${base}/stacks/${id}/{z}/{x}/{y}.${tileExtension(format)}`,
|
|
2310
|
+
preview: `${base}/stacks/${id}/preview`,
|
|
2311
|
+
},
|
|
2312
|
+
});
|
|
2313
|
+
}
|
|
2314
|
+
|
|
2315
|
+
res.setHeader(
|
|
2316
|
+
'etag',
|
|
2317
|
+
`"${crypto.createHash('sha1').update(JSON.stringify(listed)).digest('hex')}"`,
|
|
2318
|
+
);
|
|
2319
|
+
res.setHeader('cache-control', 'public, max-age=60, must-revalidate');
|
|
2320
|
+
res.json({
|
|
2321
|
+
format: 'pmtiles-swarm-stacks/1',
|
|
2322
|
+
generatedAt: new Date().toISOString(),
|
|
2323
|
+
stacks: listed,
|
|
2324
|
+
});
|
|
2325
|
+
}),
|
|
2326
|
+
);
|
|
2327
|
+
|
|
2267
2328
|
// A stable handle for "the current one". Every archive is addressed by
|
|
2268
2329
|
// infohash, which is right — it is what makes a tile immutable — but it
|
|
2269
2330
|
// leaves nothing for a style to point at that survives a rebuild. A category
|
|
@@ -3083,6 +3144,22 @@ export function createApp({
|
|
|
3083
3144
|
* @param {import('express').Request} req - The request, for its credential.
|
|
3084
3145
|
* @returns {object} - The resolution.
|
|
3085
3146
|
*/
|
|
3147
|
+
/**
|
|
3148
|
+
* Cutlines a recipe names that this node does not have.
|
|
3149
|
+
*
|
|
3150
|
+
* Serving a source unclipped because its shape could not be found would put
|
|
3151
|
+
* back exactly the data somebody asked to remove, so the stack says so and
|
|
3152
|
+
* the tile path refuses the source.
|
|
3153
|
+
* @param {object} stack - The recipe.
|
|
3154
|
+
* @returns {string[]} - One line per missing cutline.
|
|
3155
|
+
*/
|
|
3156
|
+
const cutlineProblems = (stack) =>
|
|
3157
|
+
(stack.sources ?? []).flatMap((source, index) => {
|
|
3158
|
+
if (!source?.cutline) return [];
|
|
3159
|
+
const why = cutlines?.problem(source.cutline);
|
|
3160
|
+
return why ? [`sources[${index}].cutline: ${why}`] : [];
|
|
3161
|
+
});
|
|
3162
|
+
|
|
3086
3163
|
const resolveFor = (stack, req) =>
|
|
3087
3164
|
resolveStack(stack, {
|
|
3088
3165
|
archive: (hash) => catalog.get(hash) ?? null,
|
|
@@ -3108,7 +3185,11 @@ export function createApp({
|
|
|
3108
3185
|
id: stack.id,
|
|
3109
3186
|
title: stack.title ?? stack.id,
|
|
3110
3187
|
space: stack.space ?? 'elevation',
|
|
3111
|
-
|
|
3188
|
+
// The store's own problems, plus any cutline a source names that
|
|
3189
|
+
// this node has not got. Both stop the stack being servable and
|
|
3190
|
+
// both belong in the same list -- a stack listed as fine that
|
|
3191
|
+
// refuses every tile is worse than one that says why.
|
|
3192
|
+
problems: [...stacks.problems(stack.id), ...cutlineProblems(stack)],
|
|
3112
3193
|
// Named separately from `problems` because it is not a mistake: a
|
|
3113
3194
|
// recipe asking for pixel work is perfectly valid and simply cannot
|
|
3114
3195
|
// be served yet. See docs/tile-stacks.md — "The codec problem".
|
|
@@ -3154,7 +3235,13 @@ export function createApp({
|
|
|
3154
3235
|
// that need it, rather than leaving a 501 to be discovered at the first
|
|
3155
3236
|
// tile. Null is a fact about this node, not about the recipes.
|
|
3156
3237
|
const codec = await loadCodec();
|
|
3157
|
-
res.json({
|
|
3238
|
+
res.json({
|
|
3239
|
+
stacks: list,
|
|
3240
|
+
codec: codec?.name ?? null,
|
|
3241
|
+
// So the editor can offer the shapes this node actually has rather
|
|
3242
|
+
// than asking somebody to remember a filename.
|
|
3243
|
+
cutlines: cutlines?.list() ?? [],
|
|
3244
|
+
});
|
|
3158
3245
|
}),
|
|
3159
3246
|
);
|
|
3160
3247
|
|
|
@@ -3541,6 +3628,7 @@ export function createApp({
|
|
|
3541
3628
|
signal: controller.signal,
|
|
3542
3629
|
size,
|
|
3543
3630
|
format,
|
|
3631
|
+
cutlines,
|
|
3544
3632
|
});
|
|
3545
3633
|
} catch (error) {
|
|
3546
3634
|
// The client went away mid-merge. Nothing to answer and nobody to answer
|
package/src/bake-jobs.js
CHANGED
|
@@ -3,6 +3,7 @@ import {
|
|
|
3
3
|
assertBakeable,
|
|
4
4
|
bakeRevision,
|
|
5
5
|
bakeStack,
|
|
6
|
+
bakedArchiveName,
|
|
6
7
|
bakedName,
|
|
7
8
|
mergeTileFor,
|
|
8
9
|
} from './bake.js';
|
|
@@ -34,16 +35,18 @@ export class BakeManager {
|
|
|
34
35
|
#tiles;
|
|
35
36
|
#config;
|
|
36
37
|
#loadCodec;
|
|
38
|
+
#cutlines;
|
|
37
39
|
#jobs = new Map();
|
|
38
40
|
|
|
39
41
|
/**
|
|
40
42
|
* @param {object} deps - The library, the tile store, the config and the codec probe.
|
|
41
43
|
*/
|
|
42
|
-
constructor({ library, tiles, config, loadCodec }) {
|
|
44
|
+
constructor({ library, tiles, config, loadCodec, cutlines }) {
|
|
43
45
|
this.#library = library;
|
|
44
46
|
this.#tiles = tiles;
|
|
45
47
|
this.#config = config;
|
|
46
48
|
this.#loadCodec = loadCodec;
|
|
49
|
+
this.#cutlines = cutlines;
|
|
47
50
|
}
|
|
48
51
|
|
|
49
52
|
/**
|
|
@@ -120,9 +123,15 @@ export class BakeManager {
|
|
|
120
123
|
const publishDir =
|
|
121
124
|
options.publishDir ?? (await this.#savePath(options)) ?? undefined;
|
|
122
125
|
|
|
123
|
-
//
|
|
124
|
-
//
|
|
125
|
-
|
|
126
|
+
// Both dated by default and both overridable, separately: the archive's
|
|
127
|
+
// name is what a map client shows, and the filename is what somebody finds
|
|
128
|
+
// on disk. Tying them together only means one of the two is wrong whenever
|
|
129
|
+
// they should differ.
|
|
130
|
+
const when = new Date();
|
|
131
|
+
const archiveName = bakedArchiveName(resolved, {
|
|
132
|
+
name: options.name,
|
|
133
|
+
when,
|
|
134
|
+
});
|
|
126
135
|
|
|
127
136
|
const job = {
|
|
128
137
|
stackId,
|
|
@@ -138,7 +147,7 @@ export class BakeManager {
|
|
|
138
147
|
finishedAt: null,
|
|
139
148
|
error: null,
|
|
140
149
|
infoHash: null,
|
|
141
|
-
name: bakedName(resolved, {
|
|
150
|
+
name: bakedName(resolved, { filename: options.filename, when }),
|
|
142
151
|
publishDir,
|
|
143
152
|
};
|
|
144
153
|
this.#jobs.set(stackId, job);
|
|
@@ -266,6 +275,7 @@ export class BakeManager {
|
|
|
266
275
|
tiles: this.#tiles,
|
|
267
276
|
codec,
|
|
268
277
|
pixels,
|
|
278
|
+
cutlines: this.#cutlines,
|
|
269
279
|
signal: job.controller.signal,
|
|
270
280
|
format,
|
|
271
281
|
}),
|
|
@@ -273,7 +283,10 @@ export class BakeManager {
|
|
|
273
283
|
pauseMs: this.#config.stacks?.bakePauseMs ?? 0,
|
|
274
284
|
metadata: {
|
|
275
285
|
name: job.archiveName,
|
|
276
|
-
|
|
286
|
+
// Only what was asked for. Falling back to the recipe's own
|
|
287
|
+
// description would fill in a field the dialog showed as empty, which
|
|
288
|
+
// is a worse surprise than having no description at all.
|
|
289
|
+
description: options.description,
|
|
277
290
|
attribution: resolved.stack.attribution,
|
|
278
291
|
encoding: resolved.stack.output?.encoding,
|
|
279
292
|
encodingFactors: resolved.stack.output,
|
package/src/bake.js
CHANGED
|
@@ -228,23 +228,45 @@ function stamp(when = new Date()) {
|
|
|
228
228
|
/**
|
|
229
229
|
* What to call the file a bake writes.
|
|
230
230
|
*
|
|
231
|
-
* Dated, because successive bakes of one stack are successive builds
|
|
232
|
-
* map and two files cannot share a path.
|
|
233
|
-
* see `bakedMetadata` -- so a rebuild keeps its identity the way every other
|
|
234
|
-
* rebuild here does.
|
|
231
|
+
* Dated by default, because successive bakes of one stack are successive builds
|
|
232
|
+
* of one map and two files cannot share a path.
|
|
235
233
|
*
|
|
236
|
-
*
|
|
237
|
-
*
|
|
238
|
-
*
|
|
234
|
+
* A caller may name it instead, and what they ask for is reduced to a single
|
|
235
|
+
* path segment before it is used. That is not politeness: this name is joined
|
|
236
|
+
* to a directory, and a filename is exactly the kind of field somebody puts a
|
|
237
|
+
* slash in.
|
|
239
238
|
* @param {object} resolved - The resolved stack.
|
|
240
|
-
* @param {object} [options] - `
|
|
241
|
-
* @returns {string} - A filename
|
|
239
|
+
* @param {object} [options] - `filename` to choose one outright, and `when`.
|
|
240
|
+
* @returns {string} - A filename, always ending `.pmtiles`.
|
|
242
241
|
*/
|
|
243
242
|
export function bakedName(resolved, options = {}) {
|
|
244
|
-
const
|
|
243
|
+
const requested = String(options.filename ?? '').trim();
|
|
244
|
+
if (requested) {
|
|
245
|
+
const stem = safeSegment(requested.replace(/\.pmtiles$/i, ''));
|
|
246
|
+
if (stem) return `${stem}.pmtiles`;
|
|
247
|
+
}
|
|
248
|
+
|
|
249
|
+
const title = resolved.stack.title ?? resolved.stack.id;
|
|
245
250
|
const slug = safeSegment(title) || 'stack';
|
|
246
|
-
|
|
247
|
-
|
|
251
|
+
return `${slug}-${stamp(options.when)}.pmtiles`;
|
|
252
|
+
}
|
|
253
|
+
|
|
254
|
+
/**
|
|
255
|
+
* What to call the archive itself.
|
|
256
|
+
*
|
|
257
|
+
* Dated too, and separately from the file. Nothing in this project looks an
|
|
258
|
+
* archive up by name -- `/latest/<category>/` follows a category and takes the
|
|
259
|
+
* newest by date -- so a dated name costs nothing and says which build you are
|
|
260
|
+
* looking at, which is the question somebody holding two of them has.
|
|
261
|
+
* @param {object} resolved - The resolved stack.
|
|
262
|
+
* @param {object} [options] - `name` to choose one outright, and `when`.
|
|
263
|
+
* @returns {string} - The name.
|
|
264
|
+
*/
|
|
265
|
+
export function bakedArchiveName(resolved, options = {}) {
|
|
266
|
+
const explicit = String(options.name ?? '').trim();
|
|
267
|
+
if (explicit) return explicit;
|
|
268
|
+
const title = resolved.stack.title ?? resolved.stack.id;
|
|
269
|
+
return `${title} ${stamp(options.when)}`;
|
|
248
270
|
}
|
|
249
271
|
|
|
250
272
|
/**
|
|
@@ -367,7 +389,7 @@ export function tileTypeFor(format) {
|
|
|
367
389
|
* @returns {Function} - `(z, x, y) => Promise<Buffer|null>`.
|
|
368
390
|
*/
|
|
369
391
|
export function mergeTileFor(options) {
|
|
370
|
-
const { resolved, tiles, codec, signal, pixels } = options;
|
|
392
|
+
const { resolved, tiles, codec, signal, pixels, cutlines } = options;
|
|
371
393
|
const format = options.format ?? outputFormat(resolved);
|
|
372
394
|
const size = options.size ?? outputSize(resolved.stack);
|
|
373
395
|
|
|
@@ -384,6 +406,7 @@ export function mergeTileFor(options) {
|
|
|
384
406
|
size,
|
|
385
407
|
format,
|
|
386
408
|
pixels,
|
|
409
|
+
cutlines,
|
|
387
410
|
});
|
|
388
411
|
|
|
389
412
|
// A required source that cannot be read stops the job. Baking around it
|