pmtiles-swarm 0.76.0 → 0.78.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +60 -0
- package/README.md +2 -0
- package/docs/internals.md +58 -0
- package/docs/tile-stacks.md +100 -37
- package/package.json +1 -1
- package/src/api.js +33 -0
- package/src/bake-jobs.js +28 -0
- package/src/cutline.js +25 -0
- package/src/cutlines.js +51 -4
- package/src/stack-cache.js +19 -0
- package/src/stack-tile.js +13 -7
- package/src/stacks.js +22 -3
- package/src/storage.js +283 -0
- package/src/traffic-stats.js +12 -0
- package/src/web/index.html +149 -15
- package/tools/coast-step.mjs +200 -0
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,66 @@
|
|
|
7
7
|
### 🐞 Bug fixes
|
|
8
8
|
- _...Add new stuff here..._
|
|
9
9
|
|
|
10
|
+
## 0.78.0
|
|
11
|
+
### ✨ Features and improvements
|
|
12
|
+
- **Somewhere to clear the caches and the files nothing is waiting for.** A merged tile cache with
|
|
13
|
+
no way to empty it is a directory somebody eventually finds by hand, and a temporary file a crash
|
|
14
|
+
left behind is one nobody finds at all. Settings has a **Storage** tab now: what this node is
|
|
15
|
+
holding, what each thing costs to lose, and a button per row.
|
|
16
|
+
|
|
17
|
+
Five things — merged stack tiles, left-over temporary files, stopped exports, traffic history and
|
|
18
|
+
the tile counters. `GET /api/storage` reports them, `DELETE /api/storage/:what` lets go of one.
|
|
19
|
+
Everything on the list is derived and can be rebuilt, which is what makes a button reasonable:
|
|
20
|
+
none of it asks whether you meant it, because none of it is the only copy of anything.
|
|
21
|
+
|
|
22
|
+
The archives are deliberately not on it, and neither is the resume data beside them. Both look
|
|
23
|
+
like housekeeping and neither is: retiring an archive is a decision made from its own panel with
|
|
24
|
+
what it seeds in view, and resume data thrown away is a rehash of every byte on disk.
|
|
25
|
+
|
|
26
|
+
A sweep picks files by name and by age together, because either alone is wrong: `*.tmp` and
|
|
27
|
+
`pmtiles-write-*`, untouched for an hour. Every write that uses one renames within milliseconds,
|
|
28
|
+
so the margin is not for slowness — it is because this runs while the node is serving, and a sweep
|
|
29
|
+
with none at all could take the file a catalog write is halfway through renaming into place.
|
|
30
|
+
`torrents-data` is skipped outright, being terabytes of payload with no working files in it.
|
|
31
|
+
|
|
32
|
+
Stopped exports are read off the disk rather than from what the process remembers, so one left by
|
|
33
|
+
a stack somebody has since deleted is found as well. A running export is left alone: removing the
|
|
34
|
+
directory under a running merge would have it fail on its next write, reporting a disk problem for
|
|
35
|
+
something somebody chose. This is also the first way to discard one without finding the directory
|
|
36
|
+
by hand — the API for it shipped in 0.76.0 with nothing calling it.
|
|
37
|
+
|
|
38
|
+
### 🐞 Bug fixes
|
|
39
|
+
- _...Add new stuff here..._
|
|
40
|
+
|
|
41
|
+
## 0.77.0
|
|
42
|
+
### ✨ Features and improvements
|
|
43
|
+
- **A fade can be written in metres of ground.** `featherMetres` on a source says how far to blend
|
|
44
|
+
it in as a distance, and the merge works out the pixels for each tile it builds -
|
|
45
|
+
`40075016.686 x cos(latitude) / 2^zoom / tileSize` of them. What a fade has to hide is two sources
|
|
46
|
+
disagreeing about the height of the same ground, which is a fixed number of metres, while a fade
|
|
47
|
+
in pixels is a different distance at every zoom.
|
|
48
|
+
|
|
49
|
+
It matters because a hillshade reads slope rather than height. `feather: 8` over a 7 m
|
|
50
|
+
disagreement is a gradient of 0.08 at z12 and 1.28 at z16 - invisible at one end and a saturated
|
|
51
|
+
band at the other, wider than the cliff it replaced and no less visible. The same 50 m holds 0.14
|
|
52
|
+
at every zoom, which is ordinary hillside. Below the zoom where the fade is a pixel wide it rounds
|
|
53
|
+
to nothing, and it is capped at a quarter of the tile - 128 pixels on a 512px grid, which is where
|
|
54
|
+
the old 64 came from. `featherMeters` is read as well, and the console's fade field now takes a
|
|
55
|
+
unit rather than always meaning pixels.
|
|
56
|
+
|
|
57
|
+
- **A tool that measures the step a fade has to hide.** `tools/coast-step.mjs` reports the height
|
|
58
|
+
difference where one source hands over to another, and says which kind of disagreement it is.
|
|
59
|
+
Steps that cluster are a vertical datum offset, which one `heightAdjustment` corrects. Steps that
|
|
60
|
+
scatter are a coarse source averaging land and water together in every cell that straddles a
|
|
61
|
+
coast - worst at the shore, gone offshore - which no single number corrects and a fade can only
|
|
62
|
+
hide.
|
|
63
|
+
|
|
64
|
+
### 🐞 Bug fixes
|
|
65
|
+
- **A mask range was not an edge to fade at.** A source whose only mask was a `maskRange` was told
|
|
66
|
+
its `feather` had nothing to act on, and refused - while the merge had been fading exactly that
|
|
67
|
+
edge all along. The recipe validation and the console's own warning both listed `maskValues` and
|
|
68
|
+
`maskColors` and neither had been taught about the band.
|
|
69
|
+
|
|
10
70
|
## 0.76.0
|
|
11
71
|
### ✨ Features and improvements
|
|
12
72
|
- **A stopped export stays stopped.** Stopping one left a checkpoint, and a checkpoint says what was
|
package/README.md
CHANGED
|
@@ -786,6 +786,8 @@ which the endpoint answers 501.
|
|
|
786
786
|
| `GET` `POST` `DELETE` | `/api/tokens`, `/api/tokens/:id` | Mint, list and revoke access tokens |
|
|
787
787
|
| `GET` `POST` | `/api/restart` | What a restart would do, and doing it |
|
|
788
788
|
| `GET` `DELETE` | `/api/stats` | What this node has served — per-archive counters and the last N requests; `DELETE` clears them |
|
|
789
|
+
| `GET` | `/api/storage` | What this node is holding that it could let go of — merged tiles, left-over temporary files, stopped exports, traffic history, tile counters |
|
|
790
|
+
| `DELETE` | `/api/storage/:what` | Lets go of one of them. Everything listed is derived and can be rebuilt; the archives are not on the list |
|
|
789
791
|
| `GET` | `/api/traffic` | Upload and download speed per archive over time, from `stats.db`; `infoHash`, `hours` and `buckets` narrow it, and `totals` ranks archives by bytes moved |
|
|
790
792
|
| `GET` `PATCH` | `/api/config` | Read and change settings |
|
|
791
793
|
| `POST` | `/api/login`, `/api/logout` | Console sign-in |
|
package/docs/internals.md
CHANGED
|
@@ -33,6 +33,7 @@ Operator-facing documentation is elsewhere — see [publishing](publishing.md),
|
|
|
33
33
|
- [Retiring old builds](#retiring-old-builds)
|
|
34
34
|
- [Prewarming a freshly joined archive](#prewarming-a-freshly-joined-archive)
|
|
35
35
|
- [Named save locations](#named-save-locations)
|
|
36
|
+
- [Storage](#storage)
|
|
36
37
|
|
|
37
38
|
## Marking incomplete archives
|
|
38
39
|
|
|
@@ -884,3 +885,60 @@ moving several hundred gigabytes is not something a settings screen should do as
|
|
|
884
885
|
a side effect. A move checks free space before the engine is disturbed, since
|
|
885
886
|
running out halfway through costs an hour, a partial file to clean up, and an
|
|
886
887
|
archive that has to be put back.
|
|
888
|
+
|
|
889
|
+
## Storage
|
|
890
|
+
|
|
891
|
+
A merged tile cache with no way to empty it is a directory somebody eventually
|
|
892
|
+
finds by hand, and a temporary file a crash left behind is one nobody finds at
|
|
893
|
+
all. `GET /api/storage` reports what a node is holding that it could let go of;
|
|
894
|
+
`DELETE /api/storage/:what` lets go of one of them. The console draws it as the
|
|
895
|
+
Storage tab under Settings.
|
|
896
|
+
|
|
897
|
+
Five things, and what each costs to lose:
|
|
898
|
+
|
|
899
|
+
| What | Where | Clearing it costs |
|
|
900
|
+
| ------------------------- | ------------------ | -------------------------------------------------------------- |
|
|
901
|
+
| Merged stack tiles | `stack-cache/` | The merge again: an archive read per source, and a decode each |
|
|
902
|
+
| Left-over temporary files | the data directory | Nothing — nobody is waiting for a file a crashed write left |
|
|
903
|
+
| Stopped exports | the archive's disk | The hours already spent, which is what resuming would use |
|
|
904
|
+
| Traffic history | `stats.db` | The graphs go back to empty |
|
|
905
|
+
| Tile counters | memory | The count starts again; frees nothing |
|
|
906
|
+
|
|
907
|
+
Everything on that list is derived and can be rebuilt, which is what makes a
|
|
908
|
+
button reasonable — none of it asks whether the operator meant it, because none
|
|
909
|
+
of it is the only copy of anything.
|
|
910
|
+
|
|
911
|
+
### What is deliberately not on it
|
|
912
|
+
|
|
913
|
+
**The archives**, and the resume data beside them. Both look like housekeeping
|
|
914
|
+
and neither is. An archive is the data this node exists to serve, and retiring
|
|
915
|
+
one is a decision made from its own panel with what it seeds in view. Resume
|
|
916
|
+
data thrown away is a rehash of every byte on disk — and worse than useless as
|
|
917
|
+
a repair, since a _stale partial_ resume file is the thing that cancels seed
|
|
918
|
+
mode, which is a diagnosis rather than a sweep.
|
|
919
|
+
|
|
920
|
+
### Which files a sweep is willing to touch
|
|
921
|
+
|
|
922
|
+
By name and by age together, because either alone is wrong: the name says what
|
|
923
|
+
a file was for and the age says whether anything still cares.
|
|
924
|
+
|
|
925
|
+
The names are `*.tmp` and `pmtiles-write-*`, which is what the write-then-rename
|
|
926
|
+
in `catalog.js`, `stacks.js`, `dht-state.js`, `stack-cache.js` and
|
|
927
|
+
`pmtiles-write.js` leaves behind when a process stops between the two steps. The
|
|
928
|
+
age is an hour. Every one of those renames happens within milliseconds, so an
|
|
929
|
+
hour is far past generous — the margin is not for slowness, it is because this
|
|
930
|
+
runs while the node is serving and a sweep with no margin could delete the file
|
|
931
|
+
a catalog write is halfway through renaming into place.
|
|
932
|
+
|
|
933
|
+
`torrents-data/` is skipped outright. It is terabytes of payload, nothing in it
|
|
934
|
+
is a working file, and a sweep that walks it is a sweep that spends minutes
|
|
935
|
+
finding nothing.
|
|
936
|
+
|
|
937
|
+
### The bake working directories are read from disk
|
|
938
|
+
|
|
939
|
+
Not from what the process remembers. A working directory outlives the run that
|
|
940
|
+
made it, and one belonging to a stack somebody has since deleted is exactly the
|
|
941
|
+
kind nobody thinks to look for. Whether an export is _running_ is memory's to
|
|
942
|
+
say, and that is the only thing that decides whether it may be discarded —
|
|
943
|
+
removing the directory under a running merge would have it fail on its next
|
|
944
|
+
write, reporting a disk problem for something somebody chose.
|
package/docs/tile-stacks.md
CHANGED
|
@@ -50,6 +50,8 @@ of its parts.
|
|
|
50
50
|
- [Starting one, and watching it](#starting-one-and-watching-it)
|
|
51
51
|
- [What exists now](#what-exists-now)
|
|
52
52
|
- [Clipping a source to a shape](#clipping-a-source-to-a-shape)
|
|
53
|
+
- [What a mask has to match](#what-a-mask-has-to-match)
|
|
54
|
+
- [Feathering a seam](#feathering-a-seam)
|
|
53
55
|
- [Finding a stack](#finding-a-stack)
|
|
54
56
|
- [Syncing a stack to another node](#syncing-a-stack-to-another-node)
|
|
55
57
|
- [What the offline merge got wrong](#what-the-offline-merge-got-wrong)
|
|
@@ -233,19 +235,20 @@ it differs from the snake_case rio-rgbify-merge uses.
|
|
|
233
235
|
|
|
234
236
|
### Source fields
|
|
235
237
|
|
|
236
|
-
| Field | Meaning
|
|
237
|
-
| ---------------------- |
|
|
238
|
-
| `category` | Resolve to the newest build in this category.
|
|
239
|
-
| `archive` | Or pin one infohash. Exactly one of the two.
|
|
240
|
-
| `required` | A tile fails rather than being served without this source. Defaults true for the bottom-most, false above.
|
|
241
|
-
| `encoding` | `mapbox` or `terrarium`. Elevation space only.
|
|
242
|
-
| `baseVal` / `interval` | Mapbox decode offset and step. Default `-10000` and `0.1`.
|
|
243
|
-
| `maskValues` | Decoded heights meaning "no data here". Elevation space only.
|
|
244
|
-
| `maskRange` | `[low, high]` in metres, or a list of them. Everything inside is nodata. Elevation space only.
|
|
245
|
-
| `heightAdjustment` | Metres, added **after** masking. Elevation space only.
|
|
246
|
-
| `feather` | Pixels to fade in over
|
|
247
|
-
| `
|
|
248
|
-
| `
|
|
238
|
+
| Field | Meaning |
|
|
239
|
+
| ---------------------- | ------------------------------------------------------------------------------------------------------------ |
|
|
240
|
+
| `category` | Resolve to the newest build in this category. |
|
|
241
|
+
| `archive` | Or pin one infohash. Exactly one of the two. |
|
|
242
|
+
| `required` | A tile fails rather than being served without this source. Defaults true for the bottom-most, false above. |
|
|
243
|
+
| `encoding` | `mapbox` or `terrarium`. Elevation space only. |
|
|
244
|
+
| `baseVal` / `interval` | Mapbox decode offset and step. Default `-10000` and `0.1`. |
|
|
245
|
+
| `maskValues` | Decoded heights meaning "no data here". Elevation space only. |
|
|
246
|
+
| `maskRange` | `[low, high]` in metres, or a list of them. Everything inside is nodata. Elevation space only. |
|
|
247
|
+
| `heightAdjustment` | Metres, added **after** masking. Elevation space only. |
|
|
248
|
+
| `feather` | Pixels to fade in over wherever the source stops: a `cutline`, `bounds`, or the holes a mask leaves. Max 64. |
|
|
249
|
+
| `featherMetres` | The same fade written as metres of ground, worked out per tile. Usually the one to reach for. |
|
|
250
|
+
| `opacity` | `0`–`1`, scales the source alpha. RGBA space only. |
|
|
251
|
+
| `blend` | `normal`, `multiply`, `screen`, `overlay`, `darken`, `lighten`. RGBA space only. |
|
|
249
252
|
|
|
250
253
|
`attribution` is not optional in practice. A stack is a derived work of every
|
|
251
254
|
source in it, and the thing that reliably gets lost when tiles are combined is
|
|
@@ -1085,8 +1088,8 @@ standing clear of their neighbours, which is the shape this makes.
|
|
|
1085
1088
|
|
|
1086
1089
|
## Feathering a seam
|
|
1087
1090
|
|
|
1088
|
-
_Built for a cutline
|
|
1089
|
-
the last section says
|
|
1091
|
+
_Built for a cutline, for `bounds`, and for the holes a mask leaves. A source
|
|
1092
|
+
that vanishes at a tile edge is still open, and the last section says why._
|
|
1090
1093
|
|
|
1091
1094
|
Smoothing hides the terracing **inside** an upscaled area. It does nothing about
|
|
1092
1095
|
the artefact the original discussion actually named — "artefacts at tiles and
|
|
@@ -1236,18 +1239,40 @@ lookup rather than a decode — and it is not done.
|
|
|
1236
1239
|
|
|
1237
1240
|
`feather` on a source, in pixels, `0` and absent meaning the edge is a switch as
|
|
1238
1241
|
before. It fades that source in over that many pixels measured inward from
|
|
1239
|
-
wherever it stops — the holes `maskValues` and `maskColors` leave,
|
|
1240
|
-
of a `cutline` or `bounds`:
|
|
1242
|
+
wherever it stops — the holes `maskValues`, `maskRange` and `maskColors` leave,
|
|
1243
|
+
and the edge of a `cutline` or `bounds`:
|
|
1241
1244
|
|
|
1242
1245
|
```json
|
|
1243
1246
|
{ "archive": "swissalti", "maskValues": [0], "feather": 16 }
|
|
1244
1247
|
```
|
|
1245
1248
|
|
|
1246
|
-
The step left at the seam is the height difference divided by the feather
|
|
1247
|
-
|
|
1248
|
-
|
|
1249
|
-
|
|
1250
|
-
|
|
1249
|
+
The step left at the seam is the height difference divided by the feather: two
|
|
1250
|
+
sources 40 m apart at the border, faded over 16 pixels, step 2.5 m a pixel
|
|
1251
|
+
instead of 40 m at once. Sixty-four is the most it takes — past that the ramp
|
|
1252
|
+
does not reach full weight anywhere inside a 256px tile, and the source is being
|
|
1253
|
+
turned down rather than blended in.
|
|
1254
|
+
|
|
1255
|
+
**A smaller step is not the same as an invisible one**, and that is the thing to
|
|
1256
|
+
know before picking a number. A hillshade does not read height, it reads slope:
|
|
1257
|
+
the drop divided by the ground underneath it. What decides whether a seam
|
|
1258
|
+
disappears is therefore metres per metre rather than metres per pixel — and a
|
|
1259
|
+
pixel is a different number of metres at every zoom. The same `feather: 8`, over
|
|
1260
|
+
the same 7 m disagreement, at 55°N:
|
|
1261
|
+
|
|
1262
|
+
| zoom | m/pixel | gradient |
|
|
1263
|
+
| ---- | ------- | -------- |
|
|
1264
|
+
| z12 | 11.0 | 0.08 |
|
|
1265
|
+
| z13 | 5.5 | 0.16 |
|
|
1266
|
+
| z14 | 2.7 | 0.32 |
|
|
1267
|
+
| z15 | 1.4 | 0.64 |
|
|
1268
|
+
| z16 | 0.7 | 1.28 |
|
|
1269
|
+
|
|
1270
|
+
Natural terrain is rarely over about 0.5 and a hillshade saturates around there,
|
|
1271
|
+
so a fade that vanishes at z12 is a bright band by z15 — wider than the cliff it
|
|
1272
|
+
replaced and no less visible. Widening it does help at any one zoom, since the
|
|
1273
|
+
gradient is the drop over the whole fade; the difficulty is that the number
|
|
1274
|
+
which works at z16 is eight times the one that works at z13, and a recipe has
|
|
1275
|
+
one number.
|
|
1251
1276
|
|
|
1252
1277
|
Three things fall out of the implementation and are worth stating.
|
|
1253
1278
|
|
|
@@ -1268,22 +1293,60 @@ envelope — linear in the tile's width, and chosen over a chamfer approximation
|
|
|
1268
1293
|
because a chamfer's error is largest on diagonals and a national boundary is
|
|
1269
1294
|
mostly diagonals.
|
|
1270
1295
|
|
|
1271
|
-
###
|
|
1296
|
+
### A fade in metres
|
|
1297
|
+
|
|
1298
|
+
`featherMetres` says the same thing as a distance on the ground, and the merge
|
|
1299
|
+
works out the pixels for each tile it builds:
|
|
1272
1300
|
|
|
1273
|
-
|
|
1274
|
-
|
|
1275
|
-
|
|
1276
|
-
|
|
1277
|
-
|
|
1278
|
-
on
|
|
1279
|
-
|
|
1280
|
-
|
|
1281
|
-
|
|
1282
|
-
|
|
1283
|
-
|
|
1284
|
-
|
|
1285
|
-
|
|
1286
|
-
|
|
1301
|
+
```json
|
|
1302
|
+
{ "category": "planet-bathymetry", "maskRange": [-1, 0], "featherMetres": 50 }
|
|
1303
|
+
```
|
|
1304
|
+
|
|
1305
|
+
That is the source **on top** — the one whose mask leaves the hole. A weight
|
|
1306
|
+
says how much of what is underneath shows through, so it goes on the layer with
|
|
1307
|
+
the edge and not on the one filling the hole: a fade on the bathymetry
|
|
1308
|
+
underneath is read by nothing, because `paintHeights` takes the first layer that
|
|
1309
|
+
contributes whole and only consults a weight from the second onward. See "A
|
|
1310
|
+
feathered layer over a hole stands alone" above, which is the same rule seen
|
|
1311
|
+
from the other side.
|
|
1312
|
+
|
|
1313
|
+
Web Mercator makes that arithmetic rather than a lookup: one pixel covers
|
|
1314
|
+
`40075016.686 × cos(latitude) / 2^zoom / tileSize` metres, so the conversion
|
|
1315
|
+
needs a multiply and the coordinates of the tile being built. `featherMeters` is
|
|
1316
|
+
read as well, because a key that quietly does nothing when spelled the other way
|
|
1317
|
+
is the failure this feature is most prone to.
|
|
1318
|
+
|
|
1319
|
+
Fifty metres, over the same 7 m disagreement, at 55°N on 512px tiles:
|
|
1320
|
+
|
|
1321
|
+
| zoom | m/pixel | pixels | ground | gradient |
|
|
1322
|
+
| ---- | ------- | ------ | ------ | -------- |
|
|
1323
|
+
| z12 | 11.0 | 5 | 55 m | 0.13 |
|
|
1324
|
+
| z13 | 5.5 | 9 | 49 m | 0.14 |
|
|
1325
|
+
| z14 | 2.7 | 18 | 49 m | 0.14 |
|
|
1326
|
+
| z15 | 1.4 | 36 | 49 m | 0.14 |
|
|
1327
|
+
| z16 | 0.7 | 73 | 50 m | 0.14 |
|
|
1328
|
+
|
|
1329
|
+
One number, the same slope at every zoom, and 0.14 is ordinary hillside rather
|
|
1330
|
+
than an edge.
|
|
1331
|
+
|
|
1332
|
+
It stops being exact at both ends, benignly. Below the zoom where the fade is a
|
|
1333
|
+
pixel wide it rounds to nothing — 50 m at z8 is a sixth of a pixel, and the whole
|
|
1334
|
+
coastline is inside one pixel there anyway. Above a quarter of the tile it is
|
|
1335
|
+
capped, which is 128 pixels on a 512px grid and is where the old 64 came from:
|
|
1336
|
+
past that the ramp reaches full weight nowhere inside the tile. At 55°N the cap
|
|
1337
|
+
first binds at z17.
|
|
1338
|
+
|
|
1339
|
+
**Picking the number.** It is the disagreement being hidden that sets it, not the
|
|
1340
|
+
coastline being drawn across. `tools/coast-step.mjs` measures the first against a
|
|
1341
|
+
real tile: it reports the step where one source hands over to the other, and
|
|
1342
|
+
whether those steps cluster — in which case they are a datum offset and
|
|
1343
|
+
`heightAdjustment` is the honest fix — or scatter, in which case nothing corrects
|
|
1344
|
+
them and a fade is for hiding what cannot be corrected. Divide the step by the
|
|
1345
|
+
gradient wanted: 7 m at 0.15 is about 50 m, and the same step at 0.05 would need
|
|
1346
|
+
140 m, which is wide enough to start flattening ground either side of the coast
|
|
1347
|
+
that was never wrong.
|
|
1348
|
+
|
|
1349
|
+
### What it does not do yet
|
|
1287
1350
|
|
|
1288
1351
|
**A source that vanishes at a tile edge.** A sparse archive simply has no tile
|
|
1289
1352
|
outside its extent, so the seam lands exactly on a tile boundary — the most
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pmtiles-swarm",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.78.0",
|
|
4
4
|
"description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "src/index.js",
|
package/src/api.js
CHANGED
|
@@ -37,6 +37,7 @@ import { SUMMARY_VERSION } from './pmtiles-probe.js';
|
|
|
37
37
|
import { TileReadError } from './tiles.js';
|
|
38
38
|
import { loadCodec } from './codec.js';
|
|
39
39
|
import { answerStackTile, outputFormat, outputSize } from './stack-tile.js';
|
|
40
|
+
import { clearStorage, storageReport } from './storage.js';
|
|
40
41
|
import {
|
|
41
42
|
isPinned,
|
|
42
43
|
needsCodec,
|
|
@@ -790,6 +791,38 @@ export function createApp({
|
|
|
790
791
|
}),
|
|
791
792
|
);
|
|
792
793
|
|
|
794
|
+
// What this node is holding that it could let go of. Separate from the
|
|
795
|
+
// settings it sits beside in the console: a setting says what to do next and
|
|
796
|
+
// this says what to do about what was already done.
|
|
797
|
+
app.get(
|
|
798
|
+
'/api/storage',
|
|
799
|
+
route(async (_req, res) => {
|
|
800
|
+
res.setHeader('cache-control', 'no-store');
|
|
801
|
+
res.json(
|
|
802
|
+
await storageReport({ config, stackCache, stats, traffic, bakes }),
|
|
803
|
+
);
|
|
804
|
+
}),
|
|
805
|
+
);
|
|
806
|
+
|
|
807
|
+
app.delete(
|
|
808
|
+
'/api/storage/:what',
|
|
809
|
+
route(async (req, res) => {
|
|
810
|
+
const gone = await clearStorage(req.params.what, {
|
|
811
|
+
config,
|
|
812
|
+
stackCache,
|
|
813
|
+
stats,
|
|
814
|
+
traffic,
|
|
815
|
+
bakes,
|
|
816
|
+
});
|
|
817
|
+
if (!gone) {
|
|
818
|
+
return res
|
|
819
|
+
.status(404)
|
|
820
|
+
.json({ error: 'nothing here is called that, or it is turned off' });
|
|
821
|
+
}
|
|
822
|
+
return res.json(gone);
|
|
823
|
+
}),
|
|
824
|
+
);
|
|
825
|
+
|
|
793
826
|
// Settings. Everything read per request takes effect immediately because the
|
|
794
827
|
// running config object is the one being mutated; everything bound at startup
|
|
795
828
|
// is written to the file and reported back as needing a restart, rather than
|
package/src/bake-jobs.js
CHANGED
|
@@ -205,6 +205,34 @@ export class BakeManager {
|
|
|
205
205
|
return true;
|
|
206
206
|
}
|
|
207
207
|
|
|
208
|
+
/**
|
|
209
|
+
* Every export with work still on disk, and where it is.
|
|
210
|
+
*
|
|
211
|
+
* Read from the filesystem rather than from what this process remembers: a
|
|
212
|
+
* working directory outlives the run that made it, and one belonging to a
|
|
213
|
+
* stack somebody has since deleted is exactly the kind nobody thinks to look
|
|
214
|
+
* for. Whether it is running is what decides if it may be discarded, and
|
|
215
|
+
* that is memory's to say.
|
|
216
|
+
* @returns {Promise<object[]>} - `{stackId, directory, running, stopped}`.
|
|
217
|
+
*/
|
|
218
|
+
async heldWork() {
|
|
219
|
+
const found = [];
|
|
220
|
+
for (const root of this.#workRoots()) {
|
|
221
|
+
const base = path.join(root, WORK_DIR);
|
|
222
|
+
const names = await fs.readdir(base).catch(() => []);
|
|
223
|
+
for (const stackId of names) {
|
|
224
|
+
const job = this.#jobs.get(stackId);
|
|
225
|
+
found.push({
|
|
226
|
+
stackId,
|
|
227
|
+
directory: path.join(base, stackId),
|
|
228
|
+
running: Boolean(job && !job.finishedAt),
|
|
229
|
+
stopped: this.#held.has(stackId),
|
|
230
|
+
});
|
|
231
|
+
}
|
|
232
|
+
}
|
|
233
|
+
return found;
|
|
234
|
+
}
|
|
235
|
+
|
|
208
236
|
/**
|
|
209
237
|
* Throws away what a stopped export had done.
|
|
210
238
|
*
|
package/src/cutline.js
CHANGED
|
@@ -45,6 +45,31 @@ export function worldY(lat) {
|
|
|
45
45
|
);
|
|
46
46
|
}
|
|
47
47
|
|
|
48
|
+
/** The world across the equator, in metres: what a zoom 0 tile spans. */
|
|
49
|
+
const EQUATOR = 40075016.686;
|
|
50
|
+
|
|
51
|
+
/**
|
|
52
|
+
* How much ground one pixel of a tile covers, at that tile's latitude.
|
|
53
|
+
*
|
|
54
|
+
* Web Mercator holds a pixel to a fixed fraction of the world, so the ground
|
|
55
|
+
* under it shrinks toward the poles and halves at every zoom. That is why a
|
|
56
|
+
* fade written in pixels is a different width of ground everywhere it is used,
|
|
57
|
+
* and this is the conversion that lets one written in metres mean the same
|
|
58
|
+
* thing. Taken at the middle of the tile: the scale changes across it, but a
|
|
59
|
+
* tile is a small piece of the world at any zoom where a fade is more than a
|
|
60
|
+
* pixel wide.
|
|
61
|
+
* @param {number} z - Zoom.
|
|
62
|
+
* @param {number} y - Row.
|
|
63
|
+
* @param {number} size - Pixels per side of the tile.
|
|
64
|
+
* @returns {number} - Metres per pixel.
|
|
65
|
+
*/
|
|
66
|
+
export function metresPerPixel(z, y, size) {
|
|
67
|
+
const tiles = 2 ** z;
|
|
68
|
+
const middle = (y + 0.5) / tiles;
|
|
69
|
+
const latitude = Math.atan(Math.sinh(Math.PI * (1 - 2 * middle)));
|
|
70
|
+
return (EQUATOR * Math.cos(latitude)) / (tiles * size);
|
|
71
|
+
}
|
|
72
|
+
|
|
48
73
|
/**
|
|
49
74
|
* Reads the rings out of GeoJSON, whatever shape it arrived in.
|
|
50
75
|
*
|
package/src/cutlines.js
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
import fs from 'node:fs/promises';
|
|
2
2
|
import path from 'node:path';
|
|
3
|
-
import { fromBounds, fromGeoJSON } from './cutline.js';
|
|
3
|
+
import { fromBounds, fromGeoJSON, metresPerPixel } from './cutline.js';
|
|
4
4
|
|
|
5
5
|
/**
|
|
6
6
|
* The cutlines a node has, by name.
|
|
@@ -101,16 +101,63 @@ export class CutlineStore {
|
|
|
101
101
|
*/
|
|
102
102
|
export const MAX_FEATHER = 64;
|
|
103
103
|
|
|
104
|
+
/** Past this a fade in metres is a typo rather than a distance. */
|
|
105
|
+
export const MAX_FEATHER_METRES = 100000;
|
|
106
|
+
|
|
107
|
+
/**
|
|
108
|
+
* The widest fade a grid of this size will take.
|
|
109
|
+
*
|
|
110
|
+
* A quarter of the tile, which is where 64 came from and what it still means on
|
|
111
|
+
* a 512px one: past that the ramp reaches full weight nowhere inside the tile,
|
|
112
|
+
* and the source is being turned down rather than blended in. It binds on a
|
|
113
|
+
* fade in metres, which asks for more pixels at every zoom.
|
|
114
|
+
* @param {number} size - Pixels per side of the grid the ramp runs on.
|
|
115
|
+
* @returns {number} - The most pixels a fade may be.
|
|
116
|
+
*/
|
|
117
|
+
function capFor(size) {
|
|
118
|
+
return Math.max(MAX_FEATHER, Math.round((size ?? 0) / 4));
|
|
119
|
+
}
|
|
120
|
+
|
|
104
121
|
/**
|
|
105
|
-
* How far a source fades in
|
|
122
|
+
* How far a source fades in, as metres of ground, if it is written that way.
|
|
123
|
+
*
|
|
124
|
+
* Both spellings are read. A recipe key that does nothing when it is spelled
|
|
125
|
+
* the other way is the failure this whole feature is most prone to, and the
|
|
126
|
+
* prose here says metres while half the world's config files say meters.
|
|
127
|
+
* @param {object} recipe - One source out of a recipe.
|
|
128
|
+
* @returns {number} - Metres, 0 when the fade is not written in them.
|
|
129
|
+
*/
|
|
130
|
+
export function featherMetresFor(recipe) {
|
|
131
|
+
const asked = Number(recipe?.featherMetres ?? recipe?.featherMeters);
|
|
132
|
+
return Number.isFinite(asked) && asked > 0 ? asked : 0;
|
|
133
|
+
}
|
|
134
|
+
|
|
135
|
+
/**
|
|
136
|
+
* How far a source fades in at the edge of its shape, in this tile's pixels.
|
|
106
137
|
*
|
|
107
138
|
* Zero is the old behaviour and the default: the shape is a switch, and a pixel
|
|
108
139
|
* is either the source or what is underneath it. Anything more makes it a ramp.
|
|
109
|
-
*
|
|
140
|
+
*
|
|
141
|
+
* A fade in metres is converted here, which is why this wants the tile. What
|
|
142
|
+
* the fade has to hide is two sources disagreeing about the height of the same
|
|
143
|
+
* ground, which is a fixed number of metres -- so the same number of pixels is
|
|
144
|
+
* too wide at one zoom and too steep at the next. See docs/tile-stacks.md --
|
|
145
|
+
* "Feathering a seam".
|
|
110
146
|
* @param {object} recipe - One source out of a recipe.
|
|
147
|
+
* @param {object} [tile] - `{z, y, size}` of the tile being built.
|
|
111
148
|
* @returns {number} - Pixels, 0 when it does not fade.
|
|
112
149
|
*/
|
|
113
|
-
export function featherFor(recipe) {
|
|
150
|
+
export function featherFor(recipe, tile) {
|
|
151
|
+
const size = tile?.size > 0 ? tile.size : 256;
|
|
152
|
+
const metres = featherMetresFor(recipe);
|
|
153
|
+
if (metres > 0) {
|
|
154
|
+
// With no tile there is no scale to convert against, and a fade that
|
|
155
|
+
// guessed one would be a different width from the tiles beside it.
|
|
156
|
+
if (!Number.isFinite(tile?.z)) return 0;
|
|
157
|
+
const perPixel = metresPerPixel(tile.z, tile.y ?? 0, size);
|
|
158
|
+
return Math.min(Math.round(metres / perPixel), capFor(size));
|
|
159
|
+
}
|
|
160
|
+
|
|
114
161
|
const asked = Number(recipe?.feather);
|
|
115
162
|
if (!Number.isFinite(asked) || asked <= 0) return 0;
|
|
116
163
|
return Math.min(Math.round(asked), MAX_FEATHER);
|
package/src/stack-cache.js
CHANGED
|
@@ -207,6 +207,25 @@ export class StackCache {
|
|
|
207
207
|
}
|
|
208
208
|
}
|
|
209
209
|
|
|
210
|
+
/**
|
|
211
|
+
* Throws away every tile it is holding.
|
|
212
|
+
*
|
|
213
|
+
* The hit and miss counters are left alone: they say what the cache has been
|
|
214
|
+
* doing since the process started, which emptying it does not unmake. The
|
|
215
|
+
* files go one at a time and forgivingly -- one that vanished underneath is
|
|
216
|
+
* one fewer to remove.
|
|
217
|
+
* @returns {Promise<number>} - How many tiles went.
|
|
218
|
+
*/
|
|
219
|
+
async clear() {
|
|
220
|
+
const keys = [...this.#entries.keys()];
|
|
221
|
+
this.#entries.clear();
|
|
222
|
+
this.#bytes = 0;
|
|
223
|
+
for (const key of keys) {
|
|
224
|
+
await fs.rm(this.#pathFor(key), { force: true }).catch(() => {});
|
|
225
|
+
}
|
|
226
|
+
return keys.length;
|
|
227
|
+
}
|
|
228
|
+
|
|
210
229
|
/**
|
|
211
230
|
* What the cache is holding, for the console and for tests.
|
|
212
231
|
* @returns {object} - entries, bytes, maxBytes, hits, misses.
|
package/src/stack-tile.js
CHANGED
|
@@ -151,7 +151,7 @@ export function clipsFor(resolved, cutlines, z, x, y, size = 256) {
|
|
|
151
151
|
// The feather reaches inward from the edge, so a tile wholly inside but
|
|
152
152
|
// near it still has a ramp across part of itself and cannot take the cheap
|
|
153
153
|
// answer.
|
|
154
|
-
const feather = featherFor(recipe);
|
|
154
|
+
const feather = featherFor(recipe, { z, y, size });
|
|
155
155
|
return {
|
|
156
156
|
shape,
|
|
157
157
|
feather,
|
|
@@ -524,10 +524,11 @@ async function readMaskEdges({
|
|
|
524
524
|
contributions.map(async (contribution) => {
|
|
525
525
|
if (!contribution) return;
|
|
526
526
|
const recipe = contribution.source ?? {};
|
|
527
|
-
|
|
528
|
-
|
|
529
|
-
|
|
527
|
+
// On the source's own grid rather than the output's, because that is
|
|
528
|
+
// what the ramp will be measured on.
|
|
530
529
|
const grid = contribution.raster?.width ?? size;
|
|
530
|
+
const feather = featherFor(recipe, { z, y, size: grid });
|
|
531
|
+
if (!feather || !masksAnything(recipe)) return;
|
|
531
532
|
const layout = parentsFor({ z, x, y }, grid, feather);
|
|
532
533
|
if (!layout) return;
|
|
533
534
|
|
|
@@ -612,10 +613,14 @@ async function merge({
|
|
|
612
613
|
// decides what the pixels inside it weigh. The shape is known in full, so
|
|
613
614
|
// this costs the extra rows and nothing else -- unlike a mask read out of
|
|
614
615
|
// a source's own pixels, which would need its neighbours.
|
|
615
|
-
|
|
616
|
+
//
|
|
617
|
+
// Asked again rather than taken from the clip: `clipsFor` may have been
|
|
618
|
+
// given a grid the merge did not end up using, and a fade in metres is a
|
|
619
|
+
// different number of pixels on each of them.
|
|
620
|
+
const margin = featherFor(contribution.source, { z, y, size: grid });
|
|
616
621
|
const mask = rasterizeTile(clip.shape, z, x, y, grid, margin);
|
|
617
622
|
contribution.coverage = cropMask(
|
|
618
|
-
featherMask(mask, grid + margin * 2,
|
|
623
|
+
featherMask(mask, grid + margin * 2, margin),
|
|
619
624
|
grid,
|
|
620
625
|
margin,
|
|
621
626
|
);
|
|
@@ -729,7 +734,8 @@ export async function answerStackTile(options) {
|
|
|
729
734
|
// 256 where the request did not say, which is the cautious guess: the
|
|
730
735
|
// margin is in pixels of the output grid, so assuming a smaller grid makes
|
|
731
736
|
// it wider on the ground, and too wide only costs a rasterise that was not
|
|
732
|
-
// needed while too narrow misses a ramp.
|
|
737
|
+
// needed while too narrow misses a ramp. A fade in metres does not mind
|
|
738
|
+
// either way -- fewer pixels each covering more ground is the same ground.
|
|
733
739
|
const clips = clipsFor(resolved, cutlines, z, x, y, size);
|
|
734
740
|
const format = options.format ?? outputFormat(resolved);
|
|
735
741
|
const rgba = resolved.stack.space === 'rgba';
|