pmtiles-swarm 0.91.0 ā 0.94.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +60 -0
- package/docs/configuration.md +11 -2
- package/docs/tile-stacks.md +164 -12
- package/package.json +1 -1
- package/src/api.js +44 -35
- package/src/config.js +74 -0
- package/src/index.js +22 -5
- package/src/web/index.html +100 -30
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,66 @@
|
|
|
7
7
|
### š Bug fixes
|
|
8
8
|
- _...Add new stuff here..._
|
|
9
9
|
|
|
10
|
+
## 0.94.0
|
|
11
|
+
### š Bug fixes
|
|
12
|
+
- **Saving settings rewrote a proxy list nobody had touched.** A trusted-proxy list may be stored as
|
|
13
|
+
a string ā `"loopback, 10.0.0.0/8"` is what the documentation shows ā and the box rendered that as
|
|
14
|
+
one line while reading it back as an array. The two never compared equal, so the field was sent on
|
|
15
|
+
every Save whether or not anybody had looked at it, and before the previous release that rewrite
|
|
16
|
+
was the thing that stopped the node from starting. Opening the settings page and pressing Save was
|
|
17
|
+
enough to do it.
|
|
18
|
+
|
|
19
|
+
The box now shows one entry per line whichever way the config wrote them, and records what a save
|
|
20
|
+
will read back rather than what the config holds ā so an untouched field is untouched.
|
|
21
|
+
|
|
22
|
+
## 0.93.0
|
|
23
|
+
### š Bug fixes
|
|
24
|
+
- **A trusted-proxy list typed with commas stopped the node from starting.** The settings field
|
|
25
|
+
split what was typed on newlines only, so one line reading `172.16.1.2, 172.16.1.3` was saved as
|
|
26
|
+
an array holding both addresses in one string. Express splits a comma list when it is handed a
|
|
27
|
+
bare string and never inside an array, so proxy-addr was given `172.16.1.2, 172.16.1.3` as a
|
|
28
|
+
single address and threw -- while the app was being built, before the listener binds. The node
|
|
29
|
+
would not start, could not be reached, and could not be corrected from the console that had
|
|
30
|
+
written the value; on the node that found this it was 155 restarts.
|
|
31
|
+
|
|
32
|
+
Three things were wrong and all three are fixed. The field now splits on commas as well as
|
|
33
|
+
newlines. Every shape the setting can be written in -- a string, an array, commas, spaces,
|
|
34
|
+
newlines -- is flattened to what Express wants. And an entry that is not an address is ignored
|
|
35
|
+
and logged rather than thrown: this setting is not worth a node that will not boot, and trusting
|
|
36
|
+
nobody is the safe end of being wrong about it.
|
|
37
|
+
|
|
38
|
+
- **A failed restore took the whole node down, console included.** Handing the library back to the
|
|
39
|
+
engine at startup already tolerates a failure per archive; the call coming apart as a whole was
|
|
40
|
+
unguarded, and it happens before the listener binds ā so under `Restart=always` the result is a
|
|
41
|
+
crash loop with no console to look at and no way to see why. It is now reported and the node
|
|
42
|
+
starts anyway, where every archive shows as **not loaded** until it is fixed.
|
|
43
|
+
|
|
44
|
+
- **Nothing tested that the node starts at all.** Every other test builds the pieces `src/index.js`
|
|
45
|
+
wires together and never runs the wiring, so an import cycle or a step that throws before the
|
|
46
|
+
listener binds was a failure only a real start could find. There is now a boot test that runs the
|
|
47
|
+
entry point the way the service does and asks it for a page.
|
|
48
|
+
|
|
49
|
+
## 0.92.0
|
|
50
|
+
### š Bug fixes
|
|
51
|
+
- **A stack's TileJSON published the addresses its sources are read from.** A URL source is named by
|
|
52
|
+
its URL and an S3 source by its bucket and key, and the document naming them is served to anybody
|
|
53
|
+
who can load the map ā while the archives at the other end are read with credentials nobody else
|
|
54
|
+
has. The sources are gone from it entirely, which was the right answer anyway: they are the
|
|
55
|
+
ingredients of one endpoint, not a list for a client to join, and listing them invited fetching
|
|
56
|
+
those instead of the tiles. What is left is a count and a `revision` fingerprint covering the
|
|
57
|
+
recipe and what every source resolved to ā the one thing the list was good for, in twenty bytes
|
|
58
|
+
rather than tens of kilobytes. The document for a 459-source stack is now 336 bytes.
|
|
59
|
+
|
|
60
|
+
- **Signing in threw away the view you asked for.** A link to `#stacks` on a guarded node asked for a
|
|
61
|
+
password and then showed the archives, with the address still reading `#stacks` ā which is what
|
|
62
|
+
made it look broken rather than like a redirect. Both ways into the console now land where the
|
|
63
|
+
address says.
|
|
64
|
+
|
|
65
|
+
- **An archive the engine has no record of was drawn as 0%.** That is a different fact from "none of
|
|
66
|
+
it is here", and it is the more alarming one to get wrong: a library the engine failed to take back
|
|
67
|
+
after a restart read as a library that had lost its data. The row now says **not loaded**, and says
|
|
68
|
+
where the reason is logged and that nothing on disk has been touched.
|
|
69
|
+
|
|
10
70
|
## 0.91.0
|
|
11
71
|
### ⨠Features and improvements
|
|
12
72
|
- **A source at a URL can be added by hand, not only imported.** **Add source ā an address you
|
package/docs/configuration.md
CHANGED
|
@@ -82,8 +82,17 @@ Takes effect on the next request; no restart. See
|
|
|
82
82
|
### `trustProxy`
|
|
83
83
|
|
|
84
84
|
Takes anything Express accepts: `true`, a hop count, or a subnet list such as
|
|
85
|
-
`"loopback, 10.0.0.0/8"`.
|
|
86
|
-
|
|
85
|
+
`"loopback, 10.0.0.0/8"`. A list may be written as one string or as an array,
|
|
86
|
+
and either may separate its entries with commas, spaces or newlines ā all four
|
|
87
|
+
shapes mean the same thing here. Off by default, because trusting these headers
|
|
88
|
+
from an untrusted client lets it claim any protocol or address it likes.
|
|
89
|
+
|
|
90
|
+
An entry that is not an address, a subnet, or one of `loopback`, `linklocal`
|
|
91
|
+
and `uniquelocal` is **ignored and logged**, and a value that cannot be read at
|
|
92
|
+
all leaves the node trusting nobody. Deliberately not fatal: Express compiles
|
|
93
|
+
this the moment it is set, which is before the listener binds, so a value it
|
|
94
|
+
refuses used to be a node that would not start, could not be reached, and could
|
|
95
|
+
not be corrected from the console that wrote it.
|
|
87
96
|
|
|
88
97
|
Set it when a proxy terminates TLS, or the TileJSON will advertise `http://` tile
|
|
89
98
|
URLs that browsers block as mixed content. Setting `publicUrl` instead sidesteps
|
package/docs/tile-stacks.md
CHANGED
|
@@ -60,6 +60,7 @@ of its parts.
|
|
|
60
60
|
- [Syncing a stack to another node](#syncing-a-stack-to-another-node)
|
|
61
61
|
- [What the offline merge got wrong](#what-the-offline-merge-got-wrong)
|
|
62
62
|
- [The stack editor](#the-stack-editor)
|
|
63
|
+
- [Merging vector sources](#merging-vector-sources)
|
|
63
64
|
- [Staging](#staging)
|
|
64
65
|
- [Open questions](#open-questions)
|
|
65
66
|
|
|
@@ -541,21 +542,42 @@ Derived from the resolved sources rather than from any one archive:
|
|
|
541
542
|
- `attribution` ā every source's, concatenated.
|
|
542
543
|
- `tiles` ā `/stacks/<id>/{z}/{x}/{y}.<ext>`.
|
|
543
544
|
|
|
544
|
-
Plus a non-standard `stack` block
|
|
545
|
-
|
|
546
|
-
|
|
545
|
+
Plus a non-standard `stack` block, mirroring the `latest` block on
|
|
546
|
+
`/latest/<category>/tiles.json`, so a consumer can tell one resolution from the
|
|
547
|
+
next:
|
|
547
548
|
|
|
548
549
|
```jsonc
|
|
549
550
|
"stack": {
|
|
550
551
|
"id": "planet-terrain",
|
|
551
|
-
"
|
|
552
|
-
"sources":
|
|
553
|
-
|
|
554
|
-
{ "category": "planet-bathymetry", "infohash": "ā¦", "name": "ā¦" }
|
|
555
|
-
]
|
|
552
|
+
"space": "elevation",
|
|
553
|
+
"sources": 2,
|
|
554
|
+
"revision": "9f2c1a77b3e04d16"
|
|
556
555
|
}
|
|
557
556
|
```
|
|
558
557
|
|
|
558
|
+
`revision` covers the recipe and what every source resolved to, so it moves
|
|
559
|
+
when a category resolves to a new build ā which is the whole reason the block
|
|
560
|
+
exists.
|
|
561
|
+
|
|
562
|
+
### The sources themselves are not in it
|
|
563
|
+
|
|
564
|
+
They were once, and it was wrong twice over.
|
|
565
|
+
|
|
566
|
+
A stack's sources are not a client's to join. They are the ingredients of one
|
|
567
|
+
endpoint, and this document exists to point at that endpoint: a map reads
|
|
568
|
+
`tiles`, not the archives behind it. Listing them invites somebody to fetch
|
|
569
|
+
those instead, which is the one thing a stack is there to stop them having to
|
|
570
|
+
do.
|
|
571
|
+
|
|
572
|
+
And a source may be an **address**. A URL, or a bucket and a key, published in
|
|
573
|
+
a document that is served to anybody who can load the map ā while the archive
|
|
574
|
+
at the other end of it is read with credentials nobody else has. The tiles are
|
|
575
|
+
public on purpose; where they come from is not.
|
|
576
|
+
|
|
577
|
+
What is left says the same thing the list said, in twenty bytes rather than
|
|
578
|
+
tens of kilobytes: how many sources there are, and a fingerprint that moves
|
|
579
|
+
when any of them does.
|
|
580
|
+
|
|
559
581
|
## When a source will not answer
|
|
560
582
|
|
|
561
583
|
A cache-mode source with no reachable peers, a category that resolves to an
|
|
@@ -2199,6 +2221,135 @@ Debounced, and it should not follow the map past the stack's `maxzoom`: above
|
|
|
2199
2221
|
that the client overzooms and there is nothing new to see, but the tiles are
|
|
2200
2222
|
still requested and still composited.
|
|
2201
2223
|
|
|
2224
|
+
## Merging vector sources
|
|
2225
|
+
|
|
2226
|
+
**Not built.** This is the design, written down before anything is started so
|
|
2227
|
+
that the hard part is on record rather than discovered halfway through it.
|
|
2228
|
+
|
|
2229
|
+
Most of the machinery is indifferent to what a tile contains. Source
|
|
2230
|
+
resolution, per-source `minzoom`, `maxzoom` and `bounds`, painting order,
|
|
2231
|
+
nested stacks, the cache, the ETag, `unionOfTileIds` and the bake all treat a
|
|
2232
|
+
tile as an opaque thing, and the byte-for-byte passthrough already serves a
|
|
2233
|
+
vector archive today. What a vector space adds is a merge, and the merge is
|
|
2234
|
+
where the analogy with the two pixel spaces breaks.
|
|
2235
|
+
|
|
2236
|
+
### What has no vector analogue
|
|
2237
|
+
|
|
2238
|
+
`opacity`, `blend`, `feather`, `featherMetres`, `gaussianBlurSigma`,
|
|
2239
|
+
`output.encoding`, `output.tileSize`, `maskValues`, `maskRange` and
|
|
2240
|
+
`maskColors`. Every one of them is an operation on a pixel, and there are no
|
|
2241
|
+
pixels. A recipe naming any of them under a vector space should be **refused by
|
|
2242
|
+
validation**, not accepted and ignored: a stack that says `feather: 50` and
|
|
2243
|
+
draws a hard edge is a worse answer than one that will not save.
|
|
2244
|
+
|
|
2245
|
+
The idea of a mask survives in three other forms ā clip to a **shape**, keep or
|
|
2246
|
+
drop whole **layers** by name, and filter **features** by a property. Those
|
|
2247
|
+
three are the vector mask, and none of them is a mask in the sense the
|
|
2248
|
+
elevation space means.
|
|
2249
|
+
|
|
2250
|
+
### Overlap is the whole problem
|
|
2251
|
+
|
|
2252
|
+
The pixel spaces work because the unit is a pixel, every source has one
|
|
2253
|
+
everywhere, and "the topmost covered source wins" is a complete rule. Vector's
|
|
2254
|
+
unit is a feature. Two sources that both carry a `water` layer over the same
|
|
2255
|
+
ground hold **two polygons of the same lake**, and emitting both overwrites
|
|
2256
|
+
neither. The tile carries both, the style draws both, and it shows as doubled
|
|
2257
|
+
outlines, darker semi-transparent fills, duplicated labels, and z-fighting
|
|
2258
|
+
between them.
|
|
2259
|
+
|
|
2260
|
+
So the first thing a vector space has to define is what winning means. Three
|
|
2261
|
+
answers, and the third is not a default:
|
|
2262
|
+
|
|
2263
|
+
**Per-layer replacement.** For each layer _name_, the topmost source carrying it
|
|
2264
|
+
supplies it whole, and the same-named layer of every source below is dropped for
|
|
2265
|
+
that tile. No geometry work at all: read the layer list, choose, re-emit. It is
|
|
2266
|
+
the right answer for a base plus an overlay of layers the base does not have ā
|
|
2267
|
+
an OSM base under your own `buildings`, or under contours generated from the
|
|
2268
|
+
terrain these stacks already merge ā and it is correct for a base plus a patch
|
|
2269
|
+
wherever the tile falls wholly inside the patch.
|
|
2270
|
+
|
|
2271
|
+
**Clip and union.** Each source is clipped to its own coverage, and the source
|
|
2272
|
+
below is additionally clipped _against_ the coverage of the one above it, so the
|
|
2273
|
+
two cannot both carry the same lake. This is the true counterpart of the pixel
|
|
2274
|
+
merge, and the only thing that works on a tile the boundary of a patch runs
|
|
2275
|
+
through. A **rectangle** clip in tile coordinates is cheap and exact ā
|
|
2276
|
+
Sutherland-Hodgman for polygons, Liang-Barsky for lines ā and a rectangle is
|
|
2277
|
+
what an imported file list states for every patch in it. An arbitrary
|
|
2278
|
+
**cutline** is a different matter and wants a real polygon-clipping library.
|
|
2279
|
+
|
|
2280
|
+
**Renaming.** The patch's `water` becomes `patch_water`, and the style decides
|
|
2281
|
+
what to do with it. No conflict and no geometry work, at the cost of a style
|
|
2282
|
+
that has to know. Worth having as a way out; not worth having as the rule.
|
|
2283
|
+
|
|
2284
|
+
The recommendation is per-layer replacement as the default, rectangle clipping
|
|
2285
|
+
for the base-and-patch case, and a **recipe warning** wherever two sources carry
|
|
2286
|
+
the same layer with no clip between them: _sources 0 and 1 both provide `water`,
|
|
2287
|
+
and tiles where they meet will carry both_. Silent doubling is the failure that
|
|
2288
|
+
gets found six months later by somebody squinting at a coastline.
|
|
2289
|
+
|
|
2290
|
+
### A seam cannot be feathered
|
|
2291
|
+
|
|
2292
|
+
Where two sources disagree geometrically at a boundary, a road jogs and a
|
|
2293
|
+
coastline steps, and there is nothing to fade. [Feathering a
|
|
2294
|
+
seam](#feathering-a-seam) works because a pixel can be a mixture of two values.
|
|
2295
|
+
A feature cannot be a mixture of two features.
|
|
2296
|
+
|
|
2297
|
+
Vector merging is therefore clean only where the sources were cut to agree ā the
|
|
2298
|
+
same condition that makes a provider's raster patches merge cleanly, minus the
|
|
2299
|
+
fade that hides the cases where it is not quite true. That belongs in the
|
|
2300
|
+
console rather than on the map.
|
|
2301
|
+
|
|
2302
|
+
### Overzooming a parent
|
|
2303
|
+
|
|
2304
|
+
Possible, and needed: a base to z12 under a patch to z14 has to climb the base
|
|
2305
|
+
at z13, exactly as [resampling from a parent](#resampling-from-a-parent-tile)
|
|
2306
|
+
does for pixels. Scale the parent's coordinates by the zoom difference, subtract
|
|
2307
|
+
the quadrant's offset, clip to the extent plus the buffer. Exact for geometry,
|
|
2308
|
+
and it invents nothing: the generalisation and the label placement were chosen
|
|
2309
|
+
for the parent's zoom, so an overzoomed tile is thinner than a real one at that
|
|
2310
|
+
zoom and looks it.
|
|
2311
|
+
|
|
2312
|
+
### Extent, buffer and schema
|
|
2313
|
+
|
|
2314
|
+
Sources at different **extents** ā 4096 and 512 are both common ā have to be
|
|
2315
|
+
normalised to one before anything is combined. A power-of-two ratio is exact;
|
|
2316
|
+
anything else quantises coordinates.
|
|
2317
|
+
|
|
2318
|
+
**Buffers** have to be re-clipped after any coordinate change, or features leak
|
|
2319
|
+
past the edge of the tile and draw twice at the join.
|
|
2320
|
+
|
|
2321
|
+
**Schemas** are the quiet one. Two sources whose `water` layers use `class` and
|
|
2322
|
+
`kind` merge into a layer the style half understands, and nothing in the merge
|
|
2323
|
+
can reconcile them. Worth comparing key sets on the first merged tile and
|
|
2324
|
+
reporting the disagreement as a problem on the stack, the way an unresolved
|
|
2325
|
+
source is reported.
|
|
2326
|
+
|
|
2327
|
+
**Feature ids** collide across sources. Renumbering settles the collision and
|
|
2328
|
+
breaks any client-side feature state keyed on them, so it is a choice to make
|
|
2329
|
+
out loud rather than a detail to settle inside the encoder.
|
|
2330
|
+
|
|
2331
|
+
### The libraries
|
|
2332
|
+
|
|
2333
|
+
`@mapbox/vector-tile` and `pbf` to read, `vt-pbf` to write. Small, pure
|
|
2334
|
+
JavaScript, and ordinary dependencies rather than a probe: the optional
|
|
2335
|
+
treatment [the codec gets](#the-codec-problem) exists because a native build
|
|
2336
|
+
genuinely fails to install on some platforms, and none of that applies here.
|
|
2337
|
+
|
|
2338
|
+
Not written here either, unlike the PMTiles writer and the S3 signer. A protobuf
|
|
2339
|
+
round-trip has more edge cases than either ā geometry command encoding, unknown
|
|
2340
|
+
value types, key and value pools shared across features ā and getting one wrong
|
|
2341
|
+
produces a tile that parses and is subtly wrong.
|
|
2342
|
+
|
|
2343
|
+
### Staging, when it is started
|
|
2344
|
+
|
|
2345
|
+
1. **Layer operations only.** `space: "vector"`, with keep, drop and rename by
|
|
2346
|
+
layer name, over the per-source zoom and bounds skipping that already exists.
|
|
2347
|
+
Covers base-plus-overlay, and needs no geometry code whatsoever.
|
|
2348
|
+
2. **Overzoom and rectangle clipping.** Covers base-plus-patch, which is the
|
|
2349
|
+
shape [importing a list of URLs](#importing-a-list-of-urls) makes easy to
|
|
2350
|
+
build.
|
|
2351
|
+
3. **Cutline clipping and property filters.**
|
|
2352
|
+
|
|
2202
2353
|
## Staging
|
|
2203
2354
|
|
|
2204
2355
|
1. ~~**Recipe and resolution.**~~ Done. `data/stacks.json`, load and validate,
|
|
@@ -2230,10 +2381,11 @@ handling whatsoever.
|
|
|
2230
2381
|
|
|
2231
2382
|
## Open questions
|
|
2232
2383
|
|
|
2233
|
-
- **Vector tiles.**
|
|
2234
|
-
|
|
2235
|
-
|
|
2236
|
-
|
|
2384
|
+
- **Vector tiles.** Designed rather than open now: see [merging vector
|
|
2385
|
+
sources](#merging-vector-sources). It is its own space and its own merge,
|
|
2386
|
+
and what stays open is whether the overlap rule proposed there ā layer
|
|
2387
|
+
replacement by default, clipping for a patch ā holds up against a real
|
|
2388
|
+
pair of archives.
|
|
2237
2389
|
- **Does the recipe travel?** A stack is a small JSON document and the feed
|
|
2238
2390
|
already distributes documents. Publishing recipes so a subscriber gets the
|
|
2239
2391
|
stack along with its sources is attractive, and raises an obvious question
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pmtiles-swarm",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.94.0",
|
|
4
4
|
"description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "src/index.js",
|
package/src/api.js
CHANGED
|
@@ -22,7 +22,12 @@ import {
|
|
|
22
22
|
import { mutableMagnet, trackersFromMagnet } from './mutable.js';
|
|
23
23
|
import { guessKind } from './library.js';
|
|
24
24
|
import { QBittorrentEngine } from './engines/qbittorrent.js';
|
|
25
|
-
import {
|
|
25
|
+
import {
|
|
26
|
+
RESTART_REQUIRED,
|
|
27
|
+
redactConfig,
|
|
28
|
+
saveConfig,
|
|
29
|
+
trustProxyFor,
|
|
30
|
+
} from './config.js';
|
|
26
31
|
import { freeSpace, listLocations } from './locations.js';
|
|
27
32
|
import { restart, restartMode } from './restart.js';
|
|
28
33
|
import { parseFeed, renderFeed } from './feed.js';
|
|
@@ -50,15 +55,7 @@ import {
|
|
|
50
55
|
stackCoverage,
|
|
51
56
|
stackEtag,
|
|
52
57
|
} from './stacks.js';
|
|
53
|
-
|
|
54
|
-
/**
|
|
55
|
-
* How many sources a stack's TileJSON names before it starts counting.
|
|
56
|
-
*
|
|
57
|
-
* Generous for a hand-written recipe -- those run to a handful -- and a hard
|
|
58
|
-
* stop for one imported from a provider's index, where the list is hundreds
|
|
59
|
-
* long and every entry is an address no client reads.
|
|
60
|
-
*/
|
|
61
|
-
const TILEJSON_SOURCE_LIMIT = 25;
|
|
58
|
+
import { bakeRevision } from './bake.js';
|
|
62
59
|
|
|
63
60
|
const here = path.dirname(fileURLToPath(import.meta.url));
|
|
64
61
|
|
|
@@ -268,7 +265,26 @@ export function createApp({
|
|
|
268
265
|
// the TileJSON advertises http:// tile URLs. A browser that loaded the map
|
|
269
266
|
// over https then blocks every one of them as mixed content, which looks
|
|
270
267
|
// like an empty map rather than like a configuration mistake.
|
|
271
|
-
|
|
268
|
+
//
|
|
269
|
+
// Normalised first, and then guarded anyway. Express compiles this value
|
|
270
|
+
// the moment it is set, which is before the listener binds -- so a value it
|
|
271
|
+
// cannot compile is not a setting that fails to apply, it is a node that
|
|
272
|
+
// will not start, cannot be reached, and cannot be corrected from the
|
|
273
|
+
// console that wrote it. Nothing about which addresses to trust is worth
|
|
274
|
+
// that: an unparseable one is reported and the node comes up trusting
|
|
275
|
+
// nobody, which is the safe end of being wrong.
|
|
276
|
+
const trustProxy = trustProxyFor(config.trustProxy);
|
|
277
|
+
if (trustProxy !== false) {
|
|
278
|
+
try {
|
|
279
|
+
app.set('trust proxy', trustProxy);
|
|
280
|
+
} catch (error) {
|
|
281
|
+
console.error(
|
|
282
|
+
`[config] trustProxy could not be applied (${error.message}). ` +
|
|
283
|
+
'X-Forwarded-* headers are being ignored; the node is starting ' +
|
|
284
|
+
'anyway so this can be corrected from Settings.',
|
|
285
|
+
);
|
|
286
|
+
}
|
|
287
|
+
}
|
|
272
288
|
app.use(express.json({ limit: '1mb' }));
|
|
273
289
|
|
|
274
290
|
// Tiles, TileJSON and the feed stay public ā serving them is the point.
|
|
@@ -3800,34 +3816,27 @@ export function createApp({
|
|
|
3800
3816
|
minzoom: coverage.minzoom,
|
|
3801
3817
|
maxzoom: coverage.maxzoom,
|
|
3802
3818
|
bounds: coverage.bounds,
|
|
3803
|
-
//
|
|
3804
|
-
// from the next without diffing tile URLs
|
|
3805
|
-
// `/latest/` document carries a `latest` block.
|
|
3819
|
+
// Says that it resolved, and whether the resolution has moved, so a
|
|
3820
|
+
// consumer can tell one from the next without diffing tile URLs --
|
|
3821
|
+
// the same reason the `/latest/` document carries a `latest` block.
|
|
3822
|
+
//
|
|
3823
|
+
// A count and a fingerprint rather than the sources themselves. They
|
|
3824
|
+
// were listed here once, and it was wrong twice over. A stack's
|
|
3825
|
+
// sources are not a client's to join: they are the ingredients of one
|
|
3826
|
+
// endpoint, and a document listing them invites somebody to fetch
|
|
3827
|
+
// those instead of the tiles. And a source may be an address -- a URL,
|
|
3828
|
+
// or a bucket and a key -- which this document has no business
|
|
3829
|
+
// publishing: the tiles are public where the archives behind them are
|
|
3830
|
+
// read with credentials nobody else has.
|
|
3806
3831
|
//
|
|
3807
|
-
//
|
|
3808
|
-
//
|
|
3809
|
-
//
|
|
3810
|
-
// say something no client does anything with. What the block is for is
|
|
3811
|
-
// telling one resolution from the next, and what moves between
|
|
3812
|
-
// resolutions is the infohashes, so those are what it keeps.
|
|
3832
|
+
// The fingerprint covers the recipe and what every source resolved
|
|
3833
|
+
// to, so it moves when a category resolves to a new build. That was
|
|
3834
|
+
// the only thing the list was good for.
|
|
3813
3835
|
stack: {
|
|
3814
3836
|
id: resolved.stack.id,
|
|
3815
3837
|
space: resolved.stack.space ?? 'elevation',
|
|
3816
|
-
sources: resolved.sources
|
|
3817
|
-
|
|
3818
|
-
.map((source) => ({
|
|
3819
|
-
name: source.name,
|
|
3820
|
-
infohash: source.entry?.infoHash ?? null,
|
|
3821
|
-
archive: source.entry?.name ?? null,
|
|
3822
|
-
})),
|
|
3823
|
-
// Said rather than silently truncated: a consumer comparing two of
|
|
3824
|
-
// these has to know it is looking at part of a list.
|
|
3825
|
-
...(resolved.sources.length > TILEJSON_SOURCE_LIMIT
|
|
3826
|
-
? {
|
|
3827
|
-
more: resolved.sources.length - TILEJSON_SOURCE_LIMIT,
|
|
3828
|
-
total: resolved.sources.length,
|
|
3829
|
-
}
|
|
3830
|
-
: {}),
|
|
3838
|
+
sources: resolved.sources.length,
|
|
3839
|
+
revision: bakeRevision(resolved),
|
|
3831
3840
|
},
|
|
3832
3841
|
};
|
|
3833
3842
|
// A stack says how its tiles are encoded, so a style pointing at it does
|
package/src/config.js
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import fs from 'node:fs/promises';
|
|
2
|
+
import net from 'node:net';
|
|
2
3
|
import { hashPassword } from './auth.js';
|
|
3
4
|
import path from 'node:path';
|
|
4
5
|
|
|
@@ -10,6 +11,79 @@ import path from 'node:path';
|
|
|
10
11
|
* Every setting is documented in docs/configuration.md. Comments here say only
|
|
11
12
|
* what a value is; why it is what it is belongs in the document.
|
|
12
13
|
*/
|
|
14
|
+
/** What proxy-addr accepts as a name for a group of addresses. */
|
|
15
|
+
const TRUST_KEYWORDS = new Set(['loopback', 'linklocal', 'uniquelocal']);
|
|
16
|
+
|
|
17
|
+
/**
|
|
18
|
+
* Whether one entry of a trust list is something proxy-addr can compile.
|
|
19
|
+
* @param {string} entry - One address, subnet or keyword.
|
|
20
|
+
* @returns {boolean} - True if it is usable.
|
|
21
|
+
*/
|
|
22
|
+
function isTrustEntry(entry) {
|
|
23
|
+
if (TRUST_KEYWORDS.has(entry)) return true;
|
|
24
|
+
const [address, mask, ...rest] = entry.split('/');
|
|
25
|
+
if (rest.length > 0) return false;
|
|
26
|
+
if (!net.isIP(address)) return false;
|
|
27
|
+
if (mask === undefined) return true;
|
|
28
|
+
if (net.isIP(mask)) return true;
|
|
29
|
+
const bits = Number(mask);
|
|
30
|
+
if (!Number.isInteger(bits) || bits < 0) return false;
|
|
31
|
+
return bits <= (net.isIP(address) === 6 ? 128 : 32);
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
/**
|
|
35
|
+
* What to hand Express as `trust proxy`, from what the config says.
|
|
36
|
+
*
|
|
37
|
+
* Express takes four different things here and means something different by
|
|
38
|
+
* each, and it splits a comma-separated list only when it is handed a bare
|
|
39
|
+
* string -- never inside an array. So `["10.0.0.1, 10.0.0.2"]`, which is what
|
|
40
|
+
* a settings field split on newlines alone produces from one line with a
|
|
41
|
+
* comma in it, reaches proxy-addr as a single address, and proxy-addr throws.
|
|
42
|
+
*
|
|
43
|
+
* That throw happened while the app was being built, before the listener
|
|
44
|
+
* bound: a node that could not start, could not be reached, and could not be
|
|
45
|
+
* corrected from the console that wrote the value. So every shape is
|
|
46
|
+
* flattened here, and anything left that is not an address is dropped with a
|
|
47
|
+
* warning rather than carried into Express.
|
|
48
|
+
* @param {boolean|number|string|string[]} value - `config.trustProxy`.
|
|
49
|
+
* @returns {boolean|number|string[]} - Something Express can compile.
|
|
50
|
+
*/
|
|
51
|
+
export function trustProxyFor(value) {
|
|
52
|
+
if (
|
|
53
|
+
value === true ||
|
|
54
|
+
value === false ||
|
|
55
|
+
value === undefined ||
|
|
56
|
+
value === null
|
|
57
|
+
) {
|
|
58
|
+
return value === true;
|
|
59
|
+
}
|
|
60
|
+
if (typeof value === 'number') return Number.isFinite(value) ? value : false;
|
|
61
|
+
|
|
62
|
+
const entries = (Array.isArray(value) ? value : [value])
|
|
63
|
+
.flatMap((one) => String(one).split(/[\s,]+/))
|
|
64
|
+
.map((one) => one.trim())
|
|
65
|
+
.filter(Boolean);
|
|
66
|
+
if (!entries.length) return false;
|
|
67
|
+
|
|
68
|
+
// A lone number is a hop count, and a lone `true` trusts everybody. Both
|
|
69
|
+
// are things somebody may type into a field that mostly takes addresses.
|
|
70
|
+
if (entries.length === 1) {
|
|
71
|
+
if (entries[0] === 'true') return true;
|
|
72
|
+
if (entries[0] === 'false') return false;
|
|
73
|
+
if (!Number.isNaN(Number(entries[0]))) return Number(entries[0]);
|
|
74
|
+
}
|
|
75
|
+
|
|
76
|
+
const usable = entries.filter((entry) => isTrustEntry(entry));
|
|
77
|
+
for (const entry of entries) {
|
|
78
|
+
if (usable.includes(entry)) continue;
|
|
79
|
+
console.warn(
|
|
80
|
+
`[config] trustProxy: ignoring "${entry}", which is not an address, ` +
|
|
81
|
+
'a subnet, or one of loopback, linklocal, uniquelocal.',
|
|
82
|
+
);
|
|
83
|
+
}
|
|
84
|
+
return usable.length ? usable : false;
|
|
85
|
+
}
|
|
86
|
+
|
|
13
87
|
const DEFAULTS = {
|
|
14
88
|
port: 8090,
|
|
15
89
|
host: '0.0.0.0',
|
package/src/index.js
CHANGED
|
@@ -339,11 +339,28 @@ PMTILES_SWARM_PUBLIC_URL
|
|
|
339
339
|
|
|
340
340
|
const catalogued = catalog.list().length;
|
|
341
341
|
if (catalogued > 0) {
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
|
|
345
|
-
|
|
346
|
-
|
|
342
|
+
// Reported, never fatal. Restore already tolerates a failure per archive;
|
|
343
|
+
// what this catches is the whole call coming apart -- and the node it
|
|
344
|
+
// takes down with it is the one that could have said so. Under
|
|
345
|
+
// `Restart=always` that is a crash loop with no console to look at, which
|
|
346
|
+
// is a worse failure than a library that is not being seeded: the console
|
|
347
|
+
// marks an archive the engine has no record of as `not loaded`, so a node
|
|
348
|
+
// that comes up says exactly what went wrong here.
|
|
349
|
+
try {
|
|
350
|
+
const { restored, failed } = await library.restore();
|
|
351
|
+
console.log(
|
|
352
|
+
`[restore] ${restored} of ${catalogued} archives handed back to the engine` +
|
|
353
|
+
(failed > 0 ? ` (${failed} could not be)` : ''),
|
|
354
|
+
);
|
|
355
|
+
} catch (error) {
|
|
356
|
+
console.error(
|
|
357
|
+
`[restore] could not hand the library back to the engine: ` +
|
|
358
|
+
`${error.stack ?? error.message}
|
|
359
|
+
` +
|
|
360
|
+
'[restore] the node is starting anyway; every archive will show as ' +
|
|
361
|
+
'not loaded until this is fixed and it is restarted.',
|
|
362
|
+
);
|
|
363
|
+
}
|
|
347
364
|
}
|
|
348
365
|
|
|
349
366
|
// And again if the engine loses its backing process and starts another. A
|
package/src/web/index.html
CHANGED
|
@@ -1875,6 +1875,11 @@
|
|
|
1875
1875
|
for (const entry of shown) {
|
|
1876
1876
|
const s = entry.status ?? {};
|
|
1877
1877
|
const progress = s.progress ?? 0;
|
|
1878
|
+
// The engine has never mentioned this archive. That is not the same
|
|
1879
|
+
// fact as "none of it is here", and drawing it as 0% said the second
|
|
1880
|
+
// -- so a library the engine failed to take back after a restart
|
|
1881
|
+
// read as a library that had lost its data.
|
|
1882
|
+
const known = Boolean(entry.status);
|
|
1878
1883
|
const mode = entry.mode ?? 'mirror';
|
|
1879
1884
|
const tr = document.createElement('tr');
|
|
1880
1885
|
if (entry.infoHash === selected) tr.className = 'on';
|
|
@@ -1893,12 +1898,14 @@
|
|
|
1893
1898
|
<td>${added(entry.createdAt)}</td>
|
|
1894
1899
|
<td><span class="pill ${mode}">${mode}</span></td>
|
|
1895
1900
|
<td>${originCell(entry)}</td>
|
|
1896
|
-
<td
|
|
1897
|
-
|
|
1898
|
-
|
|
1899
|
-
|
|
1900
|
-
|
|
1901
|
-
|
|
1901
|
+
<td>${
|
|
1902
|
+
known
|
|
1903
|
+
? `<div class="track">
|
|
1904
|
+
<i class="${progress >= 1 ? 'done' : ''}" style="width:${Math.min(100, progress * 100)}%"></i>
|
|
1905
|
+
</div>
|
|
1906
|
+
<div class="sub">${pct(progress)}</div>`
|
|
1907
|
+
: `<div class="sub bad" title="The engine has no record of this archive, so nothing here is being seeded or downloaded. It usually means the restore after a restart could not hand it back ā the log says why, under [restore]. Nothing has been deleted; the data on disk is untouched.">not loaded</div>`
|
|
1908
|
+
}</td>
|
|
1902
1909
|
<td title="${peersTitle(s)}">${s.peers ?? 0} / ${s.seeds ?? 0}${swarmSuffix(s)}</td>
|
|
1903
1910
|
<td>${rate(s.downloadSpeed)}</td>
|
|
1904
1911
|
<td>${rate(s.uploadSpeed)}</td>
|
|
@@ -5572,11 +5579,15 @@ Every piece is hashed against the ` +
|
|
|
5572
5579
|
type: 'addresses',
|
|
5573
5580
|
placeholder: 'off ā or one address or subnet per line',
|
|
5574
5581
|
help:
|
|
5575
|
-
'
|
|
5576
|
-
'
|
|
5577
|
-
'
|
|
5578
|
-
'
|
|
5579
|
-
'
|
|
5582
|
+
'An address or a subnet ā <code>172.16.1.49</code>, ' +
|
|
5583
|
+
'<code>172.16.1.0/24</code>, <code>loopback</code> ā one ' +
|
|
5584
|
+
'per line, or separated by commas. A lone number is read ' +
|
|
5585
|
+
'as a hop count instead, and a lone <code>true</code> ' +
|
|
5586
|
+
'trusts any caller at all, which is worth avoiding: this ' +
|
|
5587
|
+
'header is what decides which address the node believes a ' +
|
|
5588
|
+
'request came from. Anything that is not an address is ' +
|
|
5589
|
+
'ignored and said so in the log, rather than stopping the ' +
|
|
5590
|
+
'node from starting.',
|
|
5580
5591
|
},
|
|
5581
5592
|
{
|
|
5582
5593
|
key: 'publicIndex',
|
|
@@ -6300,7 +6311,14 @@ Every piece is hashed against the ` +
|
|
|
6300
6311
|
field.type === 'secret'
|
|
6301
6312
|
? `<input type="password" autocomplete="new-password" data-setting="${field.key}"${locked} data-type="secret" data-initial="""" value="" placeholder="${escapeHtml(field.placeholder ?? 'unchanged')}" />`
|
|
6302
6313
|
: field.type === 'addresses'
|
|
6303
|
-
?
|
|
6314
|
+
? // Its own initial, because this is the one field whose
|
|
6315
|
+
// box holds a different shape from the config. A
|
|
6316
|
+
// stored string never compares equal to the array
|
|
6317
|
+
// the box parses to, so sharing `initial` meant
|
|
6318
|
+
// every Save rewrote a setting nobody had touched
|
|
6319
|
+
// -- which is how a value that worked became one
|
|
6320
|
+
// the node would not start with.
|
|
6321
|
+
`<textarea data-setting="${field.key}"${locked} data-type="addresses" data-initial="${escapeHtml(JSON.stringify(parseAddresses(addressText(value))))}" rows="3" placeholder="${escapeHtml(field.placeholder ?? '')}">${escapeHtml(addressText(value))}</textarea>`
|
|
6304
6322
|
: field.type === 'list'
|
|
6305
6323
|
? `<textarea data-setting="${field.key}"${locked} data-type="list"${initial} rows="${Math.min(8, Math.max(3, (Array.isArray(value) ? value.length : 0) + 1))}" placeholder="${escapeHtml(field.placeholder ?? '')}">${escapeHtml((Array.isArray(value) ? value : []).join('\n'))}</textarea>`
|
|
6306
6324
|
: field.type === 'boolean'
|
|
@@ -6330,6 +6348,56 @@ Every piece is hashed against the ` +
|
|
|
6330
6348
|
return panel;
|
|
6331
6349
|
}
|
|
6332
6350
|
|
|
6351
|
+
/**
|
|
6352
|
+
* The entries an address field holds, however the config wrote them.
|
|
6353
|
+
*
|
|
6354
|
+
* A list may be stored as an array or as one comma-separated string --
|
|
6355
|
+
* Express accepts both, and the documented example is a string. Shown
|
|
6356
|
+
* one per line either way, so that what a save reads back is the list
|
|
6357
|
+
* that was rendered rather than a different shape of the same thing.
|
|
6358
|
+
* @param {null|boolean|number|string|string[]} value - What the config holds.
|
|
6359
|
+
* @returns {string} - The textarea's contents.
|
|
6360
|
+
*/
|
|
6361
|
+
const addressText = (value) => {
|
|
6362
|
+
if (value === undefined || value === null) return '';
|
|
6363
|
+
const entries = Array.isArray(value) ? value : [String(value)];
|
|
6364
|
+
return entries
|
|
6365
|
+
.flatMap((entry) => String(entry).split(/[\n,]/))
|
|
6366
|
+
.map((entry) => entry.trim())
|
|
6367
|
+
.filter(Boolean)
|
|
6368
|
+
.join('\n');
|
|
6369
|
+
};
|
|
6370
|
+
|
|
6371
|
+
/**
|
|
6372
|
+
* What an address field means by what it holds.
|
|
6373
|
+
*
|
|
6374
|
+
* Express takes four things here and means something different by each:
|
|
6375
|
+
* `true` trusts any caller, a number is a hop count, and a string or an
|
|
6376
|
+
* array of them names the proxies. What was typed decides which.
|
|
6377
|
+
*
|
|
6378
|
+
* Commas separate as well as newlines. Express splits a comma list only
|
|
6379
|
+
* when it is handed a bare string and never inside an array, so one line
|
|
6380
|
+
* reading `10.0.0.1, 10.0.0.2` was saved as a one-element array holding
|
|
6381
|
+
* both -- which is not an address, and which Express rejects while the
|
|
6382
|
+
* app is being built, before the listener binds. The node then would not
|
|
6383
|
+
* start, and could not be corrected from the console that wrote it.
|
|
6384
|
+
* @param {string} text - What the box holds.
|
|
6385
|
+
* @returns {null|boolean|number|string[]} - What to save.
|
|
6386
|
+
*/
|
|
6387
|
+
const parseAddresses = (text) => {
|
|
6388
|
+
const lines = String(text)
|
|
6389
|
+
.split(/[\n,]/)
|
|
6390
|
+
.map((line) => line.trim())
|
|
6391
|
+
.filter(Boolean);
|
|
6392
|
+
if (!lines.length) return null;
|
|
6393
|
+
if (lines.length === 1) {
|
|
6394
|
+
if (lines[0] === 'true') return true;
|
|
6395
|
+
if (lines[0] === 'false') return false;
|
|
6396
|
+
if (!Number.isNaN(Number(lines[0]))) return Number(lines[0]);
|
|
6397
|
+
}
|
|
6398
|
+
return lines;
|
|
6399
|
+
};
|
|
6400
|
+
|
|
6333
6401
|
/**
|
|
6334
6402
|
* Collects every schema control into an update, grouped by top-level key.
|
|
6335
6403
|
*
|
|
@@ -6361,22 +6429,8 @@ Every piece is hashed against the ` +
|
|
|
6361
6429
|
continue;
|
|
6362
6430
|
}
|
|
6363
6431
|
if (type === 'boolean') value = element.checked;
|
|
6364
|
-
else if (type === 'addresses')
|
|
6365
|
-
|
|
6366
|
-
// each: `true` trusts any caller, a number is a hop count, and a
|
|
6367
|
-
// string or an array of them names the proxies. One per line, and
|
|
6368
|
-
// what was typed decides which of the four it is.
|
|
6369
|
-
const lines = String(element.value)
|
|
6370
|
-
.split('\n')
|
|
6371
|
-
.map((line) => line.trim())
|
|
6372
|
-
.filter(Boolean);
|
|
6373
|
-
if (!lines.length) value = null;
|
|
6374
|
-
else if (lines.length === 1 && (lines[0] === 'true' || lines[0] === 'false')) {
|
|
6375
|
-
value = lines[0] === 'true';
|
|
6376
|
-
} else if (lines.length === 1 && !Number.isNaN(Number(lines[0]))) {
|
|
6377
|
-
value = Number(lines[0]);
|
|
6378
|
-
} else value = lines;
|
|
6379
|
-
} else if (type === 'list') {
|
|
6432
|
+
else if (type === 'addresses') value = parseAddresses(element.value);
|
|
6433
|
+
else if (type === 'list') {
|
|
6380
6434
|
// One per line, which is how a person reads a list of trackers ā
|
|
6381
6435
|
// and empty means an empty list rather than "unset", because a
|
|
6382
6436
|
// node with no trackers is a real thing to want and JSON would
|
|
@@ -7506,6 +7560,22 @@ Every piece is hashed against the ` +
|
|
|
7506
7560
|
return TABS.includes(named) ? named : null;
|
|
7507
7561
|
};
|
|
7508
7562
|
|
|
7563
|
+
/**
|
|
7564
|
+
* Lands on the view the address names, once there is a console to land
|
|
7565
|
+
* in.
|
|
7566
|
+
*
|
|
7567
|
+
* Two ways in, and both have to honour it: opening the page already
|
|
7568
|
+
* signed in, and signing in on arrival. Only the first did, so following
|
|
7569
|
+
* a link to `#stacks` on a guarded node asked for a password and then
|
|
7570
|
+
* showed the archives -- the address still saying `#stacks`, which is
|
|
7571
|
+
* the part that makes it read as broken rather than as a redirect.
|
|
7572
|
+
* @returns {void}
|
|
7573
|
+
*/
|
|
7574
|
+
const landFromUrl = () => {
|
|
7575
|
+
const named = tabFromUrl();
|
|
7576
|
+
if (named && named !== 'archives') showTab(named);
|
|
7577
|
+
};
|
|
7578
|
+
|
|
7509
7579
|
// The footer's year, set from the clock rather than typed into a file
|
|
7510
7580
|
// nobody will remember to edit. The version beside it arrives with the
|
|
7511
7581
|
// first status, and reads "ā¦" until then.
|
|
@@ -9451,6 +9521,7 @@ Every piece is hashed against the ` +
|
|
|
9451
9521
|
$('login').hidden = true;
|
|
9452
9522
|
$('login-pass').value = '';
|
|
9453
9523
|
refresh();
|
|
9524
|
+
landFromUrl();
|
|
9454
9525
|
} catch (error) {
|
|
9455
9526
|
$('login-error').textContent = error.message;
|
|
9456
9527
|
}
|
|
@@ -9476,8 +9547,7 @@ Every piece is hashed against the ` +
|
|
|
9476
9547
|
refresh();
|
|
9477
9548
|
// After the archives have been asked for, so the tab that opens is the
|
|
9478
9549
|
// one asked for and not the one this happens to start on.
|
|
9479
|
-
|
|
9480
|
-
if (named && named !== 'archives') showTab(named);
|
|
9550
|
+
landFromUrl();
|
|
9481
9551
|
}
|
|
9482
9552
|
|
|
9483
9553
|
start();
|