pmtiles-swarm 0.91.0 → 0.94.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -7,6 +7,66 @@
7
7
  ### šŸž Bug fixes
8
8
  - _...Add new stuff here..._
9
9
 
10
+ ## 0.94.0
11
+ ### šŸž Bug fixes
12
+ - **Saving settings rewrote a proxy list nobody had touched.** A trusted-proxy list may be stored as
13
+ a string — `"loopback, 10.0.0.0/8"` is what the documentation shows — and the box rendered that as
14
+ one line while reading it back as an array. The two never compared equal, so the field was sent on
15
+ every Save whether or not anybody had looked at it, and before the previous release that rewrite
16
+ was the thing that stopped the node from starting. Opening the settings page and pressing Save was
17
+ enough to do it.
18
+
19
+ The box now shows one entry per line whichever way the config wrote them, and records what a save
20
+ will read back rather than what the config holds — so an untouched field is untouched.
21
+
22
+ ## 0.93.0
23
+ ### šŸž Bug fixes
24
+ - **A trusted-proxy list typed with commas stopped the node from starting.** The settings field
25
+ split what was typed on newlines only, so one line reading `172.16.1.2, 172.16.1.3` was saved as
26
+ an array holding both addresses in one string. Express splits a comma list when it is handed a
27
+ bare string and never inside an array, so proxy-addr was given `172.16.1.2, 172.16.1.3` as a
28
+ single address and threw -- while the app was being built, before the listener binds. The node
29
+ would not start, could not be reached, and could not be corrected from the console that had
30
+ written the value; on the node that found this it was 155 restarts.
31
+
32
+ Three things were wrong and all three are fixed. The field now splits on commas as well as
33
+ newlines. Every shape the setting can be written in -- a string, an array, commas, spaces,
34
+ newlines -- is flattened to what Express wants. And an entry that is not an address is ignored
35
+ and logged rather than thrown: this setting is not worth a node that will not boot, and trusting
36
+ nobody is the safe end of being wrong about it.
37
+
38
+ - **A failed restore took the whole node down, console included.** Handing the library back to the
39
+ engine at startup already tolerates a failure per archive; the call coming apart as a whole was
40
+ unguarded, and it happens before the listener binds — so under `Restart=always` the result is a
41
+ crash loop with no console to look at and no way to see why. It is now reported and the node
42
+ starts anyway, where every archive shows as **not loaded** until it is fixed.
43
+
44
+ - **Nothing tested that the node starts at all.** Every other test builds the pieces `src/index.js`
45
+ wires together and never runs the wiring, so an import cycle or a step that throws before the
46
+ listener binds was a failure only a real start could find. There is now a boot test that runs the
47
+ entry point the way the service does and asks it for a page.
48
+
49
+ ## 0.92.0
50
+ ### šŸž Bug fixes
51
+ - **A stack's TileJSON published the addresses its sources are read from.** A URL source is named by
52
+ its URL and an S3 source by its bucket and key, and the document naming them is served to anybody
53
+ who can load the map — while the archives at the other end are read with credentials nobody else
54
+ has. The sources are gone from it entirely, which was the right answer anyway: they are the
55
+ ingredients of one endpoint, not a list for a client to join, and listing them invited fetching
56
+ those instead of the tiles. What is left is a count and a `revision` fingerprint covering the
57
+ recipe and what every source resolved to — the one thing the list was good for, in twenty bytes
58
+ rather than tens of kilobytes. The document for a 459-source stack is now 336 bytes.
59
+
60
+ - **Signing in threw away the view you asked for.** A link to `#stacks` on a guarded node asked for a
61
+ password and then showed the archives, with the address still reading `#stacks` — which is what
62
+ made it look broken rather than like a redirect. Both ways into the console now land where the
63
+ address says.
64
+
65
+ - **An archive the engine has no record of was drawn as 0%.** That is a different fact from "none of
66
+ it is here", and it is the more alarming one to get wrong: a library the engine failed to take back
67
+ after a restart read as a library that had lost its data. The row now says **not loaded**, and says
68
+ where the reason is logged and that nothing on disk has been touched.
69
+
10
70
  ## 0.91.0
11
71
  ### ✨ Features and improvements
12
72
  - **A source at a URL can be added by hand, not only imported.** **Add source → an address you
@@ -82,8 +82,17 @@ Takes effect on the next request; no restart. See
82
82
  ### `trustProxy`
83
83
 
84
84
  Takes anything Express accepts: `true`, a hop count, or a subnet list such as
85
- `"loopback, 10.0.0.0/8"`. Off by default, because trusting these headers from an
86
- untrusted client lets it claim any protocol or address it likes.
85
+ `"loopback, 10.0.0.0/8"`. A list may be written as one string or as an array,
86
+ and either may separate its entries with commas, spaces or newlines — all four
87
+ shapes mean the same thing here. Off by default, because trusting these headers
88
+ from an untrusted client lets it claim any protocol or address it likes.
89
+
90
+ An entry that is not an address, a subnet, or one of `loopback`, `linklocal`
91
+ and `uniquelocal` is **ignored and logged**, and a value that cannot be read at
92
+ all leaves the node trusting nobody. Deliberately not fatal: Express compiles
93
+ this the moment it is set, which is before the listener binds, so a value it
94
+ refuses used to be a node that would not start, could not be reached, and could
95
+ not be corrected from the console that wrote it.
87
96
 
88
97
  Set it when a proxy terminates TLS, or the TileJSON will advertise `http://` tile
89
98
  URLs that browsers block as mixed content. Setting `publicUrl` instead sidesteps
@@ -60,6 +60,7 @@ of its parts.
60
60
  - [Syncing a stack to another node](#syncing-a-stack-to-another-node)
61
61
  - [What the offline merge got wrong](#what-the-offline-merge-got-wrong)
62
62
  - [The stack editor](#the-stack-editor)
63
+ - [Merging vector sources](#merging-vector-sources)
63
64
  - [Staging](#staging)
64
65
  - [Open questions](#open-questions)
65
66
 
@@ -541,21 +542,42 @@ Derived from the resolved sources rather than from any one archive:
541
542
  - `attribution` — every source's, concatenated.
542
543
  - `tiles` — `/stacks/<id>/{z}/{x}/{y}.<ext>`.
543
544
 
544
- Plus a non-standard `stack` block naming what it resolved to, mirroring the
545
- `latest` block on `/latest/<category>/tiles.json`, so a consumer can tell one
546
- resolution from the next:
545
+ Plus a non-standard `stack` block, mirroring the `latest` block on
546
+ `/latest/<category>/tiles.json`, so a consumer can tell one resolution from the
547
+ next:
547
548
 
548
549
  ```jsonc
549
550
  "stack": {
550
551
  "id": "planet-terrain",
551
- "revision": 7,
552
- "sources": [
553
- { "category": "gebco", "infohash": "a074186d…", "name": "GEBCO_2026_…" },
554
- { "category": "planet-bathymetry", "infohash": "…", "name": "…" }
555
- ]
552
+ "space": "elevation",
553
+ "sources": 2,
554
+ "revision": "9f2c1a77b3e04d16"
556
555
  }
557
556
  ```
558
557
 
558
+ `revision` covers the recipe and what every source resolved to, so it moves
559
+ when a category resolves to a new build — which is the whole reason the block
560
+ exists.
561
+
562
+ ### The sources themselves are not in it
563
+
564
+ They were once, and it was wrong twice over.
565
+
566
+ A stack's sources are not a client's to join. They are the ingredients of one
567
+ endpoint, and this document exists to point at that endpoint: a map reads
568
+ `tiles`, not the archives behind it. Listing them invites somebody to fetch
569
+ those instead, which is the one thing a stack is there to stop them having to
570
+ do.
571
+
572
+ And a source may be an **address**. A URL, or a bucket and a key, published in
573
+ a document that is served to anybody who can load the map — while the archive
574
+ at the other end of it is read with credentials nobody else has. The tiles are
575
+ public on purpose; where they come from is not.
576
+
577
+ What is left says the same thing the list said, in twenty bytes rather than
578
+ tens of kilobytes: how many sources there are, and a fingerprint that moves
579
+ when any of them does.
580
+
559
581
  ## When a source will not answer
560
582
 
561
583
  A cache-mode source with no reachable peers, a category that resolves to an
@@ -2199,6 +2221,135 @@ Debounced, and it should not follow the map past the stack's `maxzoom`: above
2199
2221
  that the client overzooms and there is nothing new to see, but the tiles are
2200
2222
  still requested and still composited.
2201
2223
 
2224
+ ## Merging vector sources
2225
+
2226
+ **Not built.** This is the design, written down before anything is started so
2227
+ that the hard part is on record rather than discovered halfway through it.
2228
+
2229
+ Most of the machinery is indifferent to what a tile contains. Source
2230
+ resolution, per-source `minzoom`, `maxzoom` and `bounds`, painting order,
2231
+ nested stacks, the cache, the ETag, `unionOfTileIds` and the bake all treat a
2232
+ tile as an opaque thing, and the byte-for-byte passthrough already serves a
2233
+ vector archive today. What a vector space adds is a merge, and the merge is
2234
+ where the analogy with the two pixel spaces breaks.
2235
+
2236
+ ### What has no vector analogue
2237
+
2238
+ `opacity`, `blend`, `feather`, `featherMetres`, `gaussianBlurSigma`,
2239
+ `output.encoding`, `output.tileSize`, `maskValues`, `maskRange` and
2240
+ `maskColors`. Every one of them is an operation on a pixel, and there are no
2241
+ pixels. A recipe naming any of them under a vector space should be **refused by
2242
+ validation**, not accepted and ignored: a stack that says `feather: 50` and
2243
+ draws a hard edge is a worse answer than one that will not save.
2244
+
2245
+ The idea of a mask survives in three other forms — clip to a **shape**, keep or
2246
+ drop whole **layers** by name, and filter **features** by a property. Those
2247
+ three are the vector mask, and none of them is a mask in the sense the
2248
+ elevation space means.
2249
+
2250
+ ### Overlap is the whole problem
2251
+
2252
+ The pixel spaces work because the unit is a pixel, every source has one
2253
+ everywhere, and "the topmost covered source wins" is a complete rule. Vector's
2254
+ unit is a feature. Two sources that both carry a `water` layer over the same
2255
+ ground hold **two polygons of the same lake**, and emitting both overwrites
2256
+ neither. The tile carries both, the style draws both, and it shows as doubled
2257
+ outlines, darker semi-transparent fills, duplicated labels, and z-fighting
2258
+ between them.
2259
+
2260
+ So the first thing a vector space has to define is what winning means. Three
2261
+ answers, and the third is not a default:
2262
+
2263
+ **Per-layer replacement.** For each layer _name_, the topmost source carrying it
2264
+ supplies it whole, and the same-named layer of every source below is dropped for
2265
+ that tile. No geometry work at all: read the layer list, choose, re-emit. It is
2266
+ the right answer for a base plus an overlay of layers the base does not have —
2267
+ an OSM base under your own `buildings`, or under contours generated from the
2268
+ terrain these stacks already merge — and it is correct for a base plus a patch
2269
+ wherever the tile falls wholly inside the patch.
2270
+
2271
+ **Clip and union.** Each source is clipped to its own coverage, and the source
2272
+ below is additionally clipped _against_ the coverage of the one above it, so the
2273
+ two cannot both carry the same lake. This is the true counterpart of the pixel
2274
+ merge, and the only thing that works on a tile the boundary of a patch runs
2275
+ through. A **rectangle** clip in tile coordinates is cheap and exact —
2276
+ Sutherland-Hodgman for polygons, Liang-Barsky for lines — and a rectangle is
2277
+ what an imported file list states for every patch in it. An arbitrary
2278
+ **cutline** is a different matter and wants a real polygon-clipping library.
2279
+
2280
+ **Renaming.** The patch's `water` becomes `patch_water`, and the style decides
2281
+ what to do with it. No conflict and no geometry work, at the cost of a style
2282
+ that has to know. Worth having as a way out; not worth having as the rule.
2283
+
2284
+ The recommendation is per-layer replacement as the default, rectangle clipping
2285
+ for the base-and-patch case, and a **recipe warning** wherever two sources carry
2286
+ the same layer with no clip between them: _sources 0 and 1 both provide `water`,
2287
+ and tiles where they meet will carry both_. Silent doubling is the failure that
2288
+ gets found six months later by somebody squinting at a coastline.
2289
+
2290
+ ### A seam cannot be feathered
2291
+
2292
+ Where two sources disagree geometrically at a boundary, a road jogs and a
2293
+ coastline steps, and there is nothing to fade. [Feathering a
2294
+ seam](#feathering-a-seam) works because a pixel can be a mixture of two values.
2295
+ A feature cannot be a mixture of two features.
2296
+
2297
+ Vector merging is therefore clean only where the sources were cut to agree — the
2298
+ same condition that makes a provider's raster patches merge cleanly, minus the
2299
+ fade that hides the cases where it is not quite true. That belongs in the
2300
+ console rather than on the map.
2301
+
2302
+ ### Overzooming a parent
2303
+
2304
+ Possible, and needed: a base to z12 under a patch to z14 has to climb the base
2305
+ at z13, exactly as [resampling from a parent](#resampling-from-a-parent-tile)
2306
+ does for pixels. Scale the parent's coordinates by the zoom difference, subtract
2307
+ the quadrant's offset, clip to the extent plus the buffer. Exact for geometry,
2308
+ and it invents nothing: the generalisation and the label placement were chosen
2309
+ for the parent's zoom, so an overzoomed tile is thinner than a real one at that
2310
+ zoom and looks it.
2311
+
2312
+ ### Extent, buffer and schema
2313
+
2314
+ Sources at different **extents** — 4096 and 512 are both common — have to be
2315
+ normalised to one before anything is combined. A power-of-two ratio is exact;
2316
+ anything else quantises coordinates.
2317
+
2318
+ **Buffers** have to be re-clipped after any coordinate change, or features leak
2319
+ past the edge of the tile and draw twice at the join.
2320
+
2321
+ **Schemas** are the quiet one. Two sources whose `water` layers use `class` and
2322
+ `kind` merge into a layer the style half understands, and nothing in the merge
2323
+ can reconcile them. Worth comparing key sets on the first merged tile and
2324
+ reporting the disagreement as a problem on the stack, the way an unresolved
2325
+ source is reported.
2326
+
2327
+ **Feature ids** collide across sources. Renumbering settles the collision and
2328
+ breaks any client-side feature state keyed on them, so it is a choice to make
2329
+ out loud rather than a detail to settle inside the encoder.
2330
+
2331
+ ### The libraries
2332
+
2333
+ `@mapbox/vector-tile` and `pbf` to read, `vt-pbf` to write. Small, pure
2334
+ JavaScript, and ordinary dependencies rather than a probe: the optional
2335
+ treatment [the codec gets](#the-codec-problem) exists because a native build
2336
+ genuinely fails to install on some platforms, and none of that applies here.
2337
+
2338
+ Not written here either, unlike the PMTiles writer and the S3 signer. A protobuf
2339
+ round-trip has more edge cases than either — geometry command encoding, unknown
2340
+ value types, key and value pools shared across features — and getting one wrong
2341
+ produces a tile that parses and is subtly wrong.
2342
+
2343
+ ### Staging, when it is started
2344
+
2345
+ 1. **Layer operations only.** `space: "vector"`, with keep, drop and rename by
2346
+ layer name, over the per-source zoom and bounds skipping that already exists.
2347
+ Covers base-plus-overlay, and needs no geometry code whatsoever.
2348
+ 2. **Overzoom and rectangle clipping.** Covers base-plus-patch, which is the
2349
+ shape [importing a list of URLs](#importing-a-list-of-urls) makes easy to
2350
+ build.
2351
+ 3. **Cutline clipping and property filters.**
2352
+
2202
2353
  ## Staging
2203
2354
 
2204
2355
  1. ~~**Recipe and resolution.**~~ Done. `data/stacks.json`, load and validate,
@@ -2230,10 +2381,11 @@ handling whatsoever.
2230
2381
 
2231
2382
  ## Open questions
2232
2383
 
2233
- - **Vector tiles.** Merging MVT layers from two archives is a real want and a
2234
- completely different operation — decode protobuf, merge layer by layer,
2235
- re-encode, with feature ID collisions to settle. It should be its own feature
2236
- and should not be smuggled in under `space`.
2384
+ - **Vector tiles.** Designed rather than open now: see [merging vector
2385
+ sources](#merging-vector-sources). It is its own space and its own merge,
2386
+ and what stays open is whether the overlap rule proposed there — layer
2387
+ replacement by default, clipping for a patch — holds up against a real
2388
+ pair of archives.
2237
2389
  - **Does the recipe travel?** A stack is a small JSON document and the feed
2238
2390
  already distributes documents. Publishing recipes so a subscriber gets the
2239
2391
  stack along with its sources is attractive, and raises an obvious question
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pmtiles-swarm",
3
- "version": "0.91.0",
3
+ "version": "0.94.0",
4
4
  "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
5
5
  "type": "module",
6
6
  "main": "src/index.js",
package/src/api.js CHANGED
@@ -22,7 +22,12 @@ import {
22
22
  import { mutableMagnet, trackersFromMagnet } from './mutable.js';
23
23
  import { guessKind } from './library.js';
24
24
  import { QBittorrentEngine } from './engines/qbittorrent.js';
25
- import { RESTART_REQUIRED, redactConfig, saveConfig } from './config.js';
25
+ import {
26
+ RESTART_REQUIRED,
27
+ redactConfig,
28
+ saveConfig,
29
+ trustProxyFor,
30
+ } from './config.js';
26
31
  import { freeSpace, listLocations } from './locations.js';
27
32
  import { restart, restartMode } from './restart.js';
28
33
  import { parseFeed, renderFeed } from './feed.js';
@@ -50,15 +55,7 @@ import {
50
55
  stackCoverage,
51
56
  stackEtag,
52
57
  } from './stacks.js';
53
-
54
- /**
55
- * How many sources a stack's TileJSON names before it starts counting.
56
- *
57
- * Generous for a hand-written recipe -- those run to a handful -- and a hard
58
- * stop for one imported from a provider's index, where the list is hundreds
59
- * long and every entry is an address no client reads.
60
- */
61
- const TILEJSON_SOURCE_LIMIT = 25;
58
+ import { bakeRevision } from './bake.js';
62
59
 
63
60
  const here = path.dirname(fileURLToPath(import.meta.url));
64
61
 
@@ -268,7 +265,26 @@ export function createApp({
268
265
  // the TileJSON advertises http:// tile URLs. A browser that loaded the map
269
266
  // over https then blocks every one of them as mixed content, which looks
270
267
  // like an empty map rather than like a configuration mistake.
271
- if (config.trustProxy) app.set('trust proxy', config.trustProxy);
268
+ //
269
+ // Normalised first, and then guarded anyway. Express compiles this value
270
+ // the moment it is set, which is before the listener binds -- so a value it
271
+ // cannot compile is not a setting that fails to apply, it is a node that
272
+ // will not start, cannot be reached, and cannot be corrected from the
273
+ // console that wrote it. Nothing about which addresses to trust is worth
274
+ // that: an unparseable one is reported and the node comes up trusting
275
+ // nobody, which is the safe end of being wrong.
276
+ const trustProxy = trustProxyFor(config.trustProxy);
277
+ if (trustProxy !== false) {
278
+ try {
279
+ app.set('trust proxy', trustProxy);
280
+ } catch (error) {
281
+ console.error(
282
+ `[config] trustProxy could not be applied (${error.message}). ` +
283
+ 'X-Forwarded-* headers are being ignored; the node is starting ' +
284
+ 'anyway so this can be corrected from Settings.',
285
+ );
286
+ }
287
+ }
272
288
  app.use(express.json({ limit: '1mb' }));
273
289
 
274
290
  // Tiles, TileJSON and the feed stay public — serving them is the point.
@@ -3800,34 +3816,27 @@ export function createApp({
3800
3816
  minzoom: coverage.minzoom,
3801
3817
  maxzoom: coverage.maxzoom,
3802
3818
  bounds: coverage.bounds,
3803
- // Names what it resolved to, so a consumer can tell one resolution
3804
- // from the next without diffing tile URLs — the same reason the
3805
- // `/latest/` document carries a `latest` block.
3819
+ // Says that it resolved, and whether the resolution has moved, so a
3820
+ // consumer can tell one from the next without diffing tile URLs --
3821
+ // the same reason the `/latest/` document carries a `latest` block.
3822
+ //
3823
+ // A count and a fingerprint rather than the sources themselves. They
3824
+ // were listed here once, and it was wrong twice over. A stack's
3825
+ // sources are not a client's to join: they are the ingredients of one
3826
+ // endpoint, and a document listing them invites somebody to fetch
3827
+ // those instead of the tiles. And a source may be an address -- a URL,
3828
+ // or a bucket and a key -- which this document has no business
3829
+ // publishing: the tiles are public where the archives behind them are
3830
+ // read with credentials nobody else has.
3806
3831
  //
3807
- // Capped. A stack imported from a provider's file index has hundreds
3808
- // of sources — Mapterhorn's is 458 — and listing every one puts tens
3809
- // of kilobytes of addresses in a document every map load fetches, to
3810
- // say something no client does anything with. What the block is for is
3811
- // telling one resolution from the next, and what moves between
3812
- // resolutions is the infohashes, so those are what it keeps.
3832
+ // The fingerprint covers the recipe and what every source resolved
3833
+ // to, so it moves when a category resolves to a new build. That was
3834
+ // the only thing the list was good for.
3813
3835
  stack: {
3814
3836
  id: resolved.stack.id,
3815
3837
  space: resolved.stack.space ?? 'elevation',
3816
- sources: resolved.sources
3817
- .slice(0, TILEJSON_SOURCE_LIMIT)
3818
- .map((source) => ({
3819
- name: source.name,
3820
- infohash: source.entry?.infoHash ?? null,
3821
- archive: source.entry?.name ?? null,
3822
- })),
3823
- // Said rather than silently truncated: a consumer comparing two of
3824
- // these has to know it is looking at part of a list.
3825
- ...(resolved.sources.length > TILEJSON_SOURCE_LIMIT
3826
- ? {
3827
- more: resolved.sources.length - TILEJSON_SOURCE_LIMIT,
3828
- total: resolved.sources.length,
3829
- }
3830
- : {}),
3838
+ sources: resolved.sources.length,
3839
+ revision: bakeRevision(resolved),
3831
3840
  },
3832
3841
  };
3833
3842
  // A stack says how its tiles are encoded, so a style pointing at it does
package/src/config.js CHANGED
@@ -1,4 +1,5 @@
1
1
  import fs from 'node:fs/promises';
2
+ import net from 'node:net';
2
3
  import { hashPassword } from './auth.js';
3
4
  import path from 'node:path';
4
5
 
@@ -10,6 +11,79 @@ import path from 'node:path';
10
11
  * Every setting is documented in docs/configuration.md. Comments here say only
11
12
  * what a value is; why it is what it is belongs in the document.
12
13
  */
14
+ /** What proxy-addr accepts as a name for a group of addresses. */
15
+ const TRUST_KEYWORDS = new Set(['loopback', 'linklocal', 'uniquelocal']);
16
+
17
+ /**
18
+ * Whether one entry of a trust list is something proxy-addr can compile.
19
+ * @param {string} entry - One address, subnet or keyword.
20
+ * @returns {boolean} - True if it is usable.
21
+ */
22
+ function isTrustEntry(entry) {
23
+ if (TRUST_KEYWORDS.has(entry)) return true;
24
+ const [address, mask, ...rest] = entry.split('/');
25
+ if (rest.length > 0) return false;
26
+ if (!net.isIP(address)) return false;
27
+ if (mask === undefined) return true;
28
+ if (net.isIP(mask)) return true;
29
+ const bits = Number(mask);
30
+ if (!Number.isInteger(bits) || bits < 0) return false;
31
+ return bits <= (net.isIP(address) === 6 ? 128 : 32);
32
+ }
33
+
34
+ /**
35
+ * What to hand Express as `trust proxy`, from what the config says.
36
+ *
37
+ * Express takes four different things here and means something different by
38
+ * each, and it splits a comma-separated list only when it is handed a bare
39
+ * string -- never inside an array. So `["10.0.0.1, 10.0.0.2"]`, which is what
40
+ * a settings field split on newlines alone produces from one line with a
41
+ * comma in it, reaches proxy-addr as a single address, and proxy-addr throws.
42
+ *
43
+ * That throw happened while the app was being built, before the listener
44
+ * bound: a node that could not start, could not be reached, and could not be
45
+ * corrected from the console that wrote the value. So every shape is
46
+ * flattened here, and anything left that is not an address is dropped with a
47
+ * warning rather than carried into Express.
48
+ * @param {boolean|number|string|string[]} value - `config.trustProxy`.
49
+ * @returns {boolean|number|string[]} - Something Express can compile.
50
+ */
51
+ export function trustProxyFor(value) {
52
+ if (
53
+ value === true ||
54
+ value === false ||
55
+ value === undefined ||
56
+ value === null
57
+ ) {
58
+ return value === true;
59
+ }
60
+ if (typeof value === 'number') return Number.isFinite(value) ? value : false;
61
+
62
+ const entries = (Array.isArray(value) ? value : [value])
63
+ .flatMap((one) => String(one).split(/[\s,]+/))
64
+ .map((one) => one.trim())
65
+ .filter(Boolean);
66
+ if (!entries.length) return false;
67
+
68
+ // A lone number is a hop count, and a lone `true` trusts everybody. Both
69
+ // are things somebody may type into a field that mostly takes addresses.
70
+ if (entries.length === 1) {
71
+ if (entries[0] === 'true') return true;
72
+ if (entries[0] === 'false') return false;
73
+ if (!Number.isNaN(Number(entries[0]))) return Number(entries[0]);
74
+ }
75
+
76
+ const usable = entries.filter((entry) => isTrustEntry(entry));
77
+ for (const entry of entries) {
78
+ if (usable.includes(entry)) continue;
79
+ console.warn(
80
+ `[config] trustProxy: ignoring "${entry}", which is not an address, ` +
81
+ 'a subnet, or one of loopback, linklocal, uniquelocal.',
82
+ );
83
+ }
84
+ return usable.length ? usable : false;
85
+ }
86
+
13
87
  const DEFAULTS = {
14
88
  port: 8090,
15
89
  host: '0.0.0.0',
package/src/index.js CHANGED
@@ -339,11 +339,28 @@ PMTILES_SWARM_PUBLIC_URL
339
339
 
340
340
  const catalogued = catalog.list().length;
341
341
  if (catalogued > 0) {
342
- const { restored, failed } = await library.restore();
343
- console.log(
344
- `[restore] ${restored} of ${catalogued} archives handed back to the engine` +
345
- (failed > 0 ? ` (${failed} could not be)` : ''),
346
- );
342
+ // Reported, never fatal. Restore already tolerates a failure per archive;
343
+ // what this catches is the whole call coming apart -- and the node it
344
+ // takes down with it is the one that could have said so. Under
345
+ // `Restart=always` that is a crash loop with no console to look at, which
346
+ // is a worse failure than a library that is not being seeded: the console
347
+ // marks an archive the engine has no record of as `not loaded`, so a node
348
+ // that comes up says exactly what went wrong here.
349
+ try {
350
+ const { restored, failed } = await library.restore();
351
+ console.log(
352
+ `[restore] ${restored} of ${catalogued} archives handed back to the engine` +
353
+ (failed > 0 ? ` (${failed} could not be)` : ''),
354
+ );
355
+ } catch (error) {
356
+ console.error(
357
+ `[restore] could not hand the library back to the engine: ` +
358
+ `${error.stack ?? error.message}
359
+ ` +
360
+ '[restore] the node is starting anyway; every archive will show as ' +
361
+ 'not loaded until this is fixed and it is restarted.',
362
+ );
363
+ }
347
364
  }
348
365
 
349
366
  // And again if the engine loses its backing process and starts another. A
@@ -1875,6 +1875,11 @@
1875
1875
  for (const entry of shown) {
1876
1876
  const s = entry.status ?? {};
1877
1877
  const progress = s.progress ?? 0;
1878
+ // The engine has never mentioned this archive. That is not the same
1879
+ // fact as "none of it is here", and drawing it as 0% said the second
1880
+ // -- so a library the engine failed to take back after a restart
1881
+ // read as a library that had lost its data.
1882
+ const known = Boolean(entry.status);
1878
1883
  const mode = entry.mode ?? 'mirror';
1879
1884
  const tr = document.createElement('tr');
1880
1885
  if (entry.infoHash === selected) tr.className = 'on';
@@ -1893,12 +1898,14 @@
1893
1898
  <td>${added(entry.createdAt)}</td>
1894
1899
  <td><span class="pill ${mode}">${mode}</span></td>
1895
1900
  <td>${originCell(entry)}</td>
1896
- <td>
1897
- <div class="track">
1898
- <i class="${progress >= 1 ? 'done' : ''}" style="width:${Math.min(100, progress * 100)}%"></i>
1899
- </div>
1900
- <div class="sub">${pct(progress)}</div>
1901
- </td>
1901
+ <td>${
1902
+ known
1903
+ ? `<div class="track">
1904
+ <i class="${progress >= 1 ? 'done' : ''}" style="width:${Math.min(100, progress * 100)}%"></i>
1905
+ </div>
1906
+ <div class="sub">${pct(progress)}</div>`
1907
+ : `<div class="sub bad" title="The engine has no record of this archive, so nothing here is being seeded or downloaded. It usually means the restore after a restart could not hand it back — the log says why, under [restore]. Nothing has been deleted; the data on disk is untouched.">not loaded</div>`
1908
+ }</td>
1902
1909
  <td title="${peersTitle(s)}">${s.peers ?? 0} / ${s.seeds ?? 0}${swarmSuffix(s)}</td>
1903
1910
  <td>${rate(s.downloadSpeed)}</td>
1904
1911
  <td>${rate(s.uploadSpeed)}</td>
@@ -5572,11 +5579,15 @@ Every piece is hashed against the ` +
5572
5579
  type: 'addresses',
5573
5580
  placeholder: 'off — or one address or subnet per line',
5574
5581
  help:
5575
- 'One per line: 172.16.1.49, or 172.16.1.0/24, or several. A ' +
5576
- 'lone number is read as a hop count instead, and a lone `true` ' +
5577
- 'trusts any caller at all — worth avoiding, since this header ' +
5578
- 'is what decides which address the node believes a request ' +
5579
- 'came from.',
5582
+ 'An address or a subnet — <code>172.16.1.49</code>, ' +
5583
+ '<code>172.16.1.0/24</code>, <code>loopback</code> — one ' +
5584
+ 'per line, or separated by commas. A lone number is read ' +
5585
+ 'as a hop count instead, and a lone <code>true</code> ' +
5586
+ 'trusts any caller at all, which is worth avoiding: this ' +
5587
+ 'header is what decides which address the node believes a ' +
5588
+ 'request came from. Anything that is not an address is ' +
5589
+ 'ignored and said so in the log, rather than stopping the ' +
5590
+ 'node from starting.',
5580
5591
  },
5581
5592
  {
5582
5593
  key: 'publicIndex',
@@ -6300,7 +6311,14 @@ Every piece is hashed against the ` +
6300
6311
  field.type === 'secret'
6301
6312
  ? `<input type="password" autocomplete="new-password" data-setting="${field.key}"${locked} data-type="secret" data-initial="&quot;&quot;" value="" placeholder="${escapeHtml(field.placeholder ?? 'unchanged')}" />`
6302
6313
  : field.type === 'addresses'
6303
- ? `<textarea data-setting="${field.key}"${locked} data-type="addresses"${initial} rows="3" placeholder="${escapeHtml(field.placeholder ?? '')}">${escapeHtml(Array.isArray(value) ? value.join('\n') : value === undefined || value === null ? '' : String(value))}</textarea>`
6314
+ ? // Its own initial, because this is the one field whose
6315
+ // box holds a different shape from the config. A
6316
+ // stored string never compares equal to the array
6317
+ // the box parses to, so sharing `initial` meant
6318
+ // every Save rewrote a setting nobody had touched
6319
+ // -- which is how a value that worked became one
6320
+ // the node would not start with.
6321
+ `<textarea data-setting="${field.key}"${locked} data-type="addresses" data-initial="${escapeHtml(JSON.stringify(parseAddresses(addressText(value))))}" rows="3" placeholder="${escapeHtml(field.placeholder ?? '')}">${escapeHtml(addressText(value))}</textarea>`
6304
6322
  : field.type === 'list'
6305
6323
  ? `<textarea data-setting="${field.key}"${locked} data-type="list"${initial} rows="${Math.min(8, Math.max(3, (Array.isArray(value) ? value.length : 0) + 1))}" placeholder="${escapeHtml(field.placeholder ?? '')}">${escapeHtml((Array.isArray(value) ? value : []).join('\n'))}</textarea>`
6306
6324
  : field.type === 'boolean'
@@ -6330,6 +6348,56 @@ Every piece is hashed against the ` +
6330
6348
  return panel;
6331
6349
  }
6332
6350
 
6351
+ /**
6352
+ * The entries an address field holds, however the config wrote them.
6353
+ *
6354
+ * A list may be stored as an array or as one comma-separated string --
6355
+ * Express accepts both, and the documented example is a string. Shown
6356
+ * one per line either way, so that what a save reads back is the list
6357
+ * that was rendered rather than a different shape of the same thing.
6358
+ * @param {null|boolean|number|string|string[]} value - What the config holds.
6359
+ * @returns {string} - The textarea's contents.
6360
+ */
6361
+ const addressText = (value) => {
6362
+ if (value === undefined || value === null) return '';
6363
+ const entries = Array.isArray(value) ? value : [String(value)];
6364
+ return entries
6365
+ .flatMap((entry) => String(entry).split(/[\n,]/))
6366
+ .map((entry) => entry.trim())
6367
+ .filter(Boolean)
6368
+ .join('\n');
6369
+ };
6370
+
6371
+ /**
6372
+ * What an address field means by what it holds.
6373
+ *
6374
+ * Express takes four things here and means something different by each:
6375
+ * `true` trusts any caller, a number is a hop count, and a string or an
6376
+ * array of them names the proxies. What was typed decides which.
6377
+ *
6378
+ * Commas separate as well as newlines. Express splits a comma list only
6379
+ * when it is handed a bare string and never inside an array, so one line
6380
+ * reading `10.0.0.1, 10.0.0.2` was saved as a one-element array holding
6381
+ * both -- which is not an address, and which Express rejects while the
6382
+ * app is being built, before the listener binds. The node then would not
6383
+ * start, and could not be corrected from the console that wrote it.
6384
+ * @param {string} text - What the box holds.
6385
+ * @returns {null|boolean|number|string[]} - What to save.
6386
+ */
6387
+ const parseAddresses = (text) => {
6388
+ const lines = String(text)
6389
+ .split(/[\n,]/)
6390
+ .map((line) => line.trim())
6391
+ .filter(Boolean);
6392
+ if (!lines.length) return null;
6393
+ if (lines.length === 1) {
6394
+ if (lines[0] === 'true') return true;
6395
+ if (lines[0] === 'false') return false;
6396
+ if (!Number.isNaN(Number(lines[0]))) return Number(lines[0]);
6397
+ }
6398
+ return lines;
6399
+ };
6400
+
6333
6401
  /**
6334
6402
  * Collects every schema control into an update, grouped by top-level key.
6335
6403
  *
@@ -6361,22 +6429,8 @@ Every piece is hashed against the ` +
6361
6429
  continue;
6362
6430
  }
6363
6431
  if (type === 'boolean') value = element.checked;
6364
- else if (type === 'addresses') {
6365
- // Express takes four things here and means something different by
6366
- // each: `true` trusts any caller, a number is a hop count, and a
6367
- // string or an array of them names the proxies. One per line, and
6368
- // what was typed decides which of the four it is.
6369
- const lines = String(element.value)
6370
- .split('\n')
6371
- .map((line) => line.trim())
6372
- .filter(Boolean);
6373
- if (!lines.length) value = null;
6374
- else if (lines.length === 1 && (lines[0] === 'true' || lines[0] === 'false')) {
6375
- value = lines[0] === 'true';
6376
- } else if (lines.length === 1 && !Number.isNaN(Number(lines[0]))) {
6377
- value = Number(lines[0]);
6378
- } else value = lines;
6379
- } else if (type === 'list') {
6432
+ else if (type === 'addresses') value = parseAddresses(element.value);
6433
+ else if (type === 'list') {
6380
6434
  // One per line, which is how a person reads a list of trackers —
6381
6435
  // and empty means an empty list rather than "unset", because a
6382
6436
  // node with no trackers is a real thing to want and JSON would
@@ -7506,6 +7560,22 @@ Every piece is hashed against the ` +
7506
7560
  return TABS.includes(named) ? named : null;
7507
7561
  };
7508
7562
 
7563
+ /**
7564
+ * Lands on the view the address names, once there is a console to land
7565
+ * in.
7566
+ *
7567
+ * Two ways in, and both have to honour it: opening the page already
7568
+ * signed in, and signing in on arrival. Only the first did, so following
7569
+ * a link to `#stacks` on a guarded node asked for a password and then
7570
+ * showed the archives -- the address still saying `#stacks`, which is
7571
+ * the part that makes it read as broken rather than as a redirect.
7572
+ * @returns {void}
7573
+ */
7574
+ const landFromUrl = () => {
7575
+ const named = tabFromUrl();
7576
+ if (named && named !== 'archives') showTab(named);
7577
+ };
7578
+
7509
7579
  // The footer's year, set from the clock rather than typed into a file
7510
7580
  // nobody will remember to edit. The version beside it arrives with the
7511
7581
  // first status, and reads "…" until then.
@@ -9451,6 +9521,7 @@ Every piece is hashed against the ` +
9451
9521
  $('login').hidden = true;
9452
9522
  $('login-pass').value = '';
9453
9523
  refresh();
9524
+ landFromUrl();
9454
9525
  } catch (error) {
9455
9526
  $('login-error').textContent = error.message;
9456
9527
  }
@@ -9476,8 +9547,7 @@ Every piece is hashed against the ` +
9476
9547
  refresh();
9477
9548
  // After the archives have been asked for, so the tab that opens is the
9478
9549
  // one asked for and not the one this happens to start on.
9479
- const named = tabFromUrl();
9480
- if (named && named !== 'archives') showTab(named);
9550
+ landFromUrl();
9481
9551
  }
9482
9552
 
9483
9553
  start();