pmtiles-swarm 0.18.0 → 0.20.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +52 -1
- package/README.md +111 -77
- package/docs/architecture-diagram.md +12 -12
- package/docs/configuration.md +754 -0
- package/docs/engines.md +42 -42
- package/docs/haproxy.md +26 -26
- package/docs/internals.md +768 -0
- package/docs/publishing.md +76 -76
- package/docs/running-as-a-service.md +20 -20
- package/docs/security.md +32 -20
- package/docs/serving-tiles.md +66 -53
- package/docs/subscribing.md +37 -23
- package/docs/tilejson.md +179 -0
- package/package.json +17 -2
- package/src/api.js +213 -120
- package/src/auth.js +31 -21
- package/src/catalog.js +4 -3
- package/src/config.js +162 -657
- package/src/engines/composite.js +35 -38
- package/src/engines/libtorrent.js +6 -17
- package/src/engines/qbittorrent.js +5 -6
- package/src/engines/webtorrent.js +7 -77
- package/src/feed.js +27 -6
- package/src/hooks.js +3 -1
- package/src/incomplete.js +18 -77
- package/src/index.js +58 -32
- package/src/library.js +186 -138
- package/src/locations.js +8 -17
- package/src/lock.js +5 -1
- package/src/mbtiles.js +256 -0
- package/src/mutable.js +22 -14
- package/src/origin.js +11 -10
- package/src/pieces.js +6 -2
- package/src/pmtiles-probe.js +27 -1
- package/src/prewarm.js +9 -20
- package/src/publisher.js +41 -45
- package/src/rate-limits.js +20 -8
- package/src/retention.js +4 -24
- package/src/seeding.js +9 -14
- package/src/shutdown.js +9 -4
- package/src/sources.js +33 -78
- package/src/status-command.js +3 -1
- package/src/subscriptions.js +26 -38
- package/src/tile-stats.js +8 -3
- package/src/tilejson.js +7 -0
- package/src/tiles.js +32 -2
- package/src/torrent-create.js +29 -67
- package/src/watch.js +7 -2
- package/src/web/public.html +332 -0
package/CHANGELOG.md
CHANGED
|
@@ -16,9 +16,60 @@
|
|
|
16
16
|
WebRTC only and has no DHT, PeX or local discovery to fall back on, so `wss://` is not
|
|
17
17
|
redundancy with the rest of the list, it is the whole of that path. Two are listed because it
|
|
18
18
|
has no backstop. Every entry was checked for a completed handshake before being added.
|
|
19
|
-
-
|
|
19
|
+
- **A mirror now serves the same `ETag` as the node it followed.** BitTorrent does not carry
|
|
20
|
+
mtime — it is not in the metainfo — so a delivered archive was stamped with the moment its
|
|
21
|
+
download finished, and Apache's default `FileETag MTime Size` then gave two nodes holding
|
|
22
|
+
byte-identical archives two different validators. A client whose range requests are balanced
|
|
23
|
+
across the pair fails part-way through a read, which `pmtiles.js` reports as `EtagMismatch`.
|
|
24
|
+
The origin's timestamp now travels in the feed as `<pmtiles:mtime>` and is restored when the
|
|
25
|
+
download completes, so the bytes and the validator agree everywhere. It is restored only where
|
|
26
|
+
a peer published one; an archive that arrives without it keeps its download time, exactly as
|
|
27
|
+
before. The restore happens between the rename and the re-add, while nothing is holding the
|
|
28
|
+
file: libtorrent's resume data records each file's size and mtime and re-hashes the whole
|
|
29
|
+
store when they disagree on load, so doing it under a running torrent would trade a broken
|
|
30
|
+
ETag for an hours-long recheck of a large library.
|
|
31
|
+
- **MBTiles archives are served as tiles once their download finishes.** They could be
|
|
32
|
+
distributed but never served, which was right while one is arriving — MBTiles is SQLite, whose
|
|
33
|
+
pages are laid out for a B-tree rather than spatially, so reading one tile can touch pages
|
|
34
|
+
anywhere in the file, and over a swarm that is not a read but a download. None of that is true
|
|
35
|
+
of a complete local copy: it is an ordinary database holding the same tiles and metadata a
|
|
36
|
+
PMTiles does. A TileJSON endpoint and z/x/y tiles now answer for one, read through the built-in
|
|
37
|
+
`node:sqlite` so this costs no dependency. An incomplete archive answers 503 — "not yet" —
|
|
38
|
+
where anything that holds no tiles at all still answers 415.
|
|
39
|
+
- **`sparse` is read from the archive's own metadata.** Defaulting by tile format is a guess:
|
|
40
|
+
PMTiles records that tiles are webp, not that they are terrain. tileserver-gl reads `sparse`
|
|
41
|
+
from the metadata, so an archive built to be served there already carried the answer and it was
|
|
42
|
+
being ignored here — the same file behaved differently in the two servers unless configured
|
|
43
|
+
twice. Precedence is the entry, then the archive, then this node's default, then the format
|
|
44
|
+
guess; the archive sits above the node default because a blanket setting was chosen without
|
|
45
|
+
reference to any particular archive. It is republished in the TileJSON, so a mirror starts from
|
|
46
|
+
the same answer.
|
|
47
|
+
- **A mutable magnet now carries the build that is current, alongside the key.** A BEP 46 magnet
|
|
48
|
+
named only the public key, which needs a DHT to resolve — and browsers have none, since
|
|
49
|
+
WebTorrent stubs out `bittorrent-dht` in its browser build for want of UDP sockets. That
|
|
50
|
+
mattered because this string is routinely put in the fragment of a `tiles.json` URL, an
|
|
51
|
+
arrangement whose whole point is that one URL is self-sufficient. A key-only fragment forced a
|
|
52
|
+
browser to fetch the very document the fragment was attached to before it could join anything.
|
|
53
|
+
The magnet is now `xt=urn:btih:<build>&xs=urn:btpk:<key>`: a client resolves whichever it
|
|
54
|
+
understands, and the infohash going stale on the next build is what the key beside it is for.
|
|
55
|
+
- **The public listener has a front page.** With `adminPort` splitting the two, `/` on the public
|
|
56
|
+
port was a 404. It now lists the archives this node publishes with their tile and TileJSON
|
|
57
|
+
URLs, torrents, magnets and a preview for each. It is a view of `/api/catalog` and
|
|
58
|
+
`/api/categories`, filtered by the same `feedCategories` rule, so it can show nothing that was
|
|
59
|
+
not already published — and it is not the console, which stays on the admin port.
|
|
60
|
+
`publicIndex: false` turns it off, withdrawing the three paths it needs with it.
|
|
61
|
+
- **Lint and format tooling.** `npm run lint`, `lint:fix`, `format`, `format:check`, `tidy` and
|
|
62
|
+
`check`, wired into CI. There was no linter before, though the source carried
|
|
63
|
+
`eslint-disable` comments for one — so those suppressed nothing and the rules they named were
|
|
64
|
+
never checked.
|
|
20
65
|
|
|
21
66
|
### 🐞 Bug fixes
|
|
67
|
+
- **WebTorrent's `pieces()` was defined twice.** The later definition wins, so the first had
|
|
68
|
+
never run — which is why it still called an undefined `countHeld` and no test noticed.
|
|
69
|
+
- **A stuck HTTP connection could keep a shutdown waiting.** `closeServer` armed its force-close
|
|
70
|
+
timer after registering the handler that clears it, so a close callback arriving first would
|
|
71
|
+
`clearTimeout(undefined)` and leave the timer running.
|
|
72
|
+
- **Three thrown errors discarded the error that caused them**, losing the cause chain.
|
|
22
73
|
- **Retention no longer reaches across a folder's other entries.** A watched folder's family was
|
|
23
74
|
built from the directory alone, so several entries sharing one — which `match` exists to make
|
|
24
75
|
possible — were treated as a single family. With `keep: 1`, importing this week's `monthly`
|
package/README.md
CHANGED
|
@@ -28,6 +28,8 @@ node src/index.js --config swarm.config.json
|
|
|
28
28
|
feed contents, and updatable torrents.
|
|
29
29
|
- **[docs/serving-tiles.md](docs/serving-tiles.md)** — the TileJSON and z/x/y endpoints, the
|
|
30
30
|
`torrent` block that torrent-aware clients use, caching, and running behind a proxy.
|
|
31
|
+
- **[docs/tilejson.md](docs/tilejson.md)** — the TileJSON document in detail, and the two
|
|
32
|
+
members that are ours rather than the spec's: `torrent` and `sparse`.
|
|
31
33
|
- **[docs/security.md](docs/security.md)** — what is public, what is guarded, named tokens with
|
|
32
34
|
roles, console sign-in, and why an unauthenticated node refuses to listen on a reachable
|
|
33
35
|
address.
|
|
@@ -38,6 +40,10 @@ node src/index.js --config swarm.config.json
|
|
|
38
40
|
`/archives/<infohash>/ready`.
|
|
39
41
|
- **[docs/architecture-diagram.md](docs/architecture-diagram.md)** — how a publishing node, a
|
|
40
42
|
serving tier, the swarm and both kinds of client fit together.
|
|
43
|
+
- **[docs/configuration.md](docs/configuration.md)** — every setting, what it defaults to, and
|
|
44
|
+
what it costs to change.
|
|
45
|
+
- **[docs/internals.md](docs/internals.md)** — for anyone changing the code: the constraints
|
|
46
|
+
that are not visible from it, and the failures that produced them.
|
|
41
47
|
|
|
42
48
|
## What it does
|
|
43
49
|
|
|
@@ -62,6 +68,10 @@ save locations, access tokens and the external-program hooks.
|
|
|
62
68
|
torrent enclosures, so **qBittorrent's built-in RSS auto-downloader can subscribe today** with no
|
|
63
69
|
new software. Items also carry a namespaced description of the map — format, zoom range, bounds,
|
|
64
70
|
tile count — so a subscriber can decide whether it wants a 72 GiB download before starting one.
|
|
71
|
+
They carry the archive's mtime on the node that built it as well, which BitTorrent has no way to
|
|
72
|
+
transmit: a subscriber restores it when the download completes, so a mirror and its origin serve
|
|
73
|
+
the same `ETag` for the same bytes and a client reading ranges across both does not fail part-way
|
|
74
|
+
through.
|
|
65
75
|
|
|
66
76
|
**Follows feeds.** Subscribed feeds are polled and new archives added in one of two modes.
|
|
67
77
|
|
|
@@ -76,10 +86,10 @@ join the swarm directly — one URL serves both. See
|
|
|
76
86
|
|
|
77
87
|
The distinction is the point of the project.
|
|
78
88
|
|
|
79
|
-
| Mode
|
|
80
|
-
|
|
|
89
|
+
| Mode | Disk cost | What it is for |
|
|
90
|
+
| -------- | ----------------- | --------------------------------------------------------- |
|
|
81
91
|
| `mirror` | the whole archive | Becoming a full seeder and adding redundancy to the swarm |
|
|
82
|
-
| `cache`
|
|
92
|
+
| `cache` | only what is read | Serving tiles from a 700 GiB archive on a small disk |
|
|
83
93
|
|
|
84
94
|
Cache mode joins the swarm without downloading anything up front. A tile server reads byte ranges
|
|
85
95
|
on demand through [`pmtiles-torrent`](https://github.com/TechIdiots-LLC/pmtiles-torrent), and the node still
|
|
@@ -125,7 +135,9 @@ receive the public half on the catalog entry and hand it out in the TileJSON, wh
|
|
|
125
135
|
serving tier can be compromised without anyone being able to publish.
|
|
126
136
|
|
|
127
137
|
```json
|
|
128
|
-
{
|
|
138
|
+
{
|
|
139
|
+
"mutable": { "publish": true, "keyPath": "/etc/pmtiles-swarm/publisher.pem" }
|
|
140
|
+
}
|
|
129
141
|
```
|
|
130
142
|
|
|
131
143
|
The routing table is remembered between runs (`dht-nodes.json` in the data directory), which
|
|
@@ -200,8 +212,16 @@ merely guarded. See [Two ports](#two-ports).
|
|
|
200
212
|
}
|
|
201
213
|
],
|
|
202
214
|
"subscriptions": [
|
|
203
|
-
{
|
|
204
|
-
|
|
215
|
+
{
|
|
216
|
+
"url": "https://other.example.org/feed.xml",
|
|
217
|
+
"mode": "cache",
|
|
218
|
+
"filter": "terrain"
|
|
219
|
+
},
|
|
220
|
+
{
|
|
221
|
+
"url": "https://internal.example.org/api/catalog",
|
|
222
|
+
"mode": "mirror",
|
|
223
|
+
"token": "…"
|
|
224
|
+
}
|
|
205
225
|
]
|
|
206
226
|
}
|
|
207
227
|
```
|
|
@@ -242,7 +262,7 @@ data/torrents-data/7fae2931a9269684a7d4ed6e5fdd7d0014e6bcd1/planet.pmtiles
|
|
|
242
262
|
|
|
243
263
|
It works from a bare magnet, since the infohash is the one thing a magnet always carries. Flat
|
|
244
264
|
stays the default: it is what makes dropping a finished archive into the save path before adding
|
|
245
|
-
its torrent work, and it keeps a served filename readable. Archives
|
|
265
|
+
its torrent work, and it keeps a served filename readable. Archives _created_ from a local file
|
|
246
266
|
are unaffected either way — they keep the file they were made from — and web seed URLs are built
|
|
247
267
|
from the published location rather than the save path, so they do not change shape.
|
|
248
268
|
|
|
@@ -285,16 +305,17 @@ Three ways in, all editable from the Settings screen:
|
|
|
285
305
|
A `{...}` group is read as a date pattern — runs of Y, M and D with separators between them — so
|
|
286
306
|
it spells whatever the upstream spells. Run length decides padding, and case is ignored:
|
|
287
307
|
|
|
288
|
-
|
|
|
289
|
-
|
|
290
|
-
| `{YYYYMMDD}` → `20260807` | `{YYYY-MM-DD}` → `2026-08-07` | `{DD.MM.YYYY}` → `07.08.2026` | `{M}-{D}-{YY}` → `8-7-26`
|
|
291
|
-
| `{YYYY}` → `2026`
|
|
308
|
+
| | | | |
|
|
309
|
+
| ------------------------- | ----------------------------- | ----------------------------- | --------------------------- |
|
|
310
|
+
| `{YYYYMMDD}` → `20260807` | `{YYYY-MM-DD}` → `2026-08-07` | `{DD.MM.YYYY}` → `07.08.2026` | `{M}-{D}-{YY}` → `8-7-26` |
|
|
311
|
+
| `{YYYY}` → `2026` | `{YY}` → `26` | `{MM}` → `08` · `{M}` → `8` | `{DD}` → `07` · `{D}` → `7` |
|
|
292
312
|
|
|
293
313
|
A group that is not a date pattern is left exactly as found, so a URL containing `{id}` is not
|
|
294
314
|
quietly rewritten.
|
|
315
|
+
|
|
295
316
|
- **A directory** (`sources[].index`) — for an upstream whose naming is not predictable. The
|
|
296
317
|
listing is read (HTML autoindex or an S3 `ListBucketResult`), filtered, and the newest match
|
|
297
|
-
taken. Only links
|
|
318
|
+
taken. Only links _underneath_ the index URL are considered, since a listing is a document from
|
|
298
319
|
somewhere else and following an off-site link out of one would let that page decide what this
|
|
299
320
|
node downloads and republishes.
|
|
300
321
|
|
|
@@ -313,7 +334,7 @@ For anything finer than an hour, `everyMinutes` — a location a build pipeline
|
|
|
313
334
|
checking every few minutes, where a planet build published once a day does not. The tick underneath
|
|
314
335
|
is a minute, so that is the floor.
|
|
315
336
|
|
|
316
|
-
Monitored
|
|
337
|
+
Monitored _folders_ need no schedule: they are watched for filesystem events and pick up an archive
|
|
317
338
|
as it lands, once it has stopped growing for `stabilitySeconds`. The exception is a **network
|
|
318
339
|
share** — SMB and NFS do not deliver change notifications the way a local filesystem does, so a
|
|
319
340
|
watch on one can sit silent forever while files arrive. Set `pollSeconds` (15–60 suits most) to
|
|
@@ -324,7 +345,7 @@ terabyte archives every few seconds costs real I/O to learn nothing.
|
|
|
324
345
|
matters: a directory holding two years of daily planet builds would otherwise read as two years of
|
|
325
346
|
archives to fetch. Raise it only as far as the number of polls you expect to miss.
|
|
326
347
|
|
|
327
|
-
`POST /api/sources/preview` reports what a source
|
|
348
|
+
`POST /api/sources/preview` reports what a source _would_ take without taking any of it — the
|
|
328
349
|
**Preview** button in Settings — because a directory URL typed slightly wrong is otherwise
|
|
329
350
|
discovered by watching several hundred gigabytes arrive.
|
|
330
351
|
|
|
@@ -361,7 +382,7 @@ was built on: `port`, `host`, `dataDir`, `engine`, the per-engine blocks, `maxCo
|
|
|
361
382
|
|
|
362
383
|
How the node comes back depends on how it is run, and `GET /api/restart` reports which it will do
|
|
363
384
|
before anything happens. Under systemd, Docker, pm2 or Kubernetes it simply stops, because exiting
|
|
364
|
-
|
|
385
|
+
_is_ the restart there and starting a replacement would leave two processes fighting over one port.
|
|
365
386
|
Started by hand from a terminal it starts a replacement itself, because nothing else would.
|
|
366
387
|
|
|
367
388
|
### Seeding limits
|
|
@@ -385,7 +406,7 @@ holds, how rare each piece is across the swarm, and what each connected peer has
|
|
|
385
406
|
|
|
386
407
|
It earns its place here more than in an ordinary client. A **cache-mode** archive holds a scatter
|
|
387
408
|
of pieces on purpose — the ones some tile request happened to touch — so the bar is a picture of
|
|
388
|
-
what has actually been
|
|
409
|
+
what has actually been _viewed_, not a progress indicator.
|
|
389
410
|
|
|
390
411
|
```
|
|
391
412
|
GET /api/torrents/{infohash}/pieces?buckets=1000&peers=true
|
|
@@ -396,20 +417,20 @@ resolution does not survive the trip: a 698 GiB archive at 4 MiB pieces is 178,0
|
|
|
396
417
|
byte each is a quarter-megabyte per poll for a bar a thousand pixels wide. Three reductions, each
|
|
397
418
|
chosen for the question its bar answers:
|
|
398
419
|
|
|
399
|
-
| Bar
|
|
400
|
-
|
|
|
401
|
-
| Downloaded
|
|
402
|
-
| Availability | the **rarest** piece in it
|
|
403
|
-
| Each peer
|
|
420
|
+
| Bar | A column counts when | Why |
|
|
421
|
+
| ------------ | ----------------------------- | -------------------------------------------------------------------------------------------------------- |
|
|
422
|
+
| Downloaded | **every** piece in it is held | Otherwise a 60%-complete archive paints as almost solid |
|
|
423
|
+
| Availability | the **rarest** piece in it | "Can this still be completed" — one piece nobody has is the answer, however well supplied its neighbours |
|
|
424
|
+
| Each peer | **any** piece in it is held | "Where could I get this from" — a peer holding part of a column can still serve it |
|
|
404
425
|
|
|
405
|
-
`distributedCopies` is qBittorrent's
|
|
426
|
+
`distributedCopies` is qBittorrent's _Availability: 1.603_ — how many whole copies the swarm holds
|
|
406
427
|
between it. Below 1.0 means no complete copy is reachable from the peers currently connected.
|
|
407
428
|
|
|
408
429
|
Per-file piece ranges are on `/content` instead, as `firstPiece` and `pieceCount`. They need no
|
|
409
430
|
engine at all: a torrent is one byte stream cut into equal pieces, so a file's offset and length
|
|
410
431
|
already say which it occupies — which means they work for an archive nothing currently holds.
|
|
411
432
|
|
|
412
|
-
Supported by **libtorrent** and **WebTorrent**. qBittorrent's API reports piece
|
|
433
|
+
Supported by **libtorrent** and **WebTorrent**. qBittorrent's API reports piece _states_ but
|
|
413
434
|
neither availability nor per-peer maps, so the tab is refused there rather than half-drawn. On
|
|
414
435
|
WebTorrent, availability is counted from connected wires rather than read from a field, so it sees
|
|
415
436
|
a smaller sample than libtorrent's — the same shape over fewer peers.
|
|
@@ -426,7 +447,12 @@ is about the hours somebody else is using the line.
|
|
|
426
447
|
"uploadLimit": 20480000,
|
|
427
448
|
"downloadLimit": 40960000,
|
|
428
449
|
"alternative": { "uploadLimit": 2048000, "downloadLimit": 20480000 },
|
|
429
|
-
"schedule": {
|
|
450
|
+
"schedule": {
|
|
451
|
+
"enabled": true,
|
|
452
|
+
"from": "11:00",
|
|
453
|
+
"to": "22:00",
|
|
454
|
+
"days": "weekdays"
|
|
455
|
+
}
|
|
430
456
|
}
|
|
431
457
|
}
|
|
432
458
|
```
|
|
@@ -436,7 +462,7 @@ setting one thinks in, and the one qBittorrent's boxes use — and converts.
|
|
|
436
462
|
|
|
437
463
|
`days` takes `everyday`, `weekdays`, `weekends`, or a list of weekday numbers with 0 as Sunday. **A
|
|
438
464
|
window whose end is before its start wraps past midnight**, so `22:00`–`06:00` is one overnight
|
|
439
|
-
window rather than an empty one; on those, `days` picks the night the window
|
|
465
|
+
window rather than an empty one; on those, `days` picks the night the window _opens_, so a weekday
|
|
440
466
|
overnight window covers Friday night into Saturday morning and does not open again on Saturday.
|
|
441
467
|
|
|
442
468
|
The switch in the console header forces one set or the other, and hands control back to the
|
|
@@ -493,10 +519,10 @@ while the only caller is you, and stops being fine the moment another node wants
|
|
|
493
519
|
|
|
494
520
|
So there are named tokens, minted in **Settings → Access tokens** or at `POST /api/tokens`:
|
|
495
521
|
|
|
496
|
-
| role
|
|
497
|
-
|
|
498
|
-
| `peer`
|
|
499
|
-
| `admin` | everything the console can do.
|
|
522
|
+
| role | can |
|
|
523
|
+
| ------- | -------------------------------------------------------------------------------------------------------------------------------- |
|
|
524
|
+
| `peer` | read this node — its catalogue, feeds, tiles and `.torrent` files. What another swarm node needs to follow it, and nothing else. |
|
|
525
|
+
| `admin` | everything the console can do. |
|
|
500
526
|
|
|
501
527
|
One per person or node, so any of them can be revoked without disturbing the rest. A peer token can
|
|
502
528
|
also be narrowed to a list of categories, and then sees exactly those — not even what this node
|
|
@@ -586,7 +612,7 @@ directory is a URL that answers with a half-written archive, and every peer that
|
|
|
586
612
|
hash verification. With the marker, the URL 404s until the exact moment the file is real.
|
|
587
613
|
|
|
588
614
|
The rename is within one directory, so it is atomic and instant however large the archive is.
|
|
589
|
-
Keeping incomplete files in a
|
|
615
|
+
Keeping incomplete files in a _different_ directory would mean a completed download had to move,
|
|
590
616
|
which is instant only when both paths share a filesystem and otherwise copies the whole archive.
|
|
591
617
|
`cacheSavePath` remains available for anyone who wants cache-mode pieces on separate disk, but it
|
|
592
618
|
is now a placement choice rather than how completeness is recorded, and it ships unset.
|
|
@@ -671,8 +697,16 @@ curl -s -H "authorization: Bearer $KEY" http://172.16.1.49:8091/api/stats | jq
|
|
|
671
697
|
}
|
|
672
698
|
},
|
|
673
699
|
"recent": [
|
|
674
|
-
{
|
|
675
|
-
"
|
|
700
|
+
{
|
|
701
|
+
"at": "…",
|
|
702
|
+
"ip": "172.16.1.41",
|
|
703
|
+
"z": 14,
|
|
704
|
+
"x": 4823,
|
|
705
|
+
"y": 6155,
|
|
706
|
+
"status": 200,
|
|
707
|
+
"bytes": 41221,
|
|
708
|
+
"ms": 2
|
|
709
|
+
}
|
|
676
710
|
]
|
|
677
711
|
}
|
|
678
712
|
```
|
|
@@ -703,51 +737,51 @@ which the endpoint answers 501.
|
|
|
703
737
|
|
|
704
738
|
## API
|
|
705
739
|
|
|
706
|
-
| Method
|
|
707
|
-
|
|
|
708
|
-
| `GET`
|
|
709
|
-
| `GET`
|
|
710
|
-
| `POST`
|
|
711
|
-
| `DELETE`
|
|
712
|
-
| `GET`
|
|
713
|
-
| `GET`
|
|
714
|
-
| `GET`
|
|
715
|
-
| `GET`
|
|
716
|
-
| `PATCH`
|
|
717
|
-
| `PATCH`
|
|
718
|
-
| `PATCH`
|
|
719
|
-
| `PATCH` `GET`
|
|
720
|
-
| `POST`
|
|
721
|
-
| `POST`
|
|
722
|
-
| `POST`
|
|
723
|
-
| `DELETE`
|
|
724
|
-
| `POST`
|
|
725
|
-
| `POST`
|
|
726
|
-
| `POST`
|
|
727
|
-
| `GET` `DELETE`
|
|
728
|
-
| `GET` `POST`
|
|
729
|
-
| `GET`
|
|
730
|
-
| `POST`
|
|
731
|
-
| `POST`
|
|
732
|
-
| `POST`
|
|
733
|
-
| `POST`
|
|
734
|
-
| `POST`
|
|
735
|
-
| `POST`
|
|
736
|
-
| `GET` `POST` `DELETE` | `/api/tokens`, `/api/tokens/:id`
|
|
737
|
-
| `GET` `POST`
|
|
738
|
-
| `GET` `DELETE`
|
|
739
|
-
| `GET` `PATCH`
|
|
740
|
-
| `POST`
|
|
741
|
-
| `GET`
|
|
742
|
-
| `GET`
|
|
743
|
-
| `GET`
|
|
744
|
-
| `GET`
|
|
745
|
-
| `GET`
|
|
746
|
-
| `GET`
|
|
747
|
-
| `GET`
|
|
748
|
-
| `GET`
|
|
749
|
-
| `GET`
|
|
750
|
-
| `GET`
|
|
740
|
+
| Method | Path | Purpose |
|
|
741
|
+
| --------------------- | ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
742
|
+
| `GET` | `/api/status` | Engine health, counts, watched folders, save locations and free space |
|
|
743
|
+
| `GET` | `/api/torrents` | Catalog joined with live swarm state and what is left of each seeding limit |
|
|
744
|
+
| `POST` | `/api/torrents` | Add via `{path}`, `{url}`, `{magnet}`, `{torrentUrl}`, or a raw `.torrent` body |
|
|
745
|
+
| `DELETE` | `/api/torrents/:infoHash` | Remove (`?deleteData=true` to delete data too) |
|
|
746
|
+
| `GET` | `/api/torrents/:infoHash` | One archive, with disk usage and how it is being read |
|
|
747
|
+
| `GET` | `/api/torrents/:infoHash/file`, `/magnet` | Download the `.torrent`, or its magnet URI |
|
|
748
|
+
| `GET` | `/api/torrents/:infoHash/peers`, `/trackers`, `/content` | Per-peer, per-tracker and per-file detail |
|
|
749
|
+
| `GET` | `/api/torrents/:infoHash/pieces` | Which pieces are held, how rare each is, and what peers hold |
|
|
750
|
+
| `PATCH` | `/api/torrents/:infoHash/mode` | Switch between mirror and cache |
|
|
751
|
+
| `PATCH` | `/api/torrents/:infoHash/categories` | Set, add or remove categories |
|
|
752
|
+
| `PATCH` | `/api/torrents/:infoHash/seeding` | Per-archive seeding limit, or "use the global one" |
|
|
753
|
+
| `PATCH` `GET` | `/api/torrents/:infoHash/location` | Move the data; poll the move |
|
|
754
|
+
| `POST` | `/api/torrents/:infoHash/pause`, `/resume` | Stop offering it, without forgetting it |
|
|
755
|
+
| `POST` | `/api/torrents/:infoHash/webseeds` | Add web seeds — does not change the infohash |
|
|
756
|
+
| `POST` | `/api/torrents/:infoHash/warm` | Pre-fetch a region (`GET` for progress, `DELETE` to cancel) |
|
|
757
|
+
| `DELETE` | `/api/torrents/:infoHash/cache` | Reclaim cached pieces, keep the archive |
|
|
758
|
+
| `POST` | `/api/torrents/:infoHash/check` | Has the source changed since the torrent was made? |
|
|
759
|
+
| `POST` | `/api/torrents/:infoHash/rebuild` | Rebuild from the current source (mints a new infohash) |
|
|
760
|
+
| `POST` | `/api/check-origins` | Check every archive with a watchable source |
|
|
761
|
+
| `GET` `DELETE` | `/api/adds` | Downloads still in flight, and cancelling one by URL |
|
|
762
|
+
| `GET` `POST` | `/api/speed` | Which speed limits are in force, and the manual switch between the two sets |
|
|
763
|
+
| `GET` | `/api/categories` | Every category, with the endpoints resolving to its newest build |
|
|
764
|
+
| `POST` | `/api/adopt`, `/api/adopt/candidates` | Take over what an engine or another node holds |
|
|
765
|
+
| `POST` | `/api/sources/preview` | What a watched web location would take, without taking it |
|
|
766
|
+
| `POST` | `/api/subscriptions/preview` | Whether a peer is reachable and what it offers |
|
|
767
|
+
| `POST` | `/api/subscriptions/refresh` | Poll subscribed feeds now |
|
|
768
|
+
| `POST` | `/api/sources/check` | Check scheduled sources now, rather than at the next due time |
|
|
769
|
+
| `POST` | `/api/torrents/:infoHash/hooks/complete` | Run the completion hook again for one archive |
|
|
770
|
+
| `GET` `POST` `DELETE` | `/api/tokens`, `/api/tokens/:id` | Mint, list and revoke access tokens |
|
|
771
|
+
| `GET` `POST` | `/api/restart` | What a restart would do, and doing it |
|
|
772
|
+
| `GET` `DELETE` | `/api/stats` | What this node has served — per-archive counters and the last N requests; `DELETE` clears them |
|
|
773
|
+
| `GET` `PATCH` | `/api/config` | Read and change settings |
|
|
774
|
+
| `POST` | `/api/login`, `/api/logout` | Console sign-in |
|
|
775
|
+
| `GET` | `/api/session` | Who this request is, and whether a credential is needed at all |
|
|
776
|
+
| `GET` | `/api/catalog` | The whole catalogue, for a peer keeping itself in step |
|
|
777
|
+
| `GET` | `/archives/:infoHash/tiles.json` | TileJSON — **public** |
|
|
778
|
+
| `GET` | `/archives/:infoHash/:z/:x/:y.:ext` | One tile — **public** |
|
|
779
|
+
| `GET` | `/health` | 200 when this node can serve, 503 when its engine cannot — **public**, no credential, for a load balancer |
|
|
780
|
+
| `GET` | `/archives/:infoHash/ready` | Whether this node can serve _this_ archive yet: 200 ready, 503 not yet, 415 never — **public** |
|
|
781
|
+
| `GET` | `/archives/:infoHash/archive.torrent` | The `.torrent` a torrent-aware client joins with — **public** |
|
|
782
|
+
| `GET` | `/archives/:infoHash/preview` | Map preview for one archive — admin, not public |
|
|
783
|
+
| `GET` | `/latest/:category/tiles.json`, `/:name.torrent`, `/magnet` | The newest build in a category — **public**. The torrent name is yours to choose, so a link can read `planetiler-openmaptiles-latest.torrent`; it redirects to the immutable URL, which names the download after the build it actually is |
|
|
784
|
+
| `GET` | `/feed.xml`, `/feed/:category.xml`, `/latest/:category.xml` | RSS — **public** |
|
|
751
785
|
|
|
752
786
|
Everything under `/api/` is guarded once a credential is configured; tiles, TileJSON and the feeds
|
|
753
787
|
never are. A `peer` token may read but not change, and may be narrowed to some categories. See
|
|
@@ -17,7 +17,7 @@ One node publishes, any number serve, everything meets in the swarm.
|
|
|
17
17
|
**"Primary" and "secondary" are roles, not node types.** The only thing that makes
|
|
18
18
|
a node primary is that it creates torrents and owns the catalog others subscribe
|
|
19
19
|
to. It can sit in the load-balanced pool alongside everything else — it holds
|
|
20
|
-
complete copies, so it is the
|
|
20
|
+
complete copies, so it is the _best_ tile server in the fleet, not an exception
|
|
21
21
|
to it. The diagram separates them only to keep the arrows legible.
|
|
22
22
|
|
|
23
23
|
Likewise **mirror and cache are per-archive choices, not node identities.** A
|
|
@@ -74,7 +74,7 @@ graph LR
|
|
|
74
74
|
```
|
|
75
75
|
|
|
76
76
|
**Key points:** **orange = BitTorrent, green = RSS distribution, plain = HTTP.**
|
|
77
|
-
The publisher is the only node that
|
|
77
|
+
The publisher is the only node that _creates_ torrents; the rest learn about
|
|
78
78
|
archives from its feed and join the swarm, each choosing per archive whether to
|
|
79
79
|
mirror it or cache it.
|
|
80
80
|
|
|
@@ -147,7 +147,7 @@ A node holding a complete copy skips all of this and reads its local file.
|
|
|
147
147
|
|
|
148
148
|
## 3. What a torrent-aware client does
|
|
149
149
|
|
|
150
|
-
The server-side path above is what an
|
|
150
|
+
The server-side path above is what an _ordinary_ client triggers. A torrent-aware
|
|
151
151
|
client does something different: it takes over the archive reading itself, and
|
|
152
152
|
stops needing the tile endpoint at all.
|
|
153
153
|
|
|
@@ -206,7 +206,7 @@ Three consequences worth being clear about:
|
|
|
206
206
|
- **HTTP is never fully abandoned.** It is the fallback for anything the swarm
|
|
207
207
|
cannot answer quickly, and the only path until the swarm is connected.
|
|
208
208
|
- **The client becomes a seeder.** Every piece it pulls, it serves — so a popular
|
|
209
|
-
region gets
|
|
209
|
+
region gets _faster_ as more clients view it, which is the opposite of how a
|
|
210
210
|
tile server behaves under load.
|
|
211
211
|
- **Prefer the `.torrent` over the magnet.** A magnet carries only an infohash, so
|
|
212
212
|
the client must find peers and complete a metadata exchange before it knows
|
|
@@ -255,7 +255,7 @@ remember where it had got to.
|
|
|
255
255
|
|
|
256
256
|
### Bootstrapping without the server
|
|
257
257
|
|
|
258
|
-
A torrent-aware client still has to
|
|
258
|
+
A torrent-aware client still has to _learn_ the magnet from somewhere, and until
|
|
259
259
|
it does, the swarm — the part that depends on no server — is unreachable
|
|
260
260
|
precisely when the server is down. The fix is that the magnet travels in the
|
|
261
261
|
**fragment** of the TileJSON URL a style already carries:
|
|
@@ -277,11 +277,11 @@ string does not go stale on the next build either. See
|
|
|
277
277
|
**Decide how absolute URLs get built.** TileJSON and the RSS feed both contain
|
|
278
278
|
absolute URLs, and there are three ways to arrive at them:
|
|
279
279
|
|
|
280
|
-
| Config
|
|
281
|
-
|
|
|
282
|
-
| `publicUrl` set
|
|
280
|
+
| Config | Behaviour | Use when |
|
|
281
|
+
| ---------------- | ------------------------------------------------------------ | -------------------------------------------- |
|
|
282
|
+
| `publicUrl` set | One canonical URL, whatever the request said | There is a single public name |
|
|
283
283
|
| `trustProxy` set | Per request, from `X-Forwarded-Proto` and `X-Forwarded-Host` | One node answers on several names or schemes |
|
|
284
|
-
| neither
|
|
284
|
+
| neither | From the connection itself | Direct access, no proxy |
|
|
285
285
|
|
|
286
286
|
```json
|
|
287
287
|
{
|
|
@@ -317,7 +317,7 @@ between them.
|
|
|
317
317
|
```
|
|
318
318
|
|
|
319
319
|
Bound to loopback like that, the thing that can rewrite configuration is
|
|
320
|
-
|
|
320
|
+
_unreachable_ rather than merely guarded, and the pool in front of the public
|
|
321
321
|
port carries no route to it at all. **Peer traffic never touches the balancer** —
|
|
322
322
|
neither BitTorrent nor WebRTC — so size it for tiles alone, and see
|
|
323
323
|
[ports and reachability](engines.md#ports-and-reachability) for the peer ports,
|
|
@@ -334,14 +334,14 @@ noticing.
|
|
|
334
334
|
reads its local file and is fast from the first request. If every serving node
|
|
335
335
|
mirrors, this section does not apply to you at all.
|
|
336
336
|
|
|
337
|
-
Where nodes
|
|
337
|
+
Where nodes _do_ run in cache mode, each warms its own piece cache, so scattering
|
|
338
338
|
requests for one region across N nodes costs N first-fetches rather than one.
|
|
339
339
|
Three things reduce that, in order of effect:
|
|
340
340
|
|
|
341
341
|
1. **The CDN absorbs repeats.** Tiles are immutable, so the second request for a
|
|
342
342
|
tile never reaches any node.
|
|
343
343
|
2. **The nodes are peers in the same swarm.** A node fetching a piece a sibling
|
|
344
|
-
already holds gets it
|
|
344
|
+
already holds gets it _from that sibling_, usually over the LAN. Local service
|
|
345
345
|
discovery is on by default, so same-subnet nodes find each other without
|
|
346
346
|
configuration. The cost is one external fetch plus N−1 local ones, not N
|
|
347
347
|
external ones.
|