pmtiles-swarm 0.12.0 → 0.13.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +59 -0
- package/docs/architecture-diagram.md +31 -3
- package/docs/running-as-a-service.md +128 -1
- package/docs/security.md +50 -0
- package/docs/serving-tiles.md +37 -5
- package/package.json +1 -1
- package/src/config.js +13 -0
- package/src/engines/composite.js +45 -1
- package/src/index.js +9 -0
- package/src/library.js +40 -8
- package/src/publisher.js +24 -0
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,65 @@
|
|
|
7
7
|
### 🐞 Bug fixes
|
|
8
8
|
- _...Add new stuff here..._
|
|
9
9
|
|
|
10
|
+
## 0.13.1
|
|
11
|
+
### 🐞 Bug fixes
|
|
12
|
+
- **A node with a secondary engine no longer takes a quarter of an hour to start listening.**
|
|
13
|
+
Handing an archive to a second seeding client makes it hash every byte before it will serve any
|
|
14
|
+
— minutes for tens of gigabytes — and that hand-over was **awaited** inside `add()`. On startup,
|
|
15
|
+
where the library is restored one archive at a time, the cost landed end to end before the node
|
|
16
|
+
would bind its port. A seventeen-archive library sat silent for about fifteen minutes.
|
|
17
|
+
|
|
18
|
+
The hand-over is now queued and the caller carries on. Nothing depended on waiting for it: the
|
|
19
|
+
periodic sweep that already exists to catch archives finishing later is the same mechanism, and
|
|
20
|
+
a failed hand-over already un-marks itself so that sweep retries it.
|
|
21
|
+
|
|
22
|
+
Queued through one chain rather than fired off freely, so hand-overs still run one at a time —
|
|
23
|
+
seventeen archives hashing at once on a spinning disk is slower than seventeen in turn, and much
|
|
24
|
+
harder to reason about.
|
|
25
|
+
- **Restoring a large library reports progress.** It said nothing at all until it had finished, so
|
|
26
|
+
a node that was working and a node that was stuck looked identical for as long as it took — long
|
|
27
|
+
enough, on a real library, to go looking for a debugger:
|
|
28
|
+
|
|
29
|
+
```
|
|
30
|
+
[restore] 6 of 17 after 15s
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
On a timer rather than per archive, so a small library stays quiet.
|
|
34
|
+
|
|
35
|
+
## 0.13.0
|
|
36
|
+
### ✨ Features and improvements
|
|
37
|
+
- **The publisher's DHT socket is bound explicitly, on a configurable port.** It was left to bind
|
|
38
|
+
implicitly on its first send, which works but takes an unpredictable ephemeral port and reports
|
|
39
|
+
nothing — so there was no way to forward it, and no way to tell which one it had. `mutable.dhtPort`
|
|
40
|
+
now sets it (`0`, ephemeral, by default) and the port is logged at startup.
|
|
41
|
+
|
|
42
|
+
Publishing needs no forward either way: a put is outbound, and the replies come back on the same
|
|
43
|
+
socket the way any UDP client's do. Pinning and forwarding one makes this a *reachable* DHT node
|
|
44
|
+
instead, which earns a better routing table and contributes back — worth having on a node that
|
|
45
|
+
runs continuously.
|
|
46
|
+
|
|
47
|
+
It must not collide with an engine's port. Each seeding engine runs a DHT of its own, so a node
|
|
48
|
+
with both has three UDP participants and only this one is placed here; two sockets cannot hold
|
|
49
|
+
one port and the node would fail to start.
|
|
50
|
+
|
|
51
|
+
### 📚 Documentation
|
|
52
|
+
- **[docs/running-as-a-service.md](docs/running-as-a-service.md) covers the publisher key**:
|
|
53
|
+
generating it as the service account so ownership is right, `chmod 400` because nothing ever
|
|
54
|
+
writes it back, and why it must be backed up off the machine — losing it breaks every style
|
|
55
|
+
pointing at that public key, permanently, with no reissue. Also what happens under HA config
|
|
56
|
+
sync: the configuration replicates to the standby and the key does not, so the standby logs
|
|
57
|
+
`not publishing: ENOENT` and serves on, which is the intended outcome rather than a fault. And
|
|
58
|
+
how to confirm it works, including that `nodes: 0` in the log means nobody stored the record
|
|
59
|
+
however healthy the rest of the line looks.
|
|
60
|
+
- **The ports table lists the DHT port**, which had only been described in the publisher section —
|
|
61
|
+
not where anyone looks when deciding what to forward.
|
|
62
|
+
- **Why publishing does not reuse an engine's DHT** is recorded in
|
|
63
|
+
[src/publisher.js](src/publisher.js), since it will be asked again. libtorrent's is unreachable:
|
|
64
|
+
the 2.x Python bindings expose neither `dht_put_item` nor `dht_get_item`, though the alerts are
|
|
65
|
+
bound, so the C++ side supports BEP 44 and there is simply no method to call. WebTorrent's *is*
|
|
66
|
+
`bittorrent-dht` and could be reused to save a socket; that is a deliberate choice rather than an
|
|
67
|
+
oversight, taken to keep one code path that behaves the same whichever engine is configured.
|
|
68
|
+
|
|
10
69
|
## 0.12.0
|
|
11
70
|
### ✨ Features and improvements
|
|
12
71
|
- **A category can now be addressed without a server at all.** A node that builds can publish a
|
|
@@ -224,7 +224,7 @@ New data means a new infohash, which is what makes cache invalidation free.
|
|
|
224
224
|
%%{init: {"flowchart": {"rankSpacing": 60}}}%%
|
|
225
225
|
graph LR
|
|
226
226
|
NEW["new planet.pmtiles<br/><small>weekly build</small>"] --> HASH["primary re-hashes<br/>→ new infohash"]
|
|
227
|
-
HASH --> BEP["BEP 46 mutable entry<br/><small>same public key, seq
|
|
227
|
+
HASH --> BEP["BEP 46 mutable entry<br/><small>same public key, new seq</small>"]
|
|
228
228
|
HASH --> FEED["new RSS item"]
|
|
229
229
|
|
|
230
230
|
BEP --> FOLLOW["clients following the key<br/>see the new version"]
|
|
@@ -239,8 +239,36 @@ graph LR
|
|
|
239
239
|
**Key points:** tile URLs are content-addressed, so they never need invalidating.
|
|
240
240
|
The old infohash stays valid and servable for as long as anyone still holds it —
|
|
241
241
|
useful for clients pinned to a known-good build — while new requests move to the
|
|
242
|
-
new one as soon as they re-read `tiles.json`.
|
|
243
|
-
|
|
242
|
+
new one as soon as they re-read `tiles.json`.
|
|
243
|
+
|
|
244
|
+
**There are exactly two mutable things**, and they say the same thing by
|
|
245
|
+
different means: the `/latest/<category>/` documents, which need this server, and
|
|
246
|
+
the BEP 46 record, which does not. A style pointing at the public key resolves
|
|
247
|
+
the current build over the DHT with nothing of ours running at all. Only the node
|
|
248
|
+
that builds signs those records — see
|
|
249
|
+
[security.md](security.md#the-publisher-key-is-not-a-credential), because the key
|
|
250
|
+
behaves unlike every other secret in the system.
|
|
251
|
+
|
|
252
|
+
The sequence number is derived from the clock rather than incremented, so a
|
|
253
|
+
publisher that is rebuilt or restored from backup carries on without needing to
|
|
254
|
+
remember where it had got to.
|
|
255
|
+
|
|
256
|
+
### Bootstrapping without the server
|
|
257
|
+
|
|
258
|
+
A torrent-aware client still has to *learn* the magnet from somewhere, and until
|
|
259
|
+
it does, the swarm — the part that depends on no server — is unreachable
|
|
260
|
+
precisely when the server is down. The fix is that the magnet travels in the
|
|
261
|
+
**fragment** of the TileJSON URL a style already carries:
|
|
262
|
+
|
|
263
|
+
```
|
|
264
|
+
https://swarm.example.org/latest/openmaptiles/tiles.json#magnet:?xs=urn:btpk:…&s=openmaptiles&ws=…
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
A fragment is never sent in an HTTP request, so ordinary clients fetch the
|
|
268
|
+
TileJSON and ignore it while a swarm-aware one reads the magnet before making any
|
|
269
|
+
call. With a BEP 46 key in it (`xs=urn:btpk:`) rather than an infohash, that
|
|
270
|
+
string does not go stale on the next build either. See
|
|
271
|
+
[serving-tiles.md](serving-tiles.md#a-fragment-that-survives-a-rebuild).
|
|
244
272
|
|
|
245
273
|
---
|
|
246
274
|
|
|
@@ -354,6 +354,116 @@ What the running service actually has:
|
|
|
354
354
|
systemctl show -p ReadWritePaths -p UMask -p SupplementaryGroups pmtiles-swarm
|
|
355
355
|
```
|
|
356
356
|
|
|
357
|
+
## The publisher key
|
|
358
|
+
|
|
359
|
+
Only needed if this node publishes BEP 46 records — the signed DHT entries that
|
|
360
|
+
let a style point at a category rather than at a build that goes stale. Skip
|
|
361
|
+
this section entirely if it does not build archives.
|
|
362
|
+
|
|
363
|
+
**Generate it as the service account**, so the file is owned by the user that
|
|
364
|
+
has to read it:
|
|
365
|
+
|
|
366
|
+
```sh
|
|
367
|
+
sudo -u pmtiles-swarm -H /var/lib/pmtiles-swarm/node_modules/.bin/pmtiles-swarm publisher-key > /etc/pmtiles-swarm/publisher.pem
|
|
368
|
+
sudo chown pmtiles-swarm:pmtiles-swarm /etc/pmtiles-swarm/publisher.pem
|
|
369
|
+
sudo chmod 400 /etc/pmtiles-swarm/publisher.pem
|
|
370
|
+
```
|
|
371
|
+
|
|
372
|
+
The PEM goes to stdout and the public key to stderr, so the redirect above
|
|
373
|
+
captures only the key material and you still see the public half on the
|
|
374
|
+
terminal. Write that public key down — it is what subscribers point at, and it
|
|
375
|
+
is the one part you will want later.
|
|
376
|
+
|
|
377
|
+
`400` rather than `600`: the service only ever reads this. Nothing in the
|
|
378
|
+
product writes it back, so removing write permission costs nothing and means a
|
|
379
|
+
compromised process cannot quietly replace it.
|
|
380
|
+
|
|
381
|
+
Then in `/etc/pmtiles-swarm/swarm.config.json`:
|
|
382
|
+
|
|
383
|
+
```json
|
|
384
|
+
{
|
|
385
|
+
"mutable": {
|
|
386
|
+
"publish": true,
|
|
387
|
+
"keyPath": "/etc/pmtiles-swarm/publisher.pem"
|
|
388
|
+
}
|
|
389
|
+
}
|
|
390
|
+
```
|
|
391
|
+
|
|
392
|
+
`/etc/pmtiles-swarm` is already in `ReadWritePaths` because the console rewrites
|
|
393
|
+
the configuration there, so the unit needs no change.
|
|
394
|
+
|
|
395
|
+
### Back it up, off this machine
|
|
396
|
+
|
|
397
|
+
Losing this file breaks **every style pointing at that public key, permanently**
|
|
398
|
+
— there is no recovery, no reissue, and no way to prove to a subscriber that a
|
|
399
|
+
new key is you. It is not like an API key you can rotate.
|
|
400
|
+
|
|
401
|
+
Back it up somewhere that is not this disk, and treat the backup as seriously as
|
|
402
|
+
the original: whoever holds it can publish a signed record telling your
|
|
403
|
+
subscribers that any archive is the current build, and clients will believe it
|
|
404
|
+
because the signature checks out.
|
|
405
|
+
|
|
406
|
+
### Exactly one publisher
|
|
407
|
+
|
|
408
|
+
Two nodes publishing under one key fight over the sequence number, each
|
|
409
|
+
overwriting the other's claim about what is current. So the PEM belongs on the
|
|
410
|
+
build node and nowhere else.
|
|
411
|
+
|
|
412
|
+
**This matters if you run HA config sync.** The configuration will replicate to
|
|
413
|
+
the standby, `publish: true` and all — but the PEM will not, because you are
|
|
414
|
+
not going to copy it. The standby then logs
|
|
415
|
+
|
|
416
|
+
```
|
|
417
|
+
[mutable] not publishing: ENOENT: no such file or directory
|
|
418
|
+
```
|
|
419
|
+
|
|
420
|
+
on every start and carries on serving normally. That is the intended outcome
|
|
421
|
+
rather than a fault: the config syncing is harmless, and the key not syncing is
|
|
422
|
+
the point. If the noise bothers you, set `mutable.publish` to `false` in the
|
|
423
|
+
standby's config after the sync.
|
|
424
|
+
|
|
425
|
+
### It does not need a port forwarded
|
|
426
|
+
|
|
427
|
+
The DHT socket is separate from the seeding engine's, and by default takes an
|
|
428
|
+
ephemeral UDP port (`mutable.dhtPort: 0`). That is enough to publish: a put is
|
|
429
|
+
outbound — find nodes, then send — and the replies come back on the same
|
|
430
|
+
socket the way any UDP client's do, which NAT handles without help.
|
|
431
|
+
|
|
432
|
+
Setting a fixed port and forwarding it makes this a *reachable* DHT node, which
|
|
433
|
+
means better lookups and contributing back to the network. Worth doing if this
|
|
434
|
+
node is long-lived, but nothing here requires it.
|
|
435
|
+
|
|
436
|
+
**Do not reuse the libtorrent engine's port.** That engine runs a DHT of its own
|
|
437
|
+
on `libtorrent.listen`, and two sockets cannot hold one port — the node would
|
|
438
|
+
fail to start.
|
|
439
|
+
|
|
440
|
+
The port in use is reported at startup:
|
|
441
|
+
|
|
442
|
+
```
|
|
443
|
+
[mutable] DHT on UDP 63213
|
|
444
|
+
```
|
|
445
|
+
|
|
446
|
+
### Confirming it works
|
|
447
|
+
|
|
448
|
+
```sh
|
|
449
|
+
journalctl -u pmtiles-swarm | grep mutable
|
|
450
|
+
```
|
|
451
|
+
|
|
452
|
+
A healthy publisher says which categories it is announcing and under which key
|
|
453
|
+
at startup, then one line per category whenever a build moves:
|
|
454
|
+
|
|
455
|
+
```
|
|
456
|
+
[mutable] publishing 3 categories as 7680dc95248eb807… every 30m
|
|
457
|
+
[mutable] openmaptiles -> 4813a0e68e4b (seq 1786108931, 8 nodes)
|
|
458
|
+
```
|
|
459
|
+
|
|
460
|
+
`nodes` is how many DHT peers stored the record. **Zero means nobody did**, and
|
|
461
|
+
the record does not exist however healthy the log looks otherwise — check that
|
|
462
|
+
UDP is not blocked and that the DHT is reachable.
|
|
463
|
+
|
|
464
|
+
The first publish waits about fifteen seconds after start, because a put into a
|
|
465
|
+
DHT that has not finished bootstrapping reaches nobody.
|
|
466
|
+
|
|
357
467
|
## The sidecar
|
|
358
468
|
|
|
359
469
|
The libtorrent engine runs Python as a child process, so the service user needs
|
|
@@ -373,16 +483,33 @@ rather than exiting. Name the interpreter explicitly when the service user's
|
|
|
373
483
|
|
|
374
484
|
## Ports
|
|
375
485
|
|
|
376
|
-
|
|
486
|
+
Five listeners, and only the peer ports want a firewall rule. See
|
|
377
487
|
[ports and reachability](engines.md#ports-and-reachability) for the detail.
|
|
378
488
|
|
|
379
489
|
| | |
|
|
380
490
|
| --- | --- |
|
|
381
491
|
| `libtorrent.listen` — 6881, TCP and UDP | forward it |
|
|
382
492
|
| `webtorrent.clientOptions.torrentPort` — pin it, or it changes every start | forward it |
|
|
493
|
+
| `mutable.dhtPort` — UDP, ephemeral by default | optional; see below |
|
|
383
494
|
| `port` — 8090 | your proxy or CDN |
|
|
384
495
|
| `adminPort` — 8091, bound to `127.0.0.1` | nothing; that is the point |
|
|
385
496
|
|
|
497
|
+
`mutable.dhtPort` is the odd one. Publishing works without any forward, because
|
|
498
|
+
a put is outbound and the replies come back on the same socket the way any UDP
|
|
499
|
+
client's do. Pin it and forward it only if you want this to be a *reachable*
|
|
500
|
+
DHT node — which earns a better routing table and contributes back, and is
|
|
501
|
+
worth having on a node that runs continuously.
|
|
502
|
+
|
|
503
|
+
Pin it to something free: **not** `libtorrent.listen`, which is a DHT of its
|
|
504
|
+
own, and not `torrentPort`. `6883` sits clear of both.
|
|
505
|
+
|
|
506
|
+
```json
|
|
507
|
+
{ "mutable": { "dhtPort": 6883 } }
|
|
508
|
+
```
|
|
509
|
+
|
|
510
|
+
Note that each engine runs its own DHT as well, so a node with both engines has
|
|
511
|
+
three UDP participants. Only this one is yours to place.
|
|
512
|
+
|
|
386
513
|
## Updating
|
|
387
514
|
|
|
388
515
|
```sh
|
package/docs/security.md
CHANGED
|
@@ -188,6 +188,56 @@ Three ways past it, in order of preference:
|
|
|
188
188
|
2. Bind to `127.0.0.1` and reach it through a reverse proxy that authenticates.
|
|
189
189
|
3. Set `allowUnauthenticated: true`, if the network really is trusted.
|
|
190
190
|
|
|
191
|
+
## The publisher key is not a credential
|
|
192
|
+
|
|
193
|
+
Everything above guards *this node*: who may read the API, who may change
|
|
194
|
+
settings, who may add an archive. The publisher key is a different kind of
|
|
195
|
+
secret, and the difference is worth stating plainly because the habits do not
|
|
196
|
+
carry over.
|
|
197
|
+
|
|
198
|
+
`mutable.keyPath` holds an ed25519 private key used to sign BEP 46 records — the
|
|
199
|
+
DHT entries that say which archive is the current build of a category. It is a
|
|
200
|
+
**signing key**, closer to a code-signing certificate than to an API token.
|
|
201
|
+
|
|
202
|
+
| | An access credential (`apiKey`, a token, `password`) | The publisher key |
|
|
203
|
+
| --- | --- | --- |
|
|
204
|
+
| What it grants | access to one node | authority over what your subscribers believe |
|
|
205
|
+
| If it leaks | that node is compromised | anyone can sign a record pointing your subscribers at any archive |
|
|
206
|
+
| If you lose it | mint another | every style pointing at that public key breaks, permanently |
|
|
207
|
+
| Rotation | revoke and reissue | there is none |
|
|
208
|
+
|
|
209
|
+
That third row is the one people are unprepared for. A public key **is** the
|
|
210
|
+
identity: there is no registry to update and no way to tell a subscriber that a
|
|
211
|
+
new key is also you. Losing the file ends that identity.
|
|
212
|
+
|
|
213
|
+
And the second row is why it does not belong on a serving tier. A leaked API key
|
|
214
|
+
lets someone read a node. A leaked publisher key lets someone tell every
|
|
215
|
+
subscriber that an archive they control is your latest planet build — signed,
|
|
216
|
+
verifying correctly, and indistinguishable from a real announcement.
|
|
217
|
+
|
|
218
|
+
**So it lives on exactly one machine.** Only the node that builds needs it;
|
|
219
|
+
serving nodes carry the public half on the catalog entry and hand it out in the
|
|
220
|
+
TileJSON, which is why a serving tier can be compromised without anyone being
|
|
221
|
+
able to publish. Running two publishers under one key is separately a mistake:
|
|
222
|
+
they fight over the sequence number.
|
|
223
|
+
|
|
224
|
+
Practically:
|
|
225
|
+
|
|
226
|
+
- `chmod 400`, owned by the service account. Nothing in the product writes it
|
|
227
|
+
back, so it does not need to be writable and should not be.
|
|
228
|
+
- Back it up **off the machine**, and treat the backup as seriously as the
|
|
229
|
+
original.
|
|
230
|
+
- Do not put it in an image, a config-sync set, or anywhere a standby will
|
|
231
|
+
receive it. Under HA sync the configuration replicates and the key does not,
|
|
232
|
+
which is the intended asymmetry rather than an oversight — the standby logs
|
|
233
|
+
`not publishing: ENOENT` and serves normally.
|
|
234
|
+
- It is unrelated to `auth`. Turning authentication off does not expose it, and
|
|
235
|
+
rotating the API key does not affect it.
|
|
236
|
+
|
|
237
|
+
Generate one with `pmtiles-swarm publisher-key`. Setup detail is in
|
|
238
|
+
[running-as-a-service.md](running-as-a-service.md#the-publisher-key); what it is
|
|
239
|
+
*for* is in [serving-tiles.md](serving-tiles.md#a-fragment-that-survives-a-rebuild).
|
|
240
|
+
|
|
191
241
|
## What this is not
|
|
192
242
|
|
|
193
243
|
**It is not a defence against someone on your network.** Sessions are bearer
|
package/docs/serving-tiles.md
CHANGED
|
@@ -291,7 +291,13 @@ TileJSON documents from this server carry a non-standard `torrent` member:
|
|
|
291
291
|
"torrent": "https://swarm.example.org/archives/913d…/archive.torrent",
|
|
292
292
|
"name": "planet.pmtiles",
|
|
293
293
|
"size": 77242531840,
|
|
294
|
-
"webseeds": ["https://maps.example.org/planet.pmtiles"]
|
|
294
|
+
"webseeds": ["https://maps.example.org/planet.pmtiles"],
|
|
295
|
+
"mutable": {
|
|
296
|
+
"publicKey": "7680dc95248eb807…",
|
|
297
|
+
"salt": "openmaptiles",
|
|
298
|
+
"seq": 1786108931,
|
|
299
|
+
"magnet": "magnet:?xs=urn:btpk:7680dc95248eb807…&s=openmaptiles&ws=…"
|
|
300
|
+
}
|
|
295
301
|
}
|
|
296
302
|
}
|
|
297
303
|
```
|
|
@@ -309,10 +315,29 @@ load it. That also means a torrent-aware client gets a working map immediately
|
|
|
309
315
|
over HTTP while the swarm is still finding peers, rather than staring at an empty
|
|
310
316
|
canvas for the 90 to 240 seconds a cold magnet can take to resolve metadata.
|
|
311
317
|
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
|
|
318
|
+
### The `mutable` sub-block
|
|
319
|
+
|
|
320
|
+
Present only when a publisher is announcing this archive's category over the
|
|
321
|
+
DHT. It is the difference between a document describing *this build* and one
|
|
322
|
+
describing *the current build*:
|
|
323
|
+
|
|
324
|
+
| | |
|
|
325
|
+
| --- | --- |
|
|
326
|
+
| `publicKey` | The identity to resolve against. Public — there is nothing secret in the block |
|
|
327
|
+
| `salt` | The category, so one key can address several |
|
|
328
|
+
| `seq` | The sequence of the record this document was generated beside |
|
|
329
|
+
| `magnet` | Assembled from the above, ready to paste into a style's URL fragment |
|
|
330
|
+
|
|
331
|
+
`magnet` is built rather than left to the consumer because every node can build
|
|
332
|
+
it — it contains only the public half — so a fleet behind a balancer hands out
|
|
333
|
+
one identical string and none of them can publish. A BEP 46 client resolves it
|
|
334
|
+
and follows updates rather than pinning to the build this document happened to
|
|
335
|
+
describe.
|
|
336
|
+
|
|
337
|
+
See [a fragment that survives a rebuild](#a-fragment-that-survives-a-rebuild)
|
|
338
|
+
above for how it reaches a style, and
|
|
339
|
+
[security.md](security.md#the-publisher-key-is-not-a-credential) for why the
|
|
340
|
+
private half lives on exactly one machine.
|
|
316
341
|
|
|
317
342
|
## Health checks
|
|
318
343
|
|
|
@@ -564,6 +589,13 @@ first tile can be seconds away.
|
|
|
564
589
|
**Raster archives get the raster.** There is nothing to inspect in an image, so
|
|
565
590
|
the panel says so and the map is for checking coverage.
|
|
566
591
|
|
|
592
|
+
**What it has actually served** is on the archive's detail, as `served`, and
|
|
593
|
+
across the node at `GET /api/stats` — requests, bytes, a breakdown by zoom and
|
|
594
|
+
status, and which client addresses asked. Worth reading beside `reading`: an
|
|
595
|
+
archive being read through the swarm while serving thousands of tiles is a
|
|
596
|
+
different situation from one doing neither. See the
|
|
597
|
+
[README](../README.md#seeing-what-a-node-is-actually-serving).
|
|
598
|
+
|
|
567
599
|
No symbol layers are drawn and no glyphs are configured, deliberately: an
|
|
568
600
|
archive carries tiles, not fonts, and a preview that needed a font server to
|
|
569
601
|
render would not be a preview of the archive.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pmtiles-swarm",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.13.1",
|
|
4
4
|
"description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "src/index.js",
|
package/src/config.js
CHANGED
|
@@ -425,6 +425,19 @@ const DEFAULTS = {
|
|
|
425
425
|
keyPath: undefined,
|
|
426
426
|
/** How often to republish, in seconds. */
|
|
427
427
|
republishSeconds: 1800,
|
|
428
|
+
/**
|
|
429
|
+
* UDP port for this node's DHT socket. 0 takes an ephemeral one.
|
|
430
|
+
*
|
|
431
|
+
* Publishing does not need a forwarded port: a put is outbound, and the
|
|
432
|
+
* replies come back on the same socket the way any UDP client's do, which
|
|
433
|
+
* NAT handles. Setting a fixed port and forwarding it makes this a
|
|
434
|
+
* reachable DHT node, which gives better lookups and contributes back —
|
|
435
|
+
* but nothing here requires it.
|
|
436
|
+
*
|
|
437
|
+
* Do not reuse the libtorrent engine's port. That engine runs a DHT of its
|
|
438
|
+
* own, and two sockets cannot hold one port.
|
|
439
|
+
*/
|
|
440
|
+
dhtPort: 0,
|
|
428
441
|
},
|
|
429
442
|
/**
|
|
430
443
|
* What this node has served, at `GET /api/stats`.
|
package/src/engines/composite.js
CHANGED
|
@@ -36,6 +36,8 @@ export class CompositeEngine {
|
|
|
36
36
|
#timer;
|
|
37
37
|
/** Infohashes handed to secondaries, so they are not handed over twice. */
|
|
38
38
|
#shared = new Set();
|
|
39
|
+
/** Serialises background hand-overs, so they never run all at once. */
|
|
40
|
+
#shareChain = Promise.resolve();
|
|
39
41
|
/** Set once a stop has begun, so late timers do nothing. */
|
|
40
42
|
#stopping = false;
|
|
41
43
|
|
|
@@ -133,12 +135,54 @@ export class CompositeEngine {
|
|
|
133
135
|
const infoHash = await this.#primary.add(request);
|
|
134
136
|
|
|
135
137
|
if (this.#shareable(request) && (await this.#primaryHasItAll(infoHash))) {
|
|
136
|
-
|
|
138
|
+
// Queued rather than awaited. Handing an archive to a seeding client it
|
|
139
|
+
// has not seen before makes it hash every byte first, which is minutes
|
|
140
|
+
// for tens of gigabytes -- so awaiting it here put that cost inside the
|
|
141
|
+
// caller. On startup, where the library is restored one archive at a
|
|
142
|
+
// time, that meant a node sat silent for a quarter of an hour before it
|
|
143
|
+
// would listen, doing work the periodic sweep exists to do anyway.
|
|
144
|
+
//
|
|
145
|
+
// Serialised through one chain rather than fired off freely: seventeen
|
|
146
|
+
// archives hashing at once on a spinning disk is slower than seventeen
|
|
147
|
+
// in turn, and far harder to reason about.
|
|
148
|
+
this.#queueShare(infoHash, request);
|
|
137
149
|
}
|
|
138
150
|
|
|
139
151
|
return infoHash;
|
|
140
152
|
}
|
|
141
153
|
|
|
154
|
+
/**
|
|
155
|
+
* Resolves once every queued hand-over has finished.
|
|
156
|
+
*
|
|
157
|
+
* Nothing in normal operation waits for this — that is the point of the
|
|
158
|
+
* queue — but a caller that wants to observe the result, a test above all,
|
|
159
|
+
* needs somewhere to wait rather than a guess about microtask order.
|
|
160
|
+
* @returns {Promise<void>} - Resolves when the queue is empty.
|
|
161
|
+
*/
|
|
162
|
+
async whenShared() {
|
|
163
|
+
// Awaited twice: the first await settles what is queued now, and anything
|
|
164
|
+
// that queued more while it ran is picked up by the second.
|
|
165
|
+
await this.#shareChain;
|
|
166
|
+
await this.#shareChain;
|
|
167
|
+
}
|
|
168
|
+
|
|
169
|
+
/**
|
|
170
|
+
* Runs a share after any already queued, without making the caller wait.
|
|
171
|
+
* @param {string} infoHash - The archive.
|
|
172
|
+
* @param {object} request - The original add request.
|
|
173
|
+
* @returns {void}
|
|
174
|
+
*/
|
|
175
|
+
#queueShare(infoHash, request) {
|
|
176
|
+
this.#shareChain = this.#shareChain
|
|
177
|
+
.then(() => (this.#stopping ? undefined : this.#shareOne(infoHash, request)))
|
|
178
|
+
// #shareOne already swallows an engine refusing the archive; this is the
|
|
179
|
+
// last resort, so one failure cannot break the chain for everything
|
|
180
|
+
// queued behind it.
|
|
181
|
+
.catch((error) =>
|
|
182
|
+
console.warn(`[engine] could not hand over ${infoHash}: ${error.message}`),
|
|
183
|
+
);
|
|
184
|
+
}
|
|
185
|
+
|
|
142
186
|
/**
|
|
143
187
|
* Whether the primary actually holds the whole archive.
|
|
144
188
|
*
|
package/src/index.js
CHANGED
|
@@ -273,6 +273,15 @@ PMTILES_SWARM_PUBLIC_URL
|
|
|
273
273
|
]);
|
|
274
274
|
const pem = await fs.readFile(config.mutable.keyPath, 'utf8');
|
|
275
275
|
const dht = new DHT();
|
|
276
|
+
// Bound explicitly rather than left to bind implicitly on first send,
|
|
277
|
+
// so the port is predictable and can be forwarded if you want this to
|
|
278
|
+
// be a reachable DHT node. 0 takes an ephemeral one, which is all
|
|
279
|
+
// publishing needs.
|
|
280
|
+
await new Promise((resolve, reject) => {
|
|
281
|
+
dht.once('error', reject);
|
|
282
|
+
dht.listen(config.mutable.dhtPort ?? 0, resolve);
|
|
283
|
+
});
|
|
284
|
+
console.log(`[mutable] DHT on UDP ${dht.address().port}`);
|
|
276
285
|
publisher = new MutablePublisher({
|
|
277
286
|
catalog,
|
|
278
287
|
dht,
|
package/src/library.js
CHANGED
|
@@ -1463,9 +1463,41 @@ export class Library {
|
|
|
1463
1463
|
*/
|
|
1464
1464
|
async restore() {
|
|
1465
1465
|
const entries = this.#catalog.list();
|
|
1466
|
-
|
|
1467
|
-
|
|
1466
|
+
const tally = { restored: 0, failed: 0 };
|
|
1467
|
+
|
|
1468
|
+
// Restoring a large library is slow, and until now it said nothing until
|
|
1469
|
+
// it had finished -- so a node that was working and a node that was stuck
|
|
1470
|
+
// looked identical for as long as it took, which on a real library was
|
|
1471
|
+
// long enough to reach for a debugger. Reported on a timer rather than per
|
|
1472
|
+
// archive, so a small library stays quiet and a large one stops being a
|
|
1473
|
+
// mystery.
|
|
1474
|
+
const startedAt = Date.now();
|
|
1475
|
+
const progress = setInterval(() => {
|
|
1476
|
+
const seconds = Math.round((Date.now() - startedAt) / 1000);
|
|
1477
|
+
console.log(
|
|
1478
|
+
`[restore] ${tally.restored + tally.failed} of ${entries.length} ` +
|
|
1479
|
+
`after ${seconds}s`,
|
|
1480
|
+
);
|
|
1481
|
+
}, 15_000);
|
|
1482
|
+
progress.unref?.();
|
|
1468
1483
|
|
|
1484
|
+
try {
|
|
1485
|
+
return await this.#restoreEach(entries, tally);
|
|
1486
|
+
} finally {
|
|
1487
|
+
clearInterval(progress);
|
|
1488
|
+
}
|
|
1489
|
+
}
|
|
1490
|
+
|
|
1491
|
+
/**
|
|
1492
|
+
* The restore loop itself.
|
|
1493
|
+
*
|
|
1494
|
+
* Counts into the caller's tally rather than its own, so the progress timer
|
|
1495
|
+
* above has something to read while this is still running.
|
|
1496
|
+
* @param {object[]} entries - Catalog entries, newest first.
|
|
1497
|
+
* @param {{restored: number, failed: number}} tally - Mutated as it goes.
|
|
1498
|
+
* @returns {Promise<{restored: number, failed: number}>} - That tally.
|
|
1499
|
+
*/
|
|
1500
|
+
async #restoreEach(entries, tally) {
|
|
1469
1501
|
for (const entry of entries) {
|
|
1470
1502
|
// An engine that cannot open its port will fail every one of these, each
|
|
1471
1503
|
// after its own timeout. Stopping at the first is the difference between
|
|
@@ -1473,13 +1505,13 @@ export class Library {
|
|
|
1473
1505
|
// minutes per archive.
|
|
1474
1506
|
if (this.#engine.fatalError) {
|
|
1475
1507
|
console.error(`[restore] stopping: ${this.#engine.fatalError.message}`);
|
|
1476
|
-
failed += entries.length - restored;
|
|
1508
|
+
tally.failed += entries.length - tally.restored;
|
|
1477
1509
|
break;
|
|
1478
1510
|
}
|
|
1479
1511
|
|
|
1480
1512
|
try {
|
|
1481
1513
|
if (!entry.torrentPath && !entry.magnet) {
|
|
1482
|
-
failed++;
|
|
1514
|
+
tally.failed++;
|
|
1483
1515
|
continue;
|
|
1484
1516
|
}
|
|
1485
1517
|
|
|
@@ -1495,7 +1527,7 @@ export class Library {
|
|
|
1495
1527
|
`usable (${error.code ?? error.message}), so it cannot start. ` +
|
|
1496
1528
|
'Move it with Set location, or make that path reachable again.',
|
|
1497
1529
|
);
|
|
1498
|
-
failed++;
|
|
1530
|
+
tally.failed++;
|
|
1499
1531
|
continue;
|
|
1500
1532
|
}
|
|
1501
1533
|
}
|
|
@@ -1505,14 +1537,14 @@ export class Library {
|
|
|
1505
1537
|
// own add and skip that, which is why an archive that could not find a
|
|
1506
1538
|
// peer stayed unable to find one across every restart.
|
|
1507
1539
|
await this.#readd(entry);
|
|
1508
|
-
restored++;
|
|
1540
|
+
tally.restored++;
|
|
1509
1541
|
} catch (error) {
|
|
1510
|
-
failed++;
|
|
1542
|
+
tally.failed++;
|
|
1511
1543
|
console.error(`[restore] ${entry.name}: ${error.message}`);
|
|
1512
1544
|
}
|
|
1513
1545
|
}
|
|
1514
1546
|
|
|
1515
|
-
return
|
|
1547
|
+
return tally;
|
|
1516
1548
|
}
|
|
1517
1549
|
|
|
1518
1550
|
/**
|
package/src/publisher.js
CHANGED
|
@@ -21,6 +21,30 @@
|
|
|
21
21
|
* so republishing on a timer is not an optimisation here — it is the feature.
|
|
22
22
|
*/
|
|
23
23
|
|
|
24
|
+
/*
|
|
25
|
+
* Why this uses bittorrent-dht rather than an engine's own DHT.
|
|
26
|
+
*
|
|
27
|
+
* Both seeding engines run a DHT already, so a third one looks redundant.
|
|
28
|
+
* It is not, for two different reasons:
|
|
29
|
+
*
|
|
30
|
+
* libtorrent - its DHT lives in the Python sidecar, and the 2.x Python
|
|
31
|
+
* bindings do not expose dht_put_item or dht_get_item at all.
|
|
32
|
+
* The alerts are bound (dht_mutable_item_alert, dht_put_alert)
|
|
33
|
+
* so the C++ side supports BEP 44, but there is no method to
|
|
34
|
+
* start one. Checked against 2.0.13; worth re-checking if a
|
|
35
|
+
* later binding adds them, because a sidecar op would then be
|
|
36
|
+
* the tidier answer for a libtorrent-only node.
|
|
37
|
+
*
|
|
38
|
+
* webtorrent - its DHT *is* bittorrent-dht, reachable as `client.dht`, so
|
|
39
|
+
* reusing it is possible and would save a socket. Not done,
|
|
40
|
+
* to keep one code path that behaves the same whichever
|
|
41
|
+
* engine is configured.
|
|
42
|
+
*
|
|
43
|
+
* The dependency itself costs nothing: webtorrent already depends on the same
|
|
44
|
+
* version, and npm dedupes them to one install. Declaring it directly only
|
|
45
|
+
* removes the reliance on a transitive.
|
|
46
|
+
*/
|
|
47
|
+
|
|
24
48
|
import { mutableMagnet, publishInfoHash } from './mutable.js';
|
|
25
49
|
|
|
26
50
|
/** Republish well inside the ~2h a DHT keeps an item. */
|