pmtiles-swarm 0.12.0 → 0.13.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -7,6 +7,65 @@
7
7
  ### 🐞 Bug fixes
8
8
  - _...Add new stuff here..._
9
9
 
10
+ ## 0.13.1
11
+ ### 🐞 Bug fixes
12
+ - **A node with a secondary engine no longer takes a quarter of an hour to start listening.**
13
+ Handing an archive to a second seeding client makes it hash every byte before it will serve any
14
+ — minutes for tens of gigabytes — and that hand-over was **awaited** inside `add()`. On startup,
15
+ where the library is restored one archive at a time, the cost landed end to end before the node
16
+ would bind its port. A seventeen-archive library sat silent for about fifteen minutes.
17
+
18
+ The hand-over is now queued and the caller carries on. Nothing depended on waiting for it: the
19
+ periodic sweep that already exists to catch archives finishing later is the same mechanism, and
20
+ a failed hand-over already un-marks itself so that sweep retries it.
21
+
22
+ Queued through one chain rather than fired off freely, so hand-overs still run one at a time —
23
+ seventeen archives hashing at once on a spinning disk is slower than seventeen in turn, and much
24
+ harder to reason about.
25
+ - **Restoring a large library reports progress.** It said nothing at all until it had finished, so
26
+ a node that was working and a node that was stuck looked identical for as long as it took — long
27
+ enough, on a real library, to go looking for a debugger:
28
+
29
+ ```
30
+ [restore] 6 of 17 after 15s
31
+ ```
32
+
33
+ On a timer rather than per archive, so a small library stays quiet.
34
+
35
+ ## 0.13.0
36
+ ### ✨ Features and improvements
37
+ - **The publisher's DHT socket is bound explicitly, on a configurable port.** It was left to bind
38
+ implicitly on its first send, which works but takes an unpredictable ephemeral port and reports
39
+ nothing — so there was no way to forward it, and no way to tell which one it had. `mutable.dhtPort`
40
+ now sets it (`0`, ephemeral, by default) and the port is logged at startup.
41
+
42
+ Publishing needs no forward either way: a put is outbound, and the replies come back on the same
43
+ socket the way any UDP client's do. Pinning and forwarding one makes this a *reachable* DHT node
44
+ instead, which earns a better routing table and contributes back — worth having on a node that
45
+ runs continuously.
46
+
47
+ It must not collide with an engine's port. Each seeding engine runs a DHT of its own, so a node
48
+ with both has three UDP participants and only this one is placed here; two sockets cannot hold
49
+ one port and the node would fail to start.
50
+
51
+ ### 📚 Documentation
52
+ - **[docs/running-as-a-service.md](docs/running-as-a-service.md) covers the publisher key**:
53
+ generating it as the service account so ownership is right, `chmod 400` because nothing ever
54
+ writes it back, and why it must be backed up off the machine — losing it breaks every style
55
+ pointing at that public key, permanently, with no reissue. Also what happens under HA config
56
+ sync: the configuration replicates to the standby and the key does not, so the standby logs
57
+ `not publishing: ENOENT` and serves on, which is the intended outcome rather than a fault. And
58
+ how to confirm it works, including that `nodes: 0` in the log means nobody stored the record
59
+ however healthy the rest of the line looks.
60
+ - **The ports table lists the DHT port**, which had only been described in the publisher section —
61
+ not where anyone looks when deciding what to forward.
62
+ - **Why publishing does not reuse an engine's DHT** is recorded in
63
+ [src/publisher.js](src/publisher.js), since it will be asked again. libtorrent's is unreachable:
64
+ the 2.x Python bindings expose neither `dht_put_item` nor `dht_get_item`, though the alerts are
65
+ bound, so the C++ side supports BEP 44 and there is simply no method to call. WebTorrent's *is*
66
+ `bittorrent-dht` and could be reused to save a socket; that is a deliberate choice rather than an
67
+ oversight, taken to keep one code path that behaves the same whichever engine is configured.
68
+
10
69
  ## 0.12.0
11
70
  ### ✨ Features and improvements
12
71
  - **A category can now be addressed without a server at all.** A node that builds can publish a
@@ -224,7 +224,7 @@ New data means a new infohash, which is what makes cache invalidation free.
224
224
  %%{init: {"flowchart": {"rankSpacing": 60}}}%%
225
225
  graph LR
226
226
  NEW["new planet.pmtiles<br/><small>weekly build</small>"] --> HASH["primary re-hashes<br/>→ new infohash"]
227
- HASH --> BEP["BEP 46 mutable entry<br/><small>same public key, seq+1</small>"]
227
+ HASH --> BEP["BEP 46 mutable entry<br/><small>same public key, new seq</small>"]
228
228
  HASH --> FEED["new RSS item"]
229
229
 
230
230
  BEP --> FOLLOW["clients following the key<br/>see the new version"]
@@ -239,8 +239,36 @@ graph LR
239
239
  **Key points:** tile URLs are content-addressed, so they never need invalidating.
240
240
  The old infohash stays valid and servable for as long as anyone still holds it —
241
241
  useful for clients pinned to a known-good build — while new requests move to the
242
- new one as soon as they re-read `tiles.json`. The only mutable thing in the whole
243
- system is that one document.
242
+ new one as soon as they re-read `tiles.json`.
243
+
244
+ **There are exactly two mutable things**, and they say the same thing by
245
+ different means: the `/latest/<category>/` documents, which need this server, and
246
+ the BEP 46 record, which does not. A style pointing at the public key resolves
247
+ the current build over the DHT with nothing of ours running at all. Only the node
248
+ that builds signs those records — see
249
+ [security.md](security.md#the-publisher-key-is-not-a-credential), because the key
250
+ behaves unlike every other secret in the system.
251
+
252
+ The sequence number is derived from the clock rather than incremented, so a
253
+ publisher that is rebuilt or restored from backup carries on without needing to
254
+ remember where it had got to.
255
+
256
+ ### Bootstrapping without the server
257
+
258
+ A torrent-aware client still has to *learn* the magnet from somewhere, and until
259
+ it does, the swarm — the part that depends on no server — is unreachable
260
+ precisely when the server is down. The fix is that the magnet travels in the
261
+ **fragment** of the TileJSON URL a style already carries:
262
+
263
+ ```
264
+ https://swarm.example.org/latest/openmaptiles/tiles.json#magnet:?xs=urn:btpk:…&s=openmaptiles&ws=…
265
+ ```
266
+
267
+ A fragment is never sent in an HTTP request, so ordinary clients fetch the
268
+ TileJSON and ignore it while a swarm-aware one reads the magnet before making any
269
+ call. With a BEP 46 key in it (`xs=urn:btpk:`) rather than an infohash, that
270
+ string does not go stale on the next build either. See
271
+ [serving-tiles.md](serving-tiles.md#a-fragment-that-survives-a-rebuild).
244
272
 
245
273
  ---
246
274
 
@@ -354,6 +354,116 @@ What the running service actually has:
354
354
  systemctl show -p ReadWritePaths -p UMask -p SupplementaryGroups pmtiles-swarm
355
355
  ```
356
356
 
357
+ ## The publisher key
358
+
359
+ Only needed if this node publishes BEP 46 records — the signed DHT entries that
360
+ let a style point at a category rather than at a build that goes stale. Skip
361
+ this section entirely if it does not build archives.
362
+
363
+ **Generate it as the service account**, so the file is owned by the user that
364
+ has to read it:
365
+
366
+ ```sh
367
+ sudo -u pmtiles-swarm -H /var/lib/pmtiles-swarm/node_modules/.bin/pmtiles-swarm publisher-key > /etc/pmtiles-swarm/publisher.pem
368
+ sudo chown pmtiles-swarm:pmtiles-swarm /etc/pmtiles-swarm/publisher.pem
369
+ sudo chmod 400 /etc/pmtiles-swarm/publisher.pem
370
+ ```
371
+
372
+ The PEM goes to stdout and the public key to stderr, so the redirect above
373
+ captures only the key material and you still see the public half on the
374
+ terminal. Write that public key down — it is what subscribers point at, and it
375
+ is the one part you will want later.
376
+
377
+ `400` rather than `600`: the service only ever reads this. Nothing in the
378
+ product writes it back, so removing write permission costs nothing and means a
379
+ compromised process cannot quietly replace it.
380
+
381
+ Then in `/etc/pmtiles-swarm/swarm.config.json`:
382
+
383
+ ```json
384
+ {
385
+ "mutable": {
386
+ "publish": true,
387
+ "keyPath": "/etc/pmtiles-swarm/publisher.pem"
388
+ }
389
+ }
390
+ ```
391
+
392
+ `/etc/pmtiles-swarm` is already in `ReadWritePaths` because the console rewrites
393
+ the configuration there, so the unit needs no change.
394
+
395
+ ### Back it up, off this machine
396
+
397
+ Losing this file breaks **every style pointing at that public key, permanently**
398
+ — there is no recovery, no reissue, and no way to prove to a subscriber that a
399
+ new key is you. It is not like an API key you can rotate.
400
+
401
+ Back it up somewhere that is not this disk, and treat the backup as seriously as
402
+ the original: whoever holds it can publish a signed record telling your
403
+ subscribers that any archive is the current build, and clients will believe it
404
+ because the signature checks out.
405
+
406
+ ### Exactly one publisher
407
+
408
+ Two nodes publishing under one key fight over the sequence number, each
409
+ overwriting the other's claim about what is current. So the PEM belongs on the
410
+ build node and nowhere else.
411
+
412
+ **This matters if you run HA config sync.** The configuration will replicate to
413
+ the standby, `publish: true` and all — but the PEM will not, because you are
414
+ not going to copy it. The standby then logs
415
+
416
+ ```
417
+ [mutable] not publishing: ENOENT: no such file or directory
418
+ ```
419
+
420
+ on every start and carries on serving normally. That is the intended outcome
421
+ rather than a fault: the config syncing is harmless, and the key not syncing is
422
+ the point. If the noise bothers you, set `mutable.publish` to `false` in the
423
+ standby's config after the sync.
424
+
425
+ ### It does not need a port forwarded
426
+
427
+ The DHT socket is separate from the seeding engine's, and by default takes an
428
+ ephemeral UDP port (`mutable.dhtPort: 0`). That is enough to publish: a put is
429
+ outbound — find nodes, then send — and the replies come back on the same
430
+ socket the way any UDP client's do, which NAT handles without help.
431
+
432
+ Setting a fixed port and forwarding it makes this a *reachable* DHT node, which
433
+ means better lookups and contributing back to the network. Worth doing if this
434
+ node is long-lived, but nothing here requires it.
435
+
436
+ **Do not reuse the libtorrent engine's port.** That engine runs a DHT of its own
437
+ on `libtorrent.listen`, and two sockets cannot hold one port — the node would
438
+ fail to start.
439
+
440
+ The port in use is reported at startup:
441
+
442
+ ```
443
+ [mutable] DHT on UDP 63213
444
+ ```
445
+
446
+ ### Confirming it works
447
+
448
+ ```sh
449
+ journalctl -u pmtiles-swarm | grep mutable
450
+ ```
451
+
452
+ A healthy publisher says which categories it is announcing and under which key
453
+ at startup, then one line per category whenever a build moves:
454
+
455
+ ```
456
+ [mutable] publishing 3 categories as 7680dc95248eb807… every 30m
457
+ [mutable] openmaptiles -> 4813a0e68e4b (seq 1786108931, 8 nodes)
458
+ ```
459
+
460
+ `nodes` is how many DHT peers stored the record. **Zero means nobody did**, and
461
+ the record does not exist however healthy the log looks otherwise — check that
462
+ UDP is not blocked and that the DHT is reachable.
463
+
464
+ The first publish waits about fifteen seconds after start, because a put into a
465
+ DHT that has not finished bootstrapping reaches nobody.
466
+
357
467
  ## The sidecar
358
468
 
359
469
  The libtorrent engine runs Python as a child process, so the service user needs
@@ -373,16 +483,33 @@ rather than exiting. Name the interpreter explicitly when the service user's
373
483
 
374
484
  ## Ports
375
485
 
376
- Four listeners, and only the peer ports want a firewall rule. See
486
+ Five listeners, and only the peer ports want a firewall rule. See
377
487
  [ports and reachability](engines.md#ports-and-reachability) for the detail.
378
488
 
379
489
  | | |
380
490
  | --- | --- |
381
491
  | `libtorrent.listen` — 6881, TCP and UDP | forward it |
382
492
  | `webtorrent.clientOptions.torrentPort` — pin it, or it changes every start | forward it |
493
+ | `mutable.dhtPort` — UDP, ephemeral by default | optional; see below |
383
494
  | `port` — 8090 | your proxy or CDN |
384
495
  | `adminPort` — 8091, bound to `127.0.0.1` | nothing; that is the point |
385
496
 
497
+ `mutable.dhtPort` is the odd one. Publishing works without any forward, because
498
+ a put is outbound and the replies come back on the same socket the way any UDP
499
+ client's do. Pin it and forward it only if you want this to be a *reachable*
500
+ DHT node — which earns a better routing table and contributes back, and is
501
+ worth having on a node that runs continuously.
502
+
503
+ Pin it to something free: **not** `libtorrent.listen`, which is a DHT of its
504
+ own, and not `torrentPort`. `6883` sits clear of both.
505
+
506
+ ```json
507
+ { "mutable": { "dhtPort": 6883 } }
508
+ ```
509
+
510
+ Note that each engine runs its own DHT as well, so a node with both engines has
511
+ three UDP participants. Only this one is yours to place.
512
+
386
513
  ## Updating
387
514
 
388
515
  ```sh
package/docs/security.md CHANGED
@@ -188,6 +188,56 @@ Three ways past it, in order of preference:
188
188
  2. Bind to `127.0.0.1` and reach it through a reverse proxy that authenticates.
189
189
  3. Set `allowUnauthenticated: true`, if the network really is trusted.
190
190
 
191
+ ## The publisher key is not a credential
192
+
193
+ Everything above guards *this node*: who may read the API, who may change
194
+ settings, who may add an archive. The publisher key is a different kind of
195
+ secret, and the difference is worth stating plainly because the habits do not
196
+ carry over.
197
+
198
+ `mutable.keyPath` holds an ed25519 private key used to sign BEP 46 records — the
199
+ DHT entries that say which archive is the current build of a category. It is a
200
+ **signing key**, closer to a code-signing certificate than to an API token.
201
+
202
+ | | An access credential (`apiKey`, a token, `password`) | The publisher key |
203
+ | --- | --- | --- |
204
+ | What it grants | access to one node | authority over what your subscribers believe |
205
+ | If it leaks | that node is compromised | anyone can sign a record pointing your subscribers at any archive |
206
+ | If you lose it | mint another | every style pointing at that public key breaks, permanently |
207
+ | Rotation | revoke and reissue | there is none |
208
+
209
+ That third row is the one people are unprepared for. A public key **is** the
210
+ identity: there is no registry to update and no way to tell a subscriber that a
211
+ new key is also you. Losing the file ends that identity.
212
+
213
+ And the second row is why it does not belong on a serving tier. A leaked API key
214
+ lets someone read a node. A leaked publisher key lets someone tell every
215
+ subscriber that an archive they control is your latest planet build — signed,
216
+ verifying correctly, and indistinguishable from a real announcement.
217
+
218
+ **So it lives on exactly one machine.** Only the node that builds needs it;
219
+ serving nodes carry the public half on the catalog entry and hand it out in the
220
+ TileJSON, which is why a serving tier can be compromised without anyone being
221
+ able to publish. Running two publishers under one key is separately a mistake:
222
+ they fight over the sequence number.
223
+
224
+ Practically:
225
+
226
+ - `chmod 400`, owned by the service account. Nothing in the product writes it
227
+ back, so it does not need to be writable and should not be.
228
+ - Back it up **off the machine**, and treat the backup as seriously as the
229
+ original.
230
+ - Do not put it in an image, a config-sync set, or anywhere a standby will
231
+ receive it. Under HA sync the configuration replicates and the key does not,
232
+ which is the intended asymmetry rather than an oversight — the standby logs
233
+ `not publishing: ENOENT` and serves normally.
234
+ - It is unrelated to `auth`. Turning authentication off does not expose it, and
235
+ rotating the API key does not affect it.
236
+
237
+ Generate one with `pmtiles-swarm publisher-key`. Setup detail is in
238
+ [running-as-a-service.md](running-as-a-service.md#the-publisher-key); what it is
239
+ *for* is in [serving-tiles.md](serving-tiles.md#a-fragment-that-survives-a-rebuild).
240
+
191
241
  ## What this is not
192
242
 
193
243
  **It is not a defence against someone on your network.** Sessions are bearer
@@ -291,7 +291,13 @@ TileJSON documents from this server carry a non-standard `torrent` member:
291
291
  "torrent": "https://swarm.example.org/archives/913d…/archive.torrent",
292
292
  "name": "planet.pmtiles",
293
293
  "size": 77242531840,
294
- "webseeds": ["https://maps.example.org/planet.pmtiles"]
294
+ "webseeds": ["https://maps.example.org/planet.pmtiles"],
295
+ "mutable": {
296
+ "publicKey": "7680dc95248eb807…",
297
+ "salt": "openmaptiles",
298
+ "seq": 1786108931,
299
+ "magnet": "magnet:?xs=urn:btpk:7680dc95248eb807…&s=openmaptiles&ws=…"
300
+ }
295
301
  }
296
302
  }
297
303
  ```
@@ -309,10 +315,29 @@ load it. That also means a torrent-aware client gets a working map immediately
309
315
  over HTTP while the swarm is still finding peers, rather than staring at an empty
310
316
  canvas for the 90 to 240 seconds a cold magnet can take to resolve metadata.
311
317
 
312
- For an archive published as a mutable torrent, the block also carries
313
- `mutable.publicKey`, so a client that understands BEP 46 can follow updates
314
- rather than pinning to the version the document was generated from. See
315
- [publishing](publishing.md).
318
+ ### The `mutable` sub-block
319
+
320
+ Present only when a publisher is announcing this archive's category over the
321
+ DHT. It is the difference between a document describing *this build* and one
322
+ describing *the current build*:
323
+
324
+ | | |
325
+ | --- | --- |
326
+ | `publicKey` | The identity to resolve against. Public — there is nothing secret in the block |
327
+ | `salt` | The category, so one key can address several |
328
+ | `seq` | The sequence of the record this document was generated beside |
329
+ | `magnet` | Assembled from the above, ready to paste into a style's URL fragment |
330
+
331
+ `magnet` is built rather than left to the consumer because every node can build
332
+ it — it contains only the public half — so a fleet behind a balancer hands out
333
+ one identical string and none of them can publish. A BEP 46 client resolves it
334
+ and follows updates rather than pinning to the build this document happened to
335
+ describe.
336
+
337
+ See [a fragment that survives a rebuild](#a-fragment-that-survives-a-rebuild)
338
+ above for how it reaches a style, and
339
+ [security.md](security.md#the-publisher-key-is-not-a-credential) for why the
340
+ private half lives on exactly one machine.
316
341
 
317
342
  ## Health checks
318
343
 
@@ -564,6 +589,13 @@ first tile can be seconds away.
564
589
  **Raster archives get the raster.** There is nothing to inspect in an image, so
565
590
  the panel says so and the map is for checking coverage.
566
591
 
592
+ **What it has actually served** is on the archive's detail, as `served`, and
593
+ across the node at `GET /api/stats` — requests, bytes, a breakdown by zoom and
594
+ status, and which client addresses asked. Worth reading beside `reading`: an
595
+ archive being read through the swarm while serving thousands of tiles is a
596
+ different situation from one doing neither. See the
597
+ [README](../README.md#seeing-what-a-node-is-actually-serving).
598
+
567
599
  No symbol layers are drawn and no glyphs are configured, deliberately: an
568
600
  archive carries tiles, not fonts, and a preview that needed a font server to
569
601
  render would not be a preview of the archive.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pmtiles-swarm",
3
- "version": "0.12.0",
3
+ "version": "0.13.1",
4
4
  "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
5
5
  "type": "module",
6
6
  "main": "src/index.js",
package/src/config.js CHANGED
@@ -425,6 +425,19 @@ const DEFAULTS = {
425
425
  keyPath: undefined,
426
426
  /** How often to republish, in seconds. */
427
427
  republishSeconds: 1800,
428
+ /**
429
+ * UDP port for this node's DHT socket. 0 takes an ephemeral one.
430
+ *
431
+ * Publishing does not need a forwarded port: a put is outbound, and the
432
+ * replies come back on the same socket the way any UDP client's do, which
433
+ * NAT handles. Setting a fixed port and forwarding it makes this a
434
+ * reachable DHT node, which gives better lookups and contributes back —
435
+ * but nothing here requires it.
436
+ *
437
+ * Do not reuse the libtorrent engine's port. That engine runs a DHT of its
438
+ * own, and two sockets cannot hold one port.
439
+ */
440
+ dhtPort: 0,
428
441
  },
429
442
  /**
430
443
  * What this node has served, at `GET /api/stats`.
@@ -36,6 +36,8 @@ export class CompositeEngine {
36
36
  #timer;
37
37
  /** Infohashes handed to secondaries, so they are not handed over twice. */
38
38
  #shared = new Set();
39
+ /** Serialises background hand-overs, so they never run all at once. */
40
+ #shareChain = Promise.resolve();
39
41
  /** Set once a stop has begun, so late timers do nothing. */
40
42
  #stopping = false;
41
43
 
@@ -133,12 +135,54 @@ export class CompositeEngine {
133
135
  const infoHash = await this.#primary.add(request);
134
136
 
135
137
  if (this.#shareable(request) && (await this.#primaryHasItAll(infoHash))) {
136
- await this.#shareOne(infoHash, request);
138
+ // Queued rather than awaited. Handing an archive to a seeding client it
139
+ // has not seen before makes it hash every byte first, which is minutes
140
+ // for tens of gigabytes -- so awaiting it here put that cost inside the
141
+ // caller. On startup, where the library is restored one archive at a
142
+ // time, that meant a node sat silent for a quarter of an hour before it
143
+ // would listen, doing work the periodic sweep exists to do anyway.
144
+ //
145
+ // Serialised through one chain rather than fired off freely: seventeen
146
+ // archives hashing at once on a spinning disk is slower than seventeen
147
+ // in turn, and far harder to reason about.
148
+ this.#queueShare(infoHash, request);
137
149
  }
138
150
 
139
151
  return infoHash;
140
152
  }
141
153
 
154
+ /**
155
+ * Resolves once every queued hand-over has finished.
156
+ *
157
+ * Nothing in normal operation waits for this — that is the point of the
158
+ * queue — but a caller that wants to observe the result, a test above all,
159
+ * needs somewhere to wait rather than a guess about microtask order.
160
+ * @returns {Promise<void>} - Resolves when the queue is empty.
161
+ */
162
+ async whenShared() {
163
+ // Awaited twice: the first await settles what is queued now, and anything
164
+ // that queued more while it ran is picked up by the second.
165
+ await this.#shareChain;
166
+ await this.#shareChain;
167
+ }
168
+
169
+ /**
170
+ * Runs a share after any already queued, without making the caller wait.
171
+ * @param {string} infoHash - The archive.
172
+ * @param {object} request - The original add request.
173
+ * @returns {void}
174
+ */
175
+ #queueShare(infoHash, request) {
176
+ this.#shareChain = this.#shareChain
177
+ .then(() => (this.#stopping ? undefined : this.#shareOne(infoHash, request)))
178
+ // #shareOne already swallows an engine refusing the archive; this is the
179
+ // last resort, so one failure cannot break the chain for everything
180
+ // queued behind it.
181
+ .catch((error) =>
182
+ console.warn(`[engine] could not hand over ${infoHash}: ${error.message}`),
183
+ );
184
+ }
185
+
142
186
  /**
143
187
  * Whether the primary actually holds the whole archive.
144
188
  *
package/src/index.js CHANGED
@@ -273,6 +273,15 @@ PMTILES_SWARM_PUBLIC_URL
273
273
  ]);
274
274
  const pem = await fs.readFile(config.mutable.keyPath, 'utf8');
275
275
  const dht = new DHT();
276
+ // Bound explicitly rather than left to bind implicitly on first send,
277
+ // so the port is predictable and can be forwarded if you want this to
278
+ // be a reachable DHT node. 0 takes an ephemeral one, which is all
279
+ // publishing needs.
280
+ await new Promise((resolve, reject) => {
281
+ dht.once('error', reject);
282
+ dht.listen(config.mutable.dhtPort ?? 0, resolve);
283
+ });
284
+ console.log(`[mutable] DHT on UDP ${dht.address().port}`);
276
285
  publisher = new MutablePublisher({
277
286
  catalog,
278
287
  dht,
package/src/library.js CHANGED
@@ -1463,9 +1463,41 @@ export class Library {
1463
1463
  */
1464
1464
  async restore() {
1465
1465
  const entries = this.#catalog.list();
1466
- let restored = 0;
1467
- let failed = 0;
1466
+ const tally = { restored: 0, failed: 0 };
1467
+
1468
+ // Restoring a large library is slow, and until now it said nothing until
1469
+ // it had finished -- so a node that was working and a node that was stuck
1470
+ // looked identical for as long as it took, which on a real library was
1471
+ // long enough to reach for a debugger. Reported on a timer rather than per
1472
+ // archive, so a small library stays quiet and a large one stops being a
1473
+ // mystery.
1474
+ const startedAt = Date.now();
1475
+ const progress = setInterval(() => {
1476
+ const seconds = Math.round((Date.now() - startedAt) / 1000);
1477
+ console.log(
1478
+ `[restore] ${tally.restored + tally.failed} of ${entries.length} ` +
1479
+ `after ${seconds}s`,
1480
+ );
1481
+ }, 15_000);
1482
+ progress.unref?.();
1468
1483
 
1484
+ try {
1485
+ return await this.#restoreEach(entries, tally);
1486
+ } finally {
1487
+ clearInterval(progress);
1488
+ }
1489
+ }
1490
+
1491
+ /**
1492
+ * The restore loop itself.
1493
+ *
1494
+ * Counts into the caller's tally rather than its own, so the progress timer
1495
+ * above has something to read while this is still running.
1496
+ * @param {object[]} entries - Catalog entries, newest first.
1497
+ * @param {{restored: number, failed: number}} tally - Mutated as it goes.
1498
+ * @returns {Promise<{restored: number, failed: number}>} - That tally.
1499
+ */
1500
+ async #restoreEach(entries, tally) {
1469
1501
  for (const entry of entries) {
1470
1502
  // An engine that cannot open its port will fail every one of these, each
1471
1503
  // after its own timeout. Stopping at the first is the difference between
@@ -1473,13 +1505,13 @@ export class Library {
1473
1505
  // minutes per archive.
1474
1506
  if (this.#engine.fatalError) {
1475
1507
  console.error(`[restore] stopping: ${this.#engine.fatalError.message}`);
1476
- failed += entries.length - restored;
1508
+ tally.failed += entries.length - tally.restored;
1477
1509
  break;
1478
1510
  }
1479
1511
 
1480
1512
  try {
1481
1513
  if (!entry.torrentPath && !entry.magnet) {
1482
- failed++;
1514
+ tally.failed++;
1483
1515
  continue;
1484
1516
  }
1485
1517
 
@@ -1495,7 +1527,7 @@ export class Library {
1495
1527
  `usable (${error.code ?? error.message}), so it cannot start. ` +
1496
1528
  'Move it with Set location, or make that path reachable again.',
1497
1529
  );
1498
- failed++;
1530
+ tally.failed++;
1499
1531
  continue;
1500
1532
  }
1501
1533
  }
@@ -1505,14 +1537,14 @@ export class Library {
1505
1537
  // own add and skip that, which is why an archive that could not find a
1506
1538
  // peer stayed unable to find one across every restart.
1507
1539
  await this.#readd(entry);
1508
- restored++;
1540
+ tally.restored++;
1509
1541
  } catch (error) {
1510
- failed++;
1542
+ tally.failed++;
1511
1543
  console.error(`[restore] ${entry.name}: ${error.message}`);
1512
1544
  }
1513
1545
  }
1514
1546
 
1515
- return { restored, failed };
1547
+ return tally;
1516
1548
  }
1517
1549
 
1518
1550
  /**
package/src/publisher.js CHANGED
@@ -21,6 +21,30 @@
21
21
  * so republishing on a timer is not an optimisation here — it is the feature.
22
22
  */
23
23
 
24
+ /*
25
+ * Why this uses bittorrent-dht rather than an engine's own DHT.
26
+ *
27
+ * Both seeding engines run a DHT already, so a third one looks redundant.
28
+ * It is not, for two different reasons:
29
+ *
30
+ * libtorrent - its DHT lives in the Python sidecar, and the 2.x Python
31
+ * bindings do not expose dht_put_item or dht_get_item at all.
32
+ * The alerts are bound (dht_mutable_item_alert, dht_put_alert)
33
+ * so the C++ side supports BEP 44, but there is no method to
34
+ * start one. Checked against 2.0.13; worth re-checking if a
35
+ * later binding adds them, because a sidecar op would then be
36
+ * the tidier answer for a libtorrent-only node.
37
+ *
38
+ * webtorrent - its DHT *is* bittorrent-dht, reachable as `client.dht`, so
39
+ * reusing it is possible and would save a socket. Not done,
40
+ * to keep one code path that behaves the same whichever
41
+ * engine is configured.
42
+ *
43
+ * The dependency itself costs nothing: webtorrent already depends on the same
44
+ * version, and npm dedupes them to one install. Declaring it directly only
45
+ * removes the reliance on a transitive.
46
+ */
47
+
24
48
  import { mutableMagnet, publishInfoHash } from './mutable.js';
25
49
 
26
50
  /** Republish well inside the ~2h a DHT keeps an item. */