pmtiles-swarm 0.36.0 โ 0.37.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +62 -0
- package/docs/publishing.md +19 -0
- package/package.json +2 -2
- package/src/engines/composite.js +20 -6
- package/src/engines/libtorrent.js +52 -3
- package/src/library.js +76 -24
- package/src/torrent-create.js +44 -6
- package/src/web/index.html +40 -23
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,68 @@
|
|
|
7
7
|
### ๐ Bug fixes
|
|
8
8
|
- _...Add new stuff here..._
|
|
9
9
|
|
|
10
|
+
## 0.37.2
|
|
11
|
+
### โจ Features and improvements
|
|
12
|
+
|
|
13
|
+
### ๐ Bug fixes
|
|
14
|
+
- **Pausing an archive now stops it.** Reported from the field: an archive was paused, the row read
|
|
15
|
+
`paused`, and it went on downloading at 8.4 MiB/s. Nothing in the chain refused โ the pause was
|
|
16
|
+
asked for, reported as done, and never happened.
|
|
17
|
+
|
|
18
|
+
Three faults in a line. `LibtorrentEngine` had no `pause` or `resume` at all, so there was no way
|
|
19
|
+
to stop a libtorrent torrent from here. `CompositeEngine.pause` answered for its primary alone,
|
|
20
|
+
so an archive held by a secondary reported as not stopped when it was. And `Library.pause` tested
|
|
21
|
+
only that the engine *had* a pause method and threw away what it answered โ a composite has one
|
|
22
|
+
whatever its engines can do, so the `false` went into a void, the fallback never ran, and
|
|
23
|
+
`paused: true` went into the catalog regardless. The console prefers that flag to the engine's
|
|
24
|
+
live state, which is why the row said `paused` while Down and Up kept moving.
|
|
25
|
+
|
|
26
|
+
Requires pmtiles-torrent 0.9.0, which adds the `pause` and `resume` the sidecar never had โ and
|
|
27
|
+
makes them stick. `handle.pause()` alone is not a stop: libtorrent's auto-manager clears the
|
|
28
|
+
paused flag again within about a second, so pausing that way produces a torrent that describes
|
|
29
|
+
itself as paused while it transfers. That would have reproduced this exact symptom one layer
|
|
30
|
+
deeper.
|
|
31
|
+
|
|
32
|
+
Nothing was left in a bad state by this: because the pause never took effect, no archive was
|
|
33
|
+
half-stopped and no resume data is wrong. They were seeding and downloading throughout.
|
|
34
|
+
|
|
35
|
+
## 0.37.1
|
|
36
|
+
### โจ Features and improvements
|
|
37
|
+
- **Cancel now sits in the row it cancels.** Collected into a bar underneath the list, each button
|
|
38
|
+
had to repeat the whole filename to say which add it stopped โ two of those filled a line, and
|
|
39
|
+
pressing the right one meant matching a long name against the list above it. The rows carry their
|
|
40
|
+
own, and an add that cannot be cancelled keeps an empty cell so the columns stay lined up.
|
|
41
|
+
|
|
42
|
+
### ๐ Bug fixes
|
|
43
|
+
|
|
44
|
+
## 0.37.0
|
|
45
|
+
### โจ Features and improvements
|
|
46
|
+
- **An archive being hashed can now be cancelled, and says how far through it is.** 0.36.0 moved
|
|
47
|
+
hashing into a process of its own, which made both possible; this connects them to the console.
|
|
48
|
+
A local add is now registered with an AbortController, so `DELETE /api/adds?url=<path>` ends the
|
|
49
|
+
hasher and the Cancel button beside it works. Nothing is lost by pressing it: the archive is the
|
|
50
|
+
caller's own file and hashing only ever read it.
|
|
51
|
+
|
|
52
|
+
Shutdown reaches these too. A hasher left behind when the node exits is an orphan reading the
|
|
53
|
+
disk for hours, answering to nothing.
|
|
54
|
+
|
|
55
|
+
The piece the hasher has reached is converted to bytes against the file size โ the console draws
|
|
56
|
+
one progress bar for adds and labels it in bytes โ so a long hash reads `hashing 698 GiB ยท 41.2%
|
|
57
|
+
ยท 12m` instead of `hashing 698 GiB ยท 3m`. The hourly log line carries the percentage as well. An
|
|
58
|
+
add hashing in this process rather than out of it still reports no figure, because there is none
|
|
59
|
+
to report, and the bar is left off rather than pinned at zero.
|
|
60
|
+
|
|
61
|
+
### ๐ Bug fixes
|
|
62
|
+
- **Cancelling a hash no longer answers by hashing the same archive here instead.** A creator that
|
|
63
|
+
fails falls back to hashing in the node's own process, deliberately, since a torrent matters more
|
|
64
|
+
than the format of a torrent โ and a cancelled hash arrives as a creator that failed. So the
|
|
65
|
+
button would have answered "stop reading 698 GiB" by reading 698 GiB again, in the process
|
|
66
|
+
serving tiles, where nothing can interrupt it at all. A creator that fails for any other reason
|
|
67
|
+
still falls back.
|
|
68
|
+
- **A cancel arriving during the MD5 pass is no longer ignored until it finishes.** With `md5` on,
|
|
69
|
+
the archive is read twice, and only the second read took a signal โ so cancelling during the
|
|
70
|
+
first one waited out most of an hour of disk on a planet archive before it took effect.
|
|
71
|
+
|
|
10
72
|
## 0.36.0
|
|
11
73
|
### โจ Features and improvements
|
|
12
74
|
- **Hashing an archive now happens in a process of its own.** Building the torrent for a 698 GiB
|
package/docs/publishing.md
CHANGED
|
@@ -79,6 +79,25 @@ A magnet, a `.torrent` URL and an uploaded `.torrent` are metadata rather than
|
|
|
79
79
|
data, so there is nothing slow to wait for: those still answer `201` with the
|
|
80
80
|
entry.
|
|
81
81
|
|
|
82
|
+
### Stopping one that is running
|
|
83
|
+
|
|
84
|
+
`DELETE /api/adds?url=โฆ` stops an add before it has produced a torrent, and the
|
|
85
|
+
console offers it as a Cancel button beside each one. The identifier is whatever
|
|
86
|
+
the add was started with โ the URL for a remote one, the path for a local one.
|
|
87
|
+
|
|
88
|
+
What it costs differs by kind. A **remote** add is deleting a partial download,
|
|
89
|
+
deliberately: somebody said stop, and leaving a few hundred gigabytes behind
|
|
90
|
+
after that is waste nobody will ever find again. A **local** add deletes
|
|
91
|
+
nothing. The archive is your own file and hashing only ever read it, so
|
|
92
|
+
cancelling costs the hashing done so far and not a byte of data. If `publishDir`
|
|
93
|
+
moved the file, that already happened โ it is the cheap irreversible step and it
|
|
94
|
+
goes first โ so the archive stays where it was published to.
|
|
95
|
+
|
|
96
|
+
`GET /api/adds` reports each one's progress while it runs. A local add hashing
|
|
97
|
+
out of process says how far through the archive it is; one that has fallen back
|
|
98
|
+
to hashing in the node's own process reports no figure at all, because there is
|
|
99
|
+
none to report, and cancelling it stops the wait rather than the read.
|
|
100
|
+
|
|
82
101
|
### When the source URL is not published
|
|
83
102
|
|
|
84
103
|
Adding from a URL registers that URL as a web seed by default, because it is by
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pmtiles-swarm",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.37.2",
|
|
4
4
|
"description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "src/index.js",
|
|
@@ -46,7 +46,7 @@
|
|
|
46
46
|
"maplibre-gl": "^6.2.0",
|
|
47
47
|
"parse-torrent": "^11.0.24",
|
|
48
48
|
"pmtiles": "^4.4.1",
|
|
49
|
-
"pmtiles-torrent": "^0.
|
|
49
|
+
"pmtiles-torrent": "^0.9.0",
|
|
50
50
|
"webtorrent": "^3.0.21"
|
|
51
51
|
},
|
|
52
52
|
"engines": {
|
package/src/engines/composite.js
CHANGED
|
@@ -543,26 +543,40 @@ export class CompositeEngine {
|
|
|
543
543
|
|
|
544
544
|
/**
|
|
545
545
|
* Stops offering an archive, everywhere.
|
|
546
|
+
*
|
|
547
|
+
* Answers for the whole composite rather than for the primary alone. An
|
|
548
|
+
* archive can be held by a secondary and not by the primary, and reporting
|
|
549
|
+
* the primary's "no" for it told the caller nothing was stopped when
|
|
550
|
+
* something was -- which is a false negative that costs data, since the
|
|
551
|
+
* caller's answer to a pause it cannot get is to remove the torrent instead.
|
|
546
552
|
* @param {string} infoHash - The archive.
|
|
547
|
-
* @returns {Promise<boolean>} - Whether
|
|
553
|
+
* @returns {Promise<boolean>} - Whether any engine stopped it.
|
|
548
554
|
*/
|
|
549
555
|
async pause(infoHash) {
|
|
556
|
+
let stopped = false;
|
|
550
557
|
for (const engine of this.#secondaries) {
|
|
551
|
-
await engine.pause?.(infoHash).catch(() =>
|
|
558
|
+
stopped = (await engine.pause?.(infoHash).catch(() => false)) || stopped;
|
|
552
559
|
}
|
|
553
|
-
|
|
560
|
+
const primary = this.#primary.pause
|
|
561
|
+
? await this.#primary.pause(infoHash)
|
|
562
|
+
: false;
|
|
563
|
+
return primary || stopped;
|
|
554
564
|
}
|
|
555
565
|
|
|
556
566
|
/**
|
|
557
567
|
* Offers it again, everywhere.
|
|
558
568
|
* @param {string} infoHash - The archive.
|
|
559
|
-
* @returns {Promise<boolean>} - Whether
|
|
569
|
+
* @returns {Promise<boolean>} - Whether any engine started it.
|
|
560
570
|
*/
|
|
561
571
|
async resume(infoHash) {
|
|
572
|
+
let started = false;
|
|
562
573
|
for (const engine of this.#secondaries) {
|
|
563
|
-
await engine.resume?.(infoHash).catch(() =>
|
|
574
|
+
started = (await engine.resume?.(infoHash).catch(() => false)) || started;
|
|
564
575
|
}
|
|
565
|
-
|
|
576
|
+
const primary = this.#primary.resume
|
|
577
|
+
? await this.#primary.resume(infoHash)
|
|
578
|
+
: false;
|
|
579
|
+
return primary || started;
|
|
566
580
|
}
|
|
567
581
|
|
|
568
582
|
/**
|
|
@@ -415,6 +415,57 @@ export class LibtorrentEngine {
|
|
|
415
415
|
}
|
|
416
416
|
}
|
|
417
417
|
|
|
418
|
+
/**
|
|
419
|
+
* Stops a torrent, leaving its data and its place in the session alone.
|
|
420
|
+
*
|
|
421
|
+
* Not removal. The bytes stay, the resume data stays, and starting it again
|
|
422
|
+
* costs nothing -- which is the whole reason this exists rather than the
|
|
423
|
+
* remove-and-re-add a missing pause used to fall back to. Re-adding a
|
|
424
|
+
* 698 GiB archive means hashing the store again to arrive where it already
|
|
425
|
+
* was.
|
|
426
|
+
* @param {string} infoHash - The archive to stop.
|
|
427
|
+
* @returns {Promise<boolean>} - Whether it was stopped.
|
|
428
|
+
*/
|
|
429
|
+
async pause(infoHash) {
|
|
430
|
+
await this.#stopStart('pause', infoHash);
|
|
431
|
+
return true;
|
|
432
|
+
}
|
|
433
|
+
|
|
434
|
+
/**
|
|
435
|
+
* Offers a stopped torrent again.
|
|
436
|
+
* @param {string} infoHash - The archive to start.
|
|
437
|
+
* @returns {Promise<boolean>} - Whether it was started.
|
|
438
|
+
*/
|
|
439
|
+
async resume(infoHash) {
|
|
440
|
+
await this.#stopStart('resume', infoHash);
|
|
441
|
+
return true;
|
|
442
|
+
}
|
|
443
|
+
|
|
444
|
+
/**
|
|
445
|
+
* Pause and resume differ only in the word, including how they fail.
|
|
446
|
+
* @param {string} op - 'pause' or 'resume'.
|
|
447
|
+
* @param {string} infoHash - The archive.
|
|
448
|
+
* @returns {Promise<object>} - The sidecar's answer.
|
|
449
|
+
*/
|
|
450
|
+
async #stopStart(op, infoHash) {
|
|
451
|
+
try {
|
|
452
|
+
return await this.#call(op, { infoHash });
|
|
453
|
+
} catch (error) {
|
|
454
|
+
// An older sidecar answers "unknown op", which is true and useless: it
|
|
455
|
+
// reads as a bug in the request rather than as a package that needs
|
|
456
|
+
// updating. Worth saying plainly, because before 0.9.0 there was no
|
|
457
|
+
// pause here at all -- the request reached the catalog and stopped, so
|
|
458
|
+
// the console showed `paused` beside an archive still transferring.
|
|
459
|
+
if (/unknown op/i.test(error.message)) {
|
|
460
|
+
throw new Error(
|
|
461
|
+
`this sidecar cannot ${op}; pmtiles-torrent 0.9.0 or newer is needed`,
|
|
462
|
+
{ cause: error },
|
|
463
|
+
);
|
|
464
|
+
}
|
|
465
|
+
throw error;
|
|
466
|
+
}
|
|
467
|
+
}
|
|
468
|
+
|
|
418
469
|
async list() {
|
|
419
470
|
// A node that is shutting down still has a console polling it and a sweep
|
|
420
471
|
// or two in flight. Answering "the sidecar exited" to each of them fills
|
|
@@ -627,9 +678,7 @@ export class LibtorrentEngine {
|
|
|
627
678
|
child.on('close', (code, signal) => {
|
|
628
679
|
if (failure) return finish(failure);
|
|
629
680
|
if (result) return finish();
|
|
630
|
-
const why = signal
|
|
631
|
-
? `killed by ${signal}`
|
|
632
|
-
: `exited with code ${code}`;
|
|
681
|
+
const why = signal ? `killed by ${signal}` : `exited with code ${code}`;
|
|
633
682
|
finish(
|
|
634
683
|
new Error(
|
|
635
684
|
`hashing produced nothing (${why})${stderr ? `: ${stderr.trim()}` : ''}`,
|
package/src/library.js
CHANGED
|
@@ -272,17 +272,25 @@ export class Library {
|
|
|
272
272
|
options.onValidated?.({ path: absolute, kind: identified.kind });
|
|
273
273
|
|
|
274
274
|
// Tracked from here so the console has something to show while the hash
|
|
275
|
-
// runs.
|
|
276
|
-
//
|
|
277
|
-
//
|
|
278
|
-
//
|
|
275
|
+
// runs, and so it can be stopped. Hashing an archive this size is minutes
|
|
276
|
+
// at best and hours for a planet, all of it reading the disk everything
|
|
277
|
+
// else is served from; realising it was the wrong file should not mean
|
|
278
|
+
// waiting it out or killing the process.
|
|
279
|
+
//
|
|
280
|
+
// What it cancels is the hash. A publishDir move has already happened by
|
|
281
|
+
// here โ deliberately, since it is the cheap irreversible step โ so a
|
|
282
|
+
// cancelled add leaves the archive where it was published to, not where it
|
|
283
|
+
// was picked from. Nothing else is touched: hashing only ever reads.
|
|
284
|
+
const controller = new AbortController();
|
|
279
285
|
const { size } = await fs.stat(absolute).catch(() => ({ size: undefined }));
|
|
280
286
|
this.#running.set(requested, {
|
|
287
|
+
controller,
|
|
281
288
|
name: path.basename(absolute),
|
|
282
289
|
startedAt: new Date().toISOString(),
|
|
283
|
-
//
|
|
284
|
-
// bar
|
|
285
|
-
//
|
|
290
|
+
// Absent until the hasher says otherwise, rather than zero: a zero is
|
|
291
|
+
// drawn as a bar that has not moved, and an out-of-process hasher is not
|
|
292
|
+
// guaranteed โ config can ask for v1, and create-torrent reports nothing
|
|
293
|
+
// at all. Then it is honestly unknown rather than stuck at 0%.
|
|
286
294
|
received: undefined,
|
|
287
295
|
total: size,
|
|
288
296
|
phase: 'hashing',
|
|
@@ -302,6 +310,21 @@ export class Library {
|
|
|
302
310
|
webSeeds: [...new Set(webSeeds)],
|
|
303
311
|
comment: options.comment,
|
|
304
312
|
md5: options.md5 ?? this.#config.md5,
|
|
313
|
+
signal: controller.signal,
|
|
314
|
+
// Pieces in, bytes out. The console draws one progress bar for adds and
|
|
315
|
+
// labels it in bytes, and a piece count means nothing beside a download
|
|
316
|
+
// measured in gigabytes โ so it is converted here, where the file size
|
|
317
|
+
// is known, rather than teaching the console a second unit. Accurate to
|
|
318
|
+
// within one piece, which at 4 MiB against a planet archive is four
|
|
319
|
+
// parts in a million.
|
|
320
|
+
onHashProgress: ({ piece, pieces }) => {
|
|
321
|
+
const state = this.#running.get(requested);
|
|
322
|
+
if (!state || !pieces || !size) return;
|
|
323
|
+
state.received = Math.min(
|
|
324
|
+
size,
|
|
325
|
+
Math.round(((piece + 1) / pieces) * size),
|
|
326
|
+
);
|
|
327
|
+
},
|
|
305
328
|
});
|
|
306
329
|
|
|
307
330
|
return await this.#register(created, {
|
|
@@ -380,6 +403,10 @@ export class Library {
|
|
|
380
403
|
* trade-off: v2 clients gain per-file merkle trees and 16 KiB block
|
|
381
404
|
* verification, and v1 clients see an ordinary torrent. An engine that is
|
|
382
405
|
* only seeding is still the right one to ask for that.
|
|
406
|
+
*
|
|
407
|
+
* Everything the caller passes goes through, `signal` and `onProgress`
|
|
408
|
+
* included. Those are what make a hash out here cancellable and visible, and
|
|
409
|
+
* naming the fields it forwards would quietly drop them.
|
|
383
410
|
* @returns {Function | undefined} - A creator, or undefined to use the default.
|
|
384
411
|
*/
|
|
385
412
|
#creator() {
|
|
@@ -616,13 +643,18 @@ export class Library {
|
|
|
616
643
|
}
|
|
617
644
|
|
|
618
645
|
/**
|
|
619
|
-
* Stops an in-flight
|
|
646
|
+
* Stops an in-flight add, and discards what it had downloaded.
|
|
647
|
+
*
|
|
648
|
+
* Local adds too, now that hashing happens in a process of its own and that
|
|
649
|
+
* process can be ended. A local add discards nothing โ the archive is the
|
|
650
|
+
* caller's own file and was only ever read โ so what it costs is the hashing
|
|
651
|
+
* done so far and nothing else.
|
|
620
652
|
*
|
|
621
|
-
*
|
|
622
|
-
*
|
|
623
|
-
*
|
|
624
|
-
*
|
|
625
|
-
* @param {string} [url] - The source URL, or all of them when omitted.
|
|
653
|
+
* An add hashing in this process rather than out of it still cannot be
|
|
654
|
+
* interrupted, and libtorrent's hashing never checks for interruption
|
|
655
|
+
* either. Cancelling such an add stops the caller waiting and frees the
|
|
656
|
+
* console of it; the read finishes on its own.
|
|
657
|
+
* @param {string} [url] - The source URL or path, or all of them when omitted.
|
|
626
658
|
* @returns {string[]} - The URLs cancelled.
|
|
627
659
|
*/
|
|
628
660
|
cancelAdd(url) {
|
|
@@ -679,10 +711,12 @@ export class Library {
|
|
|
679
711
|
return [...this.#running.entries()].map(([url, state]) => ({
|
|
680
712
|
url,
|
|
681
713
|
name: state.name,
|
|
682
|
-
//
|
|
683
|
-
//
|
|
684
|
-
//
|
|
685
|
-
|
|
714
|
+
// Absent means unknown, not zero. A hash reports where it has got to
|
|
715
|
+
// only when it runs out of process; the in-process fallback reports
|
|
716
|
+
// nothing at all, and a bar pinned at 0% for forty minutes says "stuck"
|
|
717
|
+
// when the honest answer is "no idea, still working". A download always
|
|
718
|
+
// knows, so it is 0 there from the start.
|
|
719
|
+
received: state.received ?? (state.phase === 'hashing' ? undefined : 0),
|
|
686
720
|
total: state.total,
|
|
687
721
|
startedAt: state.startedAt,
|
|
688
722
|
phase: state.phase ?? 'fetching',
|
|
@@ -1832,11 +1866,24 @@ export class Library {
|
|
|
1832
1866
|
throw error;
|
|
1833
1867
|
}
|
|
1834
1868
|
|
|
1835
|
-
|
|
1836
|
-
|
|
1837
|
-
|
|
1869
|
+
// Whether it actually stopped, not whether something was asked.
|
|
1870
|
+
//
|
|
1871
|
+
// This tested only that the engine *had* a pause method and threw the
|
|
1872
|
+
// answer away. A composite has one whatever its engines can do, so a
|
|
1873
|
+
// primary with no pause of its own returned false into a void: the
|
|
1874
|
+
// fallback below never ran, `paused: true` went into the catalog, and the
|
|
1875
|
+
// console -- which prefers that flag to the engine's live state -- showed
|
|
1876
|
+
// `paused` beside an archive still transferring at 8 MiB/s. The button did
|
|
1877
|
+
// nothing and said it had worked.
|
|
1878
|
+
const stopped = this.#engine.pause
|
|
1879
|
+
? await this.#engine.pause(infoHash)
|
|
1880
|
+
: false;
|
|
1881
|
+
if (!stopped) {
|
|
1838
1882
|
// Removing without its data is a pause an engine cannot refuse; resume
|
|
1839
|
-
// adds it back and it rechecks what is already on disk.
|
|
1883
|
+
// adds it back and it rechecks what is already on disk. A last resort,
|
|
1884
|
+
// because that recheck is the whole store -- tens of minutes for a
|
|
1885
|
+
// planet archive -- which is why an engine that can really pause is
|
|
1886
|
+
// worth the two operations it takes.
|
|
1840
1887
|
await this.#engine
|
|
1841
1888
|
.remove(infoHash, { deleteData: false })
|
|
1842
1889
|
.catch(() => {});
|
|
@@ -1920,9 +1967,14 @@ export class Library {
|
|
|
1920
1967
|
throw error;
|
|
1921
1968
|
}
|
|
1922
1969
|
|
|
1923
|
-
|
|
1924
|
-
|
|
1925
|
-
|
|
1970
|
+
// Same as pause: the answer decides, not the presence of a method. An
|
|
1971
|
+
// archive stopped by the fallback above is not in the engine at all, so a
|
|
1972
|
+
// resume it merely claimed would leave the catalog saying the archive was
|
|
1973
|
+
// running while nothing held it.
|
|
1974
|
+
const started = this.#engine.resume
|
|
1975
|
+
? await this.#engine.resume(infoHash)
|
|
1976
|
+
: false;
|
|
1977
|
+
if (!started) {
|
|
1926
1978
|
await this.#readd({ ...entry, paused: false });
|
|
1927
1979
|
}
|
|
1928
1980
|
await this.#tiles?.invalidate(infoHash).catch(() => {});
|
package/src/torrent-create.js
CHANGED
|
@@ -19,7 +19,9 @@ import { DEFAULT_SUFFIX } from './incomplete.js';
|
|
|
19
19
|
* @property {number} [pieceLength] - Piece size in bytes; must be a power of two.
|
|
20
20
|
* @property {string[]} [trackers] - Announce URLs.
|
|
21
21
|
* @property {string[]} [webSeeds] - BEP 19 url-list entries.
|
|
22
|
-
* @property {AbortSignal} [signal] - Cancels a download in progress.
|
|
22
|
+
* @property {AbortSignal} [signal] - Cancels a download or a hash in progress.
|
|
23
|
+
* @property {Function} [onProgress] - Called with {phase, received, total} as the archive arrives.
|
|
24
|
+
* @property {Function} [onHashProgress] - Called with {piece, pieces} as it is hashed.
|
|
23
25
|
* @property {boolean} [md5] - Also compute an MD5 of the archive.
|
|
24
26
|
* @property {boolean} [includeSourceAsWebSeed] - Publish the source URL as a web seed. Default true.
|
|
25
27
|
* @property {string} [comment] - Free-text comment.
|
|
@@ -64,9 +66,22 @@ export async function createTorrentFromFile(filePath, options = {}) {
|
|
|
64
66
|
`[hash] ${what}: reading ${stat.size} bytes ${passes} to build the torrent`,
|
|
65
67
|
);
|
|
66
68
|
const startedAt = Date.now();
|
|
69
|
+
|
|
70
|
+
// Where the hash has got to, for the heartbeat. A hasher that reports pieces
|
|
71
|
+
// turns "still hashing after 41m" into something that says whether waiting
|
|
72
|
+
// longer is worth it; one that does not says exactly what it said before.
|
|
73
|
+
let reached;
|
|
74
|
+
const onHashProgress = (progress) => {
|
|
75
|
+
reached = progress;
|
|
76
|
+
options.onHashProgress?.(progress);
|
|
77
|
+
};
|
|
78
|
+
|
|
67
79
|
const heartbeat = setInterval(() => {
|
|
68
80
|
const minutes = Math.round((Date.now() - startedAt) / 60000);
|
|
69
|
-
|
|
81
|
+
const far = reached?.pieces
|
|
82
|
+
? ` (${(((reached.piece + 1) / reached.pieces) * 100).toFixed(1)}%)`
|
|
83
|
+
: '';
|
|
84
|
+
console.log(`[hash] ${what}: still hashing after ${minutes}m${far}`);
|
|
70
85
|
}, 60000);
|
|
71
86
|
heartbeat.unref?.();
|
|
72
87
|
options.onProgress?.({
|
|
@@ -78,12 +93,14 @@ export async function createTorrentFromFile(filePath, options = {}) {
|
|
|
78
93
|
try {
|
|
79
94
|
// A second read of the archive, which is why it is opt-in. Nothing else
|
|
80
95
|
// here touches these bytes again once the piece hashes are done.
|
|
81
|
-
const md5Digest = options.md5
|
|
96
|
+
const md5Digest = options.md5
|
|
97
|
+
? await md5File(filePath, options.signal)
|
|
98
|
+
: undefined;
|
|
82
99
|
const built = await buildTorrent(
|
|
83
100
|
filePath,
|
|
84
101
|
path.basename(filePath),
|
|
85
102
|
stat.size,
|
|
86
|
-
{ ...options, md5Digest },
|
|
103
|
+
{ ...options, md5Digest, onHashProgress },
|
|
87
104
|
);
|
|
88
105
|
const seconds = Math.round((Date.now() - startedAt) / 1000);
|
|
89
106
|
console.log(`[hash] ${what}: torrent built in ${seconds}s`);
|
|
@@ -618,13 +635,18 @@ function delay(ms, signal) {
|
|
|
618
635
|
* on disk it is a second full read. See docs/internals.md โ "Why the MD5 is
|
|
619
636
|
* opt-in".
|
|
620
637
|
* @param {string} filePath - The file to hash.
|
|
638
|
+
* @param {AbortSignal} [signal] - Stops the read.
|
|
621
639
|
* @returns {Promise<string>} - Lowercase hex digest.
|
|
622
640
|
*/
|
|
623
|
-
async function md5File(filePath) {
|
|
641
|
+
async function md5File(filePath, signal) {
|
|
624
642
|
const { createReadStream } = await import('node:fs');
|
|
625
643
|
const { createHash } = await import('node:crypto');
|
|
626
644
|
const hash = createHash('md5');
|
|
627
|
-
|
|
645
|
+
// Cancellable like the piece hashing it precedes. This is a second full read
|
|
646
|
+
// of the archive, so a cancel arriving during it would otherwise be ignored
|
|
647
|
+
// for as long as the pass it was meant to stop โ for a planet archive, most
|
|
648
|
+
// of an hour of disk after somebody pressed the button.
|
|
649
|
+
await pipelineAsync(createReadStream(filePath), hash, { signal });
|
|
628
650
|
return hash.digest('hex');
|
|
629
651
|
}
|
|
630
652
|
|
|
@@ -655,6 +677,13 @@ async function buildTorrent(input, name, size, options) {
|
|
|
655
677
|
private: options.private ?? false,
|
|
656
678
|
createdBy: 'pmtiles-swarm',
|
|
657
679
|
format: options.format ?? 'hybrid',
|
|
680
|
+
// Both only mean anything to a creator that hashes out of process,
|
|
681
|
+
// which is the whole reason there is one. `onProgress` here is the
|
|
682
|
+
// creator's own shape โ {piece, pieces} โ and is deliberately not the
|
|
683
|
+
// {phase, received, total} one this module reports to its caller; they
|
|
684
|
+
// describe different things and were briefly the same name.
|
|
685
|
+
signal: options.signal,
|
|
686
|
+
onProgress: options.onHashProgress,
|
|
658
687
|
});
|
|
659
688
|
|
|
660
689
|
const madeBy = await parseTorrent(built.torrentFile);
|
|
@@ -670,6 +699,15 @@ async function buildTorrent(input, name, size, options) {
|
|
|
670
699
|
md5: options.md5Digest,
|
|
671
700
|
};
|
|
672
701
|
} catch (error) {
|
|
702
|
+
// Unless what stopped it was somebody stopping it.
|
|
703
|
+
//
|
|
704
|
+
// Falling back here would answer "cancel this hash" by starting the same
|
|
705
|
+
// hash again in this process, where it cannot be cancelled at all โ so
|
|
706
|
+
// the button would make a 698 GiB read strictly worse than leaving it
|
|
707
|
+
// alone. Cancelling is a decision about the archive, not a fault to
|
|
708
|
+
// route around.
|
|
709
|
+
if (options.signal?.aborted) throw error;
|
|
710
|
+
|
|
673
711
|
// A torrent is more important than the format of a torrent.
|
|
674
712
|
//
|
|
675
713
|
// Said with what it costs, though, because the fallback is not a smaller
|
package/src/web/index.html
CHANGED
|
@@ -309,6 +309,16 @@
|
|
|
309
309
|
.piecerow { display: grid; grid-template-columns: 7rem 1fr 4rem; gap: 0.6rem; align-items: center; margin-bottom: 0.5rem; }
|
|
310
310
|
.piecerow > .label { font-size: 0.8rem; color: var(--muted); }
|
|
311
311
|
.piecerow > .value { font-size: 0.8rem; color: var(--muted); text-align: right; }
|
|
312
|
+
/* Like a piecerow, but the label is a filename rather than a fixed word,
|
|
313
|
+
and each row carries its own Cancel. A separate class because the
|
|
314
|
+
piecerow's 7rem label and 4rem value are sized for "downloaded" and
|
|
315
|
+
"41%": an archive name in 7rem wraps onto three lines, and the naming
|
|
316
|
+
of the buttons had to repeat the whole filename to say which was
|
|
317
|
+
which. */
|
|
318
|
+
.addrow { display: grid; grid-template-columns: minmax(0, 1fr) auto auto auto; gap: 0.6rem; align-items: center; margin-bottom: 0.5rem; }
|
|
319
|
+
.addrow > .label { font-size: 0.8rem; color: var(--muted); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
|
|
320
|
+
.addrow > .value { font-size: 0.8rem; color: var(--muted); text-align: right; white-space: nowrap; }
|
|
321
|
+
.addrow > button { padding: 0.15rem 0.55rem; font-size: 0.75rem; }
|
|
312
322
|
|
|
313
323
|
button.speed { border-color: var(--line); color: var(--muted); }
|
|
314
324
|
button.speed.on { border-color: var(--warn); color: var(--warn); }
|
|
@@ -923,7 +933,6 @@
|
|
|
923
933
|
box.innerHTML = '';
|
|
924
934
|
return;
|
|
925
935
|
}
|
|
926
|
-
const cancellable = running.filter((add) => add.cancellable !== false);
|
|
927
936
|
box.innerHTML = `
|
|
928
937
|
<h3 style="margin:1.2rem 0 0.5rem">Being added, before a torrent exists</h3>
|
|
929
938
|
<div class="sub" style="margin-bottom:0.6rem">
|
|
@@ -934,9 +943,10 @@
|
|
|
934
943
|
</div>
|
|
935
944
|
${running
|
|
936
945
|
.map((add) => {
|
|
937
|
-
//
|
|
938
|
-
//
|
|
939
|
-
//
|
|
946
|
+
// A hash says how far through it is when it runs out of process,
|
|
947
|
+
// and says nothing when it falls back to hashing in this one. So
|
|
948
|
+
// the percentage is drawn where there is one and left out where
|
|
949
|
+
// there is not โ a bar pinned at zero for twenty minutes says
|
|
940
950
|
// "stuck" when the honest answer is "no idea, still working".
|
|
941
951
|
const hashing = add.phase === 'hashing';
|
|
942
952
|
const pct =
|
|
@@ -947,32 +957,39 @@
|
|
|
947
957
|
? duration(Date.now() - new Date(add.startedAt).getTime())
|
|
948
958
|
: null;
|
|
949
959
|
const value = hashing
|
|
950
|
-
?
|
|
951
|
-
|
|
952
|
-
|
|
960
|
+
? [
|
|
961
|
+
`hashing${add.total ? ` ${bytes(add.total)}` : ''}`,
|
|
962
|
+
pct == null ? null : `${pct.toFixed(1)}%`,
|
|
963
|
+
elapsed,
|
|
964
|
+
]
|
|
965
|
+
.filter(Boolean)
|
|
966
|
+
.join(' ยท ')
|
|
953
967
|
: pct == null
|
|
954
968
|
? bytes(add.received)
|
|
955
969
|
: `${pct.toFixed(1)}%`;
|
|
970
|
+
const name = add.name ?? add.url.split('/').pop() ?? add.url;
|
|
971
|
+
// Cancel sits in the row it cancels, so the button does not have
|
|
972
|
+
// to name the archive to say which one it is. Named, they were as
|
|
973
|
+
// wide as the filename โ two of them filled a line, and reading
|
|
974
|
+
// one meant matching a long name against the list above it.
|
|
975
|
+
//
|
|
976
|
+
// An add with no controller keeps an empty cell rather than
|
|
977
|
+
// losing one, so the rows above and below it stay lined up.
|
|
978
|
+
const stop =
|
|
979
|
+
add.cancellable === false
|
|
980
|
+
? '<span></span>'
|
|
981
|
+
: `<button data-cancel="${escapeHtml(add.url)}" title="Cancel ${escapeHtml(
|
|
982
|
+
name,
|
|
983
|
+
)}" aria-label="Cancel ${escapeHtml(name)}">Cancel</button>`;
|
|
956
984
|
return `
|
|
957
|
-
<div class="
|
|
958
|
-
<span class="label" title="${escapeHtml(add.url)}">${escapeHtml(
|
|
959
|
-
add.name ?? add.url.split('/').pop() ?? add.url,
|
|
960
|
-
)}</span>
|
|
985
|
+
<div class="addrow">
|
|
986
|
+
<span class="label" title="${escapeHtml(add.url)}">${escapeHtml(name)}</span>
|
|
961
987
|
<span class="track"><i style="width:${pct == null ? 0 : pct.toFixed(1)}%"></i></span>
|
|
962
988
|
<span class="value">${value}</span>
|
|
989
|
+
${stop}
|
|
963
990
|
</div>`;
|
|
964
991
|
})
|
|
965
|
-
.join('')}
|
|
966
|
-
<div class="bar" style="margin-top:0.5rem">
|
|
967
|
-
${cancellable
|
|
968
|
-
.map(
|
|
969
|
-
(add) =>
|
|
970
|
-
`<button data-cancel="${escapeHtml(add.url)}">Cancel ${escapeHtml(
|
|
971
|
-
add.name ?? add.url.split('/').pop() ?? '',
|
|
972
|
-
)}</button>`,
|
|
973
|
-
)
|
|
974
|
-
.join('')}
|
|
975
|
-
</div>`;
|
|
992
|
+
.join('')}`;
|
|
976
993
|
|
|
977
994
|
for (const button of box.querySelectorAll('[data-cancel]')) {
|
|
978
995
|
button.onclick = async () => {
|
|
@@ -2370,7 +2387,7 @@ Every piece is hashed against the ` +
|
|
|
2370
2387
|
kind === 'url'
|
|
2371
2388
|
? 'fetching โ watch the log'
|
|
2372
2389
|
: kind === 'path'
|
|
2373
|
-
? 'hashing โ
|
|
2390
|
+
? 'hashing โ progress is below'
|
|
2374
2391
|
: 'added',
|
|
2375
2392
|
);
|
|
2376
2393
|
refresh();
|