pmtiles-swarm 0.35.5 → 0.37.0

package/CHANGELOG.md

@@ -7,6 +7,55 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.37.0
+### ✨ Features and improvements
+- **An archive being hashed can now be cancelled, and says how far through it is.** 0.36.0 moved
+  hashing into a process of its own, which made both possible; this connects them to the console.
+  A local add is now registered with an AbortController, so `DELETE /api/adds?url=<path>` ends the
+  hasher and the Cancel button beside it works. Nothing is lost by pressing it: the archive is the
+  caller's own file and hashing only ever read it.
+
+  Shutdown reaches these too. A hasher left behind when the node exits is an orphan reading the
+  disk for hours, answering to nothing.
+
+  The piece the hasher has reached is converted to bytes against the file size — the console draws
+  one progress bar for adds and labels it in bytes — so a long hash reads `hashing 698 GiB · 41.2%
+  · 12m` instead of `hashing 698 GiB · 3m`. The hourly log line carries the percentage as well. An
+  add hashing in this process rather than out of it still reports no figure, because there is none
+  to report, and the bar is left off rather than pinned at zero.
+
+### 🐞 Bug fixes
+- **Cancelling a hash no longer answers by hashing the same archive here instead.** A creator that
+  fails falls back to hashing in the node's own process, deliberately, since a torrent matters more
+  than the format of a torrent — and a cancelled hash arrives as a creator that failed. So the
+  button would have answered "stop reading 698 GiB" by reading 698 GiB again, in the process
+  serving tiles, where nothing can interrupt it at all. A creator that fails for any other reason
+  still falls back.
+- **A cancel arriving during the MD5 pass is no longer ignored until it finishes.** With `md5` on,
+  the archive is read twice, and only the second read took a signal — so cancelling during the
+  first one waited out most of an hour of disk on a planet archive before it took effect.
+
+## 0.36.0
+### ✨ Features and improvements
+- **Hashing an archive now happens in a process of its own.** Building the torrent for a 698 GiB
+  archive ran inside the libtorrent sidecar, competing with the session for the disk and for
+  Python's interpreter lock while every archive on the node was being served from that same disk.
+  It also could not be stopped: libtorrent's hashing never checks for interruption, and the sidecar
+  cannot be ended to end a hash because it holds the session and every torrent seeding from it. A
+  build started by a misclick ran its full six hours.
+
+  It is now `libtorrent_sidecar.py --create`, started per hash, holding no session and no port.
+  Killing it costs the hash and nothing else, and hashing only ever reads, so the archive is
+  untouched. It reports the piece it has reached as it goes, so a caller can draw a real fraction
+  rather than "hashing 698 GiB · 3m".
+
+  Requires pmtiles-torrent 0.8.0. Also picks up 0.7.5, which stops an archive that is hashing its
+  store from reporting itself as "paused" — libtorrent hashes one store at a time and flags every
+  torrent queued behind it as paused, so a library busy verifying itself read as one somebody had
+  stopped.
+
+### 🐞 Bug fixes
+
 ## 0.35.5
 ### ✨ Features and improvements
 

package/docs/publishing.md

@@ -79,6 +79,25 @@ A magnet, a `.torrent` URL and an uploaded `.torrent` are metadata rather than
 data, so there is nothing slow to wait for: those still answer `201` with the
 entry.
 
+### Stopping one that is running
+
+`DELETE /api/adds?url=…` stops an add before it has produced a torrent, and the
+console offers it as a Cancel button beside each one. The identifier is whatever
+the add was started with — the URL for a remote one, the path for a local one.
+
+What it costs differs by kind. A **remote** add is deleting a partial download,
+deliberately: somebody said stop, and leaving a few hundred gigabytes behind
+after that is waste nobody will ever find again. A **local** add deletes
+nothing. The archive is your own file and hashing only ever read it, so
+cancelling costs the hashing done so far and not a byte of data. If `publishDir`
+moved the file, that already happened — it is the cheap irreversible step and it
+goes first — so the archive stays where it was published to.
+
+`GET /api/adds` reports each one's progress while it runs. A local add hashing
+out of process says how far through the archive it is; one that has fallen back
+to hashing in the node's own process reports no figure at all, because there is
+none to report, and cancelling it stops the wait rather than the read.
+
 ### When the source URL is not published
 
 Adding from a URL registers that URL as a web seed by default, because it is by

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.35.5",
+  "version": "0.37.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",
@@ -46,7 +46,7 @@
     "maplibre-gl": "^6.2.0",
     "parse-torrent": "^11.0.24",
     "pmtiles": "^4.4.1",
-    "pmtiles-torrent": "^0.7.4",
+    "pmtiles-torrent": "^0.8.0",
     "webtorrent": "^3.0.21"
   },
   "engines": {

package/src/engines/libtorrent.js

@@ -491,13 +491,23 @@ export class LibtorrentEngine {
 
   /**
    * Creates a torrent from a local file.
+   *
+   * In a process of its own rather than over the pipe. libtorrent's hashing
+   * never checks for interruption, so a hash running inside the sidecar could
+   * not be stopped -- and the sidecar itself cannot be ended to stop one,
+   * because it holds the session and every torrent seeding from it. A 698 GiB
+   * build started by a misclick therefore ran its full six hours, saturating
+   * the disk the rest of the library was being served from.
+   *
+   * A process started for one hash can simply be killed. Hashing only reads,
+   * so nothing is left half-written, and the archive is untouched.
    * @param {string} filePath - The file to hash.
-   * @param {object} [options] - Piece length, trackers, web seeds, format.
+   * @param {object} [options] - Piece length, trackers, web seeds, format,
+   *   `signal` to cancel with, and `onProgress({piece, pieces})`.
    * @returns {Promise<object>} - The torrent file and what it describes.
    */
   async createTorrent(filePath, options = {}) {
-    const result = await this.#call(
-      'create',
+    const result = await this.#hashApart(
       {
         path: filePath,
         pieceLength: options.pieceLength,
@@ -508,8 +518,7 @@ export class LibtorrentEngine {
         createdBy: options.createdBy,
         format: options.format ?? 'hybrid',
       },
-      // Hashing a large archive takes as long as it takes.
-      options.timeoutMs ?? 6 * 60 * 60 * 1000,
+      options,
     );
     return {
       ...result,
@@ -517,6 +526,120 @@ export class LibtorrentEngine {
     };
   }
 
+  /**
+   * Runs one `--create` to completion, or until it is no longer wanted.
+   * @param {object} params - What to hash and how.
+   * @param {object} options - signal, onProgress, timeoutMs.
+   * @returns {Promise<object>} - The sidecar's result object.
+   */
+  #hashApart(params, options) {
+    return new Promise((resolve, reject) => {
+      const script = this.#options.script ?? resolveSidecar();
+      const child = spawn(this.#options.python, [script, '--create'], {
+        stdio: ['pipe', 'pipe', 'pipe'],
+      });
+
+      let settled = false;
+      let result;
+      let failure;
+      let pending = '';
+      let stderr = '';
+
+      /**
+       * Ends this hash once, whatever ends it.
+       * @param {Error} [error] - Why, if it failed.
+       */
+      const finish = (error) => {
+        if (settled) return;
+        settled = true;
+        clearTimeout(timer);
+        options.signal?.removeEventListener('abort', cancel);
+        if (error) reject(error);
+        else resolve(result);
+      };
+
+      const cancel = () => {
+        // The hash cannot be asked to stop, so it is ended. Nothing is lost:
+        // hashing reads and this process holds nothing else.
+        child.kill();
+        // Said the same way however it was cancelled. An AbortController with
+        // no reason gives "This operation was aborted", which in a log next to
+        // an archive name explains nothing; callers that care which kind of
+        // stop this was read `signal.reason`, which is carried as the cause.
+        finish(
+          new Error('hashing was cancelled', { cause: options.signal?.reason }),
+        );
+      };
+
+      // Hashing a large archive takes as long as it takes, but not forever:
+      // a hash that has stopped reporting is stuck, and holding the add open
+      // for six hours to discover that helps nobody.
+      const timer = setTimeout(
+        () => {
+          child.kill();
+          finish(new Error('hashing timed out'));
+        },
+        options.timeoutMs ?? 6 * 60 * 60 * 1000,
+      );
+      timer.unref?.();
+
+      if (options.signal?.aborted) return cancel();
+      options.signal?.addEventListener('abort', cancel, { once: true });
+
+      child.stdout.setEncoding('utf8');
+      child.stdout.on('data', (chunk) => {
+        pending += chunk;
+        const lines = pending.split('\n');
+        pending = lines.pop() ?? '';
+        for (const line of lines) {
+          if (!line.trim()) continue;
+          let message;
+          try {
+            message = JSON.parse(line);
+          } catch {
+            // A line that is not ours. Python's own output on the way to a
+            // crash arrives here, and is worth keeping for the error.
+            stderr += `${line}\n`;
+            continue;
+          }
+          if (message.event === 'progress') {
+            options.onProgress?.({
+              piece: message.piece,
+              pieces: message.pieces,
+            });
+          } else if (message.ok) {
+            result = message.result;
+          } else if (message.ok === false) {
+            failure = new Error(message.error ?? 'hashing failed');
+          }
+        }
+      });
+
+      child.stderr.setEncoding('utf8');
+      child.stderr.on('data', (chunk) => {
+        stderr += chunk;
+      });
+
+      child.on('error', (error) =>
+        finish(new Error(`could not start the hasher: ${error.message}`)),
+      );
+
+      child.on('close', (code, signal) => {
+        if (failure) return finish(failure);
+        if (result) return finish();
+        const why = signal ? `killed by ${signal}` : `exited with code ${code}`;
+        finish(
+          new Error(
+            `hashing produced nothing (${why})${stderr ? `: ${stderr.trim()}` : ''}`,
+          ),
+        );
+      });
+
+      child.stdin.on('error', () => {});
+      child.stdin.end(JSON.stringify(params));
+    });
+  }
+
   /**
    * Reads one piece, promoted ahead of the normal picker.
    *

package/src/library.js

@@ -272,17 +272,25 @@ export class Library {
     options.onValidated?.({ path: absolute, kind: identified.kind });
 
     // Tracked from here so the console has something to show while the hash
-    // runs. Registered without an AbortController, unlike a remote add: there
-    // is no cancelling a hash in progress, since neither libtorrent's creator
-    // nor create-torrent takes a signal. cancelAdd() skips entries without a
-    // controller, so this is inert there rather than a button that lies.
+    // runs, and so it can be stopped. Hashing an archive this size is minutes
+    // at best and hours for a planet, all of it reading the disk everything
+    // else is served from; realising it was the wrong file should not mean
+    // waiting it out or killing the process.
+    //
+    // What it cancels is the hash. A publishDir move has already happened by
+    // here — deliberately, since it is the cheap irreversible step — so a
+    // cancelled add leaves the archive where it was published to, not where it
+    // was picked from. Nothing else is touched: hashing only ever reads.
+    const controller = new AbortController();
     const { size } = await fs.stat(absolute).catch(() => ({ size: undefined }));
     this.#running.set(requested, {
+      controller,
       name: path.basename(absolute),
       startedAt: new Date().toISOString(),
-      // No byte count: hashing reports nothing until it is done, so a progress
-      // bar here would sit at zero and then vanish. `total` is the size it is
-      // working through, which with the start time is enough to show it moving.
+      // Absent until the hasher says otherwise, rather than zero: a zero is
+      // drawn as a bar that has not moved, and an out-of-process hasher is not
+      // guaranteed — config can ask for v1, and create-torrent reports nothing
+      // at all. Then it is honestly unknown rather than stuck at 0%.
       received: undefined,
       total: size,
       phase: 'hashing',
@@ -302,6 +310,21 @@ export class Library {
         webSeeds: [...new Set(webSeeds)],
         comment: options.comment,
         md5: options.md5 ?? this.#config.md5,
+        signal: controller.signal,
+        // Pieces in, bytes out. The console draws one progress bar for adds and
+        // labels it in bytes, and a piece count means nothing beside a download
+        // measured in gigabytes — so it is converted here, where the file size
+        // is known, rather than teaching the console a second unit. Accurate to
+        // within one piece, which at 4 MiB against a planet archive is four
+        // parts in a million.
+        onHashProgress: ({ piece, pieces }) => {
+          const state = this.#running.get(requested);
+          if (!state || !pieces || !size) return;
+          state.received = Math.min(
+            size,
+            Math.round(((piece + 1) / pieces) * size),
+          );
+        },
       });
 
       return await this.#register(created, {
@@ -380,6 +403,10 @@ export class Library {
    * trade-off: v2 clients gain per-file merkle trees and 16 KiB block
    * verification, and v1 clients see an ordinary torrent. An engine that is
    * only seeding is still the right one to ask for that.
+   *
+   * Everything the caller passes goes through, `signal` and `onProgress`
+   * included. Those are what make a hash out here cancellable and visible, and
+   * naming the fields it forwards would quietly drop them.
    * @returns {Function | undefined} - A creator, or undefined to use the default.
    */
   #creator() {
@@ -616,13 +643,18 @@ export class Library {
   }
 
   /**
-   * Stops an in-flight remote add, and discards what it had downloaded.
+   * Stops an in-flight add, and discards what it had downloaded.
+   *
+   * Local adds too, now that hashing happens in a process of its own and that
+   * process can be ended. A local add discards nothing — the archive is the
+   * caller's own file and was only ever read — so what it costs is the hashing
+   * done so far and nothing else.
    *
-   * Only a remote one. A local add is listed alongside them but holds no
-   * controller, because a hash in progress cannot be interrupted — neither
-   * libtorrent's creator nor create-torrent takes a signal. It is skipped
-   * below rather than special-cased, and reports as nothing cancelled.
-   * @param {string} [url] - The source URL, or all of them when omitted.
+   * An add hashing in this process rather than out of it still cannot be
+   * interrupted, and libtorrent's hashing never checks for interruption
+   * either. Cancelling such an add stops the caller waiting and frees the
+   * console of it; the read finishes on its own.
+   * @param {string} [url] - The source URL or path, or all of them when omitted.
    * @returns {string[]} - The URLs cancelled.
    */
   cancelAdd(url) {
@@ -679,10 +711,12 @@ export class Library {
     return [...this.#running.entries()].map(([url, state]) => ({
       url,
       name: state.name,
-      // Left absent rather than zeroed for a hash, which reports nothing until
-      // it finishes. A zero here is indistinguishable from a download that has
-      // not moved, and the console draws the two differently.
-      received: state.phase === 'hashing' ? undefined : (state.received ?? 0),
+      // Absent means unknown, not zero. A hash reports where it has got to
+      // only when it runs out of process; the in-process fallback reports
+      // nothing at all, and a bar pinned at 0% for forty minutes says "stuck"
+      // when the honest answer is "no idea, still working". A download always
+      // knows, so it is 0 there from the start.
+      received: state.received ?? (state.phase === 'hashing' ? undefined : 0),
       total: state.total,
       startedAt: state.startedAt,
       phase: state.phase ?? 'fetching',

package/src/torrent-create.js

@@ -19,7 +19,9 @@ import { DEFAULT_SUFFIX } from './incomplete.js';
  * @property {number} [pieceLength] - Piece size in bytes; must be a power of two.
  * @property {string[]} [trackers] - Announce URLs.
  * @property {string[]} [webSeeds] - BEP 19 url-list entries.
- * @property {AbortSignal} [signal] - Cancels a download in progress.
+ * @property {AbortSignal} [signal] - Cancels a download or a hash in progress.
+ * @property {Function} [onProgress] - Called with {phase, received, total} as the archive arrives.
+ * @property {Function} [onHashProgress] - Called with {piece, pieces} as it is hashed.
  * @property {boolean} [md5] - Also compute an MD5 of the archive.
  * @property {boolean} [includeSourceAsWebSeed] - Publish the source URL as a web seed. Default true.
  * @property {string} [comment] - Free-text comment.
@@ -64,9 +66,22 @@ export async function createTorrentFromFile(filePath, options = {}) {
     `[hash] ${what}: reading ${stat.size} bytes ${passes} to build the torrent`,
   );
   const startedAt = Date.now();
+
+  // Where the hash has got to, for the heartbeat. A hasher that reports pieces
+  // turns "still hashing after 41m" into something that says whether waiting
+  // longer is worth it; one that does not says exactly what it said before.
+  let reached;
+  const onHashProgress = (progress) => {
+    reached = progress;
+    options.onHashProgress?.(progress);
+  };
+
   const heartbeat = setInterval(() => {
     const minutes = Math.round((Date.now() - startedAt) / 60000);
-    console.log(`[hash] ${what}: still hashing after ${minutes}m`);
+    const far = reached?.pieces
+      ? ` (${(((reached.piece + 1) / reached.pieces) * 100).toFixed(1)}%)`
+      : '';
+    console.log(`[hash] ${what}: still hashing after ${minutes}m${far}`);
   }, 60000);
   heartbeat.unref?.();
   options.onProgress?.({
@@ -78,12 +93,14 @@ export async function createTorrentFromFile(filePath, options = {}) {
   try {
     // A second read of the archive, which is why it is opt-in. Nothing else
     // here touches these bytes again once the piece hashes are done.
-    const md5Digest = options.md5 ? await md5File(filePath) : undefined;
+    const md5Digest = options.md5
+      ? await md5File(filePath, options.signal)
+      : undefined;
     const built = await buildTorrent(
       filePath,
       path.basename(filePath),
       stat.size,
-      { ...options, md5Digest },
+      { ...options, md5Digest, onHashProgress },
     );
     const seconds = Math.round((Date.now() - startedAt) / 1000);
     console.log(`[hash] ${what}: torrent built in ${seconds}s`);
@@ -618,13 +635,18 @@ function delay(ms, signal) {
  * on disk it is a second full read. See docs/internals.md — "Why the MD5 is
  * opt-in".
  * @param {string} filePath - The file to hash.
+ * @param {AbortSignal} [signal] - Stops the read.
  * @returns {Promise<string>} - Lowercase hex digest.
  */
-async function md5File(filePath) {
+async function md5File(filePath, signal) {
   const { createReadStream } = await import('node:fs');
   const { createHash } = await import('node:crypto');
   const hash = createHash('md5');
-  await pipelineAsync(createReadStream(filePath), hash);
+  // Cancellable like the piece hashing it precedes. This is a second full read
+  // of the archive, so a cancel arriving during it would otherwise be ignored
+  // for as long as the pass it was meant to stop — for a planet archive, most
+  // of an hour of disk after somebody pressed the button.
+  await pipelineAsync(createReadStream(filePath), hash, { signal });
   return hash.digest('hex');
 }
 
@@ -655,6 +677,13 @@ async function buildTorrent(input, name, size, options) {
         private: options.private ?? false,
         createdBy: 'pmtiles-swarm',
         format: options.format ?? 'hybrid',
+        // Both only mean anything to a creator that hashes out of process,
+        // which is the whole reason there is one. `onProgress` here is the
+        // creator's own shape — {piece, pieces} — and is deliberately not the
+        // {phase, received, total} one this module reports to its caller; they
+        // describe different things and were briefly the same name.
+        signal: options.signal,
+        onProgress: options.onHashProgress,
       });
 
       const madeBy = await parseTorrent(built.torrentFile);
@@ -670,6 +699,15 @@ async function buildTorrent(input, name, size, options) {
         md5: options.md5Digest,
       };
     } catch (error) {
+      // Unless what stopped it was somebody stopping it.
+      //
+      // Falling back here would answer "cancel this hash" by starting the same
+      // hash again in this process, where it cannot be cancelled at all — so
+      // the button would make a 698 GiB read strictly worse than leaving it
+      // alone. Cancelling is a decision about the archive, not a fault to
+      // route around.
+      if (options.signal?.aborted) throw error;
+
       // A torrent is more important than the format of a torrent.
       //
       // Said with what it costs, though, because the fallback is not a smaller

package/src/web/index.html

@@ -934,9 +934,10 @@
           </div>
           ${running
             .map((add) => {
-              // Hashing reports nothing until it is done, so there is no
-              // percentage to draw — only how big the file is and how long it
-              // has been going. A bar pinned at zero for twenty minutes says
+              // A hash says how far through it is when it runs out of process,
+              // and says nothing when it falls back to hashing in this one. So
+              // the percentage is drawn where there is one and left out where
+              // there is not — a bar pinned at zero for twenty minutes says
               // "stuck" when the honest answer is "no idea, still working".
               const hashing = add.phase === 'hashing';
               const pct =
@@ -947,9 +948,13 @@
                 ? duration(Date.now() - new Date(add.startedAt).getTime())
                 : null;
               const value = hashing
-                ? `hashing${add.total ? ` ${bytes(add.total)}` : ''}${
-                    elapsed ? ` · ${elapsed}` : ''
-                  }`
+                ? [
+                    `hashing${add.total ? ` ${bytes(add.total)}` : ''}`,
+                    pct == null ? null : `${pct.toFixed(1)}%`,
+                    elapsed,
+                  ]
+                    .filter(Boolean)
+                    .join(' · ')
                 : pct == null
                   ? bytes(add.received)
                   : `${pct.toFixed(1)}%`;
@@ -2370,7 +2375,7 @@ Every piece is hashed against the ` +
             kind === 'url'
               ? 'fetching — watch the log'
               : kind === 'path'
-                ? 'hashing — watch the log'
+                ? 'hashing — progress is below'
                 : 'added',
           );
           refresh();