pmtiles-swarm 0.37.0 → 0.37.3

package/CHANGELOG.md

@@ -7,6 +7,59 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.37.3
+### ✨ Features and improvements
+
+### 🐞 Bug fixes
+- **An archive fetched from a URL is now hashed the same way one already on disk is.** `creator` was
+  passed on the local path and nowhere else, so every archive a schedule ever built was hashed
+  inside this process — the one serving tiles and the console — rather than in the sidecar's
+  one-shot hasher. Four things followed from that one omission: the hash competed with serving,
+  it could not be cancelled (`create-torrent` takes no signal, so the add's AbortController reached
+  the download and stopped there), it reported nothing while it ran, and it produced a v1 torrent
+  rather than a hybrid. An archive arriving from a feed got a lesser torrent, built the slower way,
+  than the same file added by path.
+- **A fetched archive now says when it has stopped downloading and started hashing.** The remote
+  add's progress callback dropped the `phase` the hasher reports, and its entry carried no phase at
+  all, so `runningAdds()` called it `fetching` from beginning to end. With the byte counts equal at
+  that point, the row sat at 100% "fetching" for the whole hash — which reads as a transfer that
+  completed and then hung, and was reported as exactly that. The bar now hands over to the hash and
+  fills again as pieces are read.
+
+## 0.37.2
+### ✨ Features and improvements
+
+### 🐞 Bug fixes
+- **Pausing an archive now stops it.** Reported from the field: an archive was paused, the row read
+  `paused`, and it went on downloading at 8.4 MiB/s. Nothing in the chain refused — the pause was
+  asked for, reported as done, and never happened.
+
+  Three faults in a line. `LibtorrentEngine` had no `pause` or `resume` at all, so there was no way
+  to stop a libtorrent torrent from here. `CompositeEngine.pause` answered for its primary alone,
+  so an archive held by a secondary reported as not stopped when it was. And `Library.pause` tested
+  only that the engine *had* a pause method and threw away what it answered — a composite has one
+  whatever its engines can do, so the `false` went into a void, the fallback never ran, and
+  `paused: true` went into the catalog regardless. The console prefers that flag to the engine's
+  live state, which is why the row said `paused` while Down and Up kept moving.
+
+  Requires pmtiles-torrent 0.9.0, which adds the `pause` and `resume` the sidecar never had — and
+  makes them stick. `handle.pause()` alone is not a stop: libtorrent's auto-manager clears the
+  paused flag again within about a second, so pausing that way produces a torrent that describes
+  itself as paused while it transfers. That would have reproduced this exact symptom one layer
+  deeper.
+
+  Nothing was left in a bad state by this: because the pause never took effect, no archive was
+  half-stopped and no resume data is wrong. They were seeding and downloading throughout.
+
+## 0.37.1
+### ✨ Features and improvements
+- **Cancel now sits in the row it cancels.** Collected into a bar underneath the list, each button
+  had to repeat the whole filename to say which add it stopped — two of those filled a line, and
+  pressing the right one meant matching a long name against the list above it. The rows carry their
+  own, and an add that cannot be cancelled keeps an empty cell so the columns stay lined up.
+
+### 🐞 Bug fixes
+
 ## 0.37.0
 ### ✨ Features and improvements
 - **An archive being hashed can now be cancelled, and says how far through it is.** 0.36.0 moved

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.37.0",
+  "version": "0.37.3",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",
@@ -46,7 +46,7 @@
     "maplibre-gl": "^6.2.0",
     "parse-torrent": "^11.0.24",
     "pmtiles": "^4.4.1",
-    "pmtiles-torrent": "^0.8.0",
+    "pmtiles-torrent": "^0.9.0",
     "webtorrent": "^3.0.21"
   },
   "engines": {

package/src/engines/composite.js

@@ -543,26 +543,40 @@ export class CompositeEngine {
 
   /**
    * Stops offering an archive, everywhere.
+   *
+   * Answers for the whole composite rather than for the primary alone. An
+   * archive can be held by a secondary and not by the primary, and reporting
+   * the primary's "no" for it told the caller nothing was stopped when
+   * something was -- which is a false negative that costs data, since the
+   * caller's answer to a pause it cannot get is to remove the torrent instead.
    * @param {string} infoHash - The archive.
-   * @returns {Promise<boolean>} - Whether the primary paused it.
+   * @returns {Promise<boolean>} - Whether any engine stopped it.
    */
   async pause(infoHash) {
+    let stopped = false;
     for (const engine of this.#secondaries) {
-      await engine.pause?.(infoHash).catch(() => {});
+      stopped = (await engine.pause?.(infoHash).catch(() => false)) || stopped;
     }
-    return this.#primary.pause ? this.#primary.pause(infoHash) : false;
+    const primary = this.#primary.pause
+      ? await this.#primary.pause(infoHash)
+      : false;
+    return primary || stopped;
   }
 
   /**
    * Offers it again, everywhere.
    * @param {string} infoHash - The archive.
-   * @returns {Promise<boolean>} - Whether the primary resumed it.
+   * @returns {Promise<boolean>} - Whether any engine started it.
    */
   async resume(infoHash) {
+    let started = false;
     for (const engine of this.#secondaries) {
-      await engine.resume?.(infoHash).catch(() => {});
+      started = (await engine.resume?.(infoHash).catch(() => false)) || started;
     }
-    return this.#primary.resume ? this.#primary.resume(infoHash) : false;
+    const primary = this.#primary.resume
+      ? await this.#primary.resume(infoHash)
+      : false;
+    return primary || started;
   }
 
   /**

package/src/engines/libtorrent.js

@@ -415,6 +415,57 @@ export class LibtorrentEngine {
     }
   }
 
+  /**
+   * Stops a torrent, leaving its data and its place in the session alone.
+   *
+   * Not removal. The bytes stay, the resume data stays, and starting it again
+   * costs nothing -- which is the whole reason this exists rather than the
+   * remove-and-re-add a missing pause used to fall back to. Re-adding a
+   * 698 GiB archive means hashing the store again to arrive where it already
+   * was.
+   * @param {string} infoHash - The archive to stop.
+   * @returns {Promise<boolean>} - Whether it was stopped.
+   */
+  async pause(infoHash) {
+    await this.#stopStart('pause', infoHash);
+    return true;
+  }
+
+  /**
+   * Offers a stopped torrent again.
+   * @param {string} infoHash - The archive to start.
+   * @returns {Promise<boolean>} - Whether it was started.
+   */
+  async resume(infoHash) {
+    await this.#stopStart('resume', infoHash);
+    return true;
+  }
+
+  /**
+   * Pause and resume differ only in the word, including how they fail.
+   * @param {string} op - 'pause' or 'resume'.
+   * @param {string} infoHash - The archive.
+   * @returns {Promise<object>} - The sidecar's answer.
+   */
+  async #stopStart(op, infoHash) {
+    try {
+      return await this.#call(op, { infoHash });
+    } catch (error) {
+      // An older sidecar answers "unknown op", which is true and useless: it
+      // reads as a bug in the request rather than as a package that needs
+      // updating. Worth saying plainly, because before 0.9.0 there was no
+      // pause here at all -- the request reached the catalog and stopped, so
+      // the console showed `paused` beside an archive still transferring.
+      if (/unknown op/i.test(error.message)) {
+        throw new Error(
+          `this sidecar cannot ${op}; pmtiles-torrent 0.9.0 or newer is needed`,
+          { cause: error },
+        );
+      }
+      throw error;
+    }
+  }
+
   async list() {
     // A node that is shutting down still has a console polling it and a sweep
     // or two in flight. Answering "the sidecar exited" to each of them fills

package/src/library.js

@@ -311,20 +311,7 @@ export class Library {
         comment: options.comment,
         md5: options.md5 ?? this.#config.md5,
         signal: controller.signal,
-        // Pieces in, bytes out. The console draws one progress bar for adds and
-        // labels it in bytes, and a piece count means nothing beside a download
-        // measured in gigabytes — so it is converted here, where the file size
-        // is known, rather than teaching the console a second unit. Accurate to
-        // within one piece, which at 4 MiB against a planet archive is four
-        // parts in a million.
-        onHashProgress: ({ piece, pieces }) => {
-          const state = this.#running.get(requested);
-          if (!state || !pieces || !size) return;
-          state.received = Math.min(
-            size,
-            Math.round(((piece + 1) / pieces) * size),
-          );
-        },
+        onHashProgress: this.#hashProgress(requested),
       });
 
       return await this.#register(created, {
@@ -352,6 +339,28 @@ export class Library {
     }
   }
 
+  /**
+   * Reports a hash's progress as bytes, which is what the console draws.
+   *
+   * Pieces are what a hasher counts; the adds list has one bar and labels it in
+   * bytes, and a piece count means nothing beside a download measured in
+   * gigabytes. Converted here, where the size is known, rather than teaching
+   * the console a second unit. Accurate to within one piece — at 4 MiB against
+   * a planet archive, four parts in a million.
+   * @param {string} key - How the add is registered in #running.
+   * @returns {Function} - An onHashProgress callback.
+   */
+  #hashProgress(key) {
+    return ({ piece, pieces }) => {
+      const state = this.#running.get(key);
+      if (!state || !pieces || !state.total) return;
+      state.received = Math.min(
+        state.total,
+        Math.round(((piece + 1) / pieces) * state.total),
+      );
+    };
+  }
+
   /**
    * Turns a caller's choice of location into a directory, and checks it.
    *
@@ -884,8 +893,33 @@ export class Library {
         fetchAttempts: this.#config.fetchAttempts,
         fetchRetryDelayMs: (this.#config.fetchRetrySeconds ?? 5) * 1000,
         signal: controller.signal,
-        onProgress: ({ received, total, done }) => {
+        // The same hasher a local add gets, which this route never asked for.
+        //
+        // Without it the hash ran in this process -- the one serving tiles and
+        // the console -- for every archive a schedule ever built, could not be
+        // cancelled because create-torrent takes no signal, reported nothing
+        // while it ran, and produced a v1 torrent rather than a hybrid. So an
+        // archive arriving from a feed got a lesser torrent, built the slower
+        // way, than the same file added by path.
+        creator: this.#creator(),
+        onHashProgress: this.#hashProgress(url),
+        onProgress: ({ phase, received, total, done }) => {
           const state = this.#running.get(url);
+          if (phase === 'hashing') {
+            // The transfer is over and a different job has started, measured
+            // in the same units. Handing the bar over rather than leaving it
+            // at 100%: a finished download that sits at 100% for the length of
+            // a 128 GiB hash reads as one that completed and then hung, which
+            // is how it was reported.
+            if (state) {
+              Object.assign(state, {
+                phase: 'hashing',
+                received: undefined,
+                total,
+              });
+            }
+            return;
+          }
           if (state) Object.assign(state, { received, total });
           const pct = total ? ((received / total) * 100).toFixed(1) : '?';
           console.log(
@@ -1866,11 +1900,24 @@ export class Library {
       throw error;
     }
 
-    if (this.#engine.pause) {
-      await this.#engine.pause(infoHash);
-    } else {
+    // Whether it actually stopped, not whether something was asked.
+    //
+    // This tested only that the engine *had* a pause method and threw the
+    // answer away. A composite has one whatever its engines can do, so a
+    // primary with no pause of its own returned false into a void: the
+    // fallback below never ran, `paused: true` went into the catalog, and the
+    // console -- which prefers that flag to the engine's live state -- showed
+    // `paused` beside an archive still transferring at 8 MiB/s. The button did
+    // nothing and said it had worked.
+    const stopped = this.#engine.pause
+      ? await this.#engine.pause(infoHash)
+      : false;
+    if (!stopped) {
       // Removing without its data is a pause an engine cannot refuse; resume
-      // adds it back and it rechecks what is already on disk.
+      // adds it back and it rechecks what is already on disk. A last resort,
+      // because that recheck is the whole store -- tens of minutes for a
+      // planet archive -- which is why an engine that can really pause is
+      // worth the two operations it takes.
       await this.#engine
         .remove(infoHash, { deleteData: false })
         .catch(() => {});
@@ -1954,9 +2001,14 @@ export class Library {
       throw error;
     }
 
-    if (this.#engine.resume) {
-      await this.#engine.resume(infoHash);
-    } else {
+    // Same as pause: the answer decides, not the presence of a method. An
+    // archive stopped by the fallback above is not in the engine at all, so a
+    // resume it merely claimed would leave the catalog saying the archive was
+    // running while nothing held it.
+    const started = this.#engine.resume
+      ? await this.#engine.resume(infoHash)
+      : false;
+    if (!started) {
       await this.#readd({ ...entry, paused: false });
     }
     await this.#tiles?.invalidate(infoHash).catch(() => {});

package/src/web/index.html

@@ -309,6 +309,16 @@
       .piecerow { display: grid; grid-template-columns: 7rem 1fr 4rem; gap: 0.6rem; align-items: center; margin-bottom: 0.5rem; }
       .piecerow > .label { font-size: 0.8rem; color: var(--muted); }
       .piecerow > .value { font-size: 0.8rem; color: var(--muted); text-align: right; }
+      /* Like a piecerow, but the label is a filename rather than a fixed word,
+         and each row carries its own Cancel. A separate class because the
+         piecerow's 7rem label and 4rem value are sized for "downloaded" and
+         "41%": an archive name in 7rem wraps onto three lines, and the naming
+         of the buttons had to repeat the whole filename to say which was
+         which. */
+      .addrow { display: grid; grid-template-columns: minmax(0, 1fr) auto auto auto; gap: 0.6rem; align-items: center; margin-bottom: 0.5rem; }
+      .addrow > .label { font-size: 0.8rem; color: var(--muted); overflow: hidden; text-overflow: ellipsis; white-space: nowrap; }
+      .addrow > .value { font-size: 0.8rem; color: var(--muted); text-align: right; white-space: nowrap; }
+      .addrow > button { padding: 0.15rem 0.55rem; font-size: 0.75rem; }
 
       button.speed { border-color: var(--line); color: var(--muted); }
       button.speed.on { border-color: var(--warn); color: var(--warn); }
@@ -923,7 +933,6 @@
           box.innerHTML = '';
           return;
         }
-        const cancellable = running.filter((add) => add.cancellable !== false);
         box.innerHTML = `
           <h3 style="margin:1.2rem 0 0.5rem">Being added, before a torrent exists</h3>
           <div class="sub" style="margin-bottom:0.6rem">
@@ -958,26 +967,29 @@
                 : pct == null
                   ? bytes(add.received)
                   : `${pct.toFixed(1)}%`;
+              const name = add.name ?? add.url.split('/').pop() ?? add.url;
+              // Cancel sits in the row it cancels, so the button does not have
+              // to name the archive to say which one it is. Named, they were as
+              // wide as the filename — two of them filled a line, and reading
+              // one meant matching a long name against the list above it.
+              //
+              // An add with no controller keeps an empty cell rather than
+              // losing one, so the rows above and below it stay lined up.
+              const stop =
+                add.cancellable === false
+                  ? '<span></span>'
+                  : `<button data-cancel="${escapeHtml(add.url)}" title="Cancel ${escapeHtml(
+                      name,
+                    )}" aria-label="Cancel ${escapeHtml(name)}">Cancel</button>`;
               return `
-                <div class="piecerow">
-                  <span class="label" title="${escapeHtml(add.url)}">${escapeHtml(
-                    add.name ?? add.url.split('/').pop() ?? add.url,
-                  )}</span>
+                <div class="addrow">
+                  <span class="label" title="${escapeHtml(add.url)}">${escapeHtml(name)}</span>
                   <span class="track"><i style="width:${pct == null ? 0 : pct.toFixed(1)}%"></i></span>
                   <span class="value">${value}</span>
+                  ${stop}
                 </div>`;
             })
-            .join('')}
-          <div class="bar" style="margin-top:0.5rem">
-            ${cancellable
-              .map(
-                (add) =>
-                  `<button data-cancel="${escapeHtml(add.url)}">Cancel ${escapeHtml(
-                    add.name ?? add.url.split('/').pop() ?? '',
-                  )}</button>`,
-              )
-              .join('')}
-          </div>`;
+            .join('')}`;
 
         for (const button of box.querySelectorAll('[data-cancel]')) {
           button.onclick = async () => {