pmtiles-swarm 0.30.0 → 0.31.0

package/CHANGELOG.md

@@ -7,6 +7,36 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.31.0
+### ✨ Features and improvements
+- **A Recheck files button, for when an archive's progress and its files disagree.** Every figure a
+  node can give you about how much of an archive is present is derived from something written down
+  earlier: the catalog's `complete` flag, the engine's resume data, the `seedOnly` claim made when
+  the torrent was added. When one of those is wrong there is no path back on its own — an archive
+  built on this node whose entry says `complete: false` is re-added without `seedOnly`, so the
+  engine goes looking for bytes that are already under its nose and sits at 0% beside a finished
+  file. Every restart reaches the same conclusion.
+
+  `POST /api/torrents/<infohash>/recheck` hashes every piece against the torrent and the result
+  wins. libtorrent does it with `force_recheck` through the sidecar (pmtiles-torrent 0.5.1 or
+  newer); qBittorrent has the same operation in its WebUI API. WebTorrent has none — it verifies on
+  add and never again — so it is re-added with the "the data is already here" claim withheld, which
+  is reported as `method: "readd"` rather than dressed up as the same mechanism.
+
+  Answers as soon as the check is under way, because hashing a planet build is tens of minutes and
+  no request should be held open for it. The archive reports state `checking` with progress as the
+  fraction hashed; progress running backwards during that is the operation working. Nothing is
+  deleted, and nothing is written to the catalog — the answer arrives where progress always does,
+  and the completion sweep already acts on it.
+
+  With two engines both are asked, since each keeps its own belief about the same file and a stale
+  one on the secondary is why a browser peer would find nothing while the primary seeds happily.
+
+- _...Add new stuff here..._
+
+### 🐞 Bug fixes
+- _...Add new stuff here..._
+
 ## 0.30.0
 ### ✨ Features and improvements
 - **The style URL now carries the `.torrent` URL as well as the magnet.** Piece hashes reach a

package/README.md

@@ -753,6 +753,7 @@ which the endpoint answers 501.
 | `PATCH`               | `/api/torrents/:infoHash/categories`                        | Set, add or remove categories                                                                                                                                                                                                             |
 | `PATCH`               | `/api/torrents/:infoHash/seeding`                           | Per-archive seeding limit, or "use the global one"                                                                                                                                                                                        |
 | `PATCH` `GET`         | `/api/torrents/:infoHash/location`                          | Move the data; poll the move                                                                                                                                                                                                              |
+| `POST`                | `/api/torrents/:infoHash/recheck`                           | Hash what is on disk again and believe the result — for an archive whose progress and whose files disagree                                                                                                                                |
 | `POST`                | `/api/torrents/:infoHash/pause`, `/resume`                  | Stop offering it, without forgetting it                                                                                                                                                                                                   |
 | `POST`                | `/api/torrents/:infoHash/webseeds`                          | Add web seeds — does not change the infohash                                                                                                                                                                                              |
 | `POST`                | `/api/torrents/:infoHash/warm`                              | Pre-fetch a region (`GET` for progress, `DELETE` to cancel)                                                                                                                                                                               |

package/docs/engines.md

@@ -344,6 +344,43 @@ Trackers live outside the torrent's `info` dictionary, so adding them **does not
 infohash** — but it only applies to torrents created after the change. An existing archive keeps
 announcing where its own torrent says to.
 
+## Rechecking what is on disk
+
+Every figure a node can give you about how much of an archive is present is derived from something
+written down earlier: the catalog's `complete` flag, the engine's resume data, the `seedOnly` claim
+made when the torrent was added. When one of those is wrong there is no path back on its own — the
+archive sits at 0% beside a finished file, downloading bytes it already has, and every restart
+repeats the same conclusion.
+
+**Recheck files** in the console (`POST /api/torrents/<infohash>/recheck`) is the one operation
+that goes and looks. It hashes every piece against the torrent and the result wins.
+
+| Engine      | How                                                             |
+| ----------- | --------------------------------------------------------------- |
+| libtorrent  | `force_recheck`, via the sidecar. Needs pmtiles-torrent ≥ 0.5.1 |
+| qBittorrent | `POST /api/v2/torrents/recheck` — the same library underneath   |
+| WebTorrent  | no such operation; re-added with `seedOnly` withheld instead    |
+
+It answers as soon as the check is under way. Hashing a planet build is tens of minutes, which is
+longer than any request should be held open for, so the archive reports state `checking` with
+progress as the fraction hashed. **Progress running backwards during that is the operation
+working**, not a fault. Nothing is deleted at any point.
+
+WebTorrent's route is cruder and is reported as `method: "readd"` rather than dressed up as the
+same thing: it verifies on add and never again, so the only way to make it look is to add the
+torrent a second time with the "the data is already here" claim withheld. It also refuses on a
+paused archive, since a re-add would start it.
+
+Nothing is written to the catalog when the check begins. The answer arrives where progress always
+does — the engine's own figures, which the completion sweep already reads. Note the one thing this
+does not repair on its own: the sweep promotes and never demotes, so a recheck finding _less_ than
+the record claims shows the truth in the console but leaves `complete: true` in place. Demoting on
+a progress figure would strip the finished name off any archive that happened to be mid-check.
+
+With two engines both are asked. Each keeps its own belief about the same file, so a stale one on
+the secondary is why a browser peer finds nothing while the primary seeds happily. A secondary that
+cannot check is reported and does not fail the operation; the primary failing does.
+
 ## Marking incomplete files
 
 An archive that is not whole yet is written under a marked name and renamed when
@@ -375,9 +412,10 @@ checked and the unreadable ones are joined by magnet instead.
 
 ## Writing another engine
 
-Implement `connect`, `add`, `remove`, `list`, `get`, `destroy`, and optionally `peers` and
-`reachability`. Anything optional that is missing is simply not offered — an engine without
-`reachability` hides the indicator rather than reporting a node unreachable.
+Implement `connect`, `add`, `remove`, `list`, `get`, `destroy`, and optionally `peers`,
+`reachability` and `recheck`. Anything optional that is missing is simply not offered — an engine
+without `reachability` hides the indicator rather than reporting a node unreachable, and one
+without `recheck` is rechecked by re-adding instead.
 The interface is deliberately small; see
 [`src/engines/types.js`](../src/engines/types.js) for the contract and
 [`src/engines/webtorrent.js`](../src/engines/webtorrent.js) for the shortest example.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.30.0",
+  "version": "0.31.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -1040,6 +1040,21 @@ export function createApp({
     }),
   );
 
+  // When the record and the disk disagree, this is the only thing that goes and
+  // looks. Answers as soon as the check is under way -- hashing a planet build
+  // is tens of minutes, and no request should be held open for it.
+  app.post(
+    '/api/torrents/:infoHash/recheck',
+    route(async (req, res) => {
+      try {
+        const result = await library.recheck(req.params.infoHash);
+        res.status(202).json(result);
+      } catch (error) {
+        res.status(error.status ?? 500).json({ error: error.message });
+      }
+    }),
+  );
+
   // Joining defaults to cache, deliberately. This is how that is changed
   // afterwards, without re-adding the archive by hand.
   app.patch(

package/src/engines/composite.js

@@ -549,6 +549,50 @@ export class CompositeEngine {
     return this.#primary.resume ? this.#primary.resume(infoHash) : false;
   }
 
+  /**
+   * Hashes what is on disk again, on every engine that can.
+   *
+   * Each engine keeps its own belief about the same file, so a stale one on the
+   * secondary is the same fault as a stale one on the primary -- it is why a
+   * browser peer would find nothing while the primary seeds happily. Both are
+   * asked.
+   *
+   * They hash concurrently, which sounds worse than it is: the second pass over
+   * a file the first just read is mostly page cache, and a recheck is a rare
+   * thing somebody asked for rather than something that runs on a timer.
+   *
+   * An engine without the operation is skipped rather than worked around. The
+   * re-add that WebTorrent needs instead is the library's to do, because it
+   * takes the catalog entry -- see Library.recheck.
+   * @param {string} infoHash - The archive to verify.
+   * @returns {Promise<object>} - The primary's answer, plus one row per engine.
+   */
+  async recheck(infoHash) {
+    const engines = [];
+    for (const engine of [this.#primary, ...this.#secondaries]) {
+      if (!engine.recheck) {
+        engines.push({ engine: engine.name, rechecking: false, skipped: true });
+        continue;
+      }
+      try {
+        const result = await engine.recheck(infoHash);
+        engines.push({ engine: engine.name, ...result });
+      } catch (error) {
+        // A secondary that cannot verify must not fail the primary's check.
+        // The primary holds the data the tiles are served from.
+        engines.push({
+          engine: engine.name,
+          rechecking: false,
+          error: error.message,
+        });
+      }
+    }
+
+    const primary = engines[0];
+    if (primary.error) throw new Error(primary.error);
+    return { ...primary, engines };
+  }
+
   /**
    * Tells every engine about a web seed.
    * @param {string} infoHash - The archive.

package/src/engines/libtorrent.js

@@ -303,6 +303,33 @@ export class LibtorrentEngine {
     }
   }
 
+  /**
+   * Hashes what is on disk again and believes the result over the record.
+   *
+   * Returns as soon as the check is under way, not when it finishes: a planet
+   * archive is tens of minutes of disk. The torrent reports state `checking`
+   * while it runs, with progress as the fraction hashed.
+   * @param {string} infoHash - The archive to verify.
+   * @returns {Promise<object>} - `{rechecking, wasPaused}`.
+   */
+  async recheck(infoHash) {
+    try {
+      return await this.#call('recheck', { infoHash });
+    } catch (error) {
+      // An older sidecar answers "unknown op", which is true and useless: it
+      // reads as a bug in the request rather than as a package that needs
+      // updating. Said plainly instead, because this is a button somebody just
+      // pressed and the next thing they do depends on which it is.
+      if (/unknown op/i.test(error.message)) {
+        throw new Error(
+          'this sidecar cannot recheck; pmtiles-torrent 0.5.1 or newer is needed',
+          { cause: error },
+        );
+      }
+      throw error;
+    }
+  }
+
   async list() {
     // A node that is shutting down still has a console polling it and a sweep
     // or two in flight. Answering "the sidecar exited" to each of them fills

package/src/engines/qbittorrent.js

@@ -211,6 +211,20 @@ export class QBittorrentEngine {
     await this.#request('/api/v2/torrents/delete', { method: 'POST', body });
   }
 
+  /**
+   * Hashes what is on disk again and believes the result over the record.
+   *
+   * qBittorrent starts the check and answers immediately, the same as the
+   * libtorrent engine does -- which is the same library underneath.
+   * @param {string} infoHash - The archive to verify.
+   * @returns {Promise<object>} - `{rechecking: true}`.
+   */
+  async recheck(infoHash) {
+    const body = new URLSearchParams({ hashes: infoHash.toLowerCase() });
+    await this.#request('/api/v2/torrents/recheck', { method: 'POST', body });
+    return { rechecking: true };
+  }
+
   /**
    * Lists every torrent qBittorrent holds.
    * @returns {Promise<import('./types.js').TorrentStatus[]>} - Normalised statuses.

package/src/engines/types.js

@@ -60,6 +60,7 @@
  * @property {(filePath: string, options?: object) => Promise<object>} [createTorrent] - Builds a torrent from a local file, where the engine can do it better than the default — libtorrent produces hybrid v1+v2, which create-torrent cannot.
  * @property {(infoHash: string) => Promise<object[]>} [trackerStatus] - Per-tracker announce results, where the engine keeps them.
  * @property {(infoHash: string) => Promise<Uint8Array | null>} [metadata] - The torrent's metainfo once known, so an archive joined by magnet can be written down rather than re-fetched over BEP 9 on every start.
+ * @property {(infoHash: string) => Promise<object>} [recheck] - Hash what is on disk again and believe the result over the stored state -- resume data, or the `seedOnly` claim made when the torrent was added. Returns once the check is under way, not when it finishes: a planet archive is tens of minutes of disk. An engine without this is rechecked by re-adding it instead, which the library does.
  * @property {() => Promise<object | null>} [reachability] - Whether peers can open a connection to this node, or only the reverse. Three states: `open` (something has connected inward), `unproven` (listening, but nothing ever has) and `offline` (not listening at all). The middle is deliberately not called firewalled -- on a node with no peers, blocked and untried are the same observation.
  * @property {() => Promise<void>} destroy - Releases resources.
  */

package/src/library.js

@@ -1657,6 +1657,68 @@ export class Library {
     return this.#catalog.put({ infoHash, paused: true });
   }
 
+  /**
+   * Hashes what is on disk again and believes the result over the record.
+   *
+   * Every other answer about how much of an archive is here comes from
+   * something written down earlier -- the catalog's `complete` flag, resume
+   * data, the `seedOnly` claim made when it was added. When one of those is
+   * wrong there is no path back on its own: an archive built here whose entry
+   * says `complete: false` is re-added without `seedOnly`, so the engine goes
+   * looking for bytes that are already under its nose and sits at 0% beside a
+   * finished file. This is the way out.
+   *
+   * Nothing is written to the catalog here. The check runs for as long as it
+   * takes to hash the archive -- tens of minutes for a planet build -- so the
+   * answer arrives long after this returns, and it arrives where it always
+   * does: the engine's own progress, which the completion sweep already reads
+   * and acts on. Recording a guess now would only have to be corrected later.
+   *
+   * Note what this cannot fix on its own. The sweep promotes, it never demotes,
+   * so a recheck that finds *less* than the record claims shows the truth in
+   * the console but leaves `complete: true` in place. That is deliberate: a
+   * torrent reports progress below 1 for perfectly ordinary reasons while it is
+   * checking, and demoting on that would strip the finished name off an archive
+   * that is merely being verified.
+   * @param {string} infoHash - The archive to verify.
+   * @returns {Promise<object>} - `{rechecking, method}`.
+   */
+  async recheck(infoHash) {
+    const entry = this.#catalog.get(infoHash);
+    if (!entry) {
+      const error = new Error('unknown archive');
+      error.status = 404;
+      throw error;
+    }
+
+    if (this.#engine.recheck) {
+      const result = await this.#engine.recheck(infoHash);
+      return { ...result, rechecking: true, method: 'recheck' };
+    }
+
+    // WebTorrent has no recheck at all: it verifies on add and never again. So
+    // the only way to make it look is to make it add the torrent again, with
+    // the claim that the data is already there withheld -- `seedOnly` is
+    // precisely "do not verify this", and it is computed from `complete`, so a
+    // copy of the entry saying otherwise is what turns the check on.
+    //
+    // Nothing is deleted and the catalog is not touched; this re-adds the same
+    // torrent against the same files. It is a slower and cruder mechanism than
+    // force_recheck, and it is reported as a different one rather than dressed
+    // up as the same thing.
+    if (entry.paused) {
+      const error = new Error(
+        'this engine rechecks by re-adding the archive, which a paused ' +
+          'archive cannot do. Resume it first.',
+      );
+      error.status = 409;
+      throw error;
+    }
+    await this.#engine.remove(infoHash, { deleteData: false }).catch(() => {});
+    await this.#readd({ ...entry, complete: false });
+    return { rechecking: true, method: 'readd' };
+  }
+
   /**
    * Starts offering a paused archive again.
    * @param {string} infoHash - The archive.

package/src/web/index.html

@@ -1641,6 +1641,7 @@
             <button id="set-location">Set location…</button>
             <button id="add-seed">Add web seed</button>
             <button id="clear-cache" ${mode === 'cache' ? '' : 'disabled title="only cache-mode archives have a cache to clear"'}>Clear cache</button>
+            <button id="recheck" title="Hash the files on disk again and believe the result. For an archive that reads 0% next to a file that is plainly there, or one that claims to be complete and is not — every other figure comes from something written down earlier, and this is the only thing that goes and looks.">Recheck files</button>
             <button id="pause">${entry.paused ? 'Resume' : 'Pause'}</button>
             <button id="remove" class="danger">Remove</button>
           </div>
@@ -1949,6 +1950,38 @@
           }
         };
 
+        $('recheck').onclick = async () => {
+          if (
+            !window.confirm(
+              `Recheck ${entry.name}.
+
+Every piece is hashed against the ` +
+                'torrent, which for a large archive is minutes to tens of ' +
+                'minutes of disk. Nothing is deleted, and the archive keeps ' +
+                'serving whatever it can while the check runs.',
+            )
+          ) {
+            return;
+          }
+          try {
+            const result = await api(`/api/torrents/${infoHash}/recheck`, {
+              method: 'POST',
+            });
+            // Said rather than left to be inferred: this returns as soon as the
+            // check is under way, and the progress bar going backwards for the
+            // next twenty minutes is the operation working, not a fault.
+            toast(
+              result.method === 'readd'
+                ? 'rechecking — this engine verifies by re-adding, so it will start from 0%'
+                : 'rechecking — progress shows the fraction hashed',
+            );
+            refresh();
+            renderDetail(infoHash);
+          } catch (error) {
+            toast(error.message);
+          }
+        };
+
         $('pause').onclick = async () => {
           const action = entry.paused ? 'resume' : 'pause';
           try {