pmtiles-swarm 0.5.0 → 0.5.2

package/CHANGELOG.md

@@ -7,6 +7,36 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.5.2
+### 🐞 Bug fixes
+- **Asks for the `pmtiles-torrent` that 0.5.1 actually needs.** It shipped declaring `^0.3.2`,
+  and a caret on a `0.x` version allows patch updates only — so that range can never install
+  0.4.0, which is where the op behind writing a `.torrent` for a magnet-joined archive lives.
+  As published, 0.5.1 could not get the sidecar its own new code depends on. It fails safely
+  either way, since an older sidecar answers "unknown op" and the call is caught, but it fails
+  silently: the feed keeps serving enclosure URLs that 404 and nothing says why.
+
+## 0.5.1
+### ✨ Features and improvements
+- **An archive joined by magnet writes its own `.torrent` on a libtorrent node.** The machinery
+  was already there — `captureMetadata` writes the metainfo down, and completion sweeps retry it
+  for anything still missing one — but it asks the engine for the metainfo, and the libtorrent
+  engine had no way to give it. Only WebTorrent did, and WebTorrent only ever receives archives
+  that are already complete. So on the engine most people run, a magnet-joined archive could
+  never produce a `.torrent`, however long it ran.
+
+  That is one gap with a long tail: a node with no `.torrent` to publish serves a feed whose
+  enclosure URLs 404, so its subscribers join by magnet too, and a magnet carrying no trackers
+  has only the DHT to find its first peer with. Needs `pmtiles-torrent` 0.4.0, which added the
+  op; against an older sidecar the call is refused and caught, exactly as before.
+
+### 🐞 Bug fixes
+- **A head-warm that was simply too early no longer waits out the full backoff.** An archive
+  joined by magnet has no metainfo until BEP 9 finishes, so a read asked for in the same second
+  the node started is refused before it reaches the swarm at all. That is a wait, not an
+  attempt: it is now retried on the next pass rather than in two minutes, and reported once
+  rather than every time.
+
 ## 0.5.0
 ### ✨ Features and improvements
 - **The head of a newly joined archive is read without waiting to be asked.** A PMTiles archive

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.5.0",
+  "version": "0.5.2",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",
@@ -39,7 +39,7 @@
     "maplibre-gl": "^6.2.0",
     "parse-torrent": "^11.0.24",
     "pmtiles": "^4.4.1",
-    "pmtiles-torrent": "^0.3.2",
+    "pmtiles-torrent": "^0.4.0",
     "webtorrent": "^3.0.21"
   },
   "engines": {

package/src/engines/libtorrent.js

@@ -324,16 +324,32 @@ export class LibtorrentEngine {
     return this.#call('peers', { infoHash });
   }
 
+  /**
+   * The metainfo of a torrent, so a magnet can stop being a magnet.
+   * @param {string} infoHash - The archive.
+   * @returns {Promise<Uint8Array|null>} - The .torrent bytes, or null.
+   */
+  async metadata(infoHash) {
+    // The same thing a torrent client's "export .torrent" offers, and for the
+    // same reason: once BEP 9 has delivered the info dictionary, this node
+    // holds everything a .torrent contains — whether or not a byte of the
+    // archive itself has arrived.
+    //
+    // Without it a node that joined by magnet has no .torrent to publish, so
+    // every subscriber following its feed also joins by magnet; and a magnet
+    // that carries no trackers has only the DHT to find its first peer with,
+    // which is minutes of waiting per archive rather than none. libtorrent
+    // could always do this. Nothing had asked it to.
+    const result = await this.#call('metadata', { infoHash });
+    if (!result?.torrentFile) return null;
+    return new Uint8Array(Buffer.from(result.torrentFile, 'base64'));
+  }
+
   /**
    * Creates a torrent from a local file.
-   *
-   * Defaults to hybrid v1+v2, which is the capability that justifies this
-   * engine: v2 gives per-file merkle trees with 16 KiB leaf blocks, so a peer
-   * can verify a small block without holding the whole hash list, while the v1
-   * half keeps every existing client working.
-   * @param {string} filePath - Path to the archive.
+   * @param {string} filePath - The file to hash.
    * @param {object} [options] - Piece length, trackers, web seeds, format.
-   * @returns {Promise<object>} - The created torrent, torrentFile as bytes.
+   * @returns {Promise<object>} - The torrent file and what it describes.
    */
   async createTorrent(filePath, options = {}) {
     const result = await this.#call(

package/src/prewarm.js

@@ -24,6 +24,24 @@
 /** How long to leave an archive alone after a failed attempt. */
 const DEFAULT_BACKOFF_MS = 120000;
 
+/**
+ * Whether a failure means "not yet" rather than "not working".
+ *
+ * An archive joined from a magnet has no metainfo until BEP 9 has finished, and
+ * until then the engine cannot say where a byte range even falls — so a read
+ * asked for in the same second the node started is refused before it reaches
+ * the swarm at all. That is a wait, not an attempt, and treating it as one cost
+ * the full backoff for something that resolves in seconds.
+ *
+ * Matched on the message because that is what the engine gives us; both the
+ * sidecar and the WebTorrent engine word it the same way.
+ * @param {Error} error - What the read threw.
+ * @returns {boolean} - True when it is worth trying again shortly.
+ */
+function tooEarly(error) {
+  return /metadata has not arrived|not held here/i.test(error?.message ?? '');
+}
+
 export class HeadWarmer {
   #tiles;
   #catalog;
@@ -31,6 +49,7 @@ export class HeadWarmer {
   #now;
   #timer;
   #tried = new Map();
+  #waiting = new Set();
   #running = false;
 
   /**
@@ -95,7 +114,6 @@ export class HeadWarmer {
     if (!entry) return null;
 
     this.#running = true;
-    this.#tried.set(entry.infoHash, this.#now());
     try {
       // The long timeout, not the interactive one: this is a byte range from
       // an archive nobody has asked for a piece of yet.
@@ -103,6 +121,9 @@ export class HeadWarmer {
         timeoutMs: this.#config.tiles?.metadataTimeoutMs ?? 120000,
       });
 
+      this.#tried.set(entry.infoHash, this.#now());
+      this.#waiting.delete(entry.infoHash);
+
       const stored = await this.#catalog.put({
         infoHash: entry.infoHash,
         pmtiles: { ...entry.pmtiles, ...summary },
@@ -115,10 +136,26 @@ export class HeadWarmer {
       );
       return stored;
     } catch (error) {
-      // Ordinary while an archive is young — the piece holding the header may
-      // not exist anywhere reachable yet. Worth saying once per attempt, since
-      // it is the answer to "why is my preview blank", and the backoff keeps
-      // it from becoming noise.
+      if (tooEarly(error)) {
+        // Not stamped, so the next pass tries again in seconds rather than in
+        // minutes — and said once, because a node that has just started will
+        // answer this way until the metainfo lands.
+        if (!this.#waiting.has(entry.infoHash)) {
+          this.#waiting.add(entry.infoHash);
+          console.log(
+            `[warm] ${entry.name}: waiting for the torrent metadata before ` +
+              'reading its head',
+          );
+        }
+        return null;
+      }
+
+      // A real attempt: it reached the swarm and found nothing. Ordinary while
+      // an archive is young, since the piece holding the header may not exist
+      // anywhere reachable yet. Worth saying, because it is the answer to "why
+      // is my preview blank", and the backoff keeps it from becoming noise.
+      this.#tried.set(entry.infoHash, this.#now());
+      this.#waiting.delete(entry.infoHash);
       console.warn(`[warm] ${entry.name}: ${error.message}`);
       return null;
     } finally {