pmtiles-swarm 0.24.3 → 0.25.0

package/CHANGELOG.md

@@ -7,6 +7,56 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.25.0
+### ✨ Features and improvements
+- **A stopped download is kept, and adding the same URL again resumes it.** Staging directories
+  were named at random, so a partial transfer became unreachable the moment the add returned:
+  nothing knew where it was, and re-adding the URL opened a fresh directory beside it and started
+  from zero. They are now named from the URL, and a fetch that runs out of attempts leaves its
+  bytes in place rather than deleting them — so the second add finds the first one's work and
+  continues with a Range request. Cancelling still removes them: somebody said stop, and leaving
+  hundreds of gigabytes behind after that is the waste the deletion was written to avoid.
+
+  Note the disk consequence. A download abandoned for good now keeps its partial file until the
+  directory is removed by hand; the give-up message and a log line both name the path.
+
+### 🐞 Bug fixes
+- **Ten network blips ended an 800 GB download, whatever it had achieved.** Two faults, and the
+  attempt count was neither of them.
+
+  The budget counted every failure rather than consecutive failures that transferred nothing, so it
+  described the whole download instead of the trouble it was in. Observed in the field: 226 GB
+  across six separate stalls, then the remaining four spent inside one bad minute, because a
+  quarter of a terabyte of progress counted for nothing. An attempt that moves bytes has reached
+  the source and got data out of it, so whatever it hits next is new trouble — progress now clears
+  the count, against a high-water mark so a short attempt after a long one is not mistaken for it.
+  A ceiling on total attempts keeps that from becoming an unbounded loop.
+
+  And the wait between attempts was flat, so ten of them covered about forty-five seconds — shorter
+  than most of the interruptions they exist to survive. It now grows with each consecutive failure
+  and the base moves from 5 seconds to 30, which spans something over twenty minutes rather than
+  under one.
+
+## 0.24.4
+### 🐞 Bug fixes
+- **The feed advertised a .torrent nobody could fetch.** Every item named
+  `/api/torrents/<infohash>/file`, and that address is unreachable to exactly the audience a feed
+  is written for: the API is not on the public listener, so it answers 404 there, and on the
+  console listener it answers 401. Our own subscriptions got one or the other, and so did any
+  ordinary torrent client pointed at the same URL.
+
+  It failed quietly, which is why it lasted. `subscriptions.js` falls back to the magnet and logs a
+  line about it, so a mirror still joined and still downloaded. What the fallback costs is not
+  visible from there: BEP 9 carries only the info dict, and a v2 torrent's piece layers live
+  outside it, so an archive joined by magnet can never obtain them. On a hybrid torrent the mirror
+  then holds metadata it cannot verify pieces against and republishes a .torrent that claims v2
+  while omitting the hashes — 413 KB against the origin's 1,074 KB, the difference being precisely
+  20,636 pieces x 32 bytes. Anything mirroring from that mirror inherits it.
+
+  Items now name `/archives/<infohash>/archive.torrent`, which is public, unauthenticated, and the
+  same URL the TileJSON has always given for the same archive. Those two disagreeing was the bug;
+  a test now pins that they agree and that what they name answers 200 on the public surface.
+
 ## 0.24.3
 ### 🐞 Bug fixes
 - **Takes pmtiles-torrent 0.4.6.** Two things a node reading pieces on demand wanted. The sidecar

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.24.3",
+  "version": "0.25.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/config.js

@@ -326,8 +326,21 @@ const DEFAULTS = {
    * docs/internals.md — "Resuming a partial download".
    */
   fetchAttempts: 10,
-  /** How long to wait before resuming a download that stopped. */
-  fetchRetrySeconds: 5,
+  /**
+   * Base wait before resuming a download that stopped, in seconds.
+   *
+   * Multiplied by the number of consecutive failures, so the wait grows while
+   * the trouble lasts: at 30 it waits 30s, then 60, then 90, and a run of ten
+   * spans something over twenty minutes. A flat five seconds made ten attempts
+   * worth about forty-five seconds in total, which is shorter than most of the
+   * interruptions this exists to survive — a download that had transferred
+   * 226 GB was ended by a single bad minute.
+   *
+   * Counted per consecutive failure rather than per attempt: anything that
+   * moves bytes clears the count, because a transfer that got data out of the
+   * source has proved the route and whatever it hit next is new trouble.
+   */
+  fetchRetrySeconds: 30,
   /** How often to check seeding limits, in seconds. Zero disables it. */
   seedingCheckIntervalSeconds: 3600,
   /**

package/src/feed.js

@@ -84,7 +84,27 @@ ${items}
  * @returns {string} - The item XML.
  */
 function renderItem(entry, baseUrl) {
-  const torrentUrl = `${baseUrl}/api/torrents/${entry.infoHash}/file`;
+  // The public route, which is the only one a subscriber can use.
+  //
+  // This named /api/torrents/<hash>/file, and that address is unreachable to
+  // everyone the feed is written for: the API is not on the public listener at
+  // all, so it answers 404 there, and on the console listener it answers 401.
+  // A feed is a public document whose whole purpose is to be followed by
+  // somebody else, and every consumer of it -- our own subscriptions, and any
+  // ordinary torrent client pointed at the same URL -- got one of those two.
+  //
+  // Nothing broke loudly, because subscriptions.js falls back to the magnet and
+  // logs a line about it. What that fallback costs is invisible from here and
+  // considerable: BEP 9 carries only the info dict, and a v2 torrent's piece
+  // layers live outside it, so an archive joined by magnet can never obtain
+  // them. On a hybrid torrent the mirror then holds metadata it cannot verify
+  // pieces against, and republishes a .torrent that claims v2 and omits the
+  // hashes -- measured at 413 KB against the origin's 1,074 KB, the difference
+  // being exactly 20,636 pieces x 32 bytes.
+  //
+  // Kept identical to what the TileJSON advertises for the same archive, in
+  // api.js. Those two disagreeing is what this was.
+  const torrentUrl = `${baseUrl}/archives/${entry.infoHash}/archive.torrent`;
   const map = entry.pmtiles;
 
   const mapFields = map

package/src/library.js

@@ -606,8 +606,22 @@ export class Library {
     // Downloaded into a directory of its own, then moved once the infohash
     // exists to be filed under. See docs/internals.md — "Staging, for an
     // archive fetched from a URL".
+    // Named from the URL rather than at random, so a download that stopped can
+    // be picked up again.
+    //
+    // A random name made every add its own directory, which meant a partial
+    // transfer was unreachable the moment the add returned: nothing knew where
+    // it was, and re-adding the same URL opened a fresh directory beside it and
+    // started from zero. For a planet archive that is hundreds of gigabytes
+    // thrown away because a network went out for a few minutes. Derived from
+    // the URL, the second add finds the first one's bytes and downloadTo
+    // resumes with a Range request.
     const staging = retain
-      ? path.join(root, INCOMING, crypto.randomBytes(8).toString('hex'))
+      ? path.join(
+          root,
+          INCOMING,
+          crypto.createHash('sha256').update(url).digest('hex').slice(0, 16),
+        )
       : undefined;
 
     // The origin is a valid web seed for exactly these bytes, so it is used as
@@ -658,12 +672,26 @@ export class Library {
         },
       });
     } catch (error) {
-      // A cancelled or failed fetch leaves a partial file in a directory
-      // nothing will ever look in again. Left alone it is invisible waste —
-      // and for a planet archive, invisible waste measured in gigabytes.
       this.#running.delete(url);
-      if (staging)
+      // Cancelling is a decision to stop wanting this; running out of attempts
+      // is not. The two used to be cleaned up identically, so a download that
+      // survived six stalls and reached 226 GB had all of it deleted by the
+      // seventh -- and the resume that the staging directory exists to make
+      // possible had nothing left to resume from.
+      //
+      // A cancelled fetch is still removed. Somebody said stop, and leaving
+      // gigabytes behind after that is the invisible waste this was written to
+      // avoid in the first place.
+      const cancelled = controller.signal.aborted;
+      if (staging && cancelled) {
         await fs.rm(staging, { recursive: true, force: true }).catch(() => {});
+      } else if (staging) {
+        console.warn(
+          `[fetch] keeping the partial download in ${staging}; adding ${url} ` +
+            'again resumes it, and discarding it is a matter of removing that ' +
+            'directory',
+        );
+      }
       throw error;
     }
 

package/src/torrent-create.js

@@ -230,7 +230,31 @@ async function downloadTo(url, target, onProgress, signal, options = {}) {
   let lastReport = 0;
   let lastError;
 
-  for (let attempt = 1; attempt <= attempts; attempt += 1) {
+  // The budget counts *consecutive* failures that moved nothing, not failures.
+  //
+  // Counting every failure made the budget a property of the whole download
+  // rather than of the trouble it is in, and for a large archive those are not
+  // the same thing at all. A 700 GiB transfer over a domestic line drops
+  // occasionally; observed in the field, a download reached 226 GB across six
+  // separate stalls and then exhausted the remaining four on a single bad
+  // minute, because nothing about a quarter of a terabyte of progress counted
+  // for anything. An attempt that transferred bytes proves the source and the
+  // route are alive, so the trouble it hit is over and the next one is new.
+  //
+  // The high-water mark rather than the last attempt's figure: an attempt can
+  // fail having written less than a previous one already had on disk, and that
+  // is not progress.
+  let consumed = 0;
+  let best = await bytesOnDisk(target);
+  // Reset does mean an unlucky download can go round more times than the
+  // budget names, which is the point, so there is a ceiling as well: a source
+  // dribbling a few bytes before dropping every time would otherwise retry
+  // for ever.
+  const ceiling = attempts * 10;
+  let taken = 0;
+
+  while (consumed < attempts && taken < ceiling) {
+    taken += 1;
     const from = await bytesOnDisk(target);
     // A file already at full length is one a previous attempt finished, and
     // that the caller died before renaming. Re-fetching it buys nothing.
@@ -249,8 +273,11 @@ async function downloadTo(url, target, onProgress, signal, options = {}) {
       // A cancelled download is a decision, not a failure to retry past.
       if (signal?.aborted) throw error;
       lastError = error;
-      if (attempt === attempts) break;
-      await delay(retryDelayMs, signal);
+      // Nothing was transferred -- the request never opened -- so this one
+      // always counts.
+      consumed += 1;
+      if (consumed >= attempts) break;
+      await delay(retryDelayMs * consumed, signal);
       continue;
     }
 
@@ -324,18 +351,38 @@ async function downloadTo(url, target, onProgress, signal, options = {}) {
       if (signal?.aborted) throw error;
       lastError = error;
       const reached = await bytesOnDisk(target);
+      // Progress clears the slate. Anything that moved bytes reached the
+      // source and got data out of it, so whatever it then ran into is a new
+      // problem rather than a continuation of the last one.
+      if (reached > best) {
+        best = reached;
+        consumed = 0;
+      } else {
+        consumed += 1;
+      }
       console.warn(
         `[fetch] ${url} stopped at ${reached} bytes ` +
-          `(attempt ${attempt}/${attempts}): ${error.message}`,
+          `(${consumed}/${attempts} consecutive without progress): ` +
+          error.message,
       );
-      if (attempt === attempts) break;
-      await delay(retryDelayMs, signal);
+      if (consumed >= attempts) break;
+      // Growing with each consecutive failure, so a budget spans an outage
+      // rather than a moment: at the default of 30s this waits 30, 60, 90 …
+      // and ten of them cover something over twenty minutes. A flat delay made
+      // ten attempts worth about forty-five seconds, which is shorter than
+      // most of the interruptions it exists to survive.
+      await delay(retryDelayMs * consumed, signal);
     }
   }
 
+  // Says what was reached as well as what failed. What matters when this lands
+  // is whether there is anything worth resuming, and the byte count is the
+  // whole of that answer -- the partial file is kept, so re-adding the same URL
+  // continues from here rather than starting again.
   throw new Error(
-    `could not finish downloading ${url} after ${attempts} attempts: ` +
-      `${lastError?.message ?? 'unknown error'}`,
+    `could not finish downloading ${url} after ${consumed} consecutive ` +
+      `attempts without progress (${best} bytes transferred, kept for a ` +
+      `resume): ${lastError?.message ?? 'unknown error'}`,
   );
 }