pmtiles-swarm 0.24.4 → 0.26.0

package/CHANGELOG.md

@@ -7,6 +7,59 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.26.0
+### 🐞 Bug fixes
+- **The add dialog stayed on screen for the length of a download.** `POST /api/torrents` awaited
+  the entire transfer before answering, so for a URL the response arrived hours after the request
+  — and the console, which closes the dialog when the response lands, sat there over an archive
+  visibly appearing behind it.
+
+  The console was always written for the other arrangement: it says "fetching — watch the log" as
+  it closes, polls `/api/adds` for progress, and offers `DELETE /api/adds` to cancel. Only the
+  route was missing. It now answers **202** as soon as the URL has been checked — it answers, it is
+  an archive of a publishable kind, it is not a credential about to be broadcast — and lets the
+  transfer run behind it. Everything a person can correct is still reported in the dialog, because
+  all of it is found before the first byte moves.
+
+  Both shortcuts inside `addRemoteArchive` had to be taught the same signal. A URL already in the
+  catalog, or one already being fetched by somebody else, returns without ever reaching the checks
+  — so a response waiting on them would have waited for something that had already happened, or
+  for the whole of a download another caller had started.
+
+  `url` bodies now answer 202 with an acknowledgement rather than 201 with the finished entry.
+  Paths, magnets and `.torrent` URLs are unchanged: they were always fast, and still answer 201
+  with the entry.
+
+## 0.25.0
+### ✨ Features and improvements
+- **A stopped download is kept, and adding the same URL again resumes it.** Staging directories
+  were named at random, so a partial transfer became unreachable the moment the add returned:
+  nothing knew where it was, and re-adding the URL opened a fresh directory beside it and started
+  from zero. They are now named from the URL, and a fetch that runs out of attempts leaves its
+  bytes in place rather than deleting them — so the second add finds the first one's work and
+  continues with a Range request. Cancelling still removes them: somebody said stop, and leaving
+  hundreds of gigabytes behind after that is the waste the deletion was written to avoid.
+
+  Note the disk consequence. A download abandoned for good now keeps its partial file until the
+  directory is removed by hand; the give-up message and a log line both name the path.
+
+### 🐞 Bug fixes
+- **Ten network blips ended an 800 GB download, whatever it had achieved.** Two faults, and the
+  attempt count was neither of them.
+
+  The budget counted every failure rather than consecutive failures that transferred nothing, so it
+  described the whole download instead of the trouble it was in. Observed in the field: 226 GB
+  across six separate stalls, then the remaining four spent inside one bad minute, because a
+  quarter of a terabyte of progress counted for nothing. An attempt that moves bytes has reached
+  the source and got data out of it, so whatever it hits next is new trouble — progress now clears
+  the count, against a high-water mark so a short attempt after a long one is not mistaken for it.
+  A ceiling on total attempts keeps that from becoming an unbounded loop.
+
+  And the wait between attempts was flat, so ten of them covered about forty-five seconds — shorter
+  than most of the interruptions they exist to survive. It now grows with each consecutive failure
+  and the base moves from 5 seconds to 30, which spans something over twenty minutes rather than
+  under one.
+
 ## 0.24.4
 ### 🐞 Bug fixes
 - **The feed advertised a .torrent nobody could fetch.** Every item named

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.24.4",
+  "version": "0.26.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -1387,7 +1387,40 @@ export function createApp({
       if (body.path) {
         entry = await library.addLocalArchive(body.path, options);
       } else if (body.url) {
-        entry = await library.addRemoteArchive(body.url, options);
+        // Answered as soon as the URL has been checked, not when the transfer
+        // finishes. Awaiting the whole thing held the response open for the
+        // length of the download -- hours for a planet archive -- so the
+        // console's add dialog stayed on screen throughout, over an archive
+        // that was visibly appearing behind it.
+        //
+        // Progress has its own route already: runningAdds() feeds /api/adds,
+        // the console polls it, and DELETE /api/adds cancels one. This is the
+        // piece that was missing rather than a new mechanism.
+        const validated = Promise.withResolvers();
+        const running = library
+          .addRemoteArchive(body.url, {
+            ...options,
+            onValidated: validated.resolve,
+          })
+          // A failure after validation has nowhere to be reported: the
+          // response has gone. It is logged where the rest of the fetch is,
+          // and swallowed here so it cannot take the process down as an
+          // unhandled rejection.
+          .catch((error) => {
+            validated.reject(error);
+            console.error(`[fetch] ${body.url}: ${error.message}`);
+          });
+
+        // Whichever comes first: the checks passing, or the whole attempt
+        // failing. A URL that does not answer, or is not an archive, still
+        // reports itself in the dialog where somebody can correct it.
+        await validated.promise;
+        void running;
+        return res.status(202).json({
+          accepted: true,
+          url: body.url,
+          message: 'fetching; progress is reported by /api/adds',
+        });
       } else if (body.magnet) {
         entry = await library.addExistingTorrent(
           { magnet: body.magnet },

package/src/config.js

@@ -326,8 +326,21 @@ const DEFAULTS = {
    * docs/internals.md — "Resuming a partial download".
    */
   fetchAttempts: 10,
-  /** How long to wait before resuming a download that stopped. */
-  fetchRetrySeconds: 5,
+  /**
+   * Base wait before resuming a download that stopped, in seconds.
+   *
+   * Multiplied by the number of consecutive failures, so the wait grows while
+   * the trouble lasts: at 30 it waits 30s, then 60, then 90, and a run of ten
+   * spans something over twenty minutes. A flat five seconds made ten attempts
+   * worth about forty-five seconds in total, which is shorter than most of the
+   * interruptions this exists to survive — a download that had transferred
+   * 226 GB was ended by a single bad minute.
+   *
+   * Counted per consecutive failure rather than per attempt: anything that
+   * moves bytes clears the count, because a transfer that got data out of the
+   * source has proved the route and whatever it hit next is new trouble.
+   */
+  fetchRetrySeconds: 30,
   /** How often to check seeding limits, in seconds. Zero disables it. */
   seedingCheckIntervalSeconds: 3600,
   /**

package/src/library.js

@@ -541,8 +541,15 @@ export class Library {
    * @returns {Promise<object>} - The catalog entry.
    */
   async addRemoteArchive(url, options = {}) {
+    // Both shortcuts below have to fire onValidated before returning, and for
+    // the same reason: a caller waiting on it to answer a request would
+    // otherwise wait for something that has already happened, or -- worse --
+    // for the whole of a download somebody else started.
     const existing = this.#catalog.findBySource(url);
-    if (existing) return existing;
+    if (existing) {
+      options.onValidated?.({ url, kind: existing.kind, held: true });
+      return existing;
+    }
 
     // A second request for a URL already being fetched joins the first; the
     // catalog cannot answer this, since an entry exists only once the download
@@ -550,6 +557,7 @@ export class Library {
     const inFlight = this.#inFlight.get(url);
     if (inFlight) {
       console.log(`[fetch] ${url} is already being fetched; joining that one`);
+      options.onValidated?.({ url, joined: true });
       return inFlight;
     }
 
@@ -590,6 +598,18 @@ export class Library {
       allowUnknown: options.allowUnknown ?? this.#config.allowUnknownArchives,
     });
 
+    // Everything a caller can do something about has now been checked: the URL
+    // answers, it is an archive of a kind this will publish, and it is not a
+    // credential being broadcast to a swarm. What remains is the transfer,
+    // which for a planet archive is hours.
+    //
+    // A caller that wants to stop waiting there says so with onValidated. That
+    // is the difference between a dialog that closes on a bad URL with the
+    // reason in it, and one that sits open for the length of the download --
+    // and the console was always written for the former, since it reports
+    // progress through runningAdds() and says "watch the log" as it closes.
+    options.onValidated?.({ url, kind: identified.kind });
+
     const summary =
       identified.kind === 'pmtiles'
         ? await probePMTiles(url).catch(() => undefined)
@@ -606,8 +626,22 @@ export class Library {
     // Downloaded into a directory of its own, then moved once the infohash
     // exists to be filed under. See docs/internals.md — "Staging, for an
     // archive fetched from a URL".
+    // Named from the URL rather than at random, so a download that stopped can
+    // be picked up again.
+    //
+    // A random name made every add its own directory, which meant a partial
+    // transfer was unreachable the moment the add returned: nothing knew where
+    // it was, and re-adding the same URL opened a fresh directory beside it and
+    // started from zero. For a planet archive that is hundreds of gigabytes
+    // thrown away because a network went out for a few minutes. Derived from
+    // the URL, the second add finds the first one's bytes and downloadTo
+    // resumes with a Range request.
     const staging = retain
-      ? path.join(root, INCOMING, crypto.randomBytes(8).toString('hex'))
+      ? path.join(
+          root,
+          INCOMING,
+          crypto.createHash('sha256').update(url).digest('hex').slice(0, 16),
+        )
       : undefined;
 
     // The origin is a valid web seed for exactly these bytes, so it is used as
@@ -658,12 +692,26 @@ export class Library {
         },
       });
     } catch (error) {
-      // A cancelled or failed fetch leaves a partial file in a directory
-      // nothing will ever look in again. Left alone it is invisible waste —
-      // and for a planet archive, invisible waste measured in gigabytes.
       this.#running.delete(url);
-      if (staging)
+      // Cancelling is a decision to stop wanting this; running out of attempts
+      // is not. The two used to be cleaned up identically, so a download that
+      // survived six stalls and reached 226 GB had all of it deleted by the
+      // seventh -- and the resume that the staging directory exists to make
+      // possible had nothing left to resume from.
+      //
+      // A cancelled fetch is still removed. Somebody said stop, and leaving
+      // gigabytes behind after that is the invisible waste this was written to
+      // avoid in the first place.
+      const cancelled = controller.signal.aborted;
+      if (staging && cancelled) {
         await fs.rm(staging, { recursive: true, force: true }).catch(() => {});
+      } else if (staging) {
+        console.warn(
+          `[fetch] keeping the partial download in ${staging}; adding ${url} ` +
+            'again resumes it, and discarding it is a matter of removing that ' +
+            'directory',
+        );
+      }
       throw error;
     }
 

package/src/torrent-create.js

@@ -230,7 +230,31 @@ async function downloadTo(url, target, onProgress, signal, options = {}) {
   let lastReport = 0;
   let lastError;
 
-  for (let attempt = 1; attempt <= attempts; attempt += 1) {
+  // The budget counts *consecutive* failures that moved nothing, not failures.
+  //
+  // Counting every failure made the budget a property of the whole download
+  // rather than of the trouble it is in, and for a large archive those are not
+  // the same thing at all. A 700 GiB transfer over a domestic line drops
+  // occasionally; observed in the field, a download reached 226 GB across six
+  // separate stalls and then exhausted the remaining four on a single bad
+  // minute, because nothing about a quarter of a terabyte of progress counted
+  // for anything. An attempt that transferred bytes proves the source and the
+  // route are alive, so the trouble it hit is over and the next one is new.
+  //
+  // The high-water mark rather than the last attempt's figure: an attempt can
+  // fail having written less than a previous one already had on disk, and that
+  // is not progress.
+  let consumed = 0;
+  let best = await bytesOnDisk(target);
+  // Reset does mean an unlucky download can go round more times than the
+  // budget names, which is the point, so there is a ceiling as well: a source
+  // dribbling a few bytes before dropping every time would otherwise retry
+  // for ever.
+  const ceiling = attempts * 10;
+  let taken = 0;
+
+  while (consumed < attempts && taken < ceiling) {
+    taken += 1;
     const from = await bytesOnDisk(target);
     // A file already at full length is one a previous attempt finished, and
     // that the caller died before renaming. Re-fetching it buys nothing.
@@ -249,8 +273,11 @@ async function downloadTo(url, target, onProgress, signal, options = {}) {
       // A cancelled download is a decision, not a failure to retry past.
       if (signal?.aborted) throw error;
       lastError = error;
-      if (attempt === attempts) break;
-      await delay(retryDelayMs, signal);
+      // Nothing was transferred -- the request never opened -- so this one
+      // always counts.
+      consumed += 1;
+      if (consumed >= attempts) break;
+      await delay(retryDelayMs * consumed, signal);
       continue;
     }
 
@@ -324,18 +351,38 @@ async function downloadTo(url, target, onProgress, signal, options = {}) {
       if (signal?.aborted) throw error;
       lastError = error;
       const reached = await bytesOnDisk(target);
+      // Progress clears the slate. Anything that moved bytes reached the
+      // source and got data out of it, so whatever it then ran into is a new
+      // problem rather than a continuation of the last one.
+      if (reached > best) {
+        best = reached;
+        consumed = 0;
+      } else {
+        consumed += 1;
+      }
       console.warn(
         `[fetch] ${url} stopped at ${reached} bytes ` +
-          `(attempt ${attempt}/${attempts}): ${error.message}`,
+          `(${consumed}/${attempts} consecutive without progress): ` +
+          error.message,
       );
-      if (attempt === attempts) break;
-      await delay(retryDelayMs, signal);
+      if (consumed >= attempts) break;
+      // Growing with each consecutive failure, so a budget spans an outage
+      // rather than a moment: at the default of 30s this waits 30, 60, 90 …
+      // and ten of them cover something over twenty minutes. A flat delay made
+      // ten attempts worth about forty-five seconds, which is shorter than
+      // most of the interruptions it exists to survive.
+      await delay(retryDelayMs * consumed, signal);
     }
   }
 
+  // Says what was reached as well as what failed. What matters when this lands
+  // is whether there is anything worth resuming, and the byte count is the
+  // whole of that answer -- the partial file is kept, so re-adding the same URL
+  // continues from here rather than starting again.
   throw new Error(
-    `could not finish downloading ${url} after ${attempts} attempts: ` +
-      `${lastError?.message ?? 'unknown error'}`,
+    `could not finish downloading ${url} after ${consumed} consecutive ` +
+      `attempts without progress (${best} bytes transferred, kept for a ` +
+      `resume): ${lastError?.message ?? 'unknown error'}`,
   );
 }