pmtiles-swarm 0.25.0 → 0.26.1

package/CHANGELOG.md

@@ -7,6 +7,48 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.26.1
+### 🐞 Bug fixes
+- **A download that finished was fetched all over again.** Resuming looked only at the
+  `.incomplete` path. A run that transferred the whole archive, had the marker removed, and then
+  stopped during the hashing left the file under its final name — so the restart found nothing to
+  resume and re-fetched every byte, with the finished copy sitting beside it the entire time.
+  Reported after 700 GB was transferred twice.
+
+  A file already under the final name is now checked against the length the source reports, and
+  hashed where it matches. Where it does not match it is not the archive being asked for, whatever
+  its name says, and the fetch proceeds — hashing it would publish the wrong bytes under the right
+  name, which is worse than transferring it again.
+- **Hashing looked like a hang.** Progress was reported for the download and then nothing at all,
+  while `createTorrentFromFile` read the whole archive to build the piece hashes — twice, with
+  `md5` on. For a planet archive that is the longer half of the work and it was completely silent,
+  so a fetch reaching 100% and going quiet read as a stall. Reported as "it completed and never
+  started making the torrent", when it had been making it for some time. It now says what it is
+  doing when it starts, reports a heartbeat every minute while it runs, and says how long it took.
+
+## 0.26.0
+### 🐞 Bug fixes
+- **The add dialog stayed on screen for the length of a download.** `POST /api/torrents` awaited
+  the entire transfer before answering, so for a URL the response arrived hours after the request
+  — and the console, which closes the dialog when the response lands, sat there over an archive
+  visibly appearing behind it.
+
+  The console was always written for the other arrangement: it says "fetching — watch the log" as
+  it closes, polls `/api/adds` for progress, and offers `DELETE /api/adds` to cancel. Only the
+  route was missing. It now answers **202** as soon as the URL has been checked — it answers, it is
+  an archive of a publishable kind, it is not a credential about to be broadcast — and lets the
+  transfer run behind it. Everything a person can correct is still reported in the dialog, because
+  all of it is found before the first byte moves.
+
+  Both shortcuts inside `addRemoteArchive` had to be taught the same signal. A URL already in the
+  catalog, or one already being fetched by somebody else, returns without ever reaching the checks
+  — so a response waiting on them would have waited for something that had already happened, or
+  for the whole of a download another caller had started.
+
+  `url` bodies now answer 202 with an acknowledgement rather than 201 with the finished entry.
+  Paths, magnets and `.torrent` URLs are unchanged: they were always fast, and still answer 201
+  with the entry.
+
 ## 0.25.0
 ### ✨ Features and improvements
 - **A stopped download is kept, and adding the same URL again resumes it.** Staging directories

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.25.0",
+  "version": "0.26.1",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -1387,7 +1387,40 @@ export function createApp({
       if (body.path) {
         entry = await library.addLocalArchive(body.path, options);
       } else if (body.url) {
-        entry = await library.addRemoteArchive(body.url, options);
+        // Answered as soon as the URL has been checked, not when the transfer
+        // finishes. Awaiting the whole thing held the response open for the
+        // length of the download -- hours for a planet archive -- so the
+        // console's add dialog stayed on screen throughout, over an archive
+        // that was visibly appearing behind it.
+        //
+        // Progress has its own route already: runningAdds() feeds /api/adds,
+        // the console polls it, and DELETE /api/adds cancels one. This is the
+        // piece that was missing rather than a new mechanism.
+        const validated = Promise.withResolvers();
+        const running = library
+          .addRemoteArchive(body.url, {
+            ...options,
+            onValidated: validated.resolve,
+          })
+          // A failure after validation has nowhere to be reported: the
+          // response has gone. It is logged where the rest of the fetch is,
+          // and swallowed here so it cannot take the process down as an
+          // unhandled rejection.
+          .catch((error) => {
+            validated.reject(error);
+            console.error(`[fetch] ${body.url}: ${error.message}`);
+          });
+
+        // Whichever comes first: the checks passing, or the whole attempt
+        // failing. A URL that does not answer, or is not an archive, still
+        // reports itself in the dialog where somebody can correct it.
+        await validated.promise;
+        void running;
+        return res.status(202).json({
+          accepted: true,
+          url: body.url,
+          message: 'fetching; progress is reported by /api/adds',
+        });
       } else if (body.magnet) {
         entry = await library.addExistingTorrent(
           { magnet: body.magnet },

package/src/library.js

@@ -541,8 +541,15 @@ export class Library {
    * @returns {Promise<object>} - The catalog entry.
    */
   async addRemoteArchive(url, options = {}) {
+    // Both shortcuts below have to fire onValidated before returning, and for
+    // the same reason: a caller waiting on it to answer a request would
+    // otherwise wait for something that has already happened, or -- worse --
+    // for the whole of a download somebody else started.
     const existing = this.#catalog.findBySource(url);
-    if (existing) return existing;
+    if (existing) {
+      options.onValidated?.({ url, kind: existing.kind, held: true });
+      return existing;
+    }
 
     // A second request for a URL already being fetched joins the first; the
     // catalog cannot answer this, since an entry exists only once the download
@@ -550,6 +557,7 @@ export class Library {
     const inFlight = this.#inFlight.get(url);
     if (inFlight) {
       console.log(`[fetch] ${url} is already being fetched; joining that one`);
+      options.onValidated?.({ url, joined: true });
       return inFlight;
     }
 
@@ -590,6 +598,18 @@ export class Library {
       allowUnknown: options.allowUnknown ?? this.#config.allowUnknownArchives,
     });
 
+    // Everything a caller can do something about has now been checked: the URL
+    // answers, it is an archive of a kind this will publish, and it is not a
+    // credential being broadcast to a swarm. What remains is the transfer,
+    // which for a planet archive is hours.
+    //
+    // A caller that wants to stop waiting there says so with onValidated. That
+    // is the difference between a dialog that closes on a bad URL with the
+    // reason in it, and one that sits open for the length of the download --
+    // and the console was always written for the former, since it reports
+    // progress through runningAdds() and says "watch the log" as it closes.
+    options.onValidated?.({ url, kind: identified.kind });
+
     const summary =
       identified.kind === 'pmtiles'
         ? await probePMTiles(url).catch(() => undefined)

package/src/torrent-create.js

@@ -49,13 +49,48 @@ export async function createTorrentFromFile(filePath, options = {}) {
   if (!stat.isFile() || stat.size === 0) {
     throw new Error(`not a usable file: ${filePath}`);
   }
-  // A second read of the archive, which is why it is opt-in. Nothing else here
-  // touches these bytes again once the piece hashes are done.
-  const md5Digest = options.md5 ? await md5File(filePath) : undefined;
-  return buildTorrent(filePath, path.basename(filePath), stat.size, {
-    ...options,
-    md5Digest,
+
+  // Says that it started, and keeps saying it is going.
+  //
+  // Hashing reads the whole archive, and with md5 on it reads it twice — for a
+  // planet archive that is the longer half of the job, and it emitted nothing
+  // at all. The download reports every second and then stops dead, so the
+  // symptom is a fetch that reaches 100% and appears to hang; reported from the
+  // field as "it completed and never started making the torrent", when it had
+  // in fact been making it for some time.
+  const what = options.sourceUrl ?? filePath;
+  const passes = options.md5 ? 'twice, once of them for the MD5' : 'once';
+  console.log(
+    `[hash] ${what}: reading ${stat.size} bytes ${passes} to build the torrent`,
+  );
+  const startedAt = Date.now();
+  const heartbeat = setInterval(() => {
+    const minutes = Math.round((Date.now() - startedAt) / 60000);
+    console.log(`[hash] ${what}: still hashing after ${minutes}m`);
+  }, 60000);
+  heartbeat.unref?.();
+  options.onProgress?.({
+    phase: 'hashing',
+    received: stat.size,
+    total: stat.size,
   });
+
+  try {
+    // A second read of the archive, which is why it is opt-in. Nothing else
+    // here touches these bytes again once the piece hashes are done.
+    const md5Digest = options.md5 ? await md5File(filePath) : undefined;
+    const built = await buildTorrent(
+      filePath,
+      path.basename(filePath),
+      stat.size,
+      { ...options, md5Digest },
+    );
+    const seconds = Math.round((Date.now() - startedAt) / 1000);
+    console.log(`[hash] ${what}: torrent built in ${seconds}s`);
+    return built;
+  } finally {
+    clearInterval(heartbeat);
+  }
 }
 
 /**
@@ -91,21 +126,53 @@ export async function createTorrentFromUrl(url, options = {}) {
     // verify. The marker means the URL 404s until the moment it is real.
     const target = path.join(options.retainPath, name);
     const marker = options.incompleteSuffix ?? DEFAULT_SUFFIX;
-    await downloadTo(
-      url,
-      `${target}${marker}`,
-      options.onProgress,
-      options.signal,
-      {
+    const partial = `${target}${marker}`;
+
+    // A run that got all the way through the download and stopped during the
+    // hashing leaves the archive under its final name, the marker already
+    // removed. Resuming looked only at the marker path, so it saw nothing to
+    // continue, and fetched the whole thing again with the finished copy
+    // sitting beside it -- 700 GB re-transferred to arrive at a file that was
+    // already there.
+    //
+    // Hashing is the longer half for a large archive and reports nothing while
+    // it runs, so being interrupted in it is not unlikely.
+    const finished = await bytesOnDisk(target);
+    const resuming = await bytesOnDisk(partial);
+    let haveIt = false;
+    if (finished > 0 && resuming === 0) {
+      const expected = await remoteLength(url, options.signal);
+      if (expected && finished === expected) {
+        haveIt = true;
+        console.log(
+          `[fetch] ${url} is already downloaded (${finished} bytes); ` +
+            'hashing what is on disk rather than fetching it again',
+        );
+      } else {
+        // Same name, different length: whatever this is, it is not the archive
+        // being asked for, and hashing it would publish the wrong bytes under
+        // the right name. The download proceeds to the marker path as usual.
+        console.warn(
+          `[fetch] ${target} exists but is ${finished} bytes against ` +
+            `${expected ?? 'an unknown length'} at the source; fetching again`,
+        );
+      }
+    }
+
+    if (!haveIt) {
+      await downloadTo(url, partial, options.onProgress, options.signal, {
         attempts: options.fetchAttempts,
         retryDelayMs: options.fetchRetryDelayMs,
-      },
-    );
-    if (marker) await fs.rename(`${target}${marker}`, target);
+      });
+      if (marker) await fs.rename(partial, target);
+    }
+
     const created = await createTorrentFromFile(target, {
       ...options,
       name,
       webSeeds,
+      onProgress: options.onProgress,
+      sourceUrl: url,
     });
     return { ...created, retainedAt: target };
   }
@@ -157,6 +224,30 @@ export async function createTorrentFromUrl(url, options = {}) {
  * @param {string} target - The partial file.
  * @returns {Promise<number>} - Bytes on disk.
  */
+/**
+ * What the source says the archive is, in bytes, or 0 when it will not say.
+ *
+ * Used to decide whether a file already under its final name is the archive
+ * being asked for. A HEAD is enough and costs nothing next to the alternative,
+ * which is transferring the whole thing a second time to find out.
+ *
+ * @param {string} url - The archive's URL.
+ * @param {AbortSignal} [signal] - Cancels the probe.
+ * @returns {Promise<number>} - Length, or 0.
+ */
+async function remoteLength(url, signal) {
+  try {
+    const response = await fetch(url, { method: 'HEAD', signal });
+    if (!response.ok) return 0;
+    return Number(response.headers.get('content-length') ?? 0) || 0;
+  } catch {
+    // A source that refuses HEAD tells us nothing, which is not the same as
+    // telling us the file is wrong. The caller re-fetches, which is what it
+    // would have done anyway.
+    return 0;
+  }
+}
+
 async function bytesOnDisk(target) {
   const stat = await fs.stat(target).catch(() => null);
   return stat?.isFile() ? stat.size : 0;