pmtiles-swarm 0.23.0 → 0.24.0

package/CHANGELOG.md

@@ -2,6 +2,23 @@
 
 ## master
 ### ✨ Features and improvements
+- _...Add new stuff here..._
+
+### 🐞 Bug fixes
+- _...Add new stuff here..._
+
+## 0.24.0
+### ✨ Features and improvements
+- **A mirror now inherits the archive summary from the feed it follows.** `renderItem` has always
+  published `pmtiles:format`, the zoom range, the bounds and the tile count, and `parseFeed` has
+  always thrown them away — so the fact that a feed is more useful than a generic torrent feed was
+  true of the XML and of nothing else. It matters more than it sounds: an archive is servable when
+  `entry.pmtiles` exists, so a fresh mirror served nothing at all until it had read the header out
+  of the swarm itself. On an 80 GiB planet archive whose only seed is busy that is hours of a node
+  that looks joined, downloads steadily, and answers every tile request with 400. The summary now
+  comes across with the item and the archive is servable the moment it is added. It is marked
+  `source: 'feed'`, because a summary taken on trust is not the same fact as one read off the
+  header, and the head warmer still replaces it with the latter as soon as it can read one.
 - **A watched folder can ask for its stable name to be a hard link.**
   `latestLinkType: "hard"` reverses the order the two kinds are attempted in,
   for a folder whose archives are read *through* that name rather than followed
@@ -84,6 +101,27 @@
   never checked.
 
 ### 🐞 Bug fixes
+- **Head warming now says whether it is running.** Every way it could decline was a bare `return`:
+  `tiles.prewarm` false, `prewarmIntervalSeconds` at zero, a node with no tile store to read with,
+  and — the one that actually bites — a pass that finds no archive eligible to warm. All four
+  produced an identical empty log, so "warming is switched off" and "warming is working and has
+  nothing to do" could not be told apart, and on a node whose mirrors were stuck at 400 the only
+  way to tell was to read the source. It now names the reason at startup, or says how often it will
+  look; and after ten idle passes it says either that everything has been summarised or which
+  archives it is skipping for want of a recognised kind. That last case is what an archive joined by
+  magnet looks like before its metainfo arrives: `guessKind` cannot tell it is PMTiles, `due`
+  refuses it, and nothing said so.
+- **One archive whose metainfo never arrived stopped every other archive being warmed.** A sweep
+  warms a single archive, chosen as the first one due, and an archive joined by magnet that has no
+  metainfo yet is answered "not yet" and deliberately left unstamped so the next pass retries in
+  seconds rather than after the full backoff. The two together meant an archive stuck that way
+  stayed due at no cost, was chosen again on every pass, and its neighbours were never attempted at
+  all — for as long as it was stuck, which where no peer ever answers is indefinitely. A node with
+  two mirrors could therefore warm neither, having genuinely tried only one. The fast retry is now
+  bounded: after five consecutive passes the wait has plainly stopped being nearly over, and it is
+  charged as an attempt like any other so the backoff spreads them out and lets its neighbours
+  through. It says so when it makes that switch, which is otherwise the quietest moment in the
+  process — the point where an archive goes from "about to work" to "may never work".
 - **A mutable magnet had nowhere to announce.** `mutableMagnet()` was never passed trackers, so the
   BEP 46 form carried `xt`, `xs`, `dn`, `s` and `ws` and no `tr=` at all. The infohash added in
   0.21.0 was therefore unusable from a browser, which has neither DHT nor peer exchange to fall

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.23.0",
+  "version": "0.24.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/feed.js

@@ -165,8 +165,62 @@ export function formatBytes(bytes) {
  * @property {string} [category] - Item category.
  * @property {string} [mtime] - The archive's mtime on the node that built it.
  * @property {string} [mutableMagnet] - BEP 46 magnet, when the publisher has an identity for it.
+ * @property {object} [pmtiles] - Coverage summary, when the feed carries one.
  */
 
+/**
+ * Reads the `pmtiles:*` summary out of a feed item, when it has one.
+ *
+ * renderItem publishes format, zoom range, bounds, tile count and attribution
+ * for exactly this — so a subscriber can tell what an archive holds before
+ * committing to 80 GiB of it. For a long time nothing read them back, which had
+ * a consequence well beyond the feed being less informative than it looked: a
+ * mirror's `servable` flag is `Boolean(entry.pmtiles)`, so an archive arrived
+ * with no summary, was unservable, and stayed unservable until the head warmer
+ * managed to read the header out of the swarm — while the answer had been in
+ * the feed all along.
+ *
+ * Deliberately partial. There is no `vectorLayers` here, because the publisher
+ * has none to give: planetiler writes that section after every tile, so on a
+ * planet archive it is the very end of the file. An entry summarised from a
+ * feed therefore stays due for warming, which is what fills the rest in.
+ *
+ * @param {string} block - The item XML.
+ * @returns {object | undefined} - The summary, or undefined if absent.
+ */
+function mapSummary(block) {
+  const format = tag(block, 'pmtiles:format');
+  // Format is the field `servable` and the tile routes key off, so a summary
+  // without it is not a summary. Better no object at all than one that reports
+  // an archive as readable and then cannot say what is in it.
+  if (!format) return undefined;
+
+  const number = (name) => {
+    const raw = tag(block, name);
+    if (raw === undefined || raw.trim() === '') return undefined;
+    const value = Number(raw);
+    return Number.isFinite(value) ? value : undefined;
+  };
+
+  const bounds = (tag(block, 'pmtiles:bounds') ?? '')
+    .split(',')
+    .map((part) => Number(part.trim()))
+    .filter((part) => Number.isFinite(part));
+
+  return {
+    format,
+    minZoom: number('pmtiles:minzoom'),
+    maxZoom: number('pmtiles:maxzoom'),
+    bounds: bounds.length === 4 ? bounds : undefined,
+    tileCount: number('pmtiles:tiles'),
+    attribution: tag(block, 'pmtiles:attribution'),
+    // Says where this came from, because a summary taken on trust from another
+    // node is not the same fact as one read off the archive's own header, and
+    // the difference matters when the two disagree.
+    source: 'feed',
+  };
+}
+
 /**
  * Parses a subscribed feed.
  *
@@ -222,6 +276,10 @@ export function parseFeed(body) {
       // more than once, so take all of them.
       md5: tag(block, 'pmtiles:md5'),
       mtime: isoDate(tag(block, 'pmtiles:mtime')),
+      // What the publisher already knows about the archive's contents. Absent
+      // from a generic torrent feed, and absent from ours for an archive the
+      // publishing node has not summarised either.
+      pmtiles: mapSummary(block),
       categories: [...block.matchAll(/<category>([\s\S]*?)<\/category>/g)]
         .map((match) => decode(match[1]).trim())
         .filter(Boolean),

package/src/library.js

@@ -836,6 +836,17 @@ export class Library {
       complete,
       // Held until the download finishes, which may be hours away.
       originMtime: options.originMtime,
+      // What the peer that offered this says it holds, where it said anything.
+      // The head warmer replaces it with what the archive's own header says as
+      // soon as it can read one; until then this is what makes the archive
+      // servable at all, since `servable` is simply whether a summary exists.
+      //
+      // Spread rather than set, because catalog.put merges by spreading the
+      // record over the existing one -- and a key present with an undefined
+      // value still overwrites. Setting it unconditionally would mean an
+      // archive re-added from a feed that carries no summary silently lost the
+      // one it had already read for itself.
+      ...(options.pmtiles ? { pmtiles: options.pmtiles } : {}),
     });
 
     // The engine's add resolves once metadata is in hand, so for a magnet

package/src/prewarm.js

@@ -20,6 +20,39 @@ const DEFAULT_MAX_BACKOFF_SECONDS = 600;
 /** How long to let the node settle before the first attempt. */
 const DEFAULT_INITIAL_DELAY_SECONDS = 10;
 
+/** How often to look for an archive that needs its head read. */
+const DEFAULT_INTERVAL_SECONDS = 30;
+
+/**
+ * How many consecutive passes may find nothing to warm before saying so.
+ *
+ * Silence is the right output for a node whose archives are all summarised, and
+ * it is also what a node produces when every archive is being skipped for a
+ * reason nobody can see — an entry whose kind is not yet known, most often,
+ * because it was joined by magnet and its metainfo has not arrived. One line
+ * after a few minutes of that separates the two without turning a healthy node
+ * into a log generator.
+ */
+const IDLE_PASSES_BEFORE_REPORTING = 10;
+
+/**
+ * How many passes in a row an archive may claim on the strength of "not yet".
+ *
+ * An archive joined from a magnet has no metainfo until BEP 9 finishes, which
+ * is usually seconds, so charging it the full backoff wastes a wait that was
+ * nearly over — hence the fast retry. What that must not do is cost nothing for
+ * ever. A pass warms one archive, chosen as the *first* that is due, and an
+ * entry that stays due at no cost is chosen again on the next pass and on every
+ * pass after it. One archive stuck this way therefore stops every other archive
+ * on the node from being warmed at all, silently, and for as long as it is
+ * stuck — which on a node whose peer never answers is indefinitely.
+ *
+ * Past this many passes the wait has plainly stopped being nearly over, and it
+ * is charged as an attempt like any other so the backoff can spread the
+ * attempts out and let its neighbours through.
+ */
+const MAX_CONSECUTIVE_EARLY_PASSES = 5;
+
 /**
  * Whether a failure means "not yet" rather than "not working".
  *
@@ -47,8 +80,11 @@ export class HeadWarmer {
   #tried = new Map();
   #attempts = new Map();
   #waiting = new Set();
+  /** Consecutive passes each archive has answered "not yet" to. */
+  #early = new Map();
   #running = false;
   #runningSince = 0;
+  #idlePasses = 0;
 
   /**
    * @param {object} tiles - The tile store, for `summarize`.
@@ -71,6 +107,31 @@ export class HeadWarmer {
     );
   }
 
+  /**
+   * Why this node is not warming, or null when it is.
+   *
+   * Exists so `start()` can say which of its several reasons applied. They are
+   * each a bare `return` on a condition, and the difference between them was
+   * invisible: a node with warming switched off, a node whose interval was set
+   * to zero, and a node warming perfectly well with nothing to do all produced
+   * exactly the same empty log.
+   * @returns {string | null} - The reason, or null.
+   */
+  get disabledReason() {
+    if (this.#config.tiles?.prewarm === false) {
+      return 'tiles.prewarm is false';
+    }
+    if (typeof this.#tiles?.summarize !== 'function') {
+      return 'this node has no tile store to read headers with';
+    }
+    const seconds =
+      this.#config.tiles?.prewarmIntervalSeconds ?? DEFAULT_INTERVAL_SECONDS;
+    if (seconds <= 0) {
+      return `tiles.prewarmIntervalSeconds is ${seconds}`;
+    }
+    return null;
+  }
+
   /**
    * Whether an archive is worth reading the head of right now.
    *
@@ -162,7 +223,11 @@ export class HeadWarmer {
     }
 
     const entry = this.#catalog.list().find((candidate) => this.due(candidate));
-    if (!entry) return null;
+    if (!entry) {
+      this.#reportIdle();
+      return null;
+    }
+    this.#idlePasses = 0;
 
     this.#running = true;
     this.#runningSince = this.#now();
@@ -175,6 +240,11 @@ export class HeadWarmer {
 
       this.#tried.set(entry.infoHash, this.#now());
       this.#waiting.delete(entry.infoHash);
+      // The metainfo evidently arrived, so the run of "not yet" answers is
+      // over. Cleared rather than left standing, since a later magnet re-add of
+      // the same archive starts its own wait and should get its own fast
+      // retries rather than inheriting a spent allowance.
+      this.#early.delete(entry.infoHash);
       // Counted even on success, because a read that got the header and not
       // the metadata has not finished and will be back. The count only matters
       // while an archive is still due, and one that is done is never asked
@@ -210,44 +280,103 @@ export class HeadWarmer {
       return stored;
     } catch (error) {
       if (tooEarly(error)) {
-        // Not stamped, so the next pass tries again in seconds rather than in
-        // minutes — and said once, because a node that has just started will
-        // answer this way until the metainfo lands.
-        if (!this.#waiting.has(entry.infoHash)) {
-          this.#waiting.add(entry.infoHash);
-          console.log(
-            `[warm] ${entry.name}: waiting for the torrent metadata before ` +
-              'reading its head',
-          );
+        const passes = (this.#early.get(entry.infoHash) ?? 0) + 1;
+        this.#early.set(entry.infoHash, passes);
+
+        if (passes <= MAX_CONSECUTIVE_EARLY_PASSES) {
+          // Not stamped, so the next pass tries again in seconds rather than in
+          // minutes — and said once, because a node that has just started will
+          // answer this way until the metainfo lands.
+          if (!this.#waiting.has(entry.infoHash)) {
+            this.#waiting.add(entry.infoHash);
+            console.log(
+              `[warm] ${entry.name}: waiting for the torrent metadata before ` +
+                'reading its head',
+            );
+          }
+          return null;
         }
-        return null;
+
+        // Waited long enough to stop calling it a wait. Said out loud, because
+        // this is the moment the archive goes from "about to work" to "may
+        // never work", and it is otherwise the quietest possible transition.
+        console.warn(
+          `[warm] ${entry.name}: still has no torrent metadata after ` +
+            `${passes} passes; treating that as a failure so other archives ` +
+            'get a turn',
+        );
+      } else {
+        // A real attempt: it reached the swarm and found nothing. Ordinary
+        // while an archive is young, since the piece holding the header may not
+        // exist anywhere reachable yet. Worth saying, because it is the answer
+        // to "why is my preview blank", and the backoff keeps it from becoming
+        // noise.
+        console.warn(`[warm] ${entry.name}: ${error.message}`);
+        this.#early.delete(entry.infoHash);
       }
 
-      // A real attempt: it reached the swarm and found nothing. Ordinary while
-      // an archive is young, since the piece holding the header may not exist
-      // anywhere reachable yet. Worth saying, because it is the answer to "why
-      // is my preview blank", and the backoff keeps it from becoming noise.
       this.#tried.set(entry.infoHash, this.#now());
       this.#waiting.delete(entry.infoHash);
       this.#attempts.set(
         entry.infoHash,
         (this.#attempts.get(entry.infoHash) ?? 0) + 1,
       );
-      console.warn(`[warm] ${entry.name}: ${error.message}`);
       return null;
     } finally {
       this.#running = false;
     }
   }
 
+  /**
+   * Says why nothing is being warmed, when that has gone on long enough to be
+   * worth explaining.
+   *
+   * Distinguishes the two ways a pass finds nothing: everything is summarised,
+   * which is the goal, and everything is being skipped, which is a fault that
+   * otherwise looks identical. The archives that are neither summarised nor
+   * eligible are named, because the reason is almost always visible in the name
+   * — an entry still called by its infohash has no metainfo yet, so `guessKind`
+   * cannot tell it is PMTiles and `due` refuses it.
+   * @returns {void}
+   */
+  #reportIdle() {
+    this.#idlePasses++;
+    if (this.#idlePasses !== IDLE_PASSES_BEFORE_REPORTING) return;
+
+    const unread = this.#catalog
+      .list()
+      .filter((entry) => !entry.pmtiles?.format);
+    if (unread.length === 0) {
+      console.log('[warm] every archive has been summarised; nothing to warm');
+      return;
+    }
+    console.warn(
+      `[warm] ${unread.length} archive(s) have no summary and none are ` +
+        'eligible to be warmed; their kind is not PMTiles, or is not yet ' +
+        `known: ${unread
+          .slice(0, 5)
+          .map((entry) => entry.name ?? entry.infoHash)
+          .join(', ')}${unread.length > 5 ? ', …' : ''}`,
+    );
+  }
+
   /**
    * Starts warming, and keeps at it.
    * @returns {void}
    */
   start() {
-    if (!this.enabled) return;
-    const seconds = this.#config.tiles?.prewarmIntervalSeconds ?? 30;
-    if (seconds <= 0) return;
+    // Said once at startup, either way. A node that is not warming is a node
+    // whose mirrored archives will not become servable on their own, and that
+    // is far too important to be inferred from the absence of log lines --
+    // which is exactly how it had to be diagnosed before.
+    const reason = this.disabledReason;
+    if (reason) {
+      console.warn(`[warm] not warming: ${reason}`);
+      return;
+    }
+    const seconds =
+      this.#config.tiles?.prewarmIntervalSeconds ?? DEFAULT_INTERVAL_SECONDS;
+    console.log(`[warm] reading archive heads every ${seconds}s`);
 
     const run = () =>
       this.sweep().catch((error) =>

package/src/subscriptions.js

@@ -409,6 +409,16 @@ export class SubscriptionManager {
       // The mtime on the node that built it, which BitTorrent will not carry.
       // Restored when the download completes.
       originMtime: item.mtime,
+      // What the publisher says the archive holds: format, zoom range, bounds.
+      //
+      // This is the difference between a mirror that can serve tiles the moment
+      // it joins and one that cannot serve them until it has read the header out
+      // of the swarm -- which, for an 80 GiB archive whose only seed is busy,
+      // can be a very long time, and used to be the only way an entry ever got
+      // a summary at all. The head warmer still runs and still overwrites this
+      // with what the archive itself says; this only means the wait is not
+      // spent unservable.
+      pmtiles: item.pmtiles,
     };
 
     // The .torrent is preferred where there is one: it carries the trackers