pmtiles-swarm 0.4.0 → 0.4.1

package/CHANGELOG.md

@@ -7,6 +7,25 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.4.1
+### 🐞 Bug fixes
+- **A feed no longer walks backwards through its own history.** An item already taken was
+  skipped and the loop carried on to the older one below it, and the cap counts what was
+  *added* — so every poll took exactly one archive and every poll took a different one, until
+  the whole backlog was on disk. Against planet.openstreetmap.org that is five 88 GiB dumps
+  arriving a quarter of an hour apart from a subscription asking for one. Items run newest
+  first, so reaching one already held now stops the pass: everything after it is older than
+  something already on disk. A build that could not be fetched stops it too — one bad fetch is
+  a reason to retry shortly, not to take last week's instead.
+- **A subscription's `mode` had no effect.** It reached the add as `paused`, which nothing
+  reads — not the library and not the engine — so every item a feed brought in arrived as a
+  cache whatever the subscription said, and a `"mode": "mirror"` feed quietly fetched nothing.
+  A cache subscription only looked correct because cache is the default. It is now passed as
+  `mode`, the name the library actually reads.
+- **The console says what an empty availability bar means.** It counts connected peers and not
+  this node, so an archive only this node holds shows nothing — the truth about the swarm rather
+  than about the file, but worth saying beside a Downloaded bar that is full.
+
 ## 0.4.0
 ### ✨ Features and improvements
 - **A watched folder can set the torrent comment**, which is where attribution and licence belong —

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.4.0",
+  "version": "0.4.1",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/subscriptions.js

@@ -141,12 +141,26 @@ export class SubscriptionManager {
         continue;
       }
       const marker = item.infoHash ?? item.magnet ?? item.torrentUrl;
-      if (this.#seen.has(marker)) continue;
-      this.#seen.add(marker);
+
+      // Stop, rather than skip past it to the next one.
+      //
+      // Items run newest first, so reaching one already taken means
+      // everything after it is older than something on disk. Skipping to the
+      // next instead takes one more archive every poll and walks backwards
+      // through the feed until the whole backlog is held — which for the
+      // planet feed is five 88 GiB dumps nobody asked for, arriving one a
+      // quarter of an hour apart. The cap counts what was added, so it does
+      // not stop this: each poll adds exactly one, and each poll adds a
+      // different one.
+      if (this.#seen.has(marker)) break;
 
       try {
         const entry = await this.#add(item, subscription);
         if (entry) {
+          // Marked only once it worked. A .torrent that 404s for an hour
+          // should be retried on the next poll rather than passed over for
+          // good on the strength of one bad fetch.
+          this.#seen.add(marker);
           added.push(entry);
           console.log(
             `[feed] ${subscription.mode ?? 'cache'} ${entry.name} from ${subscription.url}`,
@@ -154,6 +168,10 @@ export class SubscriptionManager {
         }
       } catch (error) {
         console.error(`[feed] could not add "${item.title}": ${error.message}`);
+        // And stop here too. The newest build being briefly unreachable is a
+        // reason to try again shortly, not a reason to fetch last week's
+        // instead — which is the same backwards walk by another route.
+        break;
       }
     }
     return added;
@@ -340,8 +358,15 @@ export class SubscriptionManager {
       // from one built here or added by hand, and would happily delete both.
       subscriptionUrl: subscription.url,
       // Cache mode joins the swarm without pulling the whole archive; the
-      // pieces it does hold are still served to other peers.
-      paused: (subscription.mode ?? 'cache') === 'cache',
+      // pieces it does hold are still served to other peers. Mirror commits to
+      // a whole copy.
+      //
+      // Passed as `mode`, which is the name the library reads. It had been
+      // passed as `paused`, which nothing reads — so a feed's mode did not
+      // reach the add at all, and every item arrived as cache whatever the
+      // subscription said. Defaulted here as well as there, so the two cannot
+      // drift apart into disagreeing about what an unset mode means.
+      mode: subscription.mode ?? 'cache',
     };
 
     // The .torrent is preferred where there is one: it carries the trackers

package/src/web/index.html

@@ -1190,7 +1190,9 @@
                 shown as held when <i>all</i> of them are. Availability is the
                 <i>rarest</i> piece in each column: one piece nobody has is the
                 answer to "can this still be completed", however well supplied
-                its neighbours are.
+                its neighbours are. It counts <i>connected peers</i> and not
+                this node, so an archive only this node holds shows an empty
+                bar — which is the truth about the swarm, not about the file.
               </div>
               ${
                 (info.peers ?? []).length > 0