pmtiles-swarm 0.7.0 → 0.7.1

package/CHANGELOG.md

@@ -7,6 +7,34 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.7.1
+### 🐞 Bug fixes
+- **A feed no longer deletes its only complete copy.** Retention was written for a watched folder
+  and a scheduled source, where the archive it is handed is already whole — the file was there, or
+  the fetch finished. A subscription is not like that: it joins a torrent, and the data arrives
+  hours later. So `keep: 1` on a feed removed last week's complete copy the moment this week's
+  torrent was announced, leaving nothing complete at all for the length of an 88 GiB download.
+  `keepDays` had the same exposure, a copy ageing out while its replacement was still arriving.
+
+  Retention on a subscription now waits for the newest copy to be whole, which makes `keep: 1`
+  mean *the last complete copy* — the only reading of it that is safe there. It also runs on every
+  poll rather than only on polls that took something, because what is being waited for is a
+  download finishing rather than a poll happening. Watched folders and scheduled sources are
+  unchanged: they hand over a finished archive, and asking them for a completion marker they never
+  set would have stopped their retention working.
+
+### 📚 Documentation
+- **`prune` does not apply to an RSS subscription**, which the documentation did not say and a
+  reader would reasonably have assumed otherwise — it is accepted there and quietly does nothing.
+  Absence from a bounded feed is not evidence that anything was withdrawn, so pruning needs a
+  catalogue. The two questions are now separated where they are described: whether the publisher
+  still offers an archive, and whether you still want it on your disk.
+- **Feed retention is documented**, along with the claim it replaces. The subscribing guide said a
+  node following a feed "accumulates and never sheds", which was true when it was written and is
+  what `keep` and `keepDays` on a subscription now answer.
+- **The README describes the two settings sections** rather than the single table they replaced,
+  and covers `newest`, `keep` and `keepDays` — none of which it mentioned.
+
 ## 0.7.0
 ### ✨ Features and improvements
 - **A feed can be told how long to keep what it brings in.** `keep` and `keepDays` now work on a

package/README.md

@@ -488,16 +488,28 @@ restart.
 
 ### Following other nodes
 
-`subscriptions` are the peers this node takes archives from, editable in Settings under **Remote
-nodes**. An RSS feed says "here is what is new" and is bounded by the publisher's `feedMaxItems`,
-so a node offline long enough misses things permanently; a `/api/catalog` URL says "here is
-everything", which is what makes reconciling — and pruning — possible.
+`subscriptions` are what this node takes archives from, editable in Settings as two sections:
+**RSS feeds** and **Remote nodes**. They are not one thing in two costumes. A feed says "here is
+what is new" and is bounded by the publisher's `feedMaxItems`, so a node offline long enough
+misses things permanently, and an absence from one proves nothing; a `/api/catalog` URL says
+"here is everything", which is what makes reconciling — and pruning — possible. A row belongs to
+whichever section it sits in, and is saved with that protocol rather than leaving it to be
+inferred from the URL.
 
 `mode` decides what following one costs: `cache` joins the swarm and fetches only what is read,
 `mirror` commits to a whole copy of every archive the peer lists. `token` is presented to the peer,
-which may then publish more than it publishes to the world. `prune` is off unless chosen, only ever
-considers archives that peer sent, and never acts on a filtered or partial view — start a new peer
-on `"report"` and watch it before trusting it with more.
+which may then publish more than it publishes to the world.
+
+`newest` caps how many items one check of a **feed** may take, counting from the newest, and
+defaults to 1 — planet.openstreetmap.org lists five planet dumps, and taking the lot is four
+hundred gigabytes nobody asked for. It means nothing to a catalogue, which lists everything.
+
+`keep` and `keepDays` retire what a subscription has brought in, exactly as they do for a watched
+folder or a scheduled source: only after something new has landed, and never the newest copy. They
+answer for your disk. `prune` answers for the publisher — it is off unless chosen, only ever
+considers archives that peer sent, never acts on a filtered or partial view, and **applies to a
+catalogue only**, since absence from a bounded feed is not evidence that anything was withdrawn.
+Start a new peer on `"report"` and watch it before trusting it with more.
 
 Peer tokens are redacted from `GET /api/config` like any other credential, and a save that echoes
 the placeholder back keeps the stored one.

package/docs/subscribing.md

@@ -117,9 +117,16 @@ poor trade for the archive itself.
 
 ## Removing what a peer no longer offers
 
-A feed can only ever say "here is something new". Following one, a node
-accumulates and never sheds — remove an archive at the source and every
-subscriber keeps seeding it indefinitely.
+Two different questions, with two different answers, and it is worth keeping
+them apart: whether the **publisher** still offers an archive, and whether
+**you** still want it on your disk.
+
+### The publisher stopped offering it
+
+A feed can only ever say "here is something new". Nothing about it distinguishes
+"withdrawn" from "further down a list that only carries five items", so absence
+from a feed is not evidence of anything and **`prune` does not apply to an RSS
+subscription at all** — setting it there is quietly inert.
 
 The catalogue API can say "here is everything", which is the only way a
 consumer can notice an absence. `prune` acts on that, and it is deliberately
@@ -289,6 +296,15 @@ OpenStreetMap publishes for the planet dumps:
 it is **1 by default**. That feed lists five planet dumps, so without a cap the
 first poll is roughly four hundred gigabytes. `0` takes everything it lists.
 
+`keepDays` is how long to keep what it brings in. A feed publishing weekly
+leaves a copy behind every week and the publisher goes on listing all of them,
+so without it a subscription grows for ever — which is the case pruning cannot
+help with, because nothing was ever withdrawn. `keep` says the same thing as a
+count. Both are the rules a watched folder and a scheduled source retire under,
+applied by the same code and with the same guards: **only after something new
+has landed**, and **never the newest copy**, so a feed that goes quiet cannot
+empty your disk.
+
 The items are `.osm.pbf`, not map archives, and that is fine: joining an
 existing torrent does not require the content to be anything in particular —
 only *creating* one does. The archive simply is not servable as tiles, and

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.7.0",
+  "version": "0.7.1",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/retention.js

@@ -99,6 +99,7 @@ export function expired({ family, keep, keepDays, now = Date.now() }) {
  * @param {number} [options.keep] - How many of the newest to hold.
  * @param {number} [options.keepDays] - How old a build may get, in days.
  * @param {string} options.label - How to name this family in the log.
+ * @param {boolean} [options.requireComplete] - Wait for the newest to be whole.
  * @param {number} [options.now] - The current time, for testing.
  * @returns {Promise<string[]>} - The infohashes removed.
  */
@@ -109,8 +110,23 @@ export async function retire({
   keep,
   keepDays,
   label,
+  requireComplete = false,
   now,
 }) {
+  // Where the new copy is a download rather than a file that already exists.
+  //
+  // A watched folder and a scheduled source hand over an archive that is
+  // whole: the file was there, or the fetch finished. A subscription does not
+  // — it joins a torrent, and the data arrives hours later. Retiring on the
+  // join would delete last week's complete copy the moment this week's was
+  // announced, leaving nothing complete for the length of the download.
+  //
+  // So `keep: 1` here means "the last complete copy", which is the only
+  // reading of it that is safe.
+  if (requireComplete && entry?.complete !== true) {
+    return [];
+  }
+
   const doomed = expired({ family, keep, keepDays, now });
   if (doomed.length === 0) return [];
 

package/src/subscriptions.js

@@ -180,7 +180,7 @@ export class SubscriptionManager {
     // from a bounded feed proves nothing — but age applies whatever the list
     // is, which is what makes this the answer for a feed that publishes
     // weekly for ever.
-    await this.#retire(subscription, added);
+    await this.#retire(subscription);
     return added;
   }
 
@@ -259,7 +259,7 @@ export class SubscriptionManager {
     }
 
     await this.#prune(subscription, document, archives);
-    await this.#retire(subscription, added);
+    await this.#retire(subscription);
     return added;
   }
 
@@ -279,23 +279,33 @@ export class SubscriptionManager {
    * Only after something new has landed, and never the newest copy, which are
    * the same guards a scheduled source retires under. See `retire`.
    * @param {object} subscription - The subscription.
-   * @param {object[]} added - What this pass took, newest first.
    * @returns {Promise<void>} - Resolves once anything due has gone.
    */
-  async #retire(subscription, added) {
-    if (added.length === 0 || !retains(subscription)) return;
-
+  async #retire(subscription) {
+    if (!retains(subscription)) return;
+
+    // Only what this feed brought in. An archive built here, added by hand, or
+    // taken from another peer is not this subscription's to remove.
+    const family = this.#library.catalog
+      .list()
+      .filter((entry) => entry.source?.subscription === subscription.url);
+    if (family.length === 0) return;
+
+    // Every poll, not only the polls that took something. What is being waited
+    // for is the newest copy *finishing*, which happens hours after the poll
+    // that started it — so retiring only on the way in would either act far
+    // too early or never act at all.
     await retire({
       library: this.#library,
-      // Only what this feed brought in. An archive built here, added by hand,
-      // or taken from another peer is not this subscription's to remove.
-      family: this.#library.catalog
-        .list()
-        .filter((entry) => entry.source?.subscription === subscription.url),
-      entry: added[0],
+      family,
+      entry: family[0],
       keep: subscription.keep,
       keepDays: subscription.keepDays,
       label: `[sync] ${subscription.url}`,
+      // The newest copy here is a download rather than a file that was already
+      // there, so nothing goes until it is whole. That is what makes `keep: 1`
+      // mean "the last complete copy".
+      requireComplete: true,
     });
   }