pmtiles-swarm 0.6.0 → 0.7.1

package/CHANGELOG.md

@@ -7,6 +7,58 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.7.1
+### 🐞 Bug fixes
+- **A feed no longer deletes its only complete copy.** Retention was written for a watched folder
+  and a scheduled source, where the archive it is handed is already whole — the file was there, or
+  the fetch finished. A subscription is not like that: it joins a torrent, and the data arrives
+  hours later. So `keep: 1` on a feed removed last week's complete copy the moment this week's
+  torrent was announced, leaving nothing complete at all for the length of an 88 GiB download.
+  `keepDays` had the same exposure, a copy ageing out while its replacement was still arriving.
+
+  Retention on a subscription now waits for the newest copy to be whole, which makes `keep: 1`
+  mean *the last complete copy* — the only reading of it that is safe there. It also runs on every
+  poll rather than only on polls that took something, because what is being waited for is a
+  download finishing rather than a poll happening. Watched folders and scheduled sources are
+  unchanged: they hand over a finished archive, and asking them for a completion marker they never
+  set would have stopped their retention working.
+
+### 📚 Documentation
+- **`prune` does not apply to an RSS subscription**, which the documentation did not say and a
+  reader would reasonably have assumed otherwise — it is accepted there and quietly does nothing.
+  Absence from a bounded feed is not evidence that anything was withdrawn, so pruning needs a
+  catalogue. The two questions are now separated where they are described: whether the publisher
+  still offers an archive, and whether you still want it on your disk.
+- **Feed retention is documented**, along with the claim it replaces. The subscribing guide said a
+  node following a feed "accumulates and never sheds", which was true when it was written and is
+  what `keep` and `keepDays` on a subscription now answer.
+- **The README describes the two settings sections** rather than the single table they replaced,
+  and covers `newest`, `keep` and `keepDays` — none of which it mentioned.
+
+## 0.7.0
+### ✨ Features and improvements
+- **A feed can be told how long to keep what it brings in.** `keep` and `keepDays` now work on a
+  subscription, applied by the same code a watched folder and a scheduled source retire under and
+  with the same guards: only after something new has landed, and never the newest copy. A feed
+  publishing weekly leaves a copy behind every week and the publisher goes on listing all of them,
+  so `keepDays: 10` against a weekly feed keeps a fortnight and drops the rest.
+
+  This is not what `prune` does, and the difference is why a feed needed its own answer. Pruning is
+  about the publisher — it stopped offering this, so let it go — and needs a complete listing for an
+  absence to mean anything, which is why it applies to a catalogue and not to a feed.
+  planet.openstreetmap.org lists five dumps and says nothing about the hundreds before them. Age is
+  age however short the list is.
+- **Feeds and remote nodes are two sections rather than one table with a dropdown.** They are not
+  one thing in two costumes: a feed is bounded and says "here is what is new", so it caps items per
+  check and can never prune; a catalogue says "here is everything", which is the only thing that
+  makes an absence meaningful. Half the columns applied to one and half to the other. A row is now
+  RSS or API because of the table it sits in, which also means a saved row states its protocol
+  instead of leaving it to be guessed from the URL later. Rows written before this are sorted into
+  a section by the same rule the subscription manager itself uses.
+- **A feed's save path is editable.** `savePath` was accepted in the configuration and offered
+  nowhere, so it could only be set by hand — and before the row editor learned to keep fields it
+  does not show, pressing Save would have deleted it.
+
 ## 0.6.0
 ### ✨ Features and improvements
 - **Check now, on scheduled sources and on feeds.** A schedule describes ordinary operation, and

package/README.md

@@ -488,16 +488,28 @@ restart.
 
 ### Following other nodes
 
-`subscriptions` are the peers this node takes archives from, editable in Settings under **Remote
-nodes**. An RSS feed says "here is what is new" and is bounded by the publisher's `feedMaxItems`,
-so a node offline long enough misses things permanently; a `/api/catalog` URL says "here is
-everything", which is what makes reconciling — and pruning — possible.
+`subscriptions` are what this node takes archives from, editable in Settings as two sections:
+**RSS feeds** and **Remote nodes**. They are not one thing in two costumes. A feed says "here is
+what is new" and is bounded by the publisher's `feedMaxItems`, so a node offline long enough
+misses things permanently, and an absence from one proves nothing; a `/api/catalog` URL says
+"here is everything", which is what makes reconciling — and pruning — possible. A row belongs to
+whichever section it sits in, and is saved with that protocol rather than leaving it to be
+inferred from the URL.
 
 `mode` decides what following one costs: `cache` joins the swarm and fetches only what is read,
 `mirror` commits to a whole copy of every archive the peer lists. `token` is presented to the peer,
-which may then publish more than it publishes to the world. `prune` is off unless chosen, only ever
-considers archives that peer sent, and never acts on a filtered or partial view — start a new peer
-on `"report"` and watch it before trusting it with more.
+which may then publish more than it publishes to the world.
+
+`newest` caps how many items one check of a **feed** may take, counting from the newest, and
+defaults to 1 — planet.openstreetmap.org lists five planet dumps, and taking the lot is four
+hundred gigabytes nobody asked for. It means nothing to a catalogue, which lists everything.
+
+`keep` and `keepDays` retire what a subscription has brought in, exactly as they do for a watched
+folder or a scheduled source: only after something new has landed, and never the newest copy. They
+answer for your disk. `prune` answers for the publisher — it is off unless chosen, only ever
+considers archives that peer sent, never acts on a filtered or partial view, and **applies to a
+catalogue only**, since absence from a bounded feed is not evidence that anything was withdrawn.
+Start a new peer on `"report"` and watch it before trusting it with more.
 
 Peer tokens are redacted from `GET /api/config` like any other credential, and a save that echoes
 the placeholder back keeps the stored one.

package/docs/subscribing.md

@@ -117,9 +117,16 @@ poor trade for the archive itself.
 
 ## Removing what a peer no longer offers
 
-A feed can only ever say "here is something new". Following one, a node
-accumulates and never sheds — remove an archive at the source and every
-subscriber keeps seeding it indefinitely.
+Two different questions, with two different answers, and it is worth keeping
+them apart: whether the **publisher** still offers an archive, and whether
+**you** still want it on your disk.
+
+### The publisher stopped offering it
+
+A feed can only ever say "here is something new". Nothing about it distinguishes
+"withdrawn" from "further down a list that only carries five items", so absence
+from a feed is not evidence of anything and **`prune` does not apply to an RSS
+subscription at all** — setting it there is quietly inert.
 
 The catalogue API can say "here is everything", which is the only way a
 consumer can notice an absence. `prune` acts on that, and it is deliberately
@@ -289,6 +296,15 @@ OpenStreetMap publishes for the planet dumps:
 it is **1 by default**. That feed lists five planet dumps, so without a cap the
 first poll is roughly four hundred gigabytes. `0` takes everything it lists.
 
+`keepDays` is how long to keep what it brings in. A feed publishing weekly
+leaves a copy behind every week and the publisher goes on listing all of them,
+so without it a subscription grows for ever — which is the case pruning cannot
+help with, because nothing was ever withdrawn. `keep` says the same thing as a
+count. Both are the rules a watched folder and a scheduled source retire under,
+applied by the same code and with the same guards: **only after something new
+has landed**, and **never the newest copy**, so a feed that goes quiet cannot
+empty your disk.
+
 The items are `.osm.pbf`, not map archives, and that is fine: joining an
 existing torrent does not require the content to be anything in particular —
 only *creating* one does. The archive simply is not servable as tiles, and

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.6.0",
+  "version": "0.7.1",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/config.js

@@ -555,6 +555,21 @@ const DEFAULTS = {
    * and pruning, which needs to be able to notice an absence.
    */
   subscriptions: [],
+  /*
+   * A subscription also takes `keep` and `keepDays`, which are the same rules
+   * a watched folder and a scheduled source retire under and are applied by
+   * the same code.
+   *
+   * They answer a different question from `prune`. Pruning is about the
+   * publisher — it stopped offering this, so let it go — and needs a complete
+   * listing to mean anything, which is why it applies to a catalog and not to
+   * an RSS feed. These are about the disk: a feed publishing weekly leaves a
+   * copy behind every week and goes on listing all of them, and age is age
+   * however short the list is.
+   *
+   * Retirement runs only after something new has landed, and never removes
+   * the newest copy.
+   */
   /**
    * How often to re-check whether the sources archives were built from have
    * changed, in seconds. Zero disables it. A check is one HEAD request or stat

package/src/retention.js

@@ -99,6 +99,7 @@ export function expired({ family, keep, keepDays, now = Date.now() }) {
  * @param {number} [options.keep] - How many of the newest to hold.
  * @param {number} [options.keepDays] - How old a build may get, in days.
  * @param {string} options.label - How to name this family in the log.
+ * @param {boolean} [options.requireComplete] - Wait for the newest to be whole.
  * @param {number} [options.now] - The current time, for testing.
  * @returns {Promise<string[]>} - The infohashes removed.
  */
@@ -109,8 +110,23 @@ export async function retire({
   keep,
   keepDays,
   label,
+  requireComplete = false,
   now,
 }) {
+  // Where the new copy is a download rather than a file that already exists.
+  //
+  // A watched folder and a scheduled source hand over an archive that is
+  // whole: the file was there, or the fetch finished. A subscription does not
+  // — it joins a torrent, and the data arrives hours later. Retiring on the
+  // join would delete last week's complete copy the moment this week's was
+  // announced, leaving nothing complete for the length of the download.
+  //
+  // So `keep: 1` here means "the last complete copy", which is the only
+  // reading of it that is safe.
+  if (requireComplete && entry?.complete !== true) {
+    return [];
+  }
+
   const doomed = expired({ family, keep, keepDays, now });
   if (doomed.length === 0) return [];
 

package/src/subscriptions.js

@@ -1,4 +1,5 @@
 import { parseFeed } from './feed.js';
+import { retains, retire } from './retention.js';
 
 /**
  * Follows other nodes' feeds and picks up what they publish.
@@ -174,6 +175,12 @@ export class SubscriptionManager {
         break;
       }
     }
+
+    // The disk side of following a feed. Pruning cannot apply here — absence
+    // from a bounded feed proves nothing — but age applies whatever the list
+    // is, which is what makes this the answer for a feed that publishes
+    // weekly for ever.
+    await this.#retire(subscription);
     return added;
   }
 
@@ -252,9 +259,56 @@ export class SubscriptionManager {
     }
 
     await this.#prune(subscription, document, archives);
+    await this.#retire(subscription);
     return added;
   }
 
+  /**
+   * Removes what this feed's archives have outgrown.
+   *
+   * Separate from pruning, and answering a different question. Pruning is
+   * about the *publisher*: it has stopped offering this, so let it go. This is
+   * about the disk: a feed publishing weekly leaves a copy behind every week,
+   * and the publisher will go on listing all of them.
+   *
+   * That distinction is why an RSS feed can have this and cannot have pruning.
+   * Absence from a bounded feed is not evidence that anything was withdrawn —
+   * planet.openstreetmap.org lists five dumps and says nothing about the
+   * hundreds before them — but age is age however short the list is.
+   *
+   * Only after something new has landed, and never the newest copy, which are
+   * the same guards a scheduled source retires under. See `retire`.
+   * @param {object} subscription - The subscription.
+   * @returns {Promise<void>} - Resolves once anything due has gone.
+   */
+  async #retire(subscription) {
+    if (!retains(subscription)) return;
+
+    // Only what this feed brought in. An archive built here, added by hand, or
+    // taken from another peer is not this subscription's to remove.
+    const family = this.#library.catalog
+      .list()
+      .filter((entry) => entry.source?.subscription === subscription.url);
+    if (family.length === 0) return;
+
+    // Every poll, not only the polls that took something. What is being waited
+    // for is the newest copy *finishing*, which happens hours after the poll
+    // that started it — so retiring only on the way in would either act far
+    // too early or never act at all.
+    await retire({
+      library: this.#library,
+      family,
+      entry: family[0],
+      keep: subscription.keep,
+      keepDays: subscription.keepDays,
+      label: `[sync] ${subscription.url}`,
+      // The newest copy here is a download rather than a file that was already
+      // there, so nothing goes until it is whole. That is what makes `keep: 1`
+      // mean "the last complete copy".
+      requireComplete: true,
+    });
+  }
+
   /**
    * Drops archives this subscription no longer lists.
    *

package/src/web/index.html

@@ -2419,6 +2419,14 @@
           checkNow,
         } = spec;
         rowEditorColumns[key] = columns;
+        // Where this editor's rows belong, and what they are on top of being
+        // rows. Two editors can be two views of one array — feeds and peers
+        // are both subscriptions — so the panel's own key is a DOM name, and
+        // this is where its records are written.
+        rowEditorTargets[key] = {
+          configKey: spec.configKey ?? key,
+          stamp: spec.stamp ?? {},
+        };
         // Kept so a save can put back what this editor never showed. An entry
         // holds more than there are columns for it — a watch folder's
         // pieceLength, a subscription's savePath — and a save that rebuilt each
@@ -2433,7 +2441,9 @@
           <td>
             ${
               column.options
-                ? `<select data-field="${column.field}" style="width:${column.wide ? '17rem' : '9rem'}">
+                ? `<select data-field="${column.field}"${
+                    column.onlyFor ? ` data-only-for="${column.onlyFor}"` : ''
+                  } style="width:${column.wide ? '17rem' : '9rem'}">
                      ${column.options
                        .map(
                          ([option, label]) =>
@@ -2445,6 +2455,7 @@
                    </select>`
                 : `<input
                      data-field="${column.field}"
+                     ${column.onlyFor ? `data-only-for="${column.onlyFor}"` : ''}
                      ${column.secret ? 'type="password" autocomplete="off"' : ''}
                      placeholder="${escapeHtml(column.placeholder ?? '')}"
                      style="width:${column.wide ? '17rem' : '9rem'}"
@@ -2508,6 +2519,40 @@
         const tbody = panel.querySelector('tbody');
         const note = panel.querySelector('[data-role="note"]');
 
+        /**
+         * Greys out the fields the chosen protocol does not read.
+         *
+         * The two halves of a subscription are not the same thing wearing
+         * different clothes. `newest` caps how many items a bounded feed hands
+         * over and means nothing to a catalog, which lists everything; `prune`
+         * needs a complete listing to make absence meaningful and so cannot
+         * apply to a feed. Offering both to both invites a setting that is
+         * quietly ignored, which is worse than not offering it.
+         *
+         * Left alone on "auto", since the protocol is then decided by the URL
+         * and this cannot know which it will be.
+         * @param {HTMLElement} row - The row to update.
+         * @returns {void}
+         */
+        const applyProtocol = (row) => {
+          const chosen = row.querySelector('[data-field="protocol"]')?.value;
+          for (const field of row.querySelectorAll('[data-only-for]')) {
+            const belongs = field.dataset.onlyFor;
+            const off = Boolean(chosen) && chosen !== belongs;
+            field.disabled = off;
+            field.title = off
+              ? `Only used by the ${belongs === 'rss' ? 'RSS' : 'catalog API'} protocol`
+              : '';
+          }
+        };
+
+        for (const row of tbody.querySelectorAll('tr')) applyProtocol(row);
+        panel.addEventListener('change', (event) => {
+          if (event.target.dataset?.field === 'protocol') {
+            applyProtocol(event.target.closest('tr'));
+          }
+        });
+
         panel.onclick = async (event) => {
           const token = event.target.dataset?.token;
           if (token) {
@@ -2662,6 +2707,7 @@
         for (const [key, columns] of Object.entries(specs)) {
           const panel = document.querySelector(`[data-row-editor="${key}"]`);
           if (!panel) continue;
+          const target = rowEditorTargets[key] ?? { configKey: key, stamp: {} };
           const originals = rowEditorRows[key] ?? [];
           const records = [...panel.querySelectorAll('tbody tr')]
             .map((row) => {
@@ -2675,8 +2721,17 @@
                 origin === undefined ? {} : (originals[Number(origin)] ?? {}),
               );
             })
-            .filter((record) => record[columns[0].field]);
-          updates[key] = records;
+            .filter((record) => record[columns[0].field])
+            // What the section says about every row in it. A row in the feeds
+            // table is an RSS subscription because of where it is, which is
+            // the point of having two tables rather than a dropdown.
+            .map((record) => ({ ...record, ...target.stamp }));
+
+          // Appended, not assigned: several editors may write one setting.
+          updates[target.configKey] = [
+            ...(updates[target.configKey] ?? []),
+            ...records,
+          ];
         }
         return updates;
       }
@@ -3168,6 +3223,7 @@
       // ── Settings ──────────────────────────────────────────────────────────
       let restartKeys = new Set();
       let rowEditorColumns = {};
+      let rowEditorTargets = {};
       // What each editor was rendered from, so a save can put back the fields
       // it never showed.
       let rowEditorRows = {};
@@ -3229,6 +3285,7 @@
         // are lists of small records, which is exactly what a torrent client
         // gives a grid for, and what a textarea full of braces is worst at.
         rowEditorColumns = {};
+        rowEditorTargets = {};
         rowEditorRows = {};
         await renderTokenEditor(body);
         renderHookEditor(body, config, restartKeys);
@@ -3460,35 +3517,100 @@
           </div>`;
         body.append(feedGlobals);
 
+        // Which table a subscription belongs in — the same question the
+        // subscription manager asks itself: an explicit protocol decides it, and
+        // otherwise the shape of the URL does.
+        const isPeer = (row) =>
+          row.protocol === 'api' ||
+          (!row.protocol && /\/api\/catalog\/?$/.test(row.url ?? ''));
+        const following = config.subscriptions ?? [];
+
         renderRowEditor({
           into: body,
-          key: 'subscriptions',
-          title: 'Remote nodes',
+          key: 'feeds',
+          configKey: 'subscriptions',
+          // A row here is RSS because of the table it is in, which is the
+          // point of two tables rather than a dropdown nobody reads.
+          stamp: { protocol: 'rss' },
+          title: 'RSS feeds',
           blurb:
-            'Feeds this node follows: another swarm node, or any RSS feed ' +
-            'that carries torrents — planet.openstreetmap.org publishes one ' +
-            'for the planet dumps. An RSS feed says "here is what is new" ' +
-            'and is bounded by the publisher, so a node offline long enough ' +
-            'misses things for good; a catalog URL says "here is ' +
-            'everything", which is what makes reconciling possible. A token, ' +
-            'where the peer issued one, may get you more than it publishes ' +
-            'to the world.',
+            'Any feed that carries torrents — another swarm node publishes ' +
+            'one, and so does planet.openstreetmap.org for the planet dumps. ' +
+            'A feed says "here is what is new" and is bounded by whoever ' +
+            'publishes it, so a node offline long enough misses things for ' +
+            'good, and an absence from one means nothing at all.',
           columns: [
             {
               field: 'url',
-              label: 'Feed or catalog URL',
-              placeholder: 'https://peer.example.org/feed.xml',
+              label: 'Feed URL',
+              placeholder: 'https://planet.openstreetmap.org/pbf/planet-pbf-rss.xml',
               wide: true,
             },
             {
-              field: 'protocol',
-              label: 'Protocol',
+              field: 'mode',
+              label: 'Take as',
               options: [
-                ['', 'auto'],
-                ['rss', 'RSS'],
-                ['api', 'catalog API'],
+                ['cache', 'cache — on demand'],
+                ['mirror', 'mirror — whole copy'],
               ],
             },
+            { field: 'newest', label: 'Items per check', placeholder: '1', number: true },
+            {
+              field: 'categories',
+              label: 'Categories',
+              placeholder: 'from-peer, planet',
+              list: true,
+            },
+            { field: 'filter', label: 'Name filter', placeholder: 'terrain' },
+            { field: 'savePath', label: 'Save to', placeholder: 'the default' },
+            { field: 'token', label: 'Token', placeholder: 'if issued one', secret: true },
+            { field: 'keep', label: 'Copies to keep', placeholder: 'all', number: true },
+            { field: 'keepDays', label: 'Keep for (days)', placeholder: 'for ever', number: true },
+            {
+              field: 'enabled',
+              label: 'On',
+              boolean: true,
+              options: [
+                ['', 'yes'],
+                ['false', 'no'],
+              ],
+            },
+          ],
+          rows: following.filter((row) => !isPeer(row)),
+          checkNow: '/api/subscriptions/refresh',
+          peerPreview: true,
+          footnote:
+            'Items per check is 1 by default and counts from the newest, ' +
+            'because a feed listing five planet dumps is four hundred ' +
+            'gigabytes if you take the lot; 0 takes everything it lists. ' +
+            'Copies to keep and Keep for are about your disk rather than the ' +
+            'publisher: a feed publishing weekly leaves a copy behind every ' +
+            'week and goes on listing all of them. Both run only after ' +
+            'something new has landed, and neither removes the newest copy. ' +
+            'There is no dropping here — absence from a bounded feed is not ' +
+            'evidence that anything was withdrawn, which is what a catalogue ' +
+            'is for.',
+        });
+
+        renderRowEditor({
+          into: body,
+          key: 'peers',
+          configKey: 'subscriptions',
+          stamp: { protocol: 'api' },
+          title: 'Remote nodes',
+          blurb:
+            'Another swarm node, followed through its catalogue rather than ' +
+            'its feed. A catalogue says "here is everything", which is what ' +
+            'makes reconciling possible — and dropping, which needs to be ' +
+            'able to notice an absence. A token, where the peer issued one, ' +
+            'may get you more than it publishes to the world.',
+          columns: [
+            {
+              field: 'url',
+              label: 'Catalogue URL',
+              placeholder: 'https://peer.example.org/api/catalog',
+              wide: true,
+            },
             {
               field: 'mode',
               label: 'Take as',
@@ -3497,19 +3619,14 @@
                 ['mirror', 'mirror — whole copy'],
               ],
             },
-            {
-              field: 'newest',
-              label: 'Items per check',
-              placeholder: '1',
-              number: true,
-            },
             {
               field: 'categories',
               label: 'Categories',
-              placeholder: 'from-peer, planet',
+              placeholder: 'from-peer',
               list: true,
             },
             { field: 'filter', label: 'Name filter', placeholder: 'terrain' },
+            { field: 'savePath', label: 'Save to', placeholder: 'the default' },
             { field: 'token', label: 'Token', placeholder: 'if issued one', secret: true },
             {
               field: 'prune',
@@ -3521,6 +3638,8 @@
                 ['delete', 'forget and delete'],
               ],
             },
+            { field: 'keep', label: 'Copies to keep', placeholder: 'all', number: true },
+            { field: 'keepDays', label: 'Keep for (days)', placeholder: 'for ever', number: true },
             {
               field: 'enabled',
               label: 'On',
@@ -3531,7 +3650,7 @@
               ],
             },
           ],
-          rows: config.subscriptions ?? [],
+          rows: following.filter(isPeer),
           checkNow: '/api/subscriptions/refresh',
           peerPreview: true,
           footnote:
@@ -3540,10 +3659,7 @@
             'every archive the peer lists. Dropping stays off unless chosen, ' +
             'only ever considers archives this peer sent, and never acts on a ' +
             'filtered or partial view — watch a new peer on "report only" ' +
-            'before trusting it with anything more. Items per check is 1 by ' +
-            'default and counts from the newest, because a feed listing five ' +
-            'planet dumps is four hundred gigabytes if you take the lot; 0 ' +
-            'takes everything it lists.',
+            'before trusting it with anything more.',
         });
 
         for (const [key, value] of Object.entries(config)) {