pmtiles-swarm 0.3.2 → 0.4.0

package/CHANGELOG.md

@@ -7,6 +7,62 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.4.0
+### ✨ Features and improvements
+- **A watched folder can set the torrent comment**, which is where attribution and licence belong —
+  it is the one field a torrent carries that says what the thing is, and it reaches anyone who
+  opens the file in any client. The setting was passed through from the start and offered nowhere,
+  so it could only be reached by editing the config by hand.
+- **Feeds can be followed with the controls a torrent client gives them.** A subscription takes
+  one item per check by default, counting from the newest — `newest` raises the cap and `0` lifts
+  it — because a feed like the one OpenStreetMap publishes for the planet dumps lists five of them,
+  and taking the lot is four hundred gigabytes nobody asked for. `enabled: false` switches one feed
+  off without deleting it, `subscriptionsEnabled` switches off all of them at once, and both are in
+  the console alongside the check interval.
+- **Watched folders can retire what they have outgrown**, with the `keep` and `keepDays` that
+  until now existed only on scheduled sources. A folder receiving a daily 137 GB planet build
+  fills any disk within the week, and the alternative was a `find -mtime +35` sweep in the
+  generation script — which deletes the file but leaves this node advertising a torrent for it,
+  so every peer that asks fails. Retirement takes the two together.
+- **`keepDays` retires by age rather than by count**, on both watched folders and scheduled
+  sources, which is what a `find -mtime` sweep actually said. Set alongside `keep` the two are a
+  union: whichever rule says a build has to go, it goes. Neither removes the newest build however
+  old it is — a source that stops publishing would otherwise erase itself, and a last build going
+  stale is a thing to notice rather than a thing to fix by deleting it.
+- **A watched folder can give the newest build a stable name**, with the `latestLink` that
+  until now existed only on scheduled sources — `planetiler-openmaptiles-latest.pmtiles`, the
+  `ln -sfn latest` a generation script used to run. Off unless set. The dated file stays the
+  real one and keeps its own torrent, and the link costs no extra space: a symlink where the
+  platform allows one, a hard link where it does not, since Windows refuses symlinks without
+  elevation or developer mode. The watcher ignores that one name — a hard link is
+  indistinguishable from the file it names, so without that it would be imported as a second
+  archive of bytes already being seeded.
+- **The torrent link for a category can be named whatever reads best.**
+  `/latest/openmaptiles/planetiler-openmaptiles-latest.torrent` is the same route as
+  `/latest/openmaptiles/archive.torrent` — fine in an API, poor in an href on a page. The name
+  is cosmetic: the redirect still ends at the immutable URL, which names the download after the
+  build it actually is, because a URL that could choose that would be a link on your own domain
+  that saves a file called anything at all.
+
+### 🐞 Bug fixes
+- **`/archives/<infohash>/archive.torrent` says it can be cached.** An infohash names those
+  bytes and no others, so the URL can never answer differently — the tile routes have always
+  said so and this one did not, which meant a cache or reverse proxy in front of a node had to
+  re-fetch every download from it. `/latest/<category>/archive.torrent` gets a short one, since
+  it moves on every build.
+- **A `subscriptionIntervalSeconds` of zero no longer polls as fast as the event loop allows.**
+  Zero reads as off everywhere else in the configuration; here it reached `setInterval` unchanged,
+  which is not a stopped timer.
+
+### 📚 Documentation
+- **Following someone else's RSS feed**, using the one OpenStreetMap publishes for the planet
+  dumps as the worked example, including handing what lands to a generation script with
+  `onComplete`.
+- **Piece size and network equipment.** Larger pieces are widely assumed to be gentler on a
+  router and mostly are not: peers request 16 KiB blocks whatever the piece size, so packet
+  volume for the same bytes is identical. Simultaneous connections are what exhaust a NAT table,
+  and `maxConnections` is the setting for that.
+
 ## 0.3.2
 ### ✨ Features and improvements
 - **The console has a footer naming the version it is running**, beside `© <year> TechIdiots LLC`

package/README.md

@@ -614,7 +614,7 @@ matters there is `maxConnections`, since every peer holds a NAT table entry. See
 | `GET` | `/archives/:infoHash/:z/:x/:y.:ext` | One tile — **public** |
 | `GET` | `/archives/:infoHash/archive.torrent` | The `.torrent` a torrent-aware client joins with — **public** |
 | `GET` | `/archives/:infoHash/preview` | Map preview for one archive — admin, not public |
-| `GET` | `/latest/:category/tiles.json`, `/archive.torrent`, `/magnet` | The newest build in a category — **public** |
+| `GET` | `/latest/:category/tiles.json`, `/:name.torrent`, `/magnet` | The newest build in a category — **public**. The torrent name is yours to choose, so a link can read `planetiler-openmaptiles-latest.torrent`; it redirects to the immutable URL, which names the download after the build it actually is |
 | `GET` | `/feed.xml`, `/feed/:category.xml`, `/latest/:category.xml` | RSS — **public** |
 
 Everything under `/api/` is guarded once a credential is configured; tiles, TileJSON and the feeds

package/docs/publishing.md

@@ -377,6 +377,16 @@ A new `.pmtiles` appearing in a watched folder is imported automatically.
 
 Editable in **Settings → Monitored folders** as a table, rather than by hand.
 
+`comment` goes into every torrent this folder produces, and is where the attribution and
+licence belong — it is the one field a torrent carries that says what the thing is, and
+it reaches anyone who opens the file in any client:
+
+```json
+{
+  "comment": "Planetiler OpenMapTiles export. OpenStreetMap contributors, under ODbL 1.0. OpenMapTiles under BSD 3-Clause / CC-BY 4.0"
+}
+```
+
 `stabilitySeconds` is the important one. A map build writes its output over minutes or
 hours, and hashing a half-written archive produces a torrent for bytes that no longer
 exist. Nothing is imported until the file has stopped changing for that long. Raise it if
@@ -453,8 +463,23 @@ today's URL and see whether it exists yet:
     loses that build permanently.
 
 `latestLink`
-    A symlink pointing at the newest build. The dated file stays the real one, so it
-    remains seedable under its own torrent while consumers can reference a fixed path.
+    A stable name pointing at the newest build — `planetiler-protomaps-latest.pmtiles`.
+    The dated file stays the real one, so it remains seedable under its own torrent while
+    a page links to a name that does not change. **Off unless set.**
+
+    A symlink where the platform allows one, and a hard link where it does not: Windows
+    refuses symlinks with `EPERM` unless the process is elevated or the machine is in
+    developer mode, which is not a reasonable thing to require of a daemon. Neither costs
+    extra space — both are another name for the same bytes rather than a copy, which for a
+    137 GB archive is the whole point — and both need the name and the build to be on one
+    filesystem, which a link beside the file it names always is.
+
+    Available on watched folders too, where a generation script writes the builds. There
+    the link lands in the folder being watched, so the watcher is taught to ignore that one
+    name — a hard link is indistinguishable from the file it names, and without that it
+    would be imported as a second archive of bytes already being seeded. It follows that
+    **the name must not collide with a real build's**, and that changing it leaves the old
+    one behind as a file the watcher will then import.
 
 `at`, `everyHours`, `everyMinutes`
     When to look. `at` is a time of day in UTC — `"03:30"`, or a list of them — for an
@@ -507,6 +532,23 @@ today's URL and see whether it exists yet:
     `newest` is above 1, because a poll takes candidates newest first and the *last* build
     imported is then the oldest of them.
 
+`keepDays`
+    The same thing said as a window rather than a count: how many days old a build may
+    get. `keepDays: 35` is the `find -mtime +35` sweep such a script would otherwise run
+    itself, except that this takes the torrent with the data instead of leaving the node
+    advertising an archive that is gone.
+
+    Age is read from the date in the build's name where there is one, and from when this
+    node took it otherwise. A build that can be dated neither way is never removed — a
+    guess is not good enough to delete several hundred gigabytes on.
+
+    **The newest build is never removed, however old it is.** A source that stops
+    publishing would otherwise erase itself, and a last build going stale is a thing to
+    notice rather than a thing to fix by deleting it.
+
+    Set alongside `keep` the two are a union: whichever rule says a build has to go, it
+    goes. `keep: 10, keepDays: 35` holds at most ten builds and at most five weeks.
+
 `seeding`
     A seeding limit for this source's builds, in the same shape as the global one:
     `{ "ratio": 2, "minutes": 4320, "then": "stop" }`. Useful where one source's archives
@@ -520,6 +562,27 @@ what you want — old builds stay seedable for as long as anyone still wants the
 Every candidate URL is checked with a HEAD, so a build that has not been published yet
 costs one request.
 
+### Retention on a watched folder
+
+`keep` and `keepDays` work the same way on a watched folder, where builds arrive from a
+generation script rather than from a URL:
+
+```json
+{
+  "watch": [
+    {
+      "path": "/mnt/store/generated/openmaptiles/pmtiles",
+      "categories": ["openmaptiles", "planet"],
+      "keepDays": 35
+    }
+  ]
+}
+```
+
+The family there is the folder rather than a named source: only archives this same folder
+imported are ever considered, so one dropped into the same directory by hand — or moved
+elsewhere by `publishDir` and belonging to a different folder — is not caught up in it.
+
 ### Watching a directory instead
 
 Where the naming is not predictable enough to write as a template, give an `index` — a

package/docs/subscribing.md

@@ -41,11 +41,14 @@ Its feed is then at `https://maps.example.org/feed.xml`, with per-category feeds
   "subscriptions": [
     { "url": "https://maps.example.org/feed.xml", "mode": "cache" }
   ],
+  "subscriptionsEnabled": true,
   "subscriptionIntervalSeconds": 900
 }
 ```
 
-Poll immediately rather than waiting for the interval:
+`subscriptionsEnabled` is the master switch and `subscriptionIntervalSeconds`
+is how often each feed is checked; zero or less turns polling off. Poll
+immediately rather than waiting for the interval:
 
 ```sh
 curl -X POST localhost:8090/api/subscriptions/refresh
@@ -261,6 +264,61 @@ One feed can serve subscribers with different appetites:
 
 `filter` is a case-insensitive regular expression matched against the item title.
 
+## Following someone else's RSS feed
+
+A subscription does not have to be another swarm node. Any RSS feed with
+`application/x-bittorrent` enclosures works, which includes the one
+OpenStreetMap publishes for the planet dumps:
+
+```json
+{
+  "subscriptions": [
+    {
+      "url": "https://planet.openstreetmap.org/pbf/planet-pbf-rss.xml",
+      "protocol": "rss",
+      "mode": "mirror",
+      "newest": 1,
+      "categories": ["osm-planet"],
+      "savePath": "/mnt/raid0/work/planet"
+    }
+  ]
+}
+```
+
+`newest` is how many items one check may take, counting from the newest, and
+it is **1 by default**. That feed lists five planet dumps, so without a cap the
+first poll is roughly four hundred gigabytes. `0` takes everything it lists.
+
+The items are `.osm.pbf`, not map archives, and that is fine: joining an
+existing torrent does not require the content to be anything in particular —
+only *creating* one does. The archive simply is not servable as tiles, and
+nothing tries.
+
+`enabled: false` switches one feed off without deleting it, and
+`subscriptionsEnabled: false` switches off all of them at once, for when a disk
+is filling or a build has gone wrong.
+
+Nothing is downloaded twice: an item already in the catalog is skipped, so a
+feed that keeps listing last week's dump costs one comparison per check.
+
+### Feeding a generation script
+
+The point of taking a `.osm.pbf` is usually to build something from it. That is
+what `onComplete` is for — it runs when a download finishes, so a planet dump
+landing can start the render that turns it into archives this node publishes:
+
+```json
+{
+  "onComplete": {
+    "command": "/usr/local/bin/planetilerdump-swarm.sh",
+    "args": ["%N", "%F", "%I"]
+  }
+}
+```
+
+Which is the same handover qBittorrent's "run external program on torrent
+finished" provided, with the feed replacing its RSS auto-downloader.
+
 ## Subscribing with plain qBittorrent
 
 The feed is ordinary RSS 2.0 with `application/x-bittorrent` enclosures, which is exactly

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.3.2",
+  "version": "0.4.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -1597,15 +1597,28 @@ export function createApp({
 
   // Redirects rather than serving, so what arrives is the immutable URL and a
   // client that keeps it keeps a specific build rather than a moving target.
-  app.get('/latest/:category/archive.torrent', (req, res) => {
+  // The name in the path is yours to choose: `/latest/openmaptiles/
+  // planetiler-openmaptiles-latest.torrent` is the same route as
+  // `.../archive.torrent`, and reads as what it is where it matters most —
+  // in an href on a page. It does not name the download. That is decided at
+  // the immutable URL this redirects to, which calls the file after the build
+  // it actually is, and a URL that could choose it would be a link on this
+  // domain that saves a file called anything at all.
+  app.get('/latest/:category/:name.torrent', (req, res) => {
     const entry = newestIn(req.params.category, req);
     if (!entry) return res.status(404).json({ error: 'no such category' });
+    // Short, and said explicitly. A 302 is not cacheable unless a response
+    // says so, but "unless it says so" is a thing intermediaries have been
+    // known to disagree about — and this one moves on every build, which is
+    // the whole point of it.
+    res.setHeader('cache-control', 'public, max-age=300');
     res.redirect(302, `${baseUrl(req)}/archives/${entry.infoHash}/archive.torrent`);
   });
 
   app.get('/latest/:category/magnet', (req, res) => {
     const entry = newestIn(req.params.category, req);
     if (!entry) return res.status(404).json({ error: 'no such category' });
+    res.setHeader('cache-control', 'public, max-age=300');
     res.type('text/plain').send(entry.magnet ?? '');
   });
 
@@ -1769,6 +1782,12 @@ export function createApp({
       if (!body) return res.status(404).json({ error: 'torrent file missing' });
       res.setHeader('access-control-allow-origin', '*');
       res.type('application/x-bittorrent');
+      // An infohash names these bytes and no others, so this URL can never
+      // answer differently — the same reason the tile routes say it. Worth
+      // saying out loud where a link to it is published: a cache or a reverse
+      // proxy in front of this then serves the download without touching the
+      // node at all.
+      res.setHeader('cache-control', 'public, max-age=31536000, immutable');
       res.setHeader(
         'content-disposition',
         `attachment; filename="${entry.name}.torrent"`,

package/src/config.js

@@ -475,6 +475,18 @@ const DEFAULTS = {
    *
    * `webSeedBase` on its own assumes the watched folder is already the web
    * root, since nothing is moved.
+   *
+   * `latestLink` gives the newest build a stable second name — a symlink, or
+   * a hard link where the platform refuses one. The watcher ignores that name
+   * so the link is never imported as an archive of its own.
+   *
+   * `keep` and `keepDays` retire what the folder has outgrown — the newest N
+   * builds, or a window in days. A folder receiving a daily 137 GB planet
+   * build fills any disk within the week, and this is the `find -mtime +35`
+   * sweep that would otherwise have to sit in the generation script, except
+   * that it takes the torrent with the data. Both are off unless set, only
+   * ever touch archives this same folder imported, and never remove the
+   * newest build however old it gets.
    */
   watch: [],
   /**
@@ -661,7 +673,15 @@ const DEFAULTS = {
   allowHooksFromApi: false,
   /** How often to look for finished downloads, in seconds. */
   onCompleteCheckIntervalSeconds: 60,
-  /** How often to poll subscribed feeds, in seconds. */
+  /**
+   * Whether to follow feeds at all.
+   *
+   * The master switch, separate from the per-feed one: turning it off stops
+   * every feed without editing any of them, which is what you want when a
+   * disk is filling or a run has gone wrong.
+   */
+  subscriptionsEnabled: true,
+  /** How often to poll subscribed feeds, in seconds. Zero or less is off. */
   subscriptionIntervalSeconds: 900,
   /** Republish interval for BEP 46 records, in seconds. DHT items expire. */
   republishIntervalSeconds: 3600,
@@ -886,6 +906,7 @@ export const RELOADABLE = new Map([
   ['sourceCheckIntervalHours', 'sources'],
   ['subscriptions', 'subscriptions'],
   ['subscriptionIntervalSeconds', 'subscriptions'],
+  ['subscriptionsEnabled', 'subscriptions'],
   ['seeding', 'seeding'],
   // Applied to a running session, so a schedule can be corrected at the moment
   // it turns out to be wrong rather than at the next convenient restart.

package/src/latest-link.js

@@ -0,0 +1,72 @@
+import fs from 'node:fs/promises';
+import path from 'node:path';
+
+/**
+ * Points a stable name at the newest build.
+ *
+ * The dated file stays the real one, so it remains seedable under its own
+ * torrent while a page links to a name that does not change — which is what an
+ * `ln -sfn latest` in a generation script was for.
+ *
+ * A symlink first, then a hard link. **Windows refuses symlinks with EPERM**
+ * unless the process is elevated or the machine is in developer mode, which is
+ * not a reasonable thing to require of a daemon — and a hard link needs
+ * neither. It costs no extra space either way, since both are another name for
+ * the same bytes rather than a copy, and for a 137 GB archive that distinction
+ * is the whole point. Both only work within one filesystem, which is where a
+ * link beside the file it names always is.
+ *
+ * The two are not quite equivalent, and the difference shows when the build
+ * this names is eventually deleted: a symlink is left dangling, while a hard
+ * link keeps the bytes alive until it too is gone. Retention never removes the
+ * newest build, so neither case arises from this node's own housekeeping.
+ * @param {object} options - What to link and how to say so.
+ * @param {string} options.target - The build the name should resolve to.
+ * @param {string} options.name - The stable name, absolute or beside the target.
+ * @param {string} options.label - How to name the caller in the log.
+ * @returns {Promise<string|undefined>} - The link made, or undefined on failure.
+ */
+export async function linkLatest({ target, name, label }) {
+  const link = linkPathFor(target, name);
+
+  const attempts = [
+    // The type is autodetected from the target on Windows and ignored
+    // everywhere else, which is right: the target is always a file here.
+    ['symlink', () => fs.symlink(target, link)],
+    ['hard link', () => fs.link(target, link)],
+  ];
+
+  for (const [kind, make] of attempts) {
+    try {
+      // The previous build's link, which is the usual case — this runs once
+      // per build and the name by definition already exists after the first.
+      await fs.rm(link, { force: true });
+      await make();
+      console.log(
+        `${label} latest -> ${path.basename(target)}` +
+          (kind === 'symlink' ? '' : ` (${kind})`),
+      );
+      return link;
+    } catch (error) {
+      // Try the next kind rather than giving up on the first refusal; only the
+      // last one is worth reporting.
+      if (make === attempts.at(-1)[1]) {
+        console.warn(
+          `${label} could not point ${path.basename(link)} at ` +
+            `${path.basename(target)}: ${error.message}`,
+        );
+      }
+    }
+  }
+  return undefined;
+}
+
+/**
+ * Where a stable name resolves to, absolute or beside the build it names.
+ * @param {string} target - The build being named.
+ * @param {string} name - The configured name.
+ * @returns {string} - The link's path.
+ */
+export function linkPathFor(target, name) {
+  return path.isAbsolute(name) ? name : path.join(path.dirname(target), name);
+}

package/src/library.js

@@ -197,7 +197,11 @@ export class Library {
 
     return this.#register(created, {
       categories: options.categories ?? options.category,
-      source: { type: 'file', location: absolute },
+      // `watch` names the folder that imported this, where one did. Not the
+      // same thing as the directory it sits in: with publishDir it has already
+      // moved somewhere else, and an archive dropped into a watched folder by
+      // hand is still that folder's to retire.
+      source: { type: 'file', location: absolute, watch: options.watch },
       // The torrent names the file, so the save path is its parent directory.
       savePath: path.dirname(absolute),
       pmtiles: summary,

package/src/retention.js

@@ -0,0 +1,144 @@
+const DAY_MS = 24 * 60 * 60 * 1000;
+
+/**
+ * When an archive was built, as a timestamp.
+ *
+ * The date in the name where there is one, since that is what a dated build
+ * means by "old" — falling back to when this node took it, which for a folder
+ * receiving builds as they are made is close enough to the same thing.
+ * @param {object} entry - A catalog entry.
+ * @returns {number|undefined} - Milliseconds, or undefined if it cannot be told.
+ */
+function builtAt(entry) {
+  const parsed = Date.parse(entry.buildDate ?? entry.createdAt ?? '');
+  return Number.isNaN(parsed) ? undefined : parsed;
+}
+
+/**
+ * Whether any retention rule is set at all.
+ *
+ * Worth asking before gathering the family, because gathering it means walking
+ * the whole catalog — and retention is off unless somebody turned it on, so
+ * that walk is usually for nothing.
+ * @param {object} rules - Something carrying keep and keepDays.
+ * @returns {boolean} - True if either rule would do something.
+ */
+export function retains({ keep, keepDays } = {}) {
+  return Number(keep) >= 1 || Number(keepDays) >= 1;
+}
+
+/**
+ * Which of a family of builds have outlived the rules set for them.
+ *
+ * Pure, so the decision to delete several hundred gigabytes can be tested
+ * without any of it existing.
+ * @param {object} options - The family and the rules.
+ * @param {object[]} options.family - Builds, newest first.
+ * @param {number} [options.keep] - How many of the newest to hold.
+ * @param {number} [options.keepDays] - How old a build may get, in days.
+ * @param {number} [options.now] - The current time, for testing.
+ * @returns {object[]} - The entries to retire, in the family's order.
+ */
+export function expired({ family, keep, keepDays, now = Date.now() }) {
+  const doomed = new Set();
+
+  const count = Number(keep);
+  if (Number.isFinite(count) && count >= 1) {
+    for (const entry of family.slice(count)) doomed.add(entry);
+  }
+
+  const days = Number(keepDays);
+  if (Number.isFinite(days) && days >= 1) {
+    const cutoff = now - days * DAY_MS;
+    // Never the newest, however old it is. A folder that stops receiving
+    // builds would otherwise empty itself and leave the node serving nothing —
+    // and "the last build is stale" is a thing to notice, not a thing to fix
+    // by deleting it. `find -mtime +35` had no such qualm.
+    for (const entry of family.slice(1)) {
+      const at = builtAt(entry);
+      // No date at all means no way to tell how old it is, and a guess is not
+      // good enough to delete on.
+      if (at !== undefined && at < cutoff) doomed.add(entry);
+    }
+  }
+
+  return family.filter((entry) => doomed.has(entry));
+}
+
+/**
+ * Retires the builds a family has outgrown, and removes their data.
+ *
+ * Deliberately narrow, because this deletes data:
+ *
+ *   * Off unless `keep` or `keepDays` is set. Silence has to mean "keep
+ *     everything", since the alternative is deleting archives nobody asked to
+ *     lose.
+ *   * Only the family it is given — whatever the caller can prove came from
+ *     the same source or folder. An archive added by hand, adopted from a
+ *     client, or taken from a peer is never touched, even in the same
+ *     directory.
+ *   * Never the newest build, and never the one just imported.
+ *
+ * Nothing is deleted until the new build is the one being served. A category's
+ * feed and its `/latest/<category>/tiles.json` resolve to the newest archive in
+ * it, and that is what consumers point at; once they resolve to this build, the
+ * ones it replaced are no longer where anyone is being sent. Before that —
+ * while an older build is still the answer — deleting it would break the very
+ * URL the feed is advertising. It also covers the case that makes this
+ * necessary at all: an import run that takes several builds takes them newest
+ * first, so an *older* one can be the most recent import. It has superseded
+ * nothing and must retire nothing.
+ *
+ * The torrent goes with the data. Leaving a catalog entry whose file is gone
+ * would leave the node advertising an archive it cannot serve, and every peer
+ * that asked would fail.
+ * @param {object} options - What to retire and how to say so.
+ * @param {object} options.library - The library, for removal.
+ * @param {object[]} options.family - Builds from one source, newest first.
+ * @param {object} options.entry - The build just imported.
+ * @param {number} [options.keep] - How many of the newest to hold.
+ * @param {number} [options.keepDays] - How old a build may get, in days.
+ * @param {string} options.label - How to name this family in the log.
+ * @param {number} [options.now] - The current time, for testing.
+ * @returns {Promise<string[]>} - The infohashes removed.
+ */
+export async function retire({
+  library,
+  family,
+  entry,
+  keep,
+  keepDays,
+  label,
+  now,
+}) {
+  const doomed = expired({ family, keep, keepDays, now });
+  if (doomed.length === 0) return [];
+
+  if (family[0]?.infoHash !== entry.infoHash) {
+    console.log(
+      `${label}: ${entry.name} is not the newest build here, so nothing is retired`,
+    );
+    return [];
+  }
+
+  const held = [
+    Number(keep) >= 1 ? `the newest ${Number(keep)}` : undefined,
+    Number(keepDays) >= 1 ? `${Number(keepDays)} days` : undefined,
+  ]
+    .filter(Boolean)
+    .join(' and ');
+
+  const removed = [];
+  for (const old of doomed) {
+    try {
+      await library.remove(old.infoHash, { deleteData: true });
+      removed.push(old.infoHash);
+      console.log(`${label}: retired ${old.name} — keeping ${held}`);
+    } catch (error) {
+      // Worth saying rather than swallowing: the disk this exists to protect
+      // is now not being protected.
+      console.error(`${label}: could not retire ${old.name}: ${error.message}`);
+    }
+  }
+  return removed;
+}

package/src/sources.js

@@ -28,6 +28,8 @@
 
 import fs from 'node:fs/promises';
 import path from 'node:path';
+import { linkLatest } from './latest-link.js';
+import { retains, retire } from './retention.js';
 
 /**
  * Expands date placeholders in a template.
@@ -622,79 +624,37 @@ export class ScheduledSourceManager {
   }
 
   /**
-   * Removes older builds from the same source, where `keep` says to.
+   * Removes older builds from the same source, where retention says to.
    *
    * A daily planet build is 137 GB. Kept for ever, a source like that fills
    * any disk within the week — and the older ones are rarely what anyone
    * wants, because the whole point of a dated build is that a newer one
-   * replaces it. `keep: 1` means only the newest is held.
+   * replaces it. `keep: 1` holds only the newest; `keepDays: 35` holds five
+   * weeks of them.
    *
-   * Deliberately narrow, because this deletes data:
-   *
-   *   * Only archives this same named source imported. An archive added by
-   *     hand, adopted from a client, or taken from a peer is never touched,
-   *     even if it sits in the same directory.
-   *   * Never the one just imported, and never more than the count allows.
-   *   * Off unless `keep` is set. Silence has to mean "keep everything",
-   *     since the alternative is deleting archives nobody asked to lose.
-   *
-   * The torrent goes with the data. Leaving a catalog entry whose file is
-   * gone would leave the node advertising an archive it cannot serve, and
-   * every peer that asked would fail.
+   * The family is the archives this same *named* source imported, and nothing
+   * else — one added by hand, adopted from a client, or taken from a peer is
+   * never touched, even in the same directory. See `retire` for the rest of
+   * what this will not do.
    * @param {object} source - The source definition.
    * @param {object} entry - The build just imported.
    * @returns {Promise<string[]>} - The infohashes removed.
    */
   async #retire(source, entry) {
-    const keep = Number(source.keep);
-    if (!Number.isFinite(keep) || keep < 1) return [];
     if (!source.name) return [];
-
-    // The catalog's own order, which is exactly what `/latest` follows.
-    const family = this.#catalog
-      .list()
-      .filter((candidate) => candidate.source?.name === source.name);
-
-    // Nothing is deleted until the new build is the one being served.
-    //
-    // A category's feed and its `/latest/<category>/tiles.json` resolve to the
-    // newest archive in it, and that is what consumers point at. Once they
-    // resolve to this build, the ones it replaced are no longer where anyone
-    // is being sent, and retiring them costs nobody a fetch they were about to
-    // make. Before that — while an older build is still the answer — deleting
-    // it would break the very URL the feed is advertising.
-    //
-    // It also covers the case that makes this necessary at all: a poll taking
-    // several builds takes them newest first, so an *older* one can be the
-    // most recent import. It has superseded nothing and must retire nothing.
-    if (family[0]?.infoHash !== entry.infoHash) {
-      console.log(
-        `[source] ${source.name}: ${entry.name} is not the newest build here, ` +
-          'so nothing is retired',
-      );
-      return [];
-    }
-
-    const doomed = family.slice(keep);
-    const removed = [];
-
-    for (const old of doomed) {
-      try {
-        await this.#library.remove(old.infoHash, { deleteData: true });
-        removed.push(old.infoHash);
-        console.log(
-          `[source] ${source.name}: retired ${old.name} — keeping the newest ` +
-            `${keep}`,
-        );
-      } catch (error) {
-        // Worth saying rather than swallowing: the disk this exists to protect
-        // is now not being protected.
-        console.error(
-          `[source] ${source.name}: could not retire ${old.name}: ${error.message}`,
-        );
-      }
-    }
-    return removed;
+    if (!retains(source)) return [];
+
+    return retire({
+      library: this.#library,
+      // The catalog's own order, which is exactly what `/latest` follows.
+      family: this.#catalog
+        .list()
+        .filter((candidate) => candidate.source?.name === source.name),
+      entry,
+      keep: source.keep,
+      keepDays: source.keepDays,
+      label: `[source] ${source.name}`,
+    });
   }
 
   /**
@@ -703,48 +663,16 @@ export class ScheduledSourceManager {
    * The dated file stays the real one either way, so it remains seedable under
    * its own torrent while consumers reference a fixed path.
    *
-   * A symlink first, then a hard link. Windows refuses symlinks with EPERM
-   * unless the process is elevated or the machine is in developer mode, which
-   * is not a reasonable thing to require of a daemon — and a hard link needs
-   * neither. It costs no extra space, since it is another name for the same
-   * bytes rather than a copy, and for a 137 GB archive that distinction is the
-   * whole point. It only works within one filesystem, which is where a link
-   * beside the file it names always is.
    * @param {object} source - The source definition.
    * @param {object} entry - The freshly imported entry.
    * @returns {Promise<void>} - Resolves once linked, or logs and continues.
    */
   async #linkLatest(source, entry) {
-    const target = entry.retainedAt ?? path.join(entry.savePath, entry.name);
-    const link = path.isAbsolute(source.latestLink)
-      ? source.latestLink
-      : path.join(path.dirname(target), source.latestLink);
-
-    const attempts = [
-      ['symlink', () => fs.symlink(target, link)],
-      ['hard link', () => fs.link(target, link)],
-    ];
-
-    for (const [kind, make] of attempts) {
-      try {
-        await fs.rm(link, { force: true });
-        await make();
-        console.log(
-          `[source] latest -> ${path.basename(target)}` +
-            (kind === 'symlink' ? '' : ` (${kind})`),
-        );
-        return;
-      } catch (error) {
-        // Try the next kind rather than giving up on the first refusal; only
-        // the last one is worth reporting.
-        if (make === attempts.at(-1)[1]) {
-          console.warn(
-            `[source] could not point ${path.basename(link)} at ` +
-              `${path.basename(target)}: ${error.message}`,
-          );
-        }
-      }
-    }
+    await linkLatest({
+      target: entry.retainedAt ?? path.join(entry.savePath, entry.name),
+      name: source.latestLink,
+      label: '[source]',
+    });
   }
 
   /**

package/src/subscriptions.js

@@ -39,10 +39,23 @@ export class SubscriptionManager {
   start() {
     const feeds = this.#config.subscriptions ?? [];
 
+    if (this.#config.subscriptionsEnabled === false) {
+      console.log('[feed] following is switched off');
+      return;
+    }
+
     // The timer runs even with nothing to follow. Every refresh reads the list
     // fresh, so this is what lets a peer added through the console start
     // working without a restart; an empty pass costs nothing.
-    const intervalMs = (this.#config.subscriptionIntervalSeconds ?? 900) * 1000;
+    const seconds = this.#config.subscriptionIntervalSeconds ?? 900;
+    // Zero reads as off everywhere else in the configuration, and it has to
+    // read as off here too: setInterval(fn, 0) is not a stopped timer, it is
+    // one that fires as fast as the loop will let it.
+    if (seconds <= 0) {
+      console.log('[feed] interval is zero; not polling');
+      return;
+    }
+    const intervalMs = seconds * 1000;
     // Poll once at startup, then on the interval.
     this.refresh().catch((error) =>
       console.error(`[feed] initial refresh failed: ${error.message}`),
@@ -67,7 +80,12 @@ export class SubscriptionManager {
    */
   async refresh() {
     const added = [];
+    // Checked here as well as in start(), so the switch means the same thing
+    // to a refresh asked for through the API as it does to the timer.
+    if (this.#config.subscriptionsEnabled === false) return added;
     for (const subscription of this.#config.subscriptions ?? []) {
+      // Off, but kept — what qBittorrent's per-feed checkbox does.
+      if (subscription.enabled === false) continue;
       try {
         added.push(...(await this.#poll(subscription)));
       } catch (error) {
@@ -103,7 +121,20 @@ export class SubscriptionManager {
     const items = parseFeed(await response.text());
     const added = [];
 
+    // How many items one poll may take.
+    //
+    // One by default, because a feed's items are whole files and some of them
+    // are enormous: planet.openstreetmap.org lists five planet dumps, and
+    // taking the lot on the first poll is four hundred gigabytes nobody asked
+    // for. `newest: 0` lifts the cap for feeds where that is what you want.
+    //
+    // Items arrive newest first, so the cap keeps the newest.
+    const limit = subscription.newest === 0
+      ? Number.POSITIVE_INFINITY
+      : Math.max(1, subscription.newest ?? 1);
+
     for (const item of items) {
+      if (added.length >= limit) break;
       // A regex filter lets one feed serve several consumers with different
       // appetites, e.g. only taking Europe extracts.
       if (subscription.filter && !new RegExp(subscription.filter, 'i').test(item.title)) {

package/src/watch.js

@@ -1,5 +1,8 @@
+import path from 'node:path';
 import chokidar from 'chokidar';
 import { normalizeCategories } from './catalog.js';
+import { linkLatest, linkPathFor } from './latest-link.js';
+import { retains, retire } from './retention.js';
 
 /**
  * Watches folders for new PMTiles archives and imports them automatically.
@@ -26,7 +29,7 @@ export class WatchManager {
 
   /**
    * Starts watching the configured folders.
-   * @param {object[]} folders - Entries of {path, categories, webSeedBase, publishDir, sparse, trackers, addTrackers, stabilitySeconds, pollSeconds}.
+   * @param {object[]} folders - Entries of {path, categories, webSeedBase, publishDir, sparse, trackers, addTrackers, stabilitySeconds, pollSeconds, keep, keepDays, latestLink}.
    * @returns {void}
    */
   start(folders = []) {
@@ -62,6 +65,12 @@ export class WatchManager {
 
       watcher.on('add', (file) => {
         if (!/\.pmtiles$/i.test(file)) return;
+        // The folder's own `latestLink`, which is a .pmtiles in a watched
+        // folder like any other and would otherwise be imported as a second
+        // archive — a whole extra torrent for the same bytes under a name that
+        // changes every build. A hard link is indistinguishable from the file
+        // it names, so the name is the only thing that can tell them apart.
+        if (this.#isLatestLink(file, folder)) return;
         this.#import(file, folder);
       });
       watcher.on('error', (error) => {
@@ -78,6 +87,21 @@ export class WatchManager {
     }
   }
 
+  /**
+   * Whether a file is this folder's own "latest" name.
+   * @param {string} file - The file that appeared.
+   * @param {object} folder - The watch-folder configuration.
+   * @returns {boolean} - True when it is the link, not a build.
+   */
+  #isLatestLink(file, folder) {
+    if (!folder.latestLink) return false;
+    // Compared as a path rather than a basename, so an absolute latestLink
+    // pointing somewhere else entirely does not silently exclude a real build
+    // that happens to share its name.
+    const target = path.join(folder.path, 'any.pmtiles');
+    return path.resolve(file) === path.resolve(linkPathFor(target, folder.latestLink));
+  }
+
   /**
    * Imports one archive, guarding against overlapping imports of the same file.
    * @param {string} file - Path to the archive.
@@ -110,8 +134,37 @@ export class WatchManager {
         // not on the node.
         pieceLength: folder.pieceLength,
         comment: folder.comment,
+        // Marks this as the folder's, so retention below has a family to work
+        // within and nothing outside it can be caught up in one.
+        watch: folder.path,
       });
       console.log(`[watch] imported ${entry.name} (${entry.infoHash})`);
+
+      // Before retirement, so the stable name is already pointing at the new
+      // build by the time anything older is considered for removal.
+      if (folder.latestLink) {
+        await linkLatest({
+          target: entry.retainedAt ?? path.join(entry.savePath, entry.name),
+          name: folder.latestLink,
+          label: `[watch] ${folder.path}`,
+        });
+      }
+
+      // A folder receiving a daily planet build fills any disk within the
+      // week. This is the `find -mtime +35` sweep that used to sit in the
+      // generation script, except that it takes the torrent with the data
+      // rather than leaving the node advertising an archive that is gone.
+      if (!retains(folder)) return;
+      await retire({
+        library: this.#library,
+        family: this.#library.catalog
+          .list()
+          .filter((candidate) => candidate.source?.watch === folder.path),
+        entry,
+        keep: folder.keep,
+        keepDays: folder.keepDays,
+        label: `[watch] ${folder.path}`,
+      });
     } catch (error) {
       console.error(`[watch] failed to import ${file}: ${error.message}`);
     } finally {

package/src/web/index.html

@@ -3124,9 +3124,46 @@
             },
             { field: 'publishDir', label: 'Publish to', placeholder: '/var/www/pmtiles' },
             { field: 'webSeedBase', label: 'Web seed base', placeholder: 'https://…/files' },
+            {
+              field: 'comment',
+              label: 'Torrent comment',
+              placeholder: 'attribution and licence',
+              wide: true,
+            },
             { field: 'pollSeconds', label: 'Poll every (s)', placeholder: 'events', number: true },
+            {
+              field: 'latestLink',
+              label: 'Stable name',
+              placeholder: 'off',
+            },
+            {
+              field: 'keep',
+              label: 'Builds to keep',
+              placeholder: 'all',
+              number: true,
+            },
+            {
+              field: 'keepDays',
+              label: 'Keep for (days)',
+              placeholder: 'for ever',
+              number: true,
+            },
           ],
           rows: config.watch ?? [],
+          footnote:
+            'Stable name gives the newest build a second name that does not ' +
+            'change — planet-latest.pmtiles — so a page can link to it. The ' +
+            'dated file stays the real one and keeps its own torrent; the ' +
+            'link costs no extra space, and is a symlink or a hard link ' +
+            'depending on what the platform allows. Do not give it a name a ' +
+            'real build could have. ' +
+            'Builds to keep and Keep for are both off unless set, and both ' +
+            'delete the archive as well as forgetting it — a catalog entry ' +
+            'whose file is gone leaves this node advertising something it ' +
+            'cannot serve. They only ever touch archives this same folder ' +
+            'imported, never the newest one however old it is, and nothing at ' +
+            'all until the new build is the one the feed resolves to. Set ' +
+            'both and whichever says a build has to go wins.',
         });
 
         renderRowEditor({
@@ -3193,6 +3230,12 @@
               placeholder: 'all',
               number: true,
             },
+            {
+              field: 'keepDays',
+              label: 'Keep for (days)',
+              placeholder: 'for ever',
+              number: true,
+            },
           ],
           rows: config.sources ?? [],
           preview: true,
@@ -3223,17 +3266,50 @@
             'archives this same source imported.',
         });
 
+        const feedGlobals = document.createElement('div');
+        feedGlobals.className = 'panel';
+        feedGlobals.style.marginBottom = '1rem';
+        feedGlobals.innerHTML = `
+          <h2 style="margin-top:0">Feeds</h2>
+          <div class="sub" style="margin-bottom:0.7rem">
+            Applies to every feed below.
+          </div>
+          <div class="field">
+            <label for="cfg-subscriptionsEnabled">Follow feeds</label>
+            <select id="cfg-subscriptionsEnabled" data-key="subscriptionsEnabled" data-scalar="1">
+              <option value="true"${config.subscriptionsEnabled === false ? '' : ' selected'}>on</option>
+              <option value="false"${config.subscriptionsEnabled === false ? ' selected' : ''}>off</option>
+            </select>
+            <div class="sub">
+              Off stops all of them without editing any, for when a disk is
+              filling or a build has gone wrong.
+            </div>
+          </div>
+          <div class="field">
+            <label for="cfg-subscriptionIntervalSeconds">Check every (seconds)</label>
+            <input id="cfg-subscriptionIntervalSeconds" data-key="subscriptionIntervalSeconds"
+                   data-scalar="1" style="width:9rem"
+                   value="${escapeHtml(config.subscriptionIntervalSeconds ?? 900)}" />
+            <div class="sub">
+              A poll is one request per feed. Publishers post on their own
+              schedule, so checking faster than they publish finds nothing.
+            </div>
+          </div>`;
+        body.append(feedGlobals);
+
         renderRowEditor({
           into: body,
           key: 'subscriptions',
           title: 'Remote nodes',
           blurb:
-            'Peers whose archives this node follows. An RSS feed says "here ' +
-            'is what is new" and is bounded by the publisher, so a node ' +
-            'offline long enough misses things for good; a catalog URL says ' +
-            '"here is everything", which is what makes reconciling possible. ' +
-            'A token, where the peer issued one, may get you more than it ' +
-            'publishes to the world.',
+            'Feeds this node follows: another swarm node, or any RSS feed ' +
+            'that carries torrents — planet.openstreetmap.org publishes one ' +
+            'for the planet dumps. An RSS feed says "here is what is new" ' +
+            'and is bounded by the publisher, so a node offline long enough ' +
+            'misses things for good; a catalog URL says "here is ' +
+            'everything", which is what makes reconciling possible. A token, ' +
+            'where the peer issued one, may get you more than it publishes ' +
+            'to the world.',
           columns: [
             {
               field: 'url',
@@ -3258,6 +3334,12 @@
                 ['mirror', 'mirror — whole copy'],
               ],
             },
+            {
+              field: 'newest',
+              label: 'Items per check',
+              placeholder: '1',
+              number: true,
+            },
             { field: 'category', label: 'Tag as', placeholder: 'from-peer' },
             { field: 'filter', label: 'Name filter', placeholder: 'terrain' },
             { field: 'token', label: 'Token', placeholder: 'if issued one', secret: true },
@@ -3271,6 +3353,15 @@
                 ['delete', 'forget and delete'],
               ],
             },
+            {
+              field: 'enabled',
+              label: 'On',
+              boolean: true,
+              options: [
+                ['', 'yes'],
+                ['false', 'no'],
+              ],
+            },
           ],
           rows: config.subscriptions ?? [],
           peerPreview: true,
@@ -3280,7 +3371,10 @@
             'every archive the peer lists. Dropping stays off unless chosen, ' +
             'only ever considers archives this peer sent, and never acts on a ' +
             'filtered or partial view — watch a new peer on "report only" ' +
-            'before trusting it with anything more.',
+            'before trusting it with anything more. Items per check is 1 by ' +
+            'default and counts from the newest, because a feed listing five ' +
+            'planet dumps is four hundred gigabytes if you take the lot; 0 ' +
+            'takes everything it lists.',
         });
 
         for (const [key, value] of Object.entries(config)) {
@@ -3289,6 +3383,8 @@
           // Shown as real fields above rather than as raw JSON.
           if (key === 'seeding' || key === 'speed') continue;
           if (key === 'watch' || key === 'sources' || key === 'subscriptions') continue;
+          if (key === 'subscriptionsEnabled') continue;
+          if (key === 'subscriptionIntervalSeconds') continue;
           if (key === 'onAdded' || key === 'onComplete') continue;
           const needsRestart = restartKeys.has(key);
           const field = document.createElement('div');

package/swarm.config.json.sample

@@ -71,8 +71,18 @@
     }
   ],
 
+  "subscriptionsEnabled": true,
+  "subscriptionIntervalSeconds": 900,
   "subscriptions": [
-    { "url": "https://peer.example.org/api/catalog", "protocol": "api", "mode": "cache" }
+    { "url": "https://peer.example.org/api/catalog", "protocol": "api", "mode": "cache" },
+    {
+      "url": "https://feed.example.org/pbf/planet-pbf-rss.xml",
+      "protocol": "rss",
+      "mode": "mirror",
+      "newest": 1,
+      "categories": ["osm-planet"],
+      "enabled": false
+    }
   ],
 
   "tiles": { "sparse": null },