pmtiles-swarm 0.95.0 → 0.97.0

package/CHANGELOG.md

@@ -2,6 +2,68 @@
 
 ## master
 ### ✨ Features and improvements
+- _...Add new stuff here..._
+
+### 🐞 Bug fixes
+- _...Add new stuff here..._
+
+## 0.97.0
+### ✨ Features and improvements
+- **A stack can be told to forget what it has merged, and forgets it by itself when its recipe
+  changes.** Editing a stack never served a stale tile — the cache key covers the recipe's revision,
+  so the old tiles simply stop being asked for — but they stayed on the disk, spending the budget
+  until eviction happened to reach them. Under a stack nobody is actively panning, that is a long
+  time, and a node whose stacks are edited often could end up holding mostly tiles no request can
+  reach.
+
+  Merged tiles now carry the stack they belong to in their filename, after the digest so the
+  sharding still spreads and in the name so it survives a restart. `StackStore` compares revisions
+  on every reload and says which recipes are no longer what they were; the node clears those. It
+  covers every way a recipe moves — the console's Save, `PUT /api/stacks/<id>`, an import, a delete,
+  a stack feed, and an operator editing `stacks.json` by hand — and announces nothing when a
+  rewrite leaves the recipes the same, so a restart does not read as an edit and throw away the last
+  run's work.
+
+  Alongside it, a **Clear cache** button on each stack in the console, showing what it would free,
+  and `DELETE /api/stacks/<id>/cache` behind it. That is for what a revision cannot see: an archive
+  rewritten under an address that did not change, a cutline redrawn, a codec upgraded, or simply
+  wanting the disk back now. `/api/stacks` reports `cache: {entries, bytes}` per stack.
+
+  Both take the stacks **nesting** the cleared one with them, however many levels up. An outer
+  stack's tiles were merged from the inner one's, so clearing one level and not the other leaves the
+  older answer being served from above. Tiles written by an earlier version have no stack in their
+  name; they are evicted as before and `DELETE /api/storage/merged-tiles` still empties everything.
+  See [tile-stacks.md](docs/tile-stacks.md) — "Clearing what a stack has merged".
+
+### 🐞 Bug fixes
+- **A directory the configuration named but nobody had made stopped the unit dead.**
+  `status=226/NAMESPACE`, six milliseconds of CPU, and nothing in the journal — because the program
+  never ran: with `ProtectSystem=strict`, systemd builds the mount namespace first, and a
+  `ReadWritePaths=` entry that does not exist makes that fail. Every path is now prefixed with `-`,
+  systemd's "ignore this if it does not exist", so the node starts and a write that has nowhere to
+  go fails where it is attempted, naming the path. A missing directory should be a message, not a
+  silence.
+
+  `init --systemd` also lists the directories the configuration names that are not there yet, with
+  the `install -d` line for each. See [running-as-a-service.md](docs/running-as-a-service.md) —
+  "status=226/NAMESPACE".
+
+## 0.96.0
+### ✨ Features and improvements
+- **The swarm-read limit is memory now, not a count.** `tiles.maxOpenSwarmArchives: 16` becomes
+  `tiles.swarmCacheBytes: 1 GiB`, because a count was the wrong unit for it. What is expensive about
+  a cache-mode reader is its piece cache, which is **RAM** — a map of whole pieces in the node's own
+  heap — and how big it is depends on the torrent: `max(64 MiB, 8 × pieceLength)`. Sixteen readers
+  is a gigabyte against the 4 MiB pieces this project creates and two against the 16 MiB pieces a
+  planet torrent usually has, which is not a limit anybody chose.
+
+  Each reader is asked what its cache costs rather than assumed, so the count now follows from the
+  memory: halve `pieceCacheBytes` and twice as many archives stay open for the same gigabyte. The
+  last reader is never closed, whatever the budget says — the archive just opened is the one being
+  read. `maxOpenSwarmArchives` remains for a node that wants a hard count as well; unset by default.
+
+  Complete archives are unaffected: they hold a file descriptor and no piece cache, and are bounded
+  by `maxOpenArchives`.
 - **`init --systemd` regenerates the unit without touching the configuration.** The unit is derived
   from the configuration and from how many archives the library holds, and both move — a watched
   folder added later belongs in `ReadWritePaths`, and a grown library needs longer to write its
@@ -12,7 +74,11 @@
   and a path added to the installed unit by hand will not be in the new one.
 
 ### 🐞 Bug fixes
-- _...Add new stuff here..._
+- **The unit's `ReadWritePaths` left out where an export writes.** `stackExports[].savePath` and
+  `.publishDir`, and a watched folder's `publishDir`, were not among the directories derived from
+  the configuration — so a nightly bake to a directory named nowhere else ran for an hour and was
+  refused the write at the end, with that directory's permissions perfect. Which is the exact
+  failure the derived list exists to prevent. Re-run `init --systemd` to pick them up.
 
 ## 0.95.0
 ### ✨ Features and improvements

package/docs/configuration.md

@@ -727,7 +727,8 @@ what lets a machine with 10 GiB free serve a 700 GiB planet. See
 | setting                       | default  |                                                              |
 | ----------------------------- | -------- | ------------------------------------------------------------ |
 | `tiles.maxOpenArchives`       | `128`    | complete archives kept open; each is a file descriptor       |
-| `tiles.maxOpenSwarmArchives`  | `16`     | cache-mode readers, counted apart: each holds a piece cache  |
+| `tiles.swarmCacheBytes`       | `1 GiB`  | **memory** the piece caches of swarm reads share             |
+| `tiles.maxOpenSwarmArchives`  | unset    | a hard count of swarm readers as well, if you want one       |
 | `tiles.maxOpenRemoteArchives` | `64`     | readers for a URL or a bucket; cheap to hold, dear to reopen |
 | `tiles.directoryCacheEntries` | `2000`   | header and directory cache, shared across archives           |
 | `tiles.pieceCacheBytes`       | unset    | sized from the torrent's piece length when unset             |
@@ -738,8 +739,31 @@ what lets a machine with 10 GiB free serve a 700 GiB planet. See
 | `tiles.headerTimeoutMs`       | `12000`  | how long a TileJSON request waits for a header               |
 | `tiles.sparse`                | unset    | `true` for 404 on a missing tile, `false` for 204            |
 
-Leave `pieceCacheBytes` unset unless you have a reason. A fixed budget is a trap
-with 16 MiB pieces, since 64 MiB holds only four.
+### Why a swarm read caches pieces in memory at all
+
+The swarm's unit is a **piece** — 4 MiB for archives this project creates,
+commonly 16 MiB for a planet torrent. A tile is a few kilobytes inside one. So
+fetching a piece to serve one tile and throwing it away means the next tile
+pays for the whole piece again, and PMTiles guarantees there will be a next
+tile in the same piece: its tiles are laid out in Hilbert order, so tiles that
+are neighbours on the map are neighbours in the file. A tile lookup also reads
+the header and a directory before the tile itself, and those live in pieces
+that every single request would otherwise re-fetch.
+
+Not on disk, because a disk copy of the pieces is what cache mode exists to
+avoid: the point is a machine with 10 GiB free serving a 700 GiB planet. The
+engine keeps whatever it keeps; this is the layer above it, and evicting from
+it costs nothing.
+
+**Trading cache size for open archives.** The two settings multiply. Halving
+`pieceCacheBytes` doubles how many archives stay open within the same
+`swarmCacheBytes`, at the cost of a lower hit rate on each. `16 MiB × 64
+archives` and `64 MiB × 16 archives` are the same gigabyte spent differently:
+the first suits a node reading a tile here and a tile there across many
+archives, the second a node serving a region of a few.
+
+Leave `pieceCacheBytes` unset unless you have such a reason. A fixed budget is
+a trap with 16 MiB pieces, since 64 MiB holds only four.
 
 ### Three limits, because the handles cost different things
 
@@ -752,9 +776,13 @@ directory.
 
 - **A complete archive** is a file descriptor and its share of the directory
   cache. The unit allows 65535 descriptors, so hundreds of these are nothing.
-- **A cache-mode archive** carries a piece cache sized from the torrent's piece
-  length. At 16 MiB pieces a hundred of them is gigabytes, which is why this
-  one keeps the old ceiling and is counted separately.
+- **A cache-mode archive** carries a piece cache, and it is **memory** — a map
+  of whole pieces in the node's own heap, not a directory on disk. Bounded by
+  `swarmCacheBytes` rather than by a count, because a count is the wrong unit:
+  a reader allows itself `max(64 MiB, 8 × pieceLength)`, so sixteen of them is
+  1 GiB against the 4 MiB pieces this project creates and 2 GiB against the
+  16 MiB pieces a planet torrent usually has. Stated as memory, the number
+  means what an operator cares about and the count follows from it.
 - **A URL or bucket archive** holds an HTTP reader and the summary it has
   already read. Cheap to keep, and expensive to reopen: a reopen costs a header
   and a directory fetch over the network rather than off a disk.

package/docs/running-as-a-service.md

@@ -189,6 +189,39 @@ The unit below is what it generates, with your paths in it. Worth reading
 either way — a generated file you do not understand is a hand-written one you
 have not written yet.
 
+## `status=226/NAMESPACE`
+
+The unit exits in six milliseconds, having produced nothing at all:
+
+```
+Process: 991182 ExecStart=... (code=exited, status=226/NAMESPACE)
+```
+
+Nothing is wrong with the program; it never ran. With `ProtectSystem=strict`,
+systemd builds a mount namespace before starting anything, and a directory
+named in `ReadWritePaths=` that **does not exist** makes that fail. There is no
+log from the node because there was no node.
+
+Find it:
+
+```
+systemctl show -p ReadWritePaths --value pmtiles-swarm | tr ' ' '
+'   | sed 's/^-//' | while read -r p; do [ -d "$p" ] || echo "missing: $p"; done
+```
+
+Then create what is missing and restart:
+
+```
+sudo install -d -o pmtiles-swarm -g pmtiles-swarm -m 0775 /the/missing/path
+sudo systemctl restart pmtiles-swarm
+```
+
+Units generated from version 0.97.0 onwards prefix every path with `-`, which
+is systemd for "ignore this if it does not exist" — so a missing directory
+becomes a write that fails later, naming the path, rather than a unit that will
+not start and says nothing about why. `init --systemd` also lists the
+directories the configuration names that are not there yet.
+
 ## Regenerating the unit on a node that already runs
 
 The unit is derived from two things that move: the configuration, and how many

package/docs/tile-stacks.md

@@ -38,6 +38,7 @@ of its parts.
 - [The codec problem](#the-codec-problem)
 - [Cost, and the caches that make it bearable](#cost-and-the-caches-that-make-it-bearable)
 - [Caching headers and ETags](#caching-headers-and-etags)
+  - [Clearing what a stack has merged](#clearing-what-a-stack-has-merged)
 - [TileJSON for a stack](#tilejson-for-a-stack)
 - [When a source will not answer](#when-a-source-will-not-answer)
 - [Baking a stack into an archive](#baking-a-stack-into-an-archive)
@@ -524,6 +525,50 @@ etag = H(stackId, stackRevision, resolvedInfoHashes[], z, x, y)
 `stackRevision` bumps whenever the recipe is edited, which invalidates every
 cached tile without anything having to remember to.
 
+### Clearing what a stack has merged
+
+Invalidation and clearing are two different problems, and the ETag only solves
+the first. An edited recipe is never served from the old tiles — the key
+changed — but those tiles are still on the disk, spending the budget until
+eviction reaches them. Under a stack nobody is panning, that is a long time.
+
+So the cache filename carries the stack as well as the digest:
+
+```
+<sha1(etag:ext)>.<sha1(stackId)[0:12]>.<ext>
+```
+
+The stack tag goes **after** the digest, because the first two characters of
+the name are the shard directory: putting the stack first would file every
+tile of a busy stack in one directory. It is in the name rather than in an
+index beside the tiles for the reason `load()` gives — a name survives a
+restart and cannot disagree with the file it is on.
+
+Two things use it:
+
+- **`StackStore` announces recipes that changed**, and the node clears them.
+  This covers every way a recipe moves: the console's Save, `PUT
+/api/stacks/<id>`, an import, a delete, a stack feed, and an operator with an
+  editor open on `stacks.json`. It compares revisions rather than timestamps,
+  so rewriting the file with the same recipes in it announces nothing — which
+  is what makes it safe to run on every reload, and what stops a restart from
+  reading as an edit and discarding the last run's work.
+- **`DELETE /api/stacks/<id>/cache`**, behind a per-stack button in the
+  console, for what a revision cannot see: an archive rewritten under an
+  address that did not change, a cutline redrawn, a codec upgraded, or simply
+  wanting the disk back now. `/api/stacks` reports `cache: {entries, bytes}`
+  per stack so the button can say what it would free, and `null` there means
+  the node keeps nothing at all (`stacks.cacheBytes` is 0).
+
+Both take the stacks **nesting** the changed one with them, transitively. An
+outer stack's tiles were merged from the inner one's, so clearing one level and
+not the other leaves the older answer being served from above.
+
+Tiles written before the tag existed have no stack in their name. They are
+indexed and evicted like any other, and a per-stack clear does not match them;
+they age out on their own. `DELETE /api/storage/merged-tiles` still empties the
+lot.
+
 Cache-control follows the split the tile routes already make: a stack whose
 sources are **all** pinned by `archive` can never change, so
 `public, max-age=31536000, immutable`. A stack with **any** category-resolved

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.95.0",
+  "version": "0.97.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -54,6 +54,7 @@ import {
   resolveStack,
   stackCoverage,
   stackEtag,
+  stacksAffectedBy,
 } from './stacks.js';
 import { bakeRevision } from './bake.js';
 
@@ -3394,6 +3395,11 @@ export function createApp({
           // draws progress on the row, and one poll is what keeps that
           // honest while a bake is running.
           bake: bakes?.get(stack.id) ?? null,
+          // What this stack has already merged and kept. Null rather than
+          // zeroes where the cache is off, so a row can tell "nothing kept
+          // yet" from "nothing is ever kept here" and offer the button only
+          // where it would do something.
+          cache: stackCache?.enabled ? stackCache.usage(stack.id) : null,
           // The schedules aimed at this stack, so its row can say it
           // exports itself without anybody going to look. Several is normal:
           // a nightly build to the fast disk and a weekly one published
@@ -3752,6 +3758,45 @@ export function createApp({
     }),
   );
 
+  /**
+   * Throws away the tiles this stack has already merged.
+   *
+   * Editing the recipe does this by itself — the key covers the revision, so
+   * the store drops them as it saves. This is for what a revision cannot see:
+   * an archive rewritten under an address that did not change, a cutline
+   * redrawn, a codec upgraded, or simply wanting the disk back now rather than
+   * when eviction reaches it.
+   *
+   * Takes the stacks nesting this one with it. Their tiles were merged from
+   * this one's, so clearing here and not there would leave the older answer
+   * being served one level up.
+   */
+  app.delete(
+    '/api/stacks/:id/cache',
+    route(async (req, res) => {
+      await stacks?.refresh();
+      if (!stacks?.get(req.params.id)) {
+        return res.status(404).json({ error: 'no such stack' });
+      }
+      if (!stackCache?.enabled) {
+        // 409 rather than 404: the stack is there and the request makes
+        // sense, the node is simply not keeping anything to clear.
+        return res.status(409).json({
+          error: 'nothing is cached: stacks.cacheBytes is 0 on this node',
+        });
+      }
+
+      const affected = [...stacksAffectedBy(stacks.list(), [req.params.id])];
+      let cleared = 0;
+      let bytes = 0;
+      for (const id of affected) {
+        bytes += stackCache.usage(id).bytes;
+        cleared += await stackCache.clear(id);
+      }
+      res.json({ ok: true, cleared, bytes, stacks: affected });
+    }),
+  );
+
   /**
    * Looks a stack up and resolves it, or answers why it cannot be served.
    * @param {import('express').Request} req - The request.

package/src/config.js

@@ -379,14 +379,29 @@ const DEFAULTS = {
      */
     maxOpenArchives: 128,
     /**
-     * Open archives read through the swarm, which is the expensive kind.
+     * Memory the piece caches of swarm-read archives may hold between them.
      *
-     * Counted apart from the limit above and kept where that limit used to
-     * be. A cache-mode reader holds a piece cache sized from the torrent's
-     * piece length, so with 16 MiB pieces a hundred of them is gigabytes --
-     * the reason the single old limit could not simply be raised.
+     * A count would be the wrong unit. What is expensive about a cache-mode
+     * reader is its piece cache, and how big that is depends on the torrent:
+     * `max(64 MiB, 8 x pieceLength)`, so 64 MiB for the 4 MiB pieces this
+     * project creates and 128 MiB for the 16 MiB pieces a planet torrent
+     * usually has. Sixteen readers is a gigabyte in one case and two in the
+     * other, which is not a limit anybody chose.
+     *
+     * Stated as memory, the number means what an operator actually cares
+     * about, and the count follows from it: lower `pieceCacheBytes` and more
+     * archives stay open, at the same ceiling. A budget rather than an
+     * allocation -- an archive that has served three tiles holds three
+     * pieces.
+     */
+    swarmCacheBytes: 1024 * 1024 * 1024,
+    /**
+     * A hard count of swarm-read archives, for a node that wants one.
+     *
+     * Unset by default: the budget above is the better limit, because it is
+     * the thing that runs out. Set this as well and the smaller wins.
      */
-    maxOpenSwarmArchives: 16,
+    maxOpenSwarmArchives: undefined,
     /**
      * Open archives read straight from a URL or a bucket.
      *
@@ -764,8 +779,16 @@ export function writablePaths(config, configPath) {
     // directory holding it is written to as surely as any of the above.
     configPath ? path.dirname(path.resolve(configPath)) : undefined,
     ...(config?.watch ?? []).map((entry) => entry?.path),
+    ...(config?.watch ?? []).map((entry) => entry?.publishDir),
     ...(config?.locations ?? []).map((entry) => entry?.path),
     ...(config?.subscriptions ?? []).map((entry) => entry?.savePath),
+    // Where a scheduled export puts the archive it just built, and where it
+    // publishes a copy of it. Missed for as long as this list has existed:
+    // a nightly bake to a directory named nowhere else ran for an hour and
+    // was refused the write at the end, with the directory's permissions
+    // perfect -- which is the exact failure this list exists to prevent.
+    ...(config?.stackExports ?? []).map((entry) => entry?.savePath),
+    ...(config?.stackExports ?? []).map((entry) => entry?.publishDir),
   ].filter((value) => typeof value === 'string' && value);
 
   // Deduplicated but deliberately not collapsed into common ancestors. A

package/src/index.js

@@ -254,23 +254,35 @@ PMTILES_SWARM_PUBLIC_URL
 
   const catalog = new Catalog(config.dataDir);
   await catalog.load();
+  // Merged tiles only — see the route. Indexed from disk at startup so a
+  // restart does not throw away work the node has already paid for.
+  const stackCache = new StackCache({
+    dir: config.stacks?.cacheDir ?? path.join(config.dataDir, 'stack-cache'),
+    maxBytes: config.stacks?.cacheBytes,
+  });
+  await stackCache.load();
+
   // Recipes rather than archives, so this is its own file and its own
   // store. A missing stacks.json is simply no stacks.
-  const stacks = new StackStore(config.dataDir);
+  //
+  // Built after the cache so it can drop what a changed recipe has already
+  // merged. The key covers the revision, so those tiles are never served —
+  // this is about the disk they would otherwise sit on until eviction reached
+  // them, which under a stack nobody is panning is a long time. Not awaited:
+  // a save should not wait on unlinking files nothing is going to read, and a
+  // clear that fails leaves entries that were already unreachable.
+  const stacks = new StackStore(config.dataDir, {
+    onChanged: (ids) => {
+      for (const id of ids) stackCache.clear(id).catch(() => {});
+    },
+  });
   await stacks.load();
-  // Merged tiles only -- see the route. Indexed from disk at startup so a
-  // restart does not throw away work the node has already paid for.
+
   // Shapes a recipe can clip a source to. A missing one is a problem reported
   // on the stack that names it, not a reason for this to fail.
   const cutlines = new CutlineStore(config.dataDir);
   await cutlines.load();
 
-  const stackCache = new StackCache({
-    dir: config.stacks?.cacheDir ?? path.join(config.dataDir, 'stack-cache'),
-    maxBytes: config.stacks?.cacheBytes,
-  });
-  await stackCache.load();
-
   const engine = createEngine(config);
   // The slowest, because it announces "stopped" to every tracker, and an
   // unreachable one costs a timeout each. Registered first so it stops last.

package/src/init-command.js

@@ -181,6 +181,33 @@ export async function runInit(options = {}, write = console.log) {
     write('  added to the installed unit by hand is not in this one. Move it');
     write('  into the configuration — as a save location, a watched folder or');
     write('  a cache path — and it will be derived from now on.');
+
+    // A directory the configuration names and nobody has made is the one way
+    // a correct unit still fails: systemd cannot mount what is not there, so
+    // it refuses the whole unit before the process runs.
+    const absent = [];
+    for (const dir of writablePaths(held, configPath)) {
+      const there = await fs
+        .access(dir)
+        .then(() => true)
+        .catch(() => false);
+      if (!there) absent.push(dir);
+    }
+    if (absent.length > 0) {
+      write('');
+      write(`  ${absent.length} of those directories do not exist yet:`);
+      write('');
+      for (const dir of absent) {
+        write(
+          `    sudo install -d -o ${options.user ?? 'pmtiles-swarm'} ` +
+            `-g ${options.user ?? 'pmtiles-swarm'} -m 0775 ${dir}`,
+        );
+      }
+      write('');
+      write('  The unit ignores a directory that is not there rather than');
+      write('  refusing to start, so what you would see instead is a write');
+      write('  failing later, naming the path.');
+    }
     return 0;
   }
 

package/src/stack-cache.js

@@ -28,12 +28,32 @@ import path from 'node:path';
  */
 const SHARD = 2;
 
+/**
+ * How much of the stack id's digest goes in the filename. Long enough that two
+ * of a node's stacks colliding is not a thing that happens, short enough that
+ * the name is still readable when something goes wrong in the directory.
+ */
+const TAG = 12;
+
+/**
+ * Which stack a filename belongs to, or null for one written before the tag
+ * existed. Forgiving on purpose: an untagged file is still a tile that can be
+ * counted and evicted, it simply cannot be cleared by stack.
+ * @param {string} key - The filename.
+ * @returns {string|null} - The tag.
+ */
+function tagOf(key) {
+  const parts = key.split('.');
+  return parts.length === 3 ? parts[1] : null;
+}
+
 /** Merged stack tiles kept on disk, bounded by total size. */
 export class StackCache {
   #dir;
   #maxBytes;
   #entries = new Map();
   #bytes = 0;
+  #byStack = new Map();
   #inFlight = new Map();
   #hits = 0;
   #misses = 0;
@@ -67,6 +87,7 @@ export class StackCache {
   async load() {
     if (!this.enabled) return;
     this.#entries.clear();
+    this.#byStack.clear();
     this.#bytes = 0;
     await fs.mkdir(this.#dir, { recursive: true });
 
@@ -78,8 +99,7 @@ export class StackCache {
         const file = path.join(dir, name);
         const stat = await fs.stat(file).catch(() => null);
         if (!stat?.isFile()) continue;
-        this.#entries.set(name, { size: stat.size, used: stat.mtimeMs });
-        this.#bytes += stat.size;
+        this.#index(name, stat.size, stat.mtimeMs);
       }
     }
     await this.#evict();
@@ -93,16 +113,41 @@ export class StackCache {
    * makes invalidation automatic — editing the recipe or rebuilding a source
    * changes the key rather than needing anything to remember to delete the old
    * one, and the entries nobody asks for again fall out through eviction.
+   *
+   * Which stack it was goes in the name too, after the digest rather than
+   * before it so the sharding still spreads. That is the one thing an operator
+   * clearing a single stack needs and the ETag cannot give: an ETag says
+   * whether two tiles are the same and nothing about where either came from.
+   * In the name rather than in a file beside the tiles, for the reason `load`
+   * gives — a name survives a restart, and cannot disagree with the tile.
+   * @param {string} stackId - Whose tile this is.
    * @param {string} etag - The tile's ETag, which already covers all of it.
    * @param {string} extension - The tile format, so two never collide.
    * @returns {string} - A filename.
    */
-  static key(etag, extension) {
+  static key(stackId, etag, extension) {
     const digest = crypto
       .createHash('sha1')
       .update(`${etag}:${extension}`)
       .digest('hex');
-    return `${digest}.${extension}`;
+    return `${digest}.${StackCache.tag(stackId)}.${extension}`;
+  }
+
+  /**
+   * How a stack id appears in a filename.
+   *
+   * Hashed rather than written out. A stack id is URL-safe and would make a
+   * fine filename, but it may hold the dot the name is split on, and a
+   * fixed-width tag keeps reading it a split rather than a search.
+   * @param {string} stackId - The stack id.
+   * @returns {string} - The tag.
+   */
+  static tag(stackId) {
+    return crypto
+      .createHash('sha1')
+      .update(String(stackId ?? ''))
+      .digest('hex')
+      .slice(0, TAG);
   }
 
   /**
@@ -114,6 +159,49 @@ export class StackCache {
     return path.join(this.#dir, key.slice(0, SHARD), key);
   }
 
+  /**
+   * Records one tile, in the total and against its stack.
+   *
+   * The per-stack totals are kept as entries come and go rather than counted
+   * when they are asked for. The console lists every stack on a poll, and
+   * counting would walk the whole index once per stack per poll — for a number
+   * that only ever changes here.
+   * @param {string} key - The filename.
+   * @param {number} size - Its size in bytes.
+   * @param {number} used - When it was last wanted.
+   * @returns {void}
+   */
+  #index(key, size, used) {
+    this.#forget(key);
+    const tag = tagOf(key);
+    this.#entries.set(key, { size, used, tag });
+    this.#bytes += size;
+    if (!tag) return;
+    const held = this.#byStack.get(tag) ?? { entries: 0, bytes: 0 };
+    held.entries += 1;
+    held.bytes += size;
+    this.#byStack.set(tag, held);
+  }
+
+  /**
+   * Drops one tile from the index, leaving the file to the caller.
+   * @param {string} key - The filename.
+   * @returns {object|null} - What was indexed, if anything was.
+   */
+  #forget(key) {
+    const entry = this.#entries.get(key);
+    if (!entry) return null;
+    this.#entries.delete(key);
+    this.#bytes -= entry.size;
+    const held = entry.tag ? this.#byStack.get(entry.tag) : null;
+    if (held) {
+      held.entries -= 1;
+      held.bytes -= entry.size;
+      if (held.entries <= 0) this.#byStack.delete(entry.tag);
+    }
+    return entry;
+  }
+
   /**
    * Reads a tile, or null when it is not here.
    * @param {string} key - The filename.
@@ -130,8 +218,7 @@ export class StackCache {
     if (!body) {
       // Indexed but gone — something outside this process removed it. Forget
       // it rather than trusting the index over the disk.
-      this.#entries.delete(key);
-      this.#bytes -= entry.size;
+      this.#forget(key);
       this.#misses += 1;
       return null;
     }
@@ -162,10 +249,7 @@ export class StackCache {
       return;
     }
 
-    const previous = this.#entries.get(key);
-    if (previous) this.#bytes -= previous.size;
-    this.#entries.set(key, { size: body.length, used: Date.now() });
-    this.#bytes += body.length;
+    this.#index(key, body.length, Date.now());
     await this.#evict();
   }
 
@@ -199,33 +283,68 @@ export class StackCache {
     const oldest = [...this.#entries.entries()].sort(
       (a, b) => a[1].used - b[1].used,
     );
-    for (const [key, entry] of oldest) {
+    for (const [key] of oldest) {
       if (this.#bytes <= this.#maxBytes) break;
-      this.#entries.delete(key);
-      this.#bytes -= entry.size;
+      this.#forget(key);
       await fs.rm(this.#pathFor(key), { force: true }).catch(() => {});
     }
   }
 
   /**
-   * Throws away every tile it is holding.
+   * Throws away the tiles of one stack, or every tile it is holding.
+   *
+   * Editing a recipe does not need this: the key covers the revision, so the
+   * old tiles simply stop being asked for and fall out through eviction. What
+   * this is for is the two cases the key cannot see — an operator who wants
+   * the space back now, and a source whose bytes changed underneath an address
+   * that did not.
+   *
+   * A stack this has never held is not an error. It is the ordinary answer for
+   * one whose tiles have already been evicted, and a caller clearing several
+   * at once should not have to know which.
    *
    * The hit and miss counters are left alone: they say what the cache has been
    * doing since the process started, which emptying it does not unmake. The
-   * files go one at a time and forgivingly -- one that vanished underneath is
+   * files go one at a time and forgivingly — one that vanished underneath is
    * one fewer to remove.
+   * @param {string} [stackId] - Whose tiles to drop; all of them when absent.
    * @returns {Promise<number>} - How many tiles went.
    */
-  async clear() {
-    const keys = [...this.#entries.keys()];
-    this.#entries.clear();
-    this.#bytes = 0;
+  async clear(stackId) {
+    const keys =
+      stackId === undefined
+        ? [...this.#entries.keys()]
+        : this.#keysOf(StackCache.tag(stackId));
     for (const key of keys) {
+      this.#forget(key);
       await fs.rm(this.#pathFor(key), { force: true }).catch(() => {});
     }
     return keys.length;
   }
 
+  /**
+   * The keys one stack's tiles are under.
+   * @param {string} tag - The stack's tag.
+   * @returns {string[]} - The filenames.
+   */
+  #keysOf(tag) {
+    const keys = [];
+    for (const [key, entry] of this.#entries) {
+      if (entry.tag === tag) keys.push(key);
+    }
+    return keys;
+  }
+
+  /**
+   * What one stack is holding, so its row can say so and offer to clear it.
+   * @param {string} stackId - The stack id.
+   * @returns {object} - `{entries, bytes}`, both zero for a stack with none.
+   */
+  usage(stackId) {
+    const held = this.#byStack.get(StackCache.tag(stackId));
+    return { entries: held?.entries ?? 0, bytes: held?.bytes ?? 0 };
+  }
+
   /**
    * What the cache is holding, for the console and for tests.
    * @returns {object} - entries, bytes, maxBytes, hits, misses.

package/src/stack-tile.js

@@ -974,9 +974,11 @@ export async function answerStackTile(options) {
   // Keyed by the ETag, which already covers the recipe's revision and what its
   // sources resolved to -- so an edited stack or a rebuilt source produces a
   // different key rather than needing anything to remember to invalidate the
-  // old one.
+  // old one. The stack's id goes in beside it, which the digest cannot be read
+  // back out of, so a stack's tiles can also be cleared on purpose.
   const cacheKey = stackCache?.enabled
     ? StackCache.key(
+        resolved.stack.id,
         `${stackEtag(resolved, z, x, y)}:${size ?? 'auto'}`,
         format,
       )

package/src/stacks.js

@@ -603,6 +603,43 @@ export function stacksUsing(stacks, entry, categoryInfo) {
   return found;
 }
 
+/**
+ * Every stack that would be affected by these ones changing.
+ *
+ * A stack can be a source of another, and the outer one's tiles are built from
+ * the inner one's — so a recipe that changed takes its dependents with it,
+ * however many levels up they are. Followed over both the recipes as they were
+ * and as they are, because a stack that has just stopped nesting another still
+ * holds tiles that were merged while it did.
+ * @param {Stack[]} stacks - Every stack, in any order and with duplicates.
+ * @param {Iterable<string>} ids - The stacks that changed.
+ * @returns {Set<string>} - Those ids and everything above them.
+ */
+export function stacksAffectedBy(stacks, ids) {
+  const above = new Map();
+  for (const stack of stacks ?? []) {
+    for (const source of stack.sources ?? []) {
+      if (!source?.stack) continue;
+      const users = above.get(source.stack) ?? new Set();
+      users.add(stack.id);
+      above.set(source.stack, users);
+    }
+  }
+
+  const affected = new Set();
+  const queue = [...ids];
+  while (queue.length) {
+    const id = queue.pop();
+    // A stack may name another twice, and two stacks may name the same one.
+    // Stopping at what has been seen is also what stops a loop, which the
+    // resolver refuses but the file on disk can still contain.
+    if (!id || affected.has(id)) continue;
+    affected.add(id);
+    for (const user of above.get(id) ?? []) queue.push(user);
+  }
+  return affected;
+}
+
 /**
  * Derives the zoom range, bounds and attribution a resolved stack advertises.
  *
@@ -839,13 +876,20 @@ export class StackStore {
   #mtime = null;
   #size = null;
   #checkedAt = 0;
+  #revisions = new Map();
+  #onChanged;
 
   /**
    * Creates a store backed by a file.
    * @param {string} dataDir - Directory holding stacks.json.
+   * @param {object} [options] - `onChanged(ids)`, called with the stacks whose
+   *   recipe is no longer what it was — by an edit here, by a feed, or by
+   *   somebody writing the file. Anything holding work derived from a recipe
+   *   hears about it here rather than having to watch the file itself.
    */
-  constructor(dataDir) {
+  constructor(dataDir, { onChanged } = {}) {
     this.#file = path.join(dataDir, 'stacks.json');
+    this.#onChanged = onChanged ?? null;
   }
 
   /**
@@ -857,6 +901,8 @@ export class StackStore {
    * @returns {Promise<void>} - Resolves once loaded.
    */
   async load() {
+    const were = this.#revisions;
+    const before = this.list();
     this.#stacks.clear();
     this.#problems.clear();
     let raw;
@@ -869,6 +915,9 @@ export class StackStore {
       if (error.code === 'ENOENT') {
         this.#mtime = null;
         this.#size = null;
+        // A file that has been deleted is every stack in it removed, which is
+        // as much a change as an edit.
+        this.#settle(were, before);
         return;
       }
       throw error;
@@ -881,6 +930,38 @@ export class StackStore {
       this.#stacks.set(stack.id, stack);
       if (problems.length) this.#problems.set(stack.id, problems);
     }
+    this.#settle(were, before);
+  }
+
+  /**
+   * Takes the revisions again and says which stacks are not what they were.
+   *
+   * A revision rather than a timestamp or a flag, so this is the same answer
+   * whichever way the change arrived — the console's Save, a stack feed, or
+   * an operator with an editor open on stacks.json. Rewriting the file with
+   * the same recipes in it announces nothing, which is what makes it safe to
+   * call this on every reload.
+   *
+   * The first load announces nothing either: there is no earlier revision for
+   * anything to differ from, so a restart does not read as an edit and throw
+   * away the merged tiles the last run paid for.
+   * @param {Map<string, string>} were - The revisions before.
+   * @param {Stack[]} before - The recipes before, for what nested what.
+   * @returns {void}
+   */
+  #settle(were, before) {
+    this.#revisions = new Map(
+      [...this.#stacks].map(([id, stack]) => [id, stackRevision(stack)]),
+    );
+    // Removed stacks are in here too: their revision is now undefined, which
+    // is not what it was. Added ones are not, and have nothing to clear.
+    const changed = [...were.keys()].filter(
+      (id) => this.#revisions.get(id) !== were.get(id),
+    );
+    if (!changed.length || !this.#onChanged) return;
+    this.#onChanged([
+      ...stacksAffectedBy([...before, ...this.list()], changed),
+    ]);
   }
 
   /**
@@ -958,9 +1039,12 @@ export class StackStore {
       error.problems = problems;
       throw error;
     }
+    const were = this.#revisions;
+    const before = this.list();
     this.#stacks.set(stack.id, stack);
     this.#problems.delete(stack.id);
     await this.#flush();
+    this.#settle(were, before);
     return stack;
   }
 
@@ -970,9 +1054,12 @@ export class StackStore {
    * @returns {Promise<boolean>} - Whether anything was there.
    */
   async remove(id) {
+    const were = this.#revisions;
+    const before = this.list();
     if (!this.#stacks.delete(id)) return false;
     this.#problems.delete(id);
     await this.#flush();
+    this.#settle(were, before);
     return true;
   }
 

package/src/systemd.js

@@ -101,7 +101,14 @@ export function unitFor({
 
   // Wrapped the way systemd's own examples are, because this list grows with
   // every watched folder and a single line of them is unreadable in a diff.
-  const readWrite = paths.join(' \\\n  ');
+  //
+  // Each prefixed with `-`, which is systemd for "ignore this if it does not
+  // exist". Without it a directory named here and not yet created stops the
+  // unit before the process runs: `status=226/NAMESPACE`, six milliseconds of
+  // CPU, and nothing in the journal from a program that never started. With
+  // it the node starts and the write fails where it is attempted, naming the
+  // directory. A missing directory should be a message, not a silence.
+  const readWrite = paths.map((one) => `-${one}`).join(' \\\n  ');
 
   return `# pmtiles-swarm, generated by \`pmtiles-swarm init --systemd\`.
 #
@@ -177,6 +184,11 @@ ProtectHome=read-only
 PrivateTmp=true
 NoNewPrivileges=true
 
+# Each path is prefixed with a dash, which is systemd for "ignore this if it
+# does not exist". Without it a directory named here and not yet created
+# stops the unit before the process runs -- status=226/NAMESPACE, six
+# milliseconds of CPU, and nothing in the journal from a program that never
+# started. init --systemd prints the commands that create them.
 ReadWritePaths=${readWrite}
 
 # Group-writable, so what this service creates in a folder shared with whatever

package/src/tiles.js

@@ -49,6 +49,42 @@ export class TileReadError extends Error {
   }
 }
 
+/** What a reader's piece cache holds before it has seen the torrent. */
+const STARTING_CACHE_BYTES = 64 * 1024 * 1024;
+
+/**
+ * Which swarm-read archives to close for their piece caches to fit a budget.
+ *
+ * Each reader is asked what its cache costs rather than assumed, because only
+ * it knows: the budget it sets for itself is `max(64 MiB, 8 x pieceLength)`,
+ * and the piece length is the torrent's. So a node reading 4 MiB-piece
+ * archives keeps four times as many open as one reading 16 MiB-piece
+ * archives, for the same memory -- which is the arithmetic an operator would
+ * otherwise have to do by hand with every piece length in front of them.
+ *
+ * Least recently used first, and never the last one: the archive just opened
+ * is the one being read, and closing it to meet a budget it alone exceeds
+ * would serve no tile at all.
+ * @param {Array} held - `[key, handle]` pairs, least recently used first.
+ * @param {number} budget - Bytes the piece caches may hold between them.
+ * @returns {Array} - The pairs to close, in the order to close them.
+ */
+export function overSwarmBudget(held, budget) {
+  if (!Number.isFinite(budget) || budget <= 0) return [];
+  const swarm = held.filter(([, one]) => one?.mode === 'swarm');
+  const cost = (one) => one.source?.stats?.cacheBudget ?? STARTING_CACHE_BYTES;
+
+  let total = swarm.reduce((sum, [, one]) => sum + cost(one), 0);
+  const closing = [];
+  for (const pair of swarm) {
+    if (total <= budget) break;
+    if (swarm.length - closing.length <= 1) break;
+    closing.push(pair);
+    total -= cost(pair[1]);
+  }
+  return closing;
+}
+
 /**
  * Opens archives on demand and reads tiles out of them.
  */
@@ -439,10 +475,13 @@ export class TileStore {
     // local archives reopening files it had just closed.
     const tiles = this.#config.tiles ?? {};
     await this.#evict(() => true, tiles.maxOpenArchives ?? 128);
-    await this.#evict(
-      (one) => one.mode === 'swarm',
-      tiles.maxOpenSwarmArchives ?? 16,
-    );
+    if (tiles.maxOpenSwarmArchives !== undefined) {
+      await this.#evict(
+        (one) => one.mode === 'swarm',
+        tiles.maxOpenSwarmArchives,
+      );
+    }
+    await this.#evictSwarmMemory(tiles.swarmCacheBytes ?? 1024 * 1024 * 1024);
     return handle;
   }
 
@@ -465,6 +504,19 @@ export class TileStore {
     }
   }
 
+  /**
+   * Closes swarm-read archives until their piece caches fit a byte budget.
+   * @param {number} budget - Bytes the piece caches may hold between them.
+   * @returns {Promise<void>} - When the closing is done.
+   */
+  async #evictSwarmMemory(budget) {
+    for (const [key] of overSwarmBudget([...this.#open], budget)) {
+      const victim = this.#open.get(key);
+      this.#open.delete(key);
+      await this.#release(victim);
+    }
+  }
+
   /**
    * Opens an archive, choosing between the local file and the swarm.
    * @param {object} entry - Catalog entry.

package/src/web/index.html

@@ -5289,16 +5289,21 @@ Every piece is hashed against the ` +
                 'stops doing anything.',
             },
             {
-              key: 'tiles.maxOpenSwarmArchives',
-              label: 'Cache-mode archives kept open',
+              key: 'tiles.swarmCacheBytes',
+              label: 'Piece cache memory, all swarm reads',
               type: 'number',
-              placeholder: '16',
+              placeholder: '1073741824',
               help:
-                'Counted apart from the limit above, because this is the ' +
-                'expensive kind: a cache-mode reader holds a piece cache ' +
-                'sized from the torrent’s piece length, so with 16 MiB pieces ' +
-                'a hundred of them is gigabytes. That is why the one limit ' +
-                'above could not simply be raised.',
+                'RAM, not disk. An archive read through the swarm keeps whole ' +
+                'pieces in memory, because the swarm hands out pieces and a ' +
+                'tile is a few kilobytes of one — without it, every tile pays ' +
+                'for a 4 or 16 MiB fetch. Each reader allows itself ' +
+                '<code>max(64 MiB, 8 × piece length)</code>, so this is the ' +
+                'number that decides how many stay open: lower ' +
+                '<b>Piece cache bytes</b> below and more of them fit. A ' +
+                'budget, not an allocation — a reader that has served three ' +
+                'tiles holds three pieces. Complete archives are unaffected: ' +
+                'they hold a file descriptor and no cache.',
             },
             {
               key: 'tiles.maxOpenRemoteArchives',
@@ -7769,6 +7774,12 @@ Every piece is hashed against the ` +
               : `<button type="button" data-stack-feed="${escapeHtml(stack.id)}"
                          title="The address another node follows to keep its copy of this recipe level with this one. Add it there under Settings → Feeds → Stack feeds.">RSS</button>`
           }
+          ${
+            stack.cache && stack.cache.bytes > 0
+              ? `<button type="button" data-stack-uncache="${escapeHtml(stack.id)}"
+                         title="Throws away the ${stack.cache.entries.toLocaleString()} tiles this stack has already merged, so the next request for each merges it again. Saving an edit does this by itself — this is for a source whose bytes changed under an address that did not.">Clear cache (${bytes(stack.cache.bytes)})</button>`
+              : ''
+          }
           <button type="button" data-stack-edit="${escapeHtml(stack.id)}">Edit</button>
           <button type="button" data-stack-copy="${escapeHtml(stack.id)}"
                   title="Opens the editor on a copy of this recipe, under a name of its own. Nothing is saved until you save it.">Duplicate</button>
@@ -9168,6 +9179,7 @@ Every piece is hashed against the ` +
         const drop = event.target.dataset.stackDelete;
         const bake = event.target.dataset.stackBake;
         const stopBake = event.target.dataset.stackBakeStop;
+        const uncache = event.target.dataset.stackUncache;
         if (bake !== undefined) {
           const { stacks } = await api('/api/stacks');
           await openBakeDialog(
@@ -9219,6 +9231,31 @@ Every piece is hashed against the ` +
             },
             { copy: true, taken },
           );
+        } else if (uncache !== undefined) {
+          // No confirmation, for the reason the Storage panel gives: a merged
+          // tile can be merged again. What it costs is the merge, and the row
+          // says how much disk it is getting back.
+          try {
+            const gone = await api(
+              `/api/stacks/${encodeURIComponent(uncache)}/cache`,
+              { method: 'DELETE' },
+            );
+            // The others are named rather than counted: pressing one
+            // button and clearing three stacks is surprising until you can
+            // see that the other two are built on this one.
+            const also = gone.stacks.filter((id) => id !== uncache);
+            // Nothing to free is a real answer, not a failure: eviction may
+            // have reached these tiles between the row being drawn and the
+            // button being pressed.
+            toast(
+              `${gone.bytes > 0 ? `Freed ${bytes(gone.bytes)} from` : 'Cleared'} ${uncache}${
+                also.length ? ` and ${also.join(', ')}` : ''
+              }`,
+            );
+            loadStacks().catch((e) => toast(e.message));
+          } catch (error) {
+            toast(error.message);
+          }
         } else if (drop !== undefined) {
           if (!confirm(`Delete the stack "${drop}"?`)) return;
           try {