pmtiles-swarm 0.95.0 → 0.96.0

package/CHANGELOG.md

@@ -2,6 +2,27 @@
 
 ## master
 ### ✨ Features and improvements
+- _...Add new stuff here..._
+
+### 🐞 Bug fixes
+- _...Add new stuff here..._
+
+## 0.96.0
+### ✨ Features and improvements
+- **The swarm-read limit is memory now, not a count.** `tiles.maxOpenSwarmArchives: 16` becomes
+  `tiles.swarmCacheBytes: 1 GiB`, because a count was the wrong unit for it. What is expensive about
+  a cache-mode reader is its piece cache, which is **RAM** — a map of whole pieces in the node's own
+  heap — and how big it is depends on the torrent: `max(64 MiB, 8 × pieceLength)`. Sixteen readers
+  is a gigabyte against the 4 MiB pieces this project creates and two against the 16 MiB pieces a
+  planet torrent usually has, which is not a limit anybody chose.
+
+  Each reader is asked what its cache costs rather than assumed, so the count now follows from the
+  memory: halve `pieceCacheBytes` and twice as many archives stay open for the same gigabyte. The
+  last reader is never closed, whatever the budget says — the archive just opened is the one being
+  read. `maxOpenSwarmArchives` remains for a node that wants a hard count as well; unset by default.
+
+  Complete archives are unaffected: they hold a file descriptor and no piece cache, and are bounded
+  by `maxOpenArchives`.
 - **`init --systemd` regenerates the unit without touching the configuration.** The unit is derived
   from the configuration and from how many archives the library holds, and both move — a watched
   folder added later belongs in `ReadWritePaths`, and a grown library needs longer to write its
@@ -12,7 +33,11 @@
   and a path added to the installed unit by hand will not be in the new one.
 
 ### 🐞 Bug fixes
-- _...Add new stuff here..._
+- **The unit's `ReadWritePaths` left out where an export writes.** `stackExports[].savePath` and
+  `.publishDir`, and a watched folder's `publishDir`, were not among the directories derived from
+  the configuration — so a nightly bake to a directory named nowhere else ran for an hour and was
+  refused the write at the end, with that directory's permissions perfect. Which is the exact
+  failure the derived list exists to prevent. Re-run `init --systemd` to pick them up.
 
 ## 0.95.0
 ### ✨ Features and improvements

package/docs/configuration.md

@@ -727,7 +727,8 @@ what lets a machine with 10 GiB free serve a 700 GiB planet. See
 | setting                       | default  |                                                              |
 | ----------------------------- | -------- | ------------------------------------------------------------ |
 | `tiles.maxOpenArchives`       | `128`    | complete archives kept open; each is a file descriptor       |
-| `tiles.maxOpenSwarmArchives`  | `16`     | cache-mode readers, counted apart: each holds a piece cache  |
+| `tiles.swarmCacheBytes`       | `1 GiB`  | **memory** the piece caches of swarm reads share             |
+| `tiles.maxOpenSwarmArchives`  | unset    | a hard count of swarm readers as well, if you want one       |
 | `tiles.maxOpenRemoteArchives` | `64`     | readers for a URL or a bucket; cheap to hold, dear to reopen |
 | `tiles.directoryCacheEntries` | `2000`   | header and directory cache, shared across archives           |
 | `tiles.pieceCacheBytes`       | unset    | sized from the torrent's piece length when unset             |
@@ -738,8 +739,31 @@ what lets a machine with 10 GiB free serve a 700 GiB planet. See
 | `tiles.headerTimeoutMs`       | `12000`  | how long a TileJSON request waits for a header               |
 | `tiles.sparse`                | unset    | `true` for 404 on a missing tile, `false` for 204            |
 
-Leave `pieceCacheBytes` unset unless you have a reason. A fixed budget is a trap
-with 16 MiB pieces, since 64 MiB holds only four.
+### Why a swarm read caches pieces in memory at all
+
+The swarm's unit is a **piece** — 4 MiB for archives this project creates,
+commonly 16 MiB for a planet torrent. A tile is a few kilobytes inside one. So
+fetching a piece to serve one tile and throwing it away means the next tile
+pays for the whole piece again, and PMTiles guarantees there will be a next
+tile in the same piece: its tiles are laid out in Hilbert order, so tiles that
+are neighbours on the map are neighbours in the file. A tile lookup also reads
+the header and a directory before the tile itself, and those live in pieces
+that every single request would otherwise re-fetch.
+
+Not on disk, because a disk copy of the pieces is what cache mode exists to
+avoid: the point is a machine with 10 GiB free serving a 700 GiB planet. The
+engine keeps whatever it keeps; this is the layer above it, and evicting from
+it costs nothing.
+
+**Trading cache size for open archives.** The two settings multiply. Halving
+`pieceCacheBytes` doubles how many archives stay open within the same
+`swarmCacheBytes`, at the cost of a lower hit rate on each. `16 MiB × 64
+archives` and `64 MiB × 16 archives` are the same gigabyte spent differently:
+the first suits a node reading a tile here and a tile there across many
+archives, the second a node serving a region of a few.
+
+Leave `pieceCacheBytes` unset unless you have such a reason. A fixed budget is
+a trap with 16 MiB pieces, since 64 MiB holds only four.
 
 ### Three limits, because the handles cost different things
 
@@ -752,9 +776,13 @@ directory.
 
 - **A complete archive** is a file descriptor and its share of the directory
   cache. The unit allows 65535 descriptors, so hundreds of these are nothing.
-- **A cache-mode archive** carries a piece cache sized from the torrent's piece
-  length. At 16 MiB pieces a hundred of them is gigabytes, which is why this
-  one keeps the old ceiling and is counted separately.
+- **A cache-mode archive** carries a piece cache, and it is **memory** — a map
+  of whole pieces in the node's own heap, not a directory on disk. Bounded by
+  `swarmCacheBytes` rather than by a count, because a count is the wrong unit:
+  a reader allows itself `max(64 MiB, 8 × pieceLength)`, so sixteen of them is
+  1 GiB against the 4 MiB pieces this project creates and 2 GiB against the
+  16 MiB pieces a planet torrent usually has. Stated as memory, the number
+  means what an operator cares about and the count follows from it.
 - **A URL or bucket archive** holds an HTTP reader and the summary it has
   already read. Cheap to keep, and expensive to reopen: a reopen costs a header
   and a directory fetch over the network rather than off a disk.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.95.0",
+  "version": "0.96.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/config.js

@@ -379,14 +379,29 @@ const DEFAULTS = {
      */
     maxOpenArchives: 128,
     /**
-     * Open archives read through the swarm, which is the expensive kind.
+     * Memory the piece caches of swarm-read archives may hold between them.
      *
-     * Counted apart from the limit above and kept where that limit used to
-     * be. A cache-mode reader holds a piece cache sized from the torrent's
-     * piece length, so with 16 MiB pieces a hundred of them is gigabytes --
-     * the reason the single old limit could not simply be raised.
+     * A count would be the wrong unit. What is expensive about a cache-mode
+     * reader is its piece cache, and how big that is depends on the torrent:
+     * `max(64 MiB, 8 x pieceLength)`, so 64 MiB for the 4 MiB pieces this
+     * project creates and 128 MiB for the 16 MiB pieces a planet torrent
+     * usually has. Sixteen readers is a gigabyte in one case and two in the
+     * other, which is not a limit anybody chose.
+     *
+     * Stated as memory, the number means what an operator actually cares
+     * about, and the count follows from it: lower `pieceCacheBytes` and more
+     * archives stay open, at the same ceiling. A budget rather than an
+     * allocation -- an archive that has served three tiles holds three
+     * pieces.
+     */
+    swarmCacheBytes: 1024 * 1024 * 1024,
+    /**
+     * A hard count of swarm-read archives, for a node that wants one.
+     *
+     * Unset by default: the budget above is the better limit, because it is
+     * the thing that runs out. Set this as well and the smaller wins.
      */
-    maxOpenSwarmArchives: 16,
+    maxOpenSwarmArchives: undefined,
     /**
      * Open archives read straight from a URL or a bucket.
      *
@@ -764,8 +779,16 @@ export function writablePaths(config, configPath) {
     // directory holding it is written to as surely as any of the above.
     configPath ? path.dirname(path.resolve(configPath)) : undefined,
     ...(config?.watch ?? []).map((entry) => entry?.path),
+    ...(config?.watch ?? []).map((entry) => entry?.publishDir),
     ...(config?.locations ?? []).map((entry) => entry?.path),
     ...(config?.subscriptions ?? []).map((entry) => entry?.savePath),
+    // Where a scheduled export puts the archive it just built, and where it
+    // publishes a copy of it. Missed for as long as this list has existed:
+    // a nightly bake to a directory named nowhere else ran for an hour and
+    // was refused the write at the end, with the directory's permissions
+    // perfect -- which is the exact failure this list exists to prevent.
+    ...(config?.stackExports ?? []).map((entry) => entry?.savePath),
+    ...(config?.stackExports ?? []).map((entry) => entry?.publishDir),
   ].filter((value) => typeof value === 'string' && value);
 
   // Deduplicated but deliberately not collapsed into common ancestors. A

package/src/tiles.js

@@ -49,6 +49,42 @@ export class TileReadError extends Error {
   }
 }
 
+/** What a reader's piece cache holds before it has seen the torrent. */
+const STARTING_CACHE_BYTES = 64 * 1024 * 1024;
+
+/**
+ * Which swarm-read archives to close for their piece caches to fit a budget.
+ *
+ * Each reader is asked what its cache costs rather than assumed, because only
+ * it knows: the budget it sets for itself is `max(64 MiB, 8 x pieceLength)`,
+ * and the piece length is the torrent's. So a node reading 4 MiB-piece
+ * archives keeps four times as many open as one reading 16 MiB-piece
+ * archives, for the same memory -- which is the arithmetic an operator would
+ * otherwise have to do by hand with every piece length in front of them.
+ *
+ * Least recently used first, and never the last one: the archive just opened
+ * is the one being read, and closing it to meet a budget it alone exceeds
+ * would serve no tile at all.
+ * @param {Array} held - `[key, handle]` pairs, least recently used first.
+ * @param {number} budget - Bytes the piece caches may hold between them.
+ * @returns {Array} - The pairs to close, in the order to close them.
+ */
+export function overSwarmBudget(held, budget) {
+  if (!Number.isFinite(budget) || budget <= 0) return [];
+  const swarm = held.filter(([, one]) => one?.mode === 'swarm');
+  const cost = (one) => one.source?.stats?.cacheBudget ?? STARTING_CACHE_BYTES;
+
+  let total = swarm.reduce((sum, [, one]) => sum + cost(one), 0);
+  const closing = [];
+  for (const pair of swarm) {
+    if (total <= budget) break;
+    if (swarm.length - closing.length <= 1) break;
+    closing.push(pair);
+    total -= cost(pair[1]);
+  }
+  return closing;
+}
+
 /**
  * Opens archives on demand and reads tiles out of them.
  */
@@ -439,10 +475,13 @@ export class TileStore {
     // local archives reopening files it had just closed.
     const tiles = this.#config.tiles ?? {};
     await this.#evict(() => true, tiles.maxOpenArchives ?? 128);
-    await this.#evict(
-      (one) => one.mode === 'swarm',
-      tiles.maxOpenSwarmArchives ?? 16,
-    );
+    if (tiles.maxOpenSwarmArchives !== undefined) {
+      await this.#evict(
+        (one) => one.mode === 'swarm',
+        tiles.maxOpenSwarmArchives,
+      );
+    }
+    await this.#evictSwarmMemory(tiles.swarmCacheBytes ?? 1024 * 1024 * 1024);
     return handle;
   }
 
@@ -465,6 +504,19 @@ export class TileStore {
     }
   }
 
+  /**
+   * Closes swarm-read archives until their piece caches fit a byte budget.
+   * @param {number} budget - Bytes the piece caches may hold between them.
+   * @returns {Promise<void>} - When the closing is done.
+   */
+  async #evictSwarmMemory(budget) {
+    for (const [key] of overSwarmBudget([...this.#open], budget)) {
+      const victim = this.#open.get(key);
+      this.#open.delete(key);
+      await this.#release(victim);
+    }
+  }
+
   /**
    * Opens an archive, choosing between the local file and the swarm.
    * @param {object} entry - Catalog entry.

package/src/web/index.html

@@ -5289,16 +5289,21 @@ Every piece is hashed against the ` +
                 'stops doing anything.',
             },
             {
-              key: 'tiles.maxOpenSwarmArchives',
-              label: 'Cache-mode archives kept open',
+              key: 'tiles.swarmCacheBytes',
+              label: 'Piece cache memory, all swarm reads',
               type: 'number',
-              placeholder: '16',
+              placeholder: '1073741824',
               help:
-                'Counted apart from the limit above, because this is the ' +
-                'expensive kind: a cache-mode reader holds a piece cache ' +
-                'sized from the torrent’s piece length, so with 16 MiB pieces ' +
-                'a hundred of them is gigabytes. That is why the one limit ' +
-                'above could not simply be raised.',
+                'RAM, not disk. An archive read through the swarm keeps whole ' +
+                'pieces in memory, because the swarm hands out pieces and a ' +
+                'tile is a few kilobytes of one — without it, every tile pays ' +
+                'for a 4 or 16 MiB fetch. Each reader allows itself ' +
+                '<code>max(64 MiB, 8 × piece length)</code>, so this is the ' +
+                'number that decides how many stay open: lower ' +
+                '<b>Piece cache bytes</b> below and more of them fit. A ' +
+                'budget, not an allocation — a reader that has served three ' +
+                'tiles holds three pieces. Complete archives are unaffected: ' +
+                'they hold a file descriptor and no cache.',
             },
             {
               key: 'tiles.maxOpenRemoteArchives',