pmtiles-swarm 0.77.0 → 0.79.0

package/CHANGELOG.md

@@ -7,6 +7,67 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.79.0
+### ✨ Features and improvements
+- **Duplicate a stack.** A button beside Edit on every row, which opens the editor on a copy of that
+  recipe: the same sources in the same order, the same masks, the same output, under a name of its
+  own and saved only when you save it. Most stacks after the first are a variation on one that
+  already works, and rebuilding that by hand is where a source gets left out.
+
+  Two things are changed for the copy and nothing else is. The name becomes `<name>-copy`, counting
+  up until it is one nothing is using, and the title gains `copy` — two stacks under one title are
+  two rows nobody can tell apart in the list they both appear in. Both are editable before saving.
+
+  It copies the **recipe** rather than the row. The list holds what each source resolved to, so a
+  copy taken from it would pin the infohashes the original follows by category and stop following
+  rebuilds from the moment it was made.
+
+### 🐞 Bug fixes
+- **Saving a stack with no name reported that the reply was not JSON.** `Unexpected token '<',
+  "<!DOCTYPE "... is not valid JSON`, which says nothing about a missing name. A stack with no name
+  is a `PUT /api/stacks/`, and that matches no route at all — `:id` needs a segment to be — so what
+  came back was express's own HTML error page, which the console then tried to parse.
+
+  Both halves are fixed. Every unmatched path under `/api` answers JSON now, so any future typo says
+  `no route for PUT /api/stacks/` rather than arriving as a parse error; and the dialog refuses an
+  empty or malformed name itself, since it is the one that knows what the box is for.
+
+- **A new stack could be saved over an existing one without a word.** `PUT` upserts, which is right
+  for editing and wrong for naming: typing a name already in use replaced that stack rather than
+  refusing. It is refused now when naming a new stack or a copy — where the stack it would have
+  replaced is usually the one being copied. Editing is unaffected; a stack keeps the name it has.
+
+## 0.78.0
+### ✨ Features and improvements
+- **Somewhere to clear the caches and the files nothing is waiting for.** A merged tile cache with
+  no way to empty it is a directory somebody eventually finds by hand, and a temporary file a crash
+  left behind is one nobody finds at all. Settings has a **Storage** tab now: what this node is
+  holding, what each thing costs to lose, and a button per row.
+
+  Five things — merged stack tiles, left-over temporary files, stopped exports, traffic history and
+  the tile counters. `GET /api/storage` reports them, `DELETE /api/storage/:what` lets go of one.
+  Everything on the list is derived and can be rebuilt, which is what makes a button reasonable:
+  none of it asks whether you meant it, because none of it is the only copy of anything.
+
+  The archives are deliberately not on it, and neither is the resume data beside them. Both look
+  like housekeeping and neither is: retiring an archive is a decision made from its own panel with
+  what it seeds in view, and resume data thrown away is a rehash of every byte on disk.
+
+  A sweep picks files by name and by age together, because either alone is wrong: `*.tmp` and
+  `pmtiles-write-*`, untouched for an hour. Every write that uses one renames within milliseconds,
+  so the margin is not for slowness — it is because this runs while the node is serving, and a sweep
+  with none at all could take the file a catalog write is halfway through renaming into place.
+  `torrents-data` is skipped outright, being terabytes of payload with no working files in it.
+
+  Stopped exports are read off the disk rather than from what the process remembers, so one left by
+  a stack somebody has since deleted is found as well. A running export is left alone: removing the
+  directory under a running merge would have it fail on its next write, reporting a disk problem for
+  something somebody chose. This is also the first way to discard one without finding the directory
+  by hand — the API for it shipped in 0.76.0 with nothing calling it.
+
+### 🐞 Bug fixes
+- _...Add new stuff here..._
+
 ## 0.77.0
 ### ✨ Features and improvements
 - **A fade can be written in metres of ground.** `featherMetres` on a source says how far to blend

package/README.md

@@ -786,6 +786,8 @@ which the endpoint answers 501.
 | `GET` `POST` `DELETE`  | `/api/tokens`, `/api/tokens/:id`                            | Mint, list and revoke access tokens                                                                                                                                                                                                                                                                                                        |
 | `GET` `POST`           | `/api/restart`                                              | What a restart would do, and doing it                                                                                                                                                                                                                                                                                                      |
 | `GET` `DELETE`         | `/api/stats`                                                | What this node has served — per-archive counters and the last N requests; `DELETE` clears them                                                                                                                                                                                                                                             |
+| `GET`                  | `/api/storage`                                              | What this node is holding that it could let go of — merged tiles, left-over temporary files, stopped exports, traffic history, tile counters                                                                                                                                                                                               |
+| `DELETE`               | `/api/storage/:what`                                        | Lets go of one of them. Everything listed is derived and can be rebuilt; the archives are not on the list                                                                                                                                                                                                                                  |
 | `GET`                  | `/api/traffic`                                              | Upload and download speed per archive over time, from `stats.db`; `infoHash`, `hours` and `buckets` narrow it, and `totals` ranks archives by bytes moved                                                                                                                                                                                  |
 | `GET` `PATCH`          | `/api/config`                                               | Read and change settings                                                                                                                                                                                                                                                                                                                   |
 | `POST`                 | `/api/login`, `/api/logout`                                 | Console sign-in                                                                                                                                                                                                                                                                                                                            |

package/docs/internals.md

@@ -33,6 +33,7 @@ Operator-facing documentation is elsewhere — see [publishing](publishing.md),
 - [Retiring old builds](#retiring-old-builds)
 - [Prewarming a freshly joined archive](#prewarming-a-freshly-joined-archive)
 - [Named save locations](#named-save-locations)
+- [Storage](#storage)
 
 ## Marking incomplete archives
 
@@ -884,3 +885,60 @@ moving several hundred gigabytes is not something a settings screen should do as
 a side effect. A move checks free space before the engine is disturbed, since
 running out halfway through costs an hour, a partial file to clean up, and an
 archive that has to be put back.
+
+## Storage
+
+A merged tile cache with no way to empty it is a directory somebody eventually
+finds by hand, and a temporary file a crash left behind is one nobody finds at
+all. `GET /api/storage` reports what a node is holding that it could let go of;
+`DELETE /api/storage/:what` lets go of one of them. The console draws it as the
+Storage tab under Settings.
+
+Five things, and what each costs to lose:
+
+| What                      | Where              | Clearing it costs                                              |
+| ------------------------- | ------------------ | -------------------------------------------------------------- |
+| Merged stack tiles        | `stack-cache/`     | The merge again: an archive read per source, and a decode each |
+| Left-over temporary files | the data directory | Nothing — nobody is waiting for a file a crashed write left    |
+| Stopped exports           | the archive's disk | The hours already spent, which is what resuming would use      |
+| Traffic history           | `stats.db`         | The graphs go back to empty                                    |
+| Tile counters             | memory             | The count starts again; frees nothing                          |
+
+Everything on that list is derived and can be rebuilt, which is what makes a
+button reasonable — none of it asks whether the operator meant it, because none
+of it is the only copy of anything.
+
+### What is deliberately not on it
+
+**The archives**, and the resume data beside them. Both look like housekeeping
+and neither is. An archive is the data this node exists to serve, and retiring
+one is a decision made from its own panel with what it seeds in view. Resume
+data thrown away is a rehash of every byte on disk — and worse than useless as
+a repair, since a _stale partial_ resume file is the thing that cancels seed
+mode, which is a diagnosis rather than a sweep.
+
+### Which files a sweep is willing to touch
+
+By name and by age together, because either alone is wrong: the name says what
+a file was for and the age says whether anything still cares.
+
+The names are `*.tmp` and `pmtiles-write-*`, which is what the write-then-rename
+in `catalog.js`, `stacks.js`, `dht-state.js`, `stack-cache.js` and
+`pmtiles-write.js` leaves behind when a process stops between the two steps. The
+age is an hour. Every one of those renames happens within milliseconds, so an
+hour is far past generous — the margin is not for slowness, it is because this
+runs while the node is serving and a sweep with no margin could delete the file
+a catalog write is halfway through renaming into place.
+
+`torrents-data/` is skipped outright. It is terabytes of payload, nothing in it
+is a working file, and a sweep that walks it is a sweep that spends minutes
+finding nothing.
+
+### The bake working directories are read from disk
+
+Not from what the process remembers. A working directory outlives the run that
+made it, and one belonging to a stack somebody has since deleted is exactly the
+kind nobody thinks to look for. Whether an export is _running_ is memory's to
+say, and that is the only thing that decides whether it may be discarded —
+removing the directory under a running merge would have it fail on its next
+write, reporting a disk problem for something somebody chose.

package/docs/tile-stacks.md

@@ -1299,9 +1299,17 @@ mostly diagonals.
 works out the pixels for each tile it builds:
 
 ```json
-{ "archive": "gebco", "maskRange": [-1, 0], "featherMetres": 50 }
+{ "category": "planet-bathymetry", "maskRange": [-1, 0], "featherMetres": 50 }
 ```
 
+That is the source **on top** — the one whose mask leaves the hole. A weight
+says how much of what is underneath shows through, so it goes on the layer with
+the edge and not on the one filling the hole: a fade on the bathymetry
+underneath is read by nothing, because `paintHeights` takes the first layer that
+contributes whole and only consults a weight from the second onward. See "A
+feathered layer over a hole stands alone" above, which is the same rule seen
+from the other side.
+
 Web Mercator makes that arithmetic rather than a lookup: one pixel covers
 `40075016.686 × cos(latitude) / 2^zoom / tileSize` metres, so the conversion
 needs a multiply and the coordinates of the tile being built. `featherMeters` is
@@ -1578,10 +1586,33 @@ editor exists for — an editor whose primary action only works with a mouse is
 an editor half the people cannot use. The console has no drag-and-drop anywhere
 yet, so this is the first, and the buttons are what make it safe to add.
 
+### Starting from a stack that already works
+
+**Duplicate**, beside Edit on every row. It opens the editor on a copy of that
+recipe: the same sources in the same order, the same masks, the same output —
+under a name of its own, and saved only when it is saved. Most stacks after the
+first are a variation on one that already works, and rebuilding that by hand is
+where a source gets left out.
+
+Two things are changed for the copy and nothing else is. The name becomes
+`<name>-copy`, counting up until it is one nothing is using; the title gains
+`copy`, because two stacks under one title are two rows nobody can tell apart in
+the list they both appear in. Both are editable before saving — the point is
+that neither is left matching by accident.
+
+It copies the **recipe** rather than the row. The list holds what each source
+resolved to, so a copy taken from it would pin the infohashes the original
+follows by category, and stop following rebuilds from the moment it was made.
+The editor reads `/api/stacks/<id>/raw` for the same reason.
+
 ### What the editor refuses, and what it only warns about
 
 Refuses, because the stack cannot work:
 
+- A name that is already a stack's, when naming a new one or a copy. `PUT`
+  upserts, so this is the difference between adding a stack and replacing one —
+  and for a copy, the stack it would replace is usually the one being copied.
+  Editing an existing stack is unaffected: it keeps the name it has.
 - A recipe naming an `output.tileSize` that is not 256 or 512.
 - A source that resolves to nothing — an empty category, or an archive that
   retention has removed.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.77.0",
+  "version": "0.79.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -37,6 +37,7 @@ import { SUMMARY_VERSION } from './pmtiles-probe.js';
 import { TileReadError } from './tiles.js';
 import { loadCodec } from './codec.js';
 import { answerStackTile, outputFormat, outputSize } from './stack-tile.js';
+import { clearStorage, storageReport } from './storage.js';
 import {
   isPinned,
   needsCodec,
@@ -790,6 +791,38 @@ export function createApp({
     }),
   );
 
+  // What this node is holding that it could let go of. Separate from the
+  // settings it sits beside in the console: a setting says what to do next and
+  // this says what to do about what was already done.
+  app.get(
+    '/api/storage',
+    route(async (_req, res) => {
+      res.setHeader('cache-control', 'no-store');
+      res.json(
+        await storageReport({ config, stackCache, stats, traffic, bakes }),
+      );
+    }),
+  );
+
+  app.delete(
+    '/api/storage/:what',
+    route(async (req, res) => {
+      const gone = await clearStorage(req.params.what, {
+        config,
+        stackCache,
+        stats,
+        traffic,
+        bakes,
+      });
+      if (!gone) {
+        return res
+          .status(404)
+          .json({ error: 'nothing here is called that, or it is turned off' });
+      }
+      return res.json(gone);
+    }),
+  );
+
   // Settings. Everything read per request takes effect immediately because the
   // running config object is the one being mutated; everything bound at startup
   // is written to the file and reported back as needing a restart, rather than
@@ -3725,6 +3758,19 @@ export function createApp({
     }
   }
 
+  // An API path no route claimed. Without this express's own handler answers
+  // with an HTML error page, and a caller that parses every reply as JSON --
+  // the console does -- reports `Unexpected token '<'`, which says nothing at
+  // all about what was wrong with the request.
+  //
+  // The one that produced it was a stack saved with no name: `PUT
+  // /api/stacks/` matches no route, because `:id` needs a segment to be, and
+  // the reply came back as the start of an HTML document.
+  app.use('/api', (req, res) => {
+    const where = req.originalUrl.split('?')[0];
+    res.status(404).json({ error: `no route for ${req.method} ${where}` });
+  });
+
   app.use(express.static(path.join(here, 'web')));
 
   // Four parameters, two of them unused: express identifies an error handler

package/src/bake-jobs.js

@@ -205,6 +205,34 @@ export class BakeManager {
     return true;
   }
 
+  /**
+   * Every export with work still on disk, and where it is.
+   *
+   * Read from the filesystem rather than from what this process remembers: a
+   * working directory outlives the run that made it, and one belonging to a
+   * stack somebody has since deleted is exactly the kind nobody thinks to look
+   * for. Whether it is running is what decides if it may be discarded, and
+   * that is memory's to say.
+   * @returns {Promise<object[]>} - `{stackId, directory, running, stopped}`.
+   */
+  async heldWork() {
+    const found = [];
+    for (const root of this.#workRoots()) {
+      const base = path.join(root, WORK_DIR);
+      const names = await fs.readdir(base).catch(() => []);
+      for (const stackId of names) {
+        const job = this.#jobs.get(stackId);
+        found.push({
+          stackId,
+          directory: path.join(base, stackId),
+          running: Boolean(job && !job.finishedAt),
+          stopped: this.#held.has(stackId),
+        });
+      }
+    }
+    return found;
+  }
+
   /**
    * Throws away what a stopped export had done.
    *

package/src/stack-cache.js

@@ -207,6 +207,25 @@ export class StackCache {
     }
   }
 
+  /**
+   * Throws away every tile it is holding.
+   *
+   * The hit and miss counters are left alone: they say what the cache has been
+   * doing since the process started, which emptying it does not unmake. The
+   * files go one at a time and forgivingly -- one that vanished underneath is
+   * one fewer to remove.
+   * @returns {Promise<number>} - How many tiles went.
+   */
+  async clear() {
+    const keys = [...this.#entries.keys()];
+    this.#entries.clear();
+    this.#bytes = 0;
+    for (const key of keys) {
+      await fs.rm(this.#pathFor(key), { force: true }).catch(() => {});
+    }
+    return keys.length;
+  }
+
   /**
    * What the cache is holding, for the console and for tests.
    * @returns {object} - entries, bytes, maxBytes, hits, misses.

package/src/storage.js

@@ -0,0 +1,283 @@
+import fs from 'node:fs/promises';
+import path from 'node:path';
+
+/**
+ * What this node is holding that it could let go of.
+ *
+ * Everything here is derived and can be rebuilt: a merged tile can be merged
+ * again, a traffic sample can be taken again, a half-finished write is a file
+ * nobody is waiting for. That is what makes a button reasonable — none of it
+ * asks whether the operator is sure they meant it, because none of it is the
+ * only copy of anything.
+ *
+ * The archives themselves are deliberately absent, and so is the resume data
+ * beside them. Both look like housekeeping and neither is: an archive is the
+ * data this node exists to serve, and resume data thrown away is a rehash of
+ * every byte on disk. Retiring an archive is its own decision, made where the
+ * archive is, with what it seeds in view. See docs/internals.md — "Storage".
+ */
+
+/**
+ * How long a temporary file has to sit there before it is left behind.
+ *
+ * Every write that uses one renames within milliseconds, so an hour is far
+ * past generous. The margin is not for slowness, it is because this runs while
+ * the node is serving: a sweep with no margin at all could delete the file a
+ * catalog write is in the middle of, on the one machine where it matters.
+ */
+const STALE_AFTER = 60 * 60 * 1000;
+
+/** What a half-finished write leaves behind, by name. */
+const TEMPORARY = [/\.tmp$/, /\.\d+\.tmp$/, /^pmtiles-write-/];
+
+/**
+ * Directories under the data directory that hold archives rather than working
+ * files. Skipped so a sweep cannot wander into terabytes of payload looking
+ * for kilobytes of leftovers.
+ */
+const PAYLOAD = new Set(['torrents-data']);
+
+/**
+ * How big a file is, or 0 when it is not there any more.
+ * @param {string} file - Its path.
+ * @returns {Promise<number>} - Bytes.
+ */
+async function sizeOf(file) {
+  const info = await fs.stat(file).catch(() => null);
+  return info?.isFile() ? info.size : 0;
+}
+
+/**
+ * How much a directory holds, following it down.
+ *
+ * Errors are swallowed per entry rather than per walk: a report that says
+ * nothing because one file disappeared while it was being counted is worse
+ * than one that is a few kilobytes out.
+ * @param {string} dir - Where to start.
+ * @returns {Promise<object>} - `{bytes, files}`.
+ */
+export async function directorySize(dir) {
+  let bytes = 0;
+  let files = 0;
+  const entries = await fs
+    .readdir(dir, { withFileTypes: true })
+    .catch(() => []);
+  for (const entry of entries) {
+    const here = path.join(dir, entry.name);
+    if (entry.isDirectory()) {
+      const under = await directorySize(here);
+      bytes += under.bytes;
+      files += under.files;
+    } else {
+      bytes += await sizeOf(here);
+      files += 1;
+    }
+  }
+  return { bytes, files };
+}
+
+/**
+ * Temporary files nobody is writing any more.
+ *
+ * Found by name and by age together, because either alone is wrong: a name
+ * says what a file was for and an age says whether anything still cares.
+ * @param {string} dir - Where to look.
+ * @param {number} [now] - The clock, injectable for tests.
+ * @returns {Promise<object[]>} - `{path, bytes}` per file.
+ */
+export async function staleTemporaries(dir, now = Date.now()) {
+  const found = [];
+  const entries = await fs
+    .readdir(dir, { withFileTypes: true })
+    .catch(() => []);
+
+  for (const entry of entries) {
+    const here = path.join(dir, entry.name);
+    if (entry.isDirectory()) {
+      if (PAYLOAD.has(entry.name)) continue;
+      found.push(...(await staleTemporaries(here, now)));
+      continue;
+    }
+    if (!TEMPORARY.some((pattern) => pattern.test(entry.name))) continue;
+    const info = await fs.stat(here).catch(() => null);
+    if (!info || now - info.mtimeMs < STALE_AFTER) continue;
+    found.push({ path: here, bytes: info.size });
+  }
+  return found;
+}
+
+/**
+ * Everything a node is holding that it could be asked to let go of.
+ *
+ * Reported whether or not there is anything in it, so the panel reads the same
+ * on a node that has just started as on one that has been running for a month
+ * — a row that appears only once it has something to say is a row nobody knows
+ * to look for.
+ * @param {object} deps - config, stackCache, stats, traffic, bakes.
+ * @returns {Promise<object>} - `{items}`, each with an id, a size and a note.
+ */
+export async function storageReport({
+  config,
+  stackCache,
+  stats,
+  traffic,
+  bakes,
+} = {}) {
+  const dataDir = config?.dataDir ?? './data';
+  const items = [];
+
+  const cache = stackCache?.stats?.() ?? null;
+  items.push({
+    id: 'merged-tiles',
+    title: 'Merged stack tiles',
+    where: path.join(dataDir, 'stack-cache'),
+    bytes: cache?.bytes ?? 0,
+    count: cache?.entries ?? 0,
+    unit: 'tiles',
+    available: Boolean(cache?.enabled),
+    note: cache?.enabled
+      ? 'Tiles a stack has already merged. Clearing them costs the merge again ' +
+        'the next time each is asked for — one archive read per source, a ' +
+        'decode each, and the merge itself.'
+      : 'Off: stacks.cacheBytes is 0, so nothing is kept.',
+  });
+
+  const temporaries = await staleTemporaries(dataDir);
+  items.push({
+    id: 'temporary',
+    title: 'Left-over temporary files',
+    where: dataDir,
+    bytes: temporaries.reduce((sum, file) => sum + file.bytes, 0),
+    count: temporaries.length,
+    unit: 'files',
+    available: true,
+    note:
+      'Half-finished writes, from a process that stopped between writing a ' +
+      'file and renaming it into place. Only ones untouched for an hour are ' +
+      'counted, so a write happening right now is never one of them.',
+  });
+
+  const work = (await bakes?.heldWork?.()) ?? [];
+  const sized = await Promise.all(
+    work.map(async (held) => ({
+      ...held,
+      ...(await directorySize(held.directory)),
+    })),
+  );
+  const idle = sized.filter((held) => !held.running);
+  items.push({
+    id: 'stopped-exports',
+    title: 'Stopped exports',
+    where: idle.map((held) => held.directory).join(', '),
+    bytes: idle.reduce((sum, held) => sum + held.bytes, 0),
+    count: idle.length,
+    unit: 'exports',
+    available: idle.length > 0,
+    stacks: idle.map((held) => held.stackId),
+    note:
+      'Tiles an export had buffered before it was stopped. Keeping them is ' +
+      'what lets it carry on rather than start again, so this is only worth ' +
+      'clearing for an export nobody is going to finish.',
+  });
+
+  const trafficFile = path.join(dataDir, 'stats.db');
+  items.push({
+    id: 'traffic-history',
+    title: 'Traffic history',
+    where: trafficFile,
+    bytes: await sizeOf(trafficFile),
+    count: null,
+    available: Boolean(traffic),
+    note:
+      'Per-archive upload and download samples, behind the traffic graphs. ' +
+      'It already drops anything past its retention, so this is for reclaiming ' +
+      'the file rather than for keeping it in bounds.',
+  });
+
+  items.push({
+    id: 'tile-counters',
+    title: 'Tile counters',
+    where: 'memory',
+    bytes: 0,
+    count: stats?.snapshot?.()?.total ?? null,
+    unit: 'requests',
+    available: Boolean(stats),
+    note:
+      'How many tiles each archive has served, since the counters were last ' +
+      'reset. Held in memory, so this frees nothing — it starts the count ' +
+      'again.',
+  });
+
+  return {
+    dataDir,
+    items,
+    bytes: items.reduce((sum, item) => sum + (item.bytes ?? 0), 0),
+  };
+}
+
+/**
+ * Lets go of one of them.
+ * @param {string} what - The id of an item in the report.
+ * @param {object} deps - config, stackCache, stats, traffic, bakes.
+ * @returns {Promise<object|null>} - What went, or null for an id nothing knows.
+ */
+export async function clearStorage(what, deps = {}) {
+  const { config, stackCache, stats, traffic, bakes } = deps;
+
+  if (what === 'merged-tiles') {
+    const before = stackCache?.stats?.()?.bytes ?? 0;
+    const cleared = (await stackCache?.clear?.()) ?? 0;
+    return { cleared, bytes: before };
+  }
+
+  if (what === 'temporary') {
+    const files = await staleTemporaries(config?.dataDir ?? './data');
+    let bytes = 0;
+    let cleared = 0;
+    for (const file of files) {
+      // One at a time and forgiving: a file that vanished between the walk and
+      // the unlink is a file that is gone, which is what was wanted.
+      const removed = await fs
+        .rm(file.path, { force: true })
+        .then(() => true)
+        .catch(() => false);
+      if (!removed) continue;
+      bytes += file.bytes;
+      cleared += 1;
+    }
+    return { cleared, bytes };
+  }
+
+  if (what === 'stopped-exports') {
+    const work = (await bakes?.heldWork?.()) ?? [];
+    let bytes = 0;
+    let cleared = 0;
+    for (const held of work) {
+      if (held.running) continue;
+      const size = await directorySize(held.directory);
+      // Asked of the manager rather than removed here, so an export that
+      // started between the report and the button is refused by the same rule
+      // that refuses it everywhere else.
+      if (!(await bakes.discard(held.stackId))) continue;
+      bytes += size.bytes;
+      cleared += 1;
+    }
+    return { cleared, bytes };
+  }
+
+  if (what === 'traffic-history') {
+    if (!traffic?.clear) return null;
+    const file = path.join(config?.dataDir ?? './data', 'stats.db');
+    const before = await sizeOf(file);
+    traffic.clear();
+    return { cleared: 1, bytes: before - (await sizeOf(file)) };
+  }
+
+  if (what === 'tile-counters') {
+    if (!stats?.reset) return null;
+    stats.reset();
+    return { cleared: 1, bytes: 0 };
+  }
+
+  return null;
+}

package/src/traffic-stats.js

@@ -143,6 +143,18 @@ export class TrafficStats {
     return written;
   }
 
+  /**
+   * Throws away every sample, and the file they were in.
+   *
+   * Vacuumed rather than only deleted: deleting rows returns the pages to
+   * sqlite's free list, and somebody clearing this wants the disk back.
+   * @returns {void}
+   */
+  clear() {
+    this.#db.prepare('DELETE FROM traffic').run();
+    this.#db.exec('VACUUM');
+  }
+
   /**
    * Deletes samples that have aged past the retention window.
    * @returns {number} - Rows removed.

package/src/web/index.html

@@ -840,7 +840,8 @@
 
         <div class="field">
           <label for="stack-id">Name</label>
-          <input id="stack-id" placeholder="planet-terrain" autocomplete="off" />
+          <input id="stack-id" placeholder="planet-terrain" autocomplete="off"
+                 required pattern="[A-Za-z0-9][A-Za-z0-9._-]*" />
           <div class="sub">
             Used in the URL: <code id="stack-url-preview">/stacks/…/tiles.json</code>
           </div>
@@ -4823,6 +4824,100 @@ Every piece is hashed against the ` +
        * @param {HTMLElement} into - Where to render.
        * @returns {Promise<void>} - Resolves once rendered.
        */
+      /**
+       * What this node is holding that it could let go of.
+       *
+       * Everything listed here is derived and can be rebuilt, which is why
+       * each row is one button and not a dialog: a merged tile can be merged
+       * again, a traffic sample taken again, a half-finished write is a file
+       * nobody is waiting for. The archives are deliberately not here, and
+       * neither is the resume data beside them -- retiring an archive is its
+       * own decision, made where the archive is.
+       * @param {HTMLElement} into - The settings pane to render in.
+       * @returns {Promise<void>} - Resolves once it is drawn.
+       */
+      async function renderStorageEditor(into) {
+        const panel = document.createElement('div');
+        panel.className = 'panel';
+        panel.style.marginBottom = '1rem';
+        into.append(panel);
+
+        const draw = async () => {
+          let state;
+          try {
+            state = await api('/api/storage');
+          } catch (error) {
+            panel.innerHTML = `<div class="empty">${escapeHtml(error.message)}</div>`;
+            return;
+          }
+
+          const rows = state.items
+            .map((item) => {
+              const size =
+                item.bytes > 0 ? bytes(item.bytes) : item.available ? 'empty' : EMPTY;
+              const held =
+                Number.isFinite(item.count) && item.count > 0
+                  ? `<div class="sub">${item.count.toLocaleString()} ${escapeHtml(item.unit ?? '')}</div>`
+                  : '';
+              return `
+                <tr>
+                  <td>
+                    <b>${escapeHtml(item.title)}</b>
+                    <div class="sub">${escapeHtml(item.note)}</div>
+                    <div class="sub"><code>${escapeHtml(item.where || '')}</code></div>
+                  </td>
+                  <td style="white-space:nowrap">${size}${held}</td>
+                  <td style="text-align:right">
+                    <button data-clear="${escapeHtml(item.id)}"
+                            ${item.available && (item.bytes > 0 || item.count > 0) ? '' : 'disabled'}>
+                      Clear
+                    </button>
+                  </td>
+                </tr>`;
+            })
+            .join('');
+
+          panel.innerHTML = `
+            <h2 style="margin-top:0">Storage</h2>
+            <div class="sub" style="margin-bottom:0.8rem">
+              Everything here can be rebuilt, so none of it asks whether you
+              meant it. The archives themselves are not here: retiring one is
+              its own decision, made from its own panel, with what it seeds in
+              view. Holding <b>${bytes(state.bytes) === EMPTY ? 'nothing' : bytes(state.bytes)}</b>
+              under <code>${escapeHtml(state.dataDir)}</code>.
+            </div>
+            <table>
+              <tbody>${rows}</tbody>
+            </table>
+            <div class="sub" id="storage-note" style="margin-top:0.6rem"></div>`;
+
+          for (const button of panel.querySelectorAll('[data-clear]')) {
+            button.onclick = async () => {
+              const what = button.dataset.clear;
+              button.disabled = true;
+              try {
+                const gone = await api(`/api/storage/${what}`, {
+                  method: 'DELETE',
+                });
+                await draw();
+                const note = panel.querySelector('#storage-note');
+                if (note) {
+                  note.textContent = gone.bytes
+                    ? `Freed ${bytes(gone.bytes)}.`
+                    : 'Cleared.';
+                }
+              } catch (error) {
+                const note = panel.querySelector('#storage-note');
+                if (note) note.textContent = error.message;
+                button.disabled = false;
+              }
+            };
+          }
+        };
+
+        await draw();
+      }
+
       async function renderTokenEditor(into) {
         const panel = document.createElement('div');
         panel.className = 'panel';
@@ -5044,6 +5139,7 @@ Every piece is hashed against the ` +
 
       const SETTINGS_GROUPS = [
         'Serving tiles',
+        'Storage',
         'Feeds',
         'Publishing',
         'Network',
@@ -6323,6 +6419,7 @@ Every piece is hashed against the ` +
         // Tokens are credentials, so they sit with the rest of who may reach
         // this node. Hooks run a command when a download finishes, which is
         // what the Transfers tab is about.
+        await renderStorageEditor(paneFor('Storage'));
         await renderTokenEditor(paneFor('Security'));
         renderHookEditor(paneFor('Transfers'), config, restartKeys, true);
 
@@ -7246,6 +7343,8 @@ Every piece is hashed against the ` +
                 : `<button type="button" data-stack-bake="${escapeHtml(stack.id)}" title="Run this stack over its sources and write the result as a real archive, with its own infohash, seeded like any other.">Export to archive</button>`
           }
           <button type="button" data-stack-edit="${escapeHtml(stack.id)}">Edit</button>
+          <button type="button" data-stack-copy="${escapeHtml(stack.id)}"
+                  title="Opens the editor on a copy of this recipe, under a name of its own. Nothing is saved until you save it.">Duplicate</button>
           <button type="button" data-stack-delete="${escapeHtml(stack.id)}">Delete</button>`;
 
         return `
@@ -7335,6 +7434,26 @@ Every piece is hashed against the ` +
       // space, so a form that re-derived itself from its own inputs would have
       // to know which of them are currently real.
       let stackDraft = null;
+      // Whether the dialog is naming a stack rather than editing one that is
+      // already named, and what names are already spoken for. Held here
+      // because the save is the only place it can be checked: `PUT` upserts,
+      // so a name that is taken replaces what is there rather than refusing.
+      let stackNaming = false;
+      let stackNamesTaken = new Set();
+
+      /**
+       * A name like this one that nothing is using.
+       * @param {string} base - The name being copied.
+       * @param {Set<string>} taken - What is spoken for.
+       * @returns {string} - A free name.
+       */
+      const freeStackId = (base, taken) => {
+        let candidate = `${base}-copy`;
+        for (let n = 2; taken.has(candidate); n += 1) {
+          candidate = `${base}-copy-${n}`;
+        }
+        return candidate;
+      };
 
       /** Forty characters of hex is not a label. */
       const shortHash = (hash) => (hash ? `${hash.slice(0, 12)}…` : hash);
@@ -7343,24 +7462,35 @@ Every piece is hashed against the ` +
       const draftIsRgba = () => stackDraft?.space === 'rgba';
 
       /**
-       * Opens the dialog on a new stack, or on one that already exists.
-       * @param {object} [existing] - The stack to edit.
+       * Opens the dialog on a new stack, on one that already exists, or on a
+       * copy of one.
+       * @param {object} [existing] - The stack to edit, or the copy to name.
+       * @param {object} [options] - `copy` when the draft is a duplicate, and
+       *   `taken` the names already in use.
        * @returns {Promise<void>} - Resolves once the dialog is open.
        */
-      const openStackDialog = async (existing) => {
+      const openStackDialog = async (existing, options = {}) => {
         stackDraft = existing
           ? JSON.parse(JSON.stringify(existing))
           : { id: '', title: '', space: 'elevation', sources: [], output: {} };
         stackDraft.sources ??= [];
         stackDraft.output ??= {};
 
-        $('stack-dialog-title').textContent = existing
-          ? 'Edit stack'
-          : 'Add stack';
+        // A copy arrives filled in but unnamed as far as the node is
+        // concerned, so it is edited like a new stack rather than like the one
+        // it came from.
+        stackNaming = Boolean(options.copy) || !existing;
+        stackNamesTaken = new Set(options.taken ?? []);
+
+        $('stack-dialog-title').textContent = options.copy
+          ? 'Duplicate stack'
+          : existing
+            ? 'Edit stack'
+            : 'Add stack';
         // An id is what the URL is made of, so changing one would silently
         // orphan whatever points at the old address.
         $('stack-id').value = stackDraft.id ?? '';
-        $('stack-id').disabled = Boolean(existing);
+        $('stack-id').disabled = !stackNaming;
         $('stack-title').value = stackDraft.title ?? '';
         for (const radio of document.querySelectorAll(
           'input[name="stack-space"]',
@@ -7737,9 +7867,11 @@ Every piece is hashed against the ` +
                   }>px</option>
                 </select>
                 <span class="sub">${
-                  featherWidth && !fadesAt
-                    ? 'nothing to fade at yet: mask something, or clip it'
-                    : ''
+                  featherWidth && index === 0 && !rgba
+                    ? 'the bottom source has nothing underneath to fade into'
+                    : featherWidth && !fadesAt
+                      ? 'nothing to fade at yet: mask something, or clip it'
+                      : ''
                 }</span>
               </label>
               ${
@@ -7923,6 +8055,30 @@ Every piece is hashed against the ` +
       $('stack-form').addEventListener('submit', async (event) => {
         event.preventDefault();
         const id = $('stack-id').value.trim();
+        // Caught here because the node cannot catch it: a stack with no name
+        // is a `PUT /api/stacks/`, which matches no route at all, and what
+        // comes back says only that the reply was not JSON.
+        if (!id) {
+          $('stack-error').textContent =
+            'A name is needed. It is what the stack is served under, so ' +
+            'there is nowhere to save one without it.';
+          return;
+        }
+        if (!/^[a-z0-9][a-z0-9._-]*$/i.test(id)) {
+          $('stack-error').textContent =
+            'A name may hold letters, digits, dots, dashes and underscores, ' +
+            'and has to start with a letter or a digit — it goes in a URL.';
+          return;
+        }
+        // `PUT` upserts, so a name already in use is replaced rather than
+        // refused -- which for a duplicate means quietly overwriting the very
+        // stack it was copied from.
+        if (stackNaming && stackNamesTaken.has(id)) {
+          $('stack-error').textContent =
+            `There is already a stack called "${id}". A name is what its URL ` +
+            'is made of, so saving would replace it rather than add this one.';
+          return;
+        }
         const body = {
           title: $('stack-title').value.trim() || undefined,
           space: stackDraft.space,
@@ -7988,8 +8144,16 @@ Every piece is hashed against the ` +
         loadStacks().catch((e) => toast(e.message));
       });
 
-      $('stacks-add').onclick = () =>
-        openStackDialog().catch((e) => toast(e.message));
+      $('stacks-add').onclick = async () => {
+        try {
+          const { stacks } = await api('/api/stacks');
+          await openStackDialog(undefined, {
+            taken: stacks.map((one) => one.id),
+          });
+        } catch (error) {
+          toast(error.message);
+        }
+      };
 
       /**
        * Asks where an export should land before starting it.
@@ -8133,6 +8297,7 @@ Every piece is hashed against the ` +
       // change, so they are delegated the same way the source rows are.
       $('stacks-list').addEventListener('click', async (event) => {
         const edit = event.target.dataset.stackEdit;
+        const copy = event.target.dataset.stackCopy;
         const drop = event.target.dataset.stackDelete;
         const bake = event.target.dataset.stackBake;
         const stopBake = event.target.dataset.stackBakeStop;
@@ -8162,6 +8327,23 @@ Every piece is hashed against the ` +
             const raw = await api(`/api/stacks/${encodeURIComponent(edit)}/raw`);
             await openStackDialog(raw.stack);
           }
+        } else if (copy !== undefined) {
+          const { stacks } = await api('/api/stacks');
+          // The recipe, not the report -- the same reason editing reads it.
+          // A report holds what each source resolved to, so a copy made from
+          // one would pin infohashes the original follows by category.
+          const raw = await api(`/api/stacks/${encodeURIComponent(copy)}/raw`);
+          const taken = stacks.map((one) => one.id);
+          await openStackDialog(
+            {
+              ...raw.stack,
+              id: freeStackId(copy, new Set(taken)),
+              // Named apart on purpose: two stacks under one title are two
+              // rows nobody can tell apart in the list they both appear in.
+              title: `${raw.stack.title ?? copy} copy`,
+            },
+            { copy: true, taken },
+          );
         } else if (drop !== undefined) {
           if (!confirm(`Delete the stack "${drop}"?`)) return;
           try {