pmtiles-swarm 0.26.1 → 0.28.0

package/CHANGELOG.md

@@ -7,6 +7,78 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.28.0
+### ✨ Features and improvements
+- **The archive list can be searched and sorted, and says when each archive was added.** `Added` is
+  a column now — `createdAt` has always been on every entry and returned by `/api/catalog`, it was
+  simply never shown — with a date for anything older than today and a time for today, since the
+  question a list answers is which of these is recent rather than exactly when each arrived.
+
+  Beside it, the same filter and sort the public page has: by name, infohash or category, ordered
+  by newest added, oldest added, name or size — and by download speed, upload speed or share
+  ratio, which are read off the live status the poll refreshes, so rows reorder themselves under
+  those every few seconds. That is what a torrent client does and what choosing "download speed"
+  asks for, and also why none of them is the default. Newest added is, because a list read
+  straight after adding something should have that thing at the top.
+
+  Both happen inside the render rather than where the data arrives. The list refreshes every three
+  seconds; filtering at the fetch would either clear what had been typed on each poll or refetch
+  the whole catalog on every keystroke. And the count beside the box says `4 of 37` while a filter
+  is narrowing things, because an empty table and a table filtered down to nothing look identical.
+
+### 🐞 Bug fixes
+
+## 0.27.0
+### ✨ Features and improvements
+- **Bandwidth history per archive, kept across restarts.** The tile side of this question already
+
+  The Traffic tab draws it: one chart for the node with a window of an hour, a day or a week, and
+  a table of what each archive moved over that window. Drawn as inline SVG rather than through a
+  charting library — the console is a single self-contained file the node serves itself, and a
+  dependency for two lines would have to be vendored, kept current, and shipped on every page load
+  for a panel most visits never open. Both lines share one scale, because drawing each against its
+  own maximum renders a node uploading 2 KB/s and downloading 200 MB/s as two similar lines, which
+  is the opposite of what a chart is for.
+  had an answer; the swarm side had none. An archive could seed steadily for a day and leave no
+  trace but a speed in the console that is gone the moment you look away, which makes "what is
+  using the bandwidth" and "is this archive earning its disk" unanswerable.
+
+  Upload and download speed are now sampled per archive on a timer and kept in `stats.db`, beside
+  the catalog in `dataDir` — not beside the config, which is the operator's: hand-edited, diffed,
+  copied between nodes, and the thing you reach for when a node will not start. A database that
+  grows on its own does not belong there. `node:sqlite` is built in and already used for MBTiles,
+  so this costs no dependency, and it is imported lazily for the same reason `mbtiles.js` does it:
+  the first require prints an experimental warning nobody with this switched off should have to
+  explain.
+
+  Persisted rather than held in memory, unlike `tileStats`, because it answers a question about
+  the past — restarting to pick up a new version would erase exactly the week somebody wanted to
+  look at. Application logs stay in the journal; this is only for numbers that have to survive a
+  restart.
+
+  Two settings, because they are two questions: `traffic.sampleSeconds` is how finely it looks and
+  `traffic.keepHours` how far back it remembers, defaulting to every 15 seconds for a week. Read
+  back through `GET /api/traffic`, averaged into buckets so a week of samples is a graph rather
+  than forty thousand points, with `totals` ranking archives by bytes moved. `traffic: false`
+  turns the whole thing off, and a database that cannot be opened is reported rather than fatal —
+  a node that cannot record what it moved should still move it.
+
+### 🐞 Bug fixes
+
+## 0.26.2
+### 🐞 Bug fixes
+- **The public page's sort left the categories alone.** `apply()` sorted the archives and handed
+  the categories straight to the renderer, which does not sort either — so on a node carrying
+  twenty categories and a few archives the control looked completely dead. Categories are rendered
+  first and are the list worth reading, since they follow the newest build, so sorting only the
+  other list amounted to not sorting at all for most visitors.
+
+  All three orderings now apply to both, against the fields a category actually keeps: its own
+  name, and the date and size of the build it points at. The existing test could not have caught
+  this — it checked that every option in the select had a comparator behind it, which was true, and
+  said nothing about whether the categories were ever handed to one. The comparators are now lifted
+  out of the page and called directly.
+
 ## 0.26.1
 ### 🐞 Bug fixes
 - **A download that finished was fetched all over again.** Resuming looked only at the

package/README.md

@@ -770,6 +770,7 @@ which the endpoint answers 501.
 | `GET` `POST` `DELETE` | `/api/tokens`, `/api/tokens/:id`                            | Mint, list and revoke access tokens                                                                                                                                                                                                       |
 | `GET` `POST`          | `/api/restart`                                              | What a restart would do, and doing it                                                                                                                                                                                                     |
 | `GET` `DELETE`        | `/api/stats`                                                | What this node has served — per-archive counters and the last N requests; `DELETE` clears them                                                                                                                                            |
+| `GET`                 | `/api/traffic`                                              | Upload and download speed per archive over time, from `stats.db`; `infoHash`, `hours` and `buckets` narrow it, and `totals` ranks archives by bytes moved                                                                                 |
 | `GET` `PATCH`         | `/api/config`                                               | Read and change settings                                                                                                                                                                                                                  |
 | `POST`                | `/api/login`, `/api/logout`                                 | Console sign-in                                                                                                                                                                                                                           |
 | `GET`                 | `/api/session`                                              | Who this request is, and whether a credential is needed at all                                                                                                                                                                            |

package/docs/running-as-a-service.md

@@ -354,6 +354,62 @@ What the running service actually has:
 systemctl show -p ReadWritePaths -p UMask -p SupplementaryGroups pmtiles-swarm
 ```
 
+## The statistics database
+
+Upload and download speed per archive is sampled on a timer and kept in
+`stats.db`, in `dataDir` beside the catalog. It is on by default.
+
+**It needs no new path.** `dataDir` is already in `ReadWritePaths` — the
+catalog is written there on every change — so a unit that works today works
+with this. Worth knowing rather than checking: the file appears on its own the
+first time the node starts, with no migration step and nothing to create by
+hand.
+
+SQLite writes a journal beside the database while a transaction is open, so it
+needs the **directory** writable and not only the file. That is the same
+requirement the catalog already has, and the same failure if it is missing —
+see [Two checks that lie](#two-checks-that-lie), which applies here unchanged.
+
+### The warning in the journal is expected
+
+```
+ExperimentalWarning: SQLite is an experimental feature and might change at any time
+```
+
+`node:sqlite` prints this the first time it is loaded. It is Node's warning
+about its own module, not a statement about this node's data, and it appears
+once at startup. It is mentioned here because a new warning in the journal
+immediately after an upgrade is exactly the shape of a real fault, and it is
+worth being able to dismiss it without an investigation.
+
+### How large it gets
+
+Bounded by the retention window, not by how long the node has been running:
+
+```
+rows  =  archives  x  keepHours x 3600 / sampleSeconds
+```
+
+At the defaults — every 15 seconds, kept for a week — that is about 40,000 rows
+per archive, so a node carrying twenty is a few megabytes. Samples past the
+window are deleted every ten minutes.
+
+SQLite does not return freed pages to the filesystem on its own, so the file
+settles at roughly its high-water mark rather than shrinking after a large
+retention cut. `VACUUM` reclaims it if that ever matters; at these sizes it
+will not.
+
+### Turning it down, or off
+
+```json
+{ "traffic": { "sampleSeconds": 60, "keepHours": 24 } }
+```
+
+Both are reloaded without a restart. `"traffic": false` switches it off
+entirely, and a database that cannot be opened is reported and stepped over
+rather than being fatal — a node that cannot record what it moved should still
+move it.
+
 ## The publisher key
 
 Only needed if this node publishes BEP 46 records — the signed DHT entries that

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.26.1",
+  "version": "0.28.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -105,6 +105,7 @@ export function createApp({
   config,
   speed,
   stats,
+  traffic,
   reloaders = {},
   shutdown,
 }) {
@@ -556,6 +557,33 @@ export function createApp({
   // is healthy. Admin-side: it names archives and client addresses, and a
   // public endpoint reporting who else is using a node is a privacy question
   // nobody asked for.
+  // What the swarm moved, over time. The tile side of this is /api/stats;
+  // this is the half that was invisible -- an archive could seed steadily for
+  // a day and leave no trace but a speed that vanishes when you look away.
+  app.get(
+    '/api/traffic',
+    route(async (req, res) => {
+      if (!traffic) {
+        return res
+          .status(501)
+          .json({ error: 'traffic statistics are disabled' });
+      }
+      const number = (value) =>
+        value === undefined ? undefined : Number(value);
+      const hours = number(req.query.hours);
+      const buckets = number(req.query.buckets);
+      res.json({
+        sampleSeconds: traffic.sampleSeconds,
+        keepHours: traffic.keepHours,
+        // One archive when asked for, every archive summed when not.
+        ...traffic.series({ infoHash: req.query.infoHash, hours, buckets }),
+        // Beside the series rather than behind a second request: who used the
+        // bandwidth is asked at the same moment as how much of it there was.
+        totals: traffic.totals({ hours }),
+      });
+    }),
+  );
+
   app.get(
     '/api/stats',
     route(async (req, res) => {

package/src/config.js

@@ -216,6 +216,25 @@ const DEFAULTS = {
     /** Recent requests kept for inspection. Zero keeps only the counters. */
     recent: 200,
   },
+  /**
+   * Upload and download speed per archive, sampled on a timer and kept in
+   * `stats.db` beside the catalog. `false` turns it off.
+   *
+   * Persisted rather than held in memory, unlike tileStats, because it answers
+   * a question about the past: restarting to pick up a new version would erase
+   * exactly the week somebody wanted to look at. Application logs stay in the
+   * journal -- this is only for numbers that have to survive a restart.
+   *
+   * Two settings because they are two questions: how finely it looks, and how
+   * far back it remembers. Sampling every 15 seconds for a week is roughly
+   * 40,000 rows per archive, a few megabytes for a node carrying twenty.
+   */
+  traffic: {
+    /** Seconds between samples. */
+    sampleSeconds: 15,
+    /** How far back to keep them, in hours. */
+    keepHours: 168,
+  },
   /**
    * Tile serving: a TileJSON endpoint and z/x/y tiles per archive. A node
    * holding a complete copy reads its local file; one in cache mode reads
@@ -583,6 +602,7 @@ export const RELOADABLE = new Map([
   // Read when something is added rather than held open, so a change applies to
   // the next add with nothing to restart.
   ['locations', 'none'],
+  ['traffic', 'traffic'],
   ['watch', 'watchers'],
   ['onAdded', 'hooks'],
   ['onComplete', 'hooks'],

package/src/index.js

@@ -20,6 +20,7 @@ import { closeServer, installSignalHandlers, runStoppers } from './shutdown.js';
 import { ScheduledSourceManager } from './sources.js';
 import { SubscriptionManager } from './subscriptions.js';
 import { TileStats } from './tile-stats.js';
+import { TrafficStats, openStatsDatabase } from './traffic-stats.js';
 import { TileStore } from './tiles.js';
 import { HeadWarmer } from './prewarm.js';
 import { WarmRunner } from './warm.js';
@@ -271,6 +272,29 @@ PMTILES_SWARM_PUBLIC_URL
       ? null
       : new TileStats({ recent: config.tileStats?.recent });
 
+  // What the swarm moved, as opposed to what the tile endpoint served.
+  // Persisted, because it answers a question about the past: restarting to
+  // pick up a new version would erase the week somebody wanted to look at.
+  let traffic = null;
+  if (config.traffic !== false) {
+    try {
+      traffic = new TrafficStats({
+        db: await openStatsDatabase(config),
+        engine,
+        config,
+      });
+      traffic.start();
+      console.log(
+        `[traffic] sampling every ${traffic.sampleSeconds}s, keeping ` +
+          `${traffic.keepHours}h`,
+      );
+    } catch (error) {
+      // Never fatal. A node that cannot record what it moved should still
+      // move it, and the alternative is refusing to start over a graph.
+      console.error(`[traffic] not recording: ${error.message}`);
+    }
+  }
+
   // Announcing the current build of each category over the DHT. Only ever on
   // the node that builds: the key signs what subscribers believe is current,
   // and two publishers under one key would fight over the sequence number.
@@ -364,6 +388,13 @@ PMTILES_SWARM_PUBLIC_URL
       watch.stop();
       watch.start(config.watch);
     },
+    traffic: () => {
+      // Rebuilt rather than restarted: sampleSeconds and keepHours are both
+      // read when the sampler is constructed, so this is the whole of what a
+      // restart would have achieved.
+      traffic?.stop();
+      traffic?.start();
+    },
     hooks: () => {
       hooks.stop();
       hooks.start();
@@ -402,6 +433,7 @@ PMTILES_SWARM_PUBLIC_URL
     config,
     speed,
     stats,
+    traffic,
     reloaders,
     shutdown: () => runStoppers(stoppers),
   });
@@ -537,6 +569,7 @@ PMTILES_SWARM_PUBLIC_URL
         subscriptions.stop();
         warm.stop();
         headWarmer.stop();
+        traffic?.stop();
       },
       ms: 1000,
     },

package/src/traffic-stats.js

@@ -0,0 +1,306 @@
+/**
+ * What each archive has been sending and receiving, over time.
+ *
+ * The tile side of this question already has an answer: TileStats records what
+ * this node served over HTTP. The swarm side had none. An archive could be
+ * uploading steadily for a day and the only evidence was a speed in the console
+ * that is gone the moment you look away — which makes "what is actually using
+ * the bandwidth" and "is this archive earning its disk" unanswerable.
+ *
+ * Kept in SQLite rather than in memory, unlike TileStats, and the difference is
+ * deliberate. Tile counters answer a question about now and are cheap to
+ * rebuild by waiting; a bandwidth history answers a question about the past and
+ * cannot be rebuilt at all — restarting to pick up a new version would erase
+ * exactly the week somebody wanted to look at. `node:sqlite` is built in and
+ * already used for MBTiles, so this costs no dependency.
+ *
+ * Two knobs, because they are two questions. `sampleSeconds` is how finely it
+ * looks; `keepHours` is how far back it remembers. Sampling every 15 seconds
+ * for a week is roughly 40,000 rows per archive — a few megabytes for a node
+ * carrying twenty, which is worth it for being able to answer at all.
+ */
+
+import path from 'node:path';
+
+/** How often to take a sample, when the config says nothing. */
+const DEFAULT_SAMPLE_SECONDS = 15;
+
+/** How far back to keep samples, when the config says nothing. */
+const DEFAULT_KEEP_HOURS = 168;
+
+/** How often to delete samples that have aged out. */
+const PRUNE_EVERY_MS = 10 * 60 * 1000;
+
+/**
+ * Rounds a retention window into a sensible number of buckets for a graph.
+ *
+ * A week of 15-second samples is 40,000 points and a chart a thousand pixels
+ * wide; sending all of them wastes the transfer and the browser's time to draw
+ * something no eye can resolve. Averaging into buckets keeps the shape.
+ *
+ * @param {number} from - Start of the window, unix seconds.
+ * @param {number} to - End of the window, unix seconds.
+ * @param {number} buckets - How many points are wanted.
+ * @returns {number} - Seconds per bucket, never less than one.
+ */
+export function bucketSeconds(from, to, buckets) {
+  const span = Math.max(1, to - from);
+  return Math.max(1, Math.round(span / Math.max(1, buckets)));
+}
+
+/** Per-archive upload and download speed, sampled on a timer and retained. */
+export class TrafficStats {
+  #db;
+  #insert;
+  #timer;
+  #pruneTimer;
+  #engine;
+  #now;
+  #sampleMs;
+  #keepSeconds;
+
+  /**
+   * @param {object} options - Options.
+   * @param {object} options.db - An open node:sqlite DatabaseSync.
+   * @param {object} options.engine - The engine to sample.
+   * @param {object} [options.config] - Resolved configuration.
+   * @param {Function} [options.now] - Clock returning unix seconds, for tests.
+   */
+  constructor({
+    db,
+    engine,
+    config = {},
+    now = () => Math.floor(Date.now() / 1000),
+  }) {
+    this.#db = db;
+    this.#engine = engine;
+    this.#now = now;
+
+    const sample = Number(config.traffic?.sampleSeconds);
+    this.#sampleMs =
+      (Number.isFinite(sample) && sample > 0
+        ? sample
+        : DEFAULT_SAMPLE_SECONDS) * 1000;
+    const keep = Number(config.traffic?.keepHours);
+    this.#keepSeconds =
+      (Number.isFinite(keep) && keep > 0 ? keep : DEFAULT_KEEP_HOURS) * 3600;
+
+    this.#db.exec(`
+      CREATE TABLE IF NOT EXISTS traffic (
+        info_hash TEXT NOT NULL,
+        at INTEGER NOT NULL,
+        down INTEGER NOT NULL,
+        up INTEGER NOT NULL
+      );
+      CREATE INDEX IF NOT EXISTS traffic_at ON traffic (at);
+      CREATE INDEX IF NOT EXISTS traffic_hash_at ON traffic (info_hash, at);
+    `);
+    this.#insert = this.#db.prepare(
+      'INSERT INTO traffic (info_hash, at, down, up) VALUES (?, ?, ?, ?)',
+    );
+  }
+
+  /** @returns {number} - Seconds between samples. */
+  get sampleSeconds() {
+    return this.#sampleMs / 1000;
+  }
+
+  /** @returns {number} - How far back samples are kept, in hours. */
+  get keepHours() {
+    return this.#keepSeconds / 3600;
+  }
+
+  /**
+   * Takes one sample of every torrent the engine knows about.
+   *
+   * Zero rows are written as readily as busy ones: a gap in the series would
+   * be indistinguishable from the node having been switched off, and "this
+   * archive did nothing all week" is a real answer somebody wants.
+   *
+   * @returns {Promise<number>} - How many rows were written.
+   */
+  async sample() {
+    let torrents;
+    try {
+      torrents = await this.#engine.list();
+    } catch (error) {
+      // A sampler that throws takes its timer down with it and the history
+      // simply stops, which is the one failure that cannot be noticed later.
+      console.warn(`[traffic] could not sample: ${error.message}`);
+      return 0;
+    }
+
+    const at = this.#now();
+    let written = 0;
+    for (const torrent of torrents ?? []) {
+      if (!torrent?.infoHash) continue;
+      this.#insert.run(
+        String(torrent.infoHash).toLowerCase(),
+        at,
+        Math.max(0, Math.round(torrent.downloadSpeed ?? 0)),
+        Math.max(0, Math.round(torrent.uploadSpeed ?? 0)),
+      );
+      written += 1;
+    }
+    return written;
+  }
+
+  /**
+   * Deletes samples that have aged past the retention window.
+   * @returns {number} - Rows removed.
+   */
+  prune() {
+    const cutoff = this.#now() - this.#keepSeconds;
+    const result = this.#db
+      .prepare('DELETE FROM traffic WHERE at < ?')
+      .run(cutoff);
+    return Number(result?.changes ?? 0);
+  }
+
+  /**
+   * The series for one archive, or for every archive summed.
+   *
+   * Averaged into buckets rather than returned raw — see bucketSeconds. The
+   * timestamp of a bucket is its start, so a caller plotting them gets evenly
+   * spaced points without having to know the sample interval.
+   *
+   * @param {object} [options] - Options.
+   * @param {string} [options.infoHash] - One archive, or every one summed.
+   * @param {number} [options.hours] - How far back to read.
+   * @param {number} [options.buckets] - How many points to return.
+   * @returns {object} - `{from, to, seconds, points}`.
+   */
+  series({ infoHash, hours, buckets = 240 } = {}) {
+    const to = this.#now();
+    const span =
+      Number.isFinite(hours) && hours > 0 ? hours * 3600 : this.#keepSeconds;
+    const from = to - span;
+    const seconds = bucketSeconds(from, to, buckets);
+
+    // Floored with a modulo rather than by dividing and multiplying back:
+    // node:sqlite binds a JS number as REAL, so `at / 900` is float division
+    // and every row lands in a bucket of its own -- a week of samples came
+    // back as a week of samples. SQLite's `%` casts to integer, which is
+    // exactly the flooring wanted.
+    //
+    // Summed across archives inside a bucket and then averaged over the
+    // buckets' samples: at one moment the node's throughput is the sum of what
+    // every torrent is doing, and over time it is the mean of those moments.
+    const rows = infoHash
+      ? this.#db
+          .prepare(
+            `SELECT at - (at % ?) AS bucket,
+                    AVG(down) AS down, AVG(up) AS up
+               FROM traffic
+              WHERE info_hash = ? AND at >= ?
+              GROUP BY bucket ORDER BY bucket`,
+          )
+          .all(seconds, String(infoHash).toLowerCase(), from)
+      : this.#db
+          .prepare(
+            `SELECT bucket, AVG(down) AS down, AVG(up) AS up FROM (
+               SELECT at - (at % ?) AS bucket, at,
+                      SUM(down) AS down, SUM(up) AS up
+                 FROM traffic WHERE at >= ?
+                GROUP BY at
+             ) GROUP BY bucket ORDER BY bucket`,
+          )
+          .all(seconds, from);
+
+    return {
+      from,
+      to,
+      seconds,
+      points: rows.map((row) => ({
+        at: Number(row.bucket),
+        down: Math.round(Number(row.down) || 0),
+        up: Math.round(Number(row.up) || 0),
+      })),
+    };
+  }
+
+  /**
+   * Totals per archive over a window, for ranking who used what.
+   *
+   * Speeds are a rate, so bytes are the rate multiplied by the interval it was
+   * sampled over. Approximate by construction -- it assumes each sample held
+   * until the next -- and the right shape of approximate: it cannot drift from
+   * the graph above it, because it is the same numbers.
+   *
+   * @param {object} [options] - Options.
+   * @param {number} [options.hours] - How far back to read.
+   * @returns {object[]} - `{infoHash, down, up, samples}`, busiest first.
+   */
+  totals({ hours } = {}) {
+    const span =
+      Number.isFinite(hours) && hours > 0 ? hours * 3600 : this.#keepSeconds;
+    const from = this.#now() - span;
+    const every = this.sampleSeconds;
+    return this.#db
+      .prepare(
+        `SELECT info_hash AS infoHash,
+                SUM(down) * ? AS down, SUM(up) * ? AS up,
+                COUNT(*) AS samples
+           FROM traffic WHERE at >= ?
+          GROUP BY info_hash
+          ORDER BY (SUM(down) + SUM(up)) DESC`,
+      )
+      .all(every, every, from)
+      .map((row) => ({
+        infoHash: row.infoHash,
+        down: Math.round(Number(row.down) || 0),
+        up: Math.round(Number(row.up) || 0),
+        samples: Number(row.samples) || 0,
+      }));
+  }
+
+  /** Starts sampling and pruning. @returns {void} */
+  start() {
+    if (this.#timer) return;
+    const tick = () =>
+      this.sample().catch((error) =>
+        console.warn(`[traffic] sample failed: ${error.message}`),
+      );
+    this.#timer = setInterval(tick, this.#sampleMs);
+    this.#timer.unref?.();
+
+    // Pruned on a slow timer of its own rather than after every sample: it is a
+    // whole-table delete and the sample runs every few seconds.
+    this.#pruneTimer = setInterval(() => {
+      try {
+        this.prune();
+      } catch (error) {
+        console.warn(`[traffic] could not prune: ${error.message}`);
+      }
+    }, PRUNE_EVERY_MS);
+    this.#pruneTimer.unref?.();
+  }
+
+  /** Stops sampling. @returns {void} */
+  stop() {
+    if (this.#timer) clearInterval(this.#timer);
+    if (this.#pruneTimer) clearInterval(this.#pruneTimer);
+    this.#timer = undefined;
+    this.#pruneTimer = undefined;
+  }
+}
+
+/**
+ * Opens the stats database, beside the catalog rather than beside the config.
+ *
+ * dataDir is where this node's own mutable state already lives -- the catalog,
+ * resume data, the DHT node cache. The config directory is the operator's:
+ * hand-edited, diffed, copied between nodes, and the thing you reach for when a
+ * node will not start. A database that grows on its own does not belong there.
+ *
+ * @param {object} config - Resolved configuration.
+ * @returns {Promise<object>} - An open DatabaseSync.
+ */
+export async function openStatsDatabase(config) {
+  // Imported here rather than at module load because node:sqlite prints an
+  // ExperimentalWarning the first time it is required, and a node with stats
+  // switched off should not be made to explain that warning. Same reasoning as
+  // mbtiles.js.
+  const { DatabaseSync } = await import('node:sqlite');
+  return new DatabaseSync(path.join(config.dataDir, 'stats.db'));
+}

package/src/web/index.html

@@ -324,12 +324,36 @@
             <button type="button">Feed</button>
           </a>
         </div>
+        <div class="bar">
+          <input
+            id="archive-filter"
+            type="search"
+            placeholder="Filter by name, infohash or category"
+            autocomplete="off"
+            spellcheck="false"
+            style="flex:1;min-width:12rem"
+          />
+          <label>
+            Sort
+            <select id="archive-sort">
+              <option value="added">newest added</option>
+              <option value="oldest">oldest added</option>
+              <option value="name">name</option>
+              <option value="largest">largest</option>
+              <option value="down">download speed</option>
+              <option value="up">upload speed</option>
+              <option value="ratio">share ratio</option>
+            </select>
+          </label>
+          <span class="sub" id="archive-count"></span>
+        </div>
 
         <table>
           <thead>
             <tr>
               <th>Archive</th>
               <th>Size</th>
+              <th>Added</th>
               <th>Mode</th>
               <th>Origin</th>
               <th>Progress</th>
@@ -375,6 +399,18 @@
           </label>
           <button id="traffic-reset" style="margin-left:auto">Reset counters</button>
         </div>
+        <div class="bar">
+          <label>
+            Swarm bandwidth
+            <select id="traffic-window">
+              <option value="1">last hour</option>
+              <option value="24" selected>last 24 hours</option>
+              <option value="168">last week</option>
+            </select>
+          </label>
+          <span class="sub" id="traffic-window-note"></span>
+        </div>
+        <div id="swarm-traffic"><div class="sub">loading…</div></div>
         <div id="traffic-body"><div class="sub">loading…</div></div>
       </section>
 
@@ -698,6 +734,56 @@
       const rate = (n) => (n > 0 ? `${bytes(n)}/s` : '—');
       const pct = (n) => `${Math.round((n ?? 0) * 100)}%`;
 
+      /**
+       * When an archive was added, as something short enough for a column.
+       *
+       * A date on its own for anything older than today, and a time for
+       * today: the question a list answers is "which of these is recent", and
+       * a full timestamp on every row is harder to scan than either.
+       * @param {string} value - ISO timestamp.
+       * @returns {string} - Markup for the cell.
+       */
+      const added = (value) => {
+        if (!value) return '<span class="sub">—</span>';
+        const when = new Date(value);
+        if (Number.isNaN(when.getTime())) return '<span class="sub">—</span>';
+        const today = new Date().toDateString() === when.toDateString();
+        return `<span title="${escapeHtml(when.toLocaleString())}">${
+          today
+            ? when.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' })
+            : when.toLocaleDateString()
+        }</span>`;
+      };
+
+      // The same three questions the public page asks, against an admin row.
+      // Newest first by default: a list read after adding something is a list
+      // where the thing just added should be at the top.
+      /**
+       * A live figure off an archive's status, or zero.
+       *
+       * An archive the engine has not reported on yet has no status at all,
+       * and comparing undefined gives NaN -- which makes a sort silently do
+       * nothing rather than fail, so it is worth the guard.
+       * @param {object} entry - Catalog entry with its status attached.
+       * @param {string} field - Which figure.
+       * @returns {number} - The value, or zero.
+       */
+      const speedOf = (entry, field) => Number(entry?.status?.[field] ?? 0) || 0;
+
+      const archiveSorters = {
+        added: (a, b) => new Date(b.createdAt ?? 0) - new Date(a.createdAt ?? 0),
+        oldest: (a, b) => new Date(a.createdAt ?? 0) - new Date(b.createdAt ?? 0),
+        name: (a, b) => String(a.name ?? '').localeCompare(String(b.name ?? '')),
+        largest: (a, b) => (b.size ?? 0) - (a.size ?? 0),
+        // Live values, read off the status the poll refreshes. Rows reorder
+        // themselves every few seconds under these, which is what a torrent
+        // client does and what somebody choosing "download speed" is asking
+        // for -- but it is also why they are not the default.
+        down: (a, b) => speedOf(b, 'downloadSpeed') - speedOf(a, 'downloadSpeed'),
+        up: (a, b) => speedOf(b, 'uploadSpeed') - speedOf(a, 'uploadSpeed'),
+        ratio: (a, b) => speedOf(b, 'ratio') - speedOf(a, 'ratio'),
+      };
+
       let toastTimer;
       const toast = (message) => {
         $('toast').textContent = message;
@@ -992,9 +1078,34 @@
       function renderRows() {
         const rows = $('rows');
         rows.innerHTML = '';
+
+        // Filtered and sorted here rather than where the data arrives, so the
+        // three-second poll keeps refreshing what is on screen without
+        // resetting what was typed. `archives` stays the node's answer; this
+        // is only the view of it.
+        const needle = ($('archive-filter')?.value ?? '').trim().toLowerCase();
+        const order = $('archive-sort')?.value ?? 'added';
+        const shown = archives
+          .filter(
+            (entry) =>
+              needle === '' ||
+              String(entry.name ?? '').toLowerCase().includes(needle) ||
+              String(entry.infoHash ?? '').toLowerCase().includes(needle) ||
+              (entry.categories ?? []).some((tag) =>
+                String(tag).toLowerCase().includes(needle),
+              ),
+          )
+          .sort(archiveSorters[order] ?? archiveSorters.added);
+
         $('empty').hidden = archives.length > 0;
+        // Says when a filter is hiding things, because an empty table and a
+        // table filtered down to nothing look identical otherwise.
+        $('archive-count').textContent =
+          needle === ''
+            ? ''
+            : `${shown.length} of ${archives.length}`;
 
-        for (const entry of archives) {
+        for (const entry of shown) {
           const s = entry.status ?? {};
           const progress = s.progress ?? 0;
           const mode = entry.mode ?? 'mirror';
@@ -1012,6 +1123,7 @@
                 ? `<div class="sub">${escapeHtml(entry.kind)} · not servable</div>`
                 : ''
             }</td>
+            <td>${added(entry.createdAt)}</td>
             <td><span class="pill ${mode}">${mode}</span></td>
             <td>${originCell(entry)}</td>
             <td>
@@ -3919,7 +4031,10 @@
         }
         if (name === 'settings') loadSettings().catch((e) => toast(e.message));
         if (name === 'categories') loadCategories().catch((e) => toast(e.message));
-        if (name === 'traffic') loadTraffic().catch((e) => toast(e.message));
+        if (name === 'traffic') {
+          loadTraffic().catch((e) => toast(e.message));
+          loadSwarmTraffic();
+        }
       };
       // The footer's year, set from the clock rather than typed into a file
       // nobody will remember to edit. The version beside it arrives with the
@@ -3961,6 +4076,123 @@
        * Reads the report and draws it.
        * @returns {Promise<void>} - Resolves once drawn.
        */
+      /**
+       * Draws a series as an SVG path, scaled to a shared maximum.
+       *
+       * Inline rather than through a charting library. The console is one
+       * self-contained file served by the node itself, and pulling in a
+       * dependency to draw two lines would be a larger decision than the
+       * feature -- it would have to be vendored, kept current, and would arrive
+       * on every page load for a panel most visits never open.
+       *
+       * @param {object[]} points - The series.
+       * @param {string} key - Which field to plot.
+       * @param {number} peak - Value at the top of the chart, shared by both.
+       * @param {number} width - Viewbox width.
+       * @param {number} height - Viewbox height.
+       * @returns {string} - A path's `d` attribute.
+       */
+      function trafficPath(points, key, peak, width, height) {
+        if (points.length === 0) return '';
+        const step = points.length > 1 ? width / (points.length - 1) : width;
+        return points
+          .map((point, index) => {
+            const x = (index * step).toFixed(1);
+            // Guarded against a peak of zero: an idle week is a flat line along
+            // the bottom, not a division by zero that renders nothing at all.
+            const y = (height - (peak ? (point[key] / peak) * height : 0)).toFixed(1);
+            return `${index === 0 ? 'M' : 'L'}${x} ${y}`;
+          })
+          .join(' ');
+      }
+
+      /**
+       * The whole-node bandwidth chart, and what each archive contributed.
+       * @param {object} report - From /api/traffic.
+       * @returns {string} - Markup.
+       */
+      function renderSwarmTraffic(report) {
+        const points = report.points ?? [];
+        if (points.length === 0) {
+          return `<div class="panel"><div class="sub">
+            Nothing recorded yet. Samples are taken every
+            ${report.sampleSeconds}s, so a chart appears once a couple have
+            been.</div></div>`;
+        }
+
+        // One scale for both lines. Plotting each against its own maximum
+        // would draw a node uploading 2 KB/s and downloading 200 MB/s as two
+        // similar-looking lines, which is the opposite of what a chart is for.
+        const peak = Math.max(
+          1,
+          ...points.map((point) => Math.max(point.down, point.up)),
+        );
+        const W = 1000;
+        const H = 160;
+        const down = trafficPath(points, 'down', peak, W, H);
+        const up = trafficPath(points, 'up', peak, W, H);
+        const span = (report.to - report.from) / 3600;
+
+        const rows = (report.totals ?? [])
+          .map((row) => {
+            const held = archives.find((a) => a.infoHash === row.infoHash);
+            const name = held?.name ?? `${row.infoHash.slice(0, 12)}…`;
+            return `<tr>
+              <td><b>${escapeHtml(name)}</b>
+                <div class="sub">${escapeHtml(row.infoHash.slice(0, 12))}…</div></td>
+              <td>${bytes(row.down)}</td>
+              <td>${bytes(row.up)}</td>
+            </tr>`;
+          })
+          .join('');
+
+        return `<div class="panel">
+          <svg viewBox="0 0 ${W} ${H}" preserveAspectRatio="none"
+               style="width:100%;height:160px;display:block"
+               role="img" aria-label="upload and download over the last ${span.toFixed(0)} hours">
+            <path d="${down}" fill="none" stroke="var(--accent)" stroke-width="2"
+                  vector-effect="non-scaling-stroke" />
+            <path d="${up}" fill="none" stroke="var(--ok)" stroke-width="2"
+                  vector-effect="non-scaling-stroke" />
+          </svg>
+          <div class="sub" style="display:flex;gap:1rem;flex-wrap:wrap">
+            <span><b style="color:var(--accent)">—</b> down</span>
+            <span><b style="color:var(--ok)">—</b> up</span>
+            <span>peak ${bytes(peak)}/s</span>
+            <span>${points.length} points, ${report.seconds}s each</span>
+          </div>
+        </div>
+        ${
+          rows
+            ? `<div class="panel"><table class="rows">
+                 <thead><tr><th>Archive</th><th>Downloaded</th><th>Uploaded</th></tr></thead>
+                 <tbody>${rows}</tbody></table>
+               <div class="sub">Bytes are the sampled speed held across each
+                 interval, so they follow the chart above rather than the
+                 engine's own counters.</div></div>`
+            : ''
+        }`;
+      }
+
+      /** Loads and draws the swarm bandwidth panel. @returns {Promise<void>} */
+      async function loadSwarmTraffic() {
+        const host = $('swarm-traffic');
+        const hours = Number($('traffic-window').value) || 24;
+        try {
+          const report = await api(`/api/traffic?hours=${hours}`);
+          $('traffic-window-note').textContent =
+            `sampled every ${report.sampleSeconds}s, kept for ${report.keepHours}h`;
+          host.innerHTML = renderSwarmTraffic(report);
+        } catch (error) {
+          if (error.message?.includes('disabled')) {
+            host.innerHTML =
+              '<div class="panel"><div class="sub">Swarm statistics are turned off on this node — set <code>traffic</code> in the configuration to enable them.</div></div>';
+            return;
+          }
+          host.innerHTML = `<div class="panel"><div class="sub">Could not load: ${escapeHtml(error.message)}</div></div>`;
+        }
+      }
+
       async function loadTraffic() {
         const body = $('traffic-body');
         let report;
@@ -4058,14 +4290,23 @@
         clearInterval(trafficTimer);
         if (!$('traffic-live').checked) return;
         trafficTimer = setInterval(() => {
-          if (!$('view-traffic').hidden) loadTraffic().catch(() => {});
+          if (!$('view-traffic').hidden) {
+            loadTraffic().catch(() => {});
+            loadSwarmTraffic();
+          }
         }, 5000);
       };
 
-      $('traffic-refresh').onclick = () => loadTraffic().catch((e) => toast(e.message));
+      $('traffic-refresh').onclick = () => {
+        loadTraffic().catch((e) => toast(e.message));
+        loadSwarmTraffic();
+      };
       $('traffic-live').onchange = trafficPoll;
       trafficPoll();
 
+      $('archive-filter').addEventListener('input', renderRows);
+      $('archive-sort').addEventListener('change', renderRows);
+      $('traffic-window').onchange = () => loadSwarmTraffic();
       $('traffic-reset').onclick = async () => {
         // Deliberately separate from reading: a page polling the endpoint must
         // never be able to erase the history it is drawing.

package/src/web/public.html

@@ -482,6 +482,23 @@
         largest: (a, b) => (b.size ?? 0) - (a.size ?? 0),
       };
 
+      // The same three orderings against a category, which keeps its facts in a
+      // different shape: its own name, and the build it currently points at.
+      //
+      // Sorting only the archives was the same thing as not sorting at all for
+      // most visitors. Categories are the list worth reading -- they follow the
+      // newest build, which is what a style should hold -- so they are rendered
+      // first and are what somebody watches while changing the control. A node
+      // carrying twenty of them and a handful of archives looked completely
+      // unresponsive to it.
+      const categorySorters = {
+        name: (a, b) =>
+          String(a.category ?? '').localeCompare(String(b.category ?? '')),
+        newest: (a, b) =>
+          new Date(b.newest?.createdAt ?? 0) - new Date(a.newest?.createdAt ?? 0),
+        largest: (a, b) => (b.newest?.size ?? 0) - (a.newest?.size ?? 0),
+      };
+
       // Re-renders both lists against the current filter and sort.
       const apply = () => {
         const needle = (document.getElementById('filter').value ?? '')
@@ -499,12 +516,14 @@
           )
           .sort(sorters[order] ?? sorters.name);
 
-        const categories = loadedCategories.filter(
-          (entry) =>
-            needle === '' ||
-            matches(entry.category, needle) ||
-            matches(entry.newest?.name, needle),
-        );
+        const categories = loadedCategories
+          .filter(
+            (entry) =>
+              needle === '' ||
+              matches(entry.category, needle) ||
+              matches(entry.newest?.name, needle),
+          )
+          .sort(categorySorters[order] ?? categorySorters.name);
 
         renderCategories(categories);
         render(archives);