pmtiles-swarm 0.59.0 → 0.60.0

package/CHANGELOG.md

@@ -7,6 +7,74 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.60.0
+### ✨ Features and improvements
+- **A tile URL that survives a rebuild.** Every archive is addressed by infohash, which is
+  what makes a tile cacheable for a year and what makes it useless in an application: the URL
+  changes with every build, so anything that wrote one down is pinned to a build that
+  eventually stops existing. A category is the only stable handle this system has, and now it
+  has a tile endpoint of its own — `/latest/<category>/{z}/{x}/{y}.<ext>` — which resolves to
+  whichever build is current on every request.
+
+  The category TileJSON advertises it as `tiles`, so a style written once keeps working across
+  a rebuild without being re-fetched for the URLs alone. The immutable template is still
+  published beside it as `latest.tiles`, because it is still the better URL for anything that
+  can re-read the document: it is content-addressed, so it caches for a year and never
+  revalidates. Offering only one of the two would be choosing for the consumer, and the right
+  answer differs by consumer.
+
+  The console offers the template as a copyable **XYZ** button next to the TileJSON one, which
+  is the form a Leaflet layer, an OpenLayers source or a GIS client actually wants.
+
+  Deliberately not a redirect to the immutable URL, though every other `/latest/` route is one.
+  A redirect costs a round trip and a map asks for hundreds of tiles: what is a negligible
+  indirection for a `.torrent` is the difference between a map that feels immediate and one
+  that does not. It is cached as the moving target it is — `max-age=300, must-revalidate`,
+  tagged with the build it resolved to, so a revalidation is a 304 while that build stands and
+  a miss the moment it moves.
+
+- **A benchmark for "why does this feel slower than the other one", in `tools/tile-bench.mjs`.**
+  It reads two TileJSON documents, picks tiles inside the zoom range and bounds both can serve,
+  and requests the same set from each — one server at a time, because run together they compete
+  for the same link and each measures the other's load as its own latency.
+
+  It reports percentiles rather than an average, since what makes a map feel slow is the tail
+  and a mean built from nineteen fast requests does not move for the twentieth. Time to first
+  byte is separated from the total, which is the difference between a slow lookup and a slow
+  link — they want opposite fixes.
+
+  It also detects a pool of unequal nodes, which is a common cause and an invisible one: half
+  the tiles arrive quickly and half do not, and balanced evenly the mean looks tolerable
+  throughout. Two distinct groups are reported as two, with a histogram, and `--header` tallies
+  a response header naming which backend answered. `--a-origin`/`--b-origin` send the tile
+  requests somewhere other than the document was read from, which is the only way to measure
+  one node directly: a node with `publicUrl` set answers with that name however it was asked.
+
+- **`docs/haproxy.md` covers a pool whose nodes are not the same speed.** Round robin assumes
+  the pool is interchangeable, and a tile server on an NVMe disk and one on a spinning disk are
+  not — an archive read is a seek into a large file, which is what a spinning disk is worst at.
+  `backup`, `weight` and least-connections are compared, along with what each does and does not
+  fix.
+
+### 🐞 Bug fixes
+- **An archive at 100% and seeding could serve no tiles until the node was restarted.** Which
+  source an archive is read through is decided once, when a reader opens it, and every other
+  thing that can change that answer already invalidates the reader: a pause, a resume, a mode
+  change, a move, a finished download. A finished *check* did not — and it is the easiest of
+  them to reach, because during `checking_files` libtorrent reports `progress` as the fraction
+  hashed so far, which is indistinguishable from a download sitting at the same figure.
+
+  So a tile read arriving mid-check opened against the swarm, correctly, and kept that handle
+  afterwards. The archive then read from a swarm whose only member is this node, while the
+  whole file sat on the disk beside it, and nothing evicted the handle short of a restart. The
+  completion sweep looked straight past it: an entry already recorded complete never reaches
+  the code that would have noticed.
+
+  The sweep now drops a reader that is going to the swarm for an archive the disk says is
+  whole. The disk is checked before the handle is dropped rather than after, or a reader that
+  would only re-open against the swarm anyway would be invalidated on every sweep, for ever.
+  Cache mode is left alone: it reads from the swarm because that is what it is for.
+
 ## 0.59.0
 ### ✨ Features and improvements
 - **Requires pmtiles-torrent 0.10.2**, which is what actually ends the re-checking: a

package/docs/haproxy.md

@@ -114,6 +114,70 @@ gives archive affinity, at the price of concentrating one archive on one node.
 For nodes holding complete copies there is no cold read to avoid, so this is a
 cost with no benefit.
 
+### When the nodes are not the same speed
+
+Round robin assumes the pool is interchangeable, and a tile server on an NVMe
+disk and one on a spinning disk are not. Every archive read is a seek into a
+large file — the PMTiles directory, then the leaf, then the tile — and that is
+precisely the access pattern a spinning disk is worst at. The two can easily
+differ by an order of magnitude.
+
+What that does to a pool is worse than the average suggests. Balanced evenly,
+half the tiles arrive quickly and half do not, and a map is judged by the slow
+half: panning stutters wherever the slow node answered. The mean looks
+tolerable throughout, which is why this is easy to miss and easy to blame on
+the software.
+
+Three ways out, and they answer different questions.
+
+**`backup`** keeps the slow node out of service entirely until the fast one
+fails:
+
+```
+backend pmtiles-swarm
+    option httpchk GET /health HTTP/1.1
+    http-check expect status 200
+    server node1 172.16.1.49:8090 check inter 5s
+    server node2 172.16.1.41:8090 check inter 5s backup
+```
+
+Every request goes to `node1` while it is healthy, and the pool falls to
+`node2` when it is not. Latency is then the fast node's latency, and the slow
+one is redundancy rather than capacity. This is the right answer when the fast
+node can carry the load alone, which for tile serving it usually can.
+
+**`weight`** keeps both in service and sends proportionally less to the slow
+one:
+
+```
+    server node1 172.16.1.49:8090 check inter 5s weight 200
+    server node2 172.16.1.41:8090 check inter 5s weight 20
+```
+
+Worth it only when you actually need the second node's throughput. It does not
+remove the slow answers, it makes them rarer — a tenth of requests at ten times
+the latency is still a tenth of requests a person notices.
+
+**Least Connections** is the better of the two ways to keep both in service,
+and it needs no numbers chosen by hand. A slow node holds each connection
+longer, so it accumulates open ones and stops being picked until it catches up:
+the pool tunes itself to what the disks are doing today rather than to a weight
+somebody guessed last year. It shares `weight`'s limitation — the slow answers
+become proportionally rarer, not absent — and it pairs with `backup` without
+conflict, since a backup server is held out of the pool whatever the algorithm.
+
+`tools/tile-bench.mjs` shows which situation you are in. A pool of unequal
+nodes answers in two distinct groups rather than one spread, and the tool says
+so explicitly; `--header` tallies a response header naming the backend, if
+HAProxy is set to add one:
+
+```
+http-response set-header X-Served-By %s
+```
+
+Run it before and after the change. What should move is the tail: `total p90`
+and `total p99` collapse towards `p50` once one node is answering everything.
+
 ### HTTP/2
 
 Enable it on the frontend and leave _HTTP/2 without TLS_ unchecked: the client
@@ -219,9 +283,16 @@ says so, which is worse than having no check at all.
 
 **Cache tiles by infohash aggressively.** `/archives/<infohash>/…` is immutable
 by construction — an infohash names those bytes and no others — and is served
-with `max-age=31536000, immutable`. `/latest/<category>/…` is the opposite: it
-moves on every build, and is served with `max-age=60, must-revalidate` and an
-ETag naming the build it resolved to.
+with `max-age=31536000, immutable`.
+
+Everything under `/latest/<category>/` is the opposite, because it moves on
+every build. The documents — `tiles.json`, the feeds — are served with
+`max-age=60, must-revalidate`; the tiles with `max-age=300`. Both carry an ETag
+naming the build they resolved to, so a revalidation is a 304 while the build
+stands and a miss the moment it moves. Let Cloudflare revalidate rather than
+overriding either with a page rule: the whole point of the category tile URL is
+that it is safe to write into an application, and it is only safe while the
+edge notices a rebuild.
 
 **Do not strip or rewrite the ETag on either.** It is the infohash, which is
 how a PMTiles reader notices that the archive moved underneath a read already

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.59.0",
+  "version": "0.60.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -32,7 +32,7 @@ import {
   expandTemplate,
 } from './sources.js';
 import { limitFor, remaining } from './seeding.js';
-import { buildTileJson, extensionMatches } from './tilejson.js';
+import { buildTileJson, extensionMatches, tileExtension } from './tilejson.js';
 import { SUMMARY_VERSION } from './pmtiles-probe.js';
 import { TileReadError } from './tiles.js';
 
@@ -2158,6 +2158,14 @@ export function createApp({
           servable,
           endpoints: {
             tileJson: servable ? `${base}/latest/${category}/tiles.json` : null,
+            // The XYZ template itself, which is what most things outside this
+            // system actually want: a leaflet layer, an OpenLayers source, a
+            // GIS client, anything that takes a URL with braces in it rather
+            // than a TileJSON document. It follows the category, so it can be
+            // written into an application and survive a rebuild.
+            xyz: servable
+              ? `${base}/latest/${category}/{z}/{x}/{y}.${tileExtension(newest.pmtiles?.format)}`
+              : null,
             // The same URL with the ways into the swarm in the fragment,
             // which is what a style should carry. A fragment is never sent in
             // a request, so an ordinary client fetches the TileJSON and
@@ -2253,8 +2261,23 @@ export function createApp({
       // makes that re-read cheap — a client already holding the current build
       // gets a 304 and no body, which Express does on its own once an ETag is
       // set before the response goes out.
+      // Both templates are built, because both are wanted and they are wanted
+      // for opposite reasons.
+      //
+      // `tiles` points at the category, so a style written once keeps working
+      // across a rebuild. That is what the endpoint is for, and an infohash
+      // template cannot do it: it changes every build, so anything holding it
+      // is pinned to a build that eventually stops existing.
+      //
+      // The immutable template is still published, under `latest.tiles`, and
+      // is still the better URL for anything that can re-read this document —
+      // it is content-addressed, so it caches for a year and never
+      // revalidates. Offering only one of the two would be choosing for the
+      // consumer, and the right answer differs by consumer.
+      const pinned = buildTileJson(entry, baseUrl(req));
+      const stableRoot = `${baseUrl(req)}/latest/${encodeURIComponent(req.params.category)}`;
       const doc = {
-        ...buildTileJson(entry, baseUrl(req)),
+        ...buildTileJson(entry, baseUrl(req), { tilesRoot: stableRoot }),
         // Names what it resolved to, so a consumer can tell one build from the
         // next without diffing the tile URLs.
         latest: {
@@ -2262,6 +2285,7 @@ export function createApp({
           infohash: entry.infoHash,
           name: entry.name,
           createdAt: entry.createdAt,
+          tiles: pinned.tiles,
         },
       };
       tagAsLatest(res, doc);
@@ -2833,119 +2857,191 @@ export function createApp({
     }),
   );
 
-  app.get(
-    '/archives/:infoHash/:z/:x/:y.:ext',
-    route(async (req, res) => {
-      const { infoHash, ext } = req.params;
-      let entry = catalog.get(infoHash);
-      if (!entry) return res.status(404).json({ error: 'unknown archive' });
-      // MBTiles passes through here and is turned away by the store instead,
-      // which is the only layer that knows whether the download has finished.
-      if (entry.kind && entry.kind !== 'pmtiles' && entry.kind !== 'mbtiles') {
-        return res.status(415).json({
-          error: `this is a ${entry.kind} archive, and only PMTiles and MBTiles can be served as tiles`,
+  /**
+   * Serves one tile, from an archive the caller has already resolved.
+   *
+   * Shared by the two ways of naming an archive, which differ in one respect
+   * and want the same behaviour in every other. `/archives/<infohash>/…` pins
+   * content, so the answer can be cached for a year and never revalidated.
+   * `/latest/<category>/…` resolves to whichever build is current, so it
+   * cannot: the same URL returns different bytes after a rebuild, and a client
+   * holding it for a year would hold a build that no longer exists.
+   *
+   * The difference is expressed entirely in the caching headers rather than in
+   * two implementations, because everything else — the coordinate checks, the
+   * summary backfill, the abort on a cancelled request, the 204-against-404
+   * decision for a sparse archive — has to be identical or one of the two is
+   * quietly a different endpoint.
+   * @param {object} found - The catalog entry to read from.
+   * @param {import('express').Request} req - The request.
+   * @param {import('express').Response} res - The response.
+   * @param {object} [options] - `immutable` false for a resolved category.
+   * @returns {Promise<void>} - Resolves once answered.
+   */
+  const serveTile = async (found, req, res, options = {}) => {
+    const { ext } = req.params;
+    let entry = found;
+    const infoHash = entry.infoHash;
+    // MBTiles passes through here and is turned away by the store instead,
+    // which is the only layer that knows whether the download has finished.
+    if (entry.kind && entry.kind !== 'pmtiles' && entry.kind !== 'mbtiles') {
+      return res.status(415).json({
+        error: `this is a ${entry.kind} archive, and only PMTiles and MBTiles can be served as tiles`,
+      });
+    }
+
+    const z = Number(req.params.z);
+    const x = Number(req.params.x);
+    const y = Number(req.params.y);
+    if (![z, x, y].every(Number.isInteger)) {
+      return res.status(400).json({ error: 'z, x and y must be integers' });
+    }
+    const limit = 2 ** z;
+    if (z < 0 || z > 26 || x < 0 || y < 0 || x >= limit || y >= limit) {
+      return res.status(400).json({ error: 'tile coordinates out of range' });
+    }
+    // An MBTiles archive never went through the PMTiles prober, so it
+    // reaches here with no summary and nothing to check the extension
+    // against. Read one now: it is a handful of rows out of a local SQLite
+    // file, kept in the catalog, so this is paid once per archive rather
+    // than per tile. A failure is left to the read below to report, which
+    // already knows how to say "not complete yet".
+    if (!entry.pmtiles?.format) {
+      const summary = await tiles.summarize(entry.infoHash).catch(() => null);
+      if (summary) {
+        entry = await catalog.put({
+          infoHash: entry.infoHash,
+          pmtiles: summary,
+          summarySource: 'header',
         });
       }
+    }
 
-      const z = Number(req.params.z);
-      const x = Number(req.params.x);
-      const y = Number(req.params.y);
-      if (![z, x, y].every(Number.isInteger)) {
-        return res.status(400).json({ error: 'z, x and y must be integers' });
-      }
-      const limit = 2 ** z;
-      if (z < 0 || z > 26 || x < 0 || y < 0 || x >= limit || y >= limit) {
-        return res.status(400).json({ error: 'tile coordinates out of range' });
-      }
-      // An MBTiles archive never went through the PMTiles prober, so it
-      // reaches here with no summary and nothing to check the extension
-      // against. Read one now: it is a handful of rows out of a local SQLite
-      // file, kept in the catalog, so this is paid once per archive rather
-      // than per tile. A failure is left to the read below to report, which
-      // already knows how to say "not complete yet".
-      if (!entry.pmtiles?.format) {
-        const summary = await tiles.summarize(entry.infoHash).catch(() => null);
-        if (summary) {
-          entry = await catalog.put({
-            infoHash: entry.infoHash,
-            pmtiles: summary,
-            summarySource: 'header',
-          });
-        }
-      }
+    if (!extensionMatches(entry, ext)) {
+      return res.status(400).json({
+        error: `this archive holds ${entry.pmtiles?.format ?? 'unknown'} tiles`,
+      });
+    }
 
-      if (!extensionMatches(entry, ext)) {
-        return res.status(400).json({
-          error: `this archive holds ${entry.pmtiles?.format ?? 'unknown'} tiles`,
+    const controller = new AbortController();
+    // A panning map abandons requests constantly. Without this the swarm
+    // keeps fetching pieces for tiles nobody is waiting for any more.
+    res.on('close', () => {
+      if (!res.writableEnded) controller.abort();
+    });
+
+    // Counted on the way out rather than at each return: this handler ends
+    // in six different places (200, 204, 404, 415, 400, a read error), and
+    // one hook catches all of them without any of them having to remember.
+    // Abandoned requests are not counted -- a panning map cancels constantly
+    // and those were never served.
+    if (stats) {
+      const startedAt = process.hrtime.bigint();
+      res.on('finish', () => {
+        stats.record({
+          infoHash,
+          name: entry.name,
+          z,
+          x,
+          y,
+          status: res.statusCode,
+          bytes: Number(res.getHeader('content-length')) || 0,
+          ms: Number(process.hrtime.bigint() - startedAt) / 1e6,
+          // Whatever this process can see. Behind a proxy that sends no
+          // X-Forwarded-For this is the proxy's address, which is itself
+          // the answer to "did this arrive directly or through HAProxy".
+          ip: req.ip,
         });
-      }
+      });
+    }
 
-      const controller = new AbortController();
-      // A panning map abandons requests constantly. Without this the swarm
-      // keeps fetching pieces for tiles nobody is waiting for any more.
-      res.on('close', () => {
-        if (!res.writableEnded) controller.abort();
+    let tile;
+    try {
+      tile = await tiles.getTile(infoHash, z, x, y, {
+        signal: controller.signal,
       });
+    } catch (error) {
+      if (error.name === 'AbortError') return;
+      if (error instanceof TileReadError) {
+        return res.status(error.status).json({ error: error.message });
+      }
+      throw error;
+    }
 
-      // Counted on the way out rather than at each return: this handler ends
-      // in six different places (200, 204, 404, 415, 400, a read error), and
-      // one hook catches all of them without any of them having to remember.
-      // Abandoned requests are not counted -- a panning map cancels constantly
-      // and those were never served.
-      if (stats) {
-        const startedAt = process.hrtime.bigint();
-        res.on('finish', () => {
-          stats.record({
-            infoHash,
-            name: entry.name,
-            z,
-            x,
-            y,
-            status: res.statusCode,
-            bytes: Number(res.getHeader('content-length')) || 0,
-            ms: Number(process.hrtime.bigint() - startedAt) / 1e6,
-            // Whatever this process can see. Behind a proxy that sends no
-            // X-Forwarded-For this is the proxy's address, which is itself
-            // the answer to "did this arrive directly or through HAProxy".
-            ip: req.ip,
-          });
+    res.setHeader('access-control-allow-origin', '*');
+    // The tag is the resolved infohash either way, which is what makes the
+    // category URL cheap to hold: when the build has not moved a
+    // revalidation is a 304 with no body, and when it has, the tag changes
+    // on its own without anything having to remember to invalidate.
+    res.setHeader('etag', `"${infoHash}-${z}-${x}-${y}"`);
+    res.setHeader(
+      'cache-control',
+      options.immutable === false
+        ? // A category resolves to whatever build is current, so the same
+          // URL returns different bytes after a rebuild. Five minutes bounds
+          // how long a client can be looking at the previous build, and
+          // must-revalidate is what stops a cache serving it beyond that.
+          'public, max-age=300, must-revalidate'
+        : // An infohash pins content, so a tile under one can never change.
+          // When a mutable archive is updated the infohash changes and so
+          // does this URL, which makes cache invalidation automatic.
+          'public, max-age=31536000, immutable',
+    );
+
+    // A missing tile is normal, and which status says so matters.
+    //
+    //   404 tells MapLibre the tile is absent, so it overzooms the parent —
+    //       which is the only way a sparse raster-dem renders terrain at all.
+    //   204 tells it the tile is empty but present, so it draws nothing and
+    //       does not fall back.
+    //
+    // Vector wants 204 (an empty tile means no features here); raster wants
+    // 404. Same rule and same name as tileserver-gl's `sparse`.
+    if (!tile) return res.status(isSparse(entry) ? 404 : 204).end();
+
+    res.type(entry.pmtiles?.contentType ?? 'application/octet-stream');
+    if (tile.encoding) res.setHeader('content-encoding', tile.encoding);
+    res.send(tile.data);
+  };
+
+  app.get(
+    '/archives/:infoHash/:z/:x/:y.:ext',
+    route(async (req, res) => {
+      const entry = catalog.get(req.params.infoHash);
+      if (!entry) return res.status(404).json({ error: 'unknown archive' });
+      if (entry.kind && entry.kind !== 'pmtiles' && entry.kind !== 'mbtiles') {
+        return res.status(415).json({
+          error: `this is a ${entry.kind} archive, and only PMTiles and MBTiles can be served as tiles`,
         });
       }
+      return serveTile(entry, req, res);
+    }),
+  );
 
-      let tile;
-      try {
-        tile = await tiles.getTile(infoHash, z, x, y, {
-          signal: controller.signal,
+  // The one tile URL that survives a rebuild.
+  //
+  // Every archive is addressed by infohash, which is what makes a tile
+  // cacheable for a year and what makes it useless in an application: the URL
+  // changes with every build. A category is the stable handle, so it needs a
+  // tile endpoint of its own — and this is the URL the category TileJSON
+  // advertises, so a style pointed at `/latest/<category>/tiles.json` keeps
+  // working across rebuilds without being re-fetched for the URLs alone.
+  //
+  // Not a redirect to the immutable URL, though every other `/latest/` route
+  // is one. A redirect costs a round trip, and a map asks for hundreds of
+  // tiles: what is a negligible indirection for a `.torrent` is the difference
+  // between a map that feels immediate and one that does not.
+  app.get(
+    '/latest/:category/:z/:x/:y.:ext',
+    route(async (req, res) => {
+      const entry = newestIn(req.params.category, req);
+      if (!entry) return res.status(404).json({ error: 'no such category' });
+      if (entry.kind && entry.kind !== 'pmtiles' && entry.kind !== 'mbtiles') {
+        return res.status(415).json({
+          error: `the newest archive in this category is a ${entry.kind} archive, and only PMTiles and MBTiles can be served as tiles`,
         });
-      } catch (error) {
-        if (error.name === 'AbortError') return;
-        if (error instanceof TileReadError) {
-          return res.status(error.status).json({ error: error.message });
-        }
-        throw error;
       }
-
-      res.setHeader('access-control-allow-origin', '*');
-      // An infohash pins content, so a tile under one can never change. When a
-      // mutable archive is updated the infohash changes and so does this URL,
-      // which makes cache invalidation automatic.
-      res.setHeader('cache-control', 'public, max-age=31536000, immutable');
-      res.setHeader('etag', `"${infoHash}-${z}-${x}-${y}"`);
-
-      // A missing tile is normal, and which status says so matters.
-      //
-      //   404 tells MapLibre the tile is absent, so it overzooms the parent —
-      //       which is the only way a sparse raster-dem renders terrain at all.
-      //   204 tells it the tile is empty but present, so it draws nothing and
-      //       does not fall back.
-      //
-      // Vector wants 204 (an empty tile means no features here); raster wants
-      // 404. Same rule and same name as tileserver-gl's `sparse`.
-      if (!tile) return res.status(isSparse(entry) ? 404 : 204).end();
-
-      res.type(entry.pmtiles?.contentType ?? 'application/octet-stream');
-      if (tile.encoding) res.setHeader('content-encoding', tile.encoding);
-      res.send(tile.data);
+      return serveTile(entry, req, res, { immutable: false });
     }),
   );
 

package/src/incomplete.js

@@ -191,6 +191,15 @@ export class CompletionWatcher {
         await this.#library.captureMetadata?.(entry.infoHash).catch(() => null);
       }
 
+      // Before the `complete` gate below, deliberately. An archive already
+      // recorded complete is the case this covers: it was re-checking when a
+      // tile reader opened it, so the reader went to the swarm — and when the
+      // check finished nothing noticed, because finalize only runs for a
+      // download. See Library.refreshReader.
+      if (entry.status?.progress >= 1) {
+        await this.#library.refreshReader?.(entry.infoHash).catch(() => null);
+      }
+
       if (entry.complete) continue;
 
       // The engine's account wins whenever it has one: a client allocates the

package/src/library.js

@@ -2778,6 +2778,60 @@ export class Library {
     return { cleared: before };
   }
 
+  /**
+   * Drops a reader still going to the swarm for an archive that is now whole.
+   *
+   * Which source an archive is read through is decided once, when it is
+   * opened, and every other thing that can change the answer — a pause, a
+   * resume, a mode change, a move, a finished download — invalidates the
+   * reader itself. A finished *check* is the one that does not, and it is the
+   * easiest of them to hit: during `checking_files` libtorrent reports
+   * `progress` as the fraction hashed so far, which is indistinguishable from
+   * a download at the same figure. So a tile read arriving mid-check opens
+   * against the swarm, correctly, and then keeps that handle after the check
+   * finishes and the whole file is sitting on disk.
+   *
+   * What that looks like is an archive at 100% and seeding whose tiles will
+   * not load, because the swarm it is being read from has one member: this
+   * node. It survives until something evicts the handle, which in practice
+   * means a restart.
+   *
+   * The disk is checked before the handle is dropped, and that is not
+   * belt-and-braces: without it an archive that re-opens against the swarm
+   * anyway would be invalidated again on every sweep, for ever.
+   * @param {string} infoHash - Which archive.
+   * @returns {Promise<boolean>} - Whether a stale reader was dropped.
+   */
+  async refreshReader(infoHash) {
+    // Cheap and synchronous, and false for all but a handful of archives —
+    // this runs for every entry on the completion timer.
+    if (this.#tiles?.status(infoHash)?.mode !== 'swarm') return false;
+
+    const entry = this.#catalog.get(infoHash);
+    if (!entry?.savePath || !entry?.name) return false;
+
+    // Cache mode reads from the swarm because that is what it is for, not
+    // because anything went stale.
+    if (entry.mode === 'cache') return false;
+
+    const whole = await alreadyComplete({
+      savePath: entry.savePath,
+      name: entry.name,
+      size: entry.size,
+    });
+    if (!whole) return false;
+
+    const dropped = await this.#tiles.invalidate(infoHash).catch(() => false);
+    if (dropped) {
+      console.log(
+        `[tiles] ${entry.name}: was being read from the swarm while a whole ` +
+          'copy sat on disk, which is what a finished re-check leaves behind. ' +
+          'Reopening it locally.',
+      );
+    }
+    return dropped;
+  }
+
   /**
    * Rewrites an archive's web seed list, in the .torrent and everywhere else.
    *

package/src/tilejson.js

@@ -23,21 +23,45 @@ const URL_EXTENSION = {
   mlt: 'mlt',
 };
 
+/**
+ * The extension a tile URL uses for one archive format.
+ *
+ * Exported because the console publishes the XYZ template beside the TileJSON,
+ * and a second copy of this map is a second thing to forget when a format is
+ * added — which would show up as a working document advertising a URL that
+ * answers 400.
+ * @param {string} [format] - The archive's tile format.
+ * @returns {string} - The extension, without a dot.
+ */
+export function tileExtension(format) {
+  return URL_EXTENSION[format] ?? 'bin';
+}
+
 /**
  * Builds the TileJSON document for one catalog entry.
+ *
+ * `tilesRoot` is how a category document points its tiles at itself rather
+ * than at the build it happens to have resolved to. The infohash URL is the
+ * right default — it pins content, which is what lets a tile be cached for a
+ * year — but it is the wrong thing to write into an application, because it
+ * changes with every rebuild. A category has to be able to hand out a URL that
+ * does not.
  * @param {object} entry - Catalog entry.
  * @param {string} baseUrl - Public base URL, without a trailing slash.
+ * @param {object} [options] - Overrides.
+ * @param {string} [options.tilesRoot] - Root for the tile template.
  * @returns {object} - A TileJSON 3.0.0 document.
  */
-export function buildTileJson(entry, baseUrl) {
+export function buildTileJson(entry, baseUrl, options = {}) {
   const summary = entry.pmtiles ?? {};
-  const extension = URL_EXTENSION[summary.format] ?? 'bin';
+  const extension = tileExtension(summary.format);
   const root = `${baseUrl}/archives/${entry.infoHash}`;
+  const tilesRoot = options.tilesRoot ?? root;
 
   const doc = {
     tilejson: '3.0.0',
     scheme: 'xyz',
-    tiles: [`${root}/{z}/{x}/{y}.${extension}`],
+    tiles: [`${tilesRoot}/{z}/{x}/{y}.${extension}`],
     name: summary.name ?? entry.name,
     minzoom: summary.minZoom ?? 0,
     maxzoom: summary.maxZoom ?? 14,

package/src/web/index.html

@@ -3715,6 +3715,7 @@ Every piece is hashed against the ` +
                   ends.tileJson
                     ? `<div class="links">
                          ${copyable(ends.tileJson, 'TileJSON')}
+                         ${copyable(ends.xyz, 'XYZ')}
                          ${link(ends.preview, 'preview')}
                          ${link(ends.torrent, '.torrent', true)}
                          ${copyable(ends.magnet, 'magnet', true)}