pmtiles-swarm 0.7.1 → 0.8.0

package/CHANGELOG.md

@@ -7,6 +7,36 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.8.0
+### ✨ Features and improvements
+- **`GET /health`**, for a load balancer: 200 when this node can serve, 503 when its engine
+  cannot, no credential and nothing to parse. It asks the engine rather than itself, which is the
+  distinction that makes it worth having — a feed is built from the catalogue and never touches
+  the swarm, so a balancer checking `/feed.xml`, the nearest thing that existed, gets 200 from a
+  node whose engine is dead and keeps sending it traffic. The answer is cached for two seconds,
+  because a balancer asks often and each check is an inter-process round trip, and it is sent
+  `no-store`: a stale health check keeps a dead node in rotation for as long as whatever cached
+  it says so.
+- **`GET /archives/<infohash>/ready`**, which answers a different question — whether a *particular*
+  archive has become servable on this node. 200 once its header and, for vector, its layers have
+  been read; 503 with which half is missing; **415** for an archive that can never be served,
+  since MBTiles is distributed here but cannot be read a byte range at a time and polling it would
+  be polling for ever; 404 when it is not here. It reports rather than acts, starting no read and
+  waiting for nothing — a probe that does work on demand is a probe that can be used to make a
+  node do work on demand.
+- **[docs/haproxy.md](docs/haproxy.md)**, written against the OPNsense plugin: the health monitor
+  field by field, how the check interval trades against failover time, timeouts long enough for a
+  web seed to finish, `X-Forwarded-Proto`, gating a deployment on `/ready` — and a table of what a
+  reverse proxy in front of a BitTorrent node simply cannot carry.
+
+### 🐞 Bug fixes
+- **Head-warming no longer tries to read a PMTiles header out of a `.osm.pbf`.** The guard was
+  `entry.kind && entry.kind !== 'pmtiles'`, and `guessKind` answers `undefined` for anything it
+  does not recognise — so it never fired for exactly the archives it existed to exclude. Every
+  planet dump being mirrored, and every MBTiles archive, was read as though it had a header,
+  failed, and came back on the backoff for ever. The test is positive now: the kind has to *be*
+  PMTiles, taken from the entry where it is known and from the file name where it is not.
+
 ## 0.7.1
 ### 🐞 Bug fixes
 - **A feed no longer deletes its only complete copy.** Retention was written for a watched folder

package/README.md

@@ -33,6 +33,9 @@ node src/index.js --config swarm.config.json
   address.
 - **[docs/running-as-a-service.md](docs/running-as-a-service.md)** — a systemd unit, why
   `Restart=always` is required rather than optional, and which ports want a firewall rule.
+- **[docs/haproxy.md](docs/haproxy.md)** — health monitors, what a reverse proxy carries and
+  what it cannot, timeouts long enough for a web seed, and gating a deployment on
+  `/archives/<infohash>/ready`.
 - **[docs/architecture-diagram.md](docs/architecture-diagram.md)** — how a publishing node, a
   serving tier, the swarm and both kinds of client fit together.
 
@@ -626,6 +629,8 @@ matters there is `maxConnections`, since every peer holds a NAT table entry. See
 | `GET` | `/api/catalog` | The whole catalogue, for a peer keeping itself in step |
 | `GET` | `/archives/:infoHash/tiles.json` | TileJSON — **public** |
 | `GET` | `/archives/:infoHash/:z/:x/:y.:ext` | One tile — **public** |
+| `GET` | `/health` | 200 when this node can serve, 503 when its engine cannot — **public**, no credential, for a load balancer |
+| `GET` | `/archives/:infoHash/ready` | Whether this node can serve *this* archive yet: 200 ready, 503 not yet, 415 never — **public** |
 | `GET` | `/archives/:infoHash/archive.torrent` | The `.torrent` a torrent-aware client joins with — **public** |
 | `GET` | `/archives/:infoHash/preview` | Map preview for one archive — admin, not public |
 | `GET` | `/latest/:category/tiles.json`, `/:name.torrent`, `/magnet` | The newest build in a category — **public**. The torrent name is yours to choose, so a link can read `planetiler-openmaptiles-latest.torrent`; it redirects to the immutable URL, which names the download after the build it actually is |

package/docs/haproxy.md

@@ -0,0 +1,194 @@
+# Behind HAProxy
+
+Written against the **OPNsense HAProxy plugin**, which is configured through a
+form rather than a file. The field names below are that plugin's; the generated
+configuration is shown alongside so the same thing can be written by hand.
+
+## What HAProxy carries, and what it cannot
+
+This is the first thing to get straight, because a reverse proxy in front of a
+BitTorrent node carries less than it looks like it does.
+
+| | Through HAProxy? |
+| --- | --- |
+| Tiles, TileJSON, feeds, `.torrent` files, health checks | **Yes** — ordinary HTTP on 8090 |
+| The console and API on 8091 | **No.** Do not publish it |
+| BitTorrent peers on 6881 | **No.** Not HTTP; needs its own forward |
+| WebRTC to browser peers | **No.** Outbound only, and needs nothing |
+
+A node reachable *only* through HAProxy has no incoming peers at all. It will
+still seed to peers it connects to, and still serve tiles, but it is half a
+member of the swarm. 6881 wants a firewall rule of its own, straight to the
+node, TCP **and** UDP.
+
+## The health monitor
+
+**Services → HAProxy → Settings → Health Monitors → Add**
+
+| Field | Value |
+| --- | --- |
+| Name | `pmtiles-swarm` |
+| Check type | `HTTP` |
+| Check interval | `2s` or `5s` — see below |
+| Port to check | *empty*, so it uses the real server's port |
+| HTTP method | `GET` |
+| Request URI | `/health` |
+| HTTP version | `HTTP/1.1` |
+| HTTP host | anything; `localhost` is fine |
+| Custom HTTP check → Expression | exact string match for the HTTP status code |
+| Custom HTTP check → Value | `200` |
+
+`HTTP/1.1` requires a `Host` header, which is why the field is there. The node
+does not route by it, so its value only matters if something in front of it
+does.
+
+Which produces:
+
+```
+backend pmtiles-swarm
+    option httpchk GET /health HTTP/1.1
+    http-check expect status 200
+    server node1 172.16.1.49:8090 check inter 5s
+    server node2 172.16.1.41:8090 check inter 5s
+```
+
+**Do not point the monitor at `/` or at `/feed.xml`.** Both answer 200 from a
+node whose engine is dead — a feed is built from the catalogue and never
+touches the swarm — so the balancer would keep sending traffic to a node that
+cannot serve a tile. `/health` asks the engine and answers 503 when it does not
+reply.
+
+### How often to check
+
+`/health` caches its answer for two seconds, so anything faster than that asks
+the same question twice and gets the same answer. At two seconds, near enough
+every check is a real round trip to the engine — which enumerates every torrent
+the node holds, so the cost grows with the catalogue.
+
+What the interval buys is failover speed, and it is the interval multiplied by
+HAProxy's `fall` count, which defaults to 3:
+
+| Check interval | Marked down after |
+| --- | --- |
+| `2s` | about 6 seconds |
+| `5s` | about 15 seconds |
+
+Neither is wrong. Six seconds is worth having if a node dying mid-request
+matters; fifteen is worth having if it does not, and costs the node less. What
+is not worth having is a sub-second interval, which only asks a cached answer
+more often.
+
+Note also that the node comes *back* on `rise` successful checks — 2 by
+default — so a flapping node re-enters rotation quickly whichever you choose.
+
+## Real servers
+
+**Settings → Real Servers → Add**, one per node, port **8090**.
+
+The admin port has no place in a public backend. It carries the console and the
+API, and while both are behind authentication, an internet-facing sign-in page
+is a thing to decide on deliberately rather than to acquire by copying a
+backend.
+
+## Tell the node it is behind a proxy
+
+HAProxy terminates TLS, so without being told, the node believes every request
+arrived over plain HTTP and advertises `http://` tile URLs. A browser that
+loaded the map over HTTPS then blocks every one of them as mixed content, which
+looks like an empty map rather than a configuration mistake.
+
+Two ways, and they are alternatives rather than a pair:
+
+```json
+{ "publicUrl": "https://swarm.example.org" }
+```
+
+One canonical URL whatever the request said. Simple, and right when the node is
+only ever reached one way.
+
+```json
+{ "trustProxy": "172.16.1.0/24" }
+```
+
+Derive it per request from `X-Forwarded-Proto` and the `Host`, so the same node
+answers correctly on its LAN address and through the proxy. **Name the proxy
+rather than saying `true`**: the node also listens on its LAN address directly,
+and `true` would let anything that can reach that port claim any protocol and
+host it likes, which decides what URLs your feed hands out.
+
+For that to work HAProxy has to send the header. In the plugin it is
+**Public Service → Advanced → Option pass-through**, or in configuration:
+
+```
+frontend https-in
+    http-request set-header X-Forwarded-Proto https
+```
+
+`Host` is passed through by default.
+
+## Timeouts
+
+The defaults are wrong for this. A web seed is an HTTP range request for part of
+an archive, and a peer pulling a large one holds the connection for as long as
+the transfer takes.
+
+```
+defaults
+    timeout connect 5s
+    timeout client  1h
+    timeout server  1h
+```
+
+An hour is not generous here; a client fetching tens of gigabytes over a slow
+link exceeds anything shorter, and the failure looks like a corrupt download
+rather than a timeout.
+
+## Deploying a new build
+
+`/health` says whether a node should be sent traffic. It says nothing about
+whether a *particular* archive has become servable there, which is the question
+worth asking after a build lands.
+
+```sh
+INFOHASH=5e1c143c400d15aaacfb1c748d4ab6d1b46c5df5
+for node in 172.16.1.49 172.16.1.41; do
+  until curl -fsS "http://$node:8090/archives/$INFOHASH/ready" >/dev/null; do
+    sleep 10
+  done
+  echo "$node is ready"
+done
+```
+
+Ask each node **directly**, not through the balancer — through it you learn that
+*some* node is ready, which is not the same thing and is exactly the wrong
+answer when deciding whether to move traffic.
+
+`/ready` answers 415 for an archive that can never be served, so a loop like the
+one above would never end if an MBTiles archive reached it. Treat 415 as a
+separate case if that is possible.
+
+## Cloudflare in front
+
+Two rules worth setting deliberately.
+
+**Never cache `/health` or `/archives/*/ready`.** Both send `no-store` and
+Cloudflare honours it, but a page rule that caches everything can override that.
+A cached health check keeps a dead node in rotation for as long as the cache
+says so, which is worse than having no check at all.
+
+**Cache tiles by infohash aggressively.** `/archives/<infohash>/…` is immutable
+by construction — an infohash names those bytes and no others — and is served
+with `max-age=31536000, immutable`. `/latest/<category>/…` is the opposite: it
+moves on every build and is served with `max-age=300`.
+
+WebRTC does not pass through Cloudflare, and neither does BitTorrent. Browser
+peers reach the node over ICE, and a `wss://` tracker is the only part of that
+which is HTTP at all.
+
+## When it looks like the swarm is empty
+
+Almost always the peer port rather than the proxy. HAProxy carries none of it,
+so a node behind one with no forward for 6881 shows connected peers only where
+it made the connection itself. See
+[docs/engines.md](engines.md#ports-and-reachability) for which ports want
+forwarding and which want nothing.

package/docs/serving-tiles.md

@@ -200,6 +200,85 @@ For an archive published as a mutable torrent, the block also carries
 rather than pinning to the version the document was generated from. See
 [publishing](publishing.md).
 
+## Health checks
+
+```
+GET /health
+```
+
+200 when this node can serve, 503 when it cannot, no credential and no body
+worth parsing — which is what a load balancer needs. It is on the public
+surface, so it answers on the same port the tiles do.
+
+**It asks the engine, not just itself.** A reply means the engine answered a
+round trip, which is the difference worth reporting: a feed is built from the
+catalog and never touches the swarm, so a balancer checking `/feed.xml` gets
+200 from a node whose engine is dead and keeps sending it traffic.
+
+The answer is cached for a couple of seconds, because a balancer asks often and
+each check costs an inter-process round trip. A node that has just died leaves
+rotation one check later than it otherwise would.
+
+It sends `cache-control: no-store`. A stale health check is worse than none —
+it keeps a dead node in rotation for as long as whatever cached it says so.
+
+```
+backend tiles
+    option httpchk GET /health
+    http-check expect status 200
+    server node1 10.0.0.11:8090 check inter 5s
+```
+
+Configured through a form rather than a file, and with the rest of what a proxy
+in front of this needs — timeouts, `X-Forwarded-Proto`, and the ports it cannot
+carry — in [docs/haproxy.md](haproxy.md).
+
+### Whether one archive is servable yet
+
+```
+GET /archives/<infohash>/ready
+```
+
+A different question, and worth keeping apart from the one above. `/health`
+decides whether a node should be sent traffic at all; this says whether a
+newly published archive has become servable *here* — which is what you want
+after a build lands and before pointing anything at it.
+
+| | |
+| --- | --- |
+| **200** | Ready. Its header has been read, and a vector archive has its layers |
+| **503** | Not yet — ask again. The body says which half is missing |
+| **415** | Never. MBTiles is distributed here but cannot be read a byte range at a time, so waiting would be waiting for ever |
+| **404** | Not on this node |
+
+The codes differ because the responses differ: one is "poll me", one is "stop
+polling", and a script that treats them alike either gives up too early or
+waits for something that will never happen.
+
+It **reports rather than acts** — it starts no read and waits for nothing. A
+probe that does work on demand is a probe that can be used to make a node do
+work on demand. Reading the head is the head warmer's job; this only says
+whether it has happened.
+
+So the shape of a deployment check across a serving tier is: publish, then poll
+every node until each answers 200, then move `latest`.
+
+```sh
+for node in 10.0.0.11 10.0.0.12; do
+  until curl -fsS "http://$node:8090/archives/$INFOHASH/ready" >/dev/null; do
+    sleep 10
+  done
+done
+```
+
+`curl -f` fails on 503 and on 415 alike, so treat 415 separately if an MBTiles
+archive could ever reach that loop — otherwise it never ends.
+
+An archive this node holds *completely* can still be read from disk with the
+engine down, so 503 is a statement about the node rather than about every
+request it could answer. That is the right way round for a balancer with
+somewhere else to send the traffic.
+
 ## Caching
 
 Tiles are served `Cache-Control: public, max-age=31536000, immutable`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.7.1",
+  "version": "0.8.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -13,6 +13,7 @@ import {
   isPublicSurface,
 } from './auth.js';
 import { normalizeCategories } from './catalog.js';
+import { guessKind } from './library.js';
 import { QBittorrentEngine } from './engines/qbittorrent.js';
 import { RESTART_REQUIRED, redactConfig, saveConfig } from './config.js';
 import { freeSpace, listLocations } from './locations.js';
@@ -397,6 +398,59 @@ export function createApp({
     }),
   );
 
+  /**
+   * Whether this node should be sent traffic.
+   *
+   * For a load balancer, which needs three things this did not have: no
+   * credential, a cheap answer, and a status code rather than a body to parse.
+   * A balancer checking `/feed.xml` instead — the nearest thing that existed —
+   * gets 200 from a node whose engine is dead, because a feed is built from
+   * the catalog and never touches the swarm.
+   *
+   * Readiness rather than liveness. `engine.list()` is a round trip to the
+   * engine, so a reply means the sidecar is answering and not merely that Node
+   * is; that is the difference worth reporting, since everything that makes
+   * this node useful to a swarm goes through it.
+   *
+   * Cached, because a balancer asks every couple of seconds and an IPC round
+   * trip per check is a cost with nothing to show for it. The window is short
+   * enough that a node which has just died is out of rotation within one more
+   * check than it would have been.
+   */
+  let healthChecked = 0;
+  let healthOk = true;
+  let healthError;
+  const HEALTH_TTL_MS = 2000;
+
+  app.get(
+    '/health',
+    route(async (_req, res) => {
+      const now = Date.now();
+      if (now - healthChecked >= HEALTH_TTL_MS) {
+        healthChecked = now;
+        try {
+          await engine.list();
+          healthOk = true;
+          healthError = undefined;
+        } catch (error) {
+          healthOk = false;
+          healthError = error.message;
+        }
+      }
+
+      // Never cached anywhere. A stale health check is worse than none: it
+      // keeps a dead node in rotation for as long as whatever cached it says.
+      res.setHeader('cache-control', 'no-store');
+      res.setHeader('access-control-allow-origin', '*');
+      res.status(healthOk ? 200 : 503).json({
+        status: healthOk ? 'ok' : 'unavailable',
+        engine: engine.name,
+        version: VERSION,
+        ...(healthOk ? {} : { error: healthError }),
+      });
+    }),
+  );
+
   app.get(
     '/api/status',
     route(async (_req, res) => {
@@ -1828,6 +1882,95 @@ export function createApp({
   // with authentication configured this answered 401 — to the very callers it
   // exists for, since this is the URL the TileJSON torrent block advertises and
   // the one a syncing peer follows.
+  /**
+   * Whether this node can serve *this* archive yet.
+   *
+   * A different question from `/health`, and answered separately because
+   * nobody asks it per request. `/health` decides whether a node should be
+   * sent traffic at all; this decides whether a newly published archive has
+   * become servable here — which is what you want to know after a build lands
+   * and before pointing anything at it.
+   *
+   * Reports rather than acts. It starts no read and waits for nothing: a probe
+   * that does work on demand is a probe that can be used to make a node do
+   * work on demand. The head warmer is what makes an archive ready; this only
+   * says whether it has.
+   *
+   * The three answers are deliberately different codes, because they call for
+   * different responses from whoever asked. 503 is "not yet, ask again". 415
+   * is "never" — an MBTiles archive is distributed here and cannot be read a
+   * byte range at a time, so waiting for it would be waiting for ever. 404 is
+   * "not here at all".
+   */
+  app.get(
+    '/archives/:infoHash/ready',
+    route(async (req, res) => {
+      const entry = catalog.get(req.params.infoHash);
+      // Never cached. The whole value of this is that it changes.
+      res.setHeader('cache-control', 'no-store');
+      res.setHeader('access-control-allow-origin', '*');
+
+      if (!entry) {
+        return res.status(404).json({ ready: false, reason: 'unknown archive' });
+      }
+
+      const kind = entry.kind ?? guessKind(entry.name ?? '');
+      const shape = {
+        infoHash: entry.infoHash,
+        name: entry.name,
+        kind: kind ?? 'unknown',
+        complete: entry.complete === true,
+      };
+
+      if (kind !== 'pmtiles') {
+        return res.status(415).json({
+          ...shape,
+          ready: false,
+          reason:
+            `this is ${kind ? `a ${kind}` : 'not a PMTiles'} archive, and only ` +
+            'PMTiles can be read a byte range at a time — it will not become ' +
+            'servable by waiting',
+        });
+      }
+
+      const summary = entry.pmtiles;
+      // A summary that names a format is one a header was actually read for.
+      // Anything else is a partial left by a read that did not finish.
+      if (!summary?.format) {
+        return res.status(503).json({
+          ...shape,
+          ready: false,
+          reason: 'its header has not been read yet',
+        });
+      }
+
+      // Vector tiles without their layer list can be served, but nothing can
+      // be styled from them: the TileJSON a map asks for would carry no
+      // vector_layers. The metadata sits wherever the writer put it, which for
+      // planetiler is after every tile, so it routinely arrives long after the
+      // header.
+      if (summary.format === 'pbf' && !summary.vectorLayers) {
+        return res.status(503).json({
+          ...shape,
+          ready: false,
+          format: summary.format,
+          reason: 'its metadata has not been read yet, so it carries no vector layers',
+        });
+      }
+
+      res.json({
+        ...shape,
+        ready: true,
+        format: summary.format,
+        minZoom: summary.minZoom,
+        maxZoom: summary.maxZoom,
+        ...(summary.vectorLayers
+          ? { vectorLayers: summary.vectorLayers.length }
+          : {}),
+      });
+    }),
+  );
+
   app.get(
     '/archives/:infoHash/archive.torrent',
     route(async (req, res) => {

package/src/auth.js

@@ -144,6 +144,8 @@ export function isPublicSurface(path) {
   if (/^\/archives\/[^/]+\/preview\/?$/.test(path)) return false;
 
   return (
+    // A load balancer checks this, and it checks the public port.
+    path === '/health' ||
     path === '/api/catalog' ||
     path === '/api/catalog/' ||
     path === '/feed.xml' ||

package/src/prewarm.js

@@ -21,6 +21,8 @@
  * the first few seconds if the right few kilobytes are asked for first.
  */
 
+import { guessKind } from './library.js';
+
 /** The first wait after an attempt that did not finish the job. */
 const DEFAULT_BACKOFF_SECONDS = 15;
 
@@ -91,9 +93,20 @@ export class HeadWarmer {
    * @returns {boolean} - True to attempt a read.
    */
   due(entry) {
-    // Only PMTiles has a head worth reading. A .osm.pbf from a feed is not an
-    // archive this can say anything about.
-    if (entry.kind && entry.kind !== 'pmtiles') return false;
+    // Only PMTiles has a head worth reading, and this has to be a positive
+    // test rather than the absence of a negative one.
+    //
+    // `guessKind` answers `undefined` for anything it does not recognise — a
+    // .osm.pbf from a feed, for instance — so `entry.kind && entry.kind !==
+    // 'pmtiles'` never fired for exactly the archives it was meant to exclude.
+    // Every planet dump being mirrored was read as though it had a PMTiles
+    // header, failed, and came back on the backoff for ever.
+    //
+    // Taken from the entry where it is known and from the name where it is
+    // not, since an archive joined by magnet has no kind until its metadata
+    // arrives.
+    const kind = entry.kind ?? guessKind(entry.name ?? '');
+    if (kind !== 'pmtiles') return false;
 
     // A summary that names a format is one a header was actually read for.
     // Anything else — an empty object, or one left behind by a read that raced