pmtiles-swarm 0.7.1 → 0.9.0

package/CHANGELOG.md

@@ -7,6 +7,75 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.9.0
+### ✨ Features and improvements
+- **`pmtiles-swarm status`**, which asks a running node what it is doing and reads the answer out
+  loud. It takes the same config file the node runs with, so the address, the port and the
+  credential come from one place rather than being remembered and retyped. That is the whole
+  point of it: the API is on `adminPort` rather than the public port, the node binds where `host`
+  says and that is usually not loopback, and it accepts `authorization: Bearer` and not
+  `x-api-key`. Get any one of those wrong by hand and the answer is a refused connection or a 401,
+  both of which read as a broken node rather than as a mistyped command — which is exactly how
+  they were read while diagnosing the archive fixed below.
+
+  It names the case that is otherwise silent: an archive the catalog holds and the engine does
+  not, which through `curl` is a row of empty columns and looks like a corrupt archive. Just after
+  a start it is normal and passes; persisting, the engine refused it and the log says why. Exits
+  non-zero when the node does not answer or its engine is down, so it can be the last step of a
+  deployment script, and `--json` hands back the raw replies for anything that would rather parse.
+
+  Also warns when `--config` names a file that is not there. Startup ignores that on purpose, so
+  a first run can write one — but for a question about a running node the silence is
+  misleading, since the answer then describes the default address and looks entirely real.
+
+- **[docs/haproxy.md](docs/haproxy.md) now covers the backend pool**: why round robin rather than
+  the plugin's default of Source-IP Hash, which fails quietly behind a CDN by pinning nearly all
+  traffic to one node while the rest sit idle and healthy; when least-connections or URI hash are
+  worth having instead; and what HTTP/2 on the frontend does and does not change about balancing.
+
+### 🐞 Bug fixes
+- **An archive built from a watched folder no longer sits at 0%, seeding nobody, for a quarter of
+  an hour.** The libtorrent engine dropped `seedOnly` on its way to the sidecar, so libtorrent
+  re-hashed an 81 GiB archive that had been read end to end moments earlier to produce its
+  torrent. Everything else already handled it — the library sets it in five places, the
+  composite engine checks it against what the primary reports, qBittorrent has its own flag for
+  it — and this one engine silently did not pass it on. Needs pmtiles-torrent 0.4.1, which the
+  existing dependency range picks up on a fresh install.
+- **`docs/running-as-a-service.md` no longer suggests checking a node with `curl localhost:8091`.**
+  It names loopback and sends no credential, so on a node bound to its LAN address with a key
+  configured it fails twice over, in the two ways that look most like a broken node. It now uses
+  the status command.
+
+## 0.8.0
+### ✨ Features and improvements
+- **`GET /health`**, for a load balancer: 200 when this node can serve, 503 when its engine
+  cannot, no credential and nothing to parse. It asks the engine rather than itself, which is the
+  distinction that makes it worth having — a feed is built from the catalogue and never touches
+  the swarm, so a balancer checking `/feed.xml`, the nearest thing that existed, gets 200 from a
+  node whose engine is dead and keeps sending it traffic. The answer is cached for two seconds,
+  because a balancer asks often and each check is an inter-process round trip, and it is sent
+  `no-store`: a stale health check keeps a dead node in rotation for as long as whatever cached
+  it says so.
+- **`GET /archives/<infohash>/ready`**, which answers a different question — whether a *particular*
+  archive has become servable on this node. 200 once its header and, for vector, its layers have
+  been read; 503 with which half is missing; **415** for an archive that can never be served,
+  since MBTiles is distributed here but cannot be read a byte range at a time and polling it would
+  be polling for ever; 404 when it is not here. It reports rather than acts, starting no read and
+  waiting for nothing — a probe that does work on demand is a probe that can be used to make a
+  node do work on demand.
+- **[docs/haproxy.md](docs/haproxy.md)**, written against the OPNsense plugin: the health monitor
+  field by field, how the check interval trades against failover time, timeouts long enough for a
+  web seed to finish, `X-Forwarded-Proto`, gating a deployment on `/ready` — and a table of what a
+  reverse proxy in front of a BitTorrent node simply cannot carry.
+
+### 🐞 Bug fixes
+- **Head-warming no longer tries to read a PMTiles header out of a `.osm.pbf`.** The guard was
+  `entry.kind && entry.kind !== 'pmtiles'`, and `guessKind` answers `undefined` for anything it
+  does not recognise — so it never fired for exactly the archives it existed to exclude. Every
+  planet dump being mirrored, and every MBTiles archive, was read as though it had a header,
+  failed, and came back on the backoff for ever. The test is positive now: the kind has to *be*
+  PMTiles, taken from the entry where it is known and from the file name where it is not.
+
 ## 0.7.1
 ### 🐞 Bug fixes
 - **A feed no longer deletes its only complete copy.** Retention was written for a watched folder

package/README.md

@@ -33,6 +33,9 @@ node src/index.js --config swarm.config.json
   address.
 - **[docs/running-as-a-service.md](docs/running-as-a-service.md)** — a systemd unit, why
   `Restart=always` is required rather than optional, and which ports want a firewall rule.
+- **[docs/haproxy.md](docs/haproxy.md)** — health monitors, what a reverse proxy carries and
+  what it cannot, timeouts long enough for a web seed, and gating a deployment on
+  `/archives/<infohash>/ready`.
 - **[docs/architecture-diagram.md](docs/architecture-diagram.md)** — how a publishing node, a
   serving tier, the swarm and both kinds of client fit together.
 
@@ -586,6 +589,38 @@ network equipment: peers request 16 KiB blocks whatever the piece size. The sett
 matters there is `maxConnections`, since every peer holds a NAT table entry. See
 [docs/publishing.md](docs/publishing.md).
 
+## Asking a running node what it is doing
+
+```sh
+node src/index.js status --config /etc/pmtiles-swarm/swarm.config.json
+```
+
+```
+engine  libtorrent  ready
+version 0.8.0
+17 archives, 1 the engine does not know about
+
+NAME                                             SIZE  STATE       PROGRESS
+planetiler-openmaptiles-260803.pmtiles         81 GiB  seeding         100%
+planet-260803.osm.pbf                          94 GiB  downloading      37%
+planetiler-openmaptiles-260810.pmtiles         83 GiB  —                 —
+```
+
+It reads the same config file the node runs with, so the address, the port and the credential come
+from one place rather than being remembered and retyped. That matters more than it sounds: the API
+is on `adminPort`, not the public port; the node binds where `host` says, which is usually not
+loopback; and it accepts `authorization: Bearer`, not `x-api-key`. Get any one of those wrong with
+`curl` and the answer is a refused connection or a 401 — both of which read as a broken node rather
+than a mistyped command.
+
+An archive with a state of `—` is one the catalog holds and the engine is not. Directly after a
+restart that is normal and passes. Persisting, it means the engine refused it, and the log says
+why.
+
+Exit status is 0 when the node answered and its engine is up, 1 when it did not or is not — so it
+works in a deployment script. `--json` gives the raw `/api/status` and `/api/torrents` replies for
+anything that wants to parse rather than read.
+
 ## API
 
 | Method | Path | Purpose |
@@ -626,6 +661,8 @@ matters there is `maxConnections`, since every peer holds a NAT table entry. See
 | `GET` | `/api/catalog` | The whole catalogue, for a peer keeping itself in step |
 | `GET` | `/archives/:infoHash/tiles.json` | TileJSON — **public** |
 | `GET` | `/archives/:infoHash/:z/:x/:y.:ext` | One tile — **public** |
+| `GET` | `/health` | 200 when this node can serve, 503 when its engine cannot — **public**, no credential, for a load balancer |
+| `GET` | `/archives/:infoHash/ready` | Whether this node can serve *this* archive yet: 200 ready, 503 not yet, 415 never — **public** |
 | `GET` | `/archives/:infoHash/archive.torrent` | The `.torrent` a torrent-aware client joins with — **public** |
 | `GET` | `/archives/:infoHash/preview` | Map preview for one archive — admin, not public |
 | `GET` | `/latest/:category/tiles.json`, `/:name.torrent`, `/magnet` | The newest build in a category — **public**. The torrent name is yours to choose, so a link can read `planetiler-openmaptiles-latest.torrent`; it redirects to the immutable URL, which names the download after the build it actually is |

package/docs/haproxy.md

@@ -0,0 +1,235 @@
+# Behind HAProxy
+
+Written against the **OPNsense HAProxy plugin**, which is configured through a
+form rather than a file. The field names below are that plugin's; the generated
+configuration is shown alongside so the same thing can be written by hand.
+
+## What HAProxy carries, and what it cannot
+
+This is the first thing to get straight, because a reverse proxy in front of a
+BitTorrent node carries less than it looks like it does.
+
+| | Through HAProxy? |
+| --- | --- |
+| Tiles, TileJSON, feeds, `.torrent` files, health checks | **Yes** — ordinary HTTP on 8090 |
+| The console and API on 8091 | **No.** Do not publish it |
+| BitTorrent peers on 6881 | **No.** Not HTTP; needs its own forward |
+| WebRTC to browser peers | **No.** Outbound only, and needs nothing |
+
+A node reachable *only* through HAProxy has no incoming peers at all. It will
+still seed to peers it connects to, and still serve tiles, but it is half a
+member of the swarm. 6881 wants a firewall rule of its own, straight to the
+node, TCP **and** UDP.
+
+## The health monitor
+
+**Services → HAProxy → Settings → Health Monitors → Add**
+
+| Field | Value |
+| --- | --- |
+| Name | `pmtiles-swarm` |
+| Check type | `HTTP` |
+| Check interval | `2s` or `5s` — see below |
+| Port to check | *empty*, so it uses the real server's port |
+| HTTP method | `GET` |
+| Request URI | `/health` |
+| HTTP version | `HTTP/1.1` |
+| HTTP host | anything; `localhost` is fine |
+| Custom HTTP check → Expression | exact string match for the HTTP status code |
+| Custom HTTP check → Value | `200` |
+
+`HTTP/1.1` requires a `Host` header, which is why the field is there. The node
+does not route by it, so its value only matters if something in front of it
+does.
+
+Which produces:
+
+```
+backend pmtiles-swarm
+    option httpchk GET /health HTTP/1.1
+    http-check expect status 200
+    server node1 172.16.1.49:8090 check inter 5s
+    server node2 172.16.1.41:8090 check inter 5s
+```
+
+**Do not point the monitor at `/` or at `/feed.xml`.** Both answer 200 from a
+node whose engine is dead — a feed is built from the catalogue and never
+touches the swarm — so the balancer would keep sending traffic to a node that
+cannot serve a tile. `/health` asks the engine and answers 503 when it does not
+reply.
+
+### How often to check
+
+`/health` caches its answer for two seconds, so anything faster than that asks
+the same question twice and gets the same answer. At two seconds, near enough
+every check is a real round trip to the engine — which enumerates every torrent
+the node holds, so the cost grows with the catalogue.
+
+What the interval buys is failover speed, and it is the interval multiplied by
+HAProxy's `fall` count, which defaults to 3:
+
+| Check interval | Marked down after |
+| --- | --- |
+| `2s` | about 6 seconds |
+| `5s` | about 15 seconds |
+
+Neither is wrong. Six seconds is worth having if a node dying mid-request
+matters; fifteen is worth having if it does not, and costs the node less. What
+is not worth having is a sub-second interval, which only asks a cached answer
+more often.
+
+Note also that the node comes *back* on `rise` successful checks — 2 by
+default — so a flapping node re-enters rotation quickly whichever you choose.
+
+## The backend pool
+
+**Settings → Backend Pools → Add**, mode **HTTP (Layer 7)**.
+
+### Balancing algorithm
+
+**Round Robin.** Tile requests are stateless, numerous and roughly the same
+size, which is the case round robin is for.
+
+**Not Source-IP Hash, even though it is the default here.** It exists for
+sticky sessions, and tiles have no session to be sticky about. Worse, it fails
+quietly in exactly the setup this document assumes: behind a CDN every request
+arrives from a handful of edge addresses, so hashing on the source pins almost
+all traffic to one node while the others sit idle and healthy. A NAT'd office
+or a mobile carrier does the same thing on a smaller scale.
+
+Two others are worth knowing about, for arrangements this is not:
+
+**Least Connections**, when long transfers share the backend with short ones.
+A web-seed range request can run for minutes while a tile takes milliseconds,
+and round robin will happily queue tiles behind a transfer. If web seeds are
+served from a different host — an ordinary web server in front of the published
+directory, which is the usual arrangement — that variance is not here and round
+robin is simpler.
+
+**URI Hash**, for a tier of **cache-mode** nodes. Such a node holds only the
+pieces it has read, so sending the same region to different nodes makes each of
+them pay the cold read separately — the one real cost of scaling a cache-mode
+tier horizontally. `balance uri depth 2` hashes on `/archives/<infohash>` and
+gives archive affinity, at the price of concentrating one archive on one node.
+For nodes holding complete copies there is no cold read to avoid, so this is a
+cost with no benefit.
+
+### HTTP/2
+
+Enable it on the frontend and leave *HTTP/2 without TLS* unchecked: the client
+gets HTTP/2, the node is spoken to over HTTP/1.1, which is what it speaks.
+Balancing in HTTP mode is per request rather than per connection, so a client
+multiplexing a hundred tile requests over one HTTP/2 connection still has them
+spread across the pool.
+
+## Real servers
+
+**Settings → Real Servers → Add**, one per node, port **8090**.
+
+The admin port has no place in a public backend. It carries the console and the
+API, and while both are behind authentication, an internet-facing sign-in page
+is a thing to decide on deliberately rather than to acquire by copying a
+backend.
+
+## Tell the node it is behind a proxy
+
+HAProxy terminates TLS, so without being told, the node believes every request
+arrived over plain HTTP and advertises `http://` tile URLs. A browser that
+loaded the map over HTTPS then blocks every one of them as mixed content, which
+looks like an empty map rather than a configuration mistake.
+
+Two ways, and they are alternatives rather than a pair:
+
+```json
+{ "publicUrl": "https://swarm.example.org" }
+```
+
+One canonical URL whatever the request said. Simple, and right when the node is
+only ever reached one way.
+
+```json
+{ "trustProxy": "172.16.1.0/24" }
+```
+
+Derive it per request from `X-Forwarded-Proto` and the `Host`, so the same node
+answers correctly on its LAN address and through the proxy. **Name the proxy
+rather than saying `true`**: the node also listens on its LAN address directly,
+and `true` would let anything that can reach that port claim any protocol and
+host it likes, which decides what URLs your feed hands out.
+
+For that to work HAProxy has to send the header. In the plugin it is
+**Public Service → Advanced → Option pass-through**, or in configuration:
+
+```
+frontend https-in
+    http-request set-header X-Forwarded-Proto https
+```
+
+`Host` is passed through by default.
+
+## Timeouts
+
+The defaults are wrong for this. A web seed is an HTTP range request for part of
+an archive, and a peer pulling a large one holds the connection for as long as
+the transfer takes.
+
+```
+defaults
+    timeout connect 5s
+    timeout client  1h
+    timeout server  1h
+```
+
+An hour is not generous here; a client fetching tens of gigabytes over a slow
+link exceeds anything shorter, and the failure looks like a corrupt download
+rather than a timeout.
+
+## Deploying a new build
+
+`/health` says whether a node should be sent traffic. It says nothing about
+whether a *particular* archive has become servable there, which is the question
+worth asking after a build lands.
+
+```sh
+INFOHASH=5e1c143c400d15aaacfb1c748d4ab6d1b46c5df5
+for node in 172.16.1.49 172.16.1.41; do
+  until curl -fsS "http://$node:8090/archives/$INFOHASH/ready" >/dev/null; do
+    sleep 10
+  done
+  echo "$node is ready"
+done
+```
+
+Ask each node **directly**, not through the balancer — through it you learn that
+*some* node is ready, which is not the same thing and is exactly the wrong
+answer when deciding whether to move traffic.
+
+`/ready` answers 415 for an archive that can never be served, so a loop like the
+one above would never end if an MBTiles archive reached it. Treat 415 as a
+separate case if that is possible.
+
+## Cloudflare in front
+
+Two rules worth setting deliberately.
+
+**Never cache `/health` or `/archives/*/ready`.** Both send `no-store` and
+Cloudflare honours it, but a page rule that caches everything can override that.
+A cached health check keeps a dead node in rotation for as long as the cache
+says so, which is worse than having no check at all.
+
+**Cache tiles by infohash aggressively.** `/archives/<infohash>/…` is immutable
+by construction — an infohash names those bytes and no others — and is served
+with `max-age=31536000, immutable`. `/latest/<category>/…` is the opposite: it
+moves on every build and is served with `max-age=300`.
+
+WebRTC does not pass through Cloudflare, and neither does BitTorrent. Browser
+peers reach the node over ICE, and a `wss://` tracker is the only part of that
+which is HTTP at all.
+
+## When it looks like the swarm is empty
+
+Almost always the peer port rather than the proxy. HAProxy carries none of it,
+so a node behind one with no forward for 6881 shows connected peers only where
+it made the connection itself. See
+[docs/engines.md](engines.md#ports-and-reachability) for which ports want
+forwarding and which want nothing.

package/docs/running-as-a-service.md

@@ -441,9 +441,23 @@ Then check it is actually serving:
 
 ```sh
 curl -fsS localhost:8090/feed.xml >/dev/null && echo "public surface ok"
-curl -fsS localhost:8091/api/status | head -c 200
+node /opt/pmtiles-swarm/src/index.js status \
+  --config /etc/pmtiles-swarm/swarm.config.json
 ```
 
+Ask through the status command rather than with `curl`. Reaching the API by hand
+means getting the bind address, the admin port and the credential right in one
+go, and each of them fails in a way that looks like a broken node: a node bound
+to its LAN address refuses a request to `localhost`, and the header it accepts
+is `authorization: Bearer`, so anything else is a 401. The status command reads
+all three out of the config file the service is running with. It exits non-zero
+when the node does not answer or its engine is down, so it also works as the
+last step of a deployment script.
+
+An archive listed with a state of `—` is one the catalog holds and the engine is
+not. Just after a start that is normal and passes within a minute or so. If it
+persists, the engine refused it, and the journal says why.
+
 And that it can write where it is supposed to, which nothing above proves:
 
 ```sh

package/docs/serving-tiles.md

@@ -200,6 +200,85 @@ For an archive published as a mutable torrent, the block also carries
 rather than pinning to the version the document was generated from. See
 [publishing](publishing.md).
 
+## Health checks
+
+```
+GET /health
+```
+
+200 when this node can serve, 503 when it cannot, no credential and no body
+worth parsing — which is what a load balancer needs. It is on the public
+surface, so it answers on the same port the tiles do.
+
+**It asks the engine, not just itself.** A reply means the engine answered a
+round trip, which is the difference worth reporting: a feed is built from the
+catalog and never touches the swarm, so a balancer checking `/feed.xml` gets
+200 from a node whose engine is dead and keeps sending it traffic.
+
+The answer is cached for a couple of seconds, because a balancer asks often and
+each check costs an inter-process round trip. A node that has just died leaves
+rotation one check later than it otherwise would.
+
+It sends `cache-control: no-store`. A stale health check is worse than none —
+it keeps a dead node in rotation for as long as whatever cached it says so.
+
+```
+backend tiles
+    option httpchk GET /health
+    http-check expect status 200
+    server node1 10.0.0.11:8090 check inter 5s
+```
+
+Configured through a form rather than a file, and with the rest of what a proxy
+in front of this needs — timeouts, `X-Forwarded-Proto`, and the ports it cannot
+carry — in [docs/haproxy.md](haproxy.md).
+
+### Whether one archive is servable yet
+
+```
+GET /archives/<infohash>/ready
+```
+
+A different question, and worth keeping apart from the one above. `/health`
+decides whether a node should be sent traffic at all; this says whether a
+newly published archive has become servable *here* — which is what you want
+after a build lands and before pointing anything at it.
+
+| | |
+| --- | --- |
+| **200** | Ready. Its header has been read, and a vector archive has its layers |
+| **503** | Not yet — ask again. The body says which half is missing |
+| **415** | Never. MBTiles is distributed here but cannot be read a byte range at a time, so waiting would be waiting for ever |
+| **404** | Not on this node |
+
+The codes differ because the responses differ: one is "poll me", one is "stop
+polling", and a script that treats them alike either gives up too early or
+waits for something that will never happen.
+
+It **reports rather than acts** — it starts no read and waits for nothing. A
+probe that does work on demand is a probe that can be used to make a node do
+work on demand. Reading the head is the head warmer's job; this only says
+whether it has happened.
+
+So the shape of a deployment check across a serving tier is: publish, then poll
+every node until each answers 200, then move `latest`.
+
+```sh
+for node in 10.0.0.11 10.0.0.12; do
+  until curl -fsS "http://$node:8090/archives/$INFOHASH/ready" >/dev/null; do
+    sleep 10
+  done
+done
+```
+
+`curl -f` fails on 503 and on 415 alike, so treat 415 separately if an MBTiles
+archive could ever reach that loop — otherwise it never ends.
+
+An archive this node holds *completely* can still be read from disk with the
+engine down, so 503 is a statement about the node rather than about every
+request it could answer. That is the right way round for a balancer with
+somewhere else to send the traffic.
+
 ## Caching
 
 Tiles are served `Cache-Control: public, max-age=31536000, immutable`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.7.1",
+  "version": "0.9.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -13,6 +13,7 @@ import {
   isPublicSurface,
 } from './auth.js';
 import { normalizeCategories } from './catalog.js';
+import { guessKind } from './library.js';
 import { QBittorrentEngine } from './engines/qbittorrent.js';
 import { RESTART_REQUIRED, redactConfig, saveConfig } from './config.js';
 import { freeSpace, listLocations } from './locations.js';
@@ -397,6 +398,59 @@ export function createApp({
     }),
   );
 
+  /**
+   * Whether this node should be sent traffic.
+   *
+   * For a load balancer, which needs three things this did not have: no
+   * credential, a cheap answer, and a status code rather than a body to parse.
+   * A balancer checking `/feed.xml` instead — the nearest thing that existed —
+   * gets 200 from a node whose engine is dead, because a feed is built from
+   * the catalog and never touches the swarm.
+   *
+   * Readiness rather than liveness. `engine.list()` is a round trip to the
+   * engine, so a reply means the sidecar is answering and not merely that Node
+   * is; that is the difference worth reporting, since everything that makes
+   * this node useful to a swarm goes through it.
+   *
+   * Cached, because a balancer asks every couple of seconds and an IPC round
+   * trip per check is a cost with nothing to show for it. The window is short
+   * enough that a node which has just died is out of rotation within one more
+   * check than it would have been.
+   */
+  let healthChecked = 0;
+  let healthOk = true;
+  let healthError;
+  const HEALTH_TTL_MS = 2000;
+
+  app.get(
+    '/health',
+    route(async (_req, res) => {
+      const now = Date.now();
+      if (now - healthChecked >= HEALTH_TTL_MS) {
+        healthChecked = now;
+        try {
+          await engine.list();
+          healthOk = true;
+          healthError = undefined;
+        } catch (error) {
+          healthOk = false;
+          healthError = error.message;
+        }
+      }
+
+      // Never cached anywhere. A stale health check is worse than none: it
+      // keeps a dead node in rotation for as long as whatever cached it says.
+      res.setHeader('cache-control', 'no-store');
+      res.setHeader('access-control-allow-origin', '*');
+      res.status(healthOk ? 200 : 503).json({
+        status: healthOk ? 'ok' : 'unavailable',
+        engine: engine.name,
+        version: VERSION,
+        ...(healthOk ? {} : { error: healthError }),
+      });
+    }),
+  );
+
   app.get(
     '/api/status',
     route(async (_req, res) => {
@@ -1828,6 +1882,95 @@ export function createApp({
   // with authentication configured this answered 401 — to the very callers it
   // exists for, since this is the URL the TileJSON torrent block advertises and
   // the one a syncing peer follows.
+  /**
+   * Whether this node can serve *this* archive yet.
+   *
+   * A different question from `/health`, and answered separately because
+   * nobody asks it per request. `/health` decides whether a node should be
+   * sent traffic at all; this decides whether a newly published archive has
+   * become servable here — which is what you want to know after a build lands
+   * and before pointing anything at it.
+   *
+   * Reports rather than acts. It starts no read and waits for nothing: a probe
+   * that does work on demand is a probe that can be used to make a node do
+   * work on demand. The head warmer is what makes an archive ready; this only
+   * says whether it has.
+   *
+   * The three answers are deliberately different codes, because they call for
+   * different responses from whoever asked. 503 is "not yet, ask again". 415
+   * is "never" — an MBTiles archive is distributed here and cannot be read a
+   * byte range at a time, so waiting for it would be waiting for ever. 404 is
+   * "not here at all".
+   */
+  app.get(
+    '/archives/:infoHash/ready',
+    route(async (req, res) => {
+      const entry = catalog.get(req.params.infoHash);
+      // Never cached. The whole value of this is that it changes.
+      res.setHeader('cache-control', 'no-store');
+      res.setHeader('access-control-allow-origin', '*');
+
+      if (!entry) {
+        return res.status(404).json({ ready: false, reason: 'unknown archive' });
+      }
+
+      const kind = entry.kind ?? guessKind(entry.name ?? '');
+      const shape = {
+        infoHash: entry.infoHash,
+        name: entry.name,
+        kind: kind ?? 'unknown',
+        complete: entry.complete === true,
+      };
+
+      if (kind !== 'pmtiles') {
+        return res.status(415).json({
+          ...shape,
+          ready: false,
+          reason:
+            `this is ${kind ? `a ${kind}` : 'not a PMTiles'} archive, and only ` +
+            'PMTiles can be read a byte range at a time — it will not become ' +
+            'servable by waiting',
+        });
+      }
+
+      const summary = entry.pmtiles;
+      // A summary that names a format is one a header was actually read for.
+      // Anything else is a partial left by a read that did not finish.
+      if (!summary?.format) {
+        return res.status(503).json({
+          ...shape,
+          ready: false,
+          reason: 'its header has not been read yet',
+        });
+      }
+
+      // Vector tiles without their layer list can be served, but nothing can
+      // be styled from them: the TileJSON a map asks for would carry no
+      // vector_layers. The metadata sits wherever the writer put it, which for
+      // planetiler is after every tile, so it routinely arrives long after the
+      // header.
+      if (summary.format === 'pbf' && !summary.vectorLayers) {
+        return res.status(503).json({
+          ...shape,
+          ready: false,
+          format: summary.format,
+          reason: 'its metadata has not been read yet, so it carries no vector layers',
+        });
+      }
+
+      res.json({
+        ...shape,
+        ready: true,
+        format: summary.format,
+        minZoom: summary.minZoom,
+        maxZoom: summary.maxZoom,
+        ...(summary.vectorLayers
+          ? { vectorLayers: summary.vectorLayers.length }
+          : {}),
+      });
+    }),
+  );
+
   app.get(
     '/archives/:infoHash/archive.torrent',
     route(async (req, res) => {

package/src/auth.js

@@ -144,6 +144,8 @@ export function isPublicSurface(path) {
   if (/^\/archives\/[^/]+\/preview\/?$/.test(path)) return false;
 
   return (
+    // A load balancer checks this, and it checks the public port.
+    path === '/health' ||
     path === '/api/catalog' ||
     path === '/api/catalog/' ||
     path === '/feed.xml' ||

package/src/engines/libtorrent.js

@@ -254,6 +254,12 @@ export class LibtorrentEngine {
       savePath: request.savePath ?? this.#options.savePath,
       mode: request.mode,
       paused: request.paused,
+      // The caller's claim that the data is already on disk, which for an
+      // archive created here it is: the file was read end to end a moment ago
+      // to produce the torrent. Without passing it on, libtorrent hashes the
+      // whole archive again before seeding a byte — a quarter of an hour for
+      // an 81 GiB build, during which it reads as 0% and serves nobody.
+      seedOnly: request.seedOnly,
     });
     return result.infoHash;
   }

package/src/index.js

@@ -92,20 +92,26 @@ function createOneEngine(name, config) {
  * @returns {Promise<void>} - Resolves once listening.
  */
 async function main() {
-  const { values } = parseArgs({
+  const { values, positionals } = parseArgs({
     options: {
       config: { type: 'string', short: 'c' },
       port: { type: 'string', short: 'p' },
       help: { type: 'boolean', short: 'h' },
+      json: { type: 'boolean' },
     },
-    allowPositionals: false,
+    allowPositionals: true,
   });
 
   if (values.help) {
     console.log(`pmtiles-swarm — BitTorrent distribution for PMTiles archives
 
+Usage:
+  pmtiles-swarm [--config FILE]          start the node
+  pmtiles-swarm status [--config FILE]   ask a running node what it is doing
+
   --config, -c   path to a JSON config file
   --port,   -p   override the listen port
+  --json         machine-readable output, for the status command
   --help,   -h   this message
 
 Environment: PMTILES_SWARM_PORT, PMTILES_SWARM_DATA_DIR, PMTILES_SWARM_ENGINE,
@@ -118,6 +124,23 @@ PMTILES_SWARM_PUBLIC_URL
   const config = await loadConfig(values.config);
   if (values.port) config.port = Number(values.port);
 
+  // Asking rather than starting. Everything it needs — which address the admin
+  // listener is on, which port, and the credential — comes from the same
+  // configuration the node runs with, so there is nothing to pass and nothing
+  // to get wrong.
+  if (positionals[0] === 'status') {
+    const { runStatus } = await import('./status-command.js');
+    process.exitCode = await runStatus(config, { json: values.json });
+    return;
+  }
+
+  if (positionals.length > 0) {
+    console.error(`unknown command: ${positionals[0]}`);
+    console.error('try: pmtiles-swarm status');
+    process.exitCode = 2;
+    return;
+  }
+
   // Everything that has to be stopped, in the order it should be stopped,
   // filled in as startup proceeds.
   //

package/src/prewarm.js

@@ -21,6 +21,8 @@
  * the first few seconds if the right few kilobytes are asked for first.
  */
 
+import { guessKind } from './library.js';
+
 /** The first wait after an attempt that did not finish the job. */
 const DEFAULT_BACKOFF_SECONDS = 15;
 
@@ -91,9 +93,20 @@ export class HeadWarmer {
    * @returns {boolean} - True to attempt a read.
    */
   due(entry) {
-    // Only PMTiles has a head worth reading. A .osm.pbf from a feed is not an
-    // archive this can say anything about.
-    if (entry.kind && entry.kind !== 'pmtiles') return false;
+    // Only PMTiles has a head worth reading, and this has to be a positive
+    // test rather than the absence of a negative one.
+    //
+    // `guessKind` answers `undefined` for anything it does not recognise — a
+    // .osm.pbf from a feed, for instance — so `entry.kind && entry.kind !==
+    // 'pmtiles'` never fired for exactly the archives it was meant to exclude.
+    // Every planet dump being mirrored was read as though it had a PMTiles
+    // header, failed, and came back on the backoff for ever.
+    //
+    // Taken from the entry where it is known and from the name where it is
+    // not, since an archive joined by magnet has no kind until its metadata
+    // arrives.
+    const kind = entry.kind ?? guessKind(entry.name ?? '');
+    if (kind !== 'pmtiles') return false;
 
     // A summary that names a format is one a header was actually read for.
     // Anything else — an empty object, or one left behind by a read that raced

package/src/status-command.js

@@ -0,0 +1,233 @@
+/**
+ * `pmtiles-swarm status` — asking a running node what it is doing.
+ *
+ * This exists because interrogating a node meant getting four separate things
+ * right at once: which address the admin listener is bound to, which port,
+ * which header carries the credential, and where in the JSON the answer lives.
+ * Getting any one wrong produces something that looks like a broken archive —
+ * a refused connection, a 401, or a row of nulls — rather than like a mistyped
+ * command. Every one of those is derivable from the configuration file the
+ * node is already running with, so nothing here needs to be passed or
+ * remembered.
+ */
+
+import { access } from 'node:fs/promises';
+
+/**
+ * Whether a path can be read.
+ * @param {string} path - The path.
+ * @returns {Promise<boolean>} - True when it is there.
+ */
+async function readable(path) {
+  try {
+    await access(path);
+    return true;
+  } catch {
+    return false;
+  }
+}
+
+/** Columns, and how wide the name column may grow before it is cut. */
+const NAME_WIDTH = 44;
+
+/**
+ * A size in bytes, as a person would write it.
+ * @param {number} value - Bytes.
+ * @returns {string} - e.g. "81 GiB".
+ */
+export function bytes(value) {
+  if (!Number.isFinite(value) || value <= 0) return '—';
+  const units = ['B', 'KiB', 'MiB', 'GiB', 'TiB'];
+  let size = value;
+  let unit = 0;
+  while (size >= 1024 && unit < units.length - 1) {
+    size /= 1024;
+    unit += 1;
+  }
+  return `${size < 10 && unit > 0 ? size.toFixed(1) : Math.round(size)} ${units[unit]}`;
+}
+
+/**
+ * Where this node's API is, according to its own configuration.
+ *
+ * The admin listener where there is one, since that is where the API lives on
+ * a node that separates them. A wildcard bind is reported as loopback: `::` is
+ * what the node listens on, not an address anything can connect to.
+ * @param {object} config - Resolved configuration.
+ * @returns {string} - An origin, e.g. "http://172.16.1.49:8091".
+ */
+export function adminUrl(config) {
+  const host = config.adminHost ?? config.host ?? '127.0.0.1';
+  const port = config.adminPort ?? config.port ?? 8090;
+  const reachable =
+    host === '0.0.0.0' || host === '::' || host === '' ? '127.0.0.1' : host;
+  // A bare IPv6 address needs brackets before it is a URL.
+  const bracketed =
+    reachable.includes(':') && !reachable.startsWith('[')
+      ? `[${reachable}]`
+      : reachable;
+  return `http://${bracketed}:${port}`;
+}
+
+/**
+ * The header a request to this node's API needs, if any.
+ *
+ * `authorization: Bearer`, which is the only form the node accepts — not
+ * `x-api-key`, whatever the convention elsewhere.
+ * @param {object} config - Resolved configuration.
+ * @returns {object} - Headers to send.
+ */
+export function authHeaders(config) {
+  const key = config.auth?.apiKey;
+  return key ? { authorization: `Bearer ${key}` } : {};
+}
+
+/**
+ * One line per archive, plus what the engine says about the node.
+ * @param {object} answer - `{ status, torrents }` as the API returned them.
+ * @returns {string} - The report.
+ */
+export function formatStatus({ status, torrents }) {
+  const lines = [];
+  const engine = status?.engine;
+  lines.push(
+    `engine  ${engine?.name ?? 'unknown'}` +
+      (engine?.ok === false ? `  UNAVAILABLE — ${engine.error ?? ''}` : '  ready'),
+  );
+  if (status?.version) lines.push(`version ${status.version}`);
+
+  const rows = torrents ?? [];
+  // An archive the engine has never heard of is the case worth naming. It is
+  // in the catalog, it has a size, and every live column is empty — which
+  // reads as a broken archive and is usually a node that has not finished
+  // starting, or one that could not add it.
+  const unknown = rows.filter((row) => !row.status).length;
+  lines.push(
+    `${rows.length} archive${rows.length === 1 ? '' : 's'}` +
+      (unknown > 0 ? `, ${unknown} the engine does not know about` : ''),
+  );
+  lines.push('');
+
+  if (rows.length === 0) return `${lines.join('\n')}\n`;
+
+  const head =
+    'NAME'.padEnd(NAME_WIDTH) +
+    'SIZE'.padStart(9) +
+    '  ' +
+    'STATE'.padEnd(12) +
+    'PROGRESS'.padStart(8);
+  lines.push(head);
+
+  for (const row of rows) {
+    const name =
+      row.name.length > NAME_WIDTH - 1
+        ? `${row.name.slice(0, NAME_WIDTH - 2)}…`
+        : row.name;
+    const state = row.status?.state ?? (row.paused ? 'paused' : '—');
+    const progress =
+      typeof row.status?.progress === 'number'
+        ? `${Math.round(row.status.progress * 100)}%`
+        : '—';
+    lines.push(
+      name.padEnd(NAME_WIDTH) +
+        bytes(row.size).padStart(9) +
+        '  ' +
+        String(state).padEnd(12) +
+        progress.padStart(8),
+    );
+  }
+
+  if (unknown > 0) {
+    lines.push('');
+    lines.push(
+      'An archive with no state is one the engine is not holding. If the node',
+    );
+    lines.push(
+      'has just started it may still be handing them back; if it persists, the',
+    );
+    lines.push('log will say why it could not be added.');
+  }
+
+  return `${lines.join('\n')}\n`;
+}
+
+/**
+ * Asks a running node for its status and prints it.
+ * @param {object} config - Resolved configuration.
+ * @param {object} [options] - Injectable fetch and output, for testing.
+ * @returns {Promise<number>} - The exit code.
+ */
+export async function runStatus(config, options = {}) {
+  const {
+    fetch: get = globalThis.fetch,
+    out = (text) => process.stdout.write(text),
+    err = (text) => process.stderr.write(text),
+    json = false,
+  } = options;
+
+  // A named config file that is not there is silently ignored on startup, so
+  // that a first run can write one. Here that silence is misleading: a typo in
+  // --config means this reports on the default address with no key, which is a
+  // different node from the one that was asked about, and the answer looks
+  // real. Say it, rather than letting it be discovered later.
+  if (config.configPath && !(await readable(config.configPath))) {
+    err(
+      `no config file at ${config.configPath} — using defaults, ` +
+        'which is probably not the node you meant.\n',
+    );
+  }
+
+  const base = adminUrl(config);
+  const headers = authHeaders(config);
+
+  let status;
+  let torrents;
+  try {
+    const [statusReply, torrentsReply] = await Promise.all([
+      get(`${base}/api/status`, { headers }),
+      get(`${base}/api/torrents`, { headers }),
+    ]);
+
+    // Said plainly, because a 401 here means the key in this configuration is
+    // not the key the node is running with — which is a different problem from
+    // the node being down, and looks identical without being told.
+    if (statusReply.status === 401 || statusReply.status === 403) {
+      err(
+        `${base} refused the credential in this configuration file.\n` +
+          'The node is running, but with a different auth.apiKey.\n',
+      );
+      return 1;
+    }
+    if (!statusReply.ok) {
+      err(`${base}/api/status answered ${statusReply.status}\n`);
+      return 1;
+    }
+
+    status = await statusReply.json();
+    torrents = torrentsReply.ok ? await torrentsReply.json() : [];
+  } catch (error) {
+    // A refused connection is the commonest failure and the least obvious: the
+    // node binds where the configuration says, which is often not loopback.
+    // Node's fetch reports every one of them as "fetch failed" and puts the
+    // part worth reading — refused, timed out, no such host — in `cause`.
+    const reason = error.cause?.code
+      ? `${error.message} (${error.cause.code})`
+      : error.message;
+    err(
+      `could not reach ${base}: ${reason}\n` +
+        'That address comes from adminHost and adminPort in this configuration ' +
+        'file.\nIs the node running, and bound where this says?\n',
+    );
+    return 1;
+  }
+
+  if (json) {
+    out(`${JSON.stringify({ status, torrents }, null, 2)}\n`);
+  } else {
+    out(formatStatus({ status, torrents }));
+  }
+
+  // Usable from a script: the engine being unreachable is the thing worth
+  // failing on, and it is what /health reports to a load balancer.
+  return status?.engine?.ok === false ? 1 : 0;
+}