pmtiles-swarm 0.58.1 → 0.60.0

package/CHANGELOG.md

@@ -7,6 +7,143 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.60.0
+### ✨ Features and improvements
+- **A tile URL that survives a rebuild.** Every archive is addressed by infohash, which is
+  what makes a tile cacheable for a year and what makes it useless in an application: the URL
+  changes with every build, so anything that wrote one down is pinned to a build that
+  eventually stops existing. A category is the only stable handle this system has, and now it
+  has a tile endpoint of its own — `/latest/<category>/{z}/{x}/{y}.<ext>` — which resolves to
+  whichever build is current on every request.
+
+  The category TileJSON advertises it as `tiles`, so a style written once keeps working across
+  a rebuild without being re-fetched for the URLs alone. The immutable template is still
+  published beside it as `latest.tiles`, because it is still the better URL for anything that
+  can re-read the document: it is content-addressed, so it caches for a year and never
+  revalidates. Offering only one of the two would be choosing for the consumer, and the right
+  answer differs by consumer.
+
+  The console offers the template as a copyable **XYZ** button next to the TileJSON one, which
+  is the form a Leaflet layer, an OpenLayers source or a GIS client actually wants.
+
+  Deliberately not a redirect to the immutable URL, though every other `/latest/` route is one.
+  A redirect costs a round trip and a map asks for hundreds of tiles: what is a negligible
+  indirection for a `.torrent` is the difference between a map that feels immediate and one
+  that does not. It is cached as the moving target it is — `max-age=300, must-revalidate`,
+  tagged with the build it resolved to, so a revalidation is a 304 while that build stands and
+  a miss the moment it moves.
+
+- **A benchmark for "why does this feel slower than the other one", in `tools/tile-bench.mjs`.**
+  It reads two TileJSON documents, picks tiles inside the zoom range and bounds both can serve,
+  and requests the same set from each — one server at a time, because run together they compete
+  for the same link and each measures the other's load as its own latency.
+
+  It reports percentiles rather than an average, since what makes a map feel slow is the tail
+  and a mean built from nineteen fast requests does not move for the twentieth. Time to first
+  byte is separated from the total, which is the difference between a slow lookup and a slow
+  link — they want opposite fixes.
+
+  It also detects a pool of unequal nodes, which is a common cause and an invisible one: half
+  the tiles arrive quickly and half do not, and balanced evenly the mean looks tolerable
+  throughout. Two distinct groups are reported as two, with a histogram, and `--header` tallies
+  a response header naming which backend answered. `--a-origin`/`--b-origin` send the tile
+  requests somewhere other than the document was read from, which is the only way to measure
+  one node directly: a node with `publicUrl` set answers with that name however it was asked.
+
+- **`docs/haproxy.md` covers a pool whose nodes are not the same speed.** Round robin assumes
+  the pool is interchangeable, and a tile server on an NVMe disk and one on a spinning disk are
+  not — an archive read is a seek into a large file, which is what a spinning disk is worst at.
+  `backup`, `weight` and least-connections are compared, along with what each does and does not
+  fix.
+
+### 🐞 Bug fixes
+- **An archive at 100% and seeding could serve no tiles until the node was restarted.** Which
+  source an archive is read through is decided once, when a reader opens it, and every other
+  thing that can change that answer already invalidates the reader: a pause, a resume, a mode
+  change, a move, a finished download. A finished *check* did not — and it is the easiest of
+  them to reach, because during `checking_files` libtorrent reports `progress` as the fraction
+  hashed so far, which is indistinguishable from a download sitting at the same figure.
+
+  So a tile read arriving mid-check opened against the swarm, correctly, and kept that handle
+  afterwards. The archive then read from a swarm whose only member is this node, while the
+  whole file sat on the disk beside it, and nothing evicted the handle short of a restart. The
+  completion sweep looked straight past it: an entry already recorded complete never reaches
+  the code that would have noticed.
+
+  The sweep now drops a reader that is going to the swarm for an archive the disk says is
+  whole. The disk is checked before the handle is dropped rather than after, or a reader that
+  would only re-open against the swarm anyway would be invalidated on every sweep, for ever.
+  Cache mode is left alone: it reads from the swarm because that is what it is for.
+
+## 0.59.0
+### ✨ Features and improvements
+- **Requires pmtiles-torrent 0.10.2**, which is what actually ends the re-checking: a
+  `seedOnly` add now discards resume data that would cancel the claim, and the periodic save
+  leaves a hashing torrent alone. Everything below only stops the node making more of it.
+- **`pmtiles-swarm init` writes a first configuration, and optionally the unit to run it.**
+  Every path it writes is absolute, which is the one mistake it exists to make impossible: a
+  relative path resolves against the config file, the documented layout puts that file in
+  `/etc`, and so `./data` there means a catalog, a resume directory and potentially a 700 GiB
+  archive on the configuration partition. State under `/etc` is refused rather than warned
+  about — at the moment a config is written there is nothing to migrate, and the same mistake
+  found later costs a stopped service and a careful move.
+
+  `--systemd` also writes `pmtiles-swarm.service` beside it, with `ReadWritePaths` **derived
+  from the configuration it just wrote**. That derivation is the point. A unit and a config
+  that disagree is every systemd failure this project has diagnosed, and none of them look
+  like what they are: a `savePath` missing from that line is refused inside the unit's
+  namespace before any permission bit is read, so the directory's owner and mode are both
+  perfect and the write still fails. Two files generated from one source cannot drift.
+
+  It installs nothing. The unit is written next to the configuration and the `cp` into
+  `/etc/systemd/system` is one of the commands it prints, alongside an `install -d` for every
+  directory involved with the right owner and mode.
+
+  `--password` is hashed before it is written, and left out entirely when none is given.
+  There is deliberately no placeholder: `auth.password` accepts plaintext, so `REPLACE-ME` in
+  that field is a working password until somebody notices — a credential that looks set and
+  is not.
+
+- **Two scripts for diagnosing a library that re-checks on every start**, in `tools/`.
+  `resume-doctor.py` reads a node's real configuration, catalog, stored `.torrent` files and
+  resume directory and says, for each archive, what libtorrent will do on the next start and
+  why — applying libtorrent's own rules and citing the file and line each came from. It also
+  reads the unit through `systemctl show`, does the arithmetic on every deadline that can cut
+  a resume save short, and hashes pieces rather than trusting the catalog. `resume-experiment.py`
+  proves the five behaviours involved on whatever libtorrent is actually installed, because
+  1.2, 2.0 and 2.1 differ enough that a claim verified on one is not a claim about the other.
+
+### 🐞 Bug fixes
+- **A stop no longer abandons the sidecar mid-write, which is where resume data was going.**
+  Three separate bounds decided how long the engine step had, and the smallest won: eight
+  seconds for the step, fifteen for the whole shutdown, fifteen for the shutdown RPC. The
+  sidecar allows each torrent two seconds of its resume-save budget, so past four archives the
+  node gave up first and every torrent it had not persisted re-hashed its whole store on the
+  way back up. That is the state a library gets stuck in: checking, on every start, for hours.
+
+  The engine step is now worked out from the catalog — two seconds a torrent, over a floor —
+  and the shutdown watchdog is derived from the steps it is meant to contain rather than being
+  a second deadline kept in agreement with them by hand. It was not in agreement.
+
+  The same fixed 60s applied to the periodic save, so past thirty archives the call gave up
+  before the sidecar finished, and the written/asked counts never came back — which is why the
+  shortfall this reports could not be seen from outside. **`TimeoutStopSec` in the documented
+  unit rises from 45 to 300 seconds**, and existing installs need it raised by hand.
+
+- **A `.torrent` that moved with `dataDir` is found again instead of silently becoming a
+  magnet.** `torrentPath` is recorded absolute, so moving state out of `/etc` — which
+  `docs/running-as-a-service.md` tells you to do — left every catalog entry naming a directory
+  that no longer existed. Nothing repointed them and nothing complained, because an unreadable
+  `.torrent` was treated as "use the magnet instead".
+
+  That fallback is the damage rather than a graceful degradation. A magnet carries no metadata
+  and neither does resume data, so the archive waits on BEP 9 for a file list that only a peer
+  can supply — and for an archive this node originated there is nobody to ask. Seen in the
+  field: twenty archives at 0% in `downloading_metadata`, indefinitely, after one documented
+  migration. The current `dataDir` is now tried second, the recorded path is corrected in
+  place so the warning is printed once rather than for ever, and falling back to a magnet at
+  all now says so.
+
 ## 0.58.1
 ### 🐞 Bug fixes
 - **The sample configuration put every piece of state under the config file.** `"dataDir": "./data"`,

package/docs/haproxy.md

@@ -114,6 +114,70 @@ gives archive affinity, at the price of concentrating one archive on one node.
 For nodes holding complete copies there is no cold read to avoid, so this is a
 cost with no benefit.
 
+### When the nodes are not the same speed
+
+Round robin assumes the pool is interchangeable, and a tile server on an NVMe
+disk and one on a spinning disk are not. Every archive read is a seek into a
+large file — the PMTiles directory, then the leaf, then the tile — and that is
+precisely the access pattern a spinning disk is worst at. The two can easily
+differ by an order of magnitude.
+
+What that does to a pool is worse than the average suggests. Balanced evenly,
+half the tiles arrive quickly and half do not, and a map is judged by the slow
+half: panning stutters wherever the slow node answered. The mean looks
+tolerable throughout, which is why this is easy to miss and easy to blame on
+the software.
+
+Three ways out, and they answer different questions.
+
+**`backup`** keeps the slow node out of service entirely until the fast one
+fails:
+
+```
+backend pmtiles-swarm
+    option httpchk GET /health HTTP/1.1
+    http-check expect status 200
+    server node1 172.16.1.49:8090 check inter 5s
+    server node2 172.16.1.41:8090 check inter 5s backup
+```
+
+Every request goes to `node1` while it is healthy, and the pool falls to
+`node2` when it is not. Latency is then the fast node's latency, and the slow
+one is redundancy rather than capacity. This is the right answer when the fast
+node can carry the load alone, which for tile serving it usually can.
+
+**`weight`** keeps both in service and sends proportionally less to the slow
+one:
+
+```
+    server node1 172.16.1.49:8090 check inter 5s weight 200
+    server node2 172.16.1.41:8090 check inter 5s weight 20
+```
+
+Worth it only when you actually need the second node's throughput. It does not
+remove the slow answers, it makes them rarer — a tenth of requests at ten times
+the latency is still a tenth of requests a person notices.
+
+**Least Connections** is the better of the two ways to keep both in service,
+and it needs no numbers chosen by hand. A slow node holds each connection
+longer, so it accumulates open ones and stops being picked until it catches up:
+the pool tunes itself to what the disks are doing today rather than to a weight
+somebody guessed last year. It shares `weight`'s limitation — the slow answers
+become proportionally rarer, not absent — and it pairs with `backup` without
+conflict, since a backup server is held out of the pool whatever the algorithm.
+
+`tools/tile-bench.mjs` shows which situation you are in. A pool of unequal
+nodes answers in two distinct groups rather than one spread, and the tool says
+so explicitly; `--header` tallies a response header naming the backend, if
+HAProxy is set to add one:
+
+```
+http-response set-header X-Served-By %s
+```
+
+Run it before and after the change. What should move is the tail: `total p90`
+and `total p99` collapse towards `p50` once one node is answering everything.
+
 ### HTTP/2
 
 Enable it on the frontend and leave _HTTP/2 without TLS_ unchecked: the client
@@ -219,9 +283,16 @@ says so, which is worse than having no check at all.
 
 **Cache tiles by infohash aggressively.** `/archives/<infohash>/…` is immutable
 by construction — an infohash names those bytes and no others — and is served
-with `max-age=31536000, immutable`. `/latest/<category>/…` is the opposite: it
-moves on every build, and is served with `max-age=60, must-revalidate` and an
-ETag naming the build it resolved to.
+with `max-age=31536000, immutable`.
+
+Everything under `/latest/<category>/` is the opposite, because it moves on
+every build. The documents — `tiles.json`, the feeds — are served with
+`max-age=60, must-revalidate`; the tiles with `max-age=300`. Both carry an ETag
+naming the build they resolved to, so a revalidation is a 304 while the build
+stands and a miss the moment it moves. Let Cloudflare revalidate rather than
+overriding either with a page rule: the whole point of the category tile URL is
+that it is safe to write into an application, and it is only safe while the
+edge notices a rebuild.
 
 **Do not strip or rewrite the ETag on either.** It is the infohash, which is
 how a PMTiles reader notices that the archive moved underneath a read already

package/docs/running-as-a-service.md

@@ -151,8 +151,50 @@ sudo apt-get install -y python3-libtorrent
 sudo -u pmtiles-swarm python3 -c "import libtorrent; print(libtorrent.__version__)"
 ```
 
+## The configuration, and a unit for it
+
+`init` writes both, and the unit's `ReadWritePaths` is derived from the
+configuration it just wrote. That derivation is the reason to use it: a unit and
+a configuration that disagree is the failure this whole document is arranged
+around, and two files generated from one source cannot.
+
+```sh
+sudo -u pmtiles-swarm -H /var/lib/pmtiles-swarm/node_modules/.bin/pmtiles-swarm   init --config /etc/pmtiles-swarm/swarm.config.json   --data-dir /var/lib/pmtiles-swarm/data   --save-path /mnt/store/torrent-data   --systemd --password 'the console password'
+```
+
+That writes the configuration, generates `pmtiles-swarm.service` beside it, and
+prints a command for every directory involved — each with the right owner and
+mode. Nothing privileged happens and nothing is installed: the unit is written
+next to the configuration, and copying it into `/etc/systemd/system` is one of
+the lines it hands you.
+
+Two refusals worth knowing before you type it. State under `/etc` is rejected
+rather than warned about, because at the moment a configuration is written there
+is nothing to migrate — the same mistake found six months later costs a stopped
+service and a careful move. And a configuration that already exists is left
+alone unless `--force` says otherwise, since the credentials in it are not
+recoverable.
+
+`--password` is hashed before it is written. Leave it off and there is no
+console password at all: the generated API key is the way in until you set one.
+There is deliberately no placeholder, because `auth.password` accepts plaintext
+and a placeholder in that field is a working password until somebody notices.
+
+**Re-run it after adding a watched folder**, or a subscription with a save path
+of its own. Both are directories the node writes to, and neither reaches
+`ReadWritePaths` by itself. `tools/resume-doctor.py` checks a running node's
+unit against its live configuration and says which directories are missing.
+
+The unit below is what it generates, with your paths in it. Worth reading
+either way — a generated file you do not understand is a hand-written one you
+have not written yet.
+
 ## The unit
 
+Annotated, because every directive here has cost somebody a diagnosis.
+`init --systemd` writes this file with your paths already in it; what
+follows is what those lines are for.
+
 ```ini
 [Unit]
 Description=pmtiles-swarm
@@ -177,8 +219,12 @@ Restart=always
 RestartSec=5
 
 # Stopping announces "stopped" to every tracker, releases the data directory
-# lock and cancels downloads in flight. Worst case is about 20 seconds.
-TimeoutStopSec=45
+# lock, cancels downloads in flight and writes resume data for every archive.
+# That last part is what scales: the node allows its engine two seconds per
+# torrent, so a library of fifty wants well over a minute and systemd has to
+# outlast it. Too low here and the process is killed mid-save, which costs a
+# re-hash of everything unwritten on the way back up.
+TimeoutStopSec=300
 
 # The node stops the sidecar itself, and needs it alive to do so. The default
 # signals both at once, which kills the sidecar before it can write its resume
@@ -233,6 +279,17 @@ itself, in order, and waits for the resume data to be written. Nothing is left
 running: anything still alive when `TimeoutStopSec` expires is killed, sidecar
 included.
 
+**`TimeoutStopSec`, and why it is not a round number you pick once.** Stopping
+writes resume data for every archive, and the node allows its engine two seconds
+per torrent to do it — so the figure that matters grows with the library. Three
+hundred seconds covers a hundred and forty archives; a larger node needs more.
+Set it too low and systemd kills the process mid-save, and every archive that
+had not been written re-hashes its whole store on the way back up, which for a
+700 GiB archive is hours.
+
+`tools/resume-doctor.py` prints the arithmetic for your library and says whether
+the unit's value covers it.
+
 **No `ExecStop=`.** systemd already sends `SIGTERM`, and the node handles it
 from the moment it starts. `ExecStop=/bin/kill -15 $MAINPID` is redundant, and
 becomes wrong under `KillMode=process` — that one leaves the rest of the unit
@@ -779,3 +836,111 @@ once rather than diagnosing later:
   `journalctl -u pmtiles-swarm | grep -i onComplete` — and a hook redirecting its
   own output to a file will have nothing for the journal to show, which is not
   the same as not having run.
+
+## When archives re-check on every start
+
+The symptom is always the same shape — a restart, and then a library sitting at
+0% or grinding through a re-hash of archives that are whole on the disk. The
+causes are not the same shape at all, so guessing between them is expensive:
+re-hashing a 700 GiB archive is half an hour during which it serves nobody.
+
+Two scripts in `tools/` answer it without guessing. Neither starts anything,
+neither writes anything, and both are safe against a running node.
+
+```sh
+sudo -u pmtiles-swarm python3 tools/resume-doctor.py \
+  -c /etc/pmtiles-swarm/swarm.config.json
+```
+
+**`resume-doctor.py`** reads your configuration, catalog, stored `.torrent`
+files and resume directory, and says for each archive what libtorrent will do on
+the next start and why — seed at once, stat its files, or re-hash the store. It
+applies libtorrent's own rules rather than a guess at them, and cites the file
+and line each one came from. It also reads the unit through `systemctl show`,
+does the arithmetic on every deadline that can cut a resume save short, and
+scans the journal for the failures that leave a trace.
+
+```sh
+sudo -u pmtiles-swarm python3 tools/resume-experiment.py
+```
+
+**`resume-experiment.py`** proves the behaviour on the libtorrent you actually
+have, rather than the one the documentation was written against. It builds a
+real torrent over a temporary file, saves real resume data, adds it again, and
+reports what happened. 2.0 and 2.1 differ enough to be worth the thirty seconds.
+
+Three things they exist to tell apart, because the first two look identical from
+outside and want opposite fixes:
+
+- **No resume data is not the problem.** An archive recorded complete is added
+  with `seed_mode`, and that claim stands on its own: libtorrent stats the files
+  and seeds, without hashing a byte. A missing resume file costs nothing.
+- **Stale resume data is the problem, and it is worse than none.** A resume file
+  holding a _partial_ bitfield cancels the `seed_mode` claim — libtorrent drops
+  it if a single piece is unset. The archive then comes back as a downloader
+  with every byte already on disk, and fetches again what it has. Deleting that
+  one file is the repair; with none, it seeds at once.
+- **A short or missing file is a genuine re-check.** `mismatching_file_size` is
+  the only thing here that really does mean re-hashing, and it is about the data
+  on disk rather than about resume data at all. libtorrent 2.x records no
+  mtimes, so nothing in this depends on a timestamp.
+
+Which archives carry stale resume data is not random, and the pattern is worth
+knowing before it looks like a second bug. An archive from a watched folder was
+read end to end before it was ever registered: it is complete from its first
+moment and no partial bitfield is ever written for it. An archive joined from a
+feed, a magnet or a peer is genuinely partial for hours, with partial resume
+data written for it every `resumeSaveIntervalSeconds` throughout. Only the
+second kind has anything to leave behind, so trouble concentrating there is this
+failure's ordinary shape rather than something the import path introduced.
+
+Two more things the doctor looks at, because they are the ones a snapshot alone
+gets wrong.
+
+**Whether the catalog still points at its own `.torrent` files.** Each entry
+records `torrentPath` as an absolute path, and restore reads that rather than
+recomputing it — so moving `dataDir` out of `/etc`, which this guide tells you
+to do, leaves every recorded path behind. Nothing warns, because a missing
+`.torrent` is not treated as an error: restore quietly falls back to the magnet.
+That fallback is the damage. A magnet carries no metadata, resume data does not
+carry it either, and so the archive waits on BEP 9 for a file list that only a
+peer can supply — from a swarm where this node is the origin. It sits at 0% in
+`downloading_metadata` indefinitely, which no amount of re-checking will fix.
+
+**Whether the periodic save gets through the whole library.** Every torrent is
+asked to save in the same cycle, so a healthy `resumeDir` has all its files
+within one interval of each other. Ages spread wider than that mean some cycles
+are finishing early, and the ages alone cannot say whether the same archives
+lose every time. `--watch` measures it rather than inferring it:
+
+```sh
+sudo -u pmtiles-swarm python3 tools/resume-doctor.py \
+  -c /etc/pmtiles-swarm/swarm.config.json --watch 660
+```
+
+It reports each save cycle as it happens, how many of them covered every
+archive, and which archives were never written at all.
+
+**A poisoned archive may heal itself, so run this twice.** The commonest way a
+whole archive acquires a partial bitfield is a save taken while it was
+re-checking: `write_resume_data` truncates `have_pieces` to the pieces checked
+so far, and the five-minute timer lands inside that window every time a large
+archive checks. That shortened bitfield is what cancels `seed_mode` on the next
+start — so the archive comes back partial, checks again, and writes another one.
+The loop needs no restart to keep itself going.
+
+The doctor tells the two apart by length: a bitfield shorter than the torrent
+was written mid-check, and one at full length is what a finished check
+concluded. A check that completes rewrites its own resume file correctly, so an
+archive poisoned in one run can read `seeds at once` in the next with nothing
+done to it. **Only the archives that stay poisoned across two runs, several
+minutes apart, are worth deleting a file for.**
+
+Two things it deliberately does not treat as faults. An archive whose data sits
+outside the `savePathLayout` shape is normal — one adopted from a file this node
+already holds keeps that file, and a node may keep archives beside whatever
+produced them. What matters is whether the data is where the entry says, which
+is checked directly. And every directory an archive actually lives in has to
+appear in `ReadWritePaths`, not just the `savePath` in the configuration; the
+doctor checks each one in use, because a deliberately-placed archive is exactly
+the case that gets left out of the unit.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.58.1",
+  "version": "0.60.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",
@@ -46,7 +46,7 @@
     "maplibre-gl": "^6.2.0",
     "parse-torrent": "^11.0.24",
     "pmtiles": "^4.4.1",
-    "pmtiles-torrent": "^0.9.1",
+    "pmtiles-torrent": "^0.10.2",
     "webtorrent": "^3.0.21"
   },
   "engines": {