pmtiles-swarm 0.55.3 → 0.58.1

package/CHANGELOG.md

@@ -7,6 +7,122 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.58.1
+### 🐞 Bug fixes
+- **The sample configuration put every piece of state under the config file.** `"dataDir": "./data"`,
+  `"savePath": "./data/torrents-data"` and `"resumeDir": "./data/resume"` all resolve against the
+  config file — and the service guide puts that file in `/etc`. Anyone following both documents ended
+  up with a catalog, a resume directory and their archives on the partition meant for configuration,
+  having done nothing wrong. The sample now uses absolute paths, a test enforces it, and the node
+  warns at startup if state resolves under `/etc` anyway.
+
+  It is also shorter. A first config should get a node running, not demonstrate the whole surface —
+  `docs/configuration.md` is where the rest lives.
+
+- **Moving state out of `/etc` had a trap in the instructions.** `mv OLD/data NEW/data` nests when the
+  destination exists, which it does after the documented setup — so the real directory ends up one
+  level too deep, the node writes a fresh empty catalog beside it, and an intact library reads as
+  lost. `docs/running-as-a-service.md` now gives a form that cannot nest, says to repoint
+  `libtorrent.resumeDir` as well as `dataDir`, and has you count catalog entries before and after.
+
+## 0.58.0
+### ✨ Features and improvements
+- **A stable name for every kind of import, not just watched folders.** `latestLink` and
+  `latestLinkType` are now offered on watched web locations, RSS feeds and remote nodes as well — one
+  path a consumer can hold while the build behind it changes.
+
+  A scheduled source had the feature all along and no way to ask for it: the console never showed the
+  column. It was also ignoring `latestLinkType`, so a source asking for a hard link quietly got a
+  symlink. A subscription could not ask at all.
+
+  For a subscription the name is pointed at the archive **when the download finishes**. Until then
+  there is a marker file, or a sparse one still filling in, and a name resolving to either is worse
+  than no name: whatever opens it reads zeroes rather than failing.
+
+- **The node says something when its state has landed in `/etc`.** Nobody chooses that — the
+  documented service layout puts the config file there, every path resolves relative to that file,
+  and the sample reads `"dataDir": "./data"`. So the catalog and the resume directory end up on the
+  partition meant for configuration. Warned rather than corrected: it is a real path that works, and
+  moving a running node's data would be worse than saying so.
+
+### 🐞 Bug fixes
+
+## 0.57.0
+### ✨ Features and improvements
+- **The three publishing switches are one **Local file** column on the import tables.** Three
+  dropdowns per row cost more width than the choice is worth, on tables that already scroll
+  sideways — and as an import default these are almost always decided together:
+
+  | option | serves the file | web seed | on the public page |
+  | --- | --- | --- | --- |
+  | `node` | — | — | — |
+  | `off` | no | — | — |
+  | `http` | yes | no | no |
+  | `http + catalog` | yes | no | yes |
+  | `http + web seed` | yes | yes | no |
+  | `http + web seed + catalog` | yes | yes | yes |
+
+  `http + catalog` is there rather than left out to keep a tidier ladder: a dropdown that could not
+  say it would round that state up to the nearest option it had, turning a web seed on — and
+  publishing the node to the swarm — because somebody re-saved an unrelated row. Every combination
+  survives being written and read back, which is what the round-trip test checks.
+
+  The archive details panel still offers the three separately. A row here sets a policy for what
+  arrives; the panel is where one archive gets picked over.
+
+## 0.56.2
+### 🐞 Bug fixes
+- **Adding a save location in the console did nothing.** The row was read correctly and then thrown
+  away: the settings pane below renders every config key it does not explicitly skip as a raw-JSON
+  textarea, the skip list named `watch`, `sources` and `subscriptions` but not `locations`, and in
+  `saveSettings` the textarea loop runs after the row editors — so a copy of the list as it was when
+  the pane was drawn overwrote the one with the new row in it.
+
+  The skip is now derived from the registered row editors rather than listed by hand, so this cannot
+  happen again to the next editor somebody adds. The list was the bug; keeping a list and adding one
+  more name to it would have been the same bug waiting.
+
+## 0.56.1
+### ✨ Features and improvements
+- **Comments trimmed back to house style, and the reasoning moved into the docs where it belongs.**
+  Recent work had been leaving 15–25 line explanatory blocks in the source; measured across
+  `catalog.js` that was 85 new comment lines against 76 new code lines. The argument for a decision
+  drifts out of date faster in a comment than in prose, and buries the code while it does it.
+
+  Two new sections in [docs/internals.md](docs/internals.md) give the displaced reasoning a home:
+  **Re-reading a summary an older prober wrote**, and **A validator for a URL that stays put** —
+  which is where the ETag story now lives, including why the infohash was the wrong choice and what
+  a PMTiles reader needs to see for `If-Range` and cross-origin `ETag` to work.
+
+  No behaviour change.
+
+## 0.56.0
+### ✨ Features and improvements
+- **MapLibre Tiles are recognised.** PMTiles tile type `6` is MLT, and the tile-type table stopped at
+  `5` — so an MLT archive probed as `unknown` and was refused a tile endpoint it could have served,
+  even though the extension map had known about `.mlt` for a while.
+
+  No new TileJSON key was needed for it. MapLibre spells the tile encoding of a vector source
+  `encoding`, exactly as it does the elevation packing of a raster-dem one, and applies TileJSON
+  members to both after the source is constructed — so `encoding: "mlt"` reaches a vector source the
+  same way `encoding: "terrarium"` reaches a raster one. One key, two meanings, told apart by the
+  source type. The MLT value is read from the header rather than the metadata, because an archive
+  whose tile type says MapLibre Tiles is MLT-encoded and nothing needs to say so twice.
+
+- **A resume-data shortfall is now reported.** The sidecar has been answering with how many torrents
+  were asked to write and how many managed it before the deadline, and both numbers were discarded —
+  by the engine wrapper, and again by the timer that called it. A torrent that does not write is one
+  that gets re-hashed on the next start, which for a 700 GiB archive is the difference between
+  seeding in seconds and seeding in half an hour. That is what "why is everything at 0%" looks like
+  from outside, and the silence here is part of why it was hard to see.
+
+### 🐞 Bug fixes
+- **The two pages disagreed about how large an archive was.** The console rounded to whole units
+  above ten and the public page always kept a decimal, so the same archive read as `81 GiB` on one
+  and `80.6 GiB` on the other. Both now keep a decimal from KiB up — these are mostly archive sizes,
+  and half a gigabyte is worth seeing — and a test holds the two helpers character-for-character
+  identical, since nothing about a duplicated function announces when it stops being a copy.
+
 ## 0.55.3
 ### 🐞 Bug fixes
 - **A `/latest/` document could never be updated once a client had cached it.** The `ETag` was the

package/docs/configuration.md

@@ -488,9 +488,19 @@ moment a download finishes.
 
 The node's setting is the default. A [watched folder](#watched-folders), a
 [scheduled source](#scheduled-sources), an [RSS feed](#subscriptions) and a
-remote node may each carry their own — the **Serve file**, **Web seed** and
-**Listed** columns on those tables, where `node` means "no opinion" rather than
-"off". And any individual archive can be switched in the console, under **HTTP
+remote node may each carry their own — the **Local file** column on those
+tables, which offers the three as one choice:
+
+| option                      | `serveArchive` | `selfWebSeed` | `publicDownload` |
+| --------------------------- | -------------- | ------------- | ---------------- |
+| `node`                      | unset          | unset         | unset            |
+| `off`                       | `false`        | —             | —                |
+| `http`                      | `true`         | —             | —                |
+| `http + catalog`            | `true`         | —             | `true`           |
+| `http + web seed`           | `true`         | `true`        | —                |
+| `http + web seed + catalog` | `true`         | `true`        | `true`           |
+
+`node` means "no opinion" rather than "off". And any individual archive can be switched in the console, under **HTTP
 sources** in its details.
 
 An archive that says nothing goes on following the node, so changing the node's
@@ -586,6 +596,17 @@ it, and what turns a cold tile read from tens of seconds into well under one.
 `webSeedBase` on its own assumes the watched folder is already the web root, since
 nothing is moved.
 
+**Every table that imports an archive offers this**, not just watched folders:
+a [scheduled source](#scheduled-sources), an [RSS feed](#subscriptions) and a
+remote node each take `latestLink` and `latestLinkType` too. The point is the
+same in all four — a consumer holds one path and never learns the name of any
+particular build.
+
+For a subscription the name is pointed at the archive **when the download
+finishes**, not when it is joined: until then there is a marker file or a sparse
+one still filling in, and a name resolving to either is worse than no name at
+all, because whatever opens it reads zeroes rather than failing.
+
 `latestLinkType: 'hard'` is for a name something reads the archive _through_. A
 hard link still resolves after the build it names is retired, where a symlink is
 left pointing at nothing. The other kind stays the fallback in both directions,
@@ -625,6 +646,7 @@ entry gives either a `url` template or an `index` directory:
 | `everyHours`                                    | an interval instead                                                                                           |
 | `md5`                                           | overrides the node's [`md5`](#md5) for this source alone                                                      |
 | `serveArchive`, `selfWebSeed`, `publicDownload` | override what this node offers of the archives this source fetches                                            |
+| `latestLink`, `latestLinkType`                  | a stable second name for the newest build, as on a watched folder                                             |
 
 Prefer a template where the naming is predictable: it asks a direct question,
 gets a direct answer, and needs the upstream to publish no listing at all.

package/docs/internals.md

@@ -18,6 +18,8 @@ Operator-facing documentation is elsewhere — see [publishing](publishing.md),
 - [Serving an MBTiles archive](#serving-an-mbtiles-archive)
 - [Answering for a tile that is not there](#answering-for-a-tile-that-is-not-there)
 - [Reading an archive that is still arriving](#reading-an-archive-that-is-still-arriving)
+- [Re-reading a summary an older prober wrote](#re-reading-a-summary-an-older-prober-wrote)
+- [A validator for a URL that stays put](#a-validator-for-a-url-that-stays-put)
 - [The health endpoint](#the-health-endpoint)
 - [The externally visible base URL](#the-externally-visible-base-url)
 - [Scheduled sources](#scheduled-sources)
@@ -310,6 +312,86 @@ nothing to read yet and the same archive at 100% will, so a permanent
 would put a swarm read behind each one. The limiter is in memory on purpose: a
 restart is a reasonable moment to try again.
 
+## Re-reading a summary an older prober wrote
+
+An archive's summary — format, zoom range, bounds, vector layers, encoding — is
+read once, written into the catalog, and never questioned again. That is right
+as far as it goes: re-reading a header out of the swarm is not free, and for a
+given infohash the answer cannot change.
+
+It goes wrong the moment the prober learns to read something new. Every archive
+probed before that keeps a summary with a hole in it, permanently, and the only
+way out is to remove and re-add the archive by hand.
+
+`encoding` was the case that made this plain. The key had been sitting in the
+metadata of archives a node had been serving for months; teaching the prober to
+read it changed nothing at all, because nothing ever asked again.
+
+So `summarize()` stamps a `summaryVersion`, and a summary older than the current
+prober is re-read once in the background — on a request for the archive's
+TileJSON, rate-limited to once a minute per archive. Raise `SUMMARY_VERSION`
+whenever a field is added and every archive in the catalog picks it up on its
+own, with nothing to do on upgrade.
+
+Two things this needs to get right, both learned the hard way:
+
+- **Every route that serves a TileJSON has to trigger it**, not only
+  `/archives/<infohash>/tiles.json`. `/latest/<category>/tiles.json` is the URL
+  a style points at, so leaving it out missed exactly the archives that were
+  being consumed the documented way.
+- **The write-back must not be conditioned on one field.** It began as a
+  vector-layer backfill and returned early unless layers turned up, which would
+  have discarded the encoding it had just gone to fetch.
+
+## A validator for a URL that stays put
+
+Everything under `/archives/` is addressed by infohash: the URL changes when the
+content does, so it can be cached for a year and marked `immutable`. A `/latest/`
+URL is the opposite — stable on purpose, with the content moving underneath it —
+so it needs a validator, and a short TTL alone is a guess.
+
+The obvious validator is the infohash of the archive the category resolved to.
+It is wrong, and it fails in a way that has no bottom: these documents carry more
+than which build they name. A TileJSON also carries the archive's summary; a
+magnet also carries its web seeds and trackers; the feed carries both. Enrich a
+summary or add a web seed and the body changes while the infohash does not — so
+every cache in the path revalidates, is told `304`, and goes on serving the old
+document. Not stale for a minute: unable to be updated again for as long as the
+archive exists.
+
+So these are tagged over the body they are sending. Two nodes answering
+identically still produce identical tags, which is the property the infohash was
+chosen for; two nodes answering differently no longer claim otherwise.
+
+`/latest/<category>/archive.pmtiles` and the `.torrent` redirect keep the
+infohash, because for those the infohash really is the whole content.
+
+### Where the ranges have to be careful
+
+A PMTiles reader does not fetch a file. It fetches a header, then a root
+directory, then leaf directories, then tiles, over minutes or hours. If a
+rebuild lands partway through, the offsets it read from the old build address
+bytes in the new one — which does not fail loudly. It decodes as the wrong tile,
+or as nothing, with no error anywhere naming the cause.
+
+`If-Range` is therefore honoured on the category range endpoint: a range
+conditioned on a build that is no longer current is refused _as a range_ and
+answered in full, which a reader survives. `Last-Modified` is suppressed there so
+a client cannot condition on a date instead — a build restored from a backup can
+be newer while looking older.
+
+The official reader closes the loop from its side, comparing the `ETag` of every
+response against the one it saw first and re-reading the header when they differ.
+That only works if it can see the tag: `ETag` is not exposed to cross-origin
+JavaScript by default, so these routes send `Access-Control-Expose-Headers`.
+Unexposed, the reader compares against `null`, the comparison never fires, and it
+splices two builds in silence.
+
+A weak validator is no better. The reader discards any tag beginning with `W/`,
+which is exactly what Express derives from a file's size and mtime — and that
+tag also differs per node, so two nodes behind a load balancer would hand a
+reader two tags for byte-identical archives.
+
 ## The health endpoint
 
 For a load balancer, which needs three things: no credential, a cheap answer, and

package/docs/running-as-a-service.md

@@ -256,8 +256,10 @@ directory:
   -> /etc/pmtiles-swarm/data
 ```
 
-That is rarely what you want for a service. Use absolute paths for anything
-holding data:
+That is rarely what you want for a service, and the node says so on startup —
+`/etc` is for configuration, and a catalog, a resume directory and possibly an
+archive do not belong on that partition. The shipped sample uses absolute paths
+for exactly this reason. Use them for anything holding data:
 
 ```json
 {
@@ -267,6 +269,33 @@ holding data:
 }
 ```
 
+#### Moving state that already landed in the wrong place
+
+Repoint **every** path that lived under the directory being moved, not just
+`dataDir`: `libtorrent.resumeDir` is the one that is easy to miss, because it is
+absolute and sits inside it.
+
+And move with a destination that cannot nest. `mv /etc/pmtiles-swarm/data
+/var/lib/pmtiles-swarm/data` puts the source _inside_ the destination when the
+destination already exists — which it does, since the setup above creates it —
+leaving `/var/lib/pmtiles-swarm/data/data`. The node then finds no catalog where
+it was told to look, writes a fresh empty one, and every archive appears to have
+been lost.
+
+```bash
+sudo systemctl stop pmtiles-swarm
+sudo python3 -c "import json;print(len(json.load(open('OLD/catalog.json'))['entries']))"
+
+sudo mv /etc/pmtiles-swarm/data /var/lib/pmtiles-swarm/   # note: no second 'data'
+# repoint dataDir *and* libtorrent.resumeDir, then
+
+sudo systemctl start pmtiles-swarm
+sudo python3 -c "import json;print(len(json.load(open('NEW/catalog.json'))['entries']))"
+```
+
+The entry count before and after is the check that matters: an empty new catalog
+is indistinguishable from a lost library until you count.
+
 ### 2. Every one of them in `ReadWritePaths`
 
 `ProtectSystem=strict` presents the whole filesystem as read-only inside the

package/docs/tilejson.md

@@ -172,7 +172,22 @@ MapLibre applies every TileJSON member to the source after the source is
 constructed, so an `encoding` here **overrides** one written in the style. That
 is the intended direction: the archive is the thing that knows.
 
-Anything other than the three values the style specification defines is dropped
+### `mlt`
+
+The same key carries one more value, for a different kind of source. MapLibre
+spells the tile encoding of a vector source `encoding` too, and `mlt` there means
+the tiles are [MapLibre Tiles](https://maplibre.org/maplibre-tile-spec/) rather
+than MVT — so there was no new key to invent for it, only a value to allow
+through.
+
+This one comes from the header rather than the metadata. PMTiles has a tile type
+for MLT (`6`), and an archive whose tile type says MapLibre Tiles _is_
+MLT-encoded; nothing needs to be written in the metadata to make it so.
+Elevation packing is the opposite case — the header knows the tile is WebP and
+nothing about what its three channels mean — which is why the two are read from
+different places despite sharing a key.
+
+Anything other than the values the style specification defines is dropped
 rather than passed on. A client handed an encoding it does not recognise is
 worse off than one handed nothing, because nothing at least leaves it free to
 use its own default.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.55.3",
+  "version": "0.58.1",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",