pmtiles-swarm 0.15.0 → 0.15.2

package/CHANGELOG.md

@@ -7,6 +7,31 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.15.2
+### 🐞 Bug fixes
+- **A mutable magnet is named after its category rather than a build.** `dn=` was taken from
+  whichever archive was newest when the string was generated, so a magnet whose whole purpose is to
+  resolve to *the current* build carried the name of one particular build — dated the moment the
+  next one landed. It now reads `dn=openmaptiles`.
+
+  Nothing depended on it: `dn` is a display hint, replaced by the real name as soon as metadata
+  arrives. It was simply describing the wrong thing.
+
+## 0.15.1
+### 🐞 Bug fixes
+- **The downloaded bar showed nothing on an archive that was plainly downloading.** A column on
+  that bar covers many pieces, and the sidecar reduced "held" by `all` — so a column lit only when
+  every piece beneath it had arrived, and an archive 18% complete showed an empty bar. Fixed in
+  pmtiles-torrent 0.4.4, which reports a proportion; the console now shades those columns by it,
+  with a floor so the first few percent of a download are visible rather than indistinguishable
+  from none.
+
+  Peer bars had the opposite fault and are fixed the same way, so a peer holding a little no longer
+  reads as a seed.
+
+  Renders correctly against either sidecar: the newer one never rounds a non-empty column below 2,
+  so values above 1 identify the new encoding.
+
 ## 0.15.0
 ### ✨ Features and improvements
 - **The DHT routing table is remembered between runs**, which is the difference between publishing

package/README.md

@@ -115,8 +115,31 @@ subscribers forward, and they fail differently, so publishing both is cheap insu
 
 - **RSS** — easy to consume, understood by existing clients, needs a server that stays up.
 - **BEP 46** — an ed25519-signed DHT record naming the current infohash, addressed by public key
-  rather than infohash (`magnet:?xs=urn:btpk:…`). No server needed, but the record expires and must
-  be republished. See [src/mutable.js](src/mutable.js).
+  rather than infohash (`magnet:?xs=urn:btpk:…`). No server needed at all.
+
+Publishing those records is built in. The node that builds gets a key — `pmtiles-swarm
+publisher-key` — and announces the newest archive in each category, salted by category name so one
+keypair addresses all of them. Records expire from the DHT after about two hours, so it republishes
+on a timer; that timer is the feature rather than an optimisation. Serving nodes need nothing: they
+receive the public half on the catalog entry and hand it out in the TileJSON, which is why a
+serving tier can be compromised without anyone being able to publish.
+
+```json
+{ "mutable": { "publish": true, "keyPath": "/etc/pmtiles-swarm/publisher.pem" } }
+```
+
+The routing table is remembered between runs (`dht-nodes.json` in the data directory), which
+matters more than it sounds: bootstrapping from hostnames alone was measured working about one
+start in seven on a domestic connection, and a saved table turns that into every start. It is the
+same thing libtorrent does, and why its DHT works on hosts where a fresh socket does not.
+
+**What a style should point at** is then the category's TileJSON URL with that magnet in its
+fragment — the console's Categories page has a **For a style** row that gives you the whole
+string. A fragment is never sent in an HTTP request, so ordinary clients fetch the TileJSON and
+ignore it, while a swarm-aware one reads the magnet before making any call and can start when the
+server cannot answer. With `xs=urn:btpk:` rather than an infohash it does not go stale on the next
+build either. See [docs/serving-tiles.md](docs/serving-tiles.md) and
+[docs/security.md](docs/security.md), because that key behaves unlike every other secret here.
 
 ## Configuration
 
@@ -742,9 +765,13 @@ another node, watched web locations including directory listings and date templa
 limits and what is left of them, in-place settings reload and process restart, and two-node
 subscription sync in both directions over both RSS and the catalog API.
 
-Not yet exercised: the qBittorrent engine against a real instance, watch-folder imports, and
-BEP 46 publish/resolve against a live DHT (the crypto and magnet handling are tested; interop
-with libtorrent's encoding is not).
+Running in production: watch-folder imports feeding a nightly planet build, an 18-archive library
+of roughly 2.5 TB behind HAProxy, and BEP 46 records published to a live DHT — 54 nodes stored the
+last one.
+
+Not yet exercised: the qBittorrent engine against a real instance, and **resolving** a BEP 46
+record from outside the publishing network. Nodes accept the records and report storing them, which
+is not the same claim as a stranger reading one back, and only the second matters to a subscriber.
 
 ## License and attribution
 

package/docs/publishing.md

@@ -779,4 +779,12 @@ rebuild, which fail differently, so publishing both is cheap insurance:
   public key rather than infohash (`magnet:?xs=urn:btpk:…`). No server required, but the
   record expires and must be republished.
 
-See [subscribing.md](subscribing.md).
+Publishing those records is built in: give the building node a key with `pmtiles-swarm
+publisher-key`, set `mutable.publish`, and it announces the newest archive in each category
+and republishes on a timer. Serving nodes need nothing — they receive the public half on
+the catalog entry and hand it out in the TileJSON.
+
+See [subscribing.md](subscribing.md) for the shape of it,
+[serving-tiles.md](serving-tiles.md#a-fragment-that-survives-a-rebuild) for what a style
+should then point at, and [security.md](security.md#the-publisher-key-is-not-a-credential)
+for why that key wants treating like a signing key rather than an API token.

package/docs/subscribing.md

@@ -419,9 +419,29 @@ RSS and BEP 46 fail in opposite ways, which is the argument for publishing both:
 | RSS | a server that stays up | the server goes away |
 | BEP 46 | periodic republishing | the record expires (hours, not days) |
 
-**Status:** the crypto and magnet handling are tested — 32-byte keys, 64-byte signatures
-verifying, roundtrip stable. Publishing and resolving against a live DHT, and interop
-with libtorrent's exact value encoding, are **not yet verified**.
+**This is built in.** The code above is what a host application would call; a node only
+needs a key and a setting:
+
+```sh
+pmtiles-swarm publisher-key > /etc/pmtiles-swarm/publisher.pem
+chmod 400 /etc/pmtiles-swarm/publisher.pem
+```
+
+```json
+{ "mutable": { "publish": true, "keyPath": "/etc/pmtiles-swarm/publisher.pem" } }
+```
+
+It then announces the newest archive in each category, salted by category name so one key
+addresses all of them, and republishes before the records expire. Only the node that builds
+needs the key — see
+[running-as-a-service.md](running-as-a-service.md#the-publisher-key) for setting it up and
+[security.md](security.md#the-publisher-key-is-not-a-credential) for why it is not a
+credential.
+
+**Status:** publishing is verified against a live DHT — records stored by 54 nodes at the
+last check. **Resolving** one from outside the publishing network is not yet verified:
+nodes accepting a record is not the same claim as a stranger reading it back, and only the
+second matters to a subscriber.
 
 ## Making the archives serveable
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.15.0",
+  "version": "0.15.2",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -73,7 +73,8 @@ function styleUrlFor(category, newest, base) {
   const magnet = newest?.mutable?.publicKey
     ? mutableMagnet(newest.mutable.publicKey, {
         salt: newest.mutable.salt ?? category,
-        name: newest.name,
+        // The category, since that is what this magnet resolves to.
+        name: newest.mutable.salt ?? category,
         webSeeds: newest.webSeeds,
       })
     : newest?.magnet;

package/src/publisher.js

@@ -214,7 +214,12 @@ export class MutablePublisher {
   magnetFor(category, entry) {
     return mutableMagnet(this.#key.publicKey, {
       salt: category,
-      name: entry?.name,
+      // The category rather than the build. This magnet resolves to whichever
+      // archive is current, so naming one of them dates the string the moment
+      // the next build lands. `dn` is only a label -- the real name arrives
+      // with the metadata and overrides it -- so nothing depends on this
+      // beyond being honest about what the magnet identifies.
+      name: category,
       webSeeds: entry?.webSeeds,
     });
   }

package/src/tilejson.js

@@ -100,7 +100,9 @@ function buildTorrentBlock(entry, root) {
       // and buildable by any node, because it contains only the public half.
       magnet: mutableMagnet(entry.mutable.publicKey, {
         salt: entry.mutable.salt,
-        name: entry.name,
+        // The category, not this build: the record resolves to whichever
+        // archive is current, and `dn` is a label the metadata replaces.
+        name: entry.mutable.salt,
         webSeeds: entry.webSeeds,
       }),
     };

package/src/web/index.html

@@ -2894,9 +2894,17 @@
         };
         const [r, g, b] = rgb(accent);
 
-        // Availability is a count, so it shades: the rarer the piece, the
-        // fainter the column. The other two are yes or no.
-        const ceiling = kind === 'availability' ? Math.max(1, ...bytesIn) : 1;
+        // Availability is a count, so it shades by rarity. The other two are
+        // proportions now: a column covers many pieces — 178,000 of them
+        // across a bar a thousand wide on a 698 GiB archive — so "held" is a
+        // fraction of its slice rather than a yes or no. Reduced any other way
+        // the bar only tells the truth at 0% and 100%.
+        //
+        // A sidecar older than pmtiles-torrent 0.4.4 still sends 0 or 1, and
+        // is recognised by nothing exceeding 1: the newer one never rounds a
+        // non-empty bucket below 2, precisely so this test works.
+        const ceiling = kind === 'availability' ? Math.max(1, ...bytesIn) : 255;
+        const proportional = kind !== 'availability' && Math.max(0, ...bytesIn) > 1;
 
         for (let index = 0; index < bytesIn.length; index += 1) {
           const value = bytesIn[index];
@@ -2905,7 +2913,15 @@
             image.data[at + 3] = 0;
             continue;
           }
-          const strength = kind === 'availability' ? 0.35 + 0.65 * (value / ceiling) : 1;
+          // A floor, because a column holding a little is worth seeing: without
+          // one, the first few percent of a download are indistinguishable
+          // from none at all.
+          const strength =
+            kind === 'availability'
+              ? 0.35 + 0.65 * (value / ceiling)
+              : proportional
+                ? Math.max(0.18, value / 255)
+                : 1;
           image.data[at] = r;
           image.data[at + 1] = g;
           image.data[at + 2] = b;