pmtiles-swarm 0.2.0

package/docs/serving-tiles.md

@@ -0,0 +1,218 @@
+# Serving tiles
+
+Every archive in the catalog is also a tile endpoint. A map can point straight at
+this server — no separate tile server, no unpacking the archive.
+
+```
+GET /archives/{infohash}/tiles.json          TileJSON 3.0.0
+GET /archives/{infohash}/{z}/{x}/{y}.{ext}   one tile
+GET /archives/{infohash}/archive.torrent     the .torrent
+```
+
+`{ext}` must match what the archive holds: `pbf` or `mvt` for vector, `png`,
+`jpg`, `webp`, `avif` for raster. Asking for the wrong one is a 400 rather than
+a silently wrong content type.
+
+Drop the TileJSON URL into any client that speaks TileJSON:
+
+```js
+map.addSource('basemap', {
+  type: 'vector',
+  url: 'https://swarm.example.org/archives/913d671f…/tiles.json',
+});
+```
+
+## Where the bytes come from
+
+This is the part that makes serving tiles from a torrent client worth doing at
+all. The endpoint behaves the same either way, but underneath there are two very
+different paths:
+
+| This node | Reads from | Cost |
+| --- | --- | --- |
+| Holds a complete copy | The local file, directly | Disk, no swarm involvement |
+| In cache mode | The swarm, one piece at a time | Only the pieces tiles are actually in |
+
+A cache-mode node holds almost none of a 700 GiB planet archive but can still
+answer for any tile in it. The first request for a cold area pulls the pieces
+that tile lives in, and those pieces stay — both in the piece cache and in the
+engine's store, where they are seeded back to the swarm. Reading an archive makes
+you a better peer for it.
+
+Which path is used is decided per archive, from the engine's own progress rather
+than the file size on disk: both engines preallocate the full file, so a torrent
+one piece in already looks complete by size.
+
+`GET /api/torrents/{infohash}` reports which path an open archive is using.
+
+### Engine support
+
+Cache-mode reads need piece-level control, which not every engine has:
+
+| Engine | Complete copy | Cache mode |
+| --- | --- | --- |
+| libtorrent | yes | yes — piece deadlines and per-piece priorities |
+| webtorrent | yes | yes — shares the seeding client |
+| qBittorrent | yes | **no** |
+
+qBittorrent's WebUI has per-file priorities but nothing per piece and no way to
+read one back, so there is no honest way to serve a tile from an archive it holds
+only part of. It returns 501 with an explanation rather than a generic failure.
+That is rarely a problem in practice, since qBittorrent is normally the bulk
+seeder and a bulk seeder holds complete copies.
+
+Both peer-to-peer engines reuse the client already seeding the archive rather
+than starting a second one. One peer pool, one port, one DHT node — and for
+libtorrent, one sidecar process.
+
+## The `torrent` block
+
+TileJSON documents from this server carry a non-standard `torrent` member:
+
+```json
+{
+  "tilejson": "3.0.0",
+  "tiles": ["https://swarm.example.org/archives/913d…/{z}/{x}/{y}.pbf"],
+  "minzoom": 0,
+  "maxzoom": 14,
+  "vector_layers": [{ "id": "water" }],
+  "torrent": {
+    "infohash": "913d671f3a28c5b8d605e28cf6bf01e293d36e86",
+    "magnet": "magnet:?xt=urn:btih:913d671f…",
+    "torrent": "https://swarm.example.org/archives/913d…/archive.torrent",
+    "name": "planet.pmtiles",
+    "size": 77242531840,
+    "webseeds": ["https://maps.example.org/planet.pmtiles"]
+  }
+}
+```
+
+This is progressive enhancement, and the reason it is shaped this way:
+
+- **A plain client ignores it.** TileJSON permits unknown members and MapLibre's
+  style-spec permits arbitrary source properties, so maplibre-gl-js, Leaflet and
+  anything else fetch tiles over HTTP exactly as they would from any tile server.
+- **A torrent-aware client uses it.** It joins the swarm and reads tiles from
+  pieces, falling back to the HTTP URLs whenever the swarm cannot answer.
+
+One URL serves both, so a style does not have to know which kind of client will
+load it. That also means a torrent-aware client gets a working map immediately
+over HTTP while the swarm is still finding peers, rather than staring at an empty
+canvas for the 90 to 240 seconds a cold magnet can take to resolve metadata.
+
+For an archive published as a mutable torrent, the block also carries
+`mutable.publicKey`, so a client that understands BEP 46 can follow updates
+rather than pinning to the version the document was generated from. See
+[publishing](publishing.md).
+
+## Caching
+
+Tiles are served `Cache-Control: public, max-age=31536000, immutable`.
+
+That is safe rather than optimistic. An infohash is a content hash: the bytes
+under `/archives/{infohash}/…` cannot change, because different bytes would be a
+different infohash. When a mutable archive is updated the infohash changes, so
+the URLs change with it and caches never need invalidating — the old ones simply
+stop being referenced.
+
+The TileJSON itself is not cached that way, since it is the document that points
+at the current infohash.
+
+## Configuration
+
+```json
+{
+  "tiles": {
+    "maxOpenArchives": 16,
+    "directoryCacheEntries": 200,
+    "pieceCacheBytes": null,
+    "hydrateIdleMs": null,
+    "pieceTimeoutMs": 120000,
+    "readyTimeoutMs": 60000
+  }
+}
+```
+
+## Warming a region
+
+A cache-mode node is slow exactly once per region: the first request pulls the
+pieces those tiles live in, and everything after is served from what it now
+holds. Warming moves that cost off the request path, which matters most just
+before adding a node to a load-balanced pool.
+
+```sh
+curl -X POST http://node:8090/api/torrents/$INFOHASH/warm \
+  -H 'content-type: application/json' \
+  -d '{"bounds": [5.9, 45.8, 10.5, 47.8], "minZoom": 0, "maxZoom": 12}'
+```
+
+Everything is optional. Without `bounds` it uses the archive's own; without a
+zoom range it warms from the archive's minimum up a handful of levels, because
+tile counts quadruple per level and warming to z14 globally is never what was
+meant. The zoom range is clamped to what the archive actually holds, so asking
+for more than exists costs nothing.
+
+```
+GET    /api/torrents/{infohash}/warm    progress
+DELETE /api/torrents/{infohash}/warm    cancel
+```
+
+Progress reports `total`, `done`, `hits`, `misses` and `errors`. Misses are
+normal — a bounding box over a sparse archive covers tiles that were never
+generated. A job that fails every tile without a single success gives up early
+rather than grinding through the region to prove the archive is unreadable.
+
+`maxTiles` caps a job (5000 by default) and `concurrency` sets how many tiles are
+in flight (4 by default). Raising concurrency helps when the bottleneck is swarm
+latency rather than bandwidth.
+
+**Warming a mirror node does nothing useful** — it already holds everything and
+reads its local file. The endpoint still works; it just finishes almost
+immediately.
+
+**Warming is cheaper on the second node.** Every node in the serving tier is a
+peer in the same swarm, so a node warming a region a sibling already holds
+fetches it from that sibling rather than from the original seed. Local service
+discovery is on by default, so nodes on the same subnet find each other with no
+configuration.
+
+### Absolute URLs behind a proxy
+
+TileJSON contains absolute tile URLs, so the server has to know how it is reached:
+
+```json
+{
+  "publicUrl": "https://maps.example.org",
+  "trustProxy": "loopback, 10.0.0.0/8"
+}
+```
+
+`publicUrl` pins one canonical URL. Leave it unset and set `trustProxy` instead
+to derive the URL per request from `X-Forwarded-Proto` and `X-Forwarded-Host`,
+which lets one node serve `https://maps.example.org` and `http://maps.internal`
+correctly at the same time.
+
+Set at least one of them behind a TLS-terminating proxy. Otherwise the TileJSON
+advertises `http://` tile URLs, and a browser that loaded the map over `https`
+blocks them all as mixed content.
+
+### Tuning
+
+`maxOpenArchives` bounds how many archives are held open at once; the least
+recently used is closed when the limit is passed, and reopened transparently if
+asked for again. Each open archive costs a file descriptor or a torrent reader
+plus its piece cache.
+
+Leave `pieceCacheBytes` unset unless you have a reason. It is then sized from the
+torrent's piece length, which is the safer default — a fixed byte budget is a
+trap with 16 MiB pieces, since 64 MiB holds only four of them.
+
+## What this is not
+
+This is a tile endpoint for archives this node distributes, not a general tile
+server. It does not render raster tiles from vector data, compose styles, serve
+fonts or sprites, or reproject. For any of that, point
+[tileserver-gl](https://github.com/maptiler/tileserver-gl) at the archive — it
+can read PMTiles from a torrent directly, using the same
+[pmtiles-torrent](https://github.com/TechIdiots-LLC/pmtiles-torrent) package this
+does.

package/docs/subscribing.md

@@ -0,0 +1,187 @@
+# Subscribing and distribution
+
+How a second node follows a first, and what it costs.
+
+## A two-node setup
+
+### The publisher
+
+Watches a build directory, creates torrents, publishes a feed.
+
+```json
+{
+  "port": 8090,
+  "engine": "libtorrent",
+  "publicUrl": "https://maps.example.org",
+  "libtorrent": { "savePath": "/mnt/maps/generated" },
+  "watch": [
+    {
+      "path": "/mnt/maps/generated",
+      "category": "basemaps",
+      "webSeedBase": "https://maps.example.org/files"
+    }
+  ]
+}
+```
+
+`publicUrl` matters: feed items carry absolute links, and a subscriber that cannot
+resolve them cannot fetch the `.torrent`. `webSeedBase` matters just as much — it makes
+each archive usable before it has any peers at all.
+
+Its feed is then at `https://maps.example.org/feed.xml`, with per-category feeds at
+`/feed/basemaps.xml`.
+
+### The subscriber
+
+```json
+{
+  "port": 8090,
+  "engine": "libtorrent",
+  "libtorrent": { "savePath": "/var/lib/maps" },
+  "subscriptions": [
+    { "url": "https://maps.example.org/feed.xml", "mode": "cache" }
+  ],
+  "subscriptionIntervalSeconds": 900
+}
+```
+
+Poll immediately rather than waiting for the interval:
+
+```sh
+curl -X POST localhost:8090/api/subscriptions/refresh
+```
+
+## mirror or cache
+
+The choice that decides what a subscriber costs.
+
+| Mode | Disk | The node becomes |
+| --- | --- | --- |
+| `mirror` | the whole archive | A full seeder — redundancy for the swarm |
+| `cache` | only what is read | A tile server that pays for what people look at |
+
+**Cache mode is the interesting one.** The node joins the swarm and downloads nothing.
+A tile server reads byte ranges from it on demand through `pmtiles-torrent`, so disk use
+tracks what users actually view rather than what exists — which is what makes a 700 GiB
+planet archive serveable from a small VPS. The node still seeds whatever pieces it picked
+up along the way, so a cache-mode subscriber is a contributing swarm member, not a
+freeloader.
+
+Mirror mode is what you want on at least one or two nodes per archive, so the swarm does
+not depend on a single origin.
+
+### Filtering
+
+One feed can serve subscribers with different appetites:
+
+```json
+{
+  "subscriptions": [
+    { "url": "https://maps.example.org/feed.xml", "mode": "mirror", "filter": "europe" },
+    { "url": "https://maps.example.org/feed.xml", "mode": "cache", "category": "planet" }
+  ]
+}
+```
+
+`filter` is a case-insensitive regular expression matched against the item title.
+
+## Subscribing with plain qBittorrent
+
+The feed is ordinary RSS 2.0 with `application/x-bittorrent` enclosures, which is exactly
+what qBittorrent's built-in RSS auto-downloader consumes. **An operator can follow a
+pmtiles-swarm feed today with no new software** — RSS tab, add the URL, set an
+auto-download rule.
+
+They lose cache mode, since that needs piece-level control, but for a node that wants a
+full mirror it works out of the box.
+
+## How many items a feed carries
+
+```json
+{ "feedMaxItems": 50 }
+```
+
+Newest first; `0` means no limit. A consumer can also ask for a different number with
+`?limit=`, so one publisher can serve subscribers with different poll intervals from the
+same catalog:
+
+```
+/feed.xml            the configured default
+/feed.xml?limit=1    newest build only
+/feed.xml?limit=0    everything
+```
+
+**Choose it against how often subscribers poll, not how tidy the feed looks.** A feed
+holding a single item is only safe if everyone polls more often than you publish. If a
+consumer is down overnight and you publish daily, that build drops off the feed before it
+is seen — and nothing indicates it was missed. With a daily build and subscribers polling
+every 15 minutes, 50 items is roughly seven weeks of slack.
+
+The same applies to `lookbackDays` on scheduled sources, for the same reason and in the
+other direction.
+
+## What the feed carries
+
+Beyond the standard fields, items carry a namespaced description of the map:
+
+```xml
+<item>
+  <title>planetiler-openmaptiles-latest.pmtiles</title>
+  <enclosure url="https://maps.example.org/api/torrents/5e1c…/file"
+             length="77230486744" type="application/x-bittorrent"/>
+  <pmtiles:infohash>5e1c143c400d15aaacfb1c748d4ab6d1b46c5df5</pmtiles:infohash>
+  <pmtiles:magnet>magnet:?xt=urn:btih:5e1c…</pmtiles:magnet>
+  <pmtiles:format>pbf</pmtiles:format>
+  <pmtiles:minzoom>0</pmtiles:minzoom>
+  <pmtiles:maxzoom>14</pmtiles:maxzoom>
+  <pmtiles:bounds>-180,-85.05,180,85.05</pmtiles:bounds>
+</item>
+```
+
+This is the part a generic torrent feed cannot offer. A subscriber can decide whether it
+wants a 72 GiB download from the feed alone, rather than fetching metadata to find out.
+Generic clients ignore the namespace and still work.
+
+## Updatable torrents (BEP 46)
+
+A rebuilt archive is a different torrent, because the infohash is a hash of its content.
+BEP 46 adds a level of indirection: an ed25519-signed record in the DHT whose value names
+the *current* infohash, addressed by public key. Subscribers follow the key and are
+carried across rebuilds.
+
+```js
+import { generatePublisherKey, mutableMagnet, publishInfoHash } from 'pmtiles-swarm/mutable';
+
+const key = generatePublisherKey();      // back this up — it is the archive's identity
+const magnet = mutableMagnet(key.publicKey, { name: 'planet.pmtiles' });
+// magnet:?xs=urn:btpk:200d26e8…  — note: no infohash
+
+await publishInfoHash(dht, key, currentInfoHash);
+```
+
+The public key is the permanent address; losing the private key means subscribers can
+never be moved forward again.
+
+RSS and BEP 46 fail in opposite ways, which is the argument for publishing both:
+
+| | Needs | Fails when |
+| --- | --- | --- |
+| RSS | a server that stays up | the server goes away |
+| BEP 46 | periodic republishing | the record expires (hours, not days) |
+
+**Status:** the crypto and magnet handling are tested — 32-byte keys, 64-byte signatures
+verifying, roundtrip stable. Publishing and resolving against a live DHT, and interop
+with libtorrent's exact value encoding, are **not yet verified**.
+
+## Making the archives serveable
+
+A subscriber holding archives is only useful if something serves tiles from them. Point
+tileserver-gl at the same torrents:
+
+```sh
+export PMTILES_TORRENT_PATH=/var/lib/maps
+tileserver-gl --file /var/lib/maps/torrents/5e1c….torrent
+```
+
+Both processes then share the data directory: pmtiles-swarm manages membership and the
+catalog, tileserver-gl reads ranges on demand and seeds what it fetches.

package/package.json

@@ -0,0 +1,56 @@
+{
+  "name": "pmtiles-swarm",
+  "version": "0.2.0",
+  "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
+  "type": "module",
+  "main": "src/index.js",
+  "bin": {
+    "pmtiles-swarm": "src/index.js"
+  },
+  "files": [
+    "src",
+    "docs",
+    "NOTICE.md",
+    "CHANGELOG.md"
+  ],
+  "scripts": {
+    "start": "node src/index.js",
+    "test": "node --test test/*.test.js"
+  },
+  "keywords": [
+    "pmtiles",
+    "bittorrent",
+    "torrent",
+    "map",
+    "tiles",
+    "rss",
+    "p2p",
+    "distribution"
+  ],
+  "license": "BSD-3-Clause",
+  "dependencies": {
+    "chokidar": "^5.0.0",
+    "create-torrent": "^6.1.0",
+    "express": "^5.2.1",
+    "parse-torrent": "^11.0.24",
+    "pmtiles": "^4.4.1",
+    "pmtiles-torrent": "^0.1.1"
+  },
+  "optionalDependencies": {
+    "webtorrent": "^3.0.21"
+  },
+  "engines": {
+    "node": "^22.13.0 || 24"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/TechIdiots-LLC/pmtiles-swarm.git"
+  },
+  "bugs": {
+    "url": "https://github.com/TechIdiots-LLC/pmtiles-swarm/issues"
+  },
+  "homepage": "https://github.com/TechIdiots-LLC/pmtiles-swarm#readme",
+  "publishConfig": {
+    "access": "public"
+  }
+}