pmtiles-swarm 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +53 -0
- package/LICENSE +28 -0
- package/NOTICE.md +47 -0
- package/README.md +172 -0
- package/docs/architecture-diagram.md +287 -0
- package/docs/engines.md +140 -0
- package/docs/publishing.md +387 -0
- package/docs/serving-tiles.md +218 -0
- package/docs/subscribing.md +187 -0
- package/package.json +56 -0
- package/src/api.js +472 -0
- package/src/catalog.js +159 -0
- package/src/config.js +231 -0
- package/src/engines/libtorrent.js +384 -0
- package/src/engines/qbittorrent.js +320 -0
- package/src/engines/types.js +59 -0
- package/src/engines/webtorrent.js +264 -0
- package/src/feed.js +226 -0
- package/src/file-source.js +67 -0
- package/src/index.js +177 -0
- package/src/library.js +567 -0
- package/src/mutable.js +197 -0
- package/src/origin.js +205 -0
- package/src/pmtiles-probe.js +94 -0
- package/src/read-engine.js +175 -0
- package/src/sources.js +245 -0
- package/src/subscriptions.js +158 -0
- package/src/tilejson.js +112 -0
- package/src/tiles.js +284 -0
- package/src/torrent-create.js +250 -0
- package/src/warm.js +251 -0
- package/src/watch.js +98 -0
- package/src/web/index.html +254 -0
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
# pmtiles-swarm changelog
|
|
2
|
+
|
|
3
|
+
## master
|
|
4
|
+
### ⨠Features and improvements
|
|
5
|
+
- _...Add new stuff here..._
|
|
6
|
+
|
|
7
|
+
### š Bug fixes
|
|
8
|
+
- _...Add new stuff here..._
|
|
9
|
+
|
|
10
|
+
## 0.2.0
|
|
11
|
+
### ⨠Features and improvements
|
|
12
|
+
- **Serve tiles.** Every archive now has a TileJSON endpoint and a `{z}/{x}/{y}` tile endpoint
|
|
13
|
+
under `/archives/{infohash}/`. A node holding a complete copy reads its local file; a node in
|
|
14
|
+
cache mode reads through the swarm via `pmtiles-torrent`, fetching only the pieces a requested
|
|
15
|
+
tile lives in and seeding them back.
|
|
16
|
+
- The TileJSON carries a non-standard `torrent` block ā infohash, magnet, `.torrent` URL, web
|
|
17
|
+
seeds and any BEP 46 publisher key. Ordinary clients ignore it and fetch over HTTP;
|
|
18
|
+
torrent-aware clients use it to join the swarm directly. One URL serves both.
|
|
19
|
+
- Tiles are served `immutable` with a year-long max-age. An infohash pins content, so a tile
|
|
20
|
+
under one can never change, and an updated archive gets new URLs rather than needing a purge.
|
|
21
|
+
- **Warm a region before serving it.** `POST /api/torrents/{infohash}/warm` pre-fetches the
|
|
22
|
+
tiles covering a bounding box, so a cache-mode node is useful the moment it enters a
|
|
23
|
+
load-balanced pool rather than paying for the first request to every area. Progress and
|
|
24
|
+
cancellation via `GET` and `DELETE` on the same path. The zoom range is clamped to what the
|
|
25
|
+
archive actually holds.
|
|
26
|
+
- New `trustProxy` config option. With it set, absolute URLs in TileJSON and the RSS feed are
|
|
27
|
+
derived per request from `X-Forwarded-Proto` and `X-Forwarded-Host`, so one node can answer
|
|
28
|
+
correctly on both `https://public` and `http://internal`.
|
|
29
|
+
- Depend on `pmtiles-torrent` from npm, and drop the local copy of the libtorrent sidecar in
|
|
30
|
+
favour of the one it ships. The two copies had drifted: the read side had grown `info` and
|
|
31
|
+
`set_priority` ops this project never got, which are exactly what on-demand tile reads need.
|
|
32
|
+
- Upgrade WebTorrent to 3.x, dropping the `uint8-util` override that the 2.x line needed to add
|
|
33
|
+
magnets at all. **Node 20 is no longer supported** ā WebTorrent 3 requires Node 22+, and Node
|
|
34
|
+
20 reached end of life in April 2026.
|
|
35
|
+
|
|
36
|
+
### š Bug fixes
|
|
37
|
+
- Absolute URLs used the raw `Host` header, which behind a reverse proxy is the internal address
|
|
38
|
+
the proxy dialled. They now follow `X-Forwarded-Host` when a proxy is trusted, so published
|
|
39
|
+
tile and feed URLs are reachable.
|
|
40
|
+
- Stop tracking a compiled `.pyc` that predated the `__pycache__` ignore rule.
|
|
41
|
+
|
|
42
|
+
## 0.1.0
|
|
43
|
+
### ⨠Features and improvements
|
|
44
|
+
- Initial release: BitTorrent distribution for PMTiles map archives.
|
|
45
|
+
- Pluggable seeding engines: libtorrent (via sidecar), qBittorrent (WebUI API), and embedded WebTorrent.
|
|
46
|
+
- Four ways to add an archive: local file, remote URL, existing torrent or magnet, and adoption of what the engine already seeds.
|
|
47
|
+
- Web seeds (BEP 19) registered automatically, so a new archive is usable before it has any peers.
|
|
48
|
+
- Mirror and cache modes; joining defaults to cache so a large archive cannot silently claim the disk.
|
|
49
|
+
- Scheduled sources for upstreams publishing a new dated URL per build.
|
|
50
|
+
- Origin change detection, with optional guarded auto-rebuild.
|
|
51
|
+
- RSS publish and subscribe, with map metadata (format, zoom range, bounds) in each item.
|
|
52
|
+
- Hybrid v1+v2 torrent creation through libtorrent.
|
|
53
|
+
- BEP 46 mutable-torrent helpers.
|
package/LICENSE
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
BSD 3-Clause License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026, Andrew Calcutt and contributors
|
|
4
|
+
|
|
5
|
+
Redistribution and use in source and binary forms, with or without
|
|
6
|
+
modification, are permitted provided that the following conditions are met:
|
|
7
|
+
|
|
8
|
+
1. Redistributions of source code must retain the above copyright notice, this
|
|
9
|
+
list of conditions and the following disclaimer.
|
|
10
|
+
|
|
11
|
+
2. Redistributions in binary form must reproduce the above copyright notice,
|
|
12
|
+
this list of conditions and the following disclaimer in the documentation
|
|
13
|
+
and/or other materials provided with the distribution.
|
|
14
|
+
|
|
15
|
+
3. Neither the name of the copyright holder nor the names of its contributors
|
|
16
|
+
may be used to endorse or promote products derived from this software
|
|
17
|
+
without specific prior written permission.
|
|
18
|
+
|
|
19
|
+
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
|
|
20
|
+
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
|
|
21
|
+
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
|
|
22
|
+
DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
|
|
23
|
+
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
|
|
24
|
+
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
|
|
25
|
+
SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
|
|
26
|
+
CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
|
|
27
|
+
OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
|
|
28
|
+
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
package/NOTICE.md
ADDED
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# Third-party notices
|
|
2
|
+
|
|
3
|
+
`pmtiles-swarm` is licensed BSD-3-Clause (see [LICENSE](LICENSE)). This file records the
|
|
4
|
+
third-party work it builds on.
|
|
5
|
+
|
|
6
|
+
## PMTiles ā BSD-3-Clause (reference implementation), CC0-1.0 (specification)
|
|
7
|
+
|
|
8
|
+
> Copyright 2021 and later, Protomaps LLC and contributors
|
|
9
|
+
> https://github.com/protomaps/PMTiles
|
|
10
|
+
|
|
11
|
+
The `pmtiles` npm package is used at runtime to read archive headers and metadata. No
|
|
12
|
+
implementation code is copied.
|
|
13
|
+
|
|
14
|
+
## WebTorrent ā MIT
|
|
15
|
+
|
|
16
|
+
> Copyright (c) Feross Aboukhadijeh and WebTorrent, LLC
|
|
17
|
+
> https://github.com/webtorrent/webtorrent
|
|
18
|
+
|
|
19
|
+
`src/engines/webtorrent.js` is an adapter written against WebTorrent's public API.
|
|
20
|
+
`src/mutable.js` builds on `bittorrent-dht`'s BEP 44 put/get. `create-torrent` and `parse-torrent`,
|
|
21
|
+
from the same ecosystem and also MIT, are used for torrent creation and parsing. No implementation
|
|
22
|
+
code is copied.
|
|
23
|
+
|
|
24
|
+
Note the `uint8-util` override pinned in `package.json`: version 2.3.0 rewrote `arr2hex` in a way
|
|
25
|
+
that throws when handed the hex-string infohash that webtorrent's `Torrent._onTorrentId` passes it,
|
|
26
|
+
breaking every magnet add. webtorrent declares `^2.2.5`, so the pin holds it at a working version.
|
|
27
|
+
Remove it once webtorrent fixes the call site.
|
|
28
|
+
|
|
29
|
+
## qBittorrent ā GPL-2.0-or-later
|
|
30
|
+
|
|
31
|
+
> https://github.com/qbittorrent/qBittorrent
|
|
32
|
+
|
|
33
|
+
**No qBittorrent code is used in this project.** `src/engines/qbittorrent.js` is a client for its
|
|
34
|
+
documented WebUI HTTP API, written from the public API documentation. qBittorrent is GPL-2.0+,
|
|
35
|
+
which is not compatible with redistribution under this project's BSD-3-Clause license, so code
|
|
36
|
+
must not be copied from it. Speaking to a program over its network API does not create a derived
|
|
37
|
+
work; copying its source would.
|
|
38
|
+
|
|
39
|
+
The same caution applies to libtorrent-rasterbar if a libtorrent-backed engine is added later:
|
|
40
|
+
libtorrent is BSD-3-Clause and therefore fine to link and derive from, but it must be pulled in
|
|
41
|
+
directly rather than by way of any GPL client.
|
|
42
|
+
|
|
43
|
+
## BitTorrent Enhancement Proposals
|
|
44
|
+
|
|
45
|
+
The BEPs implemented here ā BEP 19 (web seeds), BEP 44 (DHT storage) and BEP 46 (updating torrents
|
|
46
|
+
via DHT mutable items) ā are open specifications published by the BitTorrent community at
|
|
47
|
+
https://www.bittorrent.org/beps/bep_0000.html.
|
package/README.md
ADDED
|
@@ -0,0 +1,172 @@
|
|
|
1
|
+
# pmtiles-swarm
|
|
2
|
+
|
|
3
|
+
BitTorrent distribution built for PMTiles map archives.
|
|
4
|
+
|
|
5
|
+
A torrent client and server that knows what a map is: it creates torrents from PMTiles archives,
|
|
6
|
+
watches folders for new builds, publishes an RSS feed carrying each archive's coverage and zoom
|
|
7
|
+
range, and follows other nodes' feeds to mirror or cache what they publish.
|
|
8
|
+
|
|
9
|
+
It does not reimplement BitTorrent. Seeding is delegated to an engine ā your existing qBittorrent,
|
|
10
|
+
or an embedded WebTorrent client ā so libtorrent keeps doing what it is good at.
|
|
11
|
+
|
|
12
|
+
```sh
|
|
13
|
+
npm install
|
|
14
|
+
node src/index.js --config swarm.config.json
|
|
15
|
+
# then open http://localhost:8090
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
## Documentation
|
|
19
|
+
|
|
20
|
+
- **[docs/engines.md](docs/engines.md)** ā libtorrent, qBittorrent and WebTorrent: what each
|
|
21
|
+
can do, what to install, and how cache mode differs between them.
|
|
22
|
+
- **[docs/publishing.md](docs/publishing.md)** ā creating torrents, web seeds, piece size,
|
|
23
|
+
hybrid v1+v2, watch folders, and the keep-or-discard choice when adding from a URL.
|
|
24
|
+
- **[docs/subscribing.md](docs/subscribing.md)** ā a worked two-node setup, mirror vs cache,
|
|
25
|
+
feed contents, and updatable torrents.
|
|
26
|
+
- **[docs/serving-tiles.md](docs/serving-tiles.md)** ā the TileJSON and z/x/y endpoints, the
|
|
27
|
+
`torrent` block that torrent-aware clients use, caching, and running behind a proxy.
|
|
28
|
+
- **[docs/architecture-diagram.md](docs/architecture-diagram.md)** ā how a publishing node, a
|
|
29
|
+
serving tier, the swarm and both kinds of client fit together.
|
|
30
|
+
|
|
31
|
+
## What it does
|
|
32
|
+
|
|
33
|
+
**Adds archives four ways.** A local `.pmtiles` file (hashed into a new torrent, data left where
|
|
34
|
+
it is); a remote URL (see below); an existing `.torrent` or magnet, which it simply joins; or
|
|
35
|
+
adoption of everything your torrent client already seeds ā the migration path for an existing
|
|
36
|
+
library, which re-hashes nothing.
|
|
37
|
+
|
|
38
|
+
**Watches folders.** A new `.pmtiles` appearing in a watched folder is imported automatically.
|
|
39
|
+
Imports wait for the file to stop changing first, because hashing a half-written archive produces
|
|
40
|
+
a torrent for bytes that no longer exist.
|
|
41
|
+
|
|
42
|
+
**Publishes RSS.** `/feed.xml`, and `/feed/<category>.xml` per category. Plain RSS 2.0 with
|
|
43
|
+
torrent enclosures, so **qBittorrent's built-in RSS auto-downloader can subscribe today** with no
|
|
44
|
+
new software. Items also carry a namespaced description of the map ā format, zoom range, bounds,
|
|
45
|
+
tile count ā so a subscriber can decide whether it wants a 72 GiB download before starting one.
|
|
46
|
+
|
|
47
|
+
**Follows feeds.** Subscribed feeds are polled and new archives added in one of two modes.
|
|
48
|
+
|
|
49
|
+
**Serves tiles.** Every archive is also a TileJSON endpoint and a `{z}/{x}/{y}` tile endpoint, so
|
|
50
|
+
a map can point straight at it. A node holding a complete copy reads its local file; a node in
|
|
51
|
+
cache mode reads through the swarm, pulling only the pieces a requested tile lives in. The
|
|
52
|
+
TileJSON carries a `torrent` block that ordinary clients ignore and torrent-aware ones use to
|
|
53
|
+
join the swarm directly ā one URL serves both. See
|
|
54
|
+
[docs/serving-tiles.md](docs/serving-tiles.md).
|
|
55
|
+
|
|
56
|
+
## Mirror and cache
|
|
57
|
+
|
|
58
|
+
The distinction is the point of the project.
|
|
59
|
+
|
|
60
|
+
| Mode | Disk cost | What it is for |
|
|
61
|
+
| --- | --- | --- |
|
|
62
|
+
| `mirror` | the whole archive | Becoming a full seeder and adding redundancy to the swarm |
|
|
63
|
+
| `cache` | only what is read | Serving tiles from a 700 GiB archive on a small disk |
|
|
64
|
+
|
|
65
|
+
Cache mode joins the swarm without downloading anything up front. A tile server reads byte ranges
|
|
66
|
+
on demand through [`pmtiles-torrent`](https://github.com/TechIdiots-LLC/pmtiles-torrent), and the node still
|
|
67
|
+
seeds whatever pieces it has picked up along the way. Disk use tracks what people actually look
|
|
68
|
+
at rather than what exists.
|
|
69
|
+
|
|
70
|
+
Joining a torrent **defaults to cache**, because committing a disk to a copy of something that may
|
|
71
|
+
be hundreds of gigabytes should be a decision, not a side effect.
|
|
72
|
+
|
|
73
|
+
Cache mode needs piece-level control. WebTorrent has it; qBittorrent's WebUI exposes only per-file
|
|
74
|
+
priorities, which for a single-file archive is all or nothing ā so under the qBittorrent engine,
|
|
75
|
+
cache mode adds the torrent stopped and leaves on-demand reads to a client that can do them.
|
|
76
|
+
|
|
77
|
+
## Adding from a URL
|
|
78
|
+
|
|
79
|
+
Piece hashes are computed over content, so there is no way to create a torrent from a remote
|
|
80
|
+
archive without reading every byte of it. What you choose is whether those bytes are kept:
|
|
81
|
+
|
|
82
|
+
- **retain** (default) ā written to disk as they arrive, so the node is a real seeder the moment
|
|
83
|
+
the torrent is published.
|
|
84
|
+
- **discard** (`"retain": false`) ā streamed past the hasher and dropped. No disk cost, but the
|
|
85
|
+
node cannot seed what it just published; peers depend on the web seed until someone mirrors it.
|
|
86
|
+
|
|
87
|
+
Either way the origin URL is registered as a **web seed** (BEP 19), which is what makes a
|
|
88
|
+
brand-new archive usable before it has any peers at all. If your archives are already on a web
|
|
89
|
+
server or S3, always pass the URL ā it turns cold start from a dead end into an HTTP fallback that
|
|
90
|
+
gets cheaper as peers appear.
|
|
91
|
+
|
|
92
|
+
## Updatable torrents
|
|
93
|
+
|
|
94
|
+
A rebuilt archive is a different torrent: the infohash is a hash of the content. Two ways to carry
|
|
95
|
+
subscribers forward, and they fail differently, so publishing both is cheap insurance.
|
|
96
|
+
|
|
97
|
+
- **RSS** ā easy to consume, understood by existing clients, needs a server that stays up.
|
|
98
|
+
- **BEP 46** ā an ed25519-signed DHT record naming the current infohash, addressed by public key
|
|
99
|
+
rather than infohash (`magnet:?xs=urn:btpk:ā¦`). No server needed, but the record expires and must
|
|
100
|
+
be republished. See [src/mutable.js](src/mutable.js).
|
|
101
|
+
|
|
102
|
+
## Configuration
|
|
103
|
+
|
|
104
|
+
```json
|
|
105
|
+
{
|
|
106
|
+
"port": 8090,
|
|
107
|
+
"dataDir": "./data",
|
|
108
|
+
"engine": "qbittorrent",
|
|
109
|
+
"qbittorrent": { "url": "http://127.0.0.1:8080", "username": "admin", "password": "ā¦" },
|
|
110
|
+
"webtorrent": { "savePath": "./data/torrents-data" },
|
|
111
|
+
"pieceLength": 4194304,
|
|
112
|
+
"maxConnections": 100,
|
|
113
|
+
"feedMaxItems": 50,
|
|
114
|
+
"publicUrl": "https://maps.example.org",
|
|
115
|
+
"watch": [
|
|
116
|
+
{ "path": "/mnt/maps/generated", "category": "basemaps", "webSeedBase": "https://maps.example.org/files" }
|
|
117
|
+
],
|
|
118
|
+
"subscriptions": [
|
|
119
|
+
{ "url": "https://other.example.org/feed.xml", "mode": "cache", "filter": "terrain" }
|
|
120
|
+
]
|
|
121
|
+
}
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
Environment overrides: `PMTILES_SWARM_PORT`, `PMTILES_SWARM_DATA_DIR`, `PMTILES_SWARM_ENGINE`,
|
|
125
|
+
`PMTILES_SWARM_QBT_URL`, `PMTILES_SWARM_QBT_USERNAME`, `PMTILES_SWARM_QBT_PASSWORD`,
|
|
126
|
+
`PMTILES_SWARM_PUBLIC_URL`.
|
|
127
|
+
|
|
128
|
+
### Piece length
|
|
129
|
+
|
|
130
|
+
Creation tools size pieces for whole-file downloads ā 16 MiB or more for a large archive. That is
|
|
131
|
+
a poor fit for a tile server, where a cold tile costs a whole piece regardless of how few bytes it
|
|
132
|
+
needs. The default here is 4 MiB, trading a larger hash list for a quarter of the read
|
|
133
|
+
amplification. Measured against a 72 GiB archive at 16 MiB pieces, a cold tile cost roughly 30
|
|
134
|
+
seconds against a single peer.
|
|
135
|
+
|
|
136
|
+
Overridable globally, per scheduled source, and per request ā an archive nobody will read
|
|
137
|
+
randomly can keep larger pieces. Note that piece size has little bearing on load imposed on
|
|
138
|
+
network equipment: peers request 16 KiB blocks whatever the piece size. The setting that
|
|
139
|
+
matters there is `maxConnections`, since every peer holds a NAT table entry. See
|
|
140
|
+
[docs/publishing.md](docs/publishing.md).
|
|
141
|
+
|
|
142
|
+
## API
|
|
143
|
+
|
|
144
|
+
| Method | Path | Purpose |
|
|
145
|
+
| --- | --- | --- |
|
|
146
|
+
| `GET` | `/api/status` | Engine health, counts, watched folders, subscriptions |
|
|
147
|
+
| `GET` | `/api/torrents` | Catalog joined with live swarm state |
|
|
148
|
+
| `POST` | `/api/torrents` | Add via `{path}`, `{url}`, `{magnet}`, `{torrentUrl}`, or a raw `.torrent` body |
|
|
149
|
+
| `DELETE` | `/api/torrents/:infoHash` | Remove (`?deleteData=true` to delete data too) |
|
|
150
|
+
| `GET` | `/api/torrents/:infoHash/file` | Download the `.torrent` |
|
|
151
|
+
| `GET` | `/api/torrents/:infoHash/magnet` | Magnet URI |
|
|
152
|
+
| `GET` | `/api/torrents/:infoHash/peers` | Per-peer detail |
|
|
153
|
+
| `POST` | `/api/torrents/:infoHash/check` | Has the source changed since the torrent was made? |
|
|
154
|
+
| `POST` | `/api/torrents/:infoHash/rebuild` | Rebuild from the current source (mints a new infohash) |
|
|
155
|
+
| `POST` | `/api/check-origins` | Check every archive with a watchable source |
|
|
156
|
+
| `POST` | `/api/adopt` | Import what the engine already holds |
|
|
157
|
+
| `POST` | `/api/subscriptions/refresh` | Poll subscribed feeds now |
|
|
158
|
+
| `GET` | `/feed.xml`, `/feed/:category.xml` | RSS |
|
|
159
|
+
|
|
160
|
+
## Status
|
|
161
|
+
|
|
162
|
+
Working and verified end to end against a live 71.93 GiB OpenMapTiles archive: torrent creation,
|
|
163
|
+
joining existing torrents, catalog persistence, map metadata extraction, category feeds, live
|
|
164
|
+
swarm stats, and cache mode holding at zero bytes on disk.
|
|
165
|
+
|
|
166
|
+
Not yet exercised: the qBittorrent engine against a real instance, watch-folder imports, feed
|
|
167
|
+
subscription round-trips between two nodes, and BEP 46 publish/resolve against a live DHT (the
|
|
168
|
+
crypto and magnet handling are tested; interop with libtorrent's encoding is not).
|
|
169
|
+
|
|
170
|
+
## License and attribution
|
|
171
|
+
|
|
172
|
+
BSD-3-Clause. Third-party notices in [NOTICE.md](NOTICE.md).
|
|
@@ -0,0 +1,287 @@
|
|
|
1
|
+
# Architecture Diagram ā pmtiles-swarm
|
|
2
|
+
|
|
3
|
+
How a map library is ingested once, distributed over BitTorrent, and served to
|
|
4
|
+
both ordinary and torrent-aware clients from the same URLs.
|
|
5
|
+
|
|
6
|
+
Four views: **the whole topology**, **how a tile request resolves**, **what a
|
|
7
|
+
torrent-aware client does differently**, and **what happens when an archive is
|
|
8
|
+
updated**.
|
|
9
|
+
(Renders in VS Code's Markdown preview and on GitHub.)
|
|
10
|
+
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
## 1. Topology
|
|
14
|
+
|
|
15
|
+
One node publishes, any number serve, everything meets in the swarm.
|
|
16
|
+
|
|
17
|
+
**"Primary" and "secondary" are roles, not node types.** The only thing that makes
|
|
18
|
+
a node primary is that it creates torrents and owns the catalog others subscribe
|
|
19
|
+
to. It can sit in the load-balanced pool alongside everything else ā it holds
|
|
20
|
+
complete copies, so it is the *best* tile server in the fleet, not an exception
|
|
21
|
+
to it. The diagram separates them only to keep the arrows legible.
|
|
22
|
+
|
|
23
|
+
Likewise **mirror and cache are per-archive choices, not node identities.** A
|
|
24
|
+
serving node can mirror the archives it cares about and cache the rest. Mirroring
|
|
25
|
+
everything is a perfectly good deployment: every node seeds fully, every node
|
|
26
|
+
reads locally, and cache mode is there for the nodes and archives where a full
|
|
27
|
+
copy is not worth the disk.
|
|
28
|
+
|
|
29
|
+
```mermaid
|
|
30
|
+
%%{init: {"flowchart": {"rankSpacing": 80, "nodeSpacing": 50}}}%%
|
|
31
|
+
graph LR
|
|
32
|
+
subgraph IN["Ingest ā any of four ways"]
|
|
33
|
+
F1["watch folder<br/><small>new .pmtiles appears</small>"]
|
|
34
|
+
F2["scheduled source<br/><small>protomaps, mapterhornā¦</small>"]
|
|
35
|
+
F3["torrent / magnet URI<br/><small>join an existing swarm</small>"]
|
|
36
|
+
F4["local file or HTTP URL<br/><small>hashed into a new torrent</small>"]
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
F1 & F2 & F3 & F4 --> PRI
|
|
40
|
+
|
|
41
|
+
PRI["<b>pmtiles-swarm Ā· publisher</b><br/>creates torrents Ā· owns the catalog<br/>publishes /feed.xml<br/><small>mirrors ā and can serve tiles too</small>"]
|
|
42
|
+
|
|
43
|
+
PRI -->|"RSS<br/>torrents + magnets"| S1 & S2 & S3
|
|
44
|
+
|
|
45
|
+
subgraph SEC["Serving tier ā subscribed to the publisher's feed"]
|
|
46
|
+
S1["pmtiles-swarm<br/><small>mirror ā reads locally, seeds fully</small>"]
|
|
47
|
+
S2["pmtiles-swarm<br/><small>mirror ā reads locally, seeds fully</small>"]
|
|
48
|
+
S3["pmtiles-swarm<br/><small>cache ā reads pieces on demand</small>"]
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
BT{{"BitTorrent swarm<br/><small>DHT Ā· trackers Ā· web seeds</small>"}}
|
|
52
|
+
|
|
53
|
+
PRI <==>|"seeds"| BT
|
|
54
|
+
S1 & S2 <==>|"seed complete copies"| BT
|
|
55
|
+
S3 <==>|"pull pieces on demand<br/>seed back what it holds"| BT
|
|
56
|
+
|
|
57
|
+
PRI --> LB
|
|
58
|
+
S1 & S2 & S3 --> LB["Load balancer / CDN<br/>HAProxy Ā· Cloudflare<br/><small>caches tiles ā they are immutable</small>"]
|
|
59
|
+
|
|
60
|
+
LB --> C1 & C2
|
|
61
|
+
|
|
62
|
+
subgraph CL["Clients"]
|
|
63
|
+
C1["<b>Torrent-aware client</b><br/>maplibre-maui-ac<br/><small>reads the torrent block,<br/>joins the swarm directly</small>"]
|
|
64
|
+
C2["<b>Ordinary client</b><br/>maplibre-gl-js Ā· Leaflet<br/><small>ignores the torrent block,<br/>fetches tiles over HTTP</small>"]
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
C1 <==>|"tiles from pieces"| BT
|
|
68
|
+
|
|
69
|
+
%% 4-6 = RSS distribution, 7-10 and 17 = BitTorrent, rest = HTTP
|
|
70
|
+
linkStyle 4,5,6 stroke:#3F8F4F,stroke-width:2.5px;
|
|
71
|
+
linkStyle 7,8,9,10,17 stroke:#F5A623,stroke-width:3.5px;
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
**Key points:** **orange = BitTorrent, green = RSS distribution, plain = HTTP.**
|
|
75
|
+
The publisher is the only node that *creates* torrents; the rest learn about
|
|
76
|
+
archives from its feed and join the swarm, each choosing per archive whether to
|
|
77
|
+
mirror it or cache it.
|
|
78
|
+
|
|
79
|
+
Every node is both a reader and a seeder. A mirror seeds the whole archive; a
|
|
80
|
+
cache-mode node seeds whatever pieces it has pulled to answer requests. Either
|
|
81
|
+
way serving load turns into swarm capacity rather than consuming it, which is the
|
|
82
|
+
inversion that makes this worth building.
|
|
83
|
+
|
|
84
|
+
The publisher is a single point of failure for **publishing new archives only**.
|
|
85
|
+
Once a torrent exists, the swarm and the serving tier keep working without it ā
|
|
86
|
+
including the feed's existing items, which subscribers have already acted on.
|
|
87
|
+
|
|
88
|
+
---
|
|
89
|
+
|
|
90
|
+
## 2. How a tile request resolves
|
|
91
|
+
|
|
92
|
+
The same URL takes very different paths depending on who asks and what the
|
|
93
|
+
answering node holds.
|
|
94
|
+
|
|
95
|
+
```mermaid
|
|
96
|
+
%%{init: {"flowchart": {"rankSpacing": 55, "nodeSpacing": 40}}}%%
|
|
97
|
+
graph TD
|
|
98
|
+
REQ["GET /archives/{infohash}/{z}/{x}/{y}.pbf"] --> CDN{"CDN cache?"}
|
|
99
|
+
CDN -->|"hit"| DONE(["tile bytes"])
|
|
100
|
+
CDN -->|"miss"| NODE["a secondary, via the load balancer"]
|
|
101
|
+
|
|
102
|
+
NODE --> HOLD{"does this node hold<br/>a complete copy?"}
|
|
103
|
+
HOLD -->|"yes Ā· mirror"| LOCAL["read the local file<br/><small>NodeFileSource</small>"]
|
|
104
|
+
HOLD -->|"no Ā· cache mode"| ENG{"can the engine<br/>read pieces?"}
|
|
105
|
+
|
|
106
|
+
ENG -->|"libtorrent Ā· webtorrent"| SWARM["map the byte range onto pieces<br/><small>TorrentSource</small>"]
|
|
107
|
+
ENG -->|"qBittorrent"| ERR["501 ā no piece-level read"]
|
|
108
|
+
|
|
109
|
+
SWARM --> PC{"piece cached?"}
|
|
110
|
+
PC -->|"yes"| LOCAL
|
|
111
|
+
PC -->|"no"| FETCH["fetch the piece<br/><small>web seed, or peers</small>"]
|
|
112
|
+
FETCH --> SEED["keep it Ā· seed it back"]
|
|
113
|
+
SEED --> LOCAL
|
|
114
|
+
|
|
115
|
+
LOCAL --> DONE
|
|
116
|
+
|
|
117
|
+
%% 10-12 = the piece fetch and seed-back path
|
|
118
|
+
linkStyle 10,11,12 stroke:#F5A623,stroke-width:3px;
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
**Key points:** a cold tile on a cache-mode node costs one piece fetch ā and a
|
|
122
|
+
**web seed answers that in well under a second**, against 30+ seconds for a cold
|
|
123
|
+
swarm-only fetch. If your archives are also on plain HTTP storage, put that URL
|
|
124
|
+
in the torrent's `url-list`; it is the single biggest lever on this whole design.
|
|
125
|
+
|
|
126
|
+
A node holding a complete copy skips all of this and reads its local file.
|
|
127
|
+
|
|
128
|
+
---
|
|
129
|
+
|
|
130
|
+
## 3. What a torrent-aware client does
|
|
131
|
+
|
|
132
|
+
The server-side path above is what an *ordinary* client triggers. A torrent-aware
|
|
133
|
+
client does something different: it takes over the archive reading itself, and
|
|
134
|
+
stops needing the tile endpoint at all.
|
|
135
|
+
|
|
136
|
+
The important part is that it does both at once ā HTTP for the first paint, swarm
|
|
137
|
+
in the background ā so there is never a blank map waiting for metadata.
|
|
138
|
+
|
|
139
|
+
```mermaid
|
|
140
|
+
sequenceDiagram
|
|
141
|
+
autonumber
|
|
142
|
+
participant App as Map (maplibre)
|
|
143
|
+
participant P as Plugin
|
|
144
|
+
participant HTTP as pmtiles-swarm<br/>(via CDN)
|
|
145
|
+
participant BT as BitTorrent swarm
|
|
146
|
+
|
|
147
|
+
App->>P: load style ā tiles.json
|
|
148
|
+
P->>HTTP: GET /archives/{hash}/tiles.json
|
|
149
|
+
HTTP-->>P: TileJSON + torrent block
|
|
150
|
+
|
|
151
|
+
Note over P: claims the /archives/{hash}/ prefix,<br/>so only these URLs come to it
|
|
152
|
+
|
|
153
|
+
par Map is usable immediately
|
|
154
|
+
App->>P: tile 12/2145/1436
|
|
155
|
+
P->>HTTP: GET ā¦/12/2145/1436.pbf
|
|
156
|
+
HTTP-->>App: tile bytes
|
|
157
|
+
and Swarm warms up in the background
|
|
158
|
+
P->>BT: join (.torrent ā metadata already in hand)
|
|
159
|
+
BT-->>P: connected
|
|
160
|
+
P->>BT: fetch PMTiles header + root directory
|
|
161
|
+
BT-->>P: those pieces
|
|
162
|
+
Note over P: now able to resolve any tile<br/>to a byte range locally
|
|
163
|
+
end
|
|
164
|
+
|
|
165
|
+
App->>P: tile 12/2146/1436
|
|
166
|
+
Note over P: tile ā byte range (PMTiles directory)<br/>ā piece index
|
|
167
|
+
P->>BT: fetch that piece
|
|
168
|
+
BT-->>P: piece
|
|
169
|
+
P-->>App: tile bytes, no HTTP involved
|
|
170
|
+
|
|
171
|
+
Note over P,BT: the client is now a peer ā<br/>it seeds those pieces back
|
|
172
|
+
|
|
173
|
+
App->>P: tile in an unfetched region
|
|
174
|
+
P->>BT: fetch piece
|
|
175
|
+
BT--)P: too slow / unavailable
|
|
176
|
+
P->>HTTP: fall back for this tile
|
|
177
|
+
HTTP-->>App: tile bytes
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
**Key points:** the plugin resolves tiles the same way the server does ā PMTiles
|
|
181
|
+
directory lookup, byte range, piece index ā it just does it on the device. That
|
|
182
|
+
is why the `torrent` block carries the archive's `.torrent` rather than per-tile
|
|
183
|
+
URLs: **there is nothing tile-specific in the swarm.** The swarm holds one file,
|
|
184
|
+
and both ends know how to read tiles out of it.
|
|
185
|
+
|
|
186
|
+
Three consequences worth being clear about:
|
|
187
|
+
|
|
188
|
+
- **HTTP is never fully abandoned.** It is the fallback for anything the swarm
|
|
189
|
+
cannot answer quickly, and the only path until the swarm is connected.
|
|
190
|
+
- **The client becomes a seeder.** Every piece it pulls, it serves ā so a popular
|
|
191
|
+
region gets *faster* as more clients view it, which is the opposite of how a
|
|
192
|
+
tile server behaves under load.
|
|
193
|
+
- **Prefer the `.torrent` over the magnet.** A magnet carries only an infohash, so
|
|
194
|
+
the client must find peers and complete a metadata exchange before it knows
|
|
195
|
+
anything about the archive ā measured at 90 to 240 seconds against a 72 GiB
|
|
196
|
+
archive. The `.torrent` served alongside the TileJSON already contains the
|
|
197
|
+
metadata and is ready immediately.
|
|
198
|
+
|
|
199
|
+
---
|
|
200
|
+
|
|
201
|
+
## 4. Updating an archive
|
|
202
|
+
|
|
203
|
+
New data means a new infohash, which is what makes cache invalidation free.
|
|
204
|
+
|
|
205
|
+
```mermaid
|
|
206
|
+
%%{init: {"flowchart": {"rankSpacing": 60}}}%%
|
|
207
|
+
graph LR
|
|
208
|
+
NEW["new planet.pmtiles<br/><small>weekly build</small>"] --> HASH["primary re-hashes<br/>ā new infohash"]
|
|
209
|
+
HASH --> BEP["BEP 46 mutable entry<br/><small>same public key, seq+1</small>"]
|
|
210
|
+
HASH --> FEED["new RSS item"]
|
|
211
|
+
|
|
212
|
+
BEP --> FOLLOW["clients following the key<br/>see the new version"]
|
|
213
|
+
FEED --> SUBS["secondaries add the new torrent"]
|
|
214
|
+
|
|
215
|
+
SUBS --> URLS["tiles.json now points at<br/>/archives/{new infohash}/ā¦"]
|
|
216
|
+
URLS --> CACHE["old CDN entries simply<br/>stop being referenced<br/><small>no purge needed</small>"]
|
|
217
|
+
|
|
218
|
+
linkStyle 2,4 stroke:#3F8F4F,stroke-width:2.5px;
|
|
219
|
+
```
|
|
220
|
+
|
|
221
|
+
**Key points:** tile URLs are content-addressed, so they never need invalidating.
|
|
222
|
+
The old infohash stays valid and servable for as long as anyone still holds it ā
|
|
223
|
+
useful for clients pinned to a known-good build ā while new requests move to the
|
|
224
|
+
new one as soon as they re-read `tiles.json`. The only mutable thing in the whole
|
|
225
|
+
system is that one document.
|
|
226
|
+
|
|
227
|
+
---
|
|
228
|
+
|
|
229
|
+
## Deployment notes
|
|
230
|
+
|
|
231
|
+
**Decide how absolute URLs get built.** TileJSON and the RSS feed both contain
|
|
232
|
+
absolute URLs, and there are three ways to arrive at them:
|
|
233
|
+
|
|
234
|
+
| Config | Behaviour | Use when |
|
|
235
|
+
| --- | --- | --- |
|
|
236
|
+
| `publicUrl` set | One canonical URL, whatever the request said | There is a single public name |
|
|
237
|
+
| `trustProxy` set | Per request, from `X-Forwarded-Proto` and `X-Forwarded-Host` | One node answers on several names or schemes |
|
|
238
|
+
| neither | From the connection itself | Direct access, no proxy |
|
|
239
|
+
|
|
240
|
+
```json
|
|
241
|
+
{
|
|
242
|
+
"trustProxy": "loopback, 10.0.0.0/8"
|
|
243
|
+
}
|
|
244
|
+
```
|
|
245
|
+
|
|
246
|
+
`trustProxy` is what lets a single node serve **both** `https://maps.example.org`
|
|
247
|
+
to the internet and `http://maps.internal` to the LAN, rewriting every published
|
|
248
|
+
URL to match how the request arrived. It takes anything Express accepts: `true`,
|
|
249
|
+
a hop count, or a subnet list. Prefer the subnet list ā trusting these headers
|
|
250
|
+
from an untrusted client lets it claim any host it likes, and that host ends up
|
|
251
|
+
in documents you publish.
|
|
252
|
+
|
|
253
|
+
Behind a TLS-terminating proxy with neither option set, the node advertises
|
|
254
|
+
`http://` URLs, and a browser that loaded the map over `https` blocks every one
|
|
255
|
+
as mixed content ā which looks like an empty map rather than a misconfiguration.
|
|
256
|
+
|
|
257
|
+
**Load balancing needs no session affinity.** Any node serving a given infohash
|
|
258
|
+
returns byte-identical tiles, because the infohash pins the content. Round-robin
|
|
259
|
+
is fine, and a node can be added or removed mid-request-stream without a client
|
|
260
|
+
noticing.
|
|
261
|
+
|
|
262
|
+
**Cold start only applies to cache-mode nodes.** A node holding a complete copy
|
|
263
|
+
reads its local file and is fast from the first request. If every serving node
|
|
264
|
+
mirrors, this section does not apply to you at all.
|
|
265
|
+
|
|
266
|
+
Where nodes *do* run in cache mode, each warms its own piece cache, so scattering
|
|
267
|
+
requests for one region across N nodes costs N first-fetches rather than one.
|
|
268
|
+
Three things reduce that, in order of effect:
|
|
269
|
+
|
|
270
|
+
1. **The CDN absorbs repeats.** Tiles are immutable, so the second request for a
|
|
271
|
+
tile never reaches any node.
|
|
272
|
+
2. **The nodes are peers in the same swarm.** A node fetching a piece a sibling
|
|
273
|
+
already holds gets it *from that sibling*, usually over the LAN. Local service
|
|
274
|
+
discovery is on by default, so same-subnet nodes find each other without
|
|
275
|
+
configuration. The cost is one external fetch plus Nā1 local ones, not N
|
|
276
|
+
external ones.
|
|
277
|
+
3. **Warm before rotating in.** `POST /api/torrents/{infohash}/warm` pre-fetches
|
|
278
|
+
a region so the first real request is never the slow one. See
|
|
279
|
+
[serving tiles](serving-tiles.md#warming-a-region).
|
|
280
|
+
|
|
281
|
+
Path-based affinity (HAProxy's `balance uri`) helps too, but it is the smallest
|
|
282
|
+
of the four levers and it costs you even load distribution.
|
|
283
|
+
|
|
284
|
+
**Size cache-mode disks for what gets viewed, not for the archive.** Cache usage
|
|
285
|
+
grows with what people actually look at and is not bounded on its own. Watch it,
|
|
286
|
+
and mirror instead where a full copy is affordable ā a mirror is predictable,
|
|
287
|
+
faster, and a better peer.
|