pmtiles-swarm 0.48.0 → 0.50.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -7,6 +7,74 @@
7
7
  ### 🐞 Bug fixes
8
8
  - _...Add new stuff here..._
9
9
 
10
+ ## 0.50.0
11
+ ### ✨ Features and improvements
12
+ - **Three switches for what a node offers of an archive's own bytes**, on the node, on a watched
13
+ folder, on a scheduled source, and on any individual archive from **HTTP sources** in its details.
14
+ They are separate because they are three different exposures, and a node can reasonably want any
15
+ one of them without the others:
16
+
17
+ - `serveArchive` — whether `/archives/<infohash>/archive.pmtiles` answers at all. This is the one
18
+ that decides whether a stranger who knows an infohash can pull 700 GiB off the box.
19
+ - `selfWebSeed` — whether this node's own URL goes into the torrent's `url-list`, so every peer
20
+ holding the torrent fetches from here. Turning it on writes the URL into the `.torrent` and the
21
+ magnet; turning it off takes that URL back out.
22
+ - `publicDownload` — whether the public catalogue page offers it as a download. Serving a file to a
23
+ reader that was handed the URL and advertising it to every visitor are different decisions.
24
+
25
+ The last two are read as off wherever the first is, whatever the record says. A web seed URL that
26
+ answers `403` is worse than no web seed, because a client spends its retries on it, and a download
27
+ link that `403`s is worse than no link. An archive that says nothing about a setting goes on
28
+ following the node, so changing the node's answer reaches everything that never had one of its own.
29
+
30
+ - **`DELETE /api/torrents/<infohash>/webseeds`** drops web seeds, the same rewrite as adding them and
31
+ safe for the same reason: `url-list` sits outside the info dictionary, so the infohash — and every
32
+ magnet and peer depending on it — is untouched.
33
+
34
+ ### 🐞 Bug fixes
35
+ - **Serving whole archives over HTTP is no longer on for everyone.** 0.48.0 added
36
+ `/archives/<infohash>/archive.pmtiles` and left it answering for every complete archive, on every
37
+ node, to anyone who knew an infohash. Everything else this node publishes is either small — TileJSON,
38
+ a `.torrent`, a feed — or metered by the request, one tile at a time, so turning a node on had never
39
+ meant offering its disk to strangers. It does not now either: `serveArchive` defaults to off and
40
+ both range endpoints answer `403` until it is set.
41
+
42
+ ## 0.49.0
43
+ ### ✨ Features and improvements
44
+ - **The current build of a category can be read as a file.** `GET /latest/<category>/archive.pmtiles`
45
+ is what `/archives/<infohash>/archive.pmtiles` is, addressed the way a style or a long-lived config
46
+ wants to address it: by what it is rather than by which build it happens to be. Point any PMTiles
47
+ reader at it and it keeps working across rebuilds, with no infohash to chase.
48
+
49
+ - **Every `/latest/` endpoint now carries an ETag, and the tag is the infohash.** Everything under
50
+ `/archives/` is content-addressed and cached for a year, because the URL changes when the content
51
+ does. A `/latest/` URL is the opposite — stable on purpose, so the content underneath it moves and
52
+ the URL alone gives a cache no way to notice. These endpoints had a five-minute TTL and nothing
53
+ else, which is a guess: for those five minutes every client and every proxy in front of one serves
54
+ the previous build and not one of them can tell. They are now `max-age=60, must-revalidate` with a
55
+ validator that changes exactly when the archive does — and, because it is the infohash, one that
56
+ two nodes behind a load balancer agree on instead of each deriving its own from a body hash or an
57
+ mtime. The TileJSON, the `.torrent` redirect, the magnet, the single-item feed and the `/latest/`
58
+ index are all covered.
59
+
60
+ ### 🐞 Bug fixes
61
+ - **A range request could splice two builds together.** `/latest/<category>/archive.pmtiles` honours
62
+ `If-Range`, and refuses as a range — answering in full instead — anything conditioned on a build
63
+ that is no longer current. This is the failure the ETag exists to prevent: a PMTiles reader does
64
+ not fetch a file, it fetches a header, then a root directory, then leaf directories, then tiles,
65
+ over minutes or hours. A rebuild landing partway through leaves it reading old offsets against new
66
+ bytes, which decodes as the wrong tile or as nothing, with no error anywhere naming the cause.
67
+
68
+ - **`/archives/<infohash>/archive.pmtiles` was unreachable from a browser on another origin**, and
69
+ sent a validator no PMTiles reader would use. It had no CORS header at all — unlike the tile and
70
+ TileJSON routes beside it — so a page elsewhere could not fetch it. Its ETag was also whatever
71
+ Express derives from the file's size and mtime, which is weak (the official reader discards any tag
72
+ beginning with `W/`) and different on every node, so two nodes behind one load balancer would hand
73
+ a reader two tags for byte-identical archives and it would conclude the file had moved under it.
74
+ Both range routes now send the infohash as a strong tag, and both expose `ETag` and `Content-Range`
75
+ to cross-origin JavaScript — without which the reader compares against `null`, the comparison never
76
+ fires, and it splices builds in silence.
77
+
10
78
  ## 0.48.0
11
79
  ### ✨ Features and improvements
12
80
  - **An archive can now be read as a file, by byte range.** `GET /archives/<infohash>/archive.pmtiles`
package/README.md CHANGED
@@ -756,6 +756,8 @@ which the endpoint answers 501.
756
756
  | `POST` | `/api/torrents/:infoHash/recheck` | Hash what is on disk again and believe the result — for an archive whose progress and whose files disagree |
757
757
  | `POST` | `/api/torrents/:infoHash/pause`, `/resume` | Stop offering it, without forgetting it |
758
758
  | `POST` | `/api/torrents/:infoHash/webseeds` | Add web seeds — does not change the infohash |
759
+ | `DELETE` | `/api/torrents/:infoHash/webseeds` | Drop web seeds — likewise |
760
+ | `POST` | `/api/torrents/:infoHash/publish` | What this node offers of the archive itself: `serveArchive`, `selfWebSeed`, `publicDownload` |
759
761
  | `POST` | `/api/torrents/:infoHash/warm` | Pre-fetch a region (`GET` for progress, `DELETE` to cancel) |
760
762
  | `DELETE` | `/api/torrents/:infoHash/cache` | Reclaim cached pieces, keep the archive |
761
763
  | `POST` | `/api/torrents/:infoHash/check` | Has the source changed since the torrent was made? |
@@ -783,9 +785,11 @@ which the endpoint answers 501.
783
785
  | `GET` | `/health` | 200 when this node can serve, 503 when its engine cannot — **public**, no credential, for a load balancer |
784
786
  | `GET` | `/archives/:infoHash/ready` | Whether this node can serve _this_ archive yet: 200 ready, 503 not yet, 415 never — **public** |
785
787
  | `GET` | `/archives/:infoHash/archive.torrent` | The `.torrent` a torrent-aware client joins with — **public** |
786
- | `GET` | `/archives/:infoHash/archive.pmtiles` | The archive itself, by byte range — what any PMTiles reader wants, and a valid web seed. Complete archives only — **public** |
788
+ | `GET` | `/archives/:infoHash/archive.pmtiles` | The archive itself, by byte range — what any PMTiles reader wants, and a valid web seed. Complete archives only, and only where [`serveArchive`](docs/configuration.md#servearchive) is on — **public** |
787
789
  | `GET` | `/archives/:infoHash/preview` | Map preview for one archive — admin, not public |
790
+ | `GET` | `/latest/` | Every category and the endpoints that resolve to its newest build — **public**, the index of everything below |
788
791
  | `GET` | `/latest/:category/tiles.json`, `/:name.torrent`, `/magnet` | The newest build in a category — **public**. The torrent name is yours to choose, so a link can read `planetiler-openmaptiles-latest.torrent`; it redirects to the immutable URL, which names the download after the build it actually is |
792
+ | `GET` | `/latest/:category/archive.pmtiles` | The newest build in a category as a file, by byte range — **public**, and gated by [`serveArchive`](docs/configuration.md#servearchive). What to point a PMTiles reader at when you want "whichever is current" rather than one build |
789
793
  | `GET` | `/feed.xml`, `/feed/:category.xml`, `/latest/:category.xml` | RSS — **public** |
790
794
 
791
795
  Everything under `/api/` is guarded once a credential is configured; tiles, TileJSON and the feeds
@@ -13,6 +13,7 @@ Some settings only take effect on restart. Those are marked **restart**.
13
13
  - [Storage](#storage)
14
14
  - [Engines](#engines)
15
15
  - [Creating torrents](#creating-torrents)
16
+ - [Offering the archive file itself](#offering-the-archive-file-itself)
16
17
  - [Trackers](#trackers)
17
18
  - [Feeds](#feeds)
18
19
  - [Watched folders](#watched-folders)
@@ -312,6 +313,77 @@ idled. Use **Pause all** for the other half.
312
313
  It lives in the configuration rather than in memory, so a node taken out of
313
314
  rotation stays out across the restart you were probably about to do.
314
315
 
316
+ ## Offering the archive file itself
317
+
318
+ | setting | default | |
319
+ | ---------------- | ------- | ----------------------------------------------------- |
320
+ | `serveArchive` | `false` | answer `/archives/<infohash>/archive.pmtiles` |
321
+ | `selfWebSeed` | `false` | publish this node as a web seed for archives it holds |
322
+ | `publicDownload` | `false` | offer them as downloads on the public catalogue page |
323
+
324
+ Three switches rather than one, because they are three different exposures and a
325
+ node can reasonably want any of them without the others.
326
+
327
+ ### `serveArchive`
328
+
329
+ Off by default. This is the archive as a file, by byte range — which is what
330
+ every PMTiles reader actually wants, and what makes this node usable as an origin
331
+ without a copy of the file somewhere else.
332
+
333
+ It is off by default because it is the only thing a node publishes that is not
334
+ either small or metered by the request. TileJSON, a `.torrent`, a feed: kilobytes.
335
+ A tile: one tile. This is up to 700 GiB to anyone who knows an infohash, so
336
+ turning a node on has never meant offering its disk to strangers, and still does
337
+ not.
338
+
339
+ With it on, both range endpoints answer:
340
+
341
+ ```
342
+ GET /archives/<infohash>/archive.pmtiles one build, immutable, cached for a year
343
+ GET /latest/<category>/archive.pmtiles whichever is current, with an ETag
344
+ ```
345
+
346
+ With it off, both answer `403`.
347
+
348
+ ### `selfWebSeed`
349
+
350
+ Writes this node's own `archive.pmtiles` URL into the torrent's `url-list`, so
351
+ every peer holding the torrent fetches from here over HTTP. A web seed is the
352
+ difference between a cold tile taking tens of seconds and taking under one, and
353
+ it is what makes a brand-new archive usable before it has any peers at all.
354
+
355
+ It is also an open invitation, which is why it is separate: a seed URL is
356
+ followed by everyone who holds the torrent, not only by people who came to this
357
+ node. Turning it off takes the URL back out of the `.torrent` and the magnet —
358
+ but peers already holding either keep trying it until they refresh, so this
359
+ withdraws an advertisement rather than closing a door.
360
+
361
+ The node has to know what it is called. Set [`publicUrl`](#publicurl), or turn
362
+ the switch on from the console, where the request itself names the node. Without
363
+ either, the setting is refused rather than guessed at: a guessed web seed URL is
364
+ published and then followed.
365
+
366
+ ### `publicDownload`
367
+
368
+ Adds a **download** link to the public catalogue page. Separate from
369
+ `serveArchive` because serving a file to a reader that was handed the URL and
370
+ advertising it to every visitor are different decisions — the endpoint can exist
371
+ for a style or a peer without being put in front of a browser.
372
+
373
+ Both `selfWebSeed` and `publicDownload` are read as off wherever `serveArchive`
374
+ is off, whatever the file says. A web seed URL that answers `403` is worse than no
375
+ web seed, because a client spends its retries on it, and a download link that
376
+ `403`s is worse than no link.
377
+
378
+ ### Per archive, per folder, per source
379
+
380
+ The same three-level rule as [`md5`](#md5). The node's setting is the default; a
381
+ [watched folder](#watched-folders) or a [scheduled source](#scheduled-sources)
382
+ may carry its own; and any individual archive can be switched in the console,
383
+ under **HTTP sources** in its details. An archive that says nothing goes on
384
+ following the node, so changing the node's answer reaches every archive that
385
+ never had one of its own.
386
+
315
387
  ## Trackers
316
388
 
317
389
  `trackers` is baked into every torrent this node creates. It defaults to the
@@ -372,7 +444,8 @@ their own name. Archives with no category are excluded whenever this is set.
372
444
  ## Watched folders
373
445
 
374
446
  `watch` is a list of `{ path, category, match, webSeedBase, publishDir, sparse,
375
- latestLink, latestLinkType, keep, keepDays, md5 }`.
447
+ latestLink, latestLinkType, keep, keepDays, md5, serveArchive, selfWebSeed,
448
+ publicDownload }`.
376
449
 
377
450
  | field | |
378
451
  | ------------------- | ------------------------------------------------------------------------------------- |
@@ -383,6 +456,9 @@ latestLink, latestLinkType, keep, keepDays, md5 }`.
383
456
  | `latestLinkType` | `'symbolic'` (default) or `'hard'` |
384
457
  | `keep` / `keepDays` | retire what the folder has outgrown |
385
458
  | `md5` | overrides the node's [`md5`](#md5) for this folder alone |
459
+ | `serveArchive` | overrides [`serveArchive`](#servearchive) for archives from this folder |
460
+ | `selfWebSeed` | overrides [`selfWebSeed`](#selfwebseed) for them |
461
+ | `publicDownload` | overrides [`publicDownload`](#publicdownload) for them |
386
462
 
387
463
  `publishDir` and `webSeedBase` together give every imported archive a working web
388
464
  seed, which is what makes a brand-new archive usable before any peer has a copy of
@@ -420,14 +496,15 @@ remove the newest build however old it gets. See
420
496
  `sources` is a list of upstreams that publish a new archive on a schedule. Each
421
497
  entry gives either a `url` template or an `index` directory:
422
498
 
423
- | field | |
424
- | ------------ | ------------------------------------------------------------------------------------------------------------- |
425
- | `url` | a template with the date in it — `{YYYYMMDD}`, `{YYYY-MM-DD}`, `{YYYY}`, `{MM}`, `{DD}` — expanded and probed |
426
- | `index` | a directory URL, listed and filtered |
427
- | `newest` | how many listed files an index source will consider. Defaults to 1 |
428
- | `at` | a time of day in UTC, or a list of them — `"03:30"` |
429
- | `everyHours` | an interval instead |
430
- | `md5` | overrides the node's [`md5`](#md5) for this source alone |
499
+ | field | |
500
+ | ----------------------------------------------- | ------------------------------------------------------------------------------------------------------------- |
501
+ | `url` | a template with the date in it — `{YYYYMMDD}`, `{YYYY-MM-DD}`, `{YYYY}`, `{MM}`, `{DD}` — expanded and probed |
502
+ | `index` | a directory URL, listed and filtered |
503
+ | `newest` | how many listed files an index source will consider. Defaults to 1 |
504
+ | `at` | a time of day in UTC, or a list of them — `"03:30"` |
505
+ | `everyHours` | an interval instead |
506
+ | `md5` | overrides the node's [`md5`](#md5) for this source alone |
507
+ | `serveArchive`, `selfWebSeed`, `publicDownload` | override what this node offers of the archives this source fetches |
431
508
 
432
509
  Prefer a template where the naming is predictable: it asks a direct question,
433
510
  gets a direct answer, and needs the upstream to publish no listing at all.
package/docs/haproxy.md CHANGED
@@ -220,7 +220,18 @@ says so, which is worse than having no check at all.
220
220
  **Cache tiles by infohash aggressively.** `/archives/<infohash>/…` is immutable
221
221
  by construction — an infohash names those bytes and no others — and is served
222
222
  with `max-age=31536000, immutable`. `/latest/<category>/…` is the opposite: it
223
- moves on every build and is served with `max-age=300`.
223
+ moves on every build, and is served with `max-age=60, must-revalidate` and an
224
+ ETag naming the build it resolved to.
225
+
226
+ **Do not strip or rewrite the ETag on either.** It is the infohash, which is
227
+ how a PMTiles reader notices that the archive moved underneath a read already
228
+ in progress — a read that spans minutes, because the reader fetches a header,
229
+ then directories, then tiles. A proxy that drops the tag leaves the reader
230
+ comparing against nothing, and it will assemble one file out of two builds
231
+ without an error anywhere. Compression is the usual culprit: a proxy that
232
+ gzips a response is required to weaken the tag, and a weak tag is one the
233
+ reader discards. Archives are served as `application/octet-stream` and should
234
+ not be compressed at all.
224
235
 
225
236
  WebRTC does not pass through Cloudflare, and neither does BitTorrent. Browser
226
237
  peers reach the node over ICE, and a `wss://` tracker is the only part of that
@@ -112,6 +112,7 @@ category is already the grouping, so it is what "latest" is asked of:
112
112
 
113
113
  ```
114
114
  GET /latest/{category}/tiles.json TileJSON for the newest in that category
115
+ GET /latest/{category}/archive.pmtiles the newest build itself, by byte range
115
116
  GET /latest/{category}/archive.torrent 302 to that build's .torrent
116
117
  GET /latest/{category}/magnet its magnet URI
117
118
  GET /latest/{category}.xml a feed holding only the current build
@@ -120,6 +121,12 @@ GET /latest/{category}.xml a feed holding only the current build
120
121
  Point a style at `/latest/basemaps/tiles.json` and it keeps working across every
121
122
  rebuild, with no edit.
122
123
 
124
+ `archive.pmtiles` is the one of these that has to be asked for. It is off until
125
+ [`serveArchive`](configuration.md#servearchive) is set, on the node or on the
126
+ archive, because it is the only thing here that is neither small nor metered by
127
+ the request — everything else on this page is kilobytes or one tile, and this is
128
+ the whole file.
129
+
123
130
  **The tiles it names are still infohash URLs.** That is the whole point of the
124
131
  layering: this document is the only mutable thing in the system, and everything
125
132
  it refers to stays content-addressed and cached for a year. Pointing the tile
@@ -127,8 +134,7 @@ template at `/latest/` instead would make every tile a moving target and throw
127
134
  that away — a client would have no way to know whether two tiles came from the
128
135
  same build.
129
136
 
130
- So it is cached for five minutes rather than a year, and carries a `latest`
131
- block naming what it resolved to:
137
+ So it carries a `latest` block naming what it resolved to:
132
138
 
133
139
  ```json
134
140
  {
@@ -149,6 +155,45 @@ following along to the next one.
149
155
  Categories that are not published are not resolvable here either — `/latest/`
150
156
  answers 404 for them exactly as the feeds do.
151
157
 
158
+ ### How a client knows the build moved
159
+
160
+ Every one of these carries an `ETag`, and the tag is the infohash of the archive
161
+ it resolved to:
162
+
163
+ ```
164
+ ETag: "913d671f3a28c5b8d605e28cf6bf01e293d36e86"
165
+ Cache-Control: public, max-age=60, must-revalidate
166
+ ```
167
+
168
+ A short TTL on its own is a guess. At five minutes, every client and every proxy
169
+ in front of one serves the previous build for up to five minutes after a rollover
170
+ and not one of them can tell it is doing so. The infohash is the honest answer:
171
+ it changes exactly when the archive changes, never otherwise, and it is the same
172
+ value on every node in the swarm — so two nodes behind a load balancer agree
173
+ about what is current rather than each inventing a tag from a body hash or an
174
+ mtime.
175
+
176
+ For `/latest/{category}/archive.pmtiles` this is not a nicety. A PMTiles reader
177
+ does not fetch a file; it fetches a header, then a root directory, then leaf
178
+ directories, then tiles, over minutes or hours. If a rebuild lands partway
179
+ through, offsets read from the old build address bytes in the new one — which
180
+ does not fail loudly, it decodes as the wrong tile or as nothing. So `If-Range`
181
+ is honoured: a range conditioned on a build that is no longer current is refused
182
+ _as a range_ and answered in full. The official PMTiles JavaScript reader closes
183
+ the loop from the other side, comparing the ETag of every response against the
184
+ one it saw first and re-reading the header when they differ.
185
+
186
+ Two consequences worth knowing about:
187
+
188
+ - **The tag must survive the proxy.** A proxy that strips it leaves the reader
189
+ comparing against nothing. One that gzips the response is required to weaken
190
+ it, and the reader discards any tag beginning with `W/`. Archives go out as
191
+ `application/octet-stream` and should not be compressed.
192
+ - **A browser must be allowed to read it.** `ETag` is not among the handful of
193
+ response headers exposed to cross-origin JavaScript by default, so these
194
+ routes send `Access-Control-Expose-Headers`. Without it the reader sees
195
+ `null`, the comparison never fires, and it splices two builds in silence.
196
+
152
197
  ## Where the bytes come from
153
198
 
154
199
  This is the part that makes serving tiles from a torrent client worth doing at
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "pmtiles-swarm",
3
- "version": "0.48.0",
3
+ "version": "0.50.0",
4
4
  "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
5
5
  "type": "module",
6
6
  "main": "src/index.js",