pmtiles-swarm 0.29.0 → 0.30.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +33 -0
- package/README.md +6 -4
- package/docs/architecture-diagram.md +10 -7
- package/docs/configuration.md +46 -2
- package/docs/engines.md +32 -1
- package/docs/publishing.md +6 -1
- package/docs/running-as-a-service.md +9 -0
- package/docs/serving-tiles.md +58 -20
- package/package.json +1 -1
- package/src/api.js +51 -7
- package/src/web/index.html +45 -26
- package/src/web/public.html +3 -3
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,39 @@
|
|
|
7
7
|
### 🐞 Bug fixes
|
|
8
8
|
- _...Add new stuff here..._
|
|
9
9
|
|
|
10
|
+
## 0.30.0
|
|
11
|
+
### ✨ Features and improvements
|
|
12
|
+
- **The style URL now carries the `.torrent` URL as well as the magnet.** Piece hashes reach a
|
|
13
|
+
browser only from a peer, over BEP 9 — there is no other route to them — so a magnet alone
|
|
14
|
+
leaves a page waiting on a tracker connection and a WebRTC handshake before it can read a byte,
|
|
15
|
+
and never gets there at all on a network that blocks the trackers. One ordinary HTTPS request for
|
|
16
|
+
the metainfo removes that dependency, and saves a conventional client the same round trip.
|
|
17
|
+
|
|
18
|
+
Both handles now ride in the fragment of the `styleUrl` on `/api/categories` and `/latest/`, and
|
|
19
|
+
of what the console's **Copy TileJSON URL + swarm** button produces. A fragment is still never
|
|
20
|
+
sent in a request, so an ordinary client fetches the TileJSON and ignores all of it.
|
|
21
|
+
|
|
22
|
+
For a category the `.torrent` handle is category-scoped too — `/latest/<category>/archive.torrent`
|
|
23
|
+
redirects to whatever build is current — so unlike a plain magnet it does not go stale on the next
|
|
24
|
+
build. That was previously true only where the node publishes a BEP 46 key.
|
|
25
|
+
|
|
26
|
+
**This is a format change.** The fragment used to be a bare `#magnet:?…`; it is now
|
|
27
|
+
`#torrent=…&magnet=…` with both values percent-encoded, because `&` separates them and a magnet
|
|
28
|
+
is full of them. A reader that took the whole fragment for a magnet must read `magnet=` out of it
|
|
29
|
+
— `URLSearchParams` does it in one call. The bare form was not kept for the single-handle case: a
|
|
30
|
+
fragment whose shape depends on what happened to be available means every reader has to handle
|
|
31
|
+
both anyway.
|
|
32
|
+
|
|
33
|
+
- **The Added column shows the same date and time on every row.** It used to shorten today's rows
|
|
34
|
+
to a time and older ones to a date, which reads as two different quantities in one column and
|
|
35
|
+
makes the eye stop to work out which it is looking at. Seconds are dropped rather than the date,
|
|
36
|
+
since nothing here is sorted finely enough for them to matter; hovering still gives them.
|
|
37
|
+
|
|
38
|
+
- _...Add new stuff here..._
|
|
39
|
+
|
|
40
|
+
### 🐞 Bug fixes
|
|
41
|
+
- _...Add new stuff here..._
|
|
42
|
+
|
|
10
43
|
## 0.29.0
|
|
11
44
|
### ✨ Features and improvements
|
|
12
45
|
- **A connection indicator in the header, for whether the swarm can reach this node.** A node
|
package/README.md
CHANGED
|
@@ -60,9 +60,11 @@ day are followed by a dated URL template or by reading a directory listing, on a
|
|
|
60
60
|
per source.
|
|
61
61
|
|
|
62
62
|
**Has a console.** Everything above is done from a web UI in the shape of a torrent client:
|
|
63
|
-
archives with progress, ratio
|
|
64
|
-
|
|
65
|
-
|
|
63
|
+
archives with progress, ratio, expiry and when each was added, searchable and sortable by any of
|
|
64
|
+
them; tabbed detail with trackers, peers, HTTP sources and content; a chart of what the swarm has
|
|
65
|
+
been sending and receiving, kept in SQLite so it survives a restart; an indicator saying whether
|
|
66
|
+
peers can reach this node at all; and a settings screen covering monitored folders, watched web
|
|
67
|
+
locations, remote nodes, save locations, access tokens and the external-program hooks.
|
|
66
68
|
|
|
67
69
|
**Publishes RSS.** `/feed.xml`, and `/feed/<category>.xml` per category. Plain RSS 2.0 with
|
|
68
70
|
torrent enclosures, so **qBittorrent's built-in RSS auto-downloader can subscribe today** with no
|
|
@@ -739,7 +741,7 @@ which the endpoint answers 501.
|
|
|
739
741
|
|
|
740
742
|
| Method | Path | Purpose |
|
|
741
743
|
| --------------------- | ----------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
742
|
-
| `GET` | `/api/status` | Engine health, counts, watched folders, save locations and
|
|
744
|
+
| `GET` | `/api/status` | Engine health, counts, watched folders, save locations, free space, and whether the swarm can reach this node |
|
|
743
745
|
| `GET` | `/api/torrents` | Catalog joined with live swarm state and what is left of each seeding limit |
|
|
744
746
|
| `POST` | `/api/torrents` | Add via `{path}`, `{url}`, `{magnet}`, `{torrentUrl}`, or a raw `.torrent` body |
|
|
745
747
|
| `DELETE` | `/api/torrents/:infoHash` | Remove (`?deleteData=true` to delete data too) |
|
|
@@ -255,19 +255,22 @@ remember where it had got to.
|
|
|
255
255
|
|
|
256
256
|
### Bootstrapping without the server
|
|
257
257
|
|
|
258
|
-
A torrent-aware client still has to _learn_
|
|
259
|
-
it does, the swarm — the part that depends on no server — is unreachable
|
|
260
|
-
precisely when the server is down. The fix is that
|
|
258
|
+
A torrent-aware client still has to _learn_ where to join from somewhere, and
|
|
259
|
+
until it does, the swarm — the part that depends on no server — is unreachable
|
|
260
|
+
precisely when the server is down. The fix is that two handles travel in the
|
|
261
261
|
**fragment** of the TileJSON URL a style already carries:
|
|
262
262
|
|
|
263
263
|
```
|
|
264
|
-
https
|
|
264
|
+
https://…/latest/openmaptiles/tiles.json#torrent=<the .torrent URL>&magnet=<magnet:?xs=urn:btpk:…>
|
|
265
265
|
```
|
|
266
266
|
|
|
267
267
|
A fragment is never sent in an HTTP request, so ordinary clients fetch the
|
|
268
|
-
TileJSON and ignore it while a swarm-aware one
|
|
269
|
-
|
|
270
|
-
|
|
268
|
+
TileJSON and ignore it while a swarm-aware one joins before making any call. The
|
|
269
|
+
`.torrent` is the fast handle — a browser can get piece hashes no other way than
|
|
270
|
+
from a peer, so fetching the metainfo over HTTPS is what lets a page start
|
|
271
|
+
without waiting on one — and the magnet is the one that needs no server at all.
|
|
272
|
+
With a BEP 46 key in it (`xs=urn:btpk:`) rather than an infohash, neither goes
|
|
273
|
+
stale on the next build. See
|
|
271
274
|
[serving-tiles.md](serving-tiles.md#a-fragment-that-survives-a-rebuild).
|
|
272
275
|
|
|
273
276
|
---
|
package/docs/configuration.md
CHANGED
|
@@ -566,6 +566,28 @@ Note what the client address means behind a proxy: without `X-Forwarded-For` it
|
|
|
566
566
|
the proxy's own address, which still answers whether a request arrived directly or
|
|
567
567
|
through it, but not who sent it.
|
|
568
568
|
|
|
569
|
+
### `traffic`
|
|
570
|
+
|
|
571
|
+
What the swarm side has been moving, at `GET /api/traffic` and as the chart in the
|
|
572
|
+
console. Upload and download speed per archive, sampled on a timer.
|
|
573
|
+
|
|
574
|
+
| setting | default | |
|
|
575
|
+
| ----------------------- | ------- | ------------------------- |
|
|
576
|
+
| `traffic.sampleSeconds` | `15` | how finely it looks |
|
|
577
|
+
| `traffic.keepHours` | `168` | how far back it remembers |
|
|
578
|
+
|
|
579
|
+
Two knobs because they are two questions. Unlike `tileStats` this is kept in
|
|
580
|
+
SQLite, in `stats.db` beside the catalog rather than beside the configuration, and
|
|
581
|
+
the difference is deliberate: tile counters answer a question about now and are
|
|
582
|
+
cheap to rebuild by waiting, while a bandwidth history answers a question about
|
|
583
|
+
the past and cannot be rebuilt at all — restarting to pick up a new version would
|
|
584
|
+
erase exactly the week somebody wanted to look at.
|
|
585
|
+
|
|
586
|
+
A week of 15-second samples is roughly 40,000 rows per archive, a few megabytes
|
|
587
|
+
for a node carrying twenty. Both settings reload in place. See
|
|
588
|
+
[running-as-a-service.md](running-as-a-service.md#the-statistics-database) for
|
|
589
|
+
sizing and for what a locked-down unit file has to allow.
|
|
590
|
+
|
|
569
591
|
## Checking for changed sources
|
|
570
592
|
|
|
571
593
|
`originCheckIntervalSeconds` (default `0`, disabled) controls how often to
|
|
@@ -586,8 +608,8 @@ A changed source does not invalidate the existing torrent. See
|
|
|
586
608
|
|
|
587
609
|
| setting | default | |
|
|
588
610
|
| --------------------------- | ------- | ------------------------------------------------------ |
|
|
589
|
-
| `fetchAttempts` | `10` |
|
|
590
|
-
| `fetchRetrySeconds` | `
|
|
611
|
+
| `fetchAttempts` | `10` | consecutive failures without progress before giving up |
|
|
612
|
+
| `fetchRetrySeconds` | `30` | base delay, multiplied by the consecutive count |
|
|
591
613
|
| `resumeSaveIntervalSeconds` | `300` | how often to write resume data. `0` disables it |
|
|
592
614
|
|
|
593
615
|
A planet archive is hours of transfer, and a connection that drops partway is
|
|
@@ -598,6 +620,28 @@ an ETag or Last-Modified to prove the file has not changed underneath. Where it
|
|
|
598
620
|
does not, the download restarts, because splicing two builds together produces a
|
|
599
621
|
torrent for bytes that never existed.
|
|
600
622
|
|
|
623
|
+
**The budget counts consecutive failures that moved nothing, not failures.** An
|
|
624
|
+
attempt that transferred bytes proves the source and the route are alive, so it
|
|
625
|
+
resets the count. Counting every failure made the budget a property of the whole
|
|
626
|
+
download rather than of the trouble it is in, which for a large archive are not
|
|
627
|
+
the same thing: a 700 GiB transfer reached 226 GB across six separate stalls and
|
|
628
|
+
then spent its last four attempts on one bad minute, because a quarter of a
|
|
629
|
+
terabyte of progress counted for nothing. Progress is measured against the
|
|
630
|
+
high-water mark rather than the previous attempt, since an attempt can fail
|
|
631
|
+
having written less than was already on disk.
|
|
632
|
+
|
|
633
|
+
The delay grows with the consecutive count — at the default, 30s then 60s then
|
|
634
|
+
90s, so ten of them span something over twenty minutes rather than the
|
|
635
|
+
forty-five seconds a flat delay gave. Because progress resets the count, an
|
|
636
|
+
unlucky download can go round more times than `fetchAttempts` names; that is
|
|
637
|
+
intended, and bounded by a ceiling of ten times the budget so a source dribbling
|
|
638
|
+
a few bytes before dropping cannot retry for ever.
|
|
639
|
+
|
|
640
|
+
**Giving up keeps the partial file.** The staging path is derived from the URL,
|
|
641
|
+
so re-adding the same URL in the console resumes from where it stopped instead of
|
|
642
|
+
starting again, and the error says how many bytes are there to resume from. Only
|
|
643
|
+
cancelling deletes it.
|
|
644
|
+
|
|
601
645
|
Resume data is what lets a restart skip re-hashing the store — on an 800 GB
|
|
602
646
|
archive, the difference between instant and half an hour. A clean stop always
|
|
603
647
|
writes it; `resumeSaveIntervalSeconds` is for the stops that are not clean, where
|
package/docs/engines.md
CHANGED
|
@@ -223,6 +223,35 @@ unforwarded NAT can never connect to each other, so the ones that need you most
|
|
|
223
223
|
cannot serve. UPnP and NAT-PMP are on by default in both engines and will often open it for you;
|
|
224
224
|
a router with either disabled will not say so.
|
|
225
225
|
|
|
226
|
+
### The console says whether it worked
|
|
227
|
+
|
|
228
|
+
Nothing about a node's own traffic reveals that half the swarm cannot reach it. It dials out, its
|
|
229
|
+
transfers work, and it looks healthy — the cost is invisible and permanent. So the console header
|
|
230
|
+
carries an indicator, also in `GET /api/status` as `reachability`:
|
|
231
|
+
|
|
232
|
+
| Colour | State | Means |
|
|
233
|
+
| ------ | ---------- | ------------------------------------------------------ |
|
|
234
|
+
| green | `open` | something has connected inward — the port is reachable |
|
|
235
|
+
| amber | `unproven` | listening, and nothing ever has |
|
|
236
|
+
| red | `offline` | not listening at all |
|
|
237
|
+
| hidden | `unknown` | the engine cannot answer |
|
|
238
|
+
|
|
239
|
+
libtorrent answers from `net.has_incoming_connections`; WebTorrent has no such gauge, so it is
|
|
240
|
+
assembled from the wires, each of which carries the direction it was made in. Both latch: the
|
|
241
|
+
question is whether the swarm _can_ reach this node, not whether somebody is connected right now,
|
|
242
|
+
so a reachable node that is merely quiet stays green rather than dropping to amber when its last
|
|
243
|
+
peer leaves.
|
|
244
|
+
|
|
245
|
+
Reported per engine, not blended. Two engines means two listening ports, forwarded separately, and
|
|
246
|
+
one can be reachable while the other is not; the header shows the primary and names both on hover.
|
|
247
|
+
|
|
248
|
+
**Amber is not a fault.** On a node no peer has tried, blocked and untried are the same
|
|
249
|
+
observation, and nothing available separates them — which is why it reads "no incoming yet" rather
|
|
250
|
+
than "firewalled". On a busy node it will turn green within minutes; if it does not, the port is
|
|
251
|
+
worth checking. For the same reason an engine that cannot be asked hides the indicator instead of
|
|
252
|
+
showing red: not being able to ask is not the same as being unreachable. The libtorrent side needs
|
|
253
|
+
pmtiles-torrent 0.5.0 or newer.
|
|
254
|
+
|
|
226
255
|
### Every libtorrent network setting
|
|
227
256
|
|
|
228
257
|
```json
|
|
@@ -346,7 +375,9 @@ checked and the unreadable ones are joined by magnet instead.
|
|
|
346
375
|
|
|
347
376
|
## Writing another engine
|
|
348
377
|
|
|
349
|
-
Implement `connect`, `add`, `remove`, `list`, `get`, `destroy`, and optionally `peers
|
|
378
|
+
Implement `connect`, `add`, `remove`, `list`, `get`, `destroy`, and optionally `peers` and
|
|
379
|
+
`reachability`. Anything optional that is missing is simply not offered — an engine without
|
|
380
|
+
`reachability` hides the indicator rather than reporting a node unreachable.
|
|
350
381
|
The interface is deliberately small; see
|
|
351
382
|
[`src/engines/types.js`](../src/engines/types.js) for the contract and
|
|
352
383
|
[`src/engines/webtorrent.js`](../src/engines/webtorrent.js) for the shortest example.
|
package/docs/publishing.md
CHANGED
|
@@ -231,10 +231,15 @@ rather than exceptional. Each attempt continues from the bytes already on disk w
|
|
|
231
231
|
HTTP range request, so a drop costs the retry delay instead of everything transferred so
|
|
232
232
|
far.
|
|
233
233
|
|
|
234
|
+
The budget counts _consecutive_ failures that moved nothing: an attempt that transferred
|
|
235
|
+
bytes proves the source and the route are alive, so it resets the count. The delay grows
|
|
236
|
+
with that count — 30s, 60s, 90s — so ten of them span an outage rather than a moment. When
|
|
237
|
+
it does give up the partial file is kept, and re-adding the same URL resumes from it.
|
|
238
|
+
|
|
234
239
|
```json
|
|
235
240
|
{
|
|
236
241
|
"fetchAttempts": 10,
|
|
237
|
-
"fetchRetrySeconds":
|
|
242
|
+
"fetchRetrySeconds": 30
|
|
238
243
|
}
|
|
239
244
|
```
|
|
240
245
|
|
|
@@ -566,6 +566,15 @@ own, and not `torrentPort`. `6883` sits clear of both.
|
|
|
566
566
|
Note that each engine runs its own DHT as well, so a node with both engines has
|
|
567
567
|
three UDP participants. Only this one is yours to place.
|
|
568
568
|
|
|
569
|
+
### Checking the forward actually happened
|
|
570
|
+
|
|
571
|
+
A forward that did not take is invisible from here: the node still dials out, its
|
|
572
|
+
transfers still work, and nothing in its own numbers says half the swarm can
|
|
573
|
+
never open a connection to it. The console header answers it — green once
|
|
574
|
+
something has connected inward, amber while nothing has, red if the engine is not
|
|
575
|
+
listening at all. Give a new node a few minutes of seeding before reading it;
|
|
576
|
+
amber on a node no peer has tried yet means untried, not blocked.
|
|
577
|
+
|
|
569
578
|
## Updating
|
|
570
579
|
|
|
571
580
|
```sh
|
package/docs/serving-tiles.md
CHANGED
|
@@ -192,20 +192,20 @@ Both peer-to-peer engines reuse the client already seeding the archive rather
|
|
|
192
192
|
than starting a second one. One peer pool, one port, one DHT node — and for
|
|
193
193
|
libtorrent, one sidecar process.
|
|
194
194
|
|
|
195
|
-
## Carrying the
|
|
195
|
+
## Carrying the swarm in the URL fragment
|
|
196
196
|
|
|
197
197
|
The `torrent` block below solves the problem _after_ the TileJSON has been
|
|
198
198
|
fetched. It does not solve the one before it: a torrent-aware client that cannot
|
|
199
199
|
reach this server has nothing to work with, so the swarm — the part that does
|
|
200
200
|
not depend on any server — is unreachable precisely when the server is down.
|
|
201
201
|
|
|
202
|
-
The fix is to put the
|
|
202
|
+
The fix is to put the ways into the swarm in the URL **fragment**:
|
|
203
203
|
|
|
204
204
|
```json
|
|
205
205
|
"sources": {
|
|
206
206
|
"openmaptiles": {
|
|
207
207
|
"type": "vector",
|
|
208
|
-
"url": "https://swarm.example.org/latest/openmaptiles/tiles.json#magnet
|
|
208
|
+
"url": "https://swarm.example.org/latest/openmaptiles/tiles.json#torrent=https%3A%2F%2Fswarm.example.org%2Flatest%2Fopenmaptiles%2Farchive.torrent&magnet=magnet%3A%3Fxt%3Durn%3Abtih%3A4813a0e6…"
|
|
209
209
|
}
|
|
210
210
|
}
|
|
211
211
|
```
|
|
@@ -213,34 +213,72 @@ The fix is to put the magnet in the URL **fragment**:
|
|
|
213
213
|
A fragment is never sent in an HTTP request, so the same string works
|
|
214
214
|
everywhere:
|
|
215
215
|
|
|
216
|
-
| Client | What happens
|
|
217
|
-
| --------------------------------- |
|
|
218
|
-
| maplibre-gl-js, Leaflet, anything | fetches the TileJSON, ignores the fragment
|
|
219
|
-
| maplibre-native without a plugin | the same
|
|
220
|
-
| torrent-aware |
|
|
216
|
+
| Client | What happens |
|
|
217
|
+
| --------------------------------- | ------------------------------------------------------------------- |
|
|
218
|
+
| maplibre-gl-js, Leaflet, anything | fetches the TileJSON, ignores the fragment |
|
|
219
|
+
| maplibre-native without a plugin | the same |
|
|
220
|
+
| torrent-aware | joins **before any network call**, and still can if the fetch fails |
|
|
221
221
|
|
|
222
|
-
The console's **Copy TileJSON URL +
|
|
222
|
+
The console's **Copy TileJSON URL + swarm** button produces exactly this, and so
|
|
223
|
+
does the `styleUrl` on every row of `/api/categories` and `/latest/`.
|
|
223
224
|
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
225
|
+
### Two handles, and why both
|
|
226
|
+
|
|
227
|
+
They are not a ladder from worse to better. They fail in different directions.
|
|
228
|
+
|
|
229
|
+
| Handle | Needs | Gets you |
|
|
230
|
+
| ---------- | --------------- | --------------------------------------------------- |
|
|
231
|
+
| `torrent=` | this host | the metainfo itself, over one ordinary HTTP request |
|
|
232
|
+
| `magnet=` | a peer, no host | everything, eventually, from the swarm alone |
|
|
233
|
+
|
|
234
|
+
`torrent=` is not redundant with the URL it is attached to. **Piece hashes reach
|
|
235
|
+
a browser only from a peer, over BEP 9** — there is no other route to them — so
|
|
236
|
+
a magnet alone leaves a page waiting on a tracker connection and a WebRTC
|
|
237
|
+
handshake before it can read a byte, and never gets there at all on a network
|
|
238
|
+
that blocks the trackers. Fetching the metainfo over HTTPS removes that
|
|
239
|
+
dependency entirely. It also saves a conventional client the same round trip.
|
|
240
|
+
|
|
241
|
+
`magnet=` is the handle that needs nothing of this node, which is the case the
|
|
242
|
+
fragment exists for in the first place. A client with a DHT should prefer it if
|
|
243
|
+
this host is unreachable; a browser, which has neither DHT nor UDP, should reach
|
|
244
|
+
for the `.torrent` first.
|
|
245
|
+
|
|
246
|
+
For a `/latest/<category>/` URL the `.torrent` handle points at the category too,
|
|
247
|
+
so it redirects to whatever build is current and does not go stale — see the
|
|
248
|
+
caveat below, which it answers for the half of the fragment that a BEP 46 magnet
|
|
249
|
+
otherwise has to.
|
|
250
|
+
|
|
251
|
+
Both are percent-encoded, since `&` separates them and a magnet is full of them.
|
|
252
|
+
`URLSearchParams` reads the fragment; `get('magnet')` gives back the magnet
|
|
253
|
+
exactly.
|
|
254
|
+
|
|
255
|
+
> **This changed in 0.30.0.** The fragment used to be a bare `#magnet:?…` with
|
|
256
|
+
> nothing else in it. A reader that took the whole fragment for a magnet needs to
|
|
257
|
+
> read `magnet=` out of it now. The bare form was not kept for the single-handle
|
|
258
|
+
> case, because a fragment whose shape depends on what happened to be available
|
|
259
|
+
> means every reader has to handle both anyway.
|
|
227
260
|
|
|
228
261
|
### What a client should do with it
|
|
229
262
|
|
|
230
|
-
|
|
263
|
+
Four paths, each a fallback of the one above:
|
|
231
264
|
|
|
232
265
|
1. **The TileJSON URL.** One request, the full document including
|
|
233
266
|
`vector_layers`. Fastest, and what an ordinary client does anyway.
|
|
234
|
-
2. **The `
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
3. **The
|
|
267
|
+
2. **The `torrent=` metainfo.** One request, and it yields piece hashes, the
|
|
268
|
+
file name and the web seeds — enough to join and to range-read without any
|
|
269
|
+
peer having spoken yet. Works when the TileJSON route is down but this host
|
|
270
|
+
is up, and it is the only one of these a browser can use to start quickly.
|
|
271
|
+
3. **The `ws=` web seed** inside the magnet. Two HTTP range requests — the header
|
|
272
|
+
and root directory near the start of the archive, the JSON metadata at the far
|
|
273
|
+
end — and the TileJSON can be derived from them. Works when this API is down
|
|
274
|
+
but the file is still on a web server.
|
|
275
|
+
4. **The swarm.** No HTTP at all. Slowest from cold, because BEP 9 has to
|
|
239
276
|
deliver the metainfo first, and the only one that survives the server
|
|
240
277
|
disappearing entirely.
|
|
241
278
|
|
|
242
|
-
Everything those need is in the
|
|
243
|
-
the file, `tr` finds peers, `ws` gives the HTTP fallback
|
|
279
|
+
Everything those need is in the two handles: `xt` identifies the archive, `dn`
|
|
280
|
+
names the file, `tr` finds peers, `ws` gives the HTTP fallback, and `torrent=`
|
|
281
|
+
gives the metainfo without asking the swarm for it.
|
|
244
282
|
|
|
245
283
|
### One caveat on `/latest/` URLs
|
|
246
284
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pmtiles-swarm",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.30.0",
|
|
4
4
|
"description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "src/index.js",
|
package/src/api.js
CHANGED
|
@@ -66,7 +66,43 @@ function route(handler) {
|
|
|
66
66
|
* @returns {import('express').Express} - The configured app.
|
|
67
67
|
*/
|
|
68
68
|
/**
|
|
69
|
-
*
|
|
69
|
+
* Attaches whatever swarm handles exist to a URL, in its fragment.
|
|
70
|
+
*
|
|
71
|
+
* Two handles, and they are not a ladder from worse to better — they fail in
|
|
72
|
+
* different directions:
|
|
73
|
+
*
|
|
74
|
+
* - `torrent=` is the metainfo, and needs this host. Not redundant with the URL
|
|
75
|
+
* it is attached to: piece hashes reach a browser only from a peer over
|
|
76
|
+
* BEP 9, so this is the one handle that works from a page on a network that
|
|
77
|
+
* blocks the trackers, and the one that saves every client the metadata round
|
|
78
|
+
* trip that a magnet costs.
|
|
79
|
+
* - `magnet=` needs no host at all, and needs a peer. The better handle for a
|
|
80
|
+
* client with a DHT, the weaker one for a browser, which has neither DHT nor
|
|
81
|
+
* UDP.
|
|
82
|
+
*
|
|
83
|
+
* Ordered by what a client should reach for first when it can fetch.
|
|
84
|
+
*
|
|
85
|
+
* Both are percent-encoded, which is a change from the bare `#magnet:?…` this
|
|
86
|
+
* used to emit: `&` separates the handles now, and a magnet is full of them.
|
|
87
|
+
* That does break a reader that took the whole fragment for a magnet, and the
|
|
88
|
+
* alternative -- keeping the bare form when there is only one handle -- is
|
|
89
|
+
* worse, because it makes the shape of the fragment depend on what happened to
|
|
90
|
+
* be available, so every reader has to handle both anyway. One shape,
|
|
91
|
+
* `URLSearchParams` reads it, and a magnet survives the round trip exactly.
|
|
92
|
+
*
|
|
93
|
+
* @param {string} url - The URL a style points at.
|
|
94
|
+
* @param {object} handles - `{torrent, magnet}`, either of which may be absent.
|
|
95
|
+
* @returns {string} - The URL, with a fragment if there is anything to put in one.
|
|
96
|
+
*/
|
|
97
|
+
function withSwarmHandles(url, { torrent, magnet }) {
|
|
98
|
+
const parts = [];
|
|
99
|
+
if (torrent) parts.push(`torrent=${encodeURIComponent(torrent)}`);
|
|
100
|
+
if (magnet) parts.push(`magnet=${encodeURIComponent(magnet)}`);
|
|
101
|
+
return parts.length > 0 ? `${url}#${parts.join('&')}` : url;
|
|
102
|
+
}
|
|
103
|
+
|
|
104
|
+
/**
|
|
105
|
+
* A TileJSON URL carrying the ways into the swarm in its fragment.
|
|
70
106
|
* @param {string} category - Which category.
|
|
71
107
|
* @param {object} newest - Its newest entry.
|
|
72
108
|
* @param {string} base - Public base URL.
|
|
@@ -90,7 +126,14 @@ function styleUrlFor(category, newest, base) {
|
|
|
90
126
|
webSeeds: newest.webSeeds,
|
|
91
127
|
})
|
|
92
128
|
: newest?.magnet;
|
|
93
|
-
|
|
129
|
+
// Category-scoped rather than the newest build's immutable URL, for the same
|
|
130
|
+
// reason the TileJSON in front of it is: it redirects to whatever is current,
|
|
131
|
+
// so a style holding this keeps working across rebuilds. The magnet beside it
|
|
132
|
+
// only manages that when the node publishes a BEP 46 key; this handle manages
|
|
133
|
+
// it always, which makes it the more durable half of the pair as well as the
|
|
134
|
+
// faster one.
|
|
135
|
+
const torrent = `${base}/latest/${category}/archive.torrent`;
|
|
136
|
+
return withSwarmHandles(url, { torrent, magnet });
|
|
94
137
|
}
|
|
95
138
|
|
|
96
139
|
export function createApp({
|
|
@@ -1879,11 +1922,12 @@ export function createApp({
|
|
|
1879
1922
|
servable,
|
|
1880
1923
|
endpoints: {
|
|
1881
1924
|
tileJson: servable ? `${base}/latest/${category}/tiles.json` : null,
|
|
1882
|
-
// The same URL with
|
|
1883
|
-
// style should carry. A fragment is never sent in
|
|
1884
|
-
// an ordinary client fetches the TileJSON and
|
|
1885
|
-
// a swarm-aware one has
|
|
1886
|
-
// and can therefore still start when this
|
|
1925
|
+
// The same URL with the ways into the swarm in the fragment,
|
|
1926
|
+
// which is what a style should carry. A fragment is never sent in
|
|
1927
|
+
// a request, so an ordinary client fetches the TileJSON and
|
|
1928
|
+
// ignores it, while a swarm-aware one has somewhere to join before
|
|
1929
|
+
// it makes any call -- and can therefore still start when this
|
|
1930
|
+
// server cannot answer.
|
|
1887
1931
|
//
|
|
1888
1932
|
// The mutable magnet where there is one, because a category is
|
|
1889
1933
|
// precisely where an infohash goes stale: it names the category
|
package/src/web/index.html
CHANGED
|
@@ -749,11 +749,14 @@
|
|
|
749
749
|
const pct = (n) => `${Math.round((n ?? 0) * 100)}%`;
|
|
750
750
|
|
|
751
751
|
/**
|
|
752
|
-
* When an archive was added
|
|
752
|
+
* When an archive was added: the same date and time on every row.
|
|
753
753
|
*
|
|
754
|
-
*
|
|
755
|
-
*
|
|
756
|
-
* a
|
|
754
|
+
* It used to shorten today's rows to a time and older ones to a date,
|
|
755
|
+
* on the theory that a column is easier to scan that way. It is not:
|
|
756
|
+
* a column of `8/13/2026` with an `08:14 AM` in the middle of it reads
|
|
757
|
+
* as two different quantities, and the eye has to stop and work out
|
|
758
|
+
* which. Seconds are dropped rather than the date, since nothing here
|
|
759
|
+
* is sorted finely enough for them to matter; the title keeps them.
|
|
757
760
|
* @param {string} value - ISO timestamp.
|
|
758
761
|
* @returns {string} - Markup for the cell.
|
|
759
762
|
*/
|
|
@@ -761,12 +764,15 @@
|
|
|
761
764
|
if (!value) return '<span class="sub">—</span>';
|
|
762
765
|
const when = new Date(value);
|
|
763
766
|
if (Number.isNaN(when.getTime())) return '<span class="sub">—</span>';
|
|
764
|
-
|
|
765
|
-
|
|
766
|
-
|
|
767
|
-
|
|
768
|
-
:
|
|
769
|
-
|
|
767
|
+
return `<span title="${escapeHtml(when.toLocaleString())}">${escapeHtml(
|
|
768
|
+
when.toLocaleString([], {
|
|
769
|
+
year: 'numeric',
|
|
770
|
+
month: 'numeric',
|
|
771
|
+
day: 'numeric',
|
|
772
|
+
hour: '2-digit',
|
|
773
|
+
minute: '2-digit',
|
|
774
|
+
}),
|
|
775
|
+
)}</span>`;
|
|
770
776
|
};
|
|
771
777
|
|
|
772
778
|
// The same three questions the public page asks, against an admin row.
|
|
@@ -1563,7 +1569,7 @@
|
|
|
1563
1569
|
<button id="copy-magnet">Copy magnet</button>
|
|
1564
1570
|
<button id="copy-hash">Copy infohash</button>
|
|
1565
1571
|
${servable ? `<button id="copy-tilejson">Copy TileJSON URL</button>
|
|
1566
|
-
<button id="copy-tilejson-swarm" title="The same URL with the magnet in the fragment. A fragment is never sent to the server, so ordinary clients fetch the TileJSON exactly as before; a torrent-aware client reads
|
|
1572
|
+
<button id="copy-tilejson-swarm" title="The same URL with the .torrent and the magnet in the fragment. A fragment is never sent to the server, so ordinary clients fetch the TileJSON exactly as before; a torrent-aware client reads them from it, and a browser can fetch the metainfo over HTTP instead of waiting on a peer for it.">Copy TileJSON URL + swarm</button>
|
|
1567
1573
|
<a href="${tileJson}" target="_blank" rel="noreferrer"><button type="button">Open TileJSON</button></a>
|
|
1568
1574
|
<a href="${base}/archives/${entry.infoHash}/preview" target="_blank" rel="noreferrer"><button type="button">${
|
|
1569
1575
|
summary.format === 'pbf' ? 'Inspect' : 'Preview'
|
|
@@ -1835,20 +1841,30 @@
|
|
|
1835
1841
|
$('copy-magnet').onclick = () => copy(entry.magnet ?? '', 'Magnet link');
|
|
1836
1842
|
if (servable) {
|
|
1837
1843
|
$('copy-tilejson').onclick = () => copy(tileJson, 'TileJSON URL');
|
|
1838
|
-
// The
|
|
1844
|
+
// The handles ride in the fragment, which is client-side only and
|
|
1839
1845
|
// never sent in the request. So the same string serves both: an
|
|
1840
1846
|
// ordinary client fetches the TileJSON and ignores the fragment, and
|
|
1841
|
-
// a torrent-aware one
|
|
1842
|
-
// — which is what lets it start when this server is down.
|
|
1847
|
+
// a torrent-aware one has somewhere to join before making any call at
|
|
1848
|
+
// all — which is what lets it start when this server is down.
|
|
1843
1849
|
//
|
|
1844
|
-
//
|
|
1845
|
-
//
|
|
1846
|
-
//
|
|
1847
|
-
|
|
1850
|
+
// The .torrent URL is there because a browser cannot obtain piece
|
|
1851
|
+
// hashes any other way: they come only from a peer over BEP 9, so a
|
|
1852
|
+
// magnet alone leaves a page waiting on a WebRTC handshake before it
|
|
1853
|
+
// can read a byte. Same fragment, same string, both handles.
|
|
1854
|
+
//
|
|
1855
|
+
// A lone magnet keeps the bare unencoded form — legal in a fragment
|
|
1856
|
+
// (RFC 3986 allows ?, &, = and : there) and readable, which matters
|
|
1857
|
+
// for something people paste into a style file by hand.
|
|
1858
|
+
$('copy-tilejson-swarm').onclick = () => {
|
|
1859
|
+
const parts = [`torrent=${encodeURIComponent(torrentUrl)}`];
|
|
1860
|
+
if (entry.magnet) parts.push(`magnet=${encodeURIComponent(entry.magnet)}`);
|
|
1848
1861
|
copy(
|
|
1849
|
-
|
|
1850
|
-
entry.magnet
|
|
1862
|
+
`${tileJson}#${parts.join('&')}`,
|
|
1863
|
+
entry.magnet
|
|
1864
|
+
? 'TileJSON URL with torrent and magnet'
|
|
1865
|
+
: 'TileJSON URL with torrent (no magnet on this archive)',
|
|
1851
1866
|
);
|
|
1867
|
+
};
|
|
1852
1868
|
}
|
|
1853
1869
|
$('copy-hash').onclick = () => copy(entry.infoHash, 'Infohash');
|
|
1854
1870
|
|
|
@@ -2572,12 +2588,15 @@
|
|
|
2572
2588
|
url.length > 96 ? `${url.slice(0, 96)}…` : url,
|
|
2573
2589
|
)}</code>${
|
|
2574
2590
|
label === 'For a style'
|
|
2575
|
-
? `<div class="sub">the
|
|
2576
|
-
never sent to the server — an ordinary
|
|
2577
|
-
TileJSON and ignores
|
|
2578
|
-
making any call${
|
|
2579
|
-
|
|
2580
|
-
|
|
2591
|
+
? `<div class="sub">the .torrent URL and the magnet ride in the
|
|
2592
|
+
fragment, which is never sent to the server — an ordinary
|
|
2593
|
+
client fetches the TileJSON and ignores them, a swarm-aware
|
|
2594
|
+
one reads them before making any call${
|
|
2595
|
+
// Matched unanchored because the magnet is
|
|
2596
|
+
// percent-encoded in the fragment now that it
|
|
2597
|
+
// shares one with the .torrent URL.
|
|
2598
|
+
url.includes('btpk')
|
|
2599
|
+
? ', and both follow the category rather than this build'
|
|
2581
2600
|
: ''
|
|
2582
2601
|
}</div>`
|
|
2583
2602
|
: ''
|
package/src/web/public.html
CHANGED
|
@@ -401,9 +401,9 @@
|
|
|
401
401
|
card.append(links);
|
|
402
402
|
|
|
403
403
|
const urls = el('div', 'urls');
|
|
404
|
-
// The one worth copying: the TileJSON URL with the
|
|
405
|
-
// fragment, which a plain client fetches over HTTP
|
|
406
|
-
// one joins directly.
|
|
404
|
+
// The one worth copying: the TileJSON URL with the .torrent URL and
|
|
405
|
+
// the magnet in its fragment, which a plain client fetches over HTTP
|
|
406
|
+
// and a swarm-aware one joins directly.
|
|
407
407
|
if (ends.styleUrl) {
|
|
408
408
|
urls.append(
|
|
409
409
|
urlRow('style URL', new URL(ends.styleUrl, location.href).href),
|