pmtiles-swarm 0.9.0 โ 0.11.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +74 -0
- package/README.md +56 -0
- package/docs/running-as-a-service.md +5 -2
- package/docs/serving-tiles.md +66 -0
- package/package.json +2 -2
- package/src/api.js +66 -1
- package/src/config.js +19 -0
- package/src/index.js +10 -0
- package/src/tile-stats.js +249 -0
- package/src/web/index.html +24 -0
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,80 @@
|
|
|
7
7
|
### ๐ Bug fixes
|
|
8
8
|
- _...Add new stuff here..._
|
|
9
9
|
|
|
10
|
+
## 0.11.0
|
|
11
|
+
### โจ Features and improvements
|
|
12
|
+
- **The magnet can travel in the TileJSON URL's fragment**, and the console will build that string
|
|
13
|
+
for you: **Copy TileJSON URL + magnet**. A fragment is never sent in an HTTP request, so one
|
|
14
|
+
string serves every client โ maplibre-gl-js, Leaflet and plain maplibre-native fetch the
|
|
15
|
+
TileJSON and ignore it, while a torrent-aware client reads the magnet **before making any network
|
|
16
|
+
call at all**.
|
|
17
|
+
|
|
18
|
+
That last part is the point. The `torrent` block inside the TileJSON only helps once the TileJSON
|
|
19
|
+
has been fetched, which leaves the swarm โ the one part that depends on no server โ
|
|
20
|
+
unreachable exactly when the server is down. With the magnet in the fragment a client can fall
|
|
21
|
+
back to the `ws=` web seed (two range requests, and the TileJSON derives from the archive's own
|
|
22
|
+
header and metadata) or to the swarm itself.
|
|
23
|
+
|
|
24
|
+
Documented in [docs/serving-tiles.md](docs/serving-tiles.md), including the caveat worth knowing:
|
|
25
|
+
on a `/latest/<category>/` URL the fragment pins the build that was current when it was copied,
|
|
26
|
+
while the URL keeps following the category, so the two can disagree after a rebuild. Survivable,
|
|
27
|
+
since the fragment is only consulted when the TileJSON cannot be fetched and an older build
|
|
28
|
+
renders where a blank map does not โ and properly fixed by a mutable `xs=urn:btpk:` magnet,
|
|
29
|
+
which needs the BEP 46 publishing that [src/mutable.js](src/mutable.js) has machinery for and
|
|
30
|
+
nothing yet calls.
|
|
31
|
+
|
|
32
|
+
## 0.10.0
|
|
33
|
+
### โจ Features and improvements
|
|
34
|
+
- **`GET /api/stats`**, which answers what a node has actually served. Until now a tile request was
|
|
35
|
+
answered and forgotten, so the most ordinary operational questions had no answer at all: which
|
|
36
|
+
archive is carrying the load, which zooms are being pulled, whether a node behind a balancer is
|
|
37
|
+
getting its share, and whether the traffic hammering it arrived directly or through the proxy.
|
|
38
|
+
|
|
39
|
+
Per-archive counters โ requests, bytes, a breakdown by zoom and by status, p50/p95 latency, and
|
|
40
|
+
a count per client address โ plus a fixed ring of the most recent requests. Both live in memory
|
|
41
|
+
and are bounded, so the cost is the same after a billion tiles as after ten. Nothing is written
|
|
42
|
+
to disk: a restart is how you reset it, and an access log would bring retention and disk
|
|
43
|
+
questions this deliberately does not have. `DELETE /api/stats` clears it, deliberately a separate
|
|
44
|
+
verb so a dashboard polling the endpoint cannot erase the history it is drawing.
|
|
45
|
+
|
|
46
|
+
The report names the node that answered, which is the point behind a load balancer โ ask each
|
|
47
|
+
one directly and the counters say how traffic is really distributed rather than how the balancer
|
|
48
|
+
believes it is. Admin-side rather than public, because it lists archives and client addresses.
|
|
49
|
+
|
|
50
|
+
Bytes are counted **as sent**, so a gzipped vector tile counts its compressed size. That is the
|
|
51
|
+
number that matters for bandwidth and it is not what the client ends up holding.
|
|
52
|
+
|
|
53
|
+
What a client address means depends on the proxy in front. Without `X-Forwarded-For` it is the
|
|
54
|
+
proxy's own address for everything arriving through it โ still enough to separate direct
|
|
55
|
+
traffic from proxied, which is usually the question being asked, but not who sent it. For real
|
|
56
|
+
client addresses the proxy has to send the header and `trustProxy` has to name it.
|
|
57
|
+
|
|
58
|
+
Configured under `tileStats`: `recent` sets how many requests to keep, `0` keeps the counters and
|
|
59
|
+
drops the ring, and `false` turns the whole thing off, after which the endpoint answers 501.
|
|
60
|
+
- **The archive detail shows what it has served**, in the console and on
|
|
61
|
+
`GET /api/torrents/<infohash>` as a `served` block. Worth reading next to `reading`: an archive
|
|
62
|
+
being read through the swarm while serving thousands of tiles is a different situation from one
|
|
63
|
+
doing neither.
|
|
64
|
+
|
|
65
|
+
## 0.9.1
|
|
66
|
+
### ๐ Bug fixes
|
|
67
|
+
- **Requires pmtiles-torrent 0.4.2, which is what actually makes a newly built archive visible.**
|
|
68
|
+
0.9.0 said the 0% was the dropped `seedOnly`. That was half of it โ the half that made the
|
|
69
|
+
archive *slow*. The half that made it *invisible* was in the sidecar: creation defaults to a
|
|
70
|
+
hybrid v1+v2 torrent, and libtorrent answers `info_hash()` for a hybrid with the truncated v2
|
|
71
|
+
hash, while the catalog, the magnet and every v1 peer use v1. The engine held the archive under
|
|
72
|
+
a name the catalog could not look up, so a correctly seeding archive was reported as one the
|
|
73
|
+
engine had never heard of, and no tile could be served from it. The dependency floor moves to
|
|
74
|
+
0.4.2 rather than being left to whatever a fresh install happens to resolve.
|
|
75
|
+
|
|
76
|
+
Expect hybrid archives to re-check once on the first start after upgrading: their resume files
|
|
77
|
+
are now looked for under the corrected name, and the old ones are not found.
|
|
78
|
+
- **The service documentation pointed its status check at a path that does not exist.** It named
|
|
79
|
+
`/opt/pmtiles-swarm/src`, while a node installed the way the rest of that document describes has
|
|
80
|
+
its executable in `/var/lib/pmtiles-swarm/node_modules/.bin`. Running the documented command
|
|
81
|
+
found no dependencies and failed on the first import โ which is the same
|
|
82
|
+
wrong-command-in-documentation problem the status command exists to end.
|
|
83
|
+
|
|
10
84
|
## 0.9.0
|
|
11
85
|
### โจ Features and improvements
|
|
12
86
|
- **`pmtiles-swarm status`**, which asks a running node what it is doing and reads the answer out
|
package/README.md
CHANGED
|
@@ -621,6 +621,61 @@ Exit status is 0 when the node answered and its engine is up, 1 when it did not
|
|
|
621
621
|
works in a deployment script. `--json` gives the raw `/api/status` and `/api/torrents` replies for
|
|
622
622
|
anything that wants to parse rather than read.
|
|
623
623
|
|
|
624
|
+
## Seeing what a node is actually serving
|
|
625
|
+
|
|
626
|
+
```sh
|
|
627
|
+
curl -s -H "authorization: Bearer $KEY" http://172.16.1.49:8091/api/stats | jq
|
|
628
|
+
```
|
|
629
|
+
|
|
630
|
+
```json
|
|
631
|
+
{
|
|
632
|
+
"node": "planetgen",
|
|
633
|
+
"since": "2026-08-12T14:02:11.004Z",
|
|
634
|
+
"requests": 18422,
|
|
635
|
+
"bytes": 743112904,
|
|
636
|
+
"archives": {
|
|
637
|
+
"4813a0e68e4b88def6d4ef3c4eabde84ffc0c068": {
|
|
638
|
+
"name": "planetiler-openmaptiles-260803.pmtiles",
|
|
639
|
+
"requests": 18422,
|
|
640
|
+
"bytes": 743112904,
|
|
641
|
+
"byZoom": { "0": 12, "7": 3311, "14": 9022 },
|
|
642
|
+
"byStatus": { "200": 18104, "204": 301, "404": 17 },
|
|
643
|
+
"clients": { "172.16.1.2": 17980, "172.16.1.41": 442 },
|
|
644
|
+
"p50ms": 3,
|
|
645
|
+
"p95ms": 41
|
|
646
|
+
}
|
|
647
|
+
},
|
|
648
|
+
"recent": [
|
|
649
|
+
{ "at": "โฆ", "ip": "172.16.1.41", "z": 14, "x": 4823, "y": 6155,
|
|
650
|
+
"status": 200, "bytes": 41221, "ms": 2 }
|
|
651
|
+
]
|
|
652
|
+
}
|
|
653
|
+
```
|
|
654
|
+
|
|
655
|
+
Counters per archive, plus a fixed ring of the most recent requests. Both live in
|
|
656
|
+
memory and are bounded, so the cost is the same after a billion tiles as after
|
|
657
|
+
ten, and a restart is how you reset it. Nothing is written to disk โ an access
|
|
658
|
+
log brings retention and disk questions this deliberately does not have.
|
|
659
|
+
|
|
660
|
+
`node` names which machine answered, which is the point behind a load balancer:
|
|
661
|
+
ask each node directly and the counters tell you how traffic is actually
|
|
662
|
+
distributed, rather than what the balancer believes.
|
|
663
|
+
|
|
664
|
+
**What the client address means depends on your proxy.** `clients` records what
|
|
665
|
+
the process can see. Without `X-Forwarded-For` that is the proxy's own address
|
|
666
|
+
for everything arriving through it โ still enough to separate direct traffic
|
|
667
|
+
from proxied, which is usually the question, but not who sent it. For real
|
|
668
|
+
client addresses, have the proxy send the header and set `trustProxy` to name
|
|
669
|
+
it; see [docs/haproxy.md](docs/haproxy.md).
|
|
670
|
+
|
|
671
|
+
Bytes are counted **as sent**, so a gzipped vector tile counts its compressed
|
|
672
|
+
size. That is the number that matters for bandwidth, and it is not what the
|
|
673
|
+
client ends up holding.
|
|
674
|
+
|
|
675
|
+
Configured under `tileStats`: `recent` sets how many requests to keep (`0` keeps
|
|
676
|
+
the counters and drops the ring), and `"tileStats": false` turns it off, after
|
|
677
|
+
which the endpoint answers 501.
|
|
678
|
+
|
|
624
679
|
## API
|
|
625
680
|
|
|
626
681
|
| Method | Path | Purpose |
|
|
@@ -655,6 +710,7 @@ anything that wants to parse rather than read.
|
|
|
655
710
|
| `POST` | `/api/torrents/:infoHash/hooks/complete` | Run the completion hook again for one archive |
|
|
656
711
|
| `GET` `POST` `DELETE` | `/api/tokens`, `/api/tokens/:id` | Mint, list and revoke access tokens |
|
|
657
712
|
| `GET` `POST` | `/api/restart` | What a restart would do, and doing it |
|
|
713
|
+
| `GET` `DELETE` | `/api/stats` | What this node has served โ per-archive counters and the last N requests; `DELETE` clears them |
|
|
658
714
|
| `GET` `PATCH` | `/api/config` | Read and change settings |
|
|
659
715
|
| `POST` | `/api/login`, `/api/logout` | Console sign-in |
|
|
660
716
|
| `GET` | `/api/session` | Who this request is, and whether a credential is needed at all |
|
|
@@ -441,10 +441,13 @@ Then check it is actually serving:
|
|
|
441
441
|
|
|
442
442
|
```sh
|
|
443
443
|
curl -fsS localhost:8090/feed.xml >/dev/null && echo "public surface ok"
|
|
444
|
-
|
|
445
|
-
--config /etc/pmtiles-swarm/swarm.config.json
|
|
444
|
+
sudo -u pmtiles-swarm -H /var/lib/pmtiles-swarm/node_modules/.bin/pmtiles-swarm \
|
|
445
|
+
status --config /etc/pmtiles-swarm/swarm.config.json
|
|
446
446
|
```
|
|
447
447
|
|
|
448
|
+
Safe to run against a live node: it asks over HTTP and takes no lock, so it is
|
|
449
|
+
not the second process the data directory would refuse.
|
|
450
|
+
|
|
448
451
|
Ask through the status command rather than with `curl`. Reaching the API by hand
|
|
449
452
|
means getting the bind address, the admin port and the credential right in one
|
|
450
453
|
go, and each of them fails in a way that looks like a broken node: a node bound
|
package/docs/serving-tiles.md
CHANGED
|
@@ -160,6 +160,72 @@ Both peer-to-peer engines reuse the client already seeding the archive rather
|
|
|
160
160
|
than starting a second one. One peer pool, one port, one DHT node โ and for
|
|
161
161
|
libtorrent, one sidecar process.
|
|
162
162
|
|
|
163
|
+
## Carrying the magnet in the URL fragment
|
|
164
|
+
|
|
165
|
+
The `torrent` block below solves the problem *after* the TileJSON has been
|
|
166
|
+
fetched. It does not solve the one before it: a torrent-aware client that cannot
|
|
167
|
+
reach this server has nothing to work with, so the swarm โ the part that does
|
|
168
|
+
not depend on any server โ is unreachable precisely when the server is down.
|
|
169
|
+
|
|
170
|
+
The fix is to put the magnet in the URL **fragment**:
|
|
171
|
+
|
|
172
|
+
```json
|
|
173
|
+
"sources": {
|
|
174
|
+
"openmaptiles": {
|
|
175
|
+
"type": "vector",
|
|
176
|
+
"url": "https://swarm.example.org/latest/openmaptiles/tiles.json#magnet:?xt=urn:btih:4813a0e6โฆ&dn=โฆ&tr=โฆ&ws=โฆ"
|
|
177
|
+
}
|
|
178
|
+
}
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
A fragment is never sent in an HTTP request, so the same string works
|
|
182
|
+
everywhere:
|
|
183
|
+
|
|
184
|
+
| Client | What happens |
|
|
185
|
+
| --- | --- |
|
|
186
|
+
| maplibre-gl-js, Leaflet, anything | fetches the TileJSON, ignores the fragment |
|
|
187
|
+
| maplibre-native without a plugin | the same |
|
|
188
|
+
| torrent-aware | reads the magnet **before any network call**, and still has it if the fetch fails |
|
|
189
|
+
|
|
190
|
+
The console's **Copy TileJSON URL + magnet** button produces exactly this.
|
|
191
|
+
|
|
192
|
+
A magnet needs no encoding in a fragment โ RFC 3986 allows `?`, `&`, `=` and `:`
|
|
193
|
+
there โ and leaving it readable matters for something people paste into a style
|
|
194
|
+
file by hand.
|
|
195
|
+
|
|
196
|
+
### What a client should do with it
|
|
197
|
+
|
|
198
|
+
Three paths, each a strict fallback of the one above:
|
|
199
|
+
|
|
200
|
+
1. **The TileJSON URL.** One request, the full document including
|
|
201
|
+
`vector_layers`. Fastest, and what an ordinary client does anyway.
|
|
202
|
+
2. **The `ws=` web seed.** Two HTTP range requests โ the header and root
|
|
203
|
+
directory near the start of the archive, the JSON metadata at the far end โ
|
|
204
|
+
and the TileJSON can be derived from them. Works when this API is down but
|
|
205
|
+
the file is still on a web server, and it is the same order of cost as (1).
|
|
206
|
+
3. **The swarm.** No HTTP at all. Slowest from cold, because BEP 9 has to
|
|
207
|
+
deliver the metainfo first, and the only one that survives the server
|
|
208
|
+
disappearing entirely.
|
|
209
|
+
|
|
210
|
+
Everything those need is in the magnet: `xt` identifies the archive, `dn` names
|
|
211
|
+
the file, `tr` finds peers, `ws` gives the HTTP fallback.
|
|
212
|
+
|
|
213
|
+
### One caveat on `/latest/` URLs
|
|
214
|
+
|
|
215
|
+
`/latest/<category>/tiles.json` follows the category, and a magnet naming an
|
|
216
|
+
infohash does not โ it pins the build that was current when the URL was copied.
|
|
217
|
+
So the fragment goes stale on the next build while the URL does not.
|
|
218
|
+
|
|
219
|
+
That is survivable, because the fragment is only consulted when the TileJSON
|
|
220
|
+
cannot be fetched, and an older build renders where a blank map does not. But it
|
|
221
|
+
means the two halves can disagree, and the fix is a **mutable** magnet
|
|
222
|
+
(`xs=urn:btpk:โฆ`, BEP 46) whose target is resolved over the DHT rather than
|
|
223
|
+
baked into the string. See [src/mutable.js](../src/mutable.js) โ note that
|
|
224
|
+
nothing publishes those records yet.
|
|
225
|
+
|
|
226
|
+
For an immutable `/archives/<infohash>/tiles.json` URL the question does not
|
|
227
|
+
arise: both halves name the same fixed archive.
|
|
228
|
+
|
|
163
229
|
## The `torrent` block
|
|
164
230
|
|
|
165
231
|
TileJSON documents from this server carry a non-standard `torrent` member:
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pmtiles-swarm",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.11.0",
|
|
4
4
|
"description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "src/index.js",
|
|
@@ -39,7 +39,7 @@
|
|
|
39
39
|
"maplibre-gl": "^6.2.0",
|
|
40
40
|
"parse-torrent": "^11.0.24",
|
|
41
41
|
"pmtiles": "^4.4.1",
|
|
42
|
-
"pmtiles-torrent": "^0.4.
|
|
42
|
+
"pmtiles-torrent": "^0.4.2",
|
|
43
43
|
"webtorrent": "^3.0.21"
|
|
44
44
|
},
|
|
45
45
|
"engines": {
|
package/src/api.js
CHANGED
|
@@ -2,6 +2,7 @@ import crypto from 'node:crypto';
|
|
|
2
2
|
import fsSync from 'node:fs';
|
|
3
3
|
import fs from 'node:fs/promises';
|
|
4
4
|
import { createRequire } from 'node:module';
|
|
5
|
+
import os from 'node:os';
|
|
5
6
|
import path from 'node:path';
|
|
6
7
|
import { fileURLToPath } from 'node:url';
|
|
7
8
|
import express from 'express';
|
|
@@ -70,6 +71,7 @@ export function createApp({
|
|
|
70
71
|
warm,
|
|
71
72
|
config,
|
|
72
73
|
speed,
|
|
74
|
+
stats,
|
|
73
75
|
reloaders = {},
|
|
74
76
|
shutdown,
|
|
75
77
|
}) {
|
|
@@ -496,6 +498,39 @@ export function createApp({
|
|
|
496
498
|
}),
|
|
497
499
|
);
|
|
498
500
|
|
|
501
|
+
// What this node has served, which is a different question from whether it
|
|
502
|
+
// is healthy. Admin-side: it names archives and client addresses, and a
|
|
503
|
+
// public endpoint reporting who else is using a node is a privacy question
|
|
504
|
+
// nobody asked for.
|
|
505
|
+
app.get(
|
|
506
|
+
'/api/stats',
|
|
507
|
+
route(async (req, res) => {
|
|
508
|
+
if (!stats) {
|
|
509
|
+
return res.status(501).json({ error: 'tile statistics are disabled' });
|
|
510
|
+
}
|
|
511
|
+
const recent = req.query.recent === undefined ? undefined : Number(req.query.recent);
|
|
512
|
+
res.setHeader('cache-control', 'no-store');
|
|
513
|
+
res.json({
|
|
514
|
+
node: config.nodeName ?? os.hostname(),
|
|
515
|
+
...stats.snapshot({ recent }),
|
|
516
|
+
});
|
|
517
|
+
}),
|
|
518
|
+
);
|
|
519
|
+
|
|
520
|
+
// Deliberately a separate verb: reading counters should never be able to
|
|
521
|
+
// clear them, or a dashboard polling the endpoint would erase the history
|
|
522
|
+
// it is drawing.
|
|
523
|
+
app.delete(
|
|
524
|
+
'/api/stats',
|
|
525
|
+
route(async (_req, res) => {
|
|
526
|
+
if (!stats) {
|
|
527
|
+
return res.status(501).json({ error: 'tile statistics are disabled' });
|
|
528
|
+
}
|
|
529
|
+
stats.reset();
|
|
530
|
+
res.status(204).end();
|
|
531
|
+
}),
|
|
532
|
+
);
|
|
533
|
+
|
|
499
534
|
// Settings. Everything read per request takes effect immediately because the
|
|
500
535
|
// running config object is the one being mutated; everything bound at startup
|
|
501
536
|
// is written to the file and reported back as needing a restart, rather than
|
|
@@ -730,7 +765,12 @@ export function createApp({
|
|
|
730
765
|
// Optional call: a missing method throws before any .catch could see it.
|
|
731
766
|
const diskBytes =
|
|
732
767
|
(await library.diskUsage?.(entry.infoHash).catch(() => null)) ?? null;
|
|
733
|
-
|
|
768
|
+
// Null until a tile has been asked for, and reset by a restart โ the
|
|
769
|
+
// same lifetime as `reading`, and worth reading alongside it: an archive
|
|
770
|
+
// being read through the swarm while serving thousands of tiles is a
|
|
771
|
+
// different situation from one doing neither.
|
|
772
|
+
const served = stats?.forArchive(entry.infoHash) ?? null;
|
|
773
|
+
res.json({ ...entry, status, reading, diskBytes, served });
|
|
734
774
|
}),
|
|
735
775
|
);
|
|
736
776
|
|
|
@@ -2033,6 +2073,31 @@ export function createApp({
|
|
|
2033
2073
|
if (!res.writableEnded) controller.abort();
|
|
2034
2074
|
});
|
|
2035
2075
|
|
|
2076
|
+
// Counted on the way out rather than at each return: this handler ends
|
|
2077
|
+
// in six different places (200, 204, 404, 415, 400, a read error), and
|
|
2078
|
+
// one hook catches all of them without any of them having to remember.
|
|
2079
|
+
// Abandoned requests are not counted -- a panning map cancels constantly
|
|
2080
|
+
// and those were never served.
|
|
2081
|
+
if (stats) {
|
|
2082
|
+
const startedAt = process.hrtime.bigint();
|
|
2083
|
+
res.on('finish', () => {
|
|
2084
|
+
stats.record({
|
|
2085
|
+
infoHash,
|
|
2086
|
+
name: entry.name,
|
|
2087
|
+
z,
|
|
2088
|
+
x,
|
|
2089
|
+
y,
|
|
2090
|
+
status: res.statusCode,
|
|
2091
|
+
bytes: Number(res.getHeader('content-length')) || 0,
|
|
2092
|
+
ms: Number(process.hrtime.bigint() - startedAt) / 1e6,
|
|
2093
|
+
// Whatever this process can see. Behind a proxy that sends no
|
|
2094
|
+
// X-Forwarded-For this is the proxy's address, which is itself
|
|
2095
|
+
// the answer to "did this arrive directly or through HAProxy".
|
|
2096
|
+
ip: req.ip,
|
|
2097
|
+
});
|
|
2098
|
+
});
|
|
2099
|
+
}
|
|
2100
|
+
|
|
2036
2101
|
let tile;
|
|
2037
2102
|
try {
|
|
2038
2103
|
tile = await tiles.getTile(infoHash, z, x, y, {
|
package/src/config.js
CHANGED
|
@@ -397,6 +397,25 @@ const DEFAULTS = {
|
|
|
397
397
|
* instead sidesteps the question entirely.
|
|
398
398
|
*/
|
|
399
399
|
trustProxy: false,
|
|
400
|
+
/**
|
|
401
|
+
* What this node has served, at `GET /api/stats`.
|
|
402
|
+
*
|
|
403
|
+
* Per-archive counters and a fixed ring of recent requests, both in memory,
|
|
404
|
+
* so the cost does not grow with traffic. Nothing is written to disk: a
|
|
405
|
+
* restart is how you reset it, and an access log would bring retention and
|
|
406
|
+
* disk questions this deliberately does not have.
|
|
407
|
+
*
|
|
408
|
+
* `recent: 0` keeps the counters and drops the per-request ring. Setting
|
|
409
|
+
* `tileStats` to `false` turns the whole thing off.
|
|
410
|
+
*
|
|
411
|
+
* Note what the client address means behind a proxy: without
|
|
412
|
+
* `X-Forwarded-For` it is the proxy's own address, which still answers
|
|
413
|
+
* whether a request arrived directly or through it, but not who sent it.
|
|
414
|
+
*/
|
|
415
|
+
tileStats: {
|
|
416
|
+
/** Recent requests kept for inspection. */
|
|
417
|
+
recent: 200,
|
|
418
|
+
},
|
|
400
419
|
/**
|
|
401
420
|
* Tile serving: a TileJSON endpoint and z/x/y tiles per archive.
|
|
402
421
|
*
|
package/src/index.js
CHANGED
|
@@ -18,6 +18,7 @@ import { SeedingLimits } from './seeding.js';
|
|
|
18
18
|
import { closeServer, installSignalHandlers, runStoppers } from './shutdown.js';
|
|
19
19
|
import { ScheduledSourceManager } from './sources.js';
|
|
20
20
|
import { SubscriptionManager } from './subscriptions.js';
|
|
21
|
+
import { TileStats } from './tile-stats.js';
|
|
21
22
|
import { TileStore } from './tiles.js';
|
|
22
23
|
import { HeadWarmer } from './prewarm.js';
|
|
23
24
|
import { WarmRunner } from './warm.js';
|
|
@@ -236,6 +237,14 @@ PMTILES_SWARM_PUBLIC_URL
|
|
|
236
237
|
);
|
|
237
238
|
}
|
|
238
239
|
|
|
240
|
+
// What this node has served. In memory and bounded, so it costs the same
|
|
241
|
+
// after a billion tiles as after ten; `tileStats.recent: 0` keeps the
|
|
242
|
+
// counters and drops the per-request ring, and `false` turns it off.
|
|
243
|
+
const stats =
|
|
244
|
+
config.tileStats === false
|
|
245
|
+
? null
|
|
246
|
+
: new TileStats({ recent: config.tileStats?.recent });
|
|
247
|
+
|
|
239
248
|
const tiles = new TileStore({ catalog, engine, config });
|
|
240
249
|
library.attachTiles(tiles);
|
|
241
250
|
const warm = new WarmRunner(tiles);
|
|
@@ -291,6 +300,7 @@ PMTILES_SWARM_PUBLIC_URL
|
|
|
291
300
|
warm,
|
|
292
301
|
config,
|
|
293
302
|
speed,
|
|
303
|
+
stats,
|
|
294
304
|
reloaders,
|
|
295
305
|
shutdown: () => runStoppers(stoppers),
|
|
296
306
|
});
|
|
@@ -0,0 +1,249 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* What this node has actually served.
|
|
3
|
+
*
|
|
4
|
+
* A tile request is answered and forgotten, which leaves the most ordinary
|
|
5
|
+
* operational questions unanswerable: which archive is carrying the load, which
|
|
6
|
+
* zooms are being asked for, whether a node behind a balancer is getting its
|
|
7
|
+
* share, and whether the thing hammering it arrived directly or through the
|
|
8
|
+
* proxy. Everything here is in memory and bounded โ counters that never grow
|
|
9
|
+
* past the number of archives, and a fixed ring of recent requests โ so it
|
|
10
|
+
* costs the same whether the node has served a hundred tiles or a billion.
|
|
11
|
+
*
|
|
12
|
+
* Deliberately not persisted. Restarting is how you reset it, and a node that
|
|
13
|
+
* writes an access log has retention and disk questions this does not.
|
|
14
|
+
*/
|
|
15
|
+
|
|
16
|
+
/** How many recent requests to keep, when the config says nothing. */
|
|
17
|
+
const DEFAULT_RECENT = 200;
|
|
18
|
+
|
|
19
|
+
/** How many durations to keep per archive for percentiles. */
|
|
20
|
+
const SAMPLE_SIZE = 512;
|
|
21
|
+
|
|
22
|
+
/**
|
|
23
|
+
* A fixed-size sample of durations, for percentiles without unbounded memory.
|
|
24
|
+
*
|
|
25
|
+
* Keeps the most recent SAMPLE_SIZE values rather than a true reservoir: what
|
|
26
|
+
* an operator wants is "how is it behaving now", and an even sample over all
|
|
27
|
+
* time hides a node that became slow ten minutes ago.
|
|
28
|
+
*/
|
|
29
|
+
class Durations {
|
|
30
|
+
#values = [];
|
|
31
|
+
#next = 0;
|
|
32
|
+
|
|
33
|
+
/**
|
|
34
|
+
* Records one duration.
|
|
35
|
+
* @param {number} ms - How long the request took.
|
|
36
|
+
* @returns {void}
|
|
37
|
+
*/
|
|
38
|
+
add(ms) {
|
|
39
|
+
if (!Number.isFinite(ms)) return;
|
|
40
|
+
if (this.#values.length < SAMPLE_SIZE) {
|
|
41
|
+
this.#values.push(ms);
|
|
42
|
+
return;
|
|
43
|
+
}
|
|
44
|
+
this.#values[this.#next] = ms;
|
|
45
|
+
this.#next = (this.#next + 1) % SAMPLE_SIZE;
|
|
46
|
+
}
|
|
47
|
+
|
|
48
|
+
/**
|
|
49
|
+
* The value below which the given fraction of samples fall.
|
|
50
|
+
* @param {number} fraction - Between 0 and 1.
|
|
51
|
+
* @returns {number | null} - Milliseconds, or null with no samples.
|
|
52
|
+
*/
|
|
53
|
+
percentile(fraction) {
|
|
54
|
+
if (this.#values.length === 0) return null;
|
|
55
|
+
const sorted = [...this.#values].sort((a, b) => a - b);
|
|
56
|
+
const index = Math.min(
|
|
57
|
+
sorted.length - 1,
|
|
58
|
+
Math.max(0, Math.ceil(fraction * sorted.length) - 1),
|
|
59
|
+
);
|
|
60
|
+
return sorted[index];
|
|
61
|
+
}
|
|
62
|
+
}
|
|
63
|
+
|
|
64
|
+
/**
|
|
65
|
+
* Per-archive counters and a ring of recent requests.
|
|
66
|
+
*/
|
|
67
|
+
export class TileStats {
|
|
68
|
+
#archives = new Map();
|
|
69
|
+
#recent = [];
|
|
70
|
+
#recentNext = 0;
|
|
71
|
+
#limit;
|
|
72
|
+
#since = new Date().toISOString();
|
|
73
|
+
#total = 0;
|
|
74
|
+
#bytes = 0;
|
|
75
|
+
|
|
76
|
+
/**
|
|
77
|
+
* @param {object} [options] - Configuration.
|
|
78
|
+
* @param {number} [options.recent] - How many recent requests to keep.
|
|
79
|
+
*/
|
|
80
|
+
constructor(options = {}) {
|
|
81
|
+
const asked = Number(options.recent);
|
|
82
|
+
// Zero is a meaningful answer โ counters without the ring โ so it is not
|
|
83
|
+
// treated as "unset". Negative or unparseable falls back to the default.
|
|
84
|
+
this.#limit = Number.isFinite(asked) && asked >= 0 ? Math.floor(asked) : DEFAULT_RECENT;
|
|
85
|
+
}
|
|
86
|
+
|
|
87
|
+
/**
|
|
88
|
+
* Records one served tile request.
|
|
89
|
+
* @param {object} entry - What was served.
|
|
90
|
+
* @param {string} entry.infoHash - Which archive.
|
|
91
|
+
* @param {string} [entry.name] - Its filename, for readability.
|
|
92
|
+
* @param {number} entry.z - Zoom.
|
|
93
|
+
* @param {number} [entry.x] - Column.
|
|
94
|
+
* @param {number} [entry.y] - Row.
|
|
95
|
+
* @param {number} entry.status - HTTP status answered.
|
|
96
|
+
* @param {number} [entry.bytes] - Body size.
|
|
97
|
+
* @param {number} [entry.ms] - How long it took.
|
|
98
|
+
* @param {string} [entry.ip] - Who asked, as seen by this process.
|
|
99
|
+
* @param {string} [entry.at] - ISO timestamp, for tests.
|
|
100
|
+
* @returns {void}
|
|
101
|
+
*/
|
|
102
|
+
record(entry) {
|
|
103
|
+
const { infoHash, z, status } = entry;
|
|
104
|
+
if (!infoHash) return;
|
|
105
|
+
|
|
106
|
+
let archive = this.#archives.get(infoHash);
|
|
107
|
+
if (!archive) {
|
|
108
|
+
archive = {
|
|
109
|
+
name: entry.name,
|
|
110
|
+
requests: 0,
|
|
111
|
+
bytes: 0,
|
|
112
|
+
byZoom: new Map(),
|
|
113
|
+
byStatus: new Map(),
|
|
114
|
+
clients: new Map(),
|
|
115
|
+
durations: new Durations(),
|
|
116
|
+
firstSeen: entry.at ?? new Date().toISOString(),
|
|
117
|
+
lastSeen: null,
|
|
118
|
+
};
|
|
119
|
+
this.#archives.set(infoHash, archive);
|
|
120
|
+
}
|
|
121
|
+
// A rename or a later import can fill this in after the first request.
|
|
122
|
+
if (!archive.name && entry.name) archive.name = entry.name;
|
|
123
|
+
|
|
124
|
+
const bytes = Number.isFinite(entry.bytes) ? entry.bytes : 0;
|
|
125
|
+
archive.requests += 1;
|
|
126
|
+
archive.bytes += bytes;
|
|
127
|
+
archive.lastSeen = entry.at ?? new Date().toISOString();
|
|
128
|
+
archive.durations.add(entry.ms);
|
|
129
|
+
this.#total += 1;
|
|
130
|
+
this.#bytes += bytes;
|
|
131
|
+
|
|
132
|
+
if (Number.isInteger(z)) {
|
|
133
|
+
archive.byZoom.set(z, (archive.byZoom.get(z) ?? 0) + 1);
|
|
134
|
+
}
|
|
135
|
+
if (Number.isFinite(status)) {
|
|
136
|
+
archive.byStatus.set(status, (archive.byStatus.get(status) ?? 0) + 1);
|
|
137
|
+
}
|
|
138
|
+
if (entry.ip) {
|
|
139
|
+
// Counted rather than listed: a busy node sees a handful of distinct
|
|
140
|
+
// sources โ the proxy, a few LAN clients โ and the count is the answer
|
|
141
|
+
// to "is this arriving directly or through HAProxy".
|
|
142
|
+
const seen = archive.clients.get(entry.ip) ?? { requests: 0, bytes: 0 };
|
|
143
|
+
seen.requests += 1;
|
|
144
|
+
seen.bytes += bytes;
|
|
145
|
+
archive.clients.set(entry.ip, seen);
|
|
146
|
+
}
|
|
147
|
+
|
|
148
|
+
if (this.#limit === 0) return;
|
|
149
|
+
const row = {
|
|
150
|
+
at: archive.lastSeen,
|
|
151
|
+
ip: entry.ip ?? null,
|
|
152
|
+
infoHash,
|
|
153
|
+
name: archive.name ?? null,
|
|
154
|
+
z: Number.isInteger(z) ? z : null,
|
|
155
|
+
x: Number.isInteger(entry.x) ? entry.x : null,
|
|
156
|
+
y: Number.isInteger(entry.y) ? entry.y : null,
|
|
157
|
+
status: status ?? null,
|
|
158
|
+
bytes,
|
|
159
|
+
ms: Number.isFinite(entry.ms) ? entry.ms : null,
|
|
160
|
+
};
|
|
161
|
+
if (this.#recent.length < this.#limit) {
|
|
162
|
+
this.#recent.push(row);
|
|
163
|
+
} else {
|
|
164
|
+
this.#recent[this.#recentNext] = row;
|
|
165
|
+
this.#recentNext = (this.#recentNext + 1) % this.#limit;
|
|
166
|
+
}
|
|
167
|
+
}
|
|
168
|
+
|
|
169
|
+
/**
|
|
170
|
+
* Everything recorded, as plain JSON.
|
|
171
|
+
* @param {object} [options] - Shaping.
|
|
172
|
+
* @param {number} [options.recent] - How many recent rows to return.
|
|
173
|
+
* @returns {object} - The report.
|
|
174
|
+
*/
|
|
175
|
+
snapshot(options = {}) {
|
|
176
|
+
const archives = {};
|
|
177
|
+
for (const infoHash of this.#archives.keys()) {
|
|
178
|
+
archives[infoHash] = this.forArchive(infoHash);
|
|
179
|
+
}
|
|
180
|
+
|
|
181
|
+
const asked = Number(options.recent);
|
|
182
|
+
const wanted = Number.isFinite(asked) && asked >= 0 ? Math.floor(asked) : this.#limit;
|
|
183
|
+
|
|
184
|
+
return {
|
|
185
|
+
since: this.#since,
|
|
186
|
+
requests: this.#total,
|
|
187
|
+
bytes: this.#bytes,
|
|
188
|
+
archives,
|
|
189
|
+
recent: this.recent(wanted),
|
|
190
|
+
};
|
|
191
|
+
}
|
|
192
|
+
|
|
193
|
+
/**
|
|
194
|
+
* What one archive has served, or null if it has served nothing.
|
|
195
|
+
*
|
|
196
|
+
* Separate from snapshot() so the per-archive detail endpoint can answer
|
|
197
|
+
* "what is this one doing" without building a report about every other.
|
|
198
|
+
* @param {string} infoHash - Which archive.
|
|
199
|
+
* @returns {object | null} - Its counters.
|
|
200
|
+
*/
|
|
201
|
+
forArchive(infoHash) {
|
|
202
|
+
const a = this.#archives.get(infoHash);
|
|
203
|
+
if (!a) return null;
|
|
204
|
+
return {
|
|
205
|
+
name: a.name ?? null,
|
|
206
|
+
requests: a.requests,
|
|
207
|
+
bytes: a.bytes,
|
|
208
|
+
byZoom: Object.fromEntries([...a.byZoom].sort((x, y) => x[0] - y[0])),
|
|
209
|
+
byStatus: Object.fromEntries([...a.byStatus].sort((x, y) => x[0] - y[0])),
|
|
210
|
+
clients: Object.fromEntries(
|
|
211
|
+
[...a.clients]
|
|
212
|
+
.sort((x, y) => y[1].requests - x[1].requests)
|
|
213
|
+
.map(([ip, seen]) => [ip, seen.requests]),
|
|
214
|
+
),
|
|
215
|
+
p50ms: a.durations.percentile(0.5),
|
|
216
|
+
p95ms: a.durations.percentile(0.95),
|
|
217
|
+
firstSeen: a.firstSeen,
|
|
218
|
+
lastSeen: a.lastSeen,
|
|
219
|
+
};
|
|
220
|
+
}
|
|
221
|
+
|
|
222
|
+
/**
|
|
223
|
+
* The most recent requests, newest first.
|
|
224
|
+
* @param {number} [count] - How many to return.
|
|
225
|
+
* @returns {object[]} - Recent rows.
|
|
226
|
+
*/
|
|
227
|
+
recent(count = this.#limit) {
|
|
228
|
+
if (this.#recent.length === 0 || count <= 0) return [];
|
|
229
|
+
// The ring is oldest-first from the write cursor once it has wrapped.
|
|
230
|
+
const ordered =
|
|
231
|
+
this.#recent.length < this.#limit
|
|
232
|
+
? [...this.#recent]
|
|
233
|
+
: [...this.#recent.slice(this.#recentNext), ...this.#recent.slice(0, this.#recentNext)];
|
|
234
|
+
return ordered.reverse().slice(0, count);
|
|
235
|
+
}
|
|
236
|
+
|
|
237
|
+
/**
|
|
238
|
+
* Forgets everything, as a restart would.
|
|
239
|
+
* @returns {void}
|
|
240
|
+
*/
|
|
241
|
+
reset() {
|
|
242
|
+
this.#archives.clear();
|
|
243
|
+
this.#recent = [];
|
|
244
|
+
this.#recentNext = 0;
|
|
245
|
+
this.#total = 0;
|
|
246
|
+
this.#bytes = 0;
|
|
247
|
+
this.#since = new Date().toISOString();
|
|
248
|
+
}
|
|
249
|
+
}
|
package/src/web/index.html
CHANGED
|
@@ -1349,6 +1349,15 @@
|
|
|
1349
1349
|
<div><span>origin</span><b>${escapeHtml(originOf(entry).label)}</b>
|
|
1350
1350
|
<span class="sub">${escapeHtml(originOf(entry).detail)}</span></div>
|
|
1351
1351
|
<div><span>reading via</span><b>${escapeHtml(entry.reading?.mode ?? 'not open')}</b></div>
|
|
1352
|
+
<div><span>tiles served</span><b>${
|
|
1353
|
+
entry.served ? entry.served.requests.toLocaleString() : 'โ'
|
|
1354
|
+
}</b><span class="sub">${
|
|
1355
|
+
entry.served
|
|
1356
|
+
? `${bytes(entry.served.bytes)} sent ยท ${
|
|
1357
|
+
entry.served.p50ms === null ? 'โ' : `${Math.round(entry.served.p50ms)} ms`
|
|
1358
|
+
} median`
|
|
1359
|
+
: 'since this node started'
|
|
1360
|
+
}</span></div>
|
|
1352
1361
|
<div><span>format</span><b>${escapeHtml(summary.format ?? 'โ')}</b></div>
|
|
1353
1362
|
<div><span>zoom</span><b>${summary.minZoom ?? 'โ'}โ${summary.maxZoom ?? 'โ'}</b></div>
|
|
1354
1363
|
<div><span>web seeds</span><b>${(entry.webSeeds ?? []).length}</b></div>
|
|
@@ -1371,6 +1380,7 @@
|
|
|
1371
1380
|
<button id="copy-magnet">Copy magnet</button>
|
|
1372
1381
|
<button id="copy-hash">Copy infohash</button>
|
|
1373
1382
|
${servable ? `<button id="copy-tilejson">Copy TileJSON URL</button>
|
|
1383
|
+
<button id="copy-tilejson-swarm" title="The same URL with the magnet in the fragment. A fragment is never sent to the server, so ordinary clients fetch the TileJSON exactly as before; a torrent-aware client reads the magnet from it and can still join the swarm when this server is unreachable.">Copy TileJSON URL + magnet</button>
|
|
1374
1384
|
<a href="${tileJson}" target="_blank" rel="noreferrer"><button type="button">Open TileJSON</button></a>
|
|
1375
1385
|
<a href="${base}/archives/${entry.infoHash}/preview" target="_blank" rel="noreferrer"><button type="button">${
|
|
1376
1386
|
summary.format === 'pbf' ? 'Inspect' : 'Preview'
|
|
@@ -1642,6 +1652,20 @@
|
|
|
1642
1652
|
$('copy-magnet').onclick = () => copy(entry.magnet ?? '', 'Magnet link');
|
|
1643
1653
|
if (servable) {
|
|
1644
1654
|
$('copy-tilejson').onclick = () => copy(tileJson, 'TileJSON URL');
|
|
1655
|
+
// The magnet rides in the fragment, which is client-side only and
|
|
1656
|
+
// never sent in the request. So the same string serves both: an
|
|
1657
|
+
// ordinary client fetches the TileJSON and ignores the fragment, and
|
|
1658
|
+
// a torrent-aware one reads the magnet before making any call at all
|
|
1659
|
+
// โ which is what lets it start when this server is down.
|
|
1660
|
+
//
|
|
1661
|
+
// A magnet is legal in a fragment unencoded (RFC 3986 allows ?, &, =
|
|
1662
|
+
// and : there), and leaving it readable matters for something people
|
|
1663
|
+
// paste into a style file by hand.
|
|
1664
|
+
$('copy-tilejson-swarm').onclick = () =>
|
|
1665
|
+
copy(
|
|
1666
|
+
entry.magnet ? `${tileJson}#${entry.magnet}` : tileJson,
|
|
1667
|
+
entry.magnet ? 'TileJSON URL with magnet' : 'TileJSON URL (no magnet on this archive)',
|
|
1668
|
+
);
|
|
1645
1669
|
}
|
|
1646
1670
|
$('copy-hash').onclick = () => copy(entry.infoHash, 'Infohash');
|
|
1647
1671
|
|