pmtiles-swarm 0.3.0 โ 0.3.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +65 -0
- package/README.md +15 -3
- package/docs/running-as-a-service.md +305 -0
- package/package.json +3 -2
- package/src/api.js +13 -0
- package/src/config.js +25 -0
- package/src/index.js +15 -0
- package/src/web/index.html +28 -0
- package/swarm.config.json.sample +86 -0
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,71 @@
|
|
|
7
7
|
### ๐ Bug fixes
|
|
8
8
|
- _...Add new stuff here..._
|
|
9
9
|
|
|
10
|
+
## 0.3.2
|
|
11
|
+
### โจ Features and improvements
|
|
12
|
+
- **The console has a footer naming the version it is running**, beside `ยฉ <year> TechIdiots LLC`
|
|
13
|
+
and a link to the source. The version comes from `package.json` through `/api/status` rather than
|
|
14
|
+
being written into the page, since the number on screen is the one somebody quotes when reporting
|
|
15
|
+
a problem.
|
|
16
|
+
|
|
17
|
+
### ๐ Documentation
|
|
18
|
+
- **How to update an installed service**, which was missing: reinstall into the same prefix and
|
|
19
|
+
restart. The restart is not optional โ the Python sidecar is started with the process and lives
|
|
20
|
+
as long as it does, so a new one sits on disk doing nothing until then, and most of what changes
|
|
21
|
+
between releases is in there.
|
|
22
|
+
- The WebRTC check given in two places imported a directory path, which ESM refuses whatever the
|
|
23
|
+
state of the install โ so it reported a failure that was never about WebRTC. It now imports
|
|
24
|
+
`node-datachannel` by name from the install directory, which is the binary the install script
|
|
25
|
+
fetches.
|
|
26
|
+
|
|
27
|
+
## 0.3.1
|
|
28
|
+
|
|
29
|
+
Depends on pmtiles-torrent 0.3.0, which is what carries the resume-data fix below to an installed
|
|
30
|
+
copy โ 0.3.0 of this package shipped against a sidecar that could not find its own resume data.
|
|
31
|
+
|
|
32
|
+
### โจ Features and improvements
|
|
33
|
+
- **The sample configuration ships with the package**, so an installed copy has one to copy from
|
|
34
|
+
rather than only the repository โ which is the one place someone installing from npm has not got.
|
|
35
|
+
- **The `allowScripts` warning from npm 11.17 is explained.** npm is moving dependency install
|
|
36
|
+
scripts behind an allowlist; today it warns and still runs them, so an install that prints it is
|
|
37
|
+
fine. One of those scripts matters โ `node-datachannel` downloads the WebRTC binary WebTorrent
|
|
38
|
+
needs, which is not in the published tarball โ so the documentation gives a one-line check that
|
|
39
|
+
it landed, and says to leave `--strict-allow-scripts` alone, since in testing it blocked approved
|
|
40
|
+
scripts as well as unapproved ones.
|
|
41
|
+
- **The service documentation installs into the account's own directory, not globally.**
|
|
42
|
+
`sudo npm install -g` fails on some machines: a WebTorrent dependency runs `npx only-allow pnpm`
|
|
43
|
+
as a preinstall step, and under `sudo` that npx cannot write root's cache. Installing as the
|
|
44
|
+
service account avoids root's cache entirely, keeps the version pinned per service, and makes
|
|
45
|
+
upgrading one command. `--ignore-scripts` is explicitly not the answer โ `node-datachannel`
|
|
46
|
+
fetches its prebuilt binary in an install script, and without it WebTorrent cannot do WebRTC,
|
|
47
|
+
which is the only reason to run it alongside libtorrent.
|
|
48
|
+
- **Running as a systemd service is documented**, with a unit file, and now with the account setup:
|
|
49
|
+
creating a dedicated `pmtiles-swarm` system user and group, the two directories, and where Node
|
|
50
|
+
and the package go. Both directories have to be writable by the service, including the one under
|
|
51
|
+
`/etc` โ minting a token or pressing Save rewrites the configuration, so a root-owned file the
|
|
52
|
+
service can only read loses tokens on restart. Two things in it are not
|
|
53
|
+
preferences: `Restart=always` is required rather than optional, because the console's *Save &
|
|
54
|
+
Restart* detects the supervisor and exits 0 expecting to be brought back โ under
|
|
55
|
+
`Restart=on-failure` the first use of it stops the node and leaves a unit reporting success. And
|
|
56
|
+
the `ExecStop=/bin/kill -15 $MAINPID` line commonly copied between units should be omitted, since
|
|
57
|
+
systemd already sends SIGTERM and the node installs its handlers before it begins work.
|
|
58
|
+
|
|
59
|
+
### ๐ Bug fixes
|
|
60
|
+
- **A restart no longer re-hashes every archive.** Resume data was being written and never found โ
|
|
61
|
+
the lookup used an infohash no caller supplied, and `add_torrent_params.info_hashes` reads as
|
|
62
|
+
forty zeros for a torrent added from a `.torrent` file, so keying it off the torrent alone would
|
|
63
|
+
not have helped either. Fixed in pmtiles-torrent 0.3.0; on a 512 MiB archive the difference
|
|
64
|
+
measured 1.21s with a full re-hash against 0.02s with none, and it scales with the archive.
|
|
65
|
+
Resume data is also written every `resumeSaveIntervalSeconds` (five minutes by default) rather
|
|
66
|
+
than only at shutdown, so a kill or a power cut costs the last few minutes instead of everything.
|
|
67
|
+
- **Every path in the configuration resolves against the configuration file.** `dataDir`,
|
|
68
|
+
`savePath`, `cacheSavePath` and watched folders already did; `locations[].path` and
|
|
69
|
+
`libtorrent.resumeDir` were left relative, which means relative to the working directory. Started
|
|
70
|
+
by hand from the repository the two agree, so it never showed โ but a service does not run from
|
|
71
|
+
the directory its config lives in, and under systemd the working directory defaults to `/`, so
|
|
72
|
+
`./data/resume` became `/data/resume`: somewhere the unit almost certainly cannot write, for a
|
|
73
|
+
reason nothing in the config hints at.
|
|
74
|
+
|
|
10
75
|
## 0.3.0
|
|
11
76
|
### โจ Features and improvements
|
|
12
77
|
- **The peers column distinguishes who is connected from what the swarm holds.** `0 / 2` on a
|
package/README.md
CHANGED
|
@@ -31,6 +31,8 @@ node src/index.js --config swarm.config.json
|
|
|
31
31
|
- **[docs/security.md](docs/security.md)** โ what is public, what is guarded, named tokens with
|
|
32
32
|
roles, console sign-in, and why an unauthenticated node refuses to listen on a reachable
|
|
33
33
|
address.
|
|
34
|
+
- **[docs/running-as-a-service.md](docs/running-as-a-service.md)** โ a systemd unit, why
|
|
35
|
+
`Restart=always` is required rather than optional, and which ports want a firewall rule.
|
|
34
36
|
- **[docs/architecture-diagram.md](docs/architecture-diagram.md)** โ how a publishing node, a
|
|
35
37
|
serving tier, the swarm and both kinds of client fit together.
|
|
36
38
|
|
|
@@ -131,12 +133,22 @@ poll a stranger's server before you have read it.
|
|
|
131
133
|
`swarm.config.json` itself is gitignored โ along with every other
|
|
132
134
|
`swarm.config*.json` and `*.bak` โ because it holds an API key.
|
|
133
135
|
|
|
136
|
+
**`adminPort` is unset by default**, and one listener then serves everything.
|
|
137
|
+
Setting it splits the surface in two: `port` keeps the tiles, TileJSON,
|
|
138
|
+
`.torrent` files and feeds โ everything a stranger or a peer is meant to reach
|
|
139
|
+
โ while `adminPort` takes the console and the rest of `/api/`. Bind that to
|
|
140
|
+
`127.0.0.1` and the part that can rewrite this file is unreachable rather than
|
|
141
|
+
merely guarded. See [Two ports](#two-ports).
|
|
142
|
+
|
|
134
143
|
```json
|
|
135
144
|
{
|
|
136
145
|
"port": 8090,
|
|
146
|
+
"adminPort": 8091,
|
|
147
|
+
"adminHost": "127.0.0.1",
|
|
137
148
|
"dataDir": "./data",
|
|
138
|
-
"engine": "
|
|
139
|
-
"
|
|
149
|
+
"engine": "libtorrent",
|
|
150
|
+
"secondaryEngines": ["webtorrent"],
|
|
151
|
+
"libtorrent": { "python": "python3", "listen": "0.0.0.0:6881" },
|
|
140
152
|
"savePath": "./data/torrents-data",
|
|
141
153
|
"incompleteSuffix": ".incomplete",
|
|
142
154
|
"pieceLength": 4194304,
|
|
@@ -147,7 +159,7 @@ poll a stranger's server before you have read it.
|
|
|
147
159
|
"watch": [
|
|
148
160
|
{
|
|
149
161
|
"path": "/mnt/maps/incoming",
|
|
150
|
-
"
|
|
162
|
+
"categories": ["basemaps"],
|
|
151
163
|
"publishDir": "/var/www/pmtiles",
|
|
152
164
|
"webSeedBase": "https://maps.example.org/files"
|
|
153
165
|
}
|
|
@@ -0,0 +1,305 @@
|
|
|
1
|
+
# Running as a systemd service
|
|
2
|
+
|
|
3
|
+
Setting up the account, then the unit. Two lines in that unit are not optional,
|
|
4
|
+
and one line most people copy from elsewhere should be deleted โ the rest is
|
|
5
|
+
ordinary.
|
|
6
|
+
|
|
7
|
+
## An account of its own
|
|
8
|
+
|
|
9
|
+
A system account: no password, no login, and nothing on the machine belongs to
|
|
10
|
+
it except the archives.
|
|
11
|
+
|
|
12
|
+
```sh
|
|
13
|
+
sudo groupadd --system pmtiles-swarm
|
|
14
|
+
sudo useradd --system --gid pmtiles-swarm \
|
|
15
|
+
--home-dir /var/lib/pmtiles-swarm --create-home \
|
|
16
|
+
--shell /usr/sbin/nologin \
|
|
17
|
+
--comment "pmtiles-swarm service" pmtiles-swarm
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
`--system` keeps it out of the human UID range. Home is the data directory, so
|
|
21
|
+
npm and Python per-user state land with the archives.
|
|
22
|
+
|
|
23
|
+
Then the two directories it needs:
|
|
24
|
+
|
|
25
|
+
```sh
|
|
26
|
+
sudo install -d -o pmtiles-swarm -g pmtiles-swarm -m 0750 /var/lib/pmtiles-swarm
|
|
27
|
+
sudo install -d -o pmtiles-swarm -g pmtiles-swarm -m 0750 /etc/pmtiles-swarm
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
**Both have to be writable by the service, including the one under `/etc`** โ
|
|
31
|
+
minting a token or pressing Save in the console rewrites `swarm.config.json`.
|
|
32
|
+
Root-owned, tokens vanish on restart.
|
|
33
|
+
|
|
34
|
+
The file itself holds an API key, so nobody else needs to read it:
|
|
35
|
+
|
|
36
|
+
```sh
|
|
37
|
+
SAMPLE=/var/lib/pmtiles-swarm/node_modules/pmtiles-swarm/swarm.config.json.sample
|
|
38
|
+
sudo install -o pmtiles-swarm -g pmtiles-swarm -m 0600 "$SAMPLE" \
|
|
39
|
+
/etc/pmtiles-swarm/swarm.config.json
|
|
40
|
+
sudoedit /etc/pmtiles-swarm/swarm.config.json
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
Generate the key rather than inventing one:
|
|
44
|
+
|
|
45
|
+
```sh
|
|
46
|
+
node -e "console.log(require('crypto').randomBytes(32).toString('base64url'))"
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
## Node, and the package
|
|
50
|
+
|
|
51
|
+
Node from your distribution or NodeSource โ the package needs `^22.13.0 || 24`:
|
|
52
|
+
|
|
53
|
+
```sh
|
|
54
|
+
curl -fsSL https://deb.nodesource.com/setup_24.x | sudo -E bash -
|
|
55
|
+
sudo apt-get install -y nodejs
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
Then install into the service account's own directory, as that account:
|
|
59
|
+
|
|
60
|
+
```sh
|
|
61
|
+
sudo -u pmtiles-swarm -H npm install --prefix /var/lib/pmtiles-swarm pmtiles-swarm
|
|
62
|
+
sudo -u pmtiles-swarm -H /var/lib/pmtiles-swarm/node_modules/.bin/pmtiles-swarm --help
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
That path is what goes in `ExecStart`. Upgrading is the same command again.
|
|
66
|
+
|
|
67
|
+
Installing as the account rather than into a root-owned tree also sidesteps a
|
|
68
|
+
failure `sudo npm install -g` can hit: `ip-set`, a dependency of WebTorrent,
|
|
69
|
+
runs `npx only-allow pnpm` before installing, and under `sudo` that npx may not
|
|
70
|
+
be able to write root's cache.
|
|
71
|
+
|
|
72
|
+
```
|
|
73
|
+
npm error command sh -c npx only-allow pnpm
|
|
74
|
+
npm error enoent Could not read package.json: ENOENT:
|
|
75
|
+
open '/root/.npm/_npx/0b83cd9ca5e1325c/package.json'
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
If you would rather install as root and hand the tree over โ the usual pattern
|
|
79
|
+
โ give npm a cache it can write and chown afterwards:
|
|
80
|
+
|
|
81
|
+
```sh
|
|
82
|
+
sudo env npm_config_cache=/var/cache/npm \
|
|
83
|
+
npm install --prefix /var/lib/pmtiles-swarm pmtiles-swarm
|
|
84
|
+
sudo chown -R pmtiles-swarm:pmtiles-swarm /var/lib/pmtiles-swarm
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
**nvm does not affect either.** `sudo` resets `PATH` to `secure_path`, so
|
|
88
|
+
`sudo npm` is `/usr/bin/npm` whatever `nvm use` last selected. Only the
|
|
89
|
+
absolute path in `ExecStart` decides what the service runs.
|
|
90
|
+
|
|
91
|
+
### The allowScripts warning
|
|
92
|
+
|
|
93
|
+
npm 11.17 and newer print this, and it is not an error:
|
|
94
|
+
|
|
95
|
+
```
|
|
96
|
+
npm warn allow-scripts 5 packages have install scripts not yet covered by allowScripts:
|
|
97
|
+
npm warn allow-scripts node-datachannel@0.32.3 (install: prebuild-install -r napi || โฆ)
|
|
98
|
+
npm warn allow-scripts ip-set@3.0.0 (preinstall: npx only-allow pnpm)
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
npm is moving dependency install scripts behind an allowlist. Today it warns
|
|
102
|
+
and still runs them. One of those scripts matters: `node-datachannel` downloads
|
|
103
|
+
the prebuilt binary WebTorrent needs for WebRTC, and it is not in the published
|
|
104
|
+
tarball. Check it landed:
|
|
105
|
+
|
|
106
|
+
```sh
|
|
107
|
+
cd /var/lib/pmtiles-swarm && sudo -u pmtiles-swarm -H node -e \
|
|
108
|
+
"import('node-datachannel').then(() => console.log('webrtc ok'))"
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
`npm approve-scripts node-datachannel` records the approval in `package.json`
|
|
112
|
+
and silences the warning for that package:
|
|
113
|
+
|
|
114
|
+
```json
|
|
115
|
+
{ "allowScripts": { "node-datachannel@0.32.3": true } }
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
Leave `--strict-allow-scripts` alone for now. It blocks unapproved scripts
|
|
119
|
+
outright, and in testing it also blocked approved ones โ the resulting install
|
|
120
|
+
has no WebRTC binary and WebTorrent will not load.
|
|
121
|
+
|
|
122
|
+
The libtorrent engine also needs `python3` with the bindings, checked as the
|
|
123
|
+
service account:
|
|
124
|
+
|
|
125
|
+
```sh
|
|
126
|
+
sudo apt-get install -y python3-libtorrent
|
|
127
|
+
sudo -u pmtiles-swarm python3 -c "import libtorrent; print(libtorrent.__version__)"
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
## The unit
|
|
131
|
+
|
|
132
|
+
```ini
|
|
133
|
+
[Unit]
|
|
134
|
+
Description=pmtiles-swarm
|
|
135
|
+
Documentation=https://github.com/TechIdiots-LLC/pmtiles-swarm
|
|
136
|
+
After=network-online.target
|
|
137
|
+
Wants=network-online.target
|
|
138
|
+
|
|
139
|
+
[Service]
|
|
140
|
+
Type=simple
|
|
141
|
+
User=pmtiles-swarm
|
|
142
|
+
Group=pmtiles-swarm
|
|
143
|
+
|
|
144
|
+
WorkingDirectory=/var/lib/pmtiles-swarm
|
|
145
|
+
|
|
146
|
+
# Absolute: systemd reads no shell profile.
|
|
147
|
+
ExecStart=/var/lib/pmtiles-swarm/node_modules/.bin/pmtiles-swarm \
|
|
148
|
+
--config /etc/pmtiles-swarm/swarm.config.json
|
|
149
|
+
|
|
150
|
+
# Required. The console's Save & Restart exits and expects to be brought back;
|
|
151
|
+
# see below.
|
|
152
|
+
Restart=always
|
|
153
|
+
RestartSec=5
|
|
154
|
+
|
|
155
|
+
# Stopping announces "stopped" to every tracker, releases the data directory
|
|
156
|
+
# lock and cancels downloads in flight. Worst case is about 20 seconds.
|
|
157
|
+
TimeoutStopSec=45
|
|
158
|
+
|
|
159
|
+
# A seeding node holds a socket per peer, and the tile reader holds file
|
|
160
|
+
# descriptors of its own.
|
|
161
|
+
LimitNOFILE=65535
|
|
162
|
+
|
|
163
|
+
# The archives and the sidecar are the only things it needs to touch.
|
|
164
|
+
ProtectSystem=strict
|
|
165
|
+
ProtectHome=read-only
|
|
166
|
+
PrivateTmp=true
|
|
167
|
+
NoNewPrivileges=true
|
|
168
|
+
# Both, since the console rewrites the configuration when a token is minted.
|
|
169
|
+
ReadWritePaths=/var/lib/pmtiles-swarm /etc/pmtiles-swarm
|
|
170
|
+
|
|
171
|
+
[Install]
|
|
172
|
+
WantedBy=multi-user.target
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
## The two lines that matter
|
|
176
|
+
|
|
177
|
+
**`Restart=always`, not `on-failure`.** The console's *Save & Restart* applies
|
|
178
|
+
settings a running process cannot take โ the port, the data directory, the
|
|
179
|
+
torrent client. Under a supervisor the node does not relaunch itself: it shuts
|
|
180
|
+
down and **exits 0**, expecting to be brought back.
|
|
181
|
+
|
|
182
|
+
`Restart=on-failure` ignores an exit 0, so the first use of that button would
|
|
183
|
+
stop the node and leave the unit reporting success.
|
|
184
|
+
|
|
185
|
+
**No `ExecStop=`.** systemd already sends `SIGTERM`, and the node handles it
|
|
186
|
+
from the moment it starts. `ExecStop=/bin/kill -15 $MAINPID` is redundant, and
|
|
187
|
+
becomes wrong if the unit ever uses `KillMode=process` โ it would stop the node
|
|
188
|
+
while the Python sidecar kept running and kept the data directory locked.
|
|
189
|
+
|
|
190
|
+
Leave `KillMode` at its default, so the sidecar goes with its parent.
|
|
191
|
+
|
|
192
|
+
## Paths
|
|
193
|
+
|
|
194
|
+
Every path in the configuration resolves **relative to the configuration
|
|
195
|
+
file**, not to the working directory:
|
|
196
|
+
|
|
197
|
+
```
|
|
198
|
+
/etc/pmtiles-swarm/swarm.config.json with "dataDir": "./data"
|
|
199
|
+
-> /etc/pmtiles-swarm/data
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
That is usually not what you want for a service. Use absolute paths for
|
|
203
|
+
anything that holds data:
|
|
204
|
+
|
|
205
|
+
```json
|
|
206
|
+
{
|
|
207
|
+
"dataDir": "/var/lib/pmtiles-swarm",
|
|
208
|
+
"savePath": "/var/lib/pmtiles-swarm/archives",
|
|
209
|
+
"libtorrent": { "resumeDir": "/var/lib/pmtiles-swarm/resume" }
|
|
210
|
+
}
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
`ProtectSystem=strict` makes the whole filesystem read-only apart from what
|
|
214
|
+
`ReadWritePaths` names, so every one of those has to be listed. An archive
|
|
215
|
+
directory on another mount needs its own entry.
|
|
216
|
+
|
|
217
|
+
## The sidecar
|
|
218
|
+
|
|
219
|
+
The libtorrent engine runs Python as a child process, so the service user needs
|
|
220
|
+
`python3` with libtorrent importable โ not your login shell's Python:
|
|
221
|
+
|
|
222
|
+
```sh
|
|
223
|
+
sudo -u pmtiles-swarm python3 -c "import libtorrent; print(libtorrent.__version__)"
|
|
224
|
+
```
|
|
225
|
+
|
|
226
|
+
If that fails, the node still starts and reports the engine as unavailable
|
|
227
|
+
rather than exiting. Name the interpreter explicitly when the service user's
|
|
228
|
+
`PATH` is not what you tested with, which under systemd it usually is not:
|
|
229
|
+
|
|
230
|
+
```json
|
|
231
|
+
{ "libtorrent": { "python": "/usr/bin/python3" } }
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
## Ports
|
|
235
|
+
|
|
236
|
+
Four listeners, and only the peer ports want a firewall rule. See
|
|
237
|
+
[ports and reachability](engines.md#ports-and-reachability) for the detail.
|
|
238
|
+
|
|
239
|
+
| | |
|
|
240
|
+
| --- | --- |
|
|
241
|
+
| `libtorrent.listen` โ 6881, TCP and UDP | forward it |
|
|
242
|
+
| `webtorrent.clientOptions.torrentPort` โ pin it, or it changes every start | forward it |
|
|
243
|
+
| `port` โ 8090 | your proxy or CDN |
|
|
244
|
+
| `adminPort` โ 8091, bound to `127.0.0.1` | nothing; that is the point |
|
|
245
|
+
|
|
246
|
+
## Updating
|
|
247
|
+
|
|
248
|
+
```sh
|
|
249
|
+
sudo -u pmtiles-swarm -H npm install --prefix /var/lib/pmtiles-swarm pmtiles-swarm@latest
|
|
250
|
+
sudo systemctl restart pmtiles-swarm
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
**The restart is not optional.** The Python sidecar is started with the process
|
|
254
|
+
and lives as long as it does, so a new sidecar sits on disk doing nothing until
|
|
255
|
+
the service is restarted. Most of what changes between releases is in there.
|
|
256
|
+
|
|
257
|
+
Nothing under `/etc/pmtiles-swarm` is touched, and restarting does not re-check
|
|
258
|
+
the archives: a clean stop writes resume data, and `TimeoutStopSec` above
|
|
259
|
+
leaves room for it.
|
|
260
|
+
|
|
261
|
+
Confirm both halves moved, since the sidecar has its own version:
|
|
262
|
+
|
|
263
|
+
```sh
|
|
264
|
+
sudo -u pmtiles-swarm -H npm ls --prefix /var/lib/pmtiles-swarm --depth 1 \
|
|
265
|
+
pmtiles-swarm pmtiles-torrent
|
|
266
|
+
```
|
|
267
|
+
|
|
268
|
+
An install runs the dependency install scripts again, so check WebRTC survived
|
|
269
|
+
it โ see [the allowScripts warning](#the-allowscripts-warning):
|
|
270
|
+
|
|
271
|
+
```sh
|
|
272
|
+
cd /var/lib/pmtiles-swarm && sudo -u pmtiles-swarm -H node -e \
|
|
273
|
+
"import('node-datachannel').then(() => console.log('webrtc ok'))"
|
|
274
|
+
```
|
|
275
|
+
|
|
276
|
+
To pin a version, or to go back to one:
|
|
277
|
+
|
|
278
|
+
```sh
|
|
279
|
+
sudo -u pmtiles-swarm -H npm install --prefix /var/lib/pmtiles-swarm pmtiles-swarm@0.3.0
|
|
280
|
+
sudo systemctl restart pmtiles-swarm
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
## Checking it
|
|
284
|
+
|
|
285
|
+
```sh
|
|
286
|
+
sudo systemctl daemon-reload
|
|
287
|
+
sudo systemctl enable --now pmtiles-swarm
|
|
288
|
+
journalctl -u pmtiles-swarm -f
|
|
289
|
+
```
|
|
290
|
+
|
|
291
|
+
A healthy start says which engine came up, how many archives were handed back
|
|
292
|
+
to it, and which ports it is listening on. Two lines are worth reading for:
|
|
293
|
+
|
|
294
|
+
* `could not listen on โฆ port` โ something else holds it, most often a previous
|
|
295
|
+
run that has not finished stopping.
|
|
296
|
+
* `data directory is already in use by pid โฆ` โ one node per data directory,
|
|
297
|
+
enforced with a lock file. Under `Restart=always` this usually means the old
|
|
298
|
+
process outlived `TimeoutStopSec`; raise it rather than removing the lock.
|
|
299
|
+
|
|
300
|
+
Then check it is actually serving:
|
|
301
|
+
|
|
302
|
+
```sh
|
|
303
|
+
curl -fsS localhost:8090/feed.xml >/dev/null && echo "public surface ok"
|
|
304
|
+
curl -fsS localhost:8091/api/status | head -c 200
|
|
305
|
+
```
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pmtiles-swarm",
|
|
3
|
-
"version": "0.3.
|
|
3
|
+
"version": "0.3.2",
|
|
4
4
|
"description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "src/index.js",
|
|
@@ -10,6 +10,7 @@
|
|
|
10
10
|
"files": [
|
|
11
11
|
"src",
|
|
12
12
|
"docs",
|
|
13
|
+
"swarm.config.json.sample",
|
|
13
14
|
"NOTICE.md",
|
|
14
15
|
"CHANGELOG.md"
|
|
15
16
|
],
|
|
@@ -38,7 +39,7 @@
|
|
|
38
39
|
"maplibre-gl": "^6.2.0",
|
|
39
40
|
"parse-torrent": "^11.0.24",
|
|
40
41
|
"pmtiles": "^4.4.1",
|
|
41
|
-
"pmtiles-torrent": "^0.
|
|
42
|
+
"pmtiles-torrent": "^0.3.0",
|
|
42
43
|
"webtorrent": "^3.0.21"
|
|
43
44
|
},
|
|
44
45
|
"engines": {
|
package/src/api.js
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
import crypto from 'node:crypto';
|
|
2
|
+
import fsSync from 'node:fs';
|
|
2
3
|
import fs from 'node:fs/promises';
|
|
3
4
|
import { createRequire } from 'node:module';
|
|
4
5
|
import path from 'node:path';
|
|
@@ -24,6 +25,17 @@ import { TileReadError } from './tiles.js';
|
|
|
24
25
|
|
|
25
26
|
const here = path.dirname(fileURLToPath(import.meta.url));
|
|
26
27
|
|
|
28
|
+
/**
|
|
29
|
+
* This package's version, for the console to show.
|
|
30
|
+
*
|
|
31
|
+
* Read from package.json rather than repeated in a constant: a version in two
|
|
32
|
+
* places is a version that disagrees with itself, and the one on screen is the
|
|
33
|
+
* one somebody will quote when reporting a problem.
|
|
34
|
+
*/
|
|
35
|
+
const VERSION = JSON.parse(
|
|
36
|
+
fsSync.readFileSync(path.join(here, '..', 'package.json'), 'utf8'),
|
|
37
|
+
).version;
|
|
38
|
+
|
|
27
39
|
/**
|
|
28
40
|
* Wraps an async route so a rejection becomes a 500 rather than an unhandled
|
|
29
41
|
* rejection that takes the process down.
|
|
@@ -395,6 +407,7 @@ export function createApp({
|
|
|
395
407
|
engineError = error.message;
|
|
396
408
|
}
|
|
397
409
|
res.json({
|
|
410
|
+
version: VERSION,
|
|
398
411
|
engine: { name: engine.name, ok: engineOk, error: engineError },
|
|
399
412
|
archives: catalog.list().length,
|
|
400
413
|
categories: catalog.categories(),
|
package/src/config.js
CHANGED
|
@@ -570,6 +570,15 @@ const DEFAULTS = {
|
|
|
570
570
|
* because splicing two builds together produces a torrent for bytes that
|
|
571
571
|
* never existed.
|
|
572
572
|
*/
|
|
573
|
+
/**
|
|
574
|
+
* How often to write resume data, in seconds. Zero disables it.
|
|
575
|
+
*
|
|
576
|
+
* Resume data is what lets a restart skip re-hashing the store โ on an
|
|
577
|
+
* 800 GB archive, the difference between instant and half an hour. A clean
|
|
578
|
+
* stop always writes it; this is for the stops that are not clean, where
|
|
579
|
+
* everything since the last write has to be checked again.
|
|
580
|
+
*/
|
|
581
|
+
resumeSaveIntervalSeconds: 300,
|
|
573
582
|
fetchAttempts: 10,
|
|
574
583
|
/** How long to wait before resuming a download that stopped. */
|
|
575
584
|
fetchRetrySeconds: 5,
|
|
@@ -779,6 +788,22 @@ export async function loadConfig(configPath) {
|
|
|
779
788
|
path: path.resolve(base, entry.path),
|
|
780
789
|
}));
|
|
781
790
|
|
|
791
|
+
// Every remaining path, resolved the same way as the rest.
|
|
792
|
+
//
|
|
793
|
+
// These two were left relative, which means relative to the working
|
|
794
|
+
// directory rather than to the file they are written in. Started by hand
|
|
795
|
+
// from the repository those agree, so it never showed; under systemd the
|
|
796
|
+
// working directory defaults to `/`, so `./data/resume` becomes
|
|
797
|
+
// `/data/resume` โ somewhere the service almost certainly cannot write, for
|
|
798
|
+
// a reason nothing in the config hints at.
|
|
799
|
+
config.locations = (config.locations ?? []).map((entry) => ({
|
|
800
|
+
...entry,
|
|
801
|
+
path: entry.path ? path.resolve(base, entry.path) : entry.path,
|
|
802
|
+
}));
|
|
803
|
+
if (config.libtorrent?.resumeDir) {
|
|
804
|
+
config.libtorrent.resumeDir = path.resolve(base, config.libtorrent.resumeDir);
|
|
805
|
+
}
|
|
806
|
+
|
|
782
807
|
return config;
|
|
783
808
|
}
|
|
784
809
|
|
package/src/index.js
CHANGED
|
@@ -321,6 +321,20 @@ PMTILES_SWARM_PUBLIC_URL
|
|
|
321
321
|
sources.start();
|
|
322
322
|
seeding.start();
|
|
323
323
|
speed.start();
|
|
324
|
+
|
|
325
|
+
// Resume data on a timer as well as at shutdown. A clean stop writes it; a
|
|
326
|
+
// kill, a crash or a power cut does not, and whatever is lost is re-hashed
|
|
327
|
+
// on the way back up.
|
|
328
|
+
let resumeTimer;
|
|
329
|
+
const resumeSeconds = config.resumeSaveIntervalSeconds ?? 300;
|
|
330
|
+
if (resumeSeconds > 0 && engine.saveResume) {
|
|
331
|
+
resumeTimer = setInterval(() => {
|
|
332
|
+
engine
|
|
333
|
+
.saveResume()
|
|
334
|
+
.catch((error) => console.warn(`[resume] could not save: ${error.message}`));
|
|
335
|
+
}, resumeSeconds * 1000);
|
|
336
|
+
resumeTimer.unref?.();
|
|
337
|
+
}
|
|
324
338
|
hooks.start();
|
|
325
339
|
completion.start();
|
|
326
340
|
|
|
@@ -352,6 +366,7 @@ PMTILES_SWARM_PUBLIC_URL
|
|
|
352
366
|
}
|
|
353
367
|
|
|
354
368
|
stoppers.unshift(
|
|
369
|
+
{ label: 'resume timer', stop: () => resumeTimer && clearInterval(resumeTimer), ms: 500 },
|
|
355
370
|
{ label: 'origin checks', stop: () => originTimer && clearInterval(originTimer), ms: 500 },
|
|
356
371
|
{ label: 'schedulers', stop: () => {
|
|
357
372
|
sources.stop();
|
package/src/web/index.html
CHANGED
|
@@ -254,6 +254,20 @@
|
|
|
254
254
|
}
|
|
255
255
|
.choice > span { flex: 1; }
|
|
256
256
|
|
|
257
|
+
footer {
|
|
258
|
+
margin-top: 2.5rem;
|
|
259
|
+
padding: 1rem 0 0.5rem;
|
|
260
|
+
border-top: 1px solid var(--line);
|
|
261
|
+
color: var(--muted);
|
|
262
|
+
font-size: 0.8rem;
|
|
263
|
+
display: flex;
|
|
264
|
+
gap: 0.6rem;
|
|
265
|
+
flex-wrap: wrap;
|
|
266
|
+
align-items: baseline;
|
|
267
|
+
}
|
|
268
|
+
footer a { color: var(--muted); }
|
|
269
|
+
footer .version { font-variant-numeric: tabular-nums; }
|
|
270
|
+
|
|
257
271
|
/* Speed limits. Declared after .field on purpose: `.field input` is
|
|
258
272
|
width:100%, which would push the unit label onto its own line. */
|
|
259
273
|
.rates { display: flex; gap: 1rem; flex-wrap: wrap; }
|
|
@@ -362,6 +376,14 @@
|
|
|
362
376
|
</section>
|
|
363
377
|
</main>
|
|
364
378
|
|
|
379
|
+
<footer>
|
|
380
|
+
<span>© <span id="year"></span> TechIdiots LLC</span>
|
|
381
|
+
<span>ยท</span>
|
|
382
|
+
<span>pmtiles-swarm <span class="version" id="version">โฆ</span></span>
|
|
383
|
+
<span>ยท</span>
|
|
384
|
+
<a href="https://github.com/TechIdiots-LLC/pmtiles-swarm" rel="noreferrer">source</a>
|
|
385
|
+
</footer>
|
|
386
|
+
|
|
365
387
|
<dialog id="move-dialog">
|
|
366
388
|
<form method="dialog" id="move-form">
|
|
367
389
|
<h2>Set location</h2>
|
|
@@ -859,6 +881,7 @@
|
|
|
859
881
|
]);
|
|
860
882
|
renderSpeed(speed);
|
|
861
883
|
renderFetching(adds?.running);
|
|
884
|
+
if (status.version) $('version').textContent = `v${status.version}`;
|
|
862
885
|
const engine = status.engine;
|
|
863
886
|
$('status').innerHTML =
|
|
864
887
|
`engine <b>${engine.name}</b> ${engine.ok ? 'ready' : 'unavailable'}` +
|
|
@@ -3436,6 +3459,11 @@
|
|
|
3436
3459
|
if (name === 'settings') loadSettings().catch((e) => toast(e.message));
|
|
3437
3460
|
if (name === 'categories') loadCategories().catch((e) => toast(e.message));
|
|
3438
3461
|
};
|
|
3462
|
+
// The footer's year, set from the clock rather than typed into a file
|
|
3463
|
+
// nobody will remember to edit. The version beside it arrives with the
|
|
3464
|
+
// first status, and reads "โฆ" until then.
|
|
3465
|
+
$('year').textContent = String(new Date().getFullYear());
|
|
3466
|
+
|
|
3439
3467
|
$('tab-archives').onclick = () => showTab('archives');
|
|
3440
3468
|
$('tab-categories').onclick = () => showTab('categories');
|
|
3441
3469
|
$('tab-settings').onclick = () => showTab('settings');
|
|
@@ -0,0 +1,86 @@
|
|
|
1
|
+
{
|
|
2
|
+
"port": 8090,
|
|
3
|
+
"adminPort": 8091,
|
|
4
|
+
"adminHost": "127.0.0.1",
|
|
5
|
+
"publicUrl": "https://maps.example.org",
|
|
6
|
+
|
|
7
|
+
"dataDir": "./data",
|
|
8
|
+
"savePath": "./data/torrents-data",
|
|
9
|
+
"savePathLayout": "infohash",
|
|
10
|
+
"locations": [
|
|
11
|
+
{ "name": "default", "path": "./data/torrents-data" },
|
|
12
|
+
{ "name": "bulk storage", "path": "/EDIT-ME/bulk/archives" }
|
|
13
|
+
],
|
|
14
|
+
|
|
15
|
+
"engine": "libtorrent",
|
|
16
|
+
"secondaryEngines": ["webtorrent"],
|
|
17
|
+
"libtorrent": {
|
|
18
|
+
"python": "python3",
|
|
19
|
+
"resumeDir": "./data/resume",
|
|
20
|
+
"listen": "0.0.0.0:6881",
|
|
21
|
+
"upnp": true,
|
|
22
|
+
"natpmp": true
|
|
23
|
+
},
|
|
24
|
+
"webtorrent": {
|
|
25
|
+
"clientOptions": { "torrentPort": 6882 }
|
|
26
|
+
},
|
|
27
|
+
|
|
28
|
+
"torrentFormat": "hybrid",
|
|
29
|
+
"pieceLength": 4194304,
|
|
30
|
+
"maxConnections": 100,
|
|
31
|
+
"trackers": [
|
|
32
|
+
"udp://tracker.opentrackr.org:1337/announce",
|
|
33
|
+
"udp://tracker.torrent.eu.org:451/announce",
|
|
34
|
+
"wss://tracker.openwebtorrent.com"
|
|
35
|
+
],
|
|
36
|
+
|
|
37
|
+
"speed": {
|
|
38
|
+
"uploadLimit": 0,
|
|
39
|
+
"downloadLimit": 0,
|
|
40
|
+
"alternative": { "uploadLimit": 2097152, "downloadLimit": 20971520 },
|
|
41
|
+
"schedule": { "enabled": false, "from": "08:00", "to": "23:00", "days": "everyday" }
|
|
42
|
+
},
|
|
43
|
+
"seeding": { "ratio": null, "minutes": null, "then": "stop" },
|
|
44
|
+
|
|
45
|
+
"feedTitle": "Example map archives",
|
|
46
|
+
"feedCopyright": "ยฉ OpenStreetMap contributors",
|
|
47
|
+
"feedMaxItems": 50,
|
|
48
|
+
|
|
49
|
+
"watch": [
|
|
50
|
+
{
|
|
51
|
+
"path": "/EDIT-ME/maps/generated",
|
|
52
|
+
"categories": ["basemaps"],
|
|
53
|
+
"webSeedBase": "https://maps.example.org/files",
|
|
54
|
+
"stabilitySeconds": 30,
|
|
55
|
+
"pollSeconds": 0
|
|
56
|
+
}
|
|
57
|
+
],
|
|
58
|
+
|
|
59
|
+
"sources": [
|
|
60
|
+
{
|
|
61
|
+
"name": "daily-planet",
|
|
62
|
+
"url": "https://build.example.org/{YYYYMMDD}.pmtiles",
|
|
63
|
+
"filename": "planet-{YYYYMMDD}.pmtiles",
|
|
64
|
+
"categories": ["planet"],
|
|
65
|
+
"offsetDays": -1,
|
|
66
|
+
"lookbackDays": 0,
|
|
67
|
+
"at": "10:00",
|
|
68
|
+
"keep": 2,
|
|
69
|
+
"webSeed": true,
|
|
70
|
+
"latestLink": "planet-latest.pmtiles"
|
|
71
|
+
}
|
|
72
|
+
],
|
|
73
|
+
|
|
74
|
+
"subscriptions": [
|
|
75
|
+
{ "url": "https://peer.example.org/api/catalog", "protocol": "api", "mode": "cache" }
|
|
76
|
+
],
|
|
77
|
+
|
|
78
|
+
"tiles": { "sparse": null },
|
|
79
|
+
|
|
80
|
+
"auth": {
|
|
81
|
+
"apiKey": "REPLACE-WITH-A-LONG-RANDOM-STRING",
|
|
82
|
+
"username": "admin",
|
|
83
|
+
"password": "REPLACE-ME",
|
|
84
|
+
"tokens": []
|
|
85
|
+
}
|
|
86
|
+
}
|