pmtiles-swarm 0.2.0 β 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +678 -0
- package/NOTICE.md +18 -0
- package/README.md +485 -25
- package/docs/architecture-diagram.md +47 -4
- package/docs/engines.md +214 -2
- package/docs/publishing.md +349 -17
- package/docs/security.md +214 -0
- package/docs/serving-tiles.md +184 -1
- package/docs/subscribing.md +178 -2
- package/package.json +6 -4
- package/src/api.js +1431 -30
- package/src/auth.js +523 -0
- package/src/catalog.js +67 -6
- package/src/config.js +827 -8
- package/src/engines/composite.js +514 -0
- package/src/engines/libtorrent.js +94 -0
- package/src/engines/qbittorrent.js +77 -0
- package/src/engines/types.js +7 -0
- package/src/engines/webtorrent.js +395 -12
- package/src/feed.js +10 -2
- package/src/hooks.js +231 -0
- package/src/identify.js +146 -0
- package/src/incomplete.js +289 -0
- package/src/index.js +236 -28
- package/src/library.js +1833 -53
- package/src/locations.js +215 -0
- package/src/lock.js +179 -0
- package/src/pieces.js +94 -0
- package/src/pmtiles-probe.js +43 -34
- package/src/rate-limits.js +269 -0
- package/src/restart.js +79 -0
- package/src/seeding.js +273 -0
- package/src/shutdown.js +138 -0
- package/src/sources.js +553 -38
- package/src/subscriptions.js +219 -13
- package/src/tiles.js +90 -5
- package/src/torrent-create.js +336 -30
- package/src/watch.js +45 -13
- package/src/web/index.html +3387 -130
- package/src/web/preview.html +218 -0
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,684 @@
|
|
|
7
7
|
### π Bug fixes
|
|
8
8
|
- _...Add new stuff here..._
|
|
9
9
|
|
|
10
|
+
## 0.3.0
|
|
11
|
+
### β¨ Features and improvements
|
|
12
|
+
- **The peers column distinguishes who is connected from what the swarm holds.** `0 / 2` on a
|
|
13
|
+
complete, seeding archive is correct β the counts are remote clients only, and a client is never
|
|
14
|
+
its own peer β but it reads like a fault. The tracker's own totals now follow in parentheses, in
|
|
15
|
+
qBittorrent's notation, and the cell explains itself on hover. Nothing is shown until a tracker
|
|
16
|
+
has actually answered, since claiming an empty swarm on no information is worse than saying
|
|
17
|
+
nothing.
|
|
18
|
+
- **An archive fetched from a URL is filed under its infohash like every other.** It could not be
|
|
19
|
+
before: the infohash is computed from the bytes, which are the thing still arriving, so a
|
|
20
|
+
scheduled download landed in the root of the save path while everything else sat under its own
|
|
21
|
+
directory β reintroducing exactly the collision that layout exists to prevent, since two sources
|
|
22
|
+
publishing `planet.pmtiles` would write into one file. It now downloads into a randomly named
|
|
23
|
+
directory under `<savePath>/.incoming/` and is moved into place once the torrent has been hashed.
|
|
24
|
+
The move is a rename within one filesystem, so it is instant whatever the archive weighs, and the
|
|
25
|
+
random name keeps two in-flight downloads of the same filename apart. A download interrupted by a
|
|
26
|
+
kill leaves its directory behind and the next start clears it.
|
|
27
|
+
- **A watched location can keep only the newest few builds.** `keep` on a source, and **Builds to
|
|
28
|
+
keep** in the console. Each build is a whole archive, so a daily 137 GB planet build kept for
|
|
29
|
+
ever fills any disk within the week. It deletes the data of what it retires, so it is off unless
|
|
30
|
+
set, and it only ever touches archives that same named source imported β anything added by hand,
|
|
31
|
+
adopted from a client, or taken from a peer is never considered. Sources can also carry their own
|
|
32
|
+
`seeding` limit, since a daily build that has done its share deserves different treatment from
|
|
33
|
+
the only copy of something.
|
|
34
|
+
- **"Newest" now means the newest build, not the most recently added archive.** The two disagree
|
|
35
|
+
and can be opposite: a poll takes candidates newest first, so importing several at once gives the
|
|
36
|
+
newest build the *earliest* arrival time. `/latest/<category>` and the category feeds ordered by
|
|
37
|
+
arrival, which would have served the oldest of a batch, and a retention policy ordered the same
|
|
38
|
+
way would have deleted the newest. Entries record the date of the build they are, and one
|
|
39
|
+
comparison in the catalog answers it for both.
|
|
40
|
+
- **The global seeding limit has real fields.** It was editable only as a raw JSON textarea among
|
|
41
|
+
every other object setting, which is not a way to ask someone for a ratio. It now has the same
|
|
42
|
+
shape as the per-archive dialog that already existed.
|
|
43
|
+
- **A download that stops is resumed, not restarted.** A planet archive is hours of transfer and a
|
|
44
|
+
connection that drops partway is ordinary; until now that threw away everything transferred and
|
|
45
|
+
began again, repeatedly. Each attempt now continues from the bytes already on disk with an HTTP
|
|
46
|
+
range request β `fetchAttempts` and `fetchRetrySeconds` β so a drop costs the retry delay rather
|
|
47
|
+
than 49 GB. Appending only happens when it is provably safe: the response must be a 206 (a server
|
|
48
|
+
that ignores `Range` answers 200 with the whole file, and appending that gives a file that is
|
|
49
|
+
part duplicate), the ETag or Last-Modified must be unchanged (resuming across a new build splices
|
|
50
|
+
the head of one onto the tail of another), and `Content-Range` must begin where it was asked to.
|
|
51
|
+
Any of those failing restarts the download, as does a server offering no validator at all β
|
|
52
|
+
fetching an archive twice is expensive, but publishing a torrent for bytes that never existed
|
|
53
|
+
anywhere hashes perfectly well here and fails for every peer that tries it.
|
|
54
|
+
- **Downloads that have no torrent yet are visible.** An archive added from a URL is fetched whole
|
|
55
|
+
before there is anything to hash a torrent out of, so until that finishes there is no catalog
|
|
56
|
+
entry and nothing in the table β for a planet build, hours in which a watched location looks like
|
|
57
|
+
it silently did nothing. They now appear under the archive list with progress and a cancel
|
|
58
|
+
button. `/api/adds` reports bytes and totals rather than bare URLs.
|
|
59
|
+
- **A watched web location can say whether its URL is published as a web seed.** `webSeed` on a
|
|
60
|
+
source, and **Use URL as web seed** on each row in the console. The behaviour was already the
|
|
61
|
+
right default β the origin is a valid web seed for exactly those bytes, and publishing it is the
|
|
62
|
+
single biggest difference to a cold start β but it was not settable per source, and there are two
|
|
63
|
+
reasons to change it. An upstream that deletes old builds leaves a URL that outlives the file it
|
|
64
|
+
points at, so every peer that tries it fails; and where the archive also sits on public storage
|
|
65
|
+
under another address, `webSeeds` names that instead and keeps the fetch URL private. A URL that
|
|
66
|
+
appears to carry credentials is still never published unless `webSeed: true` says so explicitly,
|
|
67
|
+
because a torrent goes to the swarm and cannot be recalled.
|
|
68
|
+
- **Piece maps.** A **Pieces** tab showing what this node holds, how rare each piece is across the
|
|
69
|
+
swarm, and what each connected peer has β plus `firstPiece` and `pieceCount` per file on
|
|
70
|
+
`/content`, which need no engine at all, since a torrent is one byte stream cut into equal pieces
|
|
71
|
+
and a file's offset already says which it occupies. Worth more here than in an ordinary client: a
|
|
72
|
+
cache-mode archive holds a scatter of pieces on purpose, so the bar is a picture of what has been
|
|
73
|
+
*viewed* rather than a progress indicator. Maps arrive bucketed to the width they will be drawn
|
|
74
|
+
at, each reduced for the question its bar answers β held counts only when every piece in a column
|
|
75
|
+
is (or a 60%-complete archive paints as almost solid), availability takes the *rarest* (one piece
|
|
76
|
+
nobody has is the answer to "can this be completed"), and a peer's map takes *any* (a peer
|
|
77
|
+
holding part of a column can still serve it). Supported by libtorrent **and WebTorrent**, whose
|
|
78
|
+
`torrent.bitfield` and per-wire `peerPieces` carry the same information; qBittorrent's API has
|
|
79
|
+
piece states but neither availability nor per-peer maps, so it is refused rather than half-drawn.
|
|
80
|
+
- **Speed limits, with a schedule.** Two sets of global limits and a window that swaps them,
|
|
81
|
+
modelled on qBittorrent: `speed.uploadLimit` / `downloadLimit`, `speed.alternative`, and
|
|
82
|
+
`speed.schedule` taking `from`, `to` and `days` (`everyday`, `weekdays`, `weekends`, or weekday
|
|
83
|
+
numbers). A window whose end is before its start wraps past midnight, so `22:00`β`06:00` is one
|
|
84
|
+
overnight window rather than an empty one, and `days` picks the night it opens. The console has
|
|
85
|
+
the settings and a header switch that forces either set, handing control back to the schedule the
|
|
86
|
+
next time the window itself changes β so forcing "slow" at lunchtime does not leave the node
|
|
87
|
+
throttled tomorrow. Applied live, enforced by whichever engines can throttle, and applied whole
|
|
88
|
+
to each rather than divided between them, since they share one uplink.
|
|
89
|
+
- **A listen failure is reported rather than thrown.** The two `server.on('error')` registrations
|
|
90
|
+
had been spliced into the middle of the watch-folder reloader, so nothing was listening for the
|
|
91
|
+
event at startup: a port taken between the pre-flight check and the actual bind produced a raw
|
|
92
|
+
stack trace instead of the sentence explaining it, and every settings reload added two more
|
|
93
|
+
listeners.
|
|
94
|
+
- **libtorrent's network settings are configurable.** The sidecar has always accepted `upnp`,
|
|
95
|
+
`natpmp`, `dht`, `lsd`, `uploadLimit` and `downloadLimit`, and nothing passed them β so a node
|
|
96
|
+
could not decline UPnP however the config was written. That is the wrong default on a network
|
|
97
|
+
where port forwards are made by hand: the router has UPnP off deliberately and the client fails
|
|
98
|
+
at it quietly on every start. Unset keys still take libtorrent's own defaults.
|
|
99
|
+
- **An archive that is not whole yet is named so.** It downloads as
|
|
100
|
+
`planet.pmtiles.incomplete` and is renamed the instant it finishes. These files get published:
|
|
101
|
+
a web seed URL is predictable and goes out before the file exists, so an unmarked partial in a
|
|
102
|
+
served directory is a URL that answers with half an archive, and every peer that tries it fails
|
|
103
|
+
hash verification. Now it 404s until the file is real. The rename is inside one directory, so it
|
|
104
|
+
is atomic and instant at any size β where moving between directories is instant only when they
|
|
105
|
+
share a filesystem, and otherwise copies the whole archive. Remote downloads are marked the same
|
|
106
|
+
way, qBittorrent's own `.!qB` preference is turned on rather than overridden, and
|
|
107
|
+
`incompleteSuffix: ""` switches the whole thing off.
|
|
108
|
+
- **`cacheSavePath` is now off by default.** It existed to tell whole archives from partial ones on
|
|
109
|
+
disk, which the name above does better; it stays as a placement choice for putting cache pieces
|
|
110
|
+
on faster disk. Archives already in a catalog keep the save path they were added with.
|
|
111
|
+
- **A Categories screen in the console**, listing every tag with the endpoints that resolve to its
|
|
112
|
+
newest build β TileJSON, `.torrent`, magnet, feed and latest-only feed β each copyable. Backed by
|
|
113
|
+
a new `GET /api/categories`. A category whose newest archive is not PMTiles gets everything
|
|
114
|
+
except the tile endpoint.
|
|
115
|
+
- **Categories can be changed after an archive is added**, from its detail panel or at
|
|
116
|
+
`PATCH /api/torrents/{infohash}/categories` (whole list, or `add`/`remove` one at a time). They
|
|
117
|
+
could only be set at the moment of adding, which is the wrong time to have to know: a build
|
|
118
|
+
becomes `weekly` once there is a second one, and an archive is marked for sharing long after it
|
|
119
|
+
arrives.
|
|
120
|
+
- **Monitored folders and watched web locations are editable in Settings**, as tables rather than
|
|
121
|
+
a textarea full of JSON β the shape a torrent client gives a grid for. Folders take categories, a
|
|
122
|
+
save location, a publish directory and a web seed base; web locations take a URL template or a
|
|
123
|
+
directory to list.
|
|
124
|
+
- **The date in a watched URL is built by clicking, not by remembering.** Paste the URL of a
|
|
125
|
+
recent build, select the date in it and click a token β the token replaces what is selected.
|
|
126
|
+
A `{...}` group is now read as a date *pattern* rather than matched against a fixed list of
|
|
127
|
+
spellings, so it can say whatever the upstream says: `{M}-{D}-{YY}` gives `8-7-26`,
|
|
128
|
+
`{DD.MM.YYYY}` gives `07.08.2026`, `{YY}` gives `26`. Run length decides padding β `MM` is
|
|
129
|
+
padded, `M` is not β and case is ignored, since using case for padding as well would make `{m}`
|
|
130
|
+
and `{M}` differ with nothing to see. A group that is not a date is left exactly as found, so a
|
|
131
|
+
URL containing `{id}` is not quietly rewritten. Every spelling that worked before still does. Day offset and look-back are columns of their own
|
|
132
|
+
(protomaps publishes yesterday's build, so it wants `-1`), and Preview refuses to run on a URL
|
|
133
|
+
that still has a fixed date in it, since that would ask for the same build forever.
|
|
134
|
+
- **`onAdded`, beside the existing `onComplete`** β the same pair a torrent client offers, and
|
|
135
|
+
different moments: an archive joined in cache mode is added and will never be complete, while one
|
|
136
|
+
built here is both at once. Both are now shown in Settings under **Run external program**, with
|
|
137
|
+
the full placeholder list, laid out the way a client lays it out.
|
|
138
|
+
- **`allowHooksFromApi`.** The hooks stayed config-file-only because a token that manages torrents
|
|
139
|
+
becoming one that runs arbitrary commands as the service user is a large step to take by
|
|
140
|
+
accident β but a setting nobody can find is not much safer than one anybody can change, it is
|
|
141
|
+
just harder to use. The panel is read-only until this is set in the config file, where a token
|
|
142
|
+
cannot reach, and says so.
|
|
143
|
+
- **Adopting is a dialog now**, like adding. It lists what an engine holds that this node does not
|
|
144
|
+
yet know about β name, size, progress, format β and lets you pick, rather than importing
|
|
145
|
+
everything and reading afterwards what it did. Categories can be applied to the lot. It can also
|
|
146
|
+
adopt from **a qBittorrent instance other than the configured engine**, which is what "adopt
|
|
147
|
+
existing" sounded like it did.
|
|
148
|
+
- **Startup refuses to run two nodes over one data directory, and checks its ports first.** The
|
|
149
|
+
port is the symptom people notice; the data directory is the one that costs something, since the
|
|
150
|
+
catalog is rewritten whole by each node and the last writer silently wins. Both are checked
|
|
151
|
+
before an engine is connected or a library restored, and both explain what to change. A lock left
|
|
152
|
+
by a node that was killed rather than stopped is taken over rather than needing to be deleted.
|
|
153
|
+
- **The console and the API can have a port of their own.** `adminPort`, with an optional
|
|
154
|
+
`adminHost`, leaves tiles, TileJSON, `.torrent` files, the feeds, the `latest` endpoints and
|
|
155
|
+
`/api/catalog` on the public port and moves everything else. The public port can then face the
|
|
156
|
+
internet while the admin one is bound to loopback β so the thing that can rewrite the
|
|
157
|
+
configuration is unreachable rather than merely guarded, which is a statement a firewall can
|
|
158
|
+
enforce. On the public listener the admin surface answers 404 rather than 403, because a refusal
|
|
159
|
+
confirms there is something behind it. Routing is by the port a request arrived on, never by a
|
|
160
|
+
header, since a header is something the caller controls. The refusal to start unauthenticated now
|
|
161
|
+
reads the admin interface rather than the public one, because tiles on `0.0.0.0` is the point of
|
|
162
|
+
the tiles.
|
|
163
|
+
- **Torrents are created hybrid v1+v2 wherever libtorrent is present** β as the primary or
|
|
164
|
+
merely as a secondary, since what matters is that it is there at all. A hybrid is not a
|
|
165
|
+
trade-off: v2 clients gain per-file merkle trees over 16 KiB leaves, which is exactly the shape
|
|
166
|
+
of a tile read, and v1 clients see an ordinary torrent. `torrentFormat` takes `hybrid`, `v1` or
|
|
167
|
+
`v2`, and a node with no libtorrent falls back to v1 rather than failing. Previously every
|
|
168
|
+
torrent was v1 whatever the engine, and the docs said otherwise.
|
|
169
|
+
- **Two engines can run at once.** `secondaryEngines: ["webtorrent"]` beside a libtorrent or
|
|
170
|
+
qBittorrent primary β the arrangement the docs had been recommending without any code to do it,
|
|
171
|
+
which until now meant two processes and two catalogues. libtorrent handles the bulk and speaks
|
|
172
|
+
BitTorrent v2; WebTorrent is the only one that can talk to a browser. One rule keeps it safe:
|
|
173
|
+
only the primary writes, so a secondary is handed an archive only once it is complete and never
|
|
174
|
+
in cache mode β two clients writing one incomplete file produce a file neither one's bitfield
|
|
175
|
+
describes. Progress and state come from the primary; peers, seeds and speeds are added together.
|
|
176
|
+
A secondary that will not start is a warning, not a failure.
|
|
177
|
+
- **A map preview for every archive**, at `/archives/{infohash}/preview` and behind an
|
|
178
|
+
**Inspect** or **Preview** button in the detail panel. Vector archives get an inspector: each
|
|
179
|
+
declared layer drawn in a colour derived from its name, toggleable, with click-to-see-properties
|
|
180
|
+
on the features under the cursor, using MapLibre's own `@maplibre/maplibre-gl-inspect` β it is
|
|
181
|
+
maintained alongside the renderer, so it keeps working across major versions without this having
|
|
182
|
+
to notice. Raster archives get the raster. It is built from the archive's
|
|
183
|
+
own TileJSON, which is already a complete source description β nothing is reconstructed. No
|
|
184
|
+
symbol layers and no glyphs, since an archive carries tiles and not fonts. Both libraries are
|
|
185
|
+
ordinary dependencies served out of `node_modules`, the way tileserver-gl does it, so a node on
|
|
186
|
+
an internal network can render its own previews.
|
|
187
|
+
- **Ratio and Expires columns**, so a seeding limit can be seen coming rather than noticed
|
|
188
|
+
afterwards. Expires counts down a time limit β `42d 1h` β and says `β` where nothing applies,
|
|
189
|
+
with the reason on hover: a cache-mode archive, or one told to seed forever. A ratio target is
|
|
190
|
+
reported as progress towards a number rather than as a duration, because how long it takes
|
|
191
|
+
depends on how fast peers happen to be downloading, and the ratio is coloured as it approaches
|
|
192
|
+
the point where it would remove the archive. The detail panel carries the same countdown beside
|
|
193
|
+
the limit in effect.
|
|
194
|
+
- **A move checks there is room first**, before the engine is disturbed β running out of disk
|
|
195
|
+
halfway through several hundred gigabytes means an hour spent, a partial file to clean up and an
|
|
196
|
+
archive to put back. Only when it will actually be a copy: a move within one filesystem is a
|
|
197
|
+
rename and needs no free space at all, so checking unconditionally would refuse moves that would
|
|
198
|
+
have worked. A filesystem that will not report its free space is gone ahead with rather than
|
|
199
|
+
refused. Free space is shown beside each save location in the picker, including for a directory
|
|
200
|
+
that has not been created yet.
|
|
201
|
+
- **An archive's data can be moved after the fact** β **Set locationβ¦** in its detail panel, or
|
|
202
|
+
`PATCH /api/torrents/{infohash}/location`. The engine is told to let go, the file is moved, and
|
|
203
|
+
the torrent handed back pointed at the new path. Within one filesystem that is a rename and
|
|
204
|
+
finishes at once; across two it is a real copy, so it runs in the background and reports
|
|
205
|
+
progress rather than holding a request open for an hour, and the original is removed only after
|
|
206
|
+
the copy has been checked. An unfinished archive moves under the name it actually has, marker
|
|
207
|
+
and all.
|
|
208
|
+
- **`savePathLayout: "infohash"`**, giving each joined archive `<savePath>/<infohash>/` to itself.
|
|
209
|
+
Filenames are not unique β two builds of the same map are both `planet.pmtiles` β and this is the
|
|
210
|
+
only arrangement in which that can never matter. Flat stays the default, because the collision is
|
|
211
|
+
now refused outright when the second archive is added, and flat is what makes dropping a finished
|
|
212
|
+
archive in before adding its torrent work. Works from a bare magnet, since the infohash is the
|
|
213
|
+
one thing a magnet always carries. Archives created here are unaffected, and web seed URLs are
|
|
214
|
+
built from the published location rather than the save path, so they keep their shape.
|
|
215
|
+
- **Named save locations.** Everything used to land in one place. Name the others under
|
|
216
|
+
`locations` in Settings and they are offered wherever something is added β the add dialog, the
|
|
217
|
+
adopt dialog, each monitored folder and each watched web location β alongside the default and a
|
|
218
|
+
path given outright. qBittorrent hangs the save path off the category, which cannot work here:
|
|
219
|
+
an archive can carry several categories on purpose, and two of them naming two disks is a
|
|
220
|
+
question with no right answer. So the location is chosen rather than derived. The directory is
|
|
221
|
+
created and checked when it is chosen rather than when the first byte arrives, and a name this
|
|
222
|
+
node does not know is refused with the ones it does, since falling back quietly would put
|
|
223
|
+
several hundred gigabytes somewhere other than where it was asked for.
|
|
224
|
+
- **Most settings no longer need a restart, and there is a button for the ones that do.** Changing
|
|
225
|
+
the watched folders means restarting the watchers, not the node; the same goes for hooks, web
|
|
226
|
+
locations, remote nodes, seeding limits and the completion watcher. Those are applied on Save and
|
|
227
|
+
the console says which subsystem was restarted. What is left genuinely belongs to the process β
|
|
228
|
+
the listening socket, the data directory, the torrent client β and **Save & restart** appears
|
|
229
|
+
only for those. How the node comes back is detected rather than assumed: under systemd, Docker,
|
|
230
|
+
pm2 or Kubernetes it stops, because exiting is the restart there and a replacement would fight
|
|
231
|
+
over the port; started by hand it starts a replacement itself.
|
|
232
|
+
- **Named access tokens, with roles.** `auth.apiKey` was one credential and one power, so letting
|
|
233
|
+
another node follow this one meant handing over the key that can also delete the library. There
|
|
234
|
+
are now as many tokens as you like, each named, each `peer` (reads the catalogue, feeds, tiles
|
|
235
|
+
and torrent files β what a node needs to follow this one) or `admin` (everything). A peer token
|
|
236
|
+
can be narrowed to categories and then sees exactly those and nothing else. Minted in Settings or
|
|
237
|
+
at `POST /api/tokens`, shown once, revoked individually, and each records when it was last used
|
|
238
|
+
so retiring an old one is an informed decision. Only a SHA-256 is stored. The existing `apiKey`
|
|
239
|
+
keeps working and keeps meaning admin.
|
|
240
|
+
- **Adopt can pull from another pmtiles-swarm node**, reading its `/api/catalog` once and letting
|
|
241
|
+
you pick β which is not the same as following it, and is the right shape for "give me that one
|
|
242
|
+
build" rather than "take everything it ever publishes". What the peer already knew comes across
|
|
243
|
+
with it: the archive summary, categories, web seeds and checksum. That is what makes it better
|
|
244
|
+
than pasting the magnet, since a joined magnet has no summary until something reads its header
|
|
245
|
+
out of a swarm it has only just joined, and no web seeds at all.
|
|
246
|
+
- **Adopting across machines joins the swarm instead.** An archive whose data this node cannot read
|
|
247
|
+
β a client on another host, or a path that is not mounted here β used to be unusable, since a
|
|
248
|
+
catalog entry pointing at a file that is not there can never serve a tile. But its infohash is
|
|
249
|
+
right here, and an infohash is all it takes to join the swarm that client is already seeding
|
|
250
|
+
into, so those are joined by magnet as cache or mirror, your choice. Anything readable is still
|
|
251
|
+
adopted where it lies, and neither re-hashed nor re-downloaded.
|
|
252
|
+
- **Remote nodes are editable in Settings**, alongside folders and web locations: feed or catalog
|
|
253
|
+
URL, protocol, whether to take archives as a cache or a mirror, a tag to apply, a name filter, a
|
|
254
|
+
token and the pruning policy. A **Test** button β `POST /api/subscriptions/preview` β reports
|
|
255
|
+
whether the peer is reachable, which protocol it speaks and how many archives it is offering that
|
|
256
|
+
this node could actually take. A feed that 404s and a token the peer rejects both fail silently
|
|
257
|
+
otherwise: nothing arrives, which looks exactly like a peer with nothing new.
|
|
258
|
+
- **Polling can be finer than an hour.** `everyMinutes` on a watched web location, for somewhere a
|
|
259
|
+
build pipeline writes into rather than a daily planet build. And `pollSeconds` on a monitored
|
|
260
|
+
folder, for network shares: SMB and NFS do not deliver the change notifications a local
|
|
261
|
+
filesystem does, so a watch on one can sit silent forever while files arrive. Off by default,
|
|
262
|
+
because on a local folder it is pure waste.
|
|
263
|
+
- **Each watched location says when to check.** `at: "03:30"` β a time of day in UTC, or a list of
|
|
264
|
+
them β for an upstream that publishes on a schedule, or `everyHours` for one that publishes
|
|
265
|
+
whenever it is ready. Polling every six hours from whenever the process started found a daily
|
|
266
|
+
build up to six hours late, and those are hours during which nobody could be seeding it. Sources
|
|
267
|
+
naming neither fall back to `sourceCheckIntervalHours` as before. A source that has never run is
|
|
268
|
+
always due, so a daemon that was down over a scheduled time catches up on start.
|
|
269
|
+
- **A source can watch a directory instead of guessing filenames.** `sources[].index` reads a
|
|
270
|
+
listing β an HTML autoindex or an S3 `ListBucketResult` β filters it and takes the newest match,
|
|
271
|
+
for upstreams whose naming is not predictable enough to write as a template. Only links
|
|
272
|
+
underneath the index URL are followed: a listing is a document from somewhere else, and this node
|
|
273
|
+
is about to download gigabytes from whatever it names and republish the result under its own
|
|
274
|
+
name. `newest` bounds how many are considered and defaults to one.
|
|
275
|
+
- **`POST /api/sources/preview`**, and a Preview button beside each web location, reporting what a
|
|
276
|
+
source would take without taking any of it. A directory URL typed slightly wrong is otherwise
|
|
277
|
+
discovered by watching several hundred gigabytes arrive.
|
|
278
|
+
- Adding a scheduled source no longer needs a restart. The poll timer only started when the list
|
|
279
|
+
was already non-empty, and every pass reads the list fresh.
|
|
280
|
+
- Settings now presents the download options the way a torrent client does: a checkbox for the
|
|
281
|
+
marker, and the separate cache directory as an option that ships off.
|
|
282
|
+
- New `sparse` setting, global with a per-archive override, matching tileserver-gl.
|
|
283
|
+
- Watch folders can move each archive into the directory a web server serves (`publishDir`) and
|
|
284
|
+
advertise that URL as a web seed, rather than assuming the watched folder is already the web
|
|
285
|
+
root.
|
|
286
|
+
- Cache-mode archives can be kept under `cacheSavePath`, separate from mirrors. This began as the
|
|
287
|
+
way to tell whole archives from partial ones on disk; the marker above does that job now, and
|
|
288
|
+
this is a placement choice.
|
|
289
|
+
- **An archive can carry several categories.** A planet build can be both `basemaps` and `weekly`
|
|
290
|
+
without choosing. Feeds match on *any* tag, so it appears in both. Catalogues holding the older
|
|
291
|
+
single `category` string are read as a list of one and normalised on the next write.
|
|
292
|
+
- **Asking for the TileJSON reads the header.** A joined torrent arrives with no summary, because
|
|
293
|
+
at that moment there is nothing to read one from β and it used to stay that way, so the archive
|
|
294
|
+
was permanently unusable as a tile endpoint. The header is now read on demand, which for a
|
|
295
|
+
cache-mode archive means pulling the one piece it lives in, and kept once read. A swarm that has
|
|
296
|
+
not found peers yet says so and suggests trying again, rather than refusing outright.
|
|
297
|
+
- **Pause and resume**, in the console and at `POST /api/torrents/{infohash}/pause`. "Not right
|
|
298
|
+
now" is a different intention from "not any more", and remove was the only way to say either.
|
|
299
|
+
- **Mirror or cache is now a choice you can make, and change.** The add dialog offers it when
|
|
300
|
+
joining a magnet or a `.torrent`, and `PATCH /api/torrents/{infohash}/mode` switches an archive
|
|
301
|
+
afterwards β with buttons in the detail panel. Nothing already downloaded is discarded in
|
|
302
|
+
either direction: going to mirror keeps what the cache accumulated and fills in the rest.
|
|
303
|
+
- **Tabbed detail per archive**, as a torrent client has: General, Trackers, Peers, HTTP sources
|
|
304
|
+
and Content. Trackers are shown in their tiers, files with piece geometry, comment and creator.
|
|
305
|
+
Panes load when first opened. New `trackers` and `content` endpoints back them.
|
|
306
|
+
- **The console shows where each archive came from** β built here, adopted, added by hand, or the
|
|
307
|
+
host of the peer that sent it. Worth showing rather than inferring: an archive taken from a peer
|
|
308
|
+
is one this node seeds and serves under its own name.
|
|
309
|
+
- `prune` gained a `"report"` mode that logs what it would remove and removes nothing, so a new
|
|
310
|
+
peer can be watched before it is trusted. Pruning stays off unless asked for, only ever
|
|
311
|
+
considers archives that peer sent, and never acts on a filtered or partial view.
|
|
312
|
+
- **Optional MD5**, `md5: true` globally or per add, published as `<pmtiles:md5>` in the feed and
|
|
313
|
+
exposed in the API. Not for integrity β the torrent already verifies per piece, which is
|
|
314
|
+
stronger β but for the quick manual check and for tooling that expects a checksum. Off by
|
|
315
|
+
default because on a local file it costs a second read of the whole archive; where the bytes
|
|
316
|
+
are already streaming past it is free.
|
|
317
|
+
- **Run a command when a download finishes.** `onComplete` closes the loop for a build pipeline:
|
|
318
|
+
subscribe to a feed of source data, let the swarm fetch it, start the job that turns it into
|
|
319
|
+
something worth publishing. Placeholders match a torrent client's, so an existing
|
|
320
|
+
`torrent_finished.sh` keeps working. Command and arguments are separate rather than one shell
|
|
321
|
+
string, so a name with spaces stays one argument. Configurable from the config file only β
|
|
322
|
+
never through the API, since a token that manages archives should not also choose what code
|
|
323
|
+
runs as the service user.
|
|
324
|
+
- **A stable URL for the current build.** `/latest/{category}/tiles.json`, plus `archive.torrent`,
|
|
325
|
+
`magnet` and an `.xml` feed of just the newest. A style can point at one and survive every
|
|
326
|
+
rebuild. The tiles it names stay infohash URLs, so they remain immutable and cacheable for a
|
|
327
|
+
year β this document is the only mutable thing, and is cached for five minutes.
|
|
328
|
+
- **Seeding limits**, in the shape a torrent client uses: stop at a ratio, or after so long
|
|
329
|
+
seeding a complete copy, then stop, remove, or remove and delete the files. Global by default
|
|
330
|
+
with a per-archive override β including "seed forever", which a change to the global rule must
|
|
331
|
+
not undo. Never applies to a cache-mode archive, which holds a few pieces on purpose and has
|
|
332
|
+
not been seeding in the sense a ratio measures.
|
|
333
|
+
- **Trackers are settable wherever a torrent is created** β per watch folder, per scheduled
|
|
334
|
+
source, per request β with `trackers` replacing the global list and `addTrackers` appending to
|
|
335
|
+
it. Watch folders could not set them at all before. The add dialog shows the defaults and
|
|
336
|
+
offers a field to announce to more.
|
|
337
|
+
- **Adding is a dialog now**, with everything the API could already do: multiple categories picked
|
|
338
|
+
from those in use or typed fresh, keep-or-discard for URL fetches, and whether the source URL is
|
|
339
|
+
published as a web seed β plus a list of your own to publish instead of it.
|
|
340
|
+
- **`feedCategories` decides what leaves the node.** Category feeds let a subscriber narrow what
|
|
341
|
+
it takes; they never narrowed what was published, since `/feed.xml` carried the whole catalogue.
|
|
342
|
+
With an allow-list set, only those categories appear in any feed and other category feeds
|
|
343
|
+
answer 404. Untagged archives are excluded, because untagged means unmarked for sharing.
|
|
344
|
+
- A subscription can carry a `token`, and a credential lifts `feedCategories`. One feed then
|
|
345
|
+
serves two audiences: an internal node holding the token syncs the whole catalogue, untagged
|
|
346
|
+
archives included, while the outside world sees only the categories marked for sharing.
|
|
347
|
+
- **Access control.** Tiles, TileJSON and the feed stay public; everything under `/api/` and the
|
|
348
|
+
console are guarded whenever `auth.apiKey`, `auth.password` or `auth.passwordHash` is set. A
|
|
349
|
+
bearer token for scripts, a sign-in form and session cookie for people. Passwords set through
|
|
350
|
+
the settings screen are stored as a scrypt hash, and credentials are redacted from every
|
|
351
|
+
response. Configuring nothing keeps the previous behaviour.
|
|
352
|
+
- The startup line prints an address a browser can open. It previously printed the bind address,
|
|
353
|
+
and `http://0.0.0.0:8090` is rejected outright with `ERR_ADDRESS_INVALID`.
|
|
354
|
+
- The console's own page is public, so its sign-in form can load; only `/api/` is guarded.
|
|
355
|
+
- A node configured with only `auth.apiKey` can still use the console: the token is accepted at
|
|
356
|
+
sign-in and the form asks for a token rather than a password that does not exist. Previously
|
|
357
|
+
the console showed a sign-in form that could never succeed.
|
|
358
|
+
- **A node with no credential now refuses to start on a reachable address**, rather than warning.
|
|
359
|
+
The refusal prints the JSON to paste, into the config file it names β or how to create one when
|
|
360
|
+
there is none β along with a generated key and the `curl` that uses it. It prints without a
|
|
361
|
+
stack trace, since a configuration refusal is not a crash. Bind to loopback, configure `auth`,
|
|
362
|
+
or set `allowUnauthenticated: true`. See [docs/security.md](docs/security.md).
|
|
363
|
+
- **The web UI is now a real console.** Live-refreshing archive table with progress, peers and
|
|
364
|
+
speeds; a detail panel per archive with disk usage, web seeds and a tile preview; per-archive
|
|
365
|
+
actions for warming, clearing a cache, adding a web seed and removing; export by downloading
|
|
366
|
+
the `.torrent` or copying the magnet, TileJSON URL or infohash; and a settings screen.
|
|
367
|
+
- `GET`/`PATCH /api/config` read and write settings. Anything read per request applies
|
|
368
|
+
immediately; anything bound at startup is written to the file and reported back as needing a
|
|
369
|
+
restart, rather than being accepted and quietly ignored. Credentials are redacted on the way
|
|
370
|
+
out and never overwritten by their own placeholder.
|
|
371
|
+
- `POST /api/torrents/{infohash}/webseeds` adds web seeds to a torrent already in circulation.
|
|
372
|
+
This does not change the infohash β `url-list` sits outside the `info` dictionary β so magnets
|
|
373
|
+
and peers stay valid, and anything published without a web seed can be given one.
|
|
374
|
+
- `DELETE /api/torrents/{infohash}/cache` reclaims what on-demand reading has accumulated for one
|
|
375
|
+
archive without forgetting the archive, and `GET /api/torrents/{infohash}` reports `diskBytes`.
|
|
376
|
+
Nothing else bounded that disk usage.
|
|
377
|
+
|
|
378
|
+
### π Bug fixes
|
|
379
|
+
- **A second request for a URL already being fetched joins the first.** The catalog cannot answer
|
|
380
|
+
that question β an entry exists only once the download has finished and the torrent has been
|
|
381
|
+
hashed, so for the hours in between `findBySource` says no and every caller starts its own copy.
|
|
382
|
+
The scheduler was safe by accident, since a poll holds a flag for its whole run, but nothing
|
|
383
|
+
protected `POST /api/torrents {url}` for something a schedule was already fetching: two
|
|
384
|
+
downloads of the same hundred gigabytes, both producing the same infohash, both trying to move
|
|
385
|
+
into the same directory. A failed download is not retained, so one network error does not become
|
|
386
|
+
permanent.
|
|
387
|
+
- **One poll takes one build.** A date-based source imported *every* candidate in its lookback
|
|
388
|
+
window, where a directory-listing source has always capped at `newest` (default 1) for the stated
|
|
389
|
+
reason that each candidate is a whole archive. With a daily 137 GB planet build, `lookbackDays: 3`
|
|
390
|
+
therefore meant 411 GB from a single poll. The same cap now applies, and a candidate that is
|
|
391
|
+
already held stops the scan β candidates run newest first, so anything past one on disk is older
|
|
392
|
+
than it, and without stopping lookback walks backwards through history one archive per poll.
|
|
393
|
+
- **`latestLink` works without elevation.** Windows refuses symlinks with EPERM unless the process
|
|
394
|
+
is elevated or the machine is in developer mode, so `latest` was left pointing at nothing. It
|
|
395
|
+
falls back to a hard link, which needs neither and costs no extra space β another name for the
|
|
396
|
+
same bytes rather than a copy, which for a 137 GB archive is the point.
|
|
397
|
+
- **A poll that takes nothing says why.** A source asking only for today's date against an upstream
|
|
398
|
+
that publishes at 09:00 does nothing at all between midnight and then β and silence there is
|
|
399
|
+
indistinguishable from a broken template, a dead server, or a daemon that is not running. It now
|
|
400
|
+
names how many candidate URLs were not published yet and the first of them, and points at
|
|
401
|
+
`lookbackDays: 0` where that is the reason only one date is ever asked for. Nothing is logged
|
|
402
|
+
when the candidates are simply already held, since that is the normal state of every poll after
|
|
403
|
+
the first.
|
|
404
|
+
- **A watched location no longer restarts its download the moment one finishes.** The last-run time
|
|
405
|
+
was recorded when a poll *began* and never again, so by the time a planet build had been fetched
|
|
406
|
+
the stamp was hours old, `now - lastRun` was far past any interval, and the next tick started the
|
|
407
|
+
whole thing again β for ever, on a 72 GB archive. A failed fetch behaved the same way: one that
|
|
408
|
+
died at 35% was retried from zero immediately. The time is now recorded on the way in *and* on
|
|
409
|
+
the way out, so overlap is still prevented and the interval is measured from when the work
|
|
410
|
+
actually ended. The comment there had described this exact behaviour as the thing it was
|
|
411
|
+
avoiding.
|
|
412
|
+
- **`libtorrent` and `feedTitle` are settings the API knows about.** `DEFAULTS` doubles as the
|
|
413
|
+
allow-list, and neither key was in it β so a libtorrent node saving anything at all was answered
|
|
414
|
+
`unknown setting: libtorrent`, because the console posts back every key it was given.
|
|
415
|
+
`libtorrent` was even named in `RESTART_REQUIRED`: known everywhere except where it was checked.
|
|
416
|
+
- **A refused save now changes nothing.** Validation happened inside the loop that assigned, so a
|
|
417
|
+
save containing one bad key applied every key before it and then threw, leaving the running node
|
|
418
|
+
changed and the file on disk not. A watched location added that way started polling immediately
|
|
419
|
+
and vanished from the console. Every key is checked before any is applied.
|
|
420
|
+
- **One loaded config no longer leaks into the next.** `merge()` spread the defaults, so a nested
|
|
421
|
+
object in a loaded config *was* the one in `DEFAULTS` whenever the file did not mention it β and
|
|
422
|
+
load writes the resolved save path back into it. One process loads one config, so this only
|
|
423
|
+
surfaced in tests, but it made the defaults mutable at runtime.
|
|
424
|
+
- **Settings save again.** A setting that may only be set in the config file was refused on the
|
|
425
|
+
key's *presence* rather than on a change, and the console renders every setting it knows about
|
|
426
|
+
and posts the lot β so `allowHooksFromApi` rode along with every save and failed all of them,
|
|
427
|
+
including saves that touched nothing but a watch folder. The error even named a way out that
|
|
428
|
+
could not work: setting `allowHooksFromApi: true` unlocks the hooks, but the flag itself stays
|
|
429
|
+
guarded for ever, so the console kept echoing it and kept being refused. Echoing back the value
|
|
430
|
+
already in force is now a no-op; only a real change is refused, which is what the guard was
|
|
431
|
+
always for.
|
|
432
|
+
- **A preallocated file is no longer mistaken for a finished one.** A torrent client allocates the
|
|
433
|
+
whole file up front β libtorrent creates a 77 GB sparse file the moment a download starts β so an
|
|
434
|
+
archive 0% downloaded already measures exactly its final size. The completion sweep checked the
|
|
435
|
+
disk *first*, called that complete, and recorded it. On the next restart the composite then
|
|
436
|
+
handed a 10%-downloaded archive to a secondary as a finished seed, which is the one thing "only
|
|
437
|
+
the primary writes" exists to prevent. The engine's own progress now wins whenever it has an
|
|
438
|
+
opinion; the size check remains for the case it was written for, an archive the engine is not
|
|
439
|
+
holding at all.
|
|
440
|
+
- **A secondary is given long enough to hash what it was handed.** It is not waiting for metadata β
|
|
441
|
+
the `.torrent` carries that β it is verifying every byte against it, which is minutes for tens of
|
|
442
|
+
gigabytes. Against a default measured in seconds this appeared as `timed out after 60000ms
|
|
443
|
+
waiting for torrent metadata`, blaming the one thing that was never missing. Now
|
|
444
|
+
`secondaryShareTimeoutSeconds`, an hour by default.
|
|
445
|
+
- **The composite asks the primary before handing anything over**, rather than trusting the
|
|
446
|
+
caller's `seedOnly`. That flag is read from the catalog on restore, and a wrong `complete` there
|
|
447
|
+
was all that stood between one incomplete file and two clients writing to it.
|
|
448
|
+
- **A partial vector archive now gets its `vector_layers`, so the preview is not black.** A
|
|
449
|
+
PMTiles header is the first 127 bytes, but the JSON metadata carrying the layer list goes
|
|
450
|
+
wherever the writer put it β and planetiler puts it at the *end*, after every tile: byte
|
|
451
|
+
77,139,967,368 of a 77 GB archive. Probing a file that is 10% downloaded therefore reads a
|
|
452
|
+
perfectly good header and 1528 zero bytes where the metadata should be, and every field except
|
|
453
|
+
the one vector rendering needs looks right. The header records that offset, so the range is
|
|
454
|
+
known and fetchable: `tiles.json` now reads it out of the swarm in the background, with a
|
|
455
|
+
timeout of its own (`tiles.metadataTimeoutMs`, 120s) rather than the interactive header budget,
|
|
456
|
+
which was far too short for a piece at the far end of an archive that nobody has asked for. The
|
|
457
|
+
reply is not held up, and the next request has the layers.
|
|
458
|
+
- **The vector preview draws.** `showInspectMap: true` sets a flag on maplibre-gl-inspect and
|
|
459
|
+
nothing else β the control renders from exactly two places, a source-change handler it
|
|
460
|
+
subscribes to only when `sources` was *not* passed, and the toggle button's click. This page
|
|
461
|
+
passed `sources` and hid the button, closing both, so nothing ever called `render()` and the map
|
|
462
|
+
stayed on a style that was a background colour and nothing else: correct TileJSON, correct tiles,
|
|
463
|
+
no console error, black map. Now rendered explicitly once the map has loaded.
|
|
464
|
+
- **The preview says why a vector map is blank** instead of showing a black rectangle. Related:
|
|
465
|
+
`sources` is no longer passed to maplibre-gl-inspect when there are no layers, since passing it
|
|
466
|
+
disables the control's own lookup β though that lookup only re-reads the TileJSON, so it is the
|
|
467
|
+
metadata fix above that actually makes the map draw.
|
|
468
|
+
- **The Pieces tab had no pane to render into**, so it appeared, highlighted when clicked, and did
|
|
469
|
+
nothing. Tabs and panes are now checked against each other in both directions.
|
|
470
|
+
- **The peers tab is no longer silently empty on libtorrent.** `peer_info.utp_socket` is absent
|
|
471
|
+
from libtorrent's 2.x Python bindings, so the sidecar raised on the first peer and returned
|
|
472
|
+
nothing β an archive downloading at 10 MiB/s from a connected seed reported having no peers at
|
|
473
|
+
all. Fixed in pmtiles-torrent; a node has to be restarted to pick up a sidecar change. Three
|
|
474
|
+
layers here had each turned that exception into an empty list, so a broken engine and an empty
|
|
475
|
+
swarm produced identical output: the route now answers `{ peers, error }` and the console shows
|
|
476
|
+
the reason, and the composite engine logs which engine failed instead of swallowing it. Peer
|
|
477
|
+
rows also now carry the engine that found them and whether each is an ordinary peer, a web seed
|
|
478
|
+
or an HTTP seed β an archive pulling at full speed from one web seed looks exactly like one
|
|
479
|
+
pulling from a swarm until that single server goes away.
|
|
480
|
+
- **Restoring skipped the tracker repair**, which is the one moment somebody expects a fix to take
|
|
481
|
+
effect. It built its own add rather than going through the shared one, so an archive stored
|
|
482
|
+
without trackers stayed unable to find a peer across every restart. It now takes the same path as
|
|
483
|
+
every other re-add.
|
|
484
|
+
- **A save path that has gone is reported.** An unmounted share or a tidied-away directory left the
|
|
485
|
+
engine unable to open anything and the archive sitting at nothing, with no error of its own.
|
|
486
|
+
Restore now says which archive, which path and what to do about it.
|
|
487
|
+
- **The Trackers tab explains an empty list.** An archive with no trackers and no `.torrent` can
|
|
488
|
+
only find peers through the DHT, which on a private or quiet swarm means it may never start β
|
|
489
|
+
and "downloading, 0 peers, indefinitely" is otherwise a mystery. It now says so, and shows what
|
|
490
|
+
the magnet itself carries while the metainfo has not arrived.
|
|
491
|
+
- **An archive joined from a bare infohash never started.** It was given no trackers, so there was
|
|
492
|
+
nowhere to look for a peer, and it sat reporting "downloading" indefinitely. Two causes, both
|
|
493
|
+
now fixed: `parse-torrent` gives a bare magnet an `announce` of `[]` rather than leaving it
|
|
494
|
+
undefined, so the nullish fallback to this node's own trackers kept the empty array and never
|
|
495
|
+
fired; and a magnet supplied by hand was stored verbatim rather than rebuilt, so it kept whatever
|
|
496
|
+
it lacked. The magnet is rebuilt from what was parsed β nothing is lost, since a supplied
|
|
497
|
+
magnet's trackers and web seeds are in there β and an archive already stored without any is
|
|
498
|
+
repaired whenever it is handed back to the engine, which is how the ones added before this get
|
|
499
|
+
fixed.
|
|
500
|
+
- **Stopping a node logged a page of engine errors.** The console keeps polling and a sweep or two
|
|
501
|
+
is still in flight while the engine is being torn down, and each of them was told the sidecar had
|
|
502
|
+
exited. An engine on its way out now reports an empty library instead, which is what it has.
|
|
503
|
+
- **A second engine was handed archives that were still downloading, and wrote its own copy.**
|
|
504
|
+
`restore` and every re-add claimed `seedOnly` for any mirror-mode archive, which means "the data
|
|
505
|
+
is already here, do not fetch it" β and for a half-downloaded archive that is untrue. A composite
|
|
506
|
+
engine took it at its word and passed the archive to the secondary, which honoured the incomplete
|
|
507
|
+
marker and opened `name.incomplete` while the primary wrote `name`: two clients, two files, one
|
|
508
|
+
archive, in one directory. `seedOnly` is now claimed only for archives that are actually
|
|
509
|
+
complete, and a secondary is never given a marker at all, since it only ever receives whole
|
|
510
|
+
archives.
|
|
511
|
+
- **An archive left with both filenames retried for ever.** Finalising refused to rename over an
|
|
512
|
+
existing file β correctly β and then tried again every fifteen seconds, logging the same
|
|
513
|
+
paragraph each time and telling nobody anything they could act on. When the file under the
|
|
514
|
+
archive's own name is the right size the archive is finished, so that is now recorded and the
|
|
515
|
+
leftover named once as something that can be deleted. Nothing is deleted automatically.
|
|
516
|
+
- **Two archives could be pointed at one file.** Filenames are not unique β two builds of the same
|
|
517
|
+
map are both `planet.pmtiles`, and a rebuild keeps the name while minting a new infohash β so
|
|
518
|
+
adding the second one now fails with a 409 naming the first, instead of letting them take turns
|
|
519
|
+
writing into the same file.
|
|
520
|
+
- **"marked incomplete" appeared beside a progress bar reading 100%.** It now sits with the state,
|
|
521
|
+
and only while the file on disk actually carries the marker.
|
|
522
|
+
- **Running two engines silently disabled on-demand tile reading.** The tile reader chose how to
|
|
523
|
+
fetch pieces by switching on the engine's name, and a composite calls itself
|
|
524
|
+
`libtorrent+webtorrent` β which matched neither case, so it fell through to "cannot read pieces
|
|
525
|
+
on demand". A half-downloaded archive that pmtiles-torrent could have served a header and tiles
|
|
526
|
+
from answered a 501 instead, and the preview showed an empty map. The reader now asks the
|
|
527
|
+
primary, which is the only engine that downloads and therefore the only one that holds a partial
|
|
528
|
+
archive at all. Verified end to end: header, metadata and a tile read out of a swarm from an
|
|
529
|
+
archive this node held none of, with 16 KiB on disk afterwards β one piece.
|
|
530
|
+
- **The map preview showed nothing but "Loadingβ¦".** It imported MapLibre as a default export, and
|
|
531
|
+
MapLibre's ESM build has only named ones β which is a `SyntaxError` raised before a line of the
|
|
532
|
+
module runs, so there was no failed request and no clue in the page, only a line in the browser
|
|
533
|
+
console. It is a namespace import now, and a test reads both bundles and asserts the import form
|
|
534
|
+
matches what each actually exports, and that every `maplibregl.X` the page uses is a name the
|
|
535
|
+
bundle provides.
|
|
536
|
+
- **Stopping a node running the libtorrent engine printed a Python stack trace.** Windows delivers
|
|
537
|
+
a console Ctrl-C to every process in the group, so the sidecar received it too and reported a
|
|
538
|
+
`KeyboardInterrupt` on the way out. Nothing was wrong, but a traceback at the end of a clean stop
|
|
539
|
+
reads as a crash and buries the lines that say what actually happened. Fixed properly in the
|
|
540
|
+
sidecar, which ships with `pmtiles-torrent`, and suppressed here as well so an older sidecar is
|
|
541
|
+
quiet too. Separately, the engine no longer reports an exit it asked for as a failure β that
|
|
542
|
+
rejected a promise nobody was waiting on, which is how Node announces a crash.
|
|
543
|
+
- **An archive adopted from a client on another machine never started.** The magnet built for it
|
|
544
|
+
carried the infohash and nothing else β no trackers β so there was nowhere to look for peers but
|
|
545
|
+
the DHT, and it sat at 0% reporting "downloading" and meaning nothing of the kind. The client
|
|
546
|
+
being adopted from is seeding the archive and therefore *has* the metainfo, so that is fetched
|
|
547
|
+
and used instead: trackers, web seeds and piece geometry included, and kept on disk so a restart
|
|
548
|
+
does not need the swarm. Where a client cannot export one, the magnet at least carries this
|
|
549
|
+
node's own trackers now.
|
|
550
|
+
- **Adopting from this node's own engine restarted the download.** Whether the data could be read
|
|
551
|
+
from this process was being used to decide whether the engine held it, which are different
|
|
552
|
+
questions β so an archive under a path this process could not open was re-added as a magnet,
|
|
553
|
+
pointed at a different directory, and downloaded again from nothing. Adopting from the configured
|
|
554
|
+
engine is a catalog operation now; readability only decides whether tiles can be served straight
|
|
555
|
+
off the file.
|
|
556
|
+
- **The archives table lost a column.** Adding *Ratio* replaced the *Up* cell instead of following
|
|
557
|
+
it, so every value from there rightwards sat under the wrong heading β the upload speed appeared
|
|
558
|
+
as the ratio, and *State* was blank. A test now asserts the row builds exactly as many cells as
|
|
559
|
+
the table has headings.
|
|
560
|
+
- **"Add archiveβ¦" threw `locationPicker is not defined`.** The save-location helpers were declared
|
|
561
|
+
inside the detail panel's renderer, so the add and adopt dialogs β which are not β could not see
|
|
562
|
+
them. `node --check` accepts that happily: it is a syntax-clean script and a `ReferenceError` at
|
|
563
|
+
click time. The console script is now checked for it, by counting brace depth over a source with
|
|
564
|
+
strings, template literals, comments and regular expressions blanked out, and asserting that
|
|
565
|
+
every helper called from more than one place is declared at the top level. Verified against the
|
|
566
|
+
commit that broke it.
|
|
567
|
+
- **Magnets dropped the web seeds their torrents advertised.** Torrents created here have always
|
|
568
|
+
put them in the magnet; a torrent that was *joined* did not, and a web seed added after
|
|
569
|
+
publication reached everyone holding the `.torrent` and nobody holding the magnet β which is the
|
|
570
|
+
link that actually gets shared. Both now carry `ws=` for every seed the torrent advertises. This
|
|
571
|
+
does not weaken anything: whether a URL may be published is decided once, when the torrent is
|
|
572
|
+
created, and once it is in the `url-list` anyone holding the `.torrent` already has it.
|
|
573
|
+
- **An archive joined by magnet forgot everything the swarm told it.** A magnet carries an
|
|
574
|
+
infohash and, if you are lucky, a display name; the real name, the exact size and the piece
|
|
575
|
+
geometry arrive afterwards over BEP 9 β and arrived into nothing. Every restart asked the swarm
|
|
576
|
+
again for what the node had already been told, which needs a peer, so a restart while the swarm
|
|
577
|
+
was quiet left the archive stuck. The `.torrent` endpoint had nothing to serve and the feed
|
|
578
|
+
advertised a URL that answered 404, the Content tab was empty, and the size stayed at whatever
|
|
579
|
+
the magnet claimed β usually zero, which made the disk-space check before a move meaningless.
|
|
580
|
+
The metainfo is now written to the torrent directory as soon as the engine has it, which for a
|
|
581
|
+
magnet is the moment the add resolves. Anything joined before this is picked up by the sweep.
|
|
582
|
+
Only gaps are filled: a name chosen here is a decision about this node's copy and is not
|
|
583
|
+
overruled.
|
|
584
|
+
- **Every radio and checkbox in the console sat centred on a line of its own**, with its label
|
|
585
|
+
above it. `.field label` makes a label `display: block` and `.field input` stretches a control to
|
|
586
|
+
the full width of its dialog, and both applied to these too. They share a `choice` class now,
|
|
587
|
+
defined last in the stylesheet because the rules it has to beat match just as tightly β position,
|
|
588
|
+
not specificity, is what settles it. A test asserts both halves, since moving the block up the
|
|
589
|
+
sheet would silently revert the layout.
|
|
590
|
+
- **Shutting down could leave the port held, so the next run could not start.** Three faults in one
|
|
591
|
+
loop. The signal handlers were installed at the *end* of startup, so a Ctrl-C while the catalogue
|
|
592
|
+
was being handed back to the engine reached nothing at all and killed the process outright β port
|
|
593
|
+
still held, trackers still believing it was seeding. They are installed before any of what they
|
|
594
|
+
stop exists now. Closing the HTTP server only dropped *idle* connections, so one stuck request β
|
|
595
|
+
a tile read waiting on the swarm, say β kept it open past its own timeout; anything still
|
|
596
|
+
in-flight is now forced shortly after. And a WebTorrent client that cannot open its port reports
|
|
597
|
+
it asynchronously, long after construction: that was logged and ignored, after which every add
|
|
598
|
+
waited out a five-minute metadata timeout against a client that could never talk to anyone. It is
|
|
599
|
+
fatal now, reported with what to do about it, and restore stops at the first one rather than
|
|
600
|
+
repeating it per archive. A `.torrent` also no longer waits on the magnet timeout, since it
|
|
601
|
+
carries its own metadata.
|
|
602
|
+
- **Peer tokens were returned in plain text by `GET /api/config`.** A token is what persuades a
|
|
603
|
+
peer to publish more than it publishes to the world β the same class of thing as the qBittorrent
|
|
604
|
+
password, which was already redacted. Now redacted too, and a save that echoes the placeholder
|
|
605
|
+
back keeps the stored token rather than overwriting it with asterisks.
|
|
606
|
+
- **Adding the first peer did nothing until a restart.** The refresh timer only started when the
|
|
607
|
+
subscription list was already non-empty, so a peer added through the console was never polled.
|
|
608
|
+
The same bug as scheduled sources had; every refresh reads the list fresh.
|
|
609
|
+
- **Categories set when adding an archive never appeared.** The console read `entry.category`,
|
|
610
|
+
singular β the field the catalog folds into the list and deletes on write β so every archive
|
|
611
|
+
showed a blank tag line. The tags were stored correctly the whole time.
|
|
612
|
+
- **Pausing, resuming and switching mode silently did nothing on the WebTorrent engine.**
|
|
613
|
+
`client.get()` is async β it parses whatever identifier it is handed before matching β so the
|
|
614
|
+
promise it returned read as a perfectly good torrent whose every property was `undefined`. Every
|
|
615
|
+
guard therefore saw "no such torrent" and returned false, and `setMode` fell back to removing and
|
|
616
|
+
re-adding the torrent, which is why it appeared to work at all. Looked up directly by infohash
|
|
617
|
+
now, which needs no parsing.
|
|
618
|
+
- **The console offered a TileJSON URL for archives that can never have one.** Identification only
|
|
619
|
+
ran when an archive was created here, so a *joined* MBTiles torrent had no recorded format and
|
|
620
|
+
was treated as PMTiles: a tile endpoint was offered, and asking for it read pieces out of the
|
|
621
|
+
swarm until the reader hit the magic-number check. A joined torrent now takes an initial format
|
|
622
|
+
from its filename, the first read records what the content actually is, and both `tiles.json`
|
|
623
|
+
and the tile route answer 415 rather than retrying forever. The console hides the TileJSON,
|
|
624
|
+
preview and warm controls for anything that is not PMTiles and says why.
|
|
625
|
+
- **An archive opened in cache mode was read through the swarm forever.** Which source to use was
|
|
626
|
+
decided once, at open, so switching to mirror β or the download simply finishing β changed
|
|
627
|
+
nothing, and tiles kept being pulled a piece at a time while a complete copy sat on disk. The
|
|
628
|
+
reader is now told to forget an archive whenever its mode changes, it is paused or resumed, or
|
|
629
|
+
its cache is cleared.
|
|
630
|
+
- A TileJSON request for an archive nobody is seeding waited a full minute before saying so, which
|
|
631
|
+
reads as a hang. It is bounded at twelve seconds now (`tiles.headerTimeoutMs`) and says what is
|
|
632
|
+
actually wrong: no peers yet, and no web seed to fall back on.
|
|
633
|
+
- **Ctrl-C could hang.** Once archives were restored to the engine at startup, stopping meant
|
|
634
|
+
telling every tracker so β and an unreachable one waits for a timeout each. Every shutdown step
|
|
635
|
+
is now bounded, a watchdog exits regardless after fifteen seconds, in-flight downloads are
|
|
636
|
+
cancelled first, and a second Ctrl-C forces the issue instead of stacking another shutdown.
|
|
637
|
+
- Opening a detail tab and waiting sent you back to General. The three-second poll rebuilt the
|
|
638
|
+
whole panel; it now updates the table only, and an action that does re-render the panel returns
|
|
639
|
+
to the tab you were on.
|
|
640
|
+
- **A restart silently stopped seeding everything.** Nothing handed the catalogue back to the
|
|
641
|
+
engine, so the catalog still listed every archive and the console still showed them while the
|
|
642
|
+
engine held none. They are restored at startup now, each in the mode it was left in.
|
|
643
|
+
- **Switching mode after a restart crashed the process.** WebTorrent throws for an unknown
|
|
644
|
+
infohash, and because its `remove()` is async the rejection escaped from inside the executor
|
|
645
|
+
where a caller's `catch` could not see it. Removing something the engine does not hold is now
|
|
646
|
+
treated as already done, which is what was wanted.
|
|
647
|
+
- `webtorrent` is a plain dependency rather than an optional one. It is the *default* engine, so
|
|
648
|
+
calling it optional was wrong, and npm repeatedly dropped it from the lockfile while leaving the
|
|
649
|
+
declaration β after which `npm install webtorrent` reported "up to date" and changed nothing,
|
|
650
|
+
and the default engine failed to start.
|
|
651
|
+
- A torrent's comment and piece length were accepted by the API but never passed on.
|
|
652
|
+
- A custom `webSeeds` list was discarded when `webSeed: false` β exactly the case where the source
|
|
653
|
+
must not be published and a public URL was supplied in its place.
|
|
654
|
+
- **A pre-signed source URL was published as a web seed, credentials and all.** Adding an archive
|
|
655
|
+
from an S3 or Azure signed link baked that link β a bearer credential β into the `.torrent` and
|
|
656
|
+
broadcast it to the swarm, where it cannot be recalled. Such URLs are now detected and not
|
|
657
|
+
published; `webSeed: false` suppresses any source URL, and `webSeeds` supplies a public one
|
|
658
|
+
instead.
|
|
659
|
+
- **Creating a torrent from a local path published any readable file to a public swarm.** The
|
|
660
|
+
PMTiles probe failure was caught and discarded, so `{"path": "/etc/shadow"}` produced a
|
|
661
|
+
seeded torrent and returned its infohash. Archives are now identified by content β PMTiles
|
|
662
|
+
and MBTiles are recognised, anything else is a 400 unless `allowUnknown` is passed. Only
|
|
663
|
+
PMTiles can have its tiles served; MBTiles is SQLite, whose pages are scattered rather than
|
|
664
|
+
spatially clustered, so it is distributable but not servable.
|
|
665
|
+
- **A missing tile answered 204 for every archive, which breaks sparse raster.** MapLibre only
|
|
666
|
+
overzooms a parent tile when the child 404s, so a sparse raster-dem β Mapterhorn, or any
|
|
667
|
+
terrain built only where there is land β rendered as holes wherever data was never built.
|
|
668
|
+
Raster now answers 404 and vector keeps 204.
|
|
669
|
+
|
|
670
|
+
### π Documentation
|
|
671
|
+
- **Ports and reachability**, which nothing covered before: which of the four listeners wants a
|
|
672
|
+
forwarding rule (the peer port, exactly as in qBittorrent), why WebRTC wants none of them β it
|
|
673
|
+
is signalled over a `wss://` tracker and carried over ICE with STUN, so it needs outbound UDP
|
|
674
|
+
rather than an inbound rule β and why peer traffic never touches the load balancer. Also that
|
|
675
|
+
two engines need two peer ports, since WebTorrent picks a random one unless told otherwise, and
|
|
676
|
+
that **browser peers need a `wss://` tracker in the announce list**: the defaults are UDP-only,
|
|
677
|
+
a browser has no UDP socket and no DHT, and WebTorrent's own WebSocket trackers ship only in its
|
|
678
|
+
browser bundle. Without one, the browser half of the swarm cannot find a peer however many nodes
|
|
679
|
+
are seeding.
|
|
680
|
+
- The topology diagram shows the **browser bridge**: browsers speak WebRTC and conventional
|
|
681
|
+
clients speak TCP and uTP, so a browser peer is only ever reached by a node running WebTorrent.
|
|
682
|
+
The deployment notes cover the **two-port split**, which decides what a load balancer may be
|
|
683
|
+
pointed at. The API table gained the four routes it was missing (`/api/adds`, `/api/session`,
|
|
684
|
+
`/archives/{hash}/archive.torrent`, `/archives/{hash}/preview`), and there are now tests that a
|
|
685
|
+
diagram's `linkStyle` indices are in range, that every relative link and anchor resolves, and
|
|
686
|
+
that no route is missing a row.
|
|
687
|
+
|
|
10
688
|
## 0.2.0
|
|
11
689
|
### β¨ Features and improvements
|
|
12
690
|
- **Serve tiles.** Every archive now has a TileJSON endpoint and a `{z}/{x}/{y}` tile endpoint
|