pmtiles-swarm 0.80.0 → 0.89.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +231 -0
- package/README.md +3 -1
- package/docs/tile-stacks.md +282 -40
- package/package.json +1 -1
- package/src/api.js +201 -1
- package/src/auth.js +4 -0
- package/src/bake-jobs.js +46 -1
- package/src/config.js +47 -0
- package/src/feed.js +9 -3
- package/src/index.js +44 -0
- package/src/library.js +9 -1
- package/src/sources.js +1 -1
- package/src/stack-exports.js +285 -0
- package/src/stack-feed.js +416 -0
- package/src/stack-tile.js +23 -2
- package/src/web/index.html +234 -10
- package/src/web/public.html +39 -9
package/CHANGELOG.md
CHANGED
|
@@ -7,6 +7,237 @@
|
|
|
7
7
|
### 🐞 Bug fixes
|
|
8
8
|
- _...Add new stuff here..._
|
|
9
9
|
|
|
10
|
+
## 0.89.0
|
|
11
|
+
### ✨ Features and improvements
|
|
12
|
+
- **Both feeds are linked at the foot of the public page.** `RSS feed` is `archive RSS feed` now and
|
|
13
|
+
`stack RSS feed` is beside it. One name was unambiguous while there was one feed; with a second, a
|
|
14
|
+
reader following the old name would have got archives when they wanted recipes and had nothing on
|
|
15
|
+
the page to tell them otherwise.
|
|
16
|
+
|
|
17
|
+
|
|
18
|
+
### 🐞 Bug fixes
|
|
19
|
+
- **A feed carrying every category.** `/categories.xml`, beside `/feed.xml` and `/stacks.xml` — the
|
|
20
|
+
set was inconsistent, and following all of a node's categories meant adding each by hand and
|
|
21
|
+
remembering to add the next one, which is not following its categories at all. It is what a
|
|
22
|
+
subscriber wants when they want everything this node files: including categories added later.
|
|
23
|
+
|
|
24
|
+
The items are archives, because a category has no bytes of its own and the build it resolves to is
|
|
25
|
+
what a subscriber would join — which also means every existing consumer already knows how to read
|
|
26
|
+
it, magnets and enclosures included. That is what separates it from the whole catalogue: `/feed.xml`
|
|
27
|
+
carries every build a node holds, and this carries the current one of each category. On a node
|
|
28
|
+
keeping four builds apiece the difference is fourfold.
|
|
29
|
+
|
|
30
|
+
In the footer of the public page it replaces `categories JSON`, which was the odd one out among
|
|
31
|
+
three feeds. `/latest/` is untouched and remains the JSON index.
|
|
32
|
+
|
|
33
|
+
|
|
34
|
+
## 0.88.0
|
|
35
|
+
### ✨ Features and improvements
|
|
36
|
+
- **An archive card is built like a category's.** Copy buttons for the addresses that belong
|
|
37
|
+
somewhere else — TileJSON, magnet, and a **source URL** of its own, the TileJSON URL with the
|
|
38
|
+
`.torrent` and magnet in its fragment, pinned to that build rather than following the category.
|
|
39
|
+
The preview, `.torrent` and download stay links, because a page is followed and a file is saved.
|
|
40
|
+
The printed TileJSON row went with the button that replaced it: an address in full beside a button
|
|
41
|
+
that copies it is the same fact twice, and it was the longest line on the card.
|
|
42
|
+
|
|
43
|
+
|
|
44
|
+
### 🐞 Bug fixes
|
|
45
|
+
- **A terrain stack was offered no hillshade preview on the public page.** The listing reported its
|
|
46
|
+
encoding as whatever `output.encoding` restated, which for a recipe saying "same as the sources" —
|
|
47
|
+
the ordinary case — is nothing. So the page could not tell a terrain stack from an imagery one and
|
|
48
|
+
offered neither the raw preview nor the terrain one. It reports what the stack actually serves now,
|
|
49
|
+
and reports nothing for imagery, where a guess would have offered a hillshade of a photograph.
|
|
50
|
+
|
|
51
|
+
- **A stack whose sources disagreed about their encoding wrote tiles in more than one of them.** With
|
|
52
|
+
no `output.encoding`, the merge took the encoding of whichever source answered first — and which
|
|
53
|
+
source answers varies by tile, since the base is sparse in one place and the layer above covers in
|
|
54
|
+
another. One stack wrote one tile as mapbox and the next as terrarium, with the TileJSON in front
|
|
55
|
+
describing neither: correct in one place, a cliff face in another, for no reason the recipe showed.
|
|
56
|
+
It follows the base source now, which is a property of the recipe rather than of the tile, and the
|
|
57
|
+
listing and the merge are held to the same answer by a test.
|
|
58
|
+
|
|
59
|
+
|
|
60
|
+
## 0.87.0
|
|
61
|
+
### ✨ Features and improvements
|
|
62
|
+
- **A feed per stack, and an address to copy for it.** `/stacks/<id>.xml` carries one recipe, for
|
|
63
|
+
following a single map out of somebody's several rather than everything they publish. An **RSS**
|
|
64
|
+
button on the stack's row in the console and beside TileJSON and XYZ on the public page. Copied
|
|
65
|
+
rather than followed: the address is for another node's settings, and a browser shown an RSS
|
|
66
|
+
document mostly offers to download it. A stack this node adopted has no button and its feed 404s —
|
|
67
|
+
somebody else's recipe is theirs to publish.
|
|
68
|
+
|
|
69
|
+
|
|
70
|
+
### 🐞 Bug fixes
|
|
71
|
+
- **The copied XYZ address came back percent-encoded.** `http://…/stacks/test/%7Bz%7D/%7Bx%7D/%7By%7D.png`
|
|
72
|
+
rather than `{z}/{x}/{y}`, which no client will take. The public page resolves a relative address
|
|
73
|
+
against the page's own URL to make it absolute, and `new URL` percent-encodes braces because they
|
|
74
|
+
are not legal in a path — and an XYZ template is very little but braces. The TileJSON link was
|
|
75
|
+
unaffected, having none. Resolving still happens; the two sequences it introduces are put back.
|
|
76
|
+
|
|
77
|
+
|
|
78
|
+
## 0.86.0
|
|
79
|
+
### ✨ Features and improvements
|
|
80
|
+
- _...Add new stuff here..._
|
|
81
|
+
|
|
82
|
+
- **Stack recipes travel between nodes.** `GET /stacks.xml` is this node's own stacks as a feed;
|
|
83
|
+
**Settings → Feeds → Stack feeds** follows another node's and adopts what it carries. Archives
|
|
84
|
+
already travelled by category — a builder feeding two tile servers gave them the same archives —
|
|
85
|
+
and this is how the recipes that combine them travel with them, so a stack is written once and
|
|
86
|
+
corrected once.
|
|
87
|
+
|
|
88
|
+
This section of the docs used to argue against a feed, on the grounds that a stack is a mutable
|
|
89
|
+
document and syncing one is about conflicts. That holds where two nodes both edit a recipe and is
|
|
90
|
+
not the arrangement anybody runs: one node authors, the rest follow. What survives from the
|
|
91
|
+
objection is handled directly — **a stack made on this node is never overwritten by one arriving
|
|
92
|
+
under the same name**, which is refused and said.
|
|
93
|
+
|
|
94
|
+
A recipe is adopted under the publisher's own id, so `planet-terrain` answers at the same URL on
|
|
95
|
+
the builder and on every replica; namespacing it would have given three nodes three URLs and
|
|
96
|
+
defeated the point. A recipe naming a source this node has not got is adopted anyway and reports
|
|
97
|
+
the missing source until it arrives — refusing it would mean a replica could not be set up until
|
|
98
|
+
every archive had finished downloading, which is backwards.
|
|
99
|
+
|
|
100
|
+
What happens when a feed stops carrying a stack is the feed's own setting: keep it and say so, or
|
|
101
|
+
remove it here too.
|
|
102
|
+
|
|
103
|
+
### 🐞 Bug fixes
|
|
104
|
+
- _...Add new stuff here..._
|
|
105
|
+
|
|
106
|
+
## 0.85.1
|
|
107
|
+
### ✨ Features and improvements
|
|
108
|
+
- _...Add new stuff here..._
|
|
109
|
+
|
|
110
|
+
- **An export no longer pretends a stack has categories.** Both doors fell back to
|
|
111
|
+
`stack.categories` when none were given — a field the typedef never had, validation never checked,
|
|
112
|
+
the editor has no box for and nothing ever wrote. A category is what an *archive* is filed under,
|
|
113
|
+
and a stack has no bytes and no infohash for that to be about.
|
|
114
|
+
|
|
115
|
+
Left empty the archive is unfiled: held and seeded, in no category and no feed. The scheduled
|
|
116
|
+
export row says `unfiled` and the dialog says so in full, rather than offering a default that
|
|
117
|
+
could not exist.
|
|
118
|
+
|
|
119
|
+
### 🐞 Bug fixes
|
|
120
|
+
- _...Add new stuff here..._
|
|
121
|
+
|
|
122
|
+
## 0.85.0
|
|
123
|
+
### ✨ Features and improvements
|
|
124
|
+
- _...Add new stuff here..._
|
|
125
|
+
|
|
126
|
+
- **A scheduled export can be served by the node that baked it.** The **Local file** choice, the same
|
|
127
|
+
one every other row under Feeds offers: http, http + web seed, http + catalog. It is worth more
|
|
128
|
+
here than anywhere else — a baked archive is already on this disk, so serving it costs a route
|
|
129
|
+
rather than a download, and without it a scheduled export produces something only peers can reach.
|
|
130
|
+
A nightly build behind a URL that is always current is usually the point of scheduling one.
|
|
131
|
+
|
|
132
|
+
`publishDir` and `webSeedBase` remain the other half of the same question, for a directory
|
|
133
|
+
something else already serves. The two are not exclusive: publishing to a served directory *and*
|
|
134
|
+
offering this node as a web seed gives a client two places to get the same bytes.
|
|
135
|
+
|
|
136
|
+
### 🐞 Bug fixes
|
|
137
|
+
- _...Add new stuff here..._
|
|
138
|
+
|
|
139
|
+
## 0.84.0
|
|
140
|
+
### ✨ Features and improvements
|
|
141
|
+
- _...Add new stuff here..._
|
|
142
|
+
|
|
143
|
+
- **Scheduled exports are rows, like every other automation beside them.** `stackExports` in the
|
|
144
|
+
config, edited under Settings → Feeds with the same row editor the monitored folders and the
|
|
145
|
+
scheduled sources use: add a row, choose the stack from a dropdown, say when, and fill in the rest
|
|
146
|
+
— categories, builds to keep, keep for how many days, save location, publish directory, web seed
|
|
147
|
+
base, archive name, attribution, description.
|
|
148
|
+
|
|
149
|
+
**Several rows may name one stack.** That is what the two earlier shapes could not say: an `export`
|
|
150
|
+
block on the recipe, and then a table of one row per stack, both hold exactly one schedule — and a
|
|
151
|
+
nightly build to the fast disk beside a weekly one published elsewhere is an ordinary thing to
|
|
152
|
+
want. It also puts the schedule where the other automations already are: a watched folder says what
|
|
153
|
+
*this machine* does, not what a map is, and keeping it out of the recipe means a recipe copied to
|
|
154
|
+
another node does not quietly start baking there.
|
|
155
|
+
|
|
156
|
+
The export dialog no longer offers to repeat — a stack may have several schedules and a dialog
|
|
157
|
+
opened on the stack cannot say which it would edit. It exports once and points at the settings tab.
|
|
158
|
+
A stack's row in the Stacks view still shows the schedules aimed at it.
|
|
159
|
+
|
|
160
|
+
### 🐞 Bug fixes
|
|
161
|
+
- _...Add new stuff here..._
|
|
162
|
+
|
|
163
|
+
## 0.83.0
|
|
164
|
+
### ✨ Features and improvements
|
|
165
|
+
- _...Add new stuff here..._
|
|
166
|
+
|
|
167
|
+
- **A scheduled export is set up like the other automations beside it.** The row under Settings →
|
|
168
|
+
Feeds is the whole of one now, not only the timer: categories, builds to keep, keep for how many
|
|
169
|
+
days, archive name, attribution, description, where the data lands, a publish directory and a web
|
|
170
|
+
seed base. Those are the manual export dialog's fields plus the two retention rules every other
|
|
171
|
+
automation on that tab already had, which is the point — a scheduled export is set up in one place
|
|
172
|
+
rather than half in a settings tab and half in a dialog.
|
|
173
|
+
|
|
174
|
+
Retirement is the part that needed something new underneath. `keep` and `keepDays` are applied by
|
|
175
|
+
the same code a watched folder uses, but retiring needs a *family* — which archives are builds of
|
|
176
|
+
the same map — and a bake marked nothing, so there was nothing to compare. An archive now records
|
|
177
|
+
`source.stack`, and the family is every archive that names this stack. Without it a nightly export
|
|
178
|
+
is a disk that fills at one archive a night.
|
|
179
|
+
|
|
180
|
+
### 🐞 Bug fixes
|
|
181
|
+
- _...Add new stuff here..._
|
|
182
|
+
|
|
183
|
+
## 0.82.0
|
|
184
|
+
### ✨ Features and improvements
|
|
185
|
+
- _...Add new stuff here..._
|
|
186
|
+
|
|
187
|
+
- **Export schedules are set under Settings → Feeds.** Not only the two global settings that landed
|
|
188
|
+
there in 0.81.1 — the schedules themselves. **Scheduled exports** is a row per stack: never, every
|
|
189
|
+
day at a time, or every so many hours, saved onto that stack's recipe. Feeds is where somebody goes
|
|
190
|
+
to say when a thing runs, so it is where they are set; the export dialog's **Repeat** control still
|
|
191
|
+
writes the same schedule, for setting one while you are already there.
|
|
192
|
+
|
|
193
|
+
Turning a stack to _never_ pauses it with `enabled: false` rather than deleting the block. Where it
|
|
194
|
+
lands, what it is called and which categories it is filed under live in the same place, and those
|
|
195
|
+
should survive being paused.
|
|
196
|
+
|
|
197
|
+
### 🐞 Bug fixes
|
|
198
|
+
- _...Add new stuff here..._
|
|
199
|
+
|
|
200
|
+
## 0.81.1
|
|
201
|
+
### ✨ Features and improvements
|
|
202
|
+
- _...Add new stuff here..._
|
|
203
|
+
|
|
204
|
+
- **The scheduled-export settings are under Settings → Feeds.** `stacks.scheduledExports` and
|
|
205
|
+
`stacks.exportIntervalHours` had no schema entry, so they were config-file-only and invisible in
|
|
206
|
+
the console. Feeds rather than a group of their own, because that tab is already where the
|
|
207
|
+
automations that bring a file in on a timer sit — a scheduled source watching an upstream
|
|
208
|
+
directory, a subscription following someone else's feed. A scheduled export is the same kind of
|
|
209
|
+
thing; it just produces the file here instead of fetching it, and it lands in a category and goes
|
|
210
|
+
out over the feed exactly as a fetched one does.
|
|
211
|
+
|
|
212
|
+
### 🐞 Bug fixes
|
|
213
|
+
- _...Add new stuff here..._
|
|
214
|
+
|
|
215
|
+
## 0.81.0
|
|
216
|
+
### ✨ Features and improvements
|
|
217
|
+
- _...Add new stuff here..._
|
|
218
|
+
|
|
219
|
+
- **A stack can export itself on a schedule.** An archive is a snapshot: a stack over categories
|
|
220
|
+
follows a rebuild and a file baked from it does not, so it goes stale the moment its sources move
|
|
221
|
+
and somebody has to notice. An `export` block on the recipe says when — `at` for a time of day in
|
|
222
|
+
UTC, `everyHours` or `everyMinutes` for an interval, the same shape a scheduled source uses and
|
|
223
|
+
read by the same code — along with everything the export dialog collects. In the console, **Repeat**
|
|
224
|
+
turns the Export button into **Save schedule**, and the stack's row shows when it runs.
|
|
225
|
+
|
|
226
|
+
The hard part is remembering across a restart. The source poller keeps last-run times in memory,
|
|
227
|
+
which is fine when a missed poll costs one poll; here it would cost the whole bake, every restart,
|
|
228
|
+
for hours. So it is written to `stack-exports.json`, and written *before* the bake finishes — a
|
|
229
|
+
restart mid-export must not start it from the top, since the checkpoint is what carries it on.
|
|
230
|
+
|
|
231
|
+
A run whose sources have not moved is skipped: `bakeRevision` is recorded beside the time, and an
|
|
232
|
+
identical archive is the same map under a new infohash that then has to be seeded beside the one it
|
|
233
|
+
duplicates. It will not run two at once, will not start one over a bake already running, and does
|
|
234
|
+
not record a refusal as a run — a location that is full is something somebody fixes, and a schedule
|
|
235
|
+
that gave up quietly would hide that it ever ran. `stacks.scheduledExports: false` turns it off,
|
|
236
|
+
which is what a second node serving the same recipes wants.
|
|
237
|
+
|
|
238
|
+
### 🐞 Bug fixes
|
|
239
|
+
- _...Add new stuff here..._
|
|
240
|
+
|
|
10
241
|
## 0.80.0
|
|
11
242
|
### ✨ Features and improvements
|
|
12
243
|
- **A stack can be a source in another stack.** `{ "stack": "jaxa-with-gebco" }` beside `category`
|
package/README.md
CHANGED
|
@@ -66,7 +66,7 @@ been sending and receiving, kept in SQLite so it survives a restart; an indicato
|
|
|
66
66
|
peers can reach this node at all; and a settings screen covering monitored folders, watched web
|
|
67
67
|
locations, remote nodes, save locations, access tokens and the external-program hooks.
|
|
68
68
|
|
|
69
|
-
**Publishes RSS.** `/feed.xml
|
|
69
|
+
**Publishes RSS.** `/feed.xml` for every archive, `/categories.xml` for the build each category currently resolves to, `/stacks.xml` for the stack recipes, and `/feed/<category>.xml` per category. Plain RSS 2.0 with
|
|
70
70
|
torrent enclosures, so **qBittorrent's built-in RSS auto-downloader can subscribe today** with no
|
|
71
71
|
new software. Items also carry a namespaced description of the map — format, zoom range, bounds,
|
|
72
72
|
tile count — so a subscriber can decide whether it wants a 72 GiB download before starting one.
|
|
@@ -813,6 +813,8 @@ which the endpoint answers 501.
|
|
|
813
813
|
| `GET` | `/stacks/:id/:size/:z/:x/:y.:ext` | The same tile at 256 or 512 px. A tile's coordinates are an extent rather than a pixel count, so this is the same ground on a finer or coarser grid |
|
|
814
814
|
| `GET` | `/stacks/:id/:z/:x/:y.:ext` | One tile of a stack — **public**. Answered by the topmost source holding it; `X-Stack-Sources` says which were asked and what each said |
|
|
815
815
|
| `GET` | `/feed.xml`, `/feed/:category.xml`, `/latest/:category.xml` | RSS — **public** |
|
|
816
|
+
| `GET` | `/categories.xml` | Every category, as the build each resolves to — **public** |
|
|
817
|
+
| `GET` | `/stacks.xml`, `/stacks/:id.xml` | Stack recipes, for another node to follow — **public** |
|
|
816
818
|
|
|
817
819
|
Everything under `/api/` is guarded once a credential is configured; tiles, TileJSON and the feeds
|
|
818
820
|
never are. A `peer` token may read but not change, and may be narrowed to some categories. See
|
package/docs/tile-stacks.md
CHANGED
|
@@ -53,6 +53,7 @@ of its parts.
|
|
|
53
53
|
- [What a mask has to match](#what-a-mask-has-to-match)
|
|
54
54
|
- [Feathering a seam](#feathering-a-seam)
|
|
55
55
|
- [A stack as a source](#a-stack-as-a-source)
|
|
56
|
+
- [Exporting on a schedule](#exporting-on-a-schedule)
|
|
56
57
|
- [Finding a stack](#finding-a-stack)
|
|
57
58
|
- [Syncing a stack to another node](#syncing-a-stack-to-another-node)
|
|
58
59
|
- [What the offline merge got wrong](#what-the-offline-merge-got-wrong)
|
|
@@ -115,6 +116,24 @@ carries. **Terrain-RGB must never be resized as RGB.** Any implementation that
|
|
|
115
116
|
hands a terrain tile to an image library's `resize` is wrong even though it
|
|
116
117
|
runs.
|
|
117
118
|
|
|
119
|
+
### What it re-encodes to
|
|
120
|
+
|
|
121
|
+
`output.encoding` where the recipe says. Where it does not, the **base source's**
|
|
122
|
+
— the bottom one, which is the layer that covers everything.
|
|
123
|
+
|
|
124
|
+
The base rather than whichever source happened to answer, which is what this
|
|
125
|
+
used to read. Which source answers varies by tile: the base is sparse here, the
|
|
126
|
+
one above covers there. So a stack whose sources disagreed about their encoding
|
|
127
|
+
wrote one tile as mapbox and the next as terrarium, with the TileJSON in front
|
|
128
|
+
describing neither — a map that renders correctly in one place and as a cliff
|
|
129
|
+
face in another, for no reason the recipe shows. The base is a property of the
|
|
130
|
+
recipe and the same for every tile it serves.
|
|
131
|
+
|
|
132
|
+
It is worked out in two places — the merge, which writes the bytes, and the
|
|
133
|
+
listing, which describes them — and a test holds them together, because two
|
|
134
|
+
files deciding one fact is how a document comes to disagree with the tiles it
|
|
135
|
+
describes.
|
|
136
|
+
|
|
118
137
|
**`"space": "rgba"`** treats each tile as ordinary imagery: `opacity` scales the
|
|
119
138
|
source alpha, `blend` picks the operator, and the result is composited in
|
|
120
139
|
premultiplied RGBA. This is the Photoshop case — hillshade over satellite, a
|
|
@@ -1450,6 +1469,168 @@ than copying it is that a correction propagates.
|
|
|
1450
1469
|
pinned as what is underneath it, which is what decides whether the outer stack's
|
|
1451
1470
|
tiles are safe to cache hard.
|
|
1452
1471
|
|
|
1472
|
+
### Exporting on a schedule
|
|
1473
|
+
|
|
1474
|
+
An archive is a snapshot. A stack over categories follows its sources — when a
|
|
1475
|
+
new planet build lands, the stack serves it the next time anybody asks — and an
|
|
1476
|
+
archive baked from that stack does not. It goes stale the moment the sources
|
|
1477
|
+
move, and somebody has to notice and press Export again, which is exactly the
|
|
1478
|
+
kind of noticing that does not happen reliably.
|
|
1479
|
+
|
|
1480
|
+
So **Settings → Feeds → Scheduled exports** is a list of rows, beside the
|
|
1481
|
+
monitored folders and the scheduled sources:
|
|
1482
|
+
|
|
1483
|
+
```json
|
|
1484
|
+
"stackExports": [
|
|
1485
|
+
{
|
|
1486
|
+
"stack": "planet-terrain",
|
|
1487
|
+
"at": "03:30",
|
|
1488
|
+
"categories": ["basemaps"],
|
|
1489
|
+
"keep": 4,
|
|
1490
|
+
"savePath": "/mnt/fast",
|
|
1491
|
+
"attribution": "GEBCO 2026 | AW3D30 (JAXA)"
|
|
1492
|
+
},
|
|
1493
|
+
{
|
|
1494
|
+
"stack": "planet-terrain",
|
|
1495
|
+
"everyHours": 168,
|
|
1496
|
+
"publishDir": "/var/www/pmtiles",
|
|
1497
|
+
"webSeedBase": "https://maps.example/files"
|
|
1498
|
+
}
|
|
1499
|
+
]
|
|
1500
|
+
```
|
|
1501
|
+
|
|
1502
|
+
A row names the stack and when. `at` is a time of day in UTC, or a list of them;
|
|
1503
|
+
`everyHours` and `everyMinutes` are the interval form. That is the same shape a
|
|
1504
|
+
scheduled source uses and it is read by the same code, because they are one
|
|
1505
|
+
question and a node should not have two ways of answering it.
|
|
1506
|
+
|
|
1507
|
+
Everything else on a row is what the export dialog collects, plus the two
|
|
1508
|
+
retention rules every other automation on that tab has:
|
|
1509
|
+
|
|
1510
|
+
| Field | What it does |
|
|
1511
|
+
| ----------------------------------------------- | ---------------------------------------------------------------------------------------- |
|
|
1512
|
+
| `categories` | What each build is filed under, which is what the feed follows. Empty leaves it unfiled. |
|
|
1513
|
+
| `keep`, `keepDays` | Retirement, exactly as a watched folder does it. |
|
|
1514
|
+
| `name`, `attribution`, `description` | What the file says about itself once it is somewhere else. |
|
|
1515
|
+
| `savePath` | Where the data lands: a named location, or a path. |
|
|
1516
|
+
| `publishDir`, `webSeedBase` | A directory something already serves, and the URL it serves at. |
|
|
1517
|
+
| `serveArchive`, `selfWebSeed`, `publicDownload` | What this node itself does with the file: the **Local file** choice. |
|
|
1518
|
+
| `enabled: false` | Pauses the row without losing any of the above. |
|
|
1519
|
+
|
|
1520
|
+
#### A category belongs to the archive, not to the recipe
|
|
1521
|
+
|
|
1522
|
+
A stack has no categories, and the export is where they are said. That reads as
|
|
1523
|
+
a detail and is not: a category is what an archive is filed under — what
|
|
1524
|
+
`/latest/<category>/` follows and what a subscriber's feed carries — and a stack
|
|
1525
|
+
has no bytes and no infohash for any of that to be about. The stack editor has
|
|
1526
|
+
no field for one either.
|
|
1527
|
+
|
|
1528
|
+
Both doors used to fall back to `stack.categories`, a field the typedef never
|
|
1529
|
+
had, validation never checked and nothing ever wrote. Left empty now, the
|
|
1530
|
+
archive is **unfiled**: held and seeded, in no category and no feed. That is the
|
|
1531
|
+
right default for a one-off export and usually a mistake for a scheduled one —
|
|
1532
|
+
which is why the row says `unfiled` rather than leaving the box looking finished.
|
|
1533
|
+
|
|
1534
|
+
#### Serving what it just baked
|
|
1535
|
+
|
|
1536
|
+
**Local file** is the same choice every other row on that tab offers, and it is
|
|
1537
|
+
worth more here than anywhere else: a baked archive is already on this disk, so
|
|
1538
|
+
serving it costs a route rather than a download.
|
|
1539
|
+
|
|
1540
|
+
- **http** — this node serves the file at its archive URL.
|
|
1541
|
+
- **http + web seed** — and offers itself as a web seed for it, so a peer that
|
|
1542
|
+
finds nobody seeding can still fetch it over HTTP.
|
|
1543
|
+
- **http + catalog** — and lists it for download, so somebody with the link and
|
|
1544
|
+
no torrent client gets the file.
|
|
1545
|
+
|
|
1546
|
+
Without it a scheduled export produces an archive only peers can reach. With it,
|
|
1547
|
+
a nightly build is a URL that is always current, which is usually the point of
|
|
1548
|
+
scheduling one.
|
|
1549
|
+
|
|
1550
|
+
`publishDir` and `webSeedBase` are the other half of the same question and are
|
|
1551
|
+
for a directory **something else** already serves — nginx, a CDN. The two are
|
|
1552
|
+
not exclusive: publishing to a served directory and offering this node as a web
|
|
1553
|
+
seed gives a client two places to get the same bytes.
|
|
1554
|
+
|
|
1555
|
+
#### Rows, not a field on the recipe
|
|
1556
|
+
|
|
1557
|
+
Two other shapes were built first and both were the same mistake. An `export`
|
|
1558
|
+
block on each stack, then a table of one row per stack: neither can say that a
|
|
1559
|
+
stack has **two** schedules — a nightly build to the fast disk and a weekly one
|
|
1560
|
+
published somewhere else — which is an ordinary thing to want and impossible to
|
|
1561
|
+
express in a shape that holds one.
|
|
1562
|
+
|
|
1563
|
+
Rows also put the schedule where the other automations already are. A watched
|
|
1564
|
+
folder and a subscription are node-level: they say what _this machine_ does, not
|
|
1565
|
+
what a map is. A schedule is the same kind of statement, and keeping it out of
|
|
1566
|
+
the recipe means a recipe copied to another node does not quietly start baking
|
|
1567
|
+
there.
|
|
1568
|
+
|
|
1569
|
+
The export dialog no longer offers to repeat, for the same reason: a stack may
|
|
1570
|
+
have several schedules and a dialog opened on the stack cannot say which one it
|
|
1571
|
+
would be editing. It exports once and points at the settings tab.
|
|
1572
|
+
|
|
1573
|
+
#### Remembering across a restart
|
|
1574
|
+
|
|
1575
|
+
This is the part that differs from the source poller it borrows its schedule
|
|
1576
|
+
from, and it is the whole difficulty. The poller keeps last-run times in memory,
|
|
1577
|
+
so a restart makes everything due again — a missed poll costs one poll. A missed
|
|
1578
|
+
memory of a bake costs the whole bake: hours of reading and hundreds of gigabytes
|
|
1579
|
+
written, on every restart.
|
|
1580
|
+
|
|
1581
|
+
So it is written to `stack-exports.json` in the data directory, written-then-
|
|
1582
|
+
renamed like every other state file here, and written **before** the bake
|
|
1583
|
+
finishes rather than after. A restart in the middle of an export must not start
|
|
1584
|
+
it from the top; the checkpoint is what carries it on, and the schedule's job is
|
|
1585
|
+
only to not start a second one.
|
|
1586
|
+
|
|
1587
|
+
A row is remembered under the stack and its schedule — `planet-terrain@03:30` —
|
|
1588
|
+
rather than its position in the list. Two rows over one stack have to be told
|
|
1589
|
+
apart, and a list somebody reordered in the console must not make every schedule
|
|
1590
|
+
due again.
|
|
1591
|
+
|
|
1592
|
+
#### A run whose sources have not moved is skipped
|
|
1593
|
+
|
|
1594
|
+
`bakeRevision` already covers the recipe and what each source resolved to, and it
|
|
1595
|
+
is recorded beside the time. When the next run comes round and the revision
|
|
1596
|
+
matches, nothing is baked — the archive would be the same map under a new
|
|
1597
|
+
infohash, which then has to be seeded beside the one it duplicates, and every
|
|
1598
|
+
subscriber has to fetch it to find out it changed nothing.
|
|
1599
|
+
|
|
1600
|
+
The clock is still written down in that case. Without it the comparison would be
|
|
1601
|
+
repeated on every tick for the rest of the day, which is cheap but pointless.
|
|
1602
|
+
|
|
1603
|
+
#### Retiring what it produced
|
|
1604
|
+
|
|
1605
|
+
Without this a nightly export is a disk that fills at one archive a night, which
|
|
1606
|
+
for a planet build is the fastest way to fill one that this project has. `keep`
|
|
1607
|
+
and `keepDays` are the same two rules a watched folder uses and they are applied
|
|
1608
|
+
by the same code.
|
|
1609
|
+
|
|
1610
|
+
What was missing was a _family_: retiring needs to know which archives are builds
|
|
1611
|
+
of the same map. A watched folder marks its imports with the folder they came
|
|
1612
|
+
from; a bake marked nothing, so there was nothing to compare. It records
|
|
1613
|
+
`source.stack` now, and the family is every archive that names this stack.
|
|
1614
|
+
|
|
1615
|
+
#### What it will not do
|
|
1616
|
+
|
|
1617
|
+
**Two at once.** A bake reads every tile its sources hold; two competing for the
|
|
1618
|
+
same disk and the same cores finish later than one after the other. Anything
|
|
1619
|
+
still due is due a minute later, so the tick is the queue.
|
|
1620
|
+
|
|
1621
|
+
**Start one over a bake already running.** That is either a schedule catching up
|
|
1622
|
+
with an export taking longer than its interval, or a stack's second schedule
|
|
1623
|
+
coming round while its first is still going.
|
|
1624
|
+
|
|
1625
|
+
**Give up quietly.** A refusal — a location that is full, a codec that is not
|
|
1626
|
+
installed, a row naming a stack that has been deleted — is not recorded as a run,
|
|
1627
|
+
so the next tick tries again and says so. A schedule that recorded the attempt
|
|
1628
|
+
would wait a day before showing it had ever run.
|
|
1629
|
+
|
|
1630
|
+
`stacks.scheduledExports: false`, beside the rows, turns the whole thing off.
|
|
1631
|
+
That is what a second node serving the same stacks wants: only one of them
|
|
1632
|
+
should be the node that bakes.
|
|
1633
|
+
|
|
1453
1634
|
## Finding a stack
|
|
1454
1635
|
|
|
1455
1636
|
A stack has no infohash and appears in no feed, so nothing about it is
|
|
@@ -1466,46 +1647,107 @@ advertised as a link that answers 501.
|
|
|
1466
1647
|
|
|
1467
1648
|
## Syncing a stack to another node
|
|
1468
1649
|
|
|
1469
|
-
|
|
1470
|
-
|
|
1471
|
-
|
|
1472
|
-
|
|
1473
|
-
|
|
1474
|
-
|
|
1475
|
-
|
|
1476
|
-
|
|
1477
|
-
|
|
1478
|
-
|
|
1479
|
-
|
|
1480
|
-
|
|
1481
|
-
|
|
1482
|
-
|
|
1483
|
-
|
|
1484
|
-
|
|
1485
|
-
|
|
1486
|
-
|
|
1487
|
-
|
|
1488
|
-
|
|
1489
|
-
|
|
1490
|
-
|
|
1491
|
-
|
|
1492
|
-
|
|
1493
|
-
|
|
1494
|
-
|
|
1495
|
-
|
|
1496
|
-
|
|
1497
|
-
|
|
1498
|
-
|
|
1499
|
-
|
|
1500
|
-
|
|
1501
|
-
|
|
1502
|
-
|
|
1503
|
-
|
|
1504
|
-
|
|
1505
|
-
|
|
1506
|
-
|
|
1507
|
-
|
|
1508
|
-
|
|
1650
|
+
Archives already travel. A node follows a category and the builds arrive, which
|
|
1651
|
+
is how a builder feeds a pair of tile servers. The recipes that combine them did
|
|
1652
|
+
not travel, so those two had the same archives and no way to have the same
|
|
1653
|
+
stacks — every recipe typed twice, and corrected twice.
|
|
1654
|
+
|
|
1655
|
+
```
|
|
1656
|
+
planetgen ──── category feeds ────▶ TilerServer-01
|
|
1657
|
+
│ (archives) └──▶ TilerServer-02
|
|
1658
|
+
└────────── /stacks.xml ─────────▶ (recipes)
|
|
1659
|
+
```
|
|
1660
|
+
|
|
1661
|
+
`GET /stacks.xml` is this node's own stacks as a feed. A subscriber lists it
|
|
1662
|
+
under **Settings → Feeds → Stack feeds** with how often to check, and adopts
|
|
1663
|
+
what it carries.
|
|
1664
|
+
|
|
1665
|
+
### Why a feed, when this section used to argue against one
|
|
1666
|
+
|
|
1667
|
+
The objection was that a stack is a mutable document, so syncing it is a question
|
|
1668
|
+
about conflicts and clobbering rather than about missing pieces. That is true
|
|
1669
|
+
where two nodes both edit the same recipe, and it is not the arrangement anybody
|
|
1670
|
+
is actually running. One node authors and the rest follow — exactly as they
|
|
1671
|
+
follow a category — and in that shape there is nothing to conflict.
|
|
1672
|
+
|
|
1673
|
+
What survives from the objection is the clobbering, and it is handled directly:
|
|
1674
|
+
**a stack made on this node is never overwritten by one arriving under the same
|
|
1675
|
+
name.** That is refused and said, rather than resolved in either direction, and
|
|
1676
|
+
it is the only outcome here that could lose work.
|
|
1677
|
+
|
|
1678
|
+
### The name is the publisher's
|
|
1679
|
+
|
|
1680
|
+
A recipe is adopted under the id it has on the publisher. `planet-terrain` is
|
|
1681
|
+
`planet-terrain` on the builder and on every replica, so one URL answers on all
|
|
1682
|
+
of them — which is what puts a load balancer in front of them at all. Namespacing
|
|
1683
|
+
it as `planetgen:planet-terrain` would have been safer against collisions and
|
|
1684
|
+
would have given three nodes three different URLs, which defeats the purpose.
|
|
1685
|
+
|
|
1686
|
+
### A missing source is not a reason to refuse
|
|
1687
|
+
|
|
1688
|
+
A recipe naming a category resolves to whatever that node's newest build of it
|
|
1689
|
+
is, which is the point: the two tile servers follow the same category feeds, so
|
|
1690
|
+
they resolve to the same archive. A recipe naming an infohash needs that exact
|
|
1691
|
+
build, and a replica may not have it yet.
|
|
1692
|
+
|
|
1693
|
+
Adopted either way. The stack reports the missing source through `problems`, the
|
|
1694
|
+
console shows it as it shows any other unresolved source, and its tiles answer
|
|
1695
|
+
for what they cannot serve until the archive arrives. Refusing it instead would
|
|
1696
|
+
mean a replica could not be set up until every archive had finished downloading,
|
|
1697
|
+
which is backwards: the recipe is the small, fast half.
|
|
1698
|
+
|
|
1699
|
+
### What a feed carries, and what it does not
|
|
1700
|
+
|
|
1701
|
+
The recipe travels **inside** the item rather than behind a link, because it is a
|
|
1702
|
+
few hundred bytes: a subscriber that has read the feed has the recipe, with
|
|
1703
|
+
nothing else to fetch and no second request to authenticate. Each item also
|
|
1704
|
+
carries the recipe's revision, so a poll costs a comparison rather than a
|
|
1705
|
+
document when nothing has changed.
|
|
1706
|
+
|
|
1707
|
+
A node publishes only the stacks it authored. Republishing what it adopted would
|
|
1708
|
+
put two nodes' names on one recipe, and two nodes following each other would hand
|
|
1709
|
+
it back and forth for ever.
|
|
1710
|
+
|
|
1711
|
+
The feed is public, like the archive feeds beside it. A recipe names categories
|
|
1712
|
+
and infohashes, both of which the catalogue already publishes, so it gives away
|
|
1713
|
+
nothing that the feed next to it does not.
|
|
1714
|
+
|
|
1715
|
+
### When the publisher stops carrying one
|
|
1716
|
+
|
|
1717
|
+
Per feed, because it depends on what the far node is:
|
|
1718
|
+
|
|
1719
|
+
- **keep it, and say so** — the recipe stays and goes on serving; the console
|
|
1720
|
+
says the feed no longer carries it. A deletion on the author, accidental or
|
|
1721
|
+
not, never takes a working endpoint down across every replica at once.
|
|
1722
|
+
- **remove it here too** — a replica mirrors its author. Simple and consistent,
|
|
1723
|
+
and an accidental delete propagates within one poll.
|
|
1724
|
+
|
|
1725
|
+
A stack that comes back has the mark taken off again rather than left to puzzle
|
|
1726
|
+
over.
|
|
1727
|
+
|
|
1728
|
+
### One stack's feed, and the whole node's
|
|
1729
|
+
|
|
1730
|
+
`/stacks.xml` carries every stack this node authored, which is what a replica
|
|
1731
|
+
wants: one row in its settings and it follows all of them, including ones added
|
|
1732
|
+
later.
|
|
1733
|
+
|
|
1734
|
+
`/stacks/<id>.xml` carries one. That is for the node that wants a single map out
|
|
1735
|
+
of somebody's several and does not want the rest appearing on it every time they
|
|
1736
|
+
add one. It 404s for a stack this node adopted rather than serving it — a feed
|
|
1737
|
+
that exists and carries nothing looks like a stack that was withdrawn, which is
|
|
1738
|
+
a different thing.
|
|
1739
|
+
|
|
1740
|
+
Both are offered where the addresses are: an **RSS** button on the stack's row
|
|
1741
|
+
in the console, and beside TileJSON and XYZ on the public page. Copied rather
|
|
1742
|
+
than followed, because the address is for another node's settings and a browser
|
|
1743
|
+
shown an RSS document mostly offers to download it.
|
|
1744
|
+
|
|
1745
|
+
### Baking is still the other answer
|
|
1746
|
+
|
|
1747
|
+
A baked stack is an ordinary archive with an infohash, and archives already sync.
|
|
1748
|
+
Where what is wanted on the far node is the _output_ rather than the _recipe_,
|
|
1749
|
+
that is the mechanism, and it carries no ambiguity at all about what the sources
|
|
1750
|
+
resolved to. See "Exporting on a schedule".
|
|
1509
1751
|
|
|
1510
1752
|
## What the offline merge got wrong
|
|
1511
1753
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pmtiles-swarm",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.89.0",
|
|
4
4
|
"description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "src/index.js",
|