pmtiles-swarm 0.80.0 → 0.90.0

package/CHANGELOG.md

@@ -7,6 +7,313 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.90.0
+### ✨ Features and improvements
+- **A stack source may name a URL instead of a category, archive or stack.** `{ "url": "https://…" }`,
+  read straight over HTTP with no torrent involved — for an archive published as a plain download,
+  like Mapterhorn's terrain: a global base plus hundreds of regional patches, 11.8 TiB in total and
+  never meant to be downloaded whole. `FetchSource`, from the `pmtiles` package this project already
+  depends on, asks for byte ranges the way a swarm-backed source does; a tile costs the header once,
+  the directory once, and the tile itself.
+
+  Parent climbing works the same way it does for a category source — a shallow archive upscales for a
+  deeper request through the same code, since reading one is now only a question of which store method
+  answers. New per-source `minzoom` and `maxzoom` fields, checked before anything is opened, are what
+  make hundreds of these practical: a tile request outside a source's stated box or zoom range skips it
+  without a request leaving this node, which for a stack built from a provider's whole file list is
+  most of them, on every tile. A URL source always decodes rather than passing through raw, having no
+  infohash to answer that question with, and is not seeded, retired or rebuilt — none of the mechanisms
+  built for an archive this node actually holds apply to one it does not.
+
+- **A provider's list of PMTiles URLs can be imported as sources.** **Stacks → Import URL list…**,
+  or `POST /api/stacks/<id>/import`. Mapterhorn's `download_urls.json` names 458 files with the box
+  and zoom range of each; naming those by hand is not work anybody should do once, let alone again
+  when the provider adds one. An index with an `items` list is read, and so is a plain list of
+  addresses for a provider that publishes no index — which shape it is is worked out from the
+  document, since somebody pasting an address has no reason to know.
+
+  The global file becomes `sources[0]` and `required`: a stack is painted bottom-first, so the thing
+  covering everywhere has to sit under everything patching it, and the index does not list it first.
+  Every other entry keeps its box and zoom range, which is what lets a tile outside one skip it
+  without a request leaving the node. The encoding is asked for rather than read — an index rarely
+  states it, Mapterhorn's files are all terrarium and its JSON never says so, and a terrain source
+  read with the wrong one is a cliff face.
+
+  The list is fetched **by the node**, not the browser: the console is often on a different network,
+  and what matters is that the machine which will read the archives can reach them. **Check** shows
+  what an import would write before it writes anything.
+
+  In the editor an imported batch is one row rather than several hundred cards, with **Re-import**
+  and **Remove all**. A re-import replaces only what came from that same address — anything typed by
+  hand is left alone — and puts the batch back **where it already was** rather than on the end:
+  painting order is the whole meaning of a stack, and a batch that moved each time would quietly bury
+  a local override, a day later, on a schedule, with nothing to say why the map had changed.
+
+
+- **A stack can follow a provider's file list on a timer.** A stack feed row with **Into stack** set
+  reads a URL list rather than another node's recipes, and keeps the named stack level with it.
+  Mapterhorn's index has grown through several versions, and a node that imported it once is a node
+  serving whenever that was. The row says which of the two it is by whether **Into stack** is filled
+  in — a feed of recipes names its own stacks and an index cannot, being a list of files with no
+  opinion about what they are for, so the row supplies the stack and the encoding as well.
+
+  It reconciles exactly as **Re-import** does: hand-written sources untouched, the batch back where it
+  was, a withdrawn file dropped rather than left to 404 per tile. A poll where nothing changed writes
+  nothing at all — not the same recipe again, which would move its revision and with it every tile
+  cached against it. A stack that does not exist yet is created; one that does keeps everything it had.
+
+
+### 🐞 Bug fixes
+- **A stack reported no encoding unless its recipe stated one outright.** Reading `output.encoding`
+  alone reports null for the ordinary recipe — the one that re-encodes to nothing and writes whatever
+  its base source is written in, which is every imported stack, since an import sets the encoding per
+  source. Three places read it that way and all three were wrong about terrain: the console offered no
+  terrain preview, the TileJSON told clients nothing so terrarium heights rendered through the mapbox
+  formula, and a baked archive was written with no encoding metadata at all — an export that had to be
+  corrected by hand afterwards to be readable.
+
+- **A stack's TileJSON listed every source.** For a stack imported from a provider's index that is 458
+  addresses and 55 KB, in a document every map load fetches, describing files a client cannot use and
+  will never ask for — it reads tiles from the XYZ endpoint. It now names 25 with a count of the rest.
+  The extension it advertises is the one the endpoint actually serves, too: a stack whose sources state
+  no format has no coverage format to read, and the document said `.bin` for tiles answered as webp.
+
+- **Cancel asked you to fill the form in first.** A `<button>` in a form submits, and a submit is
+  checked against the form's required fields before it goes anywhere — so cancelling the import or
+  warm dialog answered "Please fill out this field" about a field that was about to be discarded.
+
+
+## 0.89.0
+### ✨ Features and improvements
+- **Both feeds are linked at the foot of the public page.** `RSS feed` is `archive RSS feed` now and
+  `stack RSS feed` is beside it. One name was unambiguous while there was one feed; with a second, a
+  reader following the old name would have got archives when they wanted recipes and had nothing on
+  the page to tell them otherwise.
+
+
+### 🐞 Bug fixes
+- **A feed carrying every category.** `/categories.xml`, beside `/feed.xml` and `/stacks.xml` — the
+  set was inconsistent, and following all of a node's categories meant adding each by hand and
+  remembering to add the next one, which is not following its categories at all. It is what a
+  subscriber wants when they want everything this node files: including categories added later.
+
+  The items are archives, because a category has no bytes of its own and the build it resolves to is
+  what a subscriber would join — which also means every existing consumer already knows how to read
+  it, magnets and enclosures included. That is what separates it from the whole catalogue: `/feed.xml`
+  carries every build a node holds, and this carries the current one of each category. On a node
+  keeping four builds apiece the difference is fourfold.
+
+  In the footer of the public page it replaces `categories JSON`, which was the odd one out among
+  three feeds. `/latest/` is untouched and remains the JSON index.
+
+
+## 0.88.0
+### ✨ Features and improvements
+- **An archive card is built like a category's.** Copy buttons for the addresses that belong
+  somewhere else — TileJSON, magnet, and a **source URL** of its own, the TileJSON URL with the
+  `.torrent` and magnet in its fragment, pinned to that build rather than following the category.
+  The preview, `.torrent` and download stay links, because a page is followed and a file is saved.
+  The printed TileJSON row went with the button that replaced it: an address in full beside a button
+  that copies it is the same fact twice, and it was the longest line on the card.
+
+
+### 🐞 Bug fixes
+- **A terrain stack was offered no hillshade preview on the public page.** The listing reported its
+  encoding as whatever `output.encoding` restated, which for a recipe saying "same as the sources" —
+  the ordinary case — is nothing. So the page could not tell a terrain stack from an imagery one and
+  offered neither the raw preview nor the terrain one. It reports what the stack actually serves now,
+  and reports nothing for imagery, where a guess would have offered a hillshade of a photograph.
+
+- **A stack whose sources disagreed about their encoding wrote tiles in more than one of them.** With
+  no `output.encoding`, the merge took the encoding of whichever source answered first — and which
+  source answers varies by tile, since the base is sparse in one place and the layer above covers in
+  another. One stack wrote one tile as mapbox and the next as terrarium, with the TileJSON in front
+  describing neither: correct in one place, a cliff face in another, for no reason the recipe showed.
+  It follows the base source now, which is a property of the recipe rather than of the tile, and the
+  listing and the merge are held to the same answer by a test.
+
+
+## 0.87.0
+### ✨ Features and improvements
+- **A feed per stack, and an address to copy for it.** `/stacks/<id>.xml` carries one recipe, for
+  following a single map out of somebody's several rather than everything they publish. An **RSS**
+  button on the stack's row in the console and beside TileJSON and XYZ on the public page. Copied
+  rather than followed: the address is for another node's settings, and a browser shown an RSS
+  document mostly offers to download it. A stack this node adopted has no button and its feed 404s —
+  somebody else's recipe is theirs to publish.
+
+
+### 🐞 Bug fixes
+- **The copied XYZ address came back percent-encoded.** `http://…/stacks/test/%7Bz%7D/%7Bx%7D/%7By%7D.png`
+  rather than `{z}/{x}/{y}`, which no client will take. The public page resolves a relative address
+  against the page's own URL to make it absolute, and `new URL` percent-encodes braces because they
+  are not legal in a path — and an XYZ template is very little but braces. The TileJSON link was
+  unaffected, having none. Resolving still happens; the two sequences it introduces are put back.
+
+
+## 0.86.0
+### ✨ Features and improvements
+- _...Add new stuff here..._
+
+- **Stack recipes travel between nodes.** `GET /stacks.xml` is this node's own stacks as a feed;
+  **Settings → Feeds → Stack feeds** follows another node's and adopts what it carries. Archives
+  already travelled by category — a builder feeding two tile servers gave them the same archives —
+  and this is how the recipes that combine them travel with them, so a stack is written once and
+  corrected once.
+
+  This section of the docs used to argue against a feed, on the grounds that a stack is a mutable
+  document and syncing one is about conflicts. That holds where two nodes both edit a recipe and is
+  not the arrangement anybody runs: one node authors, the rest follow. What survives from the
+  objection is handled directly — **a stack made on this node is never overwritten by one arriving
+  under the same name**, which is refused and said.
+
+  A recipe is adopted under the publisher's own id, so `planet-terrain` answers at the same URL on
+  the builder and on every replica; namespacing it would have given three nodes three URLs and
+  defeated the point. A recipe naming a source this node has not got is adopted anyway and reports
+  the missing source until it arrives — refusing it would mean a replica could not be set up until
+  every archive had finished downloading, which is backwards.
+
+  What happens when a feed stops carrying a stack is the feed's own setting: keep it and say so, or
+  remove it here too.
+
+### 🐞 Bug fixes
+- _...Add new stuff here..._
+
+## 0.85.1
+### ✨ Features and improvements
+- _...Add new stuff here..._
+
+- **An export no longer pretends a stack has categories.** Both doors fell back to
+  `stack.categories` when none were given — a field the typedef never had, validation never checked,
+  the editor has no box for and nothing ever wrote. A category is what an *archive* is filed under,
+  and a stack has no bytes and no infohash for that to be about.
+
+  Left empty the archive is unfiled: held and seeded, in no category and no feed. The scheduled
+  export row says `unfiled` and the dialog says so in full, rather than offering a default that
+  could not exist.
+
+### 🐞 Bug fixes
+- _...Add new stuff here..._
+
+## 0.85.0
+### ✨ Features and improvements
+- _...Add new stuff here..._
+
+- **A scheduled export can be served by the node that baked it.** The **Local file** choice, the same
+  one every other row under Feeds offers: http, http + web seed, http + catalog. It is worth more
+  here than anywhere else — a baked archive is already on this disk, so serving it costs a route
+  rather than a download, and without it a scheduled export produces something only peers can reach.
+  A nightly build behind a URL that is always current is usually the point of scheduling one.
+
+  `publishDir` and `webSeedBase` remain the other half of the same question, for a directory
+  something else already serves. The two are not exclusive: publishing to a served directory *and*
+  offering this node as a web seed gives a client two places to get the same bytes.
+
+### 🐞 Bug fixes
+- _...Add new stuff here..._
+
+## 0.84.0
+### ✨ Features and improvements
+- _...Add new stuff here..._
+
+- **Scheduled exports are rows, like every other automation beside them.** `stackExports` in the
+  config, edited under Settings → Feeds with the same row editor the monitored folders and the
+  scheduled sources use: add a row, choose the stack from a dropdown, say when, and fill in the rest
+  — categories, builds to keep, keep for how many days, save location, publish directory, web seed
+  base, archive name, attribution, description.
+
+  **Several rows may name one stack.** That is what the two earlier shapes could not say: an `export`
+  block on the recipe, and then a table of one row per stack, both hold exactly one schedule — and a
+  nightly build to the fast disk beside a weekly one published elsewhere is an ordinary thing to
+  want. It also puts the schedule where the other automations already are: a watched folder says what
+  *this machine* does, not what a map is, and keeping it out of the recipe means a recipe copied to
+  another node does not quietly start baking there.
+
+  The export dialog no longer offers to repeat — a stack may have several schedules and a dialog
+  opened on the stack cannot say which it would edit. It exports once and points at the settings tab.
+  A stack's row in the Stacks view still shows the schedules aimed at it.
+
+### 🐞 Bug fixes
+- _...Add new stuff here..._
+
+## 0.83.0
+### ✨ Features and improvements
+- _...Add new stuff here..._
+
+- **A scheduled export is set up like the other automations beside it.** The row under Settings →
+  Feeds is the whole of one now, not only the timer: categories, builds to keep, keep for how many
+  days, archive name, attribution, description, where the data lands, a publish directory and a web
+  seed base. Those are the manual export dialog's fields plus the two retention rules every other
+  automation on that tab already had, which is the point — a scheduled export is set up in one place
+  rather than half in a settings tab and half in a dialog.
+
+  Retirement is the part that needed something new underneath. `keep` and `keepDays` are applied by
+  the same code a watched folder uses, but retiring needs a *family* — which archives are builds of
+  the same map — and a bake marked nothing, so there was nothing to compare. An archive now records
+  `source.stack`, and the family is every archive that names this stack. Without it a nightly export
+  is a disk that fills at one archive a night.
+
+### 🐞 Bug fixes
+- _...Add new stuff here..._
+
+## 0.82.0
+### ✨ Features and improvements
+- _...Add new stuff here..._
+
+- **Export schedules are set under Settings → Feeds.** Not only the two global settings that landed
+  there in 0.81.1 — the schedules themselves. **Scheduled exports** is a row per stack: never, every
+  day at a time, or every so many hours, saved onto that stack's recipe. Feeds is where somebody goes
+  to say when a thing runs, so it is where they are set; the export dialog's **Repeat** control still
+  writes the same schedule, for setting one while you are already there.
+
+  Turning a stack to _never_ pauses it with `enabled: false` rather than deleting the block. Where it
+  lands, what it is called and which categories it is filed under live in the same place, and those
+  should survive being paused.
+
+### 🐞 Bug fixes
+- _...Add new stuff here..._
+
+## 0.81.1
+### ✨ Features and improvements
+- _...Add new stuff here..._
+
+- **The scheduled-export settings are under Settings → Feeds.** `stacks.scheduledExports` and
+  `stacks.exportIntervalHours` had no schema entry, so they were config-file-only and invisible in
+  the console. Feeds rather than a group of their own, because that tab is already where the
+  automations that bring a file in on a timer sit — a scheduled source watching an upstream
+  directory, a subscription following someone else's feed. A scheduled export is the same kind of
+  thing; it just produces the file here instead of fetching it, and it lands in a category and goes
+  out over the feed exactly as a fetched one does.
+
+### 🐞 Bug fixes
+- _...Add new stuff here..._
+
+## 0.81.0
+### ✨ Features and improvements
+- _...Add new stuff here..._
+
+- **A stack can export itself on a schedule.** An archive is a snapshot: a stack over categories
+  follows a rebuild and a file baked from it does not, so it goes stale the moment its sources move
+  and somebody has to notice. An `export` block on the recipe says when — `at` for a time of day in
+  UTC, `everyHours` or `everyMinutes` for an interval, the same shape a scheduled source uses and
+  read by the same code — along with everything the export dialog collects. In the console, **Repeat**
+  turns the Export button into **Save schedule**, and the stack's row shows when it runs.
+
+  The hard part is remembering across a restart. The source poller keeps last-run times in memory,
+  which is fine when a missed poll costs one poll; here it would cost the whole bake, every restart,
+  for hours. So it is written to `stack-exports.json`, and written *before* the bake finishes — a
+  restart mid-export must not start it from the top, since the checkpoint is what carries it on.
+
+  A run whose sources have not moved is skipped: `bakeRevision` is recorded beside the time, and an
+  identical archive is the same map under a new infohash that then has to be seeded beside the one it
+  duplicates. It will not run two at once, will not start one over a bake already running, and does
+  not record a refusal as a run — a location that is full is something somebody fixes, and a schedule
+  that gave up quietly would hide that it ever ran. `stacks.scheduledExports: false` turns it off,
+  which is what a second node serving the same recipes wants.
+
+### 🐞 Bug fixes
+- _...Add new stuff here..._
+
 ## 0.80.0
 ### ✨ Features and improvements
 - **A stack can be a source in another stack.** `{ "stack": "jaxa-with-gebco" }` beside `category`

package/README.md

@@ -66,7 +66,7 @@ been sending and receiving, kept in SQLite so it survives a restart; an indicato
 peers can reach this node at all; and a settings screen covering monitored folders, watched web
 locations, remote nodes, save locations, access tokens and the external-program hooks.
 
-**Publishes RSS.** `/feed.xml`, and `/feed/<category>.xml` per category. Plain RSS 2.0 with
+**Publishes RSS.** `/feed.xml` for every archive, `/categories.xml` for the build each category currently resolves to, `/stacks.xml` for the stack recipes, and `/feed/<category>.xml` per category. Plain RSS 2.0 with
 torrent enclosures, so **qBittorrent's built-in RSS auto-downloader can subscribe today** with no
 new software. Items also carry a namespaced description of the map — format, zoom range, bounds,
 tile count — so a subscriber can decide whether it wants a 72 GiB download before starting one.
@@ -807,12 +807,15 @@ which the endpoint answers 501.
 | `GET`                  | `/api/torrents/:infoHash/stacks`                            | Which stacks would break if this archive were removed, and how                                                                                                                                                                                                                                                                             |
 | `GET`, `PUT`, `DELETE` | `/api/stacks/:id/raw`, `/api/stacks/:id`                    | Read a recipe as written, save one, or remove it. `/raw` is the recipe; `/api/stacks` is what it resolved to                                                                                                                                                                                                                               |
 
+| `POST` | `/api/stacks/:id/import` | Import a provider's list of PMTiles URLs as sources on this stack. An index with an `items` list, or a plain list of addresses; `dryRun` says what it would write without writing it |
 | `POST`, `DELETE` | `/api/stacks/:id/bake` | Run the stack over its sources and write the result as an archive, or stop one that is running. Answers as soon as the job starts; watch it on `/api/stacks` |
 | `GET` | `/stacks/:id/preview` | A map of a stack, for looking at it — **public** |
 | `GET` | `/stacks/:id/tiles.json` | TileJSON for a stack — **public**. `maxzoom` is the maximum over its sources, not the minimum |
 | `GET` | `/stacks/:id/:size/:z/:x/:y.:ext` | The same tile at 256 or 512 px. A tile's coordinates are an extent rather than a pixel count, so this is the same ground on a finer or coarser grid |
 | `GET` | `/stacks/:id/:z/:x/:y.:ext` | One tile of a stack — **public**. Answered by the topmost source holding it; `X-Stack-Sources` says which were asked and what each said |
 | `GET` | `/feed.xml`, `/feed/:category.xml`, `/latest/:category.xml` | RSS — **public** |
+| `GET` | `/categories.xml` | Every category, as the build each resolves to — **public** |
+| `GET` | `/stacks.xml`, `/stacks/:id.xml` | Stack recipes, for another node to follow — **public** |
 
 Everything under `/api/` is guarded once a credential is configured; tiles, TileJSON and the feeds
 never are. A `peer` token may read but not change, and may be narrowed to some categories. See