pmtiles-swarm 0.89.0 → 0.90.0

package/CHANGELOG.md

@@ -7,6 +7,82 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.90.0
+### ✨ Features and improvements
+- **A stack source may name a URL instead of a category, archive or stack.** `{ "url": "https://…" }`,
+  read straight over HTTP with no torrent involved — for an archive published as a plain download,
+  like Mapterhorn's terrain: a global base plus hundreds of regional patches, 11.8 TiB in total and
+  never meant to be downloaded whole. `FetchSource`, from the `pmtiles` package this project already
+  depends on, asks for byte ranges the way a swarm-backed source does; a tile costs the header once,
+  the directory once, and the tile itself.
+
+  Parent climbing works the same way it does for a category source — a shallow archive upscales for a
+  deeper request through the same code, since reading one is now only a question of which store method
+  answers. New per-source `minzoom` and `maxzoom` fields, checked before anything is opened, are what
+  make hundreds of these practical: a tile request outside a source's stated box or zoom range skips it
+  without a request leaving this node, which for a stack built from a provider's whole file list is
+  most of them, on every tile. A URL source always decodes rather than passing through raw, having no
+  infohash to answer that question with, and is not seeded, retired or rebuilt — none of the mechanisms
+  built for an archive this node actually holds apply to one it does not.
+
+- **A provider's list of PMTiles URLs can be imported as sources.** **Stacks → Import URL list…**,
+  or `POST /api/stacks/<id>/import`. Mapterhorn's `download_urls.json` names 458 files with the box
+  and zoom range of each; naming those by hand is not work anybody should do once, let alone again
+  when the provider adds one. An index with an `items` list is read, and so is a plain list of
+  addresses for a provider that publishes no index — which shape it is is worked out from the
+  document, since somebody pasting an address has no reason to know.
+
+  The global file becomes `sources[0]` and `required`: a stack is painted bottom-first, so the thing
+  covering everywhere has to sit under everything patching it, and the index does not list it first.
+  Every other entry keeps its box and zoom range, which is what lets a tile outside one skip it
+  without a request leaving the node. The encoding is asked for rather than read — an index rarely
+  states it, Mapterhorn's files are all terrarium and its JSON never says so, and a terrain source
+  read with the wrong one is a cliff face.
+
+  The list is fetched **by the node**, not the browser: the console is often on a different network,
+  and what matters is that the machine which will read the archives can reach them. **Check** shows
+  what an import would write before it writes anything.
+
+  In the editor an imported batch is one row rather than several hundred cards, with **Re-import**
+  and **Remove all**. A re-import replaces only what came from that same address — anything typed by
+  hand is left alone — and puts the batch back **where it already was** rather than on the end:
+  painting order is the whole meaning of a stack, and a batch that moved each time would quietly bury
+  a local override, a day later, on a schedule, with nothing to say why the map had changed.
+
+
+- **A stack can follow a provider's file list on a timer.** A stack feed row with **Into stack** set
+  reads a URL list rather than another node's recipes, and keeps the named stack level with it.
+  Mapterhorn's index has grown through several versions, and a node that imported it once is a node
+  serving whenever that was. The row says which of the two it is by whether **Into stack** is filled
+  in — a feed of recipes names its own stacks and an index cannot, being a list of files with no
+  opinion about what they are for, so the row supplies the stack and the encoding as well.
+
+  It reconciles exactly as **Re-import** does: hand-written sources untouched, the batch back where it
+  was, a withdrawn file dropped rather than left to 404 per tile. A poll where nothing changed writes
+  nothing at all — not the same recipe again, which would move its revision and with it every tile
+  cached against it. A stack that does not exist yet is created; one that does keeps everything it had.
+
+
+### 🐞 Bug fixes
+- **A stack reported no encoding unless its recipe stated one outright.** Reading `output.encoding`
+  alone reports null for the ordinary recipe — the one that re-encodes to nothing and writes whatever
+  its base source is written in, which is every imported stack, since an import sets the encoding per
+  source. Three places read it that way and all three were wrong about terrain: the console offered no
+  terrain preview, the TileJSON told clients nothing so terrarium heights rendered through the mapbox
+  formula, and a baked archive was written with no encoding metadata at all — an export that had to be
+  corrected by hand afterwards to be readable.
+
+- **A stack's TileJSON listed every source.** For a stack imported from a provider's index that is 458
+  addresses and 55 KB, in a document every map load fetches, describing files a client cannot use and
+  will never ask for — it reads tiles from the XYZ endpoint. It now names 25 with a count of the rest.
+  The extension it advertises is the one the endpoint actually serves, too: a stack whose sources state
+  no format has no coverage format to read, and the document said `.bin` for tiles answered as webp.
+
+- **Cancel asked you to fill the form in first.** A `<button>` in a form submits, and a submit is
+  checked against the form's required fields before it goes anywhere — so cancelling the import or
+  warm dialog answered "Please fill out this field" about a field that was about to be discarded.
+
+
 ## 0.89.0
 ### ✨ Features and improvements
 - **Both feeds are linked at the foot of the public page.** `RSS feed` is `archive RSS feed` now and

package/README.md

@@ -807,6 +807,7 @@ which the endpoint answers 501.
 | `GET`                  | `/api/torrents/:infoHash/stacks`                            | Which stacks would break if this archive were removed, and how                                                                                                                                                                                                                                                                             |
 | `GET`, `PUT`, `DELETE` | `/api/stacks/:id/raw`, `/api/stacks/:id`                    | Read a recipe as written, save one, or remove it. `/raw` is the recipe; `/api/stacks` is what it resolved to                                                                                                                                                                                                                               |
 
+| `POST` | `/api/stacks/:id/import` | Import a provider's list of PMTiles URLs as sources on this stack. An index with an `items` list, or a plain list of addresses; `dryRun` says what it would write without writing it |
 | `POST`, `DELETE` | `/api/stacks/:id/bake` | Run the stack over its sources and write the result as an archive, or stop one that is running. Answers as soon as the job starts; watch it on `/api/stacks` |
 | `GET` | `/stacks/:id/preview` | A map of a stack, for looking at it — **public** |
 | `GET` | `/stacks/:id/tiles.json` | TileJSON for a stack — **public**. `maxzoom` is the maximum over its sources, not the minimum |

package/docs/tile-stacks.md

@@ -53,6 +53,8 @@ of its parts.
 - [What a mask has to match](#what-a-mask-has-to-match)
 - [Feathering a seam](#feathering-a-seam)
 - [A stack as a source](#a-stack-as-a-source)
+- [A source read straight from a URL](#a-source-read-straight-from-a-url)
+- [Importing a list of URLs](#importing-a-list-of-urls)
   - [Exporting on a schedule](#exporting-on-a-schedule)
 - [Finding a stack](#finding-a-stack)
 - [Syncing a stack to another node](#syncing-a-stack-to-another-node)
@@ -255,20 +257,23 @@ it differs from the snake_case rio-rgbify-merge uses.
 
 ### Source fields
 
-| Field                  | Meaning                                                                                                      |
-| ---------------------- | ------------------------------------------------------------------------------------------------------------ |
-| `category`             | Resolve to the newest build in this category.                                                                |
-| `archive`              | Or pin one infohash. Exactly one of the two.                                                                 |
-| `required`             | A tile fails rather than being served without this source. Defaults true for the bottom-most, false above.   |
-| `encoding`             | `mapbox` or `terrarium`. Elevation space only.                                                               |
-| `baseVal` / `interval` | Mapbox decode offset and step. Default `-10000` and `0.1`.                                                   |
-| `maskValues`           | Decoded heights meaning "no data here". Elevation space only.                                                |
-| `maskRange`            | `[low, high]` in metres, or a list of them. Everything inside is nodata. Elevation space only.               |
-| `heightAdjustment`     | Metres, added **after** masking. Elevation space only.                                                       |
-| `feather`              | Pixels to fade in over wherever the source stops: a `cutline`, `bounds`, or the holes a mask leaves. Max 64. |
-| `featherMetres`        | The same fade written as metres of ground, worked out per tile. Usually the one to reach for.                |
-| `opacity`              | `0`–`1`, scales the source alpha. RGBA space only.                                                           |
-| `blend`                | `normal`, `multiply`, `screen`, `overlay`, `darken`, `lighten`. RGBA space only.                             |
+| Field                  | Meaning                                                                                                        |
+| ---------------------- | -------------------------------------------------------------------------------------------------------------- |
+| `category`             | Resolve to the newest build in this category.                                                                  |
+| `archive`              | Or pin one infohash.                                                                                           |
+| `stack`                | Or another stack, merged as heights. See "A stack as a source".                                                |
+| `url`                  | Or an archive read straight from an http(s) address, no torrent involved. Exactly one of the four.             |
+| `minzoom` / `maxzoom`  | Skip a `url` source outside this zoom range before opening it. Optional; the header answers anyway without it. |
+| `required`             | A tile fails rather than being served without this source. Defaults true for the bottom-most, false above.     |
+| `encoding`             | `mapbox` or `terrarium`. Elevation space only.                                                                 |
+| `baseVal` / `interval` | Mapbox decode offset and step. Default `-10000` and `0.1`.                                                     |
+| `maskValues`           | Decoded heights meaning "no data here". Elevation space only.                                                  |
+| `maskRange`            | `[low, high]` in metres, or a list of them. Everything inside is nodata. Elevation space only.                 |
+| `heightAdjustment`     | Metres, added **after** masking. Elevation space only.                                                         |
+| `feather`              | Pixels to fade in over wherever the source stops: a `cutline`, `bounds`, or the holes a mask leaves. Max 64.   |
+| `featherMetres`        | The same fade written as metres of ground, worked out per tile. Usually the one to reach for.                  |
+| `opacity`              | `0`–`1`, scales the source alpha. RGBA space only.                                                             |
+| `blend`                | `normal`, `multiply`, `screen`, `overlay`, `darken`, `lighten`. RGBA space only.                               |
 
 `attribution` is not optional in practice. A stack is a derived work of every
 source in it, and the thing that reliably gets lost when tiles are combined is
@@ -1631,6 +1636,200 @@ would wait a day before showing it had ever run.
 That is what a second node serving the same stacks wants: only one of them
 should be the node that bakes.
 
+## A source read straight from a URL
+
+A source may name a URL instead of a category, an archive or a stack:
+
+```json
+{
+  "id": "mapterhorn",
+  "sources": [
+    {
+      "url": "https://download.mapterhorn.com/planet.pmtiles",
+      "encoding": "terrarium",
+      "minzoom": 0,
+      "maxzoom": 12
+    },
+    {
+      "url": "https://download.mapterhorn.com/6-32-31.pmtiles",
+      "encoding": "terrarium",
+      "bounds": [0, 0, 5.625, 5.625],
+      "minzoom": 13,
+      "maxzoom": 13
+    }
+  ]
+}
+```
+
+Mapterhorn publishes terrain this way: a global base to about z12 and several
+hundred regional patches reaching higher, each one a plain HTTPS download and
+none of it in a swarm this node is part of. Downloading 11.8 TiB of it first,
+to get it into a category the ordinary way, is not a step anybody wants — the
+whole point is to read a tile at a time, from wherever it already is.
+
+### Read the same way a swarm archive is
+
+`FetchSource`, from the `pmtiles` package this project already depends on,
+asks for byte ranges the way `TorrentSource` does — a request for a tile costs
+the header once, the directory once, and the tile itself, not the file. A URL
+source keeps its own small cache of open readers, separate from the one
+catalog archives share, so a stack naming hundreds of them is not competing
+with the archives this node actually seeds for the same budget.
+
+Parent climbing works the same way it does for a category source: a shallow
+archive — Mapterhorn's own base stops at z12 — is upscaled for a deeper
+request exactly as GEBCO is, through the same code, because reading one is now
+a question of which store method answers and nothing else.
+
+### Cheap to have hundreds of
+
+The one thing that makes hundreds of these practical rather than merely
+possible: `bounds` and the two new per-source fields, `minzoom` and `maxzoom`,
+are checked **before** anything is opened. A tile request outside a source's
+stated box, or at a zoom no climb from here could reach into its stated range,
+skips that source without a request leaving this node. For a stack built from
+Mapterhorn's own file list — one tile mostly intersects a global base and one
+or two regional patches — that is most of several hundred sources, on every
+tile, settled from the recipe alone.
+
+Without a stated zoom range this still works, just later: `FetchSource`'s own
+header read answers "no tile" cheaply once opened, the same way a catalog
+archive's does. The static check is what avoids opening most of them at all,
+not what makes opening one safe.
+
+### What it does not do
+
+**Never passed through raw.** A URL source has no infohash to answer the
+byte-for-byte question with, so a stack containing one always decodes —
+`needsCodec` says so from the recipe, the same answer a nested stack gets and
+for the same reason.
+
+**Not seeded, not retired, not rebuilt.** This node does not hold a copy, so
+none of the mechanisms that apply to an archive apply here. The URL is not
+content-addressed either — nothing stops whoever publishes it editing the file
+in place — so it is treated as unstable for caching, the same as a category
+and for a related reason: a category is unstable because the build under it
+moves, a URL because the operator does not control what is at the other end
+of it.
+
+**Feathering a mask edge only partly.** A URL source that both masks and fades
+needs its parents read to measure the ramp, exactly as a catalog source does —
+and that path is wired up. What is not: a parent read that fails is caught and
+skipped silently, the same forgiving rule `readMaskEdges` already applies to a
+catalog source whose parent cannot be reached, so a patchy host degrades the
+ramp rather than failing the tile.
+
+## Importing a list of URLs
+
+Naming 458 sources by hand is not work anybody should do once, let alone again
+when the provider adds a file. A provider that publishes terrain as many
+separate archives generally publishes an index of them as well, and that index
+already carries exactly what the merge needs in order to skip a source without
+opening it.
+
+**Stacks → Import URL list…** takes the address of one. Mapterhorn's is
+`https://download.mapterhorn.com/download_urls.json`.
+
+### Two shapes, detected rather than declared
+
+An **index**: an object with an `items` list, or a bare array of the same
+entries. Each entry needs a `url` and the six numbers Mapterhorn's carries —
+`min_lon`, `min_lat`, `max_lon`, `max_lat`, `min_zoom`, `max_zoom` — which
+become the source's `bounds` and its `minzoom`/`maxzoom`.
+
+A **plain list** of addresses, one per line or as a JSON array of strings, for
+a provider that publishes no index. Blank lines and `#` comments are ignored.
+Nothing states a box or a zoom range, so every source is opened for every tile
+it might cover — which is the cost of having no index, and the reason the index
+path is worth preferring.
+
+Which one it is is worked out from the document, because somebody pasting an
+address has no reason to know which their provider answers with, and guessing
+wrong would be a silent import of nothing rather than an error.
+
+### The global file becomes the base
+
+An entry whose box covers the world — Mapterhorn's `planet.pmtiles` says -180
+to 180 and ±85.0511 — is made `sources[0]` and marked `required`. A stack is
+painted bottom-first, so the thing covering everywhere has to sit underneath
+everything patching it, and the index does not list it first.
+
+It is given no `bounds`: a clip excluding nothing is a rasterise on every
+partial tile for an answer that was never in doubt. Where several entries claim
+to be global, the one reaching deepest wins and the rest become ordinary
+layers.
+
+### The encoding is the importer's to state
+
+An index rarely says. Mapterhorn's files are all terrarium and its JSON never
+mentions it, so the dialog asks and sets it on every imported source — a
+terrain source read with the wrong encoding is a cliff face rather than a
+mistake anybody has to guess at.
+
+### Re-importing
+
+Every imported source carries `importedFrom`, naming the list it came from.
+That is what lets the editor draw one row for several hundred sources, and what
+a re-import uses to know which are its to replace. A source typed by hand
+carries none and is never touched.
+
+The batch also goes back **where it already was**, rather than on the end.
+Painting order is the whole meaning of a stack: a batch that moved on every
+re-import would quietly bury a local override somebody had deliberately placed
+above it — a day later, on a schedule, with nothing to say why the map changed.
+
+### Fetched by the node, not the browser
+
+The console is often on a different network from the node, and a list is only
+useful if the machine that will actually read those archives can reach them. A
+458-entry index is also not something a browser should be parsing and posting
+back a megabyte of.
+
+`POST /api/stacks/<id>/import` takes `{ url, encoding }`. With `dryRun` it
+answers what it would write without writing it, which is what **Check** in the
+dialog shows and what the editor's **Re-import** uses to update an unsaved
+draft. A draft can send its own `sources` along, so the batch comes back merged
+into the order the operator is holding rather than into the stored one.
+
+### Following the list instead of importing it once
+
+Mapterhorn's index has grown through several versions, and a node that imported
+it in March is a node serving March. The same address goes in **Settings →
+Feeds → Stack feeds** with **Into stack** naming the stack to keep level, and
+the poll that already checks other nodes for recipes checks this for files.
+
+The row says which of the two it is by whether **Into stack** is filled in. A
+feed of recipes names its own stacks and this one cannot: an index is a list of
+files with no opinion about what they are for, so the row supplies the stack
+and the encoding the index does not state.
+
+Reconciliation is the same one **Re-import** does, for the same reasons and
+with the same guarantees: hand-written sources are left alone, the batch stays
+where it was put, and a file the provider withdrew stops being asked for. A
+poll where nothing changed writes nothing at all — not the same recipe again,
+which would move its revision and with it every tile cached against it.
+
+A stack named here that does not exist yet is created. One that does keeps
+everything it had; following a list never amounts to taking a recipe over.
+
+### What a bulk import cannot say
+
+Every imported source gets a box and a zoom range, because the index states
+them. It gets no **mask** — no per-source clip beyond its bounding box, no
+feathered edge — because there is nowhere in an index for one to come from.
+
+That is fine where the provider has already merged for you. Mapterhorn's files
+are cut so that a patch and the planet agree at the seam, so a rectangular clip
+is the right clip. It is not fine where two sources genuinely overlap at
+different quality and the edge between them matters, and there the answer is to
+import the batch and then write the overlapping source by hand, above it.
+
+Note what happens if you edit an imported source instead: a mask typed onto one
+is wiped by the next re-import, which replaces the batch entire. On a schedule
+that is a map changing at 4am with nothing to say why. A hand-written source
+carries no `importedFrom`, is never replaced, and is the place for anything the
+index could not have told us.
+
 ## Finding a stack
 
 A stack has no infohash and appears in no feed, so nothing about it is

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.89.0",
+  "version": "0.90.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",