pmtiles-swarm 0.4.5 → 0.5.0

package/CHANGELOG.md

@@ -7,6 +7,47 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.5.0
+### ✨ Features and improvements
+- **The head of a newly joined archive is read without waiting to be asked.** A PMTiles archive
+  is useless until its 127-byte header has been read: it names where the root directory and the
+  JSON metadata live, and reading it raises both to a high piece priority — so the head of the
+  file arrives out of order instead of whenever a download happens to reach byte zero. That
+  machinery existed and was spec-correct, but only ran when something read the archive, and the
+  backfill that would have followed up began by requiring a summary to already exist. A freshly
+  joined archive has none, and the one thing that would have created one was the TileJSON route,
+  which is exactly what fails without a header. So an archive being mirrored stayed unservable
+  for hours while the few kilobytes that would have made it servable sat at position zero.
+
+  It now reads one archive's head at a time — several at once turn a queue of archives into a
+  queue of stalled reads competing for the same bandwidth — with the long metadata timeout
+  rather than the interactive one, backing off between attempts, because a young archive having
+  no peer that holds its first piece is ordinary rather than exceptional. It comes back for
+  vector layers separately, since a writer may put the JSON metadata after every tile and one
+  read routinely gets the header and not the metadata.
+
+  New under `tiles`: `prewarm` (default true), `prewarmIntervalSeconds` (30) and
+  `prewarmBackoffSeconds` (120). Turn it off on a node that distributes archives but never
+  serves tiles from them.
+
+## 0.4.6
+### ✨ Features and improvements
+- **A feed's categories are a list, and are called categories.** The console offered a single
+  "Tag as" string while the configuration has always accepted `categories` as a list — one
+  concept under two names across two editors. It is now the same Categories column a watched
+  folder has. An existing `"category": "openmaptiles"` still works and needs no migration.
+  Everything user-facing now says category rather than tag, including the `%G` placeholder's
+  description and the API table.
+
+### 🐞 Bug fixes
+- **Saving in the console no longer deletes settings it does not show.** Each record was rebuilt
+  from the rendered columns alone, so every field without a column was dropped the first time
+  anyone pressed Save — a watched folder's `pieceLength`, `stabilitySeconds`, `trackers` and
+  `sparse`, a subscription's `savePath`. Nothing warned, because from the console's side the
+  save succeeded. Each row now remembers the entry it was rendered from and a save starts from
+  that, overlaying the columns; an emptied box still removes its field, since that is an
+  instruction rather than a gap.
+
 ## 0.4.5
 ### 🐞 Bug fixes
 - **A hook whose command could not be started is tried again.** Completion is recorded before

package/README.md

@@ -587,7 +587,7 @@ matters there is `maxConnections`, since every peer holds a NAT table entry. See
 | `GET` | `/api/torrents/:infoHash/peers`, `/trackers`, `/content` | Per-peer, per-tracker and per-file detail |
 | `GET` | `/api/torrents/:infoHash/pieces` | Which pieces are held, how rare each is, and what peers hold |
 | `PATCH` | `/api/torrents/:infoHash/mode` | Switch between mirror and cache |
-| `PATCH` | `/api/torrents/:infoHash/categories` | Set, add or remove tags |
+| `PATCH` | `/api/torrents/:infoHash/categories` | Set, add or remove categories |
 | `PATCH` | `/api/torrents/:infoHash/seeding` | Per-archive seeding limit, or "use the global one" |
 | `PATCH` `GET` | `/api/torrents/:infoHash/location` | Move the data; poll the move |
 | `POST` | `/api/torrents/:infoHash/pause`, `/resume` | Stop offering it, without forgetting it |
@@ -599,7 +599,7 @@ matters there is `maxConnections`, since every peer holds a NAT table entry. See
 | `POST` | `/api/check-origins` | Check every archive with a watchable source |
 | `GET` `DELETE` | `/api/adds` | Downloads still in flight, and cancelling one by URL |
 | `GET` `POST` | `/api/speed` | Which speed limits are in force, and the manual switch between the two sets |
-| `GET` | `/api/categories` | Every tag, with the endpoints resolving to its newest build |
+| `GET` | `/api/categories` | Every category, with the endpoints resolving to its newest build |
 | `POST` | `/api/adopt`, `/api/adopt/candidates` | Take over what an engine or another node holds |
 | `POST` | `/api/sources/preview` | What a watched web location would take, without taking it |
 | `POST` | `/api/subscriptions/preview` | Whether a peer is reachable and what it offers |

package/docs/subscribing.md

@@ -156,7 +156,7 @@ the peer's to retract:
 Provenance is what makes this possible: an archive records which subscription
 sent it, and only that subscription can ever propose removing it.
 
-## Sharing only what you tag
+## Sharing only what you categorise
 
 Category feeds let a *subscriber* narrow what it takes. They do not narrow what
 you publish: `/feed.xml` carries the whole catalogue, so a peer who could follow

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.4.5",
+  "version": "0.5.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -843,7 +843,7 @@ export function createApp({
   // Tags, after the fact. They could only be set when an archive was added,
   // which is the wrong moment to have to know: a build becomes "weekly" when
   // there is a second one, and an archive is marked for sharing long after it
-  // arrives. Accepts a whole list, or add/remove for one tag at a time.
+  // arrives. Accepts a whole list, or add/remove for one category at a time.
   app.patch(
     '/api/torrents/:infoHash/categories',
     route(async (req, res) => {
@@ -1426,7 +1426,7 @@ export function createApp({
   /**
    * Whether an archive may leave this node, for this caller.
    *
-   * Any tag matching is enough. A tag names one thing an archive is, not the
+   * Any category matching is enough. A category names one thing an archive is, not the
    * whole of what it is, so a planet build tagged both "basemaps" and "weekly"
    * belongs in a basemaps feed whether or not weekly is also published.
    * @param {object} entry - Catalog entry.

package/src/catalog.js

@@ -34,7 +34,7 @@ import path from 'node:path';
  * them. Accepts the older single `category` string so catalogues written before
  * tagging keep working — they are read as a list of one.
  * @param {object} source - Anything with `categories` and/or `category`.
- * @returns {string[]} - Sorted, de-duplicated, non-empty tags.
+ * @returns {string[]} - Sorted, de-duplicated, non-empty categories.
  */
 export function normalizeCategories(source) {
   const raw = [
@@ -117,7 +117,7 @@ export class Catalog {
    * @returns {CatalogEntry[]} - Matching entries.
    */
   byCategory(category) {
-    // Any match, not all: a tag names one thing an archive is, and asking for
+    // Any match, not all: a category names one thing an archive is, and asking for
     // "terrain" should find everything tagged terrain whatever else it is.
     return this.list().filter((entry) =>
       normalizeCategories(entry).includes(category),

package/src/config.js

@@ -444,6 +444,29 @@ const DEFAULTS = {
      * like a hang rather than like "not yet".
      */
     headerTimeoutMs: 12000,
+    /**
+     * Read the head of a newly joined archive without waiting to be asked.
+     *
+     * A PMTiles archive is useless until its header has been read: it names
+     * where the root directory and the JSON metadata are, and reading it also
+     * raises those to a high piece priority, so the head of the file arrives
+     * out of order rather than whenever a download happens to reach it. Left
+     * to the first request, an archive being mirrored is unservable for hours
+     * while the bytes that would make it servable sit at position zero.
+     *
+     * Off makes sense on a node that only distributes and never serves tiles.
+     */
+    prewarm: true,
+    /** How often to look for an archive whose head has not been read. */
+    prewarmIntervalSeconds: 30,
+    /**
+     * How long to leave an archive alone after a failed read.
+     *
+     * Failure here is ordinary rather than exceptional — a brand-new archive
+     * may have no peer holding its first piece yet — so this is a retry
+     * interval, not an error budget.
+     */
+    prewarmBackoffSeconds: 120,
     /**
      * What a missing tile answers with: true for 404, false for 204.
      *
@@ -643,7 +666,7 @@ const DEFAULTS = {
    *   }
    *
    * Placeholders match a torrent client's, so an existing script keeps working:
-   * %N name, %L first category, %G all tags, %F content path, %D save path,
+   * %N name, %L first category, %G all categories, %F content path, %D save path,
    * %Z size, %C file count, %I infohash.
    *
    * Command and arguments are separate rather than one string a shell pulls

package/src/hooks.js

@@ -31,7 +31,7 @@ const TAIL_LINES = 20;
  * The set mirrors a torrent client's, so an existing script keeps working:
  *
  *   %N name          %I infohash      %F content path
- *   %L category      %G tags          %D save path
+ *   %L category      %G categories    %D save path
  *   %Z size          %C file count
  *
  * @param {string} argument - An argument possibly containing placeholders.

package/src/index.js

@@ -19,6 +19,7 @@ import { closeServer, installSignalHandlers, runStoppers } from './shutdown.js';
 import { ScheduledSourceManager } from './sources.js';
 import { SubscriptionManager } from './subscriptions.js';
 import { TileStore } from './tiles.js';
+import { HeadWarmer } from './prewarm.js';
 import { WarmRunner } from './warm.js';
 import { WatchManager } from './watch.js';
 
@@ -216,6 +217,12 @@ PMTILES_SWARM_PUBLIC_URL
   library.attachTiles(tiles);
   const warm = new WarmRunner(tiles);
 
+  // Reads the head of anything joined but not yet understood — the header,
+  // then the root directory and metadata it points at. Without this an archive
+  // being mirrored is unservable until the download happens to reach byte
+  // zero, and the request that would have read it times out long before.
+  const headWarmer = new HeadWarmer(tiles, catalog, config);
+
   // Restarting one subsystem, rather than the process, for the settings that
   // only that subsystem reads. Each stops and starts from the live config, so
   // nothing here has to know what changed — only what to rebuild.
@@ -337,6 +344,11 @@ PMTILES_SWARM_PUBLIC_URL
   }
   hooks.start();
   completion.start();
+  // The header is 127 bytes and names where the root directory and the JSON
+  // metadata live (PMTiles v3 spec, fields at offsets 8 and 24). Reading it
+  // raises both to a high piece priority, so an archive becomes servable in
+  // seconds rather than at whatever hour the download reaches byte zero.
+  headWarmer.start();
 
   // Watch the sources archives were built from. A changed source does not
   // invalidate its torrent, but it does mean any web seed pointing there will
@@ -375,6 +387,7 @@ PMTILES_SWARM_PUBLIC_URL
       completion.stop();
       subscriptions.stop();
       warm.stop();
+      headWarmer.stop();
     }, ms: 1000 },
     { label: 'watchers', stop: () => watch.stop() },
     {

package/src/prewarm.js

@@ -0,0 +1,152 @@
+/**
+ * Reading the head of a newly joined archive, before anybody asks for it.
+ *
+ * A PMTiles archive is useless until its header has been read: the header is
+ * the first 127 bytes and names where the root directory and the JSON metadata
+ * live, and without it there is no TileJSON, no vector layers, and a preview
+ * that renders black. Reading it also *prioritises* what it found — the source
+ * hints the root directory as critical and the metadata as high — so the head
+ * of the file arrives out of order rather than whenever the download reaches
+ * it.
+ *
+ * None of that happened on its own. The read is on the interactive path, so it
+ * ran when somebody opened the archive; and the backfill that follows it up
+ * required a summary to already exist, which is precisely what a freshly joined
+ * archive does not have. So the first request paid for the header, and if it
+ * timed out first — which against a 72 GiB mirror with no web seed it does —
+ * nothing ever tried again.
+ *
+ * A mirror gets there eventually by downloading everything. The point of doing
+ * it deliberately is that "eventually" is hours, and the archive is servable in
+ * the first few seconds if the right few kilobytes are asked for first.
+ */
+
+/** How long to leave an archive alone after a failed attempt. */
+const DEFAULT_BACKOFF_MS = 120000;
+
+export class HeadWarmer {
+  #tiles;
+  #catalog;
+  #config;
+  #now;
+  #timer;
+  #tried = new Map();
+  #running = false;
+
+  /**
+   * @param {object} tiles - The tile store, for `summarize`.
+   * @param {object} catalog - Where the summary is written.
+   * @param {object} config - Resolved configuration.
+   * @param {Function} [now] - Clock, for testing.
+   */
+  constructor(tiles, catalog, config, now = () => Date.now()) {
+    this.#tiles = tiles;
+    this.#catalog = catalog;
+    this.#config = config;
+    this.#now = now;
+  }
+
+  /** @returns {boolean} - Whether this node can and should warm anything. */
+  get enabled() {
+    return (
+      this.#config.tiles?.prewarm !== false &&
+      typeof this.#tiles?.summarize === 'function'
+    );
+  }
+
+  /**
+   * Whether an archive is worth reading the head of right now.
+   *
+   * Deliberately narrow: an archive that has been summarised is done, unless it
+   * is vector and its layers are still missing — that section sits at the far
+   * end of the file and routinely arrives later than the header.
+   * @param {object} entry - A catalog entry.
+   * @returns {boolean} - True to attempt a read.
+   */
+  due(entry) {
+    // Only PMTiles has a head worth reading. A .osm.pbf from a feed is not an
+    // archive this can say anything about.
+    if (entry.kind && entry.kind !== 'pmtiles') return false;
+
+    const summary = entry.pmtiles;
+    if (summary && (summary.format !== 'pbf' || summary.vectorLayers)) {
+      return false;
+    }
+
+    const last = this.#tried.get(entry.infoHash) ?? 0;
+    const backoff = (this.#config.tiles?.prewarmBackoffSeconds ?? 120) * 1000;
+    return this.#now() - last >= (backoff || DEFAULT_BACKOFF_MS);
+  }
+
+  /**
+   * Reads the head of one archive that needs it.
+   *
+   * One per pass on purpose. Each read is a byte range fetched out of a swarm
+   * that may have no peer holding it yet, and starting several at once turns a
+   * queue of archives into a queue of stalled reads competing for the same
+   * bandwidth — the same mistake as prefetching leaf directories eagerly, which
+   * measured four times slower than not bothering.
+   * @returns {Promise<object|null>} - The entry warmed, or null.
+   */
+  async sweep() {
+    if (!this.enabled || this.#running) return null;
+
+    const entry = this.#catalog.list().find((candidate) => this.due(candidate));
+    if (!entry) return null;
+
+    this.#running = true;
+    this.#tried.set(entry.infoHash, this.#now());
+    try {
+      // The long timeout, not the interactive one: this is a byte range from
+      // an archive nobody has asked for a piece of yet.
+      const summary = await this.#tiles.summarize(entry.infoHash, {
+        timeoutMs: this.#config.tiles?.metadataTimeoutMs ?? 120000,
+      });
+
+      const stored = await this.#catalog.put({
+        infoHash: entry.infoHash,
+        pmtiles: { ...entry.pmtiles, ...summary },
+      });
+      console.log(
+        `[warm] read the head of ${entry.name}` +
+          (summary.vectorLayers
+            ? ` (${summary.vectorLayers.length} vector layers)`
+            : ''),
+      );
+      return stored;
+    } catch (error) {
+      // Ordinary while an archive is young — the piece holding the header may
+      // not exist anywhere reachable yet. Worth saying once per attempt, since
+      // it is the answer to "why is my preview blank", and the backoff keeps
+      // it from becoming noise.
+      console.warn(`[warm] ${entry.name}: ${error.message}`);
+      return null;
+    } finally {
+      this.#running = false;
+    }
+  }
+
+  /**
+   * Starts warming, and keeps at it.
+   * @returns {void}
+   */
+  start() {
+    if (!this.enabled) return;
+    const seconds = this.#config.tiles?.prewarmIntervalSeconds ?? 30;
+    if (seconds <= 0) return;
+
+    const run = () =>
+      this.sweep().catch((error) =>
+        console.error(`[warm] sweep failed: ${error.message}`),
+      );
+    run();
+    this.#timer = setInterval(run, seconds * 1000);
+    this.#timer.unref?.();
+  }
+
+  /** Stops warming. @returns {void} */
+  stop() {
+    if (this.#timer) clearInterval(this.#timer);
+    this.#timer = undefined;
+  }
+}

package/src/subscriptions.js

@@ -346,7 +346,7 @@ export class SubscriptionManager {
    */
   async #add(item, subscription) {
     const options = {
-      // A subscriber may file a peer's archives under its own tags; failing
+      // A subscriber may file a peer's archives under its own categories; failing
       // that, whatever the peer tagged them with comes across.
       categories:
         subscription.categories ??

package/src/web/index.html

@@ -357,7 +357,7 @@
           addressed by its infohash — which is what makes a tile immutable and
           cacheable for a year — so a style pointed at one is pinned to that
           build forever. These endpoints always resolve to the newest archive
-          carrying the tag.
+          carrying the category.
         </div>
         <div id="category-list"></div>
         <div class="empty" id="categories-empty" hidden>
@@ -543,8 +543,8 @@
             <button type="button" id="add-category">Add</button>
           </div>
           <div class="sub">
-            An archive can carry several. Feeds include anything tagged with
-            the category asked for.
+            An archive can carry several. Feeds include every archive
+            carrying the category asked for.
           </div>
         </div>
 
@@ -2327,6 +2327,11 @@
         const { into, key, title, blurb, columns, rows, restart, preview, peerPreview, footnote } =
           spec;
         rowEditorColumns[key] = columns;
+        // Kept so a save can put back what this editor never showed. An entry
+        // holds more than there are columns for it — a watch folder's
+        // pieceLength, a subscription's savePath — and a save that rebuilt each
+        // record from the columns alone deleted every one of them.
+        rowEditorRows[key] = rows;
         const panel = document.createElement('div');
         panel.className = 'panel';
         panel.style.marginBottom = '1rem';
@@ -2368,8 +2373,8 @@
             }
           </td>`;
 
-        const rowHtml = (row = {}) => `
-          <tr>
+        const rowHtml = (row = {}, index) => `
+          <tr${index === undefined ? '' : ` data-origin="${index}"`}>
             ${columns.map((column) => cell(column, row[column.field])).join('')}
             <td>
               ${preview ? '<button type="button" data-act="preview">Preview</button>' : ''}
@@ -2392,7 +2397,9 @@
                   <th></th>
                 </tr>
               </thead>
-              <tbody>${[...rows, {}].map((row) => rowHtml(row)).join('')}</tbody>
+              <tbody>${[...rows, {}]
+                .map((row, index) => rowHtml(row, index < rows.length ? index : undefined))
+                .join('')}</tbody>
             </table>
           </div>
           <div style="margin-top:0.5rem">
@@ -2492,11 +2499,19 @@
        * @param {object[]} columns - Column definitions.
        * @returns {object} - The record.
        */
-      function readRow(row, columns) {
-        const record = {};
+      function readRow(row, columns, original = {}) {
+        // Starts from what was there. The editor shows the fields worth
+        // offering, not every field an entry may hold, and rebuilding from the
+        // columns alone silently dropped the rest.
+        const record = { ...original };
         for (const column of columns) {
           const raw = row.querySelector(`[data-field="${column.field}"]`)?.value.trim() ?? '';
-          if (!raw) continue;
+          // Emptied on purpose is still an instruction, so the field goes
+          // rather than the original value surviving underneath it.
+          if (!raw) {
+            delete record[column.field];
+            continue;
+          }
           if (column.list) {
             record[column.field] = raw
               .split(',')
@@ -2527,8 +2542,19 @@
         for (const [key, columns] of Object.entries(specs)) {
           const panel = document.querySelector(`[data-row-editor="${key}"]`);
           if (!panel) continue;
+          const originals = rowEditorRows[key] ?? [];
           const records = [...panel.querySelectorAll('tbody tr')]
-            .map((row) => readRow(row, columns))
+            .map((row) => {
+              // A row added in the console has no original; one rendered from
+              // the configuration is matched back to it, so a removed row takes
+              // its own hidden fields with it and the others keep theirs.
+              const origin = row.dataset.origin;
+              return readRow(
+                row,
+                columns,
+                origin === undefined ? {} : (originals[Number(origin)] ?? {}),
+              );
+            })
             .filter((record) => record[columns[0].field]);
           updates[key] = records;
         }
@@ -3022,6 +3048,9 @@
       // ── Settings ──────────────────────────────────────────────────────────
       let restartKeys = new Set();
       let rowEditorColumns = {};
+      // What each editor was rendered from, so a save can put back the fields
+      // it never showed.
+      let rowEditorRows = {};
 
       async function loadSettings() {
         const { config, restartRequired, configPath } = await api('/api/config');
@@ -3080,6 +3109,7 @@
         // are lists of small records, which is exactly what a torrent client
         // gives a grid for, and what a textarea full of braces is worst at.
         rowEditorColumns = {};
+        rowEditorRows = {};
         await renderTokenEditor(body);
         renderHookEditor(body, config, restartKeys);
 
@@ -3352,7 +3382,12 @@
               placeholder: '1',
               number: true,
             },
-            { field: 'category', label: 'Tag as', placeholder: 'from-peer' },
+            {
+              field: 'categories',
+              label: 'Categories',
+              placeholder: 'from-peer, planet',
+              list: true,
+            },
             { field: 'filter', label: 'Name filter', placeholder: 'terrain' },
             { field: 'token', label: 'Token', placeholder: 'if issued one', secret: true },
             {