pmtiles-swarm 0.20.0 → 0.22.0

package/CHANGELOG.md

@@ -58,12 +58,33 @@
   `/api/categories`, filtered by the same `feedCategories` rule, so it can show nothing that was
   not already published — and it is not the console, which stays on the admin port.
   `publicIndex: false` turns it off, withdrawing the three paths it needs with it.
+- **`GET /latest/` lists the categories, without a credential.** Everything else under
+  `/latest/` is public — the TileJSON, the torrent, the magnet, the per-category feed — so the
+  index of what it offers belongs beside them rather than behind the console's door. The public
+  front page now leads with categories rather than only listing archives, which is the more
+  useful handle for a visitor: an archive URL names one build and goes stale on the next, while a
+  category names whichever is current. Same builder as `/api/categories`, so the two cannot drift
+  apart. The page's footer and its noscript block no longer link `/api/` paths that answer 401 to
+  exactly the audience they are shown to.
+- **The feed carries the BEP 46 identity.** `<pmtiles:mutable>` holds the magnet naming an
+  archive's publishing key, so a consumer following a category across rebuilds can read the
+  public key straight out of the feed instead of fetching a TileJSON to find it — the key rides
+  inside the string as `xs=urn:btpk:`. Absent for archives that have no identity, since an empty
+  element would be a claim.
 - **Lint and format tooling.** `npm run lint`, `lint:fix`, `format`, `format:check`, `tidy` and
   `check`, wired into CI. There was no linter before, though the source carried
   `eslint-disable` comments for one — so those suppressed nothing and the rules they named were
   never checked.
 
 ### 🐞 Bug fixes
+- **The public front page could not read its own catalogue.** `/api/catalog` is on the list of
+  paths that belong on a public listener, but that is a separate gate from the credential check,
+  which guards everything under `/api/` except login and session — so on any node with
+  authentication configured the page answered "catalog said 401" and listed nothing. It now
+  prefers the catalogue, which carries web seeds and the sparse flag, and falls back to
+  `/feed.xml`, which needs no credential and says the same things: name, infohash, magnet, size,
+  categories, format and zoom range. Deliberately not fixed by making the catalogue
+  world-readable, which would undo a decision the operator made on purpose.
 - **WebTorrent's `pieces()` was defined twice.** The later definition wins, so the first had
   never run — which is why it still called an undefined `countHeld` and no test noticed.
 - **A stuck HTTP connection could keep a shutdown waiting.** `closeServer` armed its force-close

package/docs/serving-tiles.md

@@ -41,6 +41,27 @@ swarm on every attempt, doing work that could never succeed.
 
 See [internals.md](internals.md#serving-an-mbtiles-archive).
 
+## The category index
+
+`GET /latest/` lists every category this node publishes, with the endpoints
+that resolve to each one's newest build — the TileJSON, the `.torrent`, the
+magnet, the per-category feed, and the style URL with the magnet in its
+fragment.
+
+Public, and deliberately so. Everything else under `/latest/` is — the
+TileJSON, the torrent, the magnet, the feed — so the index of what `/latest/`
+offers belongs beside them rather than behind the console's door.
+`/api/categories` returns the identical list from the identical builder, and is
+guarded only because everything under `/api/` is.
+
+A category is also the handle worth publishing. An archive URL names one build
+and goes stale on the next; a category names whichever is current, so a style
+pointing at one keeps working across rebuilds.
+
+Filtered by `feedCategories` and by whatever token was presented, the same as
+the catalogue and the feeds — so it can show nothing this node was not already
+publishing.
+
 ## Missing tiles
 
 A tile the archive does not hold answers **404 for raster and 204 for vector**,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.20.0",
+  "version": "0.22.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -1744,9 +1744,19 @@ export function createApp({
   // what makes a tile immutable, but leaves nothing for a style to point at
   // that survives a rebuild — and until now the endpoints that give you one
   // were documented rather than discoverable.
-  app.get(
-    '/api/categories',
-    route(async (req, res) => {
+  /**
+   * Every category, with the endpoints that resolve to its newest build.
+   *
+   * Shared by the guarded `/api/categories` and the public `/latest/`, which
+   * want exactly the same answer — one is the console asking and the other is
+   * a stranger, and neither is told anything the other is not. Filtering runs
+   * through `publishesEntry`, so `feedCategories` and whatever token was
+   * presented decide what appears, the same as everywhere else.
+   * @param {import('express').Request} req - The request, for its credential and base URL.
+   * @returns {object[]} - One row per category, sorted by name.
+   */
+  const describeCategories = (req) => {
+    {
       const base = baseUrl(req);
       const visible = catalog
         .list()
@@ -1759,51 +1769,82 @@ export function createApp({
         }
       }
 
-      res.json(
-        [...counts.keys()].sort().map((category) => {
-          const newest = newestIn(category, req);
-          // Only PMTiles has tiles to serve, so a category whose newest build
-          // is an MBTiles archive gets a feed and a torrent but no tile
-          // endpoint — the same rule as an individual archive.
-          const servable =
-            Boolean(newest?.pmtiles) &&
-            (newest.kind ?? 'pmtiles') === 'pmtiles';
-
-          return {
-            category,
-            archives: counts.get(category),
-            newest: newest && {
-              infoHash: newest.infoHash,
-              name: newest.name,
-              size: newest.size,
-              createdAt: newest.createdAt,
-              kind: newest.kind ?? 'pmtiles',
-            },
-            servable,
-            endpoints: {
-              tileJson: servable
-                ? `${base}/latest/${category}/tiles.json`
-                : null,
-              // The same URL with a magnet in the fragment, which is what a
-              // style should carry. A fragment is never sent in a request, so
-              // an ordinary client fetches the TileJSON and ignores it, while
-              // a swarm-aware one has the magnet before it makes any call --
-              // and can therefore still start when this server cannot answer.
-              //
-              // The mutable magnet where there is one, because a category is
-              // precisely where an infohash goes stale: it names the category
-              // and resolves the current build over the DHT. Otherwise the
-              // newest build's own magnet, which pins that build but still
-              // beats a blank map when the fallback is needed at all.
-              styleUrl: servable ? styleUrlFor(category, newest, base) : null,
-              torrent: `${base}/latest/${category}/archive.torrent`,
-              magnet: `${base}/latest/${category}/magnet`,
-              feed: `${base}/feed/${category}.xml`,
-              latestFeed: `${base}/latest/${category}.xml`,
-            },
-          };
-        }),
-      );
+      return [...counts.keys()].sort().map((category) => {
+        const newest = newestIn(category, req);
+        // Only PMTiles has tiles to serve, so a category whose newest build
+        // is an MBTiles archive gets a feed and a torrent but no tile
+        // endpoint — the same rule as an individual archive.
+        const servable =
+          Boolean(newest?.pmtiles) && (newest.kind ?? 'pmtiles') === 'pmtiles';
+
+        return {
+          category,
+          archives: counts.get(category),
+          newest: newest && {
+            infoHash: newest.infoHash,
+            name: newest.name,
+            size: newest.size,
+            createdAt: newest.createdAt,
+            kind: newest.kind ?? 'pmtiles',
+          },
+          servable,
+          endpoints: {
+            tileJson: servable ? `${base}/latest/${category}/tiles.json` : null,
+            // The same URL with a magnet in the fragment, which is what a
+            // style should carry. A fragment is never sent in a request, so
+            // an ordinary client fetches the TileJSON and ignores it, while
+            // a swarm-aware one has the magnet before it makes any call --
+            // and can therefore still start when this server cannot answer.
+            //
+            // The mutable magnet where there is one, because a category is
+            // precisely where an infohash goes stale: it names the category
+            // and resolves the current build over the DHT. Otherwise the
+            // newest build's own magnet, which pins that build but still
+            // beats a blank map when the fallback is needed at all.
+            styleUrl: servable ? styleUrlFor(category, newest, base) : null,
+            torrent: `${base}/latest/${category}/archive.torrent`,
+            magnet: `${base}/latest/${category}/magnet`,
+            feed: `${base}/feed/${category}.xml`,
+            latestFeed: `${base}/latest/${category}.xml`,
+          },
+        };
+      });
+    }
+  };
+
+  // The console's view. Identical to the public one below, and guarded only
+  // because everything under /api/ is.
+  app.get(
+    '/api/categories',
+    route(async (req, res) => {
+      res.json(describeCategories(req));
+    }),
+  );
+
+  // The same list, without a credential.
+  //
+  // Everything under /latest/ is public already — the TileJSON, the torrent,
+  // the magnet, the per-category feed — so the index of what /latest/ offers
+  // belongs here rather than behind the console's door. A category is also the
+  // handle a person actually wants: an infohash names one build, and this
+  // names whichever is current, so a style pointing at it keeps working across
+  // rebuilds.
+  //
+  // Not a second implementation. It is the same builder, filtered by the same
+  // rule, so the two cannot drift into disagreeing about what is published.
+  app.get(
+    '/latest/',
+    route(async (req, res) => {
+      res.setHeader('access-control-allow-origin', '*');
+      // Short, because the answer changes when a build lands.
+      res.setHeader('cache-control', 'public, max-age=60');
+      res.json({
+        // Named so a consumer can refuse a document it does not understand,
+        // the same as the catalogue does.
+        format: 'pmtiles-swarm-categories/1',
+        generatedAt: new Date().toISOString(),
+        categories: describeCategories(req),
+      });
     }),
   );
 

package/src/feed.js

@@ -17,6 +17,8 @@
  * subscriber restores it on completion.
  */
 
+import { mutableMagnet } from './mutable.js';
+
 const PMTILES_NS = 'https://github.com/TechIdiots-LLC/pmtiles-swarm/ns/1.0';
 
 /**
@@ -118,6 +120,18 @@ ${(entry.categories ?? (entry.category ? [entry.category] : []))
       <enclosure url="${xml(torrentUrl)}" length="${xml(entry.size)}" type="application/x-bittorrent"/>
       <pmtiles:infohash>${xml(entry.infoHash)}</pmtiles:infohash>
       <pmtiles:magnet>${xml(entry.magnet)}</pmtiles:magnet>
+${
+  entry.mutable?.publicKey
+    ? `      <pmtiles:mutable>${xml(
+        mutableMagnet(entry.mutable.publicKey, {
+          infoHash: entry.infoHash,
+          salt: entry.mutable.salt,
+          name: entry.mutable.salt ?? entry.name,
+          webSeeds: entry.webSeeds,
+        }),
+      )}</pmtiles:mutable>`
+    : ''
+}
 ${entry.md5 ? `      <pmtiles:md5>${xml(entry.md5)}</pmtiles:md5>` : ''}
 ${entry.originMtime ? `      <pmtiles:mtime>${xml(entry.originMtime)}</pmtiles:mtime>` : ''}
 ${mapFields}
@@ -150,6 +164,7 @@ export function formatBytes(bytes) {
  * @property {string} [torrentUrl] - URL of a .torrent enclosure.
  * @property {string} [category] - Item category.
  * @property {string} [mtime] - The archive's mtime on the node that built it.
+ * @property {string} [mutableMagnet] - BEP 46 magnet, when the publisher has an identity for it.
  */
 
 /**
@@ -196,6 +211,11 @@ export function parseFeed(body) {
     items.push({
       title: title ?? 'untitled',
       infoHash: tag(block, 'pmtiles:infohash')?.toLowerCase(),
+      // The BEP 46 magnet, when the publisher has an identity for this
+      // archive. Carried so a consumer can follow the series across rebuilds,
+      // and so it can read the public key out of the feed rather than being
+      // told it separately -- the key is inside the string as xs=urn:btpk:.
+      mutableMagnet: tag(block, 'pmtiles:mutable'),
       magnet,
       torrentUrl,
       // RSS allows several <category> elements, and an archive may be tagged

package/src/watch.js

@@ -42,6 +42,7 @@ export class WatchManager {
   #library;
   #watchers = [];
   #importing = new Set();
+  #ready = [];
 
   /**
    * Creates the manager.
@@ -142,6 +143,19 @@ export class WatchManager {
         console.error(`[watch] ${directory}: ${error.message}`);
       });
 
+      // chokidar scans the directory before it starts reporting changes, and
+      // a file that lands during that scan is in neither the listing nor the
+      // event stream. Nothing in the daemon depends on knowing when the scan
+      // ends — an archive dropped that early is picked up by the next one —
+      // but anything driving this deliberately does, and a caller with no way
+      // to know is a caller that has to guess with a sleep.
+      this.#ready.push(
+        new Promise((resolve) => {
+          watcher.once('ready', resolve);
+          watcher.once('error', resolve);
+        }),
+      );
+
       this.#watchers.push(watcher);
       for (const entry of entries) {
         const tags = normalizeCategories(entry);
@@ -266,5 +280,21 @@ export class WatchManager {
   async stop() {
     await Promise.all(this.#watchers.map((watcher) => watcher.close()));
     this.#watchers = [];
+    this.#ready = [];
+  }
+
+  /**
+   * Resolves once every watcher has finished its first scan.
+   *
+   * Until then a file dropped into a watched folder may be reported as an
+   * event, may be found by the scan, or may be missed by both — chokidar makes
+   * no promise about the window between opening a watch and finishing the
+   * listing. Nothing in the daemon needs this, since an archive that early is
+   * picked up on the next pass, but anything that creates a file and then
+   * expects to see it imported is racing without it.
+   * @returns {Promise<void>} - Resolves when watching has actually begun.
+   */
+  async ready() {
+    await Promise.all(this.#ready);
   }
 }

package/src/web/public.html

@@ -3,10 +3,14 @@
   The public front door, served at / on the public listener when the console
   has an admin port of its own.
 
-  Reads /api/catalog and /api/categories, both of which are already public and
-  already filtered by feedCategories — so this page shows exactly what this node
-  publishes and cannot show anything else. It is deliberately plain: no build
-  step, no dependencies, and it degrades to a readable list of links if the
+  Reads /latest/ for the categories and, for the archives, /api/catalog where a
+  credential allows it and /feed.xml where one does not — everything under /api/
+  is guarded on a node with any authentication configured, which is most of them,
+  and a front page that answers 401 to its own audience is worse than none.
+
+  All three are filtered by feedCategories, so this page shows exactly what the
+  node publishes and cannot show anything else. It is deliberately plain: no
+  build step, no dependencies, and it degrades to a readable list of links if the
   script does not run.
 -->
 <html lang="en">
@@ -167,7 +171,7 @@
       <noscript>
         <p class="failed">
           This page lists the archives using JavaScript. Without it, the same
-          information is at <a href="/api/catalog">/api/catalog</a> as JSON and
+          information is at <a href="/latest/">/latest/</a> as JSON and
           <a href="/feed.xml">/feed.xml</a> as RSS.
         </p>
       </noscript>
@@ -176,8 +180,7 @@
       <div id="archives"><p class="empty">Loading…</p></div>
 
       <footer>
-        <a href="/api/catalog">catalog JSON</a> ·
-        <a href="/api/categories">categories JSON</a> ·
+        <a href="/latest/">categories JSON</a> ·
         <a href="/feed.xml">RSS feed</a> ·
         <a href="/health">health</a>
       </footer>
@@ -280,47 +283,176 @@
 
       // Categories are the handle that survives a rebuild, so they go first —
       // a style should point at one of these rather than at an infohash.
+      // Categories first, and given the same weight as an archive, because
+      // for most visitors they are the more useful thing: an archive URL names
+      // one build and goes stale on the next, while a category names whichever
+      // is current and keeps working across rebuilds. A style should point at
+      // one of these.
       const renderCategories = (categories) => {
         const host = document.getElementById('categories');
-        const usable = categories.filter((c) => c.endpoints?.styleUrl);
+        host.textContent = '';
+        const usable = categories.filter((c) => c.endpoints);
         if (usable.length === 0) return;
 
         host.append(el('h2', null, 'Categories'));
-        const urls = el('div', 'urls');
+        host.append(
+          el(
+            'p',
+            'facts',
+            'These follow the newest build. Point a style at one of these ' +
+              'rather than at an archive below, and it keeps working when the ' +
+              'archive is rebuilt.',
+          ),
+        );
+
         for (const entry of usable) {
-          // `category` and `archives` are what /api/categories actually calls
+          // `category` and `archives` are what the endpoint actually calls
           // these; `name` and `count` would read better and are not there.
-          const label = Number.isFinite(entry.archives)
-            ? `${entry.category} (${entry.archives})`
-            : entry.category;
-          urls.append(
-            urlRow(label, new URL(entry.endpoints.styleUrl, location.href).href),
-          );
+          const card = el('div', 'archive');
+          const head = el('header');
+          head.append(el('span', 'name', entry.category));
+          if (Number.isFinite(entry.archives)) {
+            head.append(
+              el(
+                'span',
+                'tag',
+                `${entry.archives} build${entry.archives === 1 ? '' : 's'}`,
+              ),
+            );
+          }
+          card.append(head);
+
+          if (entry.newest?.name) {
+            const facts = [`newest: ${entry.newest.name}`];
+            const size = bytes(entry.newest.size);
+            if (size) facts.push(size);
+            card.append(el('p', 'facts', facts.join(' · ')));
+          }
+
+          const links = el('div', 'links');
+          const add = (href, text) => {
+            const a = el('a', null, text);
+            a.href = href;
+            links.append(a);
+          };
+          const ends = entry.endpoints;
+          if (ends.tileJson) add(ends.tileJson, 'TileJSON');
+          if (ends.torrent) add(ends.torrent, '.torrent');
+          if (ends.magnet) add(ends.magnet, 'magnet');
+          if (ends.feed) add(ends.feed, 'RSS');
+          card.append(links);
+
+          const urls = el('div', 'urls');
+          // The one worth copying: the TileJSON URL with the magnet in its
+          // fragment, which a plain client fetches over HTTP and a swarm-aware
+          // one joins directly.
+          if (ends.styleUrl) {
+            urls.append(
+              urlRow('style URL', new URL(ends.styleUrl, location.href).href),
+            );
+          } else if (ends.tileJson) {
+            urls.append(
+              urlRow('TileJSON', new URL(ends.tileJson, location.href).href),
+            );
+          }
+          if (urls.childElementCount > 0) card.append(urls);
+
+          host.append(card);
         }
-        host.append(urls);
+
         host.append(el('h2', null, 'Archives'));
       };
+      const PMTILES_NS =
+        'https://github.com/TechIdiots-LLC/pmtiles-swarm/ns/1.0';
+
+      // The feed says the same things the catalog does, in RSS. It is the
+      // fallback because it is the one that needs no credential: /api/catalog
+      // is guarded whenever this node has any authentication configured, which
+      // is most of them, and a front page that 401s is worse than none.
+      const fromFeed = (xmlText) => {
+        const doc = new DOMParser().parseFromString(xmlText, 'application/xml');
+        if (doc.querySelector('parsererror')) {
+          throw new Error('the feed is not valid XML');
+        }
+
+        const text = (item, name) =>
+          item.getElementsByTagNameNS(PMTILES_NS, name)[0]?.textContent ??
+          undefined;
+
+        return [...doc.getElementsByTagName('item')].map((item) => {
+          const zoom = (name) => {
+            const value = Number(text(item, name));
+            return Number.isFinite(value) ? value : undefined;
+          };
+          const enclosure = item.getElementsByTagName('enclosure')[0];
+          const format = text(item, 'format');
+
+          return {
+            infoHash: text(item, 'infohash'),
+            name: item.getElementsByTagName('title')[0]?.textContent,
+            size: Number(enclosure?.getAttribute('length')) || undefined,
+            categories: [...item.getElementsByTagName('category')].map(
+              (node) => node.textContent,
+            ),
+            magnet: text(item, 'magnet'),
+            createdAt: item.getElementsByTagName('pubDate')[0]?.textContent,
+            pmtiles: format
+              ? {
+                  format,
+                  minZoom: zoom('minzoom'),
+                  maxZoom: zoom('maxzoom'),
+                }
+              : undefined,
+          };
+        });
+      };
 
       const load = async () => {
+        let archives = null;
+
+        // Richer where it is reachable: the catalog carries web seeds and the
+        // sparse flag, which the feed does not.
         try {
           const response = await fetch('/api/catalog');
-          if (!response.ok) throw new Error(`catalog said ${response.status}`);
-          const body = await response.json();
-          render(body.archives ?? []);
-        } catch (error) {
-          const host = document.getElementById('archives');
-          host.textContent = '';
-          host.append(
-            el('p', 'failed', `Could not load the catalog: ${error.message}`),
-          );
-          return;
+          if (response.ok) {
+            archives = (await response.json()).archives ?? [];
+          }
+        } catch {
+          // Fall through to the feed.
         }
 
-        // Best effort, and after the archives: the page is worth showing
-        // without it.
+        if (archives === null) {
+          try {
+            const response = await fetch('/feed.xml');
+            if (!response.ok) throw new Error(`feed said ${response.status}`);
+            archives = fromFeed(await response.text());
+          } catch (error) {
+            const host = document.getElementById('archives');
+            host.textContent = '';
+            host.append(
+              el(
+                'p',
+                'failed',
+                `Could not load the archives: ${error.message}`,
+              ),
+            );
+            return;
+          }
+        }
+
+        render(archives.filter((archive) => archive.infoHash));
+
+        // /latest/ rather than /api/categories: the same list, without a
+        // credential. This is the part most visitors actually want -- an
+        // archive URL names one build, a category names whichever is current
+        // -- so it is worth more than a best-effort attempt at a guarded
+        // endpoint that answers 401 to exactly the audience this page is for.
         try {
-          const response = await fetch('/api/categories');
-          if (response.ok) renderCategories(await response.json());
+          const response = await fetch('/latest/');
+          if (response.ok) {
+            const body = await response.json();
+            renderCategories(body.categories ?? []);
+          }
         } catch {
           // Nothing to say. The archives above are the substance.
         }