pmtiles-swarm 0.50.0 → 0.52.0

package/CHANGELOG.md

@@ -7,6 +7,57 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.52.0
+### ✨ Features and improvements
+- **`publishingUrl`, for the URLs that have to be permanent.** Almost every URL this node emits is
+  worked out per request, deliberately: a node answering on several domains should name itself as
+  whichever one was asked for, and leaving `publicUrl` unset is what allows that. A web seed is not
+  read once, though — it is written into the `.torrent` and the magnet, served byte for byte to
+  everyone who fetches either, and never rewritten — so it has to be one address rather than
+  whichever the last request happened to arrive on.
+
+  `publishingUrl` is that address, and nothing else. TileJSON, tile templates, `.torrent` links,
+  style URLs and the feeds go on naming whichever host the request arrived as, so the multi-domain
+  behaviour is intact. It is taken exactly as written, port and path included: a node listening on
+  8090 behind a load balancer answering on 443 sets `https://swarm.example.org` and gets it, with
+  none of the port substitution that applies to a request reaching the node directly.
+
+- **A "Published as" field beside the web seed switch**, prefilled with whatever the node would use
+  and editable until the moment the switch is turned on. Which is the point: it is the one URL here
+  that cannot be taken back, so it should be readable — and correctable — before it is.
+
+  Resolution order is deliberate: the field, then `publishingUrl`, then `publicUrl`, then the
+  request. With none of them the setting is refused rather than invented.
+
+## 0.51.0
+### ✨ Features and improvements
+- **A web seed URL that no peer could fetch is caught before it is published.** Nothing rewrites a
+  web seed afterwards — a `.torrent` is served byte for byte as it was written, so the URL that goes
+  in is the URL every peer receives for as long as the torrent exists. A loopback address is now
+  refused outright, since `127.0.0.1` names the machine asking and every peer given it would try to
+  fetch the archive from itself and retry for ever. A private address, or a hostname with no domain
+  in it, is published with a warning rather than blocked: a node syncing to its own peers across a
+  LAN is a real arrangement, and the internal address is the right answer there.
+
+### 🐞 Bug fixes
+- **The HTTP sources tab showed the admin port beside the serve switch.** The console built the URL
+  from the address in the browser's bar, and the console is served from the admin listener — so the
+  one port that is not for the public was the one displayed next to a switch that publishes a URL to
+  the whole swarm. It now asks the node, which already answers this correctly everywhere else. What
+  was actually written into the `.torrent` and the magnet was right throughout; only the label was
+  wrong.
+
+## 0.50.1
+### ✨ Features and improvements
+
+### 🐞 Bug fixes
+- **The HTTP sources tab threw instead of opening.** The three switches were built with the base URL
+  that `renderDetail` keeps, and they are rendered by `fillPane`, which is a different function with
+  no sight of it — a `ReferenceError` the moment the tab was clicked. The console is one module
+  inside an HTML file, so there is nothing to import and nothing to lint: `node --check` accepts a
+  binding reached across a function boundary and says nothing. A test now compares what `fillPane`
+  uses against what only `renderDetail` declares.
+
 ## 0.50.0
 ### ✨ Features and improvements
 - **Three switches for what a node offers of an archive's own bytes**, on the node, on a watched

package/docs/configuration.md

@@ -315,11 +315,12 @@ rotation stays out across the restart you were probably about to do.
 
 ## Offering the archive file itself
 
-| setting          | default |                                                       |
-| ---------------- | ------- | ----------------------------------------------------- |
-| `serveArchive`   | `false` | answer `/archives/<infohash>/archive.pmtiles`         |
-| `selfWebSeed`    | `false` | publish this node as a web seed for archives it holds |
-| `publicDownload` | `false` | offer them as downloads on the public catalogue page  |
+| setting          | default |                                                         |
+| ---------------- | ------- | ------------------------------------------------------- |
+| `serveArchive`   | `false` | answer `/archives/<infohash>/archive.pmtiles`           |
+| `publishingUrl`  | unset   | the address to use where a URL must outlive the request |
+| `selfWebSeed`    | `false` | publish this node as a web seed for archives it holds   |
+| `publicDownload` | `false` | offer them as downloads on the public catalogue page    |
 
 Three switches rather than one, because they are three different exposures and a
 node can reasonably want any of them without the others.
@@ -345,6 +346,36 @@ GET /latest/<category>/archive.pmtiles     whichever is current, with an ETag
 
 With it off, both answer `403`.
 
+### `publishingUrl`
+
+The address to use for URLs that outlive the request that made them.
+
+Almost every URL this node emits is worked out per request, deliberately: a node
+answering on several domains should name itself as whichever one was asked for,
+and leaving [`publicUrl`](#publicurl) unset is what allows that. Read once, by
+whoever asked, a per-request answer is the correct answer.
+
+A web seed is not read once. It is written into the `.torrent` and the magnet,
+served byte for byte to everyone who fetches either, and never rewritten — so it
+has to be one address rather than whichever the last request happened to arrive
+on. This is that address.
+
+Narrower than `publicUrl` on purpose: `publicUrl` overrides every URL the node
+emits and so gives up the multi-domain behaviour, while this overrides only the
+ones that have to be permanent. Everything else — TileJSON, tile templates,
+`.torrent` links, style URLs, the feeds — goes on naming whichever host the
+request arrived as.
+
+```json
+{
+  "publishingUrl": "https://swarm.example.org"
+}
+```
+
+Unset falls back to `publicUrl` and then to the request. Whatever it resolves to
+is what the console shows in the **Published as** field, where it can be
+corrected before it becomes permanent.
+
 ### `selfWebSeed`
 
 Writes this node's own `archive.pmtiles` URL into the torrent's `url-list`, so
@@ -358,10 +389,35 @@ node. Turning it off takes the URL back out of the `.torrent` and the magnet —
 but peers already holding either keep trying it until they refresh, so this
 withdraws an advertisement rather than closing a door.
 
-The node has to know what it is called. Set [`publicUrl`](#publicurl), or turn
-the switch on from the console, where the request itself names the node. Without
-either, the setting is refused rather than guessed at: a guessed web seed URL is
-published and then followed.
+The node has to know what it is called, and this is the one URL where "whichever
+name was asked for" is not good enough. In order of how deliberate it is:
+
+1. **The field beside the switch in the console.** Prefilled with whatever the
+   node would use, editable until the moment the switch is turned on, and after
+   that it is what peers hold.
+2. **[`publishingUrl`](#publishingurl)** — the node's answer for exactly this
+   question, and the one to set on a node that answers on several domains.
+3. **[`publicUrl`](#publicurl)**, for a node that has overridden everything
+   anyway.
+4. **The request**, which is a guess, but the same guess every other URL makes.
+
+With none of them the setting is refused rather than invented: a guessed web
+seed URL is published and then followed.
+
+**Nothing rewrites a web seed after it is published.** A `.torrent` is served
+byte for byte as it was written, so the URL that went in is the URL every peer
+receives, for as long as the torrent exists. Two consequences:
+
+- A loopback address is refused. `127.0.0.1` names the machine asking, so every
+  peer given it would try to fetch the archive from itself and retry for ever.
+- A private address, or a hostname with no domain in it, is published with a
+  warning rather than blocked. A node syncing to its own peers across a LAN is a
+  real arrangement and the internal address is the right answer there — but it is
+  worth knowing that peers outside that network cannot use it.
+
+The console asks the node for the URL rather than building one from the address
+in the browser's bar, since the console is served from the admin listener and
+that is the one port that is not for the public.
 
 ### `publicDownload`
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.50.0",
+  "version": "0.52.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -13,7 +13,11 @@ import {
   hashToken,
   isPublicSurface,
 } from './auth.js';
-import { normalizeCategories, publishingFor } from './catalog.js';
+import {
+  normalizeCategories,
+  publishingBase,
+  publishingFor,
+} from './catalog.js';
 import { mutableMagnet, trackersFromMagnet } from './mutable.js';
 import { guessKind } from './library.js';
 import { QBittorrentEngine } from './engines/qbittorrent.js';
@@ -1014,7 +1018,26 @@ export function createApp({
         reading,
         diskBytes,
         served,
-        publishing: publishingFor(entry, config),
+        publishing: {
+          ...publishingFor(entry, config),
+          // What this node would actually publish, worked out here rather than
+          // in the browser. The console is served from the admin listener, so
+          // `location.origin` there names the one port that is not for the
+          // public — and a web seed built from it is handed to every peer in
+          // the swarm.
+          // Whatever was actually published for this archive, where one was:
+          // the record is the truth about what peers hold, and it need not be
+          // what this node would build today.
+          url:
+            entry.selfWebSeedUrl ??
+            `${publishingBase({
+              config,
+              requestBase: baseUrl(req),
+            })}/archives/${entry.infoHash}/archive.pmtiles`,
+          // The base the switch would use, so the console can offer it for
+          // editing before anything permanent is written.
+          base: publishingBase({ config, requestBase: baseUrl(req) }),
+        },
       });
     }),
   );
@@ -1137,16 +1160,17 @@ export function createApp({
     route(async (req, res) => {
       const body = req.body ?? {};
       try {
-        const result = await library.setPublishing(
-          req.params.infoHash,
-          body,
-          // A node with no publicUrl still has to name itself to be a web
-          // seed, and the request it is being asked on is the best evidence
-          // available — the same reasoning the TileJSON already uses. Given
-          // the public port, never the admin one: a seed URL on a listener
-          // bound to localhost is a URL no peer can reach.
-          { baseUrl: baseUrl(req) },
-        );
+        const result = await library.setPublishing(req.params.infoHash, body, {
+          // Typed into the field beside the switch, where it beats
+          // everything else: this is the one URL a person gets to decide
+          // deliberately, because it is the one that cannot be taken back.
+          publishingUrl: body.publishingUrl,
+          // Otherwise the request it is being asked on is the best evidence
+          // available, the same reasoning the TileJSON already uses — and
+          // with the public port rather than the admin one, since a seed URL
+          // on a listener bound to localhost reaches no peer at all.
+          baseUrl: baseUrl(req),
+        });
         res.json(result);
       } catch (error) {
         const status = /unknown archive/.test(error.message) ? 404 : 400;

package/src/catalog.js

@@ -260,3 +260,109 @@ export class Catalog {
     return this.#writing;
   }
 }
+/**
+ * Whether a URL is one other people could plausibly fetch.
+ *
+ * A web seed is not a setting; it is written into the `.torrent` and the
+ * magnet, and served to everyone who asks for either. Nothing rewrites it
+ * afterwards — the file is handed out byte for byte — so a URL that names
+ * this machine's own loopback interface is not a mistake that gets corrected
+ * later. It is distributed, followed, and retried by every peer in the swarm
+ * for as long as the torrent exists.
+ *
+ * Loopback is the only case refused outright, because it cannot be right for
+ * anybody: `127.0.0.1` means the peer's own machine, not this one. A private
+ * address is a different matter — a node syncing to its own peers across a LAN
+ * is a real arrangement, and this is not the place to overrule it — so that is
+ * reported rather than blocked.
+ * @param {string} url - The candidate web seed.
+ * @returns {{ok: boolean, why?: string, warning?: string}} - Whether to
+ *   publish it, and what to say about it either way.
+ */
+export function reachability(url) {
+  let host;
+  try {
+    host = new URL(url).hostname.replace(/^\[|\]$/g, '').toLowerCase();
+  } catch {
+    return { ok: false, why: `not a URL: ${url}` };
+  }
+
+  if (host === 'localhost' || host === '::1' || /^127\./.test(host)) {
+    return {
+      ok: false,
+      why:
+        `${host} names the machine asking, not this one, so a web seed at ` +
+        'this address is unusable by every peer that receives it. Set ' +
+        'publicUrl to the address this node is reachable at.',
+    };
+  }
+
+  const private4 =
+    /^10\./.test(host) ||
+    /^192\.168\./.test(host) ||
+    /^172\.(1[6-9]|2\d|3[01])\./.test(host) ||
+    /^169\.254\./.test(host);
+  // fc00::/7, the IPv6 unique-local range.
+  const private6 = /^f[cd][0-9a-f]{2}:/.test(host);
+  if (private4 || private6 || host.endsWith('.local')) {
+    return {
+      ok: true,
+      warning:
+        `${host} is a private address. It will be published in the .torrent ` +
+        'and the magnet as it stands, and nothing rewrites it later — peers ' +
+        'outside this network cannot use it.',
+    };
+  }
+
+  // A bare name with no dot in it resolves only where the same search domain
+  // does, which is a narrower place than the swarm.
+  if (!host.includes('.')) {
+    return {
+      ok: true,
+      warning:
+        `${host} has no domain, so it resolves only on networks that already ` +
+        'know the name. Peers elsewhere cannot use it.',
+    };
+  }
+
+  return { ok: true };
+}
+
+/**
+ * The base a URL gets when it is going to outlive the request that made it.
+ *
+ * Almost every URL this node emits is worked out per request, on purpose: a
+ * node answering on several domains should name itself as whichever one was
+ * asked, and `publicUrl` is deliberately left unset to allow that. That is the
+ * right answer for a TileJSON, a `.torrent` link or a style URL — read once,
+ * by whoever asked, and correct for them.
+ *
+ * A web seed is not that. It is written into the `.torrent` and the magnet and
+ * served byte for byte to everyone who asks for either, so it has to be one
+ * address rather than whichever the last request happened to arrive on. This
+ * is where that address comes from, in order of how deliberate it is:
+ *
+ * 1. Given outright — the field beside the switch in the console.
+ * 2. `publishingUrl`, the node's answer for exactly this question.
+ * 3. `publicUrl`, if a node has overridden everything anyway.
+ * 4. The request, which is a guess, but the same guess every other URL makes.
+ *
+ * @param {object} [options] - `explicit`, `config` and `requestBase`.
+ * @returns {string} - A base with no trailing slash, or an empty string.
+ */
+export function publishingBase({ explicit, config, requestBase } = {}) {
+  const candidates = [
+    explicit,
+    config?.publishingUrl,
+    config?.publicUrl,
+    requestBase,
+  ];
+  for (const candidate of candidates) {
+    // An empty or whitespace value means "not set", not "use an empty base" —
+    // the same reading `publicUrl` gets, and for the same reason: clearing a
+    // key by emptying it is what an operator naturally does to a JSON file.
+    const value = String(candidate ?? '').trim();
+    if (value) return value.replace(/\/+$/, '');
+  }
+  return '';
+}

package/src/config.js

@@ -234,6 +234,26 @@ const DEFAULTS = {
   allowUnauthenticated: false,
   /** Public base URL, used to build absolute links in the RSS feed and TileJSON. */
   publicUrl: undefined,
+  /**
+   * The one address to use for URLs that outlive the request that made them.
+   *
+   * Almost every URL this node emits is worked out per request, deliberately: a
+   * node answering on several domains should name itself as whichever one was
+   * asked for, and leaving `publicUrl` unset is what allows that. Read once, by
+   * whoever asked, a per-request answer is the correct answer.
+   *
+   * A web seed is not read once. It is written into the `.torrent` and the
+   * magnet, served byte for byte to everyone who fetches either, and never
+   * rewritten — so it has to be one address rather than whichever the last
+   * request happened to arrive on. Set this to that address.
+   *
+   * Narrower than `publicUrl` on purpose. `publicUrl` overrides every URL the
+   * node emits and so gives up the multi-domain behaviour; this overrides only
+   * the ones that have to be permanent. Unset falls back to `publicUrl`, then
+   * to the request — and the console lets the address be typed at the moment
+   * the switch is turned on, which beats both.
+   */
+  publishingUrl: undefined,
   /**
    * Trust X-Forwarded-* headers, for running behind a reverse proxy or CDN.
    * Takes anything Express accepts: `true`, a hop count, or a subnet list.

package/src/library.js

@@ -11,7 +11,7 @@ import {
   promote,
   suffixFor,
 } from './incomplete.js';
-import { publishingFor } from './catalog.js';
+import { publishingBase, publishingFor, reachability } from './catalog.js';
 import { checkOrigin, fingerprintOrigin } from './origin.js';
 import { probePMTiles } from './pmtiles-probe.js';
 import {
@@ -2869,18 +2869,31 @@ export class Library {
     // so nothing would ever have published it.
     const published = entry.selfWebSeedUrl ?? null;
     let webSeed = published;
+    let warning = null;
 
     if (after.selfWebSeed && !published) {
-      const base = String(options.baseUrl ?? this.#config.publicUrl ?? '')
-        .trim()
-        .replace(/\/+$/, '');
+      const base = publishingBase({
+        // Given outright beats the node's own answer, which beats the request
+        // this happens to have arrived on. See publishingBase().
+        explicit: options.publishingUrl,
+        config: this.#config,
+        requestBase: options.baseUrl,
+      });
       if (!base) {
         throw new Error(
           'cannot publish this node as a web seed without knowing its own ' +
-            'URL: set publicUrl, or pass a base URL',
+            'URL: set publishingUrl, or give one with the request',
         );
       }
       webSeed = `${base}/archives/${infoHash}/archive.pmtiles`;
+      // Checked before it goes anywhere. Nothing rewrites a web seed once it
+      // is in a .torrent -- the file is served byte for byte to everyone who
+      // asks -- so a URL that cannot work is not a mistake that gets corrected
+      // on the next request. It is distributed and then retried for ever.
+      const reach = reachability(webSeed);
+      if (!reach.ok) throw new Error(reach.why);
+      if (reach.warning) console.warn(`[web seed] ${reach.warning}`);
+      warning = reach.warning ?? null;
       await this.addWebSeeds(infoHash, [webSeed]);
       // Only the field. put() merges, so spreading the entry captured before
       // addWebSeeds would write the old webSeeds and the old magnet back over
@@ -2900,6 +2913,9 @@ export class Library {
     return {
       ...after,
       webSeed,
+      // Published anyway, and said out loud. A node syncing to its own peers
+      // across a LAN is a real arrangement and not this code's to overrule.
+      warning,
       // Named so a caller can warn about the one change that is not merely a
       // setting moving: this URL is already in the hands of every peer that
       // holds the torrent, and they will go on trying it for a while.

package/src/web/index.html

@@ -392,6 +392,13 @@
          would rearrange the panel under a click, and would leave no sign of
          why the option had gone. */
       .publish input:disabled + span { opacity: 0.55; }
+      /* The address a web seed would be published at. Beside the switch it
+         belongs to rather than in settings, because it is decided once, at the
+         moment it stops being editable. */
+      .pubbase { margin: -0.2rem 0 0 1.6rem; }
+      .pubbase.hidden { display: none; }
+      .pubbase label { display: block; font-size: 0.8rem; color: var(--muted); }
+      .pubbase input { width: 100%; max-width: 32rem; margin: 0.2rem 0; }
     </style>
   </head>
   <body>
@@ -1821,7 +1828,15 @@
             // says nothing defers to the node's own answer, and what belongs
             // in a checkbox is what is actually happening.
             const on = entry.publishing ?? {};
-            const archiveUrl = `${base}/archives/${entry.infoHash}/archive.pmtiles`;
+            // The node's own answer, not this page's. The console is served
+            // from the admin listener, so `location.origin` here is the one
+            // port that is not for the public — showing it beside a switch
+            // that publishes a URL to the whole swarm is how a useless web
+            // seed gets advertised. The fallback is for an older node that
+            // does not send one.
+            const archiveUrl =
+              on.url ??
+              `${location.origin}/archives/${entry.infoHash}/archive.pmtiles`;
 
             // Three switches rather than one, because they are three separate
             // exposures. Serving the file to a reader that was handed the URL,
@@ -1862,6 +1877,25 @@
                     while.`,
                    !on.serveArchive,
                  )}
+                 <div class="pubbase${on.serveArchive ? '' : ' hidden'}">
+                   <label for="publish-base">Published as</label>
+                   <input
+                     id="publish-base"
+                     value="${escapeHtml(on.base ?? '')}"
+                     placeholder="https://swarm.example.org"
+                     spellcheck="false"
+                     ${on.selfWebSeed ? 'disabled' : ''}
+                   />
+                   <div class="sub">
+                     The address peers will be given, and the one thing here
+                     that cannot be taken back: a <code>.torrent</code> is
+                     served exactly as it was written, so this URL reaches
+                     everyone who ever fetches it. Filled in from
+                     <code>publishingUrl</code>, or from this request when
+                     there is none — worth correcting if the node answers on
+                     more than one name.
+                   </div>
+                 </div>
                  ${toggle(
                    'publicDownload',
                    'Offer it as a download on the public page',
@@ -1922,10 +1956,25 @@
                 }
                 box.disabled = true;
                 try {
+                  // Sent only with the switch it belongs to, so editing the
+                  // field and then toggling something else cannot rewrite a
+                  // seed nobody was looking at.
+                  const typed =
+                    key === 'selfWebSeed' && want
+                      ? pane.querySelector('#publish-base')?.value.trim()
+                      : undefined;
                   const result = await api(
                     `/api/torrents/${infoHash}/publish`,
-                    { method: 'POST', body: { [key]: want } },
+                    {
+                      method: 'POST',
+                      body: { [key]: want, publishingUrl: typed || undefined },
+                    },
                   );
+                  // A seed URL nothing will rewrite later, published to a
+                  // network smaller than the swarm. Worth interrupting for:
+                  // by the time a peer fails to fetch it, the URL is in every
+                  // copy of the torrent.
+                  if (result.warning) toast(result.warning);
                   // The withdrawal is worth saying out loud. The others are
                   // visible in the checkbox that was just clicked.
                   if (result.withdrewWebSeed) {