pmtiles-swarm 0.92.0 → 0.94.0

package/CHANGELOG.md

@@ -7,6 +7,45 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.94.0
+### 🐞 Bug fixes
+- **Saving settings rewrote a proxy list nobody had touched.** A trusted-proxy list may be stored as
+  a string — `"loopback, 10.0.0.0/8"` is what the documentation shows — and the box rendered that as
+  one line while reading it back as an array. The two never compared equal, so the field was sent on
+  every Save whether or not anybody had looked at it, and before the previous release that rewrite
+  was the thing that stopped the node from starting. Opening the settings page and pressing Save was
+  enough to do it.
+
+  The box now shows one entry per line whichever way the config wrote them, and records what a save
+  will read back rather than what the config holds — so an untouched field is untouched.
+
+## 0.93.0
+### 🐞 Bug fixes
+- **A trusted-proxy list typed with commas stopped the node from starting.** The settings field
+  split what was typed on newlines only, so one line reading `172.16.1.2, 172.16.1.3` was saved as
+  an array holding both addresses in one string. Express splits a comma list when it is handed a
+  bare string and never inside an array, so proxy-addr was given `172.16.1.2, 172.16.1.3` as a
+  single address and threw -- while the app was being built, before the listener binds. The node
+  would not start, could not be reached, and could not be corrected from the console that had
+  written the value; on the node that found this it was 155 restarts.
+
+  Three things were wrong and all three are fixed. The field now splits on commas as well as
+  newlines. Every shape the setting can be written in -- a string, an array, commas, spaces,
+  newlines -- is flattened to what Express wants. And an entry that is not an address is ignored
+  and logged rather than thrown: this setting is not worth a node that will not boot, and trusting
+  nobody is the safe end of being wrong about it.
+
+- **A failed restore took the whole node down, console included.** Handing the library back to the
+  engine at startup already tolerates a failure per archive; the call coming apart as a whole was
+  unguarded, and it happens before the listener binds — so under `Restart=always` the result is a
+  crash loop with no console to look at and no way to see why. It is now reported and the node
+  starts anyway, where every archive shows as **not loaded** until it is fixed.
+
+- **Nothing tested that the node starts at all.** Every other test builds the pieces `src/index.js`
+  wires together and never runs the wiring, so an import cycle or a step that throws before the
+  listener binds was a failure only a real start could find. There is now a boot test that runs the
+  entry point the way the service does and asks it for a page.
+
 ## 0.92.0
 ### 🐞 Bug fixes
 - **A stack's TileJSON published the addresses its sources are read from.** A URL source is named by

package/docs/configuration.md

@@ -82,8 +82,17 @@ Takes effect on the next request; no restart. See
 ### `trustProxy`
 
 Takes anything Express accepts: `true`, a hop count, or a subnet list such as
-`"loopback, 10.0.0.0/8"`. Off by default, because trusting these headers from an
-untrusted client lets it claim any protocol or address it likes.
+`"loopback, 10.0.0.0/8"`. A list may be written as one string or as an array,
+and either may separate its entries with commas, spaces or newlines — all four
+shapes mean the same thing here. Off by default, because trusting these headers
+from an untrusted client lets it claim any protocol or address it likes.
+
+An entry that is not an address, a subnet, or one of `loopback`, `linklocal`
+and `uniquelocal` is **ignored and logged**, and a value that cannot be read at
+all leaves the node trusting nobody. Deliberately not fatal: Express compiles
+this the moment it is set, which is before the listener binds, so a value it
+refuses used to be a node that would not start, could not be reached, and could
+not be corrected from the console that wrote it.
 
 Set it when a proxy terminates TLS, or the TileJSON will advertise `http://` tile
 URLs that browsers block as mixed content. Setting `publicUrl` instead sidesteps

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.92.0",
+  "version": "0.94.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -22,7 +22,12 @@ import {
 import { mutableMagnet, trackersFromMagnet } from './mutable.js';
 import { guessKind } from './library.js';
 import { QBittorrentEngine } from './engines/qbittorrent.js';
-import { RESTART_REQUIRED, redactConfig, saveConfig } from './config.js';
+import {
+  RESTART_REQUIRED,
+  redactConfig,
+  saveConfig,
+  trustProxyFor,
+} from './config.js';
 import { freeSpace, listLocations } from './locations.js';
 import { restart, restartMode } from './restart.js';
 import { parseFeed, renderFeed } from './feed.js';
@@ -260,7 +265,26 @@ export function createApp({
   // the TileJSON advertises http:// tile URLs. A browser that loaded the map
   // over https then blocks every one of them as mixed content, which looks
   // like an empty map rather than like a configuration mistake.
-  if (config.trustProxy) app.set('trust proxy', config.trustProxy);
+  //
+  // Normalised first, and then guarded anyway. Express compiles this value
+  // the moment it is set, which is before the listener binds -- so a value it
+  // cannot compile is not a setting that fails to apply, it is a node that
+  // will not start, cannot be reached, and cannot be corrected from the
+  // console that wrote it. Nothing about which addresses to trust is worth
+  // that: an unparseable one is reported and the node comes up trusting
+  // nobody, which is the safe end of being wrong.
+  const trustProxy = trustProxyFor(config.trustProxy);
+  if (trustProxy !== false) {
+    try {
+      app.set('trust proxy', trustProxy);
+    } catch (error) {
+      console.error(
+        `[config] trustProxy could not be applied (${error.message}). ` +
+          'X-Forwarded-* headers are being ignored; the node is starting ' +
+          'anyway so this can be corrected from Settings.',
+      );
+    }
+  }
   app.use(express.json({ limit: '1mb' }));
 
   // Tiles, TileJSON and the feed stay public — serving them is the point.

package/src/config.js

@@ -1,4 +1,5 @@
 import fs from 'node:fs/promises';
+import net from 'node:net';
 import { hashPassword } from './auth.js';
 import path from 'node:path';
 
@@ -10,6 +11,79 @@ import path from 'node:path';
  * Every setting is documented in docs/configuration.md. Comments here say only
  * what a value is; why it is what it is belongs in the document.
  */
+/** What proxy-addr accepts as a name for a group of addresses. */
+const TRUST_KEYWORDS = new Set(['loopback', 'linklocal', 'uniquelocal']);
+
+/**
+ * Whether one entry of a trust list is something proxy-addr can compile.
+ * @param {string} entry - One address, subnet or keyword.
+ * @returns {boolean} - True if it is usable.
+ */
+function isTrustEntry(entry) {
+  if (TRUST_KEYWORDS.has(entry)) return true;
+  const [address, mask, ...rest] = entry.split('/');
+  if (rest.length > 0) return false;
+  if (!net.isIP(address)) return false;
+  if (mask === undefined) return true;
+  if (net.isIP(mask)) return true;
+  const bits = Number(mask);
+  if (!Number.isInteger(bits) || bits < 0) return false;
+  return bits <= (net.isIP(address) === 6 ? 128 : 32);
+}
+
+/**
+ * What to hand Express as `trust proxy`, from what the config says.
+ *
+ * Express takes four different things here and means something different by
+ * each, and it splits a comma-separated list only when it is handed a bare
+ * string -- never inside an array. So `["10.0.0.1, 10.0.0.2"]`, which is what
+ * a settings field split on newlines alone produces from one line with a
+ * comma in it, reaches proxy-addr as a single address, and proxy-addr throws.
+ *
+ * That throw happened while the app was being built, before the listener
+ * bound: a node that could not start, could not be reached, and could not be
+ * corrected from the console that wrote the value. So every shape is
+ * flattened here, and anything left that is not an address is dropped with a
+ * warning rather than carried into Express.
+ * @param {boolean|number|string|string[]} value - `config.trustProxy`.
+ * @returns {boolean|number|string[]} - Something Express can compile.
+ */
+export function trustProxyFor(value) {
+  if (
+    value === true ||
+    value === false ||
+    value === undefined ||
+    value === null
+  ) {
+    return value === true;
+  }
+  if (typeof value === 'number') return Number.isFinite(value) ? value : false;
+
+  const entries = (Array.isArray(value) ? value : [value])
+    .flatMap((one) => String(one).split(/[\s,]+/))
+    .map((one) => one.trim())
+    .filter(Boolean);
+  if (!entries.length) return false;
+
+  // A lone number is a hop count, and a lone `true` trusts everybody. Both
+  // are things somebody may type into a field that mostly takes addresses.
+  if (entries.length === 1) {
+    if (entries[0] === 'true') return true;
+    if (entries[0] === 'false') return false;
+    if (!Number.isNaN(Number(entries[0]))) return Number(entries[0]);
+  }
+
+  const usable = entries.filter((entry) => isTrustEntry(entry));
+  for (const entry of entries) {
+    if (usable.includes(entry)) continue;
+    console.warn(
+      `[config] trustProxy: ignoring "${entry}", which is not an address, ` +
+        'a subnet, or one of loopback, linklocal, uniquelocal.',
+    );
+  }
+  return usable.length ? usable : false;
+}
+
 const DEFAULTS = {
   port: 8090,
   host: '0.0.0.0',

package/src/index.js

@@ -339,11 +339,28 @@ PMTILES_SWARM_PUBLIC_URL
 
   const catalogued = catalog.list().length;
   if (catalogued > 0) {
-    const { restored, failed } = await library.restore();
-    console.log(
-      `[restore] ${restored} of ${catalogued} archives handed back to the engine` +
-        (failed > 0 ? ` (${failed} could not be)` : ''),
-    );
+    // Reported, never fatal. Restore already tolerates a failure per archive;
+    // what this catches is the whole call coming apart -- and the node it
+    // takes down with it is the one that could have said so. Under
+    // `Restart=always` that is a crash loop with no console to look at, which
+    // is a worse failure than a library that is not being seeded: the console
+    // marks an archive the engine has no record of as `not loaded`, so a node
+    // that comes up says exactly what went wrong here.
+    try {
+      const { restored, failed } = await library.restore();
+      console.log(
+        `[restore] ${restored} of ${catalogued} archives handed back to the engine` +
+          (failed > 0 ? ` (${failed} could not be)` : ''),
+      );
+    } catch (error) {
+      console.error(
+        `[restore] could not hand the library back to the engine: ` +
+          `${error.stack ?? error.message}
+` +
+          '[restore] the node is starting anyway; every archive will show as ' +
+          'not loaded until this is fixed and it is restarted.',
+      );
+    }
   }
 
   // And again if the engine loses its backing process and starts another. A

package/src/web/index.html

@@ -5579,11 +5579,15 @@ Every piece is hashed against the ` +
               type: 'addresses',
               placeholder: 'off — or one address or subnet per line',
               help:
-                'One per line: 172.16.1.49, or 172.16.1.0/24, or several. A ' +
-                'lone number is read as a hop count instead, and a lone `true` ' +
-                'trusts any caller at all — worth avoiding, since this header ' +
-                'is what decides which address the node believes a request ' +
-                'came from.',
+                'An address or a subnet — <code>172.16.1.49</code>, ' +
+                '<code>172.16.1.0/24</code>, <code>loopback</code> — one ' +
+                'per line, or separated by commas. A lone number is read ' +
+                'as a hop count instead, and a lone <code>true</code> ' +
+                'trusts any caller at all, which is worth avoiding: this ' +
+                'header is what decides which address the node believes a ' +
+                'request came from. Anything that is not an address is ' +
+                'ignored and said so in the log, rather than stopping the ' +
+                'node from starting.',
             },
             {
               key: 'publicIndex',
@@ -6307,7 +6311,14 @@ Every piece is hashed against the ` +
               field.type === 'secret'
                 ? `<input type="password" autocomplete="new-password" data-setting="${field.key}"${locked} data-type="secret" data-initial="&quot;&quot;" value="" placeholder="${escapeHtml(field.placeholder ?? 'unchanged')}" />`
                 : field.type === 'addresses'
-                ? `<textarea data-setting="${field.key}"${locked} data-type="addresses"${initial} rows="3" placeholder="${escapeHtml(field.placeholder ?? '')}">${escapeHtml(Array.isArray(value) ? value.join('\n') : value === undefined || value === null ? '' : String(value))}</textarea>`
+                ? // Its own initial, because this is the one field whose
+                    // box holds a different shape from the config. A
+                    // stored string never compares equal to the array
+                    // the box parses to, so sharing `initial` meant
+                    // every Save rewrote a setting nobody had touched
+                    // -- which is how a value that worked became one
+                    // the node would not start with.
+                    `<textarea data-setting="${field.key}"${locked} data-type="addresses" data-initial="${escapeHtml(JSON.stringify(parseAddresses(addressText(value))))}" rows="3" placeholder="${escapeHtml(field.placeholder ?? '')}">${escapeHtml(addressText(value))}</textarea>`
                 : field.type === 'list'
                 ? `<textarea data-setting="${field.key}"${locked} data-type="list"${initial} rows="${Math.min(8, Math.max(3, (Array.isArray(value) ? value.length : 0) + 1))}" placeholder="${escapeHtml(field.placeholder ?? '')}">${escapeHtml((Array.isArray(value) ? value : []).join('\n'))}</textarea>`
                 : field.type === 'boolean'
@@ -6337,6 +6348,56 @@ Every piece is hashed against the ` +
         return panel;
       }
 
+      /**
+       * The entries an address field holds, however the config wrote them.
+       *
+       * A list may be stored as an array or as one comma-separated string --
+       * Express accepts both, and the documented example is a string. Shown
+       * one per line either way, so that what a save reads back is the list
+       * that was rendered rather than a different shape of the same thing.
+       * @param {null|boolean|number|string|string[]} value - What the config holds.
+       * @returns {string} - The textarea's contents.
+       */
+      const addressText = (value) => {
+        if (value === undefined || value === null) return '';
+        const entries = Array.isArray(value) ? value : [String(value)];
+        return entries
+          .flatMap((entry) => String(entry).split(/[\n,]/))
+          .map((entry) => entry.trim())
+          .filter(Boolean)
+          .join('\n');
+      };
+
+      /**
+       * What an address field means by what it holds.
+       *
+       * Express takes four things here and means something different by each:
+       * `true` trusts any caller, a number is a hop count, and a string or an
+       * array of them names the proxies. What was typed decides which.
+       *
+       * Commas separate as well as newlines. Express splits a comma list only
+       * when it is handed a bare string and never inside an array, so one line
+       * reading `10.0.0.1, 10.0.0.2` was saved as a one-element array holding
+       * both -- which is not an address, and which Express rejects while the
+       * app is being built, before the listener binds. The node then would not
+       * start, and could not be corrected from the console that wrote it.
+       * @param {string} text - What the box holds.
+       * @returns {null|boolean|number|string[]} - What to save.
+       */
+      const parseAddresses = (text) => {
+        const lines = String(text)
+          .split(/[\n,]/)
+          .map((line) => line.trim())
+          .filter(Boolean);
+        if (!lines.length) return null;
+        if (lines.length === 1) {
+          if (lines[0] === 'true') return true;
+          if (lines[0] === 'false') return false;
+          if (!Number.isNaN(Number(lines[0]))) return Number(lines[0]);
+        }
+        return lines;
+      };
+
       /**
        * Collects every schema control into an update, grouped by top-level key.
        *
@@ -6368,22 +6429,8 @@ Every piece is hashed against the ` +
             continue;
           }
           if (type === 'boolean') value = element.checked;
-          else if (type === 'addresses') {
-            // Express takes four things here and means something different by
-            // each: `true` trusts any caller, a number is a hop count, and a
-            // string or an array of them names the proxies. One per line, and
-            // what was typed decides which of the four it is.
-            const lines = String(element.value)
-              .split('\n')
-              .map((line) => line.trim())
-              .filter(Boolean);
-            if (!lines.length) value = null;
-            else if (lines.length === 1 && (lines[0] === 'true' || lines[0] === 'false')) {
-              value = lines[0] === 'true';
-            } else if (lines.length === 1 && !Number.isNaN(Number(lines[0]))) {
-              value = Number(lines[0]);
-            } else value = lines;
-          } else if (type === 'list') {
+          else if (type === 'addresses') value = parseAddresses(element.value);
+          else if (type === 'list') {
             // One per line, which is how a person reads a list of trackers —
             // and empty means an empty list rather than "unset", because a
             // node with no trackers is a real thing to want and JSON would