pmtiles-swarm 0.27.0 → 0.29.0

package/CHANGELOG.md

@@ -7,6 +7,56 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.29.0
+### ✨ Features and improvements
+- **A connection indicator in the header, for whether the swarm can reach this node.** A node
+  nothing can connect to still downloads and still uploads — it dials out and its transfers work —
+  so none of its own traffic reveals that half the swarm can never start a conversation with it.
+  What it loses is invisible and permanent: fewer peers, slower starts, and a seed nobody fetches
+  from unless they were introduced to it first.
+
+  Green when something has connected inward, amber when the node is listening and nothing ever
+  has, red when it is not listening at all. libtorrent answers from
+  `net.has_incoming_connections`, which latches for the session, so a reachable node that is
+  merely quiet stays green rather than flickering when its last peer leaves. WebTorrent keeps no
+  such gauge, so it is assembled from the wires — each carries the direction it was made in — and
+  latched for the same reason.
+
+  Reported per engine rather than blended. Two engines means two listening ports, forwarded
+  separately, and one can be reachable while the other is not; a single verdict would have to hide
+  the one somebody needs to fix. The header shows the primary and names both on hover.
+
+  The amber state reads "no incoming yet", not "firewalled". On a node no peer has tried those are
+  the same observation, and claiming the first would put a warning on a node that is merely new.
+  An engine that cannot answer hides the indicator instead of showing red — not being able to ask
+  is not the same as being unreachable, and a red light on a healthy node is worse than none.
+
+  Needs pmtiles-torrent 0.5.0 for the libtorrent engine; against an older sidecar the indicator
+  simply stays hidden.
+
+### 🐞 Bug fixes
+
+## 0.28.0
+### ✨ Features and improvements
+- **The archive list can be searched and sorted, and says when each archive was added.** `Added` is
+  a column now — `createdAt` has always been on every entry and returned by `/api/catalog`, it was
+  simply never shown — with a date for anything older than today and a time for today, since the
+  question a list answers is which of these is recent rather than exactly when each arrived.
+
+  Beside it, the same filter and sort the public page has: by name, infohash or category, ordered
+  by newest added, oldest added, name or size — and by download speed, upload speed or share
+  ratio, which are read off the live status the poll refreshes, so rows reorder themselves under
+  those every few seconds. That is what a torrent client does and what choosing "download speed"
+  asks for, and also why none of them is the default. Newest added is, because a list read
+  straight after adding something should have that thing at the top.
+
+  Both happen inside the render rather than where the data arrives. The list refreshes every three
+  seconds; filtering at the fetch would either clear what had been typed on each poll or refetch
+  the whole catalog on every keystroke. And the count beside the box says `4 of 37` while a filter
+  is narrowing things, because an empty table and a table filtered down to nothing look identical.
+
+### 🐞 Bug fixes
+
 ## 0.27.0
 ### ✨ Features and improvements
 - **Bandwidth history per archive, kept across restarts.** The tile side of this question already

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.27.0",
+  "version": "0.29.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -530,6 +530,17 @@ export function createApp({
           ? config.incompleteSuffix || null
           : null,
         engine: { name: engine.name, ok: engineOk, error: engineError },
+        // Whether the swarm can reach us, as opposed to whether we can reach
+        // it. A node that cannot be connected to still downloads and still
+        // uploads, so nothing about its own traffic reveals that half the
+        // swarm can never start a conversation with it.
+        reachability:
+          typeof engine.reachability === 'function'
+            ? await engine.reachability().catch((error) => ({
+                state: 'unknown',
+                error: error.message,
+              }))
+            : null,
         archives: catalog.list().length,
         categories: catalog.categories(),
         watching: config.watch.map((w) => w.path),

package/src/engines/composite.js

@@ -303,6 +303,29 @@ export class CompositeEngine {
    * Every archive, with the peers and speeds of all engines added together.
    * @returns {Promise<import('./types.js').TorrentStatus[]>} - Merged status.
    */
+  /**
+   * Reachability, per engine rather than blended into one verdict.
+   *
+   * Two engines means two listening ports, and they are forwarded separately.
+   * One can be reachable while the other is not, so a single answer would have
+   * to either pick a winner or average two facts into something that is not
+   * true of either -- and the one it got wrong is the one somebody needs to
+   * fix. The primary leads because it is the engine that downloads.
+   * @returns {Promise<object>} - `{state, engines}`.
+   */
+  async reachability() {
+    const engines = [];
+    for (const engine of [this.#primary, ...this.#secondaries]) {
+      if (typeof engine.reachability !== 'function') continue;
+      const report = await engine.reachability().catch((error) => ({
+        state: 'unknown',
+        error: error.message,
+      }));
+      if (report) engines.push({ engine: engine.name, ...report });
+    }
+    return { ...(engines[0] ?? { state: 'unknown' }), engines };
+  }
+
   async list() {
     if (this.#stopping) return [];
     const primary = await this.#primary.list();

package/src/engines/libtorrent.js

@@ -284,6 +284,25 @@ export class LibtorrentEngine {
     return result?.trackers ?? [];
   }
 
+  /**
+   * Whether peers can open a connection to this node, or only the reverse.
+   *
+   * See the sidecar's op_reachability for what the three states mean and why
+   * the middle one is "unproven" rather than "firewalled": on a node with no
+   * peers, blocked and untried are the same observation.
+   * @returns {Promise<object|null>} - The report, or null when unavailable.
+   */
+  async reachability() {
+    if (this.#stopping) return null;
+    try {
+      return await this.#call('reachability', {});
+    } catch (error) {
+      // An engine that cannot answer is not an engine that is unreachable, and
+      // reporting it as offline would put a red light on a healthy node.
+      return { state: 'unknown', error: error.message };
+    }
+  }
+
   async list() {
     // A node that is shutting down still has a console polling it and a sweep
     // or two in flight. Answering "the sidecar exited" to each of them fills

package/src/engines/types.js

@@ -60,6 +60,7 @@
  * @property {(filePath: string, options?: object) => Promise<object>} [createTorrent] - Builds a torrent from a local file, where the engine can do it better than the default — libtorrent produces hybrid v1+v2, which create-torrent cannot.
  * @property {(infoHash: string) => Promise<object[]>} [trackerStatus] - Per-tracker announce results, where the engine keeps them.
  * @property {(infoHash: string) => Promise<Uint8Array | null>} [metadata] - The torrent's metainfo once known, so an archive joined by magnet can be written down rather than re-fetched over BEP 9 on every start.
+ * @property {() => Promise<object | null>} [reachability] - Whether peers can open a connection to this node, or only the reverse. Three states: `open` (something has connected inward), `unproven` (listening, but nothing ever has) and `offline` (not listening at all). The middle is deliberately not called firewalled -- on a node with no peers, blocked and untried are the same observation.
  * @property {() => Promise<void>} destroy - Releases resources.
  */
 

package/src/engines/webtorrent.js

@@ -34,6 +34,7 @@ import {
 export class WebTorrentSeedEngine {
   #options;
   #client = null;
+  #everIncoming = false;
   /**
    * A client error that means nothing will ever work.
    *
@@ -319,6 +320,43 @@ export class WebTorrentSeedEngine {
    * Lists every torrent the client holds.
    * @returns {Promise<import('./types.js').TorrentStatus[]>} - Normalised statuses.
    */
+  /**
+   * Whether peers can open a connection to this node, or only the reverse.
+   *
+   * WebTorrent keeps no equivalent of libtorrent's has_incoming_connections
+   * gauge, so it is assembled from the wires: every one carries the direction
+   * it was made in, and a type ending "Incoming" is somebody who reached us.
+   *
+   * Latched rather than sampled, which is the whole reason for the field. A
+   * wire is gone the moment the peer leaves, so asking "is one open now" would
+   * report a reachable node as unproven every time it went quiet. What is worth
+   * knowing is whether it has ever happened at all.
+   * @returns {Promise<object|null>} - The report, or null when not started.
+   */
+  async reachability() {
+    const client = this.#client;
+    if (!client) return null;
+
+    if (!this.#everIncoming) {
+      this.#everIncoming = (client.torrents ?? []).some((torrent) =>
+        (torrent.wires ?? []).some((wire) =>
+          String(wire.type ?? '').endsWith('Incoming'),
+        ),
+      );
+    }
+
+    const listening = Boolean(client.listening);
+    return {
+      state: !listening ? 'offline' : this.#everIncoming ? 'open' : 'unproven',
+      listening,
+      port: client.torrentPort ?? null,
+      peersConnected: (client.torrents ?? []).reduce(
+        (sum, torrent) => sum + (torrent.numPeers ?? 0),
+        0,
+      ),
+    };
+  }
+
   async list() {
     if (!this.#client) return [];
     return this.#client.torrents.map((torrent) => this.#normalise(torrent));

package/src/web/index.html

@@ -49,6 +49,19 @@
       h2 { font-size: 0.95rem; margin: 0 0 0.6rem; font-weight: 600; }
       .status { color: var(--muted); font-size: 0.85rem; }
       .status b { color: var(--fg); font-weight: 600; }
+      /* A dot rather than an image: it inherits the theme, needs no asset,
+         and stays legible at the size a header allows. */
+      .reach {
+        display: inline-flex; align-items: center; gap: 0.35rem;
+        font-size: 0.8rem; color: var(--muted); cursor: default;
+      }
+      .reach::before {
+        content: ""; width: 0.6rem; height: 0.6rem; border-radius: 50%;
+        background: var(--muted);
+      }
+      .reach.open::before { background: var(--ok); }
+      .reach.unproven::before { background: var(--warn); }
+      .reach.offline::before { background: var(--bad); }
       nav { margin-left: auto; display: flex; gap: 0.35rem; }
       nav button.on { background: var(--accent); color: #fff; border-color: var(--accent); }
       main { padding: 1.25rem; }
@@ -305,6 +318,7 @@
     <header>
       <h1>pmtiles-swarm</h1>
       <div class="status" id="status">connecting…</div>
+      <span id="reach" class="reach" hidden></span>
       <button id="speed-toggle" class="speed" hidden title="Switch between the normal and alternative speed limits"></button>
       <nav>
         <button id="tab-archives" class="on">Archives</button>
@@ -324,12 +338,36 @@
             <button type="button">Feed</button>
           </a>
         </div>
+        <div class="bar">
+          <input
+            id="archive-filter"
+            type="search"
+            placeholder="Filter by name, infohash or category"
+            autocomplete="off"
+            spellcheck="false"
+            style="flex:1;min-width:12rem"
+          />
+          <label>
+            Sort
+            <select id="archive-sort">
+              <option value="added">newest added</option>
+              <option value="oldest">oldest added</option>
+              <option value="name">name</option>
+              <option value="largest">largest</option>
+              <option value="down">download speed</option>
+              <option value="up">upload speed</option>
+              <option value="ratio">share ratio</option>
+            </select>
+          </label>
+          <span class="sub" id="archive-count"></span>
+        </div>
 
         <table>
           <thead>
             <tr>
               <th>Archive</th>
               <th>Size</th>
+              <th>Added</th>
               <th>Mode</th>
               <th>Origin</th>
               <th>Progress</th>
@@ -710,6 +748,56 @@
       const rate = (n) => (n > 0 ? `${bytes(n)}/s` : '—');
       const pct = (n) => `${Math.round((n ?? 0) * 100)}%`;
 
+      /**
+       * When an archive was added, as something short enough for a column.
+       *
+       * A date on its own for anything older than today, and a time for
+       * today: the question a list answers is "which of these is recent", and
+       * a full timestamp on every row is harder to scan than either.
+       * @param {string} value - ISO timestamp.
+       * @returns {string} - Markup for the cell.
+       */
+      const added = (value) => {
+        if (!value) return '<span class="sub">—</span>';
+        const when = new Date(value);
+        if (Number.isNaN(when.getTime())) return '<span class="sub">—</span>';
+        const today = new Date().toDateString() === when.toDateString();
+        return `<span title="${escapeHtml(when.toLocaleString())}">${
+          today
+            ? when.toLocaleTimeString([], { hour: '2-digit', minute: '2-digit' })
+            : when.toLocaleDateString()
+        }</span>`;
+      };
+
+      // The same three questions the public page asks, against an admin row.
+      // Newest first by default: a list read after adding something is a list
+      // where the thing just added should be at the top.
+      /**
+       * A live figure off an archive's status, or zero.
+       *
+       * An archive the engine has not reported on yet has no status at all,
+       * and comparing undefined gives NaN -- which makes a sort silently do
+       * nothing rather than fail, so it is worth the guard.
+       * @param {object} entry - Catalog entry with its status attached.
+       * @param {string} field - Which figure.
+       * @returns {number} - The value, or zero.
+       */
+      const speedOf = (entry, field) => Number(entry?.status?.[field] ?? 0) || 0;
+
+      const archiveSorters = {
+        added: (a, b) => new Date(b.createdAt ?? 0) - new Date(a.createdAt ?? 0),
+        oldest: (a, b) => new Date(a.createdAt ?? 0) - new Date(b.createdAt ?? 0),
+        name: (a, b) => String(a.name ?? '').localeCompare(String(b.name ?? '')),
+        largest: (a, b) => (b.size ?? 0) - (a.size ?? 0),
+        // Live values, read off the status the poll refreshes. Rows reorder
+        // themselves every few seconds under these, which is what a torrent
+        // client does and what somebody choosing "download speed" is asking
+        // for -- but it is also why they are not the default.
+        down: (a, b) => speedOf(b, 'downloadSpeed') - speedOf(a, 'downloadSpeed'),
+        up: (a, b) => speedOf(b, 'uploadSpeed') - speedOf(a, 'uploadSpeed'),
+        ratio: (a, b) => speedOf(b, 'ratio') - speedOf(a, 'ratio'),
+      };
+
       let toastTimer;
       const toast = (message) => {
         $('toast').textContent = message;
@@ -913,6 +1001,50 @@
         }
       };
 
+      /**
+       * The connection indicator: can the swarm reach us, or only we it?
+       *
+       * A node nothing can connect to still downloads and still uploads, so
+       * none of its own traffic reveals the problem -- it simply gets fewer
+       * peers, slower starts, and nobody fetching from it unless introduced
+       * first. That is the whole reason for showing it at all.
+       *
+       * The middle state says "no incoming yet" rather than "firewalled",
+       * because those are the same observation on a node no peer has tried.
+       * Calling it firewalled would put a warning on a healthy node that is
+       * merely new or quiet, which is worse than saying less.
+       *
+       * @param {object} report - status.reachability.
+       * @returns {void}
+       */
+      function renderReach(report) {
+        const host = $('reach');
+        if (!report || !report.state || report.state === 'unknown') {
+          host.hidden = true;
+          return;
+        }
+        host.hidden = false;
+        host.className = `reach ${report.state}`;
+
+        const label = {
+          open: 'reachable',
+          unproven: 'no incoming yet',
+          offline: 'not listening',
+        }[report.state];
+        host.textContent = label;
+
+        const detail = (report.engines ?? [report]).map((one) => {
+          const which = one.engine ? `${one.engine}: ` : '';
+          const port = one.port ? ` on port ${one.port}` : '';
+          if (one.state === 'offline') return `${which}not listening`;
+          if (one.state === 'open') {
+            return `${which}${one.incomingConnections ?? 0} peer(s) have connected in${port}`;
+          }
+          return `${which}listening${port}, but nothing has connected in yet — it may be firewalled, or simply untried`;
+        });
+        host.title = detail.join('\n');
+      }
+
       async function refresh() {
         try {
           const [status, list, speed, adds] = await Promise.all([
@@ -928,6 +1060,7 @@
           // nothing renames it. The engine decides, not the setting alone.
           incompleteMarker = status.incompleteMarker ?? null;
           const engine = status.engine;
+          renderReach(status.reachability);
           $('status').innerHTML =
             `engine <b>${engine.name}</b> ${engine.ok ? 'ready' : 'unavailable'}` +
             ` · <b>${status.archives}</b> archives`;
@@ -1004,9 +1137,34 @@
       function renderRows() {
         const rows = $('rows');
         rows.innerHTML = '';
+
+        // Filtered and sorted here rather than where the data arrives, so the
+        // three-second poll keeps refreshing what is on screen without
+        // resetting what was typed. `archives` stays the node's answer; this
+        // is only the view of it.
+        const needle = ($('archive-filter')?.value ?? '').trim().toLowerCase();
+        const order = $('archive-sort')?.value ?? 'added';
+        const shown = archives
+          .filter(
+            (entry) =>
+              needle === '' ||
+              String(entry.name ?? '').toLowerCase().includes(needle) ||
+              String(entry.infoHash ?? '').toLowerCase().includes(needle) ||
+              (entry.categories ?? []).some((tag) =>
+                String(tag).toLowerCase().includes(needle),
+              ),
+          )
+          .sort(archiveSorters[order] ?? archiveSorters.added);
+
         $('empty').hidden = archives.length > 0;
+        // Says when a filter is hiding things, because an empty table and a
+        // table filtered down to nothing look identical otherwise.
+        $('archive-count').textContent =
+          needle === ''
+            ? ''
+            : `${shown.length} of ${archives.length}`;
 
-        for (const entry of archives) {
+        for (const entry of shown) {
           const s = entry.status ?? {};
           const progress = s.progress ?? 0;
           const mode = entry.mode ?? 'mirror';
@@ -1024,6 +1182,7 @@
                 ? `<div class="sub">${escapeHtml(entry.kind)} · not servable</div>`
                 : ''
             }</td>
+            <td>${added(entry.createdAt)}</td>
             <td><span class="pill ${mode}">${mode}</span></td>
             <td>${originCell(entry)}</td>
             <td>
@@ -4204,6 +4363,8 @@
       $('traffic-live').onchange = trafficPoll;
       trafficPoll();
 
+      $('archive-filter').addEventListener('input', renderRows);
+      $('archive-sort').addEventListener('change', renderRows);
       $('traffic-window').onchange = () => loadSwarmTraffic();
       $('traffic-reset').onclick = async () => {
         // Deliberately separate from reading: a page polling the endpoint must