pmtiles-swarm 0.5.5 → 0.6.0

package/CHANGELOG.md

@@ -7,6 +7,53 @@
 ### 🐞 Bug fixes
 - _...Add new stuff here..._
 
+## 0.6.0
+### ✨ Features and improvements
+- **Check now, on scheduled sources and on feeds.** A schedule describes ordinary operation, and
+  setting one up is not ordinary operation — waiting six hours to find out whether a URL template
+  is right is how a typo survives a working day. `POST /api/sources/check` is new; the feed
+  equivalent existed and had no button. Both check what is *saved* rather than what is on screen,
+  since an unsaved row is not a source this node knows about.
+- **Run the completion hook again for one archive**, from its General tab or
+  `POST /api/torrents/<infohash>/hooks/complete`. Completion is recorded before the command runs,
+  so a build taking six hours is not started six times over — but a hook that failed for a reason
+  since fixed keeps that record too, and the only way to run it again was to stop the node and
+  edit the catalog by hand. Started rather than awaited, since the command may be a planet build;
+  refused with 409 if it is already running for that archive, because two builds writing the same
+  output is worse than waiting. It does not choose *which* command — that is still the config
+  file's business, and `allowHooksFromApi` still guards choosing it.
+
+### 🐞 Bug fixes
+- **The node no longer crashes while shutting down.** `child.stdin` had no `error` listener, and
+  an unhandled `'error'` event is not a rejected promise — it is a throw that takes the process
+  with it. systemd's default `KillMode` signals every process in a service's cgroup, so the Python
+  sidecar exited first and the shutdown request was written into a dead pipe:
+
+  ```
+  [shutdown] SIGTERM
+  Error: write EPIPE ... at #call (src/engines/libtorrent.js)
+  Main process exited, code=exited, status=1/FAILURE
+  ```
+
+  Every stop ended that way, and because the crash happened *inside* `destroy()`, the shutdown
+  that saves resume data never ran — so this was also a third, independent reason resume data
+  went missing. `Restart=always` brought the node back five seconds later looking healthy, which
+  is why it went unnoticed. A failed write now fails the call that made it; the pipe is checked
+  before writing; and there is a test that spawns a sidecar which reports ready and then exits.
+
+## 0.5.6
+### 🐞 Bug fixes
+- **The Pieces and Peers tabs redraw themselves.** A detail pane is filled once, the first time
+  it is opened — right for the panes describing an archive, whose name and source and trackers
+  do not change, and wrong for the two describing what is happening *now*. Pieces arrived and
+  peers came and went behind a picture taken when the tab was opened, and the only way to see
+  the current one was to close the panel and open it again, or reload the page. Those two now
+  follow the same three-second tick as the archive table. The rest stay lazy, since redrawing a
+  pane nobody is watching is requests every tick for nothing — which is why they load on demand
+  in the first place. Only while visible, and never queued behind a redraw still in flight: a
+  piece map of a large archive is a real request, and three seconds is not long enough to assume
+  the last one finished.
+
 ## 0.5.5
 ### 🐞 Bug fixes
 - **An archive could retire itself from head-warming and never say so.** The test for "already

package/README.md

@@ -604,6 +604,8 @@ matters there is `maxConnections`, since every peer holds a NAT table entry. See
 | `POST` | `/api/sources/preview` | What a watched web location would take, without taking it |
 | `POST` | `/api/subscriptions/preview` | Whether a peer is reachable and what it offers |
 | `POST` | `/api/subscriptions/refresh` | Poll subscribed feeds now |
+| `POST` | `/api/sources/check` | Check scheduled sources now, rather than at the next due time |
+| `POST` | `/api/torrents/:infoHash/hooks/complete` | Run the completion hook again for one archive |
 | `GET` `POST` `DELETE` | `/api/tokens`, `/api/tokens/:id` | Mint, list and revoke access tokens |
 | `GET` `POST` | `/api/restart` | What a restart would do, and doing it |
 | `GET` `PATCH` | `/api/config` | Read and change settings |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "pmtiles-swarm",
-  "version": "0.5.5",
+  "version": "0.6.0",
   "description": "BitTorrent distribution for PMTiles map archives: create torrents, watch folders, publish and subscribe to RSS feeds, and seed through qBittorrent or an embedded client",
   "type": "module",
   "main": "src/index.js",

package/src/api.js

@@ -63,6 +63,8 @@ export function createApp({
   catalog,
   engine,
   subscriptions,
+  sources,
+  hooks,
   tiles,
   warm,
   config,
@@ -1386,6 +1388,55 @@ export function createApp({
     }),
   );
 
+  // The same for scheduled sources: ask now rather than at the next due time.
+  // Worth having for the same reason a feed refresh is — a schedule is a
+  // statement about ordinary operation, and setting one up is not ordinary
+  // operation. Waiting six hours to find out whether a URL template is right
+  // is how a typo survives a working day.
+  app.post(
+    '/api/sources/check',
+    route(async (_req, res) => {
+      if (!sources?.sweep) {
+        return res.status(501).json({ error: 'no scheduled sources here' });
+      }
+      const taken = await sources.sweep(new Date());
+      res.json({ taken: taken.length, entries: taken });
+    }),
+  );
+
+  // Run the completion hook again for one archive.
+  //
+  // The sweep runs it once and records that it has, so a six-hour build is not
+  // started six times over — but a hook that failed for a reason since fixed
+  // keeps that record too, and the only way to run it again was to stop the
+  // node and edit the catalog by hand.
+  //
+  // A POST, and therefore refused to a read-only token: this runs a command as
+  // the service user. It does not choose *which* command — that is still the
+  // config file's business, and `allowHooksFromApi` still guards it.
+  app.post(
+    '/api/torrents/:infoHash/hooks/complete',
+    route(async (req, res) => {
+      if (!hooks?.runComplete) {
+        return res.status(501).json({ error: 'hooks are not available here' });
+      }
+      try {
+        const started = await hooks.runComplete(req.params.infoHash);
+        res.json({ started: true, ...started });
+      } catch (error) {
+        // Told apart because they mean different things to whoever pressed
+        // the button: nothing configured, nothing to run it for, or already
+        // going.
+        const status = /unknown archive/.test(error.message)
+          ? 404
+          : /already running/.test(error.message)
+            ? 409
+            : 400;
+        res.status(status).json({ error: error.message });
+      }
+    }),
+  );
+
   /**
    * How many items this feed request should return.
    *

package/src/engines/libtorrent.js

@@ -134,6 +134,24 @@ export class LibtorrentEngine {
       });
       this.#child = child;
 
+      // A pipe to a process that has gone raises 'error' on the stream, and an
+      // unhandled 'error' event is not a rejected promise — it is a throw that
+      // takes the whole node down with it. That is what happened on every
+      // stop: systemd's default KillMode signals every process in the cgroup,
+      // so the sidecar exited first and the shutdown request was written into
+      // a dead pipe. The service died with EPIPE and status=1/FAILURE, having
+      // never saved its resume data, and no `.catch()` on the call could have
+      // caught it.
+      child.stdin.on('error', (error) => {
+        if (!this.#stopping) {
+          console.warn(`[libtorrent] sidecar input failed: ${error.message}`);
+        }
+        for (const [id, waiter] of this.#pending) {
+          this.#pending.delete(id);
+          waiter.reject(error);
+        }
+      });
+
       const timer = setTimeout(() => {
         reject(
           new Error(
@@ -481,7 +499,22 @@ export class LibtorrentEngine {
         },
       });
 
-      child.stdin.write(`${JSON.stringify({ id, op, params })}\n`);
+      // Checked and reported rather than thrown at. Writing to a closed pipe
+      // is ordinary during shutdown, and a call that cannot be sent should
+      // fail as a call rather than as the process.
+      if (!child.stdin.writable || child.stdin.destroyed) {
+        this.#pending.delete(id);
+        clearTimeout(timer);
+        reject(new Error(`libtorrent ${op}: the sidecar is no longer running`));
+        return;
+      }
+
+      child.stdin.write(`${JSON.stringify({ id, op, params })}\n`, (error) => {
+        if (!error) return;
+        this.#pending.delete(id);
+        clearTimeout(timer);
+        reject(new Error(`libtorrent ${op}: ${error.message}`));
+      });
     });
   }
 

package/src/hooks.js

@@ -193,6 +193,48 @@ export class ProgramHooks {
     return fired;
   }
 
+  /**
+   * Runs the completion hook for one archive, now, whatever it did before.
+   *
+   * The sweep runs a hook once and records that it has: a build taking six
+   * hours must not be started six times over. But a hook that failed for a
+   * reason since fixed — a path that was wrong, a directory that was not
+   * writable — keeps that record too, and until now the only way to run it
+   * again was to stop the node and edit the catalog by hand.
+   *
+   * Started rather than awaited. The command may be a planet build, and a
+   * request that waited for it would time out long before it finished.
+   * @param {string} infoHash - The archive to run it for.
+   * @returns {Promise<object>} - What was started.
+   */
+  async runComplete(infoHash) {
+    if (!this.#config.onComplete?.command) {
+      throw new Error('no onComplete hook is configured');
+    }
+
+    const live = await this.#library.listWithStatus().catch(() => []);
+    const entry = live.find((candidate) => candidate.infoHash === infoHash);
+    if (!entry) throw new Error('unknown archive');
+    if (this.#running.has(infoHash)) {
+      throw new Error('it is already running for this archive');
+    }
+
+    // Recorded before starting, the same as the sweep does and for the same
+    // reason: whatever happens next, this is not a fresh completion to be
+    // picked up again a minute later.
+    await this.#library.catalog.put({
+      infoHash,
+      completedAt: new Date().toISOString(),
+    });
+
+    this.#running.add(infoHash);
+    this.#fire('onComplete', entry)
+      .catch(() => {})
+      .finally(() => this.#running.delete(infoHash));
+
+    return { name: entry.name, command: this.#config.onComplete.command };
+  }
+
   /**
    * Runs one of the configured hooks, if it is configured.
    * @param {string} which - 'onAdded' or 'onComplete'.

package/src/index.js

@@ -262,6 +262,8 @@ PMTILES_SWARM_PUBLIC_URL
     catalog,
     engine,
     subscriptions,
+    sources,
+    hooks,
     tiles,
     warm,
     config,

package/src/web/index.html

@@ -729,6 +729,21 @@
       // before the node has said whether its engine applies one.
       let incompleteMarker = null;
       let activeTab = 'general';
+      /**
+       * Panes worth redrawing while they are open.
+       *
+       * A detail pane is filled once, the first time it is shown, which is
+       * right for the ones describing an archive: its name, its source, its
+       * trackers. It is wrong for these two, which describe what is happening
+       * *now* — pieces arriving and peers coming and going — and which sat
+       * frozen until the reader thought to close the panel and open it again.
+       *
+       * The rest stay as they are. Refreshing a pane nobody is watching is
+       * four requests a tick for nothing, which is why they are loaded lazily
+       * in the first place.
+       */
+      const LIVE_PANES = new Set(['pieces', 'peers']);
+      let refreshingPane = false;
 
       // What the header switch shows, so a click knows what it is toggling to.
       let speedState = null;
@@ -894,6 +909,7 @@
             ` · <b>${status.archives}</b> archives`;
           archives = list;
           renderRows();
+          refreshOpenPane();
         } catch (error) {
           $('status').textContent = `disconnected: ${error.message}`;
         }
@@ -1072,6 +1088,29 @@
           ({ '&': '&amp;', '<': '&lt;', '>': '&gt;', '"': '&quot;', "'": '&#39;' })[c],
         );
 
+      /**
+       * Redraws the open detail pane, when it is one that changes by itself.
+       *
+       * Guarded against overlapping, because a piece map of a large archive is
+       * a real request and a slow one must not have a queue of its successors
+       * behind it.
+       * @returns {void}
+       */
+      function refreshOpenPane() {
+        if (refreshingPane || !selected || !LIVE_PANES.has(activeTab)) return;
+        const pane = document.querySelector(`[data-pane="${activeTab}"]`);
+        // Only while it is on screen: a hidden pane is one the reader moved
+        // away from, and the panel may have been replaced since.
+        if (!pane || pane.hidden) return;
+
+        refreshingPane = true;
+        Promise.resolve(fillPane(activeTab, selected))
+          .catch(() => {})
+          .finally(() => {
+            refreshingPane = false;
+          });
+      }
+
       /**
        * Fills one detail tab, the first time it is opened.
        * @param {string} name - Which pane.
@@ -1347,6 +1386,20 @@
                  </div>`
           }
 
+          <h2>Completion hook</h2>
+          <div class="actions" id="hook-box">
+            <button id="run-hook">Run it for this archive</button>
+            <span class="sub" id="hook-note"></span>
+          </div>
+          <div class="sub" style="margin-bottom:1rem">
+            Runs the configured <code>onComplete</code> command again for this
+            archive. It normally runs once, when the download finishes, and is
+            recorded as having run so a build taking hours is not started over
+            every minute — which also means a hook that failed for a reason
+            since fixed will not retry on its own. What it does is decided in
+            the config file; this only decides when.
+          </div>
+
           <h2>Mode</h2>
           <div class="actions" id="mode-box">
             <button id="mode-mirror">Switch to mirror</button>
@@ -1409,6 +1462,30 @@
         const loaded = new Set(['general']);
 
         /** Shows one tab, loading it the first time. */
+        const runHook = $('run-hook');
+        if (runHook) {
+          runHook.onclick = async () => {
+            const note = $('hook-note');
+            runHook.disabled = true;
+            note.textContent = 'starting…';
+            try {
+              const result = await api(
+                `/api/torrents/${infoHash}/hooks/complete`,
+                { method: 'POST' },
+              );
+              // Started, not finished: the command may be a planet build, and
+              // nothing here waits for one.
+              note.textContent = `started ${result.command ?? 'the hook'}`;
+              toast('completion hook started');
+            } catch (error) {
+              note.textContent = error.message;
+              toast(error.message);
+            } finally {
+              runHook.disabled = false;
+            }
+          };
+        }
+
         const showPane = (name) => {
           activeTab = name;
           for (const other of panel.querySelectorAll('.tabs button')) {
@@ -2324,8 +2401,23 @@
        * @returns {void}
        */
       function renderRowEditor(spec) {
-        const { into, key, title, blurb, columns, rows, restart, preview, peerPreview, footnote } =
-          spec;
+        const {
+          into,
+          key,
+          title,
+          blurb,
+          columns,
+          rows,
+          restart,
+          preview,
+          peerPreview,
+          footnote,
+          // Where to POST a "do it now". A schedule is a statement about
+          // ordinary operation, and setting one up is not ordinary operation:
+          // waiting six hours to learn whether a URL template is right is how
+          // a typo survives a working day.
+          checkNow,
+        } = spec;
         rowEditorColumns[key] = columns;
         // Kept so a save can put back what this editor never showed. An entry
         // holds more than there are columns for it — a watch folder's
@@ -2404,6 +2496,11 @@
           </div>
           <div style="margin-top:0.5rem">
             <button type="button" data-act="add">Add row</button>
+            ${
+              checkNow
+                ? '<button type="button" data-act="check-now">Check now</button>'
+                : ''
+            }
             <span class="sub" data-role="note"></span>
           </div>
           ${footnote ? `<div class="sub" style="margin-top:0.6rem">${escapeHtml(footnote)}</div>` : ''}`;
@@ -2436,6 +2533,29 @@
             tbody.insertAdjacentHTML('beforeend', rowHtml());
             return;
           }
+
+          if (action === 'check-now') {
+            // Against what is saved, not what is on screen. An unsaved row is
+            // not something the node knows about, and pretending otherwise
+            // would report on a source that does not exist yet.
+            const button = event.target;
+            button.disabled = true;
+            note.textContent = 'checking…';
+            try {
+              const result = await api(checkNow, { method: 'POST' });
+              const count = result.added ?? result.taken ?? 0;
+              note.textContent =
+                count > 0
+                  ? `took ${count} — save first if you have unsaved changes`
+                  : 'nothing new';
+              if (count > 0) refresh();
+            } catch (error) {
+              note.textContent = error.message;
+            } finally {
+              button.disabled = false;
+            }
+            return;
+          }
           if (action === 'drop') {
             // Never leave it with nothing to type into.
             if (tbody.children.length === 1) row.querySelectorAll('input').forEach((i) => (i.value = ''));
@@ -3280,6 +3400,7 @@
             },
           ],
           rows: config.sources ?? [],
+          checkNow: '/api/sources/check',
           preview: true,
           footnote:
             'Tokens combine however the upstream spells its filenames — ' +
@@ -3411,6 +3532,7 @@
             },
           ],
           rows: config.subscriptions ?? [],
+          checkNow: '/api/subscriptions/refresh',
           peerPreview: true,
           footnote:
             'Take as decides the disk this costs: cache joins the swarm and ' +