@martintrojer/murmur 0.1.3 → 0.1.4

package/ARCHITECTURE.md

@@ -223,11 +223,19 @@ latencies. Both are the reader pulling, and OpenSSH `ControlMaster` collapses
 them, because the persisted control socket *is* the tunnel. Push needs a
 listener, which is the thing this design does not have.
 
-**The collector never initiates authentication.** A machine that wants a
-hardware-token touch per connection makes a background collector intolerable, so
-the collector rides an existing warm control socket and fails fast otherwise.
-The consequence is correct: remote visibility is a side effect of having worked
-on that box, and a cold host shows stale until you connect for any reason.
+**The collector never prompts.** It reuses a warm `ControlMaster` socket when
+there is one and cold-connects when there is not, so with ordinary key auth a
+peer is visible whether or not you have ssh'd there recently — fleet visibility
+is not rationed by a daily chore. `BatchMode=yes` is what makes the cold path
+acceptable: no password, passphrase or host-key prompt, so a peer that cannot
+authenticate silently fails fast and shows stale instead of blocking on a
+human.
+
+The unhandled case is a host demanding a hardware-token touch per connection,
+where "cannot authenticate without a human" is not something `BatchMode` can
+detect before the token blinks. `hasWarmSocket` exists for that: gating collect
+on it per peer would restore the strict posture for those hosts only. Not wired
+up, because no peer in use needs it.
 
 **Membership is local and asymmetric.** No shared node list, no registry, no
 join protocol. Reachability is not symmetric: a laptop reaches a server, and

package/CHANGELOG.md

@@ -0,0 +1,75 @@
+# Changelog
+
+Notable changes per release. Written for someone deciding whether to upgrade,
+so it says what changed for a user rather than listing every commit.
+
+## 0.1.4
+
+A review pass over the peer-collection code, and the bugs it found.
+
+Collection is driven by tmux re-running `murmur status` on a tick, and the peer
+loop was serial: every peer paid the ssh timeout of every peer ahead of it, so
+three sleeping laptops froze the status bar for thirty seconds. Peers are now
+fetched concurrently, with a bound on the whole collect rather than only on each
+peer.
+
+Fixed, each with a symptom you could have hit:
+
+- **A large peer could never sync.** The ssh export ran into Node's default
+  1 MiB output limit, about 2,600 events. Past that the collect failed, and it
+  failed permanently: the watermark only advances on success, so every retry
+  re-requested the same oversized range. A reachable peer sat stale forever.
+- **Jumping to an agent claimed success even when it failed.** A failed
+  new-window, ssh attach, or select-window all reported success, so the picker
+  closed and nothing moved, with no message. Jump now reports the failure.
+- **A recovered host could stay marked "no tmux".** The recovery check counted
+  database inserts, which read zero on a retry after a partial write, so the
+  host stayed marked dead until it happened to author a new event.
+- **The pi extension leaked a database handle per failed write**, inside a
+  process that can run for days.
+- **ssh timeouts are sized to the status-bar tick** and deliberately
+  aggressive. A slow node is now rejected rather than allowed to hold up the
+  HUD; it shows stale until the next tick.
+- **Retention ran only when peers were configured.** A single-machine node
+  never pruned, so its event log grew without bound.
+
+Internal, no behaviour change: one shared ssh option list instead of three
+hand-rolled copies, `clear`'s queries moved behind `Store`, and `STALENESS_MS`
+states its value instead of deriving it from a collect interval nothing
+enforced.
+
+Tests went 83 to 103. Four existing tests could not fail and were rewritten;
+every new test was verified by breaking the code it covers.
+
+## 0.1.3
+
+Both fixes are about the peer columns being unreadable.
+
+- `peer list` printed tab-separated fields with no header. Now a header and
+  aligned columns.
+- `murmur pick` showed the node's self-reported hostname, which can be a
+  container id -- a string that appears nowhere else and cannot be typed at
+  `peer remove`. It now shows the peer name you configured.
+
+## 0.1.2
+
+- **The picker had a doubled border inside a tmux popup.** `display-popup` draws
+  its own, so fzf's sat one character inside it. The popup is the normal way to
+  run the picker, so this was the common case.
+- **Documented the focus-clear hooks**, which have to be wired by hand per node.
+  Without them a finished agent stays marked `done` forever and the picker fills
+  with rows that need nothing.
+
+## 0.1.1
+
+- A stale badge is now reconciled, and a shell pane no longer clears the badge
+  of the agent pane next to it.
+- Agents are searchable by tmux session, and typing matches substrings rather
+  than scattered characters.
+- The delete key drops a stuck row from the picker.
+- `--version` reads the manifest instead of a hardcoded string.
+- Only `$TMUX_PANE` decides whether we are inside tmux.
+
+## 0.1.0
+
+First release. Agent state across every machine you work on, in one view.

package/dist/cli.js

@@ -3,9 +3,6 @@
 // src/cli.ts
 import { Command } from "commander";
 
-// src/cli/clear.ts
-import Database2 from "better-sqlite3";
-
 // src/identity.ts
 import { randomUUID } from "crypto";
 import { existsSync, mkdirSync, readFileSync, writeFileSync } from "fs";
@@ -102,7 +99,7 @@ var tmux = {
   },
   attach(session, window) {
     runTmux(["switch-client", "-t", session]);
-    runTmux(["select-window", "-t", window]);
+    return runTmux(["select-window", "-t", window]) !== null;
   },
   // Window ids are what the log stores, because they are stable; names are
   // what a human recognises in a picker. Names are live tmux state, not
@@ -134,7 +131,10 @@ var tmux = {
     return null;
   },
   selectWindow(window) {
-    runTmux(["select-window", "-t", window]);
+    return runTmux(["select-window", "-t", window]) !== null;
+  },
+  newWindow(name, command) {
+    return runTmux(["new-window", "-n", name, command]) !== null;
   },
   // The window a pane belongs to, for a pane murmur has no event for. Clearing
   // a badge is a tmux operation and does not require murmur to own the pane.
@@ -322,6 +322,14 @@ function openStore() {
       const rows = database.prepare("SELECT * FROM events ORDER BY ts, host_id, seq").all();
       return rows.map(toEvent);
     },
+    latestForAgent(hostId, agentId) {
+      const row = database.prepare(
+        `SELECT * FROM events
+            WHERE host_id = ? AND agent_id = ?
+            ORDER BY seq DESC LIMIT 1`
+      ).get(hostId, agentId);
+      return row ? toEvent(row) : null;
+    },
     maxSeq(hostId) {
       return selectMaxSeq.get(hostId).seq;
     },
@@ -379,23 +387,15 @@ function openStore() {
 }
 
 // src/cli/clear.ts
-function windowHasAgent(window, focused, hostId, mux) {
+function windowHasAgent(window, focused, hostId, mux, store) {
   if (!hostId) return false;
   const siblings = mux.panesInWindow(window).filter((candidate) => candidate !== focused);
   if (siblings.length === 0) return false;
+  if (!store) return false;
   try {
-    const database = new Database2(dbPath(), { readonly: true, fileMustExist: true });
-    try {
-      for (const sibling of siblings) {
-        const row = database.prepare(
-          `SELECT state FROM events
-              WHERE host_id = ? AND agent_id = ?
-              ORDER BY seq DESC LIMIT 1`
-        ).get(hostId, `${hostId}:${sibling}`);
-        if (row && row.state !== "cleared") return true;
-      }
-    } finally {
-      database.close();
+    for (const sibling of siblings) {
+      const latest = store.latestForAgent(hostId, `${hostId}:${sibling}`);
+      if (latest && latest.state !== "cleared") return true;
     }
     return false;
   } catch {
@@ -403,6 +403,7 @@ function windowHasAgent(window, focused, hostId, mux) {
   }
 }
 function clearPane(pane, mux = tmux) {
+  let store;
   try {
     if (!pane) return;
     const window = mux.windowForPane(pane);
@@ -410,24 +411,13 @@ function clearPane(pane, mux = tmux) {
     let owner;
     if (identity) {
       try {
-        const database = new Database2(dbPath(), { readonly: true, fileMustExist: true });
-        try {
-          owner = database.prepare(
-            `SELECT agent_id, session, window, pane, session_name, window_name,
-                      agent_name, pi_session, workstream, role, cli, driver, state
-                 FROM events
-                WHERE host_id = ? AND agent_id = ?
-                ORDER BY seq DESC
-                LIMIT 1`
-          ).get(identity.host_id, `${identity.host_id}:${pane}`);
-        } finally {
-          database.close();
-        }
+        store = openStore();
+        owner = store.latestForAgent(identity.host_id, `${identity.host_id}:${pane}`) ?? void 0;
       } catch {
       }
     }
     if (!owner) {
-      if (window && !windowHasAgent(window, pane, identity?.host_id, mux)) {
+      if (window && !windowHasAgent(window, pane, identity?.host_id, mux, store)) {
         mux.setState(window, null);
       }
       return;
@@ -436,9 +426,8 @@ function clearPane(pane, mux = tmux) {
       mux.setState(owner.window, null);
       return;
     }
-    const store = openStore();
     try {
-      store.append({
+      store?.append({
         agent_id: owner.agent_id,
         session: owner.session,
         window: owner.window,
@@ -461,11 +450,15 @@ function clearPane(pane, mux = tmux) {
         reason: "",
         extra: {}
       });
-    } finally {
-      store.close();
+    } catch {
     }
     mux.setState(owner.window, null);
   } catch {
+  } finally {
+    try {
+      store?.close();
+    } catch {
+    }
   }
 }
 function registerClear(program2) {
@@ -477,8 +470,8 @@ import { execFile, execFileSync as execFileSync2 } from "child_process";
 import { promisify } from "util";
 var execFileAsync = promisify(execFile);
 var CONTROL_PATH = "~/.ssh/control/%r@%h:%p";
-var CONNECT_TIMEOUT_S = 2;
-var EXEC_TIMEOUT_MS = 1e4;
+var CONNECT_TIMEOUT_S = 1;
+var EXEC_TIMEOUT_MS = 3e3;
 var SSH_OPTIONS = [
   "-o",
   "BatchMode=yes",
@@ -489,11 +482,13 @@ var SSH_OPTIONS = [
   "-o",
   `ConnectTimeout=${CONNECT_TIMEOUT_S}`
 ];
+var MAX_EXPORT_BYTES = 64 * 1024 * 1024;
 var ssh = {
   async exec(target, argv) {
     const { stdout } = await execFileAsync("ssh", [...SSH_OPTIONS, target, ...argv], {
       encoding: "utf8",
-      timeout: EXEC_TIMEOUT_MS
+      timeout: EXEC_TIMEOUT_MS,
+      maxBuffer: MAX_EXPORT_BYTES
     });
     return stdout;
   }
@@ -693,8 +688,30 @@ function exportJsonl(store, since, isAlive, live) {
 }
 
 // src/collector.ts
-var COLLECT_INTERVAL_MS = 3e4;
-var STALENESS_MS = 2 * COLLECT_INTERVAL_MS;
+var STALENESS_MS = 6e4;
+var MAX_CONCURRENT_PEERS = 8;
+var COLLECT_DEADLINE_MS = 4e3;
+async function mapSettled(items, limit, task, deadline) {
+  const results = new Array(items.length);
+  let cursor = 0;
+  let expired = false;
+  const stop = deadline?.then(() => {
+    expired = true;
+  });
+  const worker = async () => {
+    while (cursor < items.length && !expired) {
+      const index = cursor++;
+      try {
+        results[index] = { status: "fulfilled", value: await task(items[index]) };
+      } catch (reason) {
+        results[index] = { status: "rejected", reason };
+      }
+    }
+  };
+  const pool = Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker));
+  await (stop ? Promise.race([pool, stop]) : pool);
+  return results;
+}
 function parseJsonl(output) {
   const lines = output.trim().split("\n");
   const envelope = JSON.parse(lines.shift() ?? "");
@@ -708,20 +725,35 @@ function parseJsonl(output) {
     events: lines.map((line) => eventFromWire(JSON.parse(line)))
   };
 }
-async function collect(store, channel, now = Date.now()) {
+async function collect(store, channel, now = Date.now(), deadline) {
   const results = [];
+  let timer;
   try {
-    for (const peer of store.peers()) {
+    const peers = store.peers();
+    const bounded = deadline ?? new Promise((resolve) => {
+      timer = setTimeout(resolve, COLLECT_DEADLINE_MS);
+      timer.unref?.();
+    });
+    const fetches = await mapSettled(
+      peers,
+      MAX_CONCURRENT_PEERS,
+      async (peer) => parseJsonl(
+        await channel.exec(peer.target, ["murmur", "export", "--since", String(peer.watermark)])
+      ),
+      bounded
+    );
+    for (const [index, peer] of peers.entries()) {
+      const fetch = fetches[index];
       try {
-        const output = await channel.exec(peer.target, [
-          "murmur",
-          "export",
-          "--since",
-          String(peer.watermark)
-        ]);
-        const { envelope, events } = parseJsonl(output);
+        if (!fetch) throw new Error("collect deadline passed before this peer answered");
+        if (fetch.status === "rejected") throw fetch.reason;
+        const { envelope, events } = fetch.value;
         const ingested = store.ingest(events);
-        const watermark = events.filter((event) => event.host_id === envelope.host_id).reduce((highest, event) => Math.max(highest, event.seq), peer.watermark);
+        const origin = events.filter((event) => event.host_id === envelope.host_id);
+        const watermark = origin.reduce(
+          (highest, event) => Math.max(highest, event.seq),
+          peer.watermark
+        );
         store.upsertPeer({
           name: peer.name,
           target: peer.target,
@@ -734,9 +766,16 @@ async function collect(store, channel, now = Date.now()) {
           // events: an export that returns nothing proves the binary ran, not
           // that tmux is back, which is the distinction that let a dead host
           // look healthy for three hours.
-          tmux_down_at: ingested > 0 ? null : peer.tmux_down_at
+          //
+          // Keyed on the watermark advancing, not on ingest's insert count.
+          // Two reasons the count was wrong. Ingest is INSERT OR IGNORE, so a
+          // retry after a partial apply re-sees the same events and reports
+          // zero -- leaving a recovered host marked down until it happened to
+          // author again. And the count includes rows from other origins that
+          // this peer merely relayed, which say nothing about whether this
+          // peer's tmux is back.
+          tmux_down_at: watermark > peer.watermark ? null : peer.tmux_down_at
         });
-        store.prune();
         results.push({ peer: peer.name, ok: true, ingested });
       } catch (error) {
         const message = error instanceof Error ? error.message : String(error);
@@ -748,6 +787,16 @@ async function collect(store, channel, now = Date.now()) {
   } catch (error) {
     process.stderr.write(
       `murmur: collect: ${error instanceof Error ? error.message : String(error)}
+`
+    );
+  } finally {
+    clearTimeout(timer);
+  }
+  try {
+    store.prune();
+  } catch (error) {
+    process.stderr.write(
+      `murmur: collect: prune: ${error instanceof Error ? error.message : String(error)}
 `
     );
   }
@@ -854,6 +903,22 @@ function formatTable(rows) {
   ).map((line) => `${line}
 `).join("");
 }
+function peerAddDecision(input) {
+  const { name, target, envelope, selfHostId, peers } = input;
+  if (!envelope) return null;
+  if (envelope.host_id === selfHostId) {
+    return `${target} is this node; not adding it as a peer
+`;
+  }
+  const existing = peers.find(
+    (candidate) => candidate.host_id === envelope.host_id && candidate.name !== name
+  );
+  if (existing) {
+    return `${target} is already configured as peer "${existing.name}" (${envelope.display_name}); remove it first to rename
+`;
+  }
+  return null;
+}
 function registerPeer(program2) {
   const peer = program2.command("peer").description("Manage peers");
   peer.command("add").description("Add a peer and discover its identity").argument("<name>").argument("[target]").action(async (name, target = name) => {
@@ -866,22 +931,17 @@ function registerPeer(program2) {
       } catch {
         envelope = null;
       }
-      if (envelope) {
-        if (envelope.host_id === loadIdentity()?.host_id) {
-          process.stderr.write(`${target} is this node; not adding it as a peer
-`);
-          process.exitCode = 1;
-          return;
-        }
-        const existing = store.peers().find((candidate) => candidate.host_id === envelope.host_id && candidate.name !== name);
-        if (existing) {
-          process.stderr.write(
-            `${target} is already configured as peer "${existing.name}" (${envelope.display_name}); remove it first to rename
-`
-          );
-          process.exitCode = 1;
-          return;
-        }
+      const refusal = peerAddDecision({
+        name,
+        target,
+        envelope,
+        selfHostId: loadIdentity()?.host_id ?? null,
+        peers: store.peers()
+      });
+      if (refusal) {
+        process.stderr.write(refusal);
+        process.exitCode = 1;
+        return;
       }
       store.upsertPeer({
         name,
@@ -971,6 +1031,21 @@ function terminalText(value) {
 function shellQuote(value) {
   return `'${value.replaceAll("'", `'\\''`)}'`;
 }
+var spawnRunner = (file, args, inherit = false) => {
+  const result = spawnSync(file, args, {
+    encoding: "utf8",
+    timeout: 1e4,
+    ...inherit ? { stdio: "inherit" } : {}
+  });
+  return {
+    status: result.status,
+    stdout: result.stdout ?? "",
+    // spawnSync reports a failure to even start the child in `error`, leaving
+    // status null. Collapsing both here keeps the decision table below reading
+    // as one question rather than two.
+    failed: result.error !== void 0
+  };
+};
 function forgetHostReplica(store, hostId) {
   try {
     const peer = store.peers().find((candidate) => candidate.host_id === hostId);
@@ -1003,10 +1078,10 @@ function forgetOneAgent(store, agent, mux = tmux) {
   }
   forgetReplica(store, agent.agent_id, agent.host_id);
 }
-function jumpToAgent(store, agent) {
+function jumpToAgent(store, agent, mux = tmux, run = spawnRunner) {
   const identity = loadIdentity();
   if (agent.host_id === identity?.host_id) {
-    const live = tmux.liveWindows();
+    const live = mux.liveWindows();
     if (live && !live.has(agent.window)) {
       forgetReplica(store, agent.agent_id, agent.host_id);
       return {
@@ -1015,7 +1090,13 @@ function jumpToAgent(store, agent) {
         message: `${agentLabel(agent)} is gone -- its window no longer exists. Cleared.`
       };
     }
-    tmux.attach(agent.session, agent.window);
+    if (!mux.attach(agent.session, agent.window)) {
+      return {
+        ok: false,
+        reason: "attach_failed",
+        message: `could not attach to ${agentLabel(agent)} (tmux select-window failed).`
+      };
+    }
     return { ok: true };
   }
   const peer = store.peers().find((candidate) => candidate.host_id === agent.host_id);
@@ -1027,18 +1108,18 @@ function jumpToAgent(store, agent) {
       message: `No peer configured for host ${agent.host_id.slice(0, 8)}. Try: murmur peer add <target>`
     };
   }
-  const probe = spawnSync(
-    "ssh",
-    ["-o", "BatchMode=yes", target, `tmux list-windows -a -F ${shellQuote("#{window_id}")}`],
-    { encoding: "utf8", timeout: 1e4 }
-  );
+  const probe = run("ssh", [
+    ...SSH_OPTIONS,
+    target,
+    `tmux list-windows -a -F ${shellQuote("#{window_id}")}`
+  ]);
   if (probe.status !== 0) {
-    const sshFailed = probe.status === 255 || probe.error !== void 0;
+    const sshFailed = probe.status === 255 || probe.failed;
     if (sshFailed) {
       return {
         ok: false,
         reason: "unreachable",
-        message: `cannot reach ${target} over ssh. The collector never prompts for auth, so connect once by hand to warm the connection, then retry.`
+        message: `cannot reach ${target} over ssh. Nothing here ever prompts for auth, so check the host is awake and reachable, or connect once by hand to see the real error.`
       };
     }
     forgetHostReplica(store, agent.host_id);
@@ -1048,7 +1129,7 @@ function jumpToAgent(store, agent) {
       message: `${target} has no tmux server running, so its agents are gone. Removed them; they will come back when it reports again.`
     };
   }
-  const remoteWindows = new Set((probe.stdout ?? "").split("\n").filter(Boolean));
+  const remoteWindows = new Set(probe.stdout.split("\n").filter(Boolean));
   if (!remoteWindows.has(agent.window)) {
     forgetReplica(store, agent.agent_id, agent.host_id);
     return {
@@ -1061,31 +1142,31 @@ function jumpToAgent(store, agent) {
   if (process.env.TMUX) {
     const command = `ssh -t ${shellQuote(target)} tmux attach -t ${shellQuote(attachTarget)}`;
     const name = `@${peer?.name ?? target}`;
-    const existing = tmux.windowNamed(name);
+    const existing = mux.windowNamed(name);
     if (existing) {
-      tmux.selectWindow(existing);
-      return { ok: true };
+      return mux.selectWindow(existing) ? { ok: true } : {
+        ok: false,
+        reason: "attach_failed",
+        message: `could not switch to the existing ${name} window.`
+      };
     }
-    spawnSync("tmux", ["new-window", "-n", name, command], { stdio: "ignore" });
-    return { ok: true };
+    return mux.newWindow(name, command) ? { ok: true } : {
+      ok: false,
+      reason: "attach_failed",
+      message: `could not open a window to attach to ${target}.`
+    };
   }
-  spawnSync("ssh", ["-t", target, "tmux", "attach", "-t", attachTarget], { stdio: "inherit" });
-  return { ok: true };
+  const attach = run("ssh", ["-t", target, "tmux", "attach", "-t", attachTarget], true);
+  return attach.status === 0 && !attach.failed ? { ok: true } : {
+    ok: false,
+    reason: "attach_failed",
+    message: `ssh attach to ${target} failed.`
+  };
 }
 
 // src/glance.ts
 import { execFileSync as execFileSync3 } from "child_process";
 var GLANCE_LINES = 40;
-var SSH_OPTIONS2 = [
-  "-o",
-  "BatchMode=yes",
-  "-o",
-  "ControlMaster=no",
-  "-o",
-  "ControlPath=~/.ssh/control/%r@%h:%p",
-  "-o",
-  "ConnectTimeout=2"
-];
 function glance(store, agent, lines = GLANCE_LINES) {
   if (agent.host_id === loadIdentity()?.host_id) return tmux.capture(agent.pane, lines);
   const peer = store.peers().find((candidate) => candidate.host_id === agent.host_id);
@@ -1095,7 +1176,7 @@ function glance(store, agent, lines = GLANCE_LINES) {
     return execFileSync3(
       "ssh",
       [
-        ...SSH_OPTIONS2,
+        ...SSH_OPTIONS,
         target,
         "tmux",
         "capture-pane",
@@ -1185,9 +1266,9 @@ function status(store, now = Date.now()) {
     }))
   };
 }
-async function statusWithCollect(store, now = Date.now()) {
+async function statusWithCollect(store, now = Date.now(), channel = ssh) {
   try {
-    await collect(store, ssh, now);
+    await collect(store, channel, now);
   } catch (error) {
     process.stderr.write(
       `murmur: status: collect: ${error instanceof Error ? error.message : String(error)}