@thehammer/danx-dashboard-mcp 0.1.305 → 0.1.306

package/README.md

@@ -75,7 +75,7 @@ The contract the plugin builds against:
 - **The accepted restart gap.** Only an insert that commits more than the dashboard's 30 s overlap after the event it follows can be missed, and only by a restart resume (the dashboard's own limit); a running relay never misses one. A burst of more than 64 events inside the overlap can replay the oldest once (a duplicate, never a loss).
 - **One plan.** `plan_id` is the plan the session is on. A plan different from the last call's drops the plan-scoped records buffered or held for the plan it left (dashboard events, the digest, the idle nudge), so a plan move starts clean. Session-scoped records (the pacing and auto-resume messages, which are never replayed, and the relay's own failure and revoked-key notices) are kept across a plan move and across the gap below. One wait is pending at a time: a newer call supersedes the pending one (which answers empty at once); an aborted request drops its waiter.
 - **One heartbeat window.** `POST /api/plan-sessions/me/heartbeat` every 15 s while a call is pending or was made within the last 40 s; after that the stream is closed and heartbeats stop, so a dead or unloaded plugin module reads not-healthy (the dashboard's liveness window is 60 s). The plugin therefore waits 20 s per call and backs off at most 10 s between failed calls. Stats and the transcript title ride the heartbeat when `transcript_path` is given. The sequence survives the gap, so a cursor stays valid across it.
-- **Failures inside the server.** A transient mint failure (network, timeout, 408, 429, 5xx) and a dropped stream back off 1 s doubling to 30 s (the one schedule, `listen.ts`); a lapsed ticket lease (15 min) re-mints and re-reads a failed plan inventory; three heartbeats that fail to land force a reconnect and tell the session once; a revoked key tells the session once.
+- **Failures inside the server.** A transient mint failure (network, timeout, 408, 429, 5xx) and a dropped stream back off 1 s doubling to 30 s (the one schedule, `listen.ts`); a lapsed ticket lease (15 min) re-mints and re-reads a failed plan inventory; three heartbeats in a row that fail to land on an open stream force a fresh connection silently (a line on stderr, then an immediate heartbeat on the new stream), and the session is told only once the heartbeat has failed continuously for `HEARTBEAT_SUSTAINED_OUTAGE_MS` (5 minutes), in any connection state, once per outage, so a dashboard restart never wakes it; a revoked key tells the session once.
 - **Nothing outlives the session.** The heartbeat interval, the backoff sleep and the digest hold are unref'd; `close()` (run when the transport closes and when stdin ends) aborts the stream and the mint and inventory requests and clears a pending call's wait timer, the one ref'd timer, which otherwise holds the process only until the call answers.
 - **Credential and audience.** The server's own, from its boot-time `CLAUDE_CODE_SESSION_ID`: no connection record, fingerprint check or environment variable is involved. The tool is advertised only to a person's session holding `plans.view`, never to a dispatch. The stream's advisory frames (the idle nudge, DX-4340's pacing message, DX-4342's auto-resume message) arrive as records like any event, at once and never held for a digest. The digest (agent and machine events held up to 10 minutes) is one record, `[plan events] digest`, and a held id is never acknowledged before it is returned.
 

package/dist/heartbeat-reporter.js

@@ -10,7 +10,7 @@
 import { keyRevokedHalt } from "./key-revoked-halt.js";
 import { oneLine } from "./one-line.js";
 import { DIGEST_PREFIX } from "./digest-hold.js";
-import { HEARTBEAT_INTERVAL_MS, HEARTBEAT_SILENT_FAILURE_THRESHOLD, sendHeartbeat, } from "./session-stream.js";
+import { HEARTBEAT_SILENT_FAILURE_THRESHOLD, HEARTBEAT_SUSTAINED_OUTAGE_MS, sendHeartbeat, } from "./session-stream.js";
 import { OperatorSessionStats } from "./session-stats.js";
 import { TranscriptTitleReader } from "./session-title.js";
 export class HeartbeatReporter {
@@ -23,8 +23,11 @@ export class HeartbeatReporter {
     // since an older dashboard refuses the `title` key and a refused heartbeat is a dead listener.
     dashboardTitle;
     lastTitleError = null;
-    // DX-3099 - consecutive heartbeats that failed to LAND while the owner believed it was streaming.
+    // DX-3099 - consecutive heartbeats that failed to LAND on one open stream (DX-5088: per stream, reset whenever the state leaves `streaming`).
     failureStreak = 0;
+    // DX-5088 - when the heartbeat first failed to land in the current outage (across every reconnect); null while the last one landed.
+    outageSince = null;
+    outageAnnounced = false;
     // DX-4418 - the stop notice for a revoked key goes out once; every later beat is refused the same way.
     revocationAnnounced = false;
     constructor(deps, transcriptPath) {
@@ -90,10 +93,34 @@ export class HeartbeatReporter {
             return answer;
         })
             .then((landed) => {
-            // DX-3099 - only counts while the owner still believes it is streaming: that is the exact desync the card found (the
-            // client believes it is fine; the dashboard has nothing recent for it). A failure while reconnecting or stopped is
-            // not new information: the mint backoff already owns recovery there.
-            if (!this.deps.streaming() || landed) {
+            const now = this.deps.now();
+            if (landed) {
+                this.failureStreak = 0;
+                this.outageSince = null;
+                this.outageAnnounced = false;
+                return;
+            }
+            this.outageSince ??= now;
+            // DX-5088 - the model hears of an outage only once it is sustained, in any connection state (a dashboard that stays down keeps
+            // the stream reconnecting, and that is exactly the outage worth saying); a blip such as a ~35 s deploy restart is stderr only.
+            const downMs = now - this.outageSince;
+            if (downMs >= HEARTBEAT_SUSTAINED_OUTAGE_MS && !this.outageAnnounced) {
+                this.outageAnnounced = true;
+                const minutes = Math.round(downMs / 60_000);
+                const connection = this.deps.connection();
+                // DX-5088: the wording follows the state. A stopped relay (a terminal outcome) does not reconnect by itself, so "keeps reconnecting" would be false.
+                const wording = connection.state === "stopped"
+                    ? `this session's listener has stopped${connection.reason === undefined ? "" : ` (${connection.reason})`} and does not reconnect by itself, ` +
+                        `so the dashboard has heard nothing from it for ${minutes} minutes. The next plan_events_wait call says why it stopped and what to do; ` +
+                        `a person can also restart the session.`
+                    : `this session's listener has not reached the dashboard for ${minutes} minutes (its heartbeats keep failing). ` +
+                        `It keeps reconnecting on its own, and nothing is needed from you.`;
+                this.deps.announce(oneLine(`${DIGEST_PREFIX} ${wording}`), "relay_notice");
+            }
+            // DX-3099 - the self-heal counts only while a stream is open: that is the desync the card found (the client's stream reads
+            // open; the dashboard has nothing recent for it). DX-5088: while it is reconnecting the reconnect already owns recovery, and
+            // aborting it would restart the backoff, so a failure then neither counts nor forces anything.
+            if (!this.deps.streaming()) {
                 this.failureStreak = 0;
                 return;
             }
@@ -101,15 +128,12 @@ export class HeartbeatReporter {
             if (this.failureStreak < HEARTBEAT_SILENT_FAILURE_THRESHOLD)
                 return;
             this.failureStreak = 0;
-            // The session finds out, rather than continuing to believe it is connected.
-            this.deps.announce(oneLine(`${DIGEST_PREFIX} this session's listener stopped reaching the dashboard for ` +
-                `${Math.round((HEARTBEAT_SILENT_FAILURE_THRESHOLD * HEARTBEAT_INTERVAL_MS) / 1000)}s while it still ` +
-                `believed it was streaming — forcing a fresh connection now.`), "relay_notice");
+            process.stderr.write(`plan_events_wait: ${HEARTBEAT_SILENT_FAILURE_THRESHOLD} heartbeats in a row did not land on a stream that reads open; forcing a fresh connection\n`);
             // End the CURRENT connection now, rather than wait on the stream reader's own read-idle timeout (which this exact
             // failure mode can outlive indefinitely: the connection reads as established and keeps producing keep-alive bytes,
             // only the outbound heartbeat silently never lands). The new attempt reconnects with the SAME ticket; the
             // dashboard's one-open-stream-per-ticket rule then ends the old, superseded entry, releasing the exclusive claim
-            // as a side effect of THIS reconnect, never an external watchdog.
+            // as a side effect of THIS reconnect, never an external watchdog. The new stream's `streaming` sends a heartbeat at once.
             this.deps.forceReconnect();
         })
             .catch((err) => {

package/dist/listen.js

@@ -693,6 +693,9 @@ export async function runListener(options, deps) {
         if (outcome.kind === "refused") {
             return stop("refused", `the dashboard refused the listener ticket: ${outcome.detail}`, 1);
         }
+        // DX-5088: the stream is down from here until the next attempt opens; reporting `streaming` through the backoff made the heartbeat watchdog
+        // call a dashboard outage "a stream that looks open" and abort the reconnect it was waiting on.
+        deps.onState?.("reconnecting");
         if (outcome.detail !== undefined)
             lastDropDetail = outcome.detail;
         if (outcome.healthy) {

package/dist/plan-events-relay.js

@@ -277,6 +277,7 @@ export class PlanEventRelay {
             },
             http: this.deps,
             makeStatsWatcher: this.deps.makeStatsWatcher,
+            now: this.deps.now,
             connection: () => heartbeatConnection(this.connectionState, this.stoppedReason, this.lastInventoryError),
             streaming: () => this.connectionState === "streaming",
             // `origin: null` skips the agent/machine hold: the notice reaches the session at once. It is about the session, not a plan.
@@ -471,6 +472,9 @@ export class PlanEventRelay {
                     if (!this.current(run))
                         return;
                     this.connectionState = state;
+                    // DX-5088: the heartbeat failure streak is per stream; a drop ends it, so a quick re-open never inherits the old count.
+                    if (state !== "streaming")
+                        this.reporter?.resetFailureStreak();
                     // Report `streaming` now, not up to a beat later: the session reads healthy as soon as it is.
                     if (state === "streaming" && this.deps.auth() !== null)
                         this.reporter?.tick();

package/dist/session-stream.js

@@ -57,6 +57,14 @@ export const HEARTBEAT_INTERVAL_MS = 15_000;
  * of the obvious wrong fixes this constant is not).
  */
 export const HEARTBEAT_SILENT_FAILURE_THRESHOLD = 3;
+/**
+ * DX-5088 - how long the heartbeat must keep failing to land (counted from the first failure, across every reconnect, until one
+ * lands) before the MODEL is told. 5 minutes: a production deploy replaces the single dashboard container, an outage the
+ * investigation measured at about 35 s (DX-5085 comment 14821), and the lock-convoy stalls around it ran 8-27 s more; 5 minutes is
+ * roughly 8x the restart, so a deploy never wakes a session, while an outage that outlasts it is one the session's person should hear
+ * about (the dashboard has long read the session `unattached`). Below it the self-heal is silent and logged to stderr.
+ */
+export const HEARTBEAT_SUSTAINED_OUTAGE_MS = 5 * 60_000;
 const TRANSIENT_CLIENT_STATUSES = new Set([408, 429]);
 /**
  * One authenticated request, with a timeout, classifying the failures that are

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@thehammer/danx-dashboard-mcp",
-  "version": "0.1.305",
+  "version": "0.1.306",
   "description": "Stdio MCP server fronting danxbot's dashboard API for agents (operator sessions and dispatched workers alike): the generic danxbot_api/danxbot_api_list/danxbot_api_spec tools reach any route the caller's credential may call, plus dedicated agent-specific tools (plan_connect, request_permission, attach_file, and danxbot_slack_reply) (DX-3465, PLN-11 R-25).",
   "license": "MIT",
   "type": "module",