@cohortapp/agent-sdk 2.18.14 → 2.18.16

package/docs/runbooks/fleet-rollout.md

@@ -79,12 +79,56 @@ summary}` — whose `reason` separates the cases the exit code merges:
 
 | `reason` | meaning |
 | --- | --- |
-| `verified` | every seat reports the target version (`code` 0) |
+| `verified` | every seat reports the target version **and** no seat is stuck (`code` 0) |
+| `stuck` | at least one seat has attempted `@latest` repeatedly and failed — waiting will not fix it |
 | `behind` | at least one seat is **provably** on an older version |
 | `unverifiable` | no seat is known to be behind; nothing available can prove any seat is current |
 | `empty-fleet` | no beating seats were found — never treated as done |
 | `no-fleet-read` | the org read never succeeded; says nothing about the fleet |
 
+### `behind` is waited out; `stuck` is visited
+
+Every seat is `behind` for twenty minutes after a publish. Three seats were
+behind for three weeks (2.17.0 while the fleet ran 2.18.13) and **nothing was in
+an error state**: the hourly job fired, npm answered, the install ran, the health
+gate failed honestly, the rollback worked, `last.json` recorded it and the beat
+carried it. Every one of those records was correct. Only the REPETITION says the
+seat will never arrive unaided, and no single record can hold it — so the org
+learned it the way it always had, from a person noticing an old number in a
+table.
+
+`stuck` is that repetition, made sayable. A seat earns it two ways:
+
+- **streak basis** — the seat counted its own consecutive failures
+  (`machine.upgrade.failStreak`, SDK 2.18.15+) and reached the threshold.
+- **attempt basis** — an older beat carries one outcome and no count. A failed
+  attempt with the seat *still reporting the version it failed from* proves the
+  hop did not take; older than two failed-target holds (48 h) it also proves the
+  retry it was owed came and went. Below that it reads `failing`, not `stuck`:
+  a seat that has not been ASKED again is not a stuck seat.
+
+Two things the table will not tell you, so they are said here:
+
+- **`stuck` is not a subset of `behind`.** A seat's verdict is about *this run's*
+  target; its streak is about whatever `@latest` was when it last tried. A seat
+  can be current on the version you are verifying and unable to move off it.
+- **The seat-side alert cannot cover the seats this is for.** `upgrade_stuck`
+  (`lib/telemetry/alerts.mjs`) is derived *on* the seat, so a seat too stale to
+  install the SDK never emits it — by construction the alert is blind to exactly
+  the machines it was written for. This tool is their cover, and it is automatic
+  because publishing is: stage 3 runs on every publish, and a stuck seat makes
+  the release command exit 3 with `reason: "stuck"` and the machine's name.
+  **Between publishes nothing watches them** — if a release is weeks away, run
+  `--verify-only` deliberately.
+
+**What to do with a stuck seat:** it needs a person at the machine. `maestro
+doctor` there, read `state/autoupdate/last.json` for the `reason`
+(`install-failed` is a network problem; `unhealthy-rolled-back` means the gate
+rejected the new version and the rollback worked; `rollback-unhealthy` means the
+gate rejects the OLD version too, so the seat is sick independently of the
+upgrade). Deleting `state/autoupdate/last.json` clears the 24 h hold and the
+streak, and the next hourly run retries immediately.
+
 `unverifiable` becomes `verified` the moment the one-line hq change below lands,
 with no change to this script. `propagationOutcome()` is pure and exported, so
 the mapping is pinned by test rather than by this table.

package/lib/assurance/batch.mjs

@@ -0,0 +1,353 @@
+/**
+ * lib/assurance/batch.mjs — the unit of acknowledgement is the CONVERSATION,
+ * not the message.
+ *
+ * THE COMPLAINT THIS EXISTS TO FIX
+ *
+ * "If there's a flurry of 4 messages in one go by others, instead of having a
+ *  generic reply per message, it would be responding to the batch of messages."
+ *
+ * The acknowledgement path already counts per ROOM (`room-budget.mjs`: one
+ * interim per (seat, service, channel) per fifteen minutes) and per DEBT
+ * (`interimSaid`, latched and inherited across retries). Neither of those is
+ * the thing being asked for here, and it is worth being exact about why,
+ * because "the room budget already caps it at one" is the wrong answer:
+ *
+ *   The room budget caps HOW MANY. It says nothing about WHICH, or WHEN.
+ *
+ * Under the budget alone, four messages arriving together open four debts; the
+ * sweep reaches whichever one `readdir` yielded first, composes a holding line
+ * from THAT ONE MESSAGE, sends it, and spends the room's fifteen minutes. The
+ * other three are latched silent. So the room does get exactly one line — and
+ * that line answers a quarter of what was said, chosen arbitrarily, while the
+ * sender watches three of their four messages go unmentioned. One arbitrary
+ * reply is not the same thing as one reply to the batch.
+ *
+ * Worse, the budget cannot wait. The sweep fires on the first debt to cross
+ * ACK_AFTER_MS, which is the FIRST message of the flurry. A person mid-flurry
+ * gets acknowledged for message one and then sends messages two, three and
+ * four into a room that has already spent its budget.
+ *
+ * THE MODEL
+ *
+ * A conversation is (service, channel) — deliberately the same key as
+ * `roomKey`, so this gate and the budget can never disagree about what a room
+ * is. Within a conversation, the open debts that may still speak form a BATCH.
+ * A batch has three times on it:
+ *
+ *   firstAt   the earliest arrival. The human has been waiting since here, so
+ *             ACK_AFTER_MS is measured from it — unchanged from today.
+ *   lastAt    the latest arrival. The batch is not finished until it has been
+ *             QUIET for `quietMs` after this, which is the whole point: an ack
+ *             that fires mid-flurry cannot possibly address the flurry.
+ *   span      lastAt - firstAt, capped by `maxSpanMs` so a slow-drip
+ *             conversation can never defer its acknowledgement forever.
+ *
+ * READY = aged past ACK_AFTER_MS, and (quiet, or spanned out, or urgent).
+ *
+ * WHY TWENTY SECONDS OF QUIET
+ *
+ * The quiet window is not a display-grouping heuristic (chat clients group at
+ * 30-60 s for layout; that is a rendering question, not an obligation one). It
+ * is the answer to "how long after someone's last message may I still expect
+ * another in the same breath". Twenty seconds is chosen because it is BELOW
+ * the floor at which a human expects any response at all — nothing is said
+ * before ACK_AFTER_MS (90 s) regardless, so in the common case where a flurry
+ * lands inside five seconds the quiet window costs exactly nothing and the ack
+ * still fires at 90 s — and ABOVE the inter-message gap of a real flurry,
+ * where a follow-up thought arrives while the previous message is still on
+ * screen. Raising it buys later acks for no additional coverage; lowering it
+ * starts splitting one person's two-part thought into two batches.
+ *
+ * WHAT HAPPENS TO A GENUINELY URGENT MESSAGE MID-WINDOW — the question this
+ * design has to answer honestly, because a quiet window is a delay and a delay
+ * is exactly the wrong thing to add to an emergency. Three answers, in order:
+ *
+ *   1. It was never delayed by this module in the first place. The quiet
+ *      window only ever moves an acknowledgement — a courtesy — and only
+ *      within the span between 90 s and `maxSpanMs`. The ANSWER is dispatched
+ *      by the daemon the moment the item is classified; nothing here touches
+ *      the session, the work, or the reply.
+ *   2. `isUrgent` (critical priority, or the classifier's urgent flags) takes
+ *      the record OUT of the batch into a singleton of its own, which waives
+ *      the quiet wait entirely: it acks at ACK_AFTER_MS on its own terms and
+ *      does not wait behind its noisier neighbours. It still passes the room
+ *      budget, so it cannot double up with a batch ack.
+ *   3. `maxSpanMs` (10 min) is the backstop for the case urgency flags miss: a
+ *      conversation that never falls quiet is acknowledged anyway at ten
+ *      minutes, because at that point the flurry IS the conversation and
+ *      waiting for it to end is waiting forever.
+ *
+ * PURE. Everything arrives on the argument — the records, the clock, the
+ * windows. Nothing is read from disk, the environment or the network, so the
+ * whole matrix below is pinned by a test rather than sampled from a daemon.
+ * The environment is resolved by FUNCTIONS called at the edge, never folded
+ * into a constant at import time (the lesson `room-budget.mjs` records: a
+ * constant captured at module load means an operator who exports the variable
+ * afterwards gets the stale default with no error).
+ *
+ * @module lib/assurance/batch
+ */
+
+"use strict";
+
+const num = (v, d) => { const n = parseInt(v, 10); return Number.isFinite(n) ? n : d; };
+
+/**
+ * How long a conversation must be QUIET before its batch may be acknowledged.
+ * See the header for why twenty seconds and not five or sixty.
+ */
+export const DEFAULT_BATCH_QUIET_MS = 20_000;
+
+/**
+ * The hard cap on how long a batch may stay open waiting for quiet. Past this,
+ * the batch is acknowledged whether or not the flurry has stopped — the
+ * starvation guard, and the honest answer to a conversation that never stops.
+ *
+ * Ten minutes is sized against the measured session distribution (p50 14.7
+ * min): an acknowledgement that arrives later than this is competing with the
+ * answer, at which point silence is better.
+ */
+export const DEFAULT_BATCH_MAX_SPAN_MS = 10 * 60_000;
+
+/** The configured quiet window. A FUNCTION, read at the edge — see the header. */
+export function batchQuietMs(env = process.env) {
+  return num(env && env.ASSURANCE_BATCH_QUIET_MS, DEFAULT_BATCH_QUIET_MS);
+}
+
+/** The configured maximum batch span. Same reasoning. */
+export function batchMaxSpanMs(env = process.env) {
+  return num(env && env.ASSURANCE_BATCH_MAX_SPAN_MS, DEFAULT_BATCH_MAX_SPAN_MS);
+}
+
+/**
+ * PURE. The conversation's identity.
+ *
+ * (service, channel), normalised — BYTE-FOR-BYTE the same construction as
+ * `room-budget.roomKey`, and that is a requirement rather than a coincidence.
+ * If this module batched by thread while the budget counted by channel, a
+ * four-thread room would assemble four batches and the budget would silently
+ * refuse three of them: three batches latched silent having said nothing,
+ * which is the failure mode this whole file exists to end. One definition of a
+ * room, or the two gates disagree.
+ *
+ * @param {object} [o] {service, channel}
+ * @returns {string}
+ */
+export function conversationKey(o) {
+  const a = o && typeof o === "object" ? o : {};
+  const norm = (v) => String(v == null ? "" : v).trim().toLowerCase() || "unknown";
+  return `${norm(a.service)}|${norm(a.channel)}`;
+}
+
+/**
+ * PURE. Does this debt refuse to wait for its neighbours?
+ *
+ * Deliberately narrow. "Urgent" here does not mean "important" — it means "a
+ * twenty-second wait for the room to fall quiet is itself a defect". Only two
+ * signals qualify: the classifier's own `critical` priority, and an explicit
+ * urgent flag from the poller's priority signals. Anything looser (a keyword
+ * scan over the body, a CEO sender) would take most of a busy room out of
+ * batching and hand the flood back.
+ *
+ * @param {object} rec  an obligation record
+ * @returns {boolean}
+ */
+export function isUrgent(rec) {
+  if (!rec || typeof rec !== "object") return false;
+  const p = String(rec.priority == null ? "" : rec.priority).trim().toLowerCase();
+  if (p === "critical") return true;
+  const sig = (rec.item && rec.item.priority_signals) || rec.priority_signals;
+  return !!(sig && typeof sig === "object" && sig.tagged_urgent === true);
+}
+
+/**
+ * PURE. When did this debt arrive?
+ *
+ * `openedAt`, and NOT `lastAttemptAt`: a retry restarts the work but it does
+ * not restart the conversation, and measuring a batch from a retry clock would
+ * let a single retrying debt drag its whole room's batch forward repeatedly.
+ * A record with no usable clock is treated as arriving `now`, which makes it
+ * the batch's newest member and therefore delays the ack rather than firing
+ * one early — the safe direction, because the cost is a later courtesy and the
+ * alternative cost is acknowledging a flurry that is still arriving.
+ */
+export function arrivalAt(rec, now) {
+  const t = rec && rec.openedAt;
+  return Number.isFinite(t) ? t : (Number.isFinite(now) ? now : 0);
+}
+
+/**
+ * PURE. Split interim-eligible debts into the batches that will each earn at
+ * most ONE first response.
+ *
+ * Urgent records are extracted into singleton batches BEFORE grouping, so an
+ * urgent ask is never held behind a chatty neighbour and never drags a quiet
+ * conversation's batch forward.
+ *
+ * Returned batches carry their conversation key and their members sorted
+ * OLDEST FIRST, because that is the order the acknowledgement has to read them
+ * in and the order the carrier selection assumes.
+ *
+ * @param {object} o
+ * @param {object[]} o.records  interim-eligible obligation records
+ * @param {number}   o.now
+ * @returns {{conversation:string, urgent:boolean, records:object[]}[]}
+ */
+export function groupIntoBatches(o = {}) {
+  const records = Array.isArray(o.records) ? o.records.filter((r) => r && typeof r === "object") : [];
+  const now = Number.isFinite(o.now) ? o.now : 0;
+  const batches = [];
+  const grouped = new Map();
+
+  for (const rec of records) {
+    const conv = conversationKey(rec);
+    if (isUrgent(rec)) {
+      batches.push({ conversation: conv, urgent: true, records: [rec] });
+      continue;
+    }
+    if (!grouped.has(conv)) grouped.set(conv, []);
+    grouped.get(conv).push(rec);
+  }
+
+  for (const [conversation, group] of grouped) {
+    // Oldest first. Ties broken on `key` so two records stamped in the same
+    // millisecond order identically in every process — a sweep that picked a
+    // different carrier per tick would be a different message each time.
+    group.sort((a, b) => {
+      const d = arrivalAt(a, now) - arrivalAt(b, now);
+      return d !== 0 ? d : String(a.key || "").localeCompare(String(b.key || ""));
+    });
+    batches.push({ conversation, urgent: false, records: group });
+  }
+
+  return batches;
+}
+
+/**
+ * PURE. Is this batch ready to be acknowledged, and if so, by whom?
+ *
+ * THE CARRIER IS THE NEWEST RECORD, not the oldest. The acknowledgement is
+ * about to be read directly underneath the most recent thing the person said,
+ * and a line that answers the oldest message of four reads as an agent that
+ * stopped listening three messages ago. The older members are `speaksFor`:
+ * they are latched silent by the caller and their content is handed to the
+ * composer, so the one line that does go out was written with all four
+ * messages in front of it.
+ *
+ * `ready:false` is NOT a refusal and the caller must not latch anything on it.
+ * It means "not yet" — the batch is still assembling, and the next sweep tick
+ * will ask again. That distinction is the first of the two boundary pins:
+ * a flurry's LAST message must be able to join the batch it belongs to, which
+ * it can only do if an unready batch leaves every member untouched.
+ *
+ * @param {object} o
+ * @param {{records:object[], urgent?:boolean}} o.batch
+ * @param {number} o.now
+ * @param {number} o.ackAfterMs   nothing is said before this, measured from firstAt
+ * @param {number} [o.quietMs]
+ * @param {number} [o.maxSpanMs]
+ * @returns {{ready:boolean, reason:string, carrier:object|null, speaksFor:object[],
+ *            firstAt:number, lastAt:number, size:number, waitMs:number}}
+ */
+export function batchVerdict(o = {}) {
+  const batch = o.batch && typeof o.batch === "object" ? o.batch : {};
+  const records = Array.isArray(batch.records) ? batch.records.filter(Boolean) : [];
+  const now = Number.isFinite(o.now) ? o.now : 0;
+  const ackAfterMs = Number.isFinite(o.ackAfterMs) ? o.ackAfterMs : 0;
+  const quietMs = Number.isFinite(o.quietMs) ? o.quietMs : DEFAULT_BATCH_QUIET_MS;
+  const maxSpanMs = Number.isFinite(o.maxSpanMs) ? o.maxSpanMs : DEFAULT_BATCH_MAX_SPAN_MS;
+
+  const empty = { ready: false, reason: "empty-batch", carrier: null, speaksFor: [], firstAt: 0, lastAt: 0, size: 0, waitMs: 0 };
+  if (!records.length) return empty;
+
+  const stamped = records.map((r) => ({ rec: r, at: arrivalAt(r, now) }));
+  const firstAt = Math.min(...stamped.map((s) => s.at));
+  const lastAt = Math.max(...stamped.map((s) => s.at));
+  const size = records.length;
+
+  // Newest wins the carrier; `key` breaks a same-millisecond tie the same way
+  // in every process.
+  let carrierEntry = stamped[0];
+  for (const s of stamped) {
+    if (s.at > carrierEntry.at) { carrierEntry = s; continue; }
+    if (s.at === carrierEntry.at
+      && String(s.rec.key || "").localeCompare(String(carrierEntry.rec.key || "")) > 0) carrierEntry = s;
+  }
+  const carrier = carrierEntry.rec;
+  const speaksFor = records.filter((r) => r !== carrier);
+
+  const age = now - firstAt;
+  const quiet = now - lastAt;
+  const span = lastAt - firstAt;
+
+  // TIME first, and measured from firstAt, exactly as branch (e) does today.
+  if (age < ackAfterMs) {
+    return { ready: false, reason: "not-yet-aged", carrier, speaksFor, firstAt, lastAt, size, waitMs: ackAfterMs - age };
+  }
+
+  // URGENT waives the quiet wait — see the header's answer 2.
+  if (batch.urgent === true) {
+    return { ready: true, reason: "urgent", carrier, speaksFor, firstAt, lastAt, size, waitMs: 0 };
+  }
+
+  // SPANNED OUT — the starvation guard. Checked BEFORE quiet, because a
+  // conversation that has run past the cap is ready whether or not the current
+  // instant happens to be quiet, and ordering it the other way would make the
+  // reported reason depend on the arrival jitter rather than the rule.
+  if (span >= maxSpanMs) {
+    return { ready: true, reason: "max-span", carrier, speaksFor, firstAt, lastAt, size, waitMs: 0 };
+  }
+
+  // QUIET — the flurry has stopped.
+  if (quiet < quietMs) {
+    return { ready: false, reason: "flurry-open", carrier, speaksFor, firstAt, lastAt, size, waitMs: quietMs - quiet };
+  }
+
+  return { ready: true, reason: size > 1 ? "batch-settled" : "settled", carrier, speaksFor, firstAt, lastAt, size, waitMs: 0 };
+}
+
+/**
+ * PURE. Render a batch's messages as the context the acknowledgement is
+ * composed from — OLDEST FIRST, so the line reads as a reply to a
+ * conversation rather than to a list.
+ *
+ * WHAT THIS DELIBERATELY DOES NOT DO: it does not count. "I have four
+ * messages" is a report about the inbox, not a reply to a person, and it is
+ * the exact register the owner rejected. The composer is handed the messages
+ * themselves and nothing else; how the line reads is
+ * `assurance.buildAckUserPrompt`'s to decide.
+ *
+ * @param {object[]} records
+ * @param {object} [o] {perMessageChars, maxMessages}
+ * @returns {{sender:string|null, text:string}[]}
+ */
+export function batchMessages(records, o = {}) {
+  const perMessageChars = Number.isFinite(o.perMessageChars) ? o.perMessageChars : 400;
+  // Six is the most a short acknowledgement can meaningfully address; past
+  // that the oldest are already stale and the line would be a summary.
+  const maxMessages = Number.isFinite(o.maxMessages) ? o.maxMessages : 6;
+  const list = Array.isArray(records) ? records.filter(Boolean) : [];
+  const tail = list.slice(-maxMessages);
+  const out = [];
+  for (const rec of tail) {
+    const item = (rec && rec.item) || {};
+    const text = String(item.content || item.subject || rec.summary || "").trim();
+    if (!text) continue;
+    out.push({ sender: item.sender || rec.sender || null, text: text.slice(0, perMessageChars) });
+  }
+  return out;
+}
+
+export default {
+  DEFAULT_BATCH_QUIET_MS,
+  DEFAULT_BATCH_MAX_SPAN_MS,
+  batchQuietMs,
+  batchMaxSpanMs,
+  conversationKey,
+  isUrgent,
+  arrivalAt,
+  groupIntoBatches,
+  batchVerdict,
+  batchMessages,
+};