@yemi33/minions 0.1.2075 → 0.1.2077

package/docs/cooldown-merge-semantics.md

@@ -0,0 +1,120 @@
+# `saveCooldowns` merge semantics
+
+Scoping deliverable for PRD item **P-bfa3a-scope-cooldown-merge** (Wave 3, PR10
+pre-work) of the 2026-05-27 weekly bug-audit plan. Decides the merge strategy
+the implementation PR (**P-bfa3b-cooldown-lost-update**) will apply at
+`engine/cooldown.js:63-101`. No code is modified by this scoping artefact.
+
+## Context
+
+`engine/cooldown.js:63-101` (`saveCooldowns`) handles only **deletions** across
+the debounced in-memory ↔ on-disk boundary. Any key added to `cooldowns.json`
+by a concurrent writer between two of our writes is **silently overwritten**
+when the in-memory `dispatchCooldowns` Map is flushed via
+`Object.fromEntries(dispatchCooldowns)` at line 93. The 1000 ms debounce widens
+the window.
+
+`mutateCooldowns` already runs through `mutateJsonFileLocked`, which acquires
+an exclusive `withFileLock` for the read-modify-write
+(source: `engine/shared.js:1187-1192` and `:1123-1148`), so the callback
+receives the freshly-read `diskCooldowns` snapshot — but the current code
+throws that snapshot away.
+
+## Chosen strategy: union-with-last-writer-wins, in-memory wins on collision
+
+Inside the `mutateCooldowns(diskCooldowns => …)` callback, start from a copy of
+`diskCooldowns` (the lock-acquired authoritative snapshot — it already contains
+any concurrent writer's added keys), apply the existing deletion-detection step
+unchanged, prune expired (>24 h) entries, then **union** every entry from
+`dispatchCooldowns` on top. For overlapping keys, the in-memory value wins,
+because the in-memory state was mutated within the current debounce window and
+is by construction at least as recent as anything on disk. Return that union
+object (not `Object.fromEntries(dispatchCooldowns)`) from the callback, and
+reset `_lastDiskCooldownKeys` to the keys of the union so the next
+deletion-detection cycle is anchored to what we actually persisted. This is the
+smallest correct fix given the existing lock and per-entry `timestamp` schema,
+requires no schema change, and matches the audit's
+"writer A adds X to disk, writer B's snapshot lacks X → X must survive"
+acceptance from P-bfa3b verbatim.
+
+## Acceptance bullets for the implementation PR (P-bfa3b)
+
+1. **Union semantics.** Inside the `mutateCooldowns` callback, `saveCooldowns`
+   builds the result as `{ …diskCooldowns, …inMemoryEntries }` — every key
+   present in the freshly-read `diskCooldowns` but absent from
+   `dispatchCooldowns` (and not flagged as a deletion per bullet 2) survives
+   the write. The TDD test from P-bfa3b ("writer A persists key X to disk
+   while writer B holds a snapshot lacking X — assert X survives the merge")
+   MUST pass against this semantic, with `dispatchCooldowns.has(X) === true`
+   after the next `loadCooldowns()` on writer B.
+
+2. **Deletion-merge preserved (regression).** The existing
+   `_lastDiskCooldownKeys` logic at `engine/cooldown.js:70-74` runs first,
+   before the union — any key present in `_lastDiskCooldownKeys` (i.e. we
+   wrote it on our previous flush) but absent from the freshly-read
+   `diskCooldowns` is treated as a deliberate delete by another writer and is
+   removed from BOTH the result object AND `dispatchCooldowns` so it does not
+   get re-added by the union step. A regression test asserts
+   `clearCooldown(X) → saveCooldowns()` round-trips through disk and the key
+   stays gone.
+
+3. **`_lastDiskCooldownKeys` anchored to the persisted union.** After the
+   union, the function sets `_lastDiskCooldownKeys = new Set(Object.keys(result))`
+   using the **merged** object, not `dispatchCooldowns`, so the next save's
+   deletion-detection sees exactly what we last wrote — including the
+   concurrent-writer keys that the union absorbed. Expiry-prune (>24 h) and
+   `pendingContexts` cap/truncation continue to run on every entry in the
+   merged result, not just the in-memory subset.
+
+## Trade-offs for the three options NOT chosen
+
+- **(b) per-key timestamp-tagged merge** — The entry schema already carries
+  `timestamp` (source: `engine/cooldown.js:154, 171, 195`), so this would add
+  no new field; but it adds per-key branching in the flush hot path while
+  converging on the same winner as (a) in every realistic case, because the
+  in-memory entry whose mutation triggered the debounce is by construction at
+  least as recent as the disk snapshot we just read under the same lock.
+
+- **(c) in-memory canonical** — This is precisely the current bug — calling it
+  the design simply documents the lost-update rather than fixing it, and it
+  wastes the disk snapshot that `mutateJsonFileLocked` already reads for free
+  inside the lock callback.
+
+- **(d) on-disk authoritative** — Re-reading `cooldowns.json` on every
+  `setCooldown` / `setCooldownFailure` / `isOnCooldown` call adds one fs read
+  to the per-dispatch evaluation hot path, contradicts the module-level
+  `dispatchCooldowns` Map cache that exists specifically to avoid that cost
+  (source: `engine/cooldown.js:39`), and the lock-protected merge in (a)
+  achieves the same correctness without the per-read penalty.
+
+## Implementation notes for the P-bfa3b dispatch (dallas)
+
+- Touch only `engine/cooldown.js:63-101` (`saveCooldowns`) — no API or signature
+  changes, no caller-side migration. Expected diff: ~10–15 LoC inside the
+  existing `mutateCooldowns` callback.
+- The expiry-prune loop and the `pendingContexts` cap/truncate loop must run on
+  the **merged result**, not on `dispatchCooldowns` alone, otherwise a stale
+  concurrent-writer entry inherited from disk could carry a >24 h timestamp or
+  an oversized `pendingContexts` array through the write.
+- TDD test harness: drive through real `setCooldown` calls plus a direct
+  `fs.writeFileSync(COOLDOWN_PATH, …)` between two flushes (flushing the 1000 ms
+  debounce timer via fake timers or by calling the inner write helper directly).
+  Extend the existing cooldown test file under `test/unit/cooldown*.test.js`;
+  do not add a new top-level file.
+- JSDoc paragraph for `saveCooldowns` (P-bfa3b acceptance bullet 4):
+  "Merges the in-memory `dispatchCooldowns` Map ON TOP of the lock-acquired
+  on-disk snapshot rather than replacing it. Concurrent writers' added keys
+  survive; in-memory entries win collisions because they were mutated within
+  the current debounce window. Deletions are detected via a
+  `_lastDiskCooldownKeys` diff before the union so explicit `clearCooldown`
+  calls still persist."
+
+## Source references
+
+- `engine/cooldown.js:63-101` — current lost-update site
+- `engine/cooldown.js:38-60` — `loadCooldowns` + `_lastDiskCooldownKeys` baseline
+- `engine/shared.js:1187-1192` — `mutateCooldowns` (already lock-protected)
+- `engine/shared.js:1123-1148` — `mutateJsonFileLocked` (reads disk inside the
+  lock; `skipWriteIfUnchanged` enabled for cooldowns)
+- `prd/bug-fix-plan-from-weekly-audit-2026-05-27.json` — P-bfa3a (this scoping)
+  and P-bfa3b (implementation acceptance criteria)

package/docs/deprecated.json

@@ -56,6 +56,23 @@
     "targetRemovalDate": null,
     "notes": "Do NOT set targetRemovalDate — gating is signal-based. The function is silent on no-op (returns false without logging), so the meaningful telemetry signal is the absence of the promotion log line over the sweep window, NOT the absence of function invocations (cli.js calls it every boot regardless)."
   },
+  {
+    "id": "sql-state-json-mirrors",
+    "description": "Phase X.5 follow-up to the SQL state migration (commits 62bd6a2c..1111cf54, phases 0–7). Every engine state file that previously used mutateJsonFileLocked now routes through a SQL store, but each store still writes a JSON dual-write mirror after every mutation because a handful of direct-readers (a few unit tests + a couple of inline safeJson calls) have not been migrated to the SQL read path. Once those readers are confirmed routed through the SQL store (or rewritten to use the store's read helper), the mirror writers can be deleted and the JSON files retired.",
+    "code": [
+      { "file": "engine/dispatch-store.js", "note": "_mirrorJsonFromSql + _readDispatchJsonFallback — used when SQL is empty AND JSON has content (test seeding + first-time hydrate)" },
+      { "file": "engine/work-items-store.js", "note": "_mirrorJsonFromSql + _readJsonArrayFallback (per scope) — same fallback contract as dispatch-store" },
+      { "file": "engine/pull-requests-store.js", "note": "_mirrorJsonFromSql + _readJsonArrayFallback (per scope)" },
+      { "file": "engine/logs-store.js", "note": "engine/log.json mirror written by shared._flushLogBuffer's byJsonPath loop — Phase 4.5 will retire" },
+      { "file": "engine/metrics-store.js", "note": "_mirrorJsonFromSql + _readJsonObjectFallback" },
+      { "file": "engine/watches-store.js", "note": "_mirrorJsonFromSql + _readJsonArrayFallback" },
+      { "file": "engine/small-state-store.js", "note": "_mirrorScheduleRunsJson, _mirrorPipelineRunsJson, _mirrorManagedProcessesJson, _mirrorWorktreePoolJson + each store's _readJson fallback path" },
+      { "file": "CLAUDE.md", "lines": "47-66, 240-265", "note": "State Files + Concurrency sections still describe JSON files as the source of truth; they describe a layered SQLite-then-mirror reality in places but the headline contract still reads as JSON-primary. Rewrite these sections to make SQL-as-source-of-truth the headline and the JSON mirrors a transitional compatibility detail." }
+    ],
+    "removalGate": "All direct-readers of the mirror JSON files must be confirmed routed through their respective SQL store's read helper. Specifically: (a) grep the codebase for `safeJson`, `safeJsonArr`, `safeJsonObj`, `readFileSync(...work-items.json|pull-requests.json|metrics.json|watches.json|schedule-runs.json|pipeline-runs.json|managed-processes.json|worktree-pool.json|log.json|dispatch.json...)` and confirm every hit is either (i) a test fixture that can move to the SQL helper, or (ii) intentionally documented as bypassing SQL. (b) Run the full test suite with each store's _mirrorJsonFromSql temporarily neutered (returning early before safeWrite) and confirm 0 failures — that proves no production code path depends on the mirror. Once both conditions hold, removal deletes each store's _mirrorJsonFromSql call site in shared.js (mutateWorkItems/mutatePullRequests/etc.), the corresponding _readJsonArrayFallback paths, and the JSON file gitignore entries. CLAUDE.md update can ship independently as soon as someone has bandwidth.",
+    "targetRemovalDate": null,
+    "notes": "Do NOT set targetRemovalDate — gating is signal-based, not calendar-based. The mirror writes are cheap (a few KB per write, sub-ms) so there is no production cost to keeping them indefinitely; the only reason to remove them is to simplify the codebase and lock in SQL-as-the-single-source-of-truth. Order matters: when retiring a specific store's mirror, retire the corresponding CLAUDE.md mention in the same PR so the docs never claim SQL-only while a mirror still writes."
+  },
   {
     "id": "prune-default-claude-config",
     "description": "pruneDefaultClaudeConfig: active sanitizer that strips generated `config.claude.{binary,outputFormat,allowedTools,permissionMode}` defaults from persisted config.json so the `deprecated-config-claude` warning stops tripping on stale defaults left by older `minions init` versions. Sub-cluster of `config-claude-binary-override` — the prune deliberately preserves non-default user overrides (binary/allowedTools), which is what keeps the override branch in engine/runtimes/claude.js load-bearing.",

package/engine/cooldown.js

@@ -60,6 +60,36 @@ function loadCooldowns() {
 }
 
 let _cooldownWriteTimer = null;
+/**
+ * Persist the in-memory `dispatchCooldowns` Map to disk.
+ *
+ * Merges the in-memory map ON TOP of the lock-acquired on-disk snapshot
+ * rather than replacing it (P-bfa3a → docs/cooldown-merge-semantics.md):
+ *
+ * 1. Concurrent writers' added keys SURVIVE — `saveCooldowns` starts from a
+ *    copy of `diskCooldowns` (the freshly-read snapshot inside the
+ *    `mutateCooldowns` lock) and unions in-memory entries on top, so a key
+ *    another writer persisted between our last load and this flush is no
+ *    longer silently overwritten.
+ * 2. In-memory entries WIN collisions — they were mutated within the current
+ *    debounce window and are by construction at least as recent as the disk
+ *    snapshot we just read under the same lock.
+ * 3. Deletions still PERSIST — explicit `clearCooldown` removes a key from
+ *    `dispatchCooldowns`; the merge step then drops any key that was in
+ *    `_lastDiskCooldownKeys` (i.e. we wrote it on our previous flush) but is
+ *    no longer in `dispatchCooldowns`, so the union doesn't re-add a stale
+ *    disk copy. External deletions are mirrored into memory via the existing
+ *    `_lastDiskCooldownKeys` ∧ `!diskCooldowns[k]` check before the union.
+ * 4. Expiry-prune (>24 h) and `pendingContexts` cap/truncation run on the
+ *    merged result, not just the in-memory subset, so a stale concurrent
+ *    entry inherited from disk cannot slip past the limits.
+ *
+ * Writes are debounced (1 s) so back-to-back mutations coalesce into one
+ * locked write. `_lastDiskCooldownKeys` is anchored to `dispatchCooldowns`
+ * keys after the merge — concurrent-writer keys absorbed via the union are
+ * intentionally NOT tracked as "written by us" so the next save's deletion
+ * heuristic doesn't classify them as our in-memory clears.
+ */
 function saveCooldowns() {
   // Debounce: reset timer on each call so latest state is always written
   if (_cooldownWriteTimer) clearTimeout(_cooldownWriteTimer);
@@ -67,32 +97,55 @@ function saveCooldowns() {
     _cooldownWriteTimer = null;
     try {
       mutateCooldowns((diskCooldowns) => {
+        // 1) Mirror external deletions: keys we previously wrote that a
+        //    concurrent writer has removed from disk drop out of memory too.
         for (const key of Array.from(dispatchCooldowns.keys())) {
           if (_lastDiskCooldownKeys.has(key) && !Object.prototype.hasOwnProperty.call(diskCooldowns, key)) {
             dispatchCooldowns.delete(key);
           }
         }
-        // Prune expired entries (>24h) before saving
-        const now = Date.now();
+        // 2) Start the merged result from the lock-acquired disk snapshot so
+        //    concurrent writers' added keys survive the flush.
+        const merged = { ...diskCooldowns };
+        // 3) Honor explicit in-memory deletes: any key we previously wrote
+        //    that is no longer in `dispatchCooldowns` (e.g. via clearCooldown)
+        //    must be removed from the merged result — otherwise the union
+        //    would resurrect it from the disk snapshot.
+        for (const key of _lastDiskCooldownKeys) {
+          if (!dispatchCooldowns.has(key)) delete merged[key];
+        }
+        // 4) Union in-memory entries on top — in-memory wins on collisions.
         for (const [k, v] of dispatchCooldowns) {
-          if (now - v.timestamp > 24 * 60 * 60 * 1000) dispatchCooldowns.delete(k);
+          merged[k] = v;
         }
-        // Trim pendingContexts arrays before writing to prevent bloat
+        // 5) Apply expiry-prune (>24 h) and pendingContexts cap/truncation
+        //    to EVERY entry in the merged result, including disk-inherited
+        //    entries, so a stale concurrent-writer add can't carry an
+        //    expired timestamp or an oversized payload through the write.
+        const now = Date.now();
         const cap = ENGINE_DEFAULTS.maxPendingContexts;
         const entryLimit = ENGINE_DEFAULTS.maxPendingContextEntryBytes;
-        for (const [, v] of dispatchCooldowns) {
+        for (const [k, v] of Object.entries(merged)) {
+          if (!v || typeof v !== 'object' || typeof v.timestamp !== 'number') continue;
+          if (now - v.timestamp > 24 * 60 * 60 * 1000) {
+            delete merged[k];
+            dispatchCooldowns.delete(k);
+            continue;
+          }
           if (Array.isArray(v.pendingContexts)) {
             if (v.pendingContexts.length > cap) {
               v.pendingContexts = v.pendingContexts.slice(-cap);
             }
-            // Also truncate oversized individual entries — #1167 showed
-            // 20 entries × 25 MB each still produced a 500 MB cooldowns.json.
             v.pendingContexts = v.pendingContexts.map(e => _truncateContextEntry(e, entryLimit));
           }
         }
-        const obj = Object.fromEntries(dispatchCooldowns);
-        _lastDiskCooldownKeys = new Set(Object.keys(obj));
-        return obj;
+        // 6) Anchor _lastDiskCooldownKeys to keys we actively own in memory
+        //    (NOT the full union). Absorbed concurrent-writer keys belong to
+        //    the other writer's lifecycle; tracking them here would make the
+        //    next save's deletion-detection (step 3) misclassify them as our
+        //    in-memory clears.
+        _lastDiskCooldownKeys = new Set(dispatchCooldowns.keys());
+        return merged;
       });
     } catch (err) {
       log('warn', `saveCooldowns failed writing ${COOLDOWN_PATH}: ${err.message}`);

package/engine/meeting.js

@@ -102,7 +102,7 @@ function buildFailedMeetingConclusion(meeting, agents, reason) {
   return `${base}\n\n## Conclusion Failure\n${reason || formatRoundFailuresForConclusion(meeting, 'conclude', meeting.round)}`;
 }
 
-function advanceMeetingIfRoundComplete(meeting, roundName, meetingId, config = null) {
+function advanceMeetingIfRoundComplete(meeting, roundName, meetingId, config = null, pendingInboxWrites = null) {
   if (roundName === 'investigate') {
     if (!allParticipantsFinishedRound(meeting, roundName, meeting.round)) return false;
     meeting.status = 'debating';
@@ -126,13 +126,37 @@ function advanceMeetingIfRoundComplete(meeting, roundName, meetingId, config = n
     meeting.transcript.push({ round: meeting.round, agent: 'system', type: 'conclusion', content: autoConclusion, at: ts() });
     meeting.status = 'completed';
     meeting.completedAt = ts();
-    writeMeetingTranscriptToInbox(meeting, meetingId, agents);
+    // Defer inbox write until AFTER the mutateMeeting lock releases —
+    // writeMeetingTranscriptToInbox hits the filesystem (slug dedup, write)
+    // and must not block other writers. Matches the happy-path pattern at
+    // collectMeetingFindings (concludedMeeting/configForInbox capture).
+    // Callers MUST pass a `pendingInboxWrites` array and drain it after
+    // the lock releases; the inline-write fallback is a defensive no-op
+    // for any future caller that forgets — it logs but still violates
+    // lock discipline, so do not rely on it.
+    if (Array.isArray(pendingInboxWrites)) {
+      pendingInboxWrites.push({ meeting, meetingId, agents });
+    } else {
+      log('warn', `Meeting ${meetingId}: advanceMeetingIfRoundComplete invoked without pendingInboxWrites; writing transcript inside lock (anti-pattern)`);
+      try { writeMeetingTranscriptToInbox(meeting, meetingId, agents); }
+      catch (e) { log('warn', `Meeting ${meetingId} inbox write: ${e.message}`); }
+    }
     log('warn', `Meeting ${meetingId}: conclusion failed — auto-generated fallback conclusion`);
     return true;
   }
   return false;
 }
 
+function drainPendingInboxWrites(pendingInboxWrites) {
+  if (!Array.isArray(pendingInboxWrites) || !pendingInboxWrites.length) return;
+  for (const entry of pendingInboxWrites) {
+    try {
+      writeMeetingTranscriptToInbox(entry.meeting, entry.meetingId, entry.agents || {});
+    } catch (e) { log('warn', `Meeting ${entry.meetingId} inbox write: ${e.message}`); }
+    log('info', `Meeting ${entry.meetingId} completed — transcript written to inbox`);
+  }
+}
+
 function isEmptyMeetingContent(text) {
   const value = String(text || '').trim();
   return !value || EMPTY_OUTPUT_PATTERNS.includes(value);
@@ -641,6 +665,7 @@ function collectMeetingFindings(meetingId, agentId, roundName, output, structure
 
   let concludedMeeting = null;
   let configForInbox = null;
+  const pendingInboxWrites = [];
 
   mutateMeeting(meetingId, (meeting) => {
     if (!meeting) return null; // file missing — nothing to do
@@ -685,7 +710,7 @@ function collectMeetingFindings(meetingId, agentId, roundName, output, structure
         at: ts(),
       });
       log('warn', `Meeting ${meetingId}: agent ${agentId} failed ${roundName} — ${reason}`);
-      advanceMeetingIfRoundComplete(meeting, roundName, meetingId);
+      advanceMeetingIfRoundComplete(meeting, roundName, meetingId, null, pendingInboxWrites);
       return meeting;
     }
 
@@ -707,7 +732,7 @@ function collectMeetingFindings(meetingId, agentId, roundName, output, structure
       return meeting;
     }
 
-    advanceMeetingIfRoundComplete(meeting, roundName, meetingId);
+    advanceMeetingIfRoundComplete(meeting, roundName, meetingId, null, pendingInboxWrites);
     return meeting;
   });
 
@@ -717,6 +742,12 @@ function collectMeetingFindings(meetingId, agentId, roundName, output, structure
     } catch (e) { log('warn', `Meeting ${meetingId} inbox write: ${e.message}`); }
     log('info', `Meeting ${meetingId} completed — transcript written to inbox`);
   }
+
+  // Drain inbox writes deferred by advanceMeetingIfRoundComplete's
+  // fallback-conclusion branch (conclude round failed → auto-generated
+  // conclusion). Runs OUTSIDE the mutateMeeting lock to preserve lock
+  // discipline (matches the happy-path concludedMeeting pattern above).
+  drainPendingInboxWrites(pendingInboxWrites);
 }
 
 function addMeetingNote(meetingId, note) {
@@ -888,6 +919,7 @@ function checkMeetingTimeouts(config) {
     // Re-evaluate the timeout transition under the file lock to avoid lost
     // updates if an agent finalised mid-tick. Helpers (advanceMeetingIfRoundComplete
     // etc.) operate on the locked-and-rehydrated meeting object.
+    const pendingInboxWrites = [];
     mutateMeeting(snapshot.id, (meeting) => {
       if (!meeting) return null;
       if (isTerminalMeetingStatus(meeting.status)) return null;
@@ -916,7 +948,7 @@ function checkMeetingTimeouts(config) {
         if (allParticipantsFinishedRound(meeting, roundName, meeting.round)) {
           log('warn', `Meeting ${meeting.id}: round ${meeting.round} timed out after ${Math.round(liveElapsed / 60000)}min but all participants are terminal — advancing`);
           meeting.transcript.push({ round: meeting.round, agent: 'system', type: 'timeout', content: `Round ${meeting.round} timed out after all participants finished`, at: ts() });
-          advanceMeetingIfRoundComplete(meeting, roundName, meeting.id, config);
+          advanceMeetingIfRoundComplete(meeting, roundName, meeting.id, config, pendingInboxWrites);
           return meeting;
         } else if (liveElapsed >= hardTimeout) {
           const failures = getRoundFailures(meeting, roundName, meeting.round, true);
@@ -928,7 +960,7 @@ function checkMeetingTimeouts(config) {
           }
           log('warn', `Meeting ${meeting.id}: round ${meeting.round} hit hard timeout after ${Math.round(liveElapsed / 60000)}min — marking ${stalled.length}/${totalCount} non-responders as failed and advancing`);
           meeting.transcript.push({ round: meeting.round, agent: 'system', type: 'timeout', content: `Round ${meeting.round} hard timeout — ${stalled.length} non-responder(s) marked failed`, at: ts() });
-          advanceMeetingIfRoundComplete(meeting, roundName, meeting.id, config);
+          advanceMeetingIfRoundComplete(meeting, roundName, meeting.id, config, pendingInboxWrites);
           return meeting;
         } else {
           log('warn', `Meeting ${meeting.id}: round ${meeting.round} timed out after ${Math.round(liveElapsed / 60000)}min — waiting for all participants to finish (${respondedCount}/${totalCount} succeeded)`);
@@ -942,7 +974,7 @@ function checkMeetingTimeouts(config) {
           failures[conclusionAgent] = { reason, content: '', submittedAt: ts() };
           meeting.transcript.push({ round: meeting.round, agent: conclusionAgent, type: 'failure', content: reason, at: ts() });
           log('warn', `Meeting ${meeting.id}: conclusion round hit hard timeout after ${Math.round(liveElapsed / 60000)}min — synthesising fallback conclusion`);
-          advanceMeetingIfRoundComplete(meeting, 'conclude', meeting.id, config);
+          advanceMeetingIfRoundComplete(meeting, 'conclude', meeting.id, config, pendingInboxWrites);
           return meeting;
         } else {
           log('warn', `Meeting ${meeting.id}: conclusion round timed out after ${Math.round(liveElapsed / 60000)}min — waiting for the conclusion agent to finish`);
@@ -951,6 +983,9 @@ function checkMeetingTimeouts(config) {
       }
       return null;
     });
+    // Drain deferred inbox writes OUTSIDE the meeting lock (preserves
+    // lock discipline — writeToInbox hits the filesystem).
+    drainPendingInboxWrites(pendingInboxWrites);
   }
 }
 module.exports = {

package/engine/pipeline.js

@@ -357,7 +357,7 @@ async function executeStage(stage, run, pipeline, config) {
     case STAGE_TYPE.PLAN:
       return executePlanStage(resolved, stageState, run, config, pipeline);
     case STAGE_TYPE.API:
-      return executeApiStage(resolved, stageState, run);
+      return await executeApiStage(resolved, stageState, run);
     case STAGE_TYPE.MERGE_PRS:
       return executeMergePrsStage(resolved, stageState, run, config);
     case STAGE_TYPE.SCHEDULE:
@@ -708,14 +708,27 @@ async function executePlanStage(stage, stageState, run, config, pipeline = {}) {
   };
 }
 
-function executeApiStage(stage, stageState, run) {
+// P-bfa1e-pipeline-state-machine-b — async + timeout-aware.
+// Each call is awaited end-to-end so the stage result reflects the real
+// outcome (previously the stage returned COMPLETED while requests were
+// still in flight, silently hiding API failures). On per-attempt timeout
+// the request is destroyed; the timeout flows through the existing retry
+// path. After `pipelineApiRetries` attempts exhaust on any call — or any
+// attempt times out past the retry budget — the stage returns FAILED with
+// `<endpoint>: <reason>` so updateRunStage records a terminal failure.
+async function executeApiStage(stage, stageState, run) {
   const calls = stage.calls || [{ endpoint: stage.endpoint, method: stage.method || 'POST', body: stage.body }];
+  const maxAttempts = ENGINE_DEFAULTS.pipelineApiRetries;
+  const retryDelay = ENGINE_DEFAULTS.pipelineApiRetryDelay;
+  const timeoutMs = ENGINE_DEFAULTS.pipelineApiTimeoutMs;
+
   for (const call of calls) {
     const url = `http://localhost:${process.env.MINIONS_PORT || 7331}${call.endpoint}`;
     const body = typeof call.body === 'string' ? call.body : JSON.stringify(call.body || {});
-    const maxAttempts = ENGINE_DEFAULTS.pipelineApiRetries;
-    const retryDelay = ENGINE_DEFAULTS.pipelineApiRetryDelay;
-    const makeRequest = (attempt) => {
+
+    const attemptOnce = (attempt) => new Promise((resolve) => {
+      let settled = false;
+      const finish = (result) => { if (!settled) { settled = true; resolve(result); } };
       try {
         const parsed = new URL(url);
         const req = http.request({
@@ -724,23 +737,45 @@ function executeApiStage(stage, stageState, run) {
           headers: { 'Content-Type': 'application/json' },
         }, (res) => {
           res.resume(); // drain body to free socket
-          if (res.statusCode >= 400) {
-            log('warn', `Pipeline API call to ${call.endpoint} returned ${res.statusCode} (attempt ${attempt})`);
-            if (attempt < maxAttempts) setTimeout(() => makeRequest(attempt + 1), retryDelay);
-          }
+          res.on('end', () => {
+            if (res.statusCode >= 400) {
+              log('warn', `Pipeline API call to ${call.endpoint} returned ${res.statusCode} (attempt ${attempt})`);
+              finish({ ok: false, reason: `HTTP ${res.statusCode}` });
+            } else {
+              finish({ ok: true });
+            }
+          });
+          res.on('error', (err) => finish({ ok: false, reason: err.message }));
+        });
+        req.setTimeout(timeoutMs, () => {
+          req.destroy(new Error('timeout'));
         });
         req.on('error', (err) => {
-          log('warn', `Pipeline API call to ${call.endpoint} failed: ${err.message} (attempt ${attempt})`);
-          if (attempt < maxAttempts) setTimeout(() => makeRequest(attempt + 1), retryDelay);
+          const reason = err && err.message ? err.message : (err && err.code) || 'request error';
+          log('warn', `Pipeline API call to ${call.endpoint} failed: ${reason} (attempt ${attempt})`);
+          finish({ ok: false, reason });
         });
         req.write(body);
         req.end();
       } catch (e) {
         log('warn', `Pipeline API call to ${call.endpoint} threw: ${e.message} (attempt ${attempt})`);
-        if (attempt < maxAttempts) setTimeout(() => makeRequest(attempt + 1), retryDelay);
+        finish({ ok: false, reason: e.message });
+      }
+    });
+
+    let lastReason = 'unknown';
+    let succeeded = false;
+    for (let attempt = 1; attempt <= maxAttempts; attempt++) {
+      const result = await attemptOnce(attempt);
+      if (result.ok) { succeeded = true; break; }
+      lastReason = result.reason;
+      if (attempt < maxAttempts && retryDelay > 0) {
+        await new Promise((r) => setTimeout(r, retryDelay));
       }
-    };
-    makeRequest(1);
+    }
+    if (!succeeded) {
+      return { status: PIPELINE_STATUS.FAILED, error: `${call.endpoint}: ${lastReason}`, completedAt: ts() };
+    }
   }
   return { status: PIPELINE_STATUS.COMPLETED, completedAt: ts() };
 }
@@ -1165,7 +1200,7 @@ module.exports = {
   getPipelineRuns, getActiveRun, startRun, updateRunStage, completeRun,
   discoverPipelineWork,
   evaluateCondition, // exported for testing
-  executeTaskStage, executePlanStage, executeScheduleStage, isStageComplete, resolveTemplate, // exported for testing
+  executeTaskStage, executePlanStage, executeScheduleStage, executeApiStage, isStageComplete, resolveTemplate, // exported for testing
   _resolvePipelineProjects, // exported for testing
   _findMeetingsInRun, _findExistingPlanForMeeting, _findExistingPrdForPlan, // exported for testing
 };

package/engine/shared.js

@@ -1923,6 +1923,7 @@ const ENGINE_DEFAULTS = {
   minRetryGapMs: 120000, // 2min — minimum gap between retry dispatches for the same work item; prevents tight retry loops when an idempotent agent (e.g. review bailing out on a duplicate) cannot produce the expected output (#1770)
   pipelineApiRetries: 2, // max attempts for pipeline API calls
   pipelineApiRetryDelay: 2000, // ms delay between pipeline API retries
+  pipelineApiTimeoutMs: 30000, // P-bfa1e-pipeline-state-machine-b — per-attempt request timeout for pipeline API calls; on timeout the request is destroyed and the attempt fails through the normal retry path. After all retries exhaust, executeApiStage returns FAILED instead of COMPLETED.
   prAutoLinkRetries: 3, // max attempts for gh pr list lookup when auto-linking PR after merge (3s backoff between attempts)
   rebaseQueueRetries: 3, // max rebase attempts per queued PR before giving up
   versionCheckInterval: 3600000, // 1 hour — how often to check npm for updates (ms)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@yemi33/minions",
-  "version": "0.1.2075",
+  "version": "0.1.2077",
   "description": "Multi-agent AI dev team that runs from ~/.minions/ — five autonomous agents share a single engine, dashboard, and knowledge base",
   "bin": {
     "minions": "bin/minions.js"