@yemi33/minions 0.1.2076 → 0.1.2077

package/docs/cooldown-merge-semantics.md

@@ -0,0 +1,120 @@
+# `saveCooldowns` merge semantics
+
+Scoping deliverable for PRD item **P-bfa3a-scope-cooldown-merge** (Wave 3, PR10
+pre-work) of the 2026-05-27 weekly bug-audit plan. Decides the merge strategy
+the implementation PR (**P-bfa3b-cooldown-lost-update**) will apply at
+`engine/cooldown.js:63-101`. No code is modified by this scoping artefact.
+
+## Context
+
+`engine/cooldown.js:63-101` (`saveCooldowns`) handles only **deletions** across
+the debounced in-memory ↔ on-disk boundary. Any key added to `cooldowns.json`
+by a concurrent writer between two of our writes is **silently overwritten**
+when the in-memory `dispatchCooldowns` Map is flushed via
+`Object.fromEntries(dispatchCooldowns)` at line 93. The 1000 ms debounce widens
+the window.
+
+`mutateCooldowns` already runs through `mutateJsonFileLocked`, which acquires
+an exclusive `withFileLock` for the read-modify-write
+(source: `engine/shared.js:1187-1192` and `:1123-1148`), so the callback
+receives the freshly-read `diskCooldowns` snapshot — but the current code
+throws that snapshot away.
+
+## Chosen strategy: union-with-last-writer-wins, in-memory wins on collision
+
+Inside the `mutateCooldowns(diskCooldowns => …)` callback, start from a copy of
+`diskCooldowns` (the lock-acquired authoritative snapshot — it already contains
+any concurrent writer's added keys), apply the existing deletion-detection step
+unchanged, prune expired (>24 h) entries, then **union** every entry from
+`dispatchCooldowns` on top. For overlapping keys, the in-memory value wins,
+because the in-memory state was mutated within the current debounce window and
+is by construction at least as recent as anything on disk. Return that union
+object (not `Object.fromEntries(dispatchCooldowns)`) from the callback, and
+reset `_lastDiskCooldownKeys` to the keys of the union so the next
+deletion-detection cycle is anchored to what we actually persisted. This is the
+smallest correct fix given the existing lock and per-entry `timestamp` schema,
+requires no schema change, and matches the audit's
+"writer A adds X to disk, writer B's snapshot lacks X → X must survive"
+acceptance from P-bfa3b verbatim.
+
+## Acceptance bullets for the implementation PR (P-bfa3b)
+
+1. **Union semantics.** Inside the `mutateCooldowns` callback, `saveCooldowns`
+   builds the result as `{ …diskCooldowns, …inMemoryEntries }` — every key
+   present in the freshly-read `diskCooldowns` but absent from
+   `dispatchCooldowns` (and not flagged as a deletion per bullet 2) survives
+   the write. The TDD test from P-bfa3b ("writer A persists key X to disk
+   while writer B holds a snapshot lacking X — assert X survives the merge")
+   MUST pass against this semantic, with `dispatchCooldowns.has(X) === true`
+   after the next `loadCooldowns()` on writer B.
+
+2. **Deletion-merge preserved (regression).** The existing
+   `_lastDiskCooldownKeys` logic at `engine/cooldown.js:70-74` runs first,
+   before the union — any key present in `_lastDiskCooldownKeys` (i.e. we
+   wrote it on our previous flush) but absent from the freshly-read
+   `diskCooldowns` is treated as a deliberate delete by another writer and is
+   removed from BOTH the result object AND `dispatchCooldowns` so it does not
+   get re-added by the union step. A regression test asserts
+   `clearCooldown(X) → saveCooldowns()` round-trips through disk and the key
+   stays gone.
+
+3. **`_lastDiskCooldownKeys` anchored to the persisted union.** After the
+   union, the function sets `_lastDiskCooldownKeys = new Set(Object.keys(result))`
+   using the **merged** object, not `dispatchCooldowns`, so the next save's
+   deletion-detection sees exactly what we last wrote — including the
+   concurrent-writer keys that the union absorbed. Expiry-prune (>24 h) and
+   `pendingContexts` cap/truncation continue to run on every entry in the
+   merged result, not just the in-memory subset.
+
+## Trade-offs for the three options NOT chosen
+
+- **(b) per-key timestamp-tagged merge** — The entry schema already carries
+  `timestamp` (source: `engine/cooldown.js:154, 171, 195`), so this would add
+  no new field; but it adds per-key branching in the flush hot path while
+  converging on the same winner as (a) in every realistic case, because the
+  in-memory entry whose mutation triggered the debounce is by construction at
+  least as recent as the disk snapshot we just read under the same lock.
+
+- **(c) in-memory canonical** — This is precisely the current bug — calling it
+  the design simply documents the lost-update rather than fixing it, and it
+  wastes the disk snapshot that `mutateJsonFileLocked` already reads for free
+  inside the lock callback.
+
+- **(d) on-disk authoritative** — Re-reading `cooldowns.json` on every
+  `setCooldown` / `setCooldownFailure` / `isOnCooldown` call adds one fs read
+  to the per-dispatch evaluation hot path, contradicts the module-level
+  `dispatchCooldowns` Map cache that exists specifically to avoid that cost
+  (source: `engine/cooldown.js:39`), and the lock-protected merge in (a)
+  achieves the same correctness without the per-read penalty.
+
+## Implementation notes for the P-bfa3b dispatch (dallas)
+
+- Touch only `engine/cooldown.js:63-101` (`saveCooldowns`) — no API or signature
+  changes, no caller-side migration. Expected diff: ~10–15 LoC inside the
+  existing `mutateCooldowns` callback.
+- The expiry-prune loop and the `pendingContexts` cap/truncate loop must run on
+  the **merged result**, not on `dispatchCooldowns` alone, otherwise a stale
+  concurrent-writer entry inherited from disk could carry a >24 h timestamp or
+  an oversized `pendingContexts` array through the write.
+- TDD test harness: drive through real `setCooldown` calls plus a direct
+  `fs.writeFileSync(COOLDOWN_PATH, …)` between two flushes (flushing the 1000 ms
+  debounce timer via fake timers or by calling the inner write helper directly).
+  Extend the existing cooldown test file under `test/unit/cooldown*.test.js`;
+  do not add a new top-level file.
+- JSDoc paragraph for `saveCooldowns` (P-bfa3b acceptance bullet 4):
+  "Merges the in-memory `dispatchCooldowns` Map ON TOP of the lock-acquired
+  on-disk snapshot rather than replacing it. Concurrent writers' added keys
+  survive; in-memory entries win collisions because they were mutated within
+  the current debounce window. Deletions are detected via a
+  `_lastDiskCooldownKeys` diff before the union so explicit `clearCooldown`
+  calls still persist."
+
+## Source references
+
+- `engine/cooldown.js:63-101` — current lost-update site
+- `engine/cooldown.js:38-60` — `loadCooldowns` + `_lastDiskCooldownKeys` baseline
+- `engine/shared.js:1187-1192` — `mutateCooldowns` (already lock-protected)
+- `engine/shared.js:1123-1148` — `mutateJsonFileLocked` (reads disk inside the
+  lock; `skipWriteIfUnchanged` enabled for cooldowns)
+- `prd/bug-fix-plan-from-weekly-audit-2026-05-27.json` — P-bfa3a (this scoping)
+  and P-bfa3b (implementation acceptance criteria)

package/engine/cooldown.js

@@ -60,6 +60,36 @@ function loadCooldowns() {
 }
 
 let _cooldownWriteTimer = null;
+/**
+ * Persist the in-memory `dispatchCooldowns` Map to disk.
+ *
+ * Merges the in-memory map ON TOP of the lock-acquired on-disk snapshot
+ * rather than replacing it (P-bfa3a → docs/cooldown-merge-semantics.md):
+ *
+ * 1. Concurrent writers' added keys SURVIVE — `saveCooldowns` starts from a
+ *    copy of `diskCooldowns` (the freshly-read snapshot inside the
+ *    `mutateCooldowns` lock) and unions in-memory entries on top, so a key
+ *    another writer persisted between our last load and this flush is no
+ *    longer silently overwritten.
+ * 2. In-memory entries WIN collisions — they were mutated within the current
+ *    debounce window and are by construction at least as recent as the disk
+ *    snapshot we just read under the same lock.
+ * 3. Deletions still PERSIST — explicit `clearCooldown` removes a key from
+ *    `dispatchCooldowns`; the merge step then drops any key that was in
+ *    `_lastDiskCooldownKeys` (i.e. we wrote it on our previous flush) but is
+ *    no longer in `dispatchCooldowns`, so the union doesn't re-add a stale
+ *    disk copy. External deletions are mirrored into memory via the existing
+ *    `_lastDiskCooldownKeys` ∧ `!diskCooldowns[k]` check before the union.
+ * 4. Expiry-prune (>24 h) and `pendingContexts` cap/truncation run on the
+ *    merged result, not just the in-memory subset, so a stale concurrent
+ *    entry inherited from disk cannot slip past the limits.
+ *
+ * Writes are debounced (1 s) so back-to-back mutations coalesce into one
+ * locked write. `_lastDiskCooldownKeys` is anchored to `dispatchCooldowns`
+ * keys after the merge — concurrent-writer keys absorbed via the union are
+ * intentionally NOT tracked as "written by us" so the next save's deletion
+ * heuristic doesn't classify them as our in-memory clears.
+ */
 function saveCooldowns() {
   // Debounce: reset timer on each call so latest state is always written
   if (_cooldownWriteTimer) clearTimeout(_cooldownWriteTimer);
@@ -67,32 +97,55 @@ function saveCooldowns() {
     _cooldownWriteTimer = null;
     try {
       mutateCooldowns((diskCooldowns) => {
+        // 1) Mirror external deletions: keys we previously wrote that a
+        //    concurrent writer has removed from disk drop out of memory too.
         for (const key of Array.from(dispatchCooldowns.keys())) {
           if (_lastDiskCooldownKeys.has(key) && !Object.prototype.hasOwnProperty.call(diskCooldowns, key)) {
             dispatchCooldowns.delete(key);
           }
         }
-        // Prune expired entries (>24h) before saving
-        const now = Date.now();
+        // 2) Start the merged result from the lock-acquired disk snapshot so
+        //    concurrent writers' added keys survive the flush.
+        const merged = { ...diskCooldowns };
+        // 3) Honor explicit in-memory deletes: any key we previously wrote
+        //    that is no longer in `dispatchCooldowns` (e.g. via clearCooldown)
+        //    must be removed from the merged result — otherwise the union
+        //    would resurrect it from the disk snapshot.
+        for (const key of _lastDiskCooldownKeys) {
+          if (!dispatchCooldowns.has(key)) delete merged[key];
+        }
+        // 4) Union in-memory entries on top — in-memory wins on collisions.
         for (const [k, v] of dispatchCooldowns) {
-          if (now - v.timestamp > 24 * 60 * 60 * 1000) dispatchCooldowns.delete(k);
+          merged[k] = v;
         }
-        // Trim pendingContexts arrays before writing to prevent bloat
+        // 5) Apply expiry-prune (>24 h) and pendingContexts cap/truncation
+        //    to EVERY entry in the merged result, including disk-inherited
+        //    entries, so a stale concurrent-writer add can't carry an
+        //    expired timestamp or an oversized payload through the write.
+        const now = Date.now();
         const cap = ENGINE_DEFAULTS.maxPendingContexts;
         const entryLimit = ENGINE_DEFAULTS.maxPendingContextEntryBytes;
-        for (const [, v] of dispatchCooldowns) {
+        for (const [k, v] of Object.entries(merged)) {
+          if (!v || typeof v !== 'object' || typeof v.timestamp !== 'number') continue;
+          if (now - v.timestamp > 24 * 60 * 60 * 1000) {
+            delete merged[k];
+            dispatchCooldowns.delete(k);
+            continue;
+          }
           if (Array.isArray(v.pendingContexts)) {
             if (v.pendingContexts.length > cap) {
               v.pendingContexts = v.pendingContexts.slice(-cap);
             }
-            // Also truncate oversized individual entries — #1167 showed
-            // 20 entries × 25 MB each still produced a 500 MB cooldowns.json.
             v.pendingContexts = v.pendingContexts.map(e => _truncateContextEntry(e, entryLimit));
           }
         }
-        const obj = Object.fromEntries(dispatchCooldowns);
-        _lastDiskCooldownKeys = new Set(Object.keys(obj));
-        return obj;
+        // 6) Anchor _lastDiskCooldownKeys to keys we actively own in memory
+        //    (NOT the full union). Absorbed concurrent-writer keys belong to
+        //    the other writer's lifecycle; tracking them here would make the
+        //    next save's deletion-detection (step 3) misclassify them as our
+        //    in-memory clears.
+        _lastDiskCooldownKeys = new Set(dispatchCooldowns.keys());
+        return merged;
       });
     } catch (err) {
       log('warn', `saveCooldowns failed writing ${COOLDOWN_PATH}: ${err.message}`);

package/engine/meeting.js

@@ -102,7 +102,7 @@ function buildFailedMeetingConclusion(meeting, agents, reason) {
   return `${base}\n\n## Conclusion Failure\n${reason || formatRoundFailuresForConclusion(meeting, 'conclude', meeting.round)}`;
 }
 
-function advanceMeetingIfRoundComplete(meeting, roundName, meetingId, config = null) {
+function advanceMeetingIfRoundComplete(meeting, roundName, meetingId, config = null, pendingInboxWrites = null) {
   if (roundName === 'investigate') {
     if (!allParticipantsFinishedRound(meeting, roundName, meeting.round)) return false;
     meeting.status = 'debating';
@@ -126,13 +126,37 @@ function advanceMeetingIfRoundComplete(meeting, roundName, meetingId, config = n
     meeting.transcript.push({ round: meeting.round, agent: 'system', type: 'conclusion', content: autoConclusion, at: ts() });
     meeting.status = 'completed';
     meeting.completedAt = ts();
-    writeMeetingTranscriptToInbox(meeting, meetingId, agents);
+    // Defer inbox write until AFTER the mutateMeeting lock releases —
+    // writeMeetingTranscriptToInbox hits the filesystem (slug dedup, write)
+    // and must not block other writers. Matches the happy-path pattern at
+    // collectMeetingFindings (concludedMeeting/configForInbox capture).
+    // Callers MUST pass a `pendingInboxWrites` array and drain it after
+    // the lock releases; the inline-write fallback is a defensive no-op
+    // for any future caller that forgets — it logs but still violates
+    // lock discipline, so do not rely on it.
+    if (Array.isArray(pendingInboxWrites)) {
+      pendingInboxWrites.push({ meeting, meetingId, agents });
+    } else {
+      log('warn', `Meeting ${meetingId}: advanceMeetingIfRoundComplete invoked without pendingInboxWrites; writing transcript inside lock (anti-pattern)`);
+      try { writeMeetingTranscriptToInbox(meeting, meetingId, agents); }
+      catch (e) { log('warn', `Meeting ${meetingId} inbox write: ${e.message}`); }
+    }
     log('warn', `Meeting ${meetingId}: conclusion failed — auto-generated fallback conclusion`);
     return true;
   }
   return false;
 }
 
+function drainPendingInboxWrites(pendingInboxWrites) {
+  if (!Array.isArray(pendingInboxWrites) || !pendingInboxWrites.length) return;
+  for (const entry of pendingInboxWrites) {
+    try {
+      writeMeetingTranscriptToInbox(entry.meeting, entry.meetingId, entry.agents || {});
+    } catch (e) { log('warn', `Meeting ${entry.meetingId} inbox write: ${e.message}`); }
+    log('info', `Meeting ${entry.meetingId} completed — transcript written to inbox`);
+  }
+}
+
 function isEmptyMeetingContent(text) {
   const value = String(text || '').trim();
   return !value || EMPTY_OUTPUT_PATTERNS.includes(value);
@@ -641,6 +665,7 @@ function collectMeetingFindings(meetingId, agentId, roundName, output, structure
 
   let concludedMeeting = null;
   let configForInbox = null;
+  const pendingInboxWrites = [];
 
   mutateMeeting(meetingId, (meeting) => {
     if (!meeting) return null; // file missing — nothing to do
@@ -685,7 +710,7 @@ function collectMeetingFindings(meetingId, agentId, roundName, output, structure
         at: ts(),
       });
       log('warn', `Meeting ${meetingId}: agent ${agentId} failed ${roundName} — ${reason}`);
-      advanceMeetingIfRoundComplete(meeting, roundName, meetingId);
+      advanceMeetingIfRoundComplete(meeting, roundName, meetingId, null, pendingInboxWrites);
       return meeting;
     }
 
@@ -707,7 +732,7 @@ function collectMeetingFindings(meetingId, agentId, roundName, output, structure
       return meeting;
     }
 
-    advanceMeetingIfRoundComplete(meeting, roundName, meetingId);
+    advanceMeetingIfRoundComplete(meeting, roundName, meetingId, null, pendingInboxWrites);
     return meeting;
   });
 
@@ -717,6 +742,12 @@ function collectMeetingFindings(meetingId, agentId, roundName, output, structure
     } catch (e) { log('warn', `Meeting ${meetingId} inbox write: ${e.message}`); }
     log('info', `Meeting ${meetingId} completed — transcript written to inbox`);
   }
+
+  // Drain inbox writes deferred by advanceMeetingIfRoundComplete's
+  // fallback-conclusion branch (conclude round failed → auto-generated
+  // conclusion). Runs OUTSIDE the mutateMeeting lock to preserve lock
+  // discipline (matches the happy-path concludedMeeting pattern above).
+  drainPendingInboxWrites(pendingInboxWrites);
 }
 
 function addMeetingNote(meetingId, note) {
@@ -888,6 +919,7 @@ function checkMeetingTimeouts(config) {
     // Re-evaluate the timeout transition under the file lock to avoid lost
     // updates if an agent finalised mid-tick. Helpers (advanceMeetingIfRoundComplete
     // etc.) operate on the locked-and-rehydrated meeting object.
+    const pendingInboxWrites = [];
     mutateMeeting(snapshot.id, (meeting) => {
       if (!meeting) return null;
       if (isTerminalMeetingStatus(meeting.status)) return null;
@@ -916,7 +948,7 @@ function checkMeetingTimeouts(config) {
         if (allParticipantsFinishedRound(meeting, roundName, meeting.round)) {
           log('warn', `Meeting ${meeting.id}: round ${meeting.round} timed out after ${Math.round(liveElapsed / 60000)}min but all participants are terminal — advancing`);
           meeting.transcript.push({ round: meeting.round, agent: 'system', type: 'timeout', content: `Round ${meeting.round} timed out after all participants finished`, at: ts() });
-          advanceMeetingIfRoundComplete(meeting, roundName, meeting.id, config);
+          advanceMeetingIfRoundComplete(meeting, roundName, meeting.id, config, pendingInboxWrites);
           return meeting;
         } else if (liveElapsed >= hardTimeout) {
           const failures = getRoundFailures(meeting, roundName, meeting.round, true);
@@ -928,7 +960,7 @@ function checkMeetingTimeouts(config) {
           }
           log('warn', `Meeting ${meeting.id}: round ${meeting.round} hit hard timeout after ${Math.round(liveElapsed / 60000)}min — marking ${stalled.length}/${totalCount} non-responders as failed and advancing`);
           meeting.transcript.push({ round: meeting.round, agent: 'system', type: 'timeout', content: `Round ${meeting.round} hard timeout — ${stalled.length} non-responder(s) marked failed`, at: ts() });
-          advanceMeetingIfRoundComplete(meeting, roundName, meeting.id, config);
+          advanceMeetingIfRoundComplete(meeting, roundName, meeting.id, config, pendingInboxWrites);
           return meeting;
         } else {
           log('warn', `Meeting ${meeting.id}: round ${meeting.round} timed out after ${Math.round(liveElapsed / 60000)}min — waiting for all participants to finish (${respondedCount}/${totalCount} succeeded)`);
@@ -942,7 +974,7 @@ function checkMeetingTimeouts(config) {
           failures[conclusionAgent] = { reason, content: '', submittedAt: ts() };
           meeting.transcript.push({ round: meeting.round, agent: conclusionAgent, type: 'failure', content: reason, at: ts() });
           log('warn', `Meeting ${meeting.id}: conclusion round hit hard timeout after ${Math.round(liveElapsed / 60000)}min — synthesising fallback conclusion`);
-          advanceMeetingIfRoundComplete(meeting, 'conclude', meeting.id, config);
+          advanceMeetingIfRoundComplete(meeting, 'conclude', meeting.id, config, pendingInboxWrites);
           return meeting;
         } else {
           log('warn', `Meeting ${meeting.id}: conclusion round timed out after ${Math.round(liveElapsed / 60000)}min — waiting for the conclusion agent to finish`);
@@ -951,6 +983,9 @@ function checkMeetingTimeouts(config) {
       }
       return null;
     });
+    // Drain deferred inbox writes OUTSIDE the meeting lock (preserves
+    // lock discipline — writeToInbox hits the filesystem).
+    drainPendingInboxWrites(pendingInboxWrites);
   }
 }
 module.exports = {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@yemi33/minions",
-  "version": "0.1.2076",
+  "version": "0.1.2077",
   "description": "Multi-agent AI dev team that runs from ~/.minions/ — five autonomous agents share a single engine, dashboard, and knowledge base",
   "bin": {
     "minions": "bin/minions.js"