@yemi33/minions 0.1.2149 → 0.1.2151

package/docs/worktree-lifecycle.md

@@ -0,0 +1,164 @@
+# Worktree Lifecycle
+
+Deep-dive for the four engine pieces that own git-worktree state for dispatch:
+the **pool** (recycling), the **live guard** (don't wipe an agent's work),
+the **quarantine path** (dirty/divergent → quarantine dir + retry), and the
+**Windows file-lock retry** (EPERM/EBUSY footgun). CLAUDE.md → Worktree
+Lifecycle keeps the cross-cutting invariants; the detail lives here.
+
+> Source of truth: `engine/worktree-pool.js`, `engine/shared.js#removeWorktree`
+> + `_retryFsOp`, `engine.js` (`_quarantineDirtyWorktree`, `_renameWithRetry`,
+> `_killGitDescendantsForWorktree`, `pruneOrphanWorktrees*`, `gcDispatchWorktreeIfOrphan`),
+> `engine/cleanup.js`. Last verified: 2026-06-09.
+
+## Live-worktree guard (W-mq5rwwss000f30a7)
+
+**Invariant:** every code path that wipes, renames, or recycles a worktree
+MUST call `shared.isWorktreePathLive(path, { db?, excludeDispatchId? })`
+first and skip on `true`. Originally added after W-mq5n1zx5000hcfb5 — an
+agent's worktree was wiped 4× consecutively by the reaper while the
+dispatch was still active.
+
+- **Backed by SQL** against `dispatches` (status IN ('pending','active')),
+  reading `json_extract(data, '$.worktreePath')` with a `data.meta.worktreePath`
+  fallback. The pending → active transition persists `item.worktreePath`
+  on the dispatch row so the guard has a path to correlate.
+- **Fails OPEN** (returns `true`) when SQLite is unreachable or the query
+  throws — better to leak a worktree than nuke an agent's unpushed work.
+- **Wired sites:** `shared.removeWorktree` (accepts `{ excludeDispatchId }`
+  forwarded to the guard); pool-return chain in `engine.js`
+  (`excludeDispatchId: id` so the dispatch can clean up its own worktree);
+  dispatch-end orphan GC (`gcDispatchWorktreeIfOrphan`); `_quarantineDirtyWorktree`
+  (returns `{ skipped: true, quarantinedPath: null }` on skip — callers
+  MUST honor `skipped` and not set `quarantined: true`); `engine/cleanup.js`
+  orphan-dir sweep.
+- **On skip:** drops a deduped operator note at
+  `notes/inbox/engine-worktree-skip-live-<basename>-<date>.md`.
+
+## Worktree pool (opt-in)
+
+`ENGINE_DEFAULTS.worktreePoolSize > 0` enables `engine/worktree-pool.js` to
+recycle worktree dirs across branches.
+
+- **Borrow** in `spawnAgent` only when (a) the new branch doesn't exist on
+  origin AND (b) the dispatch is not shared-branch / `useExistingBranch`.
+- **Return** in `onAgentClose` BEFORE `completeDispatch`:
+  `git reset --hard HEAD` → `git clean -fd` → `git fetch origin <main>`
+  → `git checkout --detach origin/<main>` → mark IDLE.
+- **State** at `engine/worktree-pool.json`; git ops outside any lock.
+
+## Quarantine path (dirty / divergent)
+
+When `discoverFromWorkItems` finds a worktree in a `WORKTREE_DIRTY`,
+`WORKTREE_DIVERGENT`, or post-stuck state, `_quarantineDirtyWorktree`
+moves it to `<root>/.quarantined/<basename>-<utc>` and lets the next tick
+recreate a clean worktree. Six layered defenses cover the Windows EBUSY
+race when git status descendants still hold packfile handles
+(W-mq5n1zx5000hcfb5; PR #3156).
+
+### Layer 1b — `--no-optional-locks` on every status probe (load-bearing)
+
+`_statusPorcelainCmd()` emits `git --no-optional-locks status …`. Skips
+the `.git/index.lock` acquire around the untracked-cache refresh inside
+`status`. Typical probe duration under AV scanning drops from 6–12s to
+<500ms, which removes the timeout that leaks the child in the first
+place. **Most incidents are closed by 1b alone.**
+Toggle: `ENGINE_DEFAULTS.statusProbeUseNoOptionalLocks` (default `true`).
+
+### Layer 1a — `_renameWithRetry` with jittered backoff
+
+6 attempts × 250 ms base × 2^N exponential + 200 ms random jitter
+(~16 s worst-case). Only retries on `EBUSY|EPERM|EACCES|ENOTEMPTY`;
+rethrows other codes immediately. Toggles:
+`ENGINE_DEFAULTS.quarantineRenameRetryAttempts` (6),
+`quarantineRenameRetryBaseMs` (250).
+
+### Layer 2a — `_killGitDescendantsForWorktree` (Windows-only)
+
+PowerShell sweep, 2 s timeout. Shell-out built via single-quoted literal
+(`'` escaped as `''`) so worktree-path interpolation is injection-safe.
+POSIX no-op. Best-effort — failure logged, never throws.
+Toggle: `ENGINE_DEFAULTS.statusProbeKillDescendantsWin32` (default `true`).
+
+### Layer 2b — `git worktree remove --force` fallback
+
+Triggers only when 1a exhausts retries; destroys worktree contents (no
+quarantine dir preserved). Toggle:
+`ENGINE_DEFAULTS.quarantineForceRemoveFallback` (default `true`).
+
+### Layer 1c — `FAILURE_CLASS.WORKTREE_QUARANTINE_ENV_BLOCKED`
+
+Routed when `cleanResult.quarantineError && !cleanResult.quarantined`.
+Added to `dispatch.js#isRetryableFailureReason` never-retry set so the
+per-agent retry counter doesn't bump (env failure, not the agent's fault).
+The auto-recovery loop in `engine.js#discoverFromWorkItems` recognizes
+the new class via both enum check and regex on legacy `failReason`
+strings, then re-queues under the existing
+`ENGINE_DEFAULTS.quarantineAutoRecoveryMax` (default 2) cap.
+
+### Layer 3a/3b — metrics + inbox alert
+
+Counters at
+`metrics._engine.worktreeQuarantineOutcomes.{attempts, success, successAfterRetry, fallbackForceRemove, totalFailure}`.
+Total-failure path writes a structured inbox alert with the exact
+PowerShell/POSIX recovery commands and `git worktree prune` instructions.
+
+### Auto-recovery cap
+
+`discoverFromWorkItems` auto-recovers `WORKTREE_DIRTY`/`WORKTREE_DIVERGENT`
+quarantine (#2996) up to `ENGINE_DEFAULTS.quarantineAutoRecoveryMax`
+(default 2, tracked on `_quarantineRecoveryCount`); after the cap,
+`_quarantineRecoveryGaveUp` triggers a warn-once.
+
+## Windows file-lock on removal (footgun #6 detail)
+
+`fs.rmSync` against an active git worktree can lose to lingering file
+handles (CC log streams, virus scanners, Explorer previews).
+`shared.removeWorktree` retries via `shared._retryFsOp`
+(`worktreeRemoveRetryAttempts` × exponential `worktreeRemoveRetryBaseMs`,
+codes `EPERM|EBUSY|EACCES|ENOTEMPTY`).
+
+After `worktreeStuckThreshold` consecutive failures the path is
+**escalated**:
+
+- A `notes/inbox/engine-worktree-stuck-<basename>-<date>.md` note with
+  Windows holder-identification hints is written (deduped per UTC day).
+- Per-tick warns are suppressed for `worktreeStuckSuppressMs`.
+- The retry cadence drops to `worktreeStuckSlowRetryMs`.
+- When the holder finally releases, a
+  `worktree-recovered-<basename>` note clears the alert.
+
+**Don't fight this with `fs.rmSync({force:true})` outside
+`removeWorktree`** — you'll bypass the retry, escalation, and metrics
+layers.
+
+## Periodic prune (W-mq5o6bvy000x7191)
+
+`pruneWorktreesPeriodic` runs `pruneOrphanWorktrees` (in-root) +
+`pruneOrphanWorktreesFromGitRegistry` (out-of-root via
+`git worktree list --porcelain`) per project at
+`ENGINE_DEFAULTS.worktreePruneIntervalTicks` cadence — catches Windows
+EPERM/EBUSY stragglers that the dispatch-end GC couldn't reap and sweeps
+the `git worktree list` registry for OUT-of-root entries the in-root
+scanner is blind to.
+
+## Holder identification + opt-in auto-reap (W-mq6f2fe0000557fa)
+
+Orphan-sweep escalations also run `shared.findProcessesWithCwdInside(wt)`
+(cross-platform: PowerShell `Get-CimInstance Win32_Process` on Windows;
+`/proc/*/cwd` walk on Linux; `lsof -d cwd` + `ps` on macOS) and append a
+`## Live holders` section listing pid / cmdline / age to the
+`engine-worktree-stuck-<basename>` escalation note.
+
+Setting `engine.autoReapOrphanWorktreeHolders: true`
+(Settings → Worker Pool & Worktrees → "Auto-reap orphan worktree holders")
+additionally kills any holder whose `cmdline` matches `spawn-agent.js`
+AND references the worktree basename AND whose age exceeds
+`engine.agentTimeout * 2`, then retries `removeWorktree` once. Default
+**OFF** — killing a foreign process is destructive.
+
+- Scan timeout: `engine.orphanHolderScanTimeoutMs` (default 5000ms,
+  clamped 1000–30000).
+- Motivating incident: W-mq1k8z6o003acd89 / stuck `spawn-agent.js`
+  PID 16828 that the periodic prune couldn't shake loose without manual
+  intervention.

package/engine/cli.js

@@ -163,8 +163,9 @@ function handleCommand(cmd, args) {
     //
     // `minions work --help` used to create ghost work items with title='--help'
     // because the bare-string `title` was truthy and bypassed the `!title`
-    // usage check. Same class of bug exists in `spawn`/`plan`/`complete` —
-    // every command that takes a positional arg and tests it with `if (!arg)`.
+    // usage check. The fix lives in per-command guards (`_isHelpArg` /
+    // `looksLikeFlagOrHelp`) on `work`/`spawn`/`plan`/`complete`, which print
+    // command-specific `Usage:` output. `pr` and `bridge` handle help inline.
     //
     // Intercept here so a single guard covers the whole command set. `pr` and
     // `bridge` already handle `help`/`--help`/`-h` inline (see their own
@@ -1303,15 +1304,17 @@ const commands = {
       console.log(`Engine: ${control.state} (PID ${control.pid || 'N/A'})`);
     }
 
-    // Dashboard check
+    // Dashboard check. Honors MINIONS_PORT so `minions --dev status` probes the
+    // dev dashboard (7332 by default) rather than the binary install's 7331.
     const http = require('http');
+    const dashPort = Number(process.env.MINIONS_PORT) || 7331;
     const dashCheck = new Promise(resolve => {
-      const req = http.get('http://localhost:7331/api/health', { timeout: 2000 }, () => resolve(true));
+      const req = http.get(`http://localhost:${dashPort}/api/health`, { timeout: 2000 }, () => resolve(true));
       req.on('error', () => resolve(false));
       req.on('timeout', () => { req.destroy(); resolve(false); });
     });
     dashCheck.then(dashUp => {
-      if (dashUp) console.log('Dashboard: running (http://localhost:7331)');
+      if (dashUp) console.log(`Dashboard: running (http://localhost:${dashPort})`);
       else console.log('Dashboard: not running — start with: minions dash');
     }).catch(() => {});
 

package/engine/dispatch.js

@@ -18,6 +18,18 @@ const { getConfig, INBOX_DIR } = queries;
 
 const MINIONS_DIR = shared.MINIONS_DIR;
 
+// Dispatch types that do not push code to a PR branch. These remain dispatchable
+// against `_contextOnly: true` PRs because casting a review vote, posting a
+// comment, asking a question, or running a read-only explore never mutates the
+// PR's source branch. Auto-discovery (engine.js#discoverFromPrs) is still
+// gated on `_contextOnly` separately — only explicit WIs (dashboard, watches,
+// CC) can target a context-only PR, and only via these types.
+const NON_MUTATING_DISPATCH_TYPES = new Set([
+  WORK_TYPE.REVIEW,
+  WORK_TYPE.ASK,
+  WORK_TYPE.EXPLORE,
+]);
+
 // Lazy require to break circular dependency with engine.js
 let _lifecycle = null;
 function lifecycle() { if (!_lifecycle) _lifecycle = require('./lifecycle'); return _lifecycle; }
@@ -358,7 +370,19 @@ function getStalePrDispatchReason(entry, config) {
   const prLabel = entry.meta.pr?.id || entry.meta.pr?.url || entry.id;
   if (!tracked) return `PR ${prLabel} is no longer tracked`;
   if (tracked.status !== PR_STATUS.ACTIVE) return `PR ${tracked.id || prLabel} is ${tracked.status || 'missing status'}`;
-  if (tracked._contextOnly && entry.meta?.source !== 'work-item') return `PR ${tracked.id || prLabel} is context-only`;
+  // Combined gate (W-mq5v612m001g1545 + #3126):
+  //   1. `entry.meta?.source !== 'work-item'` — blocks any non-work-item dispatch
+  //      (e.g. legacy `pr`/`pr-human-feedback` discovery paths) from running
+  //      against a context-only PR. Auto-discovery in engine.js#discoverFromPrs
+  //      already skips _contextOnly PRs at the source, so this is defense-in-depth.
+  //   2. `!NON_MUTATING_DISPATCH_TYPES.has(entry.type)` — even an explicit
+  //      work-item dispatch is dropped if it's a mutating type (fix, implement,
+  //      test, verify, decompose, docs). Only review/ask/explore — which never
+  //      push to the PR's source branch — are allowed through.
+  if (tracked._contextOnly
+      && (entry.meta?.source !== 'work-item' || !NON_MUTATING_DISPATCH_TYPES.has(entry.type))) {
+    return `PR ${tracked.id || prLabel} is context-only`;
+  }
 
   const queuedBranch = entry.meta.branch || entry.meta.pr?.branch || '';
   const trackedBranch = tracked.branch || '';
@@ -1048,4 +1072,5 @@ module.exports = {
   findActivePrOrBranchLock,
   normalizeRetryableDecision,
   isCompletedWorkItemForFailure,
+  NON_MUTATING_DISPATCH_TYPES,
 };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@yemi33/minions",
-  "version": "0.1.2149",
+  "version": "0.1.2151",
   "description": "Multi-agent AI dev team that runs from ~/.minions/ — five autonomous agents share a single engine, dashboard, and knowledge base",
   "bin": {
     "minions": "bin/minions.js"
@@ -9,6 +9,11 @@
     "package:check": "node tools/check-package-boundary.js public",
     "package:prepare:public": "node tools/prepare-package.js public",
     "package:prepare:internal": "node tools/prepare-package.js internal",
+    "dev": "node bin/minions.js --dev restart",
+    "dev:start": "node bin/minions.js --dev start",
+    "dev:stop": "node bin/minions.js --dev stop",
+    "dev:dash": "node bin/minions.js --dev dash",
+    "dev:status": "node bin/minions.js --dev status",
     "test": "node test/run-parallel.js",
     "test:sequential": "node test/unit.test.js",
     "test:unit": "node test/run-parallel.js",