@yemi33/minions 0.1.2066 → 0.1.2068

package/README.md

@@ -11,11 +11,11 @@ Inspired by and initially scaffolded from [Brady Gaster's Squad](https://bradyga
 ## Prerequisites
 
 - **Node.js** 18+ (LTS recommended)
-- **A supported runtime CLI** — Minions defaults to GitHub Copilot CLI (`npm install -g @github/copilot`). Claude Code CLI (`npm install -g @anthropic-ai/claude-code`) is also supported; switch with `minions config set-cli claude` or per-agent `cli` overrides.
-- **Auth for your runtime** — GitHub Copilot subscription (Copilot CLI handles its own auth) or an Anthropic API key / Claude Max subscription
+- **A supported runtime CLI** — Minions defaults to GitHub Copilot CLI (`npm install -g @github/copilot`). Claude Code CLI (`npm install -g @anthropic-ai/claude-code`) and OpenAI Codex CLI (`npm install -g @openai/codex` or `brew install --cask codex`) are also supported; switch with `minions config set-cli claude`, `minions config set-cli codex`, or per-agent `cli` overrides.
+- **Auth for your runtime** — GitHub Copilot subscription (Copilot CLI handles its own auth), OpenAI Codex login / API key, or an Anthropic API key / Claude Max subscription
 - **Git** — agents create worktrees for all code changes
 
-> **Note:** you do **not** need to configure your CLI for "autopilot" / "bypass permissions" / "dangerous mode". Minions passes the right bypass flag per spawn (`--dangerously-skip-permissions` for Claude; `--autopilot --allow-all --no-ask-user` for Copilot), independent of your global CLI config. Run `minions doctor` to verify your installed CLI accepts those flags.
+> **Note:** you do **not** need to configure your CLI for "autopilot" / "bypass permissions" / "dangerous mode". Minions passes the right runtime-owned flag per spawn (`--dangerously-skip-permissions` for Claude; `--autopilot --allow-all --no-ask-user` for Copilot; `codex exec --sandbox workspace-write --json -` for Codex), independent of your global CLI config. Run `minions doctor` to verify your installed CLI accepts those flags.
 
 ## Installation
 

package/docs/runtime-adapters.md

@@ -11,6 +11,7 @@ behavior is hidden behind an adapter object resolved through `resolveRuntime()`.
 | `engine/runtimes/index.js` | Adapter registry. `resolveRuntime(name)`, `listRuntimes()`, `registerRuntime()`. Engine code MUST go through these. |
 | `engine/runtimes/claude.js` | Claude Code adapter. Owns binary probe, `--system-prompt-file`, JSONL parser, model shorthands, budget cap, bare mode. |
 | `engine/runtimes/copilot.js` | GitHub Copilot CLI adapter. Owns standalone-vs-`gh-copilot` resolution, stdin-only prompt delivery, `https://api.githubcopilot.com/models` discovery, effort `'max' → 'xhigh'` mapping. |
+| `engine/runtimes/codex.js` | OpenAI Codex CLI adapter. Owns `@openai/codex`/native binary resolution, `codex exec --json -` prompt delivery, `codex debug models --bundled` discovery, `.agents/skills` roots, and Codex JSONL parsing. |
 
 `resolveRuntime(name)` throws when `name` is unknown so misconfigurations surface
 at dispatch time instead of producing silent fallbacks deep inside spawn logic.
@@ -62,24 +63,24 @@ this table is the human-readable mirror.
 Engine code branches on flags, never on runtime names. Add a flag whenever a
 real behavioral split appears between adapters.
 
-| Flag | Claude | Copilot | Gates |
-|------|--------|---------|-------|
-| `streaming` | ✓ | ✓ | JSONL events on stdout. |
-| `sessionResume` | ✓ | ✓ | `--resume <id>`. |
-| `midRunSessionId` | ✓ | ✗ | Resumable session ID emitted before the terminal `result` event; when false, steering waits for a checkpoint. |
-| `systemPromptFile` | ✓ | ✗ | sysprompt via `--system-prompt-file` (else inlined into stdin by `buildPrompt`). |
-| `effortLevels` | ✓ | ✓ | `--effort low\|medium\|high\|xhigh`. Copilot maps `'max' → 'xhigh'` inside `resolveModel`/`buildArgs`; Claude leaves `'max'` alone. |
-| `costTracking` | ✓ | ✗ | USD + token counts in the result event. Copilot only emits `premiumRequests`. |
-| `modelShorthands` | ✓ | ✗ | Bare `sonnet`/`opus`/`haiku` accepted. Copilot expects full model IDs (`claude-sonnet-4.5`, `gpt-5.4`). |
-| `modelDiscovery` | ✗ | ✓ | `listModels()` returns a real catalog (Copilot reads `https://api.githubcopilot.com/models`). |
-| `promptViaArg` | ✗ | ✗ | If true, prompt goes via `--prompt <text>` instead of stdin. False on both because Windows ARG_MAX (~32 KB) breaks `-p "<long-prompt>"` outright. |
-| `budgetCap` | ✓ | ✗ | `--max-budget-usd <n>`. |
-| `bareMode` | ✓ | ✗ | `--bare` (suppresses CLAUDE.md auto-discovery). Closest Copilot equivalent is `--no-custom-instructions`, gated separately. |
-| `fallbackModel` | ✓ | ✗ | `--fallback-model <id>` on rate-limit. |
-| `sessionPersistenceControl` | ✓ | ✗ | Engine writes `session.json`. Copilot manages session state internally in `~/.copilot/session-state/`. |
-| `resumePromptCarryover` | ✗ | ✓ | CC resume turns prepend recent visible Q&A in stdin because Copilot's session store is opaque to Minions. |
-| `resumeBookkeepingTurn` | ✓ | — | Claude CLI injects a synthetic "Continue from where you left off." meta turn on `--resume`; CC prompts must tell the model not to treat it as user intent. |
-| `streamConsumer` | ✓ | ✓ | Adapter implements `createStreamConsumer(ctx)` — required by `engine/llm.js` accumulator. |
+| Flag | Claude | Copilot | Codex | Gates |
+|------|--------|---------|-------|-------|
+| `streaming` | ✓ | ✓ | ✓ | JSONL events on stdout. |
+| `sessionResume` | ✓ | ✓ | ✓ | Runtime-specific resume args. |
+| `midRunSessionId` | ✓ | ✗ | ✗ | Resumable session ID emitted before the terminal `result` event; when false, steering waits for a checkpoint. |
+| `systemPromptFile` | ✓ | ✗ | ✗ | sysprompt via `--system-prompt-file` (else inlined into stdin by `buildPrompt`). |
+| `effortLevels` | ✓ | ✓ | ✓ | Runtime reasoning-effort controls. Codex maps through `--config model_reasoning_effort=...`; Copilot maps `'max'` to `'xhigh'`; Claude leaves `'max'` alone. |
+| `costTracking` | ✓ | ✗ | ✗ | USD + token counts in the result event. Copilot only emits `premiumRequests`; Codex accounting is not assumed. |
+| `modelShorthands` | ✓ | ✗ | ✗ | Bare `sonnet`/`opus`/`haiku` accepted by Claude only. |
+| `modelDiscovery` | ✗ | ✓ | ✓ | `listModels()` returns a real catalog when supported (Copilot API, Codex `debug models --bundled`). |
+| `promptViaArg` | ✗ | ✗ | ✗ | If true, prompt goes via `--prompt <text>` instead of stdin. False for all current runtimes. |
+| `budgetCap` | ✓ | ✗ | ✗ | `--max-budget-usd <n>`. |
+| `bareMode` | ✓ | ✗ | ✗ | `--bare` (suppresses CLAUDE.md auto-discovery). Other runtimes use their own config/suppression knobs. |
+| `fallbackModel` | ✓ | ✗ | ✗ | `--fallback-model <id>` on rate-limit. |
+| `sessionPersistenceControl` | ✓ | ✗ | ✗ | Engine writes `session.json`; Copilot and Codex own their runtime stores, while Minions records IDs only for safe resume. |
+| `resumePromptCarryover` | ✗ | ✓ | ✓ | CC resume turns prepend recent visible Q&A in stdin when the runtime session store is opaque to Minions. |
+| `resumeBookkeepingTurn` | ✓ | — | — | Claude CLI injects a synthetic "Continue from where you left off." meta turn on `--resume`; CC prompts must tell the model not to treat it as user intent. |
+| `streamConsumer` | ✓ | ✓ | ✓ | Adapter implements `createStreamConsumer(ctx)` — required by `engine/llm.js` accumulator. |
 
 When a behavior is genuinely uniform across all current adapters, it still gets
 a flag if a future runtime might disagree — flags are cheap.
@@ -165,6 +166,7 @@ contracts.
 |---------|---------------------|---------------------|-------------------------------|
 | Claude  | `~/.claude/skills`  | `<repo>/.claude/skills` | personal: `~/.claude/skills`; project: `<repo>/.claude/skills` |
 | Copilot | `~/.copilot/skills` (+ `~/.agents/skills`) | `<repo>/.github/skills`, `<repo>/.agents/skills` | personal: `~/.copilot/skills`; project: `<repo>/.github/skills` |
+| Codex | `~/.agents/skills` (+ `/etc/codex/skills`) | `<repo>/.agents/skills` | personal: `~/.agents/skills`; project: `<repo>/.agents/skills` |
 
 `getUserAssetDirs()` returns the union of runtime-native global roots and is
 passed to spawn as `--add-dir` so spawned agents read the same global assets the

package/engine/db/index.js

@@ -18,6 +18,7 @@ const path = require('path');
 const fs = require('fs');
 
 let _db = null;
+let _dbPath = null;
 let _dbInitError = null;
 
 function _resolveDbPath() {
@@ -53,13 +54,34 @@ function _installExperimentalWarningFilter() {
 }
 
 function getDb() {
-  if (_db) return _db;
+  // Re-resolve the DB path on every call so tests that swap MINIONS_TEST_DIR
+  // (or production-side configuration that changes MINIONS_HOME between
+  // operations) get a fresh connection to the new location instead of
+  // a stale handle pointing at a now-deleted tmpdir. The resolve itself
+  // is just env-var reads + path.join — sub-microsecond.
+  let dbPath;
+  try { dbPath = _resolveDbPath(); }
+  catch (e) {
+    _dbInitError = new Error(`engine/db: cannot resolve DB path — ${e.message}`);
+    throw _dbInitError;
+  }
+
+  if (_db) {
+    if (_dbPath === dbPath) return _db;
+    // Path changed (test isolation swap, or operator reconfigured
+    // MINIONS_HOME). Close the old handle and reopen against the new path.
+    try { _db.close(); } catch { /* already closed */ }
+    _db = null;
+    _dbPath = null;
+    _dbInitError = null;
+  }
+
   if (_dbInitError) throw _dbInitError;
   try {
     _installExperimentalWarningFilter();
     const { DatabaseSync } = require('node:sqlite');
-    const dbPath = _resolveDbPath();
     _db = new DatabaseSync(dbPath);
+    _dbPath = dbPath;
     // WAL mode lets the engine writer and dashboard reader hit the same
     // file from two processes without lock contention. NORMAL synchronous
     // is the recommended WAL pairing (durable enough; one OS-level fsync
@@ -72,11 +94,6 @@ function getDb() {
     runMigrations(_db);
     return _db;
   } catch (e) {
-    // Cache the failure so we don't repeatedly retry a broken setup. Most
-    // callers wrap getDb() in try/catch and treat the cache-version probe
-    // as best-effort, so a missing-Node-module install still boots the
-    // dashboard fine — it just won't benefit from cross-process cache
-    // invalidation until the install is fixed.
     _dbInitError = new Error(
       `engine/db: failed to open SQLite — ${e.message}. ` +
       `Node 22.5+ required for built-in 'node:sqlite' support.`
@@ -88,7 +105,37 @@ function getDb() {
 // Force-close — used by tests + graceful shutdown. Safe to call when not open.
 function closeDb() {
   if (_db) { try { _db.close(); } catch { /* already closed */ } _db = null; }
+  _dbPath = null;
   _dbInitError = null;
 }
 
-module.exports = { getDb, closeDb };
+// Run `fn(db)` inside a transaction. Nestable: only the OUTERMOST call
+// issues BEGIN/COMMIT; inner calls run inside the enclosing tx so multiple
+// stores (dispatches + work_items + ...) can compose into one atomic
+// mutation. Inner rollback propagates outward, which is the correct
+// semantics for cross-store invariants — if a nested mutation fails, the
+// enclosing one must also fail.
+//
+// We track depth on the db instance itself rather than module-level so
+// reopening the singleton (test isolation, path changes) doesn't leak
+// stale depth into a fresh connection.
+function withTransaction(db, fn) {
+  if (!db) throw new Error('engine/db.withTransaction: db is required');
+  const isOuter = !db._minionsTxDepth;
+  db._minionsTxDepth = (db._minionsTxDepth || 0) + 1;
+  if (isOuter) db.exec('BEGIN IMMEDIATE');
+  try {
+    const result = fn(db);
+    db._minionsTxDepth -= 1;
+    if (db._minionsTxDepth === 0) db.exec('COMMIT');
+    return result;
+  } catch (e) {
+    db._minionsTxDepth -= 1;
+    if (db._minionsTxDepth === 0) {
+      try { db.exec('ROLLBACK'); } catch { /* best-effort */ }
+    }
+    throw e;
+  }
+}
+
+module.exports = { getDb, closeDb, withTransaction };

package/engine/db/migrations/002-dispatches.js

@@ -0,0 +1,113 @@
+// engine/db/migrations/002-dispatches.js
+//
+// Phase 1: move engine/dispatch.json into a `dispatches` table.
+//
+// Schema is intentionally hybrid: a small set of indexed projection columns
+// (status, agent, type, timestamps) for hot filters/sorts, plus a single
+// `data` TEXT column holding the full record JSON. This way the
+// `mutateDispatch(fn)` API contract — "mutator receives the section-shaped
+// object, mutates in place, returns" — round-trips every field cleanly
+// without us needing to enumerate every possible column ahead of time.
+// New fields the engine adds in a future commit land in `data`
+// automatically.
+//
+// Backfill: read engine/dispatch.json's four sections (pending / active /
+// completed / review), INSERT one row per dispatch with `status` derived
+// from the section name. Timestamps are parsed from ISO strings to
+// milliseconds since epoch (matches the rest of the codebase). The JSON
+// file is renamed to engine/dispatch.json.pre-sql-<ts> for rollback —
+// NOT deleted. A separate `minions db rollback-to-json` command (next
+// phase) will read the table and reconstruct the JSON if the user pins
+// to a pre-migration version.
+
+const path = require('path');
+const fs = require('fs');
+
+function _resolveMinionsDir() {
+  const envHome = process.env.MINIONS_HOME || process.env.MINIONS_TEST_DIR;
+  if (envHome) return envHome;
+  try { return require('../../shared').MINIONS_DIR; } catch { return null; }
+}
+
+function _toMs(v) {
+  if (v == null) return null;
+  if (typeof v === 'number') return Number.isFinite(v) ? v : null;
+  const parsed = Date.parse(v);
+  return Number.isFinite(parsed) ? parsed : null;
+}
+
+module.exports = {
+  version: 2,
+  description: 'dispatches: schema + dispatch.json backfill',
+  up(db) {
+    db.exec(`
+      CREATE TABLE dispatches (
+        id            TEXT PRIMARY KEY,
+        status        TEXT NOT NULL,
+        agent         TEXT,
+        type          TEXT,
+        created_at    INTEGER,
+        started_at    INTEGER,
+        completed_at  INTEGER,
+        data          TEXT NOT NULL,
+        updated_at    INTEGER NOT NULL
+      );
+      CREATE INDEX idx_dispatches_status            ON dispatches(status);
+      CREATE INDEX idx_dispatches_agent             ON dispatches(agent);
+      CREATE INDEX idx_dispatches_status_completed  ON dispatches(status, completed_at DESC);
+      CREATE INDEX idx_dispatches_status_created    ON dispatches(status, created_at DESC);
+    `);
+
+    const minionsDir = _resolveMinionsDir();
+    if (!minionsDir) return;
+    const dispatchPath = path.join(minionsDir, 'engine', 'dispatch.json');
+    if (!fs.existsSync(dispatchPath)) return;
+
+    let raw;
+    try { raw = JSON.parse(fs.readFileSync(dispatchPath, 'utf8')); }
+    catch (e) {
+      // Corrupt JSON — abort the migration, surfacing the error. The user
+      // can fix the file (or restore from the .backup sidecar safeWrite
+      // keeps) and re-run the engine; the migration will retry.
+      throw new Error(`engine/db/002-dispatches: cannot parse dispatch.json: ${e.message}`);
+    }
+
+    const now = Date.now();
+    const insert = db.prepare(`
+      INSERT INTO dispatches (id, status, agent, type, created_at, started_at, completed_at, data, updated_at)
+      VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
+    `);
+
+    let inserted = 0;
+    for (const status of ['pending', 'active', 'completed', 'review']) {
+      const rows = raw[status];
+      if (!Array.isArray(rows)) continue;
+      for (const d of rows) {
+        if (!d || typeof d !== 'object' || !d.id) continue;
+        insert.run(
+          String(d.id),
+          status,
+          d.agent || null,
+          d.type || null,
+          _toMs(d.created_at),
+          _toMs(d.started_at),
+          _toMs(d.completed_at),
+          JSON.stringify(d),
+          now,
+        );
+        inserted += 1;
+      }
+    }
+
+    // dispatch.json stays on disk as a dual-write mirror — every successful
+    // mutateDispatch refreshes it from SQL via dispatch-store._mirrorJsonFromSql.
+    // SQL is the source of truth; the file is a read-only derivative kept
+    // for backward compatibility with legacy direct-readers (engine/routing.js,
+    // engine/queries.js fallback, test infrastructure). Rollback path: if a
+    // user pins back to a pre-Phase-1 release, dispatch.json already contains
+    // the latest committed state (last mirror write), and the SQL table can
+    // be discarded.
+    // eslint-disable-next-line no-console
+    console.log(`[db-migrate] v2: backfilled ${inserted} dispatches; dispatch.json kept as dual-write mirror`);
+  },
+};

package/engine/db/migrations/003-work-items.js

@@ -0,0 +1,128 @@
+// engine/db/migrations/003-work-items.js
+//
+// Phase 2: move work-items.json (central + per-project) into a `work_items`
+// table.
+//
+// Same hybrid schema as `dispatches`: typed projection columns for hot
+// filters (scope/status/agent/parent), plus a `data` TEXT column holding
+// the full record JSON so future fields land transparently.
+//
+// `scope` keys the file the record came from: 'central' for
+// <MINIONS_DIR>/work-items.json, otherwise the project name (matches the
+// directory under projects/<name>/). Per-scope writes diff against the
+// scope's rows so a mutation on one project's file never touches another's.
+//
+// Backfill: walk central + every projects/<name>/work-items.json. The
+// JSON files stay on disk as a dual-write mirror so legacy direct-readers
+// continue to function while we migrate them in Phase 2.5.
+
+const path = require('path');
+const fs = require('fs');
+
+function _resolveMinionsDir() {
+  const envHome = process.env.MINIONS_HOME || process.env.MINIONS_TEST_DIR;
+  if (envHome) return envHome;
+  try { return require('../../shared').MINIONS_DIR; } catch { return null; }
+}
+
+function _toMs(v) {
+  if (v == null) return null;
+  if (typeof v === 'number') return Number.isFinite(v) ? v : null;
+  const parsed = Date.parse(v);
+  return Number.isFinite(parsed) ? parsed : null;
+}
+
+function _readJsonArray(filePath) {
+  try {
+    const raw = fs.readFileSync(filePath, 'utf8');
+    const parsed = JSON.parse(raw);
+    return Array.isArray(parsed) ? parsed : [];
+  } catch {
+    return [];
+  }
+}
+
+function _listProjectDirs(minionsDir) {
+  const projectsRoot = path.join(minionsDir, 'projects');
+  let entries;
+  try { entries = fs.readdirSync(projectsRoot, { withFileTypes: true }); }
+  catch { return []; }
+  const out = [];
+  for (const e of entries) {
+    if (!e.isDirectory()) continue;
+    if (e.name === '.archived') continue;
+    out.push(e.name);
+  }
+  return out;
+}
+
+module.exports = {
+  version: 3,
+  description: 'work_items: schema + work-items.json backfill (central + per-project)',
+  up(db) {
+    db.exec(`
+      CREATE TABLE work_items (
+        id            TEXT NOT NULL,
+        scope         TEXT NOT NULL,
+        status        TEXT NOT NULL,
+        type          TEXT,
+        agent         TEXT,
+        parent_id     TEXT,
+        created_at    INTEGER,
+        completed_at  INTEGER,
+        data          TEXT NOT NULL,
+        updated_at    INTEGER NOT NULL,
+        PRIMARY KEY (scope, id)
+      );
+      CREATE INDEX idx_wi_status        ON work_items(status);
+      CREATE INDEX idx_wi_scope_status  ON work_items(scope, status);
+      CREATE INDEX idx_wi_agent         ON work_items(agent);
+      CREATE INDEX idx_wi_parent        ON work_items(parent_id);
+    `);
+
+    const minionsDir = _resolveMinionsDir();
+    if (!minionsDir) return;
+
+    const insert = db.prepare(`
+      INSERT INTO work_items (id, scope, status, type, agent, parent_id, created_at, completed_at, data, updated_at)
+      VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+    `);
+
+    const now = Date.now();
+    let inserted = 0;
+
+    const backfillFile = (scope, filePath) => {
+      const rows = _readJsonArray(filePath);
+      for (const wi of rows) {
+        if (!wi || typeof wi !== 'object' || !wi.id) continue;
+        try {
+          insert.run(
+            String(wi.id),
+            scope,
+            String(wi.status || 'pending'),
+            wi.type || null,
+            wi.dispatched_to || wi.agent || null,
+            wi.parent_id || null,
+            _toMs(wi.created_at),
+            _toMs(wi.completed_at || wi.completedAt),
+            JSON.stringify(wi),
+            now,
+          );
+          inserted += 1;
+        } catch {
+          // Duplicate (scope, id) within the same file → skip. Real-world
+          // work-items.json doesn't carry dupes, but defensive: a corrupted
+          // file shouldn't abort the whole migration.
+        }
+      }
+    };
+
+    backfillFile('central', path.join(minionsDir, 'work-items.json'));
+    for (const projectName of _listProjectDirs(minionsDir)) {
+      backfillFile(projectName, path.join(minionsDir, 'projects', projectName, 'work-items.json'));
+    }
+
+    // eslint-disable-next-line no-console
+    console.log(`[db-migrate] v3: backfilled ${inserted} work items; work-items.json files kept as dual-write mirrors`);
+  },
+};