nomarmy 0.1.0-alpha.11 → 0.1.0-alpha.13

package/README.md

@@ -52,6 +52,8 @@ Failing verification stays failed, unconditionally. A malformed report isn't aut
 
 **Checking without building** costs nothing: `mode: verify` runs a verification profile against any branch, with no worker and no model tokens.
 
+**Want deeper checks?** Three optional [validators](https://github.com/rayson-tech/nomarmy/blob/main/docs/validators.md) go further, each only adding review flags: mutation testing (do the tests pin down the changed lines?), Jev (do a scout's citations support its findings, does a report match its diff?) and a model judge (acceptance criteria, weakened tests).
+
 ## Where the work runs
 
 - **Agents** say where a job can run: an API key, your own ChatGPT or Muse Code subscription, or a local model on llama.cpp.
@@ -74,10 +76,12 @@ Developed and maintained by Rayson Technologies. This is an alpha (`0.1.0-alpha`
 | [Agents and the army](https://github.com/rayson-tech/nomarmy/blob/main/docs/agents-and-army.md) | Where a job can run, who does what, usage limits, picking an agent |
 | [`/feature` runs](https://github.com/rayson-tech/nomarmy/blob/main/docs/feature-runs.md) | A feature end to end, and watching what nomArmy is doing |
 | [Your repository](https://github.com/rayson-tech/nomarmy/blob/main/docs/your-repo.md) | `.nomarmy.yml`, verification, dependencies, private registries, what nomArmy checks |
+| [Validators](https://github.com/rayson-tech/nomarmy/blob/main/docs/validators.md) | Optional deeper checks: mutation testing, Jev, a model judge |
 | [Harnesses](https://github.com/rayson-tech/nomarmy/blob/main/docs/harnesses.md) | Ecosystem registry, detection, network levels, and requirements |
 | [Configuration](https://github.com/rayson-tech/nomarmy/blob/main/docs/configuration.md) | Settings, swapping the local model, sizing, admission |
 | [Reference](https://github.com/rayson-tech/nomarmy/blob/main/docs/reference.md) | Every CLI command and MCP tool |
 | [Security posture](https://github.com/rayson-tech/nomarmy/blob/main/docs/security.md) | What the sandbox holds back, and the one exception |
+| [FAQ](https://github.com/rayson-tech/nomarmy/blob/main/docs/faq.md) | Which model for which role, switching models, usage limits |
 | [Troubleshooting](https://github.com/rayson-tech/nomarmy/blob/main/docs/troubleshooting.md) | Symptoms and fixes |
 
 ## Security
@@ -97,6 +101,8 @@ A nom gets a writable git worktree inside a Podman sandbox and nothing else: no
 | The army and `/feature` | Driven by a real Claude Code General across three runs, about 18 implement jobs |
 | Harnesses: Go, Rust, Python, Node and mixed repos; Playwright; fake services | Live-verified offline |
 | Private registries and a verification-only network allowlist | Live-verified; each passed an independent security review |
+| Validators: mutation testing, Jev, a model judge | Unit and live tested; Jev and the judge evaluated on real job records |
+| `nomarmy stats` | Checked against a hand-built report on real job records |
 
 What we've learned from real runs, including where delegating pays and where it doesn't, is in [docs/findings.md](https://github.com/rayson-tech/nomarmy/blob/main/docs/findings.md).
 

package/bin/nomarmy.mjs

@@ -20,6 +20,10 @@ import { readGGUFMetadata, resolveModelPath, totalSplitBytes } from "../lib/gguf
 import { recommend, customRecommendation, evaluateConfig, bytesPerKvElementForCacheTypes, MIN_CONTEXT_PER_NOM } from "../lib/sizing.mjs";
 import { connectClaude, connectCodex, connectCursor, cursorAlreadyConnected, deriveWorkerModelEnv, defaultInstallDir, installMcpCopy, SCOPES, claudeUserScoped, portableServerLaunch } from "../lib/connect.mjs";
 import { compareVersions, readPackageVersion, readInstallVersions, copyIsStale } from "../lib/install-freshness.mjs";
+import { loadJobRecords, computeStats, formatStats, parseSince, resolveRepo } from "../lib/stats.mjs";
+import { requestJobStop } from "../lib/openclaw-run.mjs";
+import { loadValidators, saveJevKey, removeJev, jevSettings, askJev, validatorsPath, JEV_CHECKS, saveJudge, removeJudge, judgeSettings } from "../lib/validators.mjs";
+import { probeModel } from "../lib/model-probe.mjs";
 import { ID_RE, AUTH_ENV_NAME_RE, OPENCLAW_PROVIDER_ID_RE, openclawProviderId, isNativeProviderType } from "../lib/dispatch-schema.mjs";
 import { loadAgents, readAgentsFile, writeAgentsFile, agentsConfigPath, apiAgentAsPoolEntry, describeAgent as describeAgentLabel, agentRunsToolsOnHost, agentProviderId, AGENT_KINDS, API_PROVIDER_TYPES, RESERVED_AGENT_NAMES, BUILTIN_LOCAL_AGENT } from "../lib/agents.mjs";
 import { loadArmy, mergeArmy, describeArmy, readArmyFile, updateArmyInFile, assignRoleInFile, parseTargetSpec, armyLayerPath, globalConfigDir, DEFAULT_ARMY, ARMY_PHASES, LOCAL_CONFIG_FILENAME } from "../lib/army.mjs";
@@ -195,17 +199,25 @@ Usage: nomarmy <command> [options]
                             which agent the General is, in --global
                             (default) or --local
   config paths    where agents.yml and the three army layers live
-  jobs [--watch|--events|--prune|--wait <jobId>] [--interval N] [--older-than DAYS]
+  jobs [--watch|--events [--until-done]|--prune|--wait <jobId>|--stop <jobId> [--reason <text>]] [--interval N] [--older-than DAYS]
                   what's running across every session (agent, model, phase,
                   last tool call, files changed, heartbeat) and what just
                   finished; --watch redraws every N seconds (default 3);
                   --events prints one line per start, phase change and
-                  finish (for Claude Code's background monitor; --json for
-                  JSON lines); --prune removes the bulky runtime data
+                  finish (--json for JSON lines). It's a stream: read it
+                  with a monitor that wakes on each line. A background
+                  command is only reported when it exits, so there use
+                  --events --until-done, which exits once every job it saw
+                  running has finished (or --wait for one job). The plain
+                  stream ends on its own after 30 minutes with nothing
+                  running (--idle-minutes N); --prune removes the bulky runtime data
                   from finished jobs older than DAYS (default 2), keeping
                   their records, reports and any retained worktree;
                   --wait <jobId> [--timeout <seconds>] blocks for one job
-                  to finish (default timeout 1800; --json is supported)
+                  to finish (default timeout 1800; --json is supported);
+                  --stop <jobId> stops a running job's worker (no report
+                  recovery, no verification), keeping its worktree for
+                  continue_from
   health          check what's likely to break a run before it does:
                   expiring logins, an outdated OpenClaw or plugin, roles
                   that can't be dispatched, an unloadable agents.yml,
@@ -222,6 +234,24 @@ Usage: nomarmy <command> [options]
                   project for this repository, committed for the team
                   (.mcp.json or .cursor/mcp.json, running \`nomarmy mcp\`).
                   Codex has only the user scope.
+  stats [--since 7d|<date>] [--until <date>] [--role <role>] [--model <model>]
+        [--repo <path|name>] [--all-repos] [--json]
+                  What nomArmy's job records show for this repository (or
+                  all): volume by role and model, code committed, time,
+                  tokens and spend, how often a "done" report failed
+                  independent verification, what didn't finish, reviewers,
+                  and review flags. From verified records, never reports.
+  validators <list|add jev|test jev|remove jev|add judge|test judge|remove judge>
+                  Optional semantic checks from a model you configure with
+                  your own key. Today: Jev (TypeSafe). \`add jev\` asks for the
+                  key without echoing it (or reads --key-stdin), saves it
+                  where only you can read it, and makes one test call. Its
+                  answers only add review flags, and it sends excerpts of
+                  your code to TypeSafe. \`add judge --agent <name> --model
+                  <model>\` makes one of your agents a model judge: does the
+                  diff meet each acceptance criterion, match the report, keep
+                  its tests as strong? An agent whose tools run on this
+                  machine needs --host-tools.
   mcp             Start nomArmy's MCP server on stdio with this machine's
                   settings. What a --scope project registration runs.
   sandbox         The Podman VM every sandbox shares (macOS, Windows): its
@@ -2497,6 +2527,12 @@ function renderJobs({ running, recent }) {
  */
 async function streamJobEvents() {
   const interval = Math.max(1, Number(value("interval", "3")) || 3) * 1000;
+  // A stream nobody reads must still end: a General that ran this as a
+  // background command (reported only on exit) was never told jobs had
+  // finished, and eight of these streams were left running for days.
+  const untilDone = flag("until-done");
+  const idleLimitMs = Math.max(1, Number(value("idle-minutes", "30")) || 30) * 60000;
+  let idleSinceMs = Date.now(), sawRunning = false;
   const seen = new Map();
   const emit = (event, job, detail = "") => {
     if (json) console.log(JSON.stringify({ at: new Date().toISOString(), event, jobId: job.jobId, agent: job.agent, model: job.model, phase: job.phase, detail }));
@@ -2519,10 +2555,23 @@ async function streamJobEvents() {
     seen.clear();
     for (const [id, j] of now) seen.set(id, j);
     first = false;
+    if (running.length) { sawRunning = true; idleSinceMs = Date.now(); }
+    else if (untilDone && sawRunning) {
+      if (json) console.log(JSON.stringify({ at: new Date().toISOString(), event: "done", detail: "every job seen running has finished" }));
+      else console.log(`${new Date().toLocaleTimeString()}  done      every job seen running has finished`);
+      return;
+    } else if (Date.now() - idleSinceMs >= (untilDone ? Math.min(idleLimitMs, 120000) : idleLimitMs)) {
+      const why = untilDone ? "no job was running to wait for" : `nothing has run for ${Math.round(idleLimitMs / 60000)} minutes`;
+      if (json) console.log(JSON.stringify({ at: new Date().toISOString(), event: "idle", detail: why }));
+      else console.log(`${new Date().toLocaleTimeString()}  idle      ${why}; exiting`);
+      return;
+    }
     await new Promise((r) => setTimeout(r, interval));
   }
 }
 
+const commitSha = (commit) => (typeof commit === "string" ? commit : typeof commit?.sha === "string" ? commit.sha : null);
+
 /** Wait for one job in the shared, cross-session state directory. */
 async function waitForJobCli() {
   const requested = value("wait");
@@ -2559,7 +2608,8 @@ async function waitForJobCli() {
         outcome: meta.outcome ?? status.outcome ?? null,
         coordinatorStatus: meta.coordinatorStatus ?? status.coordinatorStatus ?? null,
         branch: meta.branch ?? status.branch ?? null,
-        commit: meta.commit?.sha ?? meta.commit ?? status.commit?.sha ?? status.commit ?? null,
+        // A job that made no commit has commit: { created: false, sha: null }; only a sha is a commit.
+        commit: commitSha(meta.commit) ?? commitSha(status.commit),
         issues,
       };
       if (json) out(result);
@@ -2600,6 +2650,13 @@ function pruneJobRuntimeCli() {
 
 async function cmdJobs() {
   if (flag("wait")) return waitForJobCli();
+  if (flag("stop")) {
+    const r = requestJobStop({ jobsRoot: jobsRootDir(), jobId: value("stop"), reason: value("reason") });
+    if (json) return out(r);
+    console.log(r.ok ? c.green(`✓ ${r.message}`) : c.red(`✗ ${r.message}`));
+    if (!r.ok) process.exitCode = 1;
+    return;
+  }
   if (flag("events")) return streamJobEvents();
   if (flag("prune")) return pruneJobRuntimeCli();
   if (json) return out(collectJobs());
@@ -2646,6 +2703,135 @@ async function cmdStatusline() {
   process.stdout.write(`${statusLineText({ session })}\n`);
 }
 
+function cmdStats() {
+  const stateRoot = process.env.NOMARMY_AGENT_STATE || path.join(os.homedir(), ".local", "share", "nomarmy-local-agents");
+  const records = loadJobRecords(path.join(stateRoot, "jobs"));
+  let repo = null;
+  if (value("repo")) repo = resolveRepo(records, value("repo"));
+  else if (!flag("all-repos")) {
+    try { repo = execFileSync("git", ["rev-parse", "--show-toplevel"], { cwd: repoDir, encoding: "utf8", stdio: ["ignore", "pipe", "ignore"] }).trim(); }
+    catch { throw new Error(`${repoDir} isn't inside a git repository; run nomarmy stats from one, or pass --repo <name> or --all-repos`); }
+  }
+  const stats = computeStats(records, { repo, sinceMs: parseSince(value("since")), untilMs: parseSince(value("until")), role: value("role"), model: value("model") });
+  if (json) return out(stats);
+  console.log(formatStats(stats));
+}
+
+// Read one line without echoing it: stty -echo around the read, restored
+// even if the read fails. Windows has no stty, so it says the input shows.
+async function readHiddenLine(prompt) {
+  const hide = process.stdin.isTTY && process.platform !== "win32";
+  if (!hide) console.log(c.yellow("(your input will be visible as you type)"));
+  const rl = createInterface({ input, output });
+  try {
+    if (hide) spawnSync("stty", ["-echo"], { stdio: ["inherit", "ignore", "ignore"] });
+    return (await rl.question(prompt)).trim();
+  } finally {
+    if (hide) { spawnSync("stty", ["echo"], { stdio: ["inherit", "ignore", "ignore"] }); process.stdout.write("\n"); }
+    rl.close();
+  }
+}
+
+async function readStdin() {
+  const chunks = [];
+  for await (const chunk of process.stdin) chunks.push(chunk);
+  return Buffer.concat(chunks).toString("utf8").trim();
+}
+
+// One tiny System One request, to prove the key and the route work.
+async function testJev(settings) {
+  const { answers } = await askJev({ key: settings.key, model: settings.model, state: { text: "The build finished and all 12 tests passed." },
+    questions: { passed: { type: "noul", instructions: "Does the text say the tests passed?", criteria: { true: "It says the tests passed", false: "It doesn't" } } } });
+  return typeof answers.passed?.noul === "number";
+}
+
+async function cmdValidators() {
+  const [sub = "list", name] = argv.slice(1).filter((a) => !a.startsWith("--"));
+  if (sub === "list") {
+    let config = {};
+    try { config = loadValidators(); } catch (error) { if (json) return out({ error: error.message }); console.log(c.red(error.message)); process.exitCode = 1; return; }
+    const judge = config.judge ? { enabled: config.judge.enabled, agent: config.judge.agent, model: config.judge.model, checks: config.judge.checks, hostTools: config.judge.host_tools } : null;
+    const jev = config.jev ? { enabled: config.jev.enabled, checks: config.jev.checks, model: config.jev.model, key: config.jev.key_env ? `env ${config.jev.key_env}` : config.jev.key_file, keyReadable: Boolean(jevSettings()) } : null;
+    if (json) return out({ path: validatorsPath(), jev, judge });
+    if (!jev && !judge) { console.log("No validators configured. Add one with: nomarmy validators add jev, or nomarmy validators add judge --agent <name> --model <model>"); return; }
+    if (jev) console.log(`Jev: ${jev.enabled ? c.green("on") : "off"} (${jev.model}); checks: ${jev.checks.join(", ")}; key: ${jev.key}${jev.keyReadable ? "" : c.red(" (not readable)")}`);
+    if (judge) console.log(`Judge: ${judge.enabled ? c.green("on") : "off"} (${judge.agent}/${judge.model}); checks: ${judge.checks.join(", ")}${judge.hostTools ? c.yellow("; host tools allowed") : ""}`);
+    return;
+  }
+  if (name === "judge") return cmdValidatorsJudge(sub);
+  if (name !== "jev") throw new Error("Usage: nomarmy validators <list|add jev|test jev|remove jev|add judge|test judge|remove judge>");
+  if (sub === "add") {
+    if (!json) {
+      console.log(c.bold("🍪 Jev (TypeSafe) for nomArmy's semantic checks\n"));
+      console.log("It checks that a scout's cited lines support its finding, and that a worker's report matches its diff.");
+      console.log("Its answers only add review flags; they never pass a check or allow a commit.");
+      console.log(c.yellow("It sends excerpts of your code (findings, cited lines, diffs, worker reports) to TypeSafe.\n"));
+    }
+    const key = flag("key-stdin") ? await readStdin() : await readHiddenLine("TypeSafe API key (not shown): ");
+    const saved = saveJevKey(key);
+    let ok = false, why = null;
+    try { ok = await testJev(jevSettings()); } catch (error) { why = error.message; }
+    if (json) return out({ saved: true, keyFile: saved.keyFile, configPath: saved.configPath, test: ok ? "pass" : "fail", reason: why });
+    console.log(c.green(`✓ Saved the key to ${saved.keyFile} (readable only by you) and turned Jev on in ${saved.configPath}.`));
+    console.log(ok ? c.green("✓ Test call answered. New jobs use it; restart open coordinator sessions to pick it up.") : c.red(`✗ Test call failed: ${why ?? "no answer"}. Check the key, then: nomarmy validators test jev`));
+    if (!ok) process.exitCode = 1;
+    return;
+  }
+  if (sub === "test") {
+    const settings = jevSettings();
+    if (!settings) throw new Error("Jev isn't configured, or its key isn't readable. Add it with: nomarmy validators add jev");
+    let ok = false, why = null;
+    try { ok = await testJev(settings); } catch (error) { why = error.message; }
+    if (json) return out({ test: ok ? "pass" : "fail", reason: why });
+    console.log(ok ? c.green("✓ Jev answered.") : c.red(`✗ Jev test call failed: ${why ?? "no answer"}`));
+    if (!ok) process.exitCode = 1;
+    return;
+  }
+  if (sub === "remove") {
+    const result = removeJev();
+    if (json) return out(result);
+    console.log(c.green(`✓ Jev is off${result.removedKey ? ", and its saved key is deleted" : ""}.`));
+    return;
+  }
+  throw new Error("Usage: nomarmy validators <list|add jev|test jev|remove jev>");
+}
+
+async function cmdValidatorsJudge(sub) {
+  const agents = loadAgents(globalConfigDir()).agents;
+  const resolve = () => judgeSettings({ agents, providerOf: agentProviderId, runsOnHost: agentRunsToolsOnHost });
+  const probe = async (settings) => probeModel({ provider: settings.provider, model: settings.model, stateRoot: process.env.NOMARMY_AGENT_STATE || path.join(os.homedir(), ".local", "share", "nomarmy-local-agents") });
+  if (sub === "add") {
+    const agent = value("agent"), model = value("model");
+    if (!agent || !model) throw new Error("Usage: nomarmy validators add judge --agent <name> --model <model> [--host-tools]");
+    if (!agents[agent]) throw new Error(`"${agent}" isn't an agent in agents.yml. Agents: ${Object.keys(agents).join(", ") || "(none)"}`);
+    if (agentRunsToolsOnHost(agents[agent]) && !flag("host-tools")) throw new Error(`agent "${agent}" runs its tools on this machine, and a judge reads text the worker wrote. Pass --host-tools to accept that, or pick a sandboxed agent (an api key, Codex, Muse).`);
+    const saved = saveJudge({ agent, model, hostTools: flag("host-tools") });
+    const settings = resolve();
+    if (settings?.problem) throw new Error(settings.problem);
+    const test = await probe(settings);
+    if (json) return out({ saved: true, configPath: saved.configPath, test: test.ok ? "pass" : test.refused ? "refused" : "inconclusive", reason: test.reason });
+    console.log(c.green(`✓ The judge is ${agent}/${model}, in ${saved.configPath}.`));
+    console.log(test.ok ? c.green("✓ Test call answered. New implement jobs use it; restart open coordinator sessions to pick it up.") : c.red(`✗ Test call ${test.refused ? "refused" : "didn't answer"}: ${test.reason ?? "no answer"}`));
+    if (!test.ok) process.exitCode = 1;
+    return;
+  }
+  if (sub === "test") {
+    const settings = resolve();
+    if (!settings) throw new Error("No judge configured. Add one with: nomarmy validators add judge --agent <name> --model <model>");
+    if (settings.problem) throw new Error(settings.problem);
+    const test = await probe(settings);
+    if (json) return out({ test: test.ok ? "pass" : "fail", reason: test.reason });
+    console.log(test.ok ? c.green(`✓ ${settings.agent}/${settings.model} answered.`) : c.red(`✗ ${test.reason ?? "no answer"}`));
+    if (!test.ok) process.exitCode = 1;
+    return;
+  }
+  if (sub === "remove") {
+    removeJudge();
+    return json ? out({ removed: true }) : console.log(c.green("✓ The judge is off."));
+  }
+  throw new Error("Usage: nomarmy validators <add judge --agent <name> --model <model> [--host-tools]|test judge|remove judge>");
+}
+
 // `nomarmy mcp`: what a --scope project registration runs. Nothing goes to
 // stdout but the server's own protocol.
 function cmdMcp() {
@@ -2655,7 +2841,7 @@ function cmdMcp() {
   child.on("exit", (code, signal) => { if (signal) process.kill(process.pid, signal); else process.exit(code ?? 1); });
 }
 
-const commands = { mcp: cmdMcp, scan: cmdScan, validate: cmdValidate, sizing: cmdSizing, init: cmdInit, setup: cmdSetup, install: cmdInstall, model: cmdModel, agents: cmdAgents, army: cmdArmy, jobs: cmdJobs, statusline: cmdStatusline, health: cmdHealth, config: cmdConfig, update: cmdUpdate, connect: cmdConnect, sandbox: cmdSandbox, start: cmdStart, stop: cmdStop, uninstall: cmdUninstall, help: () => usage(0) };
+const commands = { stats: cmdStats, validators: cmdValidators, mcp: cmdMcp, scan: cmdScan, validate: cmdValidate, sizing: cmdSizing, init: cmdInit, setup: cmdSetup, install: cmdInstall, model: cmdModel, agents: cmdAgents, army: cmdArmy, jobs: cmdJobs, statusline: cmdStatusline, health: cmdHealth, config: cmdConfig, update: cmdUpdate, connect: cmdConnect, sandbox: cmdSandbox, start: cmdStart, stop: cmdStop, uninstall: cmdUninstall, help: () => usage(0) };
 // doctor command
 async function cmdDoctor() {
   // Import lazily to avoid circular dependencies

package/lib/admission.mjs

@@ -126,6 +126,19 @@ export function createJobRuntime(deps) {
       try { return await fn(); } finally { slot.release(); }
     })();
   }
+  // Who ran the job and for whom, onto its finished record, so `nomarmy
+  // stats` can count by role and agent, and by repo for verify runs too
+  // (their records didn't carry the repo). Best-effort; never fails the job.
+  function stampJobRecord(entry, result) {
+    const file = path.join(result?.jobDir ?? path.join(jobsRoot, entry.jobId), "metadata.json");
+    try {
+      const record = JSON.parse(fs.readFileSync(file, "utf8"));
+      record.labels = { role: entry.role ?? null, agent: entry.agent ?? null, label: entry.label ?? null, runId: entry.runId ?? null };
+      record.projectDir = record.projectDir ?? projectDir;
+      fs.writeFileSync(`${file}.tmp`, JSON.stringify(record, null, 2));
+      fs.renameSync(`${file}.tmp`, file);
+    } catch { /* no record (a refused or crashed job): nothing to label */ }
+  }
   function track(jobId, meta, promise) {
     const entry = { ...meta, jobId, startedAt: new Date().toISOString(), settled: false, result: null, error: null, promise: null };
     // A machine-wide lease for as long as the job runs, so every session's
@@ -134,7 +147,7 @@ export function createJobRuntime(deps) {
     if (meta.lane) writeLease(leasesRoot, jobId, { lane: meta.lane, agent: meta.agent ?? null, runId: meta.runId ?? null, role: meta.role ?? null, model: meta.model ?? null, repo: projectDir });
     const release = () => removeLease(leasesRoot, jobId);
     entry.promise = promise.then(
-      r => { entry.settled = true; entry.result = r; release(); notifyJobFinished(entry, r, null); return r; },
+      r => { entry.settled = true; entry.result = r; release(); stampJobRecord(entry, r); notifyJobFinished(entry, r, null); return r; },
       e => { entry.settled = true; entry.error = e; release(); notifyJobFinished(entry, null, e); throw e; });
     entry.promise.catch(() => {});
     activeJobs.set(jobId, entry);

package/lib/coordinator-instructions.mjs

@@ -15,12 +15,12 @@ Before dispatching:
 - A Claude subscription agent (claude-cli) runs its tools on this machine, outside the sandbox: use it for scout and review work. nomArmy refuses implement jobs on it unless the operator set allow_host_tools; send build work to a sandboxed agent.
 - To run tests without changing anything, use mode: verify; it costs no model usage.
 - Brief outcomes, not edits: a task, explicit acceptance criteria, and the tests that prove it. Put facts you've already resolved in evidence.
-- Prefer local_worker_start for anything longer than a few minutes. Right after, if you can run a background command, run \`nomarmy jobs --wait <job_id>\` in the background so you're told the moment it finishes and can tell the operator; otherwise poll local_worker_status with wait_seconds. For a whole feature, use /feature (run_start keeps a run's jobs, spend and hours bounded).
+- Prefer local_worker_start for anything longer than a few minutes. Right after, if you can run a background command, run \`nomarmy jobs --wait <job_id>\` in the background so you're told the moment it finishes and can tell the operator (for several jobs, \`nomarmy jobs --events --until-done\`, which exits when they've all finished); otherwise poll local_worker_status with wait_seconds. Never run the plain \`nomarmy jobs --events\` stream as a background command: it only reports when it exits, so you'd never hear; it's for a monitor that reads each line. For a whole feature, use /feature (run_start keeps a run's jobs, spend and hours bounded).
 
 Trust boundary:
 - A worker's four-line report is a claim; nomArmy's verified git record and independent verification are the evidence. A job isn't complete if its report is missing or malformed, its STATUS is partial or blocked, STATUS done lacks VERIFICATION pass, or its changes aren't committed by nomArmy.
 - Read the diff of anything material before integrating it. nomArmy commits on the worker's branch and never merges into yours: integration, conflicts and pushes are yours.
-- To finish a job that came back partial, blocked or failing verification, dispatch the correction with continue_from: <that job id>. The new job starts with its unfinished work in place and verifies the whole. Don't fix and commit a worker's files yourself: that lands them unverified. If you must, commit them on a branch, run mode: verify on it before building on it, and say so in your report.
+- A job on the wrong track can be stopped with local_worker_stop; it keeps its worktree. To finish a job that came back partial, blocked, failing verification or stopped, dispatch the correction with continue_from: <that job id> (a cheaper model is fine). The new job starts with its unfinished work in place and verifies the whole. Don't fix and commit a worker's files yourself: that lands them unverified. If you must, commit them on a branch, run mode: verify on it before building on it, and say so in your report.
 - Failed and incomplete worktrees are kept for review; clean up with local_worker_cleanup or local_worker_sweep once you've decided.
 
 Never delegate deployments, production access, cloud or SSH credentials, secrets, Terraform state or kubectl contexts to a worker.`;

package/lib/execute.mjs

@@ -1,4 +1,4 @@
-import { writeStatus, shouldRetryTransientAbort, shouldAttemptScoutRecovery } from "./openclaw-run.mjs";
+import { writeStatus, shouldRetryTransientAbort, shouldAttemptScoutRecovery, readStopRequest } from "./openclaw-run.mjs";
 import fs from "node:fs";
 import path from "node:path";
 import { fileURLToPath } from "node:url";
@@ -9,7 +9,10 @@ import { parseScoutReport, verifyCitations, resolveScoutOutcome, renderScoutRepo
 import { parseDecomposeReport, buildDecomposeFindings, resolveDecomposeOutcome, checkDecompositionOverlap, renderDecomposeReport } from "./decompose.mjs";
 import { deriveTimeBudget } from "./budget.mjs";
 import { continuationProblem, continuationBase, snapshotRetainedWork, applyRetainedWork, continuationNote } from "./continue-from.mjs";
-import { estimateDisplacement } from "./transcript.mjs";
+import { checkScoutCitations, checkReportClaims } from "./jev-checks.mjs";
+import { runJudge } from "./judge.mjs";
+import { pickMutants, runMutants, describeSurvivors } from "./mutation.mjs";
+import { estimateDisplacement, readOpenClawTranscript } from "./transcript.mjs";
 import { outlineFile, findReferences } from "./repo-query.mjs";
 import { loadConfig } from "./config.mjs";
 import { linkNodePackages, nodeModulesState, repairHostInstalls } from "./sandbox-images.mjs";
@@ -18,7 +21,7 @@ import { describeRecoveryChanges, reportRecoveryPrompt } from "./worker-prompt.m
 import { parseWorkerReport } from "./report.mjs";
 import { OUTCOMES, COORDINATOR_STATUS_BY_OUTCOME } from "./outcomes.mjs";
 import { resolveOutcome, finalText, workerMetadata, applyRefactorContract, applyVerificationPolicy } from "./outcome.mjs";
-import { isTestPath, isDocumentationPath, detectScopedTestSelectionRisk, detectUnwiredNewDefinitions, detectMislabeledTestNames, detectPossibleSecrets, detectVerificationInputChanges } from "./diff-checks.mjs";
+import { parseAddedLineNumbers, isTestPath, isDocumentationPath, detectScopedTestSelectionRisk, detectUnwiredNewDefinitions, detectMislabeledTestNames, detectPossibleSecrets, detectVerificationInputChanges } from "./diff-checks.mjs";
 
 // ---------------------------------------------------------------------------
 // Job status for polling. `status.json` is written at every phase transition
@@ -36,6 +39,8 @@ export function createExecutor(deps) {
     resolveReasoningApplied, recordedBudgets, runOpenClaw, sweepStaleSandboxContainers,
     verificationFlow, normalizeVerification, runIndependentVerification,
     runRegressionCheck, repoPolicy } = deps;
+  // Optional Jev checks (lib/validators.mjs): the server wires the settings; without them (tests), none run.
+  const jevSettingsFor = (check) => { try { const s = deps.jevSettings?.(); return s?.checks?.includes(check) ? s : null; } catch { return null; } };
 
   async function executeJob({ task, acceptance, verification, mode = "implement", baseRef, timeoutSeconds = 600, profile = "coder", reasoning = "high", pool = null, subscriptionWorker = null, onBehalfOf = null, model = null, reportSize = null, workerId, evidence = null, verifyRegression = false, commitSubject = null, refactor = false, continueFrom = null, jobId: presetJobId = null }) {
     await assertRepo();
@@ -270,6 +275,9 @@ export function createExecutor(deps) {
         // failed job that changed nothing was recorded "pass" (a Senti run),
         // which reads as evidence about work that never happened.
         independentVerification = normalizeVerification({ status: "not_run", basis: "not-applicable", reason: "the worker changed nothing, so there was none of its work to verify" }, verification ?? null);
+      } else if (workerStopReason === "stopped") {
+        // Stopped on request: end now, without spending time on tests.
+        independentVerification = normalizeVerification({ status: "not_run", basis: "not-applicable", reason: "the job was stopped on request" }, verification ?? null);
       } else if (verificationFlow.verificationRunner || !reportValidation.valid) {
         independentVerification = await runIndependentVerification({ profile: verification ?? null, cwd, jobId, baseSha: base.sha, branch, mode, record: preCommit, logFile: path.join(jobDir, "verification.log") });
       }
@@ -300,6 +308,36 @@ export function createExecutor(deps) {
         if (regressionCheck.status === "restore_failed") regressionCheckFatal = true;
       }
 
+      // Mutation testing (lib/mutation.mjs), when this repo opts in: small
+      // mistakes planted one at a time in the changed lines must each fail
+      // the same profile. Survivors raise review; a failed restore is fatal.
+      let mutation = null, mutationElapsedMs = null;
+      const mutationConfig = (() => { try { return loadConfig(projectDir)?.config?.mutation ?? null; } catch { return null; } })();
+      if (mutationConfig && verification && independentVerification.status === "pass" && codeFilesChanged.length > 0 && !regressionCheckFatal) {
+        const mutationStartedMs = Date.now();
+        try {
+          const untracked = new Set((preCommit.nameStatus ?? []).filter((e) => e.untracked).map((e) => e.path));
+          const deleted = new Set((preCommit.nameStatus ?? []).filter((e) => /^D/.test(e.status)).map((e) => e.path));
+          const files = [];
+          for (const file of codeFilesChanged.filter((f) => !deleted.has(f))) {
+            const full = path.join(cwd, file);
+            let lines;
+            if (untracked.has(file)) { try { lines = fs.readFileSync(full, "utf8").split("\n").map((_, i) => i + 1); } catch { continue; } }
+            else lines = parseAddedLineNumbers(await gitRaw(["diff", "-U0", base.sha, "--", file], cwd));
+            if (lines.length) files.push({ path: file, full, lines });
+          }
+          const mutants = pickMutants(files, mutationConfig.mutants);
+          let n = 0;
+          mutation = await runMutants({ mutants, deadlineMs: mutationStartedMs + mutationConfig.max_seconds * 1000,
+            verify: () => runIndependentVerification({ profile: verification, cwd, jobId: `${jobId}-mutant-${++n}`, baseSha: base.sha, branch, mode, record: preCommit }) });
+          mutation.planned = mutants.length;
+        } catch (error) {
+          mutation = { status: "not_run", killed: 0, survived: [], inconclusive: 0, tried: 0, reason: `mutation testing failed to run: ${error.message}` };
+        }
+        mutationElapsedMs = Date.now() - mutationStartedMs;
+        fs.appendFileSync(path.join(jobDir, "coordinator.log"), `${new Date().toISOString()} mutation testing: ${mutation.killed ?? 0} killed, ${mutation.survived?.length ?? 0} survived, ${mutation.inconclusive ?? 0} inconclusive of ${mutation.tried ?? 0} tried (${Math.round(mutationElapsedMs / 1000)}s)\n`);
+      }
+
       // resolveOutcome's own contract only ever sees pass/fail/not_run for
       // regressionCheck -- a restore_failed status is substituted to not_run
       // here so resolveOutcome never needs a fourth value; the hard override
@@ -427,21 +465,67 @@ export function createExecutor(deps) {
             commitBlockedReason: `possible secret detected: ${possibleSecrets.reason}`,
             reasons: [...afterHostInstalls.reasons, `POSSIBLE SECRET DETECTED: ${possibleSecrets.reason}`] }
         : afterHostInstalls;
+      // Jev: does the worker's report match its diff? Found live: a note said
+      // "restored check.js to base commit" while the diff rewrote check.js.
+      // Only raises review; it never blocks or allows a commit.
+      let jevClaims = null, judged = null;
+      const jevImplement = jevSettingsFor("report-claims");
+      const judge = (() => { try { return deps.judgeSettings?.() ?? null; } catch { return null; } })();
+      // The whole change, new files included, for whichever validators run.
+      const jobDiff = async () => {
+        let diff = await gitRaw(["diff", base.sha, "--"], cwd);
+        for (const entry of (preCommit.nameStatus ?? []).filter((e) => e.untracked)) {
+          let text = "";
+          try { text = fs.readFileSync(path.join(cwd, entry.path), "utf8"); } catch { continue; }
+          diff += `\ndiff --git a/${entry.path} b/${entry.path}\nnew file\n--- /dev/null\n+++ b/${entry.path}\n${text.split("\n").map((l) => `+${l}`).join("\n")}\n`;
+        }
+        return diff;
+      };
+      let diffText = null;
+      if (mode === "implement" && reportValidation?.valid && repositoryChanged && (jevImplement || (judge && !judge.problem))) {
+        try { diffText = await jobDiff(); } catch { diffText = null; }
+      }
+      if (jevImplement && diffText != null) {
+        try { jevClaims = await checkReportClaims({ report: reportValidation, diff: diffText, settings: jevImplement }); }
+        catch (error) { jevClaims = { flag: null, verdict: null, error: error.message, usage: 0 }; }
+      }
+      // The model judge (lib/judge.mjs): acceptance criteria, the report and
+      // changed tests. Only raises review, never blocks or allows a commit.
+      if (judge?.problem) judged = { flags: [], answer: null, error: `judge not run: ${judge.problem}`, skipped: true };
+      else if (judge && diffText != null) {
+        const modifiedTests = preCommit.testChanges?.existing_tests_modified ?? [];
+        let testDiff = "";
+        if (modifiedTests.length) { try { testDiff = await gitRaw(["diff", base.sha, "--", ...modifiedTests], cwd); } catch { testDiff = ""; } }
+        judged = await runJudge({ settings: judge, task, acceptance: acceptance ?? [], report: reportValidation, diff: diffText, testDiff, stateRoot: path.join(jobsRoot, "..") });
+      }
+      const afterJevOnly = jevClaims?.flag
+        ? { ...afterSecrets, reviewRequired: true, reasons: [...afterSecrets.reasons, `REPORT MAY NOT MATCH THE DIFF (Jev, ${jevClaims.flag.probability.toFixed(2)}): the report says "${String(reportValidation.note ?? "").slice(0, 200)}", and the diff may show otherwise. Read the diff before accepting.`] }
+        : afterSecrets;
+      const afterJev = judged?.flags?.length
+        ? { ...afterJevOnly, reviewRequired: true, reasons: [...afterJevOnly.reasons, `JUDGE (${judge.agent}/${judge.model}): ${judged.flags.join("; ")}. Read the diff before accepting.`] }
+        : afterJevOnly;
+
       // A worker must not be judged by a check it rewrote: a changed script,
       // Makefile or package.json script that a verification command runs
       // blocks the commit; changed test-runner config only asks for review.
       const blockedInputs = verificationInputs?.blocked ?? [], flaggedInputs = verificationInputs?.flagged ?? [];
       const inputLine = blockedInputs.map((b) => `${b.file} (${b.why})`).join("; ");
       const afterInputs = blockedInputs.length
-        ? { ...afterSecrets, outcome: afterSecrets.commitAllowed || afterSecrets.outcome === OUTCOMES.WORKER_DONE ? OUTCOMES.NEEDS_REVIEW : afterSecrets.outcome,
+        ? { ...afterJev, outcome: afterJev.commitAllowed || afterJev.outcome === OUTCOMES.WORKER_DONE ? OUTCOMES.NEEDS_REVIEW : afterJev.outcome,
             reviewRequired: true, commitAllowed: false,
-            commitBlockedReason: afterSecrets.commitAllowed ? `the diff changes what verification runs: ${inputLine}` : afterSecrets.commitBlockedReason,
-            reasons: [...afterSecrets.reasons, `VERIFICATION INPUT CHANGED: the diff changes what profile '${verification}' runs, so its result can't be trusted: ${inputLine}`] }
-        : afterSecrets;
+            commitBlockedReason: afterJev.commitAllowed ? `the diff changes what verification runs: ${inputLine}` : afterJev.commitBlockedReason,
+            reasons: [...afterJev.reasons, `VERIFICATION INPUT CHANGED: the diff changes what profile '${verification}' runs, so its result can't be trusted: ${inputLine}`] }
+        : afterJev;
       const afterConfig = flaggedInputs.length
         ? { ...afterInputs, reviewRequired: true, reasons: [...afterInputs.reasons, `TEST CONFIG CHANGED: ${flaggedInputs.map((c) => `${c.file} (${c.why})`).join("; ")}`] }
         : afterInputs;
-      const finalOutcome = applyRefactorContract(applyVerificationPolicy(afterConfig, independentVerification.status, repoPolicy()),
+      const afterMutation = mutation?.status === "restore_failed"
+        ? { ...afterConfig, outcome: OUTCOMES.NEEDS_REVIEW, reviewRequired: true, commitAllowed: false,
+            commitBlockedReason: `mutation testing could not restore the worker's file: ${mutation.reason}`, reasons: [...afterConfig.reasons, `MUTATION RESTORE FAILED: ${mutation.reason}`] }
+        : mutation?.status === "survivors"
+          ? { ...afterConfig, reviewRequired: true, reasons: [...afterConfig.reasons, describeSurvivors(mutation, verification)] }
+          : afterConfig;
+      const finalOutcome = applyRefactorContract(applyVerificationPolicy(afterMutation, independentVerification.status, repoPolicy()),
         { refactor, verificationStatus: independentVerification.status, testChanges: preCommit.testChanges });
 
       progress("commit");
@@ -452,8 +536,13 @@ export function createExecutor(deps) {
 
       let coordinatorStatus = COORDINATOR_STATUS_BY_OUTCOME[finalOutcome.outcome] ?? "incomplete";
       const issues = [...finalOutcome.reasons, ...(preCommit.issues ?? [])];
-      if (workerError) issues.push(`worker error: ${String(workerError).split("\n")[0]}`);
+      if (workerStopReason === "stopped") {
+        const request = readStopRequest(jobDir);
+        issues.push(`stopped on request${request?.reason ? `: ${request.reason}` : ""}; the worktree is kept, so continue_from can pick the work up (on another model too)`);
+      } else if (workerError) issues.push(`worker error: ${String(workerError).split("\n")[0]}`);
       if ((result ?? attempted)?.salvaged) issues.push(`runner cleanup failed after the run (${(result ?? attempted).salvagedFrom}); the worker's report was recovered from the run's transcript`);
+      if (jevClaims?.error) issues.push(`Jev report check skipped (${jevClaims.error}); this job's result doesn't depend on it`);
+      if (judged?.error) issues.push(`Judge ${judged.skipped ? "skipped" : "didn't answer"} (${judged.error}); this job's result doesn't depend on it`);
       if (workerFailed || workerTimedOut) { const restarted = vmRestartIssue(vmStartedBefore, deps.podmanVmStartedAt?.() ?? null); if (restarted) issues.unshift(restarted); }
       if (repositoryChanged && !commit.created) {
         if (coordinatorStatus === "complete") coordinatorStatus = "incomplete";
@@ -464,7 +553,7 @@ export function createExecutor(deps) {
         // the diffstat right in the issue a caller actually reads -- not just
         // buried in the full manifest's git record -- is what makes "go look at
         // the worktree" worth doing instead of discarding the job.
-        issues.push(`repository changes remain uncommitted (${record.filesChanged} file(s), +${record.additions}/-${record.deletions}): ${commit.reason}`);
+        if (record.filesChanged > 0) issues.push(`repository changes remain uncommitted (${record.filesChanged} file(s), +${record.additions}/-${record.deletions}): ${commit.reason}`);
       }
       const failures = worker.toolSummary?.failures ?? 0; if (failures > 0) issues.push(`worker recorded ${failures} tool failure(s)`);
       if (record.ignoredRuntimeJunk.length) issues.push(`runtime junk ignored: ${record.ignoredRuntimeJunk.join(", ")}`);
@@ -483,6 +572,8 @@ export function createExecutor(deps) {
       const metrics = buildMetrics({ result: result ?? attempted, record, reportValidation, outcome: finalOutcome, workerElapsedMs, totalElapsedMs: Date.now() - jobStartedMs, regressionCheckElapsedMs, transientAbortRetried });
       const manifest = { version: VERSION, jobId, workerId: workerId || jobId, mode, projectDir, worktree, branch, startedAt, finishedAt,
         objective: task, acceptance: acceptance ?? [], verificationProfile: verification ?? null, ...(continuedFrom ? { continuedFrom } : {}),
+        ...(mutation ? { mutation: { ...mutation, elapsedSeconds: Math.round((mutationElapsedMs ?? 0) / 1000) } } : {}),
+        ...(jevClaims || judged ? { validators: { ...(jevClaims ? { jev: { check: "report-claims", verdict: jevClaims.verdict, flagged: Boolean(jevClaims.flag), error: jevClaims.error, truncated: Boolean(jevClaims.truncated), inputTokens: jevClaims.usage } } : {}), ...(judged ? { judge: { agent: judge?.agent ?? null, model: judge?.model ?? null, answer: judged.answer, flags: judged.flags, error: judged.error } } : {}) } } : {}),
         outcome: finalOutcome.outcome, recovered: finalOutcome.recovered, recoveryAttempted: finalOutcome.recoveryAttempted,
         reportRecoveryAttempted, reportRecovered,
         reviewRequired: finalOutcome.reviewRequired || record.testChanges.reviewRequired,
@@ -584,12 +675,18 @@ export function createExecutor(deps) {
       const remainingSeconds = timeoutSeconds - Math.round(workerElapsedMs / 1000);
       if (shouldAttemptScoutRecovery({ workerFailed, workerTimedOut, report, remainingSeconds })) {
         reportRecoveryAttempted = true;
+        // What the first run left, in case the follow-up can't see its session.
+        let filesRead = [], earlierReply = reportText;
+        try {
+          const first = await readOpenClawTranscript(path.join(runtimeDir, "state"));
+          if (first.available) { filesRead = first.filesRead ?? []; earlierReply = earlierReply || first.lastAssistantText || ""; }
+        } catch { /* the reply alone still helps */ }
         try {
           const recoveryResult = await runOpenClaw({
             task, acceptance, verification: null, mode, cwd: worktree, baseRef: base.ref, baseSha: base.sha,
             timeoutSeconds: remainingSeconds, runtimeDir, profile, reasoning, pool, subscriptionWorker, onBehalfOf, model, reportSize, jobDir, workerId: workerId || jobId,
             evidenceTool: evidencePlaced ? evidenceTool : null,
-            overridePrompt: scoutReportRecoveryPrompt({ report: used.report.scout, question: task, acceptance }), logSuffix: "-recovery",
+            overridePrompt: scoutReportRecoveryPrompt({ report: used.report.scout, question: task, acceptance, earlierReply, filesRead }), logSuffix: "-recovery",
           });
           const recoveryReport = parseScoutReport(finalText(recoveryResult), (recoveryResult?.budgetsUsed ?? used).scout);
           if (!isScoutReportUnusable(recoveryReport)) {
@@ -609,7 +706,16 @@ export function createExecutor(deps) {
       const dirty = record.repoStatusFiles.length > 0;
       const readFile = async p => { try { return await gitRaw(["show", `${base.sha}:${p}`], projectDir); } catch { return null; } };
       const verified = await verifyCitations(report.findings, { readFile, limits: used.scout });
+      // Jev: do the cited lines support each finding? Only adds flags.
+      let jevCitations = null;
+      const jevScout = jevSettingsFor("scout-citations");
+      if (jevScout && verified?.findings?.length) {
+        try { jevCitations = await checkScoutCitations({ findings: verified.findings, settings: jevScout, readFile }); }
+        catch (error) { jevCitations = { flags: [], checked: 0, errors: [error.message], usage: 0, verdicts: [] }; }
+        for (const f of jevCitations.flags) verified.findings[f.index].jev = { verdict: f.verdict, probability: f.probability };
+      }
       const outcome = resolveScoutOutcome({ report, verified, workerFailed, workerTimedOut, dirty });
+      if (jevCitations?.flags.length) outcome.reviewRequired = true;
 
       progress("record");
       if (outcome.retainWorktree) worktreeRetained = true;
@@ -621,6 +727,8 @@ export function createExecutor(deps) {
       if ((result ?? attempted)?.salvaged) issues.push(`runner cleanup failed after the run (${(result ?? attempted).salvagedFrom}); the scout's report was recovered from the run's transcript`);
       const failures = worker.toolSummary?.failures ?? 0; if (failures > 0) issues.push(`scout recorded ${failures} tool failure(s)`);
       if (dirty) issues.push(`snapshot changed: ${record.repoStatusFiles.join(", ")}`);
+      if (jevCitations?.flags.length) issues.push(`CITATIONS MAY NOT SUPPORT FINDINGS (Jev): ${jevCitations.flags.map((f) => `"${String(verified.findings[f.index].text).slice(0, 80)}${String(verified.findings[f.index].text).length > 80 ? "..." : ""}" (${f.verdict}, ${f.probability.toFixed(2)})`).join("; ")}. Read those cited lines before relying on them; they're marked [JEV] in the report.`);
+      if (jevCitations?.errors.length) issues.push(`Jev citation check skipped or incomplete (${jevCitations.errors.join("; ")}); this job's result doesn't depend on it`);
       if (reportRecoveryAttempted) {
         issues.push(reportRecovered
           ? "scout report recovered via a follow-up call after the first reply was cut off"
@@ -658,6 +766,7 @@ export function createExecutor(deps) {
       };
       const manifest = { version: VERSION, jobId, workerId: workerId || jobId, mode, projectDir, worktree: worktreeRetained ? worktree : null, branch: null, baseSha: base.sha, startedAt, finishedAt,
         objective: task, mustCover: acceptance ?? [],
+        ...(jevCitations ? { validators: { jev: { check: "scout-citations", checked: jevCitations.checked, flags: jevCitations.flags, errors: jevCitations.errors, inputTokens: jevCitations.usage } } } : {}),
         outcome: outcome.outcome, coordinatorStatus: outcome.coordinatorStatus, reviewRequired: outcome.reviewRequired, issues,
         scout: { question: report.question, confidence: report.confidence, notFound: report.notFound,
           findings: verified.findings, supported: verified.supported, unsupported: verified.unsupported, weak: verified.weak,