@surlymochan/evalid 0.5.13 → 0.5.15

package/CHANGELOG.md

@@ -3,7 +3,23 @@
 All notable changes to this project are documented here. Detailed notes live in
 [RELEASE_NOTES.md](RELEASE_NOTES.md).
 
-## [Unreleased]
+## [0.5.15] - 2026-11-09
+
+### Added
+- **`xflow audit` standalone mode (#78, M2)**: verify any git diff / PR without adopting evalid workflows — `xflow audit --pr <n>` / `--diff <patch> --base <ref>` / `--base <ref> --head <ref>` extracts the test runs the change claims to have passed (English and Chinese claim phrasing, per-file counts summed for multi-file commands, clause-split so one line's two claims don't leak onto each other), replays each in a clean clone at the change's head (deps installed with `npm ci` in the scratch copy, `NODE_TEST_*` env stripped so a nested `node --test` really runs), and compares exit codes and pass/fail counts against the claim. Diffs carrying an evalid proof seal are replayed through the existing i6b validator (seal semantics untouched). Verdicts: `verified` / `mismatch` / `no_claims` / `unverifiable` (exit 0/1/3/1). Test evidence: 11 new tests in `test/xflow-audit.test.js` (claim extraction incl. #75/#68 bodies, diff/seal parsing, replay-count parsing, fixture-repo verified/mismatch/no_claims end-to-end) 11/11 pass; local full `npm test` 475 tests / 474 pass / 0 fail / 1 skipped (pre-existing); PR CI runs ledger 42516967938 and verify 42516967984 both success.
+
+Milestone M2 "`xflow audit` standalone mode" acceptance evidence (issue #78, comment 6726208145): audits of real historical PRs #75 / #71 / #68 all `verified` (targeted run 24 passed / 0 failed / 24 total plus seal `valid` on #75; full `npm test` exit 0 on all three; #68's claimed 439/0/440 honestly counts-unparsed — multi-file runner has no single total).
+
+Acceptance debt, stated plainly: the bench-number acceptance for P1-2 / P1-3 / P2-1 / P2-3 still awaits the MiniMax full-matrix batch (15 instances × 4 arms × reps 3), which is stopped on MiniMax Token Plan quota exhaustion. No bench numbers are claimed in this release. Audit gaps carried forward: no PR from a different repo or a non-evalid agent audited yet; pytest / go test / cargo test replay paths are parser-level only.
+
+## [0.5.14] - 2026-11-09
+
+### Added
+- **PR comment ledger (#75)**: the proof-ledger workflow now posts the claim-vs-evidence ledger as a PR comment (`action/ledger-comment.mjs`, `.github/workflows/proof-ledger.yml`). A green proof gets a ✅ comment naming the verified change; a tampered proof fails the `ledger` check and gets a ❌ comment naming the exact break point, so a replay mismatch can no longer pass CI silently. Test evidence: 18 new tests in `test/github-action-ledger-comment.test.js` 18/18 pass; local full `npm test` 464 tests / 463 pass / 0 fail / 1 skipped (pre-existing); PR CI run 42505482112 success.
+
+Milestone M1 "GitHub Action end-to-end" acceptance evidence (issue #73, comment 6724800077): ① green path — PR #75 merged af88e2da, proof-ledger run 42505482067 success, ✅ ledger comment 6724656930; ② red path — demo PR #76 (closed unmerged by plan) turned the `ledger` check red (run 42505769769) and the ❌ comment 6724768829 names the break point `green_tree_mismatch: test/demo-tamper.test.js`; ③ CI replay and local replay agree on both verdicts (`valid` green, `green_tree_mismatch` tampered) through the same validator entry.
+
+Acceptance debt, stated plainly: the bench-number acceptance for P1-2 / P1-3 / P2-1 / P2-3 still awaits the MiniMax full-matrix batch (15 instances × 4 arms × reps 3), which is stopped on MiniMax Token Plan quota exhaustion. No bench numbers are claimed in this release.
 
 ## [0.5.13] - 2026-11-09
 

package/README.md

@@ -149,6 +149,88 @@ node bin/xflow.js compare codex-goal
 node bin/xflow.js launch copy
 ```
 
+## GitHub Action: verify proofs in CI
+
+evalid ships as a GitHub Action (`action.yml` at the repo root). Add it to a
+workflow to verify a sealed TDD proof on every PR: a valid proof keeps the
+check green, a broken seal (or a test edited after the seal) turns it red —
+and the claim-vs-evidence ledger is posted as a PR comment either way, so
+the reviewer sees the claim next to the replay evidence without opening
+the workflow log.
+
+```yaml
+name: Proof ledger
+on:
+  pull_request:
+    types: [opened, synchronize, reopened]
+permissions:
+  contents: read
+  pull-requests: write
+jobs:
+  ledger:
+    runs-on: ubuntu-24.04
+    steps:
+      - uses: actions/checkout@v7.0.1
+        with:
+          fetch-depth: 0  # replay materializes files from the recorded HEAD
+      - uses: actions/setup-node@v7.1.0
+        with:
+          node-version: '22'
+      - uses: surlymochan/evalid@main  # or a release tag (see CHANGELOG)
+        with:
+          change-id: my-change
+          phase: green
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+```
+
+Inputs: `change-id` (required), `phase` (`green`, default), `attempt` (`0`,
+default), `project-root` (`.`, default), `tdd-mode` (auto-detect, default),
+`post-comment` (`true`, default — set `false` to skip the comment),
+`github-token` (defaults to `$GITHUB_TOKEN`), `pr-number` (auto-detects
+from the event payload or ref when empty).
+
+The comment is upserted on a stable marker
+(`<!-- evalid-ledger: <change-id>/<phase>/<attempt> -->`), so re-runs
+update the same comment instead of posting a new one per push. A failed
+comment post never masks the verdict: the step still exits on the proof
+result, and `comment_posted` / `comment_url` are exposed as outputs.
+The check fails (exit 1) on `binding_mismatch`, `snapshot_mismatch`,
+`green_tree_mismatch`, replay mismatches, or a missing proof; the comment
+names the breakage point (for example the test files that drifted after
+the seal).
+
+Local equivalent of the CI replay — same validator, same entry:
+
+```bash
+node action/verify-proof.mjs --project-root . --change-id my-change --phase green
+```
+
+This repo dogfoods the Action on its own PRs via
+`.github/workflows/proof-ledger.yml`: it discovers every sealed proof
+added or changed by the PR (`specs/changes/*/tdd/*.json`) and verifies
+each one.
+
+## `xflow audit`: verify any PR, no evalid workflow required
+
+`xflow audit` checks a change you did *not* produce with evalid: point it
+at any git diff or PR — Claude Code, Codex, or any other agent's output —
+and it extracts the test runs the change claims to have passed, replays
+each one in a clean clone at the change's head, and compares exit codes
+and pass/fail counts against the claim. If the diff carries an evalid
+proof seal, the seal is replayed through the existing i6b validator.
+
+```bash
+xflow audit --pr 75                        # audit a GitHub PR (body = claims)
+xflow audit --diff change.patch --base HEAD~1 --claims-file claims.txt
+xflow audit --base main --head feature/x --claims "npm test: exit 0"
+```
+
+Verdicts: `verified` (every claim replayed as claimed), `mismatch` (a
+replay contradicts a claim, or a seal is invalid), `no_claims` (no test
+claims found in the supplied text), `unverifiable` (a claimed command
+could not run). Exit codes are 0 / 1 / 3 / 1 respectively (2 = misuse).
+`--json` prints the machine-readable report; `--out <file>` writes it.
+
 ## Commands
 
 The installed binary is `xflow` (`npm install -g evalid` installs the `xflow` CLI). Every
@@ -160,6 +242,7 @@ top-level command below was verified against `xflow <command> --help` at v0.5.7;
 | `xflow workflow run \| validate <yaml>` | Run a workflow (`yolo` / `corps` / a YAML path), or validate its YAML as preflight |
 | `xflow corps` | Run the heavy product/multi-agent workflow through its governed entry point |
 | `xflow proof` | Verify workflow completion evidence and write the proof file |
+| `xflow audit` | Audit any git diff / PR: replay claimed test runs, verdict |
 | `xflow doctor` | Check local xflow readiness: package scripts, workflow YAMLs, skill topology |
 | `xflow init` | Create project-local `.as-xflow/config.json` with portable defaults |
 | `xflow goal show \| set \| audit \| clear` | Manage the project-level `.xflow/GOAL.md` anchor |

package/RELEASE_NOTES.md

@@ -1,5 +1,25 @@
 # Release Notes
 
+## 0.5.15 - xflow audit standalone mode (milestone M2 accepted)
+
+Patch release. One new feature since 0.5.14, verified end to end in CI and locally.
+
+**`xflow audit` standalone mode (#79).** Verify any git diff / PR without adopting evalid workflows: `xflow audit --pr <n>` extracts the test runs the change claims to have passed, replays each one in a clean clone at the change's head, and compares exit codes and pass/fail counts against the claim; diffs carrying an evalid proof seal are replayed through the existing i6b validator. Test evidence: 11 new tests in `test/xflow-audit.test.js` 11/11 pass; local full `npm test` 475 tests / 474 pass / 0 fail / 1 skipped (pre-existing); PR CI runs ledger 42516967938 and verify 42516967984 both success.
+
+**Milestone M2 "`xflow audit` standalone mode" — accepted.** Evidence in issue #78, comment 6726208145: audits of real historical PRs #75 / #71 / #68 all `verified` — #75 targeted run 24 passed / 0 failed / 24 total plus seal `valid`, full `npm test` exit 0 on all three; #68's claimed 439/0/440 honestly counts-unparsed (the multi-file runner has no single total, so the verdict rests on exit code).
+
+Acceptance debt, stated plainly: the bench-number acceptance for P1-2, P1-3, P2-1 and P2-3 — the wall-clock/token/resolved lines from issue #5 — still awaits the MiniMax full-matrix batch (15 instances × 4 arms × reps 3), which is stopped on MiniMax Token Plan quota exhaustion. No bench numbers are claimed in this release. Audit gaps carried forward: no PR from a different repo or a non-evalid agent audited yet; pytest / go test / cargo test replay paths are parser-level only.
+
+## 0.5.14 - PR comment ledger (milestone M1 accepted)
+
+Patch release. One new feature since 0.5.13, verified end to end in CI and locally.
+
+**PR comment ledger (#75).** The proof-ledger workflow now posts the claim-vs-evidence ledger as a PR comment: a green proof gets a ✅ comment on the PR, and a tampered proof fails the `ledger` check and gets a ❌ comment naming the exact break point, so a replay mismatch can no longer pass CI silently. Test evidence: 18 new tests in `test/github-action-ledger-comment.test.js` 18/18 pass; local full `npm test` 464 tests / 463 pass / 0 fail / 1 skipped (pre-existing); PR CI run 42505482112 success.
+
+**Milestone M1 "GitHub Action end-to-end" — accepted.** Evidence in issue #73, comment 6724800077: ① green path — PR #75 merged (af88e2da), proof-ledger run 42505482067 success, ✅ ledger comment 6724656930; ② red path — demo PR #76 (closed unmerged by plan) turned the `ledger` check red (run 42505769769) and the ❌ comment 6724768829 names the break point `green_tree_mismatch: test/demo-tamper.test.js`; ③ CI replay and local replay agree on both verdicts (`valid` green, `green_tree_mismatch` tampered) through the same validator entry.
+
+Acceptance debt, stated plainly: the bench-number acceptance for P1-2, P1-3, P2-1 and P2-3 — the wall-clock/token/resolved lines from issue #5 — still awaits the MiniMax full-matrix batch (15 instances × 4 arms × reps 3), which is stopped on MiniMax Token Plan quota exhaustion. No bench numbers are claimed in this release.
+
 ## 0.5.13 - Proof-seal verification action
 
 Patch release. One new feature and one dead-code removal since 0.5.12, both verified by the test suite.

package/bin/xflow.js

@@ -7,6 +7,7 @@
  *   workflow validate <yaml>     Validate workflow YAML against schema
  *   corps                        Run the heavy corps workflow through its governed entry
  *   proof                        Verify workflow completion evidence and write proof
+ *   audit                        Audit any git diff / PR: replay claimed test runs, verdict
  *   doctor                       Check local xflow readiness
  *   evaluate                     Show one-shot external evaluator brief
  *   release status               Show release-owner publish status
@@ -40,6 +41,7 @@ import { spawnSync } from 'child_process';
 import { createHash } from 'crypto';
 import { tmpdir } from 'os';
 import { ackPendingGate } from '../src/core/pending-gates.js';
+import { runAudit, renderAuditReport } from '../src/core/audit.js';
 import { exportDomRectsToFile, resolveCliPath } from '../src/agent_team/atoms/visual/dom-rect-export.js';
 import { capturePageEvidence } from '../src/agent_team/atoms/visual/page-evidence-capture.js';
 import { computeImageDiffMetrics } from '../src/agent_team/atoms/visual/image-diff.js';
@@ -83,6 +85,9 @@ async function main() {
     case 'proof':
       await handleProof([subcommand, ...rest].filter(Boolean));
       break;
+    case 'audit':
+      await handleAudit([subcommand, ...rest].filter(Boolean));
+      break;
     case 'gate':
       await handleGate(subcommand, rest);
       break;
@@ -5742,6 +5747,77 @@ async function handleProof(args) {
   if (!proof.ok) process.exit(1);
 }
 
+// ─── audit ───────────────────────────────────────────────────────────────────
+// Standalone audit (M2, #78): verify any git diff / PR without adopting
+// evalid workflows. Claims are extracted from the PR body / claims text,
+// replayed in a clean clone at the PR head, and compared against the claim;
+// diffs carrying an evalid proof seal are replayed through the i6b validator
+// (see src/core/audit.js).
+async function handleAudit(args) {
+  if (args.some(isHelpRequest)) {
+    printAuditHelp();
+    return;
+  }
+  const parsedArgs = parseCliArgs(args);
+  const repoDir = resolve(parsedArgs.repo || process.cwd());
+  const timeoutMs = parsedArgs.timeout ? Number(parsedArgs.timeout) * 1000 : 300000;
+  let diffText = '';
+  let claimsText = '';
+  let baseRef = parsedArgs.base || null;
+  let headRef = parsedArgs.head || null;
+
+  if (parsedArgs.pr) {
+    const prNumber = String(parsedArgs.pr);
+    const view = spawnSync('gh', ['pr', 'view', prNumber, '--json', 'body,baseRefOid,mergeCommit,headRefOid'], { cwd: repoDir, encoding: 'utf8' });
+    if (view.status !== 0) {
+      console.error(`xflow audit: gh pr view ${prNumber} failed: ${(view.stderr || view.stdout || '').trim().slice(-300)}`);
+      process.exit(2);
+    }
+    let pr;
+    try {
+      pr = JSON.parse(view.stdout);
+    } catch {
+      console.error('xflow audit: could not parse gh pr view output');
+      process.exit(2);
+    }
+    claimsText = pr.body || '';
+    baseRef = pr.baseRefOid || null;
+    headRef = pr.mergeCommit?.oid || pr.headRefOid || null;
+    const diff = spawnSync('gh', ['pr', 'diff', prNumber], { cwd: repoDir, encoding: 'utf8' });
+    if (diff.status !== 0) {
+      console.error(`xflow audit: gh pr diff ${prNumber} failed: ${(diff.stderr || diff.stdout || '').trim().slice(-300)}`);
+      process.exit(2);
+    }
+    diffText = diff.stdout || '';
+  } else {
+    if (parsedArgs.diff) {
+      diffText = parsedArgs.diff === '-'
+        ? readFileSync(0, 'utf8')
+        : readFileSync(resolve(parsedArgs.diff), 'utf8');
+    }
+    if (parsedArgs.claims) claimsText = String(parsedArgs.claims);
+    if (parsedArgs.claims_file) claimsText = readFileSync(resolve(parsedArgs.claims_file), 'utf8');
+    if (!headRef && !baseRef) {
+      console.error('Usage: xflow audit --pr <n> | --diff <patch> --base <ref> | --base <ref> --head <ref>');
+      process.exit(2);
+    }
+  }
+
+  const report = runAudit({
+    repoDir,
+    diffText,
+    baseRef,
+    headRef,
+    claimsText,
+    timeoutMs,
+    keepWorkdir: Boolean(parsedArgs.keep_workdir),
+  });
+  if (parsedArgs.out) writeFileSync(resolve(parsedArgs.out), `${JSON.stringify(report, null, 2)}\n`);
+  if (parsedArgs.json) console.log(JSON.stringify(report, null, 2));
+  else console.log(renderAuditReport(report));
+  process.exit(report.exitCode);
+}
+
 async function buildWorkflowProof({ track, workflowPath, projectRoot, changeId, write = true }) {
   const { load } = await import('../src/core/workflow-loader.js');
   const { readExecutionLog } = await import('../src/core/execution-log.js');
@@ -6654,6 +6730,37 @@ Options:
 `);
 }
 
+function printAuditHelp() {
+  console.log(`
+Usage: xflow audit --pr <n> [--repo <path>] [--json] [--out <file>]
+       xflow audit --diff <patch|-> --base <ref> [--claims-file <file>] [--json]
+       xflow audit --base <ref> --head <ref> [--claims <text>] [--json]
+
+Audit any git diff / PR without adopting evalid workflows: extract the test
+runs the change claims to have passed, replay each one in a clean clone at
+the change's head, and compare exit codes and pass/fail counts against the
+claim. Diffs carrying an evalid proof seal are replayed through the existing
+i6b seal validator. No seal semantics, CLI name, or bin path changes.
+
+Verdicts / exit codes: verified 0, mismatch 1, unverifiable 1, misuse 2,
+no_claims 3 (no test claims found in the supplied text).
+
+Options:
+  --pr N              Audit a GitHub PR (body = claims, merge commit = head)
+  --diff FILE         Unified diff to apply on top of --base ('-' reads stdin)
+  --base REF          Base commit for --diff, or diff base for --head
+  --head REF          Head commit to check out directly (diff not applied)
+  --repo PATH         Local repo to clone from, default: current directory
+  --claims TEXT       Claims text (default: PR body for --pr)
+  --claims-file FILE  Read claims text from a file
+  --timeout SEC       Per-command replay timeout, default: 300
+  --json              Print the machine-readable report
+  --out FILE          Write the JSON report to a file
+  --keep-workdir      Keep the scratch clone for debugging
+  help, -h, --help    Show this help message
+`);
+}
+
 function printProofHelp() {
   console.log(`
 Usage: xflow proof [--track corps] --change-id <id> [--project-root <path>]
@@ -7148,6 +7255,7 @@ Usage:
   xflow workflow validate <yaml> Validate workflow YAML
   xflow corps --title <title>    Run heavy track through governed entry
   xflow proof --track corps      Verify final execution proof
+  xflow audit --pr 75 [--json]   Audit a PR: replay its claimed test runs
   xflow doctor [--json]          Check local xflow readiness
   xflow init [--project-root .]   Create project-local xflow config
   xflow goal set "<goal>"        Write the project-level .xflow/GOAL.md anchor

package/docs/launch-dossier.md

@@ -1,6 +1,6 @@
 # evalid Launch Dossier
 
-Generated: 2026-11-08T16:04:31.497Z
+Generated: 2026-11-08T19:55:19.000Z
 Mode: post_publish
 Verdict: conditionally ready: engine and adoption evidence are strong, registry publication remains required for a splash launch
 Launch ready: not yet
@@ -117,7 +117,7 @@ Evidence:
 - Adoption records: 1; status: pass
 - Goal audit: pass; goal: Keep evalid credible as an open-source agent delivery runtime: repo-owned workflow contracts, executable verification gates, durable project context, and honest benchmark evidence.
 - Package evidence: missing; source: registry_json
-- Package name/version: @surlymochan/evalid@0.5.13
+- Package name/version: @surlymochan/evalid@0.5.15
 
 ## Passing Evidence
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@surlymochan/evalid",
-  "version": "0.5.13",
+  "version": "0.5.15",
   "description": "Claimed != verified: a runtime verification layer for coding agents. It seals each test run and replays it from a clean snapshot, with a zero-judge gate pipeline (spec clarification, code review, TDD proof) and the xflow CLI.",
   "keywords": [
     "ai-agents",

package/src/core/audit.js

@@ -0,0 +1,497 @@
+/**
+ * xflow audit — standalone audit of any git diff / PR, without adopting
+ * evalid workflows.
+ *
+ * An audit answers one question: do the test runs this change *claims* to
+ * have passed actually pass when replayed from a clean copy at the change's
+ * head? It extracts claimed test commands from the PR body / claims text,
+ * replays each command in a scratch clone (base commit + diff applied, or
+ * the PR's merge commit checked out directly), and compares exit codes and
+ * pass/fail counts against the claim. Diffs that carry an evalid proof seal
+ * (specs/changes/<id>/tdd/<phase>-<attempt>.json) are additionally replayed
+ * through the existing i6b seal validator — the seal semantics are reused,
+ * not rebuilt.
+ *
+ * Verdicts:
+ *   verified     — every claim replayed as claimed (and every seal valid)
+ *   mismatch     — a replay contradicts a claim, or a seal is invalid
+ *   no_claims    — no test claims found in the supplied text
+ *   unverifiable — a claimed command could not run (missing runner, timeout)
+ *
+ * Exit codes mirror the verdicts: 0 verified, 1 mismatch/unverifiable,
+ * 2 misuse, 3 no_claims (inconclusive, not a failure of the change).
+ */
+import { spawnSync } from 'node:child_process';
+import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join, resolve } from 'node:path';
+
+/** Commands that count as claimed test runs. */
+export const TEST_COMMAND_PATTERNS = [
+  /npm\s+test\b/,
+  /npm\s+run\s+[\w:.-]+/,
+  /pnpm\s+test\b/,
+  /yarn\s+test\b/,
+  /node\s+--test\b/,
+  /pytest\b/,
+  /python3?\s+-m\s+pytest\b/,
+  /go\s+test\b/,
+  /cargo\s+test\b/,
+  /mvn\s+(?:-[^ ]+\s+)*test\b/,
+  /npx\s+[\w@./-]+/,
+];
+
+const COMMAND_CHARS = String.raw`[\w@./:+\-='"]`;
+const COMMAND_TAIL = new RegExp(
+  String.raw`(?:npm\s+test|npm\s+run\s+[\w:.-]+|pnpm\s+test|yarn\s+test|node\s+--test(?:\s+${COMMAND_CHARS}+)*|pytest(?:\s+${COMMAND_CHARS}+)*|python3?\s+-m\s+pytest(?:\s+${COMMAND_CHARS}+)*|go\s+test(?:\s+${COMMAND_CHARS}+)*|cargo\s+test(?:\s+${COMMAND_CHARS}+)*|mvn\s+(?:-[^\s]+\s+)*test|npx\s+[\w@./-]+(?:\s+${COMMAND_CHARS}+)*)`,
+);
+
+export function isTestCommand(text) {
+  return TEST_COMMAND_PATTERNS.some((pattern) => pattern.test(text));
+}
+
+export function isTestFile(path) {
+  return /(^|\/)test(s)?\/|\.test\.[cm]?[jt]sx?$|_test\.go$|^test_[^/]*\.py$|_test\.py$|\.spec\.[cm]?[jt]sx?$/.test(path);
+}
+
+function normalizeCommand(raw) {
+  return String(raw || '').replace(/\s+/g, ' ').trim();
+}
+
+/**
+ * Pull pass/fail/total counts out of one line of claim text.
+ * Handles English ("18 tests, all pass", "6/6 pass", "439 pass, 0 fail")
+ * and the Chinese phrasing used in evalid PR bodies ("440 项、439 通过、0 失败").
+ */
+export function parseClaimCounts(line) {
+  const text = String(line || '');
+  const counts = {};
+  let m = text.match(/(\d+)\s*\/\s*(\d+)\s*(?:pass|passed|通过|✔|✅)?/);
+  if (m) {
+    counts.passed = Number(m[1]);
+    counts.total = Number(m[2]);
+    counts.failed = counts.total - counts.passed;
+    return counts;
+  }
+  m = text.match(/(\d+)\s*(?:tests?|items?|项)[^0-9]{0,20}?(?:all\s+pass|pass|passed|通过)/i);
+  if (m) counts.passed = Number(m[1]);
+  // No \b after 通过/失败/项: \b is an ASCII word boundary and never
+  // fires after a CJK character, so "439 通过" would silently not match.
+  m = text.match(/(\d+)\s*(?:pass|passed|通过)/i);
+  if (m && counts.passed === undefined) counts.passed = Number(m[1]);
+  // "18 tests (long parenthetical ...) — all pass": the count and the
+  // verdict may be far apart, so an explicit all-pass fallback reads the
+  // first test/item count on the line as the passed count.
+  if (/all\s+pass(?:ing|es)?\b/i.test(text) && counts.passed === undefined) {
+    const n = text.match(/(\d+)\s*(?:tests?|items?|项)/);
+    if (n) counts.passed = Number(n[1]);
+  }
+  m = text.match(/(\d+)\s*(?:fail|failed|failures?|失败)/i);
+  if (m) counts.failed = Number(m[1]);
+  if (/all\s+pass(?:ing|es)?\b/i.test(text) || /全部通过/.test(text)) {
+    if (counts.failed === undefined) counts.failed = 0;
+  }
+  m = text.match(/(\d+)\s*(?:tests?|items?|项)/);
+  if (m && counts.total === undefined) counts.total = Number(m[1]);
+  return counts;
+}
+
+function parseClaimExit(line) {
+  const m = String(line || '').match(/exit(?:\s+code)?\s+(\d+)/i);
+  return m ? Number(m[1]) : null;
+}
+
+/**
+ * Extract claimed test runs from free text (a PR body, commit message, or
+ * a --claims file). Returns one claim per distinct command, with any
+ * counts/exit found on the same line or in the same paragraph attached.
+ * A line that names a test file plus counts ("18 tests ... all pass") is
+ * attached to the command claim that runs that file.
+ */
+export function extractClaims(text) {
+  const lines = String(text || '').split('\n');
+  const claims = [];
+  const byCommand = new Map();
+
+  function upsert(command, source, counts, exitCode) {
+    const key = normalizeCommand(command);
+    if (!key) return null;
+    let claim = byCommand.get(key);
+    if (!claim) {
+      claim = { command: key, claimedExit: null, claimedPassed: null, claimedFailed: null, claimedTotal: null, source };
+      byCommand.set(key, claim);
+      claims.push(claim);
+    }
+    if (counts) {
+      if (counts.passed !== undefined && claim.claimedPassed === null) claim.claimedPassed = counts.passed;
+      if (counts.failed !== undefined && claim.claimedFailed === null) claim.claimedFailed = counts.failed;
+      if (counts.total !== undefined && claim.claimedTotal === null) claim.claimedTotal = counts.total;
+    }
+    if (exitCode !== null && exitCode !== undefined && claim.claimedExit === null) claim.claimedExit = exitCode;
+    return claim;
+  }
+
+  // Counts/exit attach line-by-line, and within a line only within the
+  // same clause: one line may state two claims ("...passed 6/6 after.
+  // Full `npm test` passes (exit 0...)") and the first file's counts must
+  // not leak onto the second command. Clauses split on ". " / "; ".
+  // (File paths contain dots but never ". ", so this does not split them.)
+  for (const line of lines) {
+    const clauses = line.split(/(?:[.;]\s+|\s+;\s*)/);
+    for (const clause of clauses) {
+      const commands = new Set();
+      const spans = [...clause.matchAll(/`([^`]+)`/g)].map((span) => span[1]);
+      for (const span of spans) {
+        const found = span.match(COMMAND_TAIL);
+        if (found) commands.add(normalizeCommand(found[0]));
+      }
+      const bare = clause.match(COMMAND_TAIL);
+      if (bare) commands.add(normalizeCommand(bare[0]));
+      if (commands.size === 0) continue;
+      const counts = parseClaimCounts(clause);
+      const exitCode = parseClaimExit(clause);
+      const hasCounts = Object.keys(counts).length > 0;
+      for (const command of commands) {
+        upsert(command, clause.trim().slice(0, 200), hasCounts ? counts : null, exitCode);
+      }
+    }
+  }
+
+  // File-scoped counts: "test/foo.test.js: 18 tests, all pass" attaches to
+  // the command claim whose argv names that file. A command that runs
+  // several files sums their per-file counts (18 + 6 = 24 for the combined
+  // run), and only fills claims that have no same-line counts.
+  const fileSums = new Map();
+  for (const line of lines) {
+    const fileMatch = line.match(/([^\s`'"]+\.test\.[cm]?[jt]sx?|[^\s`'"]+_test\.go|[^\s`'"]+test_[^/\s`'"]*\.py)/);
+    if (!fileMatch) continue;
+    const counts = parseClaimCounts(line);
+    if (Object.keys(counts).length === 0) continue;
+    for (const claim of claims) {
+      if (!claim.command.includes(fileMatch[1])) continue;
+      const sum = fileSums.get(claim.command) || { passed: 0, failed: 0, total: 0, hasPassed: false, hasFailed: false, hasTotal: false };
+      if (counts.passed !== undefined) { sum.passed += counts.passed; sum.hasPassed = true; }
+      if (counts.failed !== undefined) { sum.failed += counts.failed; sum.hasFailed = true; }
+      if (counts.total !== undefined) { sum.total += counts.total; sum.hasTotal = true; }
+      fileSums.set(claim.command, sum);
+    }
+  }
+  for (const claim of claims) {
+    const sum = fileSums.get(claim.command);
+    if (!sum) continue;
+    if (claim.claimedPassed === null && sum.hasPassed) claim.claimedPassed = sum.passed;
+    if (claim.claimedFailed === null && sum.hasFailed) claim.claimedFailed = sum.failed;
+    if (claim.claimedTotal === null && sum.hasTotal) claim.claimedTotal = sum.total;
+  }
+
+  // A claim that says tests pass but states no exit code is a claim of exit 0.
+  for (const claim of claims) {
+    if (claim.claimedExit === null && (claim.claimedPassed !== null || claim.claimedFailed === 0)) {
+      claim.claimedExit = 0;
+    }
+  }
+  return claims;
+}
+
+/** Parse a unified diff: touched files, touched test files, added evalid seals. */
+export function parseDiffFiles(diffText) {
+  const files = [];
+  const seals = [];
+  for (const line of String(diffText || '').split('\n')) {
+    const m = line.match(/^\+\+\+ b\/(.+)$/) || line.match(/^\+\+\+ (.+)$/);
+    if (!m || m[1] === '/dev/null') continue;
+    const path = m[1].trim();
+    files.push(path);
+    const seal = path.match(/^specs\/changes\/([^/]+)\/tdd\/([a-z]+)-(\d+)\.json$/);
+    if (seal) seals.push({ changeId: seal[1], phase: seal[2], attempt: Number(seal[3]), path });
+  }
+  return { files, testFiles: files.filter(isTestFile), seals };
+}
+
+/**
+ * Parse pass/fail counts out of a test runner's output.
+ * node --test ("# pass 18", "# fail 0"), pytest ("18 passed, 0 failed"),
+ * mocha ("18 passing"), npm-wrapped node --test (same "# pass" lines).
+ */
+export function parseReplayCounts(output, command) {
+  const text = String(output || '');
+  // node --test prints one summary block per invocation, but multi-file
+  // runners (npm test -> run-tests.mjs, one node --test per file) print one
+  // per file. A single summary is a comparable count; several are not a
+  // total, so report unparsed and let the caller compare on exit code only.
+  const nodePasses = [...text.matchAll(/(?:#|ℹ)\s*pass\s+(\d+)/g)];
+  const nodeFails = [...text.matchAll(/(?:#|ℹ)\s*fail\s+(\d+)/g)];
+  if (nodePasses.length === 1 && nodeFails.length <= 1) {
+    const nodePass = Number(nodePasses[0][1]);
+    const nodeFail = nodeFails.length === 1 ? Number(nodeFails[0][1]) : null;
+    return { passed: nodePass, failed: nodeFail, total: nodeFail !== null ? nodePass + nodeFail : null };
+  }
+  if (nodePasses.length > 1 || nodeFails.length > 1) return null;
+  let m = text.match(/(\d+)\s+passed?(?:,\s*(\d+)\s+failed?)?/);
+  if (m) {
+    return { passed: Number(m[1]), failed: m[2] !== undefined ? Number(m[2]) : null, total: null };
+  }
+  m = text.match(/(\d+)\s+passing(?:,\s*(\d+)\s+failing)?/);
+  if (m) {
+    return { passed: Number(m[1]), failed: m[2] !== undefined ? Number(m[2]) : null, total: null };
+  }
+  if (/test[s]?\s+passed/i.test(text) && /0\s+failed/i.test(text)) return { passed: null, failed: 0, total: null };
+  return null;
+}
+
+function splitCommand(command) {
+  const parts = [];
+  let current = '';
+  let quote = null;
+  for (const char of String(command)) {
+    if (quote) {
+      if (char === quote) quote = null;
+      else current += char;
+    } else if (char === '"' || char === "'") {
+      quote = char;
+    } else if (/\s/.test(char)) {
+      if (current) { parts.push(current); current = ''; }
+    } else {
+      current += char;
+    }
+  }
+  if (current) parts.push(current);
+  return parts;
+}
+
+function runShell(cwd, argv, timeoutMs) {
+  const started = Date.now();
+  // A replay must run in a clean test-runner context: when audit itself runs
+  // under `node --test` (unit tests, or CI invoking it from a test file),
+  // NODE_TEST_CONTEXT / NODE_TEST_WORKER_ID leak into the child and make
+  // `node --test` refuse to run files ("called recursively within a test
+  // file", exit 0, zero tests). Strip them so the replay really runs.
+  const env = { ...process.env };
+  delete env.NODE_TEST_CONTEXT;
+  delete env.NODE_TEST_WORKER_ID;
+  const result = spawnSync(argv[0], argv.slice(1), { cwd, encoding: 'utf8', timeout: timeoutMs, env });
+  const output = `${result.stdout || ''}${result.stderr || ''}`;
+  return {
+    exit: result.status === null ? null : result.status,
+    timedOut: result.status === null && result.error && /timed out/i.test(String(result.error.message || '')),
+    spawnError: result.error && result.status === null ? String(result.error.message || result.error) : null,
+    output,
+    outputTail: output.split('\n').slice(-40).join('\n'),
+    durationMs: Date.now() - started,
+  };
+}
+
+function replaySeal(workdir, repoRoot, seal, python, timeoutMs) {
+  const validator = resolve(repoRoot, 'xflow', 'atoms', 'i6b_tdd_proof_validate.py');
+  if (!existsSync(validator)) {
+    return { ...seal, ok: false, verdict: 'validator_missing', error: `validator not found: ${validator}` };
+  }
+  const result = spawnSync(python, [
+    validator,
+    '--project-root', workdir,
+    '--change-id', seal.changeId,
+    '--phase', seal.phase,
+    '--attempt', String(seal.attempt),
+  ], { cwd: workdir, encoding: 'utf8', timeout: timeoutMs });
+  const lines = String(result.stdout || '').split('\n').reverse();
+  let payload = null;
+  for (const line of lines) {
+    const trimmed = line.trim();
+    if (!trimmed.startsWith('{')) continue;
+    try {
+      const parsed = JSON.parse(trimmed);
+      if (parsed && typeof parsed.verdict === 'string') { payload = parsed; break; }
+    } catch { /* not the verdict line */ }
+  }
+  return {
+    ...seal,
+    ok: payload?.ok === true && result.status === 0,
+    verdict: payload?.verdict || 'unparseable_output',
+    exit: result.status,
+    stderr: String(result.stderr || '').slice(-500),
+  };
+}
+
+/**
+ * Run a full audit. Options:
+ *   repoDir    — local repo to clone from (required)
+ *   diffText   — unified diff to apply on top of baseRef (or '' when headRef is checked out directly)
+ *   baseRef    — commit to check out before applying diffText
+ *   headRef    — commit to check out directly (diffText ignored; used for merged PRs)
+ *   claimsText — PR body / claims text to extract claims from
+ *   timeoutMs  — per-command timeout (default 300_000)
+ *   python     — python binary for the i6b validator (default 'python3')
+ *   keepWorkdir — keep the scratch clone for debugging (default false)
+ */
+export function runAudit(options = {}) {
+  const {
+    repoDir,
+    diffText = '',
+    baseRef = null,
+    headRef = null,
+    claimsText = '',
+    timeoutMs = 300000,
+    python = process.env.PYTHON || 'python3',
+    keepWorkdir = false,
+  } = options;
+  if (!repoDir || !existsSync(repoDir)) {
+    return { ok: false, verdict: 'misuse', error: `repo not found: ${repoDir}`, exitCode: 2 };
+  }
+  if (!headRef && !baseRef) {
+    return { ok: false, verdict: 'misuse', error: 'one of --head or --base is required', exitCode: 2 };
+  }
+
+  const started = Date.now();
+  const claims = extractClaims(claimsText);
+  const diff = parseDiffFiles(diffText);
+  const workdir = mkdtempSync(join(tmpdir(), 'evalid-audit-'));
+  try {
+    // --no-hardlinks: /tmp may live on another filesystem than the repo,
+    // and a hardlinked local clone then fails with "Invalid cross-device link".
+    const clone = runShell(tmpdir(), ['git', 'clone', '--quiet', '--no-hardlinks', '--local', resolve(repoDir), workdir], 120000);
+    if (clone.exit !== 0) {
+      return { ok: false, verdict: 'unverifiable', error: `git clone failed: ${clone.outputTail.slice(-300)}`, claims, diff, exitCode: 1 };
+    }
+    const checkoutRef = headRef || baseRef;
+    const checkout = runShell(workdir, ['git', 'checkout', '--quiet', checkoutRef], 60000);
+    if (checkout.exit !== 0) {
+      return { ok: false, verdict: 'unverifiable', error: `git checkout ${checkoutRef} failed: ${checkout.outputTail.slice(-300)}`, claims, diff, exitCode: 1 };
+    }
+    if (diffText && !headRef) {
+      const diffFile = join(workdir, '.audit.diff');
+      writeFileSync(diffFile, diffText);
+      const apply = runShell(workdir, ['git', 'apply', '--whitespace=nowarn', diffFile], 60000);
+      rmSync(diffFile, { force: true });
+      if (apply.exit !== 0) {
+        return { ok: false, verdict: 'unverifiable', error: `git apply failed: ${apply.outputTail.slice(-300)}`, claims, diff, exitCode: 1 };
+      }
+    }
+
+    // Replay setup: a fresh clone has no node_modules, so a claimed command
+    // that imports a dependency would fail for environmental reasons, not
+    // because the change is broken. Install deps when package.json exists
+    // and node_modules is absent (npm ci with a lockfile, npm install
+    // without one); fall back to copying the source repo's node_modules.
+    // Skipped entirely when there is nothing to replay (no claims, no seals).
+    let setup = { deps: 'none' };
+    if ((claims.length > 0 || diff.seals.length > 0)
+      && existsSync(join(workdir, 'package.json')) && !existsSync(join(workdir, 'node_modules'))) {
+      const hasLock = existsSync(join(workdir, 'package-lock.json'));
+      const installArgs = hasLock
+        ? ['npm', 'ci', '--no-audit', '--no-fund']
+        : ['npm', 'install', '--no-audit', '--no-fund'];
+      const install = runShell(workdir, installArgs, 300000);
+      if (install.exit === 0) {
+        setup = { deps: hasLock ? 'npm ci' : 'npm install' };
+      } else if (existsSync(join(resolve(repoDir), 'node_modules'))) {
+        const copy = runShell(tmpdir(), ['cp', '-r', join(resolve(repoDir), 'node_modules'), join(workdir, 'node_modules')], 120000);
+        setup = copy.exit === 0
+          ? { deps: 'copied from source repo node_modules' }
+          : { deps: 'none', error: `${installArgs[1]} failed and node_modules copy failed` };
+      } else {
+        setup = { deps: 'none', error: `${installArgs[1]} failed: ${install.outputTail.slice(-300)}` };
+      }
+    }
+    if (setup.error) {
+      return { ok: false, verdict: 'unverifiable', error: `replay setup failed (${setup.error})`, claims, diff, setup, exitCode: 1 };
+    }
+
+    const claimResults = [];
+    let mismatch = null;
+    for (const claim of claims) {
+      const argv = splitCommand(claim.command);
+      if (argv.length === 0) {
+        claimResults.push({ ...claim, replay: null, match: false, note: 'empty command' });
+        mismatch = mismatch || 'empty command';
+        continue;
+      }
+      const replay = runShell(workdir, argv, timeoutMs);
+      const counts = parseReplayCounts(replay.output, claim.command);
+      const expectedExit = claim.claimedExit ?? 0;
+      let match = replay.exit === expectedExit && !replay.timedOut && !replay.spawnError;
+      let note = null;
+      if (match && counts) {
+        if (claim.claimedPassed !== null && counts.passed !== null && counts.passed !== claim.claimedPassed) {
+          match = false;
+          note = `passed count: claimed ${claim.claimedPassed}, replayed ${counts.passed}`;
+        }
+        if (match && claim.claimedFailed !== null && counts.failed !== null && counts.failed !== claim.claimedFailed) {
+          match = false;
+          note = `failed count: claimed ${claim.claimedFailed}, replayed ${counts.failed}`;
+        }
+      } else if (match && !counts && (claim.claimedPassed !== null || claim.claimedFailed !== null)) {
+        note = 'counts_unparsed: exit matches, counts could not be parsed from replay output';
+      }
+      if (replay.timedOut) note = `replay timed out after ${timeoutMs}ms`;
+      if (replay.spawnError) note = `could not run claimed command: ${replay.spawnError}`;
+      if (!match && !note) note = `exit: claimed ${expectedExit}, replayed ${replay.exit}`;
+      claimResults.push({ ...claim, replay: { ...replay, counts }, match, note });
+      if (!match) mismatch = mismatch || note || 'claim mismatch';
+    }
+
+    const sealResults = diff.seals.map((seal) => replaySeal(workdir, resolve(repoDir), seal, python, timeoutMs));
+    const badSeal = sealResults.find((seal) => !seal.ok);
+
+    let verdict;
+    let exitCode;
+    if (claims.length === 0 && sealResults.length === 0) {
+      verdict = 'no_claims';
+      exitCode = 3;
+    } else if (mismatch || badSeal) {
+      verdict = 'mismatch';
+      exitCode = 1;
+    } else if (claimResults.some((entry) => entry.replay && (entry.replay.timedOut || entry.replay.spawnError))) {
+      verdict = 'unverifiable';
+      exitCode = 1;
+    } else {
+      verdict = 'verified';
+      exitCode = 0;
+    }
+
+    return {
+      ok: verdict === 'verified',
+      verdict,
+      exitCode,
+      claims: claimResults,
+      seals: sealResults,
+      files: diff.files,
+      testFiles: diff.testFiles,
+      setup,
+      workdir: keepWorkdir ? workdir : null,
+      durationMs: Date.now() - started,
+    };
+  } finally {
+    if (!keepWorkdir) rmSync(workdir, { recursive: true, force: true });
+  }
+}
+
+export function auditExitCode(verdict) {
+  return { verified: 0, mismatch: 1, unverifiable: 1, misuse: 2, no_claims: 3 }[verdict] ?? 1;
+}
+
+export function renderAuditReport(report) {
+  const lines = [`xflow audit: ${report.verdict}`];
+  if (report.error) lines.push(`error: ${report.error}`);
+  if (report.setup?.deps && report.setup.deps !== 'none') lines.push(`replay deps: ${report.setup.deps}`);
+  if (report.files) {
+    lines.push(`files touched: ${report.files.length} (${report.testFiles?.length ?? 0} test files)`);
+    for (const file of report.testFiles || []) lines.push(`  test: ${file}`);
+  }
+  for (const claim of report.claims || []) {
+    const replay = claim.replay;
+    const replayText = replay
+      ? `replay exit ${replay.exit}${replay.counts && replay.counts.passed !== null ? `, pass ${replay.counts.passed}` : ''}${replay.counts && replay.counts.failed !== null ? `, fail ${replay.counts.failed}` : ''}`
+      : 'not replayed';
+    const claimedText = [
+      claim.claimedExit !== null ? `exit ${claim.claimedExit}` : null,
+      claim.claimedPassed !== null ? `${claim.claimedPassed} passed` : null,
+      claim.claimedFailed !== null ? `${claim.claimedFailed} failed` : null,
+    ].filter(Boolean).join(', ') || 'no counts stated';
+    lines.push(`- ${claim.match ? '✓' : '✗'} ${claim.command} (claimed ${claimedText}; ${replayText})${claim.note ? ` — ${claim.note}` : ''}`);
+  }
+  for (const seal of report.seals || []) {
+    lines.push(`- ${seal.ok ? '✓' : '✗'} seal ${seal.changeId}/${seal.phase}/${seal.attempt}: ${seal.verdict}`);
+  }
+  if (report.durationMs !== undefined) lines.push(`duration: ${(report.durationMs / 1000).toFixed(1)}s`);
+  return lines.join('\n');
+}