ai-engineering-loop 2.5.0 → 2.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. package/.agents/skills/ael/SKILL.md +1 -1
  2. package/.agents/skills/ael/references/loop-commands.md +11 -0
  3. package/.agents/skills/ael/references/loop-goal.md +2 -0
  4. package/.agents/skills/ael/references/loop-maker.md +1 -6
  5. package/.agents/skills/ael/references/loop-report.md +8 -0
  6. package/.agents/skills/ael/references/loop-verify.md +21 -11
  7. package/.claude/agents/devil-advocate.md +6 -1
  8. package/.claude/agents/judge.md +3 -1
  9. package/.grok/agents/devil-advocate.md +6 -1
  10. package/.grok/agents/judge.md +3 -1
  11. package/CHANGELOG.md +22 -0
  12. package/agents/devil-advocate.md +8 -0
  13. package/agents/judge.md +1 -0
  14. package/agents/shared/devil-advocate.body.md +6 -1
  15. package/agents/shared/judge.body.md +3 -1
  16. package/bin/ai-engineering-loop.js +262 -20
  17. package/core/bench.md +15 -0
  18. package/core/goal-contract.md +35 -0
  19. package/core/verification-loop.md +96 -7
  20. package/lib/ablation.js +52 -0
  21. package/lib/ac-status.js +63 -0
  22. package/lib/baseline.js +138 -0
  23. package/lib/bench.js +24 -1
  24. package/lib/calibration.js +112 -0
  25. package/lib/claimed-vs-reality.js +114 -16
  26. package/lib/clean-state.js +85 -0
  27. package/lib/cli-args.js +5 -3
  28. package/lib/debt.js +110 -0
  29. package/lib/features.js +95 -0
  30. package/lib/gates.js +234 -41
  31. package/lib/harness-audit.js +124 -0
  32. package/lib/no-progress.js +6 -1
  33. package/lib/review-ingest.js +54 -2
  34. package/lib/rules.js +112 -0
  35. package/lib/stats.js +90 -0
  36. package/lib/sync-hosts.js +1 -0
  37. package/lib/verification-integrity.js +164 -0
  38. package/lib/verification-lanes.js +64 -0
  39. package/lib/verification-recorder.js +17 -5
  40. package/package.json +1 -1
  41. package/schemas/finding-ledger.schema.json +14 -0
  42. package/schemas/goal-contract.schema.json +31 -0
  43. package/schemas/verdict.schema.json +2 -1
@@ -43,9 +43,58 @@ A verification `PASS` is **strictly invalid** without concrete execution evidenc
43
43
  }
44
44
  ```
45
45
 
46
+ ## 3. Verification Layers & Command Mapping
47
+
48
+ Verification terminates through three explicit layers of evidence. Unit tests alone have systematic blind spots for cross-component and integration defects. An acceptance criterion crossing component boundaries requires Layer 3 verification or written justification if Layer 3 is unavailable.
49
+
50
+ | Layer | Scope | Configured Command (`verification.md`) | Purpose |
51
+ |---|---|---|---|
52
+ | **L1: Static / Syntax** | Syntax, static types, style, compilation | `typecheck`, `lint`, `build` | Fast static catch before runtime. Rejects type errors, lint regressions, and compilation breakage. |
53
+ | **L2: Behavioral / Runtime** | Functions, units, modules, regressions | `test_unit`, `test_related`, `test_all` | Fast behavioral feedback at named seams and regression verification. |
54
+ | **L3: End-to-End** | Cross-component flows, integrations, browser/API | `e2e` (optional) | Realistic integrated proof across boundary seams. |
55
+
56
+ Unit tests isolate and mock by design, which is exactly why they miss interface mismatches, state propagation errors, resource leaks, and environment dependencies between components.
57
+
58
+ | Tier | Required layers |
59
+ |---|---|
60
+ | T0 | L1-L2 (L3 when an AC is `crossComponent: true`) |
61
+ | T1 | L1-L2, plus L3 for every `crossComponent: true` AC |
62
+ | T2 | L1-L3 on every AC |
63
+
64
+ Enforcement (`lib/claimed-vs-reality.js`, `gate review`):
65
+ - The Layer cell accepts `L1`, `L2`, `L3`, combos such as `L2+L3`, and `L3 waived: <reason>` (at least 15 characters). Unknown tokens fail the gate.
66
+ - An AC that needs L3 without L3 or a reasoned waiver fails `gate review` on every tier, with a FIX line that says what to run.
67
+ - Legacy 3-column files stay valid when no AC needs L3.
68
+ - Tags are bound to recorded evidence (C-01): `L1` needs a passing L1 command in `verification.json`/`final-verification.json`; `L3` needs a passing `e2e` command, or Reality that describes a manual end-to-end flow (steps, URL, screenshot path). Commands not mapped in `verification.md` count as L2. Waivers need no command.
69
+ - When `e2e` is empty in `verification.md`, run the AC's real user flow and cite the observation, or write the waiver. Devil's Advocate and Judge challenge waivers like any other claim.
70
+
46
71
  ---
47
72
 
48
- ## 3. The 4 Verification Gates
73
+ ## 3a. Baseline and Default Plan
74
+
75
+ **Baseline (C-10).** After the Goal gate and before the Maker gate, `verify --baseline --run <id>` runs the default plan on the untouched tree and writes `runs/<id>/baseline.json` with status `GREEN`, `RED`, or `BROKEN`.
76
+ - `BROKEN` (a command exits 127, i.e. it does not exist): the command fails with a FIX to correct `verification.md`. Verification cannot prove anything with a broken command.
77
+ - `RED`: the tree was already failing. Later, when `verify` fails, each failing command is labelled `pre-existing` (every failing test name was already failing at baseline) or new. Unnamed failures are always treated as new. The gate still blocks; the label tells the Maker whether the change caused it.
78
+
79
+ **Default plan (C-02).** `verify` without `--cmd` runs `build -> typecheck -> lint -> test_unit` (whichever are configured), serially, and stops at the first failure. Commands that rewrite files (`--fix`, `--write`) are never run by default. Each record carries its layer (`L1`/`L2`/`L3`).
80
+
81
+ **L1 at T1/T2.** When `verification.md` configures `typecheck` or `lint`, the verification gate at T1/T2 refuses a full-scope bundle with no L1 record (`L1_MISSING`). The related lane is exempt; `verify --final` runs the full plan.
82
+
83
+ `init` writes only commands the project really has: `typecheck` from a script or a TypeScript setup, `lint` and `build` only from package scripts.
84
+
85
+ **Smoke (C-03).** An optional `smoke` key in `verification.md` is a command that starts the app, probes readiness (a health URL, a golden journey), stops it, and exits. It counts as L2. When the gated diff touches startup-risky files (migrations, `.env*`, `config/`, `*.config.*`, Docker/Procfile, `main|index|server|app` entrypoints, dependency manifests), default `verify` runs it after `test_unit`, and the verification gate fails with `SMOKE_MISSING` when it did not run. A smoke command that times out or exits non-zero blocks the gate; on timeout the recorder kills its process group.
86
+
87
+ ## 3b. Verification Integrity (C-11)
88
+
89
+ At `gate review` the CLI scans the gated diff and adds `source: "verification-integrity"` findings (HIGH; BLOCKER at T2, VALID, with a concrete fix) for:
90
+ - a skip/only/todo marker added in a test file (`it.skip`, `xit`, `fit`, `@Disabled`, `pytest.mark.skip`, `t.Skip`, ...);
91
+ - a test file deleted without a replacement test in the same directory;
92
+ - assertions removed from a test file without replacement;
93
+ - a change to verification config (`.ai-engineering-loop/verification.md`, jest/vitest/playwright/cypress config, `pytest.ini`, coverage config) that the frozen Goal Contract does not name.
94
+
95
+ These findings cannot be dropped or downgraded by the reviewer. Only a human dismissal (`dismissal: {by, evidence}`) can downgrade them; otherwise the review gate fails.
96
+
97
+ ## 4. The 4 Verification Gates
49
98
 
50
99
  1. **Gate 1: Unit & Integration Tests**: All unit and edge-case tests must pass with 0 failures.
51
100
  2. **Gate 2: Static Type Analysis**: `tsc --noEmit` or compiler type checking must exit with code `0` (zero type errors).
@@ -54,7 +103,7 @@ A verification `PASS` is **strictly invalid** without concrete execution evidenc
54
103
 
55
104
  ---
56
105
 
57
- ## 4. Rejection Triggers
106
+ ## 5. Rejection Triggers
58
107
 
59
108
  The Verification Gate immediately halts and returns to the Maker if:
60
109
  - Any test suite fails or times out.
@@ -66,7 +115,7 @@ The Verification Gate immediately halts and returns to the Maker if:
66
115
 
67
116
  ---
68
117
 
69
- ## 5. Claimed vs Reality (before Devil's Advocate)
118
+ ## 6. Claimed vs Reality (before Devil's Advocate)
70
119
 
71
120
  Write `.ai-engineering-loop/tasks/claimed-vs-reality.md` after verification logs exist:
72
121
 
@@ -74,14 +123,14 @@ Write `.ai-engineering-loop/tasks/claimed-vs-reality.md` after verification logs
74
123
  npx ai-engineering-loop context claimed-vs-reality --run <run-id>
75
124
  ```
76
125
 
77
- This scaffolds one row per frozen acceptance criterion but leaves Claimed and Reality empty. Fill them from the implemented change and actual command output; the scaffold is not verification evidence.
126
+ This scaffolds one row per frozen acceptance criterion but leaves Claimed, Reality, and Layer empty. Fill them from the implemented change and actual command output; the scaffold is not verification evidence.
78
127
 
79
128
  ```markdown
80
129
  # Claimed vs Reality
81
130
 
82
- | AC | Claimed | Reality |
83
- |---|---|---|
84
- | AC-1 | <what Maker said is done> | <command, exit code, assertion or log line> |
131
+ | AC | Claimed | Reality from command log | Layer |
132
+ |---|---|---|---|
133
+ | AC-1 | <what Maker said is done> | <command, exit code, assertion or log line> | L2 |
85
134
  ```
86
135
 
87
136
  Do **not** spawn Devil's Advocate if:
@@ -92,3 +141,43 @@ Do **not** spawn Devil's Advocate if:
92
141
  - Reality is only `seems green`
93
142
 
94
143
  Parser: `lib/claimed-vs-reality.js` (`readyForDa`). Pass that path to DA/Judge with the verification log.
144
+
145
+ ## 7. Clean State at Delivery (C-06)
146
+
147
+ `gate delivery` also checks that the next session starts from a tree it can trust:
148
+ - **Always blocks:** a `debugger` statement, merge-conflict marker or interactive breakpoint added by the change; a secret-like file in the change (`.env`, `.env.local`, keys; `.env.example` is fine); a `_templateOnly` scaffold left in the Run directory or `.ai-engineering-loop/tasks/`.
149
+ - **Untracked files that are not part of the reviewed diff** (logs, coverage, scratch files): block in UNATTENDED, warn in ASSISTED. Warnings are printed and kept in the Run history as `cleanStateWarnings`.
150
+
151
+ ## 8. Debt Ledger (C-14)
152
+
153
+ At `gate delivery` the CLI copies two kinds of deferred work into `.ai-engineering-loop/debt.json` (tracked in git, like `learnings.json`):
154
+ - VALID `MEDIUM`/`LOW` findings the Judge accepted as trade-offs (file, scenario, risk, "revisit on the next change touching <file>");
155
+ - claimed-vs-reality rows with `L3 waived: <reason>` ("revisit when an e2e command exists").
156
+
157
+ `review-pack` appends open debt on the files the new diff touches to the Devil's Advocate pack and lists it. `ael debt` lists open items (`--file <path>`); `ael debt close <id> --by <name> [--note <text>]` closes one.
158
+
159
+ ## 9. Reviewer Inputs Are Logged (C-16)
160
+
161
+ "Model-visible means logged": `review-pack` writes `runs/<id>/review-inputs.json` with the sha256 of the review pack, the gated diff, and the Devil's Advocate prompt body. `review ingest` copies them into `findings.json` (`reviewer.inputs`, `reviewer.promptSha256`). `gate review` fails the ledger binding when any recorded input changed after the reviewer saw it, so a missed defect can always be traced to exactly what the reviewer was shown.
162
+
163
+ ## 10. Reviewer Calibration (C-13)
164
+
165
+ Gates prove the change matches the Goal; only reality proves the reviewers were right. After delivery, record problems people found:
166
+
167
+ ```
168
+ ael outcome record --run <id> --source mr-comment|revert|bug|incident --location <file[:line]> --summary <text> --by <name> [--evidence <path|url>]
169
+ ael calibrate [--json]
170
+ ```
171
+
172
+ `calibrate` compares each outcome with that Run's Finding Ledger: `DA_MISS` (no finding on that file), `FALSE_DISMISS` (a finding on that file was marked INVALID or dismissed), `CAUGHT_BUT_SHIPPED` (a VALID finding was accepted as a trade-off). Results are grouped by the reviewer prompt hash (C-16), so after you tighten `agents/shared/devil-advocate.body.md` for a recurring miss you can see whether the next prompt version misses less. Plan for several tuning rounds; record each prompt change in its commit message.
173
+
174
+ ## 11. Promoting Review Feedback to Rules (C-05)
175
+
176
+ `learn` lists **promotion candidates**: learnings seen in 3 or more Runs. A human turns one into an executable check:
177
+
178
+ ```
179
+ ael learn promote L-abc123 --check "! grep -rq \"require('fs')\" src/renderer" --fix "Move file access to src/preload/file-ops.ts and call window.api" --by <name> [--why <text>]
180
+ ```
181
+
182
+ Rules live in `.ai-engineering-loop/rules.json` (tracked). Default `verify` runs every rule as an L1 check after the static checks; a violation prints `Rule <id> violated. WHY: ... FIX: ...`. The verification gate refuses a full-scope bundle in which any rule has no passing record (`RULES_MISSING`). The check must exit non-zero on a violation and is run through the shell unless `--no-shell` is given.
183
+
@@ -0,0 +1,52 @@
1
+ 'use strict';
2
+
3
+ // Harness ablation for bench (C-08, learn-harness-engineering L02/L12):
4
+ // switch one gate component off inside a bench scenario repo to measure
5
+ // whether it still earns its cost. Honoured only when bench set it up (env
6
+ // AEL_BENCH_MODE + AEL_ABLATE and a marker file bench wrote into the repo),
7
+ // and every skip is logged to the Run so it can never pass silently.
8
+
9
+ const fs = require('fs');
10
+ const path = require('path');
11
+ const { runsDir } = require('./run-state.js');
12
+
13
+ const COMPONENTS = Object.freeze(['integrity', 'l3', 'l1', 'smoke', 'rules']);
14
+ const MARKER = path.join('.ai-engineering-loop', 'bench-ablation.json');
15
+
16
+ function requested(env = process.env) {
17
+ return String(env.AEL_ABLATE || '').split(',').map((item) => item.trim()).filter(Boolean);
18
+ }
19
+
20
+ function writeMarker(repo, components) {
21
+ fs.mkdirSync(path.join(repo, '.ai-engineering-loop'), { recursive: true });
22
+ fs.writeFileSync(path.join(repo, MARKER), `${JSON.stringify({ schemaVersion: 1, components }, null, 2)}\n`);
23
+ }
24
+
25
+ function ablated(rootDir, component, runId = null, env = process.env) {
26
+ if (!env.AEL_BENCH_MODE || !requested(env).includes(component)) return false;
27
+ let marker = null;
28
+ try {
29
+ marker = JSON.parse(fs.readFileSync(path.join(rootDir, MARKER), 'utf8'));
30
+ } catch {
31
+ return false;
32
+ }
33
+ if (!Array.isArray(marker?.components) || !marker.components.includes(component)) return false;
34
+ if (runId) {
35
+ try {
36
+ fs.appendFileSync(path.join(runsDir(rootDir), runId, 'ablation.log'), `${JSON.stringify({ component, at: new Date().toISOString() })}\n`);
37
+ } catch {
38
+ // Run directory missing; the bench result still records the ablation.
39
+ }
40
+ }
41
+ return true;
42
+ }
43
+
44
+ function validateComponents(list) {
45
+ const unknown = list.filter((item) => !COMPONENTS.includes(item));
46
+ if (unknown.length) {
47
+ throw Object.assign(new Error(`Unknown --without component(s): ${unknown.join(', ')}. Use: ${COMPONENTS.join(', ')}`), { code: 'BENCH_USAGE' });
48
+ }
49
+ return list;
50
+ }
51
+
52
+ module.exports = { COMPONENTS, MARKER, ablated, requested, validateComponents, writeMarker };
@@ -0,0 +1,63 @@
1
+ 'use strict';
2
+
3
+ // Harness-owned AC status (C-07, learn-harness-engineering L08): an AC that
4
+ // names its own verify command is `proven` only when that exact command
5
+ // passed on the gated diff. Agents cannot set the status; evidence does.
6
+
7
+ const fs = require('fs');
8
+ const path = require('path');
9
+ const { runsDir } = require('./run-state.js');
10
+
11
+ const norm = (text) => String(text || '').trim().replace(/\s+/g, ' ');
12
+
13
+ function recordedCommands(rootDir, runId) {
14
+ const records = [];
15
+ for (const file of ['verification.json', 'final-verification.json']) {
16
+ try {
17
+ const bundle = JSON.parse(fs.readFileSync(path.join(runsDir(rootDir), runId, file), 'utf8'));
18
+ records.push(...(bundle.commands || []));
19
+ } catch {
20
+ // Not recorded yet.
21
+ }
22
+ }
23
+ return records;
24
+ }
25
+
26
+ // [{ id, command, status: 'proven'|'failed'|'pending'|'unbound' }]
27
+ function acStatuses(goal, records) {
28
+ return (goal?.acceptanceCriteria || []).map((criterion) => {
29
+ const command = norm(criterion.verify?.command);
30
+ if (!command) return { id: criterion.id, command: null, status: 'unbound' };
31
+ const matches = (records || []).filter((record) => norm(record.command) === command);
32
+ if (matches.some((record) => record.exitCode === 0 && record.timeoutStatus === 'COMPLETED')) {
33
+ return { id: criterion.id, command, status: 'proven' };
34
+ }
35
+ return { id: criterion.id, command, status: matches.length ? 'failed' : 'pending' };
36
+ });
37
+ }
38
+
39
+ function runAcStatuses(rootDir, run, goal) {
40
+ return acStatuses(goal, recordedCommands(rootDir, run.runId));
41
+ }
42
+
43
+ // C-09 (learn-harness-engineering L09/L11): subjective ACs (UI, copy,
44
+ // design) carry a rubric. The ledger must score every criterion with
45
+ // evidence; an unscored or failed criterion blocks Judge PASS.
46
+ function rubricProblems(goal, ledger) {
47
+ const problems = [];
48
+ const scores = Array.isArray(ledger?.rubricScores) ? ledger.rubricScores : [];
49
+ for (const criterion of goal?.acceptanceCriteria || []) {
50
+ for (const item of Array.isArray(criterion.rubric) ? criterion.rubric : []) {
51
+ const name = String(item.criterion || '').trim();
52
+ const score = scores.find((entry) => entry.ac === criterion.id && String(entry.criterion || '').trim() === name);
53
+ if (!score || typeof score.pass !== 'boolean' || !String(score.evidence || '').trim()) {
54
+ problems.push(`${criterion.id} rubric "${name}" has no score with evidence in the Finding Ledger (rubricScores)`);
55
+ } else if (!score.pass) {
56
+ problems.push(`${criterion.id} rubric "${name}" failed: ${score.evidence}`);
57
+ }
58
+ }
59
+ }
60
+ return problems;
61
+ }
62
+
63
+ module.exports = { acStatuses, recordedCommands, rubricProblems, runAcStatuses };
@@ -0,0 +1,138 @@
1
+ 'use strict';
2
+
3
+ // Baseline verification (C-10, learn-harness-engineering L06 + startup flow):
4
+ // before the Maker edits anything, run the project's own verification plan
5
+ // once on the frozen starting state. A broken command is caught before work
6
+ // starts, and failures that already existed are told apart from regressions.
7
+
8
+ const fs = require('fs');
9
+ const path = require('path');
10
+ const { atomicWriteJson, getCurrentRun, getGitRevision, loadRun, runsDir } = require('./run-state.js');
11
+
12
+ const BASELINE_FILE = 'baseline.json';
13
+ const COMMAND_NOT_FOUND = 127;
14
+
15
+ function baselineError(message, code, fix) {
16
+ return Object.assign(new Error(fix ? `${message} FIX: ${fix}` : message), { code });
17
+ }
18
+
19
+ function resolveRun(rootDir, runId) {
20
+ const run = runId ? loadRun(rootDir, runId) : getCurrentRun(rootDir);
21
+ if (!run) throw baselineError('No current run.', 'NO_CURRENT_RUN', 'pass --run <id>.');
22
+ return run;
23
+ }
24
+
25
+ // Failing test names from common runner output (node:test, TAP, jest, vitest,
26
+ // pytest). Best effort: an empty list means "could not tell", never "none".
27
+ function extractFailingTests(text) {
28
+ const names = new Set();
29
+ const patterns = [
30
+ /^\s*✖\s+(.+?)\s+\(\d+(?:\.\d+)?m?s\)\s*$/gm, // node:test spec reporter
31
+ /^not ok \d+ - (.+)$/gm, // TAP
32
+ /^\s*●\s+(.+?)\s*$/gm, // jest
33
+ /^\s*[×✗]\s+(.+?)(?:\s+\d+m?s)?\s*$/gm, // vitest
34
+ /^FAILED\s+(\S+)/gm // pytest -rf
35
+ ];
36
+ const value = String(text || '');
37
+ for (const pattern of patterns) {
38
+ for (const match of value.matchAll(pattern)) {
39
+ const name = match[1].trim();
40
+ if (name && !/^failing tests:?$/i.test(name)) names.add(name);
41
+ }
42
+ }
43
+ return [...names].sort();
44
+ }
45
+
46
+ const passedRecord = (record) => record.exitCode === 0 && record.timeoutStatus === 'COMPLETED';
47
+
48
+ // Run `plan` ([{ key, layer, command, args, shell }]) serially on the current
49
+ // tree and write runs/<id>/baseline.json. Must run before the Maker gate, so
50
+ // the tree is the frozen starting point.
51
+ async function recordBaseline(rootDir, { runId, plan, timeoutMs, runCommand } = {}) {
52
+ const run = resolveRun(rootDir, runId);
53
+ if (run.artifacts?.diff) {
54
+ throw baselineError(`Run ${run.runId} already has a gated Maker diff; a baseline must describe the tree before the change.`,
55
+ 'BASELINE_TOO_LATE', 'start a new Run, or skip the baseline for this one.');
56
+ }
57
+ if (!Array.isArray(plan) || plan.length === 0) {
58
+ throw baselineError('No baseline commands.', 'NO_BASELINE_COMMANDS',
59
+ 'add build/typecheck/lint/test_unit to .ai-engineering-loop/verification.md, or pass --cmd.');
60
+ }
61
+ const recorder = require('./verification-recorder.js');
62
+ const exec = runCommand || recorder.runCommand;
63
+ const limit = Number.isInteger(timeoutMs) && timeoutMs > 0 ? timeoutMs : recorder.DEFAULT_TIMEOUT_MS;
64
+ const records = [];
65
+ const problems = [];
66
+ for (const item of plan) {
67
+ const result = await exec(item.command, item.args || [], { cwd: rootDir, timeoutMs: limit, shell: Boolean(item.shell) });
68
+ const record = {
69
+ key: item.key || null,
70
+ layer: item.layer || null,
71
+ command: result.command,
72
+ exitCode: result.exitCode,
73
+ timeoutStatus: result.timeoutStatus,
74
+ failingTests: passedRecord(result) ? [] : extractFailingTests(`${result.stdout}\n${result.stderr}`)
75
+ };
76
+ records.push(record);
77
+ if (result.exitCode === COMMAND_NOT_FOUND || result.launchError) {
78
+ problems.push(`${record.command}: command not found (exit ${result.exitCode}). FIX: correct the ${record.key || 'command'} entry in .ai-engineering-loop/verification.md; verification cannot prove anything with it.`);
79
+ break;
80
+ }
81
+ }
82
+ const broken = problems.length > 0;
83
+ const red = records.some((record) => !passedRecord(record));
84
+ if (!broken && red) {
85
+ for (const record of records.filter((r) => !passedRecord(r))) {
86
+ problems.push(`${record.command}: already failing before any change (exit ${record.exitCode}${record.failingTests.length ? `; ${record.failingTests.length} failing test(s)` : ''}). FIX: repair the baseline in a separate Run first, or continue knowing these failures are pre-existing.`);
87
+ }
88
+ }
89
+ const status = broken ? 'BROKEN' : (red ? 'RED' : 'GREEN');
90
+ const document = {
91
+ schemaVersion: 1,
92
+ runId: run.runId,
93
+ gitRevision: getGitRevision(rootDir),
94
+ recordedAt: new Date().toISOString(),
95
+ status,
96
+ commands: records
97
+ };
98
+ const filePath = path.join(runsDir(rootDir), run.runId, BASELINE_FILE);
99
+ atomicWriteJson(filePath, document);
100
+ return {
101
+ runId: run.runId,
102
+ path: path.relative(rootDir, filePath).split(path.sep).join('/'),
103
+ status,
104
+ commands: records,
105
+ problems
106
+ };
107
+ }
108
+
109
+ function loadBaseline(rootDir, runId) {
110
+ try {
111
+ return JSON.parse(fs.readFileSync(path.join(runsDir(rootDir), runId, BASELINE_FILE), 'utf8'));
112
+ } catch {
113
+ return null;
114
+ }
115
+ }
116
+
117
+ // Label failing verification records as pre-existing (every failing test was
118
+ // already failing at baseline) or new. Records without a baseline counterpart,
119
+ // or whose failures cannot be named, stay "new": unknown is never excused.
120
+ function classifyAgainstBaseline(baseline, records) {
121
+ const byCommand = new Map((baseline?.commands || []).map((record) => [record.command, record]));
122
+ return (records || []).filter((record) => !passedRecord(record)).map((record) => {
123
+ const before = byCommand.get(record.command);
124
+ const failing = extractFailingTests(`${record.stdout || ''}\n${record.stderr || ''}`);
125
+ const known = new Set(before?.failingTests || []);
126
+ const fresh = failing.filter((name) => !known.has(name));
127
+ const preExisting = Boolean(before && !passedRecord(before) && failing.length > 0 && fresh.length === 0);
128
+ return { command: record.command, preExisting, newFailures: preExisting ? [] : (fresh.length ? fresh : failing) };
129
+ });
130
+ }
131
+
132
+ module.exports = {
133
+ BASELINE_FILE,
134
+ classifyAgainstBaseline,
135
+ extractFailingTests,
136
+ loadBaseline,
137
+ recordBaseline
138
+ };
package/lib/bench.js CHANGED
@@ -817,6 +817,9 @@ function prepareScenarioRepo(repo, scenario, { host, tokenCap, extraEnv, cli, pa
817
817
  const env = extraEnv;
818
818
  const init = spawnNode([cli, 'init'], { cwd: repo, env });
819
819
  if (init.status !== 0) throw benchError(`init failed: ${init.stdout}\n${init.stderr}`, 'BENCH_SETUP');
820
+ const ablation = require('./ablation.js');
821
+ const components = ablation.requested(env || {});
822
+ if (components.length) ablation.writeMarker(repo, components);
820
823
  installProjectHostSkills(repo, host, packageRoot);
821
824
  installLocalCli(repo, cli);
822
825
  if (host.hooks) {
@@ -1085,7 +1088,9 @@ function runBench(options = {}) {
1085
1088
  const selected = selectScenarios(all, options.scenarioIds || []);
1086
1089
  const benchId = options.benchId || newBenchId();
1087
1090
  const startedAt = new Date().toISOString();
1088
- const extraEnv = options.env || {};
1091
+ // C-08 ablation: switch gate components off inside the scenario repos.
1092
+ const without = require('./ablation.js').validateComponents(options.without || []);
1093
+ const extraEnv = { ...(options.env || {}), ...(without.length ? { AEL_ABLATE: without.join(',') } : {}) };
1089
1094
  const cli = options.cli || cliPath(options.packageRoot);
1090
1095
 
1091
1096
  let scenarioScores;
@@ -1151,14 +1156,32 @@ function runBench(options = {}) {
1151
1156
  status,
1152
1157
  unavailableReason,
1153
1158
  repeat,
1159
+ ...(without.length ? { ablation: { without } } : {}),
1154
1160
  summary: summarize(scenarioScores),
1155
1161
  scenarios: scenarioScores,
1156
1162
  instructionViolations
1157
1163
  });
1158
1164
  const directory = writeBenchOutputs(rootDir, benchId, result);
1165
+ if (without.length) appendSimplificationLog(rootDir, { benchId, without, summary: result.summary });
1159
1166
  return { ...result, directory };
1160
1167
  }
1161
1168
 
1169
+ // Simplification log (OpenAI pack QUALITY_SCORE.md): every ablation bench is
1170
+ // recorded; a human fills `decision` (keep | remove | replace) after
1171
+ // comparing it with a baseline bench (`bench compare`).
1172
+ function appendSimplificationLog(rootDir, { benchId, without, summary }) {
1173
+ const filePath = path.join(rootDir, '.ai-engineering-loop', 'simplification-log.json');
1174
+ let log = { schemaVersion: 1, entries: [] };
1175
+ try {
1176
+ log = JSON.parse(fs.readFileSync(filePath, 'utf8'));
1177
+ } catch {
1178
+ // first entry
1179
+ }
1180
+ log.entries = [...(log.entries || []), { at: new Date().toISOString(), benchId, without, complianceRate: summary?.complianceRate ?? null, decision: null }];
1181
+ fs.mkdirSync(path.dirname(filePath), { recursive: true });
1182
+ fs.writeFileSync(filePath, `${JSON.stringify(log, null, 2)}\n`);
1183
+ }
1184
+
1162
1185
  function loadBenchResult(rootDir, benchId) {
1163
1186
  const filePath = path.join(benchDir(rootDir, benchId), 'results.json');
1164
1187
  if (!fs.existsSync(filePath)) throw benchError(`No bench result ${benchId}`, 'BENCH_NOT_FOUND');
@@ -0,0 +1,112 @@
1
+ 'use strict';
2
+
3
+ // Evaluator calibration (C-13, learn-harness-engineering L11 + L14 anchors):
4
+ // pin the Devil's Advocate and Judge to reality. Post-delivery outcomes (a
5
+ // human MR comment, a revert, a bug traced to a Run) are compared with that
6
+ // Run's Finding Ledger, so misses and wrong dismissals become visible and
7
+ // can drive prompt changes (tracked by the reviewer prompt hash, C-16).
8
+
9
+ const fs = require('fs');
10
+ const path = require('path');
11
+ const { runsDir, loadRun } = require('./run-state.js');
12
+
13
+ const OUTCOMES_FILE = 'outcomes.jsonl';
14
+ const SOURCES = Object.freeze(['mr-comment', 'revert', 'bug', 'incident']);
15
+
16
+ function calibrationError(message, code) {
17
+ return Object.assign(new Error(message), { code });
18
+ }
19
+
20
+ const fileOf = (location) => String(location || '').replace(/:\d+(?::\d+)?$/, '');
21
+
22
+ function recordOutcome(rootDir, { runId, source, location, summary, evidence = null, by, now = new Date() } = {}) {
23
+ if (!runId) throw calibrationError('outcome record needs --run <id> (the delivered Run the outcome traces to).', 'INVALID_USAGE');
24
+ if (!SOURCES.includes(source)) throw calibrationError(`--source must be one of ${SOURCES.join(', ')}`, 'INVALID_USAGE');
25
+ if (!String(location || '').trim()) throw calibrationError('--location <file[:line]> is required so the outcome can be matched to findings.', 'INVALID_USAGE');
26
+ if (!String(summary || '').trim()) throw calibrationError('--summary <text> is required.', 'INVALID_USAGE');
27
+ if (!String(by || '').trim()) throw calibrationError('--by <name> is required: outcomes are human-reported evidence.', 'INVALID_USAGE');
28
+ loadRun(rootDir, runId);
29
+ const entry = { source, location: String(location).trim(), summary: String(summary).trim(), by: String(by).trim(), at: now.toISOString(), ...(evidence ? { evidence } : {}) };
30
+ fs.appendFileSync(path.join(runsDir(rootDir), runId, OUTCOMES_FILE), `${JSON.stringify(entry)}\n`, { mode: 0o600 });
31
+ return entry;
32
+ }
33
+
34
+ function readJsonLines(filePath) {
35
+ let text = '';
36
+ try {
37
+ text = fs.readFileSync(filePath, 'utf8');
38
+ } catch {
39
+ return [];
40
+ }
41
+ return text.split('\n').filter((line) => line.trim()).flatMap((line) => {
42
+ try {
43
+ return [JSON.parse(line)];
44
+ } catch {
45
+ return [];
46
+ }
47
+ });
48
+ }
49
+
50
+ // Classify one outcome against the Run's ledger findings.
51
+ function classifyOutcome(outcome, findings) {
52
+ const file = fileOf(outcome.location);
53
+ const related = (findings || []).filter((finding) => fileOf(finding.location) === file);
54
+ if (!related.length) return { verdict: 'DA_MISS', findings: [] };
55
+ const dismissed = related.filter((finding) => finding.validity === 'INVALID' || finding.dismissal);
56
+ if (dismissed.length) return { verdict: 'FALSE_DISMISS', findings: dismissed.map((finding) => finding.id) };
57
+ return { verdict: 'CAUGHT_BUT_SHIPPED', findings: related.map((finding) => finding.id) };
58
+ }
59
+
60
+ function calibrate(rootDir) {
61
+ let runs = [];
62
+ try {
63
+ runs = fs.readdirSync(runsDir(rootDir), { withFileTypes: true }).filter((entry) => entry.isDirectory()).map((entry) => entry.name).sort();
64
+ } catch {
65
+ runs = [];
66
+ }
67
+ const rows = [];
68
+ const byPrompt = {};
69
+ for (const runId of runs) {
70
+ const outcomes = readJsonLines(path.join(runsDir(rootDir), runId, OUTCOMES_FILE));
71
+ if (!outcomes.length) continue;
72
+ let ledger = null;
73
+ try {
74
+ ledger = JSON.parse(fs.readFileSync(path.join(runsDir(rootDir), runId, 'findings.json'), 'utf8'));
75
+ } catch {
76
+ ledger = null;
77
+ }
78
+ const prompt = ledger?.reviewer?.promptSha256 ? ledger.reviewer.promptSha256.slice(0, 12) : 'unknown';
79
+ for (const outcome of outcomes) {
80
+ const result = classifyOutcome(outcome, ledger?.findings);
81
+ rows.push({ runId, ...outcome, ...result, prompt });
82
+ const bucket = byPrompt[prompt] || (byPrompt[prompt] = { outcomes: 0, DA_MISS: 0, FALSE_DISMISS: 0, CAUGHT_BUT_SHIPPED: 0 });
83
+ bucket.outcomes += 1;
84
+ bucket[result.verdict] += 1;
85
+ }
86
+ }
87
+ const count = (verdict) => rows.filter((row) => row.verdict === verdict).length;
88
+ return {
89
+ outcomes: rows.length,
90
+ daMiss: count('DA_MISS'),
91
+ falseDismiss: count('FALSE_DISMISS'),
92
+ caughtButShipped: count('CAUGHT_BUT_SHIPPED'),
93
+ byPrompt,
94
+ rows
95
+ };
96
+ }
97
+
98
+ function formatCalibration(report) {
99
+ const lines = [`Outcomes: ${report.outcomes} (DA missed ${report.daMiss}, wrongly dismissed ${report.falseDismiss}, caught but shipped ${report.caughtButShipped})`];
100
+ for (const row of report.rows.filter((r) => r.verdict !== 'CAUGHT_BUT_SHIPPED')) {
101
+ lines.push(`- ${row.verdict} ${row.runId} ${row.location}: ${row.summary}${row.findings.length ? ` (findings ${row.findings.join(', ')})` : ''}`);
102
+ }
103
+ if (report.daMiss || report.falseDismiss) {
104
+ lines.push('Next: add each miss as a concrete check to agents/shared/devil-advocate.body.md (or a learn promote rule), then compare the next prompt hash here.');
105
+ }
106
+ for (const [prompt, bucket] of Object.entries(report.byPrompt)) {
107
+ lines.push(`- prompt ${prompt}: ${bucket.outcomes} outcome(s), ${bucket.DA_MISS} miss, ${bucket.FALSE_DISMISS} false dismiss`);
108
+ }
109
+ return lines.join('\n');
110
+ }
111
+
112
+ module.exports = { OUTCOMES_FILE, SOURCES, calibrate, classifyOutcome, formatCalibration, recordOutcome };