ai-engineering-loop 2.5.0 → 2.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/skills/ael/SKILL.md +1 -1
- package/.agents/skills/ael/references/loop-commands.md +11 -0
- package/.agents/skills/ael/references/loop-goal.md +2 -0
- package/.agents/skills/ael/references/loop-maker.md +1 -6
- package/.agents/skills/ael/references/loop-report.md +8 -0
- package/.agents/skills/ael/references/loop-verify.md +21 -11
- package/.claude/agents/devil-advocate.md +6 -1
- package/.claude/agents/judge.md +3 -1
- package/.grok/agents/devil-advocate.md +6 -1
- package/.grok/agents/judge.md +3 -1
- package/CHANGELOG.md +22 -0
- package/agents/devil-advocate.md +8 -0
- package/agents/judge.md +1 -0
- package/agents/shared/devil-advocate.body.md +6 -1
- package/agents/shared/judge.body.md +3 -1
- package/bin/ai-engineering-loop.js +262 -20
- package/core/bench.md +15 -0
- package/core/goal-contract.md +35 -0
- package/core/verification-loop.md +96 -7
- package/lib/ablation.js +52 -0
- package/lib/ac-status.js +63 -0
- package/lib/baseline.js +138 -0
- package/lib/bench.js +24 -1
- package/lib/calibration.js +112 -0
- package/lib/claimed-vs-reality.js +114 -16
- package/lib/clean-state.js +85 -0
- package/lib/cli-args.js +5 -3
- package/lib/debt.js +110 -0
- package/lib/features.js +95 -0
- package/lib/gates.js +234 -41
- package/lib/harness-audit.js +124 -0
- package/lib/no-progress.js +6 -1
- package/lib/review-ingest.js +54 -2
- package/lib/rules.js +112 -0
- package/lib/stats.js +90 -0
- package/lib/sync-hosts.js +1 -0
- package/lib/verification-integrity.js +164 -0
- package/lib/verification-lanes.js +64 -0
- package/lib/verification-recorder.js +17 -5
- package/package.json +1 -1
- package/schemas/finding-ledger.schema.json +14 -0
- package/schemas/goal-contract.schema.json +31 -0
- package/schemas/verdict.schema.json +2 -1
|
@@ -43,9 +43,58 @@ A verification `PASS` is **strictly invalid** without concrete execution evidenc
|
|
|
43
43
|
}
|
|
44
44
|
```
|
|
45
45
|
|
|
46
|
+
## 3. Verification Layers & Command Mapping
|
|
47
|
+
|
|
48
|
+
Verification terminates through three explicit layers of evidence. Unit tests alone have systematic blind spots for cross-component and integration defects. An acceptance criterion crossing component boundaries requires Layer 3 verification or written justification if Layer 3 is unavailable.
|
|
49
|
+
|
|
50
|
+
| Layer | Scope | Configured Command (`verification.md`) | Purpose |
|
|
51
|
+
|---|---|---|---|
|
|
52
|
+
| **L1: Static / Syntax** | Syntax, static types, style, compilation | `typecheck`, `lint`, `build` | Fast static catch before runtime. Rejects type errors, lint regressions, and compilation breakage. |
|
|
53
|
+
| **L2: Behavioral / Runtime** | Functions, units, modules, regressions | `test_unit`, `test_related`, `test_all` | Fast behavioral feedback at named seams and regression verification. |
|
|
54
|
+
| **L3: End-to-End** | Cross-component flows, integrations, browser/API | `e2e` (optional) | Realistic integrated proof across boundary seams. |
|
|
55
|
+
|
|
56
|
+
Unit tests isolate and mock by design, which is exactly why they miss interface mismatches, state propagation errors, resource leaks, and environment dependencies between components.
|
|
57
|
+
|
|
58
|
+
| Tier | Required layers |
|
|
59
|
+
|---|---|
|
|
60
|
+
| T0 | L1-L2 (L3 when an AC is `crossComponent: true`) |
|
|
61
|
+
| T1 | L1-L2, plus L3 for every `crossComponent: true` AC |
|
|
62
|
+
| T2 | L1-L3 on every AC |
|
|
63
|
+
|
|
64
|
+
Enforcement (`lib/claimed-vs-reality.js`, `gate review`):
|
|
65
|
+
- The Layer cell accepts `L1`, `L2`, `L3`, combos such as `L2+L3`, and `L3 waived: <reason>` (at least 15 characters). Unknown tokens fail the gate.
|
|
66
|
+
- An AC that needs L3 without L3 or a reasoned waiver fails `gate review` on every tier, with a FIX line that says what to run.
|
|
67
|
+
- Legacy 3-column files stay valid when no AC needs L3.
|
|
68
|
+
- Tags are bound to recorded evidence (C-01): `L1` needs a passing L1 command in `verification.json`/`final-verification.json`; `L3` needs a passing `e2e` command, or Reality that describes a manual end-to-end flow (steps, URL, screenshot path). Commands not mapped in `verification.md` count as L2. Waivers need no command.
|
|
69
|
+
- When `e2e` is empty in `verification.md`, run the AC's real user flow and cite the observation, or write the waiver. Devil's Advocate and Judge challenge waivers like any other claim.
|
|
70
|
+
|
|
46
71
|
---
|
|
47
72
|
|
|
48
|
-
##
|
|
73
|
+
## 3a. Baseline and Default Plan
|
|
74
|
+
|
|
75
|
+
**Baseline (C-10).** After the Goal gate and before the Maker gate, `verify --baseline --run <id>` runs the default plan on the untouched tree and writes `runs/<id>/baseline.json` with status `GREEN`, `RED`, or `BROKEN`.
|
|
76
|
+
- `BROKEN` (a command exits 127, i.e. it does not exist): the command fails with a FIX to correct `verification.md`. Verification cannot prove anything with a broken command.
|
|
77
|
+
- `RED`: the tree was already failing. Later, when `verify` fails, each failing command is labelled `pre-existing` (every failing test name was already failing at baseline) or new. Unnamed failures are always treated as new. The gate still blocks; the label tells the Maker whether the change caused it.
|
|
78
|
+
|
|
79
|
+
**Default plan (C-02).** `verify` without `--cmd` runs `build -> typecheck -> lint -> test_unit` (whichever are configured), serially, and stops at the first failure. Commands that rewrite files (`--fix`, `--write`) are never run by default. Each record carries its layer (`L1`/`L2`/`L3`).
|
|
80
|
+
|
|
81
|
+
**L1 at T1/T2.** When `verification.md` configures `typecheck` or `lint`, the verification gate at T1/T2 refuses a full-scope bundle with no L1 record (`L1_MISSING`). The related lane is exempt; `verify --final` runs the full plan.
|
|
82
|
+
|
|
83
|
+
`init` writes only commands the project really has: `typecheck` from a script or a TypeScript setup, `lint` and `build` only from package scripts.
|
|
84
|
+
|
|
85
|
+
**Smoke (C-03).** An optional `smoke` key in `verification.md` is a command that starts the app, probes readiness (a health URL, a golden journey), stops it, and exits. It counts as L2. When the gated diff touches startup-risky files (migrations, `.env*`, `config/`, `*.config.*`, Docker/Procfile, `main|index|server|app` entrypoints, dependency manifests), default `verify` runs it after `test_unit`, and the verification gate fails with `SMOKE_MISSING` when it did not run. A smoke command that times out or exits non-zero blocks the gate; on timeout the recorder kills its process group.
|
|
86
|
+
|
|
87
|
+
## 3b. Verification Integrity (C-11)
|
|
88
|
+
|
|
89
|
+
At `gate review` the CLI scans the gated diff and adds `source: "verification-integrity"` findings (HIGH; BLOCKER at T2, VALID, with a concrete fix) for:
|
|
90
|
+
- a skip/only/todo marker added in a test file (`it.skip`, `xit`, `fit`, `@Disabled`, `pytest.mark.skip`, `t.Skip`, ...);
|
|
91
|
+
- a test file deleted without a replacement test in the same directory;
|
|
92
|
+
- assertions removed from a test file without replacement;
|
|
93
|
+
- a change to verification config (`.ai-engineering-loop/verification.md`, jest/vitest/playwright/cypress config, `pytest.ini`, coverage config) that the frozen Goal Contract does not name.
|
|
94
|
+
|
|
95
|
+
These findings cannot be dropped or downgraded by the reviewer. Only a human dismissal (`dismissal: {by, evidence}`) can downgrade them; otherwise the review gate fails.
|
|
96
|
+
|
|
97
|
+
## 4. The 4 Verification Gates
|
|
49
98
|
|
|
50
99
|
1. **Gate 1: Unit & Integration Tests**: All unit and edge-case tests must pass with 0 failures.
|
|
51
100
|
2. **Gate 2: Static Type Analysis**: `tsc --noEmit` or compiler type checking must exit with code `0` (zero type errors).
|
|
@@ -54,7 +103,7 @@ A verification `PASS` is **strictly invalid** without concrete execution evidenc
|
|
|
54
103
|
|
|
55
104
|
---
|
|
56
105
|
|
|
57
|
-
##
|
|
106
|
+
## 5. Rejection Triggers
|
|
58
107
|
|
|
59
108
|
The Verification Gate immediately halts and returns to the Maker if:
|
|
60
109
|
- Any test suite fails or times out.
|
|
@@ -66,7 +115,7 @@ The Verification Gate immediately halts and returns to the Maker if:
|
|
|
66
115
|
|
|
67
116
|
---
|
|
68
117
|
|
|
69
|
-
##
|
|
118
|
+
## 6. Claimed vs Reality (before Devil's Advocate)
|
|
70
119
|
|
|
71
120
|
Write `.ai-engineering-loop/tasks/claimed-vs-reality.md` after verification logs exist:
|
|
72
121
|
|
|
@@ -74,14 +123,14 @@ Write `.ai-engineering-loop/tasks/claimed-vs-reality.md` after verification logs
|
|
|
74
123
|
npx ai-engineering-loop context claimed-vs-reality --run <run-id>
|
|
75
124
|
```
|
|
76
125
|
|
|
77
|
-
This scaffolds one row per frozen acceptance criterion but leaves Claimed and
|
|
126
|
+
This scaffolds one row per frozen acceptance criterion but leaves Claimed, Reality, and Layer empty. Fill them from the implemented change and actual command output; the scaffold is not verification evidence.
|
|
78
127
|
|
|
79
128
|
```markdown
|
|
80
129
|
# Claimed vs Reality
|
|
81
130
|
|
|
82
|
-
| AC | Claimed | Reality |
|
|
83
|
-
|
|
84
|
-
| AC-1 | <what Maker said is done> | <command, exit code, assertion or log line> |
|
|
131
|
+
| AC | Claimed | Reality from command log | Layer |
|
|
132
|
+
|---|---|---|---|
|
|
133
|
+
| AC-1 | <what Maker said is done> | <command, exit code, assertion or log line> | L2 |
|
|
85
134
|
```
|
|
86
135
|
|
|
87
136
|
Do **not** spawn Devil's Advocate if:
|
|
@@ -92,3 +141,43 @@ Do **not** spawn Devil's Advocate if:
|
|
|
92
141
|
- Reality is only `seems green`
|
|
93
142
|
|
|
94
143
|
Parser: `lib/claimed-vs-reality.js` (`readyForDa`). Pass that path to DA/Judge with the verification log.
|
|
144
|
+
|
|
145
|
+
## 7. Clean State at Delivery (C-06)
|
|
146
|
+
|
|
147
|
+
`gate delivery` also checks that the next session starts from a tree it can trust:
|
|
148
|
+
- **Always blocks:** a `debugger` statement, merge-conflict marker or interactive breakpoint added by the change; a secret-like file in the change (`.env`, `.env.local`, keys; `.env.example` is fine); a `_templateOnly` scaffold left in the Run directory or `.ai-engineering-loop/tasks/`.
|
|
149
|
+
- **Untracked files that are not part of the reviewed diff** (logs, coverage, scratch files): block in UNATTENDED, warn in ASSISTED. Warnings are printed and kept in the Run history as `cleanStateWarnings`.
|
|
150
|
+
|
|
151
|
+
## 8. Debt Ledger (C-14)
|
|
152
|
+
|
|
153
|
+
At `gate delivery` the CLI copies two kinds of deferred work into `.ai-engineering-loop/debt.json` (tracked in git, like `learnings.json`):
|
|
154
|
+
- VALID `MEDIUM`/`LOW` findings the Judge accepted as trade-offs (file, scenario, risk, "revisit on the next change touching <file>");
|
|
155
|
+
- claimed-vs-reality rows with `L3 waived: <reason>` ("revisit when an e2e command exists").
|
|
156
|
+
|
|
157
|
+
`review-pack` appends open debt on the files the new diff touches to the Devil's Advocate pack and lists it. `ael debt` lists open items (`--file <path>`); `ael debt close <id> --by <name> [--note <text>]` closes one.
|
|
158
|
+
|
|
159
|
+
## 9. Reviewer Inputs Are Logged (C-16)
|
|
160
|
+
|
|
161
|
+
"Model-visible means logged": `review-pack` writes `runs/<id>/review-inputs.json` with the sha256 of the review pack, the gated diff, and the Devil's Advocate prompt body. `review ingest` copies them into `findings.json` (`reviewer.inputs`, `reviewer.promptSha256`). `gate review` fails the ledger binding when any recorded input changed after the reviewer saw it, so a missed defect can always be traced to exactly what the reviewer was shown.
|
|
162
|
+
|
|
163
|
+
## 10. Reviewer Calibration (C-13)
|
|
164
|
+
|
|
165
|
+
Gates prove the change matches the Goal; only reality proves the reviewers were right. After delivery, record problems people found:
|
|
166
|
+
|
|
167
|
+
```
|
|
168
|
+
ael outcome record --run <id> --source mr-comment|revert|bug|incident --location <file[:line]> --summary <text> --by <name> [--evidence <path|url>]
|
|
169
|
+
ael calibrate [--json]
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
`calibrate` compares each outcome with that Run's Finding Ledger: `DA_MISS` (no finding on that file), `FALSE_DISMISS` (a finding on that file was marked INVALID or dismissed), `CAUGHT_BUT_SHIPPED` (a VALID finding was accepted as a trade-off). Results are grouped by the reviewer prompt hash (C-16), so after you tighten `agents/shared/devil-advocate.body.md` for a recurring miss you can see whether the next prompt version misses less. Plan for several tuning rounds; record each prompt change in its commit message.
|
|
173
|
+
|
|
174
|
+
## 11. Promoting Review Feedback to Rules (C-05)
|
|
175
|
+
|
|
176
|
+
`learn` lists **promotion candidates**: learnings seen in 3 or more Runs. A human turns one into an executable check:
|
|
177
|
+
|
|
178
|
+
```
|
|
179
|
+
ael learn promote L-abc123 --check "! grep -rq \"require('fs')\" src/renderer" --fix "Move file access to src/preload/file-ops.ts and call window.api" --by <name> [--why <text>]
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
Rules live in `.ai-engineering-loop/rules.json` (tracked). Default `verify` runs every rule as an L1 check after the static checks; a violation prints `Rule <id> violated. WHY: ... FIX: ...`. The verification gate refuses a full-scope bundle in which any rule has no passing record (`RULES_MISSING`). The check must exit non-zero on a violation and is run through the shell unless `--no-shell` is given.
|
|
183
|
+
|
package/lib/ablation.js
ADDED
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
'use strict';
|
|
2
|
+
|
|
3
|
+
// Harness ablation for bench (C-08, learn-harness-engineering L02/L12):
|
|
4
|
+
// switch one gate component off inside a bench scenario repo to measure
|
|
5
|
+
// whether it still earns its cost. Honoured only when bench set it up (env
|
|
6
|
+
// AEL_BENCH_MODE + AEL_ABLATE and a marker file bench wrote into the repo),
|
|
7
|
+
// and every skip is logged to the Run so it can never pass silently.
|
|
8
|
+
|
|
9
|
+
const fs = require('fs');
|
|
10
|
+
const path = require('path');
|
|
11
|
+
const { runsDir } = require('./run-state.js');
|
|
12
|
+
|
|
13
|
+
const COMPONENTS = Object.freeze(['integrity', 'l3', 'l1', 'smoke', 'rules']);
|
|
14
|
+
const MARKER = path.join('.ai-engineering-loop', 'bench-ablation.json');
|
|
15
|
+
|
|
16
|
+
function requested(env = process.env) {
|
|
17
|
+
return String(env.AEL_ABLATE || '').split(',').map((item) => item.trim()).filter(Boolean);
|
|
18
|
+
}
|
|
19
|
+
|
|
20
|
+
function writeMarker(repo, components) {
|
|
21
|
+
fs.mkdirSync(path.join(repo, '.ai-engineering-loop'), { recursive: true });
|
|
22
|
+
fs.writeFileSync(path.join(repo, MARKER), `${JSON.stringify({ schemaVersion: 1, components }, null, 2)}\n`);
|
|
23
|
+
}
|
|
24
|
+
|
|
25
|
+
function ablated(rootDir, component, runId = null, env = process.env) {
|
|
26
|
+
if (!env.AEL_BENCH_MODE || !requested(env).includes(component)) return false;
|
|
27
|
+
let marker = null;
|
|
28
|
+
try {
|
|
29
|
+
marker = JSON.parse(fs.readFileSync(path.join(rootDir, MARKER), 'utf8'));
|
|
30
|
+
} catch {
|
|
31
|
+
return false;
|
|
32
|
+
}
|
|
33
|
+
if (!Array.isArray(marker?.components) || !marker.components.includes(component)) return false;
|
|
34
|
+
if (runId) {
|
|
35
|
+
try {
|
|
36
|
+
fs.appendFileSync(path.join(runsDir(rootDir), runId, 'ablation.log'), `${JSON.stringify({ component, at: new Date().toISOString() })}\n`);
|
|
37
|
+
} catch {
|
|
38
|
+
// Run directory missing; the bench result still records the ablation.
|
|
39
|
+
}
|
|
40
|
+
}
|
|
41
|
+
return true;
|
|
42
|
+
}
|
|
43
|
+
|
|
44
|
+
function validateComponents(list) {
|
|
45
|
+
const unknown = list.filter((item) => !COMPONENTS.includes(item));
|
|
46
|
+
if (unknown.length) {
|
|
47
|
+
throw Object.assign(new Error(`Unknown --without component(s): ${unknown.join(', ')}. Use: ${COMPONENTS.join(', ')}`), { code: 'BENCH_USAGE' });
|
|
48
|
+
}
|
|
49
|
+
return list;
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
module.exports = { COMPONENTS, MARKER, ablated, requested, validateComponents, writeMarker };
|
package/lib/ac-status.js
ADDED
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
'use strict';
|
|
2
|
+
|
|
3
|
+
// Harness-owned AC status (C-07, learn-harness-engineering L08): an AC that
|
|
4
|
+
// names its own verify command is `proven` only when that exact command
|
|
5
|
+
// passed on the gated diff. Agents cannot set the status; evidence does.
|
|
6
|
+
|
|
7
|
+
const fs = require('fs');
|
|
8
|
+
const path = require('path');
|
|
9
|
+
const { runsDir } = require('./run-state.js');
|
|
10
|
+
|
|
11
|
+
const norm = (text) => String(text || '').trim().replace(/\s+/g, ' ');
|
|
12
|
+
|
|
13
|
+
function recordedCommands(rootDir, runId) {
|
|
14
|
+
const records = [];
|
|
15
|
+
for (const file of ['verification.json', 'final-verification.json']) {
|
|
16
|
+
try {
|
|
17
|
+
const bundle = JSON.parse(fs.readFileSync(path.join(runsDir(rootDir), runId, file), 'utf8'));
|
|
18
|
+
records.push(...(bundle.commands || []));
|
|
19
|
+
} catch {
|
|
20
|
+
// Not recorded yet.
|
|
21
|
+
}
|
|
22
|
+
}
|
|
23
|
+
return records;
|
|
24
|
+
}
|
|
25
|
+
|
|
26
|
+
// [{ id, command, status: 'proven'|'failed'|'pending'|'unbound' }]
|
|
27
|
+
function acStatuses(goal, records) {
|
|
28
|
+
return (goal?.acceptanceCriteria || []).map((criterion) => {
|
|
29
|
+
const command = norm(criterion.verify?.command);
|
|
30
|
+
if (!command) return { id: criterion.id, command: null, status: 'unbound' };
|
|
31
|
+
const matches = (records || []).filter((record) => norm(record.command) === command);
|
|
32
|
+
if (matches.some((record) => record.exitCode === 0 && record.timeoutStatus === 'COMPLETED')) {
|
|
33
|
+
return { id: criterion.id, command, status: 'proven' };
|
|
34
|
+
}
|
|
35
|
+
return { id: criterion.id, command, status: matches.length ? 'failed' : 'pending' };
|
|
36
|
+
});
|
|
37
|
+
}
|
|
38
|
+
|
|
39
|
+
function runAcStatuses(rootDir, run, goal) {
|
|
40
|
+
return acStatuses(goal, recordedCommands(rootDir, run.runId));
|
|
41
|
+
}
|
|
42
|
+
|
|
43
|
+
// C-09 (learn-harness-engineering L09/L11): subjective ACs (UI, copy,
|
|
44
|
+
// design) carry a rubric. The ledger must score every criterion with
|
|
45
|
+
// evidence; an unscored or failed criterion blocks Judge PASS.
|
|
46
|
+
function rubricProblems(goal, ledger) {
|
|
47
|
+
const problems = [];
|
|
48
|
+
const scores = Array.isArray(ledger?.rubricScores) ? ledger.rubricScores : [];
|
|
49
|
+
for (const criterion of goal?.acceptanceCriteria || []) {
|
|
50
|
+
for (const item of Array.isArray(criterion.rubric) ? criterion.rubric : []) {
|
|
51
|
+
const name = String(item.criterion || '').trim();
|
|
52
|
+
const score = scores.find((entry) => entry.ac === criterion.id && String(entry.criterion || '').trim() === name);
|
|
53
|
+
if (!score || typeof score.pass !== 'boolean' || !String(score.evidence || '').trim()) {
|
|
54
|
+
problems.push(`${criterion.id} rubric "${name}" has no score with evidence in the Finding Ledger (rubricScores)`);
|
|
55
|
+
} else if (!score.pass) {
|
|
56
|
+
problems.push(`${criterion.id} rubric "${name}" failed: ${score.evidence}`);
|
|
57
|
+
}
|
|
58
|
+
}
|
|
59
|
+
}
|
|
60
|
+
return problems;
|
|
61
|
+
}
|
|
62
|
+
|
|
63
|
+
module.exports = { acStatuses, recordedCommands, rubricProblems, runAcStatuses };
|
package/lib/baseline.js
ADDED
|
@@ -0,0 +1,138 @@
|
|
|
1
|
+
'use strict';
|
|
2
|
+
|
|
3
|
+
// Baseline verification (C-10, learn-harness-engineering L06 + startup flow):
|
|
4
|
+
// before the Maker edits anything, run the project's own verification plan
|
|
5
|
+
// once on the frozen starting state. A broken command is caught before work
|
|
6
|
+
// starts, and failures that already existed are told apart from regressions.
|
|
7
|
+
|
|
8
|
+
const fs = require('fs');
|
|
9
|
+
const path = require('path');
|
|
10
|
+
const { atomicWriteJson, getCurrentRun, getGitRevision, loadRun, runsDir } = require('./run-state.js');
|
|
11
|
+
|
|
12
|
+
const BASELINE_FILE = 'baseline.json';
|
|
13
|
+
const COMMAND_NOT_FOUND = 127;
|
|
14
|
+
|
|
15
|
+
function baselineError(message, code, fix) {
|
|
16
|
+
return Object.assign(new Error(fix ? `${message} FIX: ${fix}` : message), { code });
|
|
17
|
+
}
|
|
18
|
+
|
|
19
|
+
function resolveRun(rootDir, runId) {
|
|
20
|
+
const run = runId ? loadRun(rootDir, runId) : getCurrentRun(rootDir);
|
|
21
|
+
if (!run) throw baselineError('No current run.', 'NO_CURRENT_RUN', 'pass --run <id>.');
|
|
22
|
+
return run;
|
|
23
|
+
}
|
|
24
|
+
|
|
25
|
+
// Failing test names from common runner output (node:test, TAP, jest, vitest,
|
|
26
|
+
// pytest). Best effort: an empty list means "could not tell", never "none".
|
|
27
|
+
function extractFailingTests(text) {
|
|
28
|
+
const names = new Set();
|
|
29
|
+
const patterns = [
|
|
30
|
+
/^\s*✖\s+(.+?)\s+\(\d+(?:\.\d+)?m?s\)\s*$/gm, // node:test spec reporter
|
|
31
|
+
/^not ok \d+ - (.+)$/gm, // TAP
|
|
32
|
+
/^\s*●\s+(.+?)\s*$/gm, // jest
|
|
33
|
+
/^\s*[×✗]\s+(.+?)(?:\s+\d+m?s)?\s*$/gm, // vitest
|
|
34
|
+
/^FAILED\s+(\S+)/gm // pytest -rf
|
|
35
|
+
];
|
|
36
|
+
const value = String(text || '');
|
|
37
|
+
for (const pattern of patterns) {
|
|
38
|
+
for (const match of value.matchAll(pattern)) {
|
|
39
|
+
const name = match[1].trim();
|
|
40
|
+
if (name && !/^failing tests:?$/i.test(name)) names.add(name);
|
|
41
|
+
}
|
|
42
|
+
}
|
|
43
|
+
return [...names].sort();
|
|
44
|
+
}
|
|
45
|
+
|
|
46
|
+
const passedRecord = (record) => record.exitCode === 0 && record.timeoutStatus === 'COMPLETED';
|
|
47
|
+
|
|
48
|
+
// Run `plan` ([{ key, layer, command, args, shell }]) serially on the current
|
|
49
|
+
// tree and write runs/<id>/baseline.json. Must run before the Maker gate, so
|
|
50
|
+
// the tree is the frozen starting point.
|
|
51
|
+
async function recordBaseline(rootDir, { runId, plan, timeoutMs, runCommand } = {}) {
|
|
52
|
+
const run = resolveRun(rootDir, runId);
|
|
53
|
+
if (run.artifacts?.diff) {
|
|
54
|
+
throw baselineError(`Run ${run.runId} already has a gated Maker diff; a baseline must describe the tree before the change.`,
|
|
55
|
+
'BASELINE_TOO_LATE', 'start a new Run, or skip the baseline for this one.');
|
|
56
|
+
}
|
|
57
|
+
if (!Array.isArray(plan) || plan.length === 0) {
|
|
58
|
+
throw baselineError('No baseline commands.', 'NO_BASELINE_COMMANDS',
|
|
59
|
+
'add build/typecheck/lint/test_unit to .ai-engineering-loop/verification.md, or pass --cmd.');
|
|
60
|
+
}
|
|
61
|
+
const recorder = require('./verification-recorder.js');
|
|
62
|
+
const exec = runCommand || recorder.runCommand;
|
|
63
|
+
const limit = Number.isInteger(timeoutMs) && timeoutMs > 0 ? timeoutMs : recorder.DEFAULT_TIMEOUT_MS;
|
|
64
|
+
const records = [];
|
|
65
|
+
const problems = [];
|
|
66
|
+
for (const item of plan) {
|
|
67
|
+
const result = await exec(item.command, item.args || [], { cwd: rootDir, timeoutMs: limit, shell: Boolean(item.shell) });
|
|
68
|
+
const record = {
|
|
69
|
+
key: item.key || null,
|
|
70
|
+
layer: item.layer || null,
|
|
71
|
+
command: result.command,
|
|
72
|
+
exitCode: result.exitCode,
|
|
73
|
+
timeoutStatus: result.timeoutStatus,
|
|
74
|
+
failingTests: passedRecord(result) ? [] : extractFailingTests(`${result.stdout}\n${result.stderr}`)
|
|
75
|
+
};
|
|
76
|
+
records.push(record);
|
|
77
|
+
if (result.exitCode === COMMAND_NOT_FOUND || result.launchError) {
|
|
78
|
+
problems.push(`${record.command}: command not found (exit ${result.exitCode}). FIX: correct the ${record.key || 'command'} entry in .ai-engineering-loop/verification.md; verification cannot prove anything with it.`);
|
|
79
|
+
break;
|
|
80
|
+
}
|
|
81
|
+
}
|
|
82
|
+
const broken = problems.length > 0;
|
|
83
|
+
const red = records.some((record) => !passedRecord(record));
|
|
84
|
+
if (!broken && red) {
|
|
85
|
+
for (const record of records.filter((r) => !passedRecord(r))) {
|
|
86
|
+
problems.push(`${record.command}: already failing before any change (exit ${record.exitCode}${record.failingTests.length ? `; ${record.failingTests.length} failing test(s)` : ''}). FIX: repair the baseline in a separate Run first, or continue knowing these failures are pre-existing.`);
|
|
87
|
+
}
|
|
88
|
+
}
|
|
89
|
+
const status = broken ? 'BROKEN' : (red ? 'RED' : 'GREEN');
|
|
90
|
+
const document = {
|
|
91
|
+
schemaVersion: 1,
|
|
92
|
+
runId: run.runId,
|
|
93
|
+
gitRevision: getGitRevision(rootDir),
|
|
94
|
+
recordedAt: new Date().toISOString(),
|
|
95
|
+
status,
|
|
96
|
+
commands: records
|
|
97
|
+
};
|
|
98
|
+
const filePath = path.join(runsDir(rootDir), run.runId, BASELINE_FILE);
|
|
99
|
+
atomicWriteJson(filePath, document);
|
|
100
|
+
return {
|
|
101
|
+
runId: run.runId,
|
|
102
|
+
path: path.relative(rootDir, filePath).split(path.sep).join('/'),
|
|
103
|
+
status,
|
|
104
|
+
commands: records,
|
|
105
|
+
problems
|
|
106
|
+
};
|
|
107
|
+
}
|
|
108
|
+
|
|
109
|
+
function loadBaseline(rootDir, runId) {
|
|
110
|
+
try {
|
|
111
|
+
return JSON.parse(fs.readFileSync(path.join(runsDir(rootDir), runId, BASELINE_FILE), 'utf8'));
|
|
112
|
+
} catch {
|
|
113
|
+
return null;
|
|
114
|
+
}
|
|
115
|
+
}
|
|
116
|
+
|
|
117
|
+
// Label failing verification records as pre-existing (every failing test was
|
|
118
|
+
// already failing at baseline) or new. Records without a baseline counterpart,
|
|
119
|
+
// or whose failures cannot be named, stay "new": unknown is never excused.
|
|
120
|
+
function classifyAgainstBaseline(baseline, records) {
|
|
121
|
+
const byCommand = new Map((baseline?.commands || []).map((record) => [record.command, record]));
|
|
122
|
+
return (records || []).filter((record) => !passedRecord(record)).map((record) => {
|
|
123
|
+
const before = byCommand.get(record.command);
|
|
124
|
+
const failing = extractFailingTests(`${record.stdout || ''}\n${record.stderr || ''}`);
|
|
125
|
+
const known = new Set(before?.failingTests || []);
|
|
126
|
+
const fresh = failing.filter((name) => !known.has(name));
|
|
127
|
+
const preExisting = Boolean(before && !passedRecord(before) && failing.length > 0 && fresh.length === 0);
|
|
128
|
+
return { command: record.command, preExisting, newFailures: preExisting ? [] : (fresh.length ? fresh : failing) };
|
|
129
|
+
});
|
|
130
|
+
}
|
|
131
|
+
|
|
132
|
+
module.exports = {
|
|
133
|
+
BASELINE_FILE,
|
|
134
|
+
classifyAgainstBaseline,
|
|
135
|
+
extractFailingTests,
|
|
136
|
+
loadBaseline,
|
|
137
|
+
recordBaseline
|
|
138
|
+
};
|
package/lib/bench.js
CHANGED
|
@@ -817,6 +817,9 @@ function prepareScenarioRepo(repo, scenario, { host, tokenCap, extraEnv, cli, pa
|
|
|
817
817
|
const env = extraEnv;
|
|
818
818
|
const init = spawnNode([cli, 'init'], { cwd: repo, env });
|
|
819
819
|
if (init.status !== 0) throw benchError(`init failed: ${init.stdout}\n${init.stderr}`, 'BENCH_SETUP');
|
|
820
|
+
const ablation = require('./ablation.js');
|
|
821
|
+
const components = ablation.requested(env || {});
|
|
822
|
+
if (components.length) ablation.writeMarker(repo, components);
|
|
820
823
|
installProjectHostSkills(repo, host, packageRoot);
|
|
821
824
|
installLocalCli(repo, cli);
|
|
822
825
|
if (host.hooks) {
|
|
@@ -1085,7 +1088,9 @@ function runBench(options = {}) {
|
|
|
1085
1088
|
const selected = selectScenarios(all, options.scenarioIds || []);
|
|
1086
1089
|
const benchId = options.benchId || newBenchId();
|
|
1087
1090
|
const startedAt = new Date().toISOString();
|
|
1088
|
-
|
|
1091
|
+
// C-08 ablation: switch gate components off inside the scenario repos.
|
|
1092
|
+
const without = require('./ablation.js').validateComponents(options.without || []);
|
|
1093
|
+
const extraEnv = { ...(options.env || {}), ...(without.length ? { AEL_ABLATE: without.join(',') } : {}) };
|
|
1089
1094
|
const cli = options.cli || cliPath(options.packageRoot);
|
|
1090
1095
|
|
|
1091
1096
|
let scenarioScores;
|
|
@@ -1151,14 +1156,32 @@ function runBench(options = {}) {
|
|
|
1151
1156
|
status,
|
|
1152
1157
|
unavailableReason,
|
|
1153
1158
|
repeat,
|
|
1159
|
+
...(without.length ? { ablation: { without } } : {}),
|
|
1154
1160
|
summary: summarize(scenarioScores),
|
|
1155
1161
|
scenarios: scenarioScores,
|
|
1156
1162
|
instructionViolations
|
|
1157
1163
|
});
|
|
1158
1164
|
const directory = writeBenchOutputs(rootDir, benchId, result);
|
|
1165
|
+
if (without.length) appendSimplificationLog(rootDir, { benchId, without, summary: result.summary });
|
|
1159
1166
|
return { ...result, directory };
|
|
1160
1167
|
}
|
|
1161
1168
|
|
|
1169
|
+
// Simplification log (OpenAI pack QUALITY_SCORE.md): every ablation bench is
|
|
1170
|
+
// recorded; a human fills `decision` (keep | remove | replace) after
|
|
1171
|
+
// comparing it with a baseline bench (`bench compare`).
|
|
1172
|
+
function appendSimplificationLog(rootDir, { benchId, without, summary }) {
|
|
1173
|
+
const filePath = path.join(rootDir, '.ai-engineering-loop', 'simplification-log.json');
|
|
1174
|
+
let log = { schemaVersion: 1, entries: [] };
|
|
1175
|
+
try {
|
|
1176
|
+
log = JSON.parse(fs.readFileSync(filePath, 'utf8'));
|
|
1177
|
+
} catch {
|
|
1178
|
+
// first entry
|
|
1179
|
+
}
|
|
1180
|
+
log.entries = [...(log.entries || []), { at: new Date().toISOString(), benchId, without, complianceRate: summary?.complianceRate ?? null, decision: null }];
|
|
1181
|
+
fs.mkdirSync(path.dirname(filePath), { recursive: true });
|
|
1182
|
+
fs.writeFileSync(filePath, `${JSON.stringify(log, null, 2)}\n`);
|
|
1183
|
+
}
|
|
1184
|
+
|
|
1162
1185
|
function loadBenchResult(rootDir, benchId) {
|
|
1163
1186
|
const filePath = path.join(benchDir(rootDir, benchId), 'results.json');
|
|
1164
1187
|
if (!fs.existsSync(filePath)) throw benchError(`No bench result ${benchId}`, 'BENCH_NOT_FOUND');
|
|
@@ -0,0 +1,112 @@
|
|
|
1
|
+
'use strict';
|
|
2
|
+
|
|
3
|
+
// Evaluator calibration (C-13, learn-harness-engineering L11 + L14 anchors):
|
|
4
|
+
// pin the Devil's Advocate and Judge to reality. Post-delivery outcomes (a
|
|
5
|
+
// human MR comment, a revert, a bug traced to a Run) are compared with that
|
|
6
|
+
// Run's Finding Ledger, so misses and wrong dismissals become visible and
|
|
7
|
+
// can drive prompt changes (tracked by the reviewer prompt hash, C-16).
|
|
8
|
+
|
|
9
|
+
const fs = require('fs');
|
|
10
|
+
const path = require('path');
|
|
11
|
+
const { runsDir, loadRun } = require('./run-state.js');
|
|
12
|
+
|
|
13
|
+
const OUTCOMES_FILE = 'outcomes.jsonl';
|
|
14
|
+
const SOURCES = Object.freeze(['mr-comment', 'revert', 'bug', 'incident']);
|
|
15
|
+
|
|
16
|
+
function calibrationError(message, code) {
|
|
17
|
+
return Object.assign(new Error(message), { code });
|
|
18
|
+
}
|
|
19
|
+
|
|
20
|
+
const fileOf = (location) => String(location || '').replace(/:\d+(?::\d+)?$/, '');
|
|
21
|
+
|
|
22
|
+
function recordOutcome(rootDir, { runId, source, location, summary, evidence = null, by, now = new Date() } = {}) {
|
|
23
|
+
if (!runId) throw calibrationError('outcome record needs --run <id> (the delivered Run the outcome traces to).', 'INVALID_USAGE');
|
|
24
|
+
if (!SOURCES.includes(source)) throw calibrationError(`--source must be one of ${SOURCES.join(', ')}`, 'INVALID_USAGE');
|
|
25
|
+
if (!String(location || '').trim()) throw calibrationError('--location <file[:line]> is required so the outcome can be matched to findings.', 'INVALID_USAGE');
|
|
26
|
+
if (!String(summary || '').trim()) throw calibrationError('--summary <text> is required.', 'INVALID_USAGE');
|
|
27
|
+
if (!String(by || '').trim()) throw calibrationError('--by <name> is required: outcomes are human-reported evidence.', 'INVALID_USAGE');
|
|
28
|
+
loadRun(rootDir, runId);
|
|
29
|
+
const entry = { source, location: String(location).trim(), summary: String(summary).trim(), by: String(by).trim(), at: now.toISOString(), ...(evidence ? { evidence } : {}) };
|
|
30
|
+
fs.appendFileSync(path.join(runsDir(rootDir), runId, OUTCOMES_FILE), `${JSON.stringify(entry)}\n`, { mode: 0o600 });
|
|
31
|
+
return entry;
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
function readJsonLines(filePath) {
|
|
35
|
+
let text = '';
|
|
36
|
+
try {
|
|
37
|
+
text = fs.readFileSync(filePath, 'utf8');
|
|
38
|
+
} catch {
|
|
39
|
+
return [];
|
|
40
|
+
}
|
|
41
|
+
return text.split('\n').filter((line) => line.trim()).flatMap((line) => {
|
|
42
|
+
try {
|
|
43
|
+
return [JSON.parse(line)];
|
|
44
|
+
} catch {
|
|
45
|
+
return [];
|
|
46
|
+
}
|
|
47
|
+
});
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
// Classify one outcome against the Run's ledger findings.
|
|
51
|
+
function classifyOutcome(outcome, findings) {
|
|
52
|
+
const file = fileOf(outcome.location);
|
|
53
|
+
const related = (findings || []).filter((finding) => fileOf(finding.location) === file);
|
|
54
|
+
if (!related.length) return { verdict: 'DA_MISS', findings: [] };
|
|
55
|
+
const dismissed = related.filter((finding) => finding.validity === 'INVALID' || finding.dismissal);
|
|
56
|
+
if (dismissed.length) return { verdict: 'FALSE_DISMISS', findings: dismissed.map((finding) => finding.id) };
|
|
57
|
+
return { verdict: 'CAUGHT_BUT_SHIPPED', findings: related.map((finding) => finding.id) };
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
function calibrate(rootDir) {
|
|
61
|
+
let runs = [];
|
|
62
|
+
try {
|
|
63
|
+
runs = fs.readdirSync(runsDir(rootDir), { withFileTypes: true }).filter((entry) => entry.isDirectory()).map((entry) => entry.name).sort();
|
|
64
|
+
} catch {
|
|
65
|
+
runs = [];
|
|
66
|
+
}
|
|
67
|
+
const rows = [];
|
|
68
|
+
const byPrompt = {};
|
|
69
|
+
for (const runId of runs) {
|
|
70
|
+
const outcomes = readJsonLines(path.join(runsDir(rootDir), runId, OUTCOMES_FILE));
|
|
71
|
+
if (!outcomes.length) continue;
|
|
72
|
+
let ledger = null;
|
|
73
|
+
try {
|
|
74
|
+
ledger = JSON.parse(fs.readFileSync(path.join(runsDir(rootDir), runId, 'findings.json'), 'utf8'));
|
|
75
|
+
} catch {
|
|
76
|
+
ledger = null;
|
|
77
|
+
}
|
|
78
|
+
const prompt = ledger?.reviewer?.promptSha256 ? ledger.reviewer.promptSha256.slice(0, 12) : 'unknown';
|
|
79
|
+
for (const outcome of outcomes) {
|
|
80
|
+
const result = classifyOutcome(outcome, ledger?.findings);
|
|
81
|
+
rows.push({ runId, ...outcome, ...result, prompt });
|
|
82
|
+
const bucket = byPrompt[prompt] || (byPrompt[prompt] = { outcomes: 0, DA_MISS: 0, FALSE_DISMISS: 0, CAUGHT_BUT_SHIPPED: 0 });
|
|
83
|
+
bucket.outcomes += 1;
|
|
84
|
+
bucket[result.verdict] += 1;
|
|
85
|
+
}
|
|
86
|
+
}
|
|
87
|
+
const count = (verdict) => rows.filter((row) => row.verdict === verdict).length;
|
|
88
|
+
return {
|
|
89
|
+
outcomes: rows.length,
|
|
90
|
+
daMiss: count('DA_MISS'),
|
|
91
|
+
falseDismiss: count('FALSE_DISMISS'),
|
|
92
|
+
caughtButShipped: count('CAUGHT_BUT_SHIPPED'),
|
|
93
|
+
byPrompt,
|
|
94
|
+
rows
|
|
95
|
+
};
|
|
96
|
+
}
|
|
97
|
+
|
|
98
|
+
function formatCalibration(report) {
|
|
99
|
+
const lines = [`Outcomes: ${report.outcomes} (DA missed ${report.daMiss}, wrongly dismissed ${report.falseDismiss}, caught but shipped ${report.caughtButShipped})`];
|
|
100
|
+
for (const row of report.rows.filter((r) => r.verdict !== 'CAUGHT_BUT_SHIPPED')) {
|
|
101
|
+
lines.push(`- ${row.verdict} ${row.runId} ${row.location}: ${row.summary}${row.findings.length ? ` (findings ${row.findings.join(', ')})` : ''}`);
|
|
102
|
+
}
|
|
103
|
+
if (report.daMiss || report.falseDismiss) {
|
|
104
|
+
lines.push('Next: add each miss as a concrete check to agents/shared/devil-advocate.body.md (or a learn promote rule), then compare the next prompt hash here.');
|
|
105
|
+
}
|
|
106
|
+
for (const [prompt, bucket] of Object.entries(report.byPrompt)) {
|
|
107
|
+
lines.push(`- prompt ${prompt}: ${bucket.outcomes} outcome(s), ${bucket.DA_MISS} miss, ${bucket.FALSE_DISMISS} false dismiss`);
|
|
108
|
+
}
|
|
109
|
+
return lines.join('\n');
|
|
110
|
+
}
|
|
111
|
+
|
|
112
|
+
module.exports = { OUTCOMES_FILE, SOURCES, calibrate, classifyOutcome, formatCalibration, recordOutcome };
|