ai-engineering-loop 2.5.0 → 2.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
|
@@ -1,20 +1,28 @@
|
|
|
1
1
|
# Stage 5: Verification
|
|
2
2
|
|
|
3
|
-
- `npx ai-engineering-loop verify --run <runId> [--cmd "<command>"]...` (default `test_unit` from verification.md)
|
|
4
|
-
- Iterating (T1/T2): `verify --related` runs `test_related` on
|
|
5
|
-
-
|
|
6
|
-
- Fill `.ai-engineering-loop/tasks/claimed-vs-reality.md`: Claimed from
|
|
7
|
-
- Hand-authored artifacts: `scaffold verification|findings|delivery --run <runId>`, fill
|
|
3
|
+
- `npx ai-engineering-loop verify --run <runId> [--cmd "<command>"]...` (default `test_unit` from verification.md) runs commands in parallel (`--serial`, `--shell`), binds redacted evidence to the gated diff, and on pass compacts and scaffolds claimed-vs-reality.
|
|
4
|
+
- Iterating (T1/T2): `verify --related` runs `test_related` on changed files. After Judge PASS, `verify --final` runs the full suite once before delivery. Unchanged diff reuses passes (`--fresh` re-runs).
|
|
5
|
+
- Failed commands return non-zero and block the gate. Vague "seems green" is invalid.
|
|
6
|
+
- Fill `.ai-engineering-loop/tasks/claimed-vs-reality.md`: Claimed from change, Reality from command output, optional Layer (`L1`, `L2`, `L3`). Multi-component ACs need L3 evidence or justification if `e2e` is unconfigured. Never spawn DA if file is missing or any row lacks Reality.
|
|
7
|
+
- Hand-authored artifacts: `scaffold verification|findings|delivery --run <runId>`, fill, remove `_templateOnly`, then `validate <file>`.
|
|
8
8
|
- Pass run-summary.md and verification-summary.json to reviewers instead of raw logs.
|
|
9
9
|
|
|
10
|
+
## Verification Layers & Command Mapping
|
|
11
|
+
|
|
12
|
+
| Layer | Scope | Command (`verification.md`) | Focus |
|
|
13
|
+
|---|---|---|---|
|
|
14
|
+
| **L1** | Static | `typecheck`, `lint`, `build` | Types, lint, build |
|
|
15
|
+
| **L2** | Runtime | `test_unit`, `test_related`, `test_all` | Unit and regression tests |
|
|
16
|
+
| **L3** | End-to-End | `e2e` (optional) | Integrated boundary proof |
|
|
17
|
+
|
|
10
18
|
## No claim without fresh evidence
|
|
11
19
|
|
|
12
|
-
Before
|
|
20
|
+
Before claiming "done", "fixed", "passing", or "ready", run the proving command this iteration; earlier runs and "should work" are not evidence.
|
|
13
21
|
|
|
14
22
|
| Claim | Required evidence |
|
|
15
23
|
|---|---|
|
|
16
|
-
| Tests pass |
|
|
17
|
-
| Bug fixed |
|
|
18
|
-
| AC met |
|
|
24
|
+
| Tests pass | test output this iteration, exit 0 |
|
|
25
|
+
| Bug fixed | passing red repro, command shown |
|
|
26
|
+
| AC met | claimed-vs-reality cites output |
|
|
19
27
|
|
|
20
|
-
After `verify`, continue from run-summary.md and verification-summary.json
|
|
28
|
+
After `verify`, continue from run-summary.md and verification-summary.json.
|
package/CHANGELOG.md
CHANGED
|
@@ -6,6 +6,12 @@ All notable changes to this project are documented here. Versions follow [Semant
|
|
|
6
6
|
|
|
7
7
|
_No changes yet._
|
|
8
8
|
|
|
9
|
+
## 2.6.0 - 2026-11-10
|
|
10
|
+
|
|
11
|
+
- **Added:** three-layer verification naming in `loop-verify` (#110): L1 unit, L2 related/integration, L3 end-to-end through the existing optional `e2e` verification key, with a manual flow check plus written evidence when `e2e` is not configured. T0 needs L1-L2, T1 adds L3 when an AC crosses component boundaries, T2 always needs L1-L3.
|
|
12
|
+
- **Added:** `context claimed-vs-reality` scaffolds a 4-column table (`AC | Claimed | Reality from command log | Layer`). Existing 3-column files stay valid for `readyForDa`; empty or "seems green" Reality still fails.
|
|
13
|
+
- **Fixed:** the measure script counts characters the same on Windows (CRLF) as on macOS and Linux.
|
|
14
|
+
|
|
9
15
|
## 2.5.0 - 2026-11-02
|
|
10
16
|
|
|
11
17
|
- **Changed:** the core host skill is now `ael` (#107), so the slash command is `/ael` on every host (`~/.claude/skills/ael`, `~/.agents/skills/ael`, and each host's own folder). `sync-hosts` and `update` remove the old `ai-engineering-loop` skill files AEL wrote, so the command no longer appears twice; files you edited or added there are kept, and `hosts doctor` reports the leftover folder. Skill sources moved to `skill-src/ael/`.
|
|
@@ -43,9 +43,21 @@ A verification `PASS` is **strictly invalid** without concrete execution evidenc
|
|
|
43
43
|
}
|
|
44
44
|
```
|
|
45
45
|
|
|
46
|
+
## 3. Verification Layers & Command Mapping
|
|
47
|
+
|
|
48
|
+
Verification terminates through three explicit layers of evidence. Unit tests alone have systematic blind spots for cross-component and integration defects. An acceptance criterion crossing component boundaries requires Layer 3 verification or written justification if Layer 3 is unavailable.
|
|
49
|
+
|
|
50
|
+
| Layer | Scope | Configured Command (`verification.md`) | Purpose |
|
|
51
|
+
|---|---|---|---|
|
|
52
|
+
| **L1: Static / Syntax** | Syntax, static types, style, compilation | `typecheck`, `lint`, `build` | Fast static catch before runtime. Rejects type errors, lint regressions, and compilation breakage. |
|
|
53
|
+
| **L2: Behavioral / Runtime** | Functions, units, modules, regressions | `test_unit`, `test_related`, `test_all` | Fast behavioral feedback at named seams and regression verification. |
|
|
54
|
+
| **L3: End-to-End** | Cross-component flows, integrations, browser/API | `e2e` (optional) | Realistic integrated proof across boundary seams. |
|
|
55
|
+
|
|
56
|
+
When `e2e` is configured in `verification.md`, cross-component acceptance criteria must cite L3 command evidence. If `e2e` is empty, Maker must document in Reality why L3 is not configured.
|
|
57
|
+
|
|
46
58
|
---
|
|
47
59
|
|
|
48
|
-
##
|
|
60
|
+
## 4. The 4 Verification Gates
|
|
49
61
|
|
|
50
62
|
1. **Gate 1: Unit & Integration Tests**: All unit and edge-case tests must pass with 0 failures.
|
|
51
63
|
2. **Gate 2: Static Type Analysis**: `tsc --noEmit` or compiler type checking must exit with code `0` (zero type errors).
|
|
@@ -54,7 +66,7 @@ A verification `PASS` is **strictly invalid** without concrete execution evidenc
|
|
|
54
66
|
|
|
55
67
|
---
|
|
56
68
|
|
|
57
|
-
##
|
|
69
|
+
## 5. Rejection Triggers
|
|
58
70
|
|
|
59
71
|
The Verification Gate immediately halts and returns to the Maker if:
|
|
60
72
|
- Any test suite fails or times out.
|
|
@@ -66,7 +78,7 @@ The Verification Gate immediately halts and returns to the Maker if:
|
|
|
66
78
|
|
|
67
79
|
---
|
|
68
80
|
|
|
69
|
-
##
|
|
81
|
+
## 6. Claimed vs Reality (before Devil's Advocate)
|
|
70
82
|
|
|
71
83
|
Write `.ai-engineering-loop/tasks/claimed-vs-reality.md` after verification logs exist:
|
|
72
84
|
|
|
@@ -74,14 +86,14 @@ Write `.ai-engineering-loop/tasks/claimed-vs-reality.md` after verification logs
|
|
|
74
86
|
npx ai-engineering-loop context claimed-vs-reality --run <run-id>
|
|
75
87
|
```
|
|
76
88
|
|
|
77
|
-
This scaffolds one row per frozen acceptance criterion but leaves Claimed and
|
|
89
|
+
This scaffolds one row per frozen acceptance criterion but leaves Claimed, Reality, and Layer empty. Fill them from the implemented change and actual command output; the scaffold is not verification evidence.
|
|
78
90
|
|
|
79
91
|
```markdown
|
|
80
92
|
# Claimed vs Reality
|
|
81
93
|
|
|
82
|
-
| AC | Claimed | Reality |
|
|
83
|
-
|
|
84
|
-
| AC-1 | <what Maker said is done> | <command, exit code, assertion or log line> |
|
|
94
|
+
| AC | Claimed | Reality from command log | Layer |
|
|
95
|
+
|---|---|---|---|
|
|
96
|
+
| AC-1 | <what Maker said is done> | <command, exit code, assertion or log line> | L2 |
|
|
85
97
|
```
|
|
86
98
|
|
|
87
99
|
Do **not** spawn Devil's Advocate if:
|
|
@@ -76,8 +76,8 @@ function renderClaimedVsRealityTemplate(goal) {
|
|
|
76
76
|
'',
|
|
77
77
|
'> Generated from the frozen Goal Contract. Fill Claimed and Reality only after implementation and verification. Reality must cite observed command output; this template is not verification evidence.',
|
|
78
78
|
'',
|
|
79
|
-
'| AC | Claimed | Reality from command log |',
|
|
80
|
-
'
|
|
79
|
+
'| AC | Claimed | Reality from command log | Layer |',
|
|
80
|
+
'|---|---|---|---|'
|
|
81
81
|
];
|
|
82
82
|
for (const criterion of goal.acceptanceCriteria) {
|
|
83
83
|
const evidence = markdownCell(criterion.evidenceRequired);
|
|
@@ -87,7 +87,7 @@ function renderClaimedVsRealityTemplate(goal) {
|
|
|
87
87
|
evidence ? `Evidence: ${evidence}` : '',
|
|
88
88
|
failures ? `Failure cases: ${failures}` : ''
|
|
89
89
|
].filter(Boolean).join(' — ');
|
|
90
|
-
lines.push(`| ${markdownCell(description)} | | |`);
|
|
90
|
+
lines.push(`| ${markdownCell(description)} | | | |`);
|
|
91
91
|
}
|
|
92
92
|
return `${lines.join('\n')}\n`;
|
|
93
93
|
}
|
|
@@ -143,11 +143,15 @@ function parseClaimedVsReality(markdown) {
|
|
|
143
143
|
if (/^\|\s*-+\s*\|/.test(trimmed) || /^\|\s*AC\s*\|/i.test(trimmed)) continue;
|
|
144
144
|
const cells = trimmed.split('|').map((c) => c.trim()).filter((_, i, arr) => i > 0 && i < arr.length - 1);
|
|
145
145
|
if (cells.length < 3) continue;
|
|
146
|
-
|
|
146
|
+
const row = {
|
|
147
147
|
ac: cells[0],
|
|
148
148
|
claimed: cells[1],
|
|
149
149
|
reality: cells[2]
|
|
150
|
-
}
|
|
150
|
+
};
|
|
151
|
+
if (cells.length >= 4 && cells[3]) {
|
|
152
|
+
row.layer = cells[3];
|
|
153
|
+
}
|
|
154
|
+
rows.push(row);
|
|
151
155
|
}
|
|
152
156
|
return { rows };
|
|
153
157
|
}
|
package/package.json
CHANGED