ai-engineering-loop 2.6.0 → 2.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. package/.agents/skills/ael/SKILL.md +1 -1
  2. package/.agents/skills/ael/references/loop-commands.md +11 -0
  3. package/.agents/skills/ael/references/loop-goal.md +2 -0
  4. package/.agents/skills/ael/references/loop-maker.md +1 -6
  5. package/.agents/skills/ael/references/loop-report.md +8 -0
  6. package/.agents/skills/ael/references/loop-verify.md +11 -9
  7. package/.claude/agents/devil-advocate.md +6 -1
  8. package/.claude/agents/judge.md +3 -1
  9. package/.grok/agents/devil-advocate.md +6 -1
  10. package/.grok/agents/judge.md +3 -1
  11. package/CHANGELOG.md +16 -0
  12. package/agents/devil-advocate.md +8 -0
  13. package/agents/judge.md +1 -0
  14. package/agents/shared/devil-advocate.body.md +6 -1
  15. package/agents/shared/judge.body.md +3 -1
  16. package/bin/ai-engineering-loop.js +262 -20
  17. package/core/bench.md +15 -0
  18. package/core/goal-contract.md +35 -0
  19. package/core/verification-loop.md +78 -1
  20. package/lib/ablation.js +52 -0
  21. package/lib/ac-status.js +63 -0
  22. package/lib/baseline.js +138 -0
  23. package/lib/bench.js +24 -1
  24. package/lib/calibration.js +112 -0
  25. package/lib/claimed-vs-reality.js +106 -12
  26. package/lib/clean-state.js +85 -0
  27. package/lib/cli-args.js +5 -3
  28. package/lib/debt.js +110 -0
  29. package/lib/features.js +95 -0
  30. package/lib/gates.js +234 -41
  31. package/lib/harness-audit.js +124 -0
  32. package/lib/no-progress.js +6 -1
  33. package/lib/review-ingest.js +54 -2
  34. package/lib/rules.js +112 -0
  35. package/lib/stats.js +90 -0
  36. package/lib/sync-hosts.js +1 -0
  37. package/lib/verification-integrity.js +164 -0
  38. package/lib/verification-lanes.js +64 -0
  39. package/lib/verification-recorder.js +17 -5
  40. package/package.json +1 -1
  41. package/schemas/finding-ledger.schema.json +14 -0
  42. package/schemas/goal-contract.schema.json +31 -0
  43. package/schemas/verdict.schema.json +2 -1
@@ -18,7 +18,7 @@ Same skill on every host. Before the first DA or Judge spawn, read your host's s
18
18
 
19
19
  ## Commands
20
20
 
21
- `init|status|refresh|sync-hosts|generate-adapter|generate-workflow`: run `npx ai-engineering-loop <command>` (`loop-commands`); do not commit unless asked. No task: offer `workflow interview` or a task; no Run. Otherwise: the loop.
21
+ `init|status|refresh|sync-hosts|generate-*|stats|calibrate|debt|next`: run `npx ai-engineering-loop <command>` (`loop-commands`); do not commit unless asked. No task: offer `workflow interview` or a task; no Run. Otherwise: the loop.
22
22
 
23
23
  ## Loop
24
24
 
@@ -7,6 +7,17 @@
7
7
 
8
8
  - Never run `update apply|auto|rollback|policy` unless the user asks; the CLI applies updates itself under the user's policy (`docs runtime-safety`).
9
9
 
10
+ ## Harness health and history
11
+
12
+ Run these when the user asks how the loop is doing, before planning work, or when a gate points at them. All read local Run files; none starts a Run.
13
+ - `stats [--last <n>] [--json]`: failure causes by harness layer (`task_spec`, `context`, `environment`, `verification`, `state`, `model`), false-victory rate per tier, iterations to PASS. Fix the bottleneck layer first.
14
+ - `outcome record --run <id> --source mr-comment|revert|bug|incident --location <file[:line]> --summary <text> --by <name>`: a problem people found after delivery. Only record what the human reported; never invent one. Then `calibrate` lists DA misses and wrong dismissals per reviewer prompt version.
15
+ - `debt [--file <path>]` lists trade-offs the Judge accepted and L3 waivers; `debt close <id> --by <name>` only on the human's word.
16
+ - `next`: the next unblocked feature in `.ai-engineering-loop/features.json` and VCR (verified/activated). Finish the active feature before starting another.
17
+ - `learn` lists lessons seen in 3+ Runs; a human promotes one with `learn promote <id> --check "<cmd>" --fix "<instruction>" --by <name>`. Default `verify` then runs it as an L1 rule.
18
+ - `doctor --harness`: harness debt (entry-file size and structure, stale learnings, rule provenance, skill budget). Report findings; do not mass-delete.
19
+ - `bench --without <integrity|l3|l1|smoke|rules>` measures whether a gate component still earns its cost (`docs bench`). Never use it on a real task.
20
+
10
21
  ## Personal workflow
11
22
 
12
23
  No task (`ael`, "hi", "what can you do"): offer, numbered, 1) build my workflow, 2) start a task, 3) show status. Do not create a Run or edit files. For 1: `workflow interview --json`, show the pre-filled answers with their evidence, let the user confirm or change them, write the answers to a temp file, then `workflow propose --answers <file> --json` and show the diff. Install only after the user says yes: `workflow accept --proposal <id> --by <name>`. At a no-task start or at Deliver, show at most one pending `workflow suggest` proposal. Never accept for the user. The safety backbone cannot be removed (`docs personal-workflow`).
@@ -4,6 +4,8 @@
4
4
  - Ambiguous task the user can answer: grill first (`docs grill-policy`): design tree, recommended answers, never ask look-up facts. T2 adds `loop-risky`.
5
5
  - If the user prompt has no numbered AC, draft the contract (`docs goal-contract` if needed), list it, and wait. Chat agreement is not freeze.
6
6
  - AC is a failure table: each row has a statement, required evidence, and failure cases. Do not freeze sunny-path-only. T0 may use 1-3 rows that still list failure cases.
7
+ - Boundary-crossing AC (UI-API, service-DB, queue, IPC): `crossComponent: true`; needs L3.
8
+ - Optional per AC: `verify.command` (Judge PASS needs it green), `rubric` for subjective ACs; Goal `featureId` links `features.json` (`docs goal-contract`).
7
9
  - Write the contract JSON to a repository-local file, then `npx ai-engineering-loop goal draft --run <runId> --file <file> --revision <currentRevision>` (publishes the draft and the freeze question to Studio). Show the human the draft and its `impactPreview` in one message.
8
10
  - Wait for the explicit human freeze, then `npx ai-engineering-loop goal freeze --latest --run <runId> --by <name>` (or `--revision <r> --hash <h>`). Add `--confirm-impact` when that answer also accepts the blast radius. T0: `quick start`.
9
11
  - Otherwise the freeze writes a DRAFT blast-radius map: `impact show --run <runId>`. ASSISTED: the human confirms or edits it (`impact confirm --by <name> [--allow <glob>] [--remove <path>]`); copy/CSS/comment-only goals: `impact skip --by <name> --reason <text>` (`docs blast-radius-map`).
@@ -1,15 +1,10 @@
1
1
  # Stages 2-4: Maker
2
2
 
3
3
  - T1/T2: read `docs tdd-policy` before Maker. Use `npx ai-engineering-loop context query <term>` and `context related <file>` before opening broad source files.
4
- - Branch on mode. REPORT_ONLY: never edit repository files; write `RUN_DIR/report.json` with summary and recommendations, then the report review below, run `gate report`, and stop. ASSISTED: the parent (or intern) is Maker; delivery later needs human approval. UNATTENDED: `sandbox create --run <runId>`, Maker works only in the returned worktreePath, never the parent checkout, then `sandbox capture --run <runId>`.
4
+ - Branch on mode. REPORT_ONLY: never edit repository files; write `RUN_DIR/report.json` with summary and recommendations, then the report review (`loop-report`), run `gate report`, and stop. ASSISTED: the parent (or intern) is Maker; delivery later needs human approval. UNATTENDED: `sandbox create --run <runId>`, Maker works only in the returned worktreePath, never the parent checkout, then `sandbox capture --run <runId>`.
5
5
  - Maker intern: read `maker_intern` from workflow.md (missing or `none` = parent is Maker). Confirm it through the host model catalog; otherwise the parent is Maker and you report INVOCATION_UNAVAILABLE. The intern cannot skip later gates or spawn DA/Judge.
6
6
  - Guardrails: Claude Code enforces them after `hooks install`. Elsewhere, before an out-of-scope edit or a destructive command (`rm -rf`, `DROP TABLE`, `git push --force`, `git reset --hard`, `git clean -fd`) run `guard check --tool edit --path <file>` or `--tool bash --command "<cmd>"`: `warn` = ask the human, `block` = do not run it.
7
7
  - Bugs: red repro first. Features: one red test per AC row, then green.
8
8
  - After package and manifest edits are stable, run `npx ai-engineering-loop refresh` once (drift during red-green is expected; do not refresh per test cycle).
9
9
  - `eslint --fix` and other auto-fixers run before `gate maker`, never after.
10
10
  - ASSISTED: `npx ai-engineering-loop gate maker --from-git` writes the exact working-tree diff (tracked and new files, excluding `.ai-engineering-loop/`) to `RUN_DIR/diff.patch` and gates it; or write that file yourself and run `gate maker`. UNATTENDED: `sandbox capture` writes it, then `gate maker`. The gate can raise the tier from the diff.
11
-
12
- ## REPORT_ONLY report review (recipe `audit`)
13
-
14
- - Host has a subagent tool (`loop-hosts`): spawn ONE fresh `devil-advocate` sibling with report.json and the evidence paths it cites; it challenges unsupported or wrong findings with the usual DA budget. No Judge, no iteration loop: fold accepted findings once, drop rejected ones with evidence. Record `"review": {"executionMode": "TRUE_INDEPENDENT_AGENT", "nativeSubagent": "AVAILABLE", "reviewer": "devil-advocate", "childId": "<its own id>"}`.
15
- - No subagent tool, or the spawn failed: do the same challenge in this session and record `"executionMode": "CONTEXT_ISOLATION_ONLY"` with `"reason"` (required by `gate report` when `nativeSubagent` is `AVAILABLE`). The header shows `Execution Mode: CONTEXT_ISOLATION_ONLY (reason: <why>)`.
@@ -0,0 +1,8 @@
1
+ # REPORT_ONLY: report review
2
+
3
+ Read only in REPORT_ONLY mode, after writing `RUN_DIR/report.json` (`loop-maker`).
4
+
5
+ ## REPORT_ONLY report review (recipe `audit`)
6
+
7
+ - Host has a subagent tool (`loop-hosts`): spawn ONE fresh `devil-advocate` sibling with report.json and the evidence paths it cites; it challenges unsupported or wrong findings with the usual DA budget. No Judge, no iteration loop: fold accepted findings once, drop rejected ones with evidence. Record `"review": {"executionMode": "TRUE_INDEPENDENT_AGENT", "nativeSubagent": "AVAILABLE", "reviewer": "devil-advocate", "childId": "<its own id>"}`.
8
+ - No subagent tool, or the spawn failed: do the same challenge in this session and record `"executionMode": "CONTEXT_ISOLATION_ONLY"` with `"reason"` (required by `gate report` when `nativeSubagent` is `AVAILABLE`). The header shows `Execution Mode: CONTEXT_ISOLATION_ONLY (reason: <why>)`.
@@ -1,19 +1,21 @@
1
1
  # Stage 5: Verification
2
2
 
3
- - `npx ai-engineering-loop verify --run <runId> [--cmd "<command>"]...` (default `test_unit` from verification.md) runs commands in parallel (`--serial`, `--shell`), binds redacted evidence to the gated diff, and on pass compacts and scaffolds claimed-vs-reality.
3
+ - Before the Maker gate: `verify --baseline --run <runId>` records the untouched tree (broken command = fix verification.md first).
4
+ - `npx ai-engineering-loop verify --run <runId> [--cmd "<command>"]...` (default: L1 then `test_unit`, stops at first failure) binds redacted evidence to the gated diff, and on pass compacts and scaffolds claimed-vs-reality.
4
5
  - Iterating (T1/T2): `verify --related` runs `test_related` on changed files. After Judge PASS, `verify --final` runs the full suite once before delivery. Unchanged diff reuses passes (`--fresh` re-runs).
5
6
  - Failed commands return non-zero and block the gate. Vague "seems green" is invalid.
6
- - Fill `.ai-engineering-loop/tasks/claimed-vs-reality.md`: Claimed from change, Reality from command output, optional Layer (`L1`, `L2`, `L3`). Multi-component ACs need L3 evidence or justification if `e2e` is unconfigured. Never spawn DA if file is missing or any row lacks Reality.
7
+ - Fill `.ai-engineering-loop/tasks/claimed-vs-reality.md`: Claimed, Reality from command output, Layer (`L2+L3`). Never spawn DA if it is missing or a row lacks Reality; `gate review` refuses it.
7
8
  - Hand-authored artifacts: `scaffold verification|findings|delivery --run <runId>`, fill, remove `_templateOnly`, then `validate <file>`.
8
- - Pass run-summary.md and verification-summary.json to reviewers instead of raw logs.
9
9
 
10
10
  ## Verification Layers & Command Mapping
11
11
 
12
- | Layer | Scope | Command (`verification.md`) | Focus |
13
- |---|---|---|---|
14
- | **L1** | Static | `typecheck`, `lint`, `build` | Types, lint, build |
15
- | **L2** | Runtime | `test_unit`, `test_related`, `test_all` | Unit and regression tests |
16
- | **L3** | End-to-End | `e2e` (optional) | Integrated boundary proof |
12
+ | Layer | Command (`verification.md`) |
13
+ |---|---|
14
+ | **L1** static | `typecheck`, `lint`, `build` |
15
+ | **L2** runtime | `test_unit`, `test_related`, `test_all` |
16
+ | **L3** e2e | `e2e` (optional) |
17
+
18
+ - L3 for `crossComponent` ACs and all T2 ACs (unit mocks hide boundary bugs). No `e2e`: cite the user flow or `L3 waived: <reason>`.
17
19
 
18
20
  ## No claim without fresh evidence
19
21
 
@@ -25,4 +27,4 @@ Before claiming "done", "fixed", "passing", or "ready", run the proving command
25
27
  | Bug fixed | passing red repro, command shown |
26
28
  | AC met | claimed-vs-reality cites output |
27
29
 
28
- After `verify`, continue from run-summary.md and verification-summary.json.
30
+ After `verify`, continue from (and give reviewers) run-summary.md and verification-summary.json.
@@ -21,7 +21,8 @@ Use only:
21
21
  2. Goal Contract path (if given).
22
22
  3. Verification log path (if given).
23
23
  4. conventions.md path (if given). At most one extra read.
24
- 5. At most 8 source files that appear as paths in the diff.
24
+ 5. claimed-vs-reality.md path (if given): Claimed, Reality, and Layer (L1 static, L2 runtime tests, L3 end-to-end) per AC.
25
+ 6. At most 8 source files that appear as paths in the diff.
25
26
 
26
27
  Skip: `*.css`, files named like `*-css.ts` or `report-css.ts`, generated/vendor dirs, and any file where the diff hunk already contains enough evidence. Prefer quoting the hunk over opening the whole file.
27
28
 
@@ -39,6 +40,10 @@ Report Spec and Standards as separate findings. Do not rerank one axis with the
39
40
 
40
41
  Skip style nits unless they hide a defect.
41
42
 
43
+ Rubric ACs: when an AC in the Goal Contract has `rubric`, add `rubricScores: [{ac, criterion, pass, evidence}]` to the ledger, one per criterion. `evidence` must be observed (screenshot path, DOM snapshot, measured value), never an impression. A criterion you cannot observe is `pass: false`.
44
+
45
+ Layer check (Spec): green unit tests do not prove an AC that crosses a boundary (UI-API, service-DB, IPC, queue, external service); their mocks hide interface mismatches, state propagation, resource leaks, and environment gaps. Raise a `test-gap` finding (HIGH; BLOCKER at T2) when such an AC's Layer stops at L1/L2, when Layer says L3 but Reality cites only a unit or mocked test, or when an `L3 waived:` reason does not hold against the diff.
46
+
42
47
  Smell baseline (judgement only; skip if tooling already enforces; repo conventions override): Mysterious Name; Duplicated Code; Feature Envy; Data Clumps; Primitive Obsession; Repeated Switches; Shotgun Surgery; Divergent Change; Speculative Generality; Message Chains; Middle Man; Refused Bequest.
43
48
 
44
49
  ## Output
@@ -19,7 +19,9 @@ Use only paths in the spawn prompt: Goal Contract, verification evidence, Findin
19
19
 
20
20
  AC source is the Goal Contract **file** only. Chat `setuju` and the parent transcript are not AC. Do not PASS because the parent agreed in chat. `CONTEXT_ISOLATION_ONLY` does not relax this.
21
21
 
22
- ITERATE if verification is missing, vague, non-zero, proves a different artifact than the AC names, is only a source grep, or is happy-path-only while the Goal Contract lists empty, boundary, sibling, or error rows. ITERATE if claimed-vs-reality.md is missing or a Claimed row has empty Reality.
22
+ ITERATE if verification is missing, vague, non-zero, proves a different artifact than the AC names, is only a source grep, or is happy-path-only while the Goal Contract lists empty, boundary, sibling, or error rows. ITERATE if claimed-vs-reality.md is missing or a Claimed row has empty Reality. Findings with `source: "verification-integrity"` (skipped, deleted or weakened tests, edited verification config) are VALID unless a human dismissal says otherwise. ITERATE if an AC with `crossComponent: true`, or any AC at T2, stops at L1/L2 in the Layer column, even when all unit tests are green; accept `L3 waived: <reason>` only when the reason shows no end-to-end run or manual flow was possible.
23
+
24
+ On ITERATE or ESCALATE, set `cause` to the harness layer the failure points at: `task_spec` (AC unclear or wrong), `context` (missing project knowledge), `environment` (setup/tooling), `verification` (weak or missing proof), `state` (lost continuity), or `model` (plain implementation defect). The gate infers one when you omit it.
23
25
 
24
26
  The gate recomputes the verdict from the ledger. To block on a problem the ledger lacks (wrong-artifact or happy-path-only evidence, an AC the tests do not prove), add it to `judgeFindings` in the Finding Ledger shape: VALID, BLOCKER or HIGH, with `location`, `failureScenario`, `evidence`, and `concreteAlternativeDiff`. A verdict that blocks without such a finding is rejected. You cannot PASS what the ledger blocks.
25
27
 
@@ -22,7 +22,8 @@ Use only:
22
22
  2. Goal Contract path (if given).
23
23
  3. Verification log path (if given).
24
24
  4. conventions.md path (if given). At most one extra read.
25
- 5. At most 8 source files that appear as paths in the diff.
25
+ 5. claimed-vs-reality.md path (if given): Claimed, Reality, and Layer (L1 static, L2 runtime tests, L3 end-to-end) per AC.
26
+ 6. At most 8 source files that appear as paths in the diff.
26
27
 
27
28
  Skip: `*.css`, files named like `*-css.ts` or `report-css.ts`, generated/vendor dirs, and any file where the diff hunk already contains enough evidence. Prefer quoting the hunk over opening the whole file.
28
29
 
@@ -40,6 +41,10 @@ Report Spec and Standards as separate findings. Do not rerank one axis with the
40
41
 
41
42
  Skip style nits unless they hide a defect.
42
43
 
44
+ Rubric ACs: when an AC in the Goal Contract has `rubric`, add `rubricScores: [{ac, criterion, pass, evidence}]` to the ledger, one per criterion. `evidence` must be observed (screenshot path, DOM snapshot, measured value), never an impression. A criterion you cannot observe is `pass: false`.
45
+
46
+ Layer check (Spec): green unit tests do not prove an AC that crosses a boundary (UI-API, service-DB, IPC, queue, external service); their mocks hide interface mismatches, state propagation, resource leaks, and environment gaps. Raise a `test-gap` finding (HIGH; BLOCKER at T2) when such an AC's Layer stops at L1/L2, when Layer says L3 but Reality cites only a unit or mocked test, or when an `L3 waived:` reason does not hold against the diff.
47
+
43
48
  Smell baseline (judgement only; skip if tooling already enforces; repo conventions override): Mysterious Name; Duplicated Code; Feature Envy; Data Clumps; Primitive Obsession; Repeated Switches; Shotgun Surgery; Divergent Change; Speculative Generality; Message Chains; Middle Man; Refused Bequest.
44
49
 
45
50
  ## Output
@@ -20,7 +20,9 @@ Use only paths in the spawn prompt: Goal Contract, verification evidence, Findin
20
20
 
21
21
  AC source is the Goal Contract **file** only. Chat `setuju` and the parent transcript are not AC. Do not PASS because the parent agreed in chat. `CONTEXT_ISOLATION_ONLY` does not relax this.
22
22
 
23
- ITERATE if verification is missing, vague, non-zero, proves a different artifact than the AC names, is only a source grep, or is happy-path-only while the Goal Contract lists empty, boundary, sibling, or error rows. ITERATE if claimed-vs-reality.md is missing or a Claimed row has empty Reality.
23
+ ITERATE if verification is missing, vague, non-zero, proves a different artifact than the AC names, is only a source grep, or is happy-path-only while the Goal Contract lists empty, boundary, sibling, or error rows. ITERATE if claimed-vs-reality.md is missing or a Claimed row has empty Reality. Findings with `source: "verification-integrity"` (skipped, deleted or weakened tests, edited verification config) are VALID unless a human dismissal says otherwise. ITERATE if an AC with `crossComponent: true`, or any AC at T2, stops at L1/L2 in the Layer column, even when all unit tests are green; accept `L3 waived: <reason>` only when the reason shows no end-to-end run or manual flow was possible.
24
+
25
+ On ITERATE or ESCALATE, set `cause` to the harness layer the failure points at: `task_spec` (AC unclear or wrong), `context` (missing project knowledge), `environment` (setup/tooling), `verification` (weak or missing proof), `state` (lost continuity), or `model` (plain implementation defect). The gate infers one when you omit it.
24
26
 
25
27
  The gate recomputes the verdict from the ledger. To block on a problem the ledger lacks (wrong-artifact or happy-path-only evidence, an AC the tests do not prove), add it to `judgeFindings` in the Finding Ledger shape: VALID, BLOCKER or HIGH, with `location`, `failureScenario`, `evidence`, and `concreteAlternativeDiff`. A verdict that blocks without such a finding is rejected. You cannot PASS what the ledger blocks.
26
28
 
package/CHANGELOG.md CHANGED
@@ -6,6 +6,22 @@ All notable changes to this project are documented here. Versions follow [Semant
6
6
 
7
7
  _No changes yet._
8
8
 
9
+ ## 2.7.0 - 2026-11-16
10
+
11
+ Harness gaps from a full review against learn-harness-engineering (cards C-01..C-16 in `docs/cards/harness-gaps-2026-11-16.md`).
12
+
13
+ - **Added:** L3 enforcement (#112 follow-up): `crossComponent` ACs and every T2 AC need L3 evidence or `L3 waived: <reason>`; Layer tags must match a recorded command of that layer (C-01); `gate review` checks claimed-vs-reality on T1/T2 too.
14
+ - **Added:** `verify --baseline` records the untouched tree before the Maker gate; later failures are labelled pre-existing or new (C-10).
15
+ - **Changed:** `verify` without `--cmd` runs build -> typecheck -> lint -> test_unit and stops at the first failure; T1/T2 need an L1 record when typecheck or lint is configured (`L1_MISSING`, C-02). Optional `smoke` runs for migration/config/env/entrypoint changes (`SMOKE_MISSING`, C-03).
16
+ - **Fixed:** `init` no longer writes guessed `npx tsc --noEmit`, `npx eslint --fix` or `npm run build` for projects that do not have them; commands with `--fix`/`--write` never run by default.
17
+ - **Added:** weakened verification in the diff (new skip/only markers, deleted tests, dropped assertions, edited verification config) becomes a VALID finding only a human can dismiss (C-11).
18
+ - **Added:** every gate failure prints `WHY:` and `FIX:` (C-04); delivery blocks debugger statements, conflict markers, committed `.env`/keys and leftover `_templateOnly` scaffolds, and reports stray untracked files (C-06).
19
+ - **Added:** Goal Contract AC fields `verify.command` (Judge PASS needs it green, C-07) and `rubric` (scored per criterion with evidence, C-09); Goal `featureId` links `.ai-engineering-loop/features.json` (C-15).
20
+ - **Added:** `ael stats` (failure cause per harness layer, false-victory rate, C-12), `ael outcome record` + `ael calibrate` (C-13), `ael debt` (C-14), `ael next` (C-15), `learn promote` rules (C-05), `doctor --harness` and `bench --without` (C-08).
21
+ - **Added:** review-pack records what the Devil's Advocate saw (pack, diff, prompt hashes); the review gate rejects a ledger whose inputs changed (C-16).
22
+ - **Changed:** the REPORT_ONLY report review moved from `loop-maker` to the new `loop-report` reference, so normal runs do not load it; new harness commands are documented in `loop-commands`.
23
+ - **Added:** root `AGENTS.md` for contributors and agents working on this repository.
24
+
9
25
  ## 2.6.0 - 2026-11-10
10
26
 
11
27
  - **Added:** three-layer verification naming in `loop-verify` (#110): L1 unit, L2 related/integration, L3 end-to-end through the existing optional `e2e` verification key, with a manual flow check plus written evidence when `e2e` is not configured. T0 needs L1-L2, T1 adds L3 when an AC crosses component boundaries, T2 always needs L1-L3.
@@ -34,10 +34,18 @@ The Devil's Advocate receives **only** the objective artifact package:
34
34
  - `Project Context` (`.ai-engineering-loop/`: `architecture.md`, `conventions.md`, `verification.md`, `glossary.md`).
35
35
  - `Pure Git Diff` (`git diff <base>...HEAD`).
36
36
  - `Deterministic Verification Logs` (exit code 0 proof).
37
+ - `claimed-vs-reality.md` (Claimed, Reality, and Layer `L1`/`L2`/`L3` per AC).
37
38
  - `Prior Finding Signatures`.
38
39
 
39
40
  *Maker conversational history, intermediate attempts, and author rationalizations are strictly excluded.*
40
41
 
42
+ ### Layer check (three-layer termination)
43
+
44
+ Green unit tests do not prove an AC that crosses a component boundary: their mocks hide interface mismatches, state propagation errors, resource leaks, and environment gaps. For each AC row:
45
+ - The diff crosses a boundary (UI-API, service-DB, IPC, queue, external service) but the row's Layer stops at L1/L2: raise a `test-gap` finding (HIGH; BLOCKER at T2) naming the missing end-to-end path.
46
+ - Layer says `L3` but Reality cites only a unit or mocked test: raise a finding; the tag is wrong.
47
+ - `L3 waived: <reason>`: check the reason against the diff. Reject waivers that only say e2e is slow or missing when a manual flow could run.
48
+
41
49
  ---
42
50
 
43
51
  ## 4. Output Contract: Structured Finding Ledger
package/agents/judge.md CHANGED
@@ -51,6 +51,7 @@ flowchart LR
51
51
  - Cross-reference every Acceptance Criterion (AC-1 through AC-N) from the [Goal Contract](../core/goal-contract.md).
52
52
  - Verify that automated tests exist that explicitly exercise and prove each criterion, including non-happy-path rows in the failure table.
53
53
  - A happy-path-only suite while the contract lists empty, boundary, sibling, or error rows is `ITERATE`.
54
+ - Read the Layer column of claimed-vs-reality. An AC with `crossComponent: true`, or any AC at T2, whose evidence stops at L1/L2 is `ITERATE`, even when every unit test is green. A `L3 waived` row passes only if the reason shows no end-to-end path or manual flow could run this iteration.
54
55
  - Verify that no out-of-scope files were touched and technical constraints were respected.
55
56
 
56
57
  ### Step 3: Finding Triage & Evidence Verification
@@ -14,7 +14,8 @@ Use only:
14
14
  2. Goal Contract path (if given).
15
15
  3. Verification log path (if given).
16
16
  4. conventions.md path (if given). At most one extra read.
17
- 5. At most 8 source files that appear as paths in the diff.
17
+ 5. claimed-vs-reality.md path (if given): Claimed, Reality, and Layer (L1 static, L2 runtime tests, L3 end-to-end) per AC.
18
+ 6. At most 8 source files that appear as paths in the diff.
18
19
 
19
20
  Skip: `*.css`, files named like `*-css.ts` or `report-css.ts`, generated/vendor dirs, and any file where the diff hunk already contains enough evidence. Prefer quoting the hunk over opening the whole file.
20
21
 
@@ -32,6 +33,10 @@ Report Spec and Standards as separate findings. Do not rerank one axis with the
32
33
 
33
34
  Skip style nits unless they hide a defect.
34
35
 
36
+ Rubric ACs: when an AC in the Goal Contract has `rubric`, add `rubricScores: [{ac, criterion, pass, evidence}]` to the ledger, one per criterion. `evidence` must be observed (screenshot path, DOM snapshot, measured value), never an impression. A criterion you cannot observe is `pass: false`.
37
+
38
+ Layer check (Spec): green unit tests do not prove an AC that crosses a boundary (UI-API, service-DB, IPC, queue, external service); their mocks hide interface mismatches, state propagation, resource leaks, and environment gaps. Raise a `test-gap` finding (HIGH; BLOCKER at T2) when such an AC's Layer stops at L1/L2, when Layer says L3 but Reality cites only a unit or mocked test, or when an `L3 waived:` reason does not hold against the diff.
39
+
35
40
  Smell baseline (judgement only; skip if tooling already enforces; repo conventions override): Mysterious Name; Duplicated Code; Feature Envy; Data Clumps; Primitive Obsession; Repeated Switches; Shotgun Surgery; Divergent Change; Speculative Generality; Message Chains; Middle Man; Refused Bequest.
36
41
 
37
42
  ## Output
@@ -12,7 +12,9 @@ Use only paths in the spawn prompt: Goal Contract, verification evidence, Findin
12
12
 
13
13
  AC source is the Goal Contract **file** only. Chat `setuju` and the parent transcript are not AC. Do not PASS because the parent agreed in chat. `CONTEXT_ISOLATION_ONLY` does not relax this.
14
14
 
15
- ITERATE if verification is missing, vague, non-zero, proves a different artifact than the AC names, is only a source grep, or is happy-path-only while the Goal Contract lists empty, boundary, sibling, or error rows. ITERATE if claimed-vs-reality.md is missing or a Claimed row has empty Reality.
15
+ ITERATE if verification is missing, vague, non-zero, proves a different artifact than the AC names, is only a source grep, or is happy-path-only while the Goal Contract lists empty, boundary, sibling, or error rows. ITERATE if claimed-vs-reality.md is missing or a Claimed row has empty Reality. Findings with `source: "verification-integrity"` (skipped, deleted or weakened tests, edited verification config) are VALID unless a human dismissal says otherwise. ITERATE if an AC with `crossComponent: true`, or any AC at T2, stops at L1/L2 in the Layer column, even when all unit tests are green; accept `L3 waived: <reason>` only when the reason shows no end-to-end run or manual flow was possible.
16
+
17
+ On ITERATE or ESCALATE, set `cause` to the harness layer the failure points at: `task_spec` (AC unclear or wrong), `context` (missing project knowledge), `environment` (setup/tooling), `verification` (weak or missing proof), `state` (lost continuity), or `model` (plain implementation defect). The gate infers one when you omit it.
16
18
 
17
19
  The gate recomputes the verdict from the ledger. To block on a problem the ledger lacks (wrong-artifact or happy-path-only evidence, an AC the tests do not prove), add it to `judgeFindings` in the Finding Ledger shape: VALID, BLOCKER or HIGH, with `location`, `failureScenario`, `evidence`, and `concreteAlternativeDiff`. A verdict that blocks without such a finding is rejected. You cannot PASS what the ledger blocks.
18
20