ai-engineering-loop 1.12.0 → 1.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.agents/workflows/ai-engineering-loop.md +2 -2
- package/.claude/skills/ai-engineering-loop/SKILL.md +3 -3
- package/.gemini/skills/ai-engineering-loop/SKILL.md +3 -3
- package/.grok/skills/ai-engineering-loop/SKILL.md +3 -3
- package/README.md +12 -0
- package/bin/ai-engineering-loop.js +158 -9
- package/core/artifact-gates.md +32 -1
- package/core/runtime-safety.md +2 -1
- package/core/verification-loop.md +6 -0
- package/lib/agent-artifacts.js +250 -0
- package/lib/claimed-vs-reality.js +134 -0
- package/lib/safe-context.js +62 -4
- package/lib/sync-hosts.js +48 -3
- package/lib/verification-recorder.js +224 -0
- package/lib/workflow-runtime.js +26 -1
- package/package.json +1 -1
- package/policies/review-budget.md +1 -1
- package/schemas/verification.schema.json +2 -2
|
@@ -11,7 +11,7 @@ Deterministic gates are mandatory. Read `core/runtime-safety.md`. Stage 0 runs s
|
|
|
11
11
|
|
|
12
12
|
Inherited invariants: read `core/grill-policy.md`, `policies/tdd-policy.md`, and `.ai-engineering-loop/glossary.md`. On `adapter_type: dot`, grill includes blast radius: lifecycle sketch and load `task-impact-inquiry`. If the user prompt has no numbered AC, draft the contract, list a short menu and wait. Chat agreement is not freeze. AC is a failure table. One red test per AC row. Do not freeze sunny-path-only. Write claimed-vs-reality.md. Do not spawn Devil's Advocate if the file is missing or any Claimed row lacks Reality. Report Spec and Standards separately. Use `context query <term>` and `context related <file>` before opening broad source files. Use `reviewProfile` from runtime policy (default `lean`) for context packs; use iteration-delta review in iteration 2+. DA prompt: "at most 8 tool calls; read the diff file; skip css and generated files". Judge prompt: "at most 4 tool calls; ledger and contract only; skip css; do not re-review the whole diff". Never use `browser_subagent` as reviewer. `generate-workflow` reads workflow.md and lessons.md; missing maker_intern means INVOCATION_UNAVAILABLE. Do not type a model name.
|
|
13
13
|
|
|
14
|
-
Stage 1: if the user asks for ideas, list a short menu and wait; do not implement. Grill if the task is ambiguous (`core/grill-policy.md`). Chat agreement is not freeze. Publish the complete failure-table JSON through `npx ai-engineering-loop goal draft --run <runId> --file <repository-local-json> --revision <currentRevision>` so Studio shows the exact draft, then wait. After explicit human freeze, run `goal freeze --run <runId> --revision <reviewedRevision> --hash <reviewedHash> --by human`; `gate goal` remains legacy compatibility only. Do not start Maker before freeze. On business-logic change, and always on `adapter_type: dot`, grill includes blast radius: lifecycle sketch, four pillars (state, sibling, approval, queues), ASCII picture. Load `task-impact-inquiry` if present. Do not run a second interview. Passing unit tests are not isolation proof. Use `.ai-engineering-loop/glossary.md`. Name test seams. AC is a failure table (happy, empty/omit, boundary, sibling, error). One red test per AC row. Do not freeze sunny-path-only. Stages 2-4 branch on mode: REPORT_ONLY never edits repository files, writes RUN_DIR/report.json, runs `gate report`, and stops. ASSISTED uses normal Maker. UNATTENDED runs `sandbox create --run <runId>`, Maker only in the returned worktreePath, never edits the parent checkout, and runs `sandbox capture --run <runId>`. Read `maker_intern` from workflow.md (missing or none = parent is Maker). Do not type a model name. If intern is set, confirm it came from this host's list (`/models` if that command exists). If the list command is missing or the id is not on it, parent is Maker; say intern INVOCATION_UNAVAILABLE. Intern cannot skip verification, Devil's Advocate, or Judge. Maker: TDD at those seams (`policies/tdd-policy.md`): one red test per AC row. ASSISTED writes RUN_DIR/diff.patch; UNATTENDED capture writes it;
|
|
14
|
+
Stage 1: if the user asks for ideas, list a short menu and wait; do not implement. Grill if the task is ambiguous (`core/grill-policy.md`). Chat agreement is not freeze. Publish the complete failure-table JSON through `npx ai-engineering-loop goal draft --run <runId> --file <repository-local-json> --revision <currentRevision>` so Studio shows the exact draft, then wait. After explicit human freeze, run `goal freeze --run <runId> --revision <reviewedRevision> --hash <reviewedHash> --by human`; `gate goal` remains legacy compatibility only. Do not start Maker before freeze. On business-logic change, and always on `adapter_type: dot`, grill includes blast radius: lifecycle sketch, four pillars (state, sibling, approval, queues), ASCII picture. Load `task-impact-inquiry` if present. Do not run a second interview. Passing unit tests are not isolation proof. Use `.ai-engineering-loop/glossary.md`. Name test seams. AC is a failure table (happy, empty/omit, boundary, sibling, error). One red test per AC row. Do not freeze sunny-path-only. Stages 2-4 branch on mode: REPORT_ONLY never edits repository files, writes RUN_DIR/report.json, runs `gate report`, and stops. ASSISTED uses normal Maker. UNATTENDED runs `sandbox create --run <runId>`, Maker only in the returned worktreePath, never edits the parent checkout, and runs `sandbox capture --run <runId>`. Read `maker_intern` from workflow.md (missing or none = parent is Maker). Do not type a model name. If intern is set, confirm it came from this host's list (`/models` if that command exists). If the list command is missing or the id is not on it, parent is Maker; say intern INVOCATION_UNAVAILABLE. Intern cannot skip verification, Devil's Advocate, or Judge. Maker: TDD at those seams (`policies/tdd-policy.md`): one red test per AC row. ASSISTED writes RUN_DIR/diff.patch; UNATTENDED capture writes it. After package/manifest edits stabilize, run `npx ai-engineering-loop refresh` once; drift during red-green is expected, so do not refresh after each test cycle. Then run `npx ai-engineering-loop gate maker`. Stage 5 may use `npx ai-engineering-loop verification record --run <runId> -- <executable> [args...]` to capture bounded, redacted output and generate verification.json plus its summary without a shell. Use `--shell -- "command"` only when necessary. Failed commands stay recorded and fail the verification gate. `npx ai-engineering-loop scaffold verification --run <runId>` (or `findings` / `delivery`) creates a deliberately incomplete template; edit it into real evidence, remove `_templateOnly`, and run `npx ai-engineering-loop validate <repository-relative-file>` for diagnostics. Templates never pass a gate. Generate AC rows with `npx ai-engineering-loop context claimed-vs-reality --run <runId>` and fill Claim/Reality from actual changes and command output. Run `npx ai-engineering-loop context compact --run <runId>`, pass compact summaries to reviewers instead of raw logs, then `gate verification`. Bugs: red repro first (`core/root-cause-analysis.md`). Mid-loop stop: `core/handoff-policy.md`.
|
|
15
15
|
|
|
16
16
|
If `invoke_subagent` (or Task) exists, spawn `devil-advocate` then `judge` as siblings. Wait for each child. Do not run them in the background. Use `general-purpose` only if the named type is rejected.
|
|
17
17
|
|
|
@@ -21,4 +21,4 @@ If `invoke_subagent` (or Task) exists, spawn `devil-advocate` then `judge` as si
|
|
|
21
21
|
|
|
22
22
|
If no subagent tool exists, run the same budgets in this session as CONTEXT_ISOLATION_ONLY. Do not claim independent agent execution.
|
|
23
23
|
|
|
24
|
-
Init/status/refresh/sync-hosts/generate-adapter/generate-workflow: `npx ai-engineering-loop <command>`. generate-workflow grills Q1-Q6. Q6 Maker intern: pick from the host catalog; do not type a model name. After Judge PASS, PR/MR body includes `Steps to Reproduce & Testing (QA)` and bugfix placeholders must not stay empty. Preview Stage 8 remote payloads and never include context packs, usage ledgers, raw logs, or credentials. In ASSISTED, wait for explicit human approval and write `humanApproved: true` in RUN_DIR/delivery.json; never infer it.
|
|
24
|
+
Init/status/refresh/sync-hosts/generate-adapter/generate-workflow: `npx ai-engineering-loop <command>`. generate-workflow grills Q1-Q6. Q6 Maker intern: pick from the host catalog; do not type a model name. After Judge PASS, PR/MR body includes `Steps to Reproduce & Testing (QA)` and bugfix placeholders must not stay empty. Preview Stage 8 remote payloads and never include context packs, usage ledgers, raw logs, or credentials. In ASSISTED, wait for explicit human approval and write `humanApproved: true` in RUN_DIR/delivery.json; never infer it. Before `gate delivery`, inspect `npx ai-engineering-loop node status --run <runId>` and explicitly approve any READY approval prerequisite with `npx ai-engineering-loop node approve delivery-approval --yes --by <name> --run <runId>`. A blocked gate should explain the READY prerequisite and next command; never bypass approval. Only `gate delivery` completes the run. Confirmed process lessons go to lessons.md, not a chat dump.
|
|
@@ -47,8 +47,8 @@ Inherited invariants remain mandatory: read `core/grill-policy.md`, `policies/td
|
|
|
47
47
|
|
|
48
48
|
1. Stage 0: `npx ai-engineering-loop sync-hosts`, `status`, then `run [--mode report-only|assisted|unattended]` to create or resume the ledger. Run `state --json` and `budget status`. Read `.ai-engineering-loop/glossary.md`. Read `.ai-engineering-loop/lessons.md` and `.ai-engineering-loop/workflow.md` if they exist. Follow workflow.md hooks (`before_grill`, `after_freeze`, `after_pass`). Do not skip required gates even if workflow.md asks. If workflow.md is missing, run the default 8-stage loop. If sync-hosts copied files, tell the user a new session is needed; keep going with this session.
|
|
49
49
|
2. Stage 1: Goal Contract (`core/goal-contract.md`). If the user asks for ideas, list a short menu and wait; do not implement. If the task is ambiguous and the user can answer, grill first (`core/grill-policy.md`): design tree, recommended answers, do not ask look-up facts. On business-logic change, and always on `adapter_type: dot`, include blast radius. Chat agreement is not freeze. Emit `npx ai-engineering-loop activity goal-drafting --run <runId> --message <summary>` before drafting. Write the contract JSON to a repository-local temporary file, then publish it with `npx ai-engineering-loop goal draft --run <runId> --file <file> --revision <currentRevision>` so Workflow Studio sees the exact draft and its ready state. Wait. After the user explicitly freezes, use `npx ai-engineering-loop goal freeze --run <runId> --revision <reviewedRevision> --hash <reviewedHash> --by human`; do not edit production code unless it passes. `gate goal` remains the legacy compatibility command, not the preferred revision-bound path.
|
|
50
|
-
3. After Goal freeze, branch on mode. REPORT_ONLY: do not edit repository files; write `RUN_DIR/report.json` with summary and recommendations, then run `gate report` and stop. ASSISTED: run the normal Maker and later require human approval. UNATTENDED: run `sandbox create --run <runId>`, set Maker's working directory to the returned worktreePath, never edit the parent checkout, then run `sandbox capture --run <runId>`. For Maker, read `maker_intern` from workflow.md (missing or none = parent is Maker). Do not type a model name. Confirm it through `/models` when available; otherwise parent is Maker and report INVOCATION_UNAVAILABLE. Intern cannot skip later gates. Bugs: red repro first. Features: one red test per AC row. In ASSISTED write the exact review diff to `RUN_DIR/diff.patch`; UNATTENDED capture writes it. Then run `npx ai-engineering-loop gate maker`.
|
|
51
|
-
4. Stage 5: run
|
|
50
|
+
3. After Goal freeze, branch on mode. REPORT_ONLY: do not edit repository files; write `RUN_DIR/report.json` with summary and recommendations, then run `gate report` and stop. ASSISTED: run the normal Maker and later require human approval. UNATTENDED: run `sandbox create --run <runId>`, set Maker's working directory to the returned worktreePath, never edit the parent checkout, then run `sandbox capture --run <runId>`. For Maker, read `maker_intern` from workflow.md (missing or none = parent is Maker). Do not type a model name. Confirm it through `/models` when available; otherwise parent is Maker and report INVOCATION_UNAVAILABLE. Intern cannot skip later gates. Bugs: red repro first. Features: one red test per AC row. In ASSISTED write the exact review diff to `RUN_DIR/diff.patch`; UNATTENDED capture writes it. After all package/manifest edits are stable, run `npx ai-engineering-loop refresh` once; manifest drift during red-green is expected, so do not refresh after each test cycle. Then run `npx ai-engineering-loop gate maker`.
|
|
51
|
+
4. Stage 5: run each command from `.ai-engineering-loop/verification.md` through `npx ai-engineering-loop verification record --run <runId> -- <executable> [args...]`. It captures bounded, redacted output, binds evidence to the gated diff, writes `RUN_DIR/verification.json`, and refreshes `verification-summary.json`; commands run without a shell by default. Use `--shell -- "command"` only when shell syntax is required. Run `npx ai-engineering-loop context compact --run <runId>` and pass run-summary.md and verification-summary.json to reviewers instead of raw logs whenever possible. A failed command is still recorded and returns a non-zero CLI exit; it cannot pass `gate verification`. Vague "seems green" is invalid. Run `npx ai-engineering-loop context claimed-vs-reality --run <runId>` to scaffold AC rows from the frozen Goal; fill Claimed and Reality from actual changes and command output. For hand-authored artifacts, `npx ai-engineering-loop scaffold verification --run <runId>` (or `findings` / `delivery`) writes a deliberately incomplete `.template.json`; edit it into the real artifact, remove `_templateOnly`, then use `npx ai-engineering-loop validate <repository-relative-file>` for diagnostics. Templates are never evidence. Run `npx ai-engineering-loop gate verification`; do not spawn Devil's Advocate unless it passes.
|
|
52
52
|
5. Write artifacts, then spawn. Before Devil's Advocate:
|
|
53
53
|
- Write `git diff` to a file (for example `.ai-engineering-loop/tasks/current.diff`).
|
|
54
54
|
- Write changed paths (`git diff --name-only`) into the Task prompt as a short list.
|
|
@@ -58,7 +58,7 @@ Inherited invariants remain mandatory: read `core/grill-policy.md`, `policies/td
|
|
|
58
58
|
6. Stage 6: Run `npx ai-engineering-loop context fast-path --json`; keep the default lean path when eligible. Build `npx ai-engineering-loop context diff-hunks --run <runId>` plus `npx ai-engineering-loop context devil-advocate <files...>` only for files whose hunks are insufficient. In iteration 2+, prefer the diff-hunk pack's unresolved findings and new hunks instead of replaying the whole diff. Recheck `budget status`, then Task `subagent_type: devil-advocate`. Use `general-purpose` only if rejected. Wait and immediately `budget record` its actual usage. Normalize its ledger into `RUN_DIR/findings.json` with runId and diffHash, then run `npx ai-engineering-loop gate review`.
|
|
59
59
|
7. Stage 7: Run `npx ai-engineering-loop escalation --json` before selecting Judge. Resolve the required tier through `/models`; if unavailable, escalate to the user. Build Judge context from only run-summary.md, verification-summary.json (or verification.json if the summary is missing), open-findings.json/findings.json, and the Goal hash; Judge must not read the full diff unless those summaries contradict each other. Recheck `budget status`, spawn Judge as a sibling and wait, then immediately `budget record` actual usage. Normalize its output into `RUN_DIR/verdict.json`, include `modelTier`, then run `npx ai-engineering-loop gate judge`. A gate failure overrides optimistic prose.
|
|
60
60
|
8. ITERATE with iteration under 3: fix in the parent, re-verify, spawn a **new** Devil's Advocate (do not resume the previous child).
|
|
61
|
-
9. After Judge PASS: run the `after_pass` hook, then Stage 8 from `.ai-engineering-loop/adapter.md`. PR/MR body includes `Steps to Reproduce & Testing (QA)`; bugfix placeholders must not stay empty. Preview remote payloads; do not expose context packs, usage ledgers, raw logs, or credentials. In ASSISTED, wait for explicit human approval and write `humanApproved: true`; never infer approval.
|
|
61
|
+
9. After Judge PASS: run the `after_pass` hook, then Stage 8 from `.ai-engineering-loop/adapter.md`. PR/MR body includes `Steps to Reproduce & Testing (QA)`; bugfix placeholders must not stay empty. Preview remote payloads; do not expose context packs, usage ledgers, raw logs, or credentials. In ASSISTED, wait for explicit human approval and write `humanApproved: true`; never infer approval. For recipe-bound Runs, inspect `npx ai-engineering-loop node status --run <runId>` and explicitly approve any READY `approval` node (for example `node approve delivery-approval --yes --by <name> --run <runId>`). Then write `RUN_DIR/delivery.json` with destination and summary, validate it if useful, and run `npx ai-engineering-loop gate delivery`. If the gate says the node is not READY, follow its prerequisite status and command hint; do not bypass approval. Only this gate completes the run.
|
|
62
62
|
|
|
63
63
|
## Report header
|
|
64
64
|
|
|
@@ -40,8 +40,8 @@ Inherited invariants remain mandatory: read `core/grill-policy.md`, `policies/td
|
|
|
40
40
|
|
|
41
41
|
1. Stage 0: sync hosts, check status, then run `npx ai-engineering-loop run [--mode report-only|assisted|unattended]` to create or resume the ledger. Run `state --json` and `budget status`. Read glossary, lessons, and workflow overlays. Required gates cannot be skipped.
|
|
42
42
|
2. Stage 1: grill ambiguous decisions and business blast radius first. Emit `npx ai-engineering-loop activity goal-drafting --run <runId> --message <summary>`. Write the Goal Contract JSON to a repository-local temporary file, then publish it with `npx ai-engineering-loop goal draft --run <runId> --file <file> --revision <currentRevision>` so Workflow Studio sees it and its ready state. Wait. Chat agreement is not freeze. After explicit human freeze, use `npx ai-engineering-loop goal freeze --run <runId> --revision <reviewedRevision> --hash <reviewedHash> --by human`; do not edit production code unless it passes. `gate goal` remains the legacy compatibility command, not the preferred revision-bound path.
|
|
43
|
-
3. After Goal freeze, branch on mode. REPORT_ONLY: never edit repository files; write `RUN_DIR/report.json`, run `gate report`, and stop. ASSISTED: use the normal Maker. UNATTENDED: run `sandbox create --run <runId>`, run Maker only in the returned worktreePath, never edit the parent checkout, then run `sandbox capture --run <runId>`. Resolve maker_intern only through the host catalog; otherwise parent is Maker. Intern cannot skip gates. Use red repro or TDD. ASSISTED writes `RUN_DIR/diff.patch`; UNATTENDED capture writes it. Then run `npx ai-engineering-loop gate maker`.
|
|
44
|
-
4. Stage 5:
|
|
43
|
+
3. After Goal freeze, branch on mode. REPORT_ONLY: never edit repository files; write `RUN_DIR/report.json`, run `gate report`, and stop. ASSISTED: use the normal Maker. UNATTENDED: run `sandbox create --run <runId>`, run Maker only in the returned worktreePath, never edit the parent checkout, then run `sandbox capture --run <runId>`. Resolve maker_intern only through the host catalog; otherwise parent is Maker. Intern cannot skip gates. Use red repro or TDD. ASSISTED writes `RUN_DIR/diff.patch`; UNATTENDED capture writes it. After package/manifest edits are stable, run `npx ai-engineering-loop refresh` once; drift during red-green is expected, so avoid refreshing after each test cycle. Then run `npx ai-engineering-loop gate maker`.
|
|
44
|
+
4. Stage 5: record each verification command with `npx ai-engineering-loop verification record --run <runId> -- <executable> [args...]`. It runs without a shell by default, captures bounded/redacted output, binds evidence to the gated diff, writes `RUN_DIR/verification.json`, and updates the summary. Use `--shell -- "command"` only when shell syntax is required. Failed commands are recorded and return non-zero; they cannot pass the gate. Run `npx ai-engineering-loop context compact --run <runId>` and pass run-summary.md and verification-summary.json to reviewers instead of raw logs whenever possible. Use `npx ai-engineering-loop context claimed-vs-reality --run <runId>` to scaffold AC rows from the frozen Goal, then fill both columns from real changes and command output. For hand-authored artifacts, `npx ai-engineering-loop scaffold verification --run <runId>` (or `findings` / `delivery`) creates an incomplete `.template.json`; edit it into the real artifact, remove `_templateOnly`, then use `npx ai-engineering-loop validate <repository-relative-file>` for diagnostics. Templates are never evidence. Run `npx ai-engineering-loop gate verification`; do not spawn DA unless it passes.
|
|
45
45
|
5. Write artifacts, then spawn. Before Devil's Advocate:
|
|
46
46
|
- Write `git diff` to a file (for example `.ai-engineering-loop/tasks/current.diff`).
|
|
47
47
|
- Write changed paths (`git diff --name-only`) into the child prompt as a short list.
|
|
@@ -51,7 +51,7 @@ Inherited invariants remain mandatory: read `core/grill-policy.md`, `policies/td
|
|
|
51
51
|
6. Stage 6: Run `npx ai-engineering-loop context fast-path --json`; keep the default lean path when eligible. Build `npx ai-engineering-loop context diff-hunks --run <runId>`, then `context devil-advocate <files...>` only when hunks are insufficient. Use unresolved findings and new hunks instead of replaying accepted old findings. Check `budget status`, spawn DA with the pack path, wait, and `budget record` actual usage. Normalize its output into `RUN_DIR/findings.json` with runId and diffHash, then run `npx ai-engineering-loop gate review`.
|
|
52
52
|
7. Stage 7: Run `npx ai-engineering-loop escalation --json`, resolve the required tier through the host catalog, or escalate to the user. Build Judge context from only run-summary.md, verification-summary.json (or verification.json if the summary is missing), open-findings.json/findings.json, and the Goal hash; Judge must not read the full diff unless those summaries contradict each other. Check `budget status`, spawn Judge, wait, and `budget record` actual usage. Write `RUN_DIR/verdict.json` with modelTier, then run `npx ai-engineering-loop gate judge`.
|
|
53
53
|
8. ITERATE with iteration under 3: fix in the parent, re-verify, spawn a **new** Devil's Advocate.
|
|
54
|
-
9. After Judge PASS: run after_pass, then the configured Stage 8 adapter. PR/MR body includes `Steps to Reproduce & Testing (QA)`; bugfix placeholders must not stay empty. Preview remote payloads and never include context packs, usage ledgers, raw logs, or credentials. In ASSISTED, wait for explicit human approval and put `humanApproved: true` in `RUN_DIR/delivery.json`; never infer it. Then run `npx ai-engineering-loop gate delivery
|
|
54
|
+
9. After Judge PASS: run after_pass, then the configured Stage 8 adapter. PR/MR body includes `Steps to Reproduce & Testing (QA)`; bugfix placeholders must not stay empty. Preview remote payloads and never include context packs, usage ledgers, raw logs, or credentials. In ASSISTED, wait for explicit human approval and put `humanApproved: true` in `RUN_DIR/delivery.json`; never infer it. For recipe-bound Runs, check `node status --run <runId>` and explicitly approve every READY approval prerequisite (for example `node approve delivery-approval --yes --by <name> --run <runId>`). Then run `npx ai-engineering-loop gate delivery`; follow actionable prerequisite hints if the gate says the delivery node is not READY.
|
|
55
55
|
|
|
56
56
|
## Report header
|
|
57
57
|
|
|
@@ -58,8 +58,8 @@ Inherited invariants remain mandatory: read `core/grill-policy.md`, `policies/td
|
|
|
58
58
|
|
|
59
59
|
1. Stage 0: `npx ai-engineering-loop sync-hosts`, `status`, then `run [--mode report-only|assisted|unattended]` to create or resume the ledger. Run `state --json` and `budget status`. Read glossary, lessons, and workflow overlays. Required gates cannot be skipped.
|
|
60
60
|
2. Stage 1: grill ambiguous decisions and business blast radius first. Emit `npx ai-engineering-loop activity goal-drafting --run <runId> --message <summary>`. Write the Goal Contract JSON to a repository-local temporary file, then publish it with `npx ai-engineering-loop goal draft --run <runId> --file <file> --revision <currentRevision>` so Workflow Studio sees it and its ready state. Wait. Chat agreement is not freeze. After explicit human freeze, use `npx ai-engineering-loop goal freeze --run <runId> --revision <reviewedRevision> --hash <reviewedHash> --by human`; do not edit production code unless it passes. `gate goal` remains the legacy compatibility command, not the preferred revision-bound path.
|
|
61
|
-
3. After Goal freeze, branch on mode. REPORT_ONLY: never edit repository files; write `RUN_DIR/report.json`, run `gate report`, and stop. ASSISTED: use the normal Maker. UNATTENDED: run `sandbox create --run <runId>`, run Maker only in the returned worktreePath, never edit the parent checkout, then run `sandbox capture --run <runId>`. Resolve `maker_intern` only through `grok models`; unknown means parent Maker. Intern cannot skip gates or spawn DA/Judge. Use red repro or TDD. ASSISTED writes `RUN_DIR/diff.patch`; UNATTENDED capture writes it. Then run `npx ai-engineering-loop gate maker`.
|
|
62
|
-
4. Stage 5:
|
|
61
|
+
3. After Goal freeze, branch on mode. REPORT_ONLY: never edit repository files; write `RUN_DIR/report.json`, run `gate report`, and stop. ASSISTED: use the normal Maker. UNATTENDED: run `sandbox create --run <runId>`, run Maker only in the returned worktreePath, never edit the parent checkout, then run `sandbox capture --run <runId>`. Resolve `maker_intern` only through `grok models`; unknown means parent Maker. Intern cannot skip gates or spawn DA/Judge. Use red repro or TDD. ASSISTED writes `RUN_DIR/diff.patch`; UNATTENDED capture writes it. After package/manifest edits are stable, run `npx ai-engineering-loop refresh` once; drift during red-green is expected, so do not refresh after every test cycle. Then run `npx ai-engineering-loop gate maker`.
|
|
62
|
+
4. Stage 5: record each verification command with `npx ai-engineering-loop verification record --run <runId> -- <executable> [args...]`. It runs without a shell by default, captures bounded/redacted output, binds evidence to the gated diff, writes `RUN_DIR/verification.json`, and updates the summary. Use `--shell -- "command"` only when shell syntax is required. Failed commands are recorded and return non-zero; they cannot pass the gate. Run `npx ai-engineering-loop context compact --run <runId>` and pass run-summary.md and verification-summary.json to reviewers instead of raw logs whenever possible. Use `npx ai-engineering-loop context claimed-vs-reality --run <runId>` to scaffold AC rows from the frozen Goal, then fill both columns from real changes and command output. For hand-authored artifacts, `npx ai-engineering-loop scaffold verification --run <runId>` (or `findings` / `delivery`) creates an incomplete `.template.json`; edit it into the real artifact, remove `_templateOnly`, then use `npx ai-engineering-loop validate <repository-relative-file>` for diagnostics. Templates are never evidence. Run `npx ai-engineering-loop gate verification`; do not spawn DA unless it passes.
|
|
63
63
|
5. Write artifacts to disk so children do not need parent chat:
|
|
64
64
|
- Goal Contract path
|
|
65
65
|
- `git diff` written to `.ai-engineering-loop/tasks/current.diff`
|
|
@@ -69,7 +69,7 @@ Inherited invariants remain mandatory: read `core/grill-policy.md`, `policies/td
|
|
|
69
69
|
6. Stage 6: Run `npx ai-engineering-loop context fast-path --json`; keep the default lean path when eligible. Build `npx ai-engineering-loop context diff-hunks --run <runId>`, then `context devil-advocate <files...>` only when hunks are insufficient. Use unresolved findings and new hunks instead of replaying accepted old findings. Check `budget status`, spawn Devil's Advocate with the pack path, `background: false`, `capability_mode: "execute"`, and no `resume_from`; wait and `budget record` actual usage. Normalize its output into `RUN_DIR/findings.json` with runId and diffHash, then run `npx ai-engineering-loop gate review`.
|
|
70
70
|
7. Stage 7: Run `npx ai-engineering-loop escalation --json`; resolve `STRONG_MODEL` through `grok models`, or escalate to the user if unavailable. Build Judge context from only run-summary.md, verification-summary.json (or verification.json if the summary is missing), open-findings.json/findings.json, and the Goal hash; Judge must not read the full diff unless those summaries contradict each other. Check `budget status`, spawn a fresh sibling Judge and wait, then `budget record` actual usage. Normalize `RUN_DIR/verdict.json` with modelTier, then run `npx ai-engineering-loop gate judge`.
|
|
71
71
|
8. If Judge says `ITERATE` and iteration < 3, Maker fixes in the parent, re-verify, spawn a **fresh** DA (new spawn, no resume).
|
|
72
|
-
9. After Judge PASS: run `after_pass`, then the configured Stage 8 adapter. PR/MR body includes `Steps to Reproduce & Testing (QA)`; bugfix placeholders must not stay empty. Preview remote payloads and never include context packs, usage ledgers, raw logs, or credentials. In ASSISTED, wait for explicit human approval and put `humanApproved: true` in `RUN_DIR/delivery.json`; never infer it. Then run `npx ai-engineering-loop gate delivery
|
|
72
|
+
9. After Judge PASS: run `after_pass`, then the configured Stage 8 adapter. PR/MR body includes `Steps to Reproduce & Testing (QA)`; bugfix placeholders must not stay empty. Preview remote payloads and never include context packs, usage ledgers, raw logs, or credentials. In ASSISTED, wait for explicit human approval and put `humanApproved: true` in `RUN_DIR/delivery.json`; never infer it. For recipe-bound Runs, check `node status --run <runId>` and explicitly approve every READY approval prerequisite (for example `node approve delivery-approval --yes --by <name> --run <runId>`). Then run `npx ai-engineering-loop gate delivery`; follow actionable prerequisite hints if the gate says the delivery node is not READY.
|
|
73
73
|
|
|
74
74
|
After a proven Grok DA spawn, the report header must be:
|
|
75
75
|
|
package/README.md
CHANGED
|
@@ -55,6 +55,18 @@ Workflow Studio opens locally in your browser. Your project and run evidence sta
|
|
|
55
55
|
|
|
56
56
|
On first setup, `.ai-engineering-loop/` is added to your project’s `.gitignore` so plans and Run evidence stay private by default. Want to share that context with your team? Simply remove that line from `.gitignore`.
|
|
57
57
|
|
|
58
|
+
### Less artifact guesswork for your AI agent
|
|
59
|
+
|
|
60
|
+
Agents can record test commands directly so verification evidence is tied to the current Run and gated diff:
|
|
61
|
+
|
|
62
|
+
```bash
|
|
63
|
+
npx ai-engineering-loop verification record --run <run-id> -- npm test
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
The command captures bounded, redacted output without invoking a shell implicitly, records failures honestly, and generates a compact test summary. For other stage artifacts, use `scaffold delivery --run <run-id>` (or `verification` / `findings`) to create a private, deliberately incomplete template. Copy or rename it to the gate's real artifact filename, fill it with observed evidence, and remove `_templateOnly`; then `validate <repository-relative-file>` reports schema and Run-binding issues without advancing a gate. Templates are never accepted as evidence. `context claimed-vs-reality --run <run-id>` creates the Goal checklist while leaving claim and evidence cells blank.
|
|
67
|
+
|
|
68
|
+
When context packs exceed their budget, the CLI now reports the limit and a per-file size/token estimate so an agent can choose smaller files without trial and error. For a delivery gate blocked by a recipe approval node, inspect `node status --run <run-id>` and explicitly approve the READY prerequisite; the gate error points to the actionable command.
|
|
69
|
+
|
|
58
70
|
## Meet Workflow Studio
|
|
59
71
|
|
|
60
72
|
Workflow Studio makes agent work feel like a product, not a terminal session.
|
|
@@ -39,7 +39,8 @@ const {
|
|
|
39
39
|
createOrResumeRun,
|
|
40
40
|
getCurrentRun,
|
|
41
41
|
loadRun,
|
|
42
|
-
newRunId
|
|
42
|
+
newRunId,
|
|
43
|
+
runsDir
|
|
43
44
|
} = require('../lib/run-state.js');
|
|
44
45
|
const {
|
|
45
46
|
DEFAULT_POLICY,
|
|
@@ -80,6 +81,16 @@ const {
|
|
|
80
81
|
const {
|
|
81
82
|
writeVerificationSummary
|
|
82
83
|
} = require('../lib/verification-summary.js');
|
|
84
|
+
const {
|
|
85
|
+
recordVerificationCommand
|
|
86
|
+
} = require('../lib/verification-recorder.js');
|
|
87
|
+
const {
|
|
88
|
+
scaffoldArtifact,
|
|
89
|
+
validateArtifactFile
|
|
90
|
+
} = require('../lib/agent-artifacts.js');
|
|
91
|
+
const {
|
|
92
|
+
createClaimedVsRealityTemplate
|
|
93
|
+
} = require('../lib/claimed-vs-reality.js');
|
|
83
94
|
const {
|
|
84
95
|
runDoctor
|
|
85
96
|
} = require('../lib/doctor.js');
|
|
@@ -1163,12 +1174,72 @@ function handleSandbox() {
|
|
|
1163
1174
|
}
|
|
1164
1175
|
}
|
|
1165
1176
|
|
|
1166
|
-
function handleVerification() {
|
|
1177
|
+
async function handleVerification() {
|
|
1167
1178
|
const action = process.argv[3] || 'summarize';
|
|
1168
|
-
const
|
|
1169
|
-
const
|
|
1179
|
+
const separatorIndex = process.argv.indexOf('--');
|
|
1180
|
+
const optionArgs = action === 'record' && separatorIndex !== -1
|
|
1181
|
+
? process.argv.slice(0, separatorIndex)
|
|
1182
|
+
: process.argv;
|
|
1183
|
+
const readOption = (name) => {
|
|
1184
|
+
const index = optionArgs.indexOf(name);
|
|
1185
|
+
return index === -1 ? null : optionArgs[index + 1] || null;
|
|
1186
|
+
};
|
|
1187
|
+
const json = optionArgs.includes('--json');
|
|
1188
|
+
const runId = readOption('--run');
|
|
1170
1189
|
try {
|
|
1171
|
-
if (action
|
|
1190
|
+
if (action === 'record') {
|
|
1191
|
+
if (separatorIndex === -1 || !process.argv[separatorIndex + 1]) {
|
|
1192
|
+
throw Object.assign(
|
|
1193
|
+
new Error('Provide an executable after `--`, for example `ai-engineering-loop verification record --run <id> -- npm test`.'),
|
|
1194
|
+
{ code: 'INVALID_COMMAND' }
|
|
1195
|
+
);
|
|
1196
|
+
}
|
|
1197
|
+
const command = process.argv[separatorIndex + 1];
|
|
1198
|
+
const commandArgs = process.argv.slice(separatorIndex + 2);
|
|
1199
|
+
const timeoutValue = readOption('--timeout-ms');
|
|
1200
|
+
const result = await recordVerificationCommand(CWD, {
|
|
1201
|
+
runId,
|
|
1202
|
+
command,
|
|
1203
|
+
args: commandArgs,
|
|
1204
|
+
shell: optionArgs.includes('--shell'),
|
|
1205
|
+
timeoutMs: timeoutValue == null
|
|
1206
|
+
? undefined
|
|
1207
|
+
: parsePositiveIntegerArg('--timeout-ms', timeoutValue)
|
|
1208
|
+
});
|
|
1209
|
+
if (json) {
|
|
1210
|
+
console.log(JSON.stringify({
|
|
1211
|
+
ok: result.command.exitCode === 0 && result.command.timeoutStatus === 'COMPLETED',
|
|
1212
|
+
runId: result.runId,
|
|
1213
|
+
path: result.path,
|
|
1214
|
+
summaryPath: path.relative(CWD, path.join(runsDir(CWD), result.runId, 'verification-summary.json'))
|
|
1215
|
+
.split(path.sep).join('/'),
|
|
1216
|
+
command: result.command.command,
|
|
1217
|
+
exitCode: result.command.exitCode,
|
|
1218
|
+
timeoutStatus: result.command.timeoutStatus,
|
|
1219
|
+
testCounts: result.summary.testCounts
|
|
1220
|
+
}));
|
|
1221
|
+
} else {
|
|
1222
|
+
const passed = result.command.exitCode === 0 && result.command.timeoutStatus === 'COMPLETED';
|
|
1223
|
+
if (passed) log.success('✓ Verification command recorded');
|
|
1224
|
+
else log.error('Verification command recorded with a failing result');
|
|
1225
|
+
console.log(`- Run: ${result.runId}`);
|
|
1226
|
+
console.log(`- Command: ${result.command.command}`);
|
|
1227
|
+
console.log(`- Exit code: ${result.command.exitCode}; timeout: ${result.command.timeoutStatus}`);
|
|
1228
|
+
console.log(`- Evidence: ${result.path}`);
|
|
1229
|
+
console.log(`- Summary: ${JSON.stringify(result.summary.testCounts)}`);
|
|
1230
|
+
if (!passed) {
|
|
1231
|
+
const excerpt = [result.command.stderr, result.command.stdout]
|
|
1232
|
+
.map((value) => String(value || '').trim())
|
|
1233
|
+
.find(Boolean);
|
|
1234
|
+
if (excerpt) console.error(`- Output: ${excerpt.split('\n').slice(-3).join('\n').slice(-800)}`);
|
|
1235
|
+
}
|
|
1236
|
+
}
|
|
1237
|
+
if (result.command.exitCode !== 0 || result.command.timeoutStatus !== 'COMPLETED') {
|
|
1238
|
+
process.exitCode = result.command.exitCode > 0 ? result.command.exitCode : 1;
|
|
1239
|
+
}
|
|
1240
|
+
return;
|
|
1241
|
+
}
|
|
1242
|
+
if (action !== 'summarize') throw new Error('Unknown verification action. Use summarize|record');
|
|
1172
1243
|
const result = writeVerificationSummary(CWD, { runId });
|
|
1173
1244
|
if (json) {
|
|
1174
1245
|
console.log(JSON.stringify({
|
|
@@ -1192,13 +1263,51 @@ function handleVerification() {
|
|
|
1192
1263
|
}
|
|
1193
1264
|
}
|
|
1194
1265
|
|
|
1266
|
+
function handleArtifactTool(action) {
|
|
1267
|
+
const json = process.argv.includes('--json');
|
|
1268
|
+
try {
|
|
1269
|
+
if (action === 'scaffold') {
|
|
1270
|
+
const type = process.argv[3];
|
|
1271
|
+
const result = scaffoldArtifact(CWD, { type, runId: argValue('--run') });
|
|
1272
|
+
if (json) console.log(JSON.stringify({ ok: true, ...result, templateOnly: true }));
|
|
1273
|
+
else {
|
|
1274
|
+
log.success(`✓ ${result.type} template created`);
|
|
1275
|
+
console.log(`- Run: ${result.runId}`);
|
|
1276
|
+
console.log(`- Path: ${result.path}`);
|
|
1277
|
+
console.log('- Template is deliberately incomplete; it cannot pass its gate until replaced with observed evidence.');
|
|
1278
|
+
}
|
|
1279
|
+
return;
|
|
1280
|
+
}
|
|
1281
|
+
if (action === 'validate') {
|
|
1282
|
+
const file = process.argv[3];
|
|
1283
|
+
if (!file || file.startsWith('-')) throw new Error('validate requires a repository-relative JSON file');
|
|
1284
|
+
const result = validateArtifactFile(CWD, { file, type: argValue('--type') });
|
|
1285
|
+
if (json) console.log(JSON.stringify({ ok: result.valid, ...result }));
|
|
1286
|
+
else {
|
|
1287
|
+
console.log(`Artifact ${result.path}: ${result.valid ? 'VALID' : 'INVALID'}`);
|
|
1288
|
+
for (const error of result.errors) console.error(`- ${error}`);
|
|
1289
|
+
if (result.note) console.log(`- ${result.note}`);
|
|
1290
|
+
}
|
|
1291
|
+
if (!result.valid) process.exitCode = 1;
|
|
1292
|
+
return;
|
|
1293
|
+
}
|
|
1294
|
+
throw new Error('Use scaffold <verification|findings|delivery> or validate <file>');
|
|
1295
|
+
} catch (err) {
|
|
1296
|
+
if (json) console.log(JSON.stringify({ ok: false, code: err.code || 'ARTIFACT_TOOL_FAILED', error: err.message }));
|
|
1297
|
+
else log.error(err.message);
|
|
1298
|
+
process.exit(1);
|
|
1299
|
+
}
|
|
1300
|
+
}
|
|
1301
|
+
|
|
1195
1302
|
function handleGate() {
|
|
1196
1303
|
const gate = process.argv[3];
|
|
1197
1304
|
const runId = argValue('--run');
|
|
1198
1305
|
const json = process.argv.includes('--json');
|
|
1306
|
+
let targetRunId = runId;
|
|
1199
1307
|
try {
|
|
1200
1308
|
const target = runId ? loadRun(CWD, runId) : getCurrentRun(CWD);
|
|
1201
1309
|
if (!target) throw new Error('No current run. Start one with `ai-engineering-loop run`.');
|
|
1310
|
+
targetRunId = target.runId;
|
|
1202
1311
|
listRunLifecycle(CWD, target.runId);
|
|
1203
1312
|
let lifecycleNode = null;
|
|
1204
1313
|
if (target.workflow) {
|
|
@@ -1243,16 +1352,20 @@ function handleGate() {
|
|
|
1243
1352
|
console.log(`- State: ${run.state}, iteration ${run.iteration}`);
|
|
1244
1353
|
}
|
|
1245
1354
|
} catch (err) {
|
|
1355
|
+
const retryHint = targetRunId && gate && err.code === 'GATE_FAILED'
|
|
1356
|
+
? `After correcting the reported issue, retry: npx ai-engineering-loop gate ${gate} --run ${targetRunId}`
|
|
1357
|
+
: null;
|
|
1358
|
+
const details = [...(err.details || []), ...(retryHint ? [retryHint] : [])];
|
|
1246
1359
|
if (json) {
|
|
1247
1360
|
console.log(JSON.stringify({
|
|
1248
1361
|
ok: false,
|
|
1249
1362
|
code: err.code || 'GATE_FAILED',
|
|
1250
1363
|
error: err.message,
|
|
1251
|
-
details
|
|
1364
|
+
details
|
|
1252
1365
|
}));
|
|
1253
1366
|
} else {
|
|
1254
1367
|
log.error(err.message);
|
|
1255
|
-
for (const detail of
|
|
1368
|
+
for (const detail of details) console.error(`- ${detail}`);
|
|
1256
1369
|
}
|
|
1257
1370
|
process.exit(1);
|
|
1258
1371
|
}
|
|
@@ -1385,7 +1498,7 @@ function handleContext() {
|
|
|
1385
1498
|
const limit = Number(argValue('--limit') || 10);
|
|
1386
1499
|
const json = process.argv.includes('--json');
|
|
1387
1500
|
const includeIgnored = process.argv.includes('--include-ignored');
|
|
1388
|
-
const files = commandFiles(4, ['--run', '--profile', '--base', '--query', '--limit']);
|
|
1501
|
+
const files = commandFiles(4, ['--run', '--profile', '--base', '--query', '--limit', '--output']);
|
|
1389
1502
|
try {
|
|
1390
1503
|
if (stage === 'index') {
|
|
1391
1504
|
const result = buildContextIndex(CWD, { files, includeIgnored });
|
|
@@ -1445,6 +1558,20 @@ function handleContext() {
|
|
|
1445
1558
|
}
|
|
1446
1559
|
return;
|
|
1447
1560
|
}
|
|
1561
|
+
if (stage === 'claimed-vs-reality') {
|
|
1562
|
+
const result = createClaimedVsRealityTemplate(CWD, {
|
|
1563
|
+
runId,
|
|
1564
|
+
outputPath: argValue('--output') || undefined
|
|
1565
|
+
});
|
|
1566
|
+
if (json) console.log(JSON.stringify({ ok: true, ...result }));
|
|
1567
|
+
else {
|
|
1568
|
+
log.success('✓ Claimed-vs-reality template created from the frozen Goal');
|
|
1569
|
+
console.log(`- Run: ${result.runId}`);
|
|
1570
|
+
console.log(`- Path: ${result.path}`);
|
|
1571
|
+
console.log('- Claimed and Reality remain blank; populate Reality only from actual command output.');
|
|
1572
|
+
}
|
|
1573
|
+
return;
|
|
1574
|
+
}
|
|
1448
1575
|
if (stage === 'diff-hunks' || stage === 'diff-pack') {
|
|
1449
1576
|
const result = createDiffHunkPack(CWD, { runId, base, profile, includeIgnored });
|
|
1450
1577
|
const summary = smartContextSummary(result);
|
|
@@ -1473,9 +1600,16 @@ function handleContext() {
|
|
|
1473
1600
|
}
|
|
1474
1601
|
} catch (err) {
|
|
1475
1602
|
if (json) {
|
|
1476
|
-
console.log(JSON.stringify({
|
|
1603
|
+
console.log(JSON.stringify({
|
|
1604
|
+
ok: false,
|
|
1605
|
+
code: err.code || 'UNSAFE_CONTEXT',
|
|
1606
|
+
error: err.message,
|
|
1607
|
+
details: err.details || [],
|
|
1608
|
+
...(err.budget ? { budget: err.budget } : {})
|
|
1609
|
+
}));
|
|
1477
1610
|
} else {
|
|
1478
1611
|
log.error(err.message);
|
|
1612
|
+
for (const detail of err.details || []) console.error(`- ${detail}`);
|
|
1479
1613
|
}
|
|
1480
1614
|
process.exit(1);
|
|
1481
1615
|
}
|
|
@@ -2089,6 +2223,13 @@ Commands:
|
|
|
2089
2223
|
freeze --revision <n> --hash <sha256> [--run <id>] [--by <name>]
|
|
2090
2224
|
verification Summarize verification.json for compact Judge context
|
|
2091
2225
|
summarize [--run <id>] [--json]
|
|
2226
|
+
record [--run <id>] [--timeout-ms <n>] [--shell] [--json] -- <executable> [args...]
|
|
2227
|
+
Captures bounded, redacted output; updates verification.json and its summary
|
|
2228
|
+
--shell opts into a shell; pass one quoted command string, e.g. -- "npm test"
|
|
2229
|
+
scaffold Create a private, deliberately incomplete stage-artifact template
|
|
2230
|
+
verification | findings | delivery [--run <id>] [--json]
|
|
2231
|
+
validate Check a verification, findings, or delivery JSON file without advancing a gate
|
|
2232
|
+
<repository-relative-file> [--type verification|findings|delivery] [--json]
|
|
2092
2233
|
activity Publish safe Run lifecycle summaries for Studio
|
|
2093
2234
|
goal-drafting | goal-ready | heartbeat | waiting
|
|
2094
2235
|
node-started | node-activity | node-completed | loop-iterated | run-completed
|
|
@@ -2125,6 +2266,8 @@ Commands:
|
|
|
2125
2266
|
diff-hunks build a hunk-only review pack with unresolved findings
|
|
2126
2267
|
fast-path assess whether a small task can stay on lean review
|
|
2127
2268
|
compact write run-summary.md/json and open-findings.json
|
|
2269
|
+
claimed-vs-reality scaffold AC rows from the frozen Goal; evidence stays blank
|
|
2270
|
+
[--run <id>] [--output <repository-relative-file>]
|
|
2128
2271
|
--run <id> target a non-current run
|
|
2129
2272
|
--profile lean|standard|thorough override runtime policy for one pack
|
|
2130
2273
|
--base <git-ref> base ref for diff-hunks; default HEAD
|
|
@@ -2222,6 +2365,12 @@ switch (command) {
|
|
|
2222
2365
|
case 'verification':
|
|
2223
2366
|
handleVerification();
|
|
2224
2367
|
break;
|
|
2368
|
+
case 'scaffold':
|
|
2369
|
+
handleArtifactTool('scaffold');
|
|
2370
|
+
break;
|
|
2371
|
+
case 'validate':
|
|
2372
|
+
handleArtifactTool('validate');
|
|
2373
|
+
break;
|
|
2225
2374
|
case 'sandbox':
|
|
2226
2375
|
handleSandbox();
|
|
2227
2376
|
break;
|
package/core/artifact-gates.md
CHANGED
|
@@ -28,6 +28,37 @@ JSON sidecars follow the schemas shipped in `schemas/`. Keep the Markdown Goal C
|
|
|
28
28
|
|
|
29
29
|
Gate failures are evidence. Do not edit state to bypass one. Repair the artifact or return to Maker.
|
|
30
30
|
|
|
31
|
+
## Agent-friendly artifact diagnostics
|
|
32
|
+
|
|
33
|
+
Use the recorder for verification commands instead of hand-writing process metadata:
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
npx ai-engineering-loop verification record --run <run-id> -- npm test
|
|
37
|
+
npx ai-engineering-loop verification summarize --run <run-id>
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
The recorder runs commands without a shell by default, limits and redacts captured output, binds it to the gated Maker diff, and preserves failed results as failed evidence. Use `--shell -- "command"` only when shell syntax is required. A scaffold is only a starting point, never gate evidence:
|
|
41
|
+
|
|
42
|
+
```bash
|
|
43
|
+
npx ai-engineering-loop scaffold delivery --run <run-id>
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
Copy or rename the generated `.template.json` to the gate's real artifact filename (for example, `delivery.json`), fill it with observed evidence, and remove `_templateOnly`. Then validate the real file:
|
|
47
|
+
|
|
48
|
+
```bash
|
|
49
|
+
npx ai-engineering-loop validate .ai-engineering-loop/runs/<run-id>/delivery.json
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
`validate` reports artifact-content problems without advancing the workflow; its gate remains responsible for freshness, readiness, and human approval.
|
|
53
|
+
|
|
54
|
+
For a frozen Goal, generate the claimed-vs-reality table with blank evidence fields and fill them only from actual changes and verification output:
|
|
55
|
+
|
|
56
|
+
```bash
|
|
57
|
+
npx ai-engineering-loop context claimed-vs-reality --run <run-id>
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
In a recipe-bound run, `gate delivery` may report that a prerequisite node is ready but not approved. Inspect `node status --run <run-id>`, then explicitly approve that named approval node; never skip the human checkpoint.
|
|
61
|
+
|
|
31
62
|
## Context boundary
|
|
32
63
|
|
|
33
64
|
Create model-visible context through:
|
|
@@ -38,7 +69,7 @@ npx ai-engineering-loop context devil-advocate <files...>
|
|
|
38
69
|
npx ai-engineering-loop context judge <files...>
|
|
39
70
|
```
|
|
40
71
|
|
|
41
|
-
The command rejects sensitive paths and repository escapes, redacts recognized credentials, caps files and estimated tokens, and marks truncation. Never replace a bounded pack with the parent transcript.
|
|
72
|
+
The command rejects sensitive paths and repository escapes, redacts recognized credentials, caps files and estimated tokens, and marks truncation. Budget errors report the limit, estimated requested size, and per-file contribution (paths and counts only). Never replace a bounded pack with the parent transcript.
|
|
42
73
|
|
|
43
74
|
Before Judge, run:
|
|
44
75
|
|
package/core/runtime-safety.md
CHANGED
|
@@ -69,11 +69,12 @@ npx ai-engineering-loop context related lib/studio-server.js
|
|
|
69
69
|
npx ai-engineering-loop context fast-path --json
|
|
70
70
|
npx ai-engineering-loop context diff-hunks --run <run-id>
|
|
71
71
|
npx ai-engineering-loop context diff-hunks --run <run-id> --include-ignored
|
|
72
|
+
npx ai-engineering-loop verification record --run <run-id> -- npm test
|
|
72
73
|
npx ai-engineering-loop verification summarize --run <run-id>
|
|
73
74
|
npx ai-engineering-loop context compact --run <run-id>
|
|
74
75
|
```
|
|
75
76
|
|
|
76
|
-
The index stores file summaries and hashes, never file bodies, and reuses per-hash summaries when files are unchanged. `query` and `related` return metadata only so agents can choose the right files before reading source. `fast-path` keeps low-risk small diffs on lean review and names the reason when a task must use the standard path. The diff-hunk pack stores bounded changed hunks, file summaries, token estimates, unresolved blocking findings from the previous ledger, and a count of paths skipped by context policy. Smart context skips CSS, generated bundles, lockfiles, build output, and `.aelcontextignore` matches by default; pass `--include-ignored` only when those files are the task.
|
|
77
|
+
The index stores file summaries and hashes, never file bodies, and reuses per-hash summaries when files are unchanged. `query` and `related` return metadata only so agents can choose the right files before reading source. `fast-path` keeps low-risk small diffs on lean review and names the reason when a task must use the standard path. The diff-hunk pack stores bounded changed hunks, file summaries, token estimates, unresolved blocking findings from the previous ledger, and a count of paths skipped by context policy. Smart context skips CSS, generated bundles, lockfiles, build output, and `.aelcontextignore` matches by default; pass `--include-ignored` only when those files are the task. `verification record` executes the command without a shell by default, stores bounded output after secret and local-path redaction, binds it to the gated Maker diff, and records failures instead of hiding them. `verification summarize` extracts available test counts and short failure excerpts without forwarding raw logs. Context budget errors include per-file estimates (paths and counts only). `context compact` writes `run-summary.md`, `run-summary.json`, and `open-findings.json` so iteration 2+ reviewers and Judge start from compact artifacts and request full source or logs only when the hunk or summary cannot prove or disprove a finding.
|
|
77
78
|
|
|
78
79
|
## Package update workflow
|
|
79
80
|
|
|
@@ -70,6 +70,12 @@ The Verification Gate immediately halts and returns to the Maker if:
|
|
|
70
70
|
|
|
71
71
|
Write `.ai-engineering-loop/tasks/claimed-vs-reality.md` after verification logs exist:
|
|
72
72
|
|
|
73
|
+
```bash
|
|
74
|
+
npx ai-engineering-loop context claimed-vs-reality --run <run-id>
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
This scaffolds one row per frozen acceptance criterion but leaves Claimed and Reality empty. Fill them from the implemented change and actual command output; the scaffold is not verification evidence.
|
|
78
|
+
|
|
73
79
|
```markdown
|
|
74
80
|
# Claimed vs Reality
|
|
75
81
|
|