ai-engineering-loop 1.0.3 → 1.0.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.grok/agents/devil-advocate.md +67 -0
- package/.grok/agents/judge.md +53 -0
- package/.grok/commands/ai-engineering-loop.md +11 -0
- package/.grok/skills/ai-engineering-loop/SKILL.md +79 -0
- package/README.md +247 -19
- package/README.npm.md +7 -0
- package/agents/devil-advocate.md +9 -5
- package/agents/judge.md +2 -0
- package/bin/ai-engineering-loop.js +28 -1
- package/core/orchestration-model.md +20 -0
- package/docs/grok-cli-feasibility.md +99 -0
- package/lib/orchestration.js +182 -2
- package/package.json +22 -2
- package/tests/grok-runtime.test.js +201 -0
- package/README.full.md +0 -251
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: devil-advocate
|
|
3
|
+
description: >
|
|
4
|
+
Independent adversarial reviewer for the AI Engineering Loop. Spawn after
|
|
5
|
+
deterministic verification passes. Read-only against application source.
|
|
6
|
+
Returns a dual-axis Finding Ledger JSON. Never edits code or git branches.
|
|
7
|
+
prompt_mode: full
|
|
8
|
+
permission_mode: plan
|
|
9
|
+
agents_md: true
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
You are the Devil's Advocate for the AI Engineering Loop. You are a read-only
|
|
13
|
+
adversarial reviewer. You never modify application source, never commit, and
|
|
14
|
+
never inherit Maker conversational history.
|
|
15
|
+
|
|
16
|
+
## Input barrier
|
|
17
|
+
|
|
18
|
+
Use only what the parent put in the spawn prompt plus these artifacts on disk:
|
|
19
|
+
|
|
20
|
+
- Goal Contract
|
|
21
|
+
- `.ai-engineering-loop/` (`architecture.md`, `conventions.md`, `verification.md`)
|
|
22
|
+
- The git diff (path in the prompt, or `git diff <base>...HEAD`)
|
|
23
|
+
- Deterministic verification logs (exit code, stdout, test counts)
|
|
24
|
+
|
|
25
|
+
Do not ask the parent for Maker rationale. Do not treat parent narration as evidence.
|
|
26
|
+
|
|
27
|
+
## What to find
|
|
28
|
+
|
|
29
|
+
Priority order: correctness, error handling, security, concurrency, tests that fail to prove an acceptance criterion. Skip style nits unless they hide a defect.
|
|
30
|
+
|
|
31
|
+
## Output contract
|
|
32
|
+
|
|
33
|
+
Return a Finding Ledger as a fenced JSON block and nothing else:
|
|
34
|
+
|
|
35
|
+
```json
|
|
36
|
+
{
|
|
37
|
+
"iteration": 1,
|
|
38
|
+
"executionMode": "TRUE_INDEPENDENT_AGENT",
|
|
39
|
+
"findings": [
|
|
40
|
+
{
|
|
41
|
+
"id": "DA-01",
|
|
42
|
+
"topic": "correctness",
|
|
43
|
+
"validity": "VALID",
|
|
44
|
+
"severity": "BLOCKER",
|
|
45
|
+
"disposition": "STRONG",
|
|
46
|
+
"location": "path/to/file.ext#L12-L20",
|
|
47
|
+
"acceptanceCriteria": "AC-1",
|
|
48
|
+
"failureScenario": "Concrete failing case",
|
|
49
|
+
"reproduction": "Steps to reproduce",
|
|
50
|
+
"evidence": "What you read in the diff or source",
|
|
51
|
+
"concreteAlternativeDiff": "```diff\n- broken\n+ fixed\n```"
|
|
52
|
+
}
|
|
53
|
+
]
|
|
54
|
+
}
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
Rules:
|
|
58
|
+
|
|
59
|
+
- `validity` is VALID or INVALID. You still emit INVALID only if you opened a claim and then disproved it; otherwise omit it.
|
|
60
|
+
- `severity` is BLOCKER, HIGH, MEDIUM, or LOW.
|
|
61
|
+
- `disposition` is STRONG, ACCEPTABLE, or WEAK.
|
|
62
|
+
- Every VALID BLOCKER or HIGH finding must include `concreteAlternativeDiff`.
|
|
63
|
+
- Empty `findings` is allowed when the diff is clean against the Goal Contract.
|
|
64
|
+
|
|
65
|
+
## Tools
|
|
66
|
+
|
|
67
|
+
Use read/search and read-only shell (`git diff`, `git log`, `git show`). Do not write files. Do not spawn subagents (Grok depth limit is 1).
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: judge
|
|
3
|
+
description: >
|
|
4
|
+
Impartial magistrate for the AI Engineering Loop. Spawn after Devil's Advocate
|
|
5
|
+
returns a Finding Ledger. Issues PASS, ITERATE, or ESCALATE from Validity +
|
|
6
|
+
Severity. Does not edit application source.
|
|
7
|
+
prompt_mode: full
|
|
8
|
+
permission_mode: plan
|
|
9
|
+
agents_md: true
|
|
10
|
+
---
|
|
11
|
+
|
|
12
|
+
You are the Judge for the AI Engineering Loop. You do not write application code.
|
|
13
|
+
You evaluate evidence and issue one verdict: PASS, ITERATE, or ESCALATE.
|
|
14
|
+
|
|
15
|
+
## Inputs
|
|
16
|
+
|
|
17
|
+
Use only:
|
|
18
|
+
|
|
19
|
+
- Goal Contract (acceptance criteria, constraints, out of scope)
|
|
20
|
+
- Deterministic verification evidence (command, exit code 0, stdout, test counts)
|
|
21
|
+
- Devil's Advocate Finding Ledger
|
|
22
|
+
- The git diff if you need to fact-check a finding
|
|
23
|
+
|
|
24
|
+
Ignore Maker optimism and reviewer tone. Disposition never overrides Validity + Severity.
|
|
25
|
+
|
|
26
|
+
## Decision matrix
|
|
27
|
+
|
|
28
|
+
- Verification evidence missing, vague, or non-zero exit → ITERATE
|
|
29
|
+
- Any VALID BLOCKER or HIGH still open → ITERATE (ESCALATE if iteration >= MAX_ITERATIONS, default 3)
|
|
30
|
+
- INVALID findings → DISMISS, cannot block delivery
|
|
31
|
+
- VALID MEDIUM or LOW → ACCEPT as tradeoff; may still PASS
|
|
32
|
+
- All ACs proven, verification green, zero open blockers → PASS
|
|
33
|
+
|
|
34
|
+
## Output contract
|
|
35
|
+
|
|
36
|
+
Return a Judge verdict as a fenced JSON block:
|
|
37
|
+
|
|
38
|
+
```json
|
|
39
|
+
{
|
|
40
|
+
"verdict": "PASS",
|
|
41
|
+
"reason": "All acceptance criteria verified; 0 open blocking findings",
|
|
42
|
+
"action": "Proceed to context impact assessment and delivery adapter",
|
|
43
|
+
"blockingFindings": [],
|
|
44
|
+
"acceptableTradeoffs": [],
|
|
45
|
+
"dismissedFindings": []
|
|
46
|
+
}
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
`verdict` must be exactly PASS, ITERATE, or ESCALATE.
|
|
50
|
+
|
|
51
|
+
## Tools
|
|
52
|
+
|
|
53
|
+
Read artifacts and run read-only git. Do not edit source. Do not spawn subagents.
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: ai-engineering-loop
|
|
3
|
+
description: Run the AI Engineering Loop (init, status, refresh, or full Maker → verify → Devil's Advocate subagent → Judge subagent).
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Follow `.grok/skills/ai-engineering-loop/SKILL.md`.
|
|
7
|
+
|
|
8
|
+
Arguments:
|
|
9
|
+
|
|
10
|
+
- `init` / `status` / `refresh` → run `npx ai-engineering-loop <arg>`
|
|
11
|
+
- anything else → full 8-stage loop for that task, spawning `devil-advocate` then `judge` via `spawn_subagent`
|
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: ai-engineering-loop
|
|
3
|
+
description: >
|
|
4
|
+
Autonomous AI Engineering Operating System. On Grok CLI, run the 8-stage loop
|
|
5
|
+
with native spawn_subagent for Devil's Advocate and Judge (TRUE_INDEPENDENT_AGENT).
|
|
6
|
+
Also handles init/status/refresh of .ai-engineering-loop/ living context.
|
|
7
|
+
Triggers: /ai-engineering-loop, "run the engineering loop", "devil's advocate review".
|
|
8
|
+
---
|
|
9
|
+
|
|
10
|
+
# AI Engineering Loop — Grok CLI Runtime
|
|
11
|
+
|
|
12
|
+
Canonical specs live in this repository: `core/`, `agents/`, `policies/`.
|
|
13
|
+
This skill is the Grok host adapter. Do not paraphrase the specs; read them.
|
|
14
|
+
|
|
15
|
+
## Host detection (do this first)
|
|
16
|
+
|
|
17
|
+
1. If the `spawn_subagent` tool is in your tool list and `GROK_SUBAGENTS` is not `0`, Grok native review is **INVOCATION_AVAILABLE**.
|
|
18
|
+
2. Independent execution is **PROVEN** only after a child returns a model response with its own `subagent_id` and you did **not** pass `resume_from`.
|
|
19
|
+
3. If spawn is missing or `GROK_SUBAGENTS=0`, fall back to `CONTEXT_ISOLATION_ONLY` (artifact barrier in this session). Disclose that honestly. Never call it a subagent review.
|
|
20
|
+
|
|
21
|
+
Canonical mode ids: `TRUE_INDEPENDENT_AGENT`, `ISOLATED_AGENT_INSTANCE`, `FRESH_PROCESS_AGENT`, `CONTEXT_ISOLATION_ONLY`, `UNAVAILABLE`.
|
|
22
|
+
Skill aliases: `NATIVE_SUBAGENT` → `TRUE_INDEPENDENT_AGENT`; `ARTIFACT_ISOLATED_REVIEW` → `CONTEXT_ISOLATION_ONLY`.
|
|
23
|
+
|
|
24
|
+
## Grok process topology
|
|
25
|
+
|
|
26
|
+
Parent session is the orchestrator (and usually the Maker). Spawn **siblings**, never nested children (Grok depth limit is 1).
|
|
27
|
+
|
|
28
|
+
```
|
|
29
|
+
Parent (Maker + orchestrator)
|
|
30
|
+
├─ spawn devil-advocate capability_mode=execute isolation=none resume_from=omit
|
|
31
|
+
└─ spawn judge capability_mode=execute isolation=none resume_from=omit
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
Forbidden `subagent_type` values for DA/Judge: `caveman:cavecrew-reviewer`, `caveman:cavecrew-builder`, `caveman:cavecrew-investigator`, `explore`, `plan`. Cavecrew-reviewer uses a different finding schema and cannot feed the Judge.
|
|
35
|
+
|
|
36
|
+
Optional fallback if `devil-advocate` / `judge` types are not registered: `subagent_type: "general-purpose"` with the matching agent spec prepended to the prompt and `description` still prefixed `[devil-advocate]` or `[judge]`.
|
|
37
|
+
|
|
38
|
+
## Commands
|
|
39
|
+
|
|
40
|
+
### `/ai-engineering-loop init|status|refresh`
|
|
41
|
+
|
|
42
|
+
Run `npx ai-engineering-loop <command>` in the target repo. Do not commit unless the user asks.
|
|
43
|
+
|
|
44
|
+
### `/ai-engineering-loop [task]`
|
|
45
|
+
|
|
46
|
+
1. Stage 0: `npx ai-engineering-loop status` (init/refresh if missing or stale).
|
|
47
|
+
2. Stage 1: write a Goal Contract (`core/goal-contract.md`).
|
|
48
|
+
3. Stages 2–4: Maker work in the **parent**. Surgical diff + tests. Parent may be the Maker; do not spawn Maker as a child if you still need to spawn DA/Judge afterward from the same parent.
|
|
49
|
+
4. Stage 5: run verification from `.ai-engineering-loop/verification.md`. Keep command, exit code, stdout, test counts. Vague "seems green" is invalid.
|
|
50
|
+
5. Write artifacts to disk so children do not need parent chat:
|
|
51
|
+
- Goal Contract path
|
|
52
|
+
- `git diff <base>...HEAD` (file)
|
|
53
|
+
- verification log (file)
|
|
54
|
+
6. Stage 6: `spawn_subagent` Devil's Advocate. `background: false`. `capability_mode: "execute"`. Do **not** pass `resume_from`. Prompt contains only artifact paths + `agents/devil-advocate.md` output contract.
|
|
55
|
+
7. Stage 7: `spawn_subagent` Judge with the ledger the DA returned plus the same artifacts. Prompt follows `agents/judge.md`.
|
|
56
|
+
8. If Judge says `ITERATE` and iteration < 3, Maker fixes in the parent, re-verify, spawn a **fresh** DA (new spawn, no resume).
|
|
57
|
+
9. Stage 8: delivery adapter from `.ai-engineering-loop/adapter.md`.
|
|
58
|
+
|
|
59
|
+
After a proven Grok DA spawn, the report header must be:
|
|
60
|
+
|
|
61
|
+
```
|
|
62
|
+
Execution Mode: TRUE_INDEPENDENT_AGENT
|
|
63
|
+
Independent LLM Execution: PROVEN
|
|
64
|
+
Native Subagent Invocation: AVAILABLE
|
|
65
|
+
Review Method: True Independent Agent
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
When spawn is unavailable:
|
|
69
|
+
|
|
70
|
+
```
|
|
71
|
+
Execution Mode: CONTEXT_ISOLATION_ONLY
|
|
72
|
+
Independent LLM Execution: NOT PROVEN
|
|
73
|
+
Native Subagent Invocation: UNAVAILABLE
|
|
74
|
+
Review Method: Clean-Slate Artifact Isolation Barrier
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
## Headless fallback
|
|
78
|
+
|
|
79
|
+
If this session cannot spawn but `grok -p` is authenticated, that is `FRESH_PROCESS_AGENT` only after the child process prints a model response. Prefer `spawn_subagent` when both exist.
|
package/README.md
CHANGED
|
@@ -1,17 +1,152 @@
|
|
|
1
1
|
# AI Engineering Loop
|
|
2
2
|
|
|
3
|
+
<div align="center">
|
|
4
|
+
|
|
3
5
|
[](https://www.npmjs.com/package/ai-engineering-loop)
|
|
4
6
|
[](https://opensource.org/licenses/MIT)
|
|
7
|
+
[](https://github.com/egagofur/ai-engineering-loop/pulls)
|
|
8
|
+
[](https://github.com/egagofur/ai-engineering-loop)
|
|
9
|
+
[](https://github.com/egagofur/ai-engineering-loop/releases)
|
|
10
|
+
|
|
11
|
+
**A Reusable, Framework-Agnostic AI Engineering Operating System for Autonomous Coding Agents**
|
|
5
12
|
|
|
6
|
-
|
|
13
|
+
*Featuring living project context, strict verification evidence contracts, 3-stage capability lifecycle registry, and dual-axis Judge evaluation.*
|
|
7
14
|
|
|
8
|
-
|
|
15
|
+
[Overview](#overview--philosophy) • [Runtime Capability Registry](#runtime-capability-registry--execution-modes) • [Verification Evidence](#verification-evidence-contract) • [CLI Commands](#cli-interface--commands) • [Grok CLI](#grok-cli-integration) • [Antigravity](#antigravity-agent-integration) • [Lifecycle](#lifecycle-stages) • [Architecture](#architecture--5-layer-configuration) • [Project Profiles](#project-profiles) • [Repository Structure](#repository-structure) • [Reference Examples](#reference-examples) • [Contributing](#contributing)
|
|
16
|
+
|
|
17
|
+
</div>
|
|
9
18
|
|
|
10
19
|
---
|
|
11
20
|
|
|
12
|
-
##
|
|
21
|
+
## Overview & Philosophy
|
|
22
|
+
|
|
23
|
+
The AI Engineering Loop enforces clean architectural separation across three core layers:
|
|
24
|
+
|
|
25
|
+
```mermaid
|
|
26
|
+
flowchart TD
|
|
27
|
+
Start([User Task in Workspace]) --> PreCheck{Pre-Task Drift Check: metadata.json}
|
|
28
|
+
|
|
29
|
+
PreCheck -->|Context Missing| AutoInit[Stage 0: Bootstrap .ai-engineering-loop/]
|
|
30
|
+
PreCheck -->|Drift Detected| Reconcile[Stage 0: Reconcile Drifted Context]
|
|
31
|
+
PreCheck -->|Context Fresh| GC[Stage 1: Goal Contract: Explicit Acceptance Criteria]
|
|
32
|
+
|
|
33
|
+
AutoInit --> GC
|
|
34
|
+
Reconcile --> GC
|
|
35
|
+
|
|
36
|
+
subgraph CoreEngine [AI ENGINEERING OPERATING SYSTEM]
|
|
37
|
+
GC --> RCA[Stage 2: Root Cause Analysis]
|
|
38
|
+
RCA --> Plan[Stage 3: Implementation Plan]
|
|
39
|
+
Plan --> MA[Stage 4: Maker Agent: Surgical Diff & Tests]
|
|
40
|
+
MA --> DV{Stage 5: Deterministic Verification<br>Evidence Contract: Exit Code 0 & Full Logs}
|
|
41
|
+
|
|
42
|
+
DV -->|Fail| MA
|
|
43
|
+
DV -->|Pass| DA[Stage 6: Devil's Advocate Review<br>Capability Registry & Artifact Barrier]
|
|
44
|
+
|
|
45
|
+
DA --> JD[Stage 7: Judge Agent: Impartial Magistrate<br>Validity + Severity Decision Matrix]
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
JD -->|VALID BLOCKER / HIGH: ITERATE| MA
|
|
49
|
+
JD -->|INVALID: Dismissed / VALID LOW: Tradeoff| CheckDoD{All ACs Verified?}
|
|
50
|
+
|
|
51
|
+
CheckDoD -->|Yes: PASS| ImpactEval{Post-Task Context Impact Assessment}
|
|
52
|
+
ImpactEval -->|NONE: Typo, UI tweak| Adapter[Stage 8: Delivery Adapter: GitLab / GitHub]
|
|
53
|
+
ImpactEval -->|TARGETED: Dep/route changed| PartialRefresh[Surgical Context Update] --> Adapter
|
|
54
|
+
ImpactEval -->|MAJOR: Framework migration| FullRefresh[Full Context Reconciliation] --> Adapter
|
|
55
|
+
|
|
56
|
+
Adapter --> TargetRepo[(Target Repository)]
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
---
|
|
60
|
+
|
|
61
|
+
## Runtime Capability Registry & Execution Modes
|
|
62
|
+
|
|
63
|
+
The system maintains a strict distinction between **Configuration Support**, **Invocation Availability**, and **Execution Proof**:
|
|
64
|
+
|
|
65
|
+
```text
|
|
66
|
+
┌───────────────────────────┐ ┌───────────────────────────┐ ┌───────────────────────────┐
|
|
67
|
+
│ CONFIGURATION_SUPPORTED │ ──> │ INVOCATION_AVAILABLE │ ──> │ EXECUTION_PROVEN │
|
|
68
|
+
│ (Config is recognized) │ │ (Callable tool is active) │ │ (Child LLM response seen) │
|
|
69
|
+
└───────────────────────────┘ └───────────────────────────┘ └───────────────────────────┘
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
### 5 Standard Execution Modes (Deterministic Priority):
|
|
13
73
|
|
|
14
|
-
|
|
74
|
+
| Priority | Mode Name | Requires Independent LLM Execution? | Condition for Selection |
|
|
75
|
+
|:---:|---|:---:|---|
|
|
76
|
+
| **1** | **`TRUE_INDEPENDENT_AGENT`** | **YES** | Child session exists **AND** actual model response is captured **AND** context is independent. |
|
|
77
|
+
| **2** | **`ISOLATED_AGENT_INSTANCE`** | **YES** | Programmatic SDK agent instance with verified independent model execution. |
|
|
78
|
+
| **3** | **`FRESH_PROCESS_AGENT`** | **YES** | Separate OS process successfully executes an LLM agent with fresh context. |
|
|
79
|
+
| **4** | **`CONTEXT_ISOLATION_ONLY`** | **NO** | Clean-Slate Artifact Isolation Barrier in same session (100% prompt history excluded on disk). |
|
|
80
|
+
| **5** | **`UNAVAILABLE`** | **NO** | No review execution mechanism is available. |
|
|
81
|
+
|
|
82
|
+
### Truthful Reporting Disclosure:
|
|
83
|
+
When `CONTEXT_ISOLATION_ONLY` is selected, the report strictly produces:
|
|
84
|
+
```text
|
|
85
|
+
Execution Mode: CONTEXT_ISOLATION_ONLY
|
|
86
|
+
Independent LLM Execution: NOT PROVEN
|
|
87
|
+
Native Subagent Invocation: UNAVAILABLE
|
|
88
|
+
Review Method: Clean-Slate Artifact Isolation Barrier
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
---
|
|
92
|
+
|
|
93
|
+
## Verification Evidence Contract
|
|
94
|
+
|
|
95
|
+
A verification `PASS` is strictly invalid without concrete execution evidence. The system categorically rejects vague statements such as *"command was launched"* or *"test appears to have passed"*.
|
|
96
|
+
|
|
97
|
+
### Mandatory Execution Proof:
|
|
98
|
+
- **`command`**: Exact CLI string executed.
|
|
99
|
+
- **`executionIdentity`**: PID, execution hash, or system execution identifier.
|
|
100
|
+
- **`startTime` & `endTime`**: Documented execution duration.
|
|
101
|
+
- **`exitCode`**: Must be `0`.
|
|
102
|
+
- **`stdout` & `stderr`**: Raw machine logs captured.
|
|
103
|
+
- **`timeoutStatus`**: Must be `"COMPLETED"`.
|
|
104
|
+
- **`testCounts`**: Explicit counts of passed, failed, and skipped tests.
|
|
105
|
+
- **`assertionEvidence`**: Specific assertion proof matching the active Goal Contract's Acceptance Criteria.
|
|
106
|
+
|
|
107
|
+
---
|
|
108
|
+
|
|
109
|
+
## Dual-Axis Finding Model & Judge Decision Matrix
|
|
110
|
+
|
|
111
|
+
The Devil's Advocate categorizes findings along separate **Validity**, **Severity**, and **Disposition** axes:
|
|
112
|
+
|
|
113
|
+
```json
|
|
114
|
+
{
|
|
115
|
+
"id": "DA-01",
|
|
116
|
+
"topic": "correctness",
|
|
117
|
+
"validity": "VALID",
|
|
118
|
+
"severity": "BLOCKER",
|
|
119
|
+
"disposition": "STRONG",
|
|
120
|
+
"location": "src/services/payment.ts#L42-L58",
|
|
121
|
+
"acceptanceCriteria": "AC-2",
|
|
122
|
+
"failureScenario": "Under concurrent traffic, duplicate rows are inserted before the lock is acquired.",
|
|
123
|
+
"evidence": "Missing SELECT FOR UPDATE in findByPaymentKey query.",
|
|
124
|
+
"concreteAlternativeDiff": "```diff\n- const tx = await findByKey(key);\n+ const tx = await findByKeyWithLock(key, { mode: 'FOR UPDATE' });\n```"
|
|
125
|
+
}
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
### Judge Decision Matrix:
|
|
129
|
+
- **`VALID + BLOCKER / HIGH`** $\rightarrow$ **`ITERATE`** (Maker must apply concrete fix diff and add regression tests).
|
|
130
|
+
- **`VALID + MEDIUM / LOW`** $\rightarrow$ **`ACCEPT / TRADEOFF`** (Merged; documented as acceptable tradeoff in MR notes).
|
|
131
|
+
- **`INVALID`** $\rightarrow$ **`DISMISS`** (Reviewer hallucination disproven by code; cannot block delivery; signature recorded).
|
|
132
|
+
|
|
133
|
+
*Reviewer disposition (`STRONG`, `ACCEPTABLE`, `WEAK`) never overrides factual evidence.*
|
|
134
|
+
|
|
135
|
+
---
|
|
136
|
+
|
|
137
|
+
## Living Project Context
|
|
138
|
+
|
|
139
|
+
The `.ai-engineering-loop/` directory is **Living Context**, not a static wiki generated once.
|
|
140
|
+
|
|
141
|
+
1. **Post-Task Context Impact Assessment**: Evaluates completed tasks (`NONE`, `TARGETED`, `MAJOR`) to keep project context fresh without expensive whole-repo re-analysis.
|
|
142
|
+
2. **Context Baseline (`metadata.json`)**: Tracks `repositoryRevision` (git commit SHA) and `manifestChecksums` for instant Level 0 (0ms) drift verification.
|
|
143
|
+
3. **Strict Context Isolation**: Decouples living project context from ephemeral task logs and loop execution states.
|
|
144
|
+
|
|
145
|
+
---
|
|
146
|
+
|
|
147
|
+
## CLI Interface & Commands
|
|
148
|
+
|
|
149
|
+
The CLI package is published on NPM as [`ai-engineering-loop`](https://www.npmjs.com/package/ai-engineering-loop) and operates against the current working directory.
|
|
15
150
|
|
|
16
151
|
```bash
|
|
17
152
|
# Bootstrap .ai-engineering-loop/ context from repository discovery
|
|
@@ -29,28 +164,121 @@ npx ai-engineering-loop run
|
|
|
29
164
|
|
|
30
165
|
---
|
|
31
166
|
|
|
32
|
-
##
|
|
167
|
+
## Grok CLI Integration
|
|
33
168
|
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
169
|
+
Grok CLI is a first-class host. `spawn_subagent` is a real independent child session (own context, no parent transcript unless `resume_from` is set). After a child id and model response are captured, the registry selects **`TRUE_INDEPENDENT_AGENT`**.
|
|
170
|
+
|
|
171
|
+
| Loop role | Grok `subagent_type` | Spawn rules |
|
|
172
|
+
|---|---|---|
|
|
173
|
+
| Orchestrator / Maker | parent session | Parent stays the orchestrator (Grok nesting depth is 1) |
|
|
174
|
+
| Devil's Advocate | `devil-advocate` (fallback `general-purpose`) | `capability_mode: execute`, omit `resume_from` |
|
|
175
|
+
| Judge | `judge` (fallback `general-purpose`) | Sibling of DA, never nested under DA |
|
|
176
|
+
|
|
177
|
+
Do **not** use `caveman:cavecrew-reviewer` as Devil's Advocate or Judge — its output schema is not the Finding Ledger.
|
|
178
|
+
|
|
179
|
+
Repo-local Grok files:
|
|
180
|
+
|
|
181
|
+
- `.grok/agents/devil-advocate.md` / `.grok/agents/judge.md`
|
|
182
|
+
- `.grok/skills/ai-engineering-loop/SKILL.md`
|
|
183
|
+
- `.grok/commands/ai-engineering-loop.md` → `/ai-engineering-loop`
|
|
184
|
+
|
|
185
|
+
Fallback: `GROK_SUBAGENTS=0` or `--disallowed-tools Agent` → `CONTEXT_ISOLATION_ONLY`, disclosed as such. Optional process fallback: `grok -p` → `FRESH_PROCESS_AGENT` only after a model response is captured.
|
|
186
|
+
|
|
187
|
+
See [docs/grok-cli-feasibility.md](docs/grok-cli-feasibility.md).
|
|
43
188
|
|
|
44
189
|
---
|
|
45
190
|
|
|
46
|
-
##
|
|
191
|
+
## Antigravity Agent Integration
|
|
47
192
|
|
|
48
|
-
|
|
193
|
+
When working inside the Antigravity IDE or compatible agentic platforms, you can invoke the loop via slash commands:
|
|
49
194
|
|
|
50
|
-
|
|
195
|
+
- **`/ai-engineering-loop init`**: Initialize project context only (non-destructive bootstrap).
|
|
196
|
+
- **`/ai-engineering-loop status`**: Check repository context health & baseline freshness.
|
|
197
|
+
- **`/ai-engineering-loop refresh`**: Reconcile drifted context files non-destructively.
|
|
198
|
+
- **`/ai-engineering-loop [task description]`**: Execute the full 8-stage engineering lifecycle with pre-task drift gate and post-task impact assessment.
|
|
199
|
+
|
|
200
|
+
On Grok CLI the same slash command is provided by `.grok/commands/ai-engineering-loop.md` and runs Devil's Advocate / Judge as native subagents.
|
|
201
|
+
|
|
202
|
+
---
|
|
203
|
+
|
|
204
|
+
## Repository Structure
|
|
205
|
+
|
|
206
|
+
```text
|
|
207
|
+
ai-engineering-loop/
|
|
208
|
+
│
|
|
209
|
+
├── README.md # Operating system overview & architecture
|
|
210
|
+
├── LICENSE # MIT Open Source License
|
|
211
|
+
├── package.json # CLI package manifest
|
|
212
|
+
│
|
|
213
|
+
├── bin/ # CLI execution entrypoints
|
|
214
|
+
│ └── ai-engineering-loop.js # npx executable CLI (init, status, refresh, run)
|
|
215
|
+
│
|
|
216
|
+
├── lib/ # Core orchestration & decision engine
|
|
217
|
+
│ └── orchestration.js # 3-stage capability registry, barrier builder, Judge engine
|
|
218
|
+
│
|
|
219
|
+
├── tests/ # Deterministic test suites
|
|
220
|
+
│ ├── capability-selection.test.js # Unit tests for capability lifecycle & truthful selection
|
|
221
|
+
│ ├── orchestration.test.js # Tests for isolation, Finding schema, Judge matrix
|
|
222
|
+
│ └── grok-runtime.test.js # Grok spawn_subagent mapping, aliases, forbidden types
|
|
223
|
+
│
|
|
224
|
+
├── .grok/ # Grok CLI host adapter
|
|
225
|
+
│ ├── agents/devil-advocate.md # Native DA subagent type
|
|
226
|
+
│ ├── agents/judge.md # Native Judge subagent type
|
|
227
|
+
│ ├── skills/ai-engineering-loop/ # Grok skill (spawn protocol)
|
|
228
|
+
│ └── commands/ai-engineering-loop.md # /ai-engineering-loop slash command
|
|
229
|
+
│
|
|
230
|
+
├── core/ # Generic engineering loop specifications
|
|
231
|
+
│ ├── orchestration-model.md # 3-stage capability lifecycle & execution priority
|
|
232
|
+
│ ├── project-initialization.md # Auto-discovery & initialization lifecycle
|
|
233
|
+
│ ├── context-refresh-policy.md # Progressive drift hierarchy & living baseline
|
|
234
|
+
│ ├── context-impact-assessment.md # Post-task impact assessment (NONE, TARGETED, MAJOR)
|
|
235
|
+
│ ├── goal-contract.md # Task contract schema & acceptance criteria
|
|
236
|
+
│ ├── verification-loop.md # Dual-layer verification & Evidence Contract
|
|
237
|
+
│ ├── definition-of-done.md # 5 pillars of Done & rejection triggers
|
|
238
|
+
│ ├── iteration-policy.md # Bounded autonomous loop (MAX_ITERATIONS = 3)
|
|
239
|
+
│ ├── escalation-policy.md # Deterministic human escalation triggers
|
|
240
|
+
│ ├── judge-policy.md # Evaluation rules, triage audit, & verdicts
|
|
241
|
+
│ ├── configuration-precedence.md # 5-layer precedence & conflict resolution
|
|
242
|
+
│ └── repo-config-schema.md # Schema for target repo .ai-engineering-loop/
|
|
243
|
+
│
|
|
244
|
+
├── profiles/ # Project archetype profiles
|
|
245
|
+
│ ├── README.md # Profile catalog & auto-detection rules
|
|
246
|
+
│ ├── web-app.md # Frontend web applications
|
|
247
|
+
│ ├── backend-api.md # Backend APIs & microservices
|
|
248
|
+
│ ├── mobile-app.md # Native & cross-platform mobile apps
|
|
249
|
+
│ ├── library.md # Reusable SDKs & shared packages
|
|
250
|
+
│ └── monorepo.md # Multi-package monorepo workspaces
|
|
251
|
+
│
|
|
252
|
+
├── agents/ # Triad agent role specifications
|
|
253
|
+
│ ├── maker.md # Maker agent: surgical diffs & unit tests
|
|
254
|
+
│ ├── devil-advocate.md # Adversarial reviewer: dual-axis finding ledger & diffs
|
|
255
|
+
│ └── judge.md # Judge agent: impartial magistrate on Validity + Severity
|
|
256
|
+
│
|
|
257
|
+
├── policies/ # Operational schemas & algorithms
|
|
258
|
+
│ ├── discovery-safety-policy.md # Secret protection & non-destructive discovery rules
|
|
259
|
+
│ ├── finding-policy.md # Dual-axis finding schema & severity matrix
|
|
260
|
+
│ ├── evidence-policy.md # 5-level evidence hierarchy & Verification Evidence Contract
|
|
261
|
+
│ └── no-progress-policy.md # Finding signature hashing & stagnation detection
|
|
262
|
+
│
|
|
263
|
+
├── adapters/ # Pluggable delivery pipelines
|
|
264
|
+
│ └── dot/ # DOT Indonesia delivery adapter
|
|
265
|
+
│ ├── README.md # DOT adapter overview
|
|
266
|
+
│ ├── gitlab.md # glab CLI, issue cards, & MR generation
|
|
267
|
+
│ ├── multi-branch.md # main / staging / develop cherry-pick propagation
|
|
268
|
+
│ ├── coreview.md # @coreview-bot external review triage (Valid vs Halu)
|
|
269
|
+
│ └── mattermost.md # Channel mapping & MCP dispatch (from: "AI Agent")
|
|
270
|
+
│
|
|
271
|
+
└── templates/ # Starter templates for target repositories
|
|
272
|
+
└── repo-config/ # Ready-to-copy .ai-engineering-loop/ files
|
|
273
|
+
├── config.md # Project identity & profile binding
|
|
274
|
+
├── architecture.md # Layers & boundary invariants
|
|
275
|
+
├── conventions.md # Code standards & forbidden patterns
|
|
276
|
+
├── verification.md # CLI test/lint/build commands
|
|
277
|
+
└── adapter.md # Configured release pipeline
|
|
278
|
+
```
|
|
51
279
|
|
|
52
280
|
---
|
|
53
281
|
|
|
54
|
-
##
|
|
282
|
+
## License
|
|
55
283
|
|
|
56
|
-
MIT
|
|
284
|
+
This project is licensed under the **MIT License** — see the [LICENSE](LICENSE) file for details.
|
package/README.npm.md
CHANGED
|
@@ -36,6 +36,13 @@ npx ai-engineering-loop run
|
|
|
36
36
|
2. **Living Project Context**: Tracks repository revisions and manifest checksums in `metadata.json` for instant drift detection.
|
|
37
37
|
3. **Multi-Agent Triad**: Coordinates **Maker** (surgical diffs & tests), **Devil's Advocate** (independent adversarial review), and **Judge** (impartial evaluation & PASS certification).
|
|
38
38
|
4. **Context Impact Assessment**: Evaluates completed tasks (`NONE`, `TARGETED`, `MAJOR`) to keep project context fresh without expensive whole-repo re-analysis.
|
|
39
|
+
5. **Grok CLI host**: Native `spawn_subagent` for Devil's Advocate and Judge (`TRUE_INDEPENDENT_AGENT`). Disabled by `GROK_SUBAGENTS=0`.
|
|
40
|
+
|
|
41
|
+
---
|
|
42
|
+
|
|
43
|
+
## Grok CLI
|
|
44
|
+
|
|
45
|
+
Inside a Grok TUI session, `/ai-engineering-loop` uses repo-local `.grok/agents/` types. Devil's Advocate and Judge spawn as sibling children (`capability_mode: execute`, no `resume_from`). Do not use `caveman:cavecrew-reviewer` as the loop reviewer.
|
|
39
46
|
|
|
40
47
|
---
|
|
41
48
|
|
package/agents/devil-advocate.md
CHANGED
|
@@ -12,12 +12,16 @@ The **Devil's Advocate** is an independent adversarial reviewer. Its sole purpos
|
|
|
12
12
|
|
|
13
13
|
## 2. Review Execution Modes
|
|
14
14
|
|
|
15
|
-
The Devil's Advocate executes under one of 4 runtime modes depending on platform capabilities
|
|
15
|
+
The Devil's Advocate executes under one of 4 runtime modes depending on platform capabilities.
|
|
16
16
|
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
17
|
+
Canonical ids are listed first; skill aliases are in parentheses.
|
|
18
|
+
|
|
19
|
+
1. **`TRUE_INDEPENDENT_AGENT`** (`NATIVE_SUBAGENT`): Genuine independent child session.
|
|
20
|
+
- **Grok CLI**: `spawn_subagent` with `subagent_type: "devil-advocate"` (fallback `"general-purpose"`), `capability_mode: "execute"`, `isolation: "none"`, **omit** `resume_from`. Agent definition: `.grok/agents/devil-advocate.md`.
|
|
21
|
+
- Do **not** use `caveman:cavecrew-reviewer` (compressed review schema, not a Finding Ledger).
|
|
22
|
+
2. **`ISOLATED_AGENT_INSTANCE`** (`SDK_AGENT`): Programmatic SDK agent instance with isolated memory.
|
|
23
|
+
3. **`FRESH_PROCESS_AGENT`** (`HEADLESS_SUBPROCESS`): Fresh OS process such as `grok -p` after a model response is captured.
|
|
24
|
+
4. **`CONTEXT_ISOLATION_ONLY`** (`ARTIFACT_ISOLATED_REVIEW`): Clean-Slate Artifact Barrier in single-agent session (*strictly labeled: isolated review context, not independent agent execution*).
|
|
21
25
|
|
|
22
26
|
---
|
|
23
27
|
|
package/agents/judge.md
CHANGED
|
@@ -4,6 +4,8 @@
|
|
|
4
4
|
|
|
5
5
|
The **Judge Agent** is the final decision-maker of the AI Engineering Loop. It serves as an impartial magistrate that evaluates the complete evidence pipeline (Contract, Diff, Deterministic Verification Logs, and Devil's Advocate findings) to determine whether the iteration should **PASS**, **ITERATE**, or **ESCALATE**.
|
|
6
6
|
|
|
7
|
+
On **Grok CLI**, the parent orchestrator spawns the Judge with `spawn_subagent` (`subagent_type: "judge"`, fallback `"general-purpose"`), `capability_mode: "execute"`, and **no** `resume_from`. Agent definition: `.grok/agents/judge.md`. The Judge is a sibling of the Devil's Advocate, never its child (Grok nesting depth is 1).
|
|
8
|
+
|
|
7
9
|
```mermaid
|
|
8
10
|
flowchart LR
|
|
9
11
|
Inputs[Goal Contract + Diff + Test Logs + Review Findings] --> Judge[Judge Agent]
|
|
@@ -15,7 +15,7 @@ const path = require('path');
|
|
|
15
15
|
const crypto = require('crypto');
|
|
16
16
|
const { execSync } = require('child_process');
|
|
17
17
|
|
|
18
|
-
const VERSION = '1.0.
|
|
18
|
+
const VERSION = '1.0.5';
|
|
19
19
|
const CWD = process.cwd();
|
|
20
20
|
const CONTEXT_DIR = path.join(CWD, '.ai-engineering-loop');
|
|
21
21
|
|
|
@@ -553,6 +553,15 @@ function handleRefresh() {
|
|
|
553
553
|
handleStatus();
|
|
554
554
|
}
|
|
555
555
|
|
|
556
|
+
function detectGrokHost() {
|
|
557
|
+
try {
|
|
558
|
+
const { detectGrokRuntime } = require('../lib/orchestration.js');
|
|
559
|
+
return detectGrokRuntime(process.env, fs);
|
|
560
|
+
} catch (e) {
|
|
561
|
+
return null;
|
|
562
|
+
}
|
|
563
|
+
}
|
|
564
|
+
|
|
556
565
|
// Command: run
|
|
557
566
|
function handleRun() {
|
|
558
567
|
log.info('AI Engineering Loop — Task Execution Entrypoint (run)');
|
|
@@ -564,6 +573,8 @@ function handleRun() {
|
|
|
564
573
|
handleStatus();
|
|
565
574
|
}
|
|
566
575
|
|
|
576
|
+
const grok = detectGrokHost();
|
|
577
|
+
|
|
567
578
|
console.log('\n------------------------------------------------------------');
|
|
568
579
|
log.bold('AI Agent Ready:');
|
|
569
580
|
console.log('1. Formulate Goal Contract (core/goal-contract.md)');
|
|
@@ -574,6 +585,22 @@ function handleRun() {
|
|
|
574
585
|
console.log('6. Judge Agent evaluates DoD and issues PASS verdict');
|
|
575
586
|
console.log('7. Context Impact Assessment (NONE / TARGETED / MAJOR)');
|
|
576
587
|
console.log('8. Delivery Adapter creates MR/PR');
|
|
588
|
+
|
|
589
|
+
if (grok && grok.host === 'grok-cli') {
|
|
590
|
+
console.log('------------------------------------------------------------');
|
|
591
|
+
log.bold('Grok CLI host:');
|
|
592
|
+
console.log(`- Binary: ${grok.grokBin || 'detected'}`);
|
|
593
|
+
console.log(`- spawn_subagent: ${grok.invocationAvailable ? 'INVOCATION_AVAILABLE' : 'UNAVAILABLE'}`);
|
|
594
|
+
console.log(`- Execution proven: no (requires child subagent_id + model response)`);
|
|
595
|
+
if (grok.invocationAvailable) {
|
|
596
|
+
console.log('- Devil\'s Advocate: spawn_subagent type=devil-advocate capability_mode=execute (no resume_from)');
|
|
597
|
+
console.log('- Judge: spawn_subagent type=judge capability_mode=execute (sibling, not nested)');
|
|
598
|
+
console.log('- Forbidden types: caveman:cavecrew-reviewer, explore, plan');
|
|
599
|
+
} else {
|
|
600
|
+
console.log(`- Fallback: CONTEXT_ISOLATION_ONLY (${grok.reason})`);
|
|
601
|
+
}
|
|
602
|
+
console.log('- Skill: .grok/skills/ai-engineering-loop/SKILL.md');
|
|
603
|
+
}
|
|
577
604
|
console.log('------------------------------------------------------------\n');
|
|
578
605
|
}
|
|
579
606
|
|
|
@@ -69,6 +69,26 @@ The system categorically rejects the following as proof of independent LLM execu
|
|
|
69
69
|
|
|
70
70
|
---
|
|
71
71
|
|
|
72
|
+
## 4b. Grok CLI Empirical Discovery Record
|
|
73
|
+
|
|
74
|
+
| Investigated Surface | Tested Command / API | Classification | Status & Result |
|
|
75
|
+
|---|---|---|---|
|
|
76
|
+
| **Grok CLI binary** | `~/.grok/bin/grok` | `CONFIGURATION_SUPPORTED` | Installed (`grok 1.0.5+`). Not execution proof. |
|
|
77
|
+
| **Native subagent tool** | `spawn_subagent` | `INVOCATION_AVAILABLE` | Enabled by default. Disabled by `GROK_SUBAGENTS=0` or `--disallowed-tools Agent`. |
|
|
78
|
+
| **Child session** | `subagent_id` + child model response, `resume_from` omitted | `EXECUTION_PROVEN` → `TRUE_INDEPENDENT_AGENT` | Own context window; parent transcript is not inherited. |
|
|
79
|
+
| **Project agent types** | `.grok/agents/devil-advocate.md`, `.grok/agents/judge.md` | `CONFIGURATION_SUPPORTED` | Prefer these types; fall back to `general-purpose` with the spec prepended. |
|
|
80
|
+
| **Headless process** | `grok -p` | `FRESH_PROCESS_AGENT` | Only after the process prints a model response. |
|
|
81
|
+
| **Cavecrew reviewer** | `caveman:cavecrew-reviewer` | `GROK_COMPRESSED_REVIEW_PRESET` | **Rejected** as Devil's Advocate / Judge (wrong output schema). |
|
|
82
|
+
| **Maker resume** | `spawn_subagent(resume_from=maker)` | tainted history | **Rejected**. DA and Judge must spawn fresh. |
|
|
83
|
+
| **Artifact barrier** | `buildReviewContextBarrier()` | `CONTEXT_ISOLATION_ONLY` | Fallback when spawn is disabled. |
|
|
84
|
+
|
|
85
|
+
### Architectural Conclusion:
|
|
86
|
+
> *"Grok CLI spawn_subagent is a true independent child session. Select TRUE_INDEPENDENT_AGENT only after a child id and model response are captured without resume_from. Binary presence and agent markdown are never sufficient."*
|
|
87
|
+
|
|
88
|
+
See [docs/grok-cli-feasibility.md](../docs/grok-cli-feasibility.md).
|
|
89
|
+
|
|
90
|
+
---
|
|
91
|
+
|
|
72
92
|
## 5. Truthful Reporting Output
|
|
73
93
|
|
|
74
94
|
When `CONTEXT_ISOLATION_ONLY` is selected, the report generator strictly produces:
|