ai-engineering-loop 1.0.4 → 1.0.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/agents/devil-advocate.md +60 -0
- package/.claude/agents/judge.md +49 -0
- package/.claude/commands/ai-engineering-loop.md +10 -0
- package/.claude/settings.local.json +9 -0
- package/.claude/skills/ai-engineering-loop/SKILL.md +59 -0
- package/.grok/agents/devil-advocate.md +67 -0
- package/.grok/agents/judge.md +53 -0
- package/.grok/commands/ai-engineering-loop.md +11 -0
- package/.grok/skills/ai-engineering-loop/SKILL.md +79 -0
- package/README.md +279 -19
- package/README.npm.md +11 -0
- package/agents/devil-advocate.md +9 -5
- package/agents/judge.md +2 -0
- package/bin/ai-engineering-loop.js +40 -1
- package/core/orchestration-model.md +20 -0
- package/docs/claude-code-feasibility.md +45 -0
- package/docs/grok-cli-feasibility.md +99 -0
- package/lib/orchestration.js +182 -2
- package/package.json +24 -2
- package/tests/grok-runtime.test.js +201 -0
- package/tests/skill-host-compat.test.js +68 -0
- package/README.full.md +0 -251
package/README.md
CHANGED
|
@@ -1,17 +1,152 @@
|
|
|
1
1
|
# AI Engineering Loop
|
|
2
2
|
|
|
3
|
+
<div align="center">
|
|
4
|
+
|
|
3
5
|
[](https://www.npmjs.com/package/ai-engineering-loop)
|
|
4
6
|
[](https://opensource.org/licenses/MIT)
|
|
7
|
+
[](https://github.com/egagofur/ai-engineering-loop/pulls)
|
|
8
|
+
[](https://github.com/egagofur/ai-engineering-loop)
|
|
9
|
+
[](https://github.com/egagofur/ai-engineering-loop/releases)
|
|
10
|
+
|
|
11
|
+
**A Reusable, Framework-Agnostic AI Engineering Operating System for Autonomous Coding Agents**
|
|
12
|
+
|
|
13
|
+
*Featuring living project context, strict verification evidence contracts, 3-stage capability lifecycle registry, and dual-axis Judge evaluation.*
|
|
14
|
+
|
|
15
|
+
[Overview](#overview--philosophy) • [Runtime Capability Registry](#runtime-capability-registry--execution-modes) • [Verification Evidence](#verification-evidence-contract) • [CLI Commands](#cli-interface--commands) • [Grok CLI](#grok-cli-integration) • [Claude Code](#claude-code-integration) • [Antigravity](#antigravity-agent-integration) • [Lifecycle](#lifecycle-stages) • [Architecture](#architecture--5-layer-configuration) • [Project Profiles](#project-profiles) • [Repository Structure](#repository-structure) • [Reference Examples](#reference-examples) • [Contributing](#contributing)
|
|
16
|
+
|
|
17
|
+
</div>
|
|
18
|
+
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## Overview & Philosophy
|
|
5
22
|
|
|
6
|
-
|
|
23
|
+
The AI Engineering Loop enforces clean architectural separation across three core layers:
|
|
7
24
|
|
|
8
|
-
|
|
25
|
+
```mermaid
|
|
26
|
+
flowchart TD
|
|
27
|
+
Start([User Task in Workspace]) --> PreCheck{Pre-Task Drift Check: metadata.json}
|
|
28
|
+
|
|
29
|
+
PreCheck -->|Context Missing| AutoInit[Stage 0: Bootstrap .ai-engineering-loop/]
|
|
30
|
+
PreCheck -->|Drift Detected| Reconcile[Stage 0: Reconcile Drifted Context]
|
|
31
|
+
PreCheck -->|Context Fresh| GC[Stage 1: Goal Contract: Explicit Acceptance Criteria]
|
|
32
|
+
|
|
33
|
+
AutoInit --> GC
|
|
34
|
+
Reconcile --> GC
|
|
35
|
+
|
|
36
|
+
subgraph CoreEngine [AI ENGINEERING OPERATING SYSTEM]
|
|
37
|
+
GC --> RCA[Stage 2: Root Cause Analysis]
|
|
38
|
+
RCA --> Plan[Stage 3: Implementation Plan]
|
|
39
|
+
Plan --> MA[Stage 4: Maker Agent: Surgical Diff & Tests]
|
|
40
|
+
MA --> DV{Stage 5: Deterministic Verification<br>Evidence Contract: Exit Code 0 & Full Logs}
|
|
41
|
+
|
|
42
|
+
DV -->|Fail| MA
|
|
43
|
+
DV -->|Pass| DA[Stage 6: Devil's Advocate Review<br>Capability Registry & Artifact Barrier]
|
|
44
|
+
|
|
45
|
+
DA --> JD[Stage 7: Judge Agent: Impartial Magistrate<br>Validity + Severity Decision Matrix]
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
JD -->|VALID BLOCKER / HIGH: ITERATE| MA
|
|
49
|
+
JD -->|INVALID: Dismissed / VALID LOW: Tradeoff| CheckDoD{All ACs Verified?}
|
|
50
|
+
|
|
51
|
+
CheckDoD -->|Yes: PASS| ImpactEval{Post-Task Context Impact Assessment}
|
|
52
|
+
ImpactEval -->|NONE: Typo, UI tweak| Adapter[Stage 8: Delivery Adapter: GitLab / GitHub]
|
|
53
|
+
ImpactEval -->|TARGETED: Dep/route changed| PartialRefresh[Surgical Context Update] --> Adapter
|
|
54
|
+
ImpactEval -->|MAJOR: Framework migration| FullRefresh[Full Context Reconciliation] --> Adapter
|
|
55
|
+
|
|
56
|
+
Adapter --> TargetRepo[(Target Repository)]
|
|
57
|
+
```
|
|
9
58
|
|
|
10
59
|
---
|
|
11
60
|
|
|
12
|
-
##
|
|
61
|
+
## Runtime Capability Registry & Execution Modes
|
|
62
|
+
|
|
63
|
+
The system maintains a strict distinction between **Configuration Support**, **Invocation Availability**, and **Execution Proof**:
|
|
64
|
+
|
|
65
|
+
```text
|
|
66
|
+
┌───────────────────────────┐ ┌───────────────────────────┐ ┌───────────────────────────┐
|
|
67
|
+
│ CONFIGURATION_SUPPORTED │ ──> │ INVOCATION_AVAILABLE │ ──> │ EXECUTION_PROVEN │
|
|
68
|
+
│ (Config is recognized) │ │ (Callable tool is active) │ │ (Child LLM response seen) │
|
|
69
|
+
└───────────────────────────┘ └───────────────────────────┘ └───────────────────────────┘
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
### 5 Standard Execution Modes (Deterministic Priority):
|
|
73
|
+
|
|
74
|
+
| Priority | Mode Name | Requires Independent LLM Execution? | Condition for Selection |
|
|
75
|
+
|:---:|---|:---:|---|
|
|
76
|
+
| **1** | **`TRUE_INDEPENDENT_AGENT`** | **YES** | Child session exists **AND** actual model response is captured **AND** context is independent. |
|
|
77
|
+
| **2** | **`ISOLATED_AGENT_INSTANCE`** | **YES** | Programmatic SDK agent instance with verified independent model execution. |
|
|
78
|
+
| **3** | **`FRESH_PROCESS_AGENT`** | **YES** | Separate OS process successfully executes an LLM agent with fresh context. |
|
|
79
|
+
| **4** | **`CONTEXT_ISOLATION_ONLY`** | **NO** | Clean-Slate Artifact Isolation Barrier in same session (100% prompt history excluded on disk). |
|
|
80
|
+
| **5** | **`UNAVAILABLE`** | **NO** | No review execution mechanism is available. |
|
|
81
|
+
|
|
82
|
+
### Truthful Reporting Disclosure:
|
|
83
|
+
When `CONTEXT_ISOLATION_ONLY` is selected, the report strictly produces:
|
|
84
|
+
```text
|
|
85
|
+
Execution Mode: CONTEXT_ISOLATION_ONLY
|
|
86
|
+
Independent LLM Execution: NOT PROVEN
|
|
87
|
+
Native Subagent Invocation: UNAVAILABLE
|
|
88
|
+
Review Method: Clean-Slate Artifact Isolation Barrier
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
---
|
|
92
|
+
|
|
93
|
+
## Verification Evidence Contract
|
|
94
|
+
|
|
95
|
+
A verification `PASS` is strictly invalid without concrete execution evidence. The system categorically rejects vague statements such as *"command was launched"* or *"test appears to have passed"*.
|
|
96
|
+
|
|
97
|
+
### Mandatory Execution Proof:
|
|
98
|
+
- **`command`**: Exact CLI string executed.
|
|
99
|
+
- **`executionIdentity`**: PID, execution hash, or system execution identifier.
|
|
100
|
+
- **`startTime` & `endTime`**: Documented execution duration.
|
|
101
|
+
- **`exitCode`**: Must be `0`.
|
|
102
|
+
- **`stdout` & `stderr`**: Raw machine logs captured.
|
|
103
|
+
- **`timeoutStatus`**: Must be `"COMPLETED"`.
|
|
104
|
+
- **`testCounts`**: Explicit counts of passed, failed, and skipped tests.
|
|
105
|
+
- **`assertionEvidence`**: Specific assertion proof matching the active Goal Contract's Acceptance Criteria.
|
|
106
|
+
|
|
107
|
+
---
|
|
108
|
+
|
|
109
|
+
## Dual-Axis Finding Model & Judge Decision Matrix
|
|
110
|
+
|
|
111
|
+
The Devil's Advocate categorizes findings along separate **Validity**, **Severity**, and **Disposition** axes:
|
|
112
|
+
|
|
113
|
+
```json
|
|
114
|
+
{
|
|
115
|
+
"id": "DA-01",
|
|
116
|
+
"topic": "correctness",
|
|
117
|
+
"validity": "VALID",
|
|
118
|
+
"severity": "BLOCKER",
|
|
119
|
+
"disposition": "STRONG",
|
|
120
|
+
"location": "src/services/payment.ts#L42-L58",
|
|
121
|
+
"acceptanceCriteria": "AC-2",
|
|
122
|
+
"failureScenario": "Under concurrent traffic, duplicate rows are inserted before the lock is acquired.",
|
|
123
|
+
"evidence": "Missing SELECT FOR UPDATE in findByPaymentKey query.",
|
|
124
|
+
"concreteAlternativeDiff": "```diff\n- const tx = await findByKey(key);\n+ const tx = await findByKeyWithLock(key, { mode: 'FOR UPDATE' });\n```"
|
|
125
|
+
}
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
### Judge Decision Matrix:
|
|
129
|
+
- **`VALID + BLOCKER / HIGH`** $\rightarrow$ **`ITERATE`** (Maker must apply concrete fix diff and add regression tests).
|
|
130
|
+
- **`VALID + MEDIUM / LOW`** $\rightarrow$ **`ACCEPT / TRADEOFF`** (Merged; documented as acceptable tradeoff in MR notes).
|
|
131
|
+
- **`INVALID`** $\rightarrow$ **`DISMISS`** (Reviewer hallucination disproven by code; cannot block delivery; signature recorded).
|
|
132
|
+
|
|
133
|
+
*Reviewer disposition (`STRONG`, `ACCEPTABLE`, `WEAK`) never overrides factual evidence.*
|
|
134
|
+
|
|
135
|
+
---
|
|
13
136
|
|
|
14
|
-
|
|
137
|
+
## Living Project Context
|
|
138
|
+
|
|
139
|
+
The `.ai-engineering-loop/` directory is **Living Context**, not a static wiki generated once.
|
|
140
|
+
|
|
141
|
+
1. **Post-Task Context Impact Assessment**: Evaluates completed tasks (`NONE`, `TARGETED`, `MAJOR`) to keep project context fresh without expensive whole-repo re-analysis.
|
|
142
|
+
2. **Context Baseline (`metadata.json`)**: Tracks `repositoryRevision` (git commit SHA) and `manifestChecksums` for instant Level 0 (0ms) drift verification.
|
|
143
|
+
3. **Strict Context Isolation**: Decouples living project context from ephemeral task logs and loop execution states.
|
|
144
|
+
|
|
145
|
+
---
|
|
146
|
+
|
|
147
|
+
## CLI Interface & Commands
|
|
148
|
+
|
|
149
|
+
The CLI package is published on NPM as [`ai-engineering-loop`](https://www.npmjs.com/package/ai-engineering-loop) and operates against the current working directory.
|
|
15
150
|
|
|
16
151
|
```bash
|
|
17
152
|
# Bootstrap .ai-engineering-loop/ context from repository discovery
|
|
@@ -29,28 +164,153 @@ npx ai-engineering-loop run
|
|
|
29
164
|
|
|
30
165
|
---
|
|
31
166
|
|
|
32
|
-
##
|
|
167
|
+
## Grok CLI Integration
|
|
168
|
+
|
|
169
|
+
Grok CLI is a first-class host. `spawn_subagent` is a real independent child session (own context, no parent transcript unless `resume_from` is set). After a child id and model response are captured, the registry selects **`TRUE_INDEPENDENT_AGENT`**.
|
|
170
|
+
|
|
171
|
+
| Loop role | Grok `subagent_type` | Spawn rules |
|
|
172
|
+
|---|---|---|
|
|
173
|
+
| Orchestrator / Maker | parent session | Parent stays the orchestrator (Grok nesting depth is 1) |
|
|
174
|
+
| Devil's Advocate | `devil-advocate` (fallback `general-purpose`) | `capability_mode: execute`, omit `resume_from` |
|
|
175
|
+
| Judge | `judge` (fallback `general-purpose`) | Sibling of DA, never nested under DA |
|
|
176
|
+
|
|
177
|
+
Do **not** use `caveman:cavecrew-reviewer` as Devil's Advocate or Judge — its output schema is not the Finding Ledger.
|
|
178
|
+
|
|
179
|
+
Repo-local Grok files:
|
|
180
|
+
|
|
181
|
+
- `.grok/agents/devil-advocate.md` / `.grok/agents/judge.md`
|
|
182
|
+
- `.grok/skills/ai-engineering-loop/SKILL.md`
|
|
183
|
+
- `.grok/commands/ai-engineering-loop.md` → `/ai-engineering-loop`
|
|
184
|
+
|
|
185
|
+
Fallback: `GROK_SUBAGENTS=0` or `--disallowed-tools Agent` → `CONTEXT_ISOLATION_ONLY`, disclosed as such. Optional process fallback: `grok -p` → `FRESH_PROCESS_AGENT` only after a model response is captured.
|
|
186
|
+
|
|
187
|
+
See [docs/grok-cli-feasibility.md](docs/grok-cli-feasibility.md).
|
|
188
|
+
|
|
189
|
+
---
|
|
190
|
+
|
|
191
|
+
## Claude Code Integration
|
|
192
|
+
|
|
193
|
+
Claude Code is a first-class host. Use the **Task** (or **Agent**) tool with **only** `subagent_type`, `description`, and `prompt`.
|
|
194
|
+
|
|
195
|
+
Do **not** pass Grok keys (`spawn_subagent`, `capability_mode`, `isolation`, `resume_from`). Extra keys are the usual cause of:
|
|
196
|
+
|
|
197
|
+
```
|
|
198
|
+
API Error: 400 [kiro/claude-sonnet-5] REQUEST_BODY_INVALID
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
| Loop role | Claude Code `subagent_type` | Task keys |
|
|
202
|
+
|---|---|---|
|
|
203
|
+
| Orchestrator / Maker | parent session | n/a |
|
|
204
|
+
| Devil's Advocate | `devil-advocate` (fallback `general-purpose`) | `subagent_type`, `description`, `prompt` |
|
|
205
|
+
| Judge | `judge` (fallback `general-purpose`) | `subagent_type`, `description`, `prompt` |
|
|
206
|
+
|
|
207
|
+
Repo-local Claude Code files:
|
|
33
208
|
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
4. **Dual-Axis Finding Model & Judge Matrix**:
|
|
40
|
-
- `VALID + BLOCKER/HIGH` → `ITERATE`
|
|
41
|
-
- `VALID + MEDIUM/LOW` → `ACCEPT / TRADEOFF`
|
|
42
|
-
- `INVALID` → `DISMISS`
|
|
209
|
+
- `.claude/agents/devil-advocate.md` / `.claude/agents/judge.md`
|
|
210
|
+
- `.claude/skills/ai-engineering-loop/SKILL.md`
|
|
211
|
+
- `.claude/commands/ai-engineering-loop.md` → `/ai-engineering-loop`
|
|
212
|
+
|
|
213
|
+
See [docs/claude-code-feasibility.md](docs/claude-code-feasibility.md).
|
|
43
214
|
|
|
44
215
|
---
|
|
45
216
|
|
|
46
|
-
##
|
|
217
|
+
## Antigravity Agent Integration
|
|
218
|
+
|
|
219
|
+
When working inside the Antigravity IDE or compatible agentic platforms, you can invoke the loop via slash commands:
|
|
47
220
|
|
|
48
|
-
|
|
221
|
+
- **`/ai-engineering-loop init`**: Initialize project context only (non-destructive bootstrap).
|
|
222
|
+
- **`/ai-engineering-loop status`**: Check repository context health & baseline freshness.
|
|
223
|
+
- **`/ai-engineering-loop refresh`**: Reconcile drifted context files non-destructively.
|
|
224
|
+
- **`/ai-engineering-loop [task description]`**: Execute the full 8-stage engineering lifecycle with pre-task drift gate and post-task impact assessment.
|
|
49
225
|
|
|
50
|
-
|
|
226
|
+
On Grok CLI the same slash command is provided by `.grok/commands/ai-engineering-loop.md` and runs Devil's Advocate / Judge as native subagents.
|
|
227
|
+
|
|
228
|
+
---
|
|
229
|
+
|
|
230
|
+
## Repository Structure
|
|
231
|
+
|
|
232
|
+
```text
|
|
233
|
+
ai-engineering-loop/
|
|
234
|
+
│
|
|
235
|
+
├── README.md # Operating system overview & architecture
|
|
236
|
+
├── LICENSE # MIT Open Source License
|
|
237
|
+
├── package.json # CLI package manifest
|
|
238
|
+
│
|
|
239
|
+
├── bin/ # CLI execution entrypoints
|
|
240
|
+
│ └── ai-engineering-loop.js # npx executable CLI (init, status, refresh, run)
|
|
241
|
+
│
|
|
242
|
+
├── lib/ # Core orchestration & decision engine
|
|
243
|
+
│ └── orchestration.js # 3-stage capability registry, barrier builder, Judge engine
|
|
244
|
+
│
|
|
245
|
+
├── tests/ # Deterministic test suites
|
|
246
|
+
│ ├── capability-selection.test.js # Unit tests for capability lifecycle & truthful selection
|
|
247
|
+
│ ├── orchestration.test.js # Tests for isolation, Finding schema, Judge matrix
|
|
248
|
+
│ └── grok-runtime.test.js # Grok spawn_subagent mapping, aliases, forbidden types
|
|
249
|
+
│
|
|
250
|
+
├── .grok/ # Grok CLI host adapter
|
|
251
|
+
│ ├── agents/devil-advocate.md # Native DA subagent type
|
|
252
|
+
│ ├── agents/judge.md # Native Judge subagent type
|
|
253
|
+
│ ├── skills/ai-engineering-loop/ # Grok skill (spawn protocol)
|
|
254
|
+
│ └── commands/ai-engineering-loop.md # /ai-engineering-loop slash command
|
|
255
|
+
│
|
|
256
|
+
├── .claude/ # Claude Code host adapter (Kiro-safe)
|
|
257
|
+
│ ├── agents/devil-advocate.md # Task subagent type
|
|
258
|
+
│ ├── agents/judge.md # Task subagent type
|
|
259
|
+
│ ├── skills/ai-engineering-loop/ # Claude skill (Task keys only)
|
|
260
|
+
│ └── commands/ai-engineering-loop.md # /ai-engineering-loop slash command
|
|
261
|
+
│
|
|
262
|
+
├── core/ # Generic engineering loop specifications
|
|
263
|
+
│ ├── orchestration-model.md # 3-stage capability lifecycle & execution priority
|
|
264
|
+
│ ├── project-initialization.md # Auto-discovery & initialization lifecycle
|
|
265
|
+
│ ├── context-refresh-policy.md # Progressive drift hierarchy & living baseline
|
|
266
|
+
│ ├── context-impact-assessment.md # Post-task impact assessment (NONE, TARGETED, MAJOR)
|
|
267
|
+
│ ├── goal-contract.md # Task contract schema & acceptance criteria
|
|
268
|
+
│ ├── verification-loop.md # Dual-layer verification & Evidence Contract
|
|
269
|
+
│ ├── definition-of-done.md # 5 pillars of Done & rejection triggers
|
|
270
|
+
│ ├── iteration-policy.md # Bounded autonomous loop (MAX_ITERATIONS = 3)
|
|
271
|
+
│ ├── escalation-policy.md # Deterministic human escalation triggers
|
|
272
|
+
│ ├── judge-policy.md # Evaluation rules, triage audit, & verdicts
|
|
273
|
+
│ ├── configuration-precedence.md # 5-layer precedence & conflict resolution
|
|
274
|
+
│ └── repo-config-schema.md # Schema for target repo .ai-engineering-loop/
|
|
275
|
+
│
|
|
276
|
+
├── profiles/ # Project archetype profiles
|
|
277
|
+
│ ├── README.md # Profile catalog & auto-detection rules
|
|
278
|
+
│ ├── web-app.md # Frontend web applications
|
|
279
|
+
│ ├── backend-api.md # Backend APIs & microservices
|
|
280
|
+
│ ├── mobile-app.md # Native & cross-platform mobile apps
|
|
281
|
+
│ ├── library.md # Reusable SDKs & shared packages
|
|
282
|
+
│ └── monorepo.md # Multi-package monorepo workspaces
|
|
283
|
+
│
|
|
284
|
+
├── agents/ # Triad agent role specifications
|
|
285
|
+
│ ├── maker.md # Maker agent: surgical diffs & unit tests
|
|
286
|
+
│ ├── devil-advocate.md # Adversarial reviewer: dual-axis finding ledger & diffs
|
|
287
|
+
│ └── judge.md # Judge agent: impartial magistrate on Validity + Severity
|
|
288
|
+
│
|
|
289
|
+
├── policies/ # Operational schemas & algorithms
|
|
290
|
+
│ ├── discovery-safety-policy.md # Secret protection & non-destructive discovery rules
|
|
291
|
+
│ ├── finding-policy.md # Dual-axis finding schema & severity matrix
|
|
292
|
+
│ ├── evidence-policy.md # 5-level evidence hierarchy & Verification Evidence Contract
|
|
293
|
+
│ └── no-progress-policy.md # Finding signature hashing & stagnation detection
|
|
294
|
+
│
|
|
295
|
+
├── adapters/ # Pluggable delivery pipelines
|
|
296
|
+
│ └── dot/ # DOT Indonesia delivery adapter
|
|
297
|
+
│ ├── README.md # DOT adapter overview
|
|
298
|
+
│ ├── gitlab.md # glab CLI, issue cards, & MR generation
|
|
299
|
+
│ ├── multi-branch.md # main / staging / develop cherry-pick propagation
|
|
300
|
+
│ ├── coreview.md # @coreview-bot external review triage (Valid vs Halu)
|
|
301
|
+
│ └── mattermost.md # Channel mapping & MCP dispatch (from: "AI Agent")
|
|
302
|
+
│
|
|
303
|
+
└── templates/ # Starter templates for target repositories
|
|
304
|
+
└── repo-config/ # Ready-to-copy .ai-engineering-loop/ files
|
|
305
|
+
├── config.md # Project identity & profile binding
|
|
306
|
+
├── architecture.md # Layers & boundary invariants
|
|
307
|
+
├── conventions.md # Code standards & forbidden patterns
|
|
308
|
+
├── verification.md # CLI test/lint/build commands
|
|
309
|
+
└── adapter.md # Configured release pipeline
|
|
310
|
+
```
|
|
51
311
|
|
|
52
312
|
---
|
|
53
313
|
|
|
54
|
-
##
|
|
314
|
+
## License
|
|
55
315
|
|
|
56
|
-
MIT
|
|
316
|
+
This project is licensed under the **MIT License** — see the [LICENSE](LICENSE) file for details.
|
package/README.npm.md
CHANGED
|
@@ -36,6 +36,17 @@ npx ai-engineering-loop run
|
|
|
36
36
|
2. **Living Project Context**: Tracks repository revisions and manifest checksums in `metadata.json` for instant drift detection.
|
|
37
37
|
3. **Multi-Agent Triad**: Coordinates **Maker** (surgical diffs & tests), **Devil's Advocate** (independent adversarial review), and **Judge** (impartial evaluation & PASS certification).
|
|
38
38
|
4. **Context Impact Assessment**: Evaluates completed tasks (`NONE`, `TARGETED`, `MAJOR`) to keep project context fresh without expensive whole-repo re-analysis.
|
|
39
|
+
5. **Grok CLI host**: Native `spawn_subagent` for Devil's Advocate and Judge (`TRUE_INDEPENDENT_AGENT`). Disabled by `GROK_SUBAGENTS=0`.
|
|
40
|
+
|
|
41
|
+
---
|
|
42
|
+
|
|
43
|
+
## Grok CLI
|
|
44
|
+
|
|
45
|
+
Inside a Grok TUI session, `/ai-engineering-loop` uses repo-local `.grok/agents/` types. Devil's Advocate and Judge spawn as sibling children (`capability_mode: execute`, no `resume_from`). Do not use `caveman:cavecrew-reviewer` as the loop reviewer.
|
|
46
|
+
|
|
47
|
+
## Claude Code
|
|
48
|
+
|
|
49
|
+
Inside Claude Code, `/ai-engineering-loop` uses `.claude/agents/` types via the **Task** tool. Pass only `subagent_type`, `description`, and `prompt`. Do not pass Grok keys (`spawn_subagent`, `capability_mode`, `resume_from`) — extra keys cause Kiro `REQUEST_BODY_INVALID`.
|
|
39
50
|
|
|
40
51
|
---
|
|
41
52
|
|
package/agents/devil-advocate.md
CHANGED
|
@@ -12,12 +12,16 @@ The **Devil's Advocate** is an independent adversarial reviewer. Its sole purpos
|
|
|
12
12
|
|
|
13
13
|
## 2. Review Execution Modes
|
|
14
14
|
|
|
15
|
-
The Devil's Advocate executes under one of 4 runtime modes depending on platform capabilities
|
|
15
|
+
The Devil's Advocate executes under one of 4 runtime modes depending on platform capabilities.
|
|
16
16
|
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
17
|
+
Canonical ids are listed first; skill aliases are in parentheses.
|
|
18
|
+
|
|
19
|
+
1. **`TRUE_INDEPENDENT_AGENT`** (`NATIVE_SUBAGENT`): Genuine independent child session.
|
|
20
|
+
- **Grok CLI**: `spawn_subagent` with `subagent_type: "devil-advocate"` (fallback `"general-purpose"`), `capability_mode: "execute"`, `isolation: "none"`, **omit** `resume_from`. Agent definition: `.grok/agents/devil-advocate.md`.
|
|
21
|
+
- Do **not** use `caveman:cavecrew-reviewer` (compressed review schema, not a Finding Ledger).
|
|
22
|
+
2. **`ISOLATED_AGENT_INSTANCE`** (`SDK_AGENT`): Programmatic SDK agent instance with isolated memory.
|
|
23
|
+
3. **`FRESH_PROCESS_AGENT`** (`HEADLESS_SUBPROCESS`): Fresh OS process such as `grok -p` after a model response is captured.
|
|
24
|
+
4. **`CONTEXT_ISOLATION_ONLY`** (`ARTIFACT_ISOLATED_REVIEW`): Clean-Slate Artifact Barrier in single-agent session (*strictly labeled: isolated review context, not independent agent execution*).
|
|
21
25
|
|
|
22
26
|
---
|
|
23
27
|
|
package/agents/judge.md
CHANGED
|
@@ -4,6 +4,8 @@
|
|
|
4
4
|
|
|
5
5
|
The **Judge Agent** is the final decision-maker of the AI Engineering Loop. It serves as an impartial magistrate that evaluates the complete evidence pipeline (Contract, Diff, Deterministic Verification Logs, and Devil's Advocate findings) to determine whether the iteration should **PASS**, **ITERATE**, or **ESCALATE**.
|
|
6
6
|
|
|
7
|
+
On **Grok CLI**, the parent orchestrator spawns the Judge with `spawn_subagent` (`subagent_type: "judge"`, fallback `"general-purpose"`), `capability_mode: "execute"`, and **no** `resume_from`. Agent definition: `.grok/agents/judge.md`. The Judge is a sibling of the Devil's Advocate, never its child (Grok nesting depth is 1).
|
|
8
|
+
|
|
7
9
|
```mermaid
|
|
8
10
|
flowchart LR
|
|
9
11
|
Inputs[Goal Contract + Diff + Test Logs + Review Findings] --> Judge[Judge Agent]
|
|
@@ -15,7 +15,7 @@ const path = require('path');
|
|
|
15
15
|
const crypto = require('crypto');
|
|
16
16
|
const { execSync } = require('child_process');
|
|
17
17
|
|
|
18
|
-
const VERSION = '1.0.
|
|
18
|
+
const VERSION = '1.0.6';
|
|
19
19
|
const CWD = process.cwd();
|
|
20
20
|
const CONTEXT_DIR = path.join(CWD, '.ai-engineering-loop');
|
|
21
21
|
|
|
@@ -553,6 +553,15 @@ function handleRefresh() {
|
|
|
553
553
|
handleStatus();
|
|
554
554
|
}
|
|
555
555
|
|
|
556
|
+
function detectGrokHost() {
|
|
557
|
+
try {
|
|
558
|
+
const { detectGrokRuntime } = require('../lib/orchestration.js');
|
|
559
|
+
return detectGrokRuntime(process.env, fs);
|
|
560
|
+
} catch (e) {
|
|
561
|
+
return null;
|
|
562
|
+
}
|
|
563
|
+
}
|
|
564
|
+
|
|
556
565
|
// Command: run
|
|
557
566
|
function handleRun() {
|
|
558
567
|
log.info('AI Engineering Loop — Task Execution Entrypoint (run)');
|
|
@@ -564,6 +573,8 @@ function handleRun() {
|
|
|
564
573
|
handleStatus();
|
|
565
574
|
}
|
|
566
575
|
|
|
576
|
+
const grok = detectGrokHost();
|
|
577
|
+
|
|
567
578
|
console.log('\n------------------------------------------------------------');
|
|
568
579
|
log.bold('AI Agent Ready:');
|
|
569
580
|
console.log('1. Formulate Goal Contract (core/goal-contract.md)');
|
|
@@ -574,6 +585,34 @@ function handleRun() {
|
|
|
574
585
|
console.log('6. Judge Agent evaluates DoD and issues PASS verdict');
|
|
575
586
|
console.log('7. Context Impact Assessment (NONE / TARGETED / MAJOR)');
|
|
576
587
|
console.log('8. Delivery Adapter creates MR/PR');
|
|
588
|
+
|
|
589
|
+
if (grok && grok.host === 'grok-cli') {
|
|
590
|
+
console.log('------------------------------------------------------------');
|
|
591
|
+
log.bold('Grok CLI host:');
|
|
592
|
+
console.log(`- Binary: ${grok.grokBin || 'detected'}`);
|
|
593
|
+
console.log(`- spawn_subagent: ${grok.invocationAvailable ? 'INVOCATION_AVAILABLE' : 'UNAVAILABLE'}`);
|
|
594
|
+
console.log(`- Execution proven: no (requires child subagent_id + model response)`);
|
|
595
|
+
if (grok.invocationAvailable) {
|
|
596
|
+
console.log('- Devil\'s Advocate: spawn_subagent type=devil-advocate capability_mode=execute (no resume_from)');
|
|
597
|
+
console.log('- Judge: spawn_subagent type=judge capability_mode=execute (sibling, not nested)');
|
|
598
|
+
console.log('- Forbidden types: caveman:cavecrew-reviewer, explore, plan');
|
|
599
|
+
} else {
|
|
600
|
+
console.log(`- Fallback: CONTEXT_ISOLATION_ONLY (${grok.reason})`);
|
|
601
|
+
}
|
|
602
|
+
console.log('- Skill: .grok/skills/ai-engineering-loop/SKILL.md');
|
|
603
|
+
}
|
|
604
|
+
|
|
605
|
+
const claudeSkill = path.join(CWD, '.claude', 'skills', 'ai-engineering-loop', 'SKILL.md');
|
|
606
|
+
const claudeAgent = path.join(CWD, '.claude', 'agents', 'devil-advocate.md');
|
|
607
|
+
if (fs.existsSync(claudeSkill) || fs.existsSync(claudeAgent)) {
|
|
608
|
+
console.log('------------------------------------------------------------');
|
|
609
|
+
log.bold('Claude Code host:');
|
|
610
|
+
console.log('- Subagent tool: Task (or Agent). Keys allowed: subagent_type, description, prompt');
|
|
611
|
+
console.log('- Devil\'s Advocate: Task subagent_type=devil-advocate');
|
|
612
|
+
console.log('- Judge: Task subagent_type=judge (sibling, not nested)');
|
|
613
|
+
console.log('- Do not pass spawn_subagent, capability_mode, isolation, resume_from (Kiro 400 REQUEST_BODY_INVALID)');
|
|
614
|
+
console.log('- Skill: .claude/skills/ai-engineering-loop/SKILL.md');
|
|
615
|
+
}
|
|
577
616
|
console.log('------------------------------------------------------------\n');
|
|
578
617
|
}
|
|
579
618
|
|
|
@@ -69,6 +69,26 @@ The system categorically rejects the following as proof of independent LLM execu
|
|
|
69
69
|
|
|
70
70
|
---
|
|
71
71
|
|
|
72
|
+
## 4b. Grok CLI Empirical Discovery Record
|
|
73
|
+
|
|
74
|
+
| Investigated Surface | Tested Command / API | Classification | Status & Result |
|
|
75
|
+
|---|---|---|---|
|
|
76
|
+
| **Grok CLI binary** | `~/.grok/bin/grok` | `CONFIGURATION_SUPPORTED` | Installed (`grok 1.0.5+`). Not execution proof. |
|
|
77
|
+
| **Native subagent tool** | `spawn_subagent` | `INVOCATION_AVAILABLE` | Enabled by default. Disabled by `GROK_SUBAGENTS=0` or `--disallowed-tools Agent`. |
|
|
78
|
+
| **Child session** | `subagent_id` + child model response, `resume_from` omitted | `EXECUTION_PROVEN` → `TRUE_INDEPENDENT_AGENT` | Own context window; parent transcript is not inherited. |
|
|
79
|
+
| **Project agent types** | `.grok/agents/devil-advocate.md`, `.grok/agents/judge.md` | `CONFIGURATION_SUPPORTED` | Prefer these types; fall back to `general-purpose` with the spec prepended. |
|
|
80
|
+
| **Headless process** | `grok -p` | `FRESH_PROCESS_AGENT` | Only after the process prints a model response. |
|
|
81
|
+
| **Cavecrew reviewer** | `caveman:cavecrew-reviewer` | `GROK_COMPRESSED_REVIEW_PRESET` | **Rejected** as Devil's Advocate / Judge (wrong output schema). |
|
|
82
|
+
| **Maker resume** | `spawn_subagent(resume_from=maker)` | tainted history | **Rejected**. DA and Judge must spawn fresh. |
|
|
83
|
+
| **Artifact barrier** | `buildReviewContextBarrier()` | `CONTEXT_ISOLATION_ONLY` | Fallback when spawn is disabled. |
|
|
84
|
+
|
|
85
|
+
### Architectural Conclusion:
|
|
86
|
+
> *"Grok CLI spawn_subagent is a true independent child session. Select TRUE_INDEPENDENT_AGENT only after a child id and model response are captured without resume_from. Binary presence and agent markdown are never sufficient."*
|
|
87
|
+
|
|
88
|
+
See [docs/grok-cli-feasibility.md](../docs/grok-cli-feasibility.md).
|
|
89
|
+
|
|
90
|
+
---
|
|
91
|
+
|
|
72
92
|
## 5. Truthful Reporting Output
|
|
73
93
|
|
|
74
94
|
When `CONTEXT_ISOLATION_ONLY` is selected, the report generator strictly produces:
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
# Claude Code Feasibility and Kiro Request Safety
|
|
2
|
+
|
|
3
|
+
## 1. Problem
|
|
4
|
+
|
|
5
|
+
Claude Code users (including Kiro-backed models such as `kiro/claude-sonnet-5`) hit:
|
|
6
|
+
|
|
7
|
+
```
|
|
8
|
+
API Error: 400 [kiro/claude-sonnet-5] [400]:
|
|
9
|
+
{"message":"Improperly formed request.","reason":"REQUEST_BODY_INVALID"}
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
Kiro's `generateAssistantResponse` rejects request bodies that Claude Code will happily send. Two skill-side causes were confirmed:
|
|
13
|
+
|
|
14
|
+
1. **Extra tool keys.** The shared skill told the model to call `spawn_subagent` with `capability_mode`, `isolation`, and `resume_from`. Claude Code's Task/Agent schema does not have those fields. Proxies that validate additionalProperties fail with `REQUEST_BODY_INVALID`.
|
|
15
|
+
2. **Hostile markup in the injected skill.** Mermaid (`<br>`, nested `{braces}`), LaTeX (`$\rightarrow$`), and folded YAML descriptions inflate and corrupt the system/skill payload Kiro receives.
|
|
16
|
+
|
|
17
|
+
This is separate from 9router bugs that reject any Anthropic `system` field. Those need a router fix. The skill must still not add extra invalid keys or markup.
|
|
18
|
+
|
|
19
|
+
## 2. Claude Code mapping
|
|
20
|
+
|
|
21
|
+
| Loop role | Claude Code type | Task keys allowed |
|
|
22
|
+
|---|---|---|
|
|
23
|
+
| Orchestrator / Maker | parent session | n/a |
|
|
24
|
+
| Devil's Advocate | `devil-advocate` (fallback `general-purpose`) | `subagent_type`, `description`, `prompt` |
|
|
25
|
+
| Judge | `judge` (fallback `general-purpose`) | `subagent_type`, `description`, `prompt` |
|
|
26
|
+
|
|
27
|
+
Do not pass Grok keys (`spawn_subagent`, `capability_mode`, `resume_from`, `isolation`, `background`) on Claude Code.
|
|
28
|
+
|
|
29
|
+
Repo files:
|
|
30
|
+
|
|
31
|
+
- `.claude/skills/ai-engineering-loop/SKILL.md`
|
|
32
|
+
- `.claude/agents/devil-advocate.md`
|
|
33
|
+
- `.claude/agents/judge.md`
|
|
34
|
+
- `.claude/commands/ai-engineering-loop.md`
|
|
35
|
+
|
|
36
|
+
## 3. Mode selection
|
|
37
|
+
|
|
38
|
+
| Condition | Mode |
|
|
39
|
+
|---|---|
|
|
40
|
+
| Task/Agent tool present, child result returned | `TRUE_INDEPENDENT_AGENT` |
|
|
41
|
+
| No subagent tool | `CONTEXT_ISOLATION_ONLY` |
|
|
42
|
+
|
|
43
|
+
## 4. What stays in Grok-only files
|
|
44
|
+
|
|
45
|
+
Grok spawn details live in `.grok/skills/ai-engineering-loop/SKILL.md`. Claude Code must not load that file as its skill. Claude Code discovers `.claude/` first.
|
|
@@ -0,0 +1,99 @@
|
|
|
1
|
+
# Grok CLI Feasibility & Execution Strategy
|
|
2
|
+
|
|
3
|
+
## 1. Executive Summary
|
|
4
|
+
|
|
5
|
+
Grok CLI (verified `grok 1.0.5`) is a **first-class host** for the AI Engineering Loop. Unlike Antigravity standalone mode, Grok exposes `spawn_subagent` as a real independent child session: own context window, no parent transcript unless `resume_from` is set, and a captured `subagent_id` plus model response.
|
|
6
|
+
|
|
7
|
+
When those evidence fields are present, the capability registry **must** select `TRUE_INDEPENDENT_AGENT` (skill alias: `NATIVE_SUBAGENT`).
|
|
8
|
+
|
|
9
|
+
```text
|
|
10
|
+
Parent (Maker + orchestrator)
|
|
11
|
+
├─ spawn_subagent type=devil-advocate → Finding Ledger
|
|
12
|
+
└─ spawn_subagent type=judge → PASS | ITERATE | ESCALATE
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
Children cannot spawn children (depth 1). The parent therefore remains the orchestrator.
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## 2. Empirical Discovery Record
|
|
20
|
+
|
|
21
|
+
| Surface | Tested command / API | Classification | Result |
|
|
22
|
+
|---|---|---|---|
|
|
23
|
+
| Grok CLI binary | `~/.grok/bin/grok` (`grok 1.0.5`) | `CONFIGURATION_SUPPORTED` | Installed and authenticated in TUI sessions |
|
|
24
|
+
| Native child session | `spawn_subagent` | `INVOCATION_AVAILABLE` | Enabled by default; disabled only when `GROK_SUBAGENTS=0` or `--disallowed-tools Agent` |
|
|
25
|
+
| Child execution | child `subagent_id` + model response | `EXECUTION_PROVEN` | Proven only after the child returns; binary presence is not proof |
|
|
26
|
+
| Project agents | `.grok/agents/devil-advocate.md`, `.grok/agents/judge.md` | `CONFIGURATION_SUPPORTED` | Registered as `subagent_type` values |
|
|
27
|
+
| Skill + slash command | `.grok/skills/ai-engineering-loop/`, `.grok/commands/ai-engineering-loop.md` | `CONFIGURATION_SUPPORTED` | Grok discovers repo-local skills and command markdown |
|
|
28
|
+
| Headless child | `grok -p "..."` | `FRESH_PROCESS_AGENT` | Valid fallback **after** stdout contains a model response |
|
|
29
|
+
| Compressed reviewer | `caveman:cavecrew-reviewer` | `GROK_COMPRESSED_REVIEW_PRESET` | **Rejected** — output schema is `path:line: emoji severity`, not the Finding Ledger |
|
|
30
|
+
| Browser tools | `browser_*` | `BROWSER_AUTOMATION_TOOL` | Not an LLM reviewer |
|
|
31
|
+
| `resume_from` Maker | `spawn_subagent(resume_from=makerId)` | tainted history | **Rejected** — inherits Maker transcript |
|
|
32
|
+
|
|
33
|
+
### What is NOT execution proof on Grok
|
|
34
|
+
|
|
35
|
+
1. `~/.grok/bin/grok` existing on disk
|
|
36
|
+
2. `.grok/agents/*.md` being discoverable
|
|
37
|
+
3. Subagents being enabled in `config.toml`
|
|
38
|
+
4. A persona named "reviewer" without a child response
|
|
39
|
+
5. Cavecrew reviewer output that looks like a review
|
|
40
|
+
|
|
41
|
+
---
|
|
42
|
+
|
|
43
|
+
## 3. Mapping Loop Roles → Grok Types
|
|
44
|
+
|
|
45
|
+
| Loop role | Grok `subagent_type` | `capability_mode` | `isolation` | `resume_from` |
|
|
46
|
+
|---|---|---|---|---|
|
|
47
|
+
| Orchestrator / Maker | parent session (or `general-purpose` if parent is orchestrator-only) | `all` | `none` | n/a |
|
|
48
|
+
| Devil's Advocate | `devil-advocate` (fallback `general-purpose`) | `execute` | `none` | **omit** |
|
|
49
|
+
| Judge | `judge` (fallback `general-purpose`) | `execute` | `none` | **omit** |
|
|
50
|
+
|
|
51
|
+
`execute` allows git/read/search and forbids source edits. Do not use `read-only` if the child needs `git diff`. Do not use `worktree` isolation: DA and Judge must see the Maker working tree.
|
|
52
|
+
|
|
53
|
+
`description` prefixes `[devil-advocate]` and `[judge]` so the Grok TUI labels the child correctly when falling back to `general-purpose`.
|
|
54
|
+
|
|
55
|
+
---
|
|
56
|
+
|
|
57
|
+
## 4. Artifact Barrier (still required)
|
|
58
|
+
|
|
59
|
+
Grok children do not inherit parent chat, but they do receive compacted `AGENTS.md` / project instructions. That is acceptable (project context, not Maker thoughts).
|
|
60
|
+
|
|
61
|
+
Still write artifacts to disk and pass **paths**, not Maker narration:
|
|
62
|
+
|
|
63
|
+
- Goal Contract
|
|
64
|
+
- git diff file
|
|
65
|
+
- verification log (command, exit code, stdout, test counts)
|
|
66
|
+
- `.ai-engineering-loop/`
|
|
67
|
+
|
|
68
|
+
---
|
|
69
|
+
|
|
70
|
+
## 5. Disable / fallback matrix
|
|
71
|
+
|
|
72
|
+
| Condition | Selected mode |
|
|
73
|
+
|---|---|
|
|
74
|
+
| `spawn_subagent` available, child response captured, no `resume_from` | `TRUE_INDEPENDENT_AGENT` |
|
|
75
|
+
| `spawn_subagent` missing, `grok -p` returns a model response | `FRESH_PROCESS_AGENT` |
|
|
76
|
+
| `GROK_SUBAGENTS=0` or `--disallowed-tools Agent` | `CONTEXT_ISOLATION_ONLY` |
|
|
77
|
+
| No grok, no spawn, no headless | `UNAVAILABLE` or artifact barrier if the same session can still read the diff |
|
|
78
|
+
|
|
79
|
+
---
|
|
80
|
+
|
|
81
|
+
## 6. Answers to the eight host questions
|
|
82
|
+
|
|
83
|
+
1. **Independent sub-agents?** Yes — `spawn_subagent`, own context window.
|
|
84
|
+
2. **Pass context?** Yes — prompt + on-disk artifacts. Do not pass Maker chat.
|
|
85
|
+
3. **Same repository?** Yes — `isolation: none` shares the workspace.
|
|
86
|
+
4. **Structured findings?** Yes — DA agent contract is Finding Ledger JSON.
|
|
87
|
+
5. **Parent consume findings?** Yes — child summary returns to parent; Judge spawn gets the ledger.
|
|
88
|
+
6. **Repeat the loop?** Yes — parent iterates; each DA/Judge spawn is fresh (no `resume_from`).
|
|
89
|
+
7. **Persist state?** Yes — files under `.ai-engineering-loop/` and task artifact paths. Grok also keeps session logs under `~/.grok/sessions/`.
|
|
90
|
+
8. **Token limits?** Pass artifact paths, not full transcripts. Do not dump Maker reasoning into the DA prompt.
|
|
91
|
+
|
|
92
|
+
---
|
|
93
|
+
|
|
94
|
+
## 7. Practical limits on Grok CLI
|
|
95
|
+
|
|
96
|
+
- **Depth 1**: a DA child cannot spawn a Judge. Parent must spawn both.
|
|
97
|
+
- **Plan mode**: a write-capable child is not gated by the parent's plan-mode lock. Keep DA/Judge on `capability_mode: execute` (no writes) so they cannot edit source while the parent is in plan mode.
|
|
98
|
+
- **Headless `--disallowed-tools Agent`**: kills native review. Disclose `CONTEXT_ISOLATION_ONLY`.
|
|
99
|
+
- **Cavecrew**: useful for cheap locate/review, **not** for this loop's Judge input.
|