ai-engineering-loop 1.0.1 → 1.0.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.full.md +251 -0
- package/README.md +23 -30
- package/agents/devil-advocate.md +54 -92
- package/bin/ai-engineering-loop.js +1 -1
- package/core/judge-policy.md +48 -101
- package/core/orchestration-model.md +87 -0
- package/core/verification-loop.md +47 -83
- package/lib/orchestration.js +409 -0
- package/package.json +2 -3
- package/policies/evidence-policy.md +33 -15
- package/policies/finding-policy.md +39 -71
- package/tests/capability-selection.test.js +207 -0
- package/tests/orchestration.test.js +288 -0
- package/.README.github.bak.md +0 -272
- package/scripts/prepare-npm.js +0 -17
- package/scripts/restore-github.js +0 -15
package/README.full.md
ADDED
|
@@ -0,0 +1,251 @@
|
|
|
1
|
+
# AI Engineering Loop
|
|
2
|
+
|
|
3
|
+
<div align="center">
|
|
4
|
+
|
|
5
|
+
[](https://www.npmjs.com/package/ai-engineering-loop)
|
|
6
|
+
[](https://opensource.org/licenses/MIT)
|
|
7
|
+
[](https://github.com/egagofur/ai-engineering-loop/pulls)
|
|
8
|
+
[](https://github.com/egagofur/ai-engineering-loop)
|
|
9
|
+
[](https://github.com/egagofur/ai-engineering-loop/releases)
|
|
10
|
+
|
|
11
|
+
**A Reusable, Framework-Agnostic AI Engineering Operating System for Autonomous Coding Agents**
|
|
12
|
+
|
|
13
|
+
*Featuring living project context, strict verification evidence contracts, 3-stage capability lifecycle registry, and dual-axis Judge evaluation.*
|
|
14
|
+
|
|
15
|
+
[Overview](#overview--philosophy) • [Runtime Capability Registry](#runtime-capability-registry--execution-modes) • [Verification Evidence](#verification-evidence-contract) • [CLI Commands](#cli-interface--commands) • [Agent Integration](#antigravity-agent-integration) • [Lifecycle](#lifecycle-stages) • [Architecture](#architecture--5-layer-configuration) • [Project Profiles](#project-profiles) • [Repository Structure](#repository-structure) • [Reference Examples](#reference-examples) • [Contributing](#contributing)
|
|
16
|
+
|
|
17
|
+
</div>
|
|
18
|
+
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## Overview & Philosophy
|
|
22
|
+
|
|
23
|
+
The AI Engineering Loop enforces clean architectural separation across three core layers:
|
|
24
|
+
|
|
25
|
+
```mermaid
|
|
26
|
+
flowchart TD
|
|
27
|
+
Start([User Task in Workspace]) --> PreCheck{Pre-Task Drift Check: metadata.json}
|
|
28
|
+
|
|
29
|
+
PreCheck -->|Context Missing| AutoInit[Stage 0: Bootstrap .ai-engineering-loop/]
|
|
30
|
+
PreCheck -->|Drift Detected| Reconcile[Stage 0: Reconcile Drifted Context]
|
|
31
|
+
PreCheck -->|Context Fresh| GC[Stage 1: Goal Contract: Explicit Acceptance Criteria]
|
|
32
|
+
|
|
33
|
+
AutoInit --> GC
|
|
34
|
+
Reconcile --> GC
|
|
35
|
+
|
|
36
|
+
subgraph CoreEngine [AI ENGINEERING OPERATING SYSTEM]
|
|
37
|
+
GC --> RCA[Stage 2: Root Cause Analysis]
|
|
38
|
+
RCA --> Plan[Stage 3: Implementation Plan]
|
|
39
|
+
Plan --> MA[Stage 4: Maker Agent: Surgical Diff & Tests]
|
|
40
|
+
MA --> DV{Stage 5: Deterministic Verification<br>Evidence Contract: Exit Code 0 & Full Logs}
|
|
41
|
+
|
|
42
|
+
DV -->|Fail| MA
|
|
43
|
+
DV -->|Pass| DA[Stage 6: Devil's Advocate Review<br>Capability Registry & Artifact Barrier]
|
|
44
|
+
|
|
45
|
+
DA --> JD[Stage 7: Judge Agent: Impartial Magistrate<br>Validity + Severity Decision Matrix]
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
JD -->|VALID BLOCKER / HIGH: ITERATE| MA
|
|
49
|
+
JD -->|INVALID: Dismissed / VALID LOW: Tradeoff| CheckDoD{All ACs Verified?}
|
|
50
|
+
|
|
51
|
+
CheckDoD -->|Yes: PASS| ImpactEval{Post-Task Context Impact Assessment}
|
|
52
|
+
ImpactEval -->|NONE: Typo, UI tweak| Adapter[Stage 8: Delivery Adapter: GitLab / GitHub]
|
|
53
|
+
ImpactEval -->|TARGETED: Dep/route changed| PartialRefresh[Surgical Context Update] --> Adapter
|
|
54
|
+
ImpactEval -->|MAJOR: Framework migration| FullRefresh[Full Context Reconciliation] --> Adapter
|
|
55
|
+
|
|
56
|
+
Adapter --> TargetRepo[(Target Repository)]
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
---
|
|
60
|
+
|
|
61
|
+
## Runtime Capability Registry & Execution Modes
|
|
62
|
+
|
|
63
|
+
The system maintains a strict distinction between **Configuration Support**, **Invocation Availability**, and **Execution Proof**:
|
|
64
|
+
|
|
65
|
+
```text
|
|
66
|
+
┌───────────────────────────┐ ┌───────────────────────────┐ ┌───────────────────────────┐
|
|
67
|
+
│ CONFIGURATION_SUPPORTED │ ──> │ INVOCATION_AVAILABLE │ ──> │ EXECUTION_PROVEN │
|
|
68
|
+
│ (Config is recognized) │ │ (Callable tool is active) │ │ (Child LLM response seen) │
|
|
69
|
+
└───────────────────────────┘ └───────────────────────────┘ └───────────────────────────┘
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
### 5 Standard Execution Modes (Deterministic Priority):
|
|
73
|
+
|
|
74
|
+
| Priority | Mode Name | Requires Independent LLM Execution? | Condition for Selection |
|
|
75
|
+
|:---:|---|:---:|---|
|
|
76
|
+
| **1** | **`TRUE_INDEPENDENT_AGENT`** | **YES** | Child session exists **AND** actual model response is captured **AND** context is independent. |
|
|
77
|
+
| **2** | **`ISOLATED_AGENT_INSTANCE`** | **YES** | Programmatic SDK agent instance with verified independent model execution. |
|
|
78
|
+
| **3** | **`FRESH_PROCESS_AGENT`** | **YES** | Separate OS process successfully executes an LLM agent with fresh context. |
|
|
79
|
+
| **4** | **`CONTEXT_ISOLATION_ONLY`** | **NO** | Clean-Slate Artifact Isolation Barrier in same session (100% prompt history excluded on disk). |
|
|
80
|
+
| **5** | **`UNAVAILABLE`** | **NO** | No review execution mechanism is available. |
|
|
81
|
+
|
|
82
|
+
### Truthful Reporting Disclosure:
|
|
83
|
+
When `CONTEXT_ISOLATION_ONLY` is selected, the report strictly produces:
|
|
84
|
+
```text
|
|
85
|
+
Execution Mode: CONTEXT_ISOLATION_ONLY
|
|
86
|
+
Independent LLM Execution: NOT PROVEN
|
|
87
|
+
Native Subagent Invocation: UNAVAILABLE
|
|
88
|
+
Review Method: Clean-Slate Artifact Isolation Barrier
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
---
|
|
92
|
+
|
|
93
|
+
## Verification Evidence Contract
|
|
94
|
+
|
|
95
|
+
A verification `PASS` is strictly invalid without concrete execution evidence. The system categorically rejects vague statements such as *"command was launched"* or *"test appears to have passed"*.
|
|
96
|
+
|
|
97
|
+
### Mandatory Execution Proof:
|
|
98
|
+
- **`command`**: Exact CLI string executed.
|
|
99
|
+
- **`executionIdentity`**: PID, execution hash, or system execution identifier.
|
|
100
|
+
- **`startTime` & `endTime`**: Documented execution duration.
|
|
101
|
+
- **`exitCode`**: Must be `0`.
|
|
102
|
+
- **`stdout` & `stderr`**: Raw machine logs captured.
|
|
103
|
+
- **`timeoutStatus`**: Must be `"COMPLETED"`.
|
|
104
|
+
- **`testCounts`**: Explicit counts of passed, failed, and skipped tests.
|
|
105
|
+
- **`assertionEvidence`**: Specific assertion proof matching the active Goal Contract's Acceptance Criteria.
|
|
106
|
+
|
|
107
|
+
---
|
|
108
|
+
|
|
109
|
+
## Dual-Axis Finding Model & Judge Decision Matrix
|
|
110
|
+
|
|
111
|
+
The Devil's Advocate categorizes findings along separate **Validity**, **Severity**, and **Disposition** axes:
|
|
112
|
+
|
|
113
|
+
```json
|
|
114
|
+
{
|
|
115
|
+
"id": "DA-01",
|
|
116
|
+
"topic": "correctness",
|
|
117
|
+
"validity": "VALID",
|
|
118
|
+
"severity": "BLOCKER",
|
|
119
|
+
"disposition": "STRONG",
|
|
120
|
+
"location": "src/services/payment.ts#L42-L58",
|
|
121
|
+
"acceptanceCriteria": "AC-2",
|
|
122
|
+
"failureScenario": "Under concurrent traffic, duplicate rows are inserted before the lock is acquired.",
|
|
123
|
+
"evidence": "Missing SELECT FOR UPDATE in findByPaymentKey query.",
|
|
124
|
+
"concreteAlternativeDiff": "```diff\n- const tx = await findByKey(key);\n+ const tx = await findByKeyWithLock(key, { mode: 'FOR UPDATE' });\n```"
|
|
125
|
+
}
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
### Judge Decision Matrix:
|
|
129
|
+
- **`VALID + BLOCKER / HIGH`** $\rightarrow$ **`ITERATE`** (Maker must apply concrete fix diff and add regression tests).
|
|
130
|
+
- **`VALID + MEDIUM / LOW`** $\rightarrow$ **`ACCEPT / TRADEOFF`** (Merged; documented as acceptable tradeoff in MR notes).
|
|
131
|
+
- **`INVALID`** $\rightarrow$ **`DISMISS`** (Reviewer hallucination disproven by code; cannot block delivery; signature recorded).
|
|
132
|
+
|
|
133
|
+
*Reviewer disposition (`STRONG`, `ACCEPTABLE`, `WEAK`) never overrides factual evidence.*
|
|
134
|
+
|
|
135
|
+
---
|
|
136
|
+
|
|
137
|
+
## Living Project Context
|
|
138
|
+
|
|
139
|
+
The `.ai-engineering-loop/` directory is **Living Context**, not a static wiki generated once.
|
|
140
|
+
|
|
141
|
+
1. **Post-Task Context Impact Assessment**: Evaluates completed tasks (`NONE`, `TARGETED`, `MAJOR`) to keep project context fresh without expensive whole-repo re-analysis.
|
|
142
|
+
2. **Context Baseline (`metadata.json`)**: Tracks `repositoryRevision` (git commit SHA) and `manifestChecksums` for instant Level 0 (0ms) drift verification.
|
|
143
|
+
3. **Strict Context Isolation**: Decouples living project context from ephemeral task logs and loop execution states.
|
|
144
|
+
|
|
145
|
+
---
|
|
146
|
+
|
|
147
|
+
## CLI Interface & Commands
|
|
148
|
+
|
|
149
|
+
The CLI package is published on NPM as [`ai-engineering-loop`](https://www.npmjs.com/package/ai-engineering-loop) and operates against the current working directory.
|
|
150
|
+
|
|
151
|
+
```bash
|
|
152
|
+
# Bootstrap .ai-engineering-loop/ context from repository discovery
|
|
153
|
+
npx ai-engineering-loop init
|
|
154
|
+
|
|
155
|
+
# Check the validity, readiness, and baseline freshness of context
|
|
156
|
+
npx ai-engineering-loop status
|
|
157
|
+
|
|
158
|
+
# Reconcile drifted context against repository non-destructively
|
|
159
|
+
npx ai-engineering-loop refresh
|
|
160
|
+
|
|
161
|
+
# Verify context readiness and begin engineering loop
|
|
162
|
+
npx ai-engineering-loop run
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
---
|
|
166
|
+
|
|
167
|
+
## Antigravity Agent Integration
|
|
168
|
+
|
|
169
|
+
When working inside the Antigravity IDE or compatible agentic platforms, you can invoke the loop via slash commands:
|
|
170
|
+
|
|
171
|
+
- **`/ai-engineering-loop init`**: Initialize project context only (non-destructive bootstrap).
|
|
172
|
+
- **`/ai-engineering-loop status`**: Check repository context health & baseline freshness.
|
|
173
|
+
- **`/ai-engineering-loop refresh`**: Reconcile drifted context files non-destructively.
|
|
174
|
+
- **`/ai-engineering-loop [task description]`**: Execute the full 8-stage engineering lifecycle with pre-task drift gate and post-task impact assessment.
|
|
175
|
+
|
|
176
|
+
---
|
|
177
|
+
|
|
178
|
+
## Repository Structure
|
|
179
|
+
|
|
180
|
+
```text
|
|
181
|
+
ai-engineering-loop/
|
|
182
|
+
│
|
|
183
|
+
├── README.md # Operating system overview & architecture
|
|
184
|
+
├── LICENSE # MIT Open Source License
|
|
185
|
+
├── package.json # CLI package manifest
|
|
186
|
+
│
|
|
187
|
+
├── bin/ # CLI execution entrypoints
|
|
188
|
+
│ └── ai-engineering-loop.js # npx executable CLI (init, status, refresh, run)
|
|
189
|
+
│
|
|
190
|
+
├── lib/ # Core orchestration & decision engine
|
|
191
|
+
│ └── orchestration.js # 3-stage capability registry, barrier builder, Judge engine
|
|
192
|
+
│
|
|
193
|
+
├── tests/ # Deterministic test suites
|
|
194
|
+
│ ├── capability-selection.test.js # Unit tests for capability lifecycle & truthful selection
|
|
195
|
+
│ └── orchestration.test.js # Tests for isolation, Finding schema, Judge matrix
|
|
196
|
+
│
|
|
197
|
+
├── core/ # Generic engineering loop specifications
|
|
198
|
+
│ ├── orchestration-model.md # 3-stage capability lifecycle & execution priority
|
|
199
|
+
│ ├── project-initialization.md # Auto-discovery & initialization lifecycle
|
|
200
|
+
│ ├── context-refresh-policy.md # Progressive drift hierarchy & living baseline
|
|
201
|
+
│ ├── context-impact-assessment.md # Post-task impact assessment (NONE, TARGETED, MAJOR)
|
|
202
|
+
│ ├── goal-contract.md # Task contract schema & acceptance criteria
|
|
203
|
+
│ ├── verification-loop.md # Dual-layer verification & Evidence Contract
|
|
204
|
+
│ ├── definition-of-done.md # 5 pillars of Done & rejection triggers
|
|
205
|
+
│ ├── iteration-policy.md # Bounded autonomous loop (MAX_ITERATIONS = 3)
|
|
206
|
+
│ ├── escalation-policy.md # Deterministic human escalation triggers
|
|
207
|
+
│ ├── judge-policy.md # Evaluation rules, triage audit, & verdicts
|
|
208
|
+
│ ├── configuration-precedence.md # 5-layer precedence & conflict resolution
|
|
209
|
+
│ └── repo-config-schema.md # Schema for target repo .ai-engineering-loop/
|
|
210
|
+
│
|
|
211
|
+
├── profiles/ # Project archetype profiles
|
|
212
|
+
│ ├── README.md # Profile catalog & auto-detection rules
|
|
213
|
+
│ ├── web-app.md # Frontend web applications
|
|
214
|
+
│ ├── backend-api.md # Backend APIs & microservices
|
|
215
|
+
│ ├── mobile-app.md # Native & cross-platform mobile apps
|
|
216
|
+
│ ├── library.md # Reusable SDKs & shared packages
|
|
217
|
+
│ └── monorepo.md # Multi-package monorepo workspaces
|
|
218
|
+
│
|
|
219
|
+
├── agents/ # Triad agent role specifications
|
|
220
|
+
│ ├── maker.md # Maker agent: surgical diffs & unit tests
|
|
221
|
+
│ ├── devil-advocate.md # Adversarial reviewer: dual-axis finding ledger & diffs
|
|
222
|
+
│ └── judge.md # Judge agent: impartial magistrate on Validity + Severity
|
|
223
|
+
│
|
|
224
|
+
├── policies/ # Operational schemas & algorithms
|
|
225
|
+
│ ├── discovery-safety-policy.md # Secret protection & non-destructive discovery rules
|
|
226
|
+
│ ├── finding-policy.md # Dual-axis finding schema & severity matrix
|
|
227
|
+
│ ├── evidence-policy.md # 5-level evidence hierarchy & Verification Evidence Contract
|
|
228
|
+
│ └── no-progress-policy.md # Finding signature hashing & stagnation detection
|
|
229
|
+
│
|
|
230
|
+
├── adapters/ # Pluggable delivery pipelines
|
|
231
|
+
│ └── dot/ # DOT Indonesia delivery adapter
|
|
232
|
+
│ ├── README.md # DOT adapter overview
|
|
233
|
+
│ ├── gitlab.md # glab CLI, issue cards, & MR generation
|
|
234
|
+
│ ├── multi-branch.md # main / staging / develop cherry-pick propagation
|
|
235
|
+
│ ├── coreview.md # @coreview-bot external review triage (Valid vs Halu)
|
|
236
|
+
│ └── mattermost.md # Channel mapping & MCP dispatch (from: "AI Agent")
|
|
237
|
+
│
|
|
238
|
+
└── templates/ # Starter templates for target repositories
|
|
239
|
+
└── repo-config/ # Ready-to-copy .ai-engineering-loop/ files
|
|
240
|
+
├── config.md # Project identity & profile binding
|
|
241
|
+
├── architecture.md # Layers & boundary invariants
|
|
242
|
+
├── conventions.md # Code standards & forbidden patterns
|
|
243
|
+
├── verification.md # CLI test/lint/build commands
|
|
244
|
+
└── adapter.md # Configured release pipeline
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
---
|
|
248
|
+
|
|
249
|
+
## License
|
|
250
|
+
|
|
251
|
+
This project is licensed under the **MIT License** — see the [LICENSE](LICENSE) file for details.
|
package/README.md
CHANGED
|
@@ -2,62 +2,55 @@
|
|
|
2
2
|
|
|
3
3
|
[](https://www.npmjs.com/package/ai-engineering-loop)
|
|
4
4
|
[](https://opensource.org/licenses/MIT)
|
|
5
|
-
[](https://github.com/egagofur/ai-engineering-loop)
|
|
6
5
|
|
|
7
|
-
**A Reusable, Framework-Agnostic AI Engineering Operating System for Autonomous Coding Agents
|
|
6
|
+
**A Reusable, Framework-Agnostic AI Engineering Operating System for Autonomous Coding Agents.**
|
|
8
7
|
|
|
9
|
-
|
|
8
|
+
Features living project context, strict verification evidence contracts, 3-stage capability lifecycle registry, and dual-axis Judge evaluation.
|
|
10
9
|
|
|
11
10
|
---
|
|
12
11
|
|
|
13
|
-
##
|
|
12
|
+
## ⚡ Quick Start
|
|
14
13
|
|
|
15
|
-
|
|
14
|
+
You can initialize and manage `.ai-engineering-loop/` context directly using `npx`:
|
|
16
15
|
|
|
17
16
|
```bash
|
|
18
|
-
#
|
|
17
|
+
# Bootstrap .ai-engineering-loop/ context from repository discovery
|
|
19
18
|
npx ai-engineering-loop init
|
|
20
19
|
|
|
21
|
-
#
|
|
20
|
+
# Check the validity, readiness, and baseline freshness of context
|
|
22
21
|
npx ai-engineering-loop status
|
|
23
22
|
|
|
24
|
-
#
|
|
23
|
+
# Reconcile drifted context against repository non-destructively
|
|
25
24
|
npx ai-engineering-loop refresh
|
|
26
25
|
|
|
27
|
-
#
|
|
26
|
+
# Verify context readiness and begin engineering loop
|
|
28
27
|
npx ai-engineering-loop run
|
|
29
28
|
```
|
|
30
29
|
|
|
31
30
|
---
|
|
32
31
|
|
|
33
|
-
##
|
|
32
|
+
## 🚀 Key Architectural Features
|
|
34
33
|
|
|
35
|
-
1. **
|
|
36
|
-
2. **
|
|
37
|
-
3. **
|
|
38
|
-
|
|
34
|
+
1. **Living Project Context**: Tracks baseline git SHA and manifest checksums in `metadata.json` for 0ms drift verification and post-task impact assessment (`NONE`, `TARGETED`, `MAJOR`).
|
|
35
|
+
2. **Deterministic Verification Evidence Contract**: Rejects vague assertions ("command was launched"); strictly requires CLI exit code 0, machine logs, and assertion proofs.
|
|
36
|
+
3. **Runtime Capability Registry (3-Stage Lifecycle)**:
|
|
37
|
+
$$\text{CONFIGURATION\_SUPPORTED} \longrightarrow \text{INVOCATION\_AVAILABLE} \longrightarrow \text{EXECUTION\_PROVEN}$$
|
|
38
|
+
Prevents misleading claims of multi-agent execution by requiring proven child session model execution.
|
|
39
|
+
4. **Dual-Axis Finding Model & Judge Matrix**:
|
|
40
|
+
- `VALID + BLOCKER/HIGH` → `ITERATE`
|
|
41
|
+
- `VALID + MEDIUM/LOW` → `ACCEPT / TRADEOFF`
|
|
42
|
+
- `INVALID` → `DISMISS`
|
|
39
43
|
|
|
40
44
|
---
|
|
41
45
|
|
|
42
|
-
##
|
|
46
|
+
## 📖 Full Documentation & Specifications
|
|
43
47
|
|
|
44
|
-
|
|
48
|
+
For complete specifications, agent role definitions, project profiles, and reference walkthroughs, visit the GitHub repository:
|
|
45
49
|
|
|
46
|
-
|
|
47
|
-
- `/ai-engineering-loop status` — Check context health and living freshness.
|
|
48
|
-
- `/ai-engineering-loop refresh` — Reconcile drifted context files.
|
|
49
|
-
- `/ai-engineering-loop <task>` — Execute the full 8-stage engineering lifecycle.
|
|
50
|
+
👉 **[https://github.com/egagofur/ai-engineering-loop](https://github.com/egagofur/ai-engineering-loop)**
|
|
50
51
|
|
|
51
52
|
---
|
|
52
53
|
|
|
53
|
-
##
|
|
54
|
+
## 📄 License
|
|
54
55
|
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
👉 **[Visit the GitHub Repository](https://github.com/egagofur/ai-engineering-loop)**
|
|
58
|
-
|
|
59
|
-
---
|
|
60
|
-
|
|
61
|
-
## License
|
|
62
|
-
|
|
63
|
-
MIT License — see [LICENSE](https://github.com/egagofur/ai-engineering-loop/blob/main/LICENSE) for details.
|
|
56
|
+
MIT © [Ega Gofur](https://github.com/egagofur)
|
package/agents/devil-advocate.md
CHANGED
|
@@ -1,111 +1,73 @@
|
|
|
1
1
|
# Devil's Advocate Agent Specification
|
|
2
2
|
|
|
3
|
-
## 1. Role &
|
|
4
|
-
|
|
5
|
-
The **Devil's Advocate Agent** is the independent adversarial reviewer of the AI Engineering Loop. Its mission is to rigorously challenge the implementation, expose unhandled edge cases, question unstated assumptions, and detect architectural, security, and correctness vulnerabilities.
|
|
6
|
-
|
|
7
|
-
```mermaid
|
|
8
|
-
flowchart TD
|
|
9
|
-
Diff[Git Diff Against Base] --> DA[Devil's Advocate Agent]
|
|
10
|
-
GC[Goal Contract] --> DA
|
|
11
|
-
|
|
12
|
-
subgraph LayeredRules [Layered Review Rule Resolution]
|
|
13
|
-
G[1. Generic Engineering Invariants]
|
|
14
|
-
P[2. Project Profile Rules e.g. backend-api / web-app]
|
|
15
|
-
C[3. Repository-Local Invariants .ai-engineering-loop/]
|
|
16
|
-
T[4. Task-Specific Focus Areas]
|
|
17
|
-
G --> P --> C --> T
|
|
18
|
-
end
|
|
19
|
-
|
|
20
|
-
LayeredRules --> DA
|
|
21
|
-
DA --> FilteredReview[Targeted, Relevant Review Evaluation]
|
|
22
|
-
FilteredReview --> Findings[Standardized Finding Ledger]
|
|
23
|
-
```
|
|
24
|
-
|
|
25
|
-
---
|
|
3
|
+
## 1. Role Definition & Mandate
|
|
26
4
|
|
|
27
|
-
|
|
5
|
+
The **Devil's Advocate** is an independent adversarial reviewer. Its sole purpose is to find real, substantive flaws in the Maker's code diff before production merge:
|
|
28
6
|
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
- The Devil's Advocate **MUST NOT** directly edit codebase files or apply fixes. Its output is exclusively an evidence-backed **Finding Ledger**.
|
|
33
|
-
3. **Evidence-Backed Criticism Only**:
|
|
34
|
-
- Every critique must be substantiated with concrete repository facts, line references, or reproducible scenarios.
|
|
35
|
-
4. **Concrete Diffs Required**:
|
|
36
|
-
- For every substantive problem identified, the Devil's Advocate must provide a concrete code snippet or diff illustrating the recommended fix. Abstract complaints without constructive alternatives are rejected.
|
|
7
|
+
- **Primary Goal**: Detect correctness bugs, unhandled race conditions, missing error paths, auth/security flaws, and test gaps.
|
|
8
|
+
- **Strict Invariant**: The Devil's Advocate is strictly **read-only**. It **NEVER modifies application source code** or git branches.
|
|
9
|
+
- **Concrete Diffs Required**: Every substantive finding must provide a concrete code diff showing the fix, not just an abstract complaint.
|
|
37
10
|
|
|
38
11
|
---
|
|
39
12
|
|
|
40
|
-
##
|
|
41
|
-
|
|
42
|
-
To prevent noisy or irrelevant reviews, the Devil's Advocate dynamically resolves its active review categories using four layers:
|
|
43
|
-
|
|
44
|
-
$$\text{Generic Rules} + \text{Project Profile Rules} + \text{Repository Invariants} + \text{Task Concerns}$$
|
|
45
|
-
|
|
46
|
-
### Layer 1: Generic Review Rules (Always Active)
|
|
47
|
-
1. **Correctness & Logic Integrity**: Bugs, calculation errors, off-by-one errors, broken lifecycle logic.
|
|
48
|
-
2. **Error Handling & Failure Modes**: Swallowed exceptions, unhandled Promise rejections, missing fallbacks.
|
|
49
|
-
3. **Regression Risk**: Breaking existing functionality or altering un-targeted behaviors.
|
|
50
|
-
4. **Maintainability & Architecture**: Violating existing layer boundaries, introducing dead code.
|
|
51
|
-
5. **Testing Gaps**: Untested boundary conditions, weak/tautological assertions.
|
|
13
|
+
## 2. Review Execution Modes
|
|
52
14
|
|
|
53
|
-
|
|
54
|
-
- **`web-app` Profile**: UI responsiveness (320px–4k), accessibility (a11y/ARIA), client state lifecycle, Core Web Vitals, XSS/CSRF.
|
|
55
|
-
- **`backend-api` Profile**: IDOR & authorization scopes, database transactions & ACID rollbacks, concurrency locks (`SELECT FOR UPDATE`), N+1 query loops, API contract backwards compatibility.
|
|
56
|
-
- **`mobile-app` Profile**: Offline mutation queuing, OS lifecycle termination & state loss, permission denials, battery/GPS hygiene, secure hardware storage.
|
|
57
|
-
- **`library` Profile**: Semantic versioning & public API breaking changes, dependency bloat, cross-runtime compatibility (Node vs Browser), tree-shaking exports.
|
|
58
|
-
- **`monorepo` Profile**: Cross-package boundary leaps, workspace dependency isolation, circular package dependencies.
|
|
15
|
+
The Devil's Advocate executes under one of 4 runtime modes depending on platform capabilities:
|
|
59
16
|
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
- Specific risk areas declared in the [Goal Contract](file:///Users/egagofur/Development/work/ai-engineering-loop/core/goal-contract.md).
|
|
17
|
+
1. **`NATIVE_SUBAGENT`**: Genuine independent sub-agent session spawned by host runtime.
|
|
18
|
+
2. **`SDK_AGENT`**: Programmatic Python SDK agent instance with isolated memory.
|
|
19
|
+
3. **`HEADLESS_SUBPROCESS`**: Fresh subprocess agent spawned via CLI.
|
|
20
|
+
4. **`ARTIFACT_ISOLATED_REVIEW`**: Clean-Slate Artifact Barrier in single-agent session (*strictly labeled: isolated review context, not independent agent execution*).
|
|
65
21
|
|
|
66
22
|
---
|
|
67
23
|
|
|
68
|
-
##
|
|
24
|
+
## 3. Input Barrier (What Reviewer Sees)
|
|
69
25
|
|
|
70
|
-
|
|
26
|
+
The Devil's Advocate receives **only** the objective artifact package:
|
|
27
|
+
- `Goal Contract` (AC-1..N, constraints, out of scope).
|
|
28
|
+
- `Project Context` (`.ai-engineering-loop/`: `architecture.md`, `conventions.md`, `verification.md`).
|
|
29
|
+
- `Pure Git Diff` (`git diff <base>...HEAD`).
|
|
30
|
+
- `Deterministic Verification Logs` (exit code 0 proof).
|
|
31
|
+
- `Prior Finding Signatures`.
|
|
71
32
|
|
|
72
|
-
|
|
73
|
-
- *"Rename this local variable because I prefer shorter names."*
|
|
74
|
-
- *"We could rewrite this with a complex functional programming pattern."*
|
|
75
|
-
- *"Consider adding a generic factory pattern for future use."*
|
|
76
|
-
- *Checking mobile responsive layout on a backend API PR.*
|
|
33
|
+
*Maker conversational history, intermediate attempts, and author rationalizations are strictly excluded.*
|
|
77
34
|
|
|
78
|
-
|
|
79
|
-
- *"Line 42 accesses `user.profile.id` without checking if `profile` is null, causing runtime crash when legacy accounts log in."*
|
|
80
|
-
- *"The query in `get-attendance.ts` loads all 50,000 rows into memory without pagination."*
|
|
35
|
+
---
|
|
81
36
|
|
|
82
|
-
|
|
83
|
-
|
|
37
|
+
## 4. Output Contract: Structured Finding Ledger
|
|
38
|
+
|
|
39
|
+
The Devil's Advocate outputs a strictly structured Finding Ledger:
|
|
40
|
+
|
|
41
|
+
```json
|
|
42
|
+
{
|
|
43
|
+
"iteration": 1,
|
|
44
|
+
"executionMode": "ARTIFACT_ISOLATED_REVIEW",
|
|
45
|
+
"diffHash": "437bbfcb",
|
|
46
|
+
"findings": [
|
|
47
|
+
{
|
|
48
|
+
"id": "DA-01",
|
|
49
|
+
"topic": "correctness",
|
|
50
|
+
"validity": "VALID",
|
|
51
|
+
"severity": "BLOCKER",
|
|
52
|
+
"disposition": "STRONG",
|
|
53
|
+
"location": "src/services/queue.ts#L45-L60",
|
|
54
|
+
"acceptanceCriteria": "AC-1",
|
|
55
|
+
"failureScenario": "When the network disconnects between line 48 and 52, the transaction aborts but the queue message is not acknowledged, creating an unrecoverable poison pill.",
|
|
56
|
+
"reproduction": "Inject socket disconnect during executePaymentTransaction step.",
|
|
57
|
+
"evidence": "Observed missing error catch and dead-letter queue routing in queue.ts.",
|
|
58
|
+
"concreteAlternativeDiff": "```diff\n- queue.process(msg)\n+ try { await queue.process(msg); } catch (err) { await dlq.send(msg, err); }\n```"
|
|
59
|
+
}
|
|
60
|
+
]
|
|
61
|
+
}
|
|
62
|
+
```
|
|
84
63
|
|
|
85
64
|
---
|
|
86
65
|
|
|
87
|
-
## 5.
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
#### Finding: [ID] - [Short Title]
|
|
97
|
-
- **Severity**: `[CRITICAL | HIGH | MEDIUM | LOW]`
|
|
98
|
-
- **Category**: `[Correctness | Error Handling | Security | Concurrency | Performance | Maintainability | Testing Gaps]`
|
|
99
|
-
- **Location**: `path/to/file.ts:L20-L35`
|
|
100
|
-
- **Evidence**: [Raw code snippet or runtime behavior]
|
|
101
|
-
- **Problem**: [Technical explanation of the defect]
|
|
102
|
-
- **Impact**: [Concrete business or operational consequence]
|
|
103
|
-
- **Recommendation**:
|
|
104
|
-
```diff
|
|
105
|
-
- unsafeFunction(data);
|
|
106
|
-
+ if (data) {
|
|
107
|
-
+ safeFunction(data);
|
|
108
|
-
+ }
|
|
109
|
-
```
|
|
110
|
-
- **Confidence**: `[HIGH | MEDIUM | LOW]`
|
|
111
|
-
```
|
|
66
|
+
## 5. Review Priority & Topics
|
|
67
|
+
|
|
68
|
+
1. **Correctness & Logic**: Race conditions, unhandled edge cases, data corruption.
|
|
69
|
+
2. **Error Handling & Failure Modes**: Swallowed errors, unhandled promise rejections, missing rollbacks.
|
|
70
|
+
3. **Security & Permissions**: IDOR, auth bypass, unsanitized inputs, credential leakage.
|
|
71
|
+
4. **Concurrency & DB Locking**: Missing transactions, dirty reads, missing row locks.
|
|
72
|
+
5. **Performance**: N+1 queries, memory leaks, excessive allocations.
|
|
73
|
+
6. **Testing Gaps**: Untested failure branches, brittle assertions, missing edge cases.
|
|
@@ -15,7 +15,7 @@ const path = require('path');
|
|
|
15
15
|
const crypto = require('crypto');
|
|
16
16
|
const { execSync } = require('child_process');
|
|
17
17
|
|
|
18
|
-
const VERSION = '1.0.
|
|
18
|
+
const VERSION = '1.0.2';
|
|
19
19
|
const CWD = process.cwd();
|
|
20
20
|
const CONTEXT_DIR = path.join(CWD, '.ai-engineering-loop');
|
|
21
21
|
|