@zigrivers/scaffold 3.11.0 → 3.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +6 -5
- package/content/knowledge/core/automated-review-tooling.md +137 -140
- package/content/knowledge/core/multi-model-research-dispatch.md +27 -16
- package/content/knowledge/core/multi-model-review-dispatch.md +47 -6
- package/content/skills/multi-model-dispatch/SKILL.md +20 -22
- package/content/tools/post-implementation-review.md +71 -26
- package/content/tools/review-code.md +37 -11
- package/content/tools/review-pr.md +65 -23
- package/dist/cli/commands/build.test.js +1 -0
- package/dist/cli/commands/build.test.js.map +1 -1
- package/dist/cli/commands/init.d.ts.map +1 -1
- package/dist/cli/commands/init.js +88 -77
- package/dist/cli/commands/init.js.map +1 -1
- package/dist/cli/commands/init.test.js +30 -0
- package/dist/cli/commands/init.test.js.map +1 -1
- package/dist/cli/commands/knowledge.test.js +1 -0
- package/dist/cli/commands/knowledge.test.js.map +1 -1
- package/dist/cli/commands/run.test.js +1 -0
- package/dist/cli/commands/run.test.js.map +1 -1
- package/dist/cli/commands/skip.test.js +1 -0
- package/dist/cli/commands/skip.test.js.map +1 -1
- package/dist/cli/output/auto.d.ts +19 -6
- package/dist/cli/output/auto.d.ts.map +1 -1
- package/dist/cli/output/auto.js +10 -6
- package/dist/cli/output/auto.js.map +1 -1
- package/dist/cli/output/context.d.ts +23 -5
- package/dist/cli/output/context.d.ts.map +1 -1
- package/dist/cli/output/context.js.map +1 -1
- package/dist/cli/output/context.test.js +585 -0
- package/dist/cli/output/context.test.js.map +1 -1
- package/dist/cli/output/error-display.test.js +1 -0
- package/dist/cli/output/error-display.test.js.map +1 -1
- package/dist/cli/output/interactive.d.ts +19 -6
- package/dist/cli/output/interactive.d.ts.map +1 -1
- package/dist/cli/output/interactive.js +165 -59
- package/dist/cli/output/interactive.js.map +1 -1
- package/dist/cli/output/json.d.ts +19 -6
- package/dist/cli/output/json.d.ts.map +1 -1
- package/dist/cli/output/json.js +10 -6
- package/dist/cli/output/json.js.map +1 -1
- package/dist/core/assembly/overlay-state-resolver.test.js +1 -0
- package/dist/core/assembly/overlay-state-resolver.test.js.map +1 -1
- package/dist/e2e/game-pipeline.test.js +1 -0
- package/dist/e2e/game-pipeline.test.js.map +1 -1
- package/dist/e2e/init.test.js +1 -0
- package/dist/e2e/init.test.js.map +1 -1
- package/dist/e2e/project-type-overlays.test.js +1 -0
- package/dist/e2e/project-type-overlays.test.js.map +1 -1
- package/dist/wizard/copy/backend.d.ts +3 -0
- package/dist/wizard/copy/backend.d.ts.map +1 -0
- package/dist/wizard/copy/backend.js +49 -0
- package/dist/wizard/copy/backend.js.map +1 -0
- package/dist/wizard/copy/browser-extension.d.ts +3 -0
- package/dist/wizard/copy/browser-extension.d.ts.map +1 -0
- package/dist/wizard/copy/browser-extension.js +35 -0
- package/dist/wizard/copy/browser-extension.js.map +1 -0
- package/dist/wizard/copy/cli.d.ts +3 -0
- package/dist/wizard/copy/cli.d.ts.map +1 -0
- package/dist/wizard/copy/cli.js +40 -0
- package/dist/wizard/copy/cli.js.map +1 -0
- package/dist/wizard/copy/core.d.ts +3 -0
- package/dist/wizard/copy/core.d.ts.map +1 -0
- package/dist/wizard/copy/core.js +52 -0
- package/dist/wizard/copy/core.js.map +1 -0
- package/dist/wizard/copy/data-pipeline.d.ts +3 -0
- package/dist/wizard/copy/data-pipeline.d.ts.map +1 -0
- package/dist/wizard/copy/data-pipeline.js +66 -0
- package/dist/wizard/copy/data-pipeline.js.map +1 -0
- package/dist/wizard/copy/game.d.ts +3 -0
- package/dist/wizard/copy/game.d.ts.map +1 -0
- package/dist/wizard/copy/game.js +115 -0
- package/dist/wizard/copy/game.js.map +1 -0
- package/dist/wizard/copy/index.d.ts +8 -0
- package/dist/wizard/copy/index.d.ts.map +1 -0
- package/dist/wizard/copy/index.js +32 -0
- package/dist/wizard/copy/index.js.map +1 -0
- package/dist/wizard/copy/library.d.ts +3 -0
- package/dist/wizard/copy/library.d.ts.map +1 -0
- package/dist/wizard/copy/library.js +44 -0
- package/dist/wizard/copy/library.js.map +1 -0
- package/dist/wizard/copy/ml.d.ts +3 -0
- package/dist/wizard/copy/ml.d.ts.map +1 -0
- package/dist/wizard/copy/ml.js +45 -0
- package/dist/wizard/copy/ml.js.map +1 -0
- package/dist/wizard/copy/mobile-app.d.ts +3 -0
- package/dist/wizard/copy/mobile-app.d.ts.map +1 -0
- package/dist/wizard/copy/mobile-app.js +45 -0
- package/dist/wizard/copy/mobile-app.js.map +1 -0
- package/dist/wizard/copy/types.d.ts +60 -0
- package/dist/wizard/copy/types.d.ts.map +1 -0
- package/dist/wizard/copy/types.js +2 -0
- package/dist/wizard/copy/types.js.map +1 -0
- package/dist/wizard/copy/types.test-d.d.ts +2 -0
- package/dist/wizard/copy/types.test-d.d.ts.map +1 -0
- package/dist/wizard/copy/types.test-d.js +36 -0
- package/dist/wizard/copy/types.test-d.js.map +1 -0
- package/dist/wizard/copy/web-app.d.ts +3 -0
- package/dist/wizard/copy/web-app.d.ts.map +1 -0
- package/dist/wizard/copy/web-app.js +46 -0
- package/dist/wizard/copy/web-app.js.map +1 -0
- package/dist/wizard/questions.d.ts.map +1 -1
- package/dist/wizard/questions.js +87 -53
- package/dist/wizard/questions.js.map +1 -1
- package/dist/wizard/questions.test.js +3 -2
- package/dist/wizard/questions.test.js.map +1 -1
- package/dist/wizard/wizard.test.js +70 -0
- package/dist/wizard/wizard.test.js.map +1 -1
- package/package.json +1 -1
- package/skills/multi-model-dispatch/SKILL.md +20 -22
package/README.md
CHANGED
|
@@ -10,7 +10,7 @@ Scaffold is a composable meta-prompt pipeline built for [Claude Code](https://do
|
|
|
10
10
|
|
|
11
11
|
Here's how it works:
|
|
12
12
|
|
|
13
|
-
1. **Initialize** — run `scaffold init` in your project directory. The init wizard detects whether you're starting fresh (greenfield) or working with an existing codebase (brownfield), and lets you pick a methodology preset (deep, mvp, or custom).
|
|
13
|
+
1. **Initialize** — run `scaffold init` in your project directory. The init wizard detects whether you're starting fresh (greenfield) or working with an existing codebase (brownfield), and lets you pick a methodology preset (deep, mvp, or custom). Every question shows inline descriptions and friendly labels — type `?` at any choice prompt for detailed help.
|
|
14
14
|
|
|
15
15
|
2. **Run steps** — each step is a composable meta-prompt (a short intent declaration in `content/pipeline/`) that gets assembled at runtime into a full 7-section prompt. The assembly engine injects relevant knowledge base entries, project context from prior steps, methodology settings, and depth-appropriate instructions.
|
|
16
16
|
|
|
@@ -38,7 +38,7 @@ Either way, Scaffold constructs the prompt and the target AI tool does the work.
|
|
|
38
38
|
|
|
39
39
|
**Depth scale** (1-5) — Controls how thorough each step's output is, from "focus on the core deliverable" (1) to "explore all angles, tradeoffs, and edge cases" (5). Depth resolves with 4-level precedence: CLI flag > step override > custom default > preset default.
|
|
40
40
|
|
|
41
|
-
**Multi-model validation** — At depth 4-5, all 19 review and validation steps can dispatch independent reviews to Codex and/or Gemini CLIs. Two independent models catch more blind spots than one. When both CLIs are available, findings are reconciled by confidence level (both agree = high confidence, single model P0 = still actionable).
|
|
41
|
+
**Multi-model validation** — At depth 4-5, all 19 review and validation steps can dispatch independent reviews to Codex and/or Gemini CLIs. Two independent models catch more blind spots than one. When both CLIs are available, findings are reconciled by confidence level (both agree = high confidence, single model P0 = still actionable). When a channel is unavailable, a compensating Claude self-review pass runs in its place (labeled `[compensating: Codex-equivalent]` or `[compensating: Gemini-equivalent]`, single-source confidence). CLI commands must always run in the foreground — background execution produces empty output. See the [Multi-Model Review](#multi-model-review) section.
|
|
42
42
|
|
|
43
43
|
**State management** — Pipeline progress is tracked in `.scaffold/state.json` with atomic file writes and crash recovery. An advisory lock prevents concurrent runs. Decisions are logged to an append-only `decisions.jsonl`.
|
|
44
44
|
|
|
@@ -217,7 +217,7 @@ git init
|
|
|
217
217
|
scaffold init
|
|
218
218
|
```
|
|
219
219
|
|
|
220
|
-
The init wizard detects that this is a brand new project and
|
|
220
|
+
The init wizard detects that this is a brand new project and walks you through setup with friendly labels and inline descriptions for every option. Type `?` at any choice prompt for detailed guidance. Choose **mvp** if you want to get to working code fast — it runs only 7 critical steps instead of the full 60. You can always switch to `deep` or `custom` later.
|
|
221
221
|
|
|
222
222
|
**Open Claude Code in your project directory**, then start talking:
|
|
223
223
|
|
|
@@ -931,12 +931,13 @@ mmr review --pr 47 ──→ Dispatches to all channels in background
|
|
|
931
931
|
Agent continues working
|
|
932
932
|
|
|
933
933
|
mmr status mmr-a1b2c3 ──→ Poll progress (which channels done?)
|
|
934
|
-
Exit code: 0=done, 1=running,
|
|
934
|
+
Exit code: 0=done, 1=running, 4=failed
|
|
935
935
|
|
|
936
936
|
mmr results mmr-a1b2c3 ──→ Reconcile findings across channels
|
|
937
|
+
Run compensating passes for unavailable channels
|
|
937
938
|
Apply severity gate
|
|
938
939
|
Output unified findings
|
|
939
|
-
Exit code: 0=passed,
|
|
940
|
+
Exit code: 0=passed, 2=gate failed, 3=degraded
|
|
940
941
|
```
|
|
941
942
|
|
|
942
943
|
**Key features:**
|
|
@@ -10,194 +10,191 @@ Automated PR review leverages AI models to provide consistent, thorough code rev
|
|
|
10
10
|
|
|
11
11
|
## Summary
|
|
12
12
|
|
|
13
|
-
###
|
|
13
|
+
### Review Severity and Reconciliation
|
|
14
14
|
|
|
15
|
-
|
|
16
|
-
- **No CI secrets required** — models run locally via CLI tools
|
|
17
|
-
- **Dual-model review** — run Codex and Gemini (when available) for independent perspectives
|
|
18
|
-
- **Agent-managed loop** — Claude orchestrates the review-fix cycle locally
|
|
15
|
+
See `review-methodology` for severity definitions (P0-P3). See `multi-model-review-dispatch` for finding reconciliation rules.
|
|
19
16
|
|
|
20
|
-
|
|
21
|
-
- `AGENTS.md` — reviewer instructions with project-specific rules
|
|
22
|
-
- `docs/review-standards.md` — severity definitions (P0-P3) and criteria
|
|
23
|
-
- `scripts/cli-pr-review.sh` — dual-model review script
|
|
24
|
-
- `scripts/await-pr-review.sh` — polling script for external bot mode
|
|
17
|
+
**Action thresholds:** P0/P1/P2 findings must be fixed before proceeding to the next task. P3 findings are recorded but not actioned.
|
|
25
18
|
|
|
26
|
-
###
|
|
19
|
+
### Degraded-Mode Behavior
|
|
27
20
|
|
|
28
|
-
|
|
29
|
-
- **P0 (blocking)** — must fix before merge (security, data loss, broken functionality)
|
|
30
|
-
- **P1 (important)** — should fix before merge (bugs, missing tests, performance)
|
|
31
|
-
- **P2 (suggestion)** — consider fixing (style, naming, documentation)
|
|
32
|
-
- **P3 (nit)** — optional (personal preference, minor optimization)
|
|
21
|
+
#### Verdict Definitions
|
|
33
22
|
|
|
34
|
-
|
|
23
|
+
These are the authoritative verdict definitions. Tool files (`review-code.md`, `review-pr.md`) reference these.
|
|
35
24
|
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
25
|
+
| Verdict | Condition |
|
|
26
|
+
|---------|-----------|
|
|
27
|
+
| `pass` | All configured channels ran, no unresolved P0/P1/P2 |
|
|
28
|
+
| `degraded-pass` | Channels skipped, compensated, or have non-full coverage (e.g., partial timeout), no unresolved P0/P1/P2 |
|
|
29
|
+
| `blocked` | Unresolved P0/P1/P2 after 3 fix rounds |
|
|
30
|
+
| `needs-user-decision` | Contradictions or unresolvable findings |
|
|
41
31
|
|
|
42
|
-
|
|
32
|
+
**Verdict precedence:** `needs-user-decision` > `blocked` > `degraded-pass` > `pass`. When multiple conditions apply, the higher-precedence verdict wins.
|
|
43
33
|
|
|
44
|
-
|
|
45
|
-
1. Agent creates PR
|
|
46
|
-
2. Agent runs `scripts/cli-pr-review.sh` (or review runs automatically)
|
|
47
|
-
3. Review findings are posted as PR comments or written to a local file
|
|
48
|
-
4. Agent addresses P0/P1/P2 findings, pushes fixes
|
|
49
|
-
5. Re-review until no P0/P1/P2 findings remain
|
|
50
|
-
6. PR is ready for merge
|
|
34
|
+
**Both external channels missing:** Maximum achievable verdict is `degraded-pass` — never `pass`. Review summary must note: "All findings are single-model (Claude only). External validation was unavailable."
|
|
51
35
|
|
|
52
|
-
|
|
36
|
+
#### Status Model
|
|
53
37
|
|
|
54
|
-
|
|
38
|
+
`compensating` is a **coverage label** applied to a channel's output, not a replacement for the root-cause status. Each channel retains its root-cause status (`not_installed`, `auth_failed`, `auth_timeout`, `failed`) AND gains a coverage label (`compensating (X-equivalent)`) when a compensating pass ran. The fix cycle uses the **root-cause status** to decide whether to retry (never retry `not_installed`, `auth_failed`, `auth_timeout`). The report uses the **coverage label** to show the reader what ran.
|
|
55
39
|
|
|
56
|
-
|
|
40
|
+
#### Compensating Passes
|
|
57
41
|
|
|
58
|
-
|
|
59
|
-
# Code Review Instructions
|
|
42
|
+
When an external channel (Codex or Gemini) is unavailable, run a compensating Claude self-review pass:
|
|
60
43
|
|
|
61
|
-
|
|
62
|
-
[
|
|
44
|
+
- Same prompt structure as the missing channel, executed as a Claude self-review pass.
|
|
45
|
+
- Labeled `[compensating: Codex-equivalent]` or `[compensating: Gemini-equivalent]` in the review summary.
|
|
46
|
+
- Missing Codex → focus on implementation correctness, security, API contracts.
|
|
47
|
+
- Missing Gemini → focus on architectural patterns, design reasoning, broad context.
|
|
48
|
+
- Missing both → two compensating passes (one per missing channel's strength area).
|
|
49
|
+
- Compensating-pass findings are **single-source confidence** — they do NOT raise to high confidence even if they agree with another channel's findings.
|
|
50
|
+
- Normal mandatory-fix thresholds apply: P0/P1/P2 findings from compensating passes still require fixing.
|
|
63
51
|
|
|
64
|
-
|
|
65
|
-
- Security: [project-specific security concerns]
|
|
66
|
-
- Performance: [known hot paths or constraints]
|
|
67
|
-
- Testing: [coverage requirements, test patterns]
|
|
52
|
+
**Superpowers channel:** No compensating pass needed — Superpowers is a Claude subagent and is always available. If the Superpowers plugin is not installed, run available external CLIs and warn the user that review coverage is reduced.
|
|
68
53
|
|
|
69
|
-
|
|
70
|
-
See docs/coding-standards.md for:
|
|
71
|
-
- Naming conventions
|
|
72
|
-
- Error handling patterns
|
|
73
|
-
- Logging standards
|
|
54
|
+
#### Foreground-Only Execution
|
|
74
55
|
|
|
75
|
-
|
|
76
|
-
[Project-specific patterns reviewers should enforce]
|
|
56
|
+
Always run Codex and Gemini CLI commands as foreground Bash calls. Never use `run_in_background`, `&`, or `nohup`. Background execution produces empty or truncated output from Codex and Gemini CLIs. Multiple foreground calls can still run in parallel if the tool runner supports parallel tool invocations.
|
|
77
57
|
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
58
|
+
This constraint is intentionally duplicated from `multi-model-review-dispatch`. Knowledge entries are injected independently by the assembly engine — an agent may receive this entry without `multi-model-review-dispatch`, so both need the constraint.
|
|
59
|
+
|
|
60
|
+
## Deep Guidance
|
|
61
|
+
|
|
62
|
+
### Finding Reconciliation
|
|
63
|
+
|
|
64
|
+
After all channels complete (including compensating passes), reconcile findings using the rules in `multi-model-review-dispatch`. This orchestration entry triggers reconciliation; the dispatch entry defines how to perform it.
|
|
81
65
|
|
|
82
|
-
|
|
66
|
+
Reconciliation normalizes findings from all channels (real and compensating) to a common schema, then matches findings across channels by location and category. The purpose is to detect when multiple independent channels agree on a finding (raising confidence) and to surface contradictions that require human judgment. A finding reported by Codex alone has lower confidence than the same finding reported by both Codex and Gemini.
|
|
83
67
|
|
|
84
|
-
The
|
|
68
|
+
The reconciliation output is a deduplicated list of findings with confidence scores. High-confidence findings (agreed by 2+ real channels) are actionable without further discussion. Low-confidence findings (single-source, or from compensating passes) still require action at P0/P1/P2 but should be noted as lower-confidence in the review summary.
|
|
69
|
+
|
|
70
|
+
Findings that appear in all three channels (Codex, Gemini, Superpowers) are considered maximum-confidence and should be surfaced first in the review summary. Findings that appear in only one channel should include the channel name in the finding description to help the developer assess confidence independently.
|
|
85
71
|
|
|
86
72
|
```bash
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
#
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
# 2. Run Codex review (if available)
|
|
94
|
-
if command -v codex &>/dev/null; then
|
|
95
|
-
codex_findings=$(echo "$diff" | codex review --context AGENTS.md)
|
|
96
|
-
fi
|
|
97
|
-
|
|
98
|
-
# 3. Run Gemini review (if available)
|
|
99
|
-
if command -v gemini &>/dev/null; then
|
|
100
|
-
gemini_findings=$(echo "$diff" | gemini review --context AGENTS.md)
|
|
101
|
-
fi
|
|
102
|
-
|
|
103
|
-
# 4. Reconcile findings
|
|
104
|
-
# - Findings from both models: HIGH confidence
|
|
105
|
-
# - Findings from one model: MEDIUM confidence
|
|
106
|
-
# - Contradictions: flagged for human review
|
|
73
|
+
# Orchestration reconciliation workflow
|
|
74
|
+
# 1. Collect findings from all channels (real + compensating)
|
|
75
|
+
# 2. Normalize to common schema (severity, category, location, description)
|
|
76
|
+
# 3. Match findings across channels by location + category
|
|
77
|
+
# 4. Apply consensus rules from multi-model-review-dispatch
|
|
78
|
+
# 5. Produce reconciled findings list with confidence scores
|
|
107
79
|
```
|
|
108
80
|
|
|
109
|
-
###
|
|
81
|
+
### Channel Dispatch Pattern and Orchestration
|
|
110
82
|
|
|
111
|
-
|
|
112
|
-
- Severity levels with concrete examples per project
|
|
113
|
-
- What constitutes a blocking review (P0/P1/P2 threshold)
|
|
114
|
-
- Auto-approve criteria (when review can be skipped)
|
|
115
|
-
- Review SLA (how long before auto-approve kicks in)
|
|
83
|
+
Each external channel (Codex, Gemini) follows the same dispatch pattern: check installation, check auth, then dispatch as a foreground call. If any step fails, record the root-cause status, queue a compensating pass, and continue to the next channel. The Superpowers channel is always available as a Claude subagent and does not require installation or auth checks.
|
|
116
84
|
|
|
117
|
-
|
|
85
|
+
```bash
|
|
86
|
+
# Channel dispatch pattern
|
|
87
|
+
# For each external channel (Codex, Gemini):
|
|
88
|
+
# 1. command -v <tool> >/dev/null 2>&1 || { status=not_installed; queue_compensating; continue; }
|
|
89
|
+
# 2. <auth_check> || { status=auth_failed; queue_compensating; continue; }
|
|
90
|
+
# 3. <dispatch_foreground> || { status=failed; queue_compensating; continue; }
|
|
91
|
+
# For Superpowers: dispatch subagent (always available)
|
|
92
|
+
# After all: run queued compensating passes → reconcile → verdict
|
|
93
|
+
```
|
|
94
|
+
|
|
95
|
+
After all channels and compensating passes complete, run the reconciliation workflow above and apply the verdict decision flow. Channel results and compensating-pass labels must be preserved in the review output for auditability — do not collapse or omit them even when findings are empty.
|
|
96
|
+
|
|
97
|
+
### Degraded-Mode Worked Example
|
|
118
98
|
|
|
119
|
-
|
|
120
|
-
1. Claude performs an enhanced self-review of the diff
|
|
121
|
-
2. Focus on the AGENTS.md review criteria
|
|
122
|
-
3. Apply the same severity classification
|
|
123
|
-
4. Document that the review was single-model
|
|
99
|
+
When Codex is unavailable (not installed or auth failure), the orchestration proceeds as follows:
|
|
124
100
|
|
|
125
|
-
|
|
101
|
+
1. The installation check (`command -v codex`) fails. Codex channel status is set to `not_installed`.
|
|
102
|
+
2. A compensating Codex-equivalent pass is queued: a Claude self-review focused on implementation correctness, security, and API contracts.
|
|
103
|
+
3. Gemini and Superpowers channels run normally.
|
|
104
|
+
4. The compensating pass runs, producing findings labeled `[compensating: Codex-equivalent]`.
|
|
105
|
+
5. Reconciliation merges findings from all three sources (Gemini, Superpowers, compensating-Codex).
|
|
106
|
+
6. Maximum achievable verdict is `degraded-pass` because a real channel was absent.
|
|
107
|
+
7. The review summary notes: "Codex channel: not_installed (compensating: Codex-equivalent pass ran)."
|
|
126
108
|
|
|
127
|
-
|
|
128
|
-
- Add new review focus areas when new patterns emerge
|
|
129
|
-
- Remove rules that linters now enforce automatically
|
|
130
|
-
- Update AGENTS.md when architecture changes
|
|
131
|
-
- Track false-positive rates and adjust thresholds
|
|
109
|
+
**Fix-cycle channel rule:** Only re-run channels that originally completed or ran as compensating passes. `failed` channels are covered by their compensating pass and are not retried during fix rounds. Never retry a channel with status `not_installed`, `auth_failed`, or `auth_timeout` — these indicate persistent environment conditions that will not resolve between fix rounds.
|
|
132
110
|
|
|
133
|
-
###
|
|
111
|
+
### Verdict Decision Flow
|
|
134
112
|
|
|
135
|
-
|
|
113
|
+
Apply the following evaluation order to determine the final verdict. The first matching condition wins; all subsequent conditions are skipped.
|
|
136
114
|
|
|
137
115
|
```
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
│ Unique finding │ Found │ - │ MEDIUM confidence │
|
|
144
|
-
│ Unique finding │ - │ Found │ MEDIUM confidence │
|
|
145
|
-
│ Contradiction │ Fix X │ Keep X │ Flag for agent │
|
|
146
|
-
└─────────────────┴──────────┴──────────┴───────────────────┘
|
|
116
|
+
Verdict evaluation order:
|
|
117
|
+
1. Any contradictions or unresolvable findings? → needs-user-decision
|
|
118
|
+
2. Any unresolved P0/P1/P2 after 3 fix rounds? → blocked
|
|
119
|
+
3. Any channel not at full coverage? → degraded-pass
|
|
120
|
+
4. All channels completed, no unresolved P0/P1/P2? → pass
|
|
147
121
|
```
|
|
148
122
|
|
|
149
|
-
|
|
123
|
+
A "contradiction" exists when two channels report opposite conclusions about the same code location — for example, Codex flags a function as insecure while Gemini explicitly approves it. Contradictions cannot be resolved by the agent alone and must be surfaced to the user.
|
|
124
|
+
|
|
125
|
+
A channel is "not at full coverage" when: it ran as a compensating pass instead of a real tool, it timed out partially, or the Superpowers plugin is not installed and available channels do not cover the full diff.
|
|
126
|
+
|
|
127
|
+
**Verdict precedence reminder:** `needs-user-decision` > `blocked` > `degraded-pass` > `pass`. If multiple conditions apply simultaneously (for example, both a contradiction and an unresolved P0 exist), the higher-precedence verdict wins.
|
|
128
|
+
|
|
129
|
+
The verdict is always computed after all fix rounds are exhausted — do not emit a partial verdict mid-cycle. If a fix round resolves all P0/P1/P2 findings and no contradictions remain, the verdict upgrades from `blocked` to `pass` or `degraded-pass` depending on channel coverage. This upgrade must be verified explicitly by re-running the reconciliation step after each fix round, not assumed from the fact that fixes were applied.
|
|
150
130
|
|
|
151
131
|
### Security-Focused Review Checklist
|
|
152
132
|
|
|
153
133
|
Every automated review should check:
|
|
154
|
-
- No secrets or credentials in the diff (API keys, passwords, tokens)
|
|
155
|
-
- No `eval()` or equivalent unsafe operations introduced
|
|
156
|
-
- SQL queries use parameterized queries
|
|
157
|
-
- User input is validated before use
|
|
158
|
-
- Authentication/authorization checks are present on new endpoints
|
|
159
|
-
- Dependencies added are from trusted sources with known versions
|
|
134
|
+
- No secrets or credentials in the diff (API keys, passwords, tokens, private keys)
|
|
135
|
+
- No `eval()` or equivalent unsafe operations introduced (dynamic code execution, shell injection)
|
|
136
|
+
- SQL queries use parameterized queries — no string concatenation with user input
|
|
137
|
+
- User input is validated and sanitized before use in queries, commands, or output
|
|
138
|
+
- Authentication/authorization checks are present on all new endpoints and operations
|
|
139
|
+
- Dependencies added are from trusted sources with known, pinned versions
|
|
140
|
+
- No new global state or singletons that could cause cross-request data leaks
|
|
141
|
+
- Error messages do not expose internal paths, stack traces, or sensitive system details
|
|
142
|
+
- File system operations use safe path handling (no path traversal vulnerabilities)
|
|
143
|
+
- Cryptographic operations use approved algorithms and key lengths
|
|
144
|
+
|
|
145
|
+
When reviewing diffs that touch authentication, authorization, or data handling, elevate any security-related finding by one severity level. A finding that would normally be P2 (recommended) becomes P1 (required) in security-sensitive code paths. This conservative stance reflects the asymmetric cost of security failures versus the cost of over-caution during review.
|
|
160
146
|
|
|
161
147
|
### Performance Review Patterns
|
|
162
148
|
|
|
163
|
-
Look for these performance anti-patterns:
|
|
164
|
-
- N+1 queries (loop
|
|
165
|
-
- Missing pagination on list endpoints
|
|
166
|
-
- Synchronous operations that should be async
|
|
167
|
-
- Large objects passed by value instead of reference
|
|
168
|
-
- Missing caching for expensive computations
|
|
169
|
-
- Unbounded growth in arrays or maps
|
|
149
|
+
Look for these performance anti-patterns in the diff:
|
|
150
|
+
- N+1 queries (loop containing individual DB calls — use batch queries or eager loading)
|
|
151
|
+
- Missing pagination on list endpoints (unbounded result sets)
|
|
152
|
+
- Synchronous operations that should be async (blocking I/O in hot paths)
|
|
153
|
+
- Large objects passed by value instead of reference (unnecessary deep copies)
|
|
154
|
+
- Missing caching for expensive computations that are called repeatedly
|
|
155
|
+
- Unbounded growth in arrays or maps (no eviction, no size limits)
|
|
156
|
+
- Missing indexes on columns used in WHERE clauses of new queries
|
|
157
|
+
- Eager loading where lazy loading would suffice (over-fetching)
|
|
158
|
+
- Missing connection pooling or connection reuse for external services
|
|
170
159
|
|
|
171
|
-
###
|
|
160
|
+
### Common False Positives
|
|
172
161
|
|
|
173
|
-
|
|
162
|
+
Track and suppress recurring false positives to reduce noise in future reviews:
|
|
163
|
+
- Test files flagged for "hardcoded values" (test fixtures and expected values are intentional)
|
|
164
|
+
- Migration files flagged for "raw SQL" (migrations must use raw SQL for schema changes)
|
|
165
|
+
- Generated files flagged for style issues (generated code follows its own generator's conventions)
|
|
166
|
+
- Intentional use of `any` types in TypeScript adapter layers or third-party type overrides
|
|
167
|
+
- Deliberate `eslint-disable` comments that are already justified in surrounding context
|
|
168
|
+
- Seed data files flagged for hardcoded credentials (test-only, not production)
|
|
174
169
|
|
|
175
|
-
|
|
176
|
-
## Code Review
|
|
177
|
-
| Command | Purpose |
|
|
178
|
-
|---------|---------|
|
|
179
|
-
| `scripts/cli-pr-review.sh <PR#>` | Run dual-model review |
|
|
180
|
-
| `scripts/await-pr-review.sh <PR#>` | Poll for external review |
|
|
181
|
-
```
|
|
170
|
+
Add suppressions to AGENTS.md under "Out of Scope" to prevent repeated false findings across review cycles.
|
|
182
171
|
|
|
183
|
-
|
|
172
|
+
### Review Metrics and Continuous Improvement
|
|
184
173
|
|
|
185
|
-
|
|
174
|
+
Track these metrics over time to improve review quality and calibrate thresholds:
|
|
186
175
|
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
176
|
+
| Metric | Definition | Use |
|
|
177
|
+
|--------|------------|-----|
|
|
178
|
+
| False positive rate | Findings dismissed without action / total findings | Calibrate severity thresholds |
|
|
179
|
+
| Escape rate | Bugs reaching production despite review / total bugs | Identify coverage gaps |
|
|
180
|
+
| Time to resolve | Average time between finding logged and fix merged | Identify bottlenecks |
|
|
181
|
+
| Coverage | PRs receiving automated review / total PRs merged | Track adoption |
|
|
182
|
+
| Model agreement rate | Findings agreed by 2+ channels / total findings | Tune reconciliation rules |
|
|
183
|
+
| Compensating-pass rate | Reviews using compensating passes / total reviews | Track environment health |
|
|
191
184
|
|
|
192
|
-
|
|
185
|
+
Use the false positive rate to determine whether a severity category is over-triggering. Use the escape rate to determine whether the review is missing entire classes of bugs. Use the compensating-pass rate to identify when the review environment needs maintenance (expired auth tokens, broken CLI installs).
|
|
193
186
|
|
|
194
|
-
|
|
187
|
+
Log metric snapshots in AGENTS.md after each major project milestone. A declining model agreement rate over time suggests either that the review prompts are drifting in quality or that the codebase is accumulating technical debt in areas where models diverge. A rising escape rate despite consistent review coverage is a signal to revisit the severity thresholds or the focus areas in the review prompts.
|
|
188
|
+
|
|
189
|
+
### Fallback When Models Unavailable
|
|
190
|
+
|
|
191
|
+
When external CLIs are unavailable, the degraded-mode behavior defined in the Summary section applies. To summarize the operational steps:
|
|
195
192
|
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
-
|
|
199
|
-
-
|
|
200
|
-
-
|
|
201
|
-
|
|
193
|
+
1. For each unavailable external channel, queue a compensating Claude self-review pass focused on that channel's strength area.
|
|
194
|
+
2. Label findings as `[compensating: Codex-equivalent]` or `[compensating: Gemini-equivalent]`.
|
|
195
|
+
3. Treat compensating findings as single-source confidence — they do not raise to high confidence even when they agree with another channel.
|
|
196
|
+
4. Maximum verdict is `degraded-pass` when any channel ran as compensating instead of real.
|
|
197
|
+
5. When both external channels are unavailable, note "All findings are single-model (Claude only). External validation was unavailable." in the review summary.
|
|
198
|
+
6. Never silently drop unavailable channels — always record the channel status and compensating coverage label in the review output.
|
|
202
199
|
|
|
203
|
-
|
|
200
|
+
**Superpowers channel exception:** Superpowers is a Claude subagent and requires no external CLI or auth. It is always available as long as the Superpowers plugin is installed in the Claude Code environment. If the plugin is not installed, run available external CLIs and warn the user that review coverage is reduced — but do not run a compensating pass for Superpowers (the compensating-pass mechanism only applies to external CLIs that have an installation/auth gate).
|
|
@@ -18,12 +18,13 @@ At higher methodology depths (4+), idea exploration and adversarial challenge be
|
|
|
18
18
|
| 5 | Multi-model with reconciliation | Multi-model with reconciliation |
|
|
19
19
|
|
|
20
20
|
### Graceful Fallback Chain
|
|
21
|
-
1. Check if external CLI is available (`
|
|
22
|
-
2. If
|
|
23
|
-
3. If auth
|
|
24
|
-
4. If
|
|
25
|
-
5. If
|
|
26
|
-
6.
|
|
21
|
+
1. Check if external CLI is available (`command -v codex`, `command -v gemini`)
|
|
22
|
+
2. If not installed, skip that model silently — note in Session Metadata
|
|
23
|
+
3. If installed, check auth (`codex login status`, `NO_BROWSER=true gemini -p "respond with ok" -o json`)
|
|
24
|
+
4. If auth fails, surface loudly to the user with `!` recovery command — do NOT silently skip
|
|
25
|
+
5. If auth succeeds, dispatch with timeout
|
|
26
|
+
6. If no external models available, fall back to primary model with distinct framing prompts
|
|
27
|
+
7. Never block the session waiting for unavailable tools
|
|
27
28
|
|
|
28
29
|
### Reconciliation Rules
|
|
29
30
|
- **2+ models agree** on the same finding = **consensus** — high confidence, present as validated
|
|
@@ -36,19 +37,29 @@ At higher methodology depths (4+), idea exploration and adversarial challenge be
|
|
|
36
37
|
|
|
37
38
|
Before dispatching, verify CLI tools are installed and authenticated:
|
|
38
39
|
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
# Exit 0 = ready. Non-zero = skip Codex.
|
|
40
|
+
### Foreground-Only Execution
|
|
41
|
+
|
|
42
|
+
When an AI agent dispatches research or challenge prompts via a tool runner, always run commands in the foreground. Background execution (`run_in_background`, `&`, `nohup`) produces empty or truncated output from Codex and Gemini CLIs. Multiple foreground calls can still run in parallel if the tool runner supports parallel tool invocations.
|
|
43
43
|
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
44
|
+
```bash
|
|
45
|
+
# Codex CLI — step 1: check installed
|
|
46
|
+
command -v codex >/dev/null 2>&1 || { echo "Codex not installed — skipping"; exit 0; }
|
|
47
|
+
# step 2: check auth
|
|
48
|
+
codex login status 2>/dev/null
|
|
49
|
+
# Exit 0 = ready. Non-zero = auth failure (surface to user).
|
|
50
|
+
|
|
51
|
+
# Gemini CLI — step 1: check installed
|
|
52
|
+
command -v gemini >/dev/null 2>&1 || { echo "Gemini not installed — skipping"; exit 0; }
|
|
53
|
+
# step 2: check auth
|
|
54
|
+
NO_BROWSER=true gemini -p "respond with ok" -o json 2>&1
|
|
55
|
+
# Check for "ok" in response. Exit 41 = auth failure (surface to user).
|
|
47
56
|
```
|
|
48
57
|
|
|
49
|
-
|
|
50
|
-
-
|
|
51
|
-
-
|
|
58
|
+
Two distinct failure modes:
|
|
59
|
+
- **Not installed** (`command -v` fails): skip silently, note in Session Metadata
|
|
60
|
+
- **Auth failed** (non-zero after install check): surface loudly — tell the user which tool failed and how to fix it:
|
|
61
|
+
- Codex: "Codex auth expired — run `! codex login` to re-authenticate"
|
|
62
|
+
- Gemini: "Gemini auth expired — run `! gemini -p \"hello\"` to re-authenticate"
|
|
52
63
|
|
|
53
64
|
Auth failures are NOT silent fallbacks — surface them explicitly.
|
|
54
65
|
|
|
@@ -44,21 +44,53 @@ The review never blocks on external model availability.
|
|
|
44
44
|
|
|
45
45
|
## Deep Guidance
|
|
46
46
|
|
|
47
|
+
See `review-methodology` for severity definitions (P0-P3). This entry uses those severities but does not define them.
|
|
48
|
+
|
|
47
49
|
### Dispatch Mechanics
|
|
48
50
|
|
|
51
|
+
#### Foreground-Only Execution
|
|
52
|
+
|
|
53
|
+
When an AI agent dispatches CLI reviews via a tool runner (Claude Code Bash tool, Codex exec, etc.), always run commands in the foreground. Background execution (`run_in_background`, `&`, `nohup`) produces empty or truncated output from Codex and Gemini CLIs. Multiple foreground calls can still run in parallel if the tool runner supports parallel tool invocations.
|
|
54
|
+
|
|
49
55
|
#### CLI Availability Check
|
|
50
56
|
|
|
51
|
-
Before dispatching, verify the model CLI is installed and authenticated:
|
|
57
|
+
Before dispatching, verify the model CLI is installed and authenticated using a two-step process that produces distinct statuses for the orchestration layer:
|
|
58
|
+
|
|
59
|
+
**Step 1 — Installation check:**
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
# Codex: not found -> status: "not_installed"
|
|
63
|
+
command -v codex >/dev/null 2>&1
|
|
64
|
+
|
|
65
|
+
# Gemini: not found -> status: "not_installed"
|
|
66
|
+
command -v gemini >/dev/null 2>&1
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
If the CLI is not found, report status `not_installed` to the orchestration layer. Do not prompt the user to install it.
|
|
70
|
+
|
|
71
|
+
**Step 2 — Auth verification (only if installed):**
|
|
52
72
|
|
|
53
73
|
```bash
|
|
54
|
-
# Codex
|
|
55
|
-
|
|
74
|
+
# Codex: fail -> status: "auth_failed"
|
|
75
|
+
codex login status 2>/dev/null
|
|
56
76
|
|
|
57
|
-
# Gemini
|
|
58
|
-
|
|
77
|
+
# Gemini: exit 41 -> status: "auth_failed"
|
|
78
|
+
NO_BROWSER=true gemini -p "respond with ok" -o json 2>&1
|
|
59
79
|
```
|
|
60
80
|
|
|
61
|
-
If
|
|
81
|
+
If auth fails, report status `auth_failed` and surface recovery to the user:
|
|
82
|
+
- Codex: "Codex auth expired — run `! codex login` to re-authenticate"
|
|
83
|
+
- Gemini: "Gemini auth expired — run `! gemini -p \"hello\"` to re-authenticate"
|
|
84
|
+
|
|
85
|
+
If auth check times out (~5 seconds), retry once. If still failing, report `auth_timeout`.
|
|
86
|
+
If auth succeeds, report `ready` and proceed to dispatch.
|
|
87
|
+
|
|
88
|
+
**Post-dispatch terminal states:**
|
|
89
|
+
- `completed` — channel produced results, use normally
|
|
90
|
+
- `partial_timeout` — partial output before timeout; use what was received, note incompleteness. Does NOT trigger compensating pass.
|
|
91
|
+
- `failed` — crashed or unparseable output; triggers compensating pass.
|
|
92
|
+
|
|
93
|
+
Verdict impact: `partial_timeout` and `failed` channels mean the review is degraded. Maximum verdict is `degraded-pass` when any channel has a non-`completed` terminal state.
|
|
62
94
|
|
|
63
95
|
#### Prompt Formatting
|
|
64
96
|
|
|
@@ -241,6 +273,15 @@ Minimum standards for a multi-model review to be considered complete:
|
|
|
241
273
|
|
|
242
274
|
If the primary Claude review produces zero findings and external models are unavailable, the review should explicitly note this as unusual and recommend a targeted re-review at a later stage.
|
|
243
275
|
|
|
276
|
+
#### Degraded-Mode Gate Adaptation
|
|
277
|
+
|
|
278
|
+
When channels are skipped and compensating passes are used:
|
|
279
|
+
|
|
280
|
+
- **Minimum finding count** gate: compensating passes count toward the total but are not treated as separate external channels for consensus purposes.
|
|
281
|
+
- **Reconciliation completeness** gate (cross-model disagreement documentation): applies whenever 2+ distinct model perspectives participate (Claude + one external counts). N/A only when Claude is the sole perspective (no external models and no compensating passes that introduce genuinely different framing).
|
|
282
|
+
- **Coverage threshold** gate: compensating passes satisfy the "every pass has at least one finding or explicit no-issues note" requirement.
|
|
283
|
+
- The reconciled output must record which channels were real, which were compensating, and which were skipped, so the orchestration layer can apply appropriate verdict logic.
|
|
284
|
+
|
|
244
285
|
### Common Anti-Patterns
|
|
245
286
|
|
|
246
287
|
**Blind trust of external findings.** An external model flags an issue and the reviewer includes it without verification. External models hallucinate — they may flag a "missing section" that actually exists, or cite a "contradiction" based on a misread. Fix: every external finding must be verified against the actual artifact before inclusion in the final report.
|
|
@@ -64,7 +64,7 @@ fi
|
|
|
64
64
|
|
|
65
65
|
The `!` prefix runs the command in the user's terminal session, allowing interactive auth flows (browser OAuth, Y/n prompts) that can't work in headless mode.
|
|
66
66
|
|
|
67
|
-
**If neither CLI is available or authenticated**:
|
|
67
|
+
**If neither CLI is available or authenticated**: Queue a compensating Claude pass focused on the failed channel's strength area. Document this as "single-model review (no external CLIs available)."
|
|
68
68
|
|
|
69
69
|
## Correct Invocation Patterns
|
|
70
70
|
|
|
@@ -134,6 +134,12 @@ NO_BROWSER=true gemini -p "REVIEW_PROMPT_HERE" --output-format json -s --approva
|
|
|
134
134
|
|
|
135
135
|
**Output**: JSON on stdout with `{ response, stats, error }` structure.
|
|
136
136
|
|
|
137
|
+
## Foreground-Only Execution
|
|
138
|
+
|
|
139
|
+
Always run Codex and Gemini CLI commands as foreground Bash calls. Never use `run_in_background`, `&`, or `nohup`. Background execution produces empty or truncated output from both CLIs. Multiple foreground calls in a single message are fine — the tool runner supports parallel invocations.
|
|
140
|
+
|
|
141
|
+
This means: when dispatching reviews, make each CLI call a separate foreground Bash tool invocation. Do NOT use shell `&` or background subshells.
|
|
142
|
+
|
|
137
143
|
## Context Bundling
|
|
138
144
|
|
|
139
145
|
When dispatching a review, bundle all relevant context into the prompt. Each CLI gets the same bundle — do NOT share one model's review with the other.
|
|
@@ -208,35 +214,27 @@ You are reviewing a pull request diff. Report P0, P1, and P2 issues.
|
|
|
208
214
|
| PR diff | Full diff | If >2000 lines, split into file groups |
|
|
209
215
|
| Implementation plan | Task list + representative tasks | Include full task list, detail for flagged tasks |
|
|
210
216
|
|
|
211
|
-
##
|
|
212
|
-
|
|
213
|
-
When both CLIs produce results, reconcile findings using these rules:
|
|
217
|
+
## Finding Reconciliation
|
|
214
218
|
|
|
215
|
-
|
|
216
|
-
|----------|-----------|--------|
|
|
217
|
-
| Both flag same issue | **High** | Fix immediately — two independent models agree |
|
|
218
|
-
| Both approve (no findings) | **High** | Proceed confidently |
|
|
219
|
-
| One flags P0, other approves | **High** | Fix it — P0 is critical enough from a single source |
|
|
220
|
-
| One flags P1, other approves | **Medium** | Review the finding carefully before fixing. If the finding is specific and actionable, fix it. If vague, skip. |
|
|
221
|
-
| Models contradict each other | **Low** | Present both findings to the user for adjudication |
|
|
219
|
+
When multiple models produce findings, reconcile them using the rules defined in `multi-model-review-dispatch`. Key principles:
|
|
222
220
|
|
|
223
|
-
**Independence rule**: Never share one model's review output with the other. Each model must review the artifact independently to avoid confirmation bias.
|
|
221
|
+
- **Independence rule**: Never share one model's review output with the other. Each model must review the artifact independently to avoid confirmation bias.
|
|
222
|
+
- **Round tracking**: For iterative reviews (like PR review loops), track the round number. After 3 fix rounds with unresolved findings, stop and surface the verdict (`blocked` or `needs-user-decision`) to the user. Do NOT auto-merge.
|
|
224
223
|
|
|
225
|
-
|
|
224
|
+
For the full consensus rules, confidence scoring, and disagreement resolution process, see `multi-model-review-dispatch`.
|
|
226
225
|
|
|
227
226
|
## Fallback Behavior
|
|
228
227
|
|
|
229
228
|
| Situation | Fallback |
|
|
230
229
|
|-----------|----------|
|
|
231
|
-
| Neither CLI available |
|
|
232
|
-
| Codex only | Single-model review with Codex |
|
|
233
|
-
| Gemini only | Single-model review with Gemini |
|
|
234
|
-
| **CLI auth expired** | **Surface to user with recovery command — do NOT silently fall back** |
|
|
235
|
-
| One CLI fails mid-review
|
|
236
|
-
| Both CLIs fail
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
**Auth failures are NOT silent fallbacks.** The difference between "CLI not installed" (fall back quietly) and "CLI auth expired" (user action required) is critical. Auth can be fixed in 30 seconds with an interactive command — silently skipping wastes the user's review infrastructure.
|
|
230
|
+
| Neither CLI available | Queue two compensating Claude passes (one per missing channel's strength area). Label findings. Max verdict: `degraded-pass`. |
|
|
231
|
+
| Codex only | Single-model review with Codex + compensating Claude pass for Gemini |
|
|
232
|
+
| Gemini only | Single-model review with Gemini + compensating Claude pass for Codex |
|
|
233
|
+
| **CLI auth expired** | **Surface to user with `!` recovery command — do NOT silently fall back** |
|
|
234
|
+
| One CLI fails mid-review | Use partial results if available, else queue compensating pass. Note failure in summary. |
|
|
235
|
+
| Both CLIs fail | Two compensating passes, max verdict: `degraded-pass`. Warn user. |
|
|
236
|
+
|
|
237
|
+
Auth failures are NOT silent fallbacks.
|
|
240
238
|
|
|
241
239
|
## Integration with Review Steps
|
|
242
240
|
|