forgedock 1.1.7 → 1.1.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bin/engine/invariants.mjs +361 -0
- package/bin/hooks/pre-tool-use.mjs +182 -0
- package/bin/labels.json +10 -0
- package/bin/recall.mjs +540 -0
- package/commands/autopilot.md +957 -76
- package/commands/cleanup.md +52 -3
- package/commands/optimize.md +585 -0
- package/commands/orchestrate/phase-1-resolve.md +35 -25
- package/commands/orchestrate/phase-3-dependency.md +465 -34
- package/commands/orchestrate/phase-4-execution.md +447 -28
- package/commands/orchestrate/phase-6-report.md +198 -9
- package/commands/pipeline-health/phase-2-metrics.md +305 -1
- package/commands/pipeline-health/phase-4-proposals.md +127 -0
- package/commands/quality-gate.md +417 -0
- package/commands/review-pr.md +391 -13
- package/commands/spec-doctor.md +429 -0
- package/commands/validate.md +52 -0
- package/commands/work-on/build/architect.md +77 -0
- package/commands/work-on/build/context.md +357 -22
- package/commands/work-on/build/implement.md +2 -0
- package/commands/work-on/build/validate.md +28 -0
- package/commands/work-on/build.md +44 -0
- package/commands/work-on/close.md +621 -8
- package/commands/work-on/investigate.md +221 -0
- package/commands/work-on.md +15 -14
- package/docs/FORGE-PROTOCOL.md +101 -0
- package/docs/spec/forge-protocol-v1.md +97 -5
- package/docs/spec/label-state-machine.md +6 -1
- package/package.json +2 -2
- package/packages/protocol/src/cli.js +374 -84
- package/packages/protocol/src/types.js +75 -1
- package/scripts/build-knowledge-index.mjs +1761 -0
- package/scripts/calibration.mjs +996 -0
- package/scripts/danger-zones.mjs +782 -0
- package/scripts/doctor-pipeline-state.sh +88 -2
- package/scripts/select-fix-targets.sh +234 -0
- package/scripts/transition-label.sh +157 -6
|
@@ -0,0 +1,429 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: Evidence-backed spec evolution — reads finding concentration for a spec file, drafts a spec diff, opens a spec-evolution PR with per-change citations
|
|
3
|
+
argument-hint: [spec file path | "auto"]
|
|
4
|
+
install: extras
|
|
5
|
+
---
|
|
6
|
+
<!-- SPDX-FileCopyrightText: Copyright (c) RapierCraft Studios -->
|
|
7
|
+
<!-- SPDX-License-Identifier: AGPL-3.0-or-later -->
|
|
8
|
+
|
|
9
|
+
# /spec-doctor — Evidence-Backed Spec Evolution
|
|
10
|
+
|
|
11
|
+
**Input**: $ARGUMENTS
|
|
12
|
+
|
|
13
|
+
Closes the feedback loop from `pipeline-health` finding data into self-improving spec PRs.
|
|
14
|
+
Reads concentrated review findings for a spec file, drafts a concrete spec diff addressing
|
|
15
|
+
the recurring defect classes, and opens a `spec-evolution` PR whose body cites every finding
|
|
16
|
+
that motivated each change. The human stays the authority — **spec-evolution PRs are never
|
|
17
|
+
auto-merged**, and the eval gate must pass before merge is possible.
|
|
18
|
+
|
|
19
|
+
<!-- FORGE:SPEC_LOADED — spec-doctor.md loaded and active. Agent is bound by this spec. -->
|
|
20
|
+
|
|
21
|
+
## HARD RULES — READ BEFORE ANYTHING ELSE
|
|
22
|
+
|
|
23
|
+
1. **NEVER auto-merge a spec-evolution PR.** `gh pr create` must NEVER include `--auto-merge`.
|
|
24
|
+
The eval gate makes the proposal trustworthy; the human stays the authority.
|
|
25
|
+
2. **Every spec change must cite its evidence.** Each changed block in the PR body must list
|
|
26
|
+
the finding issues that motivated it. A spec change with no cited findings is spec drift,
|
|
27
|
+
not spec evolution.
|
|
28
|
+
3. **Do not widen scope beyond the concentrated-finding spec file.** One spec-doctor run =
|
|
29
|
+
one spec file. If adjacent specs also need fixes, create separate issues.
|
|
30
|
+
4. **The spec diff is a proposal, not a decree.** Post it as a PR for human review. Do NOT
|
|
31
|
+
update `commands/` files directly without going through the PR flow.
|
|
32
|
+
5. **NEVER use the Agent tool** — all sub-operations run inline or via `Skill(...)`.
|
|
33
|
+
|
|
34
|
+
---
|
|
35
|
+
|
|
36
|
+
## Phase 0: Resolve Target Spec
|
|
37
|
+
|
|
38
|
+
### 0A: Parse input
|
|
39
|
+
|
|
40
|
+
If `$ARGUMENTS` is a file path matching `commands/*.md`: use that spec file directly.
|
|
41
|
+
|
|
42
|
+
If `$ARGUMENTS` is `"auto"` or empty: auto-select the worst offender from the recurrence
|
|
43
|
+
data (see Phase 1 — run it with `--auto-select` mode and pick the top-ranked spec file).
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
INPUT_ARG="${ARGUMENTS:-auto}"
|
|
47
|
+
|
|
48
|
+
if echo "$INPUT_ARG" | grep -qE '^commands/'; then
|
|
49
|
+
TARGET_SPEC="$INPUT_ARG"
|
|
50
|
+
SELECT_MODE="explicit"
|
|
51
|
+
else
|
|
52
|
+
TARGET_SPEC=""
|
|
53
|
+
SELECT_MODE="auto"
|
|
54
|
+
fi
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
### 0B: Load forge.yaml
|
|
58
|
+
|
|
59
|
+
```bash
|
|
60
|
+
GH_REPO=$(yq '.project.owner + "/" + .project.repo' forge.yaml 2>/dev/null)
|
|
61
|
+
GH_FLAG="-R $GH_REPO"
|
|
62
|
+
REPO_PATH=$(yq '.paths.root' forge.yaml 2>/dev/null || echo ".")
|
|
63
|
+
STAGING_BRANCH=$(yq '.branches.staging // "staging"' forge.yaml 2>/dev/null)
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
### 0C: Ensure spec-evolution label exists
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
gh label create "spec-evolution" \
|
|
70
|
+
--color "BFD4F2" \
|
|
71
|
+
--description "Spec PR backed by eval corpus. Managed by ForgeDock." \
|
|
72
|
+
--force \
|
|
73
|
+
$GH_FLAG 2>/dev/null || true
|
|
74
|
+
```
|
|
75
|
+
|
|
76
|
+
---
|
|
77
|
+
|
|
78
|
+
## Phase 1: Query Finding Recurrence by Spec File
|
|
79
|
+
|
|
80
|
+
### 1A: Collect review findings from the last 30 days
|
|
81
|
+
|
|
82
|
+
```bash
|
|
83
|
+
SINCE=$(date -d "30 days ago" +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \
|
|
84
|
+
|| date -v-30d +%Y-%m-%dT%H:%M:%SZ 2>/dev/null \
|
|
85
|
+
|| python3 -c "import datetime; print((datetime.datetime.utcnow() - datetime.timedelta(days=30)).strftime('%Y-%m-%dT%H:%M:%SZ'))")
|
|
86
|
+
|
|
87
|
+
# Fetch review-finding issues closed in the 30-day window
|
|
88
|
+
FINDINGS_RAW=$(gh issue list $GH_FLAG \
|
|
89
|
+
--label "review-finding" \
|
|
90
|
+
--state closed \
|
|
91
|
+
--limit 200 \
|
|
92
|
+
--json number,title,body,closedAt \
|
|
93
|
+
--jq ".[] | select(.closedAt >= \"$SINCE\")" 2>/dev/null || echo "[]")
|
|
94
|
+
|
|
95
|
+
TOTAL_FINDINGS=$(echo "$FINDINGS_RAW" | grep -c '"number"' 2>/dev/null || echo "0")
|
|
96
|
+
echo "Found $TOTAL_FINDINGS review findings in the last 30 days"
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
### 1B: Group findings by referenced spec file
|
|
100
|
+
|
|
101
|
+
Findings reference a spec file when their body contains a path matching `commands/*.md` or
|
|
102
|
+
a link to a section of a command spec file. Extract the primary spec reference:
|
|
103
|
+
|
|
104
|
+
```bash
|
|
105
|
+
# For each finding, extract the first commands/*.md reference from the body
|
|
106
|
+
# Pattern: `commands/foo.md`, commands/foo/bar.md, or "commands/foo" in prose
|
|
107
|
+
echo "$FINDINGS_RAW" | jq -r '
|
|
108
|
+
. |
|
|
109
|
+
(.body | match("commands/[a-z0-9_/-]+\\.md").string // null) as $spec |
|
|
110
|
+
if $spec then "\(.number)|\($spec)" else empty end
|
|
111
|
+
' 2>/dev/null | sort | uniq > /tmp/findings_by_spec.txt
|
|
112
|
+
|
|
113
|
+
# Build concentration table: spec file → count, finding numbers
|
|
114
|
+
declare -A SPEC_COUNTS
|
|
115
|
+
declare -A SPEC_FINDINGS
|
|
116
|
+
while IFS='|' read -r num spec; do
|
|
117
|
+
[ -z "$spec" ] && continue
|
|
118
|
+
SPEC_COUNTS["$spec"]=$(( ${SPEC_COUNTS["$spec"]:-0} + 1 ))
|
|
119
|
+
SPEC_FINDINGS["$spec"]="${SPEC_FINDINGS[$spec]:+${SPEC_FINDINGS[$spec]},}#$num"
|
|
120
|
+
done < /tmp/findings_by_spec.txt
|
|
121
|
+
|
|
122
|
+
# Rank by count descending
|
|
123
|
+
for spec in "${!SPEC_COUNTS[@]}"; do
|
|
124
|
+
echo "${SPEC_COUNTS[$spec]} $spec"
|
|
125
|
+
done | sort -rn > /tmp/spec_ranked.txt
|
|
126
|
+
|
|
127
|
+
echo "Spec file recurrence ranking (last 30 days):"
|
|
128
|
+
cat /tmp/spec_ranked.txt | head -10
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
### 1C: Resolve target spec (auto-select or validate explicit)
|
|
132
|
+
|
|
133
|
+
```bash
|
|
134
|
+
if [ "$SELECT_MODE" = "auto" ]; then
|
|
135
|
+
# Pick the top-ranked spec file with ≥ 5 findings
|
|
136
|
+
TARGET_SPEC=$(awk '$1 >= 5 {print $2; exit}' /tmp/spec_ranked.txt || echo "")
|
|
137
|
+
if [ -z "$TARGET_SPEC" ]; then
|
|
138
|
+
echo "No spec file has ≥5 findings in the last 30 days. Top offender:"
|
|
139
|
+
head -1 /tmp/spec_ranked.txt
|
|
140
|
+
# Use top offender even if below threshold for explicit runs
|
|
141
|
+
TARGET_SPEC=$(awk 'NR==1{print $2}' /tmp/spec_ranked.txt || echo "")
|
|
142
|
+
if [ -z "$TARGET_SPEC" ]; then
|
|
143
|
+
echo "ERROR: No finding data available. Run pipeline-health first to accumulate findings."
|
|
144
|
+
exit 1
|
|
145
|
+
fi
|
|
146
|
+
echo "Using top offender: $TARGET_SPEC (below 5-finding threshold — proceeding with explicit run)"
|
|
147
|
+
else
|
|
148
|
+
echo "Auto-selected worst offender: $TARGET_SPEC (${SPEC_COUNTS[$TARGET_SPEC]} findings)"
|
|
149
|
+
fi
|
|
150
|
+
else
|
|
151
|
+
# Validate explicit spec file exists
|
|
152
|
+
if [ ! -f "$REPO_PATH/$TARGET_SPEC" ]; then
|
|
153
|
+
echo "ERROR: Spec file not found: $REPO_PATH/$TARGET_SPEC"
|
|
154
|
+
exit 1
|
|
155
|
+
fi
|
|
156
|
+
echo "Explicit target spec: $TARGET_SPEC"
|
|
157
|
+
fi
|
|
158
|
+
|
|
159
|
+
FINDING_COUNT=${SPEC_COUNTS["$TARGET_SPEC"]:-0}
|
|
160
|
+
FINDING_NUMBERS=${SPEC_FINDINGS["$TARGET_SPEC"]:-""}
|
|
161
|
+
echo "Target: $TARGET_SPEC — $FINDING_COUNT findings: $FINDING_NUMBERS"
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
### 1D: Load full finding bodies for the target spec
|
|
165
|
+
|
|
166
|
+
```bash
|
|
167
|
+
# Fetch full bodies for each finding that references TARGET_SPEC
|
|
168
|
+
IFS=',' read -ra FINDING_LIST <<< "$FINDING_NUMBERS"
|
|
169
|
+
FINDINGS_DETAIL=""
|
|
170
|
+
for finding_ref in "${FINDING_LIST[@]}"; do
|
|
171
|
+
num=$(echo "$finding_ref" | tr -d '#')
|
|
172
|
+
[ -z "$num" ] && continue
|
|
173
|
+
body=$(gh issue view "$num" $GH_FLAG --json number,title,body \
|
|
174
|
+
--jq '"\(.number)|\(.title)|\(.body)"' 2>/dev/null || echo "")
|
|
175
|
+
FINDINGS_DETAIL="${FINDINGS_DETAIL}
|
|
176
|
+
${body}"
|
|
177
|
+
done
|
|
178
|
+
|
|
179
|
+
echo "Loaded finding details for: $FINDING_NUMBERS"
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
---
|
|
183
|
+
|
|
184
|
+
## Phase 2: Draft Spec Diff
|
|
185
|
+
|
|
186
|
+
### 2A: Read the target spec file
|
|
187
|
+
|
|
188
|
+
```bash
|
|
189
|
+
TARGET_SPEC_CONTENT=$(cat "$REPO_PATH/$TARGET_SPEC" 2>/dev/null)
|
|
190
|
+
if [ -z "$TARGET_SPEC_CONTENT" ]; then
|
|
191
|
+
echo "ERROR: Cannot read $REPO_PATH/$TARGET_SPEC"
|
|
192
|
+
exit 1
|
|
193
|
+
fi
|
|
194
|
+
echo "Loaded target spec: $TARGET_SPEC ($(echo "$TARGET_SPEC_CONTENT" | wc -l) lines)"
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
### 2B: Analyze findings and identify defect classes
|
|
198
|
+
|
|
199
|
+
Read all finding bodies loaded in Phase 1D. Group them by defect class:
|
|
200
|
+
|
|
201
|
+
For each finding, identify:
|
|
202
|
+
- **What the builder did wrong** (the behavioral description, not the symptom)
|
|
203
|
+
- **Which section of the spec** was missing or ambiguous (or absent entirely)
|
|
204
|
+
- **What rule, check, or constraint** would prevent recurrence
|
|
205
|
+
|
|
206
|
+
Group findings by defect class — findings that share the same root behavioral failure
|
|
207
|
+
belong to one class even if their surface symptoms differ (e.g., three findings about
|
|
208
|
+
"missing predecessor check" are one class).
|
|
209
|
+
|
|
210
|
+
**Defect class analysis output** (produce for each class):
|
|
211
|
+
```
|
|
212
|
+
Class: {NAME}
|
|
213
|
+
Count: {N} (from findings: {#NNN, #NNN, ...})
|
|
214
|
+
Root behavioral failure: {what the agent does wrong — action-level}
|
|
215
|
+
Spec gap: {section name or "absent"} — {what's missing}
|
|
216
|
+
Proposed fix: {concrete spec rule/check/constraint — one paragraph max}
|
|
217
|
+
Evidence citations: {finding numbers that prove this class exists}
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
### 2C: Draft spec changes
|
|
221
|
+
|
|
222
|
+
For each defect class from 2B, draft the specific spec text change:
|
|
223
|
+
|
|
224
|
+
**Rules for spec text changes**:
|
|
225
|
+
- Describe the **bug class**, not a specific incident: write "When X condition, verify Y" not
|
|
226
|
+
"In PR #1234, the builder failed to..."
|
|
227
|
+
- One brief HTML comment `<!-- Added: forge#1742 -->` per new check (traceability)
|
|
228
|
+
- Evidence block (if used) describes the vulnerability **pattern** — not a historical narrative
|
|
229
|
+
- New checks go into the most specific relevant section (e.g., a builder check → Phase 3F
|
|
230
|
+
of work-on.md, not the header)
|
|
231
|
+
- Prefer adding a **MANDATORY** inline check over a prose warning — prose warnings are ignored
|
|
232
|
+
- If the spec section is absent: propose a new section; position it where an agent would
|
|
233
|
+
encounter it at the right moment (just-in-time)
|
|
234
|
+
|
|
235
|
+
Produce a structured diff description (not a literal `diff` output — a human-readable
|
|
236
|
+
description of what changes where):
|
|
237
|
+
|
|
238
|
+
```
|
|
239
|
+
### Change 1 — {SECTION_NAME}
|
|
240
|
+
**Type**: {Add check | Add section | Strengthen rule | Correct contradiction}
|
|
241
|
+
**Location**: {line range or section heading}
|
|
242
|
+
**Evidence**: Findings {#NNN, #NNN} — {one sentence: what recurs}
|
|
243
|
+
**Before**: {current text snippet or "section absent"}
|
|
244
|
+
**After**: {proposed text or new section content}
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
### 2D: Validate proposed changes against the spec
|
|
248
|
+
|
|
249
|
+
Before committing to the PR, validate:
|
|
250
|
+
- [ ] Each change addresses a defect class with ≥1 cited finding
|
|
251
|
+
- [ ] No change contradicts an existing HARD RULE in the same spec
|
|
252
|
+
- [ ] No change widens scope beyond the targeted spec file
|
|
253
|
+
- [ ] All inline checks are placed where an agent encounters them during execution
|
|
254
|
+
- [ ] Spec style matches surrounding content (heading levels, bash blocks, HTML comment conventions)
|
|
255
|
+
|
|
256
|
+
---
|
|
257
|
+
|
|
258
|
+
## Phase 3: Open Spec-Evolution PR
|
|
259
|
+
|
|
260
|
+
### 3A: Create branch and apply changes
|
|
261
|
+
|
|
262
|
+
```bash
|
|
263
|
+
cd "$REPO_PATH"
|
|
264
|
+
git fetch origin
|
|
265
|
+
|
|
266
|
+
# Branch slug: spec-doctor/{spec-slug}-{issue-or-timestamp}
|
|
267
|
+
SPEC_SLUG=$(echo "$TARGET_SPEC" | sed 's|commands/||;s|\.md$||;s|/|-|g' | cut -c1-40)
|
|
268
|
+
DOCTOR_BRANCH="spec-doctor/${SPEC_SLUG}-$(date +%Y%m%d)"
|
|
269
|
+
|
|
270
|
+
# Branch from staging (fast lane — spec-evolution PRs never auto-merge,
|
|
271
|
+
# so they are reviewed by a human before reaching main via staging→main)
|
|
272
|
+
git worktree add ".claude/worktrees/${DOCTOR_BRANCH}" -b "$DOCTOR_BRANCH" "origin/$STAGING_BRANCH" 2>/dev/null \
|
|
273
|
+
|| git worktree add ".claude/worktrees/${DOCTOR_BRANCH}" "$DOCTOR_BRANCH"
|
|
274
|
+
|
|
275
|
+
DOCTOR_WORKTREE="$REPO_PATH/.claude/worktrees/${DOCTOR_BRANCH}"
|
|
276
|
+
echo "Created worktree: $DOCTOR_WORKTREE on branch $DOCTOR_BRANCH"
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
### 3B: Apply spec changes
|
|
280
|
+
|
|
281
|
+
Apply each change from Phase 2C to the target spec file in the worktree:
|
|
282
|
+
|
|
283
|
+
```bash
|
|
284
|
+
TARGET_SPEC_WORKTREE="$DOCTOR_WORKTREE/$TARGET_SPEC"
|
|
285
|
+
cp "$REPO_PATH/$TARGET_SPEC" "$TARGET_SPEC_WORKTREE"
|
|
286
|
+
```
|
|
287
|
+
|
|
288
|
+
For each change from Phase 2C:
|
|
289
|
+
- Use the Edit tool to apply the change at the correct location
|
|
290
|
+
- Verify the edit was applied correctly (read back the modified section)
|
|
291
|
+
- Do NOT introduce changes beyond the scoped defect classes from Phase 2C
|
|
292
|
+
|
|
293
|
+
### 3C: Commit
|
|
294
|
+
|
|
295
|
+
```bash
|
|
296
|
+
cd "$DOCTOR_WORKTREE"
|
|
297
|
+
git add "$TARGET_SPEC"
|
|
298
|
+
git commit -s -m "fix($(basename $TARGET_SPEC .md)): address $(echo $FINDING_NUMBERS | tr ',' '\n' | wc -l | tr -d ' ') concentrated review findings (#1742)
|
|
299
|
+
|
|
300
|
+
Evidence-backed spec evolution. Defect classes addressed:
|
|
301
|
+
$(for class in "${DEFECT_CLASS_NAMES[@]}"; do echo "- $class"; done)
|
|
302
|
+
|
|
303
|
+
Cited findings: $FINDING_NUMBERS
|
|
304
|
+
|
|
305
|
+
Generated by /spec-doctor. PR is a proposal — not auto-merged.
|
|
306
|
+
Eval gate must pass before merge."
|
|
307
|
+
```
|
|
308
|
+
|
|
309
|
+
### 3D: Push and open PR
|
|
310
|
+
|
|
311
|
+
```bash
|
|
312
|
+
cd "$DOCTOR_WORKTREE"
|
|
313
|
+
git push -u origin "$DOCTOR_BRANCH"
|
|
314
|
+
|
|
315
|
+
# Build the PR body with per-change finding citations
|
|
316
|
+
PR_BODY=$(cat <<PR_BODY_EOF
|
|
317
|
+
## Spec Evolution Proposal
|
|
318
|
+
|
|
319
|
+
**Target spec**: \`$TARGET_SPEC\`
|
|
320
|
+
**Source findings** (30-day window): $FINDING_NUMBERS ($FINDING_COUNT findings)
|
|
321
|
+
**Generated by**: /spec-doctor (#1742)
|
|
322
|
+
|
|
323
|
+
> ⚠️ This PR is **never auto-merged**. The eval gate must pass, and a human must review and merge.
|
|
324
|
+
> See [eval-gate runbook](docs/design/eval-gate-runbook.md) for baseline update procedure.
|
|
325
|
+
|
|
326
|
+
## Changes
|
|
327
|
+
|
|
328
|
+
$(for change in "${CHANGES_SUMMARY[@]}"; do echo "$change"; done)
|
|
329
|
+
|
|
330
|
+
## Evidence Trail
|
|
331
|
+
|
|
332
|
+
For each change, the cited findings prove the defect class exists and recurs:
|
|
333
|
+
|
|
334
|
+
$(for finding_ref in "${FINDING_LIST[@]}"; do
|
|
335
|
+
num=$(echo "$finding_ref" | tr -d '#')
|
|
336
|
+
title=$(gh issue view "$num" $GH_FLAG --json title --jq '.title' 2>/dev/null || echo "finding $num")
|
|
337
|
+
echo "- **#$num**: $title"
|
|
338
|
+
done)
|
|
339
|
+
|
|
340
|
+
## Eval Gate
|
|
341
|
+
|
|
342
|
+
The CI eval gate (\`.github/workflows/eval-gate.yml\`) will run automatically because this PR
|
|
343
|
+
carries the \`spec-evolution\` label. The gate replays the relevant corpus slice with old vs
|
|
344
|
+
new spec and blocks merge on outcome regression (threshold: 5 percentage points).
|
|
345
|
+
|
|
346
|
+
See \`scripts/eval-gate-scorecard.mjs\` for the gate comparator implementation.
|
|
347
|
+
|
|
348
|
+
## Merge Criteria
|
|
349
|
+
|
|
350
|
+
- [ ] Eval gate passes (no regression on outcome scores)
|
|
351
|
+
- [ ] Human reviewer confirms each cited finding is addressed
|
|
352
|
+
- [ ] No new spec contradictions introduced (HARD RULES consistent with changes)
|
|
353
|
+
- [ ] Baseline update procedure followed after merge (see runbook)
|
|
354
|
+
|
|
355
|
+
---
|
|
356
|
+
Part of #1742 — evidence-backed spec evolution PRs gated by eval corpus
|
|
357
|
+
PR_BODY_EOF
|
|
358
|
+
)
|
|
359
|
+
|
|
360
|
+
# NEVER use --auto-merge here — spec-evolution PRs require human review
|
|
361
|
+
gh pr create $GH_FLAG \
|
|
362
|
+
--base "$STAGING_BRANCH" \
|
|
363
|
+
--head "$DOCTOR_BRANCH" \
|
|
364
|
+
--title "fix($(basename $TARGET_SPEC .md)): evidence-backed spec evolution — $FINDING_COUNT findings addressed" \
|
|
365
|
+
--body "$PR_BODY" \
|
|
366
|
+
--label "spec-evolution"
|
|
367
|
+
|
|
368
|
+
DOCTOR_PR_NUMBER=$(gh pr list $GH_FLAG --head "$DOCTOR_BRANCH" --json number --jq '.[0].number')
|
|
369
|
+
echo "Spec-evolution PR created: #$DOCTOR_PR_NUMBER"
|
|
370
|
+
echo "Auto-merge: EXCLUDED (structural) — eval gate must pass, human must review"
|
|
371
|
+
```
|
|
372
|
+
|
|
373
|
+
### 3E: Post completion annotation
|
|
374
|
+
|
|
375
|
+
```bash
|
|
376
|
+
gh issue comment {ORIGINATING_ISSUE_NUMBER:-1742} $GH_FLAG --body "<!-- FORGE:SPEC_DOCTOR_COMPLETE -->
|
|
377
|
+
## Spec Doctor Run Complete
|
|
378
|
+
|
|
379
|
+
**Target spec**: \`$TARGET_SPEC\`
|
|
380
|
+
**Findings addressed**: $FINDING_NUMBERS ($FINDING_COUNT findings)
|
|
381
|
+
**PR**: #$DOCTOR_PR_NUMBER
|
|
382
|
+
**Branch**: \`$DOCTOR_BRANCH\`
|
|
383
|
+
**Auto-merge**: EXCLUDED — eval gate must pass, human reviews and merges
|
|
384
|
+
|
|
385
|
+
### Defect Classes Addressed
|
|
386
|
+
$(for class in "${DEFECT_CLASS_NAMES[@]}"; do echo "- $class"; done)
|
|
387
|
+
|
|
388
|
+
The eval gate will run automatically on the \`spec-evolution\` PR. Monitor:
|
|
389
|
+
\`gh pr view $DOCTOR_PR_NUMBER $GH_FLAG --json statusCheckRollup\`"
|
|
390
|
+
```
|
|
391
|
+
|
|
392
|
+
### 3F: Worktree cleanup note
|
|
393
|
+
|
|
394
|
+
The spec-doctor worktree is retained until the PR is merged or closed.
|
|
395
|
+
After merge, clean up with:
|
|
396
|
+
|
|
397
|
+
```bash
|
|
398
|
+
cd "$REPO_PATH"
|
|
399
|
+
git worktree remove ".claude/worktrees/${DOCTOR_BRANCH}" --force
|
|
400
|
+
git branch -D "$DOCTOR_BRANCH" 2>/dev/null || true
|
|
401
|
+
```
|
|
402
|
+
|
|
403
|
+
---
|
|
404
|
+
|
|
405
|
+
## Usage Examples
|
|
406
|
+
|
|
407
|
+
```bash
|
|
408
|
+
# Auto-select worst offender from 30-day window
|
|
409
|
+
/spec-doctor auto
|
|
410
|
+
|
|
411
|
+
# Target a specific spec file
|
|
412
|
+
/spec-doctor commands/orchestrate.md
|
|
413
|
+
|
|
414
|
+
# The first real run targets commands/orchestrate.md (worst offender per issue #1742)
|
|
415
|
+
/spec-doctor commands/orchestrate.md
|
|
416
|
+
```
|
|
417
|
+
|
|
418
|
+
---
|
|
419
|
+
|
|
420
|
+
## Integration Points
|
|
421
|
+
|
|
422
|
+
- **Pipeline trigger**: `pipeline-health/phase-4-proposals.md §4A.5` queues a spec-doctor
|
|
423
|
+
run when a spec file's 30-day finding concentration exceeds threshold (≥5 findings)
|
|
424
|
+
- **Eval gate**: `.github/workflows/eval-gate.yml` triggers on the `spec-evolution` label —
|
|
425
|
+
runs the eval corpus against old vs new spec and blocks merge on regression
|
|
426
|
+
- **Auto-merge exclusion**: `review-pr.md` Phase -1 detects `spec-evolution` label and sets
|
|
427
|
+
`AUTO_MERGE=false` + adds `needs-human` — this cannot be overridden by callers
|
|
428
|
+
- **Knowledge index**: `scripts/build-knowledge-index.mjs` indexes REVIEWER cards; finding
|
|
429
|
+
queries in Phase 1 use `gh issue list` directly (no script dependency)
|
package/commands/validate.md
CHANGED
|
@@ -132,3 +132,55 @@ If NOT A PROBLEM — why the report was wrong:
|
|
|
132
132
|
If NEEDS MORE DATA — what's missing:
|
|
133
133
|
[specific questions to answer]
|
|
134
134
|
```
|
|
135
|
+
|
|
136
|
+
---
|
|
137
|
+
|
|
138
|
+
## Step 5: Finding Lifecycle Label Transition (MANDATORY when input is an issue number)
|
|
139
|
+
|
|
140
|
+
**Skip if**: Input was not an issue number (e.g., `/validate` was invoked with a plain description, not `#NNN`).
|
|
141
|
+
|
|
142
|
+
**Purpose**: If the validated issue is a review-finding with `needs-validation`, wire the verdict back into the finding lifecycle. This prevents `needs-validation` from becoming a permanent no-op label. <!-- Added: forge#1730 -->
|
|
143
|
+
|
|
144
|
+
```bash
|
|
145
|
+
# Parse issue number from input — skip if not an issue reference
|
|
146
|
+
INPUT_ISSUE=$(echo "$ARGUMENTS" | grep -oE '#?([0-9]+)' | grep -oE '[0-9]+' | head -1)
|
|
147
|
+
GH_FLAG=$(yq -r '"-R " + .project.owner + "/" + .project.repo' forge.yaml 2>/dev/null || echo "")
|
|
148
|
+
|
|
149
|
+
if [ -n "$INPUT_ISSUE" ] && [ -n "$GH_FLAG" ]; then
|
|
150
|
+
ISSUE_LABELS=$(gh issue view "$INPUT_ISSUE" "$GH_FLAG" --json labels \
|
|
151
|
+
--jq '[.labels[].name] | join(",")' 2>/dev/null || echo "")
|
|
152
|
+
|
|
153
|
+
if echo "$ISSUE_LABELS" | grep -q "needs-validation"; then
|
|
154
|
+
echo "Issue #$INPUT_ISSUE has needs-validation — applying verdict label transition..."
|
|
155
|
+
|
|
156
|
+
# Map /validate verdict to transition-label.sh verdict format
|
|
157
|
+
# CONFIRMED → validated; NOT A PROBLEM / NEEDS MORE DATA → false-positive
|
|
158
|
+
case "$VALIDATION_VERDICT" in
|
|
159
|
+
CONFIRMED) FORGE_VERDICT="CONFIRMED" ;;
|
|
160
|
+
"NOT A PROBLEM") FORGE_VERDICT="NOT-CONFIRMED" ;;
|
|
161
|
+
"NEEDS MORE DATA") FORGE_VERDICT="NOT-CONFIRMED" ;;
|
|
162
|
+
*) FORGE_VERDICT="NOT-CONFIRMED" ;;
|
|
163
|
+
esac
|
|
164
|
+
|
|
165
|
+
# Resolve and call transition-label.sh --validate
|
|
166
|
+
REPO_PATH=$(yq -r '.paths.root' forge.yaml 2>/dev/null || echo ".")
|
|
167
|
+
SCRIPT="$REPO_PATH/scripts/transition-label.sh"
|
|
168
|
+
if [ -f "$SCRIPT" ]; then
|
|
169
|
+
bash "$SCRIPT" --validate "$FORGE_VERDICT" "$INPUT_ISSUE" "$GH_FLAG" || true
|
|
170
|
+
else
|
|
171
|
+
# Prose fallback: apply directly via gh
|
|
172
|
+
if [ "$FORGE_VERDICT" = "CONFIRMED" ]; then
|
|
173
|
+
gh issue edit "$INPUT_ISSUE" "$GH_FLAG" --add-label "validated" --remove-label "needs-validation" 2>/dev/null || true
|
|
174
|
+
echo "Applied: needs-validation → validated"
|
|
175
|
+
else
|
|
176
|
+
gh issue edit "$INPUT_ISSUE" "$GH_FLAG" --add-label "false-positive" --remove-label "needs-validation" 2>/dev/null || true
|
|
177
|
+
echo "Applied: needs-validation → false-positive"
|
|
178
|
+
fi
|
|
179
|
+
fi
|
|
180
|
+
else
|
|
181
|
+
echo "Issue #$INPUT_ISSUE does not have needs-validation — no label transition needed"
|
|
182
|
+
fi
|
|
183
|
+
fi
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
**Note**: `$VALIDATION_VERDICT` is the verdict string from Step 4's output block. Extract it before this step runs. If extraction fails, default to `NOT-CONFIRMED` (conservative — do not auto-label as confirmed).
|
|
@@ -318,6 +318,76 @@ agent defaults ← Built-in behaviors
|
|
|
318
318
|
|
|
319
319
|
---
|
|
320
320
|
|
|
321
|
+
## Phase A1.5: Prior Decision Injection (MANDATORY for STANDARD/COMPLEX — skip for TRIVIAL) <!-- Added: forge#1737 -->
|
|
322
|
+
|
|
323
|
+
**Purpose**: Surface ADRs from prior pipeline runs whose anchor paths overlap the contract files before any code is written. Architect plans open with binding prior decisions ("prior decision NNN: chose process substitution over pipes because pipe-RHS subshell loss — #1689") so tradeoffs are not silently re-litigated.
|
|
324
|
+
|
|
325
|
+
**Skip if**: COMPLEXITY_BAND: TRIVIAL OR `devdocs/decisions/` does not exist at `{REPO_PATH}/devdocs/decisions/` OR the directory is empty.
|
|
326
|
+
|
|
327
|
+
**Token budget**: ≤300 tokens total (max 3 ADRs × ~100 tokens each). Truncate each ADR body to first 20 lines.
|
|
328
|
+
|
|
329
|
+
```bash
|
|
330
|
+
DECISIONS_DIR="{REPO_PATH}/devdocs/decisions"
|
|
331
|
+
|
|
332
|
+
if [ -d "$DECISIONS_DIR" ] && [ -n "$(ls -A "$DECISIONS_DIR" 2>/dev/null | grep -v '\.gitkeep')" ]; then
|
|
333
|
+
echo "Scanning $DECISIONS_DIR for ADRs overlapping contract files..."
|
|
334
|
+
|
|
335
|
+
# Build list of contract file basenames for anchor matching
|
|
336
|
+
IFS=' ' read -ra CONTRACT_FILES_ARR <<< "{AFFECTED_FILES}"
|
|
337
|
+
MATCHED_ADRS=()
|
|
338
|
+
ADR_COUNT=0
|
|
339
|
+
|
|
340
|
+
for adr_file in "$DECISIONS_DIR"/*.md; do
|
|
341
|
+
[ -f "$adr_file" ] || continue
|
|
342
|
+
# Skip needs-review ADRs — anchor is dead, don't inject as constraints
|
|
343
|
+
if grep -q "^status: needs-review" "$adr_file" 2>/dev/null; then
|
|
344
|
+
echo " Skipped (needs-review): $(basename $adr_file)"
|
|
345
|
+
continue
|
|
346
|
+
fi
|
|
347
|
+
# Check if this ADR's anchor path overlaps any contract file
|
|
348
|
+
ANCHOR=$(grep "^anchor:" "$adr_file" 2>/dev/null | head -1 | sed 's/^anchor:\s*//')
|
|
349
|
+
if [ -z "$ANCHOR" ]; then continue; fi
|
|
350
|
+
|
|
351
|
+
for contract_file in "${CONTRACT_FILES_ARR[@]}"; do
|
|
352
|
+
if echo "$ANCHOR" | grep -qF "$(basename "$contract_file" | cut -d. -f1)" || \
|
|
353
|
+
echo "$contract_file" | grep -qF "$ANCHOR"; then
|
|
354
|
+
MATCHED_ADRS+=("$adr_file")
|
|
355
|
+
ADR_COUNT=$((ADR_COUNT + 1))
|
|
356
|
+
break
|
|
357
|
+
fi
|
|
358
|
+
done
|
|
359
|
+
|
|
360
|
+
[ "$ADR_COUNT" -ge 3 ] && break # Hard cap: 3 ADRs max
|
|
361
|
+
done
|
|
362
|
+
|
|
363
|
+
if [ "${#MATCHED_ADRS[@]}" -gt 0 ]; then
|
|
364
|
+
echo ""
|
|
365
|
+
echo "### Prior Decisions (${#MATCHED_ADRS[@]} ADRs overlapping contract files)"
|
|
366
|
+
echo "<!-- Each ADR is a prior tradeoff that constrains this implementation. -->"
|
|
367
|
+
for adr_file in "${MATCHED_ADRS[@]}"; do
|
|
368
|
+
echo ""
|
|
369
|
+
echo "#### $(basename "$adr_file")"
|
|
370
|
+
head -20 "$adr_file" # ~100 tokens per ADR — stays within 300-token budget
|
|
371
|
+
done
|
|
372
|
+
echo ""
|
|
373
|
+
echo "<!-- FORGE:PRIOR_DECISIONS: ${#MATCHED_ADRS[@]} ADR(s) injected above — treat as constraints, not suggestions -->"
|
|
374
|
+
else
|
|
375
|
+
echo "No matching ADRs found for contract files — proceeding without prior decision constraints"
|
|
376
|
+
fi
|
|
377
|
+
else
|
|
378
|
+
echo "devdocs/decisions/ absent or empty — skipping prior decision injection"
|
|
379
|
+
fi
|
|
380
|
+
```
|
|
381
|
+
|
|
382
|
+
**Inject into FORGE:ARCHITECT comment**: If matching ADRs were found, add a `### Prior Decisions (Constraints)` section **before** `### Affected Paths` in the output comment. The builder reads this section before writing any code.
|
|
383
|
+
|
|
384
|
+
**Skip conditions**:
|
|
385
|
+
- COMPLEXITY_BAND: TRIVIAL — trivial changes have no cross-path consistency risk; ADR injection adds unnecessary context
|
|
386
|
+
- `devdocs/decisions/` absent or empty — no ADRs to inject
|
|
387
|
+
- All candidate ADRs are `status: needs-review` — dead-anchor ADRs are excluded from constraint injection
|
|
388
|
+
|
|
389
|
+
---
|
|
390
|
+
|
|
321
391
|
## Phase A1: Read Entry Points
|
|
322
392
|
|
|
323
393
|
Identify every entry point that initiates the affected logic. Start from the files named in the investigation report.
|
|
@@ -549,6 +619,13 @@ gh issue comment {NUMBER} {GH_FLAG} --body "<!-- FORGE:ARCHITECT -->
|
|
|
549
619
|
If devdocs path was absent or no applicable files found: 'No applicable devdocs found.' -->
|
|
550
620
|
{PROJECT_CONVENTIONS}
|
|
551
621
|
|
|
622
|
+
### Prior Decisions (Constraints)
|
|
623
|
+
<!-- Auto-ADRs from prior TRAJECTORY runs whose anchor paths overlap the contract files.
|
|
624
|
+
Injected by Phase A1.5. Builder MUST respect these tradeoffs — re-litigating requires
|
|
625
|
+
explicit justification. Absent when devdocs/decisions/ is empty or no overlap found.
|
|
626
|
+
status: needs-review ADRs are excluded — their anchors are dead. -->
|
|
627
|
+
{PRIOR_DECISIONS}
|
|
628
|
+
|
|
552
629
|
### Affected Paths (ALL must be updated)
|
|
553
630
|
| # | File | Function/Class | Change Required | Why |
|
|
554
631
|
|---|------|----------------|-----------------|-----|
|