@windyroad/risk-scorer 0.7.3-preview.303 → 0.8.0-preview.304
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/agents/pipeline.md
CHANGED
|
@@ -261,6 +261,61 @@ The hint is consumed by the calling orchestrator **after** the ADR-042 auto-appl
|
|
|
261
261
|
|
|
262
262
|
Do NOT emit `RISK_REGISTER_HINT:` when all cumulative scores are within appetite AND no confidentiality disclosure AND no user-stated-precondition fired. The hint is additive to the existing Below-Appetite Output Rule — a silent pass MUST remain silent. Do not emit an empty `RISK_REGISTER_HINT:` header with no bullets either — omit the block entirely.
|
|
263
263
|
|
|
264
|
+
## Held-Changeset Graduation Evaluation (ADR-061)
|
|
265
|
+
|
|
266
|
+
When the pipeline state indicates **within-appetite drain mode** (cumulative push and release residual both ≤ 4/25 per `RISK-POLICY.md`) AND `docs/changesets-holding/` contains entries, evaluate each held changeset against ADR-061 Rule 1's symmetric graduation criterion: **reinstate when `release-risk(pipeline with held changeset hypothetically reinstated) ≤ problem-ticket Priority`**.
|
|
267
|
+
|
|
268
|
+
This is the symmetric counterpart to ADR-042 Rule 2's move-to-holding contract. Material flows in when release-risk would exceed appetite; material flows out when release-risk falls at or below the originating problem-ticket Priority.
|
|
269
|
+
|
|
270
|
+
### Mechanism — invoke the deterministic graduation evaluator
|
|
271
|
+
|
|
272
|
+
The Rule 1a join (changeset → problem ID → ticket Priority) and the Rule 2 VP carve-out detection are deterministic lookups. Invoke the `wr-risk-scorer-evaluate-graduation` shim (ADR-049 `$PATH`-resolved) to read structured candidate lines for each held changeset:
|
|
273
|
+
|
|
274
|
+
```
|
|
275
|
+
GRADUATION_CANDIDATE: changeset=<filename> | ticket=P<NNN> | priority=<N> | class=3a | status=<resolved|vp-blocked|halt-no-resolution>
|
|
276
|
+
GRADUATION_SUMMARY: total=<N> resolved=<N> vp_blocked=<N> halts=<N>
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
The script does NOT compute release-risk and does NOT apply Rule 4 evidence-floor judgement — those are LLM-judgement surfaces you own per ADR-015's pure-scorer contract. The script's job is to emit candidates with their joined Priority; your job is to decide whether each candidate's release-risk + evidence-floor profile justifies emitting a `reinstate-from-holding` remediation line.
|
|
280
|
+
|
|
281
|
+
### Per-candidate evaluation rules
|
|
282
|
+
|
|
283
|
+
For each `status=resolved` candidate:
|
|
284
|
+
|
|
285
|
+
1. **Compute release-risk with hypothetical reinstate** (Rule 1) — re-score the current pipeline as if the held changeset were `git mv`'d back to `.changeset/`. Use the same scoring path as ADR-042 Rule 2's re-score; this is your existing pipeline-scoring competence applied to the symmetric hypothesis.
|
|
286
|
+
2. **Compare** — `release-risk ≤ priority` from the candidate line. If false, the held entry stays held — no remediation emitted this cycle.
|
|
287
|
+
3. **Verify Rule 4 evidence floor** — class-specific evidence shape per ADR-061 Rule 4:
|
|
288
|
+
- **PreToolUse:Bash gates**: ≥ 1 gate-fire log entry per intended trigger surface, with post-fire commit trail showing no false-block.
|
|
289
|
+
- **UserPromptSubmit detectors**: ≥ 1 detector firing logged to hook stderr or `.afk-run-state/<detector>.log`.
|
|
290
|
+
- **commit-hook-with-auto-fix**: ≥ 1 auto-fix commit log entry visible via `git log --grep=<hook-marker>` with the diff showing the correct fix shape.
|
|
291
|
+
- **SessionStart additionalContext hooks**: ≥ 1 session-trail entry showing the injection fired without regression in the immediate-next turn.
|
|
292
|
+
|
|
293
|
+
Per ADR-026 cite + persist + uncertainty: the evidence must ground in a re-readable artefact, not a bare count.
|
|
294
|
+
4. **Emit `reinstate-from-holding` remediation** (Rule 5) when the comparison evaluates true AND the evidence floor is met:
|
|
295
|
+
|
|
296
|
+
```
|
|
297
|
+
RISK_REMEDIATIONS:
|
|
298
|
+
- R<N> | reinstate-from-holding <changeset-name>: release-risk <release-score>/25 ≤ P<NNN> Priority <priority-value>; class 3a; evidence: <class-specific artefact citation> | S | -<release-score> | docs/changesets-holding/<changeset-name>, .changeset/<changeset-name>
|
|
299
|
+
```
|
|
300
|
+
|
|
301
|
+
The `description` column (free-form prose per ADR-042 Rule 2a open vocabulary) carries the symmetric-balance verdict, the cited evidence artefact, and the class. The agent consuming this line applies it via `git mv docs/changesets-holding/<name>.md .changeset/<name>.md`.
|
|
302
|
+
|
|
303
|
+
For each `status=vp-blocked` candidate (Rule 2 carve-out — originating ticket in Verification Pending):
|
|
304
|
+
|
|
305
|
+
- **DO NOT emit a `reinstate-from-holding` line** for this changeset. ADR-022 establishes the user-owned verify-or-reject decision surface; auto-reinstating short-circuits that surface. The `.verifying.md` → `.closed.md` transition auto-clears the carve-out; the next Step 6.5 graduation pass evaluates the changeset normally.
|
|
306
|
+
|
|
307
|
+
For each `status=halt-no-resolution` candidate (Rule 1a terminal — no ticket resolved):
|
|
308
|
+
|
|
309
|
+
- **DO NOT auto-graduate**. Surface the unresolved candidate in your report body under an "Unresolvable graduation candidates" section so the caller (orchestrator) sees the join failure and can present it as a user-decision surface per ADR-013 + ADR-044 framework-resolution boundary. Per ADR-061 Rule 1a, join ambiguity is a user-decision surface, not an agent-decision surface.
|
|
310
|
+
|
|
311
|
+
### Scope — Phase 2a only
|
|
312
|
+
|
|
313
|
+
This evaluation surface covers **orthogonal-gate class (3a) only** per ADR-061 Rule 3. Atomic-cohort class (3b — RFC-shaped held changesets that graduate as a single atomic unit per ADR-060 finding 12) requires RFC ticket cohort enumeration and is **deferred to Phase 2b**. When the holding-area contains entries that belong to an RFC cohort, the Phase 2a evaluator emits each entry as an independent 3a candidate; treat such candidates conservatively (the symmetric-balance math is identical but the evaluation unit is wrong) and prefer a `RISK_REGISTER_HINT:` over auto-emitting `reinstate-from-holding` until Phase 2b lands the cohort enumeration.
|
|
314
|
+
|
|
315
|
+
### Audit trail (Rule 6)
|
|
316
|
+
|
|
317
|
+
Every emitted `reinstate-from-holding` line MUST cite the resolved problem-ticket ID and Priority value in the description column so the audit trail extends ADR-042 Rule 6. The consuming orchestrator additionally appends to `docs/changesets-holding/README.md` "Recently reinstated" per Rule 6 § 2.
|
|
318
|
+
|
|
264
319
|
## Confidential Information Disclosure
|
|
265
320
|
|
|
266
321
|
Check diffs for business metrics (revenue, user counts, pricing, traffic volumes). Flag as a standalone risk if found.
|
package/package.json
CHANGED
|
@@ -0,0 +1,231 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# packages/risk-scorer/scripts/evaluate-graduation.sh
|
|
3
|
+
#
|
|
4
|
+
# Evaluates held-changeset graduation candidates per ADR-061
|
|
5
|
+
# (Dogfood graduation criteria for held changesets — symmetric risk balance).
|
|
6
|
+
# Phase 2a: orthogonal-gate class only (Class 3a per ADR-061 Rule 3).
|
|
7
|
+
# Atomic-cohort class (3b) requires RFC ticket cohort enumeration and is
|
|
8
|
+
# deferred to Phase 2b per the architect-approved Phase 2a/2b split.
|
|
9
|
+
#
|
|
10
|
+
# This script implements the deterministic Rule 1a join + Rule 2 VP carve-out
|
|
11
|
+
# detection. It does NOT compute release-risk and does NOT apply Rule 4
|
|
12
|
+
# evidence-floor judgement — those are LLM-judgement surfaces owned by the
|
|
13
|
+
# wr-risk-scorer:pipeline agent (per ADR-015 pure-scorer contract).
|
|
14
|
+
#
|
|
15
|
+
# Usage:
|
|
16
|
+
# evaluate-graduation.sh [<project-root>]
|
|
17
|
+
#
|
|
18
|
+
# Default <project-root> is $(pwd).
|
|
19
|
+
#
|
|
20
|
+
# Behaviour:
|
|
21
|
+
# - Globs docs/changesets-holding/*.md (excludes README.md).
|
|
22
|
+
# - For each held changeset, applies Rule 1a join:
|
|
23
|
+
# 1. Filename convention (primary): <package>-p<NNN>-<slug>.md → P<NNN>
|
|
24
|
+
# 2. Body grep fallback (secondary): grep '\bP[0-9]+\b' in changeset body
|
|
25
|
+
# 3. Multi-ticket: max(Priority) across the referenced set
|
|
26
|
+
# - Resolves the ticket file via dual-tolerant glob (ADR-031 + RFC-002):
|
|
27
|
+
# docs/problems/<NNN>-*.md (flat) AND docs/problems/*/<NNN>-*.md (per-state)
|
|
28
|
+
# - Extracts the Priority value from the ticket's `**Priority**: N (...)` line.
|
|
29
|
+
# - Detects Rule 2 VP carve-out (ticket file ends in .verifying.md).
|
|
30
|
+
# - Emits one structured candidate line per held changeset to stdout.
|
|
31
|
+
#
|
|
32
|
+
# Stdout format (one candidate per held changeset, agent-parseable):
|
|
33
|
+
# GRADUATION_CANDIDATE: changeset=<filename> | ticket=P<NNN> | priority=<N> | class=3a | status=<resolved|vp-blocked|halt-no-resolution>
|
|
34
|
+
#
|
|
35
|
+
# Stdout summary line at end:
|
|
36
|
+
# GRADUATION_SUMMARY: total=<N> resolved=<N> vp_blocked=<N> halts=<N>
|
|
37
|
+
#
|
|
38
|
+
# Exit codes:
|
|
39
|
+
# 0 — script ran to completion (any number of halts is still exit 0;
|
|
40
|
+
# halts surface via per-candidate status=halt-no-resolution lines so
|
|
41
|
+
# the agent can present them as Rule 1a halt-and-prompt candidates)
|
|
42
|
+
# 1 — no holding-area or empty holding-area (no-op caller signal)
|
|
43
|
+
# 2 — invalid project root (missing docs/)
|
|
44
|
+
#
|
|
45
|
+
# @adr ADR-061 (graduation criteria — Phase 2a Rule 1a join + Rule 2 VP carve-out)
|
|
46
|
+
# @adr ADR-049 (resolved via bin/wr-risk-scorer-evaluate-graduation shim)
|
|
47
|
+
# @adr ADR-052 (behavioural-fixture coverage at scripts/test/evaluate-graduation.bats)
|
|
48
|
+
# @adr ADR-015 (pure-scorer contract — script does deterministic join only;
|
|
49
|
+
# agent owns release-risk re-computation + evidence-floor judgement)
|
|
50
|
+
# @adr ADR-031 (dual-tolerant problem-ticket layout per RFC-002 migration window)
|
|
51
|
+
# @problem P162 (Phase 2a)
|
|
52
|
+
|
|
53
|
+
set -uo pipefail
|
|
54
|
+
|
|
55
|
+
PROJECT_ROOT="${1:-$(pwd)}"
|
|
56
|
+
HOLDING_DIR="${PROJECT_ROOT}/docs/changesets-holding"
|
|
57
|
+
PROBLEMS_DIR="${PROJECT_ROOT}/docs/problems"
|
|
58
|
+
|
|
59
|
+
if [ ! -d "${PROJECT_ROOT}/docs" ]; then
|
|
60
|
+
echo "GRADUATION_ERROR: invalid project root (missing docs/): ${PROJECT_ROOT}" >&2
|
|
61
|
+
exit 2
|
|
62
|
+
fi
|
|
63
|
+
|
|
64
|
+
if [ ! -d "$HOLDING_DIR" ]; then
|
|
65
|
+
echo "GRADUATION_SUMMARY: total=0 resolved=0 vp_blocked=0 halts=0"
|
|
66
|
+
exit 1
|
|
67
|
+
fi
|
|
68
|
+
|
|
69
|
+
# Enumerate held changesets (exclude README.md). Use null-delim shape so
|
|
70
|
+
# filenames-with-spaces never break iteration (defensive even though our
|
|
71
|
+
# convention is kebab-case).
|
|
72
|
+
HELD_FILES=()
|
|
73
|
+
while IFS= read -r -d '' f; do
|
|
74
|
+
base=$(basename "$f")
|
|
75
|
+
if [ "$base" = "README.md" ]; then
|
|
76
|
+
continue
|
|
77
|
+
fi
|
|
78
|
+
HELD_FILES+=("$f")
|
|
79
|
+
done < <(find "$HOLDING_DIR" -maxdepth 1 -type f -name '*.md' -print0 2>/dev/null)
|
|
80
|
+
|
|
81
|
+
if [ "${#HELD_FILES[@]}" -eq 0 ]; then
|
|
82
|
+
echo "GRADUATION_SUMMARY: total=0 resolved=0 vp_blocked=0 halts=0"
|
|
83
|
+
exit 1
|
|
84
|
+
fi
|
|
85
|
+
|
|
86
|
+
# Delegate the per-candidate join + VP-check to python for re-readable
|
|
87
|
+
# regex + dual-layout glob handling.
|
|
88
|
+
EVAL_RESULT=$(python3 - "$HOLDING_DIR" "$PROBLEMS_DIR" "${HELD_FILES[@]}" <<'PYEOF'
|
|
89
|
+
import os
|
|
90
|
+
import re
|
|
91
|
+
import sys
|
|
92
|
+
import glob
|
|
93
|
+
|
|
94
|
+
holding_dir = sys.argv[1]
|
|
95
|
+
problems_dir = sys.argv[2]
|
|
96
|
+
held_files = sys.argv[3:]
|
|
97
|
+
|
|
98
|
+
FILENAME_TICKET_RE = re.compile(r'-p(\d+)-', re.IGNORECASE)
|
|
99
|
+
BODY_TICKET_RE = re.compile(r'\bP(\d+)\b')
|
|
100
|
+
PRIORITY_LINE_RE = re.compile(r'^\*\*Priority\*\*:\s*(\d+)\b')
|
|
101
|
+
|
|
102
|
+
|
|
103
|
+
def find_ticket_file(ticket_id_padded: str):
|
|
104
|
+
"""Dual-tolerant glob per ADR-031 / RFC-002 migration window.
|
|
105
|
+
|
|
106
|
+
Returns (path, status_suffix) where status_suffix is one of
|
|
107
|
+
'open', 'known-error', 'verifying', 'closed', 'parked' or None
|
|
108
|
+
if no file resolves.
|
|
109
|
+
"""
|
|
110
|
+
# Per-state subdir layout
|
|
111
|
+
for state in ('open', 'known-error', 'verifying', 'closed', 'parked'):
|
|
112
|
+
candidates = glob.glob(os.path.join(problems_dir, state, f'{ticket_id_padded}-*.md'))
|
|
113
|
+
if candidates:
|
|
114
|
+
return candidates[0], state
|
|
115
|
+
# Flat layout
|
|
116
|
+
for state in ('open', 'known-error', 'verifying', 'closed', 'parked'):
|
|
117
|
+
candidates = glob.glob(os.path.join(problems_dir, f'{ticket_id_padded}-*.{state}.md'))
|
|
118
|
+
if candidates:
|
|
119
|
+
return candidates[0], state
|
|
120
|
+
return None, None
|
|
121
|
+
|
|
122
|
+
|
|
123
|
+
def extract_priority(ticket_path: str):
|
|
124
|
+
"""Read the `**Priority**: N (...)` line and return integer N, or None."""
|
|
125
|
+
try:
|
|
126
|
+
with open(ticket_path, 'r', encoding='utf-8') as f:
|
|
127
|
+
for line in f:
|
|
128
|
+
m = PRIORITY_LINE_RE.match(line.strip())
|
|
129
|
+
if m:
|
|
130
|
+
return int(m.group(1))
|
|
131
|
+
except (OSError, IOError):
|
|
132
|
+
return None
|
|
133
|
+
return None
|
|
134
|
+
|
|
135
|
+
|
|
136
|
+
def resolve_ticket_ids(changeset_path: str):
|
|
137
|
+
"""Apply Rule 1a join: filename convention primary, body-grep fallback.
|
|
138
|
+
|
|
139
|
+
Returns a list of zero-padded ticket IDs (e.g. ['085']) referenced by
|
|
140
|
+
this changeset. Empty list means halt-no-resolution per Rule 1a terminal.
|
|
141
|
+
"""
|
|
142
|
+
basename = os.path.basename(changeset_path)
|
|
143
|
+
# Primary: filename convention
|
|
144
|
+
filename_match = FILENAME_TICKET_RE.search(basename)
|
|
145
|
+
if filename_match:
|
|
146
|
+
return [f'{int(filename_match.group(1)):03d}']
|
|
147
|
+
|
|
148
|
+
# Fallback: body grep for P\d+ references
|
|
149
|
+
try:
|
|
150
|
+
with open(changeset_path, 'r', encoding='utf-8') as f:
|
|
151
|
+
body = f.read()
|
|
152
|
+
except (OSError, IOError):
|
|
153
|
+
return []
|
|
154
|
+
|
|
155
|
+
body_matches = BODY_TICKET_RE.findall(body)
|
|
156
|
+
if not body_matches:
|
|
157
|
+
return []
|
|
158
|
+
|
|
159
|
+
# De-duplicate while preserving order; zero-pad
|
|
160
|
+
seen = set()
|
|
161
|
+
ids = []
|
|
162
|
+
for raw_id in body_matches:
|
|
163
|
+
padded = f'{int(raw_id):03d}'
|
|
164
|
+
if padded not in seen:
|
|
165
|
+
seen.add(padded)
|
|
166
|
+
ids.append(padded)
|
|
167
|
+
return ids
|
|
168
|
+
|
|
169
|
+
|
|
170
|
+
total = 0
|
|
171
|
+
resolved = 0
|
|
172
|
+
vp_blocked = 0
|
|
173
|
+
halts = 0
|
|
174
|
+
|
|
175
|
+
for changeset_path in held_files:
|
|
176
|
+
total += 1
|
|
177
|
+
basename = os.path.basename(changeset_path)
|
|
178
|
+
ticket_ids = resolve_ticket_ids(changeset_path)
|
|
179
|
+
|
|
180
|
+
if not ticket_ids:
|
|
181
|
+
# Rule 1a terminal — halt-and-prompt
|
|
182
|
+
print(f'GRADUATION_CANDIDATE: changeset={basename} | ticket=- | priority=- | class=3a | status=halt-no-resolution')
|
|
183
|
+
halts += 1
|
|
184
|
+
continue
|
|
185
|
+
|
|
186
|
+
# Resolve each referenced ticket; collect (ticket_id, priority, status_suffix) triples
|
|
187
|
+
resolutions = []
|
|
188
|
+
unresolved_ids = []
|
|
189
|
+
for tid in ticket_ids:
|
|
190
|
+
path, suffix = find_ticket_file(tid)
|
|
191
|
+
if path is None:
|
|
192
|
+
unresolved_ids.append(tid)
|
|
193
|
+
continue
|
|
194
|
+
priority = extract_priority(path)
|
|
195
|
+
if priority is None:
|
|
196
|
+
unresolved_ids.append(tid)
|
|
197
|
+
continue
|
|
198
|
+
resolutions.append((tid, priority, suffix))
|
|
199
|
+
|
|
200
|
+
if not resolutions:
|
|
201
|
+
# All referenced tickets failed to resolve — halt
|
|
202
|
+
print(f'GRADUATION_CANDIDATE: changeset={basename} | ticket={",".join(f"P{i}" for i in ticket_ids)} | priority=- | class=3a | status=halt-no-resolution')
|
|
203
|
+
halts += 1
|
|
204
|
+
continue
|
|
205
|
+
|
|
206
|
+
# Rule 1a multi-ticket: max(Priority) across the referenced set
|
|
207
|
+
# Pick the resolution with the highest priority; report its ticket ID.
|
|
208
|
+
resolutions.sort(key=lambda r: r[1], reverse=True)
|
|
209
|
+
chosen_tid, chosen_priority, chosen_suffix = resolutions[0]
|
|
210
|
+
|
|
211
|
+
# Rule 2 VP carve-out
|
|
212
|
+
if chosen_suffix == 'verifying':
|
|
213
|
+
print(f'GRADUATION_CANDIDATE: changeset={basename} | ticket=P{chosen_tid} | priority={chosen_priority} | class=3a | status=vp-blocked')
|
|
214
|
+
vp_blocked += 1
|
|
215
|
+
continue
|
|
216
|
+
|
|
217
|
+
print(f'GRADUATION_CANDIDATE: changeset={basename} | ticket=P{chosen_tid} | priority={chosen_priority} | class=3a | status=resolved')
|
|
218
|
+
resolved += 1
|
|
219
|
+
|
|
220
|
+
print(f'GRADUATION_SUMMARY: total={total} resolved={resolved} vp_blocked={vp_blocked} halts={halts}')
|
|
221
|
+
PYEOF
|
|
222
|
+
)
|
|
223
|
+
PY_STATUS=$?
|
|
224
|
+
|
|
225
|
+
echo "$EVAL_RESULT"
|
|
226
|
+
|
|
227
|
+
if [ "$PY_STATUS" -ne 0 ]; then
|
|
228
|
+
exit "$PY_STATUS"
|
|
229
|
+
fi
|
|
230
|
+
|
|
231
|
+
exit 0
|
|
@@ -0,0 +1,272 @@
|
|
|
1
|
+
#!/usr/bin/env bats
|
|
2
|
+
# Behavioural-fixture coverage for packages/risk-scorer/scripts/evaluate-graduation.sh
|
|
3
|
+
# per ADR-052 (behavioural tests default) and ADR-061 (dogfood graduation criteria).
|
|
4
|
+
#
|
|
5
|
+
# Phase 2a coverage — orthogonal-gate class (Class 3a) only. Atomic-cohort
|
|
6
|
+
# class (Class 3b — Rule 3b RFC cohort enumeration) is deferred to Phase 2b.
|
|
7
|
+
# Maps to ADR-061 Confirmation criterion 2 items a-f (item g atomic-cohort
|
|
8
|
+
# lands in Phase 2b alongside the RFC enumeration logic).
|
|
9
|
+
|
|
10
|
+
setup() {
|
|
11
|
+
REPO_ROOT="$(cd "$(dirname "$BATS_TEST_FILENAME")/../../../.." && pwd)"
|
|
12
|
+
SCRIPT="$REPO_ROOT/packages/risk-scorer/scripts/evaluate-graduation.sh"
|
|
13
|
+
SHIM="$REPO_ROOT/packages/risk-scorer/bin/wr-risk-scorer-evaluate-graduation"
|
|
14
|
+
WORK_DIR="$(mktemp -d)"
|
|
15
|
+
cd "$WORK_DIR"
|
|
16
|
+
# Minimal git setup — fixture follows drain-register-queue.bats precedent
|
|
17
|
+
# so dual-layout problem-ticket lookup (flat + per-state subdir) exercises
|
|
18
|
+
# the same shape the script handles in canonical adopter trees.
|
|
19
|
+
git init --quiet
|
|
20
|
+
git config user.email "graduation-test@example.com"
|
|
21
|
+
git config user.name "Graduation Test"
|
|
22
|
+
git commit --quiet --allow-empty -m "init"
|
|
23
|
+
mkdir -p docs/changesets-holding docs/problems/open docs/problems/known-error \
|
|
24
|
+
docs/problems/verifying docs/problems/closed docs/problems/parked
|
|
25
|
+
}
|
|
26
|
+
|
|
27
|
+
teardown() {
|
|
28
|
+
cd /
|
|
29
|
+
rm -rf "$WORK_DIR"
|
|
30
|
+
}
|
|
31
|
+
|
|
32
|
+
# ----- Helpers -----
|
|
33
|
+
|
|
34
|
+
seed_problem() {
|
|
35
|
+
# seed_problem <id-padded> <state> <priority> [<extra-body>]
|
|
36
|
+
local id="$1" state="$2" priority="$3" extra="${4:-}"
|
|
37
|
+
local path
|
|
38
|
+
# Use per-state subdir layout (RFC-002 canonical post-migration shape)
|
|
39
|
+
path="docs/problems/${state}/${id}-fixture-ticket.md"
|
|
40
|
+
cat > "$path" <<EOF
|
|
41
|
+
# Problem ${id}: Fixture ticket
|
|
42
|
+
|
|
43
|
+
**Status**: ${state}
|
|
44
|
+
**Reported**: 2026-05-01
|
|
45
|
+
**Priority**: ${priority} (label) — Impact: 3 x Likelihood: $(( priority / 3 ))
|
|
46
|
+
|
|
47
|
+
## Description
|
|
48
|
+
|
|
49
|
+
Fixture body for graduation-evaluator tests.
|
|
50
|
+
|
|
51
|
+
${extra}
|
|
52
|
+
EOF
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
seed_problem_flat() {
|
|
56
|
+
# seed_problem_flat <id-padded> <state> <priority>
|
|
57
|
+
# Exercises the flat-layout half of the RFC-002 dual-tolerant glob.
|
|
58
|
+
local id="$1" state="$2" priority="$3"
|
|
59
|
+
cat > "docs/problems/${id}-fixture-flat.${state}.md" <<EOF
|
|
60
|
+
# Problem ${id}: Fixture flat ticket
|
|
61
|
+
|
|
62
|
+
**Status**: ${state}
|
|
63
|
+
**Reported**: 2026-05-01
|
|
64
|
+
**Priority**: ${priority} (label) — Impact: 3 x Likelihood: $(( priority / 3 ))
|
|
65
|
+
|
|
66
|
+
## Description
|
|
67
|
+
|
|
68
|
+
Flat-layout fixture body for graduation-evaluator tests.
|
|
69
|
+
EOF
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
seed_changeset() {
|
|
73
|
+
# seed_changeset <filename> [<body>]
|
|
74
|
+
local filename="$1" body="${2:-Default fixture changeset body.}"
|
|
75
|
+
cat > "docs/changesets-holding/${filename}" <<EOF
|
|
76
|
+
---
|
|
77
|
+
'@windyroad/itil': minor
|
|
78
|
+
---
|
|
79
|
+
|
|
80
|
+
${body}
|
|
81
|
+
EOF
|
|
82
|
+
}
|
|
83
|
+
|
|
84
|
+
# ----- Smoke tests -----
|
|
85
|
+
|
|
86
|
+
@test "shim wrapper exists and is executable" {
|
|
87
|
+
[ -x "$SHIM" ]
|
|
88
|
+
}
|
|
89
|
+
|
|
90
|
+
@test "shim resolves canonical script (not exit 127)" {
|
|
91
|
+
# Empty holding-area → exit 1 (no-op caller signal); but not exit 127.
|
|
92
|
+
run "$SHIM" "$WORK_DIR"
|
|
93
|
+
[ "$status" -ne 127 ]
|
|
94
|
+
}
|
|
95
|
+
|
|
96
|
+
@test "missing docs/ → exit 2 (invalid project root)" {
|
|
97
|
+
cd "$BATS_TEST_TMPDIR"
|
|
98
|
+
empty_dir=$(mktemp -d)
|
|
99
|
+
run bash "$SCRIPT" "$empty_dir"
|
|
100
|
+
[ "$status" -eq 2 ]
|
|
101
|
+
}
|
|
102
|
+
|
|
103
|
+
@test "missing holding dir → exit 1 (no-op signal)" {
|
|
104
|
+
rm -rf docs/changesets-holding
|
|
105
|
+
run bash "$SCRIPT" "$WORK_DIR"
|
|
106
|
+
[ "$status" -eq 1 ]
|
|
107
|
+
echo "$output" | grep -q 'GRADUATION_SUMMARY: total=0'
|
|
108
|
+
}
|
|
109
|
+
|
|
110
|
+
@test "empty holding dir → exit 1 (no-op signal)" {
|
|
111
|
+
run bash "$SCRIPT" "$WORK_DIR"
|
|
112
|
+
[ "$status" -eq 1 ]
|
|
113
|
+
echo "$output" | grep -q 'GRADUATION_SUMMARY: total=0'
|
|
114
|
+
}
|
|
115
|
+
|
|
116
|
+
@test "README.md in holding-area is ignored" {
|
|
117
|
+
echo "# Holding Area" > docs/changesets-holding/README.md
|
|
118
|
+
run bash "$SCRIPT" "$WORK_DIR"
|
|
119
|
+
# No other entries — exit 1 (empty after exclusion)
|
|
120
|
+
[ "$status" -eq 1 ]
|
|
121
|
+
echo "$output" | grep -q 'GRADUATION_SUMMARY: total=0'
|
|
122
|
+
}
|
|
123
|
+
|
|
124
|
+
# ----- ADR-061 Confirmation criterion 2 cases -----
|
|
125
|
+
|
|
126
|
+
# Case (a) — filename-convention join resolves correctly
|
|
127
|
+
@test "case (a): filename-convention join — wr-itil-p085-...md → P085 → Priority" {
|
|
128
|
+
seed_problem "085" "open" "9"
|
|
129
|
+
seed_changeset "wr-itil-p085-assistant-output-gate.md"
|
|
130
|
+
run bash "$SCRIPT" "$WORK_DIR"
|
|
131
|
+
[ "$status" -eq 0 ]
|
|
132
|
+
echo "$output" | grep -q 'changeset=wr-itil-p085-assistant-output-gate.md'
|
|
133
|
+
echo "$output" | grep -q 'ticket=P085'
|
|
134
|
+
echo "$output" | grep -q 'priority=9'
|
|
135
|
+
echo "$output" | grep -q 'status=resolved'
|
|
136
|
+
echo "$output" | grep -q 'class=3a'
|
|
137
|
+
echo "$output" | grep -q 'GRADUATION_SUMMARY: total=1 resolved=1 vp_blocked=0 halts=0'
|
|
138
|
+
}
|
|
139
|
+
|
|
140
|
+
# Case (b) — body-grep fallback resolves when filename lacks convention
|
|
141
|
+
@test "case (b): body-grep fallback — non-conventional filename resolves via P<NNN> in body" {
|
|
142
|
+
seed_problem "100" "known-error" "12"
|
|
143
|
+
seed_changeset "feature-rollout-cohort.md" "Fixes P100. Related to feature work."
|
|
144
|
+
run bash "$SCRIPT" "$WORK_DIR"
|
|
145
|
+
[ "$status" -eq 0 ]
|
|
146
|
+
echo "$output" | grep -q 'changeset=feature-rollout-cohort.md'
|
|
147
|
+
echo "$output" | grep -q 'ticket=P100'
|
|
148
|
+
echo "$output" | grep -q 'priority=12'
|
|
149
|
+
echo "$output" | grep -q 'status=resolved'
|
|
150
|
+
}
|
|
151
|
+
|
|
152
|
+
# Case (c) — multi-ticket changeset uses max(Priority)
|
|
153
|
+
@test "case (c): multi-ticket changeset uses max(Priority) across referenced set" {
|
|
154
|
+
seed_problem "200" "open" "6"
|
|
155
|
+
seed_problem "201" "open" "15"
|
|
156
|
+
seed_problem "202" "open" "9"
|
|
157
|
+
seed_changeset "multi-ticket-cohort.md" "References P200, P201, and P202 in body for max-priority test."
|
|
158
|
+
run bash "$SCRIPT" "$WORK_DIR"
|
|
159
|
+
[ "$status" -eq 0 ]
|
|
160
|
+
# Max priority across P200(6) P201(15) P202(9) is 15 → ticket=P201
|
|
161
|
+
echo "$output" | grep -q 'changeset=multi-ticket-cohort.md'
|
|
162
|
+
echo "$output" | grep -q 'ticket=P201'
|
|
163
|
+
echo "$output" | grep -q 'priority=15'
|
|
164
|
+
echo "$output" | grep -q 'status=resolved'
|
|
165
|
+
}
|
|
166
|
+
|
|
167
|
+
# Case (d) — halt-and-prompt when no resolution path succeeds
|
|
168
|
+
@test "case (d.1): halt-no-resolution when filename lacks convention AND body has no P<NNN>" {
|
|
169
|
+
seed_changeset "no-ticket-reference.md" "Body has no problem-ticket references at all."
|
|
170
|
+
run bash "$SCRIPT" "$WORK_DIR"
|
|
171
|
+
[ "$status" -eq 0 ]
|
|
172
|
+
echo "$output" | grep -q 'changeset=no-ticket-reference.md'
|
|
173
|
+
echo "$output" | grep -q 'status=halt-no-resolution'
|
|
174
|
+
echo "$output" | grep -q 'GRADUATION_SUMMARY: total=1 resolved=0 vp_blocked=0 halts=1'
|
|
175
|
+
}
|
|
176
|
+
|
|
177
|
+
@test "case (d.2): halt-no-resolution when filename references missing ticket" {
|
|
178
|
+
# Filename references P999 but ticket file does not exist in fixture
|
|
179
|
+
seed_changeset "wr-itil-p999-orphan.md"
|
|
180
|
+
run bash "$SCRIPT" "$WORK_DIR"
|
|
181
|
+
[ "$status" -eq 0 ]
|
|
182
|
+
echo "$output" | grep -q 'status=halt-no-resolution'
|
|
183
|
+
echo "$output" | grep -q 'halts=1'
|
|
184
|
+
}
|
|
185
|
+
|
|
186
|
+
# Case (e) — VP-blocked ticket emitted with vp-blocked marker
|
|
187
|
+
@test "case (e): VP-blocked ticket emits status=vp-blocked (Rule 2 carve-out)" {
|
|
188
|
+
seed_problem "300" "verifying" "10"
|
|
189
|
+
seed_changeset "wr-itil-p300-mid-verification.md"
|
|
190
|
+
run bash "$SCRIPT" "$WORK_DIR"
|
|
191
|
+
[ "$status" -eq 0 ]
|
|
192
|
+
echo "$output" | grep -q 'changeset=wr-itil-p300-mid-verification.md'
|
|
193
|
+
echo "$output" | grep -q 'ticket=P300'
|
|
194
|
+
echo "$output" | grep -q 'priority=10'
|
|
195
|
+
echo "$output" | grep -q 'status=vp-blocked'
|
|
196
|
+
echo "$output" | grep -q 'GRADUATION_SUMMARY: total=1 resolved=0 vp_blocked=1 halts=0'
|
|
197
|
+
}
|
|
198
|
+
|
|
199
|
+
# Case (f) — status-agnostic Priority lookup (open, known-error, closed all resolve)
|
|
200
|
+
@test "case (f.1): open ticket resolves" {
|
|
201
|
+
seed_problem "400" "open" "8"
|
|
202
|
+
seed_changeset "wr-itil-p400-open.md"
|
|
203
|
+
run bash "$SCRIPT" "$WORK_DIR"
|
|
204
|
+
[ "$status" -eq 0 ]
|
|
205
|
+
echo "$output" | grep -q 'ticket=P400'
|
|
206
|
+
echo "$output" | grep -q 'status=resolved'
|
|
207
|
+
}
|
|
208
|
+
|
|
209
|
+
@test "case (f.2): known-error ticket resolves" {
|
|
210
|
+
seed_problem "401" "known-error" "8"
|
|
211
|
+
seed_changeset "wr-itil-p401-ke.md"
|
|
212
|
+
run bash "$SCRIPT" "$WORK_DIR"
|
|
213
|
+
[ "$status" -eq 0 ]
|
|
214
|
+
echo "$output" | grep -q 'ticket=P401'
|
|
215
|
+
echo "$output" | grep -q 'status=resolved'
|
|
216
|
+
}
|
|
217
|
+
|
|
218
|
+
@test "case (f.3): closed ticket resolves (Priority still readable)" {
|
|
219
|
+
seed_problem "402" "closed" "8"
|
|
220
|
+
seed_changeset "wr-itil-p402-closed.md"
|
|
221
|
+
run bash "$SCRIPT" "$WORK_DIR"
|
|
222
|
+
[ "$status" -eq 0 ]
|
|
223
|
+
echo "$output" | grep -q 'ticket=P402'
|
|
224
|
+
echo "$output" | grep -q 'status=resolved'
|
|
225
|
+
}
|
|
226
|
+
|
|
227
|
+
@test "case (f.4): parked ticket resolves" {
|
|
228
|
+
seed_problem "403" "parked" "8"
|
|
229
|
+
seed_changeset "wr-itil-p403-parked.md"
|
|
230
|
+
run bash "$SCRIPT" "$WORK_DIR"
|
|
231
|
+
[ "$status" -eq 0 ]
|
|
232
|
+
echo "$output" | grep -q 'ticket=P403'
|
|
233
|
+
echo "$output" | grep -q 'status=resolved'
|
|
234
|
+
}
|
|
235
|
+
|
|
236
|
+
# Dual-tolerant layout coverage (ADR-031 / RFC-002 migration window)
|
|
237
|
+
@test "flat-layout ticket resolves (RFC-002 pre-migration shape)" {
|
|
238
|
+
seed_problem_flat "500" "open" "12"
|
|
239
|
+
seed_changeset "wr-itil-p500-flat-layout.md"
|
|
240
|
+
run bash "$SCRIPT" "$WORK_DIR"
|
|
241
|
+
[ "$status" -eq 0 ]
|
|
242
|
+
echo "$output" | grep -q 'ticket=P500'
|
|
243
|
+
echo "$output" | grep -q 'priority=12'
|
|
244
|
+
echo "$output" | grep -q 'status=resolved'
|
|
245
|
+
}
|
|
246
|
+
|
|
247
|
+
# Mixed-state holding-area — multiple changesets in one run
|
|
248
|
+
@test "mixed-state holding-area — resolved + vp-blocked + halt in single run" {
|
|
249
|
+
seed_problem "600" "open" "9"
|
|
250
|
+
seed_problem "601" "verifying" "12"
|
|
251
|
+
seed_problem "602" "open" "6"
|
|
252
|
+
seed_changeset "wr-itil-p600-resolved.md"
|
|
253
|
+
seed_changeset "wr-itil-p601-vp.md"
|
|
254
|
+
seed_changeset "no-resolution-ref.md" "Generic body, no P references."
|
|
255
|
+
run bash "$SCRIPT" "$WORK_DIR"
|
|
256
|
+
[ "$status" -eq 0 ]
|
|
257
|
+
# Counts: 3 total, 1 resolved (P600), 1 vp-blocked (P601), 1 halt
|
|
258
|
+
echo "$output" | grep -q 'GRADUATION_SUMMARY: total=3 resolved=1 vp_blocked=1 halts=1'
|
|
259
|
+
}
|
|
260
|
+
|
|
261
|
+
# Filename-convention takes precedence over body-grep when both present
|
|
262
|
+
@test "filename takes precedence over body — wr-itil-p700-fix.md with P800 in body resolves to P700" {
|
|
263
|
+
seed_problem "700" "open" "5"
|
|
264
|
+
seed_problem "800" "open" "20"
|
|
265
|
+
seed_changeset "wr-itil-p700-fix.md" "Also references P800 in body (should be ignored — filename wins)."
|
|
266
|
+
run bash "$SCRIPT" "$WORK_DIR"
|
|
267
|
+
[ "$status" -eq 0 ]
|
|
268
|
+
echo "$output" | grep -q 'ticket=P700'
|
|
269
|
+
echo "$output" | grep -q 'priority=5'
|
|
270
|
+
# Confirm body-referenced P800 was NOT picked up
|
|
271
|
+
! echo "$output" | grep -q 'ticket=P800'
|
|
272
|
+
}
|