okstra 0.54.0 → 0.56.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/bin/okstra +24 -7
- package/docs/project-structure-overview.md +0 -1
- package/docs/superpowers/plans/2026-05-25-okstra-project-root-rename.md +0 -1
- package/docs/superpowers/plans/2026-06-07-stage-conformance-qa-phase2.md +275 -0
- package/docs/superpowers/plans/2026-06-07-stage-conformance-qa-phase3.md +282 -0
- package/docs/superpowers/plans/2026-06-07-stage-conformance-qa-phase4a.md +147 -0
- package/docs/superpowers/plans/2026-06-07-stage-conformance-qa-phase4b.md +262 -0
- package/docs/superpowers/plans/2026-06-07-stage-conformance-qa-phase4c.md +184 -0
- package/docs/superpowers/plans/2026-06-07-stage-conformance-qa-phase4d.md +88 -0
- package/docs/superpowers/plans/2026-06-07-stage-conformance-qa-phase4e.md +250 -0
- package/docs/superpowers/plans/2026-06-07-stage-conformance-qa.md +409 -0
- package/docs/superpowers/specs/2026-06-07-stage-conformance-qa-design.md +169 -0
- package/package.json +1 -1
- package/runtime/BUILD.json +2 -2
- package/runtime/agents/workers/report-writer-worker.md +1 -1
- package/runtime/bin/lib/okstra/cli.sh +5 -1
- package/runtime/bin/lib/okstra/usage.sh +5 -0
- package/runtime/bin/okstra-inject-report-index.py +66 -0
- package/runtime/bin/okstra.sh +1 -0
- package/runtime/prompts/profiles/_implementation-verifier.md +23 -2
- package/runtime/prompts/profiles/final-verification.md +1 -0
- package/runtime/prompts/profiles/implementation-planning.md +4 -0
- package/runtime/prompts/profiles/improvement-discovery.md +1 -0
- package/runtime/python/okstra_ctl/clarification_items.py +10 -1
- package/runtime/python/okstra_ctl/conformance.py +270 -0
- package/runtime/python/okstra_ctl/paths.py +2 -0
- package/runtime/python/okstra_ctl/render_final_report.py +221 -2
- package/runtime/python/okstra_ctl/report_views.py +23 -4
- package/runtime/python/okstra_ctl/run.py +29 -0
- package/runtime/skills/okstra-run/SKILL.md +12 -0
- package/runtime/skills/okstra-setup/SKILL.md +35 -0
- package/runtime/templates/reports/i18n/en.json +6 -0
- package/runtime/templates/reports/i18n/ko.json +6 -0
- package/runtime/validators/lib/fixtures.sh +9 -0
- package/runtime/validators/validate-implementation-plan-stages.py +28 -3
- package/runtime/validators/validate-run.py +136 -1
- package/runtime/validators/validate_improvement_report.py +5 -1
- package/src/okstra-dirs.mjs +1 -1
- package/src/migrate.mjs +0 -146
package/bin/okstra
CHANGED
|
@@ -4,19 +4,36 @@ import { getPackageVersion } from "../src/version.mjs";
|
|
|
4
4
|
const COMMANDS = new Map([
|
|
5
5
|
["paths", () => import("../src/paths.mjs").then((m) => m.run)],
|
|
6
6
|
["install", () => import("../src/install.mjs").then((m) => m.runInstall)],
|
|
7
|
-
[
|
|
8
|
-
|
|
7
|
+
[
|
|
8
|
+
"ensure-installed",
|
|
9
|
+
() => import("../src/install.mjs").then((m) => m.runEnsureInstalled),
|
|
10
|
+
],
|
|
11
|
+
[
|
|
12
|
+
"uninstall",
|
|
13
|
+
() => import("../src/uninstall.mjs").then((m) => m.runUninstall),
|
|
14
|
+
],
|
|
9
15
|
["doctor", () => import("../src/doctor.mjs").then((m) => m.run)],
|
|
10
16
|
["setup", () => import("../src/setup.mjs").then((m) => m.run)],
|
|
11
|
-
[
|
|
17
|
+
[
|
|
18
|
+
"check-project",
|
|
19
|
+
() => import("../src/check-project.mjs").then((m) => m.run),
|
|
20
|
+
],
|
|
12
21
|
["config", () => import("../src/config.mjs").then((m) => m.run)],
|
|
13
|
-
["migrate", () => import("../src/migrate.mjs").then((m) => m.run)],
|
|
14
22
|
["task-list", () => import("../src/task-list.mjs").then((m) => m.run)],
|
|
15
23
|
["task-show", () => import("../src/task-show.mjs").then((m) => m.run)],
|
|
16
24
|
["context-cost", () => import("../src/context-cost.mjs").then((m) => m.run)],
|
|
17
|
-
[
|
|
18
|
-
|
|
19
|
-
|
|
25
|
+
[
|
|
26
|
+
"worktree-lookup",
|
|
27
|
+
() => import("../src/worktree-lookup.mjs").then((m) => m.run),
|
|
28
|
+
],
|
|
29
|
+
[
|
|
30
|
+
"plan-validate",
|
|
31
|
+
() => import("../src/plan-validate.mjs").then((m) => m.run),
|
|
32
|
+
],
|
|
33
|
+
[
|
|
34
|
+
"render-bundle",
|
|
35
|
+
() => import("../src/render-bundle.mjs").then((m) => m.run),
|
|
36
|
+
],
|
|
20
37
|
["render-views", () => import("../src/render-views.mjs").then((m) => m.run)],
|
|
21
38
|
["wizard", () => import("../src/wizard.mjs").then((m) => m.run)],
|
|
22
39
|
["token-usage", () => import("../src/token-usage.mjs").then((m) => m.run)],
|
|
@@ -137,7 +137,6 @@ So `runtime/` is not the only npm-published content. It is the install payload c
|
|
|
137
137
|
| `setup` | `src/setup.mjs` | Create/update `<PROJECT_ROOT>/.okstra/project.json` |
|
|
138
138
|
| `check-project` | `src/check-project.mjs` | Verify project registration |
|
|
139
139
|
| `config` | `src/config.mjs` | Read/write project/global settings such as PR template path |
|
|
140
|
-
| `migrate` | `src/migrate.mjs` | Move legacy `.project-docs/okstra/` state into `.okstra/` |
|
|
141
140
|
| `task-list`, `task-show` | `src/task-list.mjs`, `src/task-show.mjs` | Task/run introspection for skills |
|
|
142
141
|
| `context-cost` | `src/context-cost.mjs` | Estimate task bundle file/read context cost |
|
|
143
142
|
| `worktree-lookup` | `src/worktree-lookup.mjs` | Look up a task-key's registered worktree |
|
|
@@ -95,7 +95,6 @@
|
|
|
95
95
|
- [ ] `MigrationPlan` 은 dataclass, JSON 직렬화 가능 (dry-run 출력에 그대로 사용).
|
|
96
96
|
|
|
97
97
|
### Task 3.2: 진입점 3개 (단일 코어 호출 보장)
|
|
98
|
-
- [ ] Node CLI: `bin/commands/migrate.mjs` — argparse + Python 모듈 호출.
|
|
99
98
|
- [ ] Bash 진입점: `scripts/lib/okstra-ctl/cmd-migrate.sh` — thin adapter.
|
|
100
99
|
- [ ] `bin/okstra` dispatcher 에 `migrate` 서브커맨드 등록.
|
|
101
100
|
- [ ] `--dry-run` 기본, `--apply` 로 실제 실행, `--quiet` 로 출력 최소화.
|
|
@@ -0,0 +1,275 @@
|
|
|
1
|
+
# Stage Conformance QA — Phase 2 (게이트 판정 코어) Implementation Plan
|
|
2
|
+
|
|
3
|
+
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
4
|
+
|
|
5
|
+
**Goal:** stage 별 conformance entry + 실행 결과를 받아 "PASS / WAIVED / EXEMPT / BLOCKING" 을 결정하는 **순수 판정 로직**을 `conformance.py` 에 추가한다(미실행·FAIL·MISSING = BLOCKING, 면제·waiver = 통과). 이게 DEV-9184 의 "silent mock-green" 을 막는 두뇌다.
|
|
6
|
+
|
|
7
|
+
**Architecture:** Phase 1(매니페스트 검증 + `QA-RESULT` 파서)의 사이드카(부수효과 없는) 확장. wiring(verifier 가 결과 기록, validate-run 이 판정 적용, run.py 진입 검증, planning 이 스크립트 생성)은 **Phase 3** 로 분리한다. 본 Phase 는 외부 파일을 읽지 않는 순수 함수만 추가하므로 완전 TDD 가능하다. SSOT: [`docs/superpowers/specs/2026-06-07-stage-conformance-qa-design.md`](../specs/2026-06-07-stage-conformance-qa-design.md) §6(실행·판정), §7(우회).
|
|
8
|
+
|
|
9
|
+
**Tech Stack:** Python 3 (stdlib + pytest).
|
|
10
|
+
|
|
11
|
+
**전제:** Phase 1 완료 — `scripts/okstra_ctl/conformance.py` 에 `validate_conformance_manifest`, `parse_qa_result`, `QaResult`, `CAPABILITY_WHITELIST` 존재.
|
|
12
|
+
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
## 판정 규칙 (spec §6, §7)
|
|
16
|
+
|
|
17
|
+
단일 stage entry + 그 stage 의 실행 결과(`QaResult | None`)로부터:
|
|
18
|
+
|
|
19
|
+
| 조건 | status | ok | conditional |
|
|
20
|
+
|---|---|---|---|
|
|
21
|
+
| `exemption` 있음 | EXEMPT | True | False |
|
|
22
|
+
| `waiver` 있음 | WAIVED | True | **True** (conformance 미검증 — 사용자 확인) |
|
|
23
|
+
| 결과 없음(`None`) | BLOCKING | False | False |
|
|
24
|
+
| `overall == "MISSING"` (마커 없음) | BLOCKING | False | False |
|
|
25
|
+
| `overall == "FAIL"` | BLOCKING | False | False |
|
|
26
|
+
| `overall == "PASS"` | PASS | True | False |
|
|
27
|
+
|
|
28
|
+
판정 우선순위: exemption → waiver → 결과 평가. (Phase 3 가 "면제 선언했지만 diff 가 실제 surface 를 건드림" 교차검증을 추가한다 — 본 Phase 는 선언을 신뢰한다.)
|
|
29
|
+
|
|
30
|
+
---
|
|
31
|
+
|
|
32
|
+
## Task 1: 게이트 판정 (`decide_conformance_gate`)
|
|
33
|
+
|
|
34
|
+
**Files:**
|
|
35
|
+
- Modify: `scripts/okstra_ctl/conformance.py`
|
|
36
|
+
- Test: `tests/test_okstra_ctl_conformance.py`
|
|
37
|
+
|
|
38
|
+
- [ ] **Step 1: 실패 테스트 추가**
|
|
39
|
+
|
|
40
|
+
```python
|
|
41
|
+
# tests/test_okstra_ctl_conformance.py 의 import 에 추가
|
|
42
|
+
from okstra_ctl.conformance import ( # noqa: E402
|
|
43
|
+
ConformanceVerdict,
|
|
44
|
+
QaResult,
|
|
45
|
+
decide_conformance_gate,
|
|
46
|
+
)
|
|
47
|
+
|
|
48
|
+
|
|
49
|
+
def _gate_entry(**over):
|
|
50
|
+
base = {"stageKey": "t-stage-1", "exemption": None, "waiver": None}
|
|
51
|
+
base.update(over)
|
|
52
|
+
return base
|
|
53
|
+
|
|
54
|
+
|
|
55
|
+
def test_gate_pass_when_result_pass():
|
|
56
|
+
v = decide_conformance_gate(_gate_entry(), QaResult(overall="PASS", requirements={}))
|
|
57
|
+
assert isinstance(v, ConformanceVerdict)
|
|
58
|
+
assert v.status == "PASS" and v.ok is True and v.conditional is False
|
|
59
|
+
|
|
60
|
+
|
|
61
|
+
def test_gate_blocking_when_no_result():
|
|
62
|
+
v = decide_conformance_gate(_gate_entry(), None)
|
|
63
|
+
assert v.status == "BLOCKING" and v.ok is False
|
|
64
|
+
assert "never ran" in v.message
|
|
65
|
+
|
|
66
|
+
|
|
67
|
+
def test_gate_blocking_when_marker_missing():
|
|
68
|
+
v = decide_conformance_gate(_gate_entry(), QaResult(overall="MISSING", requirements={}))
|
|
69
|
+
assert v.status == "BLOCKING" and v.ok is False
|
|
70
|
+
assert "QA-RESULT" in v.message
|
|
71
|
+
|
|
72
|
+
|
|
73
|
+
def test_gate_blocking_when_fail():
|
|
74
|
+
v = decide_conformance_gate(_gate_entry(), QaResult(overall="FAIL", requirements={}))
|
|
75
|
+
assert v.status == "BLOCKING" and v.ok is False
|
|
76
|
+
|
|
77
|
+
|
|
78
|
+
def test_gate_exempt_passes_with_reason():
|
|
79
|
+
entry = _gate_entry(exemption={"reason": "doc-only", "declaredAt": "2026-06-07"})
|
|
80
|
+
v = decide_conformance_gate(entry, None)
|
|
81
|
+
assert v.status == "EXEMPT" and v.ok is True and v.conditional is False
|
|
82
|
+
assert "doc-only" in v.message
|
|
83
|
+
|
|
84
|
+
|
|
85
|
+
def test_gate_waiver_is_conditional():
|
|
86
|
+
entry = _gate_entry(waiver={"acknowledgedBy": "user", "reason": "replica down",
|
|
87
|
+
"at": "2026-06-07", "scope": ["db"]})
|
|
88
|
+
v = decide_conformance_gate(entry, None)
|
|
89
|
+
assert v.status == "WAIVED" and v.ok is True and v.conditional is True
|
|
90
|
+
assert "user" in v.message and "replica down" in v.message
|
|
91
|
+
|
|
92
|
+
|
|
93
|
+
def test_gate_exemption_takes_priority_over_failing_result():
|
|
94
|
+
entry = _gate_entry(exemption={"reason": "n/a", "declaredAt": "2026-06-07"})
|
|
95
|
+
v = decide_conformance_gate(entry, QaResult(overall="FAIL", requirements={}))
|
|
96
|
+
assert v.status == "EXEMPT" and v.ok is True
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
- [ ] **Step 2: 실패 확인**
|
|
100
|
+
|
|
101
|
+
Run: `python3 -m pytest tests/test_okstra_ctl_conformance.py -q`
|
|
102
|
+
Expected: FAIL — `cannot import name 'ConformanceVerdict'` / `decide_conformance_gate`
|
|
103
|
+
|
|
104
|
+
- [ ] **Step 3: 구현 추가**
|
|
105
|
+
|
|
106
|
+
`conformance.py` 의 `parse_qa_result` 정의 다음에 추가:
|
|
107
|
+
|
|
108
|
+
```python
|
|
109
|
+
@dataclass
|
|
110
|
+
class ConformanceVerdict:
|
|
111
|
+
stage_key: str
|
|
112
|
+
status: str # "PASS" | "BLOCKING" | "WAIVED" | "EXEMPT"
|
|
113
|
+
ok: bool # 진행 허용 여부 (PASS/WAIVED/EXEMPT 면 True)
|
|
114
|
+
conditional: bool # WAIVED 일 때만 True — conformance 미검증(사용자 확인)
|
|
115
|
+
message: str
|
|
116
|
+
|
|
117
|
+
|
|
118
|
+
def decide_conformance_gate(entry: dict, result: object) -> ConformanceVerdict:
|
|
119
|
+
"""단일 stage entry + 실행 결과(`QaResult | None`)로 게이트 판정.
|
|
120
|
+
|
|
121
|
+
우선순위: exemption → waiver → 결과 평가. 미실행/MISSING/FAIL 은 BLOCKING.
|
|
122
|
+
면제·waiver 의 형태 검증은 `validate_conformance_manifest` 가 이미 보장한다.
|
|
123
|
+
"""
|
|
124
|
+
key = entry.get("stageKey", "<unknown>")
|
|
125
|
+
exemption = entry.get("exemption")
|
|
126
|
+
if exemption:
|
|
127
|
+
return ConformanceVerdict(
|
|
128
|
+
key, "EXEMPT", True, False,
|
|
129
|
+
f"conformance exempted: {exemption.get('reason', '')}",
|
|
130
|
+
)
|
|
131
|
+
waiver = entry.get("waiver")
|
|
132
|
+
if waiver:
|
|
133
|
+
return ConformanceVerdict(
|
|
134
|
+
key, "WAIVED", True, True,
|
|
135
|
+
f"conformance waived by {waiver.get('acknowledgedBy', '?')}: "
|
|
136
|
+
f"{waiver.get('reason', '')}",
|
|
137
|
+
)
|
|
138
|
+
overall = getattr(result, "overall", None) if result is not None else None
|
|
139
|
+
if overall == "PASS":
|
|
140
|
+
return ConformanceVerdict(key, "PASS", True, False, "conformance PASS")
|
|
141
|
+
if overall is None:
|
|
142
|
+
return ConformanceVerdict(
|
|
143
|
+
key, "BLOCKING", False, False,
|
|
144
|
+
"conformance script never ran (no result recorded)",
|
|
145
|
+
)
|
|
146
|
+
if overall == "MISSING":
|
|
147
|
+
return ConformanceVerdict(
|
|
148
|
+
key, "BLOCKING", False, False,
|
|
149
|
+
"conformance script ran but emitted no QA-RESULT marker",
|
|
150
|
+
)
|
|
151
|
+
return ConformanceVerdict(key, "BLOCKING", False, False, f"conformance {overall}")
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
- [ ] **Step 4: 통과 확인**
|
|
155
|
+
|
|
156
|
+
Run: `python3 -m pytest tests/test_okstra_ctl_conformance.py -q`
|
|
157
|
+
Expected: PASS (24 passed — 기존 17 + 신규 7)
|
|
158
|
+
|
|
159
|
+
- [ ] **Step 5: 커밋**
|
|
160
|
+
|
|
161
|
+
```bash
|
|
162
|
+
git add scripts/okstra_ctl/conformance.py tests/test_okstra_ctl_conformance.py
|
|
163
|
+
git commit -m "feat(okstra_ctl/conformance): stage 게이트 판정(decide_conformance_gate)"
|
|
164
|
+
```
|
|
165
|
+
|
|
166
|
+
## Task 2: 사이드카 변환 + 매니페스트 일괄 평가
|
|
167
|
+
|
|
168
|
+
**Files:**
|
|
169
|
+
- Modify: `scripts/okstra_ctl/conformance.py`
|
|
170
|
+
- Test: `tests/test_okstra_ctl_conformance.py`
|
|
171
|
+
|
|
172
|
+
- [ ] **Step 1: 실패 테스트 추가**
|
|
173
|
+
|
|
174
|
+
```python
|
|
175
|
+
from okstra_ctl.conformance import ( # noqa: E402 (위 import 에 합쳐도 됨)
|
|
176
|
+
evaluate_conformance,
|
|
177
|
+
qa_result_from_dict,
|
|
178
|
+
)
|
|
179
|
+
|
|
180
|
+
|
|
181
|
+
def test_qa_result_from_dict_roundtrip():
|
|
182
|
+
r = qa_result_from_dict({"overall": "PASS", "requirements": {"R-001": {"status": "PASS"}}})
|
|
183
|
+
assert r.overall == "PASS" and r.requirements["R-001"]["status"] == "PASS"
|
|
184
|
+
|
|
185
|
+
|
|
186
|
+
def test_qa_result_from_dict_defaults_to_missing():
|
|
187
|
+
assert qa_result_from_dict(None).overall == "MISSING"
|
|
188
|
+
assert qa_result_from_dict({"overall": "bogus"}).overall == "MISSING"
|
|
189
|
+
assert qa_result_from_dict({}).requirements == {}
|
|
190
|
+
|
|
191
|
+
|
|
192
|
+
def test_evaluate_conformance_mixes_verdicts():
|
|
193
|
+
manifest = {"entries": [
|
|
194
|
+
{"stageKey": "t-stage-1", "exemption": None, "waiver": None}, # PASS
|
|
195
|
+
{"stageKey": "t-stage-2", "exemption": None, "waiver": None}, # BLOCKING (no result)
|
|
196
|
+
{"stageKey": "t-stage-3",
|
|
197
|
+
"exemption": {"reason": "doc", "declaredAt": "x"}, "waiver": None}, # EXEMPT
|
|
198
|
+
]}
|
|
199
|
+
results = {"t-stage-1": QaResult(overall="PASS", requirements={})}
|
|
200
|
+
verdicts = evaluate_conformance(manifest, results)
|
|
201
|
+
by_key = {v.stage_key: v for v in verdicts}
|
|
202
|
+
assert by_key["t-stage-1"].status == "PASS"
|
|
203
|
+
assert by_key["t-stage-2"].status == "BLOCKING"
|
|
204
|
+
assert by_key["t-stage-3"].status == "EXEMPT"
|
|
205
|
+
|
|
206
|
+
|
|
207
|
+
def test_evaluate_conformance_empty_manifest():
|
|
208
|
+
assert evaluate_conformance(None, {}) == []
|
|
209
|
+
assert evaluate_conformance({"entries": []}, {}) == []
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
- [ ] **Step 2: 실패 확인**
|
|
213
|
+
|
|
214
|
+
Run: `python3 -m pytest tests/test_okstra_ctl_conformance.py -q`
|
|
215
|
+
Expected: FAIL — `cannot import name 'evaluate_conformance'` / `qa_result_from_dict`
|
|
216
|
+
|
|
217
|
+
- [ ] **Step 3: 구현 추가**
|
|
218
|
+
|
|
219
|
+
`conformance.py` 의 `decide_conformance_gate` 다음에 추가:
|
|
220
|
+
|
|
221
|
+
```python
|
|
222
|
+
def qa_result_from_dict(data: object) -> QaResult:
|
|
223
|
+
"""결과 사이드카(JSON dict)를 `QaResult` 로 복원. Phase 3 의 verifier 가 쓴
|
|
224
|
+
`result-stage-<N>.json` 을 validate-run 이 로드할 때 쓴다. 형태가 깨졌으면
|
|
225
|
+
overall='MISSING'(=BLOCKING 취급)으로 안전하게 강등한다."""
|
|
226
|
+
if not isinstance(data, dict):
|
|
227
|
+
return QaResult(overall="MISSING", requirements={})
|
|
228
|
+
overall = data.get("overall")
|
|
229
|
+
if overall not in ("PASS", "FAIL", "MISSING"):
|
|
230
|
+
overall = "MISSING"
|
|
231
|
+
reqs = data.get("requirements")
|
|
232
|
+
return QaResult(overall=overall, requirements=reqs if isinstance(reqs, dict) else {})
|
|
233
|
+
|
|
234
|
+
|
|
235
|
+
def evaluate_conformance(manifest: object, results_by_stage: object) -> list[ConformanceVerdict]:
|
|
236
|
+
"""매니페스트 전 entry 에 대해 게이트 판정 목록을 반환.
|
|
237
|
+
|
|
238
|
+
`results_by_stage`: stageKey -> `QaResult`. 키가 없으면 미실행(None)으로 본다.
|
|
239
|
+
매니페스트 구조 검증은 호출 전에 `validate_conformance_manifest` 로 끝낸다는 전제.
|
|
240
|
+
"""
|
|
241
|
+
entries = manifest.get("entries") if isinstance(manifest, dict) else None
|
|
242
|
+
if not isinstance(entries, list):
|
|
243
|
+
return []
|
|
244
|
+
results = results_by_stage if isinstance(results_by_stage, dict) else {}
|
|
245
|
+
verdicts: list[ConformanceVerdict] = []
|
|
246
|
+
for entry in entries:
|
|
247
|
+
if not isinstance(entry, dict):
|
|
248
|
+
continue
|
|
249
|
+
result = results.get(entry.get("stageKey"))
|
|
250
|
+
verdicts.append(decide_conformance_gate(entry, result))
|
|
251
|
+
return verdicts
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
- [ ] **Step 4: 통과 확인 + 전체 회귀**
|
|
255
|
+
|
|
256
|
+
Run: `python3 -m pytest tests/test_okstra_ctl_conformance.py -q` → Expected: PASS (28 passed)
|
|
257
|
+
Run: `python3 -m pytest tests/ -q` → Expected: 전부 통과(회귀 없음)
|
|
258
|
+
|
|
259
|
+
- [ ] **Step 5: 커밋**
|
|
260
|
+
|
|
261
|
+
```bash
|
|
262
|
+
git add scripts/okstra_ctl/conformance.py tests/test_okstra_ctl_conformance.py
|
|
263
|
+
git commit -m "feat(okstra_ctl/conformance): 사이드카 변환 + evaluate_conformance 일괄 평가"
|
|
264
|
+
```
|
|
265
|
+
|
|
266
|
+
---
|
|
267
|
+
|
|
268
|
+
## Phase 2 Self-Review 체크
|
|
269
|
+
|
|
270
|
+
- [ ] spec §6 의 미실행/MISSING/FAIL = BLOCKING, PASS = 통과가 모두 커버되는가?
|
|
271
|
+
- [ ] spec §7 의 면제(EXEMPT) / waiver(WAIVED, conditional) 가 커버되는가?
|
|
272
|
+
- [ ] Phase 3 가 의존할 공개 API(`decide_conformance_gate`, `evaluate_conformance`, `qa_result_from_dict`, `ConformanceVerdict`)가 노출되는가?
|
|
273
|
+
- [ ] 순수 함수만 추가했는가(파일 I/O·외부 의존 없음)? — wiring 은 Phase 3.
|
|
274
|
+
|
|
275
|
+
Phase 2 완료 후 Phase 3 계획(verifier 결과 기록 + validate-run 게이트 적용 + run.py 진입 검증 + planning 생성)을 작성한다.
|
|
@@ -0,0 +1,282 @@
|
|
|
1
|
+
# Stage Conformance QA — Phase 3 (validate-run 게이트) Implementation Plan
|
|
2
|
+
|
|
3
|
+
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
4
|
+
|
|
5
|
+
**Goal:** `validators/validate-run.py` 가 implementation/final-verification 리포트에 대해 run 의 `qa/` 디렉터리(conformance 매니페스트 + 결과 사이드카)를 읽어 Phase 2 의 `evaluate_conformance` 로 게이트를 적용한다. BLOCKING verdict 는 run 검증 실패로 만든다 — DEV-9184 의 "정적 OK, 미검증 → silent pass" 를 실제 FAIL 로 전환하는 강제 지점.
|
|
6
|
+
|
|
7
|
+
**Architecture:** Phase 1(매니페스트 검증) + Phase 2(게이트 판정)의 순수 함수를 validate-run 에 wiring 한다. **매니페스트가 없으면 게이트는 inert**(early return) — 따라서 매니페스트를 생성하는 prompt 측(Phase 4)이 없어도 안전하게 단독 출하된다. SSOT: [`docs/superpowers/specs/2026-06-07-stage-conformance-qa-design.md`](../specs/2026-06-07-stage-conformance-qa-design.md) §6.
|
|
8
|
+
|
|
9
|
+
**Tech Stack:** Python 3 (stdlib + pytest). validate-run.py 는 importlib 로 로드되는 standalone 스크립트.
|
|
10
|
+
|
|
11
|
+
**전제:** Phase 1·2 완료 — `scripts/okstra_ctl/conformance.py` 에 `validate_conformance_manifest`, `evaluate_conformance`, `qa_result_from_dict`, `ConformanceVerdict` 존재.
|
|
12
|
+
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
## 결과 사이드카 계약 (이 Phase 에서 확정)
|
|
16
|
+
|
|
17
|
+
- 매니페스트: `<run_dir>/qa/conformance-manifest.json` (run_dir = `report_path.parent.parent`, 기존 `validate_worker_results_audit` 의 `report_path.parent.parent / "worker-results"` 패턴과 동형).
|
|
18
|
+
- 결과 사이드카(Phase 4 의 verifier 가 작성): `<run_dir>/qa/result-<stageKey>.json` = `{"stageKey","overall","ranAt","requirements"}`. 파일 부재 → 미실행(None) → BLOCKING.
|
|
19
|
+
|
|
20
|
+
---
|
|
21
|
+
|
|
22
|
+
## Task 1: conformance 게이트 함수 + 결과 로더
|
|
23
|
+
|
|
24
|
+
**Files:**
|
|
25
|
+
- Modify: `validators/validate-run.py`
|
|
26
|
+
- Test: `tests/test_validate_run_conformance.py` (신규)
|
|
27
|
+
|
|
28
|
+
- [ ] **Step 1: 실패 테스트 작성**
|
|
29
|
+
|
|
30
|
+
```python
|
|
31
|
+
# tests/test_validate_run_conformance.py
|
|
32
|
+
from __future__ import annotations
|
|
33
|
+
|
|
34
|
+
import importlib.util
|
|
35
|
+
import json
|
|
36
|
+
from pathlib import Path
|
|
37
|
+
|
|
38
|
+
REPO_ROOT = Path(__file__).resolve().parent.parent
|
|
39
|
+
VALIDATOR_PATH = REPO_ROOT / "validators" / "validate-run.py"
|
|
40
|
+
|
|
41
|
+
|
|
42
|
+
def _load_validator():
|
|
43
|
+
spec = importlib.util.spec_from_file_location("validate_run", VALIDATOR_PATH)
|
|
44
|
+
module = importlib.util.module_from_spec(spec)
|
|
45
|
+
assert spec.loader is not None
|
|
46
|
+
spec.loader.exec_module(module)
|
|
47
|
+
return module
|
|
48
|
+
|
|
49
|
+
|
|
50
|
+
def _make_run(tmp_path: Path, entries, results):
|
|
51
|
+
run_dir = tmp_path / "runs" / "implementation"
|
|
52
|
+
(run_dir / "reports").mkdir(parents=True)
|
|
53
|
+
qa = run_dir / "qa"
|
|
54
|
+
qa.mkdir()
|
|
55
|
+
(qa / "conformance-manifest.json").write_text(json.dumps({"entries": entries}))
|
|
56
|
+
for key, overall in results.items():
|
|
57
|
+
(qa / f"result-{key}.json").write_text(
|
|
58
|
+
json.dumps({"stageKey": key, "overall": overall, "requirements": {}})
|
|
59
|
+
)
|
|
60
|
+
report = run_dir / "reports" / "final-report-implementation-001.md"
|
|
61
|
+
report.write_text("# fixture\n")
|
|
62
|
+
return report
|
|
63
|
+
|
|
64
|
+
|
|
65
|
+
def _entry(key, **over):
|
|
66
|
+
base = {"stageKey": key, "script": f"qa/{key}.ts", "runCommand": f"run {key}",
|
|
67
|
+
"requirementIds": ["R-001"], "requires": ["db"],
|
|
68
|
+
"passContract": "exit 0 = PASS", "exemption": None, "waiver": None}
|
|
69
|
+
base.update(over)
|
|
70
|
+
return base
|
|
71
|
+
|
|
72
|
+
|
|
73
|
+
def test_no_manifest_is_inert(tmp_path):
|
|
74
|
+
v = _load_validator()
|
|
75
|
+
run_dir = tmp_path / "runs" / "implementation"
|
|
76
|
+
(run_dir / "reports").mkdir(parents=True)
|
|
77
|
+
report = run_dir / "reports" / "final-report-implementation-001.md"
|
|
78
|
+
report.write_text("# fixture\n")
|
|
79
|
+
failures: list[str] = []
|
|
80
|
+
v._validate_conformance(report, failures)
|
|
81
|
+
assert failures == []
|
|
82
|
+
|
|
83
|
+
|
|
84
|
+
def test_passing_result_no_failure(tmp_path):
|
|
85
|
+
v = _load_validator()
|
|
86
|
+
report = _make_run(tmp_path, [_entry("t-stage-1")], {"t-stage-1": "PASS"})
|
|
87
|
+
failures: list[str] = []
|
|
88
|
+
v._validate_conformance(report, failures)
|
|
89
|
+
assert failures == []
|
|
90
|
+
|
|
91
|
+
|
|
92
|
+
def test_missing_result_is_blocking(tmp_path):
|
|
93
|
+
v = _load_validator()
|
|
94
|
+
report = _make_run(tmp_path, [_entry("t-stage-1")], {}) # no result file
|
|
95
|
+
failures: list[str] = []
|
|
96
|
+
v._validate_conformance(report, failures)
|
|
97
|
+
assert any("BLOCKING" in f and "t-stage-1" in f for f in failures)
|
|
98
|
+
|
|
99
|
+
|
|
100
|
+
def test_failing_result_is_blocking(tmp_path):
|
|
101
|
+
v = _load_validator()
|
|
102
|
+
report = _make_run(tmp_path, [_entry("t-stage-1")], {"t-stage-1": "FAIL"})
|
|
103
|
+
failures: list[str] = []
|
|
104
|
+
v._validate_conformance(report, failures)
|
|
105
|
+
assert any("BLOCKING" in f for f in failures)
|
|
106
|
+
|
|
107
|
+
|
|
108
|
+
def test_exemption_passes(tmp_path):
|
|
109
|
+
v = _load_validator()
|
|
110
|
+
entry = _entry("t-stage-1", exemption={"reason": "doc", "declaredAt": "2026-06-07"})
|
|
111
|
+
report = _make_run(tmp_path, [entry], {})
|
|
112
|
+
failures: list[str] = []
|
|
113
|
+
v._validate_conformance(report, failures)
|
|
114
|
+
assert failures == []
|
|
115
|
+
|
|
116
|
+
|
|
117
|
+
def test_waiver_passes_but_records_conditional(tmp_path):
|
|
118
|
+
v = _load_validator()
|
|
119
|
+
entry = _entry("t-stage-1", waiver={"acknowledgedBy": "user", "reason": "replica down",
|
|
120
|
+
"at": "2026-06-07", "scope": ["db"]})
|
|
121
|
+
report = _make_run(tmp_path, [entry], {})
|
|
122
|
+
failures: list[str] = []
|
|
123
|
+
v._validate_conformance(report, failures)
|
|
124
|
+
assert failures == [] # waiver = ok (conditional, not a failure)
|
|
125
|
+
|
|
126
|
+
|
|
127
|
+
def test_malformed_manifest_reported(tmp_path):
|
|
128
|
+
v = _load_validator()
|
|
129
|
+
report = _make_run(tmp_path, [{"stageKey": ""}], {}) # invalid entry
|
|
130
|
+
failures: list[str] = []
|
|
131
|
+
v._validate_conformance(report, failures)
|
|
132
|
+
assert any("conformance manifest" in f for f in failures)
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
- [ ] **Step 2: 실패 확인**
|
|
136
|
+
|
|
137
|
+
Run: `python3 -m pytest tests/test_validate_run_conformance.py -q`
|
|
138
|
+
Expected: FAIL — `module 'validate_run' has no attribute '_validate_conformance'`
|
|
139
|
+
|
|
140
|
+
- [ ] **Step 3: 구현 추가**
|
|
141
|
+
|
|
142
|
+
`validators/validate-run.py` 상단의 `from okstra_project.dirs import tasks_root as _okstra_tasks_root` (line 33 부근) 다음에 import 추가:
|
|
143
|
+
|
|
144
|
+
```python
|
|
145
|
+
from okstra_ctl.conformance import ( # noqa: E402
|
|
146
|
+
evaluate_conformance,
|
|
147
|
+
qa_result_from_dict,
|
|
148
|
+
validate_conformance_manifest,
|
|
149
|
+
)
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
`validate_report` 함수 정의 바로 앞(`def validate_report(` 위)에 두 함수 추가:
|
|
153
|
+
|
|
154
|
+
```python
|
|
155
|
+
def _load_conformance_results(qa_dir: Path, manifest: dict) -> dict:
|
|
156
|
+
"""매니페스트 각 entry 에 대응하는 `result-<stageKey>.json` 을 로드.
|
|
157
|
+
파일 부재/파손은 키를 비워 둬 미실행(None→BLOCKING)으로 흐르게 한다."""
|
|
158
|
+
results: dict = {}
|
|
159
|
+
for entry in manifest.get("entries", []):
|
|
160
|
+
key = entry.get("stageKey") if isinstance(entry, dict) else None
|
|
161
|
+
if not isinstance(key, str) or not key:
|
|
162
|
+
continue
|
|
163
|
+
sidecar = qa_dir / f"result-{key}.json"
|
|
164
|
+
if not sidecar.is_file():
|
|
165
|
+
continue
|
|
166
|
+
try:
|
|
167
|
+
results[key] = qa_result_from_dict(json.loads(sidecar.read_text()))
|
|
168
|
+
except (OSError, json.JSONDecodeError):
|
|
169
|
+
results[key] = qa_result_from_dict(None) # → MISSING → BLOCKING
|
|
170
|
+
return results
|
|
171
|
+
|
|
172
|
+
|
|
173
|
+
def _validate_conformance(report_path: Path, failures: list[str]) -> None:
|
|
174
|
+
"""Tier 3 conformance 게이트(implementation / final-verification).
|
|
175
|
+
|
|
176
|
+
`<run_dir>/qa/conformance-manifest.json` 이 없으면 inert(선언된 conformance
|
|
177
|
+
가 없다는 뜻 — 선언을 강제하는 것은 planning 계약(Phase 4)의 몫). 매니페스트가
|
|
178
|
+
있으면 결과 사이드카와 함께 evaluate_conformance 로 판정하고 BLOCKING verdict
|
|
179
|
+
를 run 검증 실패로 승격한다. WAIVED(conditional)/EXEMPT 는 통과시킨다.
|
|
180
|
+
"""
|
|
181
|
+
qa_dir = report_path.parent.parent / "qa"
|
|
182
|
+
manifest_path = qa_dir / "conformance-manifest.json"
|
|
183
|
+
if not manifest_path.is_file():
|
|
184
|
+
return
|
|
185
|
+
try:
|
|
186
|
+
manifest = json.loads(manifest_path.read_text())
|
|
187
|
+
except (OSError, json.JSONDecodeError) as exc:
|
|
188
|
+
failures.append(f"conformance manifest unreadable at {manifest_path}: {exc}")
|
|
189
|
+
return
|
|
190
|
+
schema_errors = validate_conformance_manifest(manifest)
|
|
191
|
+
if schema_errors:
|
|
192
|
+
failures.extend(f"conformance manifest: {e}" for e in schema_errors)
|
|
193
|
+
return
|
|
194
|
+
results = _load_conformance_results(qa_dir, manifest)
|
|
195
|
+
for verdict in evaluate_conformance(manifest, results):
|
|
196
|
+
if not verdict.ok:
|
|
197
|
+
failures.append(
|
|
198
|
+
f"conformance gate BLOCKING for stage {verdict.stage_key}: "
|
|
199
|
+
f"{verdict.message}. Run the stage's conformance script (or declare "
|
|
200
|
+
f"an exemption / user waiver) — see "
|
|
201
|
+
f"docs/superpowers/specs/2026-06-07-stage-conformance-qa-design.md."
|
|
202
|
+
)
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
- [ ] **Step 4: 통과 확인**
|
|
206
|
+
|
|
207
|
+
Run: `python3 -m pytest tests/test_validate_run_conformance.py -q`
|
|
208
|
+
Expected: PASS (7 passed)
|
|
209
|
+
|
|
210
|
+
- [ ] **Step 5: 커밋**
|
|
211
|
+
|
|
212
|
+
```bash
|
|
213
|
+
git add validators/validate-run.py tests/test_validate_run_conformance.py
|
|
214
|
+
git commit -m "feat(validators): conformance Tier3 게이트 함수(_validate_conformance)"
|
|
215
|
+
```
|
|
216
|
+
|
|
217
|
+
## Task 2: 메인 dispatch 에 게이트 wiring
|
|
218
|
+
|
|
219
|
+
**Files:**
|
|
220
|
+
- Modify: `validators/validate-run.py`
|
|
221
|
+
|
|
222
|
+
- [ ] **Step 1: dispatch 에 호출 추가**
|
|
223
|
+
|
|
224
|
+
`validate-run.py` 의 메인 dispatch 에서 `validate_worker_results_audit` 호출 블록 다음, `if task_type == "improvement-discovery":` 앞에 추가:
|
|
225
|
+
|
|
226
|
+
```python
|
|
227
|
+
if task_type in ("implementation", "final-verification"):
|
|
228
|
+
_validate_conformance(report_path, failures)
|
|
229
|
+
```
|
|
230
|
+
|
|
231
|
+
(찾을 앵커 — 현재 코드:)
|
|
232
|
+
|
|
233
|
+
```python
|
|
234
|
+
if task_type:
|
|
235
|
+
validate_worker_results_audit(report_path, task_type, failures)
|
|
236
|
+
if task_type == "improvement-discovery":
|
|
237
|
+
```
|
|
238
|
+
|
|
239
|
+
- [ ] **Step 2: 기존 스위트 회귀 + workflow validator 확인**
|
|
240
|
+
|
|
241
|
+
Run: `python3 -m pytest tests/ -q`
|
|
242
|
+
Expected: 전부 통과(신규 7 포함, 회귀 없음). validate-workflow 의 final-verification fixture 는 `qa/` 디렉터리가 없으므로 게이트는 inert → 영향 없음.
|
|
243
|
+
|
|
244
|
+
Run: `bash validators/validate-workflow.sh`
|
|
245
|
+
Expected: 마지막에 PASS (게이트 inert).
|
|
246
|
+
|
|
247
|
+
- [ ] **Step 3: 코드 확인(수동)**
|
|
248
|
+
|
|
249
|
+
`grep -n "_validate_conformance" validators/validate-run.py` → 정의 1 + 호출 1 = 2건.
|
|
250
|
+
|
|
251
|
+
- [ ] **Step 4: 커밋**
|
|
252
|
+
|
|
253
|
+
```bash
|
|
254
|
+
git add validators/validate-run.py
|
|
255
|
+
git commit -m "feat(validators): implementation/final-verification 에 conformance 게이트 연결"
|
|
256
|
+
```
|
|
257
|
+
|
|
258
|
+
---
|
|
259
|
+
|
|
260
|
+
## Phase 3 Self-Review 체크
|
|
261
|
+
|
|
262
|
+
- [ ] 매니페스트 부재 시 inert(early return) 확인 — 기존 run 들이 깨지지 않는가?
|
|
263
|
+
- [ ] BLOCKING(미실행/MISSING/FAIL) → failures 추가, EXEMPT/WAIVED → 통과가 spec §6/§7 과 일치하는가?
|
|
264
|
+
- [ ] run_dir 해석(`report_path.parent.parent`)이 기존 `validate_worker_results_audit` 와 동형인가?
|
|
265
|
+
- [ ] 전체 pytest + workflow validator 통과(회귀 없음)?
|
|
266
|
+
|
|
267
|
+
---
|
|
268
|
+
|
|
269
|
+
## Phase 4 로 넘길 항목 + ⚠️ 미해결 설계 갬
|
|
270
|
+
|
|
271
|
+
본 게이트는 매니페스트/결과가 존재할 때만 작동한다. 그것을 **생성·실행**하는 다음 작업들은 Phase 4 이며, 일부는 **계획 전 설계 확정이 필요**하다(추측 금지):
|
|
272
|
+
|
|
273
|
+
1. **verifier 실행(prompt)**: `_implementation-verifier.md` 에 Tier 3 추가 — 각 entry 의 `runCommand` 실행 → `parse_qa_result` → `<run_dir>/qa/result-<stageKey>.json` 작성. (env/secret 접근, replica DB, mutation 가드 포함)
|
|
274
|
+
2. **planning 생성(prompt)**: `implementation-planning.md` 가 stage 별 스크립트 + `conformance-manifest.json` + `Conformance exemption:` 라인 산출.
|
|
275
|
+
3. **final-verification(prompt)**: 전 stage 합집합 실행.
|
|
276
|
+
4. **우회 UX**: `--qa-waiver <stageKey>:"<reason>"` CLI(okstra.sh/cli.sh) + wizard picker → `run.py` 가 매니페스트 entry 의 `waiver` 채움.
|
|
277
|
+
5. **env**: `project.json.qaEnv`(replicaDbDsn/appBaseUrl/envFile).
|
|
278
|
+
|
|
279
|
+
**⚠️ 설계 확정 필요(Phase 4 계획 전 brainstorm 보강 권장):**
|
|
280
|
+
- **매니페스트의 교차-phase 위치**: planning 이 만든 매니페스트/스크립트가 implementation run, 이어서 final-verification run 으로 어떻게 전달되는가? (task-key worktree vs stage worktree vs instruction-set 복사) — stage worktree isolation 과 정합해야 함.
|
|
281
|
+
- **"diff 가 surface 를 건드렸는데 매니페스트 entry 자체가 없음"** (planning 누락) 강제: 본 게이트는 선언된 entry 만 강제한다. 누락 탐지는 planning 계약 + diff-surface 휴리스틱이 필요(spec §7.1 선언형 면제 surface 교차검증과 함께).
|
|
282
|
+
- **env/secret 운영 모델**: replica DB·실행 env 를 누가 어떻게 주입하는가.
|