@ai-dev-methodologies/rlp-desk 0.15.3 → 0.15.5

package/docs/plans/strategic-review/rlp-desk-strategic-review.md

@@ -1,125 +0,0 @@
-# rlp-desk 전략 재평가 — autoplan 입력
-
-> **목표**: rlp-desk를 계속 발전시킬지(patch/redesign), 폐기 후 재구성할지(rebuild), 기존 도구로 pivot할지 결정.
-> **핵심 KPI**: blueprint 처음 → 끝 자율 완료. 사람 개입 최소.
-> **현재 상태**: 6주간 10개 bug, 매주 1-2개 신규 발견, manual recovery마저 broken (#10).
-
----
-
-## 1. Vision (BOS dev 합의, 2026 초)
-
-1. ralph-loop를 **fresh-context**로 매번 시작 (컨텍스트 오염 없음)
-2. idea → plan distillation
-3. PRD 정형화
-4. Worker/Verifier 사이클로 반복 개선
-5. **완전 자율화** — 사람 개입 최소
-
-→ 5번이 단일 success criterion. 1-4는 5번을 달성하기 위한 메커니즘.
-
----
-
-## 2. Reality — 6주간 측정된 사실
-
-### Bug 발생 패턴
-
-| # | 일자 | 카테고리 | 회수 가능 | 다음 fix가 노출한 다음 bug |
-|---|---|---|---|---|
-| #1 | 2026-05-01 | LLM-runtime (`.claude/` self-modification gate) | partial — codex worker로 우회 | #2 (tmux session) |
-| #2 | 2026-05-01 | tmux session lifecycle | yes | #3 |
-| #3 | 2026-05-04 | verifier no-progress | yes | #4 (regression of #3) |
-| #4 | 2026-05-05 | #3 regression | partial | #5 |
-| #5 | 2026-05-05 | worker dead on reuse (pane lifecycle) | yes | #6 |
-| #6 | 2026-05-06 | claude worker idle false-positive | yes | #7 |
-| #7 | 2026-05-06 | post-sentinel process race (1m43s drift) | yes | #8 + #10 |
-| #8 | 2026-05-06 | worker incomplete + leader A4 fallback | partial | #10 |
-| #9 | (별도) | verified_us 영속성 (status.json) | yes | — |
-| #10 | 2026-05-07 | leader ignores phase=verify on relaunch — **회수마저 깨짐** | NO (이게 회수 메커니즘) | ? |
-
-**Pattern observed:**
-- 매주 1-2개 신규 bug
-- 절반은 **이전 fix가 노출**한 새 failure mode (regression style)
-- Bug #10이 가장 심각 — 회수 메커니즘 자체가 broken → BLOCKED 시 operator의 manual recovery가 무효화
-
-### Bug 카테고리 분류
-
-| 카테고리 | Bug | 비중 | architectural 필연성? |
-|---|---|---|---|
-| (a) tmux/process lifecycle race | #2, #5, #6, #7 | 40% | **YES** — tmux pane lifecycle은 claude/codex TUI lifecycle과 분리됨. race window가 본질적 |
-| (b) artifact contract / schema | #3, #4, #8, #9 | 40% | partial — schema가 더 strict하면 줄어들지만 LLM이 schema 어기는 빈도가 본질적 |
-| (c) LLM-runtime constraint | #1 | 10% | **YES** — Claude Code의 `.claude/` self-modification gate는 외부 변수 |
-| (d) recovery hygiene | #10 | 10% | accidental — fix 가능 |
-
-→ **80%가 architectural inevitability** (a + c). schema strict + retry는 (b)를 줄이지만 LLM 비결정성이 한계.
-
-### SV gate 한계
-
-- 모든 sv-self-verify-*.sh가 **Worker/Verifier metaphor만 사용** → 실제 LLM agent run 없음
-- grep + unit test + regression의 5-category labeling
-- → 10개 bug가 SV gate 통과 후 production에서 발견됐다는 사실 = **framework이 production failure mode를 cover 못 함**
-- Production failure mode는 **LLM/tmux/network/timing 비결정성** — unit test로 잡을 수 없는 영역
-
-### In-flight 미완 branch
-
-- `feat/native-agent-revert`: P0(Bug #7 fix) + P1(slash native prose 회복) — 미완. Plan: 5 round ralplan 합의됨, codex critic APPROVED
-- `feat/bug10-relaunch-hygiene`: PR-A commit 95c0d4e + SV gate 추가 — 방금 완료, push 안 됨
-
-→ 두 branch 모두 land 안 됨 = 새 bug 발견 시 기존 fix가 무력화될 위험
-
----
-
-## 3. 4가지 옵션
-
-### Option A — Continue patching (현재 path)
-PR-A (Bug #10) → PR-B (bundler) → PR-C (patterns) → 다음 bug → 반복.
-
-### Option B — Fundamental redesign (vision 유지, architecture 재설계)
-**핵심 변경**: tmux pane orchestration 폐기 → Claude Code Native Agent() / subprocess 직접 dispatch. Sentinel file → in-memory channel. 두 변경으로 (a) 카테고리 80%가 제거.
-
-### Option C — Scrap and rebuild (vision 일부 수정)
-6주 코드 폐기. Vision 5번 (완전 자율화)은 유지하되 1번(fresh-context)은 task-isolated subprocess로 재정의. ralph-loop 자체 폐기 가능 — 그냥 plan→worker→verify 사이클로 단순화.
-
-### Option D — Pivot to existing tool
-- ralph plugin (cradle): 가벼움, 기능 부족
-- omc (oh-my-claudecode): /ralph + /ralplan + /omc-teams + /autopilot — 이미 multi-model orchestration 있음
-- superpowers: subagent-driven-development, executing-plans, brainstorming — plan→exec 사이클은 이미 있음
-- claude-devfleet: dmux 기반, multi-agent
-
----
-
-## 4. 평가 항목 (각 옵션마다)
-
-| 항목 | 정의 | 이유 |
-|---|---|---|
-| Vision 보존도 | 5개 vision 중 살아남는 개수 | "완전 자율화" 달성 가능성 |
-| Time-to-first-successful-blueprint | 처음으로 blueprint를 끝까지 자율 완료하는 데 걸리는 시간 | **단일 핵심 KPI** |
-| Sunk cost write-off % | 폐기되는 코드/SV 비율 | 결정의 reversibility |
-| Bug regression 위험 | 새 시스템에서 #1-#10 같은 bug가 다시 나올 확률 | "다음 6주에 또 10개?" |
-| Personal capacity ROI | 1주 투자 시 deliverable | sustainability |
-
----
-
-## 5. In-flight branch 처리 결정
-
-이 평가가 끝난 후 결정:
-- `feat/native-agent-revert`: land / abandon / re-scope?
-- `feat/bug10-relaunch-hygiene`: 이미 commit 95c0d4e, push할까 hold할까?
-
-선택 옵션이 무엇이든 두 branch는 처리 필요.
-
----
-
-## 6. 제약
-
-- BOS dev가 실제 캠페인을 돌려야 함 → short-term (1-2주) patching 불가피
-- rlp-desk는 npm published — breaking change는 user-facing (semver 고려)
-- 분석은 advisory only — 실제 코드 수정/commit/push는 사용자 승인 별도
-
----
-
-## 7. autoplan에 요구 — 이 문서를 입력으로
-
-CEO 관점: rlp-desk가 푸는 문제가 옳은가? 다른 도구로 같은 가치를 이미 얻을 수 있나?
-Eng 관점: 6주간의 architectural pattern을 보면 (a) 80% 카테고리가 patching으로 해결될까? 아니면 redesign이 필연인가?
-DX 관점: rlp-desk는 dev tool — operator(BOS dev)가 매번 30분씩 hand-write recovery를 해야 한다는 사실이 DX failure인가?
-
-각 phase에서 dual voices (Codex + Claude subagent) 실행, consensus table 생성, 4 options steelman.

package/docs/plans/v0.15-stabilization-phase-a-prep.md

@@ -1,130 +0,0 @@
-# Phase A — Empirical omc Baseline (Prep for Next Session)
-
-> **Plan reference**: `docs/plans/v0.15-stabilization-plan.md` §5 Phase A
-> **Goal**: measure omc /ralph + /team reliability empirically. Establishes the bar rlp-desk needs to reach.
-> **Output**: `docs/plans/v0.15-stabilization-omc-baseline.md` with per-test metrics.
-> **NOT a competition**: this is a measurement to set the stabilization target. omc is benchmark, not replacement.
-
----
-
-## Why baseline measurement comes first
-
-Before changing rlp-desk's substrate (Phase B-F), we need to know what "omc-level reliability" actually looks like for our workload. Otherwise stabilization targets are guesswork.
-
-The 4 differentiators rlp-desk MUST preserve are not measured by omc — they're rlp-desk-only by design. So the baseline measurement focuses on what omc DOES have: per-iter Worker→Verifier, PRD-driven loop, multi-agent coordination, mandatory deslop, regression re-verification.
-
-## Three test campaigns
-
-### A1 — single-iter, single-story
-- **Workload**: small TypeScript fix in a sandbox repo (or BOS apps/web/ on a contained file)
-- **Tool**: `/oh-my-claudecode:ralph "Fix the unused variable warning in <specific-file>"`
-- **Measure**:
-  - operator-touch count (target = 0)
-  - total time (entry → completion)
-  - cost (token usage if available, otherwise rough estimate)
-  - did mandatory deslop pass run?
-  - did regression re-verification pass?
-
-### A2 — multi-iter, multi-story
-- **Workload**: synthetic PRD with 3 stories, each story trivial but distinct
-- **Tool**: `/oh-my-claudecode:ralph` with auto-generated prd.json refined to 3 stories
-- **Measure**:
-  - operator-touch count per story
-  - story completion order (sequential as PRD'd, or chosen by ralph?)
-  - failure recovery if a story blocks (does it advance to next or stop?)
-  - prd.json final state (all stories `passes: true`?)
-  - reviewer behavior (was each story actually verified against acceptance criteria?)
-
-### A3 — parallel team
-- **Workload**: 3-task synthetic spec where tasks are truly independent
-- **Tool**: `/oh-my-claudecode:team 3:executor "<spec>"`
-- **Measure**:
-  - parallelism (do 3 agents really run concurrently?)
-  - lock contention (any deadlocks on shared task list?)
-  - inter-agent messaging (any SendMessage between teammates?)
-  - completion time vs sequential estimate
-  - cleanup (does TeamDelete actually run on completion?)
-
-## Sandbox setup
-
-Recommend a throwaway test repo to avoid contaminating BOS or rlp-desk:
-
-```bash
-mkdir /tmp/omc-baseline-sandbox && cd /tmp/omc-baseline-sandbox
-git init
-# Add 1-2 simple TypeScript files with intentional issues for A1
-# Add a synthetic PRD for A2
-# Add a 3-task spec for A3
-```
-
-This isolates measurement from real product work and lets us reset between tests.
-
-## Output schema
-
-`docs/plans/v0.15-stabilization-omc-baseline.md`:
-
-```markdown
-# omc Baseline Measurement — Phase A Output
-
-## A1 (single-iter, single-story)
-- Workload: <description>
-- Result: PASS / PARTIAL / FAIL
-- Operator-touch: N
-- Time: Xm Ys
-- Cost: $X.XX
-- Deslop ran: YES/NO
-- Regression re-verify: PASS/FAIL/N-A
-- Subjective notes: ...
-
-## A2 (multi-iter, multi-story)
-- Workload: <description>
-- Result: PASS / PARTIAL / FAIL
-- Stories completed: N/3
-- Operator-touch: N per story
-- Failure recovery observed: YES/NO + behavior
-- Total time: Xm Ys
-- prd.json final state: ...
-- Reviewer per-story verification: YES/NO
-
-## A3 (parallel team)
-- Workload: <description>
-- Result: PASS / PARTIAL / FAIL
-- Parallelism observed: YES/NO + concurrent count
-- Lock contention: NONE/ONE/MULTIPLE
-- Total time vs sequential: Xm vs Ym
-- Cleanup: COMPLETE/PARTIAL/MANUAL
-
-## Synthesis — what is "omc-level reliability"?
-
-[1-2 paragraphs translating the metrics into a target for rlp-desk]
-
-## Phase B implications
-
-[which omc patterns should rlp-desk Phase B adopt for tmux/process lifecycle race?]
-```
-
-## Cost estimate
-
-3 small test campaigns: total ~$5-15. Sandbox runs are bounded (no real product blast radius).
-
-## How to run (next session)
-
-Open Claude Code in /tmp/omc-baseline-sandbox (after creating it per above):
-
-```
-/oh-my-claudecode:ralph "Fix the unused variable warning in src/index.ts"
-```
-
-Wait for completion. Record metrics. Reset sandbox. Repeat for A2 and A3 with the synthetic PRD/spec.
-
-After all three: write `docs/plans/v0.15-stabilization-omc-baseline.md` in rlp-desk repo with the synthesis.
-
-## What this enables
-
-Once Phase A is done, Phase B (tmux/process lifecycle race hardening) has a concrete target: "match omc /ralph's A1 behavior on operator-touch within ±20%, while preserving multi-engine consensus + multi-mission queue."
-
-Without Phase A, Phase B targets are imaginary.
-
-## Honest scope note
-
-This is preparation work. The Phase A run itself happens in a fresh session (sandbox cwd). This prep doc + the v0.15-stabilization-plan.md are the carry-over artifacts.

package/docs/plans/v0.15-stabilization-plan.md

@@ -1,178 +0,0 @@
-# rlp-desk Stabilization Plan (v0.15.x → v0.16.x)
-
-> **Status**: ACTIVE. Replaces the misdirected 2026-05-07 "pivot to omc" decision (PR #8 redirect via this plan).
-> **Goal**: bring rlp-desk to omc /team/ralph/ralplan level of reliability **while preserving rlp-desk's self-driving advantages**.
-> **Non-goal**: pivoting away from rlp-desk. omc is the **benchmark**, not the replacement.
-
----
-
-## 0. Why this plan exists (correction note)
-
-On 2026-05-07 morning I (the assistant) ran `plan-ceo-review` on the question "rlp-desk vs omc /team" and produced a recommendation to enter maintenance mode and pivot to omc. The user immediately corrected: *the goal was always to make rlp-desk work as reliably as omc, NOT to replace it*.
-
-This plan is the corrected direction: stabilize rlp-desk by learning from omc's patterns, applying them to rlp-desk's substrate, while protecting the four real differentiators that make rlp-desk worth using in the first place.
-
-The misdirected commit `229e1b6` (the "maintenance mode" banner + FROZEN doc) is now reverted in this PR. The pivot prompt-optimizer artifact and BOS validation plan stay on disk but are deferred — they may become useful later as a comparison study, but they are not the active path.
-
----
-
-## 1. The vision (preserved verbatim)
-
-1. ralph-loop fresh-context per iteration (no context pollution)
-2. idea → plan distillation
-3. PRD formalization
-4. Worker/Verifier cycles with iterative improvement
-5. **Full autonomy — minimum operator intervention**
-
-This vision is the core. Stabilization is in service of it, not a substitute for it.
-
----
-
-## 2. Differentiators to preserve (rlp-desk-only)
-
-These four are the reason rlp-desk exists separately from omc. Stabilization work MUST NOT compromise them:
-
-1. **Multi-engine parallel consensus per iteration**: `--consensus all` runs claude AND codex on every verification, then reconciles. omc /ralph supports `--critic=codex` but as a single critic, not parallel consensus.
-2. **Multi-mission queue + cross-mission analytics**: `RLP_BACKGROUND=1` chains missions and tracks cross-mission metrics. omc /team is single-task.
-3. **BLOCK_TAGS P1-D failure taxonomy**: structured `reason_category × recoverable × suggested_action` classification. omc emits simpler verdicts (pass/fail/blocked).
-4. **Structured SV reports**: post-campaign analytics at `~/.claude/ralph-desk/analytics/<slug>/self-verification-report-NNN.md`. omc has lighter `progress.txt`.
-
-These four ARE the value proposition. The stabilization work below is about making the substrate that delivers them as reliable as omc's.
-
----
-
-## 3. The 10-bug regression pattern (what we're hardening against)
-
-Six weeks (2026-05-01 to 2026-05-07), 10 bugs, each prior fix exposing the next. Categorized:
-
-| Cat | Bugs | Root cause cluster |
-|---|---|---|
-| (a) tmux/process lifecycle race | #5, #6, #7, #10 | Long-lived TUI processes in tmux panes; sentinel polling races; recovery hygiene |
-| (b) artifact contract / schema | #3, #4, #8, #9 | Worker/Verifier output contract violations; LLM non-determinism on schema; verified_us persistence |
-| (c) LLM-runtime constraint | #1 | Claude Code `.claude/` self-modification gate blocking sentinel writes |
-| (d) recovery hygiene | #10 | Manual recovery on relaunch silently overwritten |
-
-**Per category, what omc does differently** (preliminary — to be verified empirically in §5):
-
-- **(a) Lifecycle race**: omc /team uses Claude Code native team primitives (`TeamCreate`, `TaskCreate`, `SendMessage`). No tmux, no long-lived TUI, no sentinel polling. Process lifecycle = subagent lifecycle = single Claude Code call. Race window does not exist.
-- **(b) Contract violations**: omc /ralph uses `prd.json` with `passes: bool` per story + reviewer verifies acceptance criteria. Simpler schema = less surface for LLM to violate. omc also has mandatory deslop pass + regression re-verification (`ai-slop-cleaner` + Step 7.6).
-- **(c) Self-modification gate**: omc skills are read by Claude Code via the Skill tool, not written by Workers. Workers don't touch `.claude/` paths. Gate not encountered.
-- **(d) Recovery**: omc /ralph is session-scoped (`.omc/state/sessions/{sessionId}/prd.json`). Per-session state means relaunch starts fresh; there is no "manual recovery" surface to break.
-
-These are the patterns to learn from. Adopting them does NOT require pivoting away from rlp-desk; it requires bringing equivalent semantics into rlp-desk's substrate.
-
----
-
-## 4. Stabilization principles
-
-1. **omc is benchmark, not replacement.** Every change in this plan asks "how does omc avoid this failure mode?" then engineers an equivalent for rlp-desk's stack.
-2. **Preserve all 4 differentiators.** No change should compromise multi-engine consensus, multi-mission queue, BLOCK_TAGS taxonomy, or SV reports.
-3. **Substrate first, features second.** Bug categories (a) and (d) are substrate. Categories (b) and (c) are surface. Fix substrate first; surface improvements compound on a stable base.
-4. **Real-LLM SV gate.** The current SV gate's grep+unit-test labeling missed 10 production bugs. SV must be strengthened to actually catch production failure modes (subset of campaigns run with full claude/codex worker+verifier in CI-like mode).
-5. **Increment by category.** Each PR closes ONE bug category, not multiple. Avoids "fix-of-fix-of-fix" pattern that produced #4 (regression of #3).
-
----
-
-## 5. Concrete workstream (revised, per category)
-
-### Phase A — Empirical omc baseline (W1, ~3 days)
-
-Before changing rlp-desk, measure omc reliably. Three test campaigns:
-
-| Test | Workload | Measure |
-|---|---|---|
-| A1 | omc /ralph "fix small TS error in BOS apps/web/" | operator-touch count, time, cost |
-| A2 | omc /ralph + multi-iter (3+ stories) on a synthetic PRD | operator-touch count, recovery behavior |
-| A3 | omc /team "implement small feature with 3:executor" on synthetic task | parallelism behavior, lock contention |
-
-**Output**: `docs/plans/v0.15-stabilization-omc-baseline.md` with per-test metrics. Not a competition, a *measurement*. Establishes the bar rlp-desk needs to reach.
-
-### Phase B — Category (a) substrate hardening (W1-W3, ~2 weeks)
-
-The largest cluster (4 of 10 bugs). Goal: tmux/process lifecycle race window → 0 in `--mode tmux`. `--mode native` already addresses this differently; the work here is `--mode tmux`.
-
-Sub-deliverables:
-- B1: lifecycle audit (every tmux send-keys / sentinel write / pane reuse — ASCII diagram of timing windows)
-- B2: post-sentinel reaper invariant test (extend Bug #7 fix coverage to all sentinel writes, not just per-US)
-- B3: real-LLM SV scenario for category (a) — actual claude/codex worker dispatched, lifecycle race triggered deterministically, fix verified
-- B4: lifecycle observability (debug log emits race-window measurements per iteration)
-
-### Phase C — Category (d) recovery hygiene completion (W3-W4, ~1 week)
-
-Bug #10's PR-A fix covers `phase=verify` honor. Remaining recovery surfaces:
-- C1: phase=blocked recovery (operator clears blocked sentinel + restarts) — currently honored, verify with test
-- C2: phase=worker mid-iter crash recovery (leader killed mid-worker dispatch) — verify, fix if broken
-- C3: cross-mission queue recovery (one mission BLOCKED, queue advances) — verify
-- C4: documented operator recovery cookbook with deterministic jq pipelines
-
-### Phase D — Category (b) contract hardening (W4-W6, ~2 weeks)
-
-LLM contract violations are partly inevitable, but the harness can reduce the surface:
-- D1: schema validator at every artifact write (already exists for some; extend to all done-claim/iter-signal/verdict variants)
-- D2: feedback loop — when worker violates contract, next iteration's prompt includes the schema error verbatim (omc-style)
-- D3: verified_us persistence audit (Bug #9) — `status.json` is the source of truth, memory.md is supplementary, contract clear in code
-- D4: real-LLM SV scenario for category (b)
-
-### Phase E — Category (c) LLM-runtime constraint awareness (W6-W7, ~1 week)
-
-`.claude/` self-modification gate (Bug #1):
-- E1: Worker prompt explicitly states "do NOT touch `.claude/`; sentinel paths are at `.rlp-desk/memos/`" (already done in v0.13.0 path migration; verify)
-- E2: claude worker pre-flight check — try a no-op write to `.rlp-desk/` before main work; fail fast if blocked
-- E3: cross-engine fallback — when claude worker hits permission gate, mid-flight fallback to codex worker for that iter (already partial; complete)
-
-### Phase F — Real-LLM SV gate (W7-W8, ~2 weeks)
-
-The biggest framework upgrade:
-- F1: define "SV scenario" = complete real campaign (1-3 iter, real claude/codex, real tmux or native) executed in CI nightly
-- F2: each merged PR adds at least one SV scenario covering the bug it fixed (Bug #1-#10 retroactively)
-- F3: SV gate becomes "all real-LLM scenarios PASS" before npm publish — replaces the current grep-and-label SV
-- F4: cost budget for SV gate (~$10-20/run nightly, ~$300-600/month — explicit budget approval needed before W7 starts)
-
-### Release cadence
-
-- v0.15.2 (this PR): redirect + stabilization plan + Phase A start
-- v0.15.3-v0.15.7: incremental Phase B-E PRs, each landing one category fix + real-LLM SV scenario for that category
-- v0.16.0 (~8-10 weeks from 2026-05-07): real-LLM SV gate active + 10-bug regression pattern verified eliminated empirically (3 consecutive campaigns at omc baseline parity or better)
-
----
-
-## 6. Success criteria (measurable)
-
-| Metric | Current (2026-05-07) | v0.16.0 target | Measurement |
-|---|---|---|---|
-| Bug discovery rate | 1-2/week | <1/month | git log of bug-report-* files in BOS |
-| Operator-touch per campaign | unmeasured (high) | <1 per 5 campaigns | new analytics field in `campaign.jsonl` |
-| Campaign completion rate | unmeasured (low) | >80% | new analytics field |
-| SV gate catches production bugs | 0/10 | >50% (5/10 if Bug #11 happens, caught pre-publish) | post-publish bug review |
-| Differentiator preservation | 4/4 | 4/4 | regression test per differentiator |
-
----
-
-## 7. What this plan is NOT
-
-- NOT a pivot away from rlp-desk
-- NOT a maintenance mode declaration
-- NOT a plan to delete the Node leader (`--mode tmux` and `--mode agent` Node CLI both stay; deletion is a separate decision deferred until stabilization complete)
-- NOT a promise that omc patterns will be copied verbatim — they're inspiration, the implementation is rlp-desk-native
-
-## 8. What this plan IS
-
-- A correction of the 2026-05-07 misdirection
-- A category-by-category hardening roadmap with empirical baselines (Phase A)
-- A real-LLM SV gate replacement for the current theatrical SV (Phase F)
-- A preservation contract for the 4 differentiators
-- A concrete release cadence ending in v0.16.0 with measured success criteria
-
----
-
-## 9. First action (this PR)
-
-This PR (`feat/v0.15.2-stabilization-redirect`):
-- Reverts the maintenance-mode banner in `src/node/run.mjs`
-- Replaces with stabilization-in-progress banner
-- Removes `docs/plans/v0.16-FROZEN-status.md` (misdirection artifact)
-- Adds this `docs/plans/v0.15-stabilization-plan.md`
-- Updates `tests/node/us008-cli-entrypoint.test.mjs` regex
-- Bumps to v0.15.2 + npm publish so users see the corrected banner
-
-After this lands: Phase A (omc baseline measurement) starts. That's a separate session.

package/docs/plans/v0.16-real-llm-sv-gate-spec.md

@@ -1,177 +0,0 @@
-# Real-LLM SV Gate Framework — Spec (Phase F bootstrap)
-
-> **Status**: design + first scenario bootstrap. Active stabilization Phase F.
-> **Plan reference**: `docs/plans/v0.15-stabilization-plan.md` §5 Phase F.
-> **Goal**: replace today's grep+unit-test "SV gate theater" with a framework that ACTUALLY catches production failure modes by running real claude/codex agents against real campaigns.
-
----
-
-## 1. The problem in one paragraph
-
-Today's SV gate (`tests/sv-self-verify-*.sh`, `tests/sv-gate-*.sh`) is grep-and-label theater. Each "scenario" is a shell command (grep, unit test, regression) labeled with five categories (`correctness | integration | security | perf | error-path`) and reported as "Worker → Verifier → PASS". **No real LLM agent runs**. None of the 10 production bugs (#1-#10, 2026-05-01..05-07) were caught by SV gate before shipping. Every Bug #N+1 will follow the same pattern unless the framework changes.
-
-## 2. Why current SV gate misses production failures
-
-Production failure modes that today's gate cannot reproduce:
-
-| Failure mode | Why grep+unit can't catch | Real-LLM gate can |
-|---|---|---|
-| Worker hang on `.claude/` self-modification gate (Bug #1) | LLM platform-level constraint, not testable in unit | YES (real claude worker writes sentinel, observes hang) |
-| tmux pane lifecycle race (Bug #5/#6/#7) | timing-dependent, requires real TUI lifecycle | YES (real worker + real pane + real reaper) |
-| Worker contract violation under non-determinism (Bug #3/#4/#8/#9) | LLM produces malformed/incomplete artifact non-deterministically | YES (run actual worker with real prompt; observe artifact shape over N runs) |
-| Recovery hygiene under partial state (Bug #10) | requires real campaign + real BLOCKED + real operator recovery | YES (real campaign brought to BLOCKED, real operator-style recovery, real relaunch) |
-
-The common thread: **production failures require production-like state**. Unit tests can't produce it; only real campaigns can.
-
-## 3. Framework requirements
-
-A real-LLM SV gate must:
-
-R1. **Run a complete real campaign** (1-3 iterations) per scenario, not isolated function calls.
-R2. **Use real claude or codex worker + verifier**, not stub/mock LLM responses.
-R3. **Run inside CI-like environment** (real tmux session + real Claude Code CLI / codex CLI).
-R4. **Be deterministic enough to gate a release**: same scenario produces same PASS/FAIL outcome >95% of the time.
-R5. **Stay within cost budget**: each scenario <$5; total nightly run <$50.
-R6. **Run on a schedule, not per-PR**: too expensive to gate every PR. Nightly is the target. Per-PR can run a subset (sub-$10).
-R7. **Capture failure provenance**: when scenario fails, the failure must be reproducible from the captured campaign artifacts (logs, status, sentinel state).
-R8. **Cover all 10 historical bug categories**: each merged PR adds at least one scenario covering the bug class it touched.
-
-## 4. Architecture
-
-```
-┌─────────────────────────────────────────────────────────────────┐
-│ tests/sv-real-llm/                                              │
-│                                                                  │
-│   scenarios/                                                     │
-│     bug-01-claude-self-modification-gate.test.sh                 │
-│     bug-05-worker-dead-on-reuse.test.sh                          │
-│     bug-07-post-sentinel-race.test.sh                            │
-│     bug-10-relaunch-phase-verify-hygiene.test.sh                 │
-│     ...                                                          │
-│                                                                  │
-│   harness/                                                       │
-│     run-scenario.sh         (single scenario runner)             │
-│     run-all.sh              (nightly suite runner)               │
-│     budget-guard.sh         (cost budget enforcement)            │
-│     capture-artifacts.sh    (post-run forensic bundle)           │
-│                                                                  │
-│   fixtures/                                                      │
-│     minimal-prd-1us.md       (1 user story, deterministic)       │
-│     minimal-prd-3us.md       (3 user stories)                    │
-│                                                                  │
-│   results/                                                       │
-│     <date>-<scenario>.json   (per-run outcome + cost + time)     │
-│     <date>-<scenario>.bundle/ (full campaign artifacts)          │
-└─────────────────────────────────────────────────────────────────┘
-```
-
-Each scenario is a self-contained shell test:
-- Sets up a sandbox campaign
-- Runs `node ~/.claude/ralph-desk/node/run.mjs run <slug>` or via slash command
-- Asserts on final state (sentinel + status + artifacts)
-- Reports PASS/FAIL with provenance
-
-## 5. Scenario authoring contract
-
-Every scenario must define:
-
-```bash
-# scenario header (machine-readable)
-SCENARIO_ID="bug-10-relaunch-phase-verify-hygiene"
-SCENARIO_DESCRIPTION="Relaunch with operator-written phase=verify artifacts honored"
-SCENARIO_BUG_CATEGORY="d-recovery-hygiene"
-SCENARIO_HISTORICAL_BUG="Bug #10 (BOS 2026-05-07)"
-SCENARIO_COST_BUDGET_USD="3"
-SCENARIO_TIMEOUT_SECONDS="600"
-SCENARIO_REQUIRES="claude_cli OR codex_cli; tmux; jq"
-
-# scenario body: 4 parts
-# 1. SETUP — sandbox dir, init campaign, prepare fixture state
-# 2. EXERCISE — run actual rlp-desk against the fixture
-# 3. ASSERT — check final state matches expected
-# 4. REPORT — emit JSON outcome + capture artifacts
-```
-
-The 4-part shape is non-negotiable. Scenarios that skip ASSERT or REPORT do not count toward gate coverage.
-
-## 6. Cost + time discipline
-
-- **Budget per scenario**: $5 default, $10 max. Scenarios exceeding budget must be marked `EXPENSIVE` and only run weekly, not nightly.
-- **Timeout per scenario**: 600s default, 1800s max. Scenarios exceeding timeout are FAIL automatically.
-- **Total nightly budget**: $50. Suite must fit within budget or be sharded across nights.
-- **Per-PR subset**: scenarios marked `LIGHT` (under $1, under 60s) run on every PR. Others run nightly.
-
-Cost tracking via `tests/sv-real-llm/harness/budget-guard.sh` — reads `cost-log.jsonl` from rlp-desk and aggregates pre-run/post-run deltas.
-
-## 7. First scenario (this PR's bootstrap deliverable)
-
-`tests/sv-real-llm/scenarios/bug-10-relaunch-phase-verify-hygiene.test.sh` — converts the existing PR-A test (Bug #10 fix) into a real-LLM scenario:
-
-1. SETUP: create sandbox campaign in `/tmp/sv-real-llm-bug-10/`. init with minimal PRD (1 US, "fix the typo in foo.txt" — deterministic). Run iter-1 normally to BLOCKED state (deliberately fail verifier). Operator-style recovery: write iter-signal.json + done-claim.json by hand, set status.phase=verify, remove blocked sentinel.
-2. EXERCISE: relaunch leader (`/oh-my-claudecode:autopilot` or `node run.mjs run`).
-3. ASSERT:
-   - Leader logs `[recovery] Resuming verify phase` (PR-A audit line)
-   - No new `iter-001.worker-prompt.md` written (worker dispatch skipped)
-   - Verifier dispatched and produces verdict
-   - Final state: complete sentinel OR continue to iter-2
-4. REPORT: write `<date>-bug-10.json` with `{outcome, time_seconds, cost_usd, log_path, captured_state_path}`.
-
-Cost estimate: $1-3 (single iter-2 with fast worker model + verifier).
-
-## 8. Scenario coverage roadmap
-
-| Scenario | Bug class | Status | Cost | Test type |
-|---|---|---|---|---|
-| bug-10-relaunch-phase-verify-hygiene | (d) recovery | ✅ landed PR #12 | $3 | real-LLM |
-| bug-10-blocked-recovery-counters | (d) recovery | ✅ landed (this PR set) | $2 | real-LLM |
-| bug-07-post-sentinel-race | (a) lifecycle | ✅ landed | $3 | real-LLM |
-| bug-08-worker-incomplete-leader-fallback | (b) contract | ✅ landed | $3 | real-LLM |
-| bug-01-claude-self-modification-gate | (c) LLM-runtime | ✅ landed | $3 | real-LLM |
-| bug-05-worker-dead-on-reuse | (a) lifecycle | ✅ landed | $2 | real-LLM |
-| bug-03-verifier-noprogress | (b) contract | ✅ landed | $0 | structural |
-| bug-04-verifier-noprogress-regression | (b) contract | ✅ landed | $0 | structural |
-| bug-06-claude-worker-idle-noprogress | (a) lifecycle | ✅ landed | $0 | structural |
-| bug-09-verified-us-persistence | (b) contract | ✅ landed | $0 | structural |
-
-**All 10 historical bugs (#1-#10) covered.** 6 scenarios are real-LLM (require campaigns) gated by `RLP_REAL_LLM_GATE=1`; 4 are structural (zero cost, regression-guard the fix code itself). Future PRs fixing new bug classes MUST add a scenario for that class.
-
-Real-LLM total budget: ~$16 for full nightly run (6 × ~$3). Structural scenarios contribute zero cost. Per-PR LIGHT subset: structural-only (4 scenarios, <60s, $0).
-
-## 9. Integration with existing SV gate
-
-The current `tests/sv-gate-fast.sh` becomes the LIGHT subset. New `tests/sv-gate-real-llm.sh` becomes the FULL nightly suite. Existing scenario format is preserved for grep/unit checks; real-LLM scenarios are additive, not a replacement.
-
-`tests/sv-self-verify-*.sh` pattern stays — those are per-PR scenarios. New scenarios under `tests/sv-real-llm/scenarios/` are the real-LLM additions.
-
-## 10. Out of scope (deferred)
-
-- **Continuous deployment**: nightly schedule + GHA workflow. Add after first 3 scenarios prove the harness works.
-- **Cost dashboard**: aggregate cost trends over time. Add after first $50 month spent.
-- **Cross-engine A/B**: same scenario claude vs codex. Add after baseline established.
-- **Stress scenarios**: high-load, concurrent campaigns. Add after Phase B-E stabilization complete.
-
-## 11. This PR's scope (bootstrap only)
-
-This PR ships:
-1. This spec doc
-2. `tests/sv-real-llm/harness/run-scenario.sh` — single-scenario runner skeleton
-3. `tests/sv-real-llm/scenarios/bug-10-relaunch-phase-verify-hygiene.test.sh` — first real-LLM scenario, **gated by `RLP_REAL_LLM_GATE=1`** environment variable. Default: NOT executed. Operator runs explicitly.
-4. `tests/sv-real-llm/README.md` — quick how-to-run
-
-NOT in this PR:
-- nightly schedule (deferred)
-- remaining 9 scenarios (deferred)
-- integration into `sv-gate-fast.sh` or any existing gate (deferred)
-
-The bootstrap is small ON PURPOSE. Establish the pattern, prove it works on one bug, then scale.
-
-## 12. Verification
-
-- `bash tests/sv-real-llm/scenarios/bug-10-relaunch-phase-verify-hygiene.test.sh` runs in dry-mode (env not set) and prints "SKIPPED — RLP_REAL_LLM_GATE=1 to enable". No cost.
-- With `RLP_REAL_LLM_GATE=1` AND claude CLI available: scenario runs end-to-end. PASS = audit log line present + worker prompt iter-001 NOT rewritten + verifier dispatched. Cost ~$1-3.
-- Existing sv-gate-fast: 48/48 PASS unchanged.
-- Full Node suite: 339/339 PASS unchanged.
-
-## 13. Plan integration
-
-This PR advances Phase F from §5 of `docs/plans/v0.15-stabilization-plan.md`. After this PR lands, Phase F state changes from "planned" to "bootstrap complete; coverage roadmap active". Each subsequent PR closing a Phase B/C/D/E bug-class fix MUST add the corresponding real-LLM scenario per §8 table.