@phi-code-admin/phi-code 0.87.0 → 0.89.0

package/CHANGELOG.md

@@ -1,5 +1,116 @@
 # Changelog
 
+## [0.89.0] - 2026-07-11
+
+### Added
+
+- **Two new execution-grounded commands, `/debug` and `/build`**, implementing
+  the design in `docs/design/plan-debug-build.md`. They exist because the
+  SWE-bench-lite head-to-head measured that `/plan`'s TEST/REVIEW phases approve
+  wrong code — they grade the model's own reconstruction, not a real run. These
+  modes move the burden of proof from opinion to **execution**.
+  - **`/debug <failing state>`** — turn a REAL failure green through
+    `REPRODUCE → LOCALIZE → FIX → VERIFY`. It refuses to guess: if the failure
+    does not reproduce on the current code, or cannot be run at all, it stops
+    with `BLOCKED` instead of fabricating a fix. `VERIFY` requires two real runs
+    (the reproduction now passes AND the suite does not regress) before it will
+    ever say `FIXED`. Each phase routes to a model that does not share the
+    coder's blind spot (`routing.json`).
+  - **`/build <spec>`** — `/plan`'s decomposition (`EXPLORE → PLAN → CODE`) plus
+    an execution-grounded `BUILD-VERIFY`: run the recipe, check acceptance
+    criteria derived from the spec, run an **executable** red-team against the
+    input regime the change touched (a runnable test that goes red, never a
+    critique), and route every real failure to the `/debug` protocol. It reports
+    `SUCCESS` only when a real run meets the acceptance criteria, otherwise an
+    honest `PARTIAL` that lists what still fails.
+- **Six pure, unit-tested decision cores** backing the above (no live model, no
+  fs at the boundary): `execution.ts` (the run oracle: `runCommand`/`passed`),
+  `acceptance.ts` (executable acceptance criteria; manual criteria are never
+  counted as passing), `debug-contract.ts` (`decideReproduce`/`decideVerify` —
+  BLOCKED rather than least-bad), `triage.ts` (adaptive depth — pay the pipeline
+  cost only when earned), `redteam.ts` (a break is recorded only when a test
+  actually ran red), and `build-loop.ts` (`decideBuildRound` — SUCCESS /
+  CONTINUE-to-/debug / honest PARTIAL). 73 new tests, including an end-to-end
+  integration test of the `/debug` and `/build` chain against a simulated Pi
+  runtime.
+
+### Changed
+
+- The orchestrator now carries an `orchestrationMode`. `/debug` and `/build` run
+  on a **separate linear driver** and never engage `/plan`'s review-fix cycle or
+  its five-phase bookkeeping, so the existing `/plan` path is byte-for-byte
+  unchanged (its integration suite still passes).
+
+### Honest caveat
+
+- These modes are the protocol + scaffolding. Their value depends on a real
+  executable environment; on a host that cannot run the target they downgrade to
+  `BLOCKED` rather than a fabricated pass. They are **not yet benchmarked** to
+  beat a single shot — per the design doc, that measurement (which needs a
+  per-project execution sandbox) is the next step, and no claim is made here that
+  they do.
+
+## [0.88.0] - 2026-07-10
+
+### Changed
+
+- **The /plan phase contract is now structured-primary with a text fallback**
+  (see `docs/adr/0001-phase-contract.md`). A phase agent CALLS a new
+  `phase_result` tool to emit its verdict / blocking / handoff as data;
+  `resolvePhaseOutcome` merges that structured emission field-by-field with the
+  regex-scraped markdown report, preferring the structured value. TEST and
+  REVIEW are instructed to call it. When a model does not call the tool,
+  behavior is identical to the previous text-only path — the change cannot make
+  any run worse. This replaces "regex luck" with an exact machine-read path for
+  the control-flow-critical signals (verdict, BLOCKED, the review-fix cycle).
+
+### Added
+
+- **End-to-end integration test of the orchestrator.** The real orchestrator
+  extension is now driven through a full multi-phase run against a simulated Pi
+  runtime (`test/orchestrator-integration.test.ts`): phase progression, handoff
+  propagation, the review-FAIL → fix → re-review cycle, and the BLOCKED pause
+  are all exercised together, not just the pure decision function. (Scripted
+  phase outcomes — deterministic, no live model.)
+- **An eval harness** (`evals/`) — the measurement infrastructure that did not
+  exist. Tasks with deterministic pass/fail verifiers, a runnable baseline
+  strategy (`npx tsx evals/run.ts`), unit-tested scoring/aggregation
+  (`test/evals-lib.test.ts`), and a demonstrated real run (2/2 baseline tasks on
+  a live model). `evals/README.md` documents the methodology and is honest that
+  the /plan-vs-baseline head-to-head is not yet a single number.
+- **Two ADRs** documenting the phase-contract decision and the independent-review
+  release gate (`docs/adr/`).
+
+### Fixed
+
+- Bugs found by an **independent adversarial review** of the phase-contract
+  change (see `docs/adr/0002-independent-review.md`), each now pinned by a
+  regression test:
+  - **State leak between runs**: `currentPhaseResult` was reset only in
+    `sendNextPhase`, but the first phase of each `/plan` run launches directly —
+    a stale structured result (e.g. a BLOCKED verdict) from a previous run could
+    abort a fresh run at phase 1. Reset on every run start + a phase-identity
+    stamp so a stale/late result is never mistaken for the current phase.
+  - **Field erasure on multi-call**: a second `phase_result` call replaced the
+    whole object, wiping a verdict set by the first; it now merges only the
+    fields each call sets.
+  - **Dead text fallback for PLAN/CODE**: `readPhaseReport` looked for
+    `<key>-<ts>.md` but PLAN writes `todo-<ts>.md` and CODE `progress-<ts>.md`,
+    so their handoff blocks were never read. Mapped key → report file.
+  - **`extractSection` over-matching**: a prose line starting with a section
+    word (e.g. "Blocking issues remain: 2") could be read as that section's
+    header. The plain-label form now requires `:` or end-of-line.
+- A TEST/REVIEW phase that finished with tool work but emitted no verdict at all
+  (neither structured nor a VERDICT line) is now surfaced instead of silently
+  passing.
+- **The eval runner** had two Windows bugs its first real run caught: temp-dir
+  cleanup raced `EPERM` (now retries), and `shell:true` with an args array
+  mangled the prompt (now a single quoted command string).
+
+### Tooling
+
+- Biome now also lints `evals/`.
+
 ## [0.87.0] - 2026-07-10
 
 ### Added

package/docs/adr/0001-phase-contract.md

@@ -0,0 +1,57 @@
+# ADR 0001 — Structured-primary phase contract for /plan
+
+Status: accepted (2026-07-10)
+
+## Context
+
+The /plan orchestrator chains five agent phases (explore → plan → code → test
+→ review). Each phase must report two things the orchestrator acts on:
+
+- a **verdict** (TEST/REVIEW): PASS / FAIL / BLOCKED / SKIP, and
+- a **handoff** (+ BLOCKING findings for REVIEW) carried to the next phase.
+
+Originally this was communicated **only** as markdown the model wrote to
+`.phi/plans/<phase>-<ts>.md`, which the orchestrator scraped with regexes
+(`## VERDICT:`, `## HANDOFF`, `## BLOCKING`). That is fragile: models phrase
+headers inconsistently (`**HANDOFF**`, `HANDOFF:`, mid-sentence "verdict"), and
+a mis-scrape silently degrades control flow — a missed REVIEW FAIL skips the fix
+cycle; a missed BLOCKED keeps a doomed run going. The original justification for
+text-only was that the upstream proxy did not guarantee valid structured tool
+output.
+
+## Decision
+
+Keep the markdown report (it is the human-readable artifact) but make the
+**machine-read path structured and primary**:
+
+- The orchestrator registers a `phase_result` tool. TEST and REVIEW phases are
+  instructed to call it with `{verdict, blocking, handoff}`; any phase may call
+  it to hand off.
+- `resolvePhaseOutcome(structured, reportText)` (pure, unit-tested) merges the
+  two sources **field by field**, preferring the structured value and falling
+  back to the regex-scraped report per field. When the model calls the tool the
+  outcome is exact; when it does not, behavior is byte-for-byte the pre-existing
+  text path.
+- The text parser was hardened anyway (`extractSection` accepts heading, bold,
+  and plain-label forms) so the fallback is as robust as possible.
+
+## Why not go structured-only
+
+Two reasons the text path stays as a fallback rather than being removed:
+
+1. **Provider variance.** Not every provider/proxy reliably emits tool calls on
+   every turn; a model that writes a good report but forgets the tool call must
+   still drive the pipeline correctly.
+2. **Zero-regression migration.** Making structured additive means the change
+   cannot make any existing run worse — the worst case equals the old behavior.
+
+## Consequences
+
+- Robustness of control flow (verdict/BLOCKED/fix-cycle) now depends on a
+  structured emission when available, not on regex luck.
+- Two code paths must stay in sync; `resolvePhaseOutcome` centralizes the merge
+  and is covered by unit tests, and `orchestrator-integration.test.ts` drives
+  the whole chain via the structured path.
+- Follow-up (not done here): once telemetry shows the structured path is taken
+  reliably across the providers phi ships, the text fallback can be demoted to a
+  warning-only safety net.

package/docs/adr/0002-independent-review.md

@@ -0,0 +1,47 @@
+# ADR 0002 — Independent adversarial review as a release gate
+
+Status: accepted (2026-07-10)
+
+## Context
+
+phi-code is largely built by a single author, who is also the only reviewer.
+Self-review misses the bugs the author's mental model is blind to — the code
+does what the author *thinks* it does, and they test that. There is no external
+human reviewer on hand for every change.
+
+## Decision
+
+Before shipping a non-trivial change to a load-bearing subsystem (the /plan
+orchestrator, the model/provider layer, compaction, the extension runtime), run
+an **independent adversarial review**: a reviewer with fresh context that did
+not write the code, prompted to *refute* — to find state leaks, races, contract
+mismatches, and edge cases — not to approve.
+
+Findings are triaged (verify each against the code, discard false positives),
+the real ones are fixed, and each fix is pinned with a regression test before
+the change ships.
+
+## Evidence this works
+
+The structured-phase-contract change (ADR 0001) passed the author's own unit
+and integration tests. An independent review of that change then found four
+real defects the author's tests missed:
+
+- a structured result leaking from one `/plan` run into the next (a stale
+  BLOCKED verdict could abort a fresh run at phase 1);
+- a second `phase_result` call erasing fields set by the first;
+- no phase-identity guard against a late tool call landing after a transition;
+- the text HANDOFF fallback being dead for two phases due to a report-file name
+  mismatch.
+
+All four are now fixed and covered by regression tests
+(`orchestrator-integration.test.ts`, `phase-machine.test.ts`).
+
+## Consequences
+
+- "Green tests" is necessary but not sufficient for a load-bearing change; an
+  independent refutation pass is part of the definition of done.
+- The reviewer need not be human to be useful — it needs to be *independent of
+  the authoring context* and prompted adversarially. This is not a substitute
+  for real external users, whose absence remains an honest limitation of the
+  project's validation.

package/docs/design/plan-debug-build.md

@@ -0,0 +1,266 @@
+# Design: /plan, /debug, /build — composable, execution-grounded modes
+
+Status: proposed (2026-07-11)
+Informed by: the SWE-bench-lite head-to-head (ADR 0001/0002 and evals/), which
+measured that a 5-phase, deliberation-heavy /plan ties or loses to a single shot
+at 6–14× the cost on bug-fixing, and — decisively — that its TEST/REVIEW phases
+approve wrong code because they grade the model's own reconstruction, not a real
+run. Multi-model diversity did NOT fix this: a different family (deepseek)
+independently endorsed the same wrong guard, because the error was a *shared
+plausible misconception*, not a model-specific blind spot.
+
+The one thing that would have caught it was running the real test. So this design
+moves the burden of proof from the model to **execution**, and splits the work
+into three composable commands.
+
+## Principles (each is a measured lesson, not a preference)
+
+1. **Execution is the only oracle.** Every accept/reject decision must be backed
+   by running real code — the existing test suite, a reproduction, or the built
+   app — not by a model reviewing its own output. Opinion (even adversarial,
+   even multi-model) shares misconceptions; a red test run does not.
+2. **Adversarial means executable.** "Challenge the fix" must produce a *failing
+   run*, never a critique. A run is objective; prose is not.
+3. **Pay complexity only when earned.** A one-line change must not trigger a
+   five-phase pipeline. Depth adapts to task size/type, decided by cheap signals.
+4. **Diversity proposes, the oracle disposes.** Multiple models are useful for
+   generating *candidate fixes*, filtered by real execution — not for casting
+   *review votes*, which correlate.
+5. **/plan is for building; /debug is for fixing.** They are different jobs with
+   different oracles. Do not conflate them; compose them.
+
+## The three commands
+
+| Command | Job | Oracle | Standalone use |
+|---|---|---|---|
+| `/plan <spec>` | Build from a description (decompose → implement) | Acceptance criteria derived from the spec + a smoke run | "Scaffold / build this feature" |
+| `/debug <failing-state>` | Turn a real failure green | The failing state + the existing test suite, re-run | "This test/trace is broken, fix it" |
+| `/build <spec>` | Build AND make it actually work | The full loop: run → red-team → debug → re-run | "Build this and don't stop until it runs" |
+
+`/build` is the outer loop that composes the two primitives with **execution
+between them**. All three remain usable alone.
+
+---
+
+## /plan — build from a spec
+
+Phases (each may route to its own model via routing.json):
+
+1. **EXPLORE** — map the existing code, conventions, entry points. Read-only.
+2. **PLAN** — architecture + an ordered task list. Each task is a self-contained
+   prompt (the prompt-architect skill: `[CONTEXT] → [TASK] → [FORMAT] →
+   [CONSTRAINTS]`). This decomposition is /plan's core value on large,
+   under-specified builds — the part a single shot cannot hold coherently.
+3. **CODE** — implement the tasks in dependency order.
+4. **SELF-CHECK** — the cheapest real signal available: it compiles / typechecks
+   / lints, and a smoke run of the entry point does not crash. NOT a substitute
+   for verification — just a floor.
+
+**Input:** a natural-language spec.
+**Output contract (machine-readable, consumed by /build):**
+```
+{
+  runRecipe:  { build?: string, run: string, test?: string, readySignal?: string },
+  acceptance: string[],   // testable criteria derived from the SPEC (not the code)
+  changedFiles: string[],
+  selfCheck:  "pass" | "fail" | "skipped"
+}
+```
+The `acceptance` list is the contract: crisp, checkable statements traced to the
+request ("POST /login returns 200 + a JWT for valid creds; 401 otherwise").
+Deriving these from the *spec* — before and independent of the implementation —
+is what keeps verification from becoming circular.
+
+Deliberately NOT in /plan: heavy TEST and REVIEW. Verification belongs to the
+execution loop (/build), where it can run for real. Standalone /plan stops at
+SELF-CHECK and hands its output to the user (or to /build).
+
+---
+
+## /debug — turn a real failure green
+
+The heart of the redesign. /debug never guesses what is wrong; it is *given* a
+failure and reproduces it.
+
+**Input contract — a concrete failing state (at least one):**
+```
+{
+  failingTest?:  string,   // e.g. "pytest tests/x.py::test_y"
+  trace?:        string,   // a stack trace / error output
+  reproCommand?: string,   // a command that exhibits the bug
+  expected?:     string,   // what should happen instead (from the user/spec)
+  cwd:           string
+}
+```
+If none is runnable, /debug does not fabricate one from imagination — it asks for
+one, or emits `BLOCKED: no reproducible failing state`.
+
+Phases:
+
+1. **REPRODUCE** — run the failing state, confirm it fails, capture the exact
+   symptom. If it does NOT fail on the current code, stop: `BLOCKED: cannot
+   reproduce`. (This alone kills the class of "fixes" for non-bugs.)
+2. **LOCALIZE** — drive from the real symptom: read the traceback, grep the
+   changed/nearby symbols, narrow to the fault site. Localization, not more
+   review, is the underrated lever.
+3. **FIX (generate N candidates)** — produce several minimal candidate patches
+   (different models and/or temperatures). Prefer the smallest; every added
+   guard/condition is a liability (the 3362 failure was an over-clever guard).
+4. **VERIFY (the oracle)** — for EACH candidate, run: (a) the reproduced failure
+   → must now pass, and (b) the existing test suite → must not regress. Select
+   the **minimal candidate that passes both** via `candidate-select.ts`. If none
+   passes, emit `BLOCKED` with the closest diagnostic — do NOT ship the least-bad.
+
+**Output contract:**
+```
+{ verdict: "FIXED" | "BLOCKED",
+  patch?: string,                 // minimal, verified
+  evidence: { reproBefore: "fail", reproAfter: "pass", suite: "green" },
+  reason?: string }               // when BLOCKED
+```
+`FIXED` is only ever emitted with a real before/after run pasted. There is no
+path to a green verdict without execution.
+
+---
+
+## /build — build until it runs
+
+`/build` = `/plan` then a bounded loop that closes the build→run→fix cycle with
+execution as the glue.
+
+```
+plan = /plan(spec)                      # code + runRecipe + acceptance[]
+for round in 1..MAX_ROUNDS (budgeted):
+    run = execute(plan.runRecipe)       # ORACLE #1: does it run + meet acceptance?
+    unmet = acceptance criteria not satisfied by `run`
+    redTeam = red_team(plan, acceptance)   # ORACLE #2: can an adversary break it? (below)
+    failures = unmet ∪ redTeam.breakingCases
+    if failures is empty: return SUCCESS(plan)
+    /debug({ from each failure: failingTest/trace/reproCommand, expected })  # fix REAL failures
+    # /debug mutates the working tree; loop re-runs from the top
+return PARTIAL(plan, remaining failures)   # honest: what still fails, not a false PASS
+```
+
+Key properties:
+- The `execute` step is the oracle that /plan alone lacked. A criterion is met
+  only if the running program demonstrates it.
+- Each `/debug` call is grounded in a **real** failure (a failed acceptance run
+  or a red-team breaking case), never in self-review.
+- The loop is **budgeted** and returns `PARTIAL` honestly when it cannot close —
+  listing the still-failing criteria — instead of a confident-wrong `PASS`.
+
+---
+
+## The executable red-team protocol
+
+This replaces the current REVIEW phase (which we measured as a rubber stamp). The
+adversary's deliverable is a **failing run**, not an opinion.
+
+```
+red_team(code, acceptance, changedFiles):
+    dry = 0
+    while dry < K and within budget:
+        # An adversary agent (ideally a different model family than the coder —
+        # different break intuitions) targets the BOUNDARY the change touches.
+        attempt = adversary.write_breaking_case(
+            focus = changedFiles,                     # not the whole app
+            regimes = enumerate_input_regimes(changedFiles),  # e.g. buffered vs streaming,
+                                                              # empty, null, boundary, wrong-type
+            goal = "make an acceptance criterion or an invariant FALSE, as a runnable test")
+        result = execute(attempt)                     # RUN it — this is the whole point
+        if result.green:
+            dry += 1                                  # could not break it this round
+        else:
+            dry = 0
+            record breakingCase(attempt, result.symptom)   # a concrete red run
+    return breakingCases
+```
+
+Why this works where adversarial *prose* failed (3362): the guard survived a
+different model's careful written review because the reviewer shared the
+misconception. It would NOT have survived `r.raw = io.BytesIO(...);
+assert all(isinstance(c, str) for c in r.iter_content(decode_unicode=True))`
+being *run* — that assertion is red regardless of what any model believes.
+The rule that turns "adversarial" from theatre into signal: **the adversary
+must attack the specific input regime the diff changed, and must express the
+attack as an executed test, not a claim.**
+
+---
+
+## Execution grounding (the non-negotiable prerequisite)
+
+Every oracle above requires running real code with the project's real
+dependencies. On a host that cannot run the target (e.g. an old library under a
+too-new Python — exactly what defeated the 3362 measurement), the modes must:
+- run inside a sandbox/container that has the project's real environment, or
+- when no such environment is available, DOWNGRADE honestly: /debug and /build
+  emit `BLOCKED: no executable environment` (or a low-confidence draft clearly
+  labelled unverified) — never a fabricated-reconstruction `PASS`.
+
+Reusing phi's `run` / `verify` skills for local projects is the first target;
+a per-project container is the general solution.
+
+---
+
+## Triage / adaptive depth (cost discipline)
+
+Cheap up-front classification decides the mode and the depth:
+
+| Signal | Route |
+|---|---|
+| Small edit, a known failing test/trace | `/debug` (skip planning entirely) |
+| Small self-contained feature, single shot + verify passes | ship the single shot; skip the loop |
+| Large, multi-file, under-specified build | `/build` (full loop) |
+| A single shot's output already meets acceptance on a real run | done — do not deliberate further |
+
+Measured rationale: the 6–14× overhead is only worth paying when a single shot
+*fails the real oracle*. Otherwise the cheapest passing candidate wins.
+
+---
+
+## Model routing
+
+Per-phase model assignment stays (routing.json). Corrected use, per the
+measurements:
+- **Candidate generation** (CODE, /debug FIX): diversity helps — different
+  models/temperatures produce different candidates, and the oracle filters.
+- **Red-team adversary**: a different family than the coder helps (different
+  break intuitions) — but its output is a *run*, so it is robust even to shared
+  misconceptions.
+- **Review-by-opinion**: removed. We measured it as correlated and unreliable;
+  execution replaces it.
+
+---
+
+## What is reused vs new
+
+Reused from today's orchestrator: the phase engine, per-phase model switching,
+the pure phase state machine (`phase-machine.ts`), the structured `phase_result`
+contract, the fix-cycle, `candidate-select.ts` (minimality selection).
+
+New work: (1) extract the fix-cycle into a standalone `/debug` with the input
+contract above; (2) make `acceptance[]` + `runRecipe` first-class /plan outputs;
+(3) the `/build` loop; (4) the executable red-team; (5) the execution sandbox and
+the real-run verify path. Items 1–4 are prompt/orchestration changes on existing
+machinery; item 5 is the real infrastructure investment.
+
+## How we will know it worked (this must be measured, not shipped on faith)
+
+- `/debug`: on SWE-bench-lite (bug-fixing), with a real executable environment,
+  resolved-rate vs the single-shot baseline. The hypothesis is that real-run
+  verification + candidate selection beats single-shot; if it does not, the
+  premise is wrong and we stop.
+- `/build`: on a *build* task set (multi-file features with runnable acceptance
+  criteria — the eval /plan actually deserves, which SWE-bench does not provide),
+  scored on acceptance-met + does-it-run + coherence, single-shot vs /build.
+
+No mode ships to npm before its number beats the baseline it claims to improve.
+
+## Non-goals / honest caveats
+
+- This does not add "more specialized agents" for its own sake. The measured
+  lesson is the opposite: minimal + execution beats elaborate + deliberation.
+- It does not claim /build will beat single-shot — it makes /build *measurable*
+  and gives it the one ingredient (execution) that could make it win.
+- Without the execution sandbox, this is just a reorganization; the sandbox is
+  where the value is, and it is real work.