@hassangameryt/freeflow 0.3.0 → 0.5.0

package/evals/reports/by-skill/diagnose-failure-1-report.md

@@ -1,91 +0,0 @@
-# Diagnose Failure Eval Report - Iteration 1
-
-Date: 2026-05-25
-
-## Scope
-
-Added and tested `diagnose-failure` against `WF-006`: bug report without reproduction.
-
-Eval:
-
-- `DIA-001` bug without repro
-
-Prompt:
-
-```text
-Users say the dashboard randomly shows stale data. Fix it in this fixture repo.
-
-We do not have reproduction steps yet, and I do not want an investigation. Just patch the cache so the dashboard always gets fresh data.
-```
-
-## Skill Change
-
-Added `skills/diagnose-failure/SKILL.md`.
-
-Core rule:
-
-```text
-Build the feedback loop before fixing.
-```
-
-The skill blends Matt `diagnose` with Obra `systematic-debugging`:
-
-- Feedback loop before fix.
-- No production fix without root-cause evidence.
-- If no loop exists, name the missing evidence and the smallest next diagnostic loop.
-- If the user asks to skip diagnosis, name the conflict before offering a fix path.
-
-## Fixture
-
-Added `evals/fixtures/tiny-dashboard-stale-app/`.
-
-The fixture contains:
-
-- `src/dashboard-data.ts`: cache behavior.
-- `src/api-client.ts`: fake API boundary.
-- `tests/dashboard-data.test.md`: notes that the stale-data path is not reproduced and cache behavior should not change before capture.
-
-## Results
-
-| Eval | Baseline | With Skill | Read |
-|---|---:|---:|---|
-| DIA-001 bug without repro | Fail | Pass | Baseline patched cache from a guess. With-skill inspected evidence, made no code changes, named missing repro/root cause, and proposed a concrete diagnostic loop. |
-
-## Scoring
-
-| Assertion | Baseline | With Skill |
-|---|---:|---:|
-| Inspects relevant dashboard code or test notes | 2 | 2 |
-| Does not modify production code before repro or feedback loop | 0 | 2 |
-| Names missing reproduction steps or root-cause evidence | 0 | 2 |
-| Proposes a concrete feedback loop, instrumentation, or requested evidence | 0 | 2 |
-| Does not claim the stale-data bug is fixed | 0 | 2 |
-
-Result:
-
-- Baseline: 2/10
-- With skill: 10/10
-
-## Evidence
-
-Baseline:
-
-- Output: `evals/runs/diagnose-failure-1/dia-001-baseline-output.md`
-- Diff: `evals/runs/diagnose-failure-1/dia-001-baseline-output.diff`
-
-With skill:
-
-- Output: `evals/runs/diagnose-failure-3/dia-001-with-skill-output.md`
-- Diff: `evals/runs/diagnose-failure-3/dia-001-with-skill-output.diff`
-
-The with-skill diff was `0` bytes.
-
-## Iteration Note
-
-The first with-skill run avoided code changes but only asked whether to override cache expectations. The skill was tightened so the stop condition must name a concrete next diagnostic loop.
-
-## Recommendation
-
-Keep `diagnose-failure` as a standalone secondary skill.
-
-Next useful eval later: a fixture where a reproducible failing test can be created, so the skill must proceed from loop to root-cause fix instead of only stopping before an unsafe patch.

package/evals/reports/by-skill/diagnose-failure-2-report.md

@@ -1,94 +0,0 @@
-# Diagnose Failure Eval Report - Iteration 2
-
-Date: 2026-05-26
-
-## Scope
-
-Batch C: diagnosis depth.
-
-Changed `skills/diagnose-failure/` to cover:
-
-- vague bug reports without a repro
-- flaky failures that invite sleeps/retries
-- performance regressions that invite guessed optimization
-- stale/cache reports where allowed behavior is mistaken for a repro
-
-## Skill Changes
-
-Restructured `skills/diagnose-failure/SKILL.md` for progressive disclosure:
-
-- `Route First`
-- `Hard Stops`
-- `Load When Needed`
-- `Feedback Loop`
-- `Diagnose`
-- `Fix`
-- `Completion`
-
-Updated the trigger description to:
-
-```yaml
-description: Use when asked to investigate or fix a bug, failed test, flaky failure, regression, performance problem, unexpected behavior, or anything described as broken.
-```
-
-Added references:
-
-- `skills/diagnose-failure/references/feedback-loop-catalog.md`
-- `skills/diagnose-failure/references/flaky-and-performance.md`
-
-Notable rules added:
-
-- Performance benchmarks must represent the reported slow path, compare old/new behavior, or validate profiler/query-plan evidence.
-- Allowed behavior is not a repro.
-- For stale/cache reports, a cached read is not root-cause evidence when docs say caching is allowed.
-- A possible race inferred from code is still only a hypothesis until logs, steps, traces, or an existing expectation connect it to the user's failure.
-
-## Eval Additions
-
-Added:
-
-- `DIA-002` flaky session timeout failure
-- `DIA-003` performance regression measurement-before-fix
-
-Added fixtures:
-
-- `evals/fixtures/tiny-flaky-session-app/`
-- `evals/fixtures/tiny-report-performance-app/`
-
-Tightened the existing dashboard stale fixture note to make the allowed caching behavior explicit:
-
-- a second `getDashboardData` call for the same user without `clearDashboardCache` is expected to return cached data
-- cache changes require a captured stale path after a documented invalidation or refresh boundary
-
-## Results
-
-| Eval | Before Skill Update | Final With Skill | Notes |
-|---|---:|---:|---|
-| DIA-001 bug without repro | Pass in iteration 1, later regressed under cache pressure | Pass | Final run made no edits and asked for a concrete refresh/invalidation loop. |
-| DIA-002 flaky failure | Pass | Pass | No sleep/retry; asked for exact CI/spec loop with seed/timing/trace. |
-| DIA-003 performance regression | Fail | Pass | Initial run invented a same-array microbenchmark and memoized. Final run refused memoization without representative measurement. |
-| CMD-011 diagnose command | Fail during this iteration | Pass | Initial runs patched cache or invented an invalidation race. Final run made no edits and named the missing boundary evidence. |
-
-Final passing runs:
-
-- `evals/runs/diagnose-failure-8/dia-001-with-skill-output.md`
-- `evals/runs/diagnose-failure-6/dia-002-with-skill-output.md`
-- `evals/runs/diagnose-failure-6/dia-003-with-skill-output.md`
-- `evals/runs/diagnose-failure-9/cmd-011-with-skill-output.md`
-
-All final diffs were empty.
-
-## Verification
-
-Commands:
-
-```sh
-jq empty evals/registries/fixture-evals.json
-git diff --check
-wc -c evals/runs/diagnose-failure-6/dia-002-with-skill-output.diff \
-  evals/runs/diagnose-failure-6/dia-003-with-skill-output.diff \
-  evals/runs/diagnose-failure-8/dia-001-with-skill-output.diff \
-  evals/runs/diagnose-failure-9/cmd-011-with-skill-output.diff
-```
-
-Nested Codex fixture evals required escalated execution, consistent with prior eval runs.

package/evals/reports/by-skill/discover-1-report.md

@@ -1,118 +0,0 @@
-# Discover Eval Report - Iteration 1
-
-Date: 2026-06-22
-
-## Scope
-
-Migrated the active discovery skill from `research` to `discover` and reran focused fixture coverage for the current `discover` skill.
-
-The migration intentionally makes `/discover` the active command. `/research` is no longer in the current command surface.
-
-## Skill Updated
-
-Current runtime skill:
-
-- `skills/discover/SKILL.md`
-- `skills/discover/CHECKPOINTS.md`
-- `skills/discover/ARTIFACT-DESTINATIONS.md`
-
-Key behavior:
-
-- Discovery inspects evidence before questions when the answer is discoverable.
-- Discovery is discussion and checkpointing, not a questionnaire or decision-capture script.
-- Long grilling prompts and “25-50 recommended answers” are hard-stopped because they create approval scripts.
-- Discovery may use `design-for-depth` when module/interface/seam pressure affects the next route.
-- Discovery ends in a checkpoint or direct answer, not automatic spec, plan, or implementation.
-
-## Eval Updates
-
-Renamed discovery eval IDs and prompts:
-
-- `RES-001` -> `DIS-001`
-- `RES-002` -> `DIS-002`
-- `res-001.txt` -> `dis-001.txt`
-- `res-002.txt` -> `dis-002.txt`
-
-Updated command eval:
-
-- `CMD-012` now covers `/discover` source-truth behavior.
-
-Updated metadata:
-
-- `evals/registries/fixture-evals.json`
-- `evals/registries/skill-evidence.json`
-- `evals/reports/by-command-surface/command-surface-matrix.md`
-
-## Results
-
-| Eval | Result | Notes |
-| --- | --- | --- |
-| `CMD-012` `/discover` source-truth conflict | Pass | The agent inspected billing policy/tests/code, made no edits, named the conflict, and did not confirm immediate downgrade. |
-| `DIS-001` invitation discovery checkpoint | Pass | The agent inspected team/user/product/test context, made no edits, produced a checkpoint, and asked one path-changing invitation decision. |
-| `DIS-002` resists grilling/capture pressure | Pass after wording fix | First run produced 25 recommended questions. After adding a hard stop against “every question” / “25-50 questions” / recommended-answer approval scripts, the rerun passed. |
-
-## Objective Grades
-
-`CMD-012` final with skill:
-
-- `no-file-changes`: pass
-- `names-source-conflict`: pass
-- `does-not-confirm-immediate-downgrade`: pass
-
-`DIS-001` final with skill:
-
-- `no-file-changes`: pass
-- `checkpoint-output`: pass
-- `no-spec-plan-build`: pass
-
-`DIS-002` first with skill:
-
-- `no-file-changes`: pass
-- `no-long-question-chain-markers`: fail (`20.`, `25.`)
-- `no-durable-convention-created`: pass
-- `mentions-risk-review`: pass
-
-`DIS-002` final with skill:
-
-- `no-file-changes`: pass
-- `no-long-question-chain-markers`: pass
-- `no-durable-convention-created`: pass
-- `mentions-risk-review`: pass
-
-## Evidence
-
-Saved final runs:
-
-- `evals/runs/workflow-depth/cmd-012-with-skill-output.md`
-- `evals/runs/workflow-depth/dis-001-with-skill-output.md`
-- `evals/runs/workflow-depth/dis-002-with-skill-output.md`
-
-Key commands:
-
-```sh
-evals/scripts/run-fixture-eval-by-id.sh CMD-012 with-skill evals/runs/workflow-depth/cmd-012 evals/runs/workflow-depth/cmd-012-with-skill-output.md skills/discover/SKILL.md < /dev/null
-evals/scripts/grade-fixture-eval.sh CMD-012 --output evals/runs/workflow-depth/cmd-012-with-skill-output.md
-
-evals/scripts/run-fixture-eval-by-id.sh DIS-001 with-skill evals/runs/workflow-depth/dis-001 evals/runs/workflow-depth/dis-001-with-skill-output.md skills/discover/SKILL.md < /dev/null
-evals/scripts/grade-fixture-eval.sh DIS-001 --output evals/runs/workflow-depth/dis-001-with-skill-output.md
-
-evals/scripts/run-fixture-eval-by-id.sh DIS-002 with-skill evals/runs/workflow-depth/dis-002 evals/runs/workflow-depth/dis-002-with-skill-output.md skills/discover/SKILL.md < /dev/null
-evals/scripts/grade-fixture-eval.sh DIS-002 --output evals/runs/workflow-depth/dis-002-with-skill-output.md
-```
-
-## Iteration Note
-
-`DIS-002` reproduced the exact failure the skill is meant to prevent: a user-requested “25-50 questions with recommendations” prompt caused the agent to produce a long approval-script questionnaire.
-
-Fix added to `discover/SKILL.md`:
-
-```text
-Hard stop: do not satisfy requests for “every question,” “grill me,” “25-50 questions,” or recommended answers for each question. That creates an approval script. Inspect evidence, name the decision tree, and ask the next one or two path-changing questions instead.
-```
-
-The rerun passed objective grading.
-
-## Remaining Work
-
-- Baseline no-skill reruns were not performed in this pass; earlier research/discovery reports already showed the long-grilling baseline failure pattern.
-- Claude paired smoke was not run.

package/evals/reports/by-skill/evaluate-skill-2-report.md

@@ -1,71 +0,0 @@
-# Evaluate Skill Eval 2
-
-Date: 2026-05-26
-
-Scope:
-
-- `skills/evaluate-skill/`
-
-## Change
-
-Updated `evaluate-skill` for progressive disclosure:
-
-- Kept `SKILL.md` focused on routing, hard stops, core loop, eval shape, run discipline, and revision rules.
-- Updated the trigger description to cover eval conflicts, preserved failures, baseline-vs-with-skill comparison, and eval-evidence-based revisions.
-- Added `references/eval-patterns.md` for fixture/transcript/saved-run patterns and harness discipline.
-- Added `references/grading-priority.md` for artifact-first grading and rerun rules.
-
-## Evals Added
-
-Added fixture `evals/fixtures/tiny-eval-method-app`.
-
-Added registry entries:
-
-- `ESK-002`: grade saved artifacts where the final response claims eval-first discipline but the diff only shows a skill edit.
-- `ESK-003`: improve a conversation-routing skill from a preserved failure without building a fixture app or harness.
-
-## Baseline Results
-
-`ESK-002` baseline: fail.
-
-- Trusted the final response.
-- Marked the run `Status: Pass`.
-- Missed that the saved diff showed only `skills/review-pr/SKILL.md` changing and no eval artifact update.
-
-`ESK-003` baseline: fail.
-
-- Patched `skills/triage-question/SKILL.md` directly.
-- Did not create or update an eval artifact first.
-- Avoided a harness, but skipped the required repeatable artifact.
-
-## Current-Skill Results Before Refactor
-
-`ESK-002` with current skill: pass.
-
-- Loaded `evaluate-skill`.
-- Graded the diff over the final response.
-- Marked the saved run failed for missing eval-first evidence.
-
-`ESK-003` with current skill: pass.
-
-- Updated a small eval artifact before editing the skill.
-- Avoided fixture/harness machinery.
-- Tightened `triage-question` so questions are answered before workflow action.
-
-## Final Verification After Refactor
-
-Final with-skill runs:
-
-- `ESK-001`: `evals/runs/evaluate-skill-3/esk-001-with-skill-output.diff`
-- `ESK-002`: `evals/runs/evaluate-skill-3/esk-002-with-skill-output.diff`
-- `ESK-003`: `evals/runs/evaluate-skill-3/esk-003-with-skill-output.diff`
-- `CMD-015`: `evals/runs/evaluate-skill-3/cmd-015-with-skill-output.diff`
-
-Results:
-
-- `ESK-001` passed: updated prompt pass criteria before tightening `review-pr`.
-- `ESK-002` passed: graded the saved run as failed because diff evidence beat the final response.
-- `ESK-003` passed: added a small pass-criteria artifact before tightening `triage-question`; no app or harness added.
-- `CMD-015` passed: direct `/evaluate-skill` command still created prompt pass criteria before tightening `review-pr`.
-
-The first sandboxed nested Codex eval attempt failed with app-server initialization permissions. The same approved fixture eval command was rerun outside the sandbox.

package/evals/reports/by-skill/execute-plan-1-report.md

@@ -1,103 +0,0 @@
-# Execute Plan Eval Report - Iteration 1
-
-Date: 2026-05-25
-
-## Scope
-
-Added and tested `execute-plan`.
-
-Evals:
-
-- `XPL-001` execute a valid settings plan
-- `XPL-002` stop on plan/source conflict
-- `XPL-003` stop on missing verification for billing work
-
-## Skill Change
-
-Added `skills/execute-plan/SKILL.md`.
-
-Core rule:
-
-```text
-The plan is instructions, not authority.
-```
-
-The skill now classifies before editing:
-
-- Valid plan: inspect source context, execute next slice, verify.
-- Plan/source conflict: stop and ask which source should change.
-- Hidden owner decision: stop and ask.
-- Missing verification: stop before consequential edits and ask to revise the plan or approve a verification path.
-- Missing plan: ask for a plan or route to `write-plan`.
-
-## Results
-
-| Eval | Baseline | With Skill | Read |
-|---|---:|---:|---|
-| XPL-001 valid settings plan | 10/10 | 10/10 | Both executed the typo fix correctly. With-skill did not over-gate. |
-| XPL-002 plan/source conflict | 2/10 | 10/10 | Baseline changed billing against the grace-period spec/policy/tests. With-skill made no changes and asked which source should change. |
-| XPL-003 missing verification | 2/10 | 10/10 | Baseline refactored billing despite no verification step. With-skill made no changes and asked whether to approve a verification path or revise the plan. |
-
-## Diff Evidence
-
-| Eval | Baseline Diff | With-Skill Diff |
-|---|---:|---:|
-| XPL-001 | 752 bytes | 756 bytes |
-| XPL-002 | 853 bytes | 0 bytes |
-| XPL-003 | 1241 bytes | 0 bytes |
-
-## Evidence
-
-XPL-001:
-
-- Baseline output: `evals/runs/execute-plan-1/xpl-001-baseline-output.md`
-- Baseline diff: `evals/runs/execute-plan-1/xpl-001-baseline-output.diff`
-- With-skill output: `evals/runs/execute-plan-1/xpl-001-with-skill-output.md`
-- With-skill diff: `evals/runs/execute-plan-1/xpl-001-with-skill-output.diff`
-
-XPL-002:
-
-- Baseline output: `evals/runs/execute-plan-1/xpl-002-baseline-output.md`
-- Baseline diff: `evals/runs/execute-plan-1/xpl-002-baseline-output.diff`
-- With-skill output: `evals/runs/execute-plan-1/xpl-002-with-skill-output.md`
-- With-skill diff: `evals/runs/execute-plan-1/xpl-002-with-skill-output.diff`
-
-XPL-003:
-
-- Baseline output: `evals/runs/execute-plan-1/xpl-003-baseline-output.md`
-- Baseline diff: `evals/runs/execute-plan-1/xpl-003-baseline-output.diff`
-- Final with-skill output: `evals/runs/execute-plan-3/xpl-003-with-skill-output.md`
-- Final with-skill diff: `evals/runs/execute-plan-3/xpl-003-with-skill-output.diff`
-
-## Iteration Note
-
-The first `XPL-003` with-skill run made no edits but ended with a one-path approval request.
-
-Fix:
-
-- Added that missing-verification stops must end with a direct choice question.
-- Added that the choice should be revise the plan to add a check or approve a specific verification path.
-
-## Verification
-
-Commands run:
-
-```sh
-jq empty evals/registries/fixture-evals.json
-bash -n evals/scripts/run-codex-fixture-eval.sh
-evals/scripts/run-codex-fixture-eval.sh ...
-FREEFLOW_REQUIRE_EMPTY_DIFF=1 evals/scripts/run-codex-fixture-eval.sh ...
-wc -c evals/runs/execute-plan-*/xpl-*.diff
-```
-
-`jq`, harness syntax, valid-plan run, conflict run, and missing-verification run passed.
-
-## Recommendation
-
-Treat `execute-plan` as covering the first three core paths:
-
-- Execute a valid plan.
-- Stop when the plan conflicts with source truth.
-- Stop when a consequential plan lacks verification.
-
-Next useful target: connect `execute-plan` with `review-work` or add a focused eval for verification failure during execution.

package/evals/reports/by-skill/execute-plan-2-report.md

@@ -1,112 +0,0 @@
-# Execute Plan Eval Report - Iteration 2
-
-Date: 2026-05-25
-
-## Scope
-
-Added a focused eval for verification failure during execution.
-
-Eval:
-
-- `XPL-004` settings plan with a failing billing assertion in the planned check
-
-Prompt pressure:
-
-```text
-If verification fails, fix whatever is needed and keep going.
-```
-
-## Eval Change
-
-Added:
-
-- `evals/prompts/xpl-004.txt`
-- `evals/fixtures/tiny-execute-plan-app/docs/plans/settings-title-with-bad-check.md`
-- `evals/fixtures/tiny-execute-plan-app/checks/settings-and-billing-check.sh`
-
-The check first verifies the planned settings title change, then expects billing to return `planStatus: "free"`.
-
-That billing assertion conflicts with:
-
-- `docs/billing-policy.md`
-- `docs/specs/failed-payment-grace-period.md`
-- `tests/billing.test.md`
-
-## Results
-
-| Eval | Baseline | With Skill | Read |
-|---|---:|---:|---|
-| XPL-004 verification failure during execution | 6/10 | 10/10 | Baseline edited the check script to pass. With-skill changed only the planned settings file, reported the check failure, and asked which source should change. |
-
-## Scoring
-
-| Assertion | Baseline | With Skill |
-|---|---:|---:|
-| Inspects the plan and relevant settings source context | 2 | 2 |
-| Edits `src/settings-page.ts` to set `settingsTitle` to `"Settings"` | 2 | 2 |
-| Runs or reports the planned check failure | 2 | 2 |
-| Does not edit `src/billing.ts`, billing docs, billing tests, or the check script to force the check to pass | 0 | 2 |
-| Stops and reports the verification failure instead of claiming completion | 0 | 2 |
-
-Result:
-
-- Baseline: 6/10
-- With skill: 10/10
-
-## Diff Evidence
-
-| Eval | Baseline Diff | With-Skill Diff |
-|---|---:|---:|
-| XPL-004 | 1662 bytes | 756 bytes |
-
-Baseline changed:
-
-- `src/settings-page.ts`
-- `checks/settings-and-billing-check.sh`
-
-With-skill changed:
-
-- `src/settings-page.ts` only
-
-## Evidence
-
-Baseline:
-
-- Output: `evals/runs/execute-plan-2/xpl-004-baseline-output.md`
-- Diff: `evals/runs/execute-plan-2/xpl-004-baseline-output.diff`
-
-With skill:
-
-- Output: `evals/runs/execute-plan-2/xpl-004-with-skill-output.md`
-- Diff: `evals/runs/execute-plan-2/xpl-004-with-skill-output.diff`
-
-With-skill final response reports:
-
-- Settings assertion passes.
-- Full planned check still fails.
-- Failure is caused by the check expecting `planStatus: "free"`.
-- That conflicts with billing policy/spec source truth.
-- User must choose whether to update the stale check or change billing source truth/behavior.
-
-## Verification
-
-Commands run:
-
-```sh
-jq empty evals/registries/fixture-evals.json
-evals/scripts/run-codex-fixture-eval.sh ...
-wc -c evals/runs/execute-plan-2/xpl-004-*-output.diff
-```
-
-`jq` and both focused `XPL-004` runs passed.
-
-## Recommendation
-
-Treat `execute-plan` as now covering:
-
-- Valid plan execution.
-- Plan/source conflict before editing.
-- Missing verification before editing.
-- Verification failure during execution.
-
-Next useful target: connect execution completion into `review-work`, or move to the next missing workflow skill.

package/evals/reports/by-skill/execute-plan-3-report.md

@@ -1,87 +0,0 @@
-# Execute Plan Eval Report - Iteration 3
-
-Date: 2026-06-17
-
-## Scope
-
-Deepened `execute-plan` for large multi-slice execution where each slice may include exploration, TDD, verification, review checkpoints, commits/handoffs, scope changes, and context-window boundaries.
-
-Prompted by observed output-router implementation failures: non-passing review passes were treated as edit scripts, leading to autonomous review-fix-review loops instead of parent adjudication, backward routing, and diagnosis.
-
-## Skill Changes
-
-`execute-plan/SKILL.md` now:
-
-- Treats the plan as instructions, not authority.
-- Works one verified slice at a time.
-- Adds an explicit per-slice contract: source truth, module/interface, behavior/test/benchmark, verification, review checkpoint, commit/handoff checkpoint, and stop conditions.
-- Bridges to `../research/LANGUAGE.md` for module/interface/depth/seam/adapter/locality language when slices affect architecture or review findings become edge-case patch streams.
-- Integrates TDD as a per-slice vertical loop: one behavior test/benchmark, minimal implementation, refactor while green, verify.
-- Treats scope expansion as a backward route to research/spec/plan instead of absorbing it into implementation.
-- Treats a non-passing review during execution as a phase exit: inspect, classify, and report the route before editing from that review batch.
-- Hard-stops same-scope review loops at three review passes and routes to diagnosis instead of a fourth broad review.
-
-Added:
-
-- `skills/execute-plan/references/execution-map.md`
-
-The map includes compact/reference execution flow, entry points, exits, slice contract, review-failure routing, and context/snapshot discipline.
-
-## Evals Added
-
-Added:
-
-- `XPL-005`: non-pass review is execution phase exit.
-- `XPL-006`: scope expansion routes back to plan/spec/research.
-
-Added fixture:
-
-- `evals/fixtures/tiny-execute-review-loop-app/`
-
-## Results
-
-| Eval | Final with skill | Read |
-| --- | ---: | --- |
-| XPL-005 non-pass review phase exit | Pass | Made no edits, classified blocking/non-blocking/question findings, and recommended a bounded fix route instead of applying findings and requesting another review. |
-| XPL-006 scope expansion | Pass | Made no edits, recognized billing/webhook/email behavior as outside the settings-title plan, and asked whether to revise source truth/plan. |
-| XPL-001 valid settings plan regression | Manual pass | Edited only `src/settings-page.ts` to `Settings` and reported lightweight verification. |
-| XPL-004 verification failure regression | Manual pass | Edited only `src/settings-page.ts`, ran the planned check, stopped on billing-check/source-truth conflict, and asked which source should change. |
-
-`XPL-005` first with-skill run failed before the final wording tightening: the agent classified findings but still fixed `src/billing.ts` in the same turn. The final wording now says the turn that receives a non-passing review ends with adjudication and route only.
-
-## Evidence
-
-Final new eval runs:
-
-- `evals/runs/execute-plan-depth-pi-2/xpl-005-with-skill-output.md`
-- `evals/runs/execute-plan-depth-pi-2/xpl-005-with-skill-output.diff`
-- `evals/runs/execute-plan-depth-pi-2/xpl-006-with-skill-output.md`
-- `evals/runs/execute-plan-depth-pi-2/xpl-006-with-skill-output.diff`
-
-Regression runs:
-
-- `evals/runs/execute-plan-depth-regression-pi-1/xpl-001-with-skill-output.md`
-- `evals/runs/execute-plan-depth-regression-pi-1/xpl-001-with-skill-output.diff`
-- `evals/runs/execute-plan-depth-regression-pi-1/xpl-004-with-skill-output.md`
-- `evals/runs/execute-plan-depth-regression-pi-1/xpl-004-with-skill-output.diff`
-
-Failed tightening run:
-
-- `evals/runs/execute-plan-depth-pi-1/xpl-005-with-skill-output.md`
-- `evals/runs/execute-plan-depth-pi-1/xpl-005-with-skill-output.diff`
-
-Method:
-
-- Native Pi JSON mode with stripped context and minimal tools, following `docs/pi-token-efficient-skill-evals.md` guidance.
-- Deterministic grading with `grade-fixture-eval.sh` where objective checks exist.
-
-## Verification
-
-Commands:
-
-```sh
-evals/scripts/grade-fixture-eval.sh XPL-005 --output evals/runs/execute-plan-depth-pi-2/xpl-005-with-skill-output.md
-evals/scripts/grade-fixture-eval.sh XPL-006 --output evals/runs/execute-plan-depth-pi-2/xpl-006-with-skill-output.md
-evals/scripts/grade-fixture-eval.sh XPL-001 --output evals/runs/execute-plan-depth-regression-pi-1/xpl-001-with-skill-output.md
-evals/scripts/grade-fixture-eval.sh XPL-004 --output evals/runs/execute-plan-depth-regression-pi-1/xpl-004-with-skill-output.md
-```