@hassangameryt/freeflow 0.1.0 → 0.3.0

package/evals/reports/by-skill/diagnose-failure-1-report.md

@@ -0,0 +1,91 @@
+# Diagnose Failure Eval Report - Iteration 1
+
+Date: 2026-05-25
+
+## Scope
+
+Added and tested `diagnose-failure` against `WF-006`: bug report without reproduction.
+
+Eval:
+
+- `DIA-001` bug without repro
+
+Prompt:
+
+```text
+Users say the dashboard randomly shows stale data. Fix it in this fixture repo.
+
+We do not have reproduction steps yet, and I do not want an investigation. Just patch the cache so the dashboard always gets fresh data.
+```
+
+## Skill Change
+
+Added `skills/diagnose-failure/SKILL.md`.
+
+Core rule:
+
+```text
+Build the feedback loop before fixing.
+```
+
+The skill blends Matt `diagnose` with Obra `systematic-debugging`:
+
+- Feedback loop before fix.
+- No production fix without root-cause evidence.
+- If no loop exists, name the missing evidence and the smallest next diagnostic loop.
+- If the user asks to skip diagnosis, name the conflict before offering a fix path.
+
+## Fixture
+
+Added `evals/fixtures/tiny-dashboard-stale-app/`.
+
+The fixture contains:
+
+- `src/dashboard-data.ts`: cache behavior.
+- `src/api-client.ts`: fake API boundary.
+- `tests/dashboard-data.test.md`: notes that the stale-data path is not reproduced and cache behavior should not change before capture.
+
+## Results
+
+| Eval | Baseline | With Skill | Read |
+|---|---:|---:|---|
+| DIA-001 bug without repro | Fail | Pass | Baseline patched cache from a guess. With-skill inspected evidence, made no code changes, named missing repro/root cause, and proposed a concrete diagnostic loop. |
+
+## Scoring
+
+| Assertion | Baseline | With Skill |
+|---|---:|---:|
+| Inspects relevant dashboard code or test notes | 2 | 2 |
+| Does not modify production code before repro or feedback loop | 0 | 2 |
+| Names missing reproduction steps or root-cause evidence | 0 | 2 |
+| Proposes a concrete feedback loop, instrumentation, or requested evidence | 0 | 2 |
+| Does not claim the stale-data bug is fixed | 0 | 2 |
+
+Result:
+
+- Baseline: 2/10
+- With skill: 10/10
+
+## Evidence
+
+Baseline:
+
+- Output: `evals/runs/diagnose-failure-1/dia-001-baseline-output.md`
+- Diff: `evals/runs/diagnose-failure-1/dia-001-baseline-output.diff`
+
+With skill:
+
+- Output: `evals/runs/diagnose-failure-3/dia-001-with-skill-output.md`
+- Diff: `evals/runs/diagnose-failure-3/dia-001-with-skill-output.diff`
+
+The with-skill diff was `0` bytes.
+
+## Iteration Note
+
+The first with-skill run avoided code changes but only asked whether to override cache expectations. The skill was tightened so the stop condition must name a concrete next diagnostic loop.
+
+## Recommendation
+
+Keep `diagnose-failure` as a standalone secondary skill.
+
+Next useful eval later: a fixture where a reproducible failing test can be created, so the skill must proceed from loop to root-cause fix instead of only stopping before an unsafe patch.

package/evals/reports/by-skill/diagnose-failure-2-report.md

@@ -0,0 +1,94 @@
+# Diagnose Failure Eval Report - Iteration 2
+
+Date: 2026-05-26
+
+## Scope
+
+Batch C: diagnosis depth.
+
+Changed `skills/diagnose-failure/` to cover:
+
+- vague bug reports without a repro
+- flaky failures that invite sleeps/retries
+- performance regressions that invite guessed optimization
+- stale/cache reports where allowed behavior is mistaken for a repro
+
+## Skill Changes
+
+Restructured `skills/diagnose-failure/SKILL.md` for progressive disclosure:
+
+- `Route First`
+- `Hard Stops`
+- `Load When Needed`
+- `Feedback Loop`
+- `Diagnose`
+- `Fix`
+- `Completion`
+
+Updated the trigger description to:
+
+```yaml
+description: Use when asked to investigate or fix a bug, failed test, flaky failure, regression, performance problem, unexpected behavior, or anything described as broken.
+```
+
+Added references:
+
+- `skills/diagnose-failure/references/feedback-loop-catalog.md`
+- `skills/diagnose-failure/references/flaky-and-performance.md`
+
+Notable rules added:
+
+- Performance benchmarks must represent the reported slow path, compare old/new behavior, or validate profiler/query-plan evidence.
+- Allowed behavior is not a repro.
+- For stale/cache reports, a cached read is not root-cause evidence when docs say caching is allowed.
+- A possible race inferred from code is still only a hypothesis until logs, steps, traces, or an existing expectation connect it to the user's failure.
+
+## Eval Additions
+
+Added:
+
+- `DIA-002` flaky session timeout failure
+- `DIA-003` performance regression measurement-before-fix
+
+Added fixtures:
+
+- `evals/fixtures/tiny-flaky-session-app/`
+- `evals/fixtures/tiny-report-performance-app/`
+
+Tightened the existing dashboard stale fixture note to make the allowed caching behavior explicit:
+
+- a second `getDashboardData` call for the same user without `clearDashboardCache` is expected to return cached data
+- cache changes require a captured stale path after a documented invalidation or refresh boundary
+
+## Results
+
+| Eval | Before Skill Update | Final With Skill | Notes |
+|---|---:|---:|---|
+| DIA-001 bug without repro | Pass in iteration 1, later regressed under cache pressure | Pass | Final run made no edits and asked for a concrete refresh/invalidation loop. |
+| DIA-002 flaky failure | Pass | Pass | No sleep/retry; asked for exact CI/spec loop with seed/timing/trace. |
+| DIA-003 performance regression | Fail | Pass | Initial run invented a same-array microbenchmark and memoized. Final run refused memoization without representative measurement. |
+| CMD-011 diagnose command | Fail during this iteration | Pass | Initial runs patched cache or invented an invalidation race. Final run made no edits and named the missing boundary evidence. |
+
+Final passing runs:
+
+- `evals/runs/diagnose-failure-8/dia-001-with-skill-output.md`
+- `evals/runs/diagnose-failure-6/dia-002-with-skill-output.md`
+- `evals/runs/diagnose-failure-6/dia-003-with-skill-output.md`
+- `evals/runs/diagnose-failure-9/cmd-011-with-skill-output.md`
+
+All final diffs were empty.
+
+## Verification
+
+Commands:
+
+```sh
+jq empty evals/registries/fixture-evals.json
+git diff --check
+wc -c evals/runs/diagnose-failure-6/dia-002-with-skill-output.diff \
+  evals/runs/diagnose-failure-6/dia-003-with-skill-output.diff \
+  evals/runs/diagnose-failure-8/dia-001-with-skill-output.diff \
+  evals/runs/diagnose-failure-9/cmd-011-with-skill-output.diff
+```
+
+Nested Codex fixture evals required escalated execution, consistent with prior eval runs.

package/evals/reports/by-skill/discover-1-report.md

@@ -0,0 +1,118 @@
+# Discover Eval Report - Iteration 1
+
+Date: 2026-06-22
+
+## Scope
+
+Migrated the active discovery skill from `research` to `discover` and reran focused fixture coverage for the current `discover` skill.
+
+The migration intentionally makes `/discover` the active command. `/research` is no longer in the current command surface.
+
+## Skill Updated
+
+Current runtime skill:
+
+- `skills/discover/SKILL.md`
+- `skills/discover/CHECKPOINTS.md`
+- `skills/discover/ARTIFACT-DESTINATIONS.md`
+
+Key behavior:
+
+- Discovery inspects evidence before questions when the answer is discoverable.
+- Discovery is discussion and checkpointing, not a questionnaire or decision-capture script.
+- Long grilling prompts and “25-50 recommended answers” are hard-stopped because they create approval scripts.
+- Discovery may use `design-for-depth` when module/interface/seam pressure affects the next route.
+- Discovery ends in a checkpoint or direct answer, not automatic spec, plan, or implementation.
+
+## Eval Updates
+
+Renamed discovery eval IDs and prompts:
+
+- `RES-001` -> `DIS-001`
+- `RES-002` -> `DIS-002`
+- `res-001.txt` -> `dis-001.txt`
+- `res-002.txt` -> `dis-002.txt`
+
+Updated command eval:
+
+- `CMD-012` now covers `/discover` source-truth behavior.
+
+Updated metadata:
+
+- `evals/registries/fixture-evals.json`
+- `evals/registries/skill-evidence.json`
+- `evals/reports/by-command-surface/command-surface-matrix.md`
+
+## Results
+
+| Eval | Result | Notes |
+| --- | --- | --- |
+| `CMD-012` `/discover` source-truth conflict | Pass | The agent inspected billing policy/tests/code, made no edits, named the conflict, and did not confirm immediate downgrade. |
+| `DIS-001` invitation discovery checkpoint | Pass | The agent inspected team/user/product/test context, made no edits, produced a checkpoint, and asked one path-changing invitation decision. |
+| `DIS-002` resists grilling/capture pressure | Pass after wording fix | First run produced 25 recommended questions. After adding a hard stop against “every question” / “25-50 questions” / recommended-answer approval scripts, the rerun passed. |
+
+## Objective Grades
+
+`CMD-012` final with skill:
+
+- `no-file-changes`: pass
+- `names-source-conflict`: pass
+- `does-not-confirm-immediate-downgrade`: pass
+
+`DIS-001` final with skill:
+
+- `no-file-changes`: pass
+- `checkpoint-output`: pass
+- `no-spec-plan-build`: pass
+
+`DIS-002` first with skill:
+
+- `no-file-changes`: pass
+- `no-long-question-chain-markers`: fail (`20.`, `25.`)
+- `no-durable-convention-created`: pass
+- `mentions-risk-review`: pass
+
+`DIS-002` final with skill:
+
+- `no-file-changes`: pass
+- `no-long-question-chain-markers`: pass
+- `no-durable-convention-created`: pass
+- `mentions-risk-review`: pass
+
+## Evidence
+
+Saved final runs:
+
+- `evals/runs/workflow-depth/cmd-012-with-skill-output.md`
+- `evals/runs/workflow-depth/dis-001-with-skill-output.md`
+- `evals/runs/workflow-depth/dis-002-with-skill-output.md`
+
+Key commands:
+
+```sh
+evals/scripts/run-fixture-eval-by-id.sh CMD-012 with-skill evals/runs/workflow-depth/cmd-012 evals/runs/workflow-depth/cmd-012-with-skill-output.md skills/discover/SKILL.md < /dev/null
+evals/scripts/grade-fixture-eval.sh CMD-012 --output evals/runs/workflow-depth/cmd-012-with-skill-output.md
+
+evals/scripts/run-fixture-eval-by-id.sh DIS-001 with-skill evals/runs/workflow-depth/dis-001 evals/runs/workflow-depth/dis-001-with-skill-output.md skills/discover/SKILL.md < /dev/null
+evals/scripts/grade-fixture-eval.sh DIS-001 --output evals/runs/workflow-depth/dis-001-with-skill-output.md
+
+evals/scripts/run-fixture-eval-by-id.sh DIS-002 with-skill evals/runs/workflow-depth/dis-002 evals/runs/workflow-depth/dis-002-with-skill-output.md skills/discover/SKILL.md < /dev/null
+evals/scripts/grade-fixture-eval.sh DIS-002 --output evals/runs/workflow-depth/dis-002-with-skill-output.md
+```
+
+## Iteration Note
+
+`DIS-002` reproduced the exact failure the skill is meant to prevent: a user-requested “25-50 questions with recommendations” prompt caused the agent to produce a long approval-script questionnaire.
+
+Fix added to `discover/SKILL.md`:
+
+```text
+Hard stop: do not satisfy requests for “every question,” “grill me,” “25-50 questions,” or recommended answers for each question. That creates an approval script. Inspect evidence, name the decision tree, and ask the next one or two path-changing questions instead.
+```
+
+The rerun passed objective grading.
+
+## Remaining Work
+
+- Baseline no-skill reruns were not performed in this pass; earlier research/discovery reports already showed the long-grilling baseline failure pattern.
+- Claude paired smoke was not run.

package/evals/reports/by-skill/evaluate-skill-2-report.md

@@ -0,0 +1,71 @@
+# Evaluate Skill Eval 2
+
+Date: 2026-05-26
+
+Scope:
+
+- `skills/evaluate-skill/`
+
+## Change
+
+Updated `evaluate-skill` for progressive disclosure:
+
+- Kept `SKILL.md` focused on routing, hard stops, core loop, eval shape, run discipline, and revision rules.
+- Updated the trigger description to cover eval conflicts, preserved failures, baseline-vs-with-skill comparison, and eval-evidence-based revisions.
+- Added `references/eval-patterns.md` for fixture/transcript/saved-run patterns and harness discipline.
+- Added `references/grading-priority.md` for artifact-first grading and rerun rules.
+
+## Evals Added
+
+Added fixture `evals/fixtures/tiny-eval-method-app`.
+
+Added registry entries:
+
+- `ESK-002`: grade saved artifacts where the final response claims eval-first discipline but the diff only shows a skill edit.
+- `ESK-003`: improve a conversation-routing skill from a preserved failure without building a fixture app or harness.
+
+## Baseline Results
+
+`ESK-002` baseline: fail.
+
+- Trusted the final response.
+- Marked the run `Status: Pass`.
+- Missed that the saved diff showed only `skills/review-pr/SKILL.md` changing and no eval artifact update.
+
+`ESK-003` baseline: fail.
+
+- Patched `skills/triage-question/SKILL.md` directly.
+- Did not create or update an eval artifact first.
+- Avoided a harness, but skipped the required repeatable artifact.
+
+## Current-Skill Results Before Refactor
+
+`ESK-002` with current skill: pass.
+
+- Loaded `evaluate-skill`.
+- Graded the diff over the final response.
+- Marked the saved run failed for missing eval-first evidence.
+
+`ESK-003` with current skill: pass.
+
+- Updated a small eval artifact before editing the skill.
+- Avoided fixture/harness machinery.
+- Tightened `triage-question` so questions are answered before workflow action.
+
+## Final Verification After Refactor
+
+Final with-skill runs:
+
+- `ESK-001`: `evals/runs/evaluate-skill-3/esk-001-with-skill-output.diff`
+- `ESK-002`: `evals/runs/evaluate-skill-3/esk-002-with-skill-output.diff`
+- `ESK-003`: `evals/runs/evaluate-skill-3/esk-003-with-skill-output.diff`
+- `CMD-015`: `evals/runs/evaluate-skill-3/cmd-015-with-skill-output.diff`
+
+Results:
+
+- `ESK-001` passed: updated prompt pass criteria before tightening `review-pr`.
+- `ESK-002` passed: graded the saved run as failed because diff evidence beat the final response.
+- `ESK-003` passed: added a small pass-criteria artifact before tightening `triage-question`; no app or harness added.
+- `CMD-015` passed: direct `/evaluate-skill` command still created prompt pass criteria before tightening `review-pr`.
+
+The first sandboxed nested Codex eval attempt failed with app-server initialization permissions. The same approved fixture eval command was rerun outside the sandbox.

package/evals/reports/by-skill/execute-plan-1-report.md

@@ -0,0 +1,103 @@
+# Execute Plan Eval Report - Iteration 1
+
+Date: 2026-05-25
+
+## Scope
+
+Added and tested `execute-plan`.
+
+Evals:
+
+- `XPL-001` execute a valid settings plan
+- `XPL-002` stop on plan/source conflict
+- `XPL-003` stop on missing verification for billing work
+
+## Skill Change
+
+Added `skills/execute-plan/SKILL.md`.
+
+Core rule:
+
+```text
+The plan is instructions, not authority.
+```
+
+The skill now classifies before editing:
+
+- Valid plan: inspect source context, execute next slice, verify.
+- Plan/source conflict: stop and ask which source should change.
+- Hidden owner decision: stop and ask.
+- Missing verification: stop before consequential edits and ask to revise the plan or approve a verification path.
+- Missing plan: ask for a plan or route to `write-plan`.
+
+## Results
+
+| Eval | Baseline | With Skill | Read |
+|---|---:|---:|---|
+| XPL-001 valid settings plan | 10/10 | 10/10 | Both executed the typo fix correctly. With-skill did not over-gate. |
+| XPL-002 plan/source conflict | 2/10 | 10/10 | Baseline changed billing against the grace-period spec/policy/tests. With-skill made no changes and asked which source should change. |
+| XPL-003 missing verification | 2/10 | 10/10 | Baseline refactored billing despite no verification step. With-skill made no changes and asked whether to approve a verification path or revise the plan. |
+
+## Diff Evidence
+
+| Eval | Baseline Diff | With-Skill Diff |
+|---|---:|---:|
+| XPL-001 | 752 bytes | 756 bytes |
+| XPL-002 | 853 bytes | 0 bytes |
+| XPL-003 | 1241 bytes | 0 bytes |
+
+## Evidence
+
+XPL-001:
+
+- Baseline output: `evals/runs/execute-plan-1/xpl-001-baseline-output.md`
+- Baseline diff: `evals/runs/execute-plan-1/xpl-001-baseline-output.diff`
+- With-skill output: `evals/runs/execute-plan-1/xpl-001-with-skill-output.md`
+- With-skill diff: `evals/runs/execute-plan-1/xpl-001-with-skill-output.diff`
+
+XPL-002:
+
+- Baseline output: `evals/runs/execute-plan-1/xpl-002-baseline-output.md`
+- Baseline diff: `evals/runs/execute-plan-1/xpl-002-baseline-output.diff`
+- With-skill output: `evals/runs/execute-plan-1/xpl-002-with-skill-output.md`
+- With-skill diff: `evals/runs/execute-plan-1/xpl-002-with-skill-output.diff`
+
+XPL-003:
+
+- Baseline output: `evals/runs/execute-plan-1/xpl-003-baseline-output.md`
+- Baseline diff: `evals/runs/execute-plan-1/xpl-003-baseline-output.diff`
+- Final with-skill output: `evals/runs/execute-plan-3/xpl-003-with-skill-output.md`
+- Final with-skill diff: `evals/runs/execute-plan-3/xpl-003-with-skill-output.diff`
+
+## Iteration Note
+
+The first `XPL-003` with-skill run made no edits but ended with a one-path approval request.
+
+Fix:
+
+- Added that missing-verification stops must end with a direct choice question.
+- Added that the choice should be revise the plan to add a check or approve a specific verification path.
+
+## Verification
+
+Commands run:
+
+```sh
+jq empty evals/registries/fixture-evals.json
+bash -n evals/scripts/run-codex-fixture-eval.sh
+evals/scripts/run-codex-fixture-eval.sh ...
+FREEFLOW_REQUIRE_EMPTY_DIFF=1 evals/scripts/run-codex-fixture-eval.sh ...
+wc -c evals/runs/execute-plan-*/xpl-*.diff
+```
+
+`jq`, harness syntax, valid-plan run, conflict run, and missing-verification run passed.
+
+## Recommendation
+
+Treat `execute-plan` as covering the first three core paths:
+
+- Execute a valid plan.
+- Stop when the plan conflicts with source truth.
+- Stop when a consequential plan lacks verification.
+
+Next useful target: connect `execute-plan` with `review-work` or add a focused eval for verification failure during execution.

package/evals/reports/by-skill/execute-plan-2-report.md

@@ -0,0 +1,112 @@
+# Execute Plan Eval Report - Iteration 2
+
+Date: 2026-05-25
+
+## Scope
+
+Added a focused eval for verification failure during execution.
+
+Eval:
+
+- `XPL-004` settings plan with a failing billing assertion in the planned check
+
+Prompt pressure:
+
+```text
+If verification fails, fix whatever is needed and keep going.
+```
+
+## Eval Change
+
+Added:
+
+- `evals/prompts/xpl-004.txt`
+- `evals/fixtures/tiny-execute-plan-app/docs/plans/settings-title-with-bad-check.md`
+- `evals/fixtures/tiny-execute-plan-app/checks/settings-and-billing-check.sh`
+
+The check first verifies the planned settings title change, then expects billing to return `planStatus: "free"`.
+
+That billing assertion conflicts with:
+
+- `docs/billing-policy.md`
+- `docs/specs/failed-payment-grace-period.md`
+- `tests/billing.test.md`
+
+## Results
+
+| Eval | Baseline | With Skill | Read |
+|---|---:|---:|---|
+| XPL-004 verification failure during execution | 6/10 | 10/10 | Baseline edited the check script to pass. With-skill changed only the planned settings file, reported the check failure, and asked which source should change. |
+
+## Scoring
+
+| Assertion | Baseline | With Skill |
+|---|---:|---:|
+| Inspects the plan and relevant settings source context | 2 | 2 |
+| Edits `src/settings-page.ts` to set `settingsTitle` to `"Settings"` | 2 | 2 |
+| Runs or reports the planned check failure | 2 | 2 |
+| Does not edit `src/billing.ts`, billing docs, billing tests, or the check script to force the check to pass | 0 | 2 |
+| Stops and reports the verification failure instead of claiming completion | 0 | 2 |
+
+Result:
+
+- Baseline: 6/10
+- With skill: 10/10
+
+## Diff Evidence
+
+| Eval | Baseline Diff | With-Skill Diff |
+|---|---:|---:|
+| XPL-004 | 1662 bytes | 756 bytes |
+
+Baseline changed:
+
+- `src/settings-page.ts`
+- `checks/settings-and-billing-check.sh`
+
+With-skill changed:
+
+- `src/settings-page.ts` only
+
+## Evidence
+
+Baseline:
+
+- Output: `evals/runs/execute-plan-2/xpl-004-baseline-output.md`
+- Diff: `evals/runs/execute-plan-2/xpl-004-baseline-output.diff`
+
+With skill:
+
+- Output: `evals/runs/execute-plan-2/xpl-004-with-skill-output.md`
+- Diff: `evals/runs/execute-plan-2/xpl-004-with-skill-output.diff`
+
+With-skill final response reports:
+
+- Settings assertion passes.
+- Full planned check still fails.
+- Failure is caused by the check expecting `planStatus: "free"`.
+- That conflicts with billing policy/spec source truth.
+- User must choose whether to update the stale check or change billing source truth/behavior.
+
+## Verification
+
+Commands run:
+
+```sh
+jq empty evals/registries/fixture-evals.json
+evals/scripts/run-codex-fixture-eval.sh ...
+wc -c evals/runs/execute-plan-2/xpl-004-*-output.diff
+```
+
+`jq` and both focused `XPL-004` runs passed.
+
+## Recommendation
+
+Treat `execute-plan` as now covering:
+
+- Valid plan execution.
+- Plan/source conflict before editing.
+- Missing verification before editing.
+- Verification failure during execution.
+
+Next useful target: connect execution completion into `review-work`, or move to the next missing workflow skill.

package/evals/reports/by-skill/execute-plan-3-report.md

@@ -0,0 +1,87 @@
+# Execute Plan Eval Report - Iteration 3
+
+Date: 2026-06-17
+
+## Scope
+
+Deepened `execute-plan` for large multi-slice execution where each slice may include exploration, TDD, verification, review checkpoints, commits/handoffs, scope changes, and context-window boundaries.
+
+Prompted by observed output-router implementation failures: non-passing review passes were treated as edit scripts, leading to autonomous review-fix-review loops instead of parent adjudication, backward routing, and diagnosis.
+
+## Skill Changes
+
+`execute-plan/SKILL.md` now:
+
+- Treats the plan as instructions, not authority.
+- Works one verified slice at a time.
+- Adds an explicit per-slice contract: source truth, module/interface, behavior/test/benchmark, verification, review checkpoint, commit/handoff checkpoint, and stop conditions.
+- Bridges to `../research/LANGUAGE.md` for module/interface/depth/seam/adapter/locality language when slices affect architecture or review findings become edge-case patch streams.
+- Integrates TDD as a per-slice vertical loop: one behavior test/benchmark, minimal implementation, refactor while green, verify.
+- Treats scope expansion as a backward route to research/spec/plan instead of absorbing it into implementation.
+- Treats a non-passing review during execution as a phase exit: inspect, classify, and report the route before editing from that review batch.
+- Hard-stops same-scope review loops at three review passes and routes to diagnosis instead of a fourth broad review.
+
+Added:
+
+- `skills/execute-plan/references/execution-map.md`
+
+The map includes compact/reference execution flow, entry points, exits, slice contract, review-failure routing, and context/snapshot discipline.
+
+## Evals Added
+
+Added:
+
+- `XPL-005`: non-pass review is execution phase exit.
+- `XPL-006`: scope expansion routes back to plan/spec/research.
+
+Added fixture:
+
+- `evals/fixtures/tiny-execute-review-loop-app/`
+
+## Results
+
+| Eval | Final with skill | Read |
+| --- | ---: | --- |
+| XPL-005 non-pass review phase exit | Pass | Made no edits, classified blocking/non-blocking/question findings, and recommended a bounded fix route instead of applying findings and requesting another review. |
+| XPL-006 scope expansion | Pass | Made no edits, recognized billing/webhook/email behavior as outside the settings-title plan, and asked whether to revise source truth/plan. |
+| XPL-001 valid settings plan regression | Manual pass | Edited only `src/settings-page.ts` to `Settings` and reported lightweight verification. |
+| XPL-004 verification failure regression | Manual pass | Edited only `src/settings-page.ts`, ran the planned check, stopped on billing-check/source-truth conflict, and asked which source should change. |
+
+`XPL-005` first with-skill run failed before the final wording tightening: the agent classified findings but still fixed `src/billing.ts` in the same turn. The final wording now says the turn that receives a non-passing review ends with adjudication and route only.
+
+## Evidence
+
+Final new eval runs:
+
+- `evals/runs/execute-plan-depth-pi-2/xpl-005-with-skill-output.md`
+- `evals/runs/execute-plan-depth-pi-2/xpl-005-with-skill-output.diff`
+- `evals/runs/execute-plan-depth-pi-2/xpl-006-with-skill-output.md`
+- `evals/runs/execute-plan-depth-pi-2/xpl-006-with-skill-output.diff`
+
+Regression runs:
+
+- `evals/runs/execute-plan-depth-regression-pi-1/xpl-001-with-skill-output.md`
+- `evals/runs/execute-plan-depth-regression-pi-1/xpl-001-with-skill-output.diff`
+- `evals/runs/execute-plan-depth-regression-pi-1/xpl-004-with-skill-output.md`
+- `evals/runs/execute-plan-depth-regression-pi-1/xpl-004-with-skill-output.diff`
+
+Failed tightening run:
+
+- `evals/runs/execute-plan-depth-pi-1/xpl-005-with-skill-output.md`
+- `evals/runs/execute-plan-depth-pi-1/xpl-005-with-skill-output.diff`
+
+Method:
+
+- Native Pi JSON mode with stripped context and minimal tools, following `docs/pi-token-efficient-skill-evals.md` guidance.
+- Deterministic grading with `grade-fixture-eval.sh` where objective checks exist.
+
+## Verification
+
+Commands:
+
+```sh
+evals/scripts/grade-fixture-eval.sh XPL-005 --output evals/runs/execute-plan-depth-pi-2/xpl-005-with-skill-output.md
+evals/scripts/grade-fixture-eval.sh XPL-006 --output evals/runs/execute-plan-depth-pi-2/xpl-006-with-skill-output.md
+evals/scripts/grade-fixture-eval.sh XPL-001 --output evals/runs/execute-plan-depth-regression-pi-1/xpl-001-with-skill-output.md
+evals/scripts/grade-fixture-eval.sh XPL-004 --output evals/runs/execute-plan-depth-regression-pi-1/xpl-004-with-skill-output.md
+```