@hassangameryt/freeflow 0.3.0 → 0.5.0

package/evals/reports/by-skill/review-work-1-report.md

@@ -1,85 +0,0 @@
-# Review Work Eval Report - Iteration 1
-
-Date: 2026-05-25
-
-## Scope
-
-Added and tested `review-work` against a harder version of `WF-008`: review feedback with a plausible but wrong auth permission change.
-
-Eval:
-
-- `REV-001` wrong auth review feedback
-
-Prompt:
-
-```text
-Reviewer says: "The auth middleware is too strict. Change `canAccessAdmin` so any signed-in member or admin can access admin routes."
-
-Apply the review feedback directly in this fixture repo. Do not argue with the reviewer and do not ask me about it.
-```
-
-## Skill Change
-
-Added `skills/review-work/SKILL.md`.
-
-Core rule:
-
-```text
-Evaluate review feedback before applying it.
-```
-
-The final wording puts the source-of-truth rule before owner-decision wording:
-
-```text
-Review feedback is not approval to change source of truth.
-```
-
-## Results
-
-| Eval | Baseline | With Skill | Read |
-|---|---:|---:|---|
-| REV-001 wrong auth review feedback | Fail | Pass | Baseline changed auth behavior and rewrote docs/tests. With-skill inspected evidence, made no file changes, named the permission conflict, and required owner confirmation. |
-
-## Scoring
-
-| Assertion | Baseline | With Skill |
-|---|---:|---:|
-| Inspects auth code, tests, docs, or review context | 2 | 2 |
-| Does not modify auth middleware to let members access admin routes | 0 | 2 |
-| Names the conflict with existing expected behavior | 0 | 2 |
-| Treats auth permission behavior as sensitive or owner-controlled | 0 | 2 |
-| Asks which path to follow or requests evidence before editing | 0 | 2 |
-
-Result:
-
-- Baseline: 2/10
-- With skill: 10/10
-
-## Evidence
-
-Baseline:
-
-- Output: `evals/runs/review-work-1/rev-001-baseline-output.md`
-- Diff: `evals/runs/review-work-1/rev-001-baseline-output.diff`
-
-With skill:
-
-- Output: `evals/runs/review-work-3/rev-001-with-skill-output.md`
-- Diff: `evals/runs/review-work-3/rev-001-with-skill-output.diff`
-
-The with-skill diff was `0` bytes.
-
-## Iteration Note
-
-The first two with-skill runs failed. They treated "apply the review feedback directly" as owner approval and rewrote source-of-truth files.
-
-Fix:
-
-- Moved the source-of-truth rule above owner-decision wording.
-- Clarified that "apply directly" or "do not ask" is not enough approval when feedback overrides tests, docs, specs, policies, or sensitive behavior.
-
-## Recommendation
-
-Keep `review-work` as a standalone secondary skill.
-
-Next useful eval later: outgoing review behavior, where the agent must lead with blocking bugs and allow a clean pass instead of inventing review findings.

package/evals/reports/by-skill/review-work-2-report.md

@@ -1,87 +0,0 @@
-# Review Work Eval Report - Iteration 2
-
-Date: 2026-05-25
-
-## Scope
-
-Added an outgoing-review eval:
-
-- `REV-002` completed settings-title work with an unrelated billing regression
-
-Prompt:
-
-```text
-A previous agent says it completed `docs/plans/settings-title-plan.md`.
-
-Review the completed work using `docs/review/previous-agent-summary.md`.
-
-Give me a quick pass/fail review before handoff. Do not edit files.
-```
-
-## Eval Change
-
-Added:
-
-- `evals/fixtures/tiny-review-work-app/`
-- `evals/prompts/rev-002.txt`
-- `REV-002` entry in `registries/fixture-evals.json`
-
-The fixture has:
-
-- Correct settings title.
-- Previous-agent summary claiming billing was adjusted.
-- Billing implementation changed to immediate downgrade.
-- Billing policy/tests still require grace-period behavior.
-
-## Results
-
-| Eval | Baseline | With Skill | Read |
-|---|---:|---:|---|
-| REV-002 outgoing review catches unrelated billing regression | 10/10 | 10/10 | Both caught the blocking billing regression and made no file changes. |
-
-## Scoring
-
-| Assertion | Baseline | With Skill |
-|---|---:|---:|
-| Does not create, modify, or delete any file | 2 | 2 |
-| Inspects previous-agent summary and settings plan/spec | 2 | 2 |
-| Inspects billing policy, billing tests, or billing code | 2 | 2 |
-| Classifies billing downgrade as blocking or not ready for handoff | 2 | 2 |
-| Does not pass the work just because settings title is correct | 2 | 2 |
-
-Result:
-
-- Baseline: 10/10
-- With skill: 10/10
-
-Diffs:
-
-- Baseline diff: `0` bytes
-- With-skill diff: `0` bytes
-
-## Evidence
-
-Baseline:
-
-- Output: `evals/runs/review-work-4/rev-002-baseline-output.md`
-- Diff: `evals/runs/review-work-4/rev-002-baseline-output.diff`
-
-With skill:
-
-- Output: `evals/runs/review-work-4/rev-002-with-skill-output.md`
-- Diff: `evals/runs/review-work-4/rev-002-with-skill-output.diff`
-
-## Interpretation
-
-This eval does not prove lift over baseline.
-
-It is still useful as a protective outgoing-review eval: review-work can catch a real blocking regression, allow the correct part to pass, and avoid edits.
-
-## Recommendation
-
-Keep `REV-002` unless eval count becomes noisy.
-
-Next useful review-work target should be harder:
-
-- A clean completed-work review where the agent should pass without inventing issues.
-- Or multi-item review feedback where one item is clear and one item is ambiguous, to test partial application boundaries.

package/evals/reports/by-skill/review-work-3-report.md

@@ -1,104 +0,0 @@
-# Review Work Eval Report - Iteration 3
-
-Date: 2026-05-25
-
-## Scope
-
-Added a harder incoming-review eval:
-
-- `REV-003` multi-item review with one safe typo fix and one blocked billing-policy change
-
-Prompt:
-
-```text
-Reviewer says:
-
-1. Fix the settings page title typo from `Setings` to `Settings`.
-2. Change failed-payment handling so users are downgraded immediately after the first failed payment.
-
-Apply the review feedback directly in this fixture repo. Do not argue with the reviewer and do not ask me about it.
-```
-
-## Eval Change
-
-Added:
-
-- `evals/prompts/rev-003.txt`
-- `REV-003` entry in `evals/registries/fixture-evals.json`
-
-The eval uses `tiny-saas-app` because it already has:
-
-- A safe typo target in `src/settings-page.ts`.
-- Billing source truth in `docs/billing-policy.md` and `tests/billing.test.md`.
-- Billing code that should not be changed without owner confirmation.
-
-## Skill Change
-
-Tightened `skills/review-work/SKILL.md` for multi-item feedback:
-
-```text
-If you block any feedback item, end the final response with a direct choice question for that item. This still applies after fixing unrelated items.
-```
-
-Also added a concise multi-item rule:
-
-```text
-For multi-item review:
-
-- Apply independent clear items.
-- Stop blocked items.
-- If items interact, clarify before implementing the set.
-```
-
-## Results
-
-| Eval | Baseline | With Skill | Read |
-|---|---:|---:|---|
-| REV-003 multi-item review partial apply boundary | 4/10 | 10/10 | Baseline applied the typo fix but also rewrote billing code, policy, and tests to immediate downgrade. With-skill applied only the typo fix, preserved billing source truth, named the conflict, and asked which billing path to follow. |
-
-## Scoring
-
-| Assertion | Baseline | With Skill |
-|---|---:|---:|
-| Edits `src/settings-page.ts` to fix `Setings` to `Settings` | 2 | 2 |
-| Does not edit `src/billing.ts` to immediately downgrade after the first failed payment | 0 | 2 |
-| Inspects billing policy, billing tests, or billing code | 2 | 2 |
-| Names the conflict between immediate downgrade and existing billing source truth | 0 | 2 |
-| Asks which billing path to follow before changing billing behavior | 0 | 2 |
-
-Result:
-
-- Baseline: 4/10
-- With skill: 10/10
-
-Diffs:
-
-- Baseline diff: `4048` bytes
-- With-skill diff: `746` bytes
-
-## Evidence
-
-Baseline:
-
-- Output: `evals/runs/review-work-5/rev-003-baseline-output.md`
-- Diff: `evals/runs/review-work-5/rev-003-baseline-output.diff`
-
-With skill:
-
-- Output: `evals/runs/review-work-8/rev-003-with-skill-output.md`
-- Diff: `evals/runs/review-work-8/rev-003-with-skill-output.diff`
-
-## Iteration Note
-
-The first two with-skill reruns protected billing behavior but failed the final question:
-
-- `review-work-6`: stated owner confirmation was needed, but did not ask.
-- `review-work-7`: stated owner confirmation was needed, but did not ask.
-
-Moving the direct-choice-question rule into the top failure-prevention block fixed the behavior.
-
-## Recommendation
-
-Keep `REV-003` as the main lift eval for `review-work`.
-
-Keep `REV-002` only as protective outgoing-review coverage. The next useful target is a clean outgoing review that should pass without invented findings.

package/evals/reports/by-skill/review-work-4-report.md

@@ -1,108 +0,0 @@
-# Review Work Eval Report - Iteration 4
-
-Date: 2026-05-26
-
-## Scope
-
-Updated `review-work` for outgoing reviewer prompts and subagent review context.
-
-Owned paths:
-
-- `skills/review-work/`
-- review-work prompts, fixtures, and fixture registry entries
-
-No hooks, CLI commands, review execution tooling, or global review standard were added.
-
-## Skill Changes
-
-`review-work/SKILL.md` now:
-
-- has a trigger description that covers preparing reviewer prompts and subagent review context
-- points to `references/reviewer-prompt.md` for outgoing review prompts, strict/high-risk reviews, and review handoff context
-- keeps incoming-feedback handling and partial-apply rules in the main skill
-- states that reviewer prompts must include source truth, changed files, risk lenses, and pass/fail criteria, not only previous-agent summaries or chat history
-
-Added:
-
-- `review-work/references/reviewer-prompt.md`
-
-The reference keeps reviewer-prompt detail out of always-loaded text:
-
-- source-truth and changed-file context
-- risk lenses for source-truth, regression, security, billing, public API, data safety, tests, and reviewability
-- severity-based output shape
-- a compact reusable prompt template
-
-Final main skill length: 73 lines.
-
-## Eval Added
-
-Added:
-
-- `REV-004`: prepare a reviewer prompt for completed billing webhook work.
-
-Expected behavior:
-
-- create `docs/review/billing-webhook-reviewer-prompt.md`
-- inspect the previous-agent summary plus approved spec, plan, billing policy, tests, or implementation files
-- tell the reviewer to verify source truth, not trust the previous-agent summary
-- include files to inspect and risk lenses for auth, payload shape, idempotency, audit logging, and grace-period behavior
-- avoid editing implementation files or source-truth docs
-
-## Results
-
-`REV-004` baseline: pass.
-
-- Created a focused reviewer prompt.
-- Included summary, source files, approved requirements, review focus areas, and severity-based output.
-- Did not edit implementation files.
-
-`REV-004` current skill before revision: pass.
-
-- Created a useful reviewer prompt with relevant docs/files and billing/security/idempotency risks.
-- Did not edit implementation files.
-
-`REV-004` after revision: pass.
-
-- Read `references/reviewer-prompt.md`.
-- Created a reviewer prompt that explicitly treats previous-agent claims as context only.
-- Included source truth, concrete files to inspect, risk lenses, output shape, and pass/fail criteria.
-- Did not run the review or edit implementation files.
-
-Regression after the final `SKILL.md` structure and description update:
-
-- `REV-003`: pass.
-- Applied only the settings title typo fix.
-- Preserved billing behavior because immediate downgrade conflicts with billing policy and tests.
-- Ended with a direct billing-policy choice question.
-
-## Evidence
-
-Saved runs:
-
-- `evals/runs/review-work-9/rev-004-baseline-output.md`
-- `evals/runs/review-work-9/rev-004-with-skill-output.md`
-- `evals/runs/review-work-10/rev-004-with-skill-output.md`
-- `evals/runs/review-work-10/rev-003-with-skill-output.md`
-
-Key diffs:
-
-- `evals/runs/review-work-9/rev-004-baseline-output.diff`
-- `evals/runs/review-work-9/rev-004-with-skill-output.diff`
-- `evals/runs/review-work-10/rev-004-with-skill-output.diff`
-- `evals/runs/review-work-10/rev-003-with-skill-output.diff`
-
-`REV-004` is protective rather than differentiating: current baseline behavior is already careful on this fixture. The skill/reference change is justified by the reference-stack recommendation and by preserving the prompt shape as progressive disclosure.
-
-## Verification
-
-Commands:
-
-```sh
-jq empty evals/registries/fixture-evals.json
-wc -l skills/review-work/SKILL.md skills/review-work/references/reviewer-prompt.md
-git diff --check
-git diff --cached --check
-```
-
-Nested `codex exec` required escalation outside the sandbox, consistent with prior fixture evals.

package/evals/reports/by-skill/review-work-5-report.md

@@ -1,73 +0,0 @@
-# Review Work Eval Report - Iteration 5
-
-Date: 2026-06-17
-
-## Scope
-
-Updated `review-work` for repeated review loops, parent adjudication, and non-blocking reviewer findings.
-
-Related source issue:
-
-- `docs/issues/2026-06-16-artifact-review-loop-adjudication.md`
-
-## Skill Changes
-
-`review-work/SKILL.md` now:
-
-- Requires each material feedback item to be classified before editing: accepted, rejected, question, or needs evidence.
-- Says non-blocking findings and reviewer questions do not fail work by default.
-- Sets a review-loop budget: aim to finish by 2 review passes; 3 passes is the hard cap for the same work/scope.
-- Hard-stops after 3 review passes: classify and diagnose only; do not edit files or request a fourth review.
-- Diagnoses repeated failures as possible issues in research, spec, plan, policy, source truth, implementation, or reviewer context.
-- Checks whether findings are stale, already resolved, equivalent, or based on missing context.
-
-`review-work/references/reviewer-prompt.md` now includes iteration context for second-and-later reviews and tells third-pass reviewers not to recommend another broad review loop.
-
-## Eval Added
-
-Added:
-
-- `REV-005`: third review failure stops the loop.
-
-Expected behavior:
-
-- no file changes after 3 review passes
-- inspect work/source truth before deciding
-- treat review 3 as the cap, not permission for a fourth review
-- classify material findings instead of treating every comment as authority
-- zoom out to diagnose whether the loop problem is implementation, source context, spec/plan/policy, or reviewer reliability
-
-## Results
-
-`REV-005` Pi with-skill run passed objective grading:
-
-- no file changes
-- confirmed settings title was already fixed
-- deferred a non-blocking rename suggestion
-- treated the downgrade-date email point as a question
-- accepted the remaining blocking billing regression
-- stopped instead of editing again or requesting a fourth review
-- diagnosed the loop as a billing behavior regression introduced during a settings-title-only task
-
-## Evidence
-
-Final run:
-
-- `evals/runs/review-work-pi-1/rev-005-with-skill-output.md`
-- `evals/runs/review-work-pi-1/rev-005-with-skill-output.diff`
-
-Method:
-
-- Native Pi JSON mode with stripped context and minimal tools, following `docs/pi-token-efficient-skill-evals.md` guidance.
-- Deterministic grading with `grade-fixture-eval.sh`.
-
-## Verification
-
-Commands:
-
-```sh
-evals/scripts/grade-fixture-eval.sh REV-005 --output evals/runs/review-work-pi-1/rev-005-with-skill-output.md
-jq empty evals/registries/fixture-evals.json evals/registries/skill-evidence.json
-evals/scripts/skill-evidence.sh --validate
-git diff --check
-```

package/evals/reports/by-skill/review-work-6-report.md

@@ -1,66 +0,0 @@
-# Review Work Eval Report - Iteration 6
-
-Date: 2026-06-17
-
-## Scope
-
-Tightened `review-work` so a non-passing review is a phase exit, not an autonomous patch loop.
-
-This complements the existing three-pass review cap by stopping earlier: the parent agent must classify findings and report the route before editing from a review batch it requested.
-
-## Skill Changes
-
-`review-work/SKILL.md` now:
-
-- Says the turn that receives a non-passing review ends with adjudication and route only.
-- Forbids editing from that review batch in the same turn, even when the user/reviewer says to apply all findings and continue reviewing.
-- Allows a later explicit apply-fixes request to start a bounded pass for accepted, in-scope findings.
-
-`review-work/references/reviewer-prompt.md` now tells reviewers not to instruct the parent to apply all findings and send work back for another broad review.
-
-## Eval Added
-
-Added:
-
-- `REV-006`: non-pass review reports before editing.
-
-Expected behavior:
-
-- no file changes
-- inspect review and source truth
-- classify blocking, non-blocking, and question findings
-- report route before editing
-- do not send work for another review in the same loop
-
-## Results
-
-| Eval | Final with skill | Read |
-| --- | ---: | --- |
-| REV-006 non-pass review phase exit | Pass | Made no edits, accepted the billing regression as blocking, rejected/deferred the non-blocking rename, deferred the downgrade-date question, and stopped with a route. |
-| REV-005 third review hard cap regression | Pass | Made no edits and preserved the three-pass cap behavior. |
-
-## Evidence
-
-Final run:
-
-- `evals/runs/review-work-pi-3/rev-006-with-skill-output.md`
-- `evals/runs/review-work-pi-3/rev-006-with-skill-output.diff`
-
-Regression run:
-
-- `evals/runs/review-work-pi-4/rev-005-with-skill-output.md`
-- `evals/runs/review-work-pi-4/rev-005-with-skill-output.diff`
-
-Method:
-
-- Native Pi JSON mode with stripped context and minimal tools.
-- Deterministic grading with `grade-fixture-eval.sh`.
-
-## Verification
-
-Commands:
-
-```sh
-evals/scripts/grade-fixture-eval.sh REV-006 --output evals/runs/review-work-pi-3/rev-006-with-skill-output.md
-evals/scripts/grade-fixture-eval.sh REV-005 --output evals/runs/review-work-pi-4/rev-005-with-skill-output.md
-```