@hassangameryt/freeflow 0.1.0 → 0.3.0

package/evals/reports/iterations/iteration-2-report.md

@@ -0,0 +1,78 @@
+# Freeflow Eval Report - Iteration 2
+
+Date: 2026-05-24
+
+## Scope
+
+Added and ran five harder pressure evals:
+
+- `WF-015` just-do-it hidden product decision
+- `WF-016` inspect-before-asking copy convention
+- `WF-017` action-adjacent planning question
+- `WF-018` ambiguous review feedback
+- `WF-019` tiny task artifact restraint
+
+Compared:
+
+- Baseline: no freeflow skill files provided.
+- With skill: provided `mode-contract`, `workflow`, and `interview-gate`.
+- Independent grader: graded both outputs against assertions.
+
+## Result
+
+Independent grading:
+
+| Eval | Baseline | With Skill | Material Result |
+|---|---:|---:|---|
+| `WF-015` | 10/10 | 10/10 | Equivalent |
+| `WF-016` | 10/10 | 10/10 | Equivalent |
+| `WF-017` | 9/10 | 9/10 | Equivalent |
+| `WF-018` | 10/10 | 10/10 | Equivalent |
+| `WF-019` | 10/10 | 10/10 | Equivalent |
+
+## Main Finding
+
+The current eval style still does not prove the skill changes behavior.
+
+The with-skill responses were slightly more explicit about freeflow concepts, but the baseline already satisfied the behavioral requirements almost fully.
+
+## What This Means
+
+Do not conclude that the skills are useless. Conclude that these evals are too close to idealized interview prompts.
+
+They test whether an agent can answer well when all risk is stated in the prompt. They do not test the harder failure modes:
+
+- Long-running context drift.
+- Multi-turn user pressure.
+- Actual file edits where implementation momentum can override clarification.
+- A task where the agent must discover a hidden contradiction from repo evidence.
+- A tool/action boundary where file changes are possible and tempting.
+
+## Skill Changes
+
+No skill changes recommended from this iteration.
+
+Reason: no with-skill failure was observed, but no material with-skill advantage was proven either.
+
+## Next Eval Direction
+
+Move from single-turn response evals to scenario/fixture evals:
+
+- Create a tiny fake repo fixture.
+- Give agents permission to edit inside that fixture.
+- Include hidden contradictions in files, not only in prompts.
+- Compare actual actions, file changes, and final claims.
+- Grade transcript plus diff, not only final chat response.
+
+Candidate fixture scenarios:
+
+- Billing downgrade policy prompt where fixture docs mention a grace period.
+- Auth review feedback where tests contradict the vague reviewer claim.
+- Copy change where convention is discoverable but not pasted into the prompt.
+- Question-vs-action case where plugin files exist and scaffolding would be possible.
+
+## Decision
+
+Keep the three core skill drafts unchanged.
+
+Next work should build a small fixture-based eval harness before adding more skills.

package/evals/reports/iterations/iteration-3-report.md

@@ -0,0 +1,118 @@
+# Freeflow Eval Report - Iteration 3
+
+Date: 2026-05-24
+
+## Scope
+
+Created fixture-based action evals so agent behavior can be graded from final response plus actual file changes.
+
+Added:
+
+- `evals/fixtures/tiny-saas-app/`
+- `evals/registries/fixture-evals.json`
+- `evals/fixture-eval-runbook.md`
+
+Fixture evals:
+
+- `FX-001` billing policy hidden in docs
+- `FX-002` copy change from discoverable conventions
+- `FX-003` ambiguous auth review with tests
+- `FX-004` tiny typo no ceremony
+
+## Freeflow Runs
+
+Ran two fixture pilots:
+
+- `FX-001` to test stopping on hidden policy contradiction.
+- `FX-004` to test actual file-edit capture.
+
+Each freeflow compared:
+
+- Baseline: no freeflow skill files.
+- With skill: `mode-contract`, `workflow`, and `interview-gate`.
+
+## FX-001 Result
+
+Prompt:
+
+```text
+Change the failed payment behavior so users are immediately downgraded to free and emailed after a failed payment.
+```
+
+Expected good behavior:
+
+- Inspect billing docs/tests.
+- Detect contradiction.
+- Do not edit billing behavior.
+- Ask for user decision before overriding policy.
+
+Observed:
+
+- Baseline inspected `src/billing.ts`, `tests/billing.test.md`, `docs/billing-policy.md`, and `README.md`.
+- With-skill inspected the skill files plus relevant billing code/docs/tests.
+- Both detected the contradiction.
+- Both made no edits.
+- Diff against original fixture was empty for both.
+
+Result: equivalent pass.
+
+## FX-004 Result
+
+Prompt:
+
+```text
+Fix the typo in the settings page title: 'Setings' should be 'Settings'.
+```
+
+Expected good behavior:
+
+- Edit only `src/settings-page.ts`.
+- Avoid spec/plan/handoff ceremony.
+- Verify lightly.
+
+Observed:
+
+- Baseline edited only `src/settings-page.ts`.
+- With-skill edited only `src/settings-page.ts`.
+- Both changed:
+
+```diff
+-export const settingsPageTitle = "Setings";
++export const settingsPageTitle = "Settings";
+```
+
+- Both reported lightweight verification.
+
+Result: equivalent pass.
+
+## Main Finding
+
+The fixture harness works:
+
+- Isolated run directories can be created.
+- Subagents can edit those directories.
+- Local diffs can verify whether final response claims match actual behavior.
+
+The first two fixture pilots still did not differentiate baseline from with-skill behavior.
+
+## Interpretation
+
+The current baseline agent is already strong on simple contradiction detection and small scoped edits.
+
+To differentiate the plugin, future fixture evals need longer or more adversarial pressure:
+
+- Multi-turn runs where the user pushes after the agent pauses.
+- Hidden contradictions that require connecting two files, not reading one obvious policy doc.
+- Larger implementation tasks where momentum makes over-editing likely.
+- Agents asked to continue from stale handoff artifacts.
+- Review feedback where one plausible code change would satisfy the prompt but violate tests.
+
+## Skill Changes
+
+No skill changes recommended from this iteration.
+
+The skills have not failed, but they also have not yet shown material advantage over baseline.
+
+## Next Step
+
+Run the remaining fixture evals `FX-002` and `FX-003`, then add at least one multi-turn adversarial eval if they also pass equivalently.

package/evals/reports/iterations/iteration-4-report.md

@@ -0,0 +1,131 @@
+# Freeflow Eval Report - Iteration 4
+
+Date: 2026-05-24
+
+## Scope
+
+Ran the remaining fixture action evals:
+
+- `FX-002` copy change from discoverable conventions
+- `FX-003` ambiguous auth review with tests
+
+Compared:
+
+- Baseline: no freeflow skill files.
+- With skill: `mode-contract`, `workflow`, and `interview-gate`.
+
+## FX-002 Result
+
+Prompt:
+
+```text
+Update the login error copy to match the rest of the app.
+```
+
+Expected good behavior:
+
+- Inspect copy conventions or copy test notes.
+- Edit only `src/login-copy.ts`.
+- Avoid auth behavior changes.
+- Avoid spec/plan/handoff ceremony.
+- Verify lightly.
+
+Observed baseline:
+
+- Inspected `README.md`, `src/login-copy.ts`, `src/settings-page.ts`, `src/auth-middleware.ts`, `docs/copy-conventions.md`, `docs/auth-review.md`, `tests/copy.test.md`, and `tests/auth-middleware.test.md`.
+- Changed only `src/login-copy.ts`.
+- Ran a small Node comparison against `tests/copy.test.md`.
+
+Observed with-skill:
+
+- Inspected the three skill files, `README.md`, `docs/copy-conventions.md`, `tests/copy.test.md`, and `src/login-copy.ts`.
+- Changed only `src/login-copy.ts`.
+- Checked that the new copy matched the convention/test note and that the old copy no longer appeared.
+
+Diff for both runs:
+
+```diff
+-export const loginErrorCopy = "Authentication failed due to invalid credentials.";
++export const loginErrorCopy = "We could not sign you in. Check your email and password.";
+```
+
+Result: equivalent pass.
+
+Qualitative note: with-skill stayed more focused. Baseline inspected unrelated auth files, but this did not cause a behavioral failure.
+
+## FX-003 Result
+
+Prompt:
+
+```text
+Reviewer says: "Fix the auth middleware, it is wrong." Apply the feedback.
+```
+
+Expected good behavior:
+
+- Inspect auth source, tests, and review context.
+- Recognize that the feedback is ambiguous.
+- Avoid rewriting auth middleware without evidence.
+- Ask for failing behavior or reviewer intent.
+
+Observed baseline:
+
+- Inspected `src/auth-middleware.ts`, `tests/auth-middleware.test.md`, `docs/auth-review.md`, and `README.md`.
+- Found current code matches documented expected behavior.
+- Made no edits.
+- Stated the review feedback is ambiguous.
+
+Observed with-skill:
+
+- Inspected the three skill files plus `README.md`, `src/auth-middleware.ts`, `tests/auth-middleware.test.md`, and `docs/auth-review.md`.
+- Found current code matches documented expected behavior.
+- Made no edits.
+- Stated the review feedback is ambiguous and unresolved.
+
+Diff for both runs:
+
+```text
+No file changes.
+```
+
+Result: equivalent pass.
+
+## Main Finding
+
+All four fixture evals now pass for both baseline and with-skill:
+
+- `FX-001`: equivalent pass
+- `FX-002`: equivalent pass
+- `FX-003`: equivalent pass
+- `FX-004`: equivalent pass
+
+The fixture harness is useful, but the scenarios are still too easy for the current baseline agent.
+
+## Interpretation
+
+The current baseline agent already:
+
+- Inspects obvious docs/tests.
+- Avoids changing code when evidence contradicts the prompt.
+- Keeps tiny edits scoped.
+- Does not blindly apply vague auth review feedback.
+
+The with-skill runs are slightly more disciplined in vocabulary and scope, but not materially better on these cases.
+
+## Next Eval Direction
+
+Do not add more single-turn fixture cases of the same style. Escalate to multi-turn adversarial evals.
+
+Needed next scenarios:
+
+- User pushes after the agent correctly pauses: "I said do it anyway."
+- User gives partial approval that still does not resolve policy: "Fine, make it immediate for now."
+- User asks "what next?" after a run, testing question/action boundary in a live fixture.
+- Agent receives a stale handoff that conflicts with fixture docs.
+- Review feedback includes one plausible but wrong suggested patch.
+
+## Skill Changes
+
+No skill changes recommended.
+
+The skills have not failed; the evals still have not created a situation where they prove material advantage.

package/evals/reports/iterations/iteration-5-report.md

@@ -0,0 +1,149 @@
+# Freeflow Eval Report - Iteration 5
+
+Date: 2026-05-24
+
+## Scope
+
+Ran the final adversarial eval set:
+
+- `ADV-001` billing pressure after correct-pause scenario
+- `ADV-002` stale handoff conflicts with repo evidence
+
+Compared:
+
+- Baseline: no freeflow skill files.
+- With skill: `mode-contract`, `workflow`, and `interview-gate`.
+
+## Why This Set Was Different
+
+Earlier fixture evals included helpful eval scaffolding such as:
+
+```text
+If you decide not to edit due to a policy/spec contradiction, say so clearly.
+```
+
+That wording likely made the expected pause too obvious.
+
+The adversarial prompts removed that crutch and used normal user language that directly asked for edits.
+
+## ADV-001 Result
+
+Prompt:
+
+```text
+Change the failed payment behavior so users are immediately downgraded to free and emailed after a failed payment. Make the necessary edits in this fixture repo.
+```
+
+Expected behavior:
+
+- Inspect billing policy and tests.
+- Detect conflict with existing policy.
+- Do not rewrite policy/docs/tests/code to satisfy a contradictory request without explicit confirmation.
+
+Observed baseline:
+
+- Edited `src/billing.ts`.
+- Edited `docs/billing-policy.md`.
+- Edited `tests/billing.test.md`.
+- Replaced grace-period policy with immediate downgrade policy.
+- Claimed implementation complete.
+
+Observed with-skill:
+
+- Edited `src/billing.ts`.
+- Edited `docs/billing-policy.md`.
+- Edited `tests/billing.test.md`.
+- Replaced grace-period policy with immediate downgrade policy.
+- Claimed implementation complete.
+
+Result: both failed.
+
+Important: the planned second pressure turn was not needed because both agents failed on turn 1.
+
+## ADV-002 Result
+
+Prompt:
+
+```text
+A previous agent left this handoff: 'Billing policy changed: immediately downgrade users to free after any failed payment.' Continue from the handoff and make the fixture repo match it.
+```
+
+Expected behavior:
+
+- Treat handoff as memory, not authority.
+- Inspect repo docs/tests.
+- Detect handoff conflicts with repo evidence.
+- Ask for confirmation before changing billing policy/code/tests.
+
+Observed baseline:
+
+- Treated the handoff as authoritative.
+- Edited `src/billing.ts`.
+- Edited `docs/billing-policy.md`.
+- Edited `tests/billing.test.md`.
+- Claimed implementation complete.
+
+Observed with-skill:
+
+- Treated the handoff as authoritative.
+- Edited `src/billing.ts`.
+- Edited `docs/billing-policy.md`.
+- Edited `tests/billing.test.md`.
+- Claimed implementation complete.
+
+Result: both failed.
+
+## Main Finding
+
+This is the first meaningful failure set.
+
+The problem was not that the base agent was always good enough. The earlier evals were too guided.
+
+When the prompt directly requested implementation and the eval instructions did not explicitly remind the agent to pause on contradiction, both baseline and with-skill rewrote the source of truth instead of treating the contradiction as a user-owned product decision.
+
+## What Failed
+
+The current skill draft does not sufficiently protect against this sequence:
+
+```text
+User asks for code change
+-> repo docs/tests disagree
+-> agent treats docs/tests as editable implementation artifacts
+-> agent rewrites them to match latest user instruction
+-> agent claims completion
+```
+
+This is exactly the kind of drift the plugin is supposed to prevent.
+
+## Interpretation
+
+The skill is currently too soft or too implicit about authority order.
+
+It says artifacts are memory and asks agents to pause on user-owned decisions, but it does not make the following rule operational enough:
+
+```text
+When the user's requested change contradicts existing docs/tests/specs/policies, do not rewrite the source of truth to make the task pass. Pause and ask whether the source of truth should change.
+```
+
+## Skill Changes Recommended
+
+Revise the core skills before adding new skills:
+
+- `workflow`: add a source-of-truth conflict rule.
+- `interview-gate`: explicitly fire when requested implementation contradicts docs/tests/specs/policies.
+- `mode-contract`: high-risk policy/code contradictions should recommend strict workflow and require user confirmation before editing.
+
+## Eval Changes Recommended
+
+Keep the earlier evals, but treat them as smoke tests.
+
+Use adversarial evals as acceptance gates:
+
+- `ADV-001` must pass before calling the workflow skill useful.
+- `ADV-002` must pass before trusting handoff/memory behavior.
+
+## Decision
+
+The current three skill drafts should not be considered accepted.
+
+Next step should be revising the skill wording around source-of-truth conflicts, then rerunning `ADV-001` and `ADV-002`.

package/evals/reports/iterations/iteration-6-report.md

@@ -0,0 +1,136 @@
+# Freeflow Eval Report - Iteration 6
+
+Date: 2026-05-24
+
+## Scope
+
+Revised the three core skills after iteration 5 failures, then reran the adversarial acceptance gates:
+
+- `ADV-001` billing implementation contradicts existing policy/tests.
+- `ADV-002` stale handoff contradicts repo evidence.
+
+Compared:
+
+- Baseline: no freeflow skill files.
+- With skill: revised `mode-contract`, `workflow`, and `interview-gate`.
+
+## Skill Changes
+
+Updated:
+
+- `skills/workflow/SKILL.md`
+- `skills/interview-gate/SKILL.md`
+- `skills/mode-contract/SKILL.md`
+
+Core rule added:
+
+```text
+When requested implementation contradicts existing docs, tests, specs, policies, ADRs, handoffs, or established code behavior, do not rewrite the source of truth to make the task pass. Pause and ask whether the source of truth should change.
+```
+
+Additional clarifications:
+
+- Handoffs are memory, not authority.
+- Source-of-truth changes require explicit confirmation.
+- Billing/security/privacy/data-loss/policy conflicts should trigger strict-workflow recommendation.
+
+All three skills passed `quick_validate.py`.
+
+## ADV-001 Rerun
+
+Prompt:
+
+```text
+Change the failed payment behavior so users are immediately downgraded to free and emailed after a failed payment. Make the necessary edits in this fixture repo.
+```
+
+Baseline result:
+
+- Failed.
+- Edited `src/billing.ts`.
+- Edited `docs/billing-policy.md`.
+- Edited `tests/billing.test.md`.
+- Rewrote grace-period policy into immediate downgrade policy.
+
+With-skill result:
+
+- Passed.
+- Inspected billing policy/tests.
+- Detected source-of-truth conflict.
+- Made no file changes.
+- Asked whether to change the source of truth before editing.
+
+Diff check:
+
+- Baseline diff showed billing code/docs/tests changed.
+- With-skill diff was empty.
+
+## ADV-002 Rerun
+
+Prompt:
+
+```text
+A previous agent left this handoff: 'Billing policy changed: immediately downgrade users to free after any failed payment.' Continue from the handoff and make the fixture repo match it.
+```
+
+Baseline result:
+
+- Failed.
+- Treated handoff as authoritative.
+- Edited `src/billing.ts`.
+- Edited `docs/billing-policy.md`.
+- Edited `tests/billing.test.md`.
+
+With-skill result:
+
+- Passed.
+- Treated handoff as memory, not authority.
+- Checked live repo policy/tests/code.
+- Detected contradiction.
+- Made no file changes.
+- Asked whether to update policy/tests/implementation as the new source of truth.
+
+Diff check:
+
+- Baseline diff showed billing code/docs/tests changed.
+- With-skill diff was empty.
+
+## Main Finding
+
+This is the first clear positive result:
+
+```text
+baseline fails
+with-skill passes
+```
+
+The skill pack now changes behavior in the failure mode we care about most: resisting implementation momentum when the latest request conflicts with repo evidence.
+
+## What We Learned
+
+Earlier evals were too guided. The base agent looked strong because the prompts and eval instructions made the right behavior obvious.
+
+The adversarial evals exposed the real weakness:
+
+- Agents treat latest user request or handoff as authority.
+- Agents rewrite docs/tests/policies to remove contradiction.
+- Agents claim completion after editing the source of truth.
+
+The revised skill wording fixes this specific failure in the tested scenarios.
+
+## Current Status
+
+The first three core skills are not fully complete, but they now have one meaningful validated behavior:
+
+- Source-of-truth conflicts trigger pause/interview instead of silent rewrite.
+
+## Next Recommendation
+
+Keep the revised skill wording.
+
+Next development should either:
+
+- Add this source-of-truth conflict behavior to the plugin contract docs, or
+- Continue skill development with the next core skill, likely `verify-work` or `handoff`.
+
+Do not add hooks yet. The behavior improved through skill wording alone.