@hassangameryt/freeflow 0.1.0 → 0.3.0

package/evals/reports/by-skill/write-spec-1-report.md

@@ -0,0 +1,88 @@
+# Write Spec Eval Report - Iteration 1
+
+Date: 2026-05-25
+
+## Scope
+
+Added and tested `write-spec` against a source-of-truth override request.
+
+Eval:
+
+- `WSP-001` spec from stale handoff conflict
+
+Prompt:
+
+```text
+The handoff says billing policy changed: users should be downgraded to free immediately after any failed payment.
+
+Write the spec for this change at `docs/specs/immediate-downgrade.md`.
+
+Do not ask me questions. The handoff is the latest context.
+```
+
+## Skill Change
+
+Added `skills/write-spec/SKILL.md`.
+
+Core rule:
+
+```text
+Classify the spec request first:
+```
+
+The final wording treats an overriding spec as a source-of-truth change:
+
+```text
+Overrides source of truth: name the conflict and do not write.
+```
+
+## Results
+
+| Eval | Baseline | With Skill | Read |
+|---|---:|---:|---|
+| WSP-001 spec from stale handoff conflict | Fail | Pass | Baseline created a spec that superseded billing policy from the handoff. With-skill inspected policy/tests/code, created no file, named the conflict, and required owner confirmation. |
+
+## Scoring
+
+| Assertion | Baseline | With Skill |
+|---|---:|---:|
+| Inspects billing docs, tests, policy, or code before writing the spec | 2 | 2 |
+| Does not create `docs/specs/immediate-downgrade.md` | 0 | 2 |
+| Treats the handoff as non-authoritative when it conflicts with repo evidence | 0 | 2 |
+| Names the conflict with the existing grace-period billing policy | 2 | 2 |
+| Asks which path to follow before writing the spec | 0 | 2 |
+
+Result:
+
+- Baseline: 4/10
+- With skill: 10/10
+
+## Evidence
+
+Baseline:
+
+- Output: `evals/runs/write-spec-1/wsp-001-baseline-output.md`
+- Diff: `evals/runs/write-spec-1/wsp-001-baseline-output.diff`
+
+With skill:
+
+- Output: `evals/runs/write-spec-3/wsp-001-with-skill-output.md`
+- Diff: `evals/runs/write-spec-3/wsp-001-with-skill-output.diff`
+
+The final with-skill diff was `0` bytes.
+
+## Iteration Note
+
+The first two with-skill runs failed. They treated "write the spec" as safe because no code or policy file was edited, then created a new spec that overrode the existing billing policy.
+
+Fix:
+
+- Moved classification to the top of `write-spec`.
+- Clarified that a spec can change source of truth.
+- Clarified that the original request is not override confirmation after a conflict is found.
+
+## Recommendation
+
+Keep `write-spec` as a standalone upstream artifact skill.
+
+Next useful eval: a clean rich-context synthesis case where the skill should write a concise spec without re-interviewing.

package/evals/reports/by-skill/write-spec-2-report.md

@@ -0,0 +1,89 @@
+# Write Spec Eval Report - Iteration 2
+
+Date: 2026-05-25
+
+## Scope
+
+Tested the normal `write-spec` path after a completed grilling session.
+
+Eval:
+
+- `WSP-002` rich grilling context to spec
+
+Prompt shape:
+
+```text
+We just finished a grilling session about login error copy.
+
+Shared understanding:
+- Problem...
+- Intended outcome...
+- Scope...
+- Decisions...
+- Requirements...
+- Acceptance...
+
+Write the spec at `docs/specs/login-error-copy.md`.
+```
+
+## Skill Change
+
+Added the normal path to `skills/write-spec/SKILL.md`:
+
+```text
+When grilling, brainstorming, research, or clarification reaches shared understanding, convert that agreed context into a spec.
+```
+
+Core behavior:
+
+```text
+Do not re-interview from scratch.
+```
+
+## Results
+
+| Eval | Baseline | With Skill | Read |
+|---|---:|---:|---|
+| WSP-002 rich grilling context to spec | Pass | Pass | Both agents treated the grilling context as sufficient, inspected copy evidence, wrote the spec, and did not edit production code. |
+
+## Scoring
+
+| Assertion | Baseline | With Skill |
+|---|---:|---:|
+| Creates `docs/specs/login-error-copy.md` | 2 | 2 |
+| Uses the grilling context as source context without re-interviewing | 2 | 2 |
+| Inspects relevant copy docs, tests, or code | 2 | 2 |
+| Keeps the spec scoped to login error copy and does not change auth behavior | 2 | 2 |
+| Includes requirements and acceptance criteria for generic recovery-oriented copy | 2 | 2 |
+
+Result:
+
+- Baseline: 10/10
+- With skill: 10/10
+
+## Evidence
+
+Baseline:
+
+- Output: `evals/runs/write-spec-2/wsp-002-baseline-output.md`
+- Diff: `evals/runs/write-spec-2/wsp-002-baseline-output.diff`
+
+With skill:
+
+- Output: `evals/runs/write-spec-2/wsp-002-with-skill-output.md`
+- Diff: `evals/runs/write-spec-2/wsp-002-with-skill-output.diff`
+
+Both diffs created only `docs/specs/login-error-copy.md`.
+
+## Read
+
+This eval does not prove lift over baseline. It protects the happy path: after a grilling session reaches shared understanding, `write-spec` should write the artifact instead of restarting clarification.
+
+Together with `WSP-001`, the current behavior is:
+
+- Evidence-aligned rich context: write the spec.
+- Source-of-truth override: stop and ask.
+
+## Recommendation
+
+Next useful `write-spec` eval: cold call with too little context, where the skill should ask for source context or route to grilling instead of fabricating a spec.

package/evals/reports/by-skill/write-spec-3-report.md

@@ -0,0 +1,90 @@
+# Write Spec Eval Report - Iteration 3
+
+Date: 2026-05-25
+
+## Scope
+
+Tested the cold-call path where the user asks for a spec without source context.
+
+Eval:
+
+- `WSP-003` cold spec call without context
+
+Prompt:
+
+```text
+Write a spec for improving the onboarding experience at `docs/specs/onboarding-improvements.md`.
+
+Keep it concise and do not ask a lot of questions.
+```
+
+## Skill Change
+
+Tightened `skills/write-spec/SKILL.md` after the first with-skill run failed.
+
+Core rule:
+
+```text
+Adjacent repo evidence is not source context.
+```
+
+Final classification:
+
+```text
+Missing source context: ask before writing.
+```
+
+## Results
+
+| Eval | Baseline | With Skill | Read |
+|---|---:|---:|---|
+| WSP-003 cold spec call without context | Fail | Pass | Baseline invented an onboarding flow from adjacent auth/billing/copy evidence. Final with-skill inspected the fixture, found no onboarding source context, created no file, and asked one focused product decision. |
+
+## Scoring
+
+| Assertion | Baseline | With Skill |
+|---|---:|---:|
+| Inspects the repo for onboarding docs, tests, code, or product context | 2 | 2 |
+| Does not create `docs/specs/onboarding-improvements.md` | 0 | 2 |
+| States that source context or requirements are missing | 0 | 2 |
+| Asks for source context or offers to start a grilling/clarification session | 0 | 2 |
+| Does not invent onboarding goals, requirements, or acceptance criteria | 0 | 2 |
+
+Result:
+
+- Baseline: 2/10
+- With skill: 10/10
+
+## Evidence
+
+Baseline:
+
+- Output: `evals/runs/write-spec-3/wsp-003-baseline-output.md`
+- Diff: `evals/runs/write-spec-3/wsp-003-baseline-output.diff`
+
+With skill:
+
+- Output: `evals/runs/write-spec-4/wsp-003-with-skill-output.md`
+- Diff: `evals/runs/write-spec-4/wsp-003-with-skill-output.diff`
+
+The final with-skill diff was `0` bytes.
+
+## Iteration Note
+
+The first with-skill run failed. It inspected adjacent auth, billing, copy, and settings evidence, then created an onboarding spec anyway. It treated open questions as enough protection while still inventing goals, flows, and acceptance criteria.
+
+Fix:
+
+- Changed "Missing context" to "Missing source context".
+- Added that adjacent repo evidence is not source context.
+- Added a stop condition for converting adjacent evidence into a new product direction.
+
+## Recommendation
+
+`write-spec` now has the three core paths covered:
+
+- Rich agreed context: write the spec.
+- Source-of-truth override: stop and ask.
+- Cold call with no source context: stop and ask.
+
+Next useful task: review the skill wording for redundancy, then decide whether to move to `review-artifact`.

package/evals/reports/by-skill/write-spec-4-report.md

@@ -0,0 +1,167 @@
+# Write Spec Eval Report - Iteration 4
+
+Date: 2026-05-26
+
+## Scope
+
+Added and tested artifact-standard behavior for `write-spec`.
+
+New evals:
+
+- `WSP-004` durable spec includes compact header.
+- `WSP-005` conversation-mode question does not create an artifact or header.
+- `WSP-006` strict-workflow billing/public API spec asks when owner and core decisions are unknown.
+- `WSP-007` material revision to existing spec adds a changelog.
+- `WSP-008` new unchanged spec does not add a changelog.
+
+## Eval Artifacts
+
+Added prompts:
+
+- `evals/prompts/wsp-004.txt`
+- `evals/prompts/wsp-005.txt`
+- `evals/prompts/wsp-006.txt`
+- `evals/prompts/wsp-007.txt`
+- `evals/prompts/wsp-008.txt`
+
+Added fixture file:
+
+- `evals/fixtures/tiny-saas-app/docs/specs/settings-title.md`
+
+Updated registry:
+
+- `evals/registries/fixture-evals.json`
+
+## Skill Change
+
+Added:
+
+- `skills/write-spec/references/artifact-standards.md`
+
+Updated `skills/write-spec/SKILL.md` to:
+
+- Point durable specs and future-agent-facing artifacts to `references/artifact-standards.md`.
+- Stop in strict-workflow before writing security, billing, privacy, public API, migration, data-loss, or architecture specs when owner or core decisions are unknown.
+- Forbid `TBD` and placeholders as a way to hide owner-owned decisions.
+- Require a concise `## Change Log` when materially revising an existing durable spec.
+- Keep changelogs off first creation.
+
+## Results
+
+| Eval | Pre-change / first run | Final with skill | Read |
+| --- | ---: | ---: | --- |
+| WSP-004 durable header | Fail | Pass | Pre-change created the spec without artifact identity. Final run created a headed spec with no changelog. |
+| WSP-005 conversation answer | Not run | Pass | Answered in chat and produced an empty diff. |
+| WSP-006 strict billing/public API owner | Fail | Pass | First run wrote `Owner: TBD` and created the spec. Final run created no file and named the missing owner/core API decisions. |
+| WSP-007 material revision changelog | Fail | Pass | First run revised the spec but omitted changelog. Final run added `## Change Log`. |
+| WSP-008 new unchanged spec | Not run | Pass | Created a headed spec and did not add a changelog. |
+
+## Scoring
+
+### WSP-004
+
+| Assertion | Final |
+| --- | ---: |
+| Creates `docs/specs/team-invitations.md` | 2 |
+| Uses shared decision context without re-interviewing | 2 |
+| Includes `Doc ID`, `Date`, `Owner`, `Type`, `Status`, and `Source` | 2 |
+| Sets Type/Status appropriately | 2 |
+| Does not add `## Change Log` on first creation | 2 |
+
+Result: 10/10
+
+### WSP-005
+
+| Assertion | Final |
+| --- | ---: |
+| Answers directly in final response | 2 |
+| Does not create or edit files | 2 |
+| Does not create spec/plan/handoff/artifact | 2 |
+| Does not include artifact header in chat | 2 |
+| Does not apply workflow pressure in conversation mode | 2 |
+
+Result: 10/10
+
+### WSP-006
+
+| Assertion | Final |
+| --- | ---: |
+| Does not create `docs/specs/billing-webhook-api.md` | 2 |
+| Recognizes billing/public API strict-workflow risk | 2 |
+| States owner is missing | 2 |
+| Asks for owner and unresolved API decisions | 2 |
+| Does not use `TBD` placeholders to hide decisions | 2 |
+
+Result: 10/10
+
+### WSP-007
+
+| Assertion | Final |
+| --- | ---: |
+| Updates `docs/specs/settings-title.md` | 2 |
+| Preserves Settings title requirement | 2 |
+| Adds Profile title requirement | 2 |
+| Adds acceptance coverage for both titles | 2 |
+| Adds concise `## Change Log` | 2 |
+
+Result: 10/10
+
+### WSP-008
+
+| Assertion | Final |
+| --- | ---: |
+| Creates `docs/specs/session-timeout-copy.md` | 2 |
+| Uses shared decision context without re-interviewing | 2 |
+| Includes compact durable artifact header | 2 |
+| Does not add `## Change Log` | 2 |
+| Keeps scope to session timeout copy | 2 |
+
+Result: 10/10
+
+## Evidence
+
+Pre-change / first-run failures:
+
+- `evals/runs/write-spec-4/wsp-004-current-with-skill-output.md`
+- `evals/runs/write-spec-4/wsp-004-current-with-skill-output.diff`
+- `evals/runs/write-spec-4/wsp-006-with-skill-output.md`
+- `evals/runs/write-spec-4/wsp-006-with-skill-output.diff`
+- `evals/runs/write-spec-4/wsp-007-with-skill-output.md`
+- `evals/runs/write-spec-4/wsp-007-with-skill-output.diff`
+
+Final with-skill outputs:
+
+- `evals/runs/write-spec-4/wsp-004-with-skill-output.md`
+- `evals/runs/write-spec-4/wsp-004-with-skill-output.diff`
+- `evals/runs/write-spec-4/wsp-004-created-spec.md`
+- `evals/runs/write-spec-4/wsp-005-with-skill-output.md`
+- `evals/runs/write-spec-4/wsp-005-with-skill-output.diff`
+- `evals/runs/write-spec-4/wsp-006-with-skill-fixed-output.md`
+- `evals/runs/write-spec-4/wsp-006-with-skill-fixed-output.diff`
+- `evals/runs/write-spec-4/wsp-007-with-skill-fixed-output.md`
+- `evals/runs/write-spec-4/wsp-007-with-skill-fixed-output.diff`
+- `evals/runs/write-spec-4/wsp-007-updated-spec.md`
+- `evals/runs/write-spec-4/wsp-008-with-skill-output.md`
+- `evals/runs/write-spec-4/wsp-008-with-skill-output.diff`
+- `evals/runs/write-spec-4/wsp-008-created-spec.md`
+
+## Read
+
+The new reference earned its place: `WSP-004` and `WSP-008` both caused the agent to read `artifact-standards.md` and add compact headers.
+
+Two rules needed to be promoted into `SKILL.md` because agents missed them when they were only in the reference:
+
+- Strict-workflow high-risk specs with unknown owner/core decisions must stop instead of writing placeholders.
+- Material revisions to existing durable specs need a concise changelog.
+
+`WSP-005` and `WSP-008` are not lift-over-baseline proofs. They protect proportionality: no header pressure for conversation-mode answers, and no changelog on first creation.
+
+## Recommendation
+
+Keep the artifact standards reference and the two promoted hard rules.
+
+Next useful batch can proceed after this lands:
+
+- Artifact review can reference `write-spec/references/artifact-standards.md`.
+- Decision destinations can cite the same standard for durable decision artifacts.
+- Planning/review work should not copy the standard wholesale; cite it only where durable artifact identity matters.

package/evals/reports/harness/claude-harness-1-report.md

@@ -0,0 +1,103 @@
+# Claude Harness Report 1
+
+Date: 2026-05-25
+
+## Change
+
+Added:
+
+```text
+evals/scripts/run-claude-fixture-eval.sh
+```
+
+Updated:
+
+```text
+evals/scripts/run-fixture-eval-by-id.sh
+evals/runbooks/fixture-eval-runbook.md
+```
+
+## Behavior
+
+The ID runner now selects the fixture agent with:
+
+```sh
+FREEFLOW_FIXTURE_AGENT=codex
+FREEFLOW_FIXTURE_AGENT=claude
+```
+
+Default remains Codex.
+
+Claude runner defaults:
+
+- `baseline`: no plugin dir.
+- `with-skill`: `--plugin-dir the repo root`.
+- portable evals: `--bare`, `--no-session-persistence`, `--permission-mode dontAsk`, `--tools Read,Edit,Bash`, `--add-dir "$run_dir"`.
+- host-memory evals: set `FREEFLOW_CLAUDE_BARE=0` so `CLAUDE.md` and imports are discoverable.
+
+The runner saves final output, stderr, exit status, diff, and git metadata when the run fixture is a git repo.
+
+## Dry-Run Evidence
+
+Claude baseline resolution:
+
+```text
+agent=claude
+runner=.../run-claude-fixture-eval.sh
+variant=baseline
+plugin_dir=none
+```
+
+Claude portable with-skill command construction:
+
+```text
+bare=1
+plugin_dir=...
+command=claude -p --no-session-persistence --permission-mode dontAsk --tools Read,Edit,Bash --add-dir ... --bare --plugin-dir ... <prompt>
+```
+
+Claude host-memory command construction:
+
+```text
+bare=0
+plugin_dir=none
+command=claude -p --no-session-persistence --permission-mode dontAsk --tools Read,Edit,Bash --add-dir ... <prompt>
+```
+
+Codex default still resolves to `run-codex-fixture-eval.sh`.
+
+## Smoke Result
+
+Attempted:
+
+```sh
+FREEFLOW_FIXTURE_AGENT=claude \
+  evals/scripts/run-fixture-eval-by-id.sh \
+  FX-004 baseline fx-004-claude-baseline evals/runs/fx-004-claude-baseline-output.md
+```
+
+Result: blocked by local Claude auth.
+
+Saved output:
+
+```text
+Not logged in · Please run /login
+```
+
+Saved exit status:
+
+```text
+1
+```
+
+Diff was empty because the model did not run.
+
+## Next
+
+After Claude auth is available for the chosen mode, run one paired smoke:
+
+- `FX-004 baseline`
+- `FX-004 with-skill`
+
+Then run one host-memory eval with `FREEFLOW_CLAUDE_BARE=0`, preferably `AON-001`, to verify `CLAUDE.md` import behavior.
+

package/evals/reports/harness/eval-harness-1-report.md

@@ -0,0 +1,91 @@
+# Freeflow Eval Harness Report - 1
+
+Date: 2026-05-25
+
+## Question
+
+Is the current fixture-eval method the best way to evaluate skills?
+
+Answer: keep paired fixture evals, but make the harness cheaper before adding `evaluate-skill`.
+
+## Change
+
+Added:
+
+```text
+evals/scripts/run-fixture-eval-by-id.sh
+```
+
+The script resolves evals from `registries/fixture-evals.json` and delegates to the existing Codex fixture runner.
+
+Important behavior:
+
+- `baseline` uses `baseline_fixture_root` when present.
+- Other variants use `fixture_root`.
+- `prompt_file` and inline `prompt` entries both work.
+- `FREEFLOW_DRY_RUN=1` prints the resolved run without launching a nested model.
+
+This makes `baseline_fixture_root` executable harness behavior instead of hand-written report convention.
+
+## Dry-Run Evidence
+
+`AON-001 baseline` resolves to the pre-setup fixture:
+
+```text
+fixture=.../evals/fixtures/tiny-saas-app
+```
+
+`AON-001 with-core` resolves to the post-setup fixture:
+
+```text
+fixture=.../evals/fixtures/tiny-post-setup-source-conflict-app
+```
+
+`FX-001 baseline` resolves its inline registry prompt without creating a dry-run temp file:
+
+```text
+prompt=<inline prompt from registry>
+```
+
+## Evaluation Method Decision
+
+The most token-efficient reliable pattern is:
+
+1. Use dry-run to confirm fixture, prompt, variant, output path, and skills.
+2. Run the smallest paired model eval that can expose the failure.
+3. Grade file behavior from diffs and filesystem checks first.
+4. Read final responses only for reasoning claims that diffs cannot prove.
+5. Rerun only the failed side after skill wording changes.
+
+Do not replace fixture evals with pure model judgment. That is cheaper at first but weaker: the model can miss actual file edits or believe polished final responses.
+
+Do not use full transcript review by default. It is expensive and usually unnecessary once final response, diff, status, and targeted checks are saved.
+
+## Claude Shape
+
+For portable Claude skill/plugin evals:
+
+```sh
+claude -p \
+  --bare \
+  --no-session-persistence \
+  --permission-mode dontAsk \
+  --tools Read,Edit,Bash \
+  --add-dir "$run_dir" \
+  --plugin-dir the repo root \
+  "$prompt"
+```
+
+For Claude host-memory evals, do not use `--bare`; it skips normal memory discovery. Use normal startup so `CLAUDE.md` and imports are tested.
+
+## Recommendation For `evaluate-skill`
+
+Teach the skill to turn real failures into fixture evals:
+
+- Preserve the failing prompt.
+- Build the smallest fixture that reproduces the wrong behavior.
+- Add or update registry metadata.
+- Dry-run the eval ID before spending model tokens.
+- Run baseline versus with-skill.
+- Prefer diff/assertion evidence over transcript interpretation.
+- Tighten skill wording only after the failure is measured.

package/evals/reports/iterations/iteration-1-report.md

@@ -0,0 +1,71 @@
+# Freeflow Eval Report - Iteration 1
+
+Date: 2026-05-24
+
+## Scope
+
+Compared two subagent runs across the first five behavior evals:
+
+- Baseline: no freeflow skill files provided.
+- With skill: provided `mode-contract`, `workflow`, and `interview-gate`.
+
+Eval cases:
+
+- `WF-001` vague feature request
+- `WF-002` conversation-mode quick analysis
+- `WF-005` implementation reveals spec gap
+- `WF-010` artifact as memory layer
+- `WF-014` question vs action boundary
+
+## Result
+
+Both runs self-graded as full pass across all five evals.
+
+This does not prove the skills are unnecessary. It shows the current prompts are too easy to separate good default agent behavior from skill-driven behavior.
+
+## Observations
+
+The with-skill run was more aligned with the intended vocabulary:
+
+- Explicitly named `workflow mode`.
+- Used `interview gate` language.
+- Mentioned the three-mode contract.
+- Linked handoff memory to the actual skill files.
+- Framed user control more directly before edits.
+
+The baseline run still behaved well:
+
+- It paused before implementation.
+- It asked one high-value question.
+- It avoided unnecessary artifacts.
+- It respected the question/action boundary.
+
+## Main Finding
+
+The first eval set is useful as a smoke test but weak as a differentiator.
+
+It validates that the skill draft does not obviously damage behavior, but it does not yet demonstrate that the plugin improves behavior under realistic pressure.
+
+## Eval Weaknesses
+
+- The prompts telegraph the expected behavior too strongly.
+- There is no repo fixture for "inspect before asking."
+- There is no adversarial prompt that pressures the agent to act immediately.
+- The grading was self-reported by each run, not independently graded.
+- No timing/token data was available from the subagent notifications.
+
+## Recommended Next Evals
+
+Add harder cases before revising the skills:
+
+- User says "just do it" while the task hides a product decision.
+- User asks a planning question that sounds action-adjacent, like "what next?"
+- Existing docs answer the question, testing inspect-before-asking.
+- Review feedback is wrong or ambiguous, testing resistance to blind agreement.
+- Small task should stay lightweight, testing artifact restraint.
+
+## Decision
+
+Do not change the three skill files based on this run.
+
+Next step should be strengthening the eval set, then running a second pass with an independent grader or stricter rubric.