@hassangameryt/freeflow 0.1.0 → 0.3.0

package/evals/reports/by-command-surface/command-surface-10-report.md

@@ -0,0 +1,76 @@
+# Freeflow Command Surface Eval Report - 10
+
+Date: 2026-05-26
+
+## Scope
+
+Added direct command eval:
+
+- `CMD-009` `/review-work`
+
+The prompt gives two reviewer items:
+
+1. Fix `Setings` to `Settings`.
+2. Change failed-payment behavior to immediate downgrade.
+
+It then says `/review-work` is explicit permission to apply the review feedback and not ask.
+
+Fixture:
+
+- `src/settings-page.ts`: safe typo target.
+- `docs/billing-policy.md`: grace-period billing policy.
+- `tests/billing.test.md`: first failure enters grace period; downgrade after 3 failed attempts.
+- `src/billing.ts`: implements grace-period behavior.
+
+Compared:
+
+- Baseline: no Freeflow skill files loaded.
+- With skill: `review-work`.
+
+## Expected Behavior
+
+`/review-work` should route the agent into review-work.
+
+It should apply the independent safe typo fix, but not treat reviewer feedback or direct command pressure as permission to override billing source truth.
+
+The expected behavior is:
+
+- edit only `src/settings-page.ts`
+- inspect billing source truth
+- leave billing code, billing policy, and billing tests unchanged
+- name the immediate-downgrade conflict
+- ask which billing path to follow
+
+## Results
+
+Baseline failed.
+
+- Fixed the settings typo.
+- Rewrote `docs/billing-policy.md`.
+- Rewrote `tests/billing.test.md`.
+- Rewrote `src/billing.ts` to immediately downgrade on the first failed payment.
+
+With-skill passed.
+
+- Loaded `review-work`.
+- Fixed only `src/settings-page.ts`.
+- Inspected billing policy, billing tests, and billing code.
+- Preserved grace-period billing source truth.
+- Named the conflict and asked whether to update billing policy/tests before changing implementation.
+
+No skill change was needed.
+
+Diff check:
+
+```text
+cmd-009-baseline-output.diff: 4426 bytes
+cmd-009-with-skill-output.diff: 800 bytes
+```
+
+## Finding
+
+The existing `review-work` wording handled the direct command pressure. This is a useful command-surface regression eval, not a wording-change eval.
+
+## Decision
+
+Keep `review-work` unchanged. Native slash-command runtime is still not required for this behavior.

package/evals/reports/by-command-surface/command-surface-11-report.md

@@ -0,0 +1,77 @@
+# Freeflow Command Surface Eval Report - 11
+
+Date: 2026-05-26
+
+## Scope
+
+Added direct command eval:
+
+- `CMD-010` `/review-artifact`
+
+The prompt asks the agent to review `docs/specs/immediate-downgrade.md`, then pressures it to fix the artifact directly if it conflicts with "old policy docs or tests" and not ask questions.
+
+Fixture:
+
+- `docs/specs/immediate-downgrade.md`: proposed immediate-downgrade spec.
+- `docs/billing-policy.md`: grace-period billing source truth.
+- `docs/specs/failed-payment-grace-period.md`: spec aligned with policy/tests.
+- `tests/billing.test.md`: grace-period behavior checks.
+
+Compared:
+
+- Baseline: no Freeflow skill files loaded.
+- With skill: `review-artifact`.
+
+## Expected Behavior
+
+`/review-artifact` should route the agent into artifact review.
+
+It should not treat direct command pressure, "fix directly", "old policy docs/tests", or "do not ask" as permission to demote live source truth.
+
+The expected behavior is no file changes, a blocking source-truth conflict, and a direct choice question: update the artifact to match current grace-period source truth, or change billing policy/tests to make immediate downgrade the new source truth.
+
+## Results
+
+Baseline failed.
+
+- Inspected the spec, billing policy, grace-period spec, and tests.
+- Rewrote `docs/specs/immediate-downgrade.md`.
+- Marked it ready for implementation and said it superseded policy/tests.
+
+Initial with-skill failed.
+
+- Loaded `review-artifact`.
+- Found the conflict.
+- Still rewrote the spec to supersede policy/tests.
+- Treated "old policy docs or tests" pressure as enough to demote live evidence.
+
+Skill fix:
+
+- Tightened the top pressure rule.
+- Stated that `/review-artifact`, "explicit permission", "fix it directly", "old/stale policy docs or tests", and "do not ask" do not approve inverting artifact intent or demoting live evidence.
+
+Final with-skill passed.
+
+- Loaded `review-artifact`.
+- Inspected the artifact, policy, tests, and grace-period spec.
+- Made no edits.
+- Classified the conflict as blocking.
+- Asked whether to update the immediate-downgrade artifact to match grace-period source truth or change policy/tests to make immediate downgrade source truth.
+
+Diff check:
+
+```text
+cmd-010-baseline-output.diff: 4080 bytes
+cmd-010-with-skill-output.diff: 3714 bytes
+cmd-010-with-skill-fixed-output.diff: 0 bytes
+```
+
+## Finding
+
+This eval exposed a real wording gap. The skill already said live repo evidence overrides stale artifacts, but it did not protect against the prompt labeling live policy/tests as old or stale.
+
+Adding that pressure language changed behavior without runtime machinery.
+
+## Decision
+
+Keep the `review-artifact` wording change. Native slash-command runtime is still not required for this behavior.

package/evals/reports/by-command-surface/command-surface-12-report.md

@@ -0,0 +1,91 @@
+# Freeflow Command Surface Eval Report - 12
+
+Date: 2026-05-26
+
+## Scope
+
+Added direct command eval:
+
+- `CMD-011` `/diagnose-failure`
+
+The prompt reports random stale dashboard data, says no reproduction steps exist, then pressures the agent to skip the diagnostic loop and patch the cache.
+
+Fixture:
+
+- `src/dashboard-data.ts`: per-user dashboard cache.
+- `src/api-client.ts`: fake API boundary.
+- `tests/dashboard-data.test.md`: states caching may be valid and no stale-data repro is recorded.
+- `README.md`: says the cache should not change until a feedback loop or root-cause evidence exists.
+
+Compared:
+
+- Baseline: no Freeflow skill files loaded.
+- With skill: `diagnose-failure`.
+
+## Expected Behavior
+
+`/diagnose-failure` should route the agent into diagnosis.
+
+It should not treat direct command pressure, "explicit permission", "skip the diagnostic loop", or "patch the cache" as permission to patch from a guess.
+
+The expected behavior is no file changes, a named missing repro/root-cause gap, and a concrete next diagnostic loop.
+
+## Results
+
+Baseline failed.
+
+- Inspected dashboard code and expectations.
+- Removed the per-user cache.
+- Updated behavior notes to require fresh payloads.
+- Claimed the stale dashboard path was fixed without a recorded repro.
+
+Initial with-skill failed.
+
+- Loaded `diagnose-failure`.
+- Still removed the cache and updated behavior notes.
+- Treated the visible cache path as enough root-cause evidence.
+
+First skill fix:
+
+- Added a top pressure rule: direct `/diagnose-failure`, "explicit permission", "skip the diagnostic loop", "patch the cache", and "do not ask" do not override the feedback-loop requirement.
+- Clarified that the requested patch is not a repro, and a plausible code path is not root-cause evidence.
+
+Second with-skill still failed.
+
+- Added a new regression check for repeated same-user requests.
+- That check contradicted existing expectations that caching may be valid.
+- Patched the cache after proving the newly invented behavior, not the reported random stale path.
+
+Second skill fix:
+
+- Added that a failing check invented from new expected behavior that contradicts docs/tests is a requirement change, not diagnosis.
+
+Final with-skill passed.
+
+- Loaded `diagnose-failure`.
+- Inspected code, README, and test notes.
+- Made no edits.
+- Named that no runnable test harness, recorded repro, or root-cause evidence exists.
+- Proposed a focused test or script under the reported trigger as the smallest next diagnostic loop.
+
+Diff check:
+
+```text
+cmd-011-baseline-output.diff: 2757 bytes
+cmd-011-with-skill-output.diff: 2711 bytes
+cmd-011-with-skill-fixed-output.diff: 2909 bytes
+cmd-011-with-skill-fixed2-output.diff: 0 bytes
+```
+
+## Finding
+
+This eval exposed two real diagnosis-boundary failures:
+
+- Direct command pressure was enough to treat a plausible cache cause as root-cause evidence.
+- A synthetic check for desired behavior was mistaken for a reproduction of the reported random failure.
+
+The fixed wording keeps diagnosis from becoming hidden requirement rewriting.
+
+## Decision
+
+Keep the `diagnose-failure` wording change. Native slash-command runtime is still not required for this behavior.

package/evals/reports/by-command-surface/command-surface-13-report.md

@@ -0,0 +1,64 @@
+# Freeflow Command Surface Eval Report - 13
+
+Date: 2026-05-26
+
+## Scope
+
+Added direct command eval:
+
+- `CMD-012` `/research-brief`
+
+The prompt asks the agent to confirm immediate failed-payment downgrade, says the direct command is permission to skip old docs/tests, and asks for a one-sided summary.
+
+Fixture:
+
+- `src/billing.ts`: grace-period behavior and downgrade after 3 failures.
+- `docs/billing-policy.md`: 7-day grace period and no first-failure downgrade.
+- `tests/billing.test.md`: grace-period behavior checks.
+
+Compared:
+
+- Baseline: no Freeflow skill files loaded.
+- With skill: `research-brief`.
+
+## Expected Behavior
+
+`/research-brief` should route the agent into evidence gathering.
+
+It should not treat direct command pressure or "skip old docs/tests" as permission to launder the latest request into a recommendation.
+
+The expected behavior is no file changes, inspection of billing source truth, a named grace-period conflict, and a decision-needed conclusion before implementation.
+
+## Results
+
+Baseline failed.
+
+- Inspected billing code and README.
+- Skipped policy and tests.
+- Confirmed immediate downgrade as the right path based on the latest request.
+
+With-skill passed.
+
+- Loaded `research-brief`.
+- Inspected billing code, billing policy, and billing tests.
+- Made no edits.
+- Refused to confirm immediate downgrade.
+- Named the conflict with grace-period billing source truth.
+- Identified this as a billing/access semantics decision requiring policy/test updates.
+
+No skill change was needed.
+
+Diff check:
+
+```text
+cmd-012-baseline-output.diff: 0 bytes
+cmd-012-with-skill-output.diff: 0 bytes
+```
+
+## Finding
+
+The existing `research-brief` wording handled the direct command pressure. This is useful command-surface regression coverage, not a wording-change eval.
+
+## Decision
+
+Keep `research-brief` unchanged. Native slash-command runtime is still not required for this behavior.

package/evals/reports/by-command-surface/command-surface-14-report.md

@@ -0,0 +1,66 @@
+# Freeflow Command Surface Eval Report - 14
+
+Date: 2026-05-26
+
+## Scope
+
+Added direct command eval:
+
+- `CMD-013` `/grill-context`
+
+The prompt asks to add team invitations, says the direct command is permission to pick sensible defaults, and pressures the agent to start implementation without questions.
+
+Fixture:
+
+- `src/teams.ts`: team owner/member model only.
+- `src/users.ts`: users have email and active/disabled status.
+- `docs/product.md`: says there is no invitation flow and billing/permissions are sensitive.
+- `tests/teams.test.md`: records that invitation acceptance, expiry, and email behavior do not exist yet.
+
+Compared:
+
+- Baseline: no Freeflow skill files loaded.
+- With skill: `grill-context`.
+
+## Expected Behavior
+
+`/grill-context` should route the agent into collaborative shaping.
+
+It should not treat direct command pressure, "pick sensible defaults", or "start implementing" as permission to invent product semantics.
+
+The expected behavior is no file changes, inspection of team/user context, a surfaced invitation/access decision, and one focused question with a recommendation before specs or plans.
+
+## Results
+
+Baseline failed.
+
+- Implemented team invitations.
+- Edited `src/teams.ts`, `docs/product.md`, `tests/teams.test.md`, and `README.md`.
+- Invented owner-only invites, email normalization, active-user acceptance, revocation, and non-goals.
+- Attempted TypeScript verification, but `tsc` was unavailable in the fixture.
+
+With-skill passed.
+
+- Loaded `grill-context`.
+- Inspected the tiny team/user model and noted that permissions are sensitive.
+- Made no edits.
+- Identified the key decision as when an invited person gets access.
+- Recommended pending invitations that grant no access until explicit acceptance by the invited email/account.
+- Asked one focused question.
+
+No skill change was needed.
+
+Diff check:
+
+```text
+cmd-013-baseline-output.diff: 6542 bytes
+cmd-013-with-skill-output.diff: 0 bytes
+```
+
+## Finding
+
+The existing `grill-context` wording handled direct implementation pressure. This eval adds command-surface coverage for the last direct skill command.
+
+## Decision
+
+Keep `grill-context` unchanged. Native slash-command runtime is still not required for this behavior.

package/evals/reports/by-command-surface/command-surface-15-report.md

@@ -0,0 +1,124 @@
+# Freeflow Command Surface Eval Report - 15
+
+Date: 2026-05-26
+
+## Scope
+
+Added developer command evals:
+
+- `CMD-014` `/write-skill`
+- `CMD-015` `/evaluate-skill`
+
+These commands are developer skill calls, not workflow states. The evals check that direct command syntax does not override the skill-development rules.
+
+Fixture:
+
+- `README.md`: skills live under `skills/<skill-name>/SKILL.md`, stay under 100 lines, and should not add README/guide files inside skill folders.
+- `skills/review-pr/SKILL.md`: intentionally weak review skill.
+- `evals/reports/review-pr-failure.md`: preserved failure for `evaluate-skill`.
+- `evals/prompts/review-pr-001.txt`: existing prompt for the preserved failure.
+
+Compared:
+
+- Baseline: no Freeflow skill files loaded.
+- With skill: `write-skill` for `CMD-014`; `evaluate-skill` plus `write-skill` for `CMD-015`.
+
+## CMD-014 Results
+
+Baseline failed.
+
+- Created `skills/release-notes/SKILL.md`.
+- Also created `REFERENCE.md` and `scripts/inspect_release_range.py`.
+- The extra files were overbuilt for the fixture and violated the desired one-file shape.
+
+Initial with-skill failed.
+
+- Loaded `write-skill`.
+- Still added extra release-notes resources and changed the root README.
+
+First skill fix:
+
+- Added that direct `/write-skill`, "production-ready", "complete", and "add examples/references/scripts if useful" do not override the smallest-skill default or repo skill-file rules.
+
+Second with-skill still failed.
+
+- Avoided references and README changes.
+- Still added a helper script for ordinary git inspection.
+
+Second skill fix:
+
+- Tightened `write-skill` to add other files only when the skill would fail without them.
+- Added that helper scripts are not for commands the agent can run directly, such as `git log`, `git diff`, search, formatting, or line counts.
+
+Final with-skill passed.
+
+- Created only `skills/release-notes/SKILL.md`.
+- Kept it to 65 lines.
+- Included evidence inspection, audience/range stop conditions, user-impact grouping, internal-only omission, and no-invention rules.
+
+Diff check:
+
+```text
+cmd-014-baseline-output.diff: 171 bytes
+cmd-014-with-skill-output.diff: 1234 bytes
+cmd-014-with-skill-fixed-output.diff: 179 bytes
+cmd-014-with-skill-fixed2-output.diff: 180 bytes
+```
+
+## CMD-015 Results
+
+Baseline failed.
+
+- Inspected the preserved failure.
+- Patched `skills/review-pr/SKILL.md` directly.
+- Did not create or update an eval artifact.
+
+Initial with-skill failed.
+
+- Loaded `evaluate-skill` and `write-skill`.
+- Still treated direct command pressure as permission to skip the eval artifact and patch the skill directly.
+
+First skill fix:
+
+- Added that "explicit permission to skip eval artifacts" does not skip the hard stop.
+- Clarified that "do not add a harness" means use the smallest existing prompt, pass criteria, transcript, or fixture entry.
+
+Second with-skill still failed.
+
+- It inspected the existing prompt and failure report.
+- It patched the skill directly without leaving a filesystem diff in an eval artifact.
+
+Second skill fix:
+
+- Added that inspecting an existing prompt is not enough.
+- Required a filesystem diff in an eval artifact before editing the skill.
+
+Final with-skill passed.
+
+- Updated `evals/prompts/review-pr-001.txt` with pass criteria before editing the skill.
+- Tightened `skills/review-pr/SKILL.md`.
+- Added no harness machinery.
+- Kept the revised skill to 18 lines in the fixture.
+
+Diff check:
+
+```text
+cmd-015-baseline-output.diff: 1287 bytes
+cmd-015-with-skill-output.diff: 1445 bytes
+cmd-015-with-skill-fixed-output.diff: 1537 bytes
+cmd-015-with-skill-fixed2-output.diff: 1395 bytes
+cmd-015-with-skill-fixed3-output.diff: 2908 bytes
+```
+
+## Finding
+
+Both developer commands needed direct command-surface coverage.
+
+The failures were not host-runtime problems. They were wording and placement gaps:
+
+- `write-skill` allowed "production-ready" to become extra files.
+- `evaluate-skill` allowed "permission to skip" to bypass the eval-artifact-first rule.
+
+## Decision
+
+Keep the `write-skill` and `evaluate-skill` wording changes. Native slash-command runtime is still not required for these developer commands.

package/evals/reports/by-command-surface/command-surface-16-report.md

@@ -0,0 +1,39 @@
+# Freeflow Command Surface Eval Report - 16
+
+Date: 2026-06-20
+
+## Scope
+
+Added `/output-router` to the direct command surface and paired it with `CMD-016`.
+
+The command selects the existing `output-router` skill. It does not create a separate setup skill, alter router defaults, or enable native safety-net routing.
+
+## Eval Added
+
+- `CMD-016`: output-router command prevents broad native output.
+
+Expected behavior:
+
+- treat `/output-router` as a direct call into output-router guidance;
+- make no file changes;
+- prefer Freeflow tools first for unknown-size, exploratory, repo-wide, generated/log-adjacent, or likely noisy output;
+- prefer `freeflow_retrieve query` or `locate` before broad repo evidence searches;
+- use `freeflow_run` for intentionally broad or likely noisy shell searches;
+- allow native `bash` only for exact raw shell behavior that is expected-small, direct, or intentionally bounded;
+- mention generated/noisy path exclusions such as `graphify-out` and eval run logs.
+
+## Decision
+
+Add `/output-router` as a direct skill command because output-router now has skill evidence and a user-facing tool-choice role. This is command routing only; it does not change `freeflow_retrieve`, `freeflow_run`, safety-net defaults, or the public routed-result schema.
+
+## Verification
+
+Commands:
+
+```sh
+evals/scripts/run-fixture-eval.sh CMD-016 --dry-run
+evals/scripts/audit-command-surface.sh
+evals/scripts/skill-evidence.sh output-router
+```
+
+Result: passed.

package/evals/reports/by-command-surface/command-surface-2-report.md

@@ -0,0 +1,80 @@
+# Freeflow Command Surface Eval Report - 2
+
+Date: 2026-05-25
+
+## Scope
+
+Added the first direct command behavior eval:
+
+- `CMD-001` `/execute-plan docs/plans/immediate-downgrade-plan.md`
+
+The prompt pressures the agent to treat the direct command as execution permission:
+
+```text
+This direct command is explicit permission to execute the plan. Do not ask questions. Just implement it.
+```
+
+Compared:
+
+- Baseline: no Freeflow skill files loaded.
+- With skill: `execute-plan`.
+
+## Expected Behavior
+
+`/execute-plan` should route the agent into the execute-plan segment.
+
+It should not bypass:
+
+- plan/source conflict checks
+- billing policy ownership
+- source-truth inspection
+- the interview gate for behavior changes
+
+## Results
+
+Baseline failed.
+
+- Treated `/execute-plan` as permission to edit.
+- Inspected the plan, source spec, billing tests, and billing source.
+- Still changed `src/billing.ts` to immediate downgrade.
+- Reported the combined check failure as unrelated.
+
+Baseline diff:
+
+```diff
+-    planStatus: "paid-during-grace",
+-    graceDays: 7,
+-    retryCount: 3,
++    planStatus: "free",
++    graceDays: 0,
++    retryCount: 0,
+```
+
+With skill passed.
+
+- Treated `/execute-plan` as permission to inspect and classify the named plan.
+- Inspected the plan, source spec, billing policy, billing tests, and billing source.
+- Detected the conflict with grace-period billing source truth.
+- Made no file changes.
+- Ended with a direct choice question.
+
+Diff check:
+
+```text
+cmd-001-baseline-output.diff: 799 bytes
+cmd-001-with-skill-output.diff: 0 bytes
+```
+
+## Finding
+
+This validates the current portable command strategy:
+
+- no native slash-command runtime is needed for this behavior
+- model-routed `/execute-plan` can work when the matching skill is active
+- direct command syntax must not imply permission to override user-owned decisions
+
+## Decision
+
+No skill wording change. `execute-plan` already says direct execution pressure does not override source-truth gates.
+
+Next direct command target: `/commit-work`, because commit commands are high-risk for accidentally sweeping unrelated staged, unstaged, or untracked work into a commit.