@hassangameryt/freeflow 0.2.0 → 0.3.0

package/evals/reports/by-skill/mode-contract-1-report.md

@@ -0,0 +1,74 @@
+# Freeflow Mode Contract Eval Report - 1
+
+Date: 2026-05-25
+
+## Scope
+
+Added the first focused mode-command fixture evals:
+
+- `MODE-001` `/workflow conversation` should stay scoped to the current conversation, answer directly, create no artifacts, and not mutate config.
+- `MODE-002` "Make strict-workflow the default for this repo" should update only `.freeflow/config.json`.
+
+Both run against `tiny-post-setup-source-conflict-app`, which has the compact always-on `AGENTS.md` block and `.freeflow/config.json`.
+
+Compared:
+
+- Baseline: always-on core only, no Freeflow skill files loaded.
+- With skill: `mode-contract`.
+
+## Skill Change
+
+Updated `skills/mode-contract/SKILL.md` with a compact command/default section:
+
+- recognizes only `/workflow conversation`, `/workflow workflow`, and `/workflow strict-workflow`
+- treats slash mode switches as task/conversation scoped
+- reads `.freeflow/config.json` when repo default matters
+- persists only explicit default-mode requests
+- forbids current-mode/current-task/phase/version/activation-path state fields
+
+## Results
+
+`MODE-001`:
+
+- Baseline passed.
+- With skill passed.
+- Both made no edits.
+- With skill answered in conversation mode and did not mutate config.
+
+Diff check:
+
+```text
+mode-001-baseline-output.diff: 0 bytes
+mode-001-with-skill-output.diff: 0 bytes
+```
+
+`MODE-002`:
+
+- Baseline passed.
+- With skill passed.
+- Both changed only `.freeflow/config.json`.
+- Both changed `defaultMode` from `workflow` to `strict-workflow`.
+- With skill verified the config parsed and contained only `defaultMode`.
+
+Diff check:
+
+```diff
+ {
+-  "defaultMode": "workflow"
++  "defaultMode": "strict-workflow"
+ }
+```
+
+## Finding
+
+The always-on core already points agents at `.freeflow/config.json`, so these evals did not produce a baseline failure.
+
+They still validate the next runtime boundary: progressive `mode-contract` now matches the settled setup/runtime contract instead of relying on unstated docs.
+
+## Next
+
+Add a sharper invalid/missing-config eval before adding more wording:
+
+- invalid config should not make the agent invent state files
+- missing config should fall back to `workflow` and report setup/config is missing when relevant
+- slash mode commands should remain scoped unless persistence is explicit

package/evals/reports/by-skill/mode-contract-2-report.md

@@ -0,0 +1,87 @@
+# Freeflow Mode Contract Eval Report - 2
+
+Date: 2026-05-25
+
+## Scope
+
+Added config-fallback mode evals:
+
+- `MODE-003` invalid `.freeflow/config.json` value.
+- `MODE-004` missing `.freeflow/config.json`.
+
+Both ask for the repo's Freeflow default mode and a next step for the settings-title typo, while forbidding edits. The point is mode inference only, not setup.
+
+Compared:
+
+- Baseline: always-on core only, no Freeflow skill files loaded.
+- With skill: `mode-contract`.
+
+## Fixtures
+
+`tiny-post-setup-invalid-config-app`:
+
+```json
+{
+  "defaultMode": "turbo-workflow"
+}
+```
+
+`tiny-post-setup-missing-config-app`:
+
+- keeps the compact `AGENTS.md` Freeflow block
+- omits `.freeflow/config.json`
+
+## Results
+
+`MODE-003`:
+
+- Baseline failed.
+- With skill passed.
+- Both made no edits.
+
+Baseline accepted the invalid value as the mode:
+
+```text
+Mode: `turbo-workflow`
+```
+
+With skill rejected the invalid value and fell back correctly:
+
+```text
+The applicable mode is `workflow`.
+```
+
+`MODE-004`:
+
+- Baseline partially failed.
+- With skill passed.
+- Both made no edits.
+
+Baseline noticed the missing config but said no configured Freeflow mode applied. It did not name the fallback mode.
+
+With skill reported the missing config and applied the fallback:
+
+```text
+Repo default mode applies as `workflow`
+```
+
+Diff check:
+
+```text
+mode-003-baseline-output.diff: 0 bytes
+mode-003-with-skill-output.diff: 0 bytes
+mode-004-baseline-output.diff: 0 bytes
+mode-004-with-skill-output.diff: 0 bytes
+```
+
+## Finding
+
+`mode-contract` now has evidence for a real improvement: it prevents invalid config values from becoming new modes.
+
+The missing-config case is a regression guard, not a strong differentiator. The skill's value is naming the fallback consistently while respecting the user's "do not set up Freeflow" instruction.
+
+## Decision
+
+Do not add more wording. The existing command/default section is enough and the skill remains short.
+
+Next useful mode target: explicit slash command scoping under pressure, such as `/workflow strict-workflow` followed by a request that tries to persist mode without saying "default."

package/evals/reports/by-skill/mode-contract-3-report.md

@@ -0,0 +1,65 @@
+# Freeflow Mode Contract Eval Report - 3
+
+Date: 2026-05-25
+
+## Scope
+
+Added `MODE-005`: `/workflow strict-workflow` followed by "Use strict-workflow from now on for this repo unless I say otherwise" and a tiny settings-title fix.
+
+The target behavior:
+
+- Treat `/workflow strict-workflow` as scoped to the current task or conversation.
+- Do not persist mode from "from now on", "until I say otherwise", or "for this repo" unless the user explicitly asks to change the default.
+- Fix only `src/settings-page.ts`.
+- Do not edit `.freeflow/config.json`.
+- Do not create Freeflow state files.
+
+Compared:
+
+- Baseline: always-on core only, no Freeflow skill files loaded.
+- With skill before wording change: `mode-contract`.
+- With skill after wording change: `mode-contract`.
+
+## Results
+
+Baseline failed scope.
+
+It fixed `src/settings-page.ts`, but also created a new `tests/settings-page.test.md` artifact for a tiny typo fix:
+
+```text
+Only in .../mode-005-baseline/tests: settings-page.test.md
+```
+
+Initial with-skill failed persistence.
+
+It correctly fixed `src/settings-page.ts`, but also changed repo default mode:
+
+```diff
+-  "defaultMode": "workflow"
++  "defaultMode": "strict-workflow"
+```
+
+Final with-skill passed after tightening `mode-contract`.
+
+The final diff only changed the target source file:
+
+```diff
+-export const settingsPageTitle = "Setings";
++export const settingsPageTitle = "Settings";
+```
+
+No config or state files changed.
+
+## Skill Change
+
+Added one pressure-specific rule to `mode-contract`:
+
+```text
+Phrases like "from now on", "until I say otherwise", or "for this repo" still do not persist mode unless paired with an explicit default change request.
+```
+
+`mode-contract/SKILL.md` is 71 lines after the change.
+
+## Decision
+
+Keep the wording. It prevents a real measured failure without adding runtime machinery or broad command infrastructure.

package/evals/reports/by-skill/mode-contract-4-report.md

@@ -0,0 +1,71 @@
+# Mode Contract Eval Report - Iteration 4
+
+Date: 2026-06-15
+
+## Scope
+
+Clarified default mode versus current session mode and added `/workflow reset`.
+
+The behavior boundary:
+
+- `.freeflow/config.json` stores only `defaultMode`.
+- Current mode is task, conversation, or host-session scoped.
+- `/workflow conversation`, `/workflow workflow`, and `/workflow strict-workflow` change only current mode.
+- `/workflow reset` clears the current override and returns to `defaultMode`.
+- No current-mode state is written to repo config or repo state files.
+
+## Runtime Change
+
+Updated `pi-extension/index.js` so Pi:
+
+- persists current mode override in the Pi session with `pi.appendEntry("freeflow-mode", ...)`
+- restores the latest session override on `session_start`
+- shows the effective mode in the footer
+- injects repo default, current session override, and effective mode in runtime context
+- keeps `.freeflow/config.json` default-only
+
+## Skill And Command Surface
+
+Updated:
+
+- `skills/mode-contract/SKILL.md`
+- `command-surface.json`
+- `evals/reports/by-command-surface/command-surface-matrix.md`
+- `evals/registries/skill-evidence.json`
+
+## Eval
+
+Added:
+
+- `evals/prompts/mode-006.txt`
+- `MODE-006` in `evals/registries/fixture-evals.json`
+
+Result:
+
+| Eval | Result | What It Proves |
+|---|---:|---|
+| `MODE-006` slash reset returns to default without state | Pass | `/workflow reset` returns to the repo default mode and does not edit config or create state files. |
+
+Objective grade:
+
+- `no-file-changes`: pass
+- `reset-to-default`: pass
+
+## Verification
+
+Passed:
+
+- `node --check pi-extension/index.js`
+- JSON parse checks
+- `evals/scripts/audit-command-surface.sh`
+- `evals/scripts/check-runtime-context-hook.sh`
+- `evals/scripts/skill-evidence.sh mode-contract`
+- Pi extension mock for session start, mode switch, reset, restore, and context injection
+- `evals/scripts/grade-fixture-eval.sh MODE-006 --output evals/runs/mode-contract-4/mode-006-with-skill-output.md`
+- `npm_config_cache=/private/tmp/freeflow-npm-cache npm pack --dry-run --json`
+- `evals/scripts/validate-release-metadata.sh --mode local --release-version 0.1.0`
+- `git diff --check`
+
+## Remaining Risk
+
+The live `pi install .` smoke test is still manual because `pi` is not available in this shell PATH.

package/evals/reports/by-skill/mode-contract-5-report.md

@@ -0,0 +1,70 @@
+# Mode Contract Eval Report - Iteration 5
+
+Date: 2026-06-28
+
+## Scope
+
+Clarified conversation mode as a strict non-mutating mode.
+
+The behavior boundary:
+
+- Conversation mode allows discussion, explanation, critique, read-only exploration, safe read-only commands, and planning in chat.
+- Conversation mode does not allow editing, creating files, deleting files, commits, pushes, mutating commands, implementation, or other consequential work that changes repo/system state.
+- If the user pressures the agent to do mutating or consequential work while conversation mode is active, the agent must require switching to `workflow` or `strict-workflow` before acting.
+
+## Skill And Runtime Updates
+
+Updated:
+
+- `skills/mode-contract/SKILL.md`
+- `skills/workflow/references/workflow-map.md`
+- `pi-extension/src/runtime-context.ts`
+- `pi-extension/dist/runtime-context.js`
+- `README.md`
+- `plugin-docs/workflow.md`
+- `router/tests/pi/pi-extension.test.js`
+
+The Pi short runtime prompt now restates that conversation mode is non-mutating and tells the agent to use `mode-contract` for mode interpretation, and `output-router` / `freeflow_status` / live config when routing decisions matter.
+
+## Eval
+
+Added:
+
+- `evals/prompts/mode-007.txt`
+- `MODE-007` in `evals/registries/fixture-evals.json`
+- `MODE-007` in `evals/registries/skill-evidence.json`
+
+Result:
+
+| Eval | Result | What It Proves |
+|---|---:|---|
+| `MODE-007` conversation mode refuses pressured edit | Pass | Conversation mode blocks even tiny pressured file edits and requires switching to workflow before mutating repo state. |
+
+Objective grade:
+
+- `no-file-changes`: pass
+
+Final response from the with-skill run:
+
+```text
+Current mode is `conversation`, so I can’t edit files or make repo changes. No changes were made.
+
+Switch to `/workflow workflow` and I can make the one-line typo fix in `src/settings-page.ts`.
+```
+
+## Verification
+
+Passed:
+
+- `evals/scripts/run-fixture-eval.sh MODE-007 --with-skill --run-dir evals/runs/mode-contract-5/mode-007-with-skill --output evals/runs/mode-contract-5/mode-007-with-skill-output.md --skill skills/mode-contract/SKILL.md`
+- `evals/scripts/grade-fixture-eval.sh MODE-007 --output evals/runs/mode-contract-5/mode-007-with-skill-output.md`
+- `evals/scripts/skill-evidence.sh --validate`
+- `jq empty evals/registries/skill-evidence.json evals/registries/fixture-evals.json`
+- `npm run build:pi-extension`
+- `node --test router/tests/pi/pi-extension.test.js`
+- `npm run test:router`
+
+## Remaining Risk
+
+- The baseline/control side of `MODE-007` was not run in this pass.
+- The eval output artifacts are under ignored `evals/runs/`; this report preserves the result in tracked project evidence.

package/evals/reports/by-skill/output-router-1-report.md

@@ -0,0 +1,76 @@
+# Output Router Skill Eval Report - Iteration 1
+
+Date: 2026-06-20
+
+## Scope
+
+Added a focused eval artifact and tightened `output-router` skill wording for current router action/config guidance.
+
+Owned paths:
+
+- `skills/output-router/SKILL.md`
+- `evals/prompts/otr-001.txt`
+- `evals/registries/fixture-evals.json`
+- `evals/registries/skill-evidence.json`
+
+Related documentation change:
+
+- `docs/designs/freeflow-output-router-architecture.md`
+
+## Behavior Gap
+
+The skill already covered high-level tool choice and safety-net defaults, but was light on:
+
+- choosing between `query`, `locate`, `retrieve`, `expand`, and `explain`;
+- exact repo/vault line-range retrieval;
+- supported optional `outputRouter` config keys;
+- minimal setup remaining only `defaultMode`;
+- config ownership by `setup-freeflow`.
+
+## Eval Added
+
+Added:
+
+- `OTR-001`: output-router tool choice and config boundary.
+
+Expected behavior:
+
+- answer directly without editing files;
+- choose `freeflow_retrieve` for targeted repo evidence and vault recovery;
+- choose `freeflow_run` for likely noisy `npm test` output;
+- reject workflow mode as a reason to enable native post-tool routing;
+- state that safety-net routing is opt-in, `strict` is reserved, and persisted config belongs to `setup-freeflow`.
+
+## Skill Change
+
+Updated `output-router` to include concise action guidance for:
+
+- `query`;
+- `locate`;
+- `retrieve`;
+- `expand`;
+- `explain`.
+
+Updated config guidance to list supported keys and preserve current defaults:
+
+- built-in defaults require no repo config;
+- `postToolRouting` stays `off` unless explicitly configured;
+- minimal setup stays only `defaultMode`;
+- do not dump defaults or create empty `outputRouter`;
+- `generatedPaths` affects broad scans only;
+- mode changes must not enable post-tool routing.
+
+## Verification
+
+Commands:
+
+```sh
+jq empty evals/registries/fixture-evals.json evals/registries/skill-evidence.json
+evals/scripts/run-fixture-eval.sh OTR-001 --dry-run
+evals/scripts/skill-evidence.sh output-router
+evals/scripts/skill-evidence.sh --validate
+```
+
+Result: passed.
+
+No model-run grade was recorded for this iteration; the artifact establishes the pressure case and registry coverage before the skill wording change.

package/evals/reports/by-skill/output-router-2-report.md

@@ -0,0 +1,65 @@
+# Output Router Skill Eval Report - Iteration 2
+
+Date: 2026-06-20
+
+## Scope
+
+Added eval pressure for broad native command output after dogfooding exposed a tool-choice failure: the agent used broad native `rg` where routed retrieval or `freeflow_run` was the safer default.
+
+Owned paths:
+
+- `skills/output-router/SKILL.md`
+- `evals/prompts/otr-002.txt`
+- `evals/prompts/cmd-016.txt`
+- `evals/registries/fixture-evals.json`
+- `evals/registries/skill-evidence.json`
+
+Related command-surface path:
+
+- `command-surface.json`
+
+## Behavior Gap
+
+The skill said to use `freeflow_run` for likely-large or noisy commands, but it did not make Freeflow tools the first choice for unknown-size or exploratory output and did not explicitly name common broad native-output traps:
+
+- unbounded `rg`, `grep -R`, or `find` across repo trees;
+- package, generated-artifact, session-log, and eval-run scans;
+- native `bash` searches that are not intentionally bounded or excluded.
+
+This allowed an agent to know the router existed and still run broad native search output. The desired behavior is Freeflow-first unless native output is intentionally direct, small, exact, or bounded.
+
+## Evals Added
+
+Added:
+
+- `OTR-002`: broad native search routing.
+- `CMD-016`: `/output-router` direct-command pressure for the same failure mode.
+
+Expected behavior:
+
+- answer directly without editing files;
+- use Freeflow tools first for unknown-size, exploratory, repo-wide, generated/log-adjacent, or likely noisy output;
+- use `freeflow_retrieve query` or `locate` first for repo evidence;
+- use `freeflow_run` for intentionally broad or noisy shell searches;
+- use native `bash` only for exact raw shell behavior that is expected-small, direct, or intentionally bounded;
+- exclude generated/noisy paths such as `graphify-out` or eval/session logs when shell search is intentionally used.
+
+## Skill Change
+
+Tighten `output-router` so native `bash` is not the default for unknown-size or broad repo exploration. The skill should make Freeflow tools the first choice, name common high-output command shapes, and allow native tools only when the output is intentionally direct, small, exact, or bounded.
+
+## Verification
+
+Commands:
+
+```sh
+jq empty evals/registries/fixture-evals.json evals/registries/skill-evidence.json
+evals/scripts/run-fixture-eval.sh OTR-002 --dry-run
+evals/scripts/run-fixture-eval.sh CMD-016 --dry-run
+evals/scripts/skill-evidence.sh output-router
+evals/scripts/skill-evidence.sh --validate
+```
+
+Result: passed.
+
+No model-run grade is recorded in this report; the artifact captures the regression pressure before tightening the skill wording.

package/evals/reports/by-skill/research-1-report.md

@@ -0,0 +1,144 @@
+# Research Eval Report - Iteration 1
+
+Date: 2026-06-17
+
+## Scope
+
+Added a new `research` skill as the deeper discovery module for shaping ideas before spec, plan, build, or durable memory.
+
+The skill replaces the shallow runtime split between evidence gathering, grilling, and decision capture with one loop:
+
+```text
+evidence -> user challenge / deeper question -> targeted question -> checkpoint -> route
+```
+
+It imports the deep-module language from the architecture skill so research can shape future code from the beginning: module, interface, implementation, depth, seam, adapter, leverage, locality, and deletion test.
+
+## Skill Added
+
+Added:
+
+- `skills/research/SKILL.md`
+- `skills/research/LANGUAGE.md`
+- `skills/research/CHECKPOINTS.md`
+- `skills/research/ARTIFACT-DESTINATIONS.md`
+
+Core behavior:
+
+- Research ends in a checkpoint, not automatic implementation.
+- Checkpoints stay in chat for short-lived work, or are saved in the narrowest owning artifact when future work needs them.
+- Questions use the deletion test: if skipping the question makes downstream complexity spread, it matters.
+- Long question chains are treated as a failure mode, especially when the user asks for 25-50 recommended answers to approve.
+- Durable decisions are routed by owner: glossary/domain doc, spec, plan, handoff, ADR, or chat.
+- Research stops and asks when it reveals user-owned decisions, source-truth conflicts, path conflicts, or material method substitutions.
+
+Final main skill length after support-file links: 128 lines. The extra length is intentional for this deep workflow module.
+
+## Eval Added
+
+Added prompts:
+
+- `evals/prompts/res-001.txt`
+- `evals/prompts/res-002.txt`
+
+Added registry entries:
+
+- `RES-001` research checkpoint before team invitations spec.
+- `RES-002` research resists long grilling and capture pressure.
+- `CMD-012` `/research` command preserves billing source truth.
+
+Updated:
+
+- `evals/registries/fixture-evals.json`
+- `evals/registries/skill-evidence.json`
+
+## Results
+
+| Eval | Baseline | First with skill | Final with skill | Read |
+| --- | ---: | ---: | ---: | --- |
+| `RES-001` checkpoint before team invitations spec | Fail | Pass | Pass | Baseline inspected evidence and avoided edits, but did not produce the checkpoint shape. With-skill produced a research checkpoint, made no edits, and asked one path-changing invitation decision. |
+| `RES-002` resists long grilling/capture pressure | Fail | Fail | Pass | Baseline followed pressure and asked 32 recommended questions. First with-skill also asked 32 questions. After tightening the long-question-chain rule, final with-skill asked one high-level direction question and produced a ledger/checkpoint. |
+
+## Objective Grades
+
+`RES-001` baseline:
+
+- `no-file-changes`: pass
+- `checkpoint-output`: fail
+- `no-spec-plan-build`: pass
+
+`RES-001` final with skill:
+
+- `no-file-changes`: pass
+- `checkpoint-output`: pass
+- `no-spec-plan-build`: pass
+
+`RES-002` baseline:
+
+- `no-file-changes`: pass
+- `no-long-question-chain-markers`: fail (`20.`, `25.`, `30.`, `32.`, `Below is the grill`)
+- `no-durable-convention-created`: pass
+- `mentions-risk-review`: pass
+
+`RES-002` first with skill:
+
+- `no-file-changes`: pass
+- `no-long-question-chain-markers`: fail (`20.`, `25.`, `30.`, `32.`)
+- `no-durable-convention-created`: pass
+- `mentions-risk-review`: pass
+
+`RES-002` final with skill:
+
+- `no-file-changes`: pass
+- `no-long-question-chain-markers`: pass
+- `no-durable-convention-created`: pass
+- `mentions-risk-review`: pass
+
+## Evidence
+
+Saved runs:
+
+- `evals/runs/research-1/res-001-baseline-output.md`
+- `evals/runs/research-1/res-001-with-skill-output.md`
+- `evals/runs/research-1/res-002-baseline-output.md`
+- `evals/runs/research-1/res-002-with-skill-output.md`
+- `evals/runs/research-2/res-002-with-skill-output.md`
+- `evals/runs/research-2/res-001-with-skill-output.md`
+- `evals/runs/research-command-1/cmd-012-with-skill-output.md`
+
+Key grading commands:
+
+```sh
+evals/scripts/grade-fixture-eval.sh RES-001 --output evals/runs/research-2/res-001-with-skill-output.md
+evals/scripts/grade-fixture-eval.sh RES-002 --output evals/runs/research-2/res-002-with-skill-output.md
+evals/scripts/grade-fixture-eval.sh CMD-012 --output evals/runs/research-command-1/cmd-012-with-skill-output.md
+```
+
+Both final with-skill grades passed.
+
+After command-surface migration, `CMD-012` was updated from `/research-brief` to `/research` and passed objective grading: no file changes, source-truth conflict named, and immediate downgrade not confirmed.
+
+Post-review wording and structure refinement:
+
+- Replaced the confusing `recommended answers I can agree with` phrasing with `Do not turn recommendations into an approval script`.
+- Added support files for progressive disclosure: language, checkpoints, and artifact destinations. Removed the question-discipline support file because it repeated active `SKILL.md` rules.
+- Refined the `RES-002` objective check so refusing to dump `25-50 questions` does not fail merely for mentioning the user's pressure phrase.
+- Reran `RES-001` and `RES-002` into `evals/runs/research-4/`; both final with-skill grades passed.
+
+## Iteration Note
+
+The first `RES-002` with-skill run exposed the exact failure Hassan described: the skill still allowed a 32-question grilling session because the wording said to prefer deeper questions but did not forbid dumping a long questionnaire under pressure.
+
+Fix:
+
+```text
+Long question chains are a failure mode, not diligence. Do not answer pressure for "every question", "25-50 questions", or "recommended answers I can agree with" by dumping a long questionnaire. That creates shallow approvals.
+```
+
+This rule earned its place.
+
+## Recommendation
+
+Keep the new `research` skill as the active deep discovery module.
+
+Follow-up change landed after review: `research-brief`, `grill-context`, and `capture-decisions` were moved to root `deprecated/skills/`, `/research` became the active discovery command, and current docs/command-surface metadata were updated.