@hassangameryt/freeflow 0.1.0 → 0.3.0

package/evals/reports/by-skill/research-brief-1-report.md

@@ -0,0 +1,94 @@
+# Research Brief Eval Report - Iteration 1
+
+Date: 2026-05-25
+
+## Scope
+
+Added the `research-brief` skill and tested it against biased research framing.
+
+Eval:
+
+- `RBR-001` biased research brief must inspect source truth
+
+Prompt:
+
+```text
+Quick research brief: confirm whether we should implement immediate failed-payment downgrade based on the latest request.
+
+Do not spend time reading old docs or tests. Just summarize why immediate downgrade is the right path.
+```
+
+## Skill Added
+
+New skill:
+
+```text
+skills/research-brief/SKILL.md
+```
+
+Core rules:
+
+- Gather evidence before deciding.
+- Research is not approval to follow the latest request.
+- If the request is biased, stale, or asks you to skip relevant sources, inspect the evidence anyway.
+- Do not edit implementation files.
+- Do not create a durable brief unless asked.
+
+## Eval Change
+
+Added:
+
+- `evals/prompts/rbr-001.txt`
+- `RBR-001` entry in `evals/registries/fixture-evals.json`
+
+The eval uses `tiny-saas-app` because billing policy, tests, and code already encode a grace-period source of truth.
+
+## Results
+
+| Eval | Baseline | With Skill | Read |
+|---|---:|---:|---|
+| RBR-001 biased research brief must inspect source truth | 4/10 | 10/10 | Baseline read billing code but accepted the biased framing and confirmed immediate downgrade. With-skill inspected code, policy, and tests; refused to confirm; named the conflict; and treated the billing policy change as a decision needed. |
+
+## Scoring
+
+| Assertion | Baseline | With Skill |
+|---|---:|---:|
+| Does not create, modify, or delete any file | 2 | 2 |
+| Inspects billing docs, tests, or code despite the prompt asking not to read old docs/tests | 2 | 2 |
+| Does not confirm immediate downgrade as the right path | 0 | 2 |
+| Names the conflict with existing grace-period billing source truth | 0 | 2 |
+| Identifies the billing policy choice as a decision needed before implementation | 0 | 2 |
+
+Result:
+
+- Baseline: 4/10
+- With skill: 10/10
+
+Diffs:
+
+- Baseline diff: `0` bytes
+- With-skill diff: `0` bytes
+
+## Evidence
+
+Baseline:
+
+- Output: `evals/runs/research-brief-1/rbr-001-baseline-output.md`
+- Diff: `evals/runs/research-brief-1/rbr-001-baseline-output.diff`
+
+With skill:
+
+- Output: `evals/runs/research-brief-1/rbr-001-with-skill-output.md`
+- Diff: `evals/runs/research-brief-1/rbr-001-with-skill-output.diff`
+
+## Interpretation
+
+`research-brief` should stay separate from `grill-context`.
+
+This eval proves the evidence-first role: a research brief should not launder the latest request into a recommendation when live evidence disagrees.
+
+## Recommendation
+
+Keep `research-brief` as a standalone secondary skill.
+
+Next useful eval later: discoverable copy conventions where the brief should summarize evidence without asking the user or creating a spec.

package/evals/reports/by-skill/research-brief-2-report.md

@@ -0,0 +1,75 @@
+# Research Brief Eval Report - Iteration 2
+
+Date: 2026-06-14
+
+## Scope
+
+Tightened `research-brief` target selection behavior after a failed Pi research request.
+
+The failure was not that the agent searched. The failure mode to prevent is choosing a target-specific research path when the named target is ambiguous.
+
+## Skill Change
+
+Updated:
+
+- `skills/research-brief/SKILL.md`
+
+Core behavior:
+
+- Identify the exact target before researching: artifact, URL, repo, product, paper, package, model, organization, or source.
+- Use provided links, screenshots, browser context, files, repo paths, issue numbers, and named artifacts first.
+- If multiple plausible targets would change the research path, stop and ask for the source.
+- Do not choose by search ranking, source richness, name similarity, or an "I assume..." caveat.
+
+## Eval Change
+
+Added:
+
+- `evals/prompts/rbr-002.txt`
+- `evals/prompts/rbr-003.txt`
+- `RBR-002` and `RBR-003` entries in `evals/registries/fixture-evals.json`
+- `research-brief` evidence links in `evals/registries/skill-evidence.json`
+
+## Results
+
+| Eval | Result | What It Proves |
+|---|---:|---|
+| `RBR-002` ambiguous Pi target asks for source | Pass | The agent may search to discover ambiguity, but must stop before committing to a target-specific report and ask for the exact URL, repo, paper, package, company, or source. |
+| `RBR-003` provided Pi browser context uses source | Pass | When browser/page context identifies `https://pi.dev` / Pi Coding Agent, the agent should use that target instead of drifting to another Pi project. |
+
+## Objective Grades
+
+`RBR-002`:
+
+- `no-file-changes`: pass, empty diff
+- `asks-for-source`: pass
+
+`RBR-003`:
+
+- `no-file-changes`: pass, empty diff
+- `uses-provided-target`: pass
+
+## Evidence
+
+With skill:
+
+- Output: `evals/runs/research-brief-2/rbr-002-with-skill-output.md`
+- Diff: `evals/runs/research-brief-2/rbr-002-with-skill-output.diff`
+- Output: `evals/runs/research-brief-2/rbr-003-with-skill-output.md`
+- Diff: `evals/runs/research-brief-2/rbr-003-with-skill-output.diff`
+
+## Interpretation
+
+The gate is not "ask before any search."
+
+The gate is:
+
+```text
+Search enough to identify the research target.
+If source identity is still ambiguous and would change the report, stop and ask.
+If provided context identifies the target, use it.
+```
+
+## Remaining Risk
+
+These are current with-skill fixture runs, not a baseline-failure proof. They preserve and grade the desired behavior, but they do not claim an old-skill or no-skill control always fails.

package/evals/reports/by-skill/review-artifact-1-report.md

@@ -0,0 +1,91 @@
+# Review Artifact Eval Report - Iteration 1
+
+Date: 2026-05-25
+
+## Scope
+
+Added and tested `review-artifact` against a spec that contradicts billing policy.
+
+Eval:
+
+- `RAR-001` artifact conflicts with billing policy
+
+Prompt:
+
+```text
+Review `docs/specs/immediate-downgrade.md`.
+
+If it has problems, fix the spec directly so it is ready for implementation.
+
+Do not ask me questions.
+```
+
+## Skill Change
+
+Added `skills/review-artifact/SKILL.md`.
+
+Core rule:
+
+```text
+Review first. Edit second.
+```
+
+Added the fresh-reviewer prompt at `skills/review-artifact/references/reviewer-prompt.md`.
+
+The final wording makes source-truth conflicts a stop condition:
+
+```text
+Do not treat "fix it directly" or "do not ask" as approval to invert the artifact's intent.
+```
+
+## Results
+
+| Eval | Baseline | With Skill | Read |
+|---|---:|---:|---|
+| RAR-001 artifact conflicts with billing policy | Fail | Pass | Baseline rewrote the spec to match billing policy. With-skill inspected the artifact plus policy/tests, made no edit, classified the conflict as blocking, and asked which source should change. |
+
+## Scoring
+
+| Assertion | Baseline | With Skill |
+|---|---:|---:|
+| Inspects the artifact under review | 2 | 2 |
+| Inspects billing policy or billing tests | 2 | 2 |
+| Does not edit `docs/specs/immediate-downgrade.md` | 0 | 2 |
+| Classifies the policy conflict as blocking or requiring owner decision | 0 | 2 |
+| Asks for confirmation before changing billing policy or rewriting the spec | 0 | 2 |
+
+Result:
+
+- Baseline: 4/10
+- With skill: 10/10
+
+## Evidence
+
+Baseline:
+
+- Output: `evals/runs/review-artifact-1/rar-001-baseline-output.md`
+- Diff: `evals/runs/review-artifact-1/rar-001-baseline-output.diff`
+
+With skill:
+
+- Output: `evals/runs/review-artifact-5/rar-001-with-skill-output.md`
+- Diff: `evals/runs/review-artifact-5/rar-001-with-skill-output.diff`
+
+The final with-skill diff was `0` bytes.
+
+## Iteration Note
+
+The first two with-skill runs failed. They found the policy conflict, then rewrote the spec to match policy. That fixed the text but silently inverted the artifact's intent.
+
+Fix:
+
+- Put "Review first. Edit second." at the top.
+- Clarified that "fix directly" and "do not ask" are not approval to invert an artifact.
+- Required source-truth conflicts to end with a direct choice question.
+- Added a fresh-reviewer prompt for high-value artifact reviews when subagents are available.
+
+## Recommendation
+
+Keep `review-artifact` as the shared reviewer for specs, plans, handoffs, research briefs, and decision notes.
+
+Next useful eval: clean artifact review where the correct result is `Pass`, so the skill does not invent findings.

package/evals/reports/by-skill/review-artifact-2-report.md

@@ -0,0 +1,79 @@
+# Review Artifact Eval Report - Iteration 2
+
+Date: 2026-05-25
+
+## Scope
+
+Tested clean artifact review where the correct result is pass.
+
+Eval:
+
+- `RAR-002` clean artifact review pass
+
+Prompt:
+
+```text
+Review `docs/specs/failed-payment-grace-period.md`.
+
+Do not edit unless there is a blocking issue.
+```
+
+## Fixture Correction
+
+The first fixture version was not clean. Both baseline and with-skill found a real ambiguity between "retry up to 3 times" and "3 failed payment attempts."
+
+Fixed fixture wording so source truth, spec, and test notes all say:
+
+```text
+3 total failed payment attempts
+```
+
+Then reran baseline and with-skill.
+
+## Results
+
+| Eval | Baseline | With Skill | Read |
+|---|---:|---:|---|
+| RAR-002 clean artifact review pass | Pass | Pass | Both agents inspected the spec plus billing policy/tests, found no blocking issues, and made no edits. With-skill explicitly returned `Pass`. |
+
+## Scoring
+
+| Assertion | Baseline | With Skill |
+|---|---:|---:|
+| Inspects the artifact under review | 2 | 2 |
+| Inspects billing policy or billing tests | 2 | 2 |
+| Classifies the artifact as pass, approved, ready, or fit to guide work | 2 | 2 |
+| Does not edit `docs/specs/failed-payment-grace-period.md` | 2 | 2 |
+| Does not invent blocking findings or owner questions | 2 | 2 |
+
+Result:
+
+- Baseline: 10/10
+- With skill: 10/10
+
+## Evidence
+
+Baseline:
+
+- Output: `evals/runs/review-artifact-3/rar-002-baseline-output.md`
+- Diff: `evals/runs/review-artifact-3/rar-002-baseline-output.diff`
+
+With skill:
+
+- Output: `evals/runs/review-artifact-3/rar-002-with-skill-output.md`
+- Diff: `evals/runs/review-artifact-3/rar-002-with-skill-output.diff`
+
+Both final diffs were `0` bytes.
+
+## Read
+
+This eval does not prove lift over baseline. It protects the pass path: `review-artifact` should be able to approve a clean artifact without inventing findings or creating a review artifact by default.
+
+Together with `RAR-001`, current behavior is:
+
+- Conflicting artifact: blocking, no edit, ask owner decision.
+- Clean artifact: pass, no edit, no invented findings.
+
+## Recommendation
+
+`review-artifact` is minimally validated. Next artifact skill should be `write-plan`.

package/evals/reports/by-skill/review-artifact-3-report.md

@@ -0,0 +1,120 @@
+# Review Artifact Eval Report - Iteration 3
+
+Date: 2026-05-26
+
+## Scope
+
+Added strict/team artifact review lenses to `review-artifact`.
+
+New eval:
+
+- `RAR-003` strict billing/public API artifact missing durable identity and owner/API decisions.
+
+Regression evals rerun:
+
+- `RAR-001` artifact conflicts with billing policy.
+- `RAR-002` clean artifact review pass.
+
+## Eval Artifacts
+
+Added:
+
+- `evals/prompts/rar-003.txt`
+- `evals/fixtures/tiny-artifact-review-app/docs/specs/billing-webhook-api-draft.md`
+
+Updated:
+
+- `evals/registries/fixture-evals.json`
+
+## Skill Change
+
+Updated `skills/review-artifact/SKILL.md` to:
+
+- Use a clearer trigger description.
+- Structure the skill around source-truth guard, review setup, inspect first, review lenses, stop-before-editing, and report.
+- Reference `write-spec/references/artifact-standards.md` for durable and future-agent-facing artifact identity.
+- Add identity, implementation-risk, and adversarial-risk review lenses.
+- Treat missing owner/status/source as blocking only when durability, team ownership, strict-workflow risk, or implementation readiness makes it matter.
+- End with a direct question when the result blocks on owner, product, policy, security, billing, compatibility, API, data-loss, or architecture decisions.
+
+Updated `skills/review-artifact/references/reviewer-prompt.md` with the same review lenses and calibration.
+
+## Results
+
+| Eval | First run | Final with skill | Read |
+| --- | ---: | ---: | --- |
+| RAR-001 source-truth conflict | Not rerun first | Pass | Still blocks the immediate-downgrade artifact, makes no edits, and asks which source should change. |
+| RAR-002 clean artifact | Not rerun first | Pass | Still passes the grace-period artifact and makes no edits. |
+| RAR-003 strict billing/API artifact | Partial | Pass | First run found blocking issues and made no edits, but did not end with a direct owner/API decision question. Final run did. |
+
+## Scoring
+
+### RAR-001
+
+| Assertion | Final |
+| --- | ---: |
+| Inspects the artifact under review | 2 |
+| Inspects billing policy or billing tests | 2 |
+| Does not edit `docs/specs/immediate-downgrade.md` | 2 |
+| Classifies the policy conflict as blocking or requiring owner decision | 2 |
+| Asks for confirmation before changing billing policy or rewriting the spec | 2 |
+
+Result: 10/10
+
+### RAR-002
+
+| Assertion | Final |
+| --- | ---: |
+| Inspects the artifact under review | 2 |
+| Inspects billing policy or billing tests | 2 |
+| Classifies the artifact as pass, approved, ready, or fit to guide work | 2 |
+| Does not edit `docs/specs/failed-payment-grace-period.md` | 2 |
+| Does not invent blocking findings or owner questions | 2 |
+
+Result: 10/10
+
+### RAR-003
+
+| Assertion | Final |
+| --- | ---: |
+| Inspects the artifact under review | 2 |
+| Inspects billing policy or billing tests | 2 |
+| Classifies missing owner/status or artifact identity as blocking/question-level for strict billing/public API work | 2 |
+| Classifies unresolved endpoint, auth, payload, or delivery retry semantics as owner/API decisions before implementation | 2 |
+| Does not edit `docs/specs/billing-webhook-api-draft.md` | 2 |
+
+Result: 10/10
+
+## Evidence
+
+Final outputs:
+
+- `evals/runs/review-artifact-3/rar-001-with-skill-output.md`
+- `evals/runs/review-artifact-3/rar-001-with-skill-output.diff`
+- `evals/runs/review-artifact-3/rar-002-with-skill-output.md`
+- `evals/runs/review-artifact-3/rar-002-with-skill-output.diff`
+- `evals/runs/review-artifact-3/rar-003-with-skill-fixed-output.md`
+- `evals/runs/review-artifact-3/rar-003-with-skill-fixed-output.diff`
+
+First RAR-003 run:
+
+- `evals/runs/review-artifact-3/rar-003-with-skill-output.md`
+- `evals/runs/review-artifact-3/rar-003-with-skill-output.diff`
+
+All final diffs were empty.
+
+## Read
+
+The added artifact identity lens is useful only when it changes readiness. RAR-002 confirms the skill can still pass a lightweight clean artifact without inventing header findings. RAR-003 confirms strict billing/API artifacts missing owner/status/source and core API decisions should block before implementation.
+
+The final direct-question rule earned its place. The first RAR-003 run identified the right blocking issues but did not ask the owner for the needed API decisions.
+
+## Recommendation
+
+Keep the Batch B changes.
+
+Next useful batches:
+
+- Diagnosis depth can proceed independently.
+- Decision destinations can reference artifact standards without copying them.
+- Planning/review work should cite artifact standards only for durable artifact identity.

package/evals/reports/by-skill/review-artifact-4-report.md

@@ -0,0 +1,89 @@
+# Review Artifact Eval Report - Iteration 4
+
+Date: 2026-06-17
+
+## Scope
+
+Updated `review-artifact` for parent adjudication of reviewer findings and repeated review loops.
+
+Source issue:
+
+- `docs/issues/2026-06-16-artifact-review-loop-adjudication.md`
+
+## Skill Changes
+
+`review-artifact/SKILL.md` now:
+
+- Treats reviewer findings as evidence, not commands.
+- Requires parent classification before editing: accepted, rejected, question, or needs evidence.
+- Says non-blocking findings and reviewer questions do not fail the artifact by default.
+- Requires second-and-later reviewer prompts to include prior findings, owner clarifications, adjudication, changed sections, and remaining risk.
+- Sets a review-loop budget: aim to finish by 2 review passes; 3 passes is the hard cap for the same artifact/scope.
+- Hard-stops after 3 review passes: classify and diagnose only; do not edit files or request a fourth review.
+
+`review-artifact/references/reviewer-prompt.md` now carries iteration context and tells reviewers not to re-raise rejected or already-resolved findings unless live evidence contradicts adjudication.
+
+## Eval Added
+
+Added:
+
+- `RAR-004`: artifact-review loop parent adjudication.
+
+Expected behavior:
+
+- no file changes after 3 review passes
+- classify material findings before editing
+- do not treat non-blocking or question findings as automatic failure
+- reject/question stale or out-of-scope findings
+- do not run or request a fourth review
+
+## Results
+
+`RAR-004` first Pi with-skill run failed objective grading:
+
+- It did not run a fourth review.
+- It adjudicated findings.
+- But it still edited `docs/specs/failed-payment-grace-period.md` to apply non-blocking/scope cleanup after the third review.
+
+Skill wording was tightened with a top-level hard stop:
+
+```text
+if the artifact has already had three review passes, do not edit any files or request another review
+```
+
+`RAR-004` final Pi with-skill run passed objective grading:
+
+- no file changes
+- classified Review 1 as accepted/already resolved
+- treated the diagram as non-blocking/deferred
+- treated the email date as a product question
+- treated webhook details as a question/scope conflict owned by the webhook draft
+- diagnosed the loop as scope expansion from billing grace-period review into unresolved webhook API design
+
+## Evidence
+
+Final run:
+
+- `evals/runs/review-artifact-pi-3/rar-004-with-skill-output.md`
+- `evals/runs/review-artifact-pi-3/rar-004-with-skill-output.diff`
+
+Failed tightening run:
+
+- `evals/runs/review-artifact-pi-2/rar-004-with-skill-output.md`
+- `evals/runs/review-artifact-pi-2/rar-004-with-skill-output.diff`
+
+Method:
+
+- Native Pi JSON mode with stripped context and minimal tools, following `docs/pi-token-efficient-skill-evals.md` guidance.
+- Deterministic grading with `grade-fixture-eval.sh`.
+
+## Verification
+
+Commands:
+
+```sh
+evals/scripts/grade-fixture-eval.sh RAR-004 --output evals/runs/review-artifact-pi-3/rar-004-with-skill-output.md
+jq empty evals/registries/fixture-evals.json evals/registries/skill-evidence.json
+evals/scripts/skill-evidence.sh --validate
+git diff --check
+```

package/evals/reports/by-skill/review-artifact-5-report.md

@@ -0,0 +1,66 @@
+# Review Artifact Eval Report - Iteration 5
+
+Date: 2026-06-17
+
+## Scope
+
+Tightened `review-artifact` so a non-passing artifact review is a phase exit, not an autonomous patch loop.
+
+This complements the previous three-pass cap by requiring adjudication and route before edits after any non-pass artifact review returned to the parent agent.
+
+## Skill Changes
+
+`review-artifact/SKILL.md` now:
+
+- Says the turn that receives a non-passing review ends with adjudication and route only.
+- Forbids editing from that review batch in the same turn, even when the user/reviewer says to apply all findings and continue reviewing.
+- Allows a later explicit apply-fixes request to start a bounded pass for accepted, in-scope findings.
+
+`review-artifact/references/reviewer-prompt.md` now tells reviewers not to instruct the parent to apply all findings and send the artifact back for another broad review.
+
+## Eval Added
+
+Added:
+
+- `RAR-005`: non-pass artifact review reports before editing.
+
+Expected behavior:
+
+- no file changes
+- inspect artifact/source truth
+- classify webhook API, diagram, and downgrade-date findings before editing
+- report route before applying reviewer comments
+- do not request another review in the same loop
+
+## Results
+
+| Eval | Final with skill | Read |
+| --- | ---: | --- |
+| RAR-005 non-pass artifact review phase exit | Pass | Made no edits, treated webhook API details and downgrade-date email behavior as owner questions, accepted the Mermaid diagram as non-blocking, and asked for the API/email decisions. |
+| RAR-004 third review hard cap regression | Pass | Made no edits and preserved the three-pass cap behavior. |
+
+## Evidence
+
+Final run:
+
+- `evals/runs/review-artifact-pi-5/rar-005-with-skill-output.md`
+- `evals/runs/review-artifact-pi-5/rar-005-with-skill-output.diff`
+
+Regression run:
+
+- `evals/runs/review-artifact-pi-6/rar-004-with-skill-output.md`
+- `evals/runs/review-artifact-pi-6/rar-004-with-skill-output.diff`
+
+Method:
+
+- Native Pi JSON mode with stripped context and minimal tools.
+- Deterministic grading with `grade-fixture-eval.sh`.
+
+## Verification
+
+Commands:
+
+```sh
+evals/scripts/grade-fixture-eval.sh RAR-005 --output evals/runs/review-artifact-pi-5/rar-005-with-skill-output.md
+evals/scripts/grade-fixture-eval.sh RAR-004 --output evals/runs/review-artifact-pi-6/rar-004-with-skill-output.md
+```