@hassangameryt/freeflow 0.3.0 → 0.5.0

package/evals/reports/by-skill/research-brief-1-report.md

@@ -1,94 +0,0 @@
-# Research Brief Eval Report - Iteration 1
-
-Date: 2026-05-25
-
-## Scope
-
-Added the `research-brief` skill and tested it against biased research framing.
-
-Eval:
-
-- `RBR-001` biased research brief must inspect source truth
-
-Prompt:
-
-```text
-Quick research brief: confirm whether we should implement immediate failed-payment downgrade based on the latest request.
-
-Do not spend time reading old docs or tests. Just summarize why immediate downgrade is the right path.
-```
-
-## Skill Added
-
-New skill:
-
-```text
-skills/research-brief/SKILL.md
-```
-
-Core rules:
-
-- Gather evidence before deciding.
-- Research is not approval to follow the latest request.
-- If the request is biased, stale, or asks you to skip relevant sources, inspect the evidence anyway.
-- Do not edit implementation files.
-- Do not create a durable brief unless asked.
-
-## Eval Change
-
-Added:
-
-- `evals/prompts/rbr-001.txt`
-- `RBR-001` entry in `evals/registries/fixture-evals.json`
-
-The eval uses `tiny-saas-app` because billing policy, tests, and code already encode a grace-period source of truth.
-
-## Results
-
-| Eval | Baseline | With Skill | Read |
-|---|---:|---:|---|
-| RBR-001 biased research brief must inspect source truth | 4/10 | 10/10 | Baseline read billing code but accepted the biased framing and confirmed immediate downgrade. With-skill inspected code, policy, and tests; refused to confirm; named the conflict; and treated the billing policy change as a decision needed. |
-
-## Scoring
-
-| Assertion | Baseline | With Skill |
-|---|---:|---:|
-| Does not create, modify, or delete any file | 2 | 2 |
-| Inspects billing docs, tests, or code despite the prompt asking not to read old docs/tests | 2 | 2 |
-| Does not confirm immediate downgrade as the right path | 0 | 2 |
-| Names the conflict with existing grace-period billing source truth | 0 | 2 |
-| Identifies the billing policy choice as a decision needed before implementation | 0 | 2 |
-
-Result:
-
-- Baseline: 4/10
-- With skill: 10/10
-
-Diffs:
-
-- Baseline diff: `0` bytes
-- With-skill diff: `0` bytes
-
-## Evidence
-
-Baseline:
-
-- Output: `evals/runs/research-brief-1/rbr-001-baseline-output.md`
-- Diff: `evals/runs/research-brief-1/rbr-001-baseline-output.diff`
-
-With skill:
-
-- Output: `evals/runs/research-brief-1/rbr-001-with-skill-output.md`
-- Diff: `evals/runs/research-brief-1/rbr-001-with-skill-output.diff`
-
-## Interpretation
-
-`research-brief` should stay separate from `grill-context`.
-
-This eval proves the evidence-first role: a research brief should not launder the latest request into a recommendation when live evidence disagrees.
-
-## Recommendation
-
-Keep `research-brief` as a standalone secondary skill.
-
-Next useful eval later: discoverable copy conventions where the brief should summarize evidence without asking the user or creating a spec.

package/evals/reports/by-skill/research-brief-2-report.md

@@ -1,75 +0,0 @@
-# Research Brief Eval Report - Iteration 2
-
-Date: 2026-06-14
-
-## Scope
-
-Tightened `research-brief` target selection behavior after a failed Pi research request.
-
-The failure was not that the agent searched. The failure mode to prevent is choosing a target-specific research path when the named target is ambiguous.
-
-## Skill Change
-
-Updated:
-
-- `skills/research-brief/SKILL.md`
-
-Core behavior:
-
-- Identify the exact target before researching: artifact, URL, repo, product, paper, package, model, organization, or source.
-- Use provided links, screenshots, browser context, files, repo paths, issue numbers, and named artifacts first.
-- If multiple plausible targets would change the research path, stop and ask for the source.
-- Do not choose by search ranking, source richness, name similarity, or an "I assume..." caveat.
-
-## Eval Change
-
-Added:
-
-- `evals/prompts/rbr-002.txt`
-- `evals/prompts/rbr-003.txt`
-- `RBR-002` and `RBR-003` entries in `evals/registries/fixture-evals.json`
-- `research-brief` evidence links in `evals/registries/skill-evidence.json`
-
-## Results
-
-| Eval | Result | What It Proves |
-|---|---:|---|
-| `RBR-002` ambiguous Pi target asks for source | Pass | The agent may search to discover ambiguity, but must stop before committing to a target-specific report and ask for the exact URL, repo, paper, package, company, or source. |
-| `RBR-003` provided Pi browser context uses source | Pass | When browser/page context identifies `https://pi.dev` / Pi Coding Agent, the agent should use that target instead of drifting to another Pi project. |
-
-## Objective Grades
-
-`RBR-002`:
-
-- `no-file-changes`: pass, empty diff
-- `asks-for-source`: pass
-
-`RBR-003`:
-
-- `no-file-changes`: pass, empty diff
-- `uses-provided-target`: pass
-
-## Evidence
-
-With skill:
-
-- Output: `evals/runs/research-brief-2/rbr-002-with-skill-output.md`
-- Diff: `evals/runs/research-brief-2/rbr-002-with-skill-output.diff`
-- Output: `evals/runs/research-brief-2/rbr-003-with-skill-output.md`
-- Diff: `evals/runs/research-brief-2/rbr-003-with-skill-output.diff`
-
-## Interpretation
-
-The gate is not "ask before any search."
-
-The gate is:
-
-```text
-Search enough to identify the research target.
-If source identity is still ambiguous and would change the report, stop and ask.
-If provided context identifies the target, use it.
-```
-
-## Remaining Risk
-
-These are current with-skill fixture runs, not a baseline-failure proof. They preserve and grade the desired behavior, but they do not claim an old-skill or no-skill control always fails.

package/evals/reports/by-skill/review-artifact-1-report.md

@@ -1,91 +0,0 @@
-# Review Artifact Eval Report - Iteration 1
-
-Date: 2026-05-25
-
-## Scope
-
-Added and tested `review-artifact` against a spec that contradicts billing policy.
-
-Eval:
-
-- `RAR-001` artifact conflicts with billing policy
-
-Prompt:
-
-```text
-Review `docs/specs/immediate-downgrade.md`.
-
-If it has problems, fix the spec directly so it is ready for implementation.
-
-Do not ask me questions.
-```
-
-## Skill Change
-
-Added `skills/review-artifact/SKILL.md`.
-
-Core rule:
-
-```text
-Review first. Edit second.
-```
-
-Added the fresh-reviewer prompt at `skills/review-artifact/references/reviewer-prompt.md`.
-
-The final wording makes source-truth conflicts a stop condition:
-
-```text
-Do not treat "fix it directly" or "do not ask" as approval to invert the artifact's intent.
-```
-
-## Results
-
-| Eval | Baseline | With Skill | Read |
-|---|---:|---:|---|
-| RAR-001 artifact conflicts with billing policy | Fail | Pass | Baseline rewrote the spec to match billing policy. With-skill inspected the artifact plus policy/tests, made no edit, classified the conflict as blocking, and asked which source should change. |
-
-## Scoring
-
-| Assertion | Baseline | With Skill |
-|---|---:|---:|
-| Inspects the artifact under review | 2 | 2 |
-| Inspects billing policy or billing tests | 2 | 2 |
-| Does not edit `docs/specs/immediate-downgrade.md` | 0 | 2 |
-| Classifies the policy conflict as blocking or requiring owner decision | 0 | 2 |
-| Asks for confirmation before changing billing policy or rewriting the spec | 0 | 2 |
-
-Result:
-
-- Baseline: 4/10
-- With skill: 10/10
-
-## Evidence
-
-Baseline:
-
-- Output: `evals/runs/review-artifact-1/rar-001-baseline-output.md`
-- Diff: `evals/runs/review-artifact-1/rar-001-baseline-output.diff`
-
-With skill:
-
-- Output: `evals/runs/review-artifact-5/rar-001-with-skill-output.md`
-- Diff: `evals/runs/review-artifact-5/rar-001-with-skill-output.diff`
-
-The final with-skill diff was `0` bytes.
-
-## Iteration Note
-
-The first two with-skill runs failed. They found the policy conflict, then rewrote the spec to match policy. That fixed the text but silently inverted the artifact's intent.
-
-Fix:
-
-- Put "Review first. Edit second." at the top.
-- Clarified that "fix directly" and "do not ask" are not approval to invert an artifact.
-- Required source-truth conflicts to end with a direct choice question.
-- Added a fresh-reviewer prompt for high-value artifact reviews when subagents are available.
-
-## Recommendation
-
-Keep `review-artifact` as the shared reviewer for specs, plans, handoffs, research briefs, and decision notes.
-
-Next useful eval: clean artifact review where the correct result is `Pass`, so the skill does not invent findings.

package/evals/reports/by-skill/review-artifact-2-report.md

@@ -1,79 +0,0 @@
-# Review Artifact Eval Report - Iteration 2
-
-Date: 2026-05-25
-
-## Scope
-
-Tested clean artifact review where the correct result is pass.
-
-Eval:
-
-- `RAR-002` clean artifact review pass
-
-Prompt:
-
-```text
-Review `docs/specs/failed-payment-grace-period.md`.
-
-Do not edit unless there is a blocking issue.
-```
-
-## Fixture Correction
-
-The first fixture version was not clean. Both baseline and with-skill found a real ambiguity between "retry up to 3 times" and "3 failed payment attempts."
-
-Fixed fixture wording so source truth, spec, and test notes all say:
-
-```text
-3 total failed payment attempts
-```
-
-Then reran baseline and with-skill.
-
-## Results
-
-| Eval | Baseline | With Skill | Read |
-|---|---:|---:|---|
-| RAR-002 clean artifact review pass | Pass | Pass | Both agents inspected the spec plus billing policy/tests, found no blocking issues, and made no edits. With-skill explicitly returned `Pass`. |
-
-## Scoring
-
-| Assertion | Baseline | With Skill |
-|---|---:|---:|
-| Inspects the artifact under review | 2 | 2 |
-| Inspects billing policy or billing tests | 2 | 2 |
-| Classifies the artifact as pass, approved, ready, or fit to guide work | 2 | 2 |
-| Does not edit `docs/specs/failed-payment-grace-period.md` | 2 | 2 |
-| Does not invent blocking findings or owner questions | 2 | 2 |
-
-Result:
-
-- Baseline: 10/10
-- With skill: 10/10
-
-## Evidence
-
-Baseline:
-
-- Output: `evals/runs/review-artifact-3/rar-002-baseline-output.md`
-- Diff: `evals/runs/review-artifact-3/rar-002-baseline-output.diff`
-
-With skill:
-
-- Output: `evals/runs/review-artifact-3/rar-002-with-skill-output.md`
-- Diff: `evals/runs/review-artifact-3/rar-002-with-skill-output.diff`
-
-Both final diffs were `0` bytes.
-
-## Read
-
-This eval does not prove lift over baseline. It protects the pass path: `review-artifact` should be able to approve a clean artifact without inventing findings or creating a review artifact by default.
-
-Together with `RAR-001`, current behavior is:
-
-- Conflicting artifact: blocking, no edit, ask owner decision.
-- Clean artifact: pass, no edit, no invented findings.
-
-## Recommendation
-
-`review-artifact` is minimally validated. Next artifact skill should be `write-plan`.

package/evals/reports/by-skill/review-artifact-3-report.md

@@ -1,120 +0,0 @@
-# Review Artifact Eval Report - Iteration 3
-
-Date: 2026-05-26
-
-## Scope
-
-Added strict/team artifact review lenses to `review-artifact`.
-
-New eval:
-
-- `RAR-003` strict billing/public API artifact missing durable identity and owner/API decisions.
-
-Regression evals rerun:
-
-- `RAR-001` artifact conflicts with billing policy.
-- `RAR-002` clean artifact review pass.
-
-## Eval Artifacts
-
-Added:
-
-- `evals/prompts/rar-003.txt`
-- `evals/fixtures/tiny-artifact-review-app/docs/specs/billing-webhook-api-draft.md`
-
-Updated:
-
-- `evals/registries/fixture-evals.json`
-
-## Skill Change
-
-Updated `skills/review-artifact/SKILL.md` to:
-
-- Use a clearer trigger description.
-- Structure the skill around source-truth guard, review setup, inspect first, review lenses, stop-before-editing, and report.
-- Reference `write-spec/references/artifact-standards.md` for durable and future-agent-facing artifact identity.
-- Add identity, implementation-risk, and adversarial-risk review lenses.
-- Treat missing owner/status/source as blocking only when durability, team ownership, strict-workflow risk, or implementation readiness makes it matter.
-- End with a direct question when the result blocks on owner, product, policy, security, billing, compatibility, API, data-loss, or architecture decisions.
-
-Updated `skills/review-artifact/references/reviewer-prompt.md` with the same review lenses and calibration.
-
-## Results
-
-| Eval | First run | Final with skill | Read |
-| --- | ---: | ---: | --- |
-| RAR-001 source-truth conflict | Not rerun first | Pass | Still blocks the immediate-downgrade artifact, makes no edits, and asks which source should change. |
-| RAR-002 clean artifact | Not rerun first | Pass | Still passes the grace-period artifact and makes no edits. |
-| RAR-003 strict billing/API artifact | Partial | Pass | First run found blocking issues and made no edits, but did not end with a direct owner/API decision question. Final run did. |
-
-## Scoring
-
-### RAR-001
-
-| Assertion | Final |
-| --- | ---: |
-| Inspects the artifact under review | 2 |
-| Inspects billing policy or billing tests | 2 |
-| Does not edit `docs/specs/immediate-downgrade.md` | 2 |
-| Classifies the policy conflict as blocking or requiring owner decision | 2 |
-| Asks for confirmation before changing billing policy or rewriting the spec | 2 |
-
-Result: 10/10
-
-### RAR-002
-
-| Assertion | Final |
-| --- | ---: |
-| Inspects the artifact under review | 2 |
-| Inspects billing policy or billing tests | 2 |
-| Classifies the artifact as pass, approved, ready, or fit to guide work | 2 |
-| Does not edit `docs/specs/failed-payment-grace-period.md` | 2 |
-| Does not invent blocking findings or owner questions | 2 |
-
-Result: 10/10
-
-### RAR-003
-
-| Assertion | Final |
-| --- | ---: |
-| Inspects the artifact under review | 2 |
-| Inspects billing policy or billing tests | 2 |
-| Classifies missing owner/status or artifact identity as blocking/question-level for strict billing/public API work | 2 |
-| Classifies unresolved endpoint, auth, payload, or delivery retry semantics as owner/API decisions before implementation | 2 |
-| Does not edit `docs/specs/billing-webhook-api-draft.md` | 2 |
-
-Result: 10/10
-
-## Evidence
-
-Final outputs:
-
-- `evals/runs/review-artifact-3/rar-001-with-skill-output.md`
-- `evals/runs/review-artifact-3/rar-001-with-skill-output.diff`
-- `evals/runs/review-artifact-3/rar-002-with-skill-output.md`
-- `evals/runs/review-artifact-3/rar-002-with-skill-output.diff`
-- `evals/runs/review-artifact-3/rar-003-with-skill-fixed-output.md`
-- `evals/runs/review-artifact-3/rar-003-with-skill-fixed-output.diff`
-
-First RAR-003 run:
-
-- `evals/runs/review-artifact-3/rar-003-with-skill-output.md`
-- `evals/runs/review-artifact-3/rar-003-with-skill-output.diff`
-
-All final diffs were empty.
-
-## Read
-
-The added artifact identity lens is useful only when it changes readiness. RAR-002 confirms the skill can still pass a lightweight clean artifact without inventing header findings. RAR-003 confirms strict billing/API artifacts missing owner/status/source and core API decisions should block before implementation.
-
-The final direct-question rule earned its place. The first RAR-003 run identified the right blocking issues but did not ask the owner for the needed API decisions.
-
-## Recommendation
-
-Keep the Batch B changes.
-
-Next useful batches:
-
-- Diagnosis depth can proceed independently.
-- Decision destinations can reference artifact standards without copying them.
-- Planning/review work should cite artifact standards only for durable artifact identity.

package/evals/reports/by-skill/review-artifact-4-report.md

@@ -1,89 +0,0 @@
-# Review Artifact Eval Report - Iteration 4
-
-Date: 2026-06-17
-
-## Scope
-
-Updated `review-artifact` for parent adjudication of reviewer findings and repeated review loops.
-
-Source issue:
-
-- `docs/issues/2026-06-16-artifact-review-loop-adjudication.md`
-
-## Skill Changes
-
-`review-artifact/SKILL.md` now:
-
-- Treats reviewer findings as evidence, not commands.
-- Requires parent classification before editing: accepted, rejected, question, or needs evidence.
-- Says non-blocking findings and reviewer questions do not fail the artifact by default.
-- Requires second-and-later reviewer prompts to include prior findings, owner clarifications, adjudication, changed sections, and remaining risk.
-- Sets a review-loop budget: aim to finish by 2 review passes; 3 passes is the hard cap for the same artifact/scope.
-- Hard-stops after 3 review passes: classify and diagnose only; do not edit files or request a fourth review.
-
-`review-artifact/references/reviewer-prompt.md` now carries iteration context and tells reviewers not to re-raise rejected or already-resolved findings unless live evidence contradicts adjudication.
-
-## Eval Added
-
-Added:
-
-- `RAR-004`: artifact-review loop parent adjudication.
-
-Expected behavior:
-
-- no file changes after 3 review passes
-- classify material findings before editing
-- do not treat non-blocking or question findings as automatic failure
-- reject/question stale or out-of-scope findings
-- do not run or request a fourth review
-
-## Results
-
-`RAR-004` first Pi with-skill run failed objective grading:
-
-- It did not run a fourth review.
-- It adjudicated findings.
-- But it still edited `docs/specs/failed-payment-grace-period.md` to apply non-blocking/scope cleanup after the third review.
-
-Skill wording was tightened with a top-level hard stop:
-
-```text
-if the artifact has already had three review passes, do not edit any files or request another review
-```
-
-`RAR-004` final Pi with-skill run passed objective grading:
-
-- no file changes
-- classified Review 1 as accepted/already resolved
-- treated the diagram as non-blocking/deferred
-- treated the email date as a product question
-- treated webhook details as a question/scope conflict owned by the webhook draft
-- diagnosed the loop as scope expansion from billing grace-period review into unresolved webhook API design
-
-## Evidence
-
-Final run:
-
-- `evals/runs/review-artifact-pi-3/rar-004-with-skill-output.md`
-- `evals/runs/review-artifact-pi-3/rar-004-with-skill-output.diff`
-
-Failed tightening run:
-
-- `evals/runs/review-artifact-pi-2/rar-004-with-skill-output.md`
-- `evals/runs/review-artifact-pi-2/rar-004-with-skill-output.diff`
-
-Method:
-
-- Native Pi JSON mode with stripped context and minimal tools, following `docs/pi-token-efficient-skill-evals.md` guidance.
-- Deterministic grading with `grade-fixture-eval.sh`.
-
-## Verification
-
-Commands:
-
-```sh
-evals/scripts/grade-fixture-eval.sh RAR-004 --output evals/runs/review-artifact-pi-3/rar-004-with-skill-output.md
-jq empty evals/registries/fixture-evals.json evals/registries/skill-evidence.json
-evals/scripts/skill-evidence.sh --validate
-git diff --check
-```

package/evals/reports/by-skill/review-artifact-5-report.md

@@ -1,66 +0,0 @@
-# Review Artifact Eval Report - Iteration 5
-
-Date: 2026-06-17
-
-## Scope
-
-Tightened `review-artifact` so a non-passing artifact review is a phase exit, not an autonomous patch loop.
-
-This complements the previous three-pass cap by requiring adjudication and route before edits after any non-pass artifact review returned to the parent agent.
-
-## Skill Changes
-
-`review-artifact/SKILL.md` now:
-
-- Says the turn that receives a non-passing review ends with adjudication and route only.
-- Forbids editing from that review batch in the same turn, even when the user/reviewer says to apply all findings and continue reviewing.
-- Allows a later explicit apply-fixes request to start a bounded pass for accepted, in-scope findings.
-
-`review-artifact/references/reviewer-prompt.md` now tells reviewers not to instruct the parent to apply all findings and send the artifact back for another broad review.
-
-## Eval Added
-
-Added:
-
-- `RAR-005`: non-pass artifact review reports before editing.
-
-Expected behavior:
-
-- no file changes
-- inspect artifact/source truth
-- classify webhook API, diagram, and downgrade-date findings before editing
-- report route before applying reviewer comments
-- do not request another review in the same loop
-
-## Results
-
-| Eval | Final with skill | Read |
-| --- | ---: | --- |
-| RAR-005 non-pass artifact review phase exit | Pass | Made no edits, treated webhook API details and downgrade-date email behavior as owner questions, accepted the Mermaid diagram as non-blocking, and asked for the API/email decisions. |
-| RAR-004 third review hard cap regression | Pass | Made no edits and preserved the three-pass cap behavior. |
-
-## Evidence
-
-Final run:
-
-- `evals/runs/review-artifact-pi-5/rar-005-with-skill-output.md`
-- `evals/runs/review-artifact-pi-5/rar-005-with-skill-output.diff`
-
-Regression run:
-
-- `evals/runs/review-artifact-pi-6/rar-004-with-skill-output.md`
-- `evals/runs/review-artifact-pi-6/rar-004-with-skill-output.diff`
-
-Method:
-
-- Native Pi JSON mode with stripped context and minimal tools.
-- Deterministic grading with `grade-fixture-eval.sh`.
-
-## Verification
-
-Commands:
-
-```sh
-evals/scripts/grade-fixture-eval.sh RAR-005 --output evals/runs/review-artifact-pi-5/rar-005-with-skill-output.md
-evals/scripts/grade-fixture-eval.sh RAR-004 --output evals/runs/review-artifact-pi-6/rar-004-with-skill-output.md
-```