@hassangameryt/freeflow 0.8.0 → 0.9.0

package/skills/evaluate-skill/references/evaluation-design.md

@@ -1,112 +0,0 @@
-# Evaluation Design
-
-Read this before choosing or materially changing the behavioral question, evidence class, group shape, pressure, or baseline/candidate boundary.
-
-This reference is judgment: adapt it to the question being answered. Its binding statements restate [Evaluate Skill](../SKILL.md) Rules or the definition schema.
-
-## Map Claims To Evidence
-
-| Claim | Required evidence |
-| --- | --- |
-| Description routes the earliest useful prompt | Natural activation |
-| Body works with guaranteed context | Explicit body delivery |
-| Nearby prompt is not hijacked | Natural activation plus behavioral output |
-| Declared dependencies compose | Exact ordered composition |
-| Guidance remains useful later | Multi-turn evidence |
-| Files or structured state match | Artifact outcome |
-| Behavior holds across named hosts | The same group semantics on every named host |
-
-Record separately:
-
-- declaration;
-- resource materialization;
-- delivery;
-- observed reads;
-- behavior;
-- artifacts;
-- derived grades;
-- later semantic judgment.
-
-A read does not prove compliance. Correct behavior does not prove which skill caused it.
-
-## Choose The Smallest Group Shape
-
-- **Description prompt:** natural activation and exact read timing.
-- **Explicit body task:** first-read body behavior without activation ambiguity.
-- **Fixture task:** files or repository state matter.
-- **Stateful turns:** ordered conversation and retained use matter.
-- **Saved-result review:** canonical evidence already answers the question.
-- **Suite:** several independent questions require one ordered serial invocation.
-
-Keep activation and body behavior separate unless their integrated interaction is the exact question and the current runner supports that observing boundary.
-
-## Fix Exact Variants
-
-Every group contains exactly:
-
-- `baseline`;
-- `candidate`.
-
-For a new skill, baseline has no target.
-
-For a revision, baseline names an exact immutable previous package. If no exact baseline exists, return to Write Skill before revision.
-
-To learn whether a skill's body, not just its description, changes behavior, add a separate group whose baseline is a neutral package: the same name and description with a placeholder body. A candidate that matches the neutral baseline gains nothing from its body under that pressure.
-
-For a description-only revision, body and resources remain byte-identical.
-
-Freeze prompts or turns, fixture, tools, model, thinking, other skills, context, runtime, and criteria across variants.
-
-## Design Common Questions
-
-### Description activation
-
-Use the earliest natural prompt where the target should become useful.
-
-For a nearby prompt, predeclare whether success means:
-
-- non-trigger; or
-- safe behavior after activation.
-
-### First-read body
-
-Explicitly deliver the exact target body with only guaranteed context. Ambient package context cannot repair a hidden dependency.
-
-### Dependency composition
-
-Materialize exact ordered skills and context. Vary only the target snapshot. Do not provide a hidden base stack.
-
-### Retained use
-
-Use ordered turns in one persistent subject process. Repeated body delivery does not prove retained use.
-
-### Artifact outcome
-
-Grade files, changed paths, structured state, and events before relying on the final response. Preserve turn-scoped workspace evidence when later turns may change earlier state.
-
-## Apply Real Pressure
-
-A useful group:
-
-- creates a natural temptation to violate the intended behavior;
-- keeps criteria and review questions outside the prompt;
-- exposes deterministic evidence where possible;
-- distinguishes baseline from candidate behavior;
-- remains realistic enough that passing behavior is useful.
-
-Choose the pressure by the kind of instruction under test:
-
-- **A Rule:** make breaking it the easy or rewarded path, such as urgency, a user request that conflicts with it, or a shortcut that saves work. The candidate passes only if the Rule holds.
-- **Judgment:** include at least one case where following the default literally is the worse choice, next to a case where the default is right. The candidate passes on the outcome in both, so a variant that only obeys the default cannot pass.
-
-## Grade By Requirement And Respect Variance
-
-Grade each requirement a group checks, not only pass or fail. Label each as derivable from the request and code, dependent on a clarification, or imposed by the evaluator (an exact name or message only the checker knows); a failure on an evaluator-imposed requirement says little about the skill.
-
-One run per cell cannot separate a difference of one or two items from run-to-run variance. Repeat the cell, or report such a difference as unresolved.
-
-Freeze the candidate before a batch runs. Items used to develop a revision are not evidence for it: judge it on held-out items, and stop iterating on the items it was tuned against.
-
-## Keep Separate Conclusions Separate
-
-An explicitly delivered body may support first-read behavior while a natural prompt still fails activation. Preserve both conclusions; one does not repair the other.

package/skills/evaluate-skill/references/execution-and-evidence.md

@@ -1,111 +0,0 @@
-# Execution And Evidence
-
-Read this when operating `run|view|review`, resolving paths, interpreting states, or reasoning about Pi execution, isolation, persistence, cancellation, and safeguards.
-
-This reference is a contract: it states how the evaluator behaves. Rely on that behavior as written; its binding requirements on you restate [Evaluate Skill](../SKILL.md) Rules.
-
-## Command And Path Contract
-
-Run commands from the repository or fixture root that owns the definitions:
-
-```text
-node <evaluate-skill-directory>/scripts/skill-eval.mjs run <suite-or-group-path> [--group <id-or-position>] [--variant baseline|candidate] [--trials <n>] [--max-cost <total>]
-node <evaluate-skill-directory>/scripts/skill-eval.mjs view <result-id-or-directory> [--group <id-or-position>] [--variant baseline|candidate]
-node <evaluate-skill-directory>/scripts/skill-eval.mjs review <result-id-or-directory> --model <provider/model> [--group <id-or-position>] [--thinking <level>]
-```
-
-- The current working directory is the definition root and result root.
-- The `run` target resolves from that directory.
-- Suite group references resolve relative to the suite file.
-- Fixture, skill, and context declarations resolve from the definition root.
-- Results are stored under `<cwd>/.skill-eval/runs/<result-id>`.
-- A bare `view` target is tried first as a stored result ID, then as a path from the working directory.
-- An absolute or path-like `view` target resolves as a directory path.
-
-Examples:
-
-```text
-skill-eval run .skill-eval/my-skill/suite.json
-skill-eval run .skill-eval/my-skill/suite.json --variant candidate
-skill-eval view 20260720173209-a3649b8b
-skill-eval view 20260720173209-a3649b8b --group 2 --variant candidate
-skill-eval view /absolute/path/to/result
-```
-
-With no selectors, `run` or `view` selects every suite group and both variants. `--group` is invalid for a direct group definition/result.
-
-## Current Execution Boundary
-
-`run` executes description and body groups. Unsupported group types are rejected before subject execution. Description groups allow no tools or `read`; body groups allow no tools or path-guarded `read`, `write`, and `edit`, the evaluator's `run_command` tool for declared commands, plus non-native custom tools supplied by a declared runtime extension bundle. Native `bash`, `powershell`, `grep`, `find`, and `ls` tools are unsupported.
-
-A runtime profile selects the installed native Pi host, the `isolated` or `host` prompt mode, whether the variant uses an isolated persistent session directory, zero or more ordered extension bundles, and declarative environment sources. Literal values are non-secret configuration; only explicitly inherited parent variables are passed to the child without being persisted, alongside a small host-runtime baseline. Missing inherited variables invalidate the variant. Launch-control and loader keys are rejected from inherited values, and literal and inherited names must not overlap. For body runs, delivery remains `unavailable` until target setup succeeds; `natural-prompt` or `explicit-skill-command` records a prompt attempt. The evaluator launches the base guard first, its command tool when the group declares commands, declared extensions next, and, when a group declares context assertions, a final non-mutating observer last. The observer records both the effective `before_agent_start` system prompt and each provider-neutral `context` message projection; it does not replace or sanitize extension injections after the base guard has established evaluator-owned isolation.
-
-Use fresh JSON-mode execution for one-shot descriptions. Use one persistent RPC process per selected variant for ordered description turns and all body groups. RPC correlates responses, waits for `agent_settled`, disables automatic retry and compaction, preserves directly observed partial-turn evidence, and cleans the process tree.
-
-A body target is matched to the exact snapshotted `SKILL.md` and explicitly delivered on turn one. Later turns remain unchanged. Description groups use natural prompts and never explicit target delivery.
-
-## Isolation And Resources
-
-For each selected variant:
-
-- snapshot exact declared skills, context, and the shared runtime bundle resources;
-- copy the shared runtime bundles identically into the variant and verify their fingerprints before and after execution;
-- snapshot the group fixture once from the definition-root working tree—never from a variant Git source—then create an independent writable copy;
-- preserve skill order and target index;
-- preserve runtime host, session mode, and extension order;
-- preserve runtime bundle and provider-context observation evidence when requested;
-- replace ambient instructions with evaluator-owned declared context;
-- keep criteria and review questions outside the subject prompt;
-- allow reads only from workspace and declared immutable resources;
-- allow writes and edits only inside the workspace;
-- reject traversal, symlink escape, unsupported Git entries, and invalid UTF-8;
-- verify immutable resource fingerprints after execution.
-
-Working-tree sources snapshot current paths. Git sources resolve one exact commit and record both the declared ref and resolved identity.
-
-## Persistence And States
-
-Each selected run is persisted before its counterpart. Grade and group evidence are persisted before later groups.
-
-Run states are `complete`, `invalid`, `infrastructure-failed`, `cancelled`, and `not-selected`. Group and batch states are `complete`, `partially-complete`, `invalid` where applicable, and `cancelled`.
-
-- Failed deterministic checks are ordinary complete behavioral evidence and exit zero when no infrastructure/grade failure exists.
-- Variant-local invalidity or infrastructure failure preserves evidence and does not stop safe queued work.
-- Shared fixture failure invalidates the group.
-- Thrown grading or recoverable grade/group publication failure becomes explicit `grade-error` fallback evidence.
-- If a run or required fallback cannot be preserved, queued work stops and no completed summary is published.
-- Cancellation starts no queued subject and preserves observed diagnostics.
-
-Invalidity, infrastructure failure, cancellation, or `grade-error` makes the final command exit nonzero after safe queued work finishes.
-
-## Views And Raw Evidence
-
-Views can render a complete result, all baseline or candidate variants, one group, or one group/variant. Grade appears before selected run evidence. The displayed absolute result path anchors result-relative artifact and workspace paths.
-
-Path cells escape literal backslashes before tab, carriage return, and newline. Variant views exclude expectation-owned errors from the other variant while retaining comparison, group, and system errors.
-
-Use ordinary file tools for raw `run.json`, events, transcript, final response, stderr, workspace, definition, grade, context observations, and group artifacts. Views remove transport noise; they do not replace canonical evidence. Tool-call grades distinguish attempted, succeeded, failed, and not-called outcomes.
-
-## Safeguards And Limits
-
-Normal completion follows settlement. Path guards, no-progress detection, cancellation, process-tree cleanup, and very high emergency ceilings stop runaway or unsafe infrastructure. Subjects run until they settle, with no turn, token, output, or short time cap, because a guessed cap cuts off the behavior the evaluation is trying to observe.
-
-`--max-cost <total>` is an explicit spend ceiling, not a subject cap. After each subject finishes, its host-reported `usage.cost.total` is added to the invocation's spend; once spend reaches the ceiling, no further subject starts and queued variants become `cancelled`. The subject already running completes, so spend can exceed the ceiling by one subject. The summary records `budget` with the ceiling, spend, and whether it was exhausted. A host that reports no cost never exhausts the ceiling.
-
-## Trials
-
-`--trials <n>` runs the selection as `n` independent complete invocations. Each trial is an ordinary result with its own ID. A separate aggregate under `.skill-eval/runs/<aggregate-id>/aggregate.json` lists the trials and, per group, counts pass, fail, and unavailable for every deterministic check and comparison, plus comparison transitions. `--max-cost` is shared across trials, and exhaustion stops starting further trials. `view <aggregate-id>` renders the counts; view an individual trial for its evidence.
-
-## Advisory Review
-
-`review` answers each group's `review_questions` with a separate reviewer model:
-
-- only groups with review questions and two complete runs are reviewed;
-- the reviewer sees the task turns, each run's responses, tool calls, and changed paths as Run A and Run B in random order, and never sees variant names or deterministic grades;
-- it runs without tools, skills, extensions, context files, or session, with only the base process environment;
-- `groups/<id>/semantic-grade.json` records the label mapping, prompt hash, answers mapped back to variants, and errors; `groups/<id>/review/` keeps the prompt, events, final response, and stderr;
-- malformed or incomplete reviewer output becomes `review-error` with the raw response preserved.
-
-A review is advisory. It never changes run or deterministic grade evidence, and rerunning it replaces only the previous advisory review.
-
-When an unsupported operation changes the question, reject it or run a clearly separate direct comparison with explicit limits. Never invoke archived evaluators as a fallback.

package/skills/evaluate-skill/references/review-and-revision.md

@@ -1,94 +0,0 @@
-# Review And Revision
-
-Read this after canonical evidence and deterministic grades exist, when evidence conflicts, meaning remains unresolved, the failed boundary must be classified, or revision and rerun scope must be chosen.
-
-This reference is judgment: adapt it to the evidence in hand. Its rerun and no-edit requirements restate [Evaluate Skill](../SKILL.md) Rules.
-
-## Use The Strongest Evidence
-
-Prefer, in order:
-
-1. filesystem state, typed workspace manifests, diffs, and changed paths;
-2. structured events, tool activity, usage, process results, logs, and saved reports;
-3. final response;
-4. full transcript when stronger surfaces cannot answer the question;
-5. subject self-assessment.
-
-Lower-priority evidence cannot repair a contradiction above it.
-
-If a response claims success but a required file is absent, preserve both facts and judge from the stronger artifact evidence.
-
-## Keep Deterministic Facts Separate
-
-Built-in checks establish exact observable facts. Missing evidence remains unavailable. A failed check remains behavioral evidence and does not relabel a complete run as infrastructure failure.
-
-The active agent or user may judge:
-
-- reasoning quality;
-- authority preservation;
-- architectural fitness;
-- recommendation quality;
-- semantic completeness.
-
-Keep semantic judgment, cited evidence, uncertainty, and limits outside canonical run and deterministic grade state.
-
-Do not turn semantic judgment into evaluator-owned readiness or promotion status.
-
-## Diagnose The Failed Boundary
-
-Classify before revising:
-
-- activation timing;
-- description language unavailable before body read;
-- first-read body comprehension;
-- undeclared, undelivered, unread, or ambient-only dependency;
-- normal guidance hidden behind the wrong resource boundary;
-- nearby behavior that hijacks the task;
-- retained-use or multi-turn failure;
-- wording, placement, example, repetition, or stop-condition pressure;
-- fixture, environment, isolation, host, runner, or deterministic-check failure.
-
-Do not add broad prose before identifying which boundary failed.
-
-## Return Revision To Write Skill
-
-When evidence supports a skill change:
-
-1. name the exact failed boundary;
-2. cite the canonical evidence;
-3. identify one measured pressure point;
-4. preserve unrelated instructions and fixed criteria;
-5. return the revision target to [Write Skill](../../write-skill/SKILL.md).
-
-Write Skill owns package revision.
-
-Evaluate Skill does not edit the skill or replace its baseline while evaluating it.
-
-When evidence proves the group, fixture, or criterion is wrong, revise the evaluation instead. Preserve the old result and state that it no longer answers the revised question.
-
-## Rerun The Complete Group
-
-After a measured skill revision or resolved infrastructure cause:
-
-- rerun the complete fixed group;
-- run both baseline and candidate;
-- keep every result visible;
-- do not reuse an earlier baseline run as a fresh counterpart;
-- do not rerun only the preferred variant;
-- do not use partial evidence as a comparable full result;
-- do not rerun merely to erase failure.
-
-For variance, use `--trials` to run independent complete invocations rather than repeated preferred variants.
-
-## Report
-
-Report:
-
-- failed boundary and evidence;
-- exact skill revision or evaluation correction;
-- criteria that remained fixed;
-- old and new result paths;
-- deterministic transitions;
-- remaining unsupported claims.
-
-The user decides whether to revise further, use, publish, or reject the skill.