@hassangameryt/freeflow 0.7.3 → 0.8.0

package/skills/evaluate-skill/SKILL.md

@@ -12,31 +12,42 @@ A **group** asks one behavioral question through exactly two variants:
 - `baseline`: no target skill for a new skill, or one exact immutable previous package for a revision;
 - `candidate`: one exact new or updated package.
 
-The **subject** performs the declared task under one variant. The **evaluator** isolates subjects and preserves canonical evidence. The **deterministic grader** derives fixed mechanical facts after run persistence. A **suite** runs an ordered set of independent groups.
+The **subject** performs the declared task under one variant. The **evaluator** isolates subjects and preserves canonical evidence. The **deterministic grader** derives fixed mechanical facts after the run is saved. A **suite** runs an ordered set of independent groups.
 
-Evaluate Skill does not author or revise skills, decide semantic quality, claim readiness, or choose publication. [Write Skill](../write-skill/SKILL.md) owns package preservation and revision. The active agent or user judges unresolved meaning. The user decides whether to revise, use, publish, or reject the skill.
+Evaluate Skill does not author or revise skills, decide semantic quality, claim readiness, or choose publication. [Write Skill](../write-skill/SKILL.md) owns package preservation and revision. The active agent or user judges unresolved meaning; the user decides whether to revise, use, publish, or reject the skill.
 
-Inline examples teach the subject and are part of the candidate intervention. They are not independent evaluation evidence. Keep deterministic criteria and review questions outside natural subject prompts.
+The **Rules** below are binding; if one cannot be met, stop and say why. Everything else is **judgment**: follow its stated goal, and adapt when a different choice serves that goal better.
 
-## Route From Existing Evidence First
+## Rules
+
+- Never mutate canonical evidence or erase an unfavorable result. Preserve partial and infrastructure evidence without converting unavailable behavior into pass or fail.
+- Never fabricate behavioral evidence. Missing evidence stays unavailable.
+- Compare a revision only against an exact immutable baseline. If none exists, return to Write Skill to preserve the current package; never create a baseline by editing the package being evaluated.
+- Keep deterministic criteria and review questions out of subject prompts and declared subject context.
+- Declare every subject resource. Ambient installation or context is not declared composition.
+- Claim only what the evidence class observed; one class never proves another.
+- Never edit the skill from Evaluate Skill. Return one measured revision target to Write Skill.
+- After a behavioral change, rerun the complete fixed group with both variants. Never reuse an earlier run as the fresh counterpart or rerun only the preferred variant.
+- Advisory review and semantic judgment never enter canonical run or grade state, and neither establishes readiness.
+- Never invoke archived evaluators.
+
+## Start From Existing Evidence
+
+Goal: answer the question with the least new evidence that can answer it.
 
 Before proposing a run or revision:
 
-- inspect adequate saved evidence;
+- inspect adequate saved evidence, through views and raw artifact reads when a complete result already answers the question;
 - reuse an existing group unchanged when it still preserves the question, pressure, variants, and fixed criteria;
-- use saved views and raw artifact reads when a complete result already answers the question;
-- preserve partial infrastructure evidence without converting unavailable behavior into pass or fail;
-- do not mutate canonical evidence or erase an unfavorable result.
-
-For a draft-only request, use Write Skill. Do not fabricate behavioral evidence.
+- for a draft-only request, use Write Skill instead.
 
-A revision comparison requires an exact immutable baseline identity. If none exists, return to Write Skill to preserve the complete current package before revision. Evaluate Skill consumes baseline and candidate identities; it does not create a baseline by editing the package it is evaluating.
+Do not rerun merely to demonstrate process.
 
 ## Name One Behavioral Question
 
-Name the exact claim and observing mechanism before choosing the group shape.
+Goal: one question, observed at the boundary where the claim lives.
 
-Evidence classes include:
+Name the exact claim and observing mechanism before choosing the group shape. Evidence classes include:
 
 - **Description activation:** did a natural prompt cause the exact target to be read, and when?
 - **First-read body behavior:** did explicit body delivery guide the subject using only guaranteed context?
@@ -44,9 +55,7 @@ Evidence classes include:
 - **Dependency composition:** did exact ordered skills, resources, and context work together?
 - **Retained use:** did guidance remain useful on later declared turns without redelivery?
 - **Artifact outcome:** did files, structured state, events, and responses match fixed criteria?
-- **Cross-host behavior:** did the same behavioral question hold on every named host?
-
-One evidence class does not prove another.
+- **Cross-host behavior:** did the same behavioral question hold on every named host? The evaluator currently runs only native Pi hosts.
 
 ```text
 successful read
@@ -65,10 +74,14 @@ correct behavior
 ≠ proof that the target caused it
 ```
 
+Test a skill's Rules and its judgment differently. A Rule needs pressure to break it; it passes only if it holds. Judgment needs a case where following the default literally is the worse choice; it passes on outcome, so a candidate that only obeys cannot pass.
+
 Read [Evaluation Design](references/evaluation-design.md) before choosing or materially changing the question, evidence class, group shape, pressure, or baseline/candidate boundary.
 
 ## Design The Smallest Valid Comparison
 
+Goal: the only difference between variants is the target, and the pressure is strong enough to show it.
+
 Before subject output exists:
 
 1. fix one behavioral question;
@@ -78,80 +91,51 @@ Before subject output exists:
 5. keep prompt or turns, fixture, tools, model, thinking, other skills, context, runtime, and criteria fixed except for the target difference;
 6. run one group or an ordered suite serially;
 7. inspect canonical evidence and deterministic facts before judging meaning;
-8. classify the failed boundary before proposing revision;
-9. return one measured revision target to Write Skill;
-10. rerun the complete fixed group with both variants after a behavioral change.
-
-For a new skill:
-
-```text
-baseline: no target skill
-candidate: exact candidate package
-```
-
-For a revision:
-
-```text
-baseline: exact previous package
-candidate: exact revised package
-```
+8. classify the failed boundary before proposing revision.
 
-For a description-only revision, keep body, references, scripts, and other resources byte-identical.
+For a new skill, the baseline has no target skill. For a revision, the baseline is the exact previous package. For a description-only revision, keep body, references, scripts, and other resources byte-identical.
 
 If both variants pass, the pressure may be weak or the baseline already sufficient. If both fail, distinguish skill, fixture, dependency, environment, host, isolation, and criterion failures before revising instructions.
 
-## Declare The Complete Environment
+Inline examples in the skill teach the subject and are part of the candidate intervention; they are not independent evaluation evidence.
 
-Use the exact [Evaluation Definition Schema](references/evaluation-definition-schema.md) when authoring or checking group and suite JSON.
+## Declare The Complete Environment
 
-Declare:
+Use the exact [Evaluation Definition Schema](references/evaluation-definition-schema.md) when authoring or checking group and suite JSON. Declare:
 
 - prompt or ordered turns;
 - fixture;
 - exact ordered skill packages and target index;
 - working-tree or Git source identity;
 - context resources;
-- tools;
+- tools and any declared commands;
 - model and thinking;
-- runtime host and extension bundles when needed;
+- runtime host, prompt mode, and extension bundles when needed;
 - literal and inherited environment sources;
 - deterministic expectations and comparison IDs;
 - review questions.
 
-Snapshot every declared subject resource. Ambient installation is not declared composition.
-
-Criteria and review questions remain outside the subject prompt.
-
 ## Run And Preserve Canonical Evidence
 
 Before operating the evaluator, resolving paths, interpreting states, or reasoning about host execution, read [Execution and Evidence](references/execution-and-evidence.md).
 
-Use the canonical [skill evaluator](scripts/skill-eval.mjs):
+Use the canonical [skill evaluator](scripts/skill-eval.mjs) from the definition root:
 
 ```text
-node <evaluate-skill-directory>/scripts/skill-eval.mjs run <suite-or-group-path> [--group <id-or-position>] [--variant baseline|candidate]
+node <evaluate-skill-directory>/scripts/skill-eval.mjs run <suite-or-group-path> [--group <id-or-position>] [--variant baseline|candidate] [--trials <n>] [--max-cost <total>]
 node <evaluate-skill-directory>/scripts/skill-eval.mjs view <result-id-or-directory> [--group <id-or-position>] [--variant baseline|candidate]
+node <evaluate-skill-directory>/scripts/skill-eval.mjs review <result-id-or-directory> --model <provider/model> [--group <id-or-position>] [--thinking <level>]
 ```
 
-Run from the definition root.
+One run per variant cannot separate a behavioral difference from sampling noise. Use `--trials <n>` before claiming a difference; each trial is an independent complete result, and the aggregate only counts their deterministic grades. Use `--max-cost` when trials or suites could spend more than intended.
 
 With no selectors, run or view every suite group and both variants. Selectors narrow execution or viewing scope; they do not modify definitions.
 
-Canonical results live under:
-
-```text
-<definition-root>/.skill-eval/runs/<result-id>
-```
-
-A view is a projection of saved evidence, not its canonical source. Use ordinary file tools when exact run, event, transcript, response, stderr, workspace, definition, grade, context-observation, or group artifacts matter.
-
-Do not invoke archived evaluators.
+Canonical results live under `<definition-root>/.skill-eval/runs/<result-id>`. A view is a projection of saved evidence, not its canonical source; use ordinary file tools when the exact run, event, transcript, response, stderr, workspace, definition, grade, context-observation, or group artifact matters.
 
 ## Derive Facts Without Inventing Meaning
 
-Append deterministic grades only after canonical run evidence exists.
-
-Deterministic checks may establish exact facts such as:
+Deterministic grades are appended only after canonical run evidence exists. They can establish exact facts such as:
 
 - skill or resource reads;
 - file and path state;
@@ -162,25 +146,15 @@ Deterministic checks may establish exact facts such as:
 - context text at a selected surface, request, and turn;
 - factual baseline-to-candidate transitions.
 
-A failed check remains ordinary behavioral evidence. It does not turn a valid subject run into infrastructure failure.
-
-Missing evidence remains unavailable.
-
-Malformed expectations or grading failures produce separate `grade-error` evidence and do not rewrite canonical run evidence.
+A failed check remains ordinary behavioral evidence; it does not turn a valid subject run into infrastructure failure. Malformed expectations or grading failures produce separate `grade-error` evidence and do not rewrite run evidence.
 
 A transition such as `fail-to-pass` is a factual comparison. It does not mean bad-to-good, not-ready-to-ready, or rejected-to-promotable.
 
 ## Review Meaning After Evidence Exists
 
-After canonical evidence and deterministic grades exist, read [Review and Revision](references/review-and-revision.md) when:
+After canonical evidence and deterministic grades exist, read [Review and Revision](references/review-and-revision.md) when evidence conflicts, semantic meaning remains unresolved, the failed boundary must be classified, a revision target must be selected, or rerun scope is uncertain.
 
-- evidence conflicts;
-- semantic meaning remains unresolved;
-- the failed boundary must be classified;
-- a revision target must be selected;
-- rerun scope is uncertain.
-
-The evaluator does not launch an automatic semantic grader.
+The evaluator never grades meaning automatically. `review` records an optional advisory answer to each review question from a separate model that sees the two runs unlabeled, in `semantic-grade.json`.
 
 The active agent or user may judge reasoning quality, authority preservation, architectural fitness, recommendation quality, or semantic completeness. Keep that judgment, cited evidence, uncertainty, and limits outside canonical run and grade state.
 
@@ -190,12 +164,9 @@ When evidence supports a candidate change:
 
 1. identify the exact failed boundary;
 2. preserve the canonical result path and relevant evidence;
-3. state one measured revision target;
-4. keep unrelated instructions and criteria stable;
-5. return revision to Write Skill;
-6. rerun the complete fixed group with both variants after the change.
-
-Do not edit the skill from Evaluate Skill.
+3. state one measured revision target, keeping unrelated instructions and criteria stable;
+4. return revision to Write Skill;
+5. rerun the complete fixed group with both variants after the change.
 
 If the evaluation definition or criterion is wrong, preserve the old result and state that it no longer answers the revised question. Do not reinterpret it as evidence for the new question.
 
@@ -214,6 +185,4 @@ Report:
 - what it does not prove;
 - any supported failed boundary or revision target.
 
-Stop rather than overclaim when required activation, first-read, composition, retained-use, isolation, artifact, or host evidence is unavailable.
-
-The active agent or user judges unresolved behavior. The user decides whether to revise, use, publish, or reject the skill.
+Stop rather than overclaim when required activation, first-read, composition, retained-use, isolation, artifact, or host evidence is unavailable. The user decides whether to revise, use, publish, or reject the skill.

package/skills/evaluate-skill/references/evaluation-definition-schema.md

@@ -2,6 +2,8 @@
 
 Read this when authoring or checking schema-version-1 group and suite JSON.
 
+This reference is a contract: the shapes and constraints below are binding as written. Structural violations are rejected before any subject runs; a malformed expectation surfaces afterward as `grade-error` evidence.
+
 This file owns user-authored definition shapes. Read [execution and evidence](execution-and-evidence.md) for commands, path resolution, run states, persistence, views, isolation, and safeguards.
 
 ## Definition Root
@@ -62,13 +64,14 @@ Rules:
 - `input` contains exactly one of:
   - `prompt`: one non-empty string;
   - `turns`: a non-empty ordered array of non-empty strings. Repeated natural turns are allowed.
-- `fixture` is `null` or one contained relative directory/file path.
+- `fixture` is `null` or one contained relative directory path. The directory's contents become the workspace.
 - `tools` is an ordered array with no duplicates.
 - `variants` contains exactly `baseline` and `candidate`.
 - `expectations` is an ordered array with unique expectation IDs.
 - `review_questions` is an ordered string array. Questions are rendered for the reviewer and never sent to the subject.
 - `model` is optional. `model.model` names the provider/model; optional `model.thinking` selects its thinking level. The CLI has no model override.
-- `runtime` is optional. When omitted, the group runs under a plain `pi` host with no declared extensions. When present, it declares the host, ordered extension bundles, and string environment overrides.
+- `runtime` is optional. When omitted, the group runs under a plain `pi` host with the isolated prompt and no declared extensions. When present, it declares the host, prompt mode, ordered extension bundles, and string environment overrides.
+- `commands` is optional and only for body groups that list the `run_command` tool. See [Declared Commands](#declared-commands).
 
 ## Description Group Example
 
@@ -204,7 +207,7 @@ A body group explicitly delivers each target on turn one and observes first-read
 }
 ```
 
-Description groups allow no tools or only built-in `read`. Body groups allow built-in `read`, `write`, and `edit`, plus custom tools declared by a runtime extension bundle. Native `bash`, `powershell`, `grep`, `find`, and `ls` tools are unsupported. A no-target baseline or runtime-only group may use `skills: []` and `target: null`; a candidate that declares skills must identify its target.
+Description groups allow no tools or only built-in `read`. Body groups allow built-in `read`, `write`, and `edit`, the evaluator's `run_command` tool for declared commands, plus custom tools declared by a runtime extension bundle. Native `bash`, `powershell`, `grep`, `find`, and `ls` tools are unsupported. A no-target baseline or runtime-only group may use `skills: []` and `target: null`; a candidate that declares skills must identify its target.
 
 The examples are complete definition shapes. Replace fixture, resource, model, and Git-ref values with paths and identities that exist under your definition root.
 
@@ -233,7 +236,12 @@ A runtime profile is shared by both baseline and candidate variants. It changes
 }
 ```
 
-`host` is `pi` or `piflow`; the evaluator maps it to the corresponding installed host command and records the selected host. `session` controls whether the variant receives an isolated persistent session directory; it defaults to `false` in omitted runtime profiles. A runtime may declare zero or more bundles. Each bundle has an entry file and one or more directory resources copied under their original relative paths, so sibling imports and multi-tree extensions remain intact. Bundle bytes are snapshotted once and copied identically to both variants.
+`host` is `pi`; the evaluator maps it to the installed Pi command and records the selected host. Optional `prompt` is `isolated` (default) or `host`:
+
+- `isolated` replaces the host's system prompt with a minimal evaluator-owned prompt. Use it to isolate the target skill.
+- `host` keeps the host's own system prompt and whatever declared extensions compose onto it, appending declared context. Use it when evidence must reflect the production prompt, for example by declaring the Freeflow Pi extension bundle with a fixture containing `.freeflow/config.json`.
+
+Both modes enforce the same tool and path guards. `session` controls whether the variant receives an isolated persistent session directory; it defaults to `false` in omitted runtime profiles. A runtime may declare zero or more bundles. Each bundle has an entry file and one or more directory resources copied under their original relative paths, so sibling imports and multi-tree extensions remain intact. Bundle bytes are snapshotted once and copied identically to both variants.
 
 `environment.literal` contains non-secret string configuration and is recorded in run evidence. `environment.inherit` names parent-process variables whose values are passed to the child but never persisted; other parent variables are not passed. Missing inherited variables invalidate the variant. The evaluator supplies only a small host-runtime baseline such as `PATH` and `HOME`. Launch-control and loader keys are rejected from inherited values, literal and inherited names must not overlap, and secret-like literal keys are rejected.
 
@@ -355,7 +363,29 @@ Two expectations may share one comparison ID only when they have opposite varian
 
 Definition loading validates common structure and unique IDs. Kind-specific expectation validation occurs during deterministic grading after run persistence. A malformed expectation therefore becomes separate `grade-error` evidence rather than changing the subject run. Use the documented shapes to avoid spending a subject run on an invalid check.
 
-Definition-supplied commands and tests are unsupported.
+Definition-supplied commands run only through the subject's `run_command` tool. Grading never runs commands.
+
+## Declared Commands
+
+A body group that must let the subject run tests or other programs lists the `run_command` tool and declares each allowed command:
+
+```json
+{
+  "tools": ["read", "edit", "run_command"],
+  "commands": [
+    { "id": "unit-tests", "argv": ["node", "--test"], "timeout_ms": 120000 },
+    { "id": "lint", "argv": ["npm", "run", "lint"] }
+  ]
+}
+```
+
+- `run_command` and `commands` require each other and are valid only in body groups.
+- Each command has a unique `id` (group-ID syntax), a non-empty `argv`, and optional integer `timeout_ms` from 1 to 600000 (default 120000).
+- The subject chooses only a declared `id`; it cannot change `argv`. The argv runs without a shell, in the workspace, without evaluator control variables.
+- Output is capped per stream and marked when truncated. A nonzero exit or timeout is an ordinary tool result, not a tool error.
+- Grade calls with `tool-call` expectations, for example `tool: "run_command"` with `argumentContains: ["unit-tests"]`.
+
+Commands are trusted author input with the evaluator's host permissions, like extension bundles; the subject only selects among them.
 
 ## Suite
 

package/skills/evaluate-skill/references/evaluation-design.md

@@ -2,16 +2,7 @@
 
 Read this before choosing or materially changing the behavioral question, evidence class, group shape, pressure, or baseline/candidate boundary.
 
-## Keep Roles Distinct
-
-- **Subject:** performs the declared task under one variant.
-- **Evaluator:** isolates subjects and preserves canonical evidence.
-- **Deterministic grader:** derives fixed mechanical facts after run persistence.
-- **Reviewer:** the active agent or user judges unresolved meaning.
-- **Author:** uses Write Skill to revise one measured pressure point.
-- **User:** decides whether to revise, use, publish, or reject the skill.
-
-The evaluator does not revise the skill, launch an automatic semantic grader, or own readiness, promotion, or production status.
+This reference is judgment: adapt it to the question being answered. Its binding statements restate [Evaluate Skill](../SKILL.md) Rules or the definition schema.
 
 ## Map Claims To Evidence
 
@@ -60,6 +51,8 @@ For a new skill, baseline has no target.
 
 For a revision, baseline names an exact immutable previous package. If no exact baseline exists, return to Write Skill before revision.
 
+To learn whether a skill's body, not just its description, changes behavior, add a separate group whose baseline is a neutral package: the same name and description with a placeholder body. A candidate that matches the neutral baseline gains nothing from its body under that pressure.
+
 For a description-only revision, body and resources remain byte-identical.
 
 Freeze prompts or turns, fixture, tools, model, thinking, other skills, context, runtime, and criteria across variants.
@@ -101,12 +94,19 @@ A useful group:
 - distinguishes baseline from candidate behavior;
 - remains realistic enough that passing behavior is useful.
 
-If both variants pass, the pressure may be weak or the baseline sufficient.
+Choose the pressure by the kind of instruction under test:
+
+- **A Rule:** make breaking it the easy or rewarded path, such as urgency, a user request that conflicts with it, or a shortcut that saves work. The candidate passes only if the Rule holds.
+- **Judgment:** include at least one case where following the default literally is the worse choice, next to a case where the default is right. The candidate passes on the outcome in both, so a variant that only obeys the default cannot pass.
+
+## Grade By Requirement And Respect Variance
+
+Grade each requirement a group checks, not only pass or fail. Label each as derivable from the request and code, dependent on a clarification, or imposed by the evaluator (an exact name or message only the checker knows); a failure on an evaluator-imposed requirement says little about the skill.
 
-If both fail, classify skill, fixture, dependency, environment, host, isolation, or criterion before editing.
+One run per cell cannot separate a difference of one or two items from run-to-run variance. Repeat the cell, or report such a difference as unresolved.
 
-## Reuse Existing Evidence
+Freeze the candidate before a batch runs. Items used to develop a revision are not evidence for it: judge it on held-out items, and stop iterating on the items it was tuned against.
 
-When saved canonical evidence already answers the fixed question, inspect it through views and raw artifacts. Do not rerun merely to demonstrate process.
+## Keep Separate Conclusions Separate
 
 An explicitly delivered body may support first-read behavior while a natural prompt still fails activation. Preserve both conclusions; one does not repair the other.

package/skills/evaluate-skill/references/execution-and-evidence.md

@@ -1,14 +1,17 @@
 # Execution And Evidence
 
-Read this when operating `run|view`, resolving paths, interpreting states, or reasoning about Pi execution, isolation, persistence, cancellation, and safeguards.
+Read this when operating `run|view|review`, resolving paths, interpreting states, or reasoning about Pi execution, isolation, persistence, cancellation, and safeguards.
+
+This reference is a contract: it states how the evaluator behaves. Rely on that behavior as written; its binding requirements on you restate [Evaluate Skill](../SKILL.md) Rules.
 
 ## Command And Path Contract
 
 Run commands from the repository or fixture root that owns the definitions:
 
 ```text
-node <evaluate-skill-directory>/scripts/skill-eval.mjs run <suite-or-group-path> [--group <id-or-position>] [--variant baseline|candidate]
+node <evaluate-skill-directory>/scripts/skill-eval.mjs run <suite-or-group-path> [--group <id-or-position>] [--variant baseline|candidate] [--trials <n>] [--max-cost <total>]
 node <evaluate-skill-directory>/scripts/skill-eval.mjs view <result-id-or-directory> [--group <id-or-position>] [--variant baseline|candidate]
+node <evaluate-skill-directory>/scripts/skill-eval.mjs review <result-id-or-directory> --model <provider/model> [--group <id-or-position>] [--thinking <level>]
 ```
 
 - The current working directory is the definition root and result root.
@@ -33,9 +36,9 @@ With no selectors, `run` or `view` selects every suite group and both variants.
 
 ## Current Execution Boundary
 
-`run` executes description and body groups. Unsupported group types are rejected before subject execution. Description groups allow no tools or `read`; body groups allow no tools or path-guarded `read`, `write`, and `edit`, plus non-native custom tools supplied by a declared runtime extension bundle. Native `bash`, `powershell`, `grep`, `find`, and `ls` tools and definition-supplied commands are unsupported.
+`run` executes description and body groups. Unsupported group types are rejected before subject execution. Description groups allow no tools or `read`; body groups allow no tools or path-guarded `read`, `write`, and `edit`, the evaluator's `run_command` tool for declared commands, plus non-native custom tools supplied by a declared runtime extension bundle. Native `bash`, `powershell`, `grep`, `find`, and `ls` tools are unsupported.
 
-A runtime profile selects the installed `pi` or `piflow` host, whether the variant uses an isolated persistent session directory, zero or more ordered extension bundles, and declarative environment sources. Literal values are non-secret configuration; only explicitly inherited parent variables are passed to the child without being persisted, alongside a small host-runtime baseline. Missing inherited variables invalidate the variant. Launch-control and loader keys are rejected from inherited values, and literal and inherited names must not overlap. For body runs, delivery remains `unavailable` until target setup succeeds; `natural-prompt` or `explicit-skill-command` records a prompt attempt. When a group declares context assertions, the evaluator launches the base guard first, declared extensions second, and a final non-mutating observer last. The observer records both the effective `before_agent_start` system prompt and each provider-neutral `context` message projection; it does not replace or sanitize extension injections after the base guard has established evaluator-owned isolation.
+A runtime profile selects the installed native Pi host, the `isolated` or `host` prompt mode, whether the variant uses an isolated persistent session directory, zero or more ordered extension bundles, and declarative environment sources. Literal values are non-secret configuration; only explicitly inherited parent variables are passed to the child without being persisted, alongside a small host-runtime baseline. Missing inherited variables invalidate the variant. Launch-control and loader keys are rejected from inherited values, and literal and inherited names must not overlap. For body runs, delivery remains `unavailable` until target setup succeeds; `natural-prompt` or `explicit-skill-command` records a prompt attempt. The evaluator launches the base guard first, its command tool when the group declares commands, declared extensions next, and, when a group declares context assertions, a final non-mutating observer last. The observer records both the effective `before_agent_start` system prompt and each provider-neutral `context` message projection; it does not replace or sanitize extension injections after the base guard has established evaluator-owned isolation.
 
 Use fresh JSON-mode execution for one-shot descriptions. Use one persistent RPC process per selected variant for ordered description turns and all body groups. RPC correlates responses, waits for `agent_settled`, disables automatic retry and compaction, preserves directly observed partial-turn evidence, and cleans the process tree.
 
@@ -85,6 +88,24 @@ Use ordinary file tools for raw `run.json`, events, transcript, final response,
 
 ## Safeguards And Limits
 
-Normal completion follows settlement. Path guards, no-progress detection, cancellation, process-tree cleanup, and very high emergency ceilings stop runaway or unsafe infrastructure. Do not impose ordinary guessed turn, token, spend, output, or short time caps.
+Normal completion follows settlement. Path guards, no-progress detection, cancellation, process-tree cleanup, and very high emergency ceilings stop runaway or unsafe infrastructure. Subjects run until they settle, with no turn, token, output, or short time cap, because a guessed cap cuts off the behavior the evaluation is trying to observe.
+
+`--max-cost <total>` is an explicit spend ceiling, not a subject cap. After each subject finishes, its host-reported `usage.cost.total` is added to the invocation's spend; once spend reaches the ceiling, no further subject starts and queued variants become `cancelled`. The subject already running completes, so spend can exceed the ceiling by one subject. The summary records `budget` with the ceiling, spend, and whether it was exhausted. A host that reports no cost never exhausts the ceiling.
+
+## Trials
+
+`--trials <n>` runs the selection as `n` independent complete invocations. Each trial is an ordinary result with its own ID. A separate aggregate under `.skill-eval/runs/<aggregate-id>/aggregate.json` lists the trials and, per group, counts pass, fail, and unavailable for every deterministic check and comparison, plus comparison transitions. `--max-cost` is shared across trials, and exhaustion stops starting further trials. `view <aggregate-id>` renders the counts; view an individual trial for its evidence.
+
+## Advisory Review
+
+`review` answers each group's `review_questions` with a separate reviewer model:
+
+- only groups with review questions and two complete runs are reviewed;
+- the reviewer sees the task turns, each run's responses, tool calls, and changed paths as Run A and Run B in random order, and never sees variant names or deterministic grades;
+- it runs without tools, skills, extensions, context files, or session, with only the base process environment;
+- `groups/<id>/semantic-grade.json` records the label mapping, prompt hash, answers mapped back to variants, and errors; `groups/<id>/review/` keeps the prompt, events, final response, and stderr;
+- malformed or incomplete reviewer output becomes `review-error` with the raw response preserved.
+
+A review is advisory. It never changes run or deterministic grade evidence, and rerunning it replaces only the previous advisory review.
 
 When an unsupported operation changes the question, reject it or run a clearly separate direct comparison with explicit limits. Never invoke archived evaluators as a fallback.

package/skills/evaluate-skill/references/review-and-revision.md

@@ -2,6 +2,8 @@
 
 Read this after canonical evidence and deterministic grades exist, when evidence conflicts, meaning remains unresolved, the failed boundary must be classified, or revision and rerun scope must be chosen.
 
+This reference is judgment: adapt it to the evidence in hand. Its rerun and no-edit requirements restate [Evaluate Skill](../SKILL.md) Rules.
+
 ## Use The Strongest Evidence
 
 Prefer, in order:
@@ -76,7 +78,7 @@ After a measured skill revision or resolved infrastructure cause:
 - do not use partial evidence as a comparable full result;
 - do not rerun merely to erase failure.
 
-For variance, use independent complete invocations rather than repeated preferred variants.
+For variance, use `--trials` to run independent complete invocations rather than repeated preferred variants.
 
 ## Report
 

package/skills/evaluate-skill/scripts/lib/definitions.mjs

@@ -2,9 +2,12 @@ import { readFile, realpath } from "node:fs/promises";
 import path from "node:path";
 
 const ID = /^[a-z][a-z0-9]*(?:-[a-z0-9]+)*$/;
-const EVALUATION_TYPES = new Set(["description", "body", "end-to-end"]);
+const EVALUATION_TYPES = new Set(["description", "body"]);
 const VARIANTS = ["baseline", "candidate"];
-const EVALUATION_HOSTS = new Set(["pi", "piflow"]);
+const EVALUATION_HOSTS = new Set(["pi"]);
+const PROMPT_MODES = new Set(["isolated", "host"]);
+export const COMMAND_TOOL = "run_command";
+const MAX_COMMAND_TIMEOUT_MS = 10 * 60 * 1000;
 const RESERVED_ENVIRONMENT_KEYS = new Set([
   "PATH",
   "HOME",
@@ -15,8 +18,6 @@ const RESERVED_ENVIRONMENT_KEYS = new Set([
   "NODE_OPTIONS",
   "NODE_PATH",
   "PI_PACKAGE_DIR",
-  "PIFLOW_CODING_AGENT_DIR",
-  "PIFLOW_CODING_AGENT_SESSION_DIR",
 ]);
 
 export class DefinitionError extends Error {
@@ -147,12 +148,15 @@ function validateRuntime(value, definitionPath) {
   const runtime = requireExactKeys(
     value,
     ["host", "extensions", "environment", "session"],
-    [],
+    ["prompt"],
     "group.runtime",
     definitionPath,
   );
   if (!EVALUATION_HOSTS.has(runtime.host)) {
-    fail("invalid-definition", "group.runtime.host must be pi or piflow", definitionPath);
+    fail("invalid-definition", "group.runtime.host must be pi", definitionPath);
+  }
+  if (Object.hasOwn(runtime, "prompt") && !PROMPT_MODES.has(runtime.prompt)) {
+    fail("invalid-definition", "group.runtime.prompt must be isolated or host", definitionPath);
   }
   if (!Array.isArray(runtime.extensions)) {
     fail("invalid-definition", "group.runtime.extensions must be an array", definitionPath);
@@ -253,6 +257,42 @@ function validateExpectations(value, definitionPath) {
   }
 }
 
+// Declared commands are fixed by the definition author; the subject may only choose which one runs.
+function validateCommands(group, definitionPath) {
+  const toolDeclared = group.tools.includes(COMMAND_TOOL);
+  if (!Object.hasOwn(group, "commands")) {
+    if (toolDeclared) fail("invalid-definition", `${COMMAND_TOOL} requires declared group.commands`, definitionPath);
+    return;
+  }
+  if (!toolDeclared) fail("invalid-definition", `group.commands requires the ${COMMAND_TOOL} tool`, definitionPath);
+  if (group.type !== "body")
+    fail("invalid-definition", "group.commands are supported only by body groups", definitionPath);
+  if (!Array.isArray(group.commands) || group.commands.length === 0) {
+    fail("invalid-definition", "group.commands must be a non-empty array", definitionPath);
+  }
+  const ids = new Set();
+  for (const [index, entry] of group.commands.entries()) {
+    const label = `group.commands[${index}]`;
+    const command = requireExactKeys(entry, ["id", "argv"], ["timeout_ms"], label, definitionPath);
+    requireId(command.id, `${label}.id`, definitionPath);
+    if (ids.has(command.id))
+      fail("invalid-definition", `group.commands contains duplicate ID ${command.id}`, definitionPath);
+    ids.add(command.id);
+    const argv = requireStringArray(command.argv, `${label}.argv`, definitionPath);
+    if (argv.length === 0) fail("invalid-definition", `${label}.argv must not be empty`, definitionPath);
+    if (
+      Object.hasOwn(command, "timeout_ms") &&
+      (!Number.isInteger(command.timeout_ms) || command.timeout_ms < 1 || command.timeout_ms > MAX_COMMAND_TIMEOUT_MS)
+    ) {
+      fail(
+        "invalid-definition",
+        `${label}.timeout_ms must be an integer from 1 to ${MAX_COMMAND_TIMEOUT_MS}`,
+        definitionPath,
+      );
+    }
+  }
+}
+
 function validateModel(value, definitionPath) {
   const model = requireExactKeys(value, ["model"], ["thinking"], "group.model", definitionPath);
   requireString(model.model, "group.model.model", definitionPath);
@@ -274,7 +314,7 @@ function validateGroup(value, definitionPath) {
       "expectations",
       "review_questions",
     ],
-    ["model", "runtime"],
+    ["model", "runtime", "commands"],
     "group",
     definitionPath,
   );
@@ -282,11 +322,12 @@ function validateGroup(value, definitionPath) {
   if (group.kind !== "group") fail("invalid-definition", "group.kind must be group", definitionPath);
   requireId(group.id, "group.id", definitionPath);
   if (!EVALUATION_TYPES.has(group.type)) {
-    fail("invalid-definition", "group.type must be description, body, or end-to-end", definitionPath);
+    fail("invalid-definition", "group.type must be description or body", definitionPath);
   }
   validateInput(group.input, definitionPath);
   if (group.fixture !== null) requireSafeRelativePath(group.fixture, "group.fixture", definitionPath);
   requireStringArray(group.tools, "group.tools", definitionPath, { unique: true });
+  validateCommands(group, definitionPath);
   if (Object.hasOwn(group, "runtime")) validateRuntime(group.runtime, definitionPath);
   const variants = requireExactKeys(group.variants, VARIANTS, [], "group.variants", definitionPath);
   for (const variant of VARIANTS) validateEnvironment(variants[variant], `group.variants.${variant}`, definitionPath);

package/skills/evaluate-skill/scripts/lib/pi.mjs

@@ -30,8 +30,9 @@ import {
 } from "./sandbox.mjs";
 
 const guardExtension = fileURLToPath(new URL("../pi-guard.mjs", import.meta.url));
+const commandToolExtension = fileURLToPath(new URL("../pi-command-tool.mjs", import.meta.url));
 const observerExtension = fileURLToPath(new URL("../pi-observer.mjs", import.meta.url));
-const HOST_COMMANDS = { pi: "pi", piflow: "piflow" };
+const HOST_COMMANDS = { pi: "pi" };
 const BASE_PROCESS_ENVIRONMENT_KEYS = [
   "PATH",
   "HOME",
@@ -43,8 +44,6 @@ const BASE_PROCESS_ENVIRONMENT_KEYS = [
   "XDG_CONFIG_HOME",
   "XDG_DATA_HOME",
   "XDG_CACHE_HOME",
-  "PIFLOW_CODING_AGENT_DIR",
-  "PIFLOW_CODING_AGENT_SESSION_DIR",
   "PI_PACKAGE_DIR",
   "PI_OFFLINE",
 ];
@@ -150,6 +149,7 @@ function subjectRun({ group, variant }, subject, observation) {
     completedAt: new Date().toISOString(),
     workspace: subject.workspace,
     tools: [...group.tools],
+    commands: group.commands ?? [],
     model: {
       declared: group.model ?? null,
       observed: observation.model,
@@ -329,9 +329,7 @@ function createProcessEnvironment({
   contextObservationPath,
   requiresContextObservation,
 }) {
-  const processEnvironment = Object.fromEntries(
-    BASE_PROCESS_ENVIRONMENT_KEYS.flatMap((key) => (process.env[key] === undefined ? [] : [[key, process.env[key]]])),
-  );
+  const processEnvironment = baseProcessEnvironment();
   Object.assign(processEnvironment, {
     PWD: workspace,
     ...runtime.environment.literal,
@@ -342,7 +340,9 @@ function createProcessEnvironment({
     SKILL_EVAL_ALLOWED_TOOLS: JSON.stringify(group.tools),
     SKILL_EVAL_CONTEXT_MANIFEST: contextManifestPath ?? "",
     SKILL_EVAL_HOST: runtime.host,
+    SKILL_EVAL_PROMPT_MODE: runtime.prompt,
   });
+  if (group.commands !== undefined) processEnvironment.SKILL_EVAL_COMMANDS = JSON.stringify(group.commands);
   for (const key of runtime.environment.inherit) {
     if (process.env[key] === undefined) {
       throw new VariantSetupError(`inherited runtime environment variable is unavailable: ${key}`);
@@ -353,6 +353,12 @@ function createProcessEnvironment({
   return processEnvironment;
 }
 
+export function baseProcessEnvironment() {
+  return Object.fromEntries(
+    BASE_PROCESS_ENVIRONMENT_KEYS.flatMap((key) => (process.env[key] === undefined ? [] : [[key, process.env[key]]])),
+  );
+}
+
 async function readContextObservations(file, required) {
   const contents = await readFile(file, "utf8").catch((error) => {
     if (error?.code === "ENOENT") {
@@ -426,6 +432,7 @@ function piArguments(group, environment, runtime, mode, observeContext, sessionD
   if (runtime.session) args.push("--session-dir", sessionDirectory);
   else args.push("--no-session");
   args.push("--no-extensions", "--extension", guardExtension);
+  if (group.commands !== undefined) args.push("--extension", commandToolExtension);
   for (const bundle of runtime.extensions) args.push("--extension", bundle.entry);
   if (observeContext) args.push("--extension", observerExtension);
   args.push("--no-skills", "--no-prompt-templates", "--no-themes", "--no-context-files", "--no-approve");
@@ -438,7 +445,7 @@ function piArguments(group, environment, runtime, mode, observeContext, sessionD
   return args;
 }
 
-async function runPiProcess({ command, args, cwd, eventsFile, stderrFile, signal, environment }) {
+export async function runPiProcess({ command, args, cwd, eventsFile, stderrFile, signal, environment }) {
   const child = spawn(command, args, {
     cwd,
     detached: process.platform !== "win32",