@hassangameryt/freeflow 0.7.3 → 0.9.0

package/skills/evaluate-behavior/references/authoring.md

@@ -0,0 +1,381 @@
+# Authoring Definitions
+
+This reference is a contract: the shapes and constraints below are binding as written, for schema-version-2 group, suite, and configuration JSON. The loader checks them: it refuses a definition that breaks one when it loads, before any subject runs, by `run` and `check-tasks` alike, and the refusal names the group, its file and the field. Run `skill-eval check-tasks <path>`, which calls no model, after writing a definition and fix what it names; field rules it checks appear below as a pointer, and what stays is what its errors cannot teach: meanings, defaults and choices. Only an expectation of an unknown kind loads and surfaces afterward, as `grade-error` evidence.
+
+Mode-specific fields are with their mode's guidance: [coding tasks](coding-tasks.md) (command graders, reference solutions, declared commands, shell, container, network, and the Claude Code host) and [conversations, pipelines and judges](conversations-pipelines-judges.md) (participants, rubric and judges, runtime profile). Commands, path resolution, run states, persistence, views, isolation, and safeguards are in [running and evidence](running-and-evidence.md).
+
+## Definition Root
+
+Run `skill-eval` from the repository or fixture root that owns the evaluation inputs.
+
+| Input | Resolves from |
+| --- | --- |
+| `run <suite-or-group-path>` | current working directory |
+| suite `groups` entries | directory containing the suite file |
+| group `fixture` | definition root/current working directory |
+| `configuration.extends` | directory containing the group file |
+| subject `skills` and `context` | definition root/current working directory, then the selected working tree or Git commit |
+
+Paths are relative and stay inside the root; the loader checks both.
+
+## Common Group Shape
+
+A group is a **task** (what the agent is asked to do and how it is graded) and a **configuration** (the agent environment that does it). Two or more named arms, usually `baseline` and `candidate`, each overlay the shared configuration, and `component` names the only places where an arm may differ from the reference arm; a group with one arm measures one setup and needs no `component`. Every group contains exactly these fields, plus optional `reference`, `commands` and `reference_solution` ([coding tasks](coding-tasks.md)), `revision`, `participants`, and the judged-evidence fields `rubric`, `judges` and `combine` ([conversations, pipelines and judges](conversations-pipelines-judges.md)):
+
+```json
+{
+  "schema_version": 2,
+  "kind": "group",
+  "id": "one-behavioral-question",
+  "type": "description",
+  "input": { "prompt": "Natural task prompt." },
+  "fixture": null,
+  "tools": ["read"],
+  "configuration": {
+    "host": { "name": "pi" },
+    "model": { "model": "xai/grok-4.7", "thinking": "low" },
+    "subject": { "kind": "skill", "source": { "kind": "working-tree" }, "context": [] }
+  },
+  "arms": {
+    "baseline": { "subject": { "skills": [], "target": null } },
+    "candidate": { "subject": { "skills": ["skills/target"], "target": 0 } }
+  },
+  "component": ["subject.skills", "subject.target"],
+  "expectations": [
+    { "id": "baseline-read", "kind": "skill-read", "variant": "baseline", "expect": "by-turn", "turn": 1, "comparison": "target-read" },
+    { "id": "candidate-read", "kind": "skill-read", "variant": "candidate", "expect": "by-turn", "turn": 1, "comparison": "target-read" }
+  ],
+  "review_questions": []
+}
+```
+
+The loader checks every field above (`schema_version` 2, `kind`, the `id` pattern, `type`, exactly one of `input.prompt` and `input.turns`, a contained `fixture` directory, unique `tools` and expectation IDs, at least one expectation, review question or rubric dimension) and names the one it refuses. What it cannot teach:
+
+- `input.turns` may repeat a natural turn. The `fixture` directory's contents become the workspace.
+- `input.resume_before` lists turns (from 2) before which the subject's Pi session is closed and resumed from its own file (Pi's RPC `switch_session`, so extensions see `session_shutdown`, then `session_start` with reason `resume`), as a user resuming it would; it needs a body group on `pi` with `configuration.runtime.session: true` and no participants. Each resumed turn records `resumed.sessionFile` in `run.json`.
+- `review_questions` are rendered for the reviewer and never sent to the subject. The reviewer sees every arm's run as Run A, Run B, Run C, ... in a random order, must answer every question for every label, and its answers are mapped back to arm names; `review --swap` repeats the review with the order reversed and records agreement per question, and per arm when the arms are not `baseline` and `candidate`.
+- A group with only review questions has no deterministic checks; its evidence comes from the advisory review alone. A group with only a rubric is graded by its judges alone.
+- `revision: "description-only"`, for a description group whose two arms both name a target: before any subject runs, the evaluator snapshots both targets and refuses the group as `invalid` unless every file other than `SKILL.md` is byte-identical and `SKILL.md` differs only in its frontmatter `description`.
+- Tools: a description group takes none or `read` (`skill-read` is its activation check); a body group takes `read`, `write`, `edit`, `ls`, `find`, `grep`, `run_command` for [declared commands](coding-tasks.md#declared-commands), and custom tools from a runtime extension bundle; `bash` needs a [shell](coding-tasks.md#shell) or a [container](coding-tasks.md#container-environment).
+
+## Configuration And Arms
+
+A configuration has these fields; each arm's effective configuration is the shared `configuration` with that arm's overlay merged in:
+
+| Field | Required in the effective configuration | Meaning |
+| --- | --- | --- |
+| `host` | yes | `pi` or `claude-code`, with optional `version`, `extensions` and `settings`; see [Host Extensions And Settings](#host-extensions-and-settings) and [the Claude Code host](coding-tasks.md#claude-code-host). |
+| `subject` | yes | What is under test: a `skill` (see [Subject](#subject)), or `none` when the comparison is about the environment, not a skill. |
+| `model` | no | `model` names the provider/model; optional `thinking` selects its thinking level. The CLI has no model override, and a subject run without a model gets Pi's default and no copied login. |
+| `runtime` | no | Host session, prompt mode, extension bundles and environment; see [Runtime Profile](conversations-pipelines-judges.md#runtime-profile). |
+| `shell` | no | A confined `bash` for the subject; see [Shell](coding-tasks.md#shell). |
+| `environment` | no | A container the subject's shell and the command graders run in; see [Container Environment](coding-tasks.md#container-environment). |
+| `network` | no | What code the subject could have written may reach: `"none"` (sealed) or `"host"`; see [Network](coding-tasks.md#network). |
+| `participants` | no | More than one participant in each arm: a conversation with a simulated user, or a pipeline of stages; shared, or different per arm when `component` names it; see [Participants](conversations-pipelines-judges.md#participants). |
+| `freeflow` | no | Contributor-only (needs a Freeflow repository as the definition root): Freeflow as part of the environment; see [Freeflow](#freeflow). |
+
+The loader checks each field's shape (host names, the arm-name pattern, `reference`, `component`, `extends`) and names the one it refuses. What it cannot teach:
+
+- `configuration.extends` names a configuration file, relative to the group file; its configuration is merged first, then the group's own fields.
+- Each arm overlays the shared configuration: objects merge key by key; an array, scalar, or `null` replaces the shared value.
+- `reference` names the arm the others are compared with; it defaults to `baseline`. Evidence, views and the aggregate list the reference first, then the other arms in declaration order.
+- `component` lists dot paths into the configuration, such as `subject.source` or `model.thinking`: the only places an arm may differ from the reference, compared leaf by leaf (an array is one leaf). The loader refuses an arm that differs elsewhere, and a `component` path no arm differs in. `component: []` declares an A/A comparison (or A/A/A): every arm identical, which measures run-to-run variation.
+- **One arm** measures one setup: it is its own reference, with no `component` and no `comparison`. Checks, pass^k, metrics, judged scores (`review --rubric`), reference solutions and command graders work as usual; the aggregate's claim is a rate on the suite, and a suite `decision` or a pairwise `review` refuses it. Two or more arms need `component`.
+- `host.name`, `host.version` and `runtime` are the same in every arm, except `runtime.environment.literal` (see [Runtime Profile](conversations-pipelines-judges.md#runtime-profile)); `host.extensions`, `host.settings`, `shell`, `environment` and `freeflow` may differ, as the component under test.
+
+A group with exactly `baseline` and `candidate`, and `baseline` as the reference, keeps the evidence shapes it always had. Any other group records comparisons, transitions and pairs keyed by arm name. An A/A/B group, where `same` repeats the reference and `changed` adds the skill:
+
+```json
+{
+  "arms": {
+    "baseline": {},
+    "same": {},
+    "changed": { "subject": { "skills": ["skills/target"], "target": 0 } }
+  },
+  "component": ["subject.skills", "subject.target"]
+}
+```
+
+## Body Group Example
+
+A body group explicitly delivers each target on turn one and observes first-read or later-turn behavior:
+
+```json
+{
+  "schema_version": 2,
+  "kind": "group",
+  "id": "retained-artifact-behavior",
+  "type": "body",
+  "input": {
+    "turns": [
+      "Inspect the task and prepare the work.",
+      "Complete it and write result.json."
+    ]
+  },
+  "fixture": "fixtures/task",
+  "tools": ["read", "write", "edit"],
+  "configuration": {
+    "host": { "name": "pi" },
+    "model": { "model": "xai/grok-4.7", "thinking": "medium" },
+    "subject": {
+      "kind": "skill",
+      "skills": ["skills/support", "skills/target"],
+      "target": 1,
+      "context": ["context/contract.md"]
+    }
+  },
+  "arms": {
+    "baseline": { "subject": { "source": { "kind": "git", "ref": "before-body-change" } } },
+    "candidate": { "subject": { "source": { "kind": "working-tree" } } }
+  },
+  "component": ["subject.source"],
+  "expectations": [
+    {
+      "id": "baseline-result",
+      "kind": "path",
+      "variant": "baseline",
+      "path": "result.json",
+      "turn": 2,
+      "expect": "exists",
+      "comparison": "result-exists"
+    },
+    {
+      "id": "candidate-result",
+      "kind": "path",
+      "variant": "candidate",
+      "path": "result.json",
+      "turn": 2,
+      "expect": "exists",
+      "comparison": "result-exists"
+    },
+    {
+      "id": "candidate-status",
+      "kind": "json",
+      "variant": "candidate",
+      "path": "result.json",
+      "turn": 2,
+      "expect": "field-equals",
+      "pointer": "/status",
+      "value": "complete"
+    },
+    {
+      "id": "candidate-changes",
+      "kind": "changed-paths",
+      "variant": "candidate",
+      "turn": 2,
+      "expect": "equals",
+      "paths": ["result.json"]
+    }
+  ],
+  "review_questions": [
+    "Did the second turn apply the first-turn guidance without redelivery?"
+  ]
+}
+```
+
+The examples are complete definition shapes. Replace fixture, resource, model, and Git-ref values with paths and identities that exist under your definition root.
+
+## Host Extensions And Settings
+
+```json
+{
+  "host": {
+    "name": "pi",
+    "extensions": [{ "package": "@gotgenes/pi-anthropic-auth", "version": "3.4.2" }],
+    "settings": { "retry": { "enabled": false }, "compaction": { "enabled": true } }
+  }
+}
+```
+
+- `extensions` lists Pi packages by exact name and version, found among the user's installed Pi packages (`<Pi agent directory>/npm/node_modules`). They load right after the evaluator's guard, in order, and are recorded under `resources.hostExtensions`. Extensions can change what the host sends (an auth extension may reshape the system prompt), so they belong to the configuration, and arms that differ in them must declare it in `component`.
+- `settings` is any object of Pi settings. It becomes the subject's `settings.json` in its isolated agent directory, merged over the evaluator's two defaults: `retry.enabled` and `compaction.enabled` are `false` unless declared, so a provider error or a compaction never happens silently inside a measured run. Any other setting it does not name takes Pi's default; the user's own settings never apply. `run.json` records the effective settings under `isolation.settings`, and body runs send the same retry and compaction values to Pi over RPC. Extensions load only from `extensions`.
+- An extension can also be built from source, so arms can compare versions of an extension under development:
+
+  ```json
+  {
+    "extensions": [
+      { "source": { "kind": "git", "ref": "3f2a9c1", "path": "packages/my-extension" }, "build": ["npm", "run", "build"], "entry": "dist/index.js" },
+      { "path": "extensions/local-probe", "entry": "index.mjs" }
+    ]
+  }
+  ```
+
+  - `source` takes a directory (`path`, or `"."`) at an exact commit (a branch or tag is refused, because it moves) of the Git repository holding the definition root, extracted with `git archive`. `path` instead takes a directory under the definition root from the working tree, snapshotted, never used live.
+  - Optional `build` runs once in the extracted copy, without a shell, in an isolated home, temp and Pi directory with no copied login, with a 10-minute timeout. A build that fails, times out or cannot start makes the group `invalid` before any subject runs.
+  - `entry` is the file Pi loads, relative to the built directory.
+  - Each distinct source and build is built once per group, under `resources/host-extensions/<n>/source` in the group directory. `resources.hostExtensions` records the source (with the resolved `commit` for a Git source), the entry, a `sha256` over the built directory's file list and contents, and the build's argv, exit code, duration and output tails.
+- The advisory reviewer takes the same fields from a configuration file given with `review --configuration`.
+
+## Subject
+
+A `skill` subject:
+
+```json
+{
+  "kind": "skill",
+  "source": { "kind": "git", "ref": "v1.2.3" },
+  "skills": ["skills/target", "skills/support"],
+  "target": 0,
+  "context": ["context/base-guidance.md"]
+}
+```
+
+A `none` subject is `{ "kind": "none" }`. `source` is `{ "kind": "working-tree" }` or `{ "kind": "git", "ref": "<exact-ref>" }`.
+
+A source applies only to declared skill and context paths in that arm. It never supplies the group fixture: `fixture` is snapshotted once from the definition-root working tree, then copied independently for selected variants. `skills` preserves registration order, and `target` is a zero-based index into it (a non-reference arm that declares skills names one). `context` preserves declaration order and may name files or directories.
+
+## Expectations
+
+Every expectation has an `id`, a `kind` from the table below, a `variant` (the arm it checks), an optional `comparison` (see [Comparison](#comparison)) and an optional one-based `turn`. The loader runs each kind's own grader check on the definition, so a field, value, arm or turn that grader would refuse is refused before any subject runs.
+
+Without `turn`, response checks use the final response and workspace checks use final state. Turn-scoped workspace checks cause an immutable workspace snapshot after each settled turn.
+
+| `kind` | Group type | Kind-specific fields | `expect` |
+| --- | --- | --- | --- |
+| `skill-read` | description | timing `turn` except for `never` | `never`, `on-turn`, `by-turn`, `not-before-turn` |
+| `resource-read` | either | `resource`, optional `index`, `path`, optional `turn` | `read`, `not-read` |
+| `path` | body | workspace `path`, optional `turn` | `exists`, `absent` |
+| `changed-paths` | body | unique workspace `paths`, optional `turn` | `equals`, `excludes` |
+| `file-text` | body | workspace `path`, string `value`, optional `turn`, optional `line` | `contains`, `not-contains`, `equals` |
+| `json` | body | workspace `path`, optional `turn`, conditional `pointer` and `value` | `available`, `missing`, `valid`, `malformed`, `field-present`, `field-absent`, `field-equals` |
+| `response-text` | body | string `value`, optional `turn` | `contains`, `not-contains`, `equals` |
+| `tool-call` | body | string `tool`, optional `turn`, optional `argumentContains`/`argumentNotContains` arrays | `called`, `succeeded`, `failed`, `not-called` |
+| `context-text` | body | string `value`, optional `surface` (`system-prompt`, the default, or `provider-context`), optional `request` (`first`, `last`, the default, or a positive request number within the turn), optional `turn` | `contains`, `not-contains`, `equals` |
+| `command` | body | `argv`, optional `files`, optional `timeout_seconds`, `pass`; no `turn` | none; see [command graders](coding-tasks.md#command-graders) |
+
+With `line` (1-based), a `file-text` expectation compares only that line, without its line ending. A line the file does not have fails every expectation, including `not-contains`, so an empty file cannot pass by having nothing to read. Use it to check one line, such as a review's first-line judgment, when the rest of the file may quote the same text.
+
+For `tool-call` with `expect: "not-called"`, omitted argument filters require zero calls to the named tool. Supplied filters scope the absence check to calls matching those argument predicates.
+
+### Resource paths
+
+- `resource: "workspace"`: omit `index`; `path` is relative to the writable workspace.
+- `resource: "skill"`: `index` selects the zero-based declared skill; `path` is relative to that skill directory.
+- `resource: "context"`: `index` selects the zero-based declared context item; `path` is relative to that materialized item.
+
+### Changed paths
+
+- `equals`: the complete changed-file set must equal `paths`.
+- `excludes`: none of `paths` may appear in the changed-file set.
+
+Directory-only presence does not count as a changed file.
+
+### JSON states
+
+```json
+[
+  {
+    "id": "file-missing",
+    "kind": "json",
+    "variant": "candidate",
+    "path": "missing.json",
+    "expect": "missing"
+  },
+  {
+    "id": "file-malformed",
+    "kind": "json",
+    "variant": "candidate",
+    "path": "broken.json",
+    "expect": "malformed"
+  },
+  {
+    "id": "field-absent",
+    "kind": "json",
+    "variant": "candidate",
+    "path": "result.json",
+    "expect": "field-absent",
+    "pointer": "/optional"
+  },
+  {
+    "id": "field-null",
+    "kind": "json",
+    "variant": "candidate",
+    "path": "result.json",
+    "expect": "field-equals",
+    "pointer": "/value",
+    "value": null
+  }
+]
+```
+
+### Comparison
+
+A comparison ID joins one expectation per arm, all the same check. For each non-reference arm the report records a factual transition from the reference, such as `fail-to-pass`, never a quality verdict. A `baseline`/`candidate` group records it as `transition`; any other group records `reference`, each arm's check and state under `arms`, and each non-reference arm's transition under `transitions`.
+
+## Suite
+
+```json
+{
+  "schema_version": 2,
+  "kind": "suite",
+  "id": "workflow-core",
+  "groups": [
+    "groups/natural-activation.json",
+    "groups/body-behavior.json"
+  ]
+}
+```
+
+A suite's `groups` run in order and resolve from the suite's directory; `decision` is optional. The loader checks the rest.
+
+### Decision
+
+A suite may declare the rule its trials are judged by, before any run. Without one, an aggregate supports only a rate on the suite. A direct group run is one case and never yields "beats".
+
+```json
+"decision": {
+  "measure": { "comparison": "target-read" },
+  "arms": ["baseline", "candidate"],
+  "direction": "higher",
+  "method": "sign-test",
+  "alpha": 0.05,
+  "min_cases": 10,
+  "trials": { "initial": 3, "max": 9, "step": 3 }
+}
+```
+
+- `measure` names exactly one of: `comparison`, a comparison ID from the groups' expectations (a pass counts 1, a fail 0); `metric`, a name in the aggregate's `metrics` section; `judged`, a dimension ID in its `judged` section; or `score`, the comparison ID of test-graded command expectations, read from the `scores` section (an expectation's own ID scores its one arm only). Section values are read from `perTrial` (`true` counts 1, `false` 0, numbers as given). Null and unavailable values are excluded and counted.
+- `arms` names two arms, the reference first; every group needs both, and `--variant` is refused. The loader refuses a measure a group records in one decision arm only, or that no group records in both; a group that records it in neither arm, such as a control case, loads, and its case is excluded and counted.
+- `direction`: whether `higher` or `lower` is better.
+- `method` defaults to `sign-test`, `alpha` to 0.05, and `min_cases` to 10.
+- The unit is the case. Each arm's values over a case's trials are averaged, and the second arm is better, worse or equal. An exact two-sided sign test over the better and worse cases (ties and excluded cases are counted, not tested) gives the verdict that [design and measurement](design-and-measurement.md#fix-the-decision-rule-first) defines, with `min_cases` as the minimum and `alpha` as the level.
+- A suite with a decision always runs as trials and records an aggregate. `trials` is optional: without it, `--trials` sets the count (default 1). After `initial` trials, each case whose sign is not yet stable (the arms tie, or differ by less than one trial's worth: the observed range divided by the fewer values) runs up to `step` more trials of that group alone, until `max`. A case with an arm that has no usable value is not extended.
+
+What the aggregate records for a decision is in [running and evidence](running-and-evidence.md#trials).
+
+## Selection
+
+`--group <id | position>` and `--variant <arm>` narrow what runs (`skill-eval --help`); with neither, every group and arm runs. Excluded arms are `not-selected`, never failed, and the saved definitions do not change.
+
+## Freeflow
+
+For Freeflow contributors only: this field builds Freeflow itself with `scripts/freeflow-snapshot.mjs`, so it works only when the definition root is a Freeflow repository; elsewhere the group is refused before any subject runs. To compare a host extension, declare it under `host.extensions`.
+
+A configuration with `freeflow` runs the subject with Freeflow loaded, built from an exact commit:
+
+```json
+{
+  "configuration": {
+    "host": { "name": "pi" },
+    "model": { "model": "xai/grok-4.7", "thinking": "medium" },
+    "subject": { "kind": "none" },
+    "runtime": { "prompt": "host" },
+    "freeflow": {
+      "source": { "kind": "git", "ref": "<commit>" },
+      "repository": { "taskMemory": { "enabled": false } },
+      "personal": {}
+    }
+  },
+  "arms": {
+    "baseline": {},
+    "candidate": { "freeflow": { "repository": { "taskMemory": { "enabled": true } } } }
+  },
+  "component": ["freeflow.repository.taskMemory.enabled"]
+}
+```
+
+- `source` is a Git commit; the package is built from that commit with the definition root's `scripts/freeflow-snapshot.mjs`, once per distinct ref per group, into the group's `resources/freeflow/`. The user's development snapshot is never used or changed.
+- `repository` is written to the workspace's `.freeflow/config.json` and `personal`, when not empty, to `.freeflow/local.json`, before the workspace's starting state is taken, so neither counts as a subject change. Both default to `{}`, and that build's own validator checks them.
+- `runtime.prompt` is `host`, because the isolated prompt replaces the host's system prompt, which would remove Freeflow's own instructions.
+- Freeflow's extension loads right after the evaluator's guard, the package directory is readable by the subject, and Freeflow may register custom tools that `tools` then names.
+- `run.json` records the ref, resolved commit, package name and version, tarball hash, and the settings under `resources.freeflow`.

package/skills/evaluate-behavior/references/coding-tasks.md

@@ -0,0 +1,145 @@
+# Coding Tasks
+
+This reference holds a coding task's guidance and the definition fields it uses, and the Claude Code host's. The guidance is judgment; the field shapes and constraints are a contract, binding as written, like the rest of [authoring](authoring.md), whose loader checks them: run `skill-eval check-tasks` and fix the field it names.
+
+## Choose And Declare A Coding Task
+
+- **Answers:** does the agent's change work, judged by running it rather than reading it?
+- **Declare:** a body group with the repository as its fixture and tools to edit it; `command` expectations whose `argv` runs hidden tests, a linter or a build on a copy of the final workspace, with grader `files` kept outside the fixture and a `pass` rule (an exit code, or named required tests); declared `commands` and `run_command` when the subject should run tests itself; a `reference_solution` proving the task solvable; and a `shell` or a container `environment` only when the task needs `bash`, or a toolchain the host lacks.
+- **Grade both directions:** tests of the new behavior, which fail on the unchanged fixture and must pass after the change, and the existing tests the change must not break, which pass before and must still pass. Passing only the first rewards a change that breaks something else.
+- **Yields:** each command's exit code, output tails and per-test results, with the share of required tests passed as a score; a reference check per group.
+- **Pitfalls:**
+  - Run `check-tasks` first: it applies the reference solution and runs the graders with no model (Reference Solution, below). A task the reference fails measures the grader, not the agent; a task already solved measures nothing. For every group it also lists what the copy lacks against the setting the task came from: Git history (a workspace is never a repository), ignored files the fixture holds (a clone lacks them) or the task names (such as `.env`), and tools or network the task refers to but the subject lacks. Judge each gap before running: give the subject what the task needs, or reword the task.
+  - A required test name that matches nothing fails every arm; the reference check catches it.
+  - Grader files inside the fixture, or named in the prompt, leak the answer.
+  - A subject can make tests pass by editing them. List every test you rely on, existing ones included, in the grader `files`: a grader file replaces whatever the subject left at that path. A `changed-paths` expectation with `excludes` also shows whether the subject touched them.
+  - A container image is pinned by digest and never pulled by the evaluator; when it is missing, relay the refusal and ask the user before pulling. Declare `network: "none"` when the subject's code must not reach the network, the host's files or the login (Network, below).
+  - Grader commands run with the evaluator's host permissions, and run whatever the subject wrote. Run only tasks whose fixture and graders you trust.
+  - A subject without a shell runs only the declared commands, which is often enough. A shell needs a confining isolation that the evaluator probes (Shell, below); when it needs software this machine lacks, relay the refusal to the user and ask before installing anything.
+
+## Command Graders
+
+A `command` expectation grades a body run's final work by running a program on it, such as hidden tests or a linter:
+
+```json
+{
+  "id": "candidate-hidden-tests",
+  "kind": "command",
+  "variant": "candidate",
+  "comparison": "hidden-tests",
+  "argv": ["node", "--test", "--test-reporter=tap", "test/hidden.test.mjs"],
+  "files": { "test/hidden.test.mjs": "graders/calc/hidden.test.mjs" },
+  "timeout_seconds": 600,
+  "pass": { "tests": { "format": "node-test", "required": ["adds two numbers", "adds negatives"] } }
+}
+```
+
+- After the run, the final workspace is copied to `<arm>/commands/<id>/workspace`, each `files` entry is copied in (key: path in the copy; value: path under the definition root), and `argv` runs there without a shell, with the subject's isolated home, temp, XDG and Pi directories. The workspace itself is never changed, so file graders still verify its hashes.
+- `files` sources resolve from the definition root like `fixture`, never from inside the fixture, which the subject sees; they are inputs of the run, so an uncommitted grader file makes the run non-canonical. A destination may not pass through a symlink the subject left in the workspace.
+- `timeout_seconds` (default 600) stops the command's whole process tree. The loader checks these fields and names the one it refuses.
+- `pass` is one rule. `{ "exit_code": 0 }` passes on that exit code. `{ "tests": { "format", "required", "report" } }` passes when every `required` test passed: `node-test` and `tap` read TAP from stdout (give Node `--test-reporter=tap`, since newer Node versions default to another reporter), and `junit` reads the JUnit XML file at `report` in the copy, or stdout without one. A required name matches a test's own name or its full name, parents joined by ` > ` (for JUnit, `<classname> > <name>`); skipped, missing and failed tests do not pass. The check records `score`, the passed share of `required`. The aggregate's `scores` section gives each such check's score per arm (`count`, `unavailable`, `mean`, `median`) and per trial as `score.<expectation id>`, and also as `score.<comparison id>` when the expectation belongs to a comparison, so a suite decision can measure it with `{ "score": "<comparison id>" }`.
+- `run.json` records each command under `postRun.commands.<id>`: exit code, signal, timeout, duration, the last 20,000 characters of stdout and stderr, and parsed test results. A run that did not complete, or a command that could not start, grades `unavailable`; a timeout fails.
+
+Commands are trusted author input with the evaluator's host permissions; whatever the subject wrote runs with them too when the command executes it.
+
+## Reference Solution
+
+`reference_solution` is optional: `{ "patch": "<file>" }`, a unified diff applied to the fixture copy with `git apply`, or `{ "files": "<dir>" }`, whose regular files overlay the fixture copy; both are inputs of the run, kept outside the fixture. The reference is one workspace shared by every arm, so `skill-eval check-tasks` grades each arm's workspace and command expectations with no model, reporting per arm: `solvable` only when every arm's applicable expectations pass on the reference and fail before it, `unsolvable` with the failing graders named per arm, `already-solved` when an arm's expectations all pass on the unchanged fixture (the task cannot show a change), or `not-checkable` when a check cannot run (interaction expectations need a transcript). The same graders run on the unchanged fixture: each arm records `before` with every check's state there and, for test graders, the required tests that fail before and pass on the reference (`failToPass`) and the ones that pass both times (`passToPass`). A pass-to-pass test is fine as long as some required test or the exit-code rule fails before. An optional `"arm"` restricts the check to one arm. `run` refuses an unsolvable or already-solved group before any subject starts (`--skip-reference-check` overrides); the result records the check in `groups/<id>/reference-check.json`.
+
+## Declared Commands
+
+A body group that must let the subject run tests or other programs lists the `run_command` tool and declares each allowed command:
+
+```json
+{
+  "tools": ["read", "edit", "run_command"],
+  "commands": [
+    { "id": "unit-tests", "argv": ["node", "--test"], "timeout_ms": 120000 },
+    { "id": "lint", "argv": ["npm", "run", "lint"] }
+  ]
+}
+```
+
+- The loader checks the pairing (`run_command` with `commands`, body groups only) and each command's `id`, `argv` and `timeout_ms` (default 120000).
+- The subject chooses only a declared `id`; it cannot change `argv`. The argv runs without a shell, in the workspace (in the arm's container when it has a container environment), without evaluator control variables or a pointer to the login.
+- Output is capped per stream and marked when truncated. A nonzero exit or timeout is an ordinary tool result, not a tool error.
+- Grade calls with `tool-call` expectations, for example `tool: "run_command"` with `argumentContains: ["unit-tests"]`.
+
+A command's argv is trusted author input, like an extension bundle, and the subject only selects among them; but what it runs, such as the workspace's tests, may be the subject's own code. Without a container it runs on the host with the user's permissions and network; declare `network: "none"` and a container environment to seal it (Network, below).
+
+Group `commands` run only through the subject's `run_command` tool. Grading runs only the `argv` of `command` expectations, after the subject has finished.
+
+## Shell
+
+A body group may give its subject `bash` only through a host extension that runs the shell in isolation, such as a micro-VM. The evaluator does not choose or ship a sandbox; it verifies the one the configuration names:
+
+```json
+{
+  "tools": ["read", "write", "bash"],
+  "configuration": {
+    "host": { "name": "pi", "extensions": [{ "path": "isolation/vm-shell", "build": ["npm", "ci", "--ignore-scripts"], "entry": "index.ts" }] },
+    "shell": { "isolation": "isolation/vm-shell", "network": "none", "workspace": "/workspace", "requires": ["gondolin"] }
+  }
+}
+```
+
+- `bash` needs a `shell` or a [container environment](#container-environment) in every arm; without one, `run` refuses it before any subject starts. The loader checks a declared `shell`'s fields: `isolation` names one of the arm's `host.extensions` by its label (the package name, a working-tree `path`, or a Git source's `path`), and `network` is `"none"`, so the shell cannot reach the network.
+- Optional `workspace` is where the isolation shows the workspace (a micro-VM often mounts it at `/workspace`). The guard checks a path under it as the same path in the host workspace, so `/workspace/src/a.js` is allowed and `/etc/hosts` is not.
+- Optional `requires` lists what the isolation needs on this machine: a known name (`gondolin` for QEMU and Node.js 23.6+, or `qemu` or `node-23.6` alone), or a custom entry naming the programs, the reason and how to install them.
+
+Before an arm's first subject, the evaluator:
+
+1. Checks `requires`. When something is missing, the group is `invalid` before any subject runs, and the refusal names each missing item, what was found, why the isolation needs it, the install command for this system (Homebrew on macOS, apt or dnf on Linux), and the expected download when known. Ask the user before installing anything.
+2. Runs the probes: the subject's own Pi, with the guard and the arm's host extensions, `bash` as its only tool, and a local scripted model (an OpenAI-compatible endpoint the evaluator serves, so no model is called) that issues six bash calls. A control must succeed: writing a file inside the workspace. Five must fail: writing a host file outside the workspace, reading a marker beside the isolated home, reading a marker in the directory where the host keeps its login (Pi's agent directory), reaching the network (a listener on the host's own network address, plus `https://example.com/` unless `SKILL_EVAL_PROBE_PUBLIC_URL` names another URL or is empty), and printing a variable the evaluator plants in Pi's environment. Each is judged by what the host observes: files, connections and output.
+
+Any probe that does not hold makes the group `invalid` before any subject runs. Each run records the arm's result under `confinement` (every probe's command, whether it ran and held, and its output), also kept in `confinement/<arm>/` in the group directory with Pi's events and stderr.
+
+## Container Environment
+
+A body group can run its subject's `bash` and its [command graders](#command-graders) in a task image with its own toolchain, as benchmark-style coding tasks need:
+
+```json
+{
+  "tools": ["read", "write", "edit", "bash"],
+  "configuration": {
+    "environment": { "kind": "container", "image": "example.org/task@sha256:<64 hex characters>", "workdir": "/testbed", "network": "none" }
+  }
+}
+```
+
+- The image is pinned by digest, because a tag can change under the run. `workdir` names the workspace in the container, such as `/testbed`, and must be empty or absent in the image (checked when the container starts). The loader checks the rest: `network` `"none"`, and a container or a `shell`, not both, since the container provides the shell. Arms may differ in `environment` when `component` declares it, such as `environment.image`.
+- The evaluator uses the standard `docker` command line (Docker Desktop, Colima, OrbStack or Podman's `docker` all provide it) and looks for it only for groups that declare a container. A missing CLI, a stopped engine, or an image that is not present locally makes the group `invalid` before any subject runs, with the same refusal as a shell prerequisite; for a missing image it gives the `docker pull` command and, when `docker manifest inspect` reports it, the compressed download size. The evaluator never pulls an image; ask the user first.
+- With `bash` in `tools`, each subject gets its own container from the image (`--network none`, 4 GB of memory, 2 CPUs, 1024 processes), removed when the subject ends, cancelled or not. The workspace is mounted at its own host path, where `bash` and declared commands start, and `workdir` is a link to it, so `pwd`, the working directory the host reports and the paths its file tools take all agree, and the image's own references to `workdir` (such as `GIT_WORK_TREE=/testbed`) reach the workspace. Graders see their copy at `workdir`. Each declared resource the subject may read (its skills, context, runtime extension resources and Freeflow's package) is mounted read-only at the same path it has on the host, so a path the subject was given, such as `node <skill>/scripts/check.mjs`, works from bash as it does for `read`; the workspace is the only writable mount. The guard rewrites each bash call to `docker exec` into it, so the command runs in the image and Pi runs only the docker client; paths under `workdir` are checked as workspace paths. The [Shell](#shell) probes run first, in a container of their own, and must all hold.
+- With Freeflow loaded on Pi, its Tool Execution tools may be declared in `tools` and are confined the same way: `bash_background` and `monitor` run their command in the subject's container (every arm must declare a container environment; a shell isolation extension does not confine them), the subject's temp folder, where their output is saved, becomes readable and is mounted read-only at its own path, `stop_background` needs nothing more, and `apply_patch` may name only paths inside the workspace. Every command the guard sends to the container is stopped there when the host stops its docker client (a timeout, an abort, `stop_background`, the session's end), when the image has `setsid`.
+- Each `command` expectation runs in a fresh container from the same image on the graded copy of the workspace (`docker run --rm`, same mount, network and limits), removed afterwards even on a timeout. Its record names the image and container. `check-tasks` runs them in the image too, and reports the group `not-checkable`, with the same refusal, when docker or the image is missing.
+
+## Network
+
+Three paths run code the subject could have written: its `bash`, its [declared commands](#declared-commands) (which run the workspace's own tests and scripts), and [command graders](#command-graders) (hidden tests that import its code). `network` declares one policy for all three:
+
+- `"none"`: sealed. `bash` runs behind a probed [Shell](#shell) or in the arm's container, and declared commands and command graders run in the arm's [container](#container-environment), each with no network and none of the host's files, so the provider login on disk is out of reach. `"none"` needs a container environment for declared commands and command graders.
+- `"host"`: unsealed, on purpose. Declared commands and command graders run on the host as the user, with the network and the user's files, including the provider login on disk. A declared command gets no pointer to the login (no `PI_CODING_AGENT_DIR` or login variable), which keeps it out of easy reach, not out of reach.
+
+Without `network`, an arm gets what its paths allow: sealed when every path it has is confined, unsealed when it runs declared commands or command graders without a container. The plan line names the unsealed arms before anything starts, and each run records `sealing`: the `network` its code had, the `declared` policy (or null), where `bash`, `commands` and `graders` ran (`container`, `shell`, `host`, or null when it has none), and whether the `login` was out of reach: with `bash`, only when the arm's login-directory probe held, and otherwise `not shown out of reach`. The subject's host process (Pi or Claude Code) always keeps the network, which it needs to reach its model, and reads only its own copied login.
+
+## Claude Code Host
+
+`"host": { "name": "claude-code" }` runs subjects, judges and simulated users in headless Claude Code (`claude -p` with stream-json input and output). A group runs on one host; compare hosts with one group per host.
+
+```json
+{
+  "host": { "name": "claude-code", "version": "2.1.296" },
+  "model": { "model": "anthropic/claude-haiku-5-5", "thinking": "low" },
+  "subject": { "kind": "none" },
+  "environment": { "kind": "container", "image": "python@sha256:<64 hex characters>", "workdir": "/testbed", "network": "none" }
+}
+```
+
+- **Login.** Claude Code's login reaches a subject only from the evaluator's environment: `CLAUDE_CODE_OAUTH_TOKEN` (on a Mac whose login is in the keychain, `claude setup-token` prints one), `ANTHROPIC_API_KEY`, `ANTHROPIC_AUTH_TOKEN` or `CLAUDE_SESSION_INGRESS_TOKEN_FILE`, with `ANTHROPIC_BASE_URL` and proxy variables when set. No credential file is copied. `run.json` records the variable names under `isolation.credentials`, never their values. Without one, the arm is invalid before it starts.
+- **Model.** Every arm declares a Claude model; `thinking` is Claude Code's `--effort`. The loader checks both.
+- **Tools.** `read`, `write`, `edit` and `grep` run as Claude Code's Read, Write, Edit and Grep; `ls` and `find` as Glob. Nothing else is enabled: no Agent, web or MCP tools. A Glob or Grep pattern must be relative to its path. `bash` runs as Bash only in a [container environment](#container-environment), because Claude Code has no isolation extension.
+- **Isolation.** Each process has its own home, temp and `CLAUDE_CONFIG_DIR`, and starts with `--setting-sources ""`, `--strict-mcp-config`, `--disable-slash-commands`, `--permission-mode dontAsk` and the evaluator's guard as a PreToolUse hook. CLAUDE.md files, auto memory, automatic compaction, the updater and nonessential traffic are off. Client retries stay at Claude Code's default.
+- **Prompt.** `isolated` (the default) replaces Claude Code's system prompt with the evaluator's, which lists the declared tools, skills and context; `host` keeps Claude Code's and appends the skills listing and context. Claude Code still adds its own identity line and an environment note to every request; the note says whether a Git repository encloses the workspace, so keep results outside a repository (`--results`) when that matters. A local relay records each request in `<arm>/host-context.json`, with what Claude Code reported loading (`init`), and serves `context-text` expectations.
+- **Skills.** Claude Code's own skill loader is off. Declared skills are listed in the system prompt; a body run's target arrives as the text Pi's `/skill:<name>` command expands to, recorded as an `explicit-skill-command` delivery.
+- **Settings.** `settings` is a Claude Code settings object, passed with `--settings`; the loader refuses those that would replace the guard, permissions, login or environment, or load plugins, MCP servers or a status line.
+- **Not supported yet:** `host.extensions`, runtime extension bundles, `freeflow`, `commands`, and `shell`; the loader names each.