stageflow 0.40.0 → 0.44.0

package/skills/stageflow/SKILL.md

@@ -5,13 +5,14 @@ description: >-
   optional Docker Compose local try), session-capture (reuse this chat or a past
   session), author (a loop the human can explain), run (start, watch, or answer
   a pipeline), or delegate (a repeating pattern). Use when the user mentions
-  Stageflow, sf, a pipeline, HITL, or a repeating Stageflow workflow.
+  Stageflow, sf, a pipeline, HITL, an eval of a stage or an agent, or a repeating
+  Stageflow workflow.
 compatibility: Requires the Stageflow CLI (sf). An MCP host (sf ui or sf mcp) is optional.
 ---
 
 # Stageflow
 
-Read [references/control-surface.md](references/control-surface.md) before talking to a run. Probe with [scripts/detect-host.mjs](scripts/detect-host.mjs). Do not restate that policy here.
+Read [references/control-surface.md](references/control-surface.md) before talking to a run (and [references/evals.md](references/evals.md) before talking about evals). Probe with [scripts/detect-host.mjs](scripts/detect-host.mjs). Do not restate that policy here.
 
 Read the matching job `SKILL.md` and follow it. Do not invent job behavior in this router.
 
@@ -23,4 +24,6 @@ Read the matching job `SKILL.md` and follow it. Do not invent job behavior in th
 | The request needs to start, watch, or answer a run | [../stageflow-run/SKILL.md](../stageflow-run/SKILL.md) |
 | The request repeats a known pattern, or could become one | [../stageflow-delegate/SKILL.md](../stageflow-delegate/SKILL.md) |
 
+Evals are not a sixth job. When the request is to measure, test or compare a stage or an agent (is this stage any good, did my prompt change help, does this agent still behave, run the eval), read [references/evals.md](references/evals.md): `sf eval validate` is free, `sf eval run` calls models and costs money (an agent eval costs more: agent, simulator and judge), so confirm before running. Writing the eval and dataset files belongs to `stageflow-author`; starting and reading an eval run follows the same MCP-vs-CLI rule as a pipeline run.
+
 If none of those fit, name the five jobs in one line and ask which.

package/skills/stageflow/references/control-surface.md

@@ -23,14 +23,16 @@ Start a host with `sf ui` or `sf mcp` when the user wants MCP. Do not auto-start
 
 Use MCP tools over the Streamable HTTP endpoint at `{baseUrl}/mcp`. Send `Authorization: Bearer <drive-token>` when `STAGEFLOW_CONTROL_TOKEN` (or a named drive token) is configured. Tool names and payloads live in Stageflow docs: MCP (checkout: [docs/mcp.md](../../../docs/mcp.md)). Typical talking-job tools: `list_pipelines`, `list_tasks`, `start_run`, `run_stage`, `get_run`, `wait_run`, `list_waiting`, `answer_gate`, `decide_feedback_loop`, `get_envelope`, `read_artifact`, `list_checkout_changes`, `get_run_diff`, `read_checkout_file`, `validate`, `describe_pipeline`, `get_health`, `list_providers`, `list_models`, `list_project_mcp`, `probe_project_mcp`, `export_run`, `list_skills`, `list_stage_events`, `get_stage_verification`.
 
+Eval tools (`list_evals`, `get_eval`, `validate_eval`, `run_eval`, `get_eval_run`, `list_eval_runs`, `get_eval_results`, `get_eval_result`, `cancel_eval_run`, `compare_eval_runs`, `get_eval_failure_suggestions`, `get_eval_failure_sort_status`, `sort_eval_failures`) follow the same rule; `run_eval` costs money, and `sort_eval_failures` sends failed results to an external classifier (manual only). See [evals.md](evals.md).
+
 `get_health` returns capacity fields plus `stageflow_home`, `boot_providers` / `providers_live` (prefer `list_providers` for live auth), and an on-demand `disk` breakdown (`runs_bytes`, `worktrees_bytes`, `repos_bytes`, `state_db_bytes`, `a2a_artifacts_bytes`, `free_bytes`) for the durable root. When `slotsAvailable` is `0`, starts may still succeed as queued until `STAGEFLOW_MAX_QUEUED` is full — see `start_run` in the MCP docs.
 
 **Skills cwd ≠ catalog root:** `STAGEFLOW_OPERATOR_CWD` resolves operator skills; MCP/HTTP catalog browse uses seeded ∪ registered `project_root` values. See Stageflow docs: MCP — Skills cwd vs catalog project_root.
 
-Some capabilities stay **CLI-only** (or `docker exec` on a container Host): `sf graph`, `sf migrate-yaml`, `sf skills install` (durable host install; prefer `start_run.skills` for harnesses), A2A config mutate/validate, and provider OAuth login. Provider API keys for Host boot use `STAGEFLOW_PROVIDER_<ID>_API_KEY(_FILE)`, not an MCP login tool. Use the MCP **CLI-only capabilities** decision table (Stageflow docs: MCP) instead of inventing MCP substitutes.
+Some capabilities stay **CLI-only** (or `docker exec` on a container Host): `sf graph`, `sf migrate-yaml`, `sf eval engine install`, `sf skills install` (durable host install; prefer `start_run.skills` for harnesses), A2A config mutate/validate, and provider OAuth login. Provider API keys for Host boot use `STAGEFLOW_PROVIDER_<ID>_API_KEY(_FILE)`, not an MCP login tool. Use the MCP **CLI-only capabilities** decision table (Stageflow docs: MCP) instead of inventing MCP substitutes.
 
 ## When the host is down
 
-Use the `sf` CLI. Command names and flags live in Stageflow docs: CLI reference (checkout: [docs/cli-reference.md](../../../docs/cli-reference.md)). Typical talking-job commands: `sf run`, `sf runs waiting`, `sf runs answer`, `sf runs feedback-decide`, `sf runs wait`, `sf validate`, `sf envelope get`, `sf artifact read`, `sf providers`.
+Use the `sf` CLI. Command names and flags live in Stageflow docs: CLI reference (checkout: [docs/cli-reference.md](../../../docs/cli-reference.md)). Typical talking-job commands: `sf run`, `sf runs waiting`, `sf runs answer`, `sf runs feedback-decide`, `sf runs wait`, `sf validate`, `sf envelope get`, `sf artifact read`, `sf providers`; for evals, `sf eval validate`, `sf eval run --json`, `sf eval runs`, `sf eval compare`.
 
 Probe before each mutating `sf runs` verb. If the probe is **up** or the command refuses because a host is up, continue that gate via MCP — do not start a second mutating writer, and do not start `sf mcp` as a disposable bridge.

package/skills/stageflow/references/evals.md

@@ -0,0 +1,68 @@
+# Evals
+
+An eval runs one stage, or one agent, over a dataset of cases, grades every output, repeats each case, and reports a pass rate with honest error bars. Use it when the human asks whether a stage or an agent is any good, or whether a change to it made it better or worse. It works for any kind of stage or agent, not only code. For an agent a case is a whole conversation (see Agent evals below). This file is the one place the suite describes evals; job skills cite it.
+
+Full docs: Stageflow docs: Evals (checkout: [docs/evals.md](../../../docs/evals.md); agents: [Evals for agents](../../../docs/evals.md#evals-for-agents)), [CLI reference: `sf eval`](../../../docs/cli-reference.md#sf-eval), [MCP: eval tools](../../../docs/mcp.md#eval-tools), [CI: evals](../../../docs/ci.md#evals-in-ci).
+
+## Files
+
+An eval is `evals/<id>.eval.yaml` (target stage or agent, dataset, graders, defaults). Its dataset is `datasets/<name>.dataset.jsonl` (one case per line: `id`, `input`, optional `expected`, `tags`). Samples: `examples/evals/` in the Stageflow repo (a stage sample and an agent sample). YAML keys and validation codes: [docs/yaml-catalog.md](../../../docs/yaml-catalog.md#evals-eval-yaml). Write these with the native Write tool like any catalog file; the cases must match the stage's `io.input.schema`.
+
+## Free and paid
+
+| Action | CLI | MCP | Cost |
+|---|---|---|---|
+| Check an eval and its dataset | `sf eval validate [<id\|path>] [--strict] [--json]` | `validate_eval` | Free, no model call |
+| List, show, read runs and results, compare | `sf eval list`, `show`, `runs`, `runs show`, `compare`, `datasets` (all take `--json`) | `list_evals`, `get_eval`, `list_eval_runs`, `get_eval_run`, `get_eval_results`, `get_eval_result`, `compare_eval_runs` | Free |
+| Check the engine | `sf eval engine status --json` | none | Free |
+| Install the engine | `sf eval engine install` | none (CLI only) | Free, but downloads about 1.2 GB; needs Node 22.22.0 or newer |
+| **Run an eval** | `sf eval run <id\|path> --json` | `run_eval` | **Calls models and costs real money** |
+| Read failure suggestions | `sf eval suggestions <evalRunId> [--status ...] --json` | `get_eval_failure_suggestions`, `get_eval_failure_sort_status` | Free |
+| **Sort failures** | `sf eval sort <evalRunId> [--classifier auto\|jev\|judge] [--limit n] [--resort] --json` | `sort_eval_failures` | **Sends failed results to TypeSafe Jev or the judge model; small cost.** Manual only; tags nothing |
+| Stop a run | `sf eval runs cancel <evalRunId>` | `cancel_eval_run` | Stops spending |
+
+Never start an eval run on your own. Tell the human what it will do (cases x repeats stage calls, plus judge calls for rubric graders; for an agent, conversations of several turns, plus a simulator model for simulated scenarios and a judge for `rubric` and `goal_reached`) and the cost cap (`defaults.max_cost_usd`, default 2; `--max-cost` overrides it), and wait for a yes. Only one eval run is active per Host.
+
+## Before a run
+
+1. `sf eval validate <id> --strict --json` must pass. Fix findings first. A stage with no model needs a default `model:` in `stageflow.yaml` (`eval.target_unresolved` otherwise).
+2. `sf eval engine status --json`: `installed` and `node_ok` must be true. If not, name the command and ask before installing.
+3. Rubric graders need a judge from `anthropic`, `openai` or `openrouter` and an API key for it. Other providers cannot judge. A judge equal to the stage's model warns.
+4. Pick MCP or CLI by [control-surface.md](control-surface.md). `run_eval` starts in the background by default; poll `get_eval_run`.
+
+## Reading the result
+
+`sf eval run --json` prints one document and exits `0` only when the run passed.
+
+| `outcome` | Exit | Meaning |
+|---|---|---|
+| `passed` | `0` | Completed and the pass rate reached the threshold |
+| `below_threshold` | `1` | Completed, pass rate under the threshold. Read the failing cases |
+| `failed` | `1` | Could not finish (engine, invalid eval, Host error) |
+| `cancelled` | `1` | Stopped by the user; completed results are kept |
+| `budget_stopped` | `1` | Hit the cost cap; results are partial |
+
+Report `pass_rate` with `pass_rate_ci` (a 95% interval computed over cases, so it is wide with few cases), the `threshold`, `results.errored`, `reliability` (share of cases that pass on every repeat; `null` with one repeat), `cost_total` and the `url`. Do not call a small difference real: for a before-and-after, use `sf eval compare <a> <b> --json` and report its `verdict` (`improved`, `regressed`, `no_clear_difference` means inconclusive, `insufficient_data` means fewer than 5 shared cases). To see why cases failed, use `get_eval_results` / `get_eval_result` or the console URL.
+
+## Sorting failures
+
+After a run with failures, `sf eval sort` (or `sort_eval_failures`) suggests a bucket for each failed result: `agent_wrong` (change the agent or stage), `grader_wrong` (fix the grader), `case_wrong` (fix the dataset) or `unclear`, plus a failure type and a confidence. Ask first like a run: it sends each failed result's case, output and grader reasons to TypeSafe Jev (needs a key) or to the judge model's provider. Never accept suggestions for the human; accepting tags results and is done in the console or API. Read the buckets before editing the agent: a failure that belongs to a grader or a scenario is not fixed by changing the agent.
+
+## Agent evals
+
+An agent eval has `target: { kind: agent, agent: <id> }`; the agent must exist as `agents/<id>.agent.yaml` in the project (disabled is fine, archived is not). Cases are **scenarios**: `input.script` (a list of user messages, deterministic and cheap) or `input.persona` + `goal` + `opening` (a simulator model plays the user). A `conversation:` block sets `mode`, `max_turns`, `tools` and `approvals`. Graders add `tool_calls`, `conversation_check` and `goal_reached`; `schema` does not apply. There is no `sf eval new`: write the files by hand (the sample is `examples/evals/support-sample.*`).
+
+- Validate with `sf eval validate <id> --strict --json` (free). Fix `eval.agent_not_found`, `eval.grader_not_applicable` and `dataset.input_mismatch` findings first.
+- A run costs more than a stage run: agent turns, simulator calls and judge calls, with a default cap of `max_cost_usd: 1`. Say so before asking for a yes. A conversation that has started always finishes, so spend can pass the cap slightly.
+- **Tools are read-only by default**: write, shell and unlisted MCP tools are stubbed and recorded, never executed, and calls that would ask first are recorded and denied. Never turn on `conversation.tools: real` yourself. It needs `allow_real_tools: true` in the file and `--allow-real-tools` (CLI) or `allow_real_tools: true` (`run_eval`) on the run, and it lets the agent change the project. Ask the human and recommend a throwaway copy.
+- `sf eval run --json` for an agent eval adds `target_kind: "agent"`, `agent_id`, `agent_revision`, `mean_turns`, `mean_tool_calls` and `tools_policy`. Report those with the usual numbers. Conversations vary more than single outputs, so use 3 or more repeats and quote `reliability` (pass^k), which is stricter than the pass rate.
+- To see why a conversation failed, `get_eval_result` returns the conversation (messages, tool calls with `policy_outcome` of `ran`, `stubbed`, `mocked`, `would_ask_first` or `denied`, `final_reply`, `stop_reason`). A stubbed tool is expected under `read_only`, not a bug. Statuses `EVAL_AGENT_FAILED`, `EVAL_AGENT_TIMEOUT`, `EVAL_AGENT_EMPTY_REPLY` (the provider returned nothing; not graded) and `EVAL_SIMULATOR_ERROR` mean the conversation did not finish. After 5 consecutive empty-reply or failed cases the run stops as `failed`; check the provider key and rate limits (an OpenRouter `:free` model can hit its daily limit) before rerunning.
+- The run uses the agent's current revision and records it. Compare two runs of the same agent with `sf eval compare`; it shows what changed in the agent.
+- Running sends the scenario to the agent's model, and the conversation to the simulator and the judge. Eval sessions are hidden from the agent's lists and Inbox.
+
+## Limits to state when relevant
+
+- Pipeline-level overlays (entry `mcp`, `secrets`, `requires`, wrapper `completion`, `channels`, `schemas`, `include`) are not applied when a pipeline stage is evaluated, so the stage can behave differently than inside its pipeline.
+- Cases run with no checkout; a stage that needs a repository cannot be evaluated yet. A stage that stops at a HITL gate records an error for that case.
+- Not built: `command` graders, judge calibration, scheduled or CI-triggered evals, deleting an eval in the console. For agents also: red-teaming and generated scenarios, voice and channel evals, a `groundedness` grader, a revision picker.
+- Running sends case inputs to the stage's model and, for rubric graders, inputs and outputs to the judge's provider. `code` graders run JavaScript on this machine.

package/skills/stageflow-author/SKILL.md

@@ -75,3 +75,7 @@ Write one external stage YAML per step and one pipeline YAML that wires them wit
 Read [`references/validate-and-report.md`](references/validate-and-report.md). Follow it to the end.
 
 **Done when** that file's success report is printed.
+
+## Evals
+
+Only when the human asks whether a stage or an agent is any good, or wants to check a prompt or instruction change. Read [`../stageflow/references/evals.md`](../stageflow/references/evals.md). Write `evals/<id>.eval.yaml` and its `datasets/<name>.dataset.jsonl` with the native Write tool (cases must match the stage's `io.input.schema`; the sample is `examples/evals/`), then check them with `sf eval validate <id> --strict` (free). For an agent, target `{ kind: agent, agent: <id> }` (the agent file must exist in `agents/`), write scenarios in `input` (`script` messages, or `persona`, `goal` and `opening`), give each `expected.tools` and `expected.forbidden_tools`, and add a `conversation:` block; leave `tools: read_only` and never set `tools: real` or `allow_real_tools` unless the human asks. The sample is `examples/evals/support-sample.*`; there is no `sf eval new`. Do not run `sf eval run` here: it calls models and costs money, and `stageflow-run` owns starting and reading runs.

package/skills/stageflow-run/SKILL.md

@@ -205,6 +205,10 @@ MCP `get_run` / `wait_run` and `sf runs wait` / `sf runs show` `--json` carry li
 
 **Done when** that five-part report is printed.
 
+## Eval runs
+
+An eval run is not a pipeline run, so none of the steps above apply to it. When the human wants to run an eval, read [`../stageflow/references/evals.md`](../stageflow/references/evals.md) and follow it: `sf eval validate` first (free), say what the run will cost and wait for a yes, then `run_eval` (host up) or `sf eval run --json` (host down), and report `outcome`, `pass_rate` with its interval, `threshold`, `cost_total` and `url`. An eval of an agent costs more (agent turns, a simulator, the judge), keeps tools read-only unless the file and the run both allow real tools (`allow_real_tools` / `--allow-real-tools`, which you never pass on your own), and adds `agent_id`, `agent_revision`, `mean_turns` and `mean_tool_calls` to the document; read the conversation with `get_eval_result`. Eval tools are not in the list under [MCP tools](#mcp-tools) and `mcp-call.mjs` does not call them: use this harness's native Stageflow MCP tools when their names are in the tool list, otherwise the `sf eval` CLI.
+
 ## Non-goals
 
 This job starts, watches, and answers runs. It does not require `sf ui`. The operator console is not the answer path. It does not register or invent an MCP tool. It does not start `sf mcp`. It does not retry, resume, abandon, recover, cancel, delete, gc, or rerun stages or runs.