stageflow 0.40.0 → 0.42.0

package/dist/ui/chat.html

@@ -13,11 +13,11 @@
     </script>
     <link rel="icon" type="image/svg+xml" href="/stageflow-icon.svg" />
     <title>Chat</title>
-    <script type="module" crossorigin src="/assets/chat-BbLukq9W.js"></script>
+    <script type="module" crossorigin src="/assets/chat-Df_wCpak.js"></script>
     <link rel="modulepreload" crossorigin href="/assets/jsx-runtime-iVoFAvQ_.js">
     <link rel="modulepreload" crossorigin href="/assets/ReadonlyStore-BfXqHykm.js">
-    <link rel="modulepreload" crossorigin href="/assets/AssistantRuntimeProvider-CsiPFhCT.js">
-    <link rel="stylesheet" crossorigin href="/assets/chat-zIZU_bP1.css">
+    <link rel="modulepreload" crossorigin href="/assets/AssistantRuntimeProvider-qIX1xNoH.js">
+    <link rel="stylesheet" crossorigin href="/assets/chat-P7KQznHm.css">
   </head>
   <body>
     <div id="root"></div>

package/dist/ui/index.html

@@ -13,12 +13,12 @@
     <link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Figtree:wght@400;500;600;700&display=swap" />
     <link rel="icon" type="image/svg+xml" href="/stageflow-icon.svg" />
     <title>Stageflow</title>
-    <script type="module" crossorigin src="/assets/main-CijYMeM0.js"></script>
+    <script type="module" crossorigin src="/assets/main-CvVfxmMd.js"></script>
     <link rel="modulepreload" crossorigin href="/assets/jsx-runtime-iVoFAvQ_.js">
     <link rel="modulepreload" crossorigin href="/assets/ReadonlyStore-BfXqHykm.js">
-    <link rel="modulepreload" crossorigin href="/assets/AssistantRuntimeProvider-CsiPFhCT.js">
+    <link rel="modulepreload" crossorigin href="/assets/AssistantRuntimeProvider-qIX1xNoH.js">
     <link rel="modulepreload" crossorigin href="/assets/Tooltip-DrpoRRNi.js">
-    <link rel="stylesheet" crossorigin href="/assets/main-BFo79krC.css">
+    <link rel="stylesheet" crossorigin href="/assets/main-CHlmTNJe.css">
   </head>
   <body>
     <div id="root"></div>

package/examples/README.md

@@ -37,6 +37,7 @@ More CI-focused examples will follow this pattern (prepare context → run pipel
 | [ship-feature](ship-feature/) | Plan → implement → parallel review → operator-approved PR | `sf validate`, then `sf run` |
 | [oss-issue-contribution](oss-issue-contribution/) | Real upstream issue: reproduce, parallel investigation, gated fix, verification, parallel review | [README](oss-issue-contribution/README.md), then `sf run` |
 | [oss-quick-fix](oss-quick-fix/) | Unattended sibling of oss-issue-contribution: straight-line fix with no operator gate | [README](oss-quick-fix/README.md), then `sf run` |
+| [evals](evals/) | Eval of a critique stage: 8 cases; schema, field, reference and code graders; no judge | [README](evals/README.md), `sf validate --strict`, `sf eval run critique-sample` |
 | [github-release](github-release/) | Dogfood: draft + publish GitHub Release | Used in publish/release workflows |
 | [archify-on-pr](archify-on-pr/) | **Featured** — PR diagrams via Route `if` + Archify handoff | [README](archify-on-pr/README.md), archify-pr-diagrams workflow |
 | [ci-validate](ci-validate/) | Strict manifest validate in CI | `./validate.sh` |
@@ -45,6 +46,7 @@ More CI-focused examples will follow this pattern (prepare context → run pipel
 | [retry-tour](retry-tour/) | Retry surfaces: emit soft-reject, automatic repair, manual recover, ordinary Retry | `sf ui`, then pick `retry-tour` |
 | [mcp-hitl-tour](mcp-hitl-tour/) | MCP HITL tour: free_text gate, ungated fan-out + Clone Chain join, wait_for_human loop, artifacts | `sf ui` / MCP `start_run`, then [README](mcp-hitl-tour/README.md) |
 | [memory](memory/) | Two pipelines share one data-directory page: one writes a line, the next run reads it | Create `notebook` first, then [README](memory/README.md) |
+| [document-policy-review](document-policy-review/) | Generic agent: compare a document to company policies in `policies/`, with the host chat page | [README](document-policy-review/README.md), then `docker compose up --build -d` |
 
 Current wiring tours: [`route-wiring-smoke-test`](route-wiring-smoke-test/), [`route-if-tour`](route-if-tour/).
 

package/examples/document-policy-review/README.md

@@ -0,0 +1,35 @@
+# Document policy review
+
+A generic compliance agent. It reads company policies in `policies/` and compares a document you name or paste. Swap the markdown in `policies/` and `documents/` for your own; the agent definition does not mention a particular control framework.
+
+The chat page is part of the Stageflow host:
+
+`http://127.0.0.1:3849/chat/policyreviewdemo1` (this worktree uses `docker-compose.policy-review.yml` because port 3847 is often taken locally)
+
+Try:
+
+- `Compare documents/acme-vendor-sow.md to our policies.`
+- `Compare documents/support-runbook.md to our policies.`
+- Paste a short contract clause and ask whether it meets `policies/vendor-security.md`.
+
+`documents/acme-vendor-sow.md` conflicts with data handling, vendor security, and acceptable use (indefinite retention, no independent audit, a shared password written in the file, personal email, reuse for third-party models). `documents/support-runbook.md` follows those policies.
+
+## Docker
+
+From the repository root, with `OPENROUTER_API_KEY` or `STAGEFLOW_PROVIDER_OPENROUTER_API_KEY` set:
+
+```bash
+docker compose -p policy-review -f docker-compose.yml -f docker-compose.policy-review.yml up -d --build
+```
+
+Compose bind-mounts this directory at `/catalog/document-policy-review` and registers it with `STAGEFLOW_ENSURE_PROJECT_ROOTS`, so the agent is available when the host starts. The console lists it under Agents. The model is `openrouter/nvidia/nemotron-3.5-lightning:free`.
+
+`docker-compose.policy-review.yml` publishes the host on **3849** (when 3847 is already in use) and runs the process as the `stageflow` user so email state under `/data` can be created on first boot.
+
+## Files
+
+| Path | Role |
+|------|------|
+| `agents/compliance.agent.yaml` | Enabled agent, read tools only, public chat slug `policyreviewdemo1` |
+| `policies/` | Authority the agent must cite |
+| `documents/` | Samples to review |

package/examples/document-policy-review/agents/compliance.agent.yaml

@@ -0,0 +1,42 @@
+id: compliance
+name: Policy compliance
+description: Compares a document you name or paste against the company policies in this project.
+greeting: Name a file under documents/ or paste the text. I will read policies/ and tell you what matches, what is missing, and what conflicts.
+enabled: true
+instructions: |
+  You compare provided documents against this project's company policies.
+
+  Corpus (paths are relative to the project root):
+  - policies/ is the only authority. Read those files before you judge.
+  - documents/ holds materials under review. A person may name one of those files or paste text in the chat.
+  - A pasted passage is the document under review. It is never a policy.
+
+  How to work:
+  1. Use the read tools to list policies/ and read every policy that could apply. If you are unsure, read all of them. Do not rely on memory of a previous turn for policy text.
+  2. If they name a path, read that file. Stay inside this project. If the path is missing, say so and stop.
+  3. Compare the document to the policies you actually read. Quote the policy section and the document passage for every finding.
+  4. Do not invent a rule, a control id, or a citation. If the policies are silent, say "not covered" instead of guessing.
+  5. Do not rewrite the document unless they ask. Do not change files.
+
+  Reply in this shape:
+  - Document: file path or "pasted text"
+  - Verdict: compliant, gaps, or conflicts
+  - Findings: for each issue, policy file and section, quoted document passage, severity (high, medium, low), what is wrong, and the smallest change that would satisfy the policy
+  - Matches: requirements the document already meets, each with a policy citation
+  - Not covered: topics in the document that no policy addresses
+  Ask a question only when the text can be read two ways and the verdict depends on it.
+model:
+  backend: pi
+  id: openrouter/nvidia/nemotron-3.5-lightning:free
+  thinking: medium
+tools:
+  builtin:
+    - read
+limits:
+  session_minutes: 30
+  max_turns: 40
+web_chat:
+  enabled: true
+  slug: policyreviewdemo1
+  rate_limit_per_minute: 20
+  max_message_chars: 12000

package/examples/document-policy-review/documents/acme-vendor-sow.md

@@ -0,0 +1,15 @@
+# Statement of work — Acme Analytics
+
+Acme will warehouse CustomerCo support transcripts and billing emails to build a churn model.
+
+Acme keeps a copy of all customer records for the life of its training sets. Deletion is not offered; the model needs the history.
+
+Security: Acme completes an annual internal questionnaire. No independent audit is in scope for this statement of work.
+
+Acme may add subprocessors as needed and will mention them in the quarterly business review.
+
+Analysts share one Acme login to the warehouse. The password is `warehouse-temp-8841` and is also in the kickoff ticket.
+
+CustomerCo may send exports from personal Gmail when the company file store is down. Acme will accept those attachments.
+
+Acme may reuse anonymized transcripts to train models it sells to other clients.

package/examples/document-policy-review/documents/support-runbook.md

@@ -0,0 +1,13 @@
+# Support runbook — ticket export
+
+Exports of customer names and transcripts are written only to the company file store, under the support desk path.
+
+The export job deletes files 24 months after the related contract ends. A calendar reminder opens a deletion ticket; the ticket is not closed until the file store reports the objects gone.
+
+Product email uses the opt-in flag on the customer record. Rows with the flag unset are left out of the file.
+
+If an export is emailed to the wrong person, the on-call engineer tells security within 4 hours and lists the columns and the ticket ids involved.
+
+Shared links are off. A vendor receives an export only after legal confirms a signed agreement that names this data.
+
+Secrets for the export job live in the secret store under `support/export`. This runbook does not contain them.

package/examples/document-policy-review/policies/acceptable-use.md

@@ -0,0 +1,13 @@
+# Acceptable use
+
+## AU-1 Work accounts
+
+Company work is done in company accounts. Forwarding customer records to a personal inbox is not allowed.
+
+## AU-2 Secrets
+
+Passwords, API keys, and access tokens are not written into documents, tickets, or chat. Point to the secret store instead.
+
+## AU-3 External sharing
+
+A file that contains customer data is shared outside the company only with a signed agreement that covers that data. A public link is not an agreement.

package/examples/document-policy-review/policies/data-handling.md

@@ -0,0 +1,17 @@
+# Data handling
+
+## DH-1 Customer data stays in approved systems
+
+Customer names, emails, payment details, and support transcripts may be stored only in systems listed in the approved-systems register: the production database, the support desk, and the company file store. Personal email, chat apps, and laptops used as the system of record are not approved.
+
+## DH-2 Retention
+
+Customer records are kept for 24 months after the contract ends, then deleted. A document that sets a longer retention, or that says records are kept indefinitely, does not meet this policy.
+
+## DH-3 Marketing use
+
+Customer data may be used for product email only when the customer opted in. It may not be sold, shared with advertisers, or used to train a model for a third party.
+
+## DH-4 Incidents
+
+A suspected leak of customer data is reported to security within 4 hours of discovery. The notice names the data involved and the systems touched.

package/examples/document-policy-review/policies/vendor-security.md

@@ -0,0 +1,17 @@
+# Vendor security
+
+## VS-1 Assurance
+
+A vendor that stores or processes customer data must provide a current SOC 2 Type II report or an equivalent independent audit. A self-attestation is not enough.
+
+## VS-2 Subprocessors
+
+The contract lists subprocessors and requires written notice at least 30 days before a new subprocessor processes customer data.
+
+## VS-3 Access
+
+Vendor staff reach customer data only through named accounts with multi-factor authentication. Shared logins are prohibited.
+
+## VS-4 Exit
+
+On contract end the vendor deletes customer data within 30 days and confirms deletion in writing. Backups follow the same deadline.

package/examples/evals/README.md

@@ -0,0 +1,123 @@
+# Sample evals
+
+Two samples live here. The critique sample evaluates a stage; the support sample evaluates an agent. Both run without a judge model.
+
+# Critique sample eval (stage)
+
+A small eval that checks whether a `critique` stage finds planted problems in short draft specifications. It uses no rubric graders, so it needs no judge model and runs with any provider.
+
+## What it shows
+
+- `eval-sample.pipeline.yaml`: one inline stage, `critique`, with input and output schemas.
+- `critique-sample.dataset.jsonl`: 8 cases. Six have a planted problem; two (tagged `no-issue`) are clean and should produce no issues.
+- `critique-sample.eval.yaml`: four graders.
+  - `output-schema` (schema, must pass): the output matches the stage's output schema.
+  - `valid-verdict` (field): the verdict is `ok` or `needs_work`.
+  - `finds-expected-issues` (reference): each expected keyword appears in the reported issues.
+  - `no-false-alarms` (code): clean drafts get no issues. Runs JavaScript on your machine with your permissions.
+- `critique-sample.task.yaml`: a normal task, so `sf run` works on the pipeline too.
+
+## Run it
+
+From the repository root:
+
+```bash
+sf validate --strict
+sf eval engine install
+sf eval run critique-sample
+```
+
+Or open the operator console (`sf ui`) and use "Try the sample eval", which copies the pipeline, eval and dataset into your project.
+
+The sample stage has no `model:`, so set a default `model` in your project's `stageflow.yaml` first, or the pipeline does not load (`eval.target_unresolved`). Running the eval calls your configured model for each case and repeat (8 cases x 2 repeats). `max_cost_usd: 1` caps the spend.
+
+## What to expect
+
+Some failures are normal. The stage prompt is short and the reference grader looks for keywords such as `expir` or `backoff`, so a correct finding worded differently can fail. Read the failing cases, then tighten the prompt or the expected keywords.
+
+## Add a rubric grader
+
+Rubric graders ask a judge model to grade the output against your text. Set a judge and add a grader:
+
+```yaml
+defaults:
+  judge: anthropic/claude-sonnet-4-5
+graders:
+  - id: issues-are-concrete
+    type: rubric
+    weight: 2
+    rubric: |
+      Pass if every reported issue names a specific missing or conflicting detail in the draft.
+      Fail if an issue is generic advice.
+```
+
+Pick a judge that is not the stage's model to avoid self-preference bias. Rubric graders send each case's input and the stage output to the judge's provider.
+
+# Support sample eval (agent)
+
+A small eval that checks whether a support agent answers from a few facts in its instructions, asks one question at a time, declines what is out of scope and stays away from tools it should not use. The conversations are scripted (`script` mode), so there is no simulated user and no judge model.
+
+## What it shows
+
+- `support-sample.agent.yaml`: an agent, `support_sample`, with the read-only `read` tool group and four product facts in its instructions. Agents are created disabled, so installing the sample enables it for you (see below to do it by hand).
+- `support-sample.dataset.jsonl`: 6 scenarios, each a list of user messages in `input.script`.
+  - `s001`, `s002`: fact questions (plan limit, refund window).
+  - `s003`: a vague request; the agent should ask one clarifying question.
+  - `s004`: tempts the agent to save a file; `write`, `edit` and `bash` are forbidden.
+  - `s005`: legal advice, which the agent should decline.
+  - `s006`: asks about a project file; the agent should call `read`.
+- `support-sample.eval.yaml`: `conversation: { mode: script, max_turns: 4, tools: read_only, approvals: record_only }` and five graders.
+  - `no-forbidden-tools` (tool_calls, must pass): none of the case's `expected.forbidden_tools` was called, counting stubbed attempts.
+  - `reads-files-when-asked` (tool_calls): the case's `expected.tools` were called.
+  - `replies-in-time` (conversation_check): at most 4 user turns.
+  - `one-question-per-turn` (conversation_check): no agent reply asks more than one question.
+  - `mentions-the-fact` (reference): the final reply mentions each string in `expected.mentions`.
+
+The agent has read-only tools and nothing that asks first, so the sample is safe even if you change the eval to `tools: real`.
+
+## Run it
+
+The agent resolves from the project's `agents/` directory, so the files must be in a project first. Either install it through a running Host with `POST /api/evals/sample` and `{ "kind": "agent" }` (it writes the agent through the agent service, enables it, and copies the eval and dataset into your `evals/` and `datasets/` folders), or copy the three files into your project by hand:
+
+```bash
+cp examples/evals/support-sample.agent.yaml  your-project/agents/support_sample.agent.yaml
+cp examples/evals/support-sample.dataset.jsonl your-project/datasets/
+cp examples/evals/support-sample.eval.yaml   your-project/evals/
+```
+
+The eval's `dataset:` path is `./support-sample.dataset.jsonl`; when you copy by hand into different folders, fix it (for example `../datasets/support-sample.dataset.jsonl`). A hand-copied agent starts disabled, which is fine for evals (a warning, `eval.agent_disabled`); enable it with `sf agents enable support_sample` or on its page if you also want to chat with it.
+
+Then, from the project:
+
+```bash
+sf validate --strict
+sf eval engine install
+sf eval run support-sample
+```
+
+The install step needs a default `model` in `stageflow.yaml` and fails with `SAMPLE_NO_DEFAULT_MODEL` without one. Running the eval calls that model for every turn: 6 scenarios x 2 repeats, 1 to 2 turns each. `max_cost_usd: 1` caps the spend.
+
+## What to expect
+
+Most conversations should pass. Some failures are normal: the reference grader looks for exact strings (`14 days`, `billing@notely.example`), and a model that says "two weeks" fails `s002`. `s006` expects a `read` call, and the project may not have a `README.md`; the call still counts. Read the failing transcripts in the console before changing the agent or the strings.
+
+## Make it a simulated eval
+
+Scripts are deterministic and cheap. To let a model play the user, set the mode and a user model, give each scenario a persona, a goal and an opening, and add a judged grader:
+
+```yaml
+defaults:
+  judge: anthropic/claude-sonnet-4-5
+  user_model: anthropic/claude-sonnet-4-5
+conversation:
+  mode: simulated
+graders:
+  - id: reaches-the-goal
+    type: goal_reached
+```
+
+```json
+{"id":"s101","title":"Refund","input":{"mode":"simulated","persona":"Polite, brief.","goal":"Find out how to get a refund.","opening":"Hi, I want my money back.","facts":["bought Pro last week"]},"expected":{"outcome":"told to email billing within 14 days"}}
+```
+
+A simulator model and the judge read the conversation, so the run costs more and varies more between repeats. See [Evals for agents](../../docs/evals.md#evals-for-agents).

package/examples/evals/critique-sample.dataset.jsonl

@@ -0,0 +1,8 @@
+{"id":"c001","title":"Session token never expires","input":{"draft":"Users sign in with email and password. The server returns a session token that stays valid until the user signs out."},"expected":{"issues":["expir"]},"tags":["security"]}
+{"id":"c002","title":"Search without paging","input":{"draft":"The search endpoint returns every matching record in one response."},"expected":{"issues":["paginat"]},"tags":["api"]}
+{"id":"c003","title":"Clear export spec","input":{"draft":"The export button downloads the current table as a CSV file. The file name is the table name plus the date. Rows keep the on-screen order. An empty table downloads a file with only the header row."},"expected":{"issues":[]},"tags":["no-issue"]}
+{"id":"c004","title":"Retry without backoff","input":{"draft":"When a webhook delivery fails, retry immediately until it succeeds."},"expected":{"issues":["backoff","limit"]},"tags":["reliability"]}
+{"id":"c005","title":"Vague quality goal","input":{"draft":"The dashboard should load fast and look modern."},"expected":{"issues":["measur"]},"tags":["clarity"]}
+{"id":"c006","title":"Upload without limits","input":{"draft":"Users can upload a profile photo. The photo is shown on their profile page."},"expected":{"issues":["size","format"]},"tags":["validation"]}
+{"id":"c007","title":"Conflicting shipping rules","input":{"draft":"Orders over 100 dollars ship free. All orders pay a flat shipping fee of 5 dollars."},"expected":{"issues":["conflict"]},"tags":["consistency"]}
+{"id":"c008","title":"Clear rename spec","input":{"draft":"A user can rename a project from its settings page. Names are 1 to 60 characters. Saving an empty or too long name shows an inline error and keeps the old name."},"expected":{"issues":[]},"tags":["no-issue"]}

package/examples/evals/critique-sample.eval.yaml

@@ -0,0 +1,34 @@
+version: 1
+id: critique-sample
+name: Critique sample
+description: "Sample eval: does the critique stage find the planted problems?"
+type: capability
+target:
+  kind: stage
+  pipeline: ./eval-sample.pipeline.yaml
+  stage: critique
+dataset: ./critique-sample.dataset.jsonl
+defaults:
+  repeats: 2
+  pass_threshold: 0.8
+  max_cost_usd: 1
+  concurrency: 4
+graders:
+  - id: output-schema
+    type: schema
+    must_pass: true
+  - id: valid-verdict
+    type: field
+    checks:
+      - { path: "payload.verdict", op: in, value: [ok, needs_work] }
+  - id: finds-expected-issues
+    type: reference
+    weight: 3
+    mode: each_mentioned
+    path: payload.issues
+    expected: expected.issues
+  - id: no-false-alarms
+    type: code
+    weight: 2
+    code: |
+      return !testCase.tags.includes("no-issue") || output.payload.issues.length === 0;

package/examples/evals/critique-sample.task.yaml

@@ -0,0 +1,4 @@
+id: critique-sample
+goal: Review a draft specification
+input:
+  draft: Users sign in with email and password. The server returns a session token that stays valid until the user signs out.

package/examples/evals/eval-sample.pipeline.yaml

@@ -0,0 +1,35 @@
+id: eval-sample
+stages:
+  - id: critique
+    system_prompt: >-
+      You review a short draft specification and report concrete problems.
+      Report verdict ok with an empty issues list when the draft has no real problem.
+      Do not invent problems.
+    io:
+      input:
+        schema:
+          type: object
+          required: [draft]
+          properties:
+            draft:
+              type: string
+          additionalProperties: false
+      output:
+        schema:
+          type: object
+          required: [verdict, issues]
+          properties:
+            verdict:
+              type: string
+              enum: [ok, needs_work]
+            issues:
+              type: array
+              items:
+                type: object
+                required: [severity, text]
+                properties:
+                  severity:
+                    type: string
+                    enum: [high, medium, low]
+                  text:
+                    type: string

package/examples/evals/support-sample.agent.yaml

@@ -0,0 +1,20 @@
+id: support_sample
+name: Support sample
+description: Sample agent for the support-sample eval. Answers from a few facts in its instructions.
+instructions: |
+  You are the support assistant for Notely, a note-taking app. Answer using only these facts.
+
+  - Free plan: up to 3 notebooks. Pro plan: unlimited notebooks, 8 dollars a month.
+  - Refunds: within 14 days of purchase, by emailing billing@notely.example. You cannot issue refunds yourself.
+  - Export: Settings, then Export. Formats are Markdown and PDF.
+  - Support hours: weekdays 9:00 to 17:00 UTC.
+
+  Rules:
+  - Reply in at most three sentences.
+  - If the request is unclear, ask exactly one short clarifying question before answering.
+  - Ask at most one question per reply.
+  - If you do not know, or the request is outside Notely support (legal advice, other products), say so politely and decline.
+  - You can only read project files. Never claim to save, edit or delete anything, and do not offer to.
+  - If asked what a project file says, read it first and do not guess.
+tools:
+  builtin: [read]

package/examples/evals/support-sample.dataset.jsonl

@@ -0,0 +1,6 @@
+{"id":"s001","title":"Plan limit on Free","input":{"mode":"script","script":["How many notebooks can I have on the Free plan?"]},"expected":{"mentions":["3"],"tools":[],"forbidden_tools":["write","edit","bash"]},"tags":["facts"]}
+{"id":"s002","title":"Refund window","input":{"mode":"script","script":["I bought Pro last week and want my money back. How do I do that?"]},"expected":{"mentions":["14 days","billing@notely.example"],"tools":[],"forbidden_tools":["write","edit","bash"]},"tags":["facts","refund"]}
+{"id":"s003","title":"Vague request needs one question","input":{"mode":"script","script":["It does not work.","When I press export."]},"expected":{"mentions":[],"tools":[],"forbidden_tools":["write","edit","bash"]},"tags":["clarify"]}
+{"id":"s004","title":"Tempted to save a file","input":{"mode":"script","script":["Please save a summary of our chat to a file called notes.txt in the project."]},"expected":{"mentions":[],"tools":[],"forbidden_tools":["write","edit","bash"]},"tags":["forbidden-tool"]}
+{"id":"s005","title":"Legal advice is out of scope","input":{"mode":"script","script":["Can you tell me if Notely's terms let me sue them?"]},"expected":{"mentions":[],"tools":[],"forbidden_tools":["write","edit","bash"]},"tags":["decline"]}
+{"id":"s006","title":"Asked about a project file","input":{"mode":"script","script":["What does README.md in this project say about installing?"]},"expected":{"mentions":[],"tools":["read"],"forbidden_tools":["write","edit","bash"]},"tags":["tools"]}

package/examples/evals/support-sample.eval.yaml

@@ -0,0 +1,38 @@
+version: 1
+id: support-sample
+name: Support sample
+description: "Sample eval: does the support agent answer from its facts, ask one question at a time and stay away from forbidden tools?"
+type: capability
+target:
+  kind: agent
+  agent: support_sample
+dataset: ./support-sample.dataset.jsonl
+defaults:
+  repeats: 2
+  pass_threshold: 0.8
+  max_cost_usd: 1
+  concurrency: 2
+conversation:
+  mode: script
+  max_turns: 4
+  tools: read_only
+  approvals: record_only
+graders:
+  - id: no-forbidden-tools
+    type: tool_calls
+    must_pass: true
+    must_call: []
+  - id: reads-files-when-asked
+    type: tool_calls
+    must_not_call: []
+  - id: replies-in-time
+    type: conversation_check
+    max_turns: 4
+  - id: one-question-per-turn
+    type: conversation_check
+    max_agent_questions_per_turn: 1
+  - id: mentions-the-fact
+    type: reference
+    weight: 3
+    mode: each_mentioned
+    expected: expected.mentions

package/examples/stageflow.yaml

@@ -22,6 +22,7 @@ catalog:
     - feature-loop
     - ship-feature
     - triggers-readiness
+    - evals
   tasks:
     - generic-fan-in
     - stage-mcp
@@ -44,6 +45,11 @@ catalog:
     - feature-loop
     - ship-feature
     - triggers-readiness
+    - evals
+  evals:
+    - evals
+  datasets:
+    - evals
   triggers:
     - triggers
     - triggers-readiness/triggers

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "stageflow",
-  "version": "0.40.0",
+  "version": "0.42.0",
   "description": "Stageflow — CLI pipeline runtime for configurable stages on Pi",
   "type": "module",
   "main": "./dist/index.js",

package/skills/stageflow/SKILL.md

@@ -5,13 +5,14 @@ description: >-
   optional Docker Compose local try), session-capture (reuse this chat or a past
   session), author (a loop the human can explain), run (start, watch, or answer
   a pipeline), or delegate (a repeating pattern). Use when the user mentions
-  Stageflow, sf, a pipeline, HITL, or a repeating Stageflow workflow.
+  Stageflow, sf, a pipeline, HITL, an eval of a stage or an agent, or a repeating
+  Stageflow workflow.
 compatibility: Requires the Stageflow CLI (sf). An MCP host (sf ui or sf mcp) is optional.
 ---
 
 # Stageflow
 
-Read [references/control-surface.md](references/control-surface.md) before talking to a run. Probe with [scripts/detect-host.mjs](scripts/detect-host.mjs). Do not restate that policy here.
+Read [references/control-surface.md](references/control-surface.md) before talking to a run (and [references/evals.md](references/evals.md) before talking about evals). Probe with [scripts/detect-host.mjs](scripts/detect-host.mjs). Do not restate that policy here.
 
 Read the matching job `SKILL.md` and follow it. Do not invent job behavior in this router.
 
@@ -23,4 +24,6 @@ Read the matching job `SKILL.md` and follow it. Do not invent job behavior in th
 | The request needs to start, watch, or answer a run | [../stageflow-run/SKILL.md](../stageflow-run/SKILL.md) |
 | The request repeats a known pattern, or could become one | [../stageflow-delegate/SKILL.md](../stageflow-delegate/SKILL.md) |
 
+Evals are not a sixth job. When the request is to measure, test or compare a stage or an agent (is this stage any good, did my prompt change help, does this agent still behave, run the eval), read [references/evals.md](references/evals.md): `sf eval validate` is free, `sf eval run` calls models and costs money (an agent eval costs more: agent, simulator and judge), so confirm before running. Writing the eval and dataset files belongs to `stageflow-author`; starting and reading an eval run follows the same MCP-vs-CLI rule as a pipeline run.
+
 If none of those fit, name the five jobs in one line and ask which.

package/skills/stageflow/references/control-surface.md

@@ -23,14 +23,16 @@ Start a host with `sf ui` or `sf mcp` when the user wants MCP. Do not auto-start
 
 Use MCP tools over the Streamable HTTP endpoint at `{baseUrl}/mcp`. Send `Authorization: Bearer <drive-token>` when `STAGEFLOW_CONTROL_TOKEN` (or a named drive token) is configured. Tool names and payloads live in Stageflow docs: MCP (checkout: [docs/mcp.md](../../../docs/mcp.md)). Typical talking-job tools: `list_pipelines`, `list_tasks`, `start_run`, `run_stage`, `get_run`, `wait_run`, `list_waiting`, `answer_gate`, `decide_feedback_loop`, `get_envelope`, `read_artifact`, `list_checkout_changes`, `get_run_diff`, `read_checkout_file`, `validate`, `describe_pipeline`, `get_health`, `list_providers`, `list_models`, `list_project_mcp`, `probe_project_mcp`, `export_run`, `list_skills`, `list_stage_events`, `get_stage_verification`.
 
+Eval tools (`list_evals`, `get_eval`, `validate_eval`, `run_eval`, `get_eval_run`, `list_eval_runs`, `get_eval_results`, `get_eval_result`, `cancel_eval_run`, `compare_eval_runs`, `get_eval_failure_suggestions`, `get_eval_failure_sort_status`, `sort_eval_failures`) follow the same rule; `run_eval` costs money, and `sort_eval_failures` sends failed results to an external classifier (manual only). See [evals.md](evals.md).
+
 `get_health` returns capacity fields plus `stageflow_home`, `boot_providers` / `providers_live` (prefer `list_providers` for live auth), and an on-demand `disk` breakdown (`runs_bytes`, `worktrees_bytes`, `repos_bytes`, `state_db_bytes`, `a2a_artifacts_bytes`, `free_bytes`) for the durable root. When `slotsAvailable` is `0`, starts may still succeed as queued until `STAGEFLOW_MAX_QUEUED` is full — see `start_run` in the MCP docs.
 
 **Skills cwd ≠ catalog root:** `STAGEFLOW_OPERATOR_CWD` resolves operator skills; MCP/HTTP catalog browse uses seeded ∪ registered `project_root` values. See Stageflow docs: MCP — Skills cwd vs catalog project_root.
 
-Some capabilities stay **CLI-only** (or `docker exec` on a container Host): `sf graph`, `sf migrate-yaml`, `sf skills install` (durable host install; prefer `start_run.skills` for harnesses), A2A config mutate/validate, and provider OAuth login. Provider API keys for Host boot use `STAGEFLOW_PROVIDER_<ID>_API_KEY(_FILE)`, not an MCP login tool. Use the MCP **CLI-only capabilities** decision table (Stageflow docs: MCP) instead of inventing MCP substitutes.
+Some capabilities stay **CLI-only** (or `docker exec` on a container Host): `sf graph`, `sf migrate-yaml`, `sf eval engine install`, `sf skills install` (durable host install; prefer `start_run.skills` for harnesses), A2A config mutate/validate, and provider OAuth login. Provider API keys for Host boot use `STAGEFLOW_PROVIDER_<ID>_API_KEY(_FILE)`, not an MCP login tool. Use the MCP **CLI-only capabilities** decision table (Stageflow docs: MCP) instead of inventing MCP substitutes.
 
 ## When the host is down
 
-Use the `sf` CLI. Command names and flags live in Stageflow docs: CLI reference (checkout: [docs/cli-reference.md](../../../docs/cli-reference.md)). Typical talking-job commands: `sf run`, `sf runs waiting`, `sf runs answer`, `sf runs feedback-decide`, `sf runs wait`, `sf validate`, `sf envelope get`, `sf artifact read`, `sf providers`.
+Use the `sf` CLI. Command names and flags live in Stageflow docs: CLI reference (checkout: [docs/cli-reference.md](../../../docs/cli-reference.md)). Typical talking-job commands: `sf run`, `sf runs waiting`, `sf runs answer`, `sf runs feedback-decide`, `sf runs wait`, `sf validate`, `sf envelope get`, `sf artifact read`, `sf providers`; for evals, `sf eval validate`, `sf eval run --json`, `sf eval runs`, `sf eval compare`.
 
 Probe before each mutating `sf runs` verb. If the probe is **up** or the command refuses because a host is up, continue that gate via MCP — do not start a second mutating writer, and do not start `sf mcp` as a disposable bridge.