@evolvingmachines/evolve 0.0.65 → 0.0.66

package/skills/evolve-evals/references/core-concepts/datasets.md

@@ -0,0 +1,234 @@
+---
+title: "Datasets"
+description: "Browse ready-to-run tasks or publish your own versioned collection."
+---
+
+A dataset is a named collection of tasks. Use `name@version` for a specific version, or a bare name for the active version.
+
+```bash
+evolve dataset list
+evolve dataset show harbor-examples@1.0
+```
+
+`show` lists tasks, timeouts, and provider compatibility. Use `--search <text>` with `list` to narrow the catalog.
+
+## Publish your tasks
+
+```text
+my-dataset/
+├── hello-world/
+│   ├── instruction.md
+│   ├── task.toml
+│   ├── environment/Dockerfile
+│   └── tests/test.sh
+└── another-task/
+    └── ...
+```
+
+You can publish a single task directory, a directory of tasks, or a directory containing `tasks/`.
+
+### 1. Check the configuration
+
+```bash
+evolve dataset check ./my-dataset
+```
+
+This sends configuration files for validation. It does not upload the corpus or build images. Fix rejected declarations before publishing.
+
+### 2. Publish a version
+
+### Local directory
+
+```bash
+evolve dataset publish \
+  --dir ./my-dataset \
+  --name my-dataset --version 1.0 \
+  --watch
+```
+
+### Git repository
+
+```bash
+evolve dataset publish \
+  --git https://github.com/acme/my-dataset.git \
+  --ref v1.0.0 \
+  --name my-dataset --version 1.0 \
+  --watch
+```
+
+Use a tag or full 40-character commit SHA. Branch names are rejected. `--path <folder>` selects a subdirectory.
+
+### Harbor Hub
+
+```bash
+evolve dataset publish --from hub:cookbook/hello-world --watch
+```
+
+Public Harbor Hub packages supply their own name and version. `--from` also accepts a public HTTPS tarball URL.
+
+The dataset is private to your organization. Its owner can publish more versions under the same name.
+
+### 3. Inspect the result
+
+```bash
+evolve dataset show my-dataset@1.0
+```
+
+Check both ready and failed task counts. If you disconnect during publication, use `evolve dataset watch my-dataset` to resume watching.
+
+## What publication does
+
+```mermaid
+%%{init: {"flowchart": {"nodeSpacing": 16, "rankSpacing": 20, "padding": 10}, "themeVariables": {"fontSize": "14px", "fontFamily": "Google Sans Flex, sans-serif"}}}%%
+flowchart TB
+  A["1. Receive task package"] --> B["2. Validate and import tasks"]
+  B --> C["3. Prepare task images"]
+  C --> D{"Any tasks ready?"}
+  D -->|Yes| E["4. Version READY"]
+  D -->|No| F["4. Version FAILED"]
+```
+
+Tasks build independently. A version can be `READY` while some tasks failed. Whole-dataset and glob selections run its ready tasks and report `build_exclusions`; explicitly selecting a failed task rejects the job.
+
+A ready version becomes active on your dataset. Publishing a new version label leaves the previous active version in place if the new publication fails.
+
+Use a new version label for changed tasks. Publishing an existing label re-imports it in place, so `name@version` selects a label rather than immutable content.
+
+**Note:**
+
+Configuration validation cannot prove that images build or tests are correct. Run [task checks](/core-concepts/check) to inspect task quality.
+
+## Pin or switch versions
+
+```bash
+# Run an exact version.
+evolve run -d my-dataset@1.0 -a codex -m gpt-5.6-luna
+
+# Change what the bare name resolves to.
+evolve dataset activate my-dataset 1.0
+```
+
+Only ready versions accept jobs. Jobs record the version they resolve when created.
+
+## Download your source package
+
+```bash
+evolve dataset download my-dataset@1.0 -o ./corpora
+```
+
+The download includes the original corpus and its `solution/` files. It is owner-only; platform datasets are not downloadable through this command.
+
+## dataset.toml manifest
+
+An optional `dataset.toml` selects the tasks in a package and pins their content. Put it beside the task directories. If the package contains `tasks/`, a manifest in `tasks/dataset.toml` takes precedence over one at the package root.
+
+```toml
+schema_version = "1.0"
+
+[dataset]
+name = "acme/my-dataset"
+version = "1.0"
+description = "Our coding tasks"
+authors = [{ name = "Acme", email = "evals@example.com" }]
+keywords = ["coding"]
+
+[[tasks]]
+name = "acme/hello-world"
+digest = "sha256:<task-content-hash>"
+
+[[files]]
+path = "README.md"
+# Optional: digest = "sha256:<file-content-hash>"
+```
+
+Replace the task digest placeholder with the helper's output below before publishing. A local package with a valid manifest can supply the publication name and version:
+
+```bash
+evolve dataset publish --dir ./my-dataset --watch
+```
+
+The catalog name comes from the short part (`my-dataset`); the `acme/` prefix does not choose the owning Evolve team. Explicit `--name` and `--version` values win. Without a manifest version, pass `--version`. Git and HTTPS archive sources still require explicit publication name and version; public Harbor Hub packages supply defaults.
+
+### Manifest fields and selection rules
+
+| Field | Accepted value and behavior |
+| --- | --- |
+| `schema_version` | String, default `"1.0"`. |
+| `dataset.name` | Required `org/name`. Each part starts with a letter or digit and contains only letters, digits, `.`, `_`, or `-`; `..` is forbidden. |
+| `dataset.version` | Optional nonempty string. |
+| `dataset.description` | String, default `""`. |
+| `dataset.authors` | Array of tables with required nonempty `name` and optional string `email`; default `[]`. |
+| `dataset.keywords` | Array of strings, default `[]`. |
+| `tasks` | Required nonempty array. Each entry has `name` in the same `org/name` grammar and a required `digest`: `sha256:` followed by 64 lowercase hex characters. |
+| `files` | Optional array. Each entry has a simple filename `path` (no directory separators) and optional `digest` in the same SHA-256 spelling. Empty or omitted digest skips hashing that file. |
+
+Unknown root and `[dataset]` fields are rejected. Each listed dataset file must exist beside the manifest, even without a digest.
+
+Task matching first uses an exact string match against `[task].name` in a local `task.toml`, then falls back to the reference's short name matching a task directory. Only listed tasks are imported. Every reference must resolve locally; missing tasks are not fetched from a registry.
+
+Duplicate identical name/digest pairs are collapsed. An ambiguous declared task name, or references mapping the same directory to different digests, rejects the manifest. Missing tasks and any task/file digest mismatch fail publication.
+
+### Compute a task content digest
+
+Task digests use Harbor's content-hash algorithm, not the archive hash or a hash of `task.toml` alone. The helper below requires Python and `pathspec`:
+
+```bash
+python3 -m pip install 'pathspec==1.0.3'
+python3 - ./my-dataset/hello-world <<'PY'
+import hashlib
+import sys
+from pathlib import Path
+import pathspec
+
+root = Path(sys.argv[1])
+paths = [root / name for name in
+         ("task.toml", "instruction.md", "README.md", "trajectory.json")
+         if (root / name).is_file()]
+for name in ("environment", "tests", "solution", "steps"):
+    paths.extend(path for path in (root / name).rglob("*") if path.is_file())
+ignore_file = root / ".gitignore"
+patterns = (ignore_file.read_text().splitlines() if ignore_file.is_file() else
+            ["__pycache__/", "*.pyc", ".DS_Store", "*.swp", "*.swo", "*~"])
+ignored = pathspec.PathSpec.from_lines("gitignore", patterns)
+digest = hashlib.sha256()
+for path in sorted(paths, key=lambda path: path.relative_to(root).as_posix()):
+    relative = path.relative_to(root).as_posix()
+    if any(part.startswith("._") and part != "._" for part in path.relative_to(root).parts):
+        continue
+    if ignored.match_file(relative):
+        continue
+    file_hash = hashlib.sha256(path.read_bytes()).hexdigest()
+    digest.update(f"{relative}\0{file_hash}\n".encode())
+print("sha256:" + digest.hexdigest())
+PY
+```
+
+The hash covers the four listed root files when present and regular files recursively under `environment/`, `tests/`, `solution/`, and `steps/`. Other root files are outside this digest. Paths are relative, use `/`, and sort lexicographically. For each file, the outer SHA-256 receives its path, a NUL byte, the lowercase SHA-256 of its bytes, and a newline.
+
+A task's `.gitignore` supplies the filters when present; otherwise the default patterns shown above apply. This uses PathSpec's `gitignore` pattern semantics, which differ from Git's directory re-inclusion behavior. macOS `._` sidecars are skipped. Use regular files: publication does not support task symlinks.
+
+Dataset-level `[[files]]` digests are simpler: prefix the SHA-256 of that file's bytes with `sha256:`. Recompute affected digests whenever you edit content.
+
+### Package details
+
+Local publication preserves executable permissions and dotfiles. It excludes `.git`, `.DS_Store`, `.venv`, and symlinks.
+
+Custom `metric.py` dataset metrics are not supported. A script beside the manifest or listed in `files` rejects publication; use the platform's metrics.
+
+Large local uploads are resumable. During receipt, the import can be `QUEUED` with `receiving: true`; it starts building after the corpus arrives.
+
+### Ownership and updates
+
+A dataset name belongs to its first publisher. Another owner cannot publish under that name.
+
+The SDK supports deleting an owned dataset, but deletion is rejected once a job references it. The CLI has no dataset delete command.
+
+Git-backed datasets can report when their source tag moves. Automatic upstream import is opt-in through the SDK's `datasets().update()`; a normal publication does not enable it.
+
+**[Dataset commands](/cli-reference/dataset)**
+
+Sources, validation, publication, and downloads.
+
+**[Dataset SDK](/sdk-reference/datasets)**
+
+The Python and TypeScript clients.

package/skills/evolve-evals/references/core-concepts/index.md

@@ -0,0 +1,52 @@
+---
+title: "How evaluations fit together"
+description: "The few concepts you need to read a job and understand its results."
+---
+
+A dataset supplies the tasks. An agent and model attempt them. A verifier scores the work.
+
+```mermaid
+%%{init: {"flowchart": {"nodeSpacing": 16, "rankSpacing": 20, "padding": 10}, "themeVariables": {"fontSize": "14px", "fontFamily": "Google Sans Flex, sans-serif"}}}%%
+flowchart TD
+  D["Dataset version"] --> T["Tasks"]
+  A["Agent configurations<br/>Harness + model + settings"] --> J["Job"]
+  T --> J
+  J --> R["Trials<br/>Task × configuration × attempt"]
+  R --> V["Verifier → reward"]
+  R --> F["Trace + logs + files"]
+```
+
+| Concept | What it means | Example |
+| --- | --- | --- |
+| [Task](/core-concepts/tasks) | An instruction, environment, and grading logic. | Fix a bug and pass the tests. |
+| [Dataset](/core-concepts/datasets) | A named, versioned collection of tasks. | `harbor-examples@1.0`. |
+| [Agent](/core-concepts/agents) | A harness running a selected model. | Codex with `gpt-5.6-luna`. |
+| [Sandbox](/core-concepts/sandboxes) | The isolated environment where a trial runs. | A task image on Daytona, E2B, or Modal. |
+| [Job](/core-concepts/jobs) | A group of trials with shared run settings. | 10 tasks × 2 configurations × 3 attempts = 60 trials. |
+| [Trial](/core-concepts/trials) | One attempt at one task. | One score and its supporting evidence. |
+| [Verifier](/core-concepts/task-verifiers) | The task's code that evaluates the output. | A script writing a reward of `0` or `1`. |
+| [Trace](/core-concepts/trial-outputs) | The recorded agent conversation and actions. | Messages, tool calls, and results. |
+
+## Two ways to examine quality
+
+**[Check the task](/core-concepts/check)**
+
+Review the instruction, environment, solution, and verifier before scaling the evaluation.
+
+**[Analyze the attempt](/core-concepts/analyze)**
+
+Review a recorded trial against a rubric to understand the agent's behavior.
+
+Checks and analyses are optional model-powered operations with their own results and costs. The task's verifier produces the evaluation reward.
+
+## Read the result at the right level
+
+| Level | What to read |
+| --- | --- |
+| **Job configuration** | What was selected and resolved |
+| **Job summary** | Counts, scores, and costs |
+| **Each trial: status** | Whether execution and grading completed |
+| **Each trial: reward** | How the verifier scored the work |
+| **Each trial: evidence** | Trace, logs, and recorded files |
+
+Start with the [quick start](/getting-started/quick-start), then learn how to [configure a job](/core-concepts/jobs).

package/skills/evolve-evals/references/core-concepts/jobs.md

@@ -0,0 +1,223 @@
+---
+title: "Jobs"
+description: "Run a task set across agents and models. Follow each attempt to its result."
+---
+
+A job runs selected tasks against one or more **agent arms**. An arm is an agent, model, and configuration you want to evaluate.
+
+```text
+3 selected tasks
+× 2 agent arms
+× 2 attempts
+= 12 trials
+```
+
+Each of the three tasks runs twice in Arm A and twice in Arm B.
+
+## Start a job
+
+```bash
+evolve run \
+  -d harbor-examples@1.0 -i hello-world \
+  -a codex -m gpt-5.6-luna \
+  --max-trial-spend 1 -r 0 \
+  --watch
+```
+
+| Choose | Flag | Meaning |
+| --- | --- | --- |
+| Task set | `-d name@version` | Repeat for several datasets. |
+| Agent | `-a codex` | The harness that runs the model. |
+| Model | `-m model` | Repeat to compare models with the same harness. |
+| Attempts | `-k 2` | Attempts per task and arm. Default: 1. |
+| Parallel trials | `-n 4` | Job concurrency. Default: 4; maximum: 150. |
+| Label | `--job-name nightly` | A readable name. The job ID remains its identity. |
+
+Use a [config file](#config-files) to compare different harnesses in one job.
+
+## Narrow the task set
+
+```text
+1. Select dataset version
+          ↓
+2. Include names matching -i
+          ↓
+3. Remove names matching -x
+          ↓
+4. Keep up to -l tasks per dataset
+```
+
+`-i` and `-x` accept task-name globs. Quote them so your shell does not expand them.
+
+```bash
+evolve run -d my-dataset@1.0 -a codex -m gpt-5.6-luna \
+  -i 'auth-*' -x 'auth-legacy' -l 20 \
+  --max-trial-spend 2 -r 0 --watch
+```
+
+## Watch it
+
+With `--watch`, the CLI follows events until the job finishes. Without it, the CLI returns the accepted job immediately.
+
+Use the returned ID as `$JOB_ID`:
+
+```bash
+evolve job show "$JOB_ID"
+evolve job trials "$JOB_ID"
+evolve job tasks "$JOB_ID"
+```
+
+**Note:**
+
+**Accepted** means the job was created. **Completed** means every trial settled. Neither means every trial succeeded.
+
+## Spend and retries
+
+Set `--max-trial-spend` to cap each trial attempt’s **agent** model spend. The response records the resolved cap and its model-spend estimate for the full job:
+
+```text
+worst_case_spend_usd
+  = per-trial cap
+  × trial count
+  × (1 + max_retries)
+```
+
+| Cost | Where to read it |
+| --- | --- |
+| Job model spend, including retired retry attempts | `stats.cost_usd` |
+| Verifier judge spend, included in that total | `stats.judge_cost_usd` |
+| Trace analysis spend, separate from trial spend | `stats.analysis.cost_usd` |
+| GPU compute estimate, separate from model spend | `stats.gpu_cost_usd` |
+
+An LLM verifier judge has its own cap: the smaller of the agent cap and $5. Its spend is additional to the estimate above. Analysis also has a separate budget.
+
+The fleet defaults are $200 per trial and 2 infrastructure retries; deployment settings can change them. The job response records the values used. There is no job-wide spend-cap setting.
+
+A trial that reaches its own cap proceeds to verification. An account, team, or platform budget refusal is a separate `BUDGET` outcome.
+
+### Automatic retries and capacity waits
+
+`-r 0` disables configured infrastructure retries. With retries enabled, only eligible `INFRASTRUCTURE_ERROR` trials rerun automatically. Each retry receives a fresh per-trial cap; previous spend remains in the job total.
+
+`--retry-include` and `--retry-exclude` filter exception types. Exclusions win. The same infrastructure failure twice in a row can stop retries early.
+
+A provider that cannot allocate a sandbox can cause a **capacity wait**, even with `-r 0`. Capacity waits use a separate bounded wait budget and do not consume configured retries.
+
+## Read the results
+
+```bash
+evolve job trials "$JOB_ID" --status INFRASTRUCTURE_ERROR,SCORING_ERROR
+evolve job compare "$JOB_ID" "$OTHER_JOB_ID"
+evolve job list --search nightly
+```
+
+Results are grouped by agent arm and dataset. Compare 2–10 jobs at once.
+
+### How to read pass@k
+
+pass@k estimates the chance of at least one successful attempt among `k` attempts, averaged across tasks.
+
+It appears for settled groups with one binary reward per trial. Missing rewards count as failures for this statistic; the trial's own reward remains absent. Non-binary or multi-key rewards do not produce pass@k.
+
+Available `k` values are powers of two and multiples of five up to the smallest per-task attempt count. A single-attempt group has no pass@k.
+
+## Statuses
+
+| Status | Meaning |
+| --- | --- |
+| `QUEUED` | Accepted; waiting for dispatch. |
+| `RUNNING` | Trials are being processed. |
+| `CANCELLING` | Cancellation requested; active trials are stopping. |
+| `COMPLETED` | All trials settled. Some may have errors. |
+| `CANCELLED` | The job was cancelled. |
+| `FAILED` | Reserved in the API; the current worker does not assign it. |
+
+Read [trial statuses](/core-concepts/trials#statuses) to understand individual outcomes.
+
+## Derive a new job
+
+Each command below creates a new job linked through `source_jobs`. The original result stays intact.
+
+| Action | Runs again | Selects |
+| --- | --- | --- |
+| **Resume** | Agent and verifier | Failed or stopped trials from a finished job. |
+| **Retry** | Agent and verifier | All, failed, or explicitly selected settled trials. Scored trials are allowed. |
+| **Regrade** | Verifier only | Eligible trials with recorded verifier inputs. |
+
+```bash
+evolve job resume "$JOB_ID"
+evolve job retry "$JOB_ID" --failed-only
+evolve job retry "$JOB_ID" -t "$TRIAL_ID"
+evolve job regrade "$JOB_ID" --task hello-world
+```
+
+Use full trial IDs with `-t`. Named settled trials can be retried while other trials in the source job are running. Whole-job retry and resume require a finished source.
+
+### Regrade eligibility
+
+Regrade re-runs the recorded verifier snapshot on retained inputs. It does not accept a replacement verifier or pick up edits to the task's tests.
+
+Regrade needs recorded inputs from a separate verifier environment. Shared-verifier, multi-step, and LLM-judge tasks are not supported for regrade.
+
+A whole-job regrade skips ineligible trials and refuses if none remain. A single-trial regrade reports the reason it is ineligible. Uploaded jobs cannot be resumed, retried, or regraded.
+
+## Stop, cancel, delete
+
+| Command | Effect |
+| --- | --- |
+| `evolve job cancel "$JOB_ID"` | Stop the whole job. |
+| `evolve job stop "$JOB_ID" --dataset my-dataset` | Stop trials from one dataset. Other trials continue. |
+| `evolve job delete "$JOB_ID" --yes` | Permanently remove the finished job and its stored results. |
+
+Deletion is restricted to the creator. Active analyses or derived regrades can prevent deletion until they finish.
+
+## Share
+
+```bash
+evolve job share "$JOB_ID" --link
+```
+
+Share by link or email. Recipients get read access; see [Sharing](/core-concepts/sharing).
+
+## Download
+
+```bash
+evolve job download "$JOB_ID" -o results/
+```
+
+This unpacks a Harbor-format job directory under `results/job-<id>/`, with job records and one folder per trial. See [Trial outputs](/core-concepts/trial-outputs).
+
+## Config files
+
+Use YAML or JSON for a repeatable comparison:
+
+```yaml comparison.yaml
+datasets:
+  - name: harbor-examples
+    version: "1.0"
+    task_names: [hello-world]
+agents:
+  - name: codex
+    model_name: gpt-5.6-luna
+  - name: claude
+    model_name: fable
+n_attempts: 2
+max_trial_spend_usd: 1
+retry:
+  max_retries: 0
+```
+
+```bash
+evolve run -c comparison.yaml --print-config
+evolve run -c comparison.yaml --watch
+```
+
+`--print-config` shows the request without submitting it. It does not resolve every server default or validate server acceptance. Use `--org` for team ownership.
+
+**[Run reference](/cli-reference/run)**
+
+Flags, configuration overrides, and output modes.
+
+**[Inspect a trial](/core-concepts/trials)**
+
+Read the score, failure, and trace.

package/skills/evolve-evals/references/core-concepts/models.md

@@ -0,0 +1,73 @@
+---
+title: "Models"
+description: "Select models, compare them, and control metered spend."
+---
+
+`-m` selects the model for an evaluation arm. Every arm must name a model; there is no default.
+
+```bash
+evolve run -d harbor-examples@1.0 -i hello-world \
+  -a codex \
+  -m gpt-5.6-luna -m gpt-5.6-terra \
+  --max-trial-spend 1 --max-retries 0 --watch
+```
+
+Repeating `-m` creates one arm per model. This command runs the selected task once with each model.
+
+## Find a supported model
+
+Each harness has its own model names. Read the live roster instead of assuming a name works with every harness:
+
+```bash
+curl -sS https://dashboard.evolvingmachines.ai/api/meta
+```
+
+The `agents` list includes each harness's models, effort support, and configuration capabilities. The same document is available through [`meta()` in both SDKs](/sdk-reference/meta), without authentication.
+
+## Model access and spend
+
+```text
+1. Agent harness
+         ↓
+2. Evolve gateway
+   Meter usage against the cap
+         ↓
+3. Model provider
+```
+
+An `EVOLVE_API_KEY` supplies platform model access. `--max-trial-spend` limits the agent’s metered spend per trial attempt. Verifier judges and trace analysis have separate budgets. See [job costs](/core-concepts/jobs#spend-and-retries).
+
+Requests that bypass the gateway with separately supplied credentials are not metered by Evolve. A task's `no-network` agent policy restricts model access to the platform gateway.
+
+If you enable your own provider key, supported model requests bill that provider account. Gateway metering and the cap still apply. Job creation still rejects accounts with an exhausted platform credit balance.
+
+## Reasoning effort
+
+`--effort` applies to every arm in the CLI run. Accepted values depend on the harness:
+
+| Harness behavior | How to configure it |
+| --- | --- |
+| Supports effort levels | Use a value from `limits.job.reasoning_efforts` |
+| Qwen thinking mode | Use `off` or `minimal` to disable thinking; `medium` or `thinking` to enable it |
+| Gemini | Do not pass effort; it is unsupported |
+| Custom agent | Configure reasoning in its run command |
+
+When omitted, Evolve resolves the harness's default and records it in `agent_info.reasoning_effort`. The same harness and model at different efforts are distinct arms.
+
+In `/api/meta`, `agents[].effort_support` identifies the mode: `level`, `binary`, or `none`. Qwen's accepted values are listed in `limits.job.binary_effort_values`.
+
+## Analysis and check models
+
+Trace analysis and task checks use the Claude Code harness. Select their model separately:
+
+| Operation | Model option |
+| --- | --- |
+| `evolve analyze` | `-m` |
+| `evolve check` | `-m` |
+| Analysis attached to `evolve run` | `--analyze-model` |
+
+Their default is `openrouter/deepseek/deepseek-v4.1-flash`. Choose from the supported Claude Code roster.
+
+**[Configure a run](/cli-reference/run)**
+
+All model, effort, concurrency, and budget options.

package/skills/evolve-evals/references/core-concepts/multi-step.md

@@ -0,0 +1,93 @@
+---
+title: "Multi-step tasks"
+description: "Give the agent a sequence of instructions while preserving its files."
+---
+
+A multi-step task runs ordered instructions in one shared environment. Files persist between steps; each step starts a fresh agent conversation and receives its own verification result.
+
+```mermaid
+%%{init: {"flowchart": {"nodeSpacing": 16, "rankSpacing": 20, "padding": 10, "subGraphTitleMargin": {"top": 8, "bottom": 24}}, "themeVariables": {"fontSize": "14px", "fontFamily": "Google Sans Flex, sans-serif"}}}%%
+flowchart TB
+  subgraph shared["One shared environment"]
+    A["1. Start environment"] --> B["2. Step 1<br/>New conversation"]
+    B --> C["3. Verify step 1"]
+    C -->|"Threshold met"| D["4. Step 2<br/>New conversation · same files"]
+    D --> E["5. Verify step 2"]
+  end
+  E --> F["6. Select trial reward<br/>mean or final"]
+  C -->|"Threshold missed<br/>Skip step 2"| F
+```
+
+This two-step example uses the same environment for both agent steps and their verifiers. Reward thresholds and failure rules decide whether later steps run.
+
+## Directory layout
+
+- migrate-then-test/
+  - task.toml
+  - environment/
+    - Dockerfile
+  - tests/
+    - test.sh
+  - steps/
+    - 01-migrate/
+      - instruction.md
+      - tests/
+        - test.sh
+      - workdir/
+        - setup.sh
+    - 02-test/
+      - instruction.md
+      - tests/
+        - test.sh
+
+The root `tests/` folder can hold shared verifier files. Each step can include optional `workdir/` files and a `setup.sh` inside that folder.
+
+Each step needs an instruction and a test script, either its own or the shared root `tests/test.sh`. Step test files override shared files with the same path.
+
+## Declare the order
+
+```toml task.toml
+multi_step_reward_strategy = "mean"
+
+[[steps]]
+name = "01-migrate"
+min_reward = 0.5
+
+[[steps]]
+name = "02-test"
+
+[steps.agent]
+timeout_sec = 600
+```
+
+Every declared step must have a matching directory, and every step directory must be declared. There is no root instruction for this layout.
+
+Before a step starts, its optional `workdir/` files are copied into the working directory and `setup.sh` runs if present.
+
+## Rewards and stopping
+
+| Setting | Meaning |
+| --- | --- |
+| `mean` | Average each metric across steps with a verifier result; a missing metric counts as `0`. This is the default |
+| `final` | Use the last executed step's result, including a step that stopped the trial early |
+| `min_reward = 0.5` | Require the `reward` key to be at least `0.5`; a missing key also stops the trial |
+| `min_reward = { accuracy = 0.8 }` | Require each named metric to meet its threshold; a missing metric also stops the trial |
+
+A setup or healthcheck failure stops the remaining steps. A step error with no verifier result also stops the trial. An agent error alone can still continue when verification returns a result and the reward thresholds pass. Infrastructure failures stop the trial.
+
+The trial exposes individual outcomes in `step_results`. A retry starts again at step 1.
+
+## Current limits
+
+| Supported per step | Set once for the whole task |
+| --- | --- |
+| Instruction and tests | Environment image and agent user |
+| Agent and verifier timeouts | Network policies |
+| Verifier environment variables | Shared verifier mode |
+| Healthcheck | Sandbox provider |
+
+Multi-step tasks require shared verification. Compose, separate verification, per-step agent users, per-step network policies, per-step verifier environments, and per-step collection hooks are rejected. Verifiers run as root. These trials cannot be regraded.
+
+`steps.artifacts` does not create a separate snapshot after each step. Save intermediate outputs under distinct paths when you need to inspect them later.
+
+To seed the first conversation on Claude Code or Codex, put an ATIF `trajectory.json` inside the first step’s directory, beside its `instruction.md`. A root trajectory or a trajectory in a later step is rejected. Later steps start fresh conversations.