@evolvingmachines/evolve 0.0.55-project-sable.20260916.db0bc0bb

package/docs-evals/core-concepts/tasks.mdx

@@ -0,0 +1,216 @@
+---
+title: "Tasks"
+description: "The task format: one directory with an instruction, an environment, a verifier, and an optional reference solution."
+---
+
+A task is a directory. The agent reads the instruction and works inside the environment. The verifier runs afterwards and scores the work.
+
+```text
+my-task/
+├── instruction.md    what the agent is asked to do
+├── task.toml         timeouts, network, resources, verifier mode
+├── pre_artifacts.sh  optional: collects the agent's work after the run
+├── environment/
+│   └── Dockerfile    the box the agent works in
+├── tests/
+│   └── test.sh       the verifier: writes the reward
+└── solution/
+    └── solve.sh      optional reference solution
+```
+
+`task.toml`, `instruction.md` and `tests/test.sh` are required.
+
+## instruction.md
+
+Plain Markdown, handed to the agent unchanged at the start of the trial. Say what done looks like, use absolute paths (`/app/out.json`, not `out.json`), and describe the output's shape, so the instruction and the verifier agree.
+
+## environment/
+
+A `Dockerfile` that builds the box the agent works in, or a `docker-compose.yaml` when the task needs several containers; the agent then runs in the `main` service. The image is built once, when the dataset is published. A task can instead name a prebuilt image with `docker_image` in `task.toml` and omit the folder: any public registry, with the tag or digest pinned. An untagged or `:latest` reference is refused at import.
+
+The agent works in `/app`, and its git state is the submission, so seed a git baseline in the Dockerfile: `git init`, `git add -A`, one commit.
+
+The image's `ENTRYPOINT` runs before the agent starts, so a service the image starts is running when the agent arrives. Keep-alive entrypoints such as `sleep infinity` start nothing, and `CMD` is never run. A service that dies before the agent starts fails the trial with its exit code. An image that starts a service cannot be verified in `separate` mode: verify in `shared` mode, or give the verifier its own `tests/Dockerfile`.
+
+## tests/
+
+`tests/test.sh` is the verifier. It runs in the task's working directory after the agent finishes, checks the work, and writes the reward to one of two files.
+
+- `/logs/verifier/reward.txt`: one number in `[0, 1]`, usually `1` or `0`.
+- `/logs/verifier/reward.json`: a JSON object of named numbers, for a score with several parts.
+
+When both exist, `reward.json` wins. The primary reward is the value under `"reward"`, else the single value when the object has one key. Several named scores with no `"reward"` still score the trial, as metrics with no primary reward, and the task is left out of pass@k. Everything the script prints is kept as the verifier log.
+
+```bash
+#!/bin/bash
+set -uo pipefail
+
+if uvx pytest /tests/test_outputs.py; then
+  echo 1 > /logs/verifier/reward.txt
+else
+  echo 0 > /logs/verifier/reward.txt
+fi
+```
+
+Everything under `tests/` is carried onto the verifier beside `test.sh`.
+
+`tests/Dockerfile` is built only for a `separate` verifier that pins no `docker_image`. Otherwise it is never built, the task imports with the `tests_dockerfile_not_built` note, and anything the recipe would install must already be in the image.
+
+## pre_artifacts.sh
+
+Optional. It runs in the agent's sandbox after the agent finishes and writes the work to hand on under `/logs/artifacts`. Uncommitted work is committed first. Without it, the paths listed under `artifacts` are collected for you.
+
+```bash
+#!/bin/bash
+set -uo pipefail
+mkdir -p /logs/artifacts
+git diff --binary "$(git rev-list --max-parents=0 HEAD)" HEAD > /logs/artifacts/model.patch
+```
+
+## solution/
+
+Optional. `solution/solve.sh` is the reference solution. `evolve check` runs it to confirm the task is solvable. It is archived at publish, gates nothing, and the agent is never given it.
+
+## task.toml
+
+The keys a task author sets. Everything else has a default.
+
+```toml
+[agent]
+timeout_sec = 3600
+
+[verifier]
+timeout_sec = 600
+environment_mode = "shared"
+
+[environment]
+network_mode = "allowlist"
+allowed_hosts = ["pypi.org"]
+cpus = 2
+memory_mb = 4096
+storage_mb = 10240
+```
+
+- **Timeouts.** `agent.timeout_sec` and `verifier.timeout_sec`, in seconds. The defaults are 3600 s for the agent and 600 s for the verifier; a declared value always wins. A job can stretch them with `--timeout-multiplier`.
+- **Network access.** `environment.network_mode` is `no-network` (the agent reaches nothing but its model), `allowlist` (only the hosts in `allowed_hosts`, accepted with this mode only), or `public` (the open internet, and the default). The older `allow_internet = false` means `no-network`, `true` means `public`. Only `no-network` makes the spend cap airtight; see [models](/core-concepts/models).
+- **Verifier network.** A `shared` verifier sees the agent box's network; a `[verifier] network_mode` that differs from it is refused at import. A `separate` verifier box uses the mode of `[verifier.environment]`, else a copy of `[environment]`'s, with `[verifier] network_mode` overriding either. The platform never seals a verifier on its own.
+- **Verifier mode.** `verifier.environment_mode` is `shared`, the verifier runs inside the agent's sandbox after the agent finishes and its credentials are revoked, or `separate`, the verifier boots a fresh copy of the environment and judges only what the task lists under `artifacts`. Without that list a separate verifier sees none of the agent's work, and even the reference solution scores 0. A `[verifier.environment]` table implies `separate`. Only settled separate-mode trials can be [regraded](/core-concepts/jobs#derive-a-new-job).
+- **Resources.** `environment.cpus`, `memory_mb`, `storage_mb`, and `gpus` with `gpu_types`. Omitted, the provider's default sizing applies. A size above a provider's ceiling refuses on that provider; above every ceiling it is refused at import. The ceilings are on [sandboxes](/core-concepts/sandboxes).
+- **Artifacts.** A top-level `artifacts = ["/app/out.json"]` list names the paths carried out of the agent's sandbox: into a separate verifier, and, on a separate-mode trial, into the trial's stored artifacts.
+
+## Environment variables
+
+A task's `[environment.env]` table has two value kinds.
+
+```toml
+[environment.env]
+APP_MODE = "ci"                        # literal: lands in the box as written
+GITHUB_TOKEN = "${GITHUB_TOKEN}"       # template: a secret the job must attach
+LOG_LEVEL = "${LOG_LEVEL:-info}"       # template with a fallback
+```
+
+A literal lands in the box as written; it is dataset content, so it must never be a secret. A template, `${VAR}` as the whole value, asks for a secret the job attaches with `--secret`, matched by env name; see [secrets](/core-concepts/secrets). `${VAR:-default}` falls back to the default when nothing is attached. A job that leaves a template unsatisfied is refused with `secret_not_attached`, and a `${...}` inside a longer string is refused at import.
+
+The image's own start process sees the literals only; templates reach the agent and everything it launches.
+
+## LLM judges
+
+A verifier can grade with a language model. The task asks for the credential with an environment template in `task.toml`:
+
+```toml
+[verifier.env]
+ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"
+```
+
+The verifier receives a credential for the platform's gateway under that name, with the matching base-URL variable set beside it, so `litellm`-style clients and `rewardkit` work unchanged and offline. The recognized names are `ANTHROPIC_API_KEY` and `OPENAI_API_KEY` with their `_API_BASE` and `_BASE_URL` companions, each as the entire value; `"Bearer ${ANTHROPIC_API_KEY}"` is refused at import. Any other `${VAR:-default}` resolves to its default, and a non-judge template with no default is refused.
+
+The rubric names the judge model, or `rewardkit`'s own default, `anthropic/claude-sonnet-4-6`, applies; the credential admits its key's model family only. `--ve REWARDKIT_JUDGE=<agent>` and `--ve REWARDKIT_MODEL=<model>` override the rubric's `[judge]` for one job, in both verifier modes; no other `--ve` key is accepted. The judge's spend is shown apart from the agent's, on the trial and on the job. Judge tasks are not regradable yet.
+
+## Multi-step tasks
+
+A task can walk the agent through ordered steps in one shared environment. A `steps/` directory holds one sub-directory per step in place of the root `instruction.md`, `tests/` and `solution/`, and `[[steps]]` entries in `task.toml` declare them in order. `environment/` stays at the root and the filesystem persists from step to step.
+
+```text
+migrate-then-prove/
+├── task.toml                [[steps]] entries, in order
+├── environment/
+│   └── Dockerfile
+├── tests/                   optional shared helpers for every step's verifier
+└── steps/
+    ├── 01-migrate/
+    │   ├── instruction.md
+    │   ├── tests/           merged over the shared tests/
+    │   └── workdir/         optional files landed before the step; its setup.sh runs first
+    └── 02-prove/
+        ├── instruction.md
+        └── tests/
+```
+
+```toml
+multi_step_reward_strategy = "mean"    # "mean" (default) or "final"
+
+[[steps]]
+name = "01-migrate"
+min_reward = 0.5                       # below this the trial stops here
+
+[[steps]]
+name = "02-prove"
+
+[steps.agent]
+timeout_sec = 600                      # this step's override of the task-level timeout
+```
+
+Each step has its own instruction, its own verifier (`tests/` merged over the shared `tests/`), an optional `workdir/` whose `setup.sh` runs first, and its own healthcheck, `timeout_sec` under `[steps.agent]` and `[steps.verifier]`, `[steps.verifier].env` and `artifacts` list. Each executed step is verified when it ends, and the results are on the trial as `step_results`.
+
+`multi_step_reward_strategy` is `"mean"` (the default, over the steps that produced a result) or `"final"` (the last executed step's result). `min_reward` on a step ends the trial early when the step under-scores: a number gates the primary reward, a map gates each key. A step that fails, or whose healthcheck fails, ends the trial there. A retried trial starts again from the first step.
+
+Refused at import: a per-step `user`, network declaration, or verifier environment or mode. `[[steps]]` and `steps/` must agree both ways.
+
+## MCP servers
+
+A task can hand its agent MCP tools, registered into the harness's native MCP configuration beside any the arm carries.
+
+```toml
+[[environment.mcp_servers]]
+name = "docs-search"
+transport = "streamable-http"          # or "sse" (the default); "http" is read as streamable-http
+url = "http://mcp-server:8000/mcp"     # a compose service, reachable with no egress
+
+[[environment.mcp_servers]]
+name = "sqlite"
+transport = "stdio"                    # launched inside the box by the harness
+command = "uvx"
+args = ["mcp-server-sqlite", "--db-path", "/app/data.db"]
+```
+
+A `stdio` server declares `command` and `args`; it inherits `[environment.env]`, and there is no `env` field. A remote server declares an `http` or `https` `url` with no credential in it. The URL must be reachable under the task's own declarations, checked at import: the agent's own container, a compose service by name, or a public host the network policy admits. A `stdio` server that downloads itself (`npx …`, `uvx …`) fails at run time under `no-network`; bake it into the image.
+
+## Healthcheck and the agent user
+
+`[environment.healthcheck]` keeps a slow-starting environment from meeting the agent too early: the command runs before the agent is installed, and the agent starts only once it exits `0`.
+
+```toml
+[environment.healthcheck]
+command = "curl -fsS http://localhost:8000/health"
+interval_sec = 5           # every field below command is optional; these are the defaults
+timeout_sec = 30
+start_period_sec = 0
+start_interval_sec = 5
+retries = 3
+```
+
+Inside `start_period_sec` a failure does not count. After it, `retries` failures spaced by `interval_sec`, each bounded by `timeout_sec`, fail the trial before any agent spend. On a multi-container task the check sees the `[environment.env]` literals; on a single-container task it sees none of the table, and templates are visible to no healthcheck.
+
+`[agent] user = "dev"` runs the agent as that user, by name or numeric uid. It must exist in the image (`RUN useradd -m dev`); a missing user fails the trial before any spend. Omitted, the image's `USER` applies, root by default. `USER app:group` in the Dockerfile is refused at import, and so is `[verifier] user` other than `root`.
+
+## Not supported
+
+- **Computer-use and desktop tasks.** `[environment].os` other than Linux is refused at import. A task whose instruction assumes a desktop is not refused; it runs on a Linux box without one.
+- **Run-level trajectory seeding.** No job flag names a session to resume from. Task-level seeding works: a `trajectory.json` beside `instruction.md` (or in the first step's directory) is seeded into the agent's session on `claude` and `codex`; other harnesses refuse before any spend. `[agent] load_trajectory` and `resume_trajectory` keys are refused.
+- **Verifier scripts whose PEP-723 header needs a package index.** Under `no-network` the verifier box has no index (only `rewardkit` and `litellm` resolve, on a judge task), so such a task is refused at import. Under `allowlist` or `public`, or without `uv run` or `uvx` in `tests/`, nothing changes.
+
+## From tasks to a dataset
+
+A dataset is a folder of task directories, published to the catalog with `evolve dataset publish`. See [datasets](/core-concepts/datasets).
+
+This is the Harbor task format; the full specification is at [docs.harborframework.com](https://docs.harborframework.com/core-concepts/tasks/overview).

package/docs-evals/core-concepts/trial-outputs.mdx

@@ -0,0 +1,103 @@
+---
+title: "Trial outputs"
+description: "What a finished trial gives you, file by file, and in which formats."
+---
+
+A finished trial leaves five kinds of output: its record, its parsed trace, its trajectory, the raw streams of the agent and the verifier, and the agent's home folder. All of them download as one directory, and each can be read on its own.
+
+## The trial directory
+
+`evolve trial download <trial-id> -o trials/` saves the trial under `trials/<trial-id>/`. A file the trial never produced is absent, never empty.
+
+```text
+<trial-id>/
+├── config.json             the trial's identity: its task and its agent
+├── result.json             the outcome: status, reward, verifier verdict, exception
+├── evolve.json             the platform's record: cost and tokens per lane, provider, regrade lineage
+├── exception.txt           present when the trial ended with an exception
+├── agent/
+│   ├── trajectory.json     the ATIF trajectory, when stored
+│   ├── trace-parsed.jsonl  the parsed trace, one {seq, type, data} per line
+│   ├── codex.txt           the harness's stdout, under the harness's own name
+│   ├── stderr.log          the harness's stderr
+│   ├── agent-home.json     the home capture record
+│   └── .codex/             the captured home, at its real names
+└── verifier/
+    ├── test-stdout.txt     everything the verifier printed
+    └── reward.json         the rewards map
+```
+
+The stdout file takes the harness's name: `claude-code.txt`, `codex.txt`, `gemini-cli.txt`, `qwen-code.txt`, `kimi-code.txt`, `opencode.txt` or `droid.txt`, and `stdout.log` for an agent you registered yourself. The tree also holds `agent/sessions/` (claude, codex), `agent/qwen-sessions/` (qwen) and `agent/opencode/` (opencode): copies of the session files at the paths other tools expect.
+
+`result.json` states `agent_result.cost_usd` only once the spend is final; until then it is null there, and `evolve.json` carries the current figure with its `spend_source`. Tokens are always stated.
+
+The job download, `evolve job download`, adds to every trial its `lock.json`, its `trial.log`, the raw `verifier/reward.txt`, `steps/<name>/verifier/reward.json` on a multi-step trial, the home byte for byte, and `artifacts/` with a `manifest.json` and each collected artifact at its source path (separate-mode trials only). Everything platform-specific in the archive's record files rides under an `x_evolve` key: a trial's spend is `x_evolve.spentUsd` with its `spendSource`, and the job config's agents carry `x_reasoning_effort` and `x_preset`.
+
+## The agent's home folder
+
+The agent's home is the harness's own working state: its session files, its settings, and the transcripts of any subagents it ran. The platform captures it after the run and keeps it with the trial, so the run can be inspected the way the agent saw it. In the download it sits under `agent/` at its real names, with `agent-home.json` beside it: every captured file's path, size, sha256 and mode, and every file that was skipped, with the reason.
+
+The trial download carries the text view of the home. A file that is not UTF-8 text, such as a database, is named in the record and left out; the job download carries every byte. Two caches are left out on purpose, codex's `.codex/.tmp/plugins` and opencode's `.config/opencode/node_modules`; the record lists them under `excluded`, and anything else it could not carry under `skipped`, with the reason.
+
+From the SDK, `trials().artifact(id, "agent-home")` returns the same text view as a map from sandbox path to content. A home too large to serve as one text view is refused with a 413 `invalid_input`; the job download carries it whole.
+
+## The parsed trace
+
+Every harness prints its run in its own format. The platform parses that stream into one vocabulary, the parsed trace, which the dashboard shows and `evolve trial trace` prints. Each event is one JSON object: `seq`, a position that also serves as a resume cursor; `type`, the kind of event; and `data`, the harness's update as parsed.
+
+```json
+{"seq": 0, "type": "prompt", "data": {"_prompt": {"text": "Create hello.txt containing Hello."}}}
+{"seq": 1, "type": "agent_message_chunk", "data": {"update": {"sessionUpdate": "agent_message_chunk", "content": {"type": "text", "text": "I'll create the file."}}}}
+{"seq": 2, "type": "tool_call", "data": {"update": {"sessionUpdate": "tool_call", "toolCallId": "call-1", "title": "Write hello.txt", "kind": "edit", "status": "pending"}}}
+{"seq": 3, "type": "tool_call_update", "data": {"update": {"sessionUpdate": "tool_call_update", "toolCallId": "call-1", "status": "completed"}}}
+{"seq": 4, "type": "usage", "data": {"update": {"sessionUpdate": "usage", "scope": "call", "usage": {"promptTokens": 812, "completionTokens": 64}}}}
+```
+
+The kinds are `prompt` for the instruction the agent received, `user_message_chunk`, `agent_message_chunk` and `agent_thought_chunk` for text, `tool_call` and `tool_call_update` for a tool call and its later status and output, `plan` for the agent's plan entries, `usage` for token counts, `error` for a failure the harness reported, and `trace_loss` when events were lost. A tool call's `kind` is one of the parser's kinds, such as `read`, `edit` or `execute`, and its `status` moves through `pending`, `in_progress`, `completed` or `failed`.
+
+Once the trial is finished, the gateway's own `usage` events follow the harness's, one per model call, with the tokens and the cost as priced. Their `data.update.source` is `gateway`, and they are the only usage lines that carry tokens and money.
+
+```bash
+evolve trial trace <trial-id>
+evolve trial trace <trial-id> --grep 'permission denied' --tail 50
+evolve trial trace <trial-id> --cursor 140
+```
+
+`trial trace` prints one line per event: the position, the type, and the data cut to 140 characters. A gateway usage event prints as `gateway <model>` with its tokens and cost. `--type`, `--grep` and `--tail` filter on the server; `--cursor <seq>` resumes after the last event you saw; `--json` prints each event in full. While the trial runs, the dashboard's trial viewer shows the trace as it grows.
+
+## The ATIF trajectory
+
+`agent/trajectory.json` is the same run in ATIF, the Agent Trajectory Interchange Format, the trajectory format Harbor defines, at version `ATIF-v1.7`. The platform builds it from the stored parsed trace.
+
+Its top-level fields are `schema_version`, `agent` with the harness's name, version and model, and `steps`, the ordered system, user and agent turns, with `session_id`, `trajectory_id`, `final_metrics`, `subagent_trajectories` and `extra` optional. The instruction becomes the first `user` step. An `agent` step carries its message, its `tool_calls` under the harness's own tool names, and the `observation` those calls returned.
+
+```json
+{
+  "schema_version": "ATIF-v1.7",
+  "agent": {"name": "codex", "version": "0.31.0", "model_name": "gpt-5.5"},
+  "steps": [
+    {"step_id": 1, "source": "user", "message": "Create hello.txt containing Hello."},
+    {
+      "step_id": 2,
+      "source": "agent",
+      "message": "I'll create the file.",
+      "tool_calls": [{"tool_call_id": "call-1", "function_name": "apply_patch", "arguments": {"path": "hello.txt"}}],
+      "observation": {"results": [{"source_call_id": "call-1", "content": "Success"}]}
+    }
+  ]
+}
+```
+
+```bash
+evolve trial download <trial-id> --stream trace-atif
+```
+
+## Raw streams and the verifier log
+
+Three artifacts are kept byte for byte: `trace-stdout`, the harness process's stdout; `trace-stderr`, its stderr; and `verifier`, everything the verifier printed. The reward itself is in `result.json` and in `verifier/reward.json`. Print one of them instead of saving:
+
+```bash
+evolve trial download <trial-id> --stream verifier
+```
+
+From the SDK, `trials().artifact(id, "verifier")` returns the same text, and `null` when the trial never stored it.

package/docs-evals/core-concepts/trials.mdx

@@ -0,0 +1,103 @@
+---
+title: "Trials"
+description: "One agent's attempt at one task: its record, its trace, and its artifacts."
+---
+
+A trial is one attempt of one task by one agent arm. Trial ids are global: every trial command takes the trial id alone, and the trial's record carries its job id. Treat ids as opaque strings.
+
+```bash
+evolve trial show <trial-id>
+```
+
+`trial show` prints the task, the agent and model, the status, the reward, the tokens, what the trial spent and, when it failed, the exception. Spend prints as a plain amount once the meter has settled it, as `at least $X` while it is still a lower bound, and as `-` when nobody has measured it yet.
+
+While a trial runs, `attempt_phase` says which step it is in (`prepare`, `build`, `boot`, `install`, `agent`, `verify`, `persist`), and `live_spent_usd` is a lower bound on its spend so far, refreshed about every 30 seconds and null before the first reading; `usage` carries the same reading with its tokens. On a settled trial read `agent_result.cost_usd`; `spend_source` says whether it is final (`measured`).
+
+## The trace
+
+Every trial records a parsed event trace: the instruction, each agent turn, every tool call and its result. Print it, filtered on the server.
+
+```bash
+evolve trial trace <trial-id>
+evolve trial trace <trial-id> --type <event-type>
+evolve trial trace <trial-id> --grep 'permission denied' --tail 50
+```
+
+`--type` keeps events of exactly one type. `--grep` is a case-insensitive regex over type and content. `--tail` keeps the last N matching events. To search every trial of a job in one pass:
+
+```bash
+evolve job grep <job-id> 'permission denied'
+```
+
+## Artifacts
+
+Beside the parsed trace, a trial keeps the raw record of its run. One vocabulary names the pieces everywhere; every file and format is described on [trial outputs](/core-concepts/trial-outputs).
+
+- `trace-parsed`: the parsed event trace
+- `trace-stdout`: the agent process's stdout, byte for byte
+- `trace-stderr`: the agent process's stderr
+- `trace-atif`: the normalized trajectory, in the ATIF format
+- `agent-home`: the text view of the agent's home folder, captured after the run
+- `verifier`: everything the scoring step printed
+- `trajectory`: reserved for the harness's own native session file, not served yet
+
+Stream one artifact to stdout:
+
+```bash
+evolve trial download <trial-id> --stream verifier
+```
+
+## Download the trial tree
+
+Without `--stream`, the trial is saved whole under `<dir>/<trial-id>/` in the trial tree layout, plus `evolve.json`.
+
+```bash
+evolve trial download <trial-id> -o trials/
+```
+
+```text
+config.json               task and agent
+result.json               status, reward, verifier verdict, exception, phase clocks
+agent/trajectory.json     the normalized ATIF trajectory
+agent/<harness>.txt       the harness's stdout stream (claude-code.txt, codex.txt, ...)
+agent/stderr.log          the harness's stderr stream
+agent/trace-parsed.jsonl  the parsed event trace
+agent/.claude/ ...        the captured agent home at its real names
+agent/agent-home.json     the capture record: every file's path, size and digest
+verifier/test-stdout.txt  the verifier log
+verifier/reward.json      the rewards map, when the verifier produced one
+exception.txt             when the trial carries an exception
+evolve.json               the platform's own record: cost, tokens, provider
+```
+
+An artifact the trial never recorded is an absent file, never an empty placeholder. `--overwrite` replaces an existing folder. The job archive from `evolve job download` holds the same tree per trial, plus what only the server can add: `lock.json`, `trial.log`, `artifacts/`, and the raw `verifier/reward.txt`.
+
+## Act on one trial
+
+```bash
+evolve trial retry <trial-id>
+evolve trial regrade <trial-id>
+evolve trial stop <trial-id> <trial-id>
+```
+
+`retry` runs a settled trial again, `regrade` re-runs only its verifier, and each produces a new job holding the one new trial. `stop` ends in-flight trials without cancelling their job; each stopped trial settles `CANCELLED` and is charged its spend so far.
+
+## Statuses
+
+A valid reward, 0 included, is `SCORED`; a failure is never reported as a fabricated zero.
+
+- `QUEUED`: waiting for a sandbox slot.
+- `RUNNING`: the agent phase; `attempt_phase` says which step.
+- `SCORING`: the agent finished; the verifier is running.
+- `SCORED`: a valid reward is in `reward`.
+- `SCORING_ERROR`: the verifier crashed or wrote a reward outside `[0, 1]`; read `exception_info`.
+- `INFRASTRUCTURE_ERROR`: the trial was lost before a result was recorded; retried automatically, then `resume`.
+- `BUDGET`: a budget above the trial's own cap refused it. The exception type is `ApiUsageLimitError` and the message carries `user:` (your credits), `team:` (the organization's monthly budget) or `other:` (the platform's global stop). Never retried on its own; add credits or raise the budget, then `resume`.
+- `INDETERMINATE`: the platform cannot tell whether the trial completed.
+- `CANCELLED`: stopped before settling; the exception type is `CancelledError`.
+
+`SCORING_ERROR` and `INDETERMINATE` carry the last 4000 characters of the verifier's own output in `exception_message`. Job statuses are on [jobs](/core-concepts/jobs#statuses).
+
+<Card title="trial reference" icon="terminal" href="/cli-reference/trial">
+  Every flag of `evolve trial`.
+</Card>

package/docs-evals/core-concepts/upload.mdx

@@ -0,0 +1,45 @@
+---
+title: "Upload a job"
+description: "Bring a job you ran elsewhere onto the platform as a finished job."
+---
+
+A job you ran elsewhere, in the Harbor job layout, uploads as a terminal job. Its trials, traces and rewards become a job you read like any other.
+
+```bash
+evolve upload ./job-2026-08-27__12-00-00 -d terminal-bench-4@4.0
+```
+
+`upload` takes the job directory, or its `.tar.gz`. With `--from <url>` it takes instead a public https URL of the archive, which the server fetches itself. `-d name[@version]` links the uploaded trials to a published dataset version by task name. The command follows the import until the job exists.
+
+`--no-wait` returns at once with the import id.
+
+```bash
+evolve job imports --status RUNNING
+evolve job import <import-id> --watch
+```
+
+`job imports` lists your uploads, newest first. `job import --watch` re-attaches to one and follows it to the job, or to its typed failure. A large upload resumes after a dropped connection, and its import id is available before the transfer ends.
+
+## What lands
+
+The trials' facts, never a re-judgment: a rewarded trial arrives `SCORED` with its rewards untouched, a trial with no rewards arrives `INDETERMINATE`, and an errored trial keeps its exception. The trajectory, the agent's logs and home folder, and the verifier log land where a native trial's do, so every trace and artifact read works unchanged. Prior `analysis.json`, `lock.json`, `artifacts/` and `steps/` content are not imported.
+
+The ingested job carries `upload`: the archive's own `original_job_id` and `original_job_name`, `uploaded_at`, the uploader's `reported_totals`, and `task_links`. It is null on every job this platform ran. Resume, retry and regrade refuse an uploaded job with `job_uploaded`; analyze works on it unchanged.
+
+An uploaded job never claims an execution: its `sandbox_provider` is null, the CLI prints `ported`, and the metered spend and token fields stay null. The archive's own figures ride `trial.upload.reported_agent_result` and `upload.reported_totals`, and `job show` prints them as `reported $X.XX (N/M trials reporting)`.
+
+## Task linkage
+
+Every trial's `lock.json` names its task by a content hash, and every task the platform imports carries the same hash, so an uploaded job links to the dataset it ran against by itself. A linked trial's analyses run with the task folder; an unlinked trial's run without it, and the record says why: `hash_mismatch`, `task_not_in_dataset`, `no_task_digest`, `no_hash_match`, `no_dataset_named` or `dataset_ambiguous`. A name alone never links.
+
+`-d` is the override: it links by task name to that one version, whatever the hashes say, and settles an ambiguous hash. A hint you cannot use fails the import. The outcome is printed by the upload follow, repeated by `evolve analyze` before it spends, and recorded on the job's `upload.task_links`, the import's `task_links` and each trial's `upload.link`.
+
+## Bounds and duplicates
+
+The only size limit is the archive ceiling published as `limits.uploads.job_archive_bytes` (`upload_too_large` past it). A trial with a single file over 5 GiB, or a trajectory too large to parse, is skipped instead of failing the job: the import lists it under `skipped_trials` as `trial_too_large`, and `job import` prints one line per skipped trial. A trial whose `result.json` cannot be read fails the whole import with `invalid_trial`.
+
+Re-uploading a job you already uploaded fails with `job_already_uploaded`, naming the existing job; delete it, then upload again. Another user uploading the same archive gets their own copy.
+
+<Card title="upload reference" icon="terminal" href="/cli-reference/upload">
+  Every flag of `evolve upload`.
+</Card>

package/docs-evals/dashboard/checks.mdx

@@ -0,0 +1,15 @@
+---
+title: "Checks"
+description: "Read a job's task quality checks in the dashboard."
+---
+
+A job's CHECK tab lists the task quality checks run on the job's datasets.
+
+{/* retake after the UI pass, from a non-admin account */}
+<Frame caption="The CHECK tab of a job whose tasks have not been checked">
+  <img src="/images/job-check.png" alt="The CHECK tab: the sentence that no task of this job has been checked, and the evolve check command that would check its dataset." />
+</Frame>
+
+Until a check exists, the tab shows the command that would start one, `evolve check -d <dataset@version>`, with the job's dataset filled in. Checks start from the command line; see [check](/core-concepts/check).
+
+Once a dataset has been checked, the tab lists one row per task with its dataset and its result. Click a row to open the check in the [trial viewer](/dashboard/trial-viewer): the events are the checker's own trace, and its result document is on the DOWNLOAD menu.

package/docs-evals/dashboard/datasets.mdx

@@ -0,0 +1,24 @@
+---
+title: "Datasets"
+description: "Browse the catalog, and each dataset's tasks and versions, in the dashboard."
+---
+
+The Datasets page lists every dataset you can run: the platform's public datasets and your own private ones.
+
+{/* retake after the UI pass, from a non-admin account */}
+<Frame caption="The datasets catalog">
+  <img src="/images/datasets.png" alt="The Datasets page: a search box, an ALL, PUBLIC and PRIVATE filter, and a table of datasets with task count, version, visibility, status and update time." />
+</Frame>
+
+Search by name, and filter with ALL, PUBLIC or PRIVATE. Each row shows the dataset's task count, its active version, its visibility, its status and when it last changed. Tick rows to delete datasets you own. There is no publish button: publishing is [`evolve dataset publish`](/core-concepts/datasets).
+
+Click a dataset to open it.
+
+{/* retake after the UI pass, from a non-admin account */}
+<Frame caption="One dataset: its tasks, with a verdict per sandbox provider">
+  <img src="/images/dataset.png" alt="One dataset's page: the visibility, version and task count in the header, TASKS and VERSIONS tabs, and a table of tasks with GPUs, agent timeout, verifier timeout, skills and providers." />
+</Frame>
+
+The header shows the visibility, the active version and the task count. The TASKS tab lists every task of that version with its GPU count, its agent and verifier timeouts, its skills, and the sandbox providers that can run it. An arrow after a provider means the trial is rerouted to the provider after the arrow. Hover a refused provider for the reason.
+
+The VERSIONS tab lists the dataset's versions. DELETE appears only on a dataset you may delete: your own private dataset, or a public one when you are the platform admin. It removes the dataset.

package/docs-evals/dashboard/jobs.mdx

@@ -0,0 +1,65 @@
+---
+title: "Jobs"
+description: "Start a job from the browser, follow it, and read its results."
+---
+
+The Jobs page lists your jobs, newest first.
+
+{/* retake after the UI pass, from a non-admin account */}
+<Frame caption="The jobs list">
+  <img src="/images/jobs.png" alt="The Jobs page: a NEW JOB button, a search box, a status filter, and a table of jobs with status, datasets, agents, trials, cost and creation time." />
+</Frame>
+
+Search by name, and filter by status: ALL, QUEUED, RUNNING, CANCELLING, COMPLETED, CANCELLED or FAILED. Each row shows the job's datasets, its agents with their models, how many trials have settled out of the total, what it has spent, and when it started. Tick rows to cancel live jobs or delete finished ones.
+
+## New job
+
+NEW JOB opens the form. It takes the same choices as `evolve run`.
+
+{/* retake after the UI pass, from a non-admin account */}
+<Frame caption="The New job form">
+  <img src="/images/new-job.png" alt="The New job form: a Sources list of datasets to tick, an Agents section with agent, model and reasoning effort, an ADD AGENT button, and a Summary rail with LAUNCH and VALIDATE buttons." />
+</Frame>
+
+- **Sources**: tick one or more datasets, cap the task count, and narrow the tasks with include and exclude patterns over their names.
+- **Agents**: one arm per entry, with its agent, its model (required) and its reasoning effort. Arm options hold the version pin, a preset, a config file and skills. ADD AGENT adds another arm.
+- **Secrets**: attach a secret from your vault, or save and attach a one-off value.
+- **Job settings**: job name, attempts per task, parallel trials, auto-retries, and the spend cap per trial.
+- **Advanced**, collapsed by default: the sandbox provider, the timeout multipliers, the judge overrides for the verifier, and the exceptions to retry on.
+
+The Summary rail counts datasets × agents × attempts. VALIDATE checks the whole form without starting anything. LAUNCH starts the job and opens its page.
+
+## One job
+
+{/* retake after the UI pass, from a non-admin account */}
+<Frame caption="A job's page, on its OVERVIEW tab">
+  <img src="/images/job.png" alt="A job's page: the job id and status in the header, CANCEL, RESUME, DOWNLOAD and RE-RUN JOB buttons, tiles for trials, errors, average reward, cost and max cost, and the OVERVIEW tab with a progress bar and a per-agent table." />
+</Frame>
+
+The header carries the job id and its status. CANCEL stops a running job. RESUME runs a finished job's failed or stopped trials again as a new job.
+
+DOWNLOAD saves a finished job's results as a `.tar.gz` in the standard job layout; while the job runs it is disabled, with the reason on hover. RE-RUN JOB opens the New job form filled in with this job's choices.
+
+Five tiles sum the job up: trials settled out of the total, errors, average reward, cost so far, and the most the job can cost. The average reward shows a value for a job with one agent and one dataset; with more, it shows a dash and the OVERVIEW rows carry the means. Below them, five tabs.
+
+- **OVERVIEW**: a progress bar by trial status, and one row per agent and dataset with its scored trials, errors, mean reward and pass@k.
+- **TRIALS**: every trial, with ALL or ERRORS, its task, agent, status, reward, cost, duration and token counts. Click a row to open the [trial viewer](/dashboard/trial-viewer).
+- **ANALYZE**: the trace analyses of the job's trials, below.
+- **CHECK**: the task quality checks of the job's datasets. See [checks](/dashboard/checks).
+- **CONFIG**: the stored job record as JSON: its id, status, counts, stats and resolved settings.
+
+{/* retake after the UI pass, from a non-admin account */}
+<Frame caption="The TRIALS tab">
+  <img src="/images/job-trials.png" alt="The TRIALS tab: ALL and ERRORS filters and a table of trials with task, agent, status, reward, cost, duration, input tokens and output tokens." />
+</Frame>
+
+## The ANALYZE tab
+
+{/* retake after the UI pass, from a non-admin account */}
+<Frame caption="The ANALYZE tab after every trial was analyzed">
+  <img src="/images/job-analyze.png" alt="The ANALYZE tab: tiles for completed, failed and pending analyses and the analyzer's cost, the policy with model, effort and criteria, and a table of pass, fail, not applicable and unknown counts per criterion." />
+</Frame>
+
+The tab counts completed, failed and pending analyses and the analyzer's own spend. The policy block names the analyzer's model, its reasoning effort and the rubric's criteria. The first table counts, per criterion, how many analyses ruled pass, fail, not applicable or unknown.
+
+Below it, one row per trial carries its label, its verdicts, the analyzer's spend and its summary; click a row to open the analysis in the trial viewer. Start analyses with `evolve analyze <job-id>` or with `--analyze` on `evolve run`, see [analyze](/core-concepts/analyze).

package/docs-evals/dashboard/traces.mdx

@@ -0,0 +1,15 @@
+---
+title: "Traces"
+description: "Every recorded run in one list."
+---
+
+The Traces page lists every run the platform recorded for you, newest first: the trials of your jobs, their regrades, analyses and checks, and the managed-agent sessions the platform recorded, which these pages do not cover.
+
+{/* retake after the UI pass, from a non-admin account */}
+<Frame caption="The traces list">
+  <img src="/images/traces.png" alt="The Traces page: a search box and a table of runs with a trace tag, the sandbox provider, the sandbox id, the agent and model, the start time, the cost and the status." />
+</Frame>
+
+Each row carries a tag naming the kind of run (EVAL for a trial, AGENT for a managed-agent session, REGRADE, ANALYZE, or CHECK), the task or session name, the dataset, the reward when there is one, the sandbox provider and sandbox id, the agent and model, when it started, what it spent, and its status. Search by name. Tick rows to stop running ones or delete finished ones.
+
+Click a row to open it in the [trial viewer](/dashboard/trial-viewer).

package/docs-evals/dashboard/trial-viewer.mdx

@@ -0,0 +1,30 @@
+---
+title: "Trial viewer"
+description: "Read one trial's trace event by event, its tool calls, its spend, and download its artifacts."
+---
+
+Click a trial on a job's TRIALS tab, or a row on the Traces page, to open it.
+
+{/* retake after the UI pass, from a non-admin account */}
+<Frame caption="A scored trial in the viewer">
+  <img src="/images/trial.png" alt="The trial viewer: a header with the task, agent, model, sandbox id, cost and status, a progress bar with previous and next event controls, an EVENTS list on the left, and the selected event's content on the right." />
+</Frame>
+
+The header names the run: its kind (EVAL for a trial, AGENT for a managed-agent session, REGRADE, ANALYZE, or CHECK), its task, its agent and model, its sandbox id, what it spent, and its status.
+
+BACK returns to where you came from. RETRY runs a settled trial again as a new one-trial job. STOP ends a run that is still going.
+
+DOWNLOAD on a trial offers the parsed trace, the ATIF trajectory, the raw stdout, the raw stderr, the agent home folder, the verifier log, or everything at once; on an analysis or a check it offers the verdict document, the raw stdout, the raw stderr, the agent home folder, or everything.
+
+Under the header, the reward and the event navigation: previous, next, and a progress bar you can drag. Type `/` to search the events.
+
+The EVENTS list runs down the left: the prompt, each assistant message and thought, each tool call and its result, and the gateway's own records: one per model call with its tokens and cost, and a run total at the end. Click an event to read it in full on the right. A running trial keeps appending events while you watch.
+
+{/* retake after the UI pass, from a non-admin account */}
+<Frame caption="An analysis run, with the TOOLS panel open">
+  <img src="/images/trial-analysis.png" alt="The trial viewer on an analysis run: a TOOLS summary with the number of calls, the success rate and a bar per tool, then the events list and the selected event's content." />
+</Frame>
+
+The panel above the events has two tabs. TOOLS counts the run's tool calls, its success rate and how many failed, with a bar per tool. SPEND shows what the run spent. The gateway meter at the end of the run states the prompt, cached, cache-write and completion tokens and the cost in dollars.
+
+An analysis, a check or a regrade is read the same way. An analysis's events are the analyzer's own trace, and its verdict document is on the DOWNLOAD menu.