@evolvingmachines/evolve 0.0.65 → 0.0.66

package/skills/evolve-evals/references/core-concepts/tasks.mdx

@@ -1,216 +0,0 @@
----
-title: "Tasks"
-description: "The task format: one directory with an instruction, an environment, a verifier, and an optional reference solution."
----
-
-A task is a directory. The agent reads the instruction and works inside the environment. The verifier runs afterwards and scores the work.
-
-```text
-my-task/
-├── instruction.md    what the agent is asked to do
-├── task.toml         timeouts, network, resources, verifier mode
-├── pre_artifacts.sh  optional: collects the agent's work after the run
-├── environment/
-│   └── Dockerfile    the box the agent works in
-├── tests/
-│   └── test.sh       the verifier: writes the reward
-└── solution/
-    └── solve.sh      optional reference solution
-```
-
-`task.toml`, `instruction.md` and `tests/test.sh` are required.
-
-## instruction.md
-
-Plain Markdown, handed to the agent unchanged at the start of the trial. Say what done looks like, use absolute paths (`/app/out.json`, not `out.json`), and describe the output's shape, so the instruction and the verifier agree.
-
-## environment/
-
-A `Dockerfile` that builds the box the agent works in, or a `docker-compose.yaml` when the task needs several containers; the agent then runs in the `main` service. The image is built once, when the dataset is published. A task can instead name a prebuilt image with `docker_image` in `task.toml` and omit the folder: any public registry, with the tag or digest pinned. An untagged or `:latest` reference is refused at import.
-
-The agent works in `/app`, and its git state is the submission, so seed a git baseline in the Dockerfile: `git init`, `git add -A`, one commit.
-
-The image's `ENTRYPOINT` runs before the agent starts, so a service the image starts is running when the agent arrives. Keep-alive entrypoints such as `sleep infinity` start nothing, and `CMD` is never run. A service that dies before the agent starts fails the trial with its exit code. An image that starts a service cannot be verified in `separate` mode: verify in `shared` mode, or give the verifier its own `tests/Dockerfile`.
-
-## tests/
-
-`tests/test.sh` is the verifier. It runs in the task's working directory after the agent finishes, checks the work, and writes the reward to one of two files.
-
-- `/logs/verifier/reward.txt`: one number in `[0, 1]`, usually `1` or `0`.
-- `/logs/verifier/reward.json`: a JSON object of named numbers, for a score with several parts.
-
-When both exist, `reward.json` wins. The primary reward is the value under `"reward"`, else the single value when the object has one key. Several named scores with no `"reward"` still score the trial, as metrics with no primary reward, and the task is left out of pass@k. Everything the script prints is kept as the verifier log.
-
-```bash
-#!/bin/bash
-set -uo pipefail
-
-if uvx pytest /tests/test_outputs.py; then
-  echo 1 > /logs/verifier/reward.txt
-else
-  echo 0 > /logs/verifier/reward.txt
-fi
-```
-
-Everything under `tests/` is carried onto the verifier beside `test.sh`.
-
-`tests/Dockerfile` is built only for a `separate` verifier that pins no `docker_image`. Otherwise it is never built, the task imports with the `tests_dockerfile_not_built` note, and anything the recipe would install must already be in the image.
-
-## pre_artifacts.sh
-
-Optional. It runs in the agent's sandbox after the agent finishes and writes the work to hand on under `/logs/artifacts`. Uncommitted work is committed first. Without it, the paths listed under `artifacts` are collected for you.
-
-```bash
-#!/bin/bash
-set -uo pipefail
-mkdir -p /logs/artifacts
-git diff --binary "$(git rev-list --max-parents=0 HEAD)" HEAD > /logs/artifacts/model.patch
-```
-
-## solution/
-
-Optional. `solution/solve.sh` is the reference solution. `evolve check` runs it to confirm the task is solvable. It is archived at publish, gates nothing, and the agent is never given it.
-
-## task.toml
-
-The keys a task author sets. Everything else has a default.
-
-```toml
-[agent]
-timeout_sec = 3600
-
-[verifier]
-timeout_sec = 600
-environment_mode = "shared"
-
-[environment]
-network_mode = "allowlist"
-allowed_hosts = ["pypi.org"]
-cpus = 2
-memory_mb = 4096
-storage_mb = 10240
-```
-
-- **Timeouts.** `agent.timeout_sec` and `verifier.timeout_sec`, in seconds. The defaults are 3600 s for the agent and 600 s for the verifier; a declared value always wins. A job can stretch them with `--timeout-multiplier`.
-- **Network access.** `environment.network_mode` is `no-network` (the agent reaches nothing but its model), `allowlist` (only the hosts in `allowed_hosts`, accepted with this mode only), or `public` (the open internet, and the default). The older `allow_internet = false` means `no-network`, `true` means `public`. Only `no-network` makes the spend cap airtight; see [models](/core-concepts/models).
-- **Verifier network.** A `shared` verifier sees the agent box's network; a `[verifier] network_mode` that differs from it is refused at import. A `separate` verifier box uses the mode of `[verifier.environment]`, else a copy of `[environment]`'s, with `[verifier] network_mode` overriding either. The platform never seals a verifier on its own.
-- **Verifier mode.** `verifier.environment_mode` is `shared`, the verifier runs inside the agent's sandbox after the agent finishes and its credentials are revoked, or `separate`, the verifier boots a fresh copy of the environment and judges only what the task lists under `artifacts`. Without that list a separate verifier sees none of the agent's work, and even the reference solution scores 0. A `[verifier.environment]` table implies `separate`. Only settled separate-mode trials can be [regraded](/core-concepts/jobs#derive-a-new-job).
-- **Resources.** `environment.cpus`, `memory_mb`, `storage_mb`, and `gpus` with `gpu_types`. Omitted, the provider's default sizing applies. A size above a provider's ceiling refuses on that provider; above every ceiling it is refused at import. The ceilings are on [sandboxes](/core-concepts/sandboxes).
-- **Artifacts.** A top-level `artifacts = ["/app/out.json"]` list names the paths carried out of the agent's sandbox: into a separate verifier, and, on a separate-mode trial, into the trial's stored artifacts.
-
-## Environment variables
-
-A task's `[environment.env]` table has two value kinds.
-
-```toml
-[environment.env]
-APP_MODE = "ci"                        # literal: lands in the box as written
-GITHUB_TOKEN = "${GITHUB_TOKEN}"       # template: a secret the job must attach
-LOG_LEVEL = "${LOG_LEVEL:-info}"       # template with a fallback
-```
-
-A literal lands in the box as written; it is dataset content, so it must never be a secret. A template, `${VAR}` as the whole value, asks for a secret the job attaches with `--secret`, matched by env name; see [secrets](/core-concepts/secrets). `${VAR:-default}` falls back to the default when nothing is attached. A job that leaves a template unsatisfied is refused with `secret_not_attached`, and a `${...}` inside a longer string is refused at import.
-
-The image's own start process sees the literals only; templates reach the agent and everything it launches.
-
-## LLM judges
-
-A verifier can grade with a language model. The task asks for the credential with an environment template in `task.toml`:
-
-```toml
-[verifier.env]
-ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"
-```
-
-The verifier receives a credential for the platform's gateway under that name, with the matching base-URL variable set beside it, so `litellm`-style clients and `rewardkit` work unchanged and offline. The recognized names are `ANTHROPIC_API_KEY` and `OPENAI_API_KEY` with their `_API_BASE` and `_BASE_URL` companions, each as the entire value; `"Bearer ${ANTHROPIC_API_KEY}"` is refused at import. Any other `${VAR:-default}` resolves to its default, and a non-judge template with no default is refused.
-
-The rubric names the judge model, or `rewardkit`'s own default, `anthropic/claude-sonnet-4-6`, applies; the credential admits its key's model family only. `--ve REWARDKIT_JUDGE=<agent>` and `--ve REWARDKIT_MODEL=<model>` override the rubric's `[judge]` for one job, in both verifier modes; no other `--ve` key is accepted. The judge's spend is shown apart from the agent's, on the trial and on the job. Judge tasks are not regradable yet.
-
-## Multi-step tasks
-
-A task can walk the agent through ordered steps in one shared environment. A `steps/` directory holds one sub-directory per step in place of the root `instruction.md`, `tests/` and `solution/`, and `[[steps]]` entries in `task.toml` declare them in order. `environment/` stays at the root and the filesystem persists from step to step.
-
-```text
-migrate-then-prove/
-├── task.toml                [[steps]] entries, in order
-├── environment/
-│   └── Dockerfile
-├── tests/                   optional shared helpers for every step's verifier
-└── steps/
-    ├── 01-migrate/
-    │   ├── instruction.md
-    │   ├── tests/           merged over the shared tests/
-    │   └── workdir/         optional files landed before the step; its setup.sh runs first
-    └── 02-prove/
-        ├── instruction.md
-        └── tests/
-```
-
-```toml
-multi_step_reward_strategy = "mean"    # "mean" (default) or "final"
-
-[[steps]]
-name = "01-migrate"
-min_reward = 0.5                       # below this the trial stops here
-
-[[steps]]
-name = "02-prove"
-
-[steps.agent]
-timeout_sec = 600                      # this step's override of the task-level timeout
-```
-
-Each step has its own instruction, its own verifier (`tests/` merged over the shared `tests/`), an optional `workdir/` whose `setup.sh` runs first, and its own healthcheck, `timeout_sec` under `[steps.agent]` and `[steps.verifier]`, `[steps.verifier].env` and `artifacts` list. Each executed step is verified when it ends, and the results are on the trial as `step_results`.
-
-`multi_step_reward_strategy` is `"mean"` (the default, over the steps that produced a result) or `"final"` (the last executed step's result). `min_reward` on a step ends the trial early when the step under-scores: a number gates the primary reward, a map gates each key. A step that fails, or whose healthcheck fails, ends the trial there. A retried trial starts again from the first step.
-
-Refused at import: a per-step `user`, network declaration, or verifier environment or mode. `[[steps]]` and `steps/` must agree both ways.
-
-## MCP servers
-
-A task can hand its agent MCP tools, registered into the harness's native MCP configuration beside any the arm carries.
-
-```toml
-[[environment.mcp_servers]]
-name = "docs-search"
-transport = "streamable-http"          # or "sse" (the default); "http" is read as streamable-http
-url = "http://mcp-server:8000/mcp"     # a compose service, reachable with no egress
-
-[[environment.mcp_servers]]
-name = "sqlite"
-transport = "stdio"                    # launched inside the box by the harness
-command = "uvx"
-args = ["mcp-server-sqlite", "--db-path", "/app/data.db"]
-```
-
-A `stdio` server declares `command` and `args`; it inherits `[environment.env]`, and there is no `env` field. A remote server declares an `http` or `https` `url` with no credential in it. The URL must be reachable under the task's own declarations, checked at import: the agent's own container, a compose service by name, or a public host the network policy admits. A `stdio` server that downloads itself (`npx …`, `uvx …`) fails at run time under `no-network`; bake it into the image.
-
-## Healthcheck and the agent user
-
-`[environment.healthcheck]` keeps a slow-starting environment from meeting the agent too early: the command runs before the agent is installed, and the agent starts only once it exits `0`.
-
-```toml
-[environment.healthcheck]
-command = "curl -fsS http://localhost:8000/health"
-interval_sec = 5           # every field below command is optional; these are the defaults
-timeout_sec = 30
-start_period_sec = 0
-start_interval_sec = 5
-retries = 3
-```
-
-Inside `start_period_sec` a failure does not count. After it, `retries` failures spaced by `interval_sec`, each bounded by `timeout_sec`, fail the trial before any agent spend. On a multi-container task the check sees the `[environment.env]` literals; on a single-container task it sees none of the table, and templates are visible to no healthcheck.
-
-`[agent] user = "dev"` runs the agent as that user, by name or numeric uid. It must exist in the image (`RUN useradd -m dev`); a missing user fails the trial before any spend. Omitted, the image's `USER` applies, root by default. `USER app:group` in the Dockerfile is refused at import, and so is `[verifier] user` other than `root`.
-
-## Not supported
-
-- **Computer-use and desktop tasks.** `[environment].os` other than Linux is refused at import. A task whose instruction assumes a desktop is not refused; it runs on a Linux box without one.
-- **Run-level trajectory seeding.** No job flag names a session to resume from. Task-level seeding works: a `trajectory.json` beside `instruction.md` (or in the first step's directory) is seeded into the agent's session on `claude` and `codex`; other harnesses refuse before any spend. `[agent] load_trajectory` and `resume_trajectory` keys are refused.
-- **Verifier scripts whose PEP-723 header needs a package index.** Under `no-network` the verifier box has no index (only `rewardkit` and `litellm` resolve, on a judge task), so such a task is refused at import. Under `allowlist` or `public`, or without `uv run` or `uvx` in `tests/`, nothing changes.
-
-## From tasks to a dataset
-
-A dataset is a folder of task directories, published to the catalog with `evolve dataset publish`. See [datasets](/core-concepts/datasets).
-
-This is the Harbor task format; the full specification is at [docs.harborframework.com](https://docs.harborframework.com/core-concepts/tasks/overview).

package/skills/evolve-evals/references/core-concepts/teams.mdx

@@ -1,36 +0,0 @@
----
-title: "Teams"
-description: "Create a team, invite people, run under it, and see each other's work."
----
-
-Every account has a personal organization from the start. A team is an organization you create and invite people into: jobs, checks, sessions, datasets, agents and skills belong to one organization, and every member sees them.
-
-```bash
-evolve auth org create acme --display-name "Acme Lab"
-evolve auth org invite acme
-```
-
-`invite` prints a token once. Whoever runs `evolve auth org join <token>` while signed in joins as a member. Owners manage members and invites; members run and read.
-
-## Run under a team
-
-```bash
-evolve run -d terminal-bench-4@4.0 -a claude -m haiku -l 5 --org acme
-evolve job list --scope org
-```
-
-`--org` names the organization a job lands in. Without it, the job goes to your default organization (`evolve auth org use acme` sets one, `evolve auth org use personal` clears it) or to your personal one. `dataset publish --org` places a new dataset the same way; a new version of an existing dataset stays where that dataset is. The public catalog is visible to every organization. `job list --scope` picks what you see: `my` (yours), `shared` (what teammates created, and runs shared with you by email) or `org` (everything in your organizations, yours included).
-
-In the SDK the same choice is `org: "acme"` on `jobs().start()` and `datasets().publish()`, or once for a client with `hosted({ apiKey, org: "acme" })`.
-
-## In the dashboard
-
-The box under your account in the sidebar is the workspace switcher. It always names one workspace: the one in the page's URL, otherwise the one you last opened in this tab, otherwise Personal. Its menu lists Personal and every team you belong to, each with your role, and **Create team**. Pick a workspace and Datasets, Jobs, Traces, Members and Settings show that workspace's work; on a page that is yours alone (Skills, Integrations, Secrets, API Keys, Usage, Billing) the pick is remembered for the next one.
-
-TEAM in the sidebar has two pages. Members lists people with their role: an owner has MAKE OWNER, DEMOTE and REMOVE on every other row, and everyone has LEAVE on their own; below the list, an owner has NEW INVITE LINK, shown once, and REVOKE. A person who opens an invite link signs in, or signs up, and joins. Settings holds the team's name, and DELETE TEAM for an owner. In Personal, Members lists only you and Settings shows your workspace's name.
-
-Traces is under the workspace too. In a team it is one list of every run in that team, with a CREATED BY column naming who ran each. In Personal it has two tabs: MINE (yours, in any team) and SHARED WITH ME (what teammates made, plus runs shared with you by email). Skills lists your own uploads.
-
-<Card title="auth reference" icon="terminal" href="/cli-reference/auth">
-  Every `evolve auth org` command.
-</Card>

package/skills/evolve-evals/references/core-concepts/trial-outputs.mdx

@@ -1,117 +0,0 @@
----
-title: "Trial outputs"
-description: "What a finished trial gives you, file by file, and in which formats."
----
-
-A finished trial leaves five kinds of output: its record, its parsed trace, its trajectory, the raw streams of the agent and the verifier, and the agent's home folder. All of them download as one directory, and each can be read on its own.
-
-## The trial directory
-
-`evolve trial download <trial-id> -o trials/` saves the trial under `trials/<trial-id>/`. A file the trial never produced is absent, never empty.
-
-```text
-<trial-id>/
-├── config.json             the trial's identity: its task and its agent
-├── result.json             the outcome: status, reward, verifier verdict, exception
-├── evolve.json             the platform's record: cost and tokens per lane, provider, regrade lineage
-├── exception.txt           present when the trial ended with an exception
-├── agent/
-│   ├── trajectory.json     the ATIF trajectory, when stored
-│   ├── trace-parsed.jsonl  the parsed trace, one {seq, type, data} per line
-│   ├── codex.txt           the harness's stdout, under the harness's own name
-│   ├── stderr.log          the harness's stderr
-│   ├── agent-home.json     the home capture record
-│   └── .codex/             the captured home, at its real names
-└── verifier/
-    ├── test-stdout.txt     everything the verifier printed
-    └── reward.json         the rewards map
-```
-
-The stdout file takes the harness's name: `claude-code.txt`, `codex.txt`, `gemini-cli.txt`, `qwen-code.txt`, `kimi-code.txt`, `opencode.txt` or `droid.txt`, and `stdout.log` for an agent you registered yourself. The tree also holds `agent/sessions/` (claude, codex), `agent/qwen-sessions/` (qwen) and `agent/opencode/` (opencode): copies of the session files at the paths other tools expect.
-
-`result.json` states `agent_result.cost_usd` only once the spend is final; until then it is null there, and `evolve.json` carries the current figure with its `spend_source`. Tokens are always stated.
-
-The job download, `evolve job download`, adds to every trial its `lock.json`, its `trial.log`, the raw `verifier/reward.txt`, `steps/<name>/verifier/reward.json` on a multi-step trial, the home byte for byte, and `artifacts/` with a `manifest.json` and each collected artifact at its source path (separate-mode trials only). Everything platform-specific in the archive's record files rides under an `x_evolve` key: a trial's spend is `x_evolve.spentUsd` with its `spendSource`, and the job config's agents carry `x_reasoning_effort` and `x_preset`.
-
-## The agent's home folder
-
-The agent's home is the harness's own working state: its session files, its settings, and the transcripts of any subagents it ran. The platform captures it after the run and keeps it with the trial, so the run can be inspected the way the agent saw it. In the download it sits under `agent/` at its real names, with `agent-home.json` beside it: every captured file's path, size, sha256 and mode, and every file that was skipped, with the reason.
-
-The trial download carries the text view of the home. A file that is not UTF-8 text, such as a database, is named in the record and left out; the job download carries every byte. Two caches are left out on purpose, codex's `.codex/.tmp/plugins` and opencode's `.config/opencode/node_modules`; the record lists them under `excluded`, and anything else it could not carry under `skipped`, with the reason.
-
-From the SDK, `trials().artifact(id, "agent-home")` returns the same text view as a map from sandbox path to content. A home too large to serve as one text view is refused with a 413 `invalid_input`; the job download carries it whole.
-
-## The parsed trace
-
-Every harness prints its run in its own format. The platform parses that stream into one vocabulary, the parsed trace, which the dashboard shows and `evolve trial trace` prints. Each event is one JSON object: `seq`, a position that also serves as a resume cursor; `type`, the kind of event; and `data`, the harness's update as parsed.
-
-```json
-{"seq": 0, "type": "prompt", "data": {"_prompt": {"text": "Create hello.txt containing Hello."}}}
-{"seq": 1, "type": "agent_message_chunk", "data": {"update": {"sessionUpdate": "agent_message_chunk", "content": {"type": "text", "text": "I'll create the file."}}}}
-{"seq": 2, "type": "tool_call", "data": {"update": {"sessionUpdate": "tool_call", "toolCallId": "call-1", "title": "Write hello.txt", "kind": "edit", "status": "pending"}}}
-{"seq": 3, "type": "tool_call_update", "data": {"update": {"sessionUpdate": "tool_call_update", "toolCallId": "call-1", "status": "completed"}}}
-{"seq": 4, "type": "usage", "data": {"update": {"sessionUpdate": "usage", "scope": "call", "usage": {"promptTokens": 812, "completionTokens": 64}}}}
-```
-
-The kinds are `prompt` for the instruction the agent received, `user_message_chunk`, `agent_message_chunk` and `agent_thought_chunk` for text, `tool_call` and `tool_call_update` for a tool call and its later status and output, `plan` for the agent's plan entries, `usage` for token counts, `error` for a failure the harness reported, and `trace_loss` when events were lost. A tool call's `kind` is one of the parser's kinds, such as `read`, `edit` or `execute`, and its `status` moves through `pending`, `in_progress`, `completed` or `failed`.
-
-Once the trial is finished, the gateway's own `usage` events follow the harness's, one per model call, with the tokens and the cost as priced. Their `data.update.source` is `gateway`, and they are the only usage lines that carry tokens and money.
-
-```bash
-evolve trial trace <trial-id>
-evolve trial trace <trial-id> --grep 'permission denied' --tail 50
-evolve trial trace <trial-id> --cursor 140
-```
-
-`trial trace` prints one line per event: the position, the type, and the data cut to 140 characters. A gateway usage event prints as `gateway <model>` with its tokens and cost. `--type`, `--grep` and `--tail` filter on the server; `--cursor <seq>` resumes after the last event you saw; `--json` prints each event in full. While the trial runs, the dashboard's trial viewer shows the trace as it grows.
-
-## The ATIF trajectory
-
-`agent/trajectory.json` is the same run in ATIF, the Agent Trajectory Interchange Format, the trajectory format Harbor defines, at version `ATIF-v1.7`. The platform builds it from the stored parsed trace.
-
-Its top-level fields are `schema_version`, `agent` with the harness's name, version and model, and `steps`, the ordered system, user and agent turns, with `session_id`, `trajectory_id`, `final_metrics`, `subagent_trajectories` and `extra` optional. The instruction becomes the first `user` step. An `agent` step carries its message, its `tool_calls` under the harness's own tool names, and the `observation` those calls returned.
-
-```json
-{
-  "schema_version": "ATIF-v1.7",
-  "agent": {"name": "codex", "version": "0.31.0", "model_name": "gpt-6-astra"},
-  "steps": [
-    {"step_id": 1, "source": "user", "message": "Create hello.txt containing Hello."},
-    {
-      "step_id": 2,
-      "source": "agent",
-      "message": "I'll create the file.",
-      "tool_calls": [{"tool_call_id": "call-1", "function_name": "apply_patch", "arguments": {"path": "hello.txt"}}],
-      "observation": {"results": [{"source_call_id": "call-1", "content": "Success"}]}
-    }
-  ]
-}
-```
-
-```bash
-evolve trial download <trial-id> --stream trace-atif
-```
-
-## Raw streams and the verifier log
-
-Three artifacts are kept byte for byte: `trace-stdout`, the harness process's stdout; `trace-stderr`, its stderr; and `verifier`, everything the verifier printed. The reward itself is in `result.json` and in `verifier/reward.json`. Print one of them instead of saving:
-
-```bash
-evolve trial download <trial-id> --stream verifier
-```
-
-From the SDK, `trials().artifact(id, "verifier")` returns the same text, and `null` when the trial never stored it.
-
-## The run's file system
-
-Beyond the artifacts above, every trial's whole file system can be browsed: while the box runs you read it live, and after the run you read the kept tree — the files the run created or modified, with the files it never touched listed but held only by the image. The same reads work on analysis runs and task checks, and a published task's own files can be read from its package.
-
-```bash
-evolve trial files status <trial-id>
-evolve trial files ls <trial-id> /app/work
-evolve trial files cat <trial-id> /app/work/main.py
-evolve trial files changes <trial-id>
-evolve trial files archive <trial-id> --path /app/work -o archives/
-```
-
-The box's own streams are read the same way: `evolve trial logs <trial-id> --stream agent --follow` prints the harness's output as it arrives; `verifier` once the verifier ran, `system` when the job was started with `--system-log`; `setup` and `metrics` are named but not recorded today, so their pages are empty and say why. `evolve trial procs <trial-id>` lists the box's processes while it runs. From the SDK it is `trials().filesystem(id)`; see the [SDK reference](/sdk-reference/trials#filesystem).

package/skills/evolve-evals/references/core-concepts/trials.mdx

@@ -1,103 +0,0 @@
----
-title: "Trials"
-description: "One agent's attempt at one task: its record, its trace, and its artifacts."
----
-
-A trial is one attempt of one task by one agent arm. Trial ids are global: every trial command takes the trial id alone, and the trial's record carries its job id. Treat ids as opaque strings.
-
-```bash
-evolve trial show <trial-id>
-```
-
-`trial show` prints the task, the agent and model, the status, the reward, the tokens, what the trial spent and, when it failed, the exception. Spend prints as a plain amount once the meter has settled it, as `at least $X` while it is still a lower bound, and as `-` when nobody has measured it yet.
-
-While a trial runs, `attempt_phase` says which step it is in (`prepare`, `build`, `boot`, `install`, `agent`, `verify`, `persist`), and `live_spent_usd` is a lower bound on its spend so far, refreshed about every 30 seconds and null before the first reading; `usage` carries the same reading with its tokens. On a settled trial read `agent_result.cost_usd`; `spend_source` says whether it is final (`measured`).
-
-## The trace
-
-Every trial records a parsed event trace: the instruction, each agent turn, every tool call and its result. Print it, filtered on the server.
-
-```bash
-evolve trial trace <trial-id>
-evolve trial trace <trial-id> --type <event-type>
-evolve trial trace <trial-id> --grep 'permission denied' --tail 50
-```
-
-`--type` keeps events of exactly one type. `--grep` is a case-insensitive regex over type and content. `--tail` keeps the last N matching events. To search every trial of a job in one pass:
-
-```bash
-evolve job grep <job-id> 'permission denied'
-```
-
-## Artifacts
-
-Beside the parsed trace, a trial keeps the raw record of its run. One vocabulary names the pieces everywhere; every file and format is described on [trial outputs](/core-concepts/trial-outputs).
-
-- `trace-parsed`: the parsed event trace
-- `trace-stdout`: the agent process's stdout, byte for byte
-- `trace-stderr`: the agent process's stderr
-- `trace-atif`: the normalized trajectory, in the ATIF format
-- `agent-home`: the text view of the agent's home folder, captured after the run
-- `verifier`: everything the scoring step printed
-- `trajectory`: reserved for the harness's own native session file, not served yet
-
-Stream one artifact to stdout:
-
-```bash
-evolve trial download <trial-id> --stream verifier
-```
-
-## Download the trial tree
-
-Without `--stream`, the trial is saved whole under `<dir>/<trial-id>/` in the trial tree layout, plus `evolve.json`.
-
-```bash
-evolve trial download <trial-id> -o trials/
-```
-
-```text
-config.json               task and agent
-result.json               status, reward, verifier verdict, exception, phase clocks
-agent/trajectory.json     the normalized ATIF trajectory
-agent/<harness>.txt       the harness's stdout stream (claude-code.txt, codex.txt, ...)
-agent/stderr.log          the harness's stderr stream
-agent/trace-parsed.jsonl  the parsed event trace
-agent/.claude/ ...        the captured agent home at its real names
-agent/agent-home.json     the capture record: every file's path, size and digest
-verifier/test-stdout.txt  the verifier log
-verifier/reward.json      the rewards map, when the verifier produced one
-exception.txt             when the trial carries an exception
-evolve.json               the platform's own record: cost, tokens, provider
-```
-
-An artifact the trial never recorded is an absent file, never an empty placeholder. `--overwrite` replaces an existing folder. The job archive from `evolve job download` holds the same tree per trial, plus what only the server can add: `lock.json`, `trial.log`, `artifacts/`, and the raw `verifier/reward.txt`.
-
-## Act on one trial
-
-```bash
-evolve trial retry <trial-id>
-evolve trial regrade <trial-id>
-evolve trial stop <trial-id> <trial-id>
-```
-
-`retry` runs a settled trial again, `regrade` re-runs only its verifier, and each produces a new job holding the one new trial. `stop` ends in-flight trials without cancelling their job; each stopped trial settles `CANCELLED` and is charged its spend so far.
-
-## Statuses
-
-A valid reward, 0 included, is `SCORED`; a failure is never reported as a fabricated zero.
-
-- `QUEUED`: waiting for a sandbox slot.
-- `RUNNING`: the agent phase; `attempt_phase` says which step.
-- `SCORING`: the agent finished; the verifier is running.
-- `SCORED`: a valid reward is in `reward`.
-- `SCORING_ERROR`: the verifier crashed or wrote a reward outside `[0, 1]`; read `exception_info`.
-- `INFRASTRUCTURE_ERROR`: the trial was lost before a result was recorded; retried automatically, then `resume`.
-- `BUDGET`: a budget above the trial's own cap refused it. The exception type is `ApiUsageLimitError` and the message carries `user:` (your credits), `team:` (the organization's monthly budget) or `other:` (the platform's global stop). Never retried on its own; add credits or raise the budget, then `resume`.
-- `INDETERMINATE`: the platform cannot tell whether the trial completed.
-- `CANCELLED`: stopped before settling; the exception type is `CancelledError`.
-
-`SCORING_ERROR` and `INDETERMINATE` carry the last 4000 characters of the verifier's own output in `exception_message`. Job statuses are on [jobs](/core-concepts/jobs#statuses).
-
-<Card title="trial reference" icon="terminal" href="/cli-reference/trial">
-  Every flag of `evolve trial`.
-</Card>

package/skills/evolve-evals/references/core-concepts/upload.mdx

@@ -1,93 +0,0 @@
----
-title: "Upload a job"
-description: "Bring a job you ran elsewhere onto the platform: a Harbor job folder, or Evolve SDK runs packed as one."
----
-
-A job you ran elsewhere uploads as a finished job. Its trials, traces and rewards become a job you read, analyze and share like any other.
-
-A job from `harbor run` — local or through the Harbor hub — uploads as it is: point `upload` at the job folder, nothing to repack. Runs made with the Evolve SDK are packed into that same layout first (below).
-
-```bash
-evolve upload ./job-2026-08-27__12-00-00 -d terminal-bench-4@4.0
-```
-
-`upload` takes the job folder, or its `.tar.gz`. With `--from <url>` it takes a public https URL of the archive instead. `-d name[@version]` links the trials to a published dataset version by task name. The command follows the import until the job exists; `--no-wait` returns at once with the import id.
-
-```bash
-evolve job imports --status RUNNING
-evolve job import <import-id> --watch
-```
-
-## What a job folder holds
-
-The Harbor job layout, exactly what `evolve job download` and `harbor job download` produce:
-
-```text
-job/
-├── config.json                   REQUIRED — the job record
-├── result.json                   REQUIRED
-└── <trial>/                      one folder per trial
-    ├── result.json               REQUIRED — the trial's facts (below)
-    ├── agent/
-    │   ├── trajectory.json       the trace, in Harbor's trajectory format
-    │   ├── stdout.log            the harness's output (any agent/*.txt or *.log also counts)
-    │   └── …                     the agent's home, stored byte for byte
-    ├── verifier/                 test-stdout.txt, reward.txt
-    ├── artifacts/                the agent's deliverables
-    ├── steps/                    multi-step trials, as Harbor writes them
-    └── trial.log, exception.txt
-```
-
-The job root needs both record files; inside a trial only `result.json` is required. What is absent stays absent: a trial without a trace shows no trace, a trial without a home shows no files.
-
-## What `result.json` must hold
-
-```json
-{
-  "task_name": "my-task",
-  "trial_name": "my-task__deepseek",
-  "trial_uri": "",
-  "task_id": {},
-  "task_checksum": "",
-  "config": {},
-  "agent_info": { "name": "codex", "version": "0.154.0", "model_info": { "name": "deepseek-v4.1-flash" } },
-  "verifier_result": { "rewards": { "reward": 1.0 } },
-  "exception_info": null,
-  "agent_execution": { "started_at": "2026-09-16T18:08:27Z", "finished_at": "2026-09-16T18:08:38Z" }
-}
-```
-
-The first seven keys are required. `verifier_result.rewards` is the score: names to numbers, `reward` the primary. Harbor's exception names (`ScoringError`, `InfrastructureError`, `ApiUsageLimitError`, `CancelledError`) map to their statuses; any other name stays as written. `agent_result` carries tokens and cost when you have them.
-
-## Runs made with the Evolve SDK
-
-Every SDK run is logged to a local file under `~/.evolve-sdk/observability/sessions/`. Its header line names the agent and model, and its raw lines are the transcript. To bring such runs onto the platform, pack one trial folder per run: the header's agent and model into `agent_info`, the raw lines at the harness's own transcript path under `agent/` (for codex `agent/.codex/sessions/<yyyy>/<mm>/<dd>/rollout-<id>.jsonl`, for claude `agent/.claude/projects/<project>/<session>.jsonl`), and the score, when you have one, in `verifier_result.rewards`. Raw transcripts anywhere else under `agent/` are stored as files but not read as the trace.
-
-The `systemPrompt` of an SDK run is not in the session file. To show it, describe the run in `agent/trajectory.json` instead of the raw lines: a first step whose `source` is `system` holding the prompt, then the conversation. A `system` step is drawn as the system prompt, never as an error. When the raw lines are at the harness path, they are the trace and `agent/trajectory.json` is not read.
-
-## What lands
-
-- Rewards → `SCORED`, rewards untouched.
-- No rewards → no score. Not an error.
-- `exception_info` → the exception, kept as written.
-- Trace, logs, home, verifier log, artifacts, steps → where a native trial's are.
-- A prior `analysis.json` → not imported. Analyses run here.
-
-The job carries `upload`: the archive's `original_job_id` and `original_job_name`, `uploaded_at`, the uploader's `reported_totals`, `task_links`. Resume, retry and regrade refuse an uploaded job (`job_uploaded`); analyze works on it. The CLI prints `ported` as the provider, and after the import `scored N of M trial(s)`. Zero of many is a warning: check `verifier_result.rewards`.
-
-## Task linkage
-
-Each trial's `lock.json` names its task by content hash; a task the platform holds with the same hash links by itself. A linked trial's analyses run with the task folder, an unlinked trial's without it, and the record says why: `hash_mismatch`, `task_not_in_dataset`, `no_task_digest`, `no_hash_match`, `no_dataset_named`, `dataset_ambiguous`. A name alone never links.
-
-`-d name@version` overrides: it links by task name to that version and settles an ambiguous hash.
-
-## Bounds and duplicates
-
-- One limit: the archive ceiling `limits.uploads.job_archive_bytes` (`upload_too_large`).
-- A trial with a file over 5 GiB, or a trajectory too large to parse, is skipped (`trial_too_large`); the rest lands. A `job.log` over that size refuses the job (`upload_too_large`).
-- A `result.json` that cannot be read fails the import (`invalid_trial`).
-- The same job twice: `job_already_uploaded`. Delete it, then upload again.
-
-<Card title="upload reference" icon="terminal" href="/cli-reference/upload">
-  Every flag of `evolve upload`.
-</Card>

package/skills/evolve-evals/references/dashboard/checks.mdx

@@ -1,21 +0,0 @@
----
-title: "Checks"
-description: "Read task quality checks in the dashboard, on a job's CHECK tab or on the check's own page."
----
-
-A job's CHECK tab lists the task quality checks run on the job's datasets.
-
-{/* retake after the UI pass, from a non-admin account */}
-<Frame caption="The CHECK tab of a job whose tasks have not been checked">
-  <img src="/images/job-check.png" alt="The CHECK tab: the sentence that no task of this job has been checked, and the evolve check command that would check its dataset." />
-</Frame>
-
-Until a check exists, the tab shows the command that would start one, `evolve check -d <dataset@version>`, with the job's dataset filled in. Checks start from the command line; see [check](/core-concepts/check).
-
-Once a dataset has been checked, the tab lists one row per task with its dataset and its result. Click a row to open the check in the [trial viewer](/dashboard/trial-viewer): the events are the checker's own trace, and its result document is on the DOWNLOAD menu.
-
-## The check page
-
-A check is a job, so it sits in the [Jobs](/dashboard/jobs) list with KIND `CHECK`, and its row opens the check's own page. The band names the check, its status, what was checked, the checker's model and effort, its cost and when it started. SHARE is the creator's and opens the share dialog; STOP, for the creator, ends every task check still queued or running while any is live. DOWNLOAD saves a completed check as a `.tar.gz` in the standard check layout.
-
-Under the band, one row per task check with its dataset and its result, the same table as a job's CHECK tab. Click a row to open it in the [trial viewer](/dashboard/trial-viewer).

package/skills/evolve-evals/references/dashboard/datasets.mdx

@@ -1,24 +0,0 @@
----
-title: "Datasets"
-description: "Browse the catalog, and each dataset's tasks and versions, in the dashboard."
----
-
-The Datasets page lists every dataset you can run: the platform's public datasets and your own private ones.
-
-{/* retake after the UI pass, from a non-admin account */}
-<Frame caption="The datasets catalog">
-  <img src="/images/datasets.png" alt="The Datasets page: a search box, an ALL, PUBLIC and PRIVATE filter, and a table of datasets with task count, version, visibility, status and update time." />
-</Frame>
-
-Search by name, and filter with ALL, PUBLIC or PRIVATE. Each row shows the dataset's task count, its active version, its visibility, its status and when it last changed. Tick rows to delete datasets you own. There is no publish button: publishing is [`evolve dataset publish`](/core-concepts/datasets).
-
-Click a dataset to open it.
-
-{/* retake after the UI pass, from a non-admin account */}
-<Frame caption="One dataset: its tasks, with a verdict per sandbox provider">
-  <img src="/images/dataset.png" alt="One dataset's page: the visibility, version and task count in the header, TASKS and VERSIONS tabs, and a table of tasks with GPUs, agent timeout, verifier timeout, skills and providers." />
-</Frame>
-
-The header shows the visibility, the active version and the task count. The TASKS tab lists every task of that version with its GPU count, its agent and verifier timeouts, its skills, and the sandbox providers that can run it. An arrow after a provider means the trial is rerouted to the provider after the arrow. Hover a refused provider for the reason.
-
-The VERSIONS tab lists the dataset's versions. DELETE appears only on a dataset you may delete: your own private dataset, or a public one when you are the platform admin. It removes the dataset.