@evolvingmachines/evolve 0.0.55-project-sable.20260916.db0bc0bb

package/docs-evals/core-concepts/datasets.mdx

@@ -0,0 +1,138 @@
+---
+title: "Datasets"
+description: "The catalog, and publishing your own dataset."
+---
+
+A dataset is a named, versioned folder of [tasks](/core-concepts/tasks) in the catalog. You name a version as `name@version`. A bare name means the dataset's active version.
+
+## Browse the catalog
+
+```bash
+evolve dataset list
+evolve dataset show terminal-bench-4@4.0
+```
+
+`dataset list` prints every dataset you can run: the platform's public datasets and your own. `--search <text>` filters by name and description. `dataset show` prints one dataset's versions, its tasks with their timeouts, and for each task which sandbox providers can run it.
+
+## Publish your own
+
+Any folder of task directories can be published: a single task directory, a directory with a `tasks/` folder, or a directory of task directories. What you publish is private to your organization; another account asking for its name reads `dataset_not_found`. A name belongs to its first publisher: re-publishing your own name adds a version, and a name owned by anyone else is refused with `dataset_name_taken`.
+
+Check it first: the pre-flight is a dry run that uploads nothing and writes nothing.
+
+```bash
+evolve dataset check ./my-swe
+```
+
+Then publish from a local directory, from a git repository, or from a source the server fetches itself.
+
+<Tabs>
+  <Tab title="Local directory">
+    ```bash
+    evolve dataset publish \
+      --dir ./my-swe \
+      --name my-swe \
+      --version 1.0 \
+      --watch
+    ```
+
+    When the folder carries a `dataset.toml` manifest, `--name` and `--version` come from it and may be omitted.
+
+    Everything in the directory is packed, dotfiles included, and executables stay executable. Only `.git`, `.DS_Store` and `.venv` are left out, and symlinks are never packed. A large upload resumes after a dropped connection, and `evolve dataset watch` can follow it from another machine while it uploads: it reads `QUEUED (receiving)` until the corpus has arrived.
+  </Tab>
+  <Tab title="Git repository">
+    ```bash
+    evolve dataset publish \
+      --git https://github.com/acme/my-swe.git \
+      --ref v1.0.0 \
+      --name my-swe \
+      --version 1.0 \
+      --watch
+    ```
+
+    `--ref` must be pinned: a tag, or a full 40-character commit sha. A branch name is refused with `unpinned_git_ref`, and the refusal's `details.commit` is the sha to pin. `--path <subfolder>` imports one folder of a larger repository. The URL must be `https://`; for a private repository put a token in it.
+  </Tab>
+  <Tab title="Fetchable source">
+    ```bash
+    evolve dataset publish --from hub:cookbook/hello-world --watch
+    ```
+
+    `--from` takes a public https tarball URL, or `hub:org/name[@ref]` for a public package on the Harbor hub. Both are public only, with no credentials in the URL. A tarball may wrap the corpus in one top-level directory, so a repository's archive URL publishes as is.
+
+    For a hub package the name and version default to the package's own, and the ref is pinned when the publish is accepted. A package the hub does not show is refused with `hub_package_not_found`; a hub that cannot be reached is `hub_unreachable`, retry the publish.
+  </Tab>
+</Tabs>
+
+`--watch` follows the publish until the version is READY or FAILED. Each task builds on its own, so one broken task does not block the others; `--watch` ends with how many built. If the terminal is gone, `evolve dataset watch <name>` re-attaches to the same follow, from any machine. `--skip-preflight` uploads without the check; a task the check would have refused then fails at import instead.
+
+### The pre-flight
+
+The pre-flight sends each task's `task.toml`, and the `dataset.toml` if there is one, and answers with a verdict per task. It checks what a `task.toml` alone can decide, so an all-ok answer means no config refuses, not that every image will build. A `NOTE` is not a refusal; today it is `tests_dockerfile_not_built`. A refused task stops the publish before anything is uploaded.
+
+### The manifest
+
+A corpus with a `dataset.toml` imports what the manifest says: only the tasks listed under `[[tasks]]`, each verified against its pinned digest. A listed task the checkout lacks fails the publish with `manifest_task_missing`, a digest mismatch with `manifest_digest_mismatch`. The `[dataset]` description reaches the catalog, and every version records the identity it imported under as `manifest`. A `metric.py` custom metric is refused with `custom_metric_not_supported`.
+
+## What happens when you publish
+
+```mermaid
+%%{init: {"themeVariables": {"fontSize": "14px", "fontFamily": "Google Sans Flex, sans-serif"}, "flowchart": {"nodeSpacing": 20, "rankSpacing": 24, "padding": 10, "wrappingWidth": 120, "subGraphTitleMargin": {"top": 6, "bottom": 6}}}}%%
+flowchart TB
+  subgraph PUB[At publish]
+    direction LR
+    P[Pre-flight: nothing uploaded] --> U[Upload: the version is IMPORTING, then BUILDING]
+    U --> B[Parse and build: one image per task]
+    B --> RD[READY, or FAILED when no task built]
+  end
+  subgraph RUN[At the first run]
+    direction LR
+    F[The first trial on a provider is slower] --> L[Every later trial starts from the cache]
+  end
+  PUB --> RUN
+```
+
+Every declaration in `task.toml` is honored or refused with the reason; a task never runs on weaker semantics than it declares.
+
+Each task's `environment/` is built into an image once. Tasks build independently: one that fails any step is recorded FAILED with a typed reason, and the others keep building.
+
+The version lands READY when at least one task built, and FAILED only when none did or on a corpus-level refusal; a FAILED version changes nothing, the dataset keeps serving what it served. On your own dataset, READY also makes the version active, so the bare name runs it. A job refuses any version that is not READY with `version_not_ready`.
+
+The first trial of a task on a provider takes a few minutes longer than the rest; it is not a hang.
+
+The import's `warnings` name what a version will permanently lack: `no_solutions_archived` or `partial_solutions_archived` for the reference-solution record, `tasks_failed_to_build` when some tasks failed, `tests_dockerfile_not_built` for the tasks whose verifier never builds their `tests/Dockerfile`.
+
+### Partially built versions
+
+On the version, `task_count` counts the READY tasks and `n_failed_tasks` the rest. `dataset show name@version` lists every failed task with its reason, and `dataset publish --watch` prints each task's outcome. A whole-dataset or glob job on such a version runs the READY tasks and says so in its `build_exclusions`. Naming a failed task explicitly at job create is refused with `task_failed_to_build`; a fix is a new version.
+
+### Version states
+
+A version moves `DRAFT` → `RECEIVING` (a large upload still streaming) → `IMPORTING` → `BUILDING` → `READY`, with `FAILED` and `ARCHIVED` as off-ramps. `READY` is the only state that accepts jobs.
+
+An import is `QUEUED` (`receiving: true` while the corpus is still uploading), `RUNNING`, `COMPLETED` (the version is READY) or `FAILED` (read `failure`). A finished import stays readable as long as its dataset exists.
+
+## Versions
+
+Each publish creates a version, and a version that lands READY is built and active. To point the bare name at a different READY version:
+
+```bash
+evolve dataset activate my-swe 1.0
+```
+
+Activating the version that is already active succeeds without change. A version still building is refused with `version_not_ready`; a FAILED or ARCHIVED one with `version_not_activatable`.
+
+The owner of a dataset can download the original package back:
+
+```bash
+evolve dataset download my-swe@1.0 -o corpora/
+```
+
+It is the whole corpus you published, `solution/` included, and the one call that returns task files. Only the owner may: a platform dataset has no owner and cannot be downloaded, and someone else's dataset reads as not found.
+
+Deleting a dataset removes its versions, tasks and archived solutions. You must own it (`dataset_not_owned` on a platform dataset), and a dataset any job ran against is never deleted: `dataset_in_use` names the blocking jobs.
+
+A dataset published from git records what it was built from, and `dataset list` and `dataset show` print one line when the ref has moved upstream. A new version is always one you publish, unless you opt in with `datasets().update(name, { upstream_auto_import: true })`; that is refused with `upstream_not_watchable` on a dataset with no moving git ref.
+
+<Card title="dataset reference" icon="terminal" href="/cli-reference/dataset">
+  Every flag of `evolve dataset`.
+</Card>

package/docs-evals/core-concepts/index.mdx

@@ -0,0 +1,42 @@
+---
+title: "Core concepts"
+description: "How a dataset becomes trials, rewards, analyses and results."
+---
+
+```mermaid
+%%{init: {"themeVariables": {"fontSize": "15px", "fontFamily": "Google Sans Flex, sans-serif"}, "flowchart": {"nodeSpacing": 24, "rankSpacing": 28, "padding": 10, "wrappingWidth": 170, "subGraphTitleMargin": {"top": 6, "bottom": 6}}}}%%
+flowchart TB
+  subgraph S[Set up]
+    D[Dataset: a folder of tasks] --> J[Job: arms of harness x model, one sandbox provider]
+    J --> T[Trials: one per task per arm]
+  end
+  subgraph RUN[Run, per trial]
+    SB[Sandbox: the agent works on the task] --> V[Verifier: scores the result]
+    SB -.-> L[Live trace to the dashboard]
+    V --> R[Reward and outputs: trajectory, files, logs]
+  end
+  subgraph AFTER[After]
+    AN[Analyze: trace analysis, verdicts]
+    C[Check: task quality]
+    O[Results: jobs, trials, analyses and checks, in the dashboard and as downloads]
+  end
+  T --> SB
+  R --> AN
+  D --> C
+  R ~~~ C
+  R --> O
+  AN --> O
+  C --> O
+```
+
+- **Dataset**: a named, versioned folder of tasks in the catalog. `name@version` names one version; a bare name means the active one.
+- **Task**: one directory with an instruction, an environment, and a verifier. See [tasks](/core-concepts/tasks).
+- **Job**: one run of every task of its datasets by every arm, a fixed number of attempts each. See [jobs](/core-concepts/jobs).
+- **Arm**: one harness driving one model, with its optional effort, preset, config and skills. A job with two models has two arms, and every arm runs on the job's one sandbox provider.
+- **Trial**: one attempt of one task by one arm, in its own sandbox. See [trials](/core-concepts/trials).
+- **Trajectory**: the record of what the agent did: the instruction, every agent turn, every tool call and its result. It streams live to the dashboard while the trial runs and is stored with the trial.
+- **Verifier**: the task's test script. It runs after the agent has finished and produces the reward.
+- **Reward**: the verifier's score for the trial, usually `1` or `0`.
+- **Outputs**: what a finished trial keeps: the reward, the trajectory, the agent's files and the logs, captured when the trial ends. See [trial outputs](/core-concepts/trial-outputs).
+- **Analysis**: a rubric judgment of a settled trial's trace, one analyzer run per trial. See [analyze](/core-concepts/analyze).
+- **Check**: a rubric judgment of a task itself, before you spend a job on it. See [check](/core-concepts/check).

package/docs-evals/core-concepts/jobs.mdx

@@ -0,0 +1,136 @@
+---
+title: "Jobs"
+description: "Start a job, watch it, read its results, and derive new jobs from it."
+---
+
+A job runs every task of its datasets with every agent arm, a fixed number of attempts each. Start one with `evolve run`. It is the short form of `evolve job start`, and both take the same flags.
+
+```bash
+evolve run \
+  -d terminal-bench-4@4.0 \
+  -a codex \
+  -m gpt-5.5 \
+  -k 2 \
+  -l 5 \
+  --watch
+```
+
+`-d` names a dataset and may repeat: one job can span several. `-a` names the agent. `-m` names the model and may repeat too; each model is one arm, so two models against one agent run every task twice.
+
+`-k` is the number of attempts per task and arm (default 1). `-n` is how many trials run in parallel (default 4, at most 150). `--job-name` gives the job a label; omitted, the server generates one.
+
+## Narrow the task set
+
+```bash
+evolve run \
+  -d terminal-bench-4@4.0 \
+  -a codex \
+  -m gpt-5.5 \
+  -i 'auth-*' \
+  -x 'auth-legacy' \
+  -l 20 \
+  --watch
+```
+
+`-i` and `-x` are include and exclude globs over task names, applied to every dataset. `-l` caps the task count of each dataset after the filters.
+
+## Watch it
+
+`--watch` streams the job's events until it finishes: each trial as it starts, scores and settles, then the final block. `-q` keeps only the final block. Without `--watch` the command prints the job id and returns; read the job later with `evolve job show`.
+
+## Spend and retries
+
+Every trial runs under a model-spend cap, `--max-trial-spend <usd>`; omitted, the server's default applies, $200. It is the only spend limit: there is no job-wide budget. The job states the cap it applied as `max_trial_spend_usd` and the most it can cost as `worst_case_spend_usd`, the cap times the trials times the attempts the retry policy allows. A trial that exhausts its cap is not a failure; it is scored on what it produced.
+
+Your credit balance sits under every cap. A job started at a zero balance is refused with `insufficient_credits`, and a balance that runs out stops spending mid-job.
+
+Trials that fail on infrastructure errors are retried automatically. `-r <n>` sets how many times; the default is 2, and `0` turns it off. No other status re-runs on its own. `--retry-include` and `--retry-exclude` choose by exception type, exclude wins, and the names are on the [run reference](/cli-reference/run#spend-and-retries).
+
+A retried trial keeps its history: `n_retries` and the `retries` list on the trial, `stats.n_retries` on the job, and every retried attempt's spend stays in the job total. Two infrastructure failures of the same kind in a row stop the retries early; the trial stays `INFRASTRUCTURE_ERROR`, its message says why, and the event stream carries `trial.retry_circuit_broken`.
+
+## Read the results
+
+```bash
+evolve job show <job-id>
+evolve job trials <job-id> --status INFRASTRUCTURE_ERROR,SCORING_ERROR
+evolve job tasks <job-id>
+evolve job compare <job-id> <job-id>
+evolve job list --search nightly
+```
+
+`job show` prints the whole job, with pass@k once every attempt has settled. `job trials` lists the trials, `--status` and `--dataset` narrow the list. `job tasks` is one row per task.
+
+`job compare` puts two to ten jobs side by side. `job list` is your jobs, newest first; `--scope shared` lists your organizations' jobs that teammates created.
+
+A job's spend prints as `at least $X` while it holds settled trials whose spend has not been measured yet.
+
+pass@k is the chance that k attempts contain at least one success, averaged over the tasks of an arm. The k values are the powers of two and the multiples of five up to the smallest attempt count, so a single-attempt job has none. An attempt that produced no reward counts as a failure, and a group with non-binary rewards or attempts still in flight shows nothing. The archive's `result.json` carries the same numbers.
+
+## Statuses
+
+A job is `QUEUED` (accepted, waiting for dispatch), `RUNNING`, `CANCELLING` (cancel requested, in-flight trials winding down), `COMPLETED` (every trial settled), `CANCELLED`, or `FAILED`. `FAILED` is reserved: nothing sets it today, so `failure` is null on every job. A job that goes wrong does so one trial at a time and still reaches `COMPLETED`; read `trials.byStatus` for where. Trial statuses are on [trials](/core-concepts/trials#statuses).
+
+## Derive a new job
+
+Every derived run is a new job linked to its source through `source_jobs`. The source is never changed.
+
+```bash
+evolve job retry <job-id> --failed-only
+evolve job retry <job-id> -t <trial-id> -t <trial-id>
+evolve job resume <job-id> -f InfrastructureError
+evolve job regrade <job-id> --task tricky-task
+```
+
+`resume` takes a finished job's failed or stopped trials and runs them: `ScoringError`, `InfrastructureError`, `ApiUsageLimitError` and `IncompleteTrialError`, plus stopped trials and the still-queued trials of a cancelled job. `-f <type>` narrows it by exception type. Scored trials are never re-run. A live source is refused with `job_not_terminal`, an empty selection with `no_failed_trials`.
+
+`retry` runs selected trials again: every trial, only the failed ones with `--failed-only` (`SCORING_ERROR`, `INFRASTRUCTURE_ERROR`, `BUDGET`, `INDETERMINATE`), or exactly the ones named with `-t`. A scored trial is a legitimate target. The whole-job and `--failed-only` forms need a finished source; the named form works on a running job once every named trial has settled, else `trial_not_settled`, or `trial_not_found` for an id the job does not own. Naming trials and `--failed-only` together is refused.
+
+`regrade` re-runs only the verifier of a finished job, under the network policy the source trial ran with; `--status` and `--task` narrow the set. Only settled separate-mode trials are eligible; shared-mode trials and judge tasks are not. A single-trial regrade of an ineligible source is refused with `regrade_source_ineligible`; a whole-job regrade skips the ineligible trials and answers `no_regradable_trials` when none remain. The result is a job with `is_regrade` true.
+
+## Stop, cancel, delete
+
+```bash
+evolve job cancel <job-id>
+evolve job stop <job-id> --dataset terminal-bench-4
+evolve job delete <job-id> --yes
+```
+
+`cancel` asks the whole job to stop; a finished job is a no-op. `stop` ends one dataset's live trials and leaves the job running. `delete` removes a job you created permanently: its trials, traces, analyses and stored files. Without `--yes` it asks first.
+
+Only the job's creator may delete it; another member of the organization is refused with `org_forbidden`. The job must be finished: a live job, a running analysis wave or a live regrade derived from it refuses, naming what to wait for. Regrade jobs and the `source_jobs` entries of derived jobs stay. Deleting an uploaded job is how it is replaced; see [upload](/core-concepts/upload).
+
+## Download
+
+```bash
+evolve job download <job-id> -o results/
+```
+
+The results unpack as the standard job directory, `results/job-<id>/`: the job's `config.json`, `lock.json`, `result.json` and `job.log`, and one directory per trial. See [trial outputs](/core-concepts/trial-outputs) for what each trial directory holds.
+
+## Config files
+
+Put the job in a file and run it with `-c`. The file is YAML or JSON with the same field names the SDK's `jobs().start()` takes. Flags on the command line override the file's fields, one field at a time. `--print-config` prints the resolved job body as JSON and exits without running anything.
+
+```yaml nightly.yaml
+datasets:
+  - name: terminal-bench-4
+    version: "4.0"
+  - name: harbor-examples
+    version: "1.0"
+agents:
+  - name: codex
+    model_name: gpt-5.5
+  - name: claude
+    model_name: fable
+n_attempts: 2
+max_trial_spend_usd: 25
+```
+
+```bash
+evolve run -c nightly.yaml --print-config
+evolve run -c nightly.yaml --watch
+```
+
+<Card title="run reference" icon="terminal" href="/cli-reference/run">
+  Every flag of `evolve run`.
+</Card>

package/docs-evals/core-concepts/models.mdx

@@ -0,0 +1,68 @@
+---
+title: "Models"
+description: "How a model is named on -m, and why one API key covers every provider."
+---
+
+`-m` names the model an arm runs, as the harness knows it. One `EVOLVE_API_KEY` covers every provider: you never configure a provider key for a job, and every model call that goes through the platform's gateway is metered against the trial's spend cap. The cap is airtight only on a task that declares `no-network`; under `allowlist` or `public` an agent can reach a provider directly with a key of its own, and that traffic is neither metered nor capped.
+
+```bash
+evolve run \
+  -d terminal-bench-4@4.0 \
+  -a claude \
+  -m fable \
+  -l 5 \
+  --watch
+evolve run \
+  -d terminal-bench-4@4.0 \
+  -a codex \
+  -m gpt-5.5 \
+  -m gpt-5.6-terra \
+  -l 5 \
+  --watch
+```
+
+`-m` repeats. Each model is one arm, so the second command runs every task twice, once per model. Every arm must name a model; there is no default.
+
+With a provider key of your own enabled, model calls bill that provider account instead of credits; the cap still applies, and a zero credit balance still refuses a job create, resume or retry.
+
+## The rosters
+
+Each harness has its own roster of model names. These are the names the SDK ships at this release. The platform publishes the live list for each harness in its capability document, which the SDK reads with `meta()`.
+
+<Tabs>
+  <Tab title="claude">
+    `fable`, `opus`, `sonnet`, `haiku`, `opus[1m]`, `sonnet[1m]`, `glm-5.3`, `glm-5.3-flash`, `openrouter/deepseek/deepseek-v4.1-flash`, `fireworks/deepseek-v4.1-flash`
+  </Tab>
+  <Tab title="codex">
+    `gpt-6-astra`, `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`, `gpt-5.5`, `gpt-5.3-codex`
+  </Tab>
+  <Tab title="gemini">
+    `gemini-3.5-flash`, `gemini-3.5-flash-lite`, `gemini-3.1-pro-preview`
+  </Tab>
+  <Tab title="qwen">
+    `qwen3.7-max`, `qwen3.7-plus`, `qwen3.6-flash`
+  </Tab>
+  <Tab title="kimi">
+    `kimi-k3`, `kimi-k2.7-code`, `kimi-k3-raptor`, `kimi-k2p7-code-raptor`
+  </Tab>
+  <Tab title="opencode">
+    `openrouter/anthropic/claude-fable-5.1`, `openrouter/anthropic/claude-opus-5`, `openrouter/anthropic/claude-sonnet-5`, `openrouter/anthropic/claude-haiku-4.5`, `openrouter/openai/gpt-6-astra`, `openrouter/openai/gpt-5.6-sol`, `openrouter/openai/gpt-5.6-terra`, `openrouter/openai/gpt-5.6-luna`, `openrouter/google/gemini-3.6-flash`, `openrouter/qwen/qwen3.7-max`, `openrouter/moonshotai/kimi-k3`, `openrouter/z-ai/glm-5.3`, `openrouter/z-ai/glm-5.3-flash`, `openrouter/deepseek/deepseek-v4.1-flash`, `fireworks/deepseek-v4.1-flash`
+  </Tab>
+  <Tab title="droid">
+    `claude-fable-5.1`, `claude-opus-5`, `claude-sonnet-5`, `claude-haiku-4-5`, `gpt-6-astra`, `gpt-5.6-sol`, `gpt-5.6-terra`, `gpt-5.6-luna`, `gemini-3.6-flash`, `qwen3.7-max`, `kimi-k3`, `glm-5.3`, `glm-5.3-flash`, `openrouter/deepseek/deepseek-v4.1-flash`, `fireworks/deepseek-v4.1-flash`
+  </Tab>
+</Tabs>
+
+## Reasoning effort
+
+`--effort <value>` sets the reasoning effort for every arm. The values each harness accepts are in the [capability document](/sdk-reference/meta), under its `effort_support`: `gemini` accepts no effort value, `qwen` switches thinking on or off rather than taking a level, and the other five harnesses take a level. An effort a harness cannot honor is refused with `invalid_input` when the job is created, never silently skipped.
+
+An omitted effort resolves to the harness's own default and is stamped as if declared: every trial echoes it on `agent_info.reasoning_effort`, and the job's `stats.evals` keys carry it as the segment before the dataset, `agent__model__effort__dataset`. Effort is part of an arm's identity: the same agent and model at two efforts are two arms.
+
+## The analyzer's and the checker's model
+
+The analyzer and the checker run on the `claude` roster. Their default model is `openrouter/deepseek/deepseek-v4.1-flash`; `-m` on `evolve analyze` and `evolve check`, and `--analyze-model` on `evolve run`, pick another.
+
+<Card title="Agents" icon="bot" href="/core-concepts/agents">
+  The seven harnesses, and how to bring your own.
+</Card>

package/docs-evals/core-concepts/sandboxes.mdx

@@ -0,0 +1,52 @@
+---
+title: "Sandboxes"
+description: "Where trials run: e2b, daytona, or modal."
+---
+
+Every trial runs in its own sandbox on one of three providers. Pick the provider per job with `-e`; the default is `daytona`. Once chosen, the provider is fixed for the job's life, and a resumed job inherits it.
+
+- `e2b`: [E2B](https://e2b.dev/docs)
+- `daytona`: [Daytona](https://www.daytona.io/docs), the default
+- `modal`: [Modal](https://modal.com/docs)
+
+```bash
+evolve run \
+  -d terminal-bench-4@4.0 \
+  -a codex \
+  -m gpt-5.5 \
+  -e modal \
+  -l 5 \
+  --watch
+```
+
+## Which tasks run where
+
+Not every task can run on every provider, and the catalog says so before any money is spent. `evolve dataset show` lists a verdict per provider for every task.
+
+```bash
+evolve dataset show terminal-bench-4@4.0
+```
+
+A verdict is `ok`, or refused with a reason. A GPU task carries a third form: `ok` with a note that the trial degrades to `modal`, the provider that reserves GPUs. Starting a job whose selected tasks include one that is refused on the chosen provider is rejected at creation with `provider_unsupported`, so nothing is spent on a trial that cannot execute.
+
+Three refusals appear in the verdicts: multi-container tasks run on `e2b` and `daytona` only; a multi-container task under `no-network` runs nowhere, so declare `allowlist` or `public`; and a size above a provider's ceiling refuses on that provider.
+
+| Provider | Max vCPUs | Max memory | Disk |
+| --- | --- | --- | --- |
+| `e2b` | 8 | 8192 MB | fixed, 20 GB |
+| `daytona` | 4 | 8192 MB | sized per task, up to 10 GB |
+| `modal` | 16 | 32768 MB | fixed, 512 GB |
+
+Two limits show only when the sandbox is created, as an `INFRASTRUCTURE_ERROR` with the reason in the trial's failure detail. On `daytona` an `allowlist` holds one kind of entry: IP addresses and CIDRs for a single-container task (at most 10), hostnames for a multi-container task (at most 20, `*.` covering a domain). `e2b` and `modal` take both kinds.
+
+Each provider keeps a box alive for at most its own plan's lifetime, 24 hours on `e2b` and `modal`. A timeout stretched past it fails the trial.
+
+## GPU tasks
+
+A task declares `gpus` and `gpu_types` in `[environment]`; `gpu_types` null means any type. GPU compute is paid by the platform; a GPU trial draws your credits for its model calls only. `modal` is the provider that reserves GPUs today, so a GPU task runs on `modal` whatever the job picked, and the trial records the move in `sandbox_provider_degrade`. A queued GPU trial past the fleet's `gpu_concurrency_cap` waits; a GPU count above `modal`'s ceiling is refused at import.
+
+Every settled GPU trial states its compute as an estimate, `gpu_cost`, from the sandbox's lifetime at the provider's list price. It is a separate figure, never added to the model spend; the job sums it as `stats.gpu_cost_usd`. `evolve trial show` prints it as `gpu compute (est.)`, and `unpriced_reason` says why when no honest number exists.
+
+## The analyzer's and the checker's sandbox
+
+Trace analysis and task checks run in sandboxes too. `-e` on `evolve analyze` and on `evolve check`, and `--analyze-provider` on `evolve run`, pick the provider. Omitted, the platform's analysis default applies, `daytona`.

package/docs-evals/core-concepts/secrets.mdx

@@ -0,0 +1,41 @@
+---
+title: "Secrets"
+description: "Store env secrets once and attach them to jobs by name."
+---
+
+An env secret is a value stored under a name, with a delivery mode. Values are write-only: every read returns metadata, never the value.
+
+```bash
+printf %s "$GITHUB_TOKEN" | evolve secrets set GITHUB_TOKEN --delivery direct
+```
+
+Pipe the value on stdin, as above, to keep it out of your shell history; `--value <value>` is the other channel. `--delivery` is required: `direct` places the value in the sandbox environment, `brokered` keeps it out of every sandbox and needs the `--allowed-host`, `--allowed-path-prefix` and `--allowed-method` scoping. Eval trials take `direct` secrets only; a brokered one is refused with `secret_brokered_unsupported`. `--label <label>` keeps several values of one name side by side; the default label is `default`.
+
+Restating the same value re-shapes its delivery or scoping. A different value under an existing name and label is refused with `secret_exists`: delete it first, or use another label.
+
+```bash
+evolve secrets list
+evolve secrets delete GITHUB_TOKEN --label staging
+```
+
+## Attach a secret to a job
+
+```bash
+evolve run \
+  -d terminal-bench-4@4.0 \
+  -a codex \
+  -m gpt-5.5 \
+  --secret GITHUB_TOKEN \
+  -l 5 \
+  --watch
+```
+
+`--secret NAME[@LABEL][=ENVNAME]` attaches a stored secret to every agent run. `@LABEL` picks a labeled row; omitted, the `default` row or the only row, and several labels with no `default` is refused with `secret_ambiguous`. `=ENVNAME` renames the variable inside the sandbox; a name the platform owns, such as the `EVOLVE_` prefix or `REWARDKIT_JUDGE`, is refused. The job stores a reference, and the value never rides the command line.
+
+In the SDK the same attachment is `secrets: [{ name, label?, as? }]` on `jobs().start()`. `--secret-inline NAME[@LABEL]:DELIVERY=VALUE`, or `{ name, value, delivery, label?, as? }` in that list, saves the value and attaches it in one step; re-running with the same value converges, a different value is refused with `secret_exists`.
+
+A task's `[environment.env]` can ask for a secret with a `${VAR}` template, satisfied by a secret attached under that env name; see [tasks](/core-concepts/tasks#environment-variables). Plain, non-secret environment does not go on a job: the server refuses `agent_env` (`--ae`). Declare it in the task's `[environment.env]` table instead.
+
+<Card title="secrets reference" icon="terminal" href="/cli-reference/secrets">
+  Every flag of `evolve secrets`.
+</Card>

package/docs-evals/core-concepts/skills.mdx

@@ -0,0 +1,42 @@
+---
+title: "Skills"
+description: "Mount skills into every agent run, from a git source or from your uploads."
+---
+
+A skill is a folder of instructions mounted into the agent's run. Give a job one or more with `--skills`; every arm gets them.
+
+```bash
+evolve run \
+  -d terminal-bench-4@4.0 \
+  -a claude \
+  -m fable \
+  --skills skills.sh/acme/skills/pdf \
+  -l 5 \
+  --watch
+```
+
+`--skills` (also spelled `--skill`) repeats, and accepts:
+
+- `skills.sh/<owner>/<repo>[/<skill>]`
+- `org/repo[@ref]`
+- an https git URL
+- `upload:<id>`, a skill you uploaded
+- `name:<skill-name>`, your moving name pointer, resolved on the server to its current upload
+- a local folder, which is uploaded first and then referenced
+
+## Upload a skill
+
+```bash
+evolve skill upload ./my-skill
+evolve skill list
+evolve skill show name:my-skill
+evolve skill delete <id>
+```
+
+The folder's name becomes your moving name pointer: `name:my-skill` resolves to the latest upload with that name, and an unknown name is refused with `skill_name_not_found`. `skill show` prints the metadata and the `SKILL.md`. A skill that a running job references cannot be deleted, and past jobs keep the version they ran with.
+
+The job's arm records what mounted under `skill_locks`: name, pinned source, digest and, for a git skill, the repository and commit. A skill that cannot be fetched fails the trial as an infrastructure error, never as a score.
+
+<Card title="skill reference" icon="terminal" href="/cli-reference/skill">
+  Every flag of `evolve skill`.
+</Card>