@evolvingmachines/evolve 0.0.55-project-sable.20260916.db0bc0bb

package/docs-evals/cli-reference/secrets.mdx

@@ -0,0 +1,95 @@
+---
+title: "evolve secrets"
+description: "Store and manage env secrets. Values are write-only; reads return metadata."
+---
+
+import GlobalOptions from '/snippets/global-options.mdx';
+
+```bash
+evolve secrets <command> [options]
+```
+
+`secrets` is the one plural noun; `secret` is a hidden alias. It is the same vault the dashboard's Secrets page writes, reached with your API key.
+
+The same three operations exist on the SDK's `Evolve.managedSecrets()`.
+
+A read-only API key may `list` but not `set` or `delete` (`read_only_key`). LLM provider keys cannot be stored here; they decide who pays and stay on the dashboard.
+
+## secrets set
+
+```bash
+evolve secrets set <NAME> [options]
+```
+
+Store an env secret, or re-shape its delivery or scoping by restating the same value. A different value under an existing name and label is refused (`secret_exists`, 409): delete first, or use another label. Never a silent overwrite. `NAME` matches `[A-Z_][A-Z0-9_]{0,127}`, uppercased for you; the `EVOLVE_` prefix is reserved.
+
+<ParamField path="--value" type="value">
+  The secret value, at most 190 bytes of UTF-8. Omit it to pipe the value on stdin, which keeps it out of shell history; one trailing newline is stripped from the pipe. A terminal with neither channel is a usage error, never a hang.
+</ParamField>
+
+<ParamField path="--label" type="label" default="default">
+  Labeled-row identity, at most 80 characters of `[A-Za-z0-9._-]`. Several values of one name live side by side.
+</ParamField>
+
+<ParamField path="--delivery" type="mode" required>
+  `brokered`: the value never enters a sandbox, and needs the `--allowed-*` scoping. `direct`: the raw value in the sandbox env; scoping refused.
+</ParamField>
+
+<ParamField path="--allowed-host" type="host">
+  Brokered scoping: a hostname, or a wildcard like `*.example.com`. Repeatable.
+</ParamField>
+
+<ParamField path="--allowed-path-prefix" type="/prefix">
+  Brokered scoping: an allowed URL path prefix. Repeatable.
+</ParamField>
+
+<ParamField path="--allowed-method" type="METHOD">
+  Brokered scoping: an allowed HTTP method. Repeatable.
+</ParamField>
+
+```bash
+printf %s "$GITHUB_TOKEN" | evolve secrets set GITHUB_TOKEN --delivery brokered \
+  --allowed-host api.github.com --allowed-path-prefix / --allowed-method GET
+```
+
+## secrets list
+
+```bash
+evolve secrets list [options]
+```
+
+List your env secrets. Metadata only; values never leave the server.
+
+<ParamField path="--columns" type="keys | all | help">
+  Choose and order columns, comma-separated. `help` lists the keys.
+</ParamField>
+
+<ParamField path="-q, --quiet">
+  Print only `name[:label]`, one per line.
+</ParamField>
+
+<ParamField path="--no-trunc">
+  Full cell content instead of one-line truncation.
+</ParamField>
+
+<ParamField path="--no-headers">
+  Omit the header row in piped (TSV) output.
+</ParamField>
+
+## secrets delete
+
+```bash
+evolve secrets delete <NAME> [options]
+```
+
+Delete an env secret by name. Revokes every runtime grant riding the row. Rotate a value by `delete` then `set`, or store the new value under another label.
+
+<ParamField path="--label" type="label">
+  The labeled row to delete. Omitted, it resolves as an attachment does: the `default` row, else the only row, else refused with `secret_ambiguous` naming every label.
+</ParamField>
+
+```bash
+evolve secrets delete GITHUB_TOKEN --label staging
+```
+
+<GlobalOptions />

package/docs-evals/cli-reference/skill.mdx

@@ -0,0 +1,51 @@
+---
+title: "evolve skill"
+description: "Upload and manage platform-stored skills."
+---
+
+import GlobalOptions from '/snippets/global-options.mdx';
+import ListFlags from '/snippets/list-flags.mdx';
+
+```bash
+evolve skill <command> [options]
+```
+
+## skill list
+
+```bash
+evolve skill list [options]
+```
+
+List your uploaded skills, newest first.
+
+<ListFlags />
+
+## skill upload
+
+```bash
+evolve skill upload <dir>
+```
+
+Upload a skill folder. Its name becomes your moving name pointer.
+
+## skill show
+
+```bash
+evolve skill show <id | name:<skill-name>>
+```
+
+Show one uploaded skill: its metadata and its `SKILL.md`.
+
+```bash
+evolve skill show name:my-skill
+```
+
+## skill delete
+
+```bash
+evolve skill delete <id>
+```
+
+Delete an uploaded skill record. Past jobs keep their locks. Refused (`skill_in_use`, 409) while a running job references it.
+
+<GlobalOptions />

package/docs-evals/cli-reference/skills.mdx

@@ -0,0 +1,81 @@
+---
+title: "evolve skills"
+description: "The skills the CLI serves to coding agents."
+---
+
+import GlobalOptions from '/snippets/global-options.mdx';
+
+```bash
+evolve skills <command> [options]
+```
+
+The skills ship with the CLI and match its version. `skills get evals` is the index of these docs and `skills get evals <page>` one page of them. The `evolve` pointer skill that `skills install` writes is served but never listed. `EVOLVE_SKILLS_DIR` names another checkout or package root to serve.
+
+## skills list
+
+```bash
+evolve skills list
+```
+
+List the skills the installed version serves, one line each. Bare `evolve skills` does the same.
+
+## skills get
+
+```bash
+evolve skills get <name> [name...]
+evolve skills get <name> <page>
+```
+
+Print one or more skills, or one page. A second word that is not a skill name is a page of the first skill: this site's own path without the suffix (`core-concepts/tasks`), or a file under `references/` for a task-authoring skill.
+
+<ParamField path="--full">
+  Also print every page of the skill (a docs folder's pages, or `references/` and `templates/`), each behind a `--- <path> ---` line.
+</ParamField>
+
+<ParamField path="--all">
+  Every skill, instead of naming them.
+</ParamField>
+
+```bash
+evolve skills get evals
+evolve skills get evals core-concepts/tasks
+evolve skills get create-task --full
+```
+
+## skills path
+
+```bash
+evolve skills path [name]
+```
+
+Print the root the skills are served from, or one skill's folder.
+
+## skills install
+
+```bash
+evolve skills install [options]
+```
+
+Install the `evolve` pointer skill for your agents. Each target gets `<folder>/evolve/SKILL.md`; the path written is printed. A file already there with different content is left alone unless `--force`.
+
+<ParamField path="--target" type="claude | codex | cursor | copilot | gemini | opencode | agents | all">
+  Which agent home: `~/.claude`, `~/.codex`, `~/.cursor`, `~/.copilot`, `~/.gemini`, `~/.config/opencode` or `~/.agents`, each under `skills/`. Defaults to `all`.
+</ParamField>
+
+<ParamField path="--path" type="dir">
+  A skills directory of your own instead of `--target`.
+</ParamField>
+
+<ParamField path="--force">
+  Overwrite an installed `SKILL.md` whose content differs.
+</ParamField>
+
+```bash
+evolve skills install
+evolve skills install --target claude
+evolve skills install --path ./skills
+```
+
+With the [skills CLI](https://github.com/vercel-labs/skills) instead, the same pointer installs from the repository: `npx skills add evolving-machines-lab/evolve`.
+
+<GlobalOptions />

package/docs-evals/cli-reference/trial.mdx

@@ -0,0 +1,112 @@
+---
+title: "evolve trial"
+description: "Inspect, download, and act on single trials."
+---
+
+import GlobalOptions from '/snippets/global-options.mdx';
+
+```bash
+evolve trial <command> [options]
+```
+
+Every command below takes the [global options](/cli-reference#global-options).
+
+## trial show
+
+```bash
+evolve trial show <trial-id>
+```
+
+Show one trial in full detail: its status, reward, `tokens`, `spent` and `spent (judge)` rows, `gpu compute (est.)` on a GPU trial, the exception when it failed, and its latest analysis with the verdicts, the summary and the typed failure. An uploaded trial prints its provider as `ported` and its reported figures apart from the metered ones.
+
+There is no `trial files` or `trial read` verb; listing and reading a trial's stored files is SDK-only, `trials().files()` and `file()`.
+
+## trial trace
+
+```bash
+evolve trial trace <trial-id> [options]
+```
+
+Print a trial's parsed trace, filtered server-side.
+
+<ParamField path="--type" type="event-type">
+  Only events of exactly this type. At most 100 characters.
+</ParamField>
+
+<ParamField path="--grep" type="pattern">
+  Only events matching this case-insensitive regex over type and content. A plain string is a plain substring. At most 512 characters; an invalid pattern is refused by the server with `invalid_input`.
+</ParamField>
+
+<ParamField path="--tail" type="n">
+  Only the last N matching events, 1 to 10000, then paged oldest first.
+</ParamField>
+
+<ParamField path="--cursor" type="seq">
+  Resume after this sequence number. The filters compose with it.
+</ParamField>
+
+<ParamField path="-l, --limit" type="n" default="200">
+  Events per page fetch, not a total: the command drains the filtered trace. Max 1000.
+</ParamField>
+
+```bash
+evolve trial trace d1a10c4e-… --grep 'permission denied' --tail 50
+```
+
+## trial download
+
+```bash
+evolve trial download <trial-id> [options]
+```
+
+Save a trial as its trial tree plus `evolve.json`, or stream one artifact to stdout. The two modes are exclusive.
+
+<ParamField path="-o, --output-dir" type="dir" default="trials/">
+  Directory to save under. Files land in `<dir>/<trial-id>/`.
+</ParamField>
+
+<ParamField path="--overwrite">
+  Replace an existing `<dir>/<trial-id>/`.
+</ParamField>
+
+<ParamField path="--stream" type="artifact">
+  Print one artifact to stdout instead of saving: `trace-parsed`, `verifier`, `trace-stdout`, `trace-stderr`, `trace-atif` (the ATIF trajectory), `trajectory` (reserved: the harness-native session file) or `agent-home`.
+</ParamField>
+
+<ParamField path="--cursor" type="seq">
+  With `--stream trace-parsed`: resume after this sequence number.
+</ParamField>
+
+<ParamField path="--limit" type="n">
+  With `--stream trace-parsed`: max events per page.
+</ParamField>
+
+```bash
+evolve trial download d1a10c4e-… --stream trace-stdout
+```
+
+## trial retry
+
+```bash
+evolve trial retry <trial-id>
+```
+
+Run one settled trial again. The result is a job.
+
+## trial regrade
+
+```bash
+evolve trial regrade <trial-id>
+```
+
+Verifier-only re-run of one trial. The result is a job.
+
+## trial stop
+
+```bash
+evolve trial stop <trial-id> [trial-id...]
+```
+
+Stop in-flight trials without cancelling their job.
+
+<GlobalOptions />

package/docs-evals/cli-reference/upload.mdx

@@ -0,0 +1,36 @@
+---
+title: "evolve upload"
+description: "Upload a job directory as a finished job."
+---
+
+import GlobalOptions from '/snippets/global-options.mdx';
+
+```bash
+evolve upload <job_dir> [options]
+```
+
+Upload a job directory in the Harbor job layout, its `.tar.gz`, or a public URL of one, as a terminal job. The command follows the import to the job, prints the record with the task linkage (`linked N of M trials to <dataset>` and the not-linked tasks with their reasons), names any skipped trials, and ends with the analyze hint.
+
+```bash
+evolve upload ./job-2026-08-27__12-00-00 -d terminal-bench-4@4.0
+```
+
+## Options
+
+<ParamField path="-d, --dataset" type="name[@version]">
+  Link the uploaded trials to a published dataset version by task name. Without it the trials link by their task hash on their own; this is the override, and the way to settle an ambiguous hash. See [upload](/core-concepts/upload#task-linkage).
+</ParamField>
+
+<ParamField path="--from" type="url">
+  A public https URL of the job archive (`.tar.gz`) that the server fetches itself, instead of `<job_dir>`. No local bytes.
+</ParamField>
+
+<ParamField path="--no-wait">
+  Return after the import is accepted, with the import id, instead of following it to the job. Re-attach with `evolve job import <id> --watch`.
+</ParamField>
+
+<GlobalOptions />
+
+## Following an import
+
+`evolve job imports` lists your uploads, and `evolve job import <import-id> --watch` follows one. See the [`job`](/cli-reference/job) group.

package/docs-evals/core-concepts/agents.mdx

@@ -0,0 +1,106 @@
+---
+title: "Agents"
+description: "The built-in harnesses, how to configure an arm, and how to register your own agent."
+---
+
+An agent is a harness plus a model. `-a` names the harness, `-m` the model it drives. Seven harnesses are built in.
+
+| Name on `-a` | Harness |
+| --- | --- |
+| `claude` | Claude Code |
+| `codex` | Codex |
+| `gemini` | Gemini CLI |
+| `qwen` | Qwen Code |
+| `kimi` | Kimi Code |
+| `opencode` | OpenCode |
+| `droid` | Droid |
+
+```bash
+evolve run \
+  -d terminal-bench-4@4.0 \
+  -a claude \
+  -m fable \
+  -l 5 \
+  --watch
+```
+
+`-a <name[@version]>` can pin a harness version. Omitted, the latest is chosen once, when the job is created, so every trial installs the same version; it is recorded on every trial as `agent_info.version`. A pin that does not exist is refused with `agent_version_not_found`, and a pin on a registered agent is refused.
+
+## Configure the arm
+
+These flags apply to every arm of the job.
+
+```bash
+evolve run \
+  -d terminal-bench-4@4.0 \
+  -a codex \
+  -m gpt-5.5 \
+  -l 5 \
+  --effort high \
+  --preset no-internet \
+  --ak config=./codex-settings.json
+```
+
+`--effort <value>` sets the reasoning effort; see [models](/core-concepts/models#reasoning-effort). `--preset <name>` applies a named settings bundle: `no-internet` turns the vendor's server-side web tools off, `pinned-context` fixes the context window at 200000 tokens. `--ak config=<path|inline JSON>` becomes the harness's native settings file inside the sandbox; your document is the base, the platform's routing sits on top, and a preset wins where they disagree.
+
+Only `claude` and `codex` take a config or a preset; the [capability document](/sdk-reference/meta) publishes `supports_config` and `presets` per harness. An effort, preset or config a harness cannot honor is refused when the job is created, never silently skipped, and so is a config key that touches billing, base URLs, routing or environment.
+
+Environment for the agent comes from two places, a task's own `[environment.env]` table and a secret attached with `--secret`; a job cannot add a third, and the server refuses `agent_env` (the `--ae` flag). See [secrets](/core-concepts/secrets) and [tasks](/core-concepts/tasks#environment-variables).
+
+## Register your own agent
+
+Any CLI can run as an agent. Register it once, and its name works on `-a` like a built-in.
+
+```bash
+evolve agent add acme-cli --install-script ./install.sh --run "acme-cli --headless"
+evolve run \
+  -d terminal-bench-4@4.0 \
+  -a acme-cli \
+  -m gpt-5.5 \
+  -l 5 \
+  --watch
+```
+
+`--install-script <path>` names a file; its contents are uploaded. `--dir <path>` uploads a local directory instead. `--run <command>` is required and runs with `sh -c`. `--ae KEY=VALUE` injects environment at run time and may repeat.
+
+```bash
+evolve agent list
+evolve agent show acme-cli
+evolve agent remove acme-cli
+```
+
+Removing an agent keeps the record of the past jobs that ran it. Registered agents are private, so another account's name reads as `agent_not_found`. The number you may hold is published as `agent_registration.max_per_user`; past it, `agent_limit_reached`.
+
+The install script, or the uploaded directory, runs once with internet and no secrets: fetch only public sources and leave the executables in `$PREFIX/bin`. To change an agent without a gap, use `upsert` rather than remove and add.
+
+### The run contract
+
+Your `run_command` runs with `sh -c` at the task's working directory, and can rely on exactly this:
+
+- The instruction arrives twice: on stdin, and at the path in `$EVOLVE_INSTRUCTION_FILE`.
+- `$EVOLVE_GATEWAY_BASE_URL` is an OpenAI-compatible base URL that already ends in `/v1`, and `$EVOLVE_GATEWAY_API_KEY` is its credential. The same two values are also `$OPENAI_BASE_URL` and `$OPENAI_API_KEY`, so a CLI that reads its endpoint from the environment works unchanged.
+- `$EVOLVE_MODEL` is the `model_name` of the arm.
+- Your declared env may not override those six keys, and may not look like a credential; both are refused at registration with `agent_invalid_env`.
+
+A CLI that routes through a config file must write that file inside `run_command`, from the contract values. `codex` is the worked example:
+
+```bash
+mkdir -p ~/.codex && cat > ~/.codex/config.toml <<EOF
+model_provider = "evolve"
+[model_providers.evolve]
+name = "evolve"
+base_url = "$EVOLVE_GATEWAY_BASE_URL"
+env_key = "EVOLVE_GATEWAY_API_KEY"
+wire_api = "responses"
+EOF
+codex login --with-api-key <<< "$EVOLVE_GATEWAY_API_KEY"
+codex exec --skip-git-repo-check -
+```
+
+A CLI that ignores `OPENAI_BASE_URL` without this reaches for its vendor's endpoint, finds the box sealed, and spends the agent budget failing to connect. The spend cap is airtight only under `no-network`; see [models](/core-concepts/models).
+
+Against a built-in, a registered agent has no live trace (there is no parser for an unknown CLI) and no `reasoning_effort` (put the flag in `run_command`); everything else is recorded the same way.
+
+<Card title="agent reference" icon="terminal" href="/cli-reference/agent">
+  Every flag of `evolve agent`.
+</Card>

package/docs-evals/core-concepts/analyze.mdx

@@ -0,0 +1,86 @@
+---
+title: "Analyze"
+description: "Judge every trial's trace against a rubric, on the platform."
+---
+
+An analysis reads a settled trial's whole record, the trace and the task, and rules on every criterion of a rubric. It runs on the platform, one analyzer run per trial. Ask for it on a finished job:
+
+```bash
+evolve analyze <job-id> --watch
+```
+
+Or arm it when you start the job, so each trial is analyzed the moment it settles:
+
+```bash
+evolve run \
+  -d terminal-bench-4@4.0 \
+  -a codex \
+  -m gpt-5.5 \
+  --analyze \
+  -l 5 \
+  --watch
+```
+
+## Rubrics and prompts
+
+The default rubric has seven criteria, `score_is_earned` first. Bring your own as TOML, YAML or JSON in the shape below and, if you want, your own prompt for the analyzer.
+
+```toml rubric.toml
+[[criteria]]
+name = "score_is_earned"
+description = "The reward reflects work the agent actually did."
+guidance = "Read the verifier log and the trajectory side by side."
+```
+
+```bash
+evolve analyze <job-id> \
+  -r rubric.toml \
+  -p prompt.txt \
+  --failing \
+  -l 20 \
+  -n 2 \
+  --watch
+```
+
+`-r` is the rubric file, `-p` the prompt file. `--passing` and `--failing` restrict the batch of analyses to trials with reward 1.0, or below it; both together is refused. `-l` caps the number of trials, `-n` how many analyses run at once, beneath the organization's `max_concurrent_analyses` (fleet default 60). `-m` picks the analyzer's model, `--effort` its reasoning effort, `-e` its sandbox provider.
+
+A result that misses or invents a criterion is a failed analysis, never a partial pass. A rubric with unknown keys, empty or duplicate criteria, or a length bound exceeded is refused with `invalid_rubric`; an off-roster model or an unknown effort or provider with `invalid_input`; a job with no analyzable trial, or a `--passing` or `--failing` filter that selects none, with `no_analyzable_trials`; a job still running with `job_not_terminal`.
+
+On `evolve run` the same knobs are `--analyze-model`, `--analyze-rubric`, `--analyze-prompt`, `--analyze-effort` and `--analyze-provider`. Each of them implies `--analyze`.
+
+Running `evolve analyze` again on the same job, with another rubric or model, is the re-analysis path: one wave at a time (`analysis_already_running` meanwhile), and each trial then serves its newest analysis while earlier ones stay under their own ids. Cancelled trials are never analyzed.
+
+## The result
+
+Each analysis produces a short summary of what happened in the trial, and one entry per criterion:
+
+- `outcome`: `pass`, `fail`, `not_applicable` or `unknown`
+- `explanation`: the analyzer's reasoning, in plain words
+- `evidence`: what in the trial record supports it
+
+Under the default rubric, the platform derives one label from the outcomes:
+
+- `flagged`: a fail on `score_is_earned`, `score_is_correct`, `task_was_fair` or `report_is_truthful`
+- `env_fault`: otherwise, a fail on `environment_worked`
+- `unclear`: otherwise, an unknown on any of those five, or `not_applicable` on `score_is_earned` or `score_is_correct`
+- `clean`: everything else
+
+A custom rubric carries its per-criterion outcomes and no label.
+
+The analyzer's spend is shown apart from the trial's: `estimated_cost_usd` on the analysis, `stats.analysis.cost_usd` on the job. A failed analysis carries `failure` with a `phase` and a `message`; a run that produces no valid result is retried once.
+
+## Read it back
+
+```bash
+evolve analysis list --job <job-id> --status failed
+evolve analysis show <analysis-id>
+evolve analysis trace <analysis-id>
+evolve analysis download <analysis-id> -o analyses/
+evolve analysis download <analysis-id> --stream analysis
+```
+
+`analysis show` prints the verdict document. Every analysis verb also accepts a trial id, which names that trial's latest analysis. `analysis trace` prints the analyzer's own transcript. `analysis download` saves the run as a trial folder, or streams one artifact: `analysis`, `trace-parsed`, `trace-stdout`, `trace-stderr` or `agent-home`.
+
+<Card title="analyze reference" icon="terminal" href="/cli-reference/analyze">
+  Every flag of `evolve analyze` and `evolve analysis`.
+</Card>

package/docs-evals/core-concepts/check.mdx

@@ -0,0 +1,51 @@
+---
+title: "Check"
+description: "Check the quality of tasks against a rubric before you spend a job on them."
+---
+
+A check reads a task and, when it can, runs its environment, its reference solution and its verifier, then rules on every criterion of a rubric. Point it at a task directory, or at a directory of tasks:
+
+```bash
+evolve check ./tasks \
+  -i 'abs-*' \
+  -l 5 \
+  --watch
+```
+
+Or at a published dataset version. Nothing is uploaded; the checker reads the retained task package.
+
+```bash
+evolve check -d my-swe@1.0 --watch
+```
+
+When the path is a task directory, that task is checked; otherwise every top-level directory that is one, in sorted order. `-i` and `-x` are globs over task names and `-l` caps the count, in that order; a selection that leaves nothing is refused with `no_checkable_tasks`. There is no task-count ceiling.
+
+`-r` is a rubric file, `-p` a prompt file. `-m` and `--effort` set the checker's model and effort, `-e` its sandbox provider, `-n` how many checks run at once, beneath the organization's `max_concurrent_analyses`.
+
+The default rubric has eleven criteria: `instruction_is_sufficient`, `tests_match_the_instruction`, `verifier_is_correct`, `no_answer_leakage`, `grader_is_out_of_reach`, `reference_solution_is_valid`, `verifier_rejects_non_solutions`, `environment_builds_and_runs`, `verification_is_stable`, `limits_allow_the_task` and `task_is_solvable`. The checker's default model is the analyzer's; a custom rubric, model, prompt and effort follow the same rules and refusals as [analyze](/core-concepts/analyze).
+
+## The result
+
+Each task gets one entry per criterion, with an `outcome` (`pass`, `fail`, `not_applicable` or `unknown`), an `explanation` and `evidence`. Under the default rubric, one label is derived:
+
+- `has_a_problem`: a fail on any criterion
+- `unclear`: otherwise, an unknown on any of the six file-based criteria
+- `no_problem_found`: everything else
+
+`executed` says whether the checker actually ran the task's environment, so a reading-only `no_problem_found` is never mistaken for a run. A check never fails as a whole: refusals happen at accept, and a task's failure is on that task. A check is readable by its creator and by every member of the organization; anyone else reads `check_not_found`.
+
+## Read it back
+
+```bash
+evolve check list --status running
+evolve check show <check-id>
+evolve check trace <task-check-id> --since 200
+evolve check download <check-id> -o checks/
+evolve check download <task-check-id> --stream task-check
+```
+
+`check show` prints the check with one result per task. `check trace` prints the checker's transcript for one task check. `check download` saves the whole check as a check folder by its check id, one task's folder by its task check id, or streams one artifact of a task check: `task-check`, `trace-parsed`, `trace-stdout`, `trace-stderr` or `agent-home`.
+
+<Card title="check reference" icon="terminal" href="/cli-reference/check">
+  Every flag of `evolve check`.
+</Card>