@evolvingmachines/evolve 0.0.65 → 0.0.67-train-harness-wave-2026-09-25.20260926.e00a8a6d
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/{chunk-DK2MGUPB.js → chunk-DI6LPW2B.js} +2 -2
- package/dist/cli/index.cjs +29 -29
- package/dist/cli/index.d.cts +3 -3
- package/dist/cli/index.d.ts +3 -3
- package/dist/cli/index.js +18 -18
- package/dist/index.cjs +64 -60
- package/dist/index.d.cts +297 -18
- package/dist/index.d.ts +297 -18
- package/dist/index.js +48 -44
- package/dist/{types-B_qmzlMk.d.cts → types-Dj0iMl5V.d.cts} +70 -90
- package/dist/{types-B_qmzlMk.d.ts → types-Dj0iMl5V.d.ts} +70 -90
- package/harness-capabilities.json +285 -0
- package/package.json +9 -6
- package/skills/create-adapter/SKILL.md +71 -26
- package/skills/create-task/SKILL.md +124 -66
- package/skills/evolve/SKILL.md +37 -20
- package/skills/evolve-evals/SKILL.md +142 -69
- package/skills/evolve-evals/references/cli-reference/agent.md +76 -0
- package/skills/evolve-evals/references/cli-reference/analysis.md +78 -0
- package/skills/evolve-evals/references/cli-reference/analyze.md +97 -0
- package/skills/evolve-evals/references/cli-reference/auth.md +96 -0
- package/skills/evolve-evals/references/cli-reference/check.md +157 -0
- package/skills/evolve-evals/references/cli-reference/dataset.md +154 -0
- package/skills/evolve-evals/references/cli-reference/filesystem.md +168 -0
- package/skills/evolve-evals/references/cli-reference/index.md +149 -0
- package/skills/evolve-evals/references/cli-reference/job.md +198 -0
- package/skills/evolve-evals/references/cli-reference/run.md +188 -0
- package/skills/evolve-evals/references/cli-reference/secrets.md +77 -0
- package/skills/evolve-evals/references/cli-reference/session.md +56 -0
- package/skills/evolve-evals/references/cli-reference/skill.md +52 -0
- package/skills/evolve-evals/references/cli-reference/skills.md +85 -0
- package/skills/evolve-evals/references/cli-reference/trial.md +115 -0
- package/skills/evolve-evals/references/cli-reference/upload.md +72 -0
- package/skills/evolve-evals/references/core-concepts/agents.md +101 -0
- package/skills/evolve-evals/references/core-concepts/analyze.md +143 -0
- package/skills/evolve-evals/references/core-concepts/check.md +116 -0
- package/skills/evolve-evals/references/core-concepts/compatibility.md +39 -0
- package/skills/evolve-evals/references/core-concepts/compose.md +103 -0
- package/skills/evolve-evals/references/core-concepts/datasets.md +234 -0
- package/skills/evolve-evals/references/core-concepts/index.md +52 -0
- package/skills/evolve-evals/references/core-concepts/jobs.md +223 -0
- package/skills/evolve-evals/references/core-concepts/models.md +73 -0
- package/skills/evolve-evals/references/core-concepts/multi-step.md +93 -0
- package/skills/evolve-evals/references/core-concepts/rewardkit.md +341 -0
- package/skills/evolve-evals/references/core-concepts/sandboxes.md +73 -0
- package/skills/evolve-evals/references/core-concepts/secrets.md +86 -0
- package/skills/evolve-evals/references/core-concepts/sharing.md +81 -0
- package/skills/evolve-evals/references/core-concepts/skills.md +100 -0
- package/skills/evolve-evals/references/core-concepts/task-config.md +169 -0
- package/skills/evolve-evals/references/core-concepts/task-environment.md +127 -0
- package/skills/evolve-evals/references/core-concepts/task-reference.md +191 -0
- package/skills/evolve-evals/references/core-concepts/task-verifiers.md +209 -0
- package/skills/evolve-evals/references/core-concepts/tasks.md +90 -0
- package/skills/evolve-evals/references/core-concepts/teams.md +128 -0
- package/skills/evolve-evals/references/core-concepts/trial-outputs.md +225 -0
- package/skills/evolve-evals/references/core-concepts/trials.md +129 -0
- package/skills/evolve-evals/references/core-concepts/upload-sdk-session.md +240 -0
- package/skills/evolve-evals/references/core-concepts/upload.md +166 -0
- package/skills/evolve-evals/references/dashboard/checks.md +54 -0
- package/skills/evolve-evals/references/dashboard/datasets.md +65 -0
- package/skills/evolve-evals/references/dashboard/jobs.md +106 -0
- package/skills/evolve-evals/references/dashboard/traces.md +54 -0
- package/skills/evolve-evals/references/dashboard/trial-viewer.md +77 -0
- package/skills/evolve-evals/references/getting-started/first-task.md +228 -0
- package/skills/evolve-evals/references/getting-started/installation.md +79 -0
- package/skills/evolve-evals/references/getting-started/quick-start.md +100 -0
- package/skills/evolve-evals/references/index.md +58 -0
- package/skills/evolve-evals/references/sdk/http.md +110 -0
- package/skills/evolve-evals/references/sdk/python.md +108 -0
- package/skills/evolve-evals/references/sdk/typescript.md +104 -0
- package/skills/evolve-evals/references/sdk-reference/agents.md +75 -0
- package/skills/evolve-evals/references/sdk-reference/analyses.md +125 -0
- package/skills/evolve-evals/references/sdk-reference/auth.md +171 -0
- package/skills/evolve-evals/references/sdk-reference/checks.md +146 -0
- package/skills/evolve-evals/references/sdk-reference/datasets.md +170 -0
- package/skills/evolve-evals/references/sdk-reference/error-codes.md +231 -0
- package/skills/evolve-evals/references/sdk-reference/errors.md +98 -0
- package/skills/evolve-evals/references/sdk-reference/filesystem.md +166 -0
- package/skills/evolve-evals/references/sdk-reference/index.md +161 -0
- package/skills/evolve-evals/references/sdk-reference/jobs.md +250 -0
- package/skills/evolve-evals/references/sdk-reference/meta.md +54 -0
- package/skills/evolve-evals/references/sdk-reference/methods/agents.md +154 -0
- package/skills/evolve-evals/references/sdk-reference/methods/analyses.md +211 -0
- package/skills/evolve-evals/references/sdk-reference/methods/auth.md +257 -0
- package/skills/evolve-evals/references/sdk-reference/methods/checks.md +462 -0
- package/skills/evolve-evals/references/sdk-reference/methods/datasets.md +687 -0
- package/skills/evolve-evals/references/sdk-reference/methods/filesystem.md +527 -0
- package/skills/evolve-evals/references/sdk-reference/methods/jobs.md +1023 -0
- package/skills/evolve-evals/references/sdk-reference/methods/meta.md +213 -0
- package/skills/evolve-evals/references/sdk-reference/methods/secrets.md +157 -0
- package/skills/evolve-evals/references/sdk-reference/methods/skills.md +125 -0
- package/skills/evolve-evals/references/sdk-reference/methods/trials.md +371 -0
- package/skills/evolve-evals/references/sdk-reference/secrets.md +104 -0
- package/skills/evolve-evals/references/sdk-reference/skills.md +66 -0
- package/skills/evolve-evals/references/sdk-reference/trials.md +164 -0
- package/skills/evolve-evals/references/sdk-reference/types.md +1255 -0
- package/skills/publish/SKILL.md +103 -79
- package/skills/rewardkit/SKILL.md +78 -23
- package/spec/openapi.yaml +164 -160
- package/skills/evolve-evals/references/cli-reference/agent.mdx +0 -79
- package/skills/evolve-evals/references/cli-reference/analysis.mdx +0 -160
- package/skills/evolve-evals/references/cli-reference/analyze.mdx +0 -84
- package/skills/evolve-evals/references/cli-reference/auth.mdx +0 -133
- package/skills/evolve-evals/references/cli-reference/check.mdx +0 -263
- package/skills/evolve-evals/references/cli-reference/dataset.mdx +0 -151
- package/skills/evolve-evals/references/cli-reference/index.mdx +0 -108
- package/skills/evolve-evals/references/cli-reference/job.mdx +0 -278
- package/skills/evolve-evals/references/cli-reference/run.mdx +0 -204
- package/skills/evolve-evals/references/cli-reference/secrets.mdx +0 -95
- package/skills/evolve-evals/references/cli-reference/skill.mdx +0 -61
- package/skills/evolve-evals/references/cli-reference/skills.mdx +0 -81
- package/skills/evolve-evals/references/cli-reference/trial.mdx +0 -181
- package/skills/evolve-evals/references/cli-reference/upload.mdx +0 -36
- package/skills/evolve-evals/references/core-concepts/agents.mdx +0 -106
- package/skills/evolve-evals/references/core-concepts/analyze.mdx +0 -86
- package/skills/evolve-evals/references/core-concepts/check.mdx +0 -64
- package/skills/evolve-evals/references/core-concepts/datasets.mdx +0 -138
- package/skills/evolve-evals/references/core-concepts/index.mdx +0 -42
- package/skills/evolve-evals/references/core-concepts/jobs.mdx +0 -151
- package/skills/evolve-evals/references/core-concepts/models.mdx +0 -68
- package/skills/evolve-evals/references/core-concepts/sandboxes.mdx +0 -52
- package/skills/evolve-evals/references/core-concepts/secrets.mdx +0 -41
- package/skills/evolve-evals/references/core-concepts/sharing.mdx +0 -35
- package/skills/evolve-evals/references/core-concepts/skills.mdx +0 -42
- package/skills/evolve-evals/references/core-concepts/tasks.mdx +0 -216
- package/skills/evolve-evals/references/core-concepts/teams.mdx +0 -36
- package/skills/evolve-evals/references/core-concepts/trial-outputs.mdx +0 -117
- package/skills/evolve-evals/references/core-concepts/trials.mdx +0 -103
- package/skills/evolve-evals/references/core-concepts/upload.mdx +0 -93
- package/skills/evolve-evals/references/dashboard/checks.mdx +0 -21
- package/skills/evolve-evals/references/dashboard/datasets.mdx +0 -24
- package/skills/evolve-evals/references/dashboard/jobs.mdx +0 -65
- package/skills/evolve-evals/references/dashboard/traces.mdx +0 -17
- package/skills/evolve-evals/references/dashboard/trial-viewer.mdx +0 -30
- package/skills/evolve-evals/references/getting-started/installation.mdx +0 -56
- package/skills/evolve-evals/references/getting-started/quick-start.mdx +0 -94
- package/skills/evolve-evals/references/index.mdx +0 -48
- package/skills/evolve-evals/references/sdk/python.mdx +0 -161
- package/skills/evolve-evals/references/sdk/typescript.mdx +0 -163
- package/skills/evolve-evals/references/sdk-reference/agents.mdx +0 -108
- package/skills/evolve-evals/references/sdk-reference/analyses.mdx +0 -222
- package/skills/evolve-evals/references/sdk-reference/auth.mdx +0 -116
- package/skills/evolve-evals/references/sdk-reference/checks.mdx +0 -300
- package/skills/evolve-evals/references/sdk-reference/datasets.mdx +0 -346
- package/skills/evolve-evals/references/sdk-reference/errors.mdx +0 -210
- package/skills/evolve-evals/references/sdk-reference/index.mdx +0 -60
- package/skills/evolve-evals/references/sdk-reference/jobs.mdx +0 -822
- package/skills/evolve-evals/references/sdk-reference/meta.mdx +0 -52
- package/skills/evolve-evals/references/sdk-reference/skills.mdx +0 -69
- package/skills/evolve-evals/references/sdk-reference/trials.mdx +0 -302
- package/skills/evolve-evals/references/sdk-reference/types.mdx +0 -274
- package/skills/evolve-evals/references/snippets/global-options.mdx +0 -13
- package/skills/evolve-evals/references/snippets/list-flags.mdx +0 -23
|
@@ -1,81 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
title: "evolve skills"
|
|
3
|
-
description: "The skills the CLI serves to coding agents."
|
|
4
|
-
---
|
|
5
|
-
|
|
6
|
-
import GlobalOptions from '/snippets/global-options.mdx';
|
|
7
|
-
|
|
8
|
-
```bash
|
|
9
|
-
evolve skills <command> [options]
|
|
10
|
-
```
|
|
11
|
-
|
|
12
|
-
The skills ship with the CLI and match its version. `skills get evals` is the index of these docs and `skills get evals <page>` one page of them. The `evolve` pointer skill that `skills install` writes is served but never listed. The SDK skill is not served; it installs from the repository with `npx skills add evolving-machines-lab/evolve --skill evolve-agents`. `EVOLVE_SKILLS_DIR` names another skills directory to serve.
|
|
13
|
-
|
|
14
|
-
## skills list
|
|
15
|
-
|
|
16
|
-
```bash
|
|
17
|
-
evolve skills list
|
|
18
|
-
```
|
|
19
|
-
|
|
20
|
-
List the skills the installed version serves, one line each. Bare `evolve skills` does the same.
|
|
21
|
-
|
|
22
|
-
## skills get
|
|
23
|
-
|
|
24
|
-
```bash
|
|
25
|
-
evolve skills get <name> [name...]
|
|
26
|
-
evolve skills get <name> <page>
|
|
27
|
-
```
|
|
28
|
-
|
|
29
|
-
Print one or more skills, or one page. A second word that is not a skill name is a page of the first skill: its path under `references/` without the suffix, as this site spells it (`core-concepts/tasks`).
|
|
30
|
-
|
|
31
|
-
<ParamField path="--full">
|
|
32
|
-
Also print every file under `references/` and `templates/`, each behind a `--- <path> ---` line.
|
|
33
|
-
</ParamField>
|
|
34
|
-
|
|
35
|
-
<ParamField path="--all">
|
|
36
|
-
Every skill, instead of naming them.
|
|
37
|
-
</ParamField>
|
|
38
|
-
|
|
39
|
-
```bash
|
|
40
|
-
evolve skills get evals
|
|
41
|
-
evolve skills get evals core-concepts/tasks
|
|
42
|
-
evolve skills get create-task --full
|
|
43
|
-
```
|
|
44
|
-
|
|
45
|
-
## skills path
|
|
46
|
-
|
|
47
|
-
```bash
|
|
48
|
-
evolve skills path [name]
|
|
49
|
-
```
|
|
50
|
-
|
|
51
|
-
Print the skills directory, or one skill's folder.
|
|
52
|
-
|
|
53
|
-
## skills install
|
|
54
|
-
|
|
55
|
-
```bash
|
|
56
|
-
evolve skills install [options]
|
|
57
|
-
```
|
|
58
|
-
|
|
59
|
-
Install the `evolve` pointer skill for your agents. Each target gets `<folder>/evolve/SKILL.md`; the path written is printed. A file already there with different content is left alone unless `--force`.
|
|
60
|
-
|
|
61
|
-
<ParamField path="--target" type="claude | codex | cursor | copilot | gemini | opencode | agents | all">
|
|
62
|
-
Which agent home: `~/.claude`, `~/.codex`, `~/.cursor`, `~/.copilot`, `~/.gemini`, `~/.config/opencode` or `~/.agents`, each under `skills/`. Defaults to `all`.
|
|
63
|
-
</ParamField>
|
|
64
|
-
|
|
65
|
-
<ParamField path="--path" type="dir">
|
|
66
|
-
A skills directory of your own instead of `--target`.
|
|
67
|
-
</ParamField>
|
|
68
|
-
|
|
69
|
-
<ParamField path="--force">
|
|
70
|
-
Overwrite an installed `SKILL.md` whose content differs.
|
|
71
|
-
</ParamField>
|
|
72
|
-
|
|
73
|
-
```bash
|
|
74
|
-
evolve skills install
|
|
75
|
-
evolve skills install --target claude
|
|
76
|
-
evolve skills install --path ./skills
|
|
77
|
-
```
|
|
78
|
-
|
|
79
|
-
With the [skills CLI](https://github.com/vercel-labs/skills) instead, the same pointer installs from the repository: `npx skills add evolving-machines-lab/evolve`.
|
|
80
|
-
|
|
81
|
-
<GlobalOptions />
|
|
@@ -1,181 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
title: "evolve trial"
|
|
3
|
-
description: "Inspect, download, and act on single trials."
|
|
4
|
-
---
|
|
5
|
-
|
|
6
|
-
import GlobalOptions from '/snippets/global-options.mdx';
|
|
7
|
-
|
|
8
|
-
```bash
|
|
9
|
-
evolve trial <command> [options]
|
|
10
|
-
```
|
|
11
|
-
|
|
12
|
-
Every command below takes the [global options](/cli-reference#global-options).
|
|
13
|
-
|
|
14
|
-
## trial show
|
|
15
|
-
|
|
16
|
-
```bash
|
|
17
|
-
evolve trial show <trial-id>
|
|
18
|
-
```
|
|
19
|
-
|
|
20
|
-
Show one trial in full detail: its status, reward, `tokens`, `spent` and `spent (judge)` rows, `gpu compute (est.)` on a GPU trial, the exception when it failed, and its latest analysis with the verdicts, the summary and the typed failure. An uploaded trial prints its provider as `ported` and its reported figures apart from the metered ones.
|
|
21
|
-
|
|
22
|
-
The run's own files are under [`trial files`](#trial-files); the platform's stored artifacts (session files, the verifier log) are SDK-only, `trials().files()` and `file()`.
|
|
23
|
-
|
|
24
|
-
## trial trace
|
|
25
|
-
|
|
26
|
-
```bash
|
|
27
|
-
evolve trial trace <trial-id> [options]
|
|
28
|
-
```
|
|
29
|
-
|
|
30
|
-
Print a trial's parsed trace, filtered server-side.
|
|
31
|
-
|
|
32
|
-
<ParamField path="--type" type="event-type">
|
|
33
|
-
Only events of exactly this type. At most 100 characters.
|
|
34
|
-
</ParamField>
|
|
35
|
-
|
|
36
|
-
<ParamField path="--grep" type="pattern">
|
|
37
|
-
Only events matching this case-insensitive regex over type and content. A plain string is a plain substring. At most 512 characters; an invalid pattern is refused by the server with `invalid_input`.
|
|
38
|
-
</ParamField>
|
|
39
|
-
|
|
40
|
-
<ParamField path="--tail" type="n">
|
|
41
|
-
Only the last N matching events, 1 to 10000, then paged oldest first.
|
|
42
|
-
</ParamField>
|
|
43
|
-
|
|
44
|
-
<ParamField path="--cursor" type="seq">
|
|
45
|
-
Resume after this sequence number. The filters compose with it.
|
|
46
|
-
</ParamField>
|
|
47
|
-
|
|
48
|
-
<ParamField path="-l, --limit" type="n" default="200">
|
|
49
|
-
Events per page fetch, not a total: the command drains the filtered trace. Max 1000.
|
|
50
|
-
</ParamField>
|
|
51
|
-
|
|
52
|
-
```bash
|
|
53
|
-
evolve trial trace d1a10c4e-… --grep 'permission denied' --tail 50
|
|
54
|
-
```
|
|
55
|
-
|
|
56
|
-
## trial download
|
|
57
|
-
|
|
58
|
-
```bash
|
|
59
|
-
evolve trial download <trial-id> [options]
|
|
60
|
-
```
|
|
61
|
-
|
|
62
|
-
Save a trial as its trial tree plus `evolve.json`, or stream one artifact to stdout. The two modes are exclusive.
|
|
63
|
-
|
|
64
|
-
<ParamField path="-o, --output-dir" type="dir" default="trials/">
|
|
65
|
-
Directory to save under. Files land in `<dir>/<trial-id>/`.
|
|
66
|
-
</ParamField>
|
|
67
|
-
|
|
68
|
-
<ParamField path="--overwrite">
|
|
69
|
-
Replace an existing `<dir>/<trial-id>/`.
|
|
70
|
-
</ParamField>
|
|
71
|
-
|
|
72
|
-
<ParamField path="--stream" type="artifact">
|
|
73
|
-
Print one artifact to stdout instead of saving: `trace-parsed`, `verifier`, `trace-stdout`, `trace-stderr`, `trace-atif` (the ATIF trajectory), `trajectory` (reserved: the harness-native session file), `agent-home`, or `filesystem` (the run's file system as one `.tar.gz`, raw bytes — pipe it to a file; the same archive as `trial files archive`).
|
|
74
|
-
</ParamField>
|
|
75
|
-
|
|
76
|
-
<ParamField path="--cursor" type="seq">
|
|
77
|
-
With `--stream trace-parsed`: resume after this sequence number.
|
|
78
|
-
</ParamField>
|
|
79
|
-
|
|
80
|
-
<ParamField path="--limit" type="n">
|
|
81
|
-
With `--stream trace-parsed`: max events per page.
|
|
82
|
-
</ParamField>
|
|
83
|
-
|
|
84
|
-
```bash
|
|
85
|
-
evolve trial download d1a10c4e-… --stream trace-stdout
|
|
86
|
-
```
|
|
87
|
-
|
|
88
|
-
## trial files
|
|
89
|
-
|
|
90
|
-
```bash
|
|
91
|
-
evolve trial files <status|ls|cat|search|changes|archive> <trial-id> [path] [options]
|
|
92
|
-
```
|
|
93
|
-
|
|
94
|
-
The run's file system: while the box runs, `files` reads it as it changes; after the run, the same commands read the kept tree. `files status` says which you get.
|
|
95
|
-
|
|
96
|
-
- `files status <trial-id>` — `live`, `captured`, `capturing` (the kept tree is still being written) or `none`; the box, how change events arrive (`watcher`), and the capture record (`ready`, `incomplete` with what was left out, or `failed`).
|
|
97
|
-
- `files ls <trial-id> [path]` — one folder, sorted by name (default `/`). Each row shows the type, mode, size, mtime and name; `created (agent)` or `modified (verifier)` marks what the run changed and when, `image only` marks a file the run never touched (it lists but does not open on the kept tree).
|
|
98
|
-
- `files cat <trial-id> <path>` — the raw bytes of one file, to stdout. `--range bytes=a-b` (or `bytes=a-`, `bytes=-n`) reads a slice.
|
|
99
|
-
- `files search <trial-id> <text>` — every line matching the text, as `path:line: snippet`. `--path` narrows the search to one folder (the whole box takes seconds); `--regex` treats the text as a pattern; a `(truncated)` note means there was more.
|
|
100
|
-
- `files changes <trial-id>` — the files the run created, modified or removed, with the phase (`--phase agent|verifier`).
|
|
101
|
-
- `files archive <trial-id>` — a `.tar.gz` of one subtree (`--path`, default the whole tree) to stdout, or saved under `-o <dir>`. The kept tree's archive lists the files it does not carry in `MANIFEST.tsv`; while the box runs, a subtree too large to read out in time is refused (`feature_unsupported`).
|
|
102
|
-
|
|
103
|
-
<ParamField path="--source" type="live|capture">
|
|
104
|
-
Force the source. Without it, whichever the run has. A source that is not there is refused with `filesystem_state`.
|
|
105
|
-
</ParamField>
|
|
106
|
-
|
|
107
|
-
<ParamField path="-l, --limit" type="n">
|
|
108
|
-
Page size on `ls`, `search` and `changes` (`ls`, `changes` default 500; `search` default 200; max 1000).
|
|
109
|
-
</ParamField>
|
|
110
|
-
|
|
111
|
-
<ParamField path="--cursor" type="c">
|
|
112
|
-
Resume a paged `ls` or `changes` after the previous page's `next_cursor`.
|
|
113
|
-
</ParamField>
|
|
114
|
-
|
|
115
|
-
```bash
|
|
116
|
-
evolve trial files ls d1a10c4e-… /app/work
|
|
117
|
-
evolve trial files search d1a10c4e-… 'permission denied' --path /app/work
|
|
118
|
-
evolve trial files archive d1a10c4e-… --path /app/work -o archives/
|
|
119
|
-
```
|
|
120
|
-
|
|
121
|
-
## trial logs
|
|
122
|
-
|
|
123
|
-
```bash
|
|
124
|
-
evolve trial logs <trial-id> --stream <name> [--follow] [options]
|
|
125
|
-
```
|
|
126
|
-
|
|
127
|
-
The box's own streams: `agent` (the harness's stdout and stderr), `verifier` and `system` (the box's system log, only when the job asked for it with `--system-log`); `setup` and `metrics` are named but not recorded today, so their pages are empty and say why. One page by default; `--follow` keeps printing lines as they arrive while the box lives, and returns once it is gone.
|
|
128
|
-
|
|
129
|
-
<ParamField path="--stream" type="name" required>
|
|
130
|
-
`agent`, `verifier`, `system`, `setup` or `metrics`.
|
|
131
|
-
</ParamField>
|
|
132
|
-
|
|
133
|
-
<ParamField path="-f, --follow">
|
|
134
|
-
Keep printing new lines while the box lives.
|
|
135
|
-
</ParamField>
|
|
136
|
-
|
|
137
|
-
<ParamField path="-l, --limit" type="n" default="1000">
|
|
138
|
-
Lines per page, max 1000.
|
|
139
|
-
</ParamField>
|
|
140
|
-
|
|
141
|
-
<ParamField path="--cursor" type="seq">
|
|
142
|
-
Resume after this line.
|
|
143
|
-
</ParamField>
|
|
144
|
-
|
|
145
|
-
```bash
|
|
146
|
-
evolve trial logs d1a10c4e-… --stream agent --follow
|
|
147
|
-
```
|
|
148
|
-
|
|
149
|
-
## trial procs
|
|
150
|
-
|
|
151
|
-
```bash
|
|
152
|
-
evolve trial procs <trial-id>
|
|
153
|
-
```
|
|
154
|
-
|
|
155
|
-
The processes running in the box right now. Live only: once the box is gone the command is refused with `filesystem_state`.
|
|
156
|
-
|
|
157
|
-
## trial retry
|
|
158
|
-
|
|
159
|
-
```bash
|
|
160
|
-
evolve trial retry <trial-id>
|
|
161
|
-
```
|
|
162
|
-
|
|
163
|
-
Run one settled trial again. The result is a job.
|
|
164
|
-
|
|
165
|
-
## trial regrade
|
|
166
|
-
|
|
167
|
-
```bash
|
|
168
|
-
evolve trial regrade <trial-id>
|
|
169
|
-
```
|
|
170
|
-
|
|
171
|
-
Verifier-only re-run of one trial. The result is a job.
|
|
172
|
-
|
|
173
|
-
## trial stop
|
|
174
|
-
|
|
175
|
-
```bash
|
|
176
|
-
evolve trial stop <trial-id> [trial-id...]
|
|
177
|
-
```
|
|
178
|
-
|
|
179
|
-
Stop in-flight trials without cancelling their job.
|
|
180
|
-
|
|
181
|
-
<GlobalOptions />
|
|
@@ -1,36 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
title: "evolve upload"
|
|
3
|
-
description: "Upload a job folder as a finished job."
|
|
4
|
-
---
|
|
5
|
-
|
|
6
|
-
import GlobalOptions from '/snippets/global-options.mdx';
|
|
7
|
-
|
|
8
|
-
```bash
|
|
9
|
-
evolve upload <job_dir> [options]
|
|
10
|
-
```
|
|
11
|
-
|
|
12
|
-
Upload a job folder in the Harbor job layout, its `.tar.gz`, or a public URL of one, as a terminal job. What the folder and each trial's `result.json` must hold is on [Upload a job](/core-concepts/upload). The command follows the import to the job, prints the record with the task linkage (`linked N of M trials to <dataset>` and the not-linked tasks with their reasons), names any skipped trials, and ends with the analyze hint.
|
|
13
|
-
|
|
14
|
-
```bash
|
|
15
|
-
evolve upload ./job-2026-08-27__12-00-00 -d terminal-bench-4@4.0
|
|
16
|
-
```
|
|
17
|
-
|
|
18
|
-
## Options
|
|
19
|
-
|
|
20
|
-
<ParamField path="-d, --dataset" type="name[@version]">
|
|
21
|
-
Link the uploaded trials to a published dataset version by task name. Without it the trials link by their task hash on their own; this is the override, and the way to settle an ambiguous hash. See [upload](/core-concepts/upload#task-linkage).
|
|
22
|
-
</ParamField>
|
|
23
|
-
|
|
24
|
-
<ParamField path="--from" type="url">
|
|
25
|
-
A public https URL of the job archive (`.tar.gz`) that the server fetches itself, instead of `<job_dir>`. No local bytes.
|
|
26
|
-
</ParamField>
|
|
27
|
-
|
|
28
|
-
<ParamField path="--no-wait">
|
|
29
|
-
Return after the import is accepted, with the import id, instead of following it to the job. Re-attach with `evolve job import <id> --watch`.
|
|
30
|
-
</ParamField>
|
|
31
|
-
|
|
32
|
-
<GlobalOptions />
|
|
33
|
-
|
|
34
|
-
## Following an import
|
|
35
|
-
|
|
36
|
-
`evolve job imports` lists your uploads, and `evolve job import <import-id> --watch` follows one. See the [`job`](/cli-reference/job) group.
|
|
@@ -1,106 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
title: "Agents"
|
|
3
|
-
description: "The built-in harnesses, how to configure an arm, and how to register your own agent."
|
|
4
|
-
---
|
|
5
|
-
|
|
6
|
-
An agent is a harness plus a model. `-a` names the harness, `-m` the model it drives. Seven harnesses are built in.
|
|
7
|
-
|
|
8
|
-
| Name on `-a` | Harness |
|
|
9
|
-
| --- | --- |
|
|
10
|
-
| `claude` | Claude Code |
|
|
11
|
-
| `codex` | Codex |
|
|
12
|
-
| `gemini` | Gemini CLI |
|
|
13
|
-
| `qwen` | Qwen Code |
|
|
14
|
-
| `kimi` | Kimi Code |
|
|
15
|
-
| `opencode` | OpenCode |
|
|
16
|
-
| `droid` | Droid |
|
|
17
|
-
|
|
18
|
-
```bash
|
|
19
|
-
evolve run \
|
|
20
|
-
-d terminal-bench-4@4.0 \
|
|
21
|
-
-a claude \
|
|
22
|
-
-m fable \
|
|
23
|
-
-l 5 \
|
|
24
|
-
--watch
|
|
25
|
-
```
|
|
26
|
-
|
|
27
|
-
`-a <name[@version]>` can pin a harness version. Omitted, the latest is chosen once, when the job is created, so every trial installs the same version; it is recorded on every trial as `agent_info.version`. A pin that does not exist is refused with `agent_version_not_found`, and a pin on a registered agent is refused.
|
|
28
|
-
|
|
29
|
-
## Configure the arm
|
|
30
|
-
|
|
31
|
-
These flags apply to every arm of the job.
|
|
32
|
-
|
|
33
|
-
```bash
|
|
34
|
-
evolve run \
|
|
35
|
-
-d terminal-bench-4@4.0 \
|
|
36
|
-
-a codex \
|
|
37
|
-
-m gpt-6-astra \
|
|
38
|
-
-l 5 \
|
|
39
|
-
--effort high \
|
|
40
|
-
--preset no-internet \
|
|
41
|
-
--ak config=./codex-settings.json
|
|
42
|
-
```
|
|
43
|
-
|
|
44
|
-
`--effort <value>` sets the reasoning effort; see [models](/core-concepts/models#reasoning-effort). `--preset <name>` applies a named settings bundle: `no-internet` turns the vendor's server-side web tools off, `pinned-context` fixes the context window at 200000 tokens. `--ak config=<path|inline JSON>` becomes the harness's native settings file inside the sandbox; your document is the base, the platform's routing sits on top, and a preset wins where they disagree.
|
|
45
|
-
|
|
46
|
-
Only `claude` and `codex` take a config or a preset; the [capability document](/sdk-reference/meta) publishes `supports_config` and `presets` per harness. An effort, preset or config a harness cannot honor is refused when the job is created, never silently skipped, and so is a config key that touches billing, base URLs, routing or environment.
|
|
47
|
-
|
|
48
|
-
Environment for the agent comes from two places, a task's own `[environment.env]` table and a secret attached with `--secret`; a job cannot add a third. See [secrets](/core-concepts/secrets) and [tasks](/core-concepts/tasks#environment-variables).
|
|
49
|
-
|
|
50
|
-
## Register your own agent
|
|
51
|
-
|
|
52
|
-
Any CLI can run as an agent. Register it once, and its name works on `-a` like a built-in.
|
|
53
|
-
|
|
54
|
-
```bash
|
|
55
|
-
evolve agent add acme-cli --install-script ./install.sh --run "acme-cli --headless"
|
|
56
|
-
evolve run \
|
|
57
|
-
-d terminal-bench-4@4.0 \
|
|
58
|
-
-a acme-cli \
|
|
59
|
-
-m gpt-6-astra \
|
|
60
|
-
-l 5 \
|
|
61
|
-
--watch
|
|
62
|
-
```
|
|
63
|
-
|
|
64
|
-
`--install-script <path>` names a file; its contents are uploaded. `--dir <path>` uploads a local directory instead. `--run <command>` is required and runs with `sh -c`. `--ae KEY=VALUE` injects environment at run time and may repeat.
|
|
65
|
-
|
|
66
|
-
```bash
|
|
67
|
-
evolve agent list
|
|
68
|
-
evolve agent show acme-cli
|
|
69
|
-
evolve agent remove acme-cli
|
|
70
|
-
```
|
|
71
|
-
|
|
72
|
-
Removing an agent keeps the record of the past jobs that ran it. A registered agent belongs to its owner and to an organization: teammates may name it in their jobs and open it by name; a name outside your organizations reads as `agent_not_found`. The number you may hold is published as `agent_registration.max_per_user`; past it, `agent_limit_reached`.
|
|
73
|
-
|
|
74
|
-
The install script, or the uploaded directory, runs once with internet and no secrets: fetch only public sources and leave the executables in `$PREFIX/bin`. To change an agent without a gap, use the SDK's `upsert`; the CLI removes the registration and adds it again.
|
|
75
|
-
|
|
76
|
-
### The run contract
|
|
77
|
-
|
|
78
|
-
Your `run_command` runs with `sh -c` at the task's working directory, and can rely on exactly this:
|
|
79
|
-
|
|
80
|
-
- The instruction arrives twice: on stdin, and at the path in `$EVOLVE_INSTRUCTION_FILE`.
|
|
81
|
-
- `$EVOLVE_GATEWAY_BASE_URL` is an OpenAI-compatible base URL that already ends in `/v1`, and `$EVOLVE_GATEWAY_API_KEY` is its credential. The same two values are also `$OPENAI_BASE_URL` and `$OPENAI_API_KEY`, so a CLI that reads its endpoint from the environment works unchanged.
|
|
82
|
-
- `$EVOLVE_MODEL` is the `model_name` of the arm.
|
|
83
|
-
- Your declared env may not override those six keys, and may not look like a credential; both are refused at registration with `agent_invalid_env`.
|
|
84
|
-
|
|
85
|
-
A CLI that routes through a config file must write that file inside `run_command`, from the contract values. `codex` is the worked example:
|
|
86
|
-
|
|
87
|
-
```bash
|
|
88
|
-
mkdir -p ~/.codex && cat > ~/.codex/config.toml <<EOF
|
|
89
|
-
model_provider = "evolve"
|
|
90
|
-
[model_providers.evolve]
|
|
91
|
-
name = "evolve"
|
|
92
|
-
base_url = "$EVOLVE_GATEWAY_BASE_URL"
|
|
93
|
-
env_key = "EVOLVE_GATEWAY_API_KEY"
|
|
94
|
-
wire_api = "responses"
|
|
95
|
-
EOF
|
|
96
|
-
codex login --with-api-key <<< "$EVOLVE_GATEWAY_API_KEY"
|
|
97
|
-
codex exec --skip-git-repo-check -
|
|
98
|
-
```
|
|
99
|
-
|
|
100
|
-
A CLI that ignores `OPENAI_BASE_URL` without this reaches for its vendor's endpoint, finds the box sealed, and spends the agent budget failing to connect. The spend cap is airtight only under `no-network`; see [models](/core-concepts/models).
|
|
101
|
-
|
|
102
|
-
Against a built-in, a registered agent has no live trace (there is no parser for an unknown CLI) and no `reasoning_effort` (put the flag in `run_command`); everything else is recorded the same way.
|
|
103
|
-
|
|
104
|
-
<Card title="agent reference" icon="terminal" href="/cli-reference/agent">
|
|
105
|
-
Every flag of `evolve agent`.
|
|
106
|
-
</Card>
|
|
@@ -1,86 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
title: "Analyze"
|
|
3
|
-
description: "Judge every trial's trace against a rubric, on the platform."
|
|
4
|
-
---
|
|
5
|
-
|
|
6
|
-
An analysis reads a settled trial's whole record, the trace and the task, and rules on every criterion of a rubric. It runs on the platform, one analyzer run per trial. Ask for it on a finished job:
|
|
7
|
-
|
|
8
|
-
```bash
|
|
9
|
-
evolve analyze <job-id> --watch
|
|
10
|
-
```
|
|
11
|
-
|
|
12
|
-
Or arm it when you start the job, so each trial is analyzed the moment it settles:
|
|
13
|
-
|
|
14
|
-
```bash
|
|
15
|
-
evolve run \
|
|
16
|
-
-d terminal-bench-4@4.0 \
|
|
17
|
-
-a codex \
|
|
18
|
-
-m gpt-6-astra \
|
|
19
|
-
--analyze \
|
|
20
|
-
-l 5 \
|
|
21
|
-
--watch
|
|
22
|
-
```
|
|
23
|
-
|
|
24
|
-
## Rubrics and prompts
|
|
25
|
-
|
|
26
|
-
The default rubric has seven criteria, `score_is_earned` first. Bring your own as TOML, YAML or JSON in the shape below and, if you want, your own prompt for the analyzer.
|
|
27
|
-
|
|
28
|
-
```toml rubric.toml
|
|
29
|
-
[[criteria]]
|
|
30
|
-
name = "score_is_earned"
|
|
31
|
-
description = "The reward reflects work the agent actually did."
|
|
32
|
-
guidance = "Read the verifier log and the trajectory side by side."
|
|
33
|
-
```
|
|
34
|
-
|
|
35
|
-
```bash
|
|
36
|
-
evolve analyze <job-id> \
|
|
37
|
-
-r rubric.toml \
|
|
38
|
-
-p prompt.txt \
|
|
39
|
-
--failing \
|
|
40
|
-
-l 20 \
|
|
41
|
-
-n 2 \
|
|
42
|
-
--watch
|
|
43
|
-
```
|
|
44
|
-
|
|
45
|
-
`-r` is the rubric file, `-p` the prompt file. `--passing` and `--failing` restrict the batch of analyses to trials with reward 1.0, or below it; both together is refused. `-t` names one trial to analyze, repeatable, applied with the filter and before `-l`. `-l` caps the number of trials, `-n` how many analyses run at once, beneath the organization's `max_concurrent_analyses` (fleet default 60). `-m` picks the analyzer's model, `--effort` its reasoning effort, `-e` its sandbox provider. `--show-defaults` prints the defaults a bare `evolve analyze` runs under and exits.
|
|
46
|
-
|
|
47
|
-
A result that misses or invents a criterion is a failed analysis, never a partial pass. A rubric with unknown keys, empty or duplicate criteria, or a length bound exceeded is refused with `invalid_rubric`; an off-roster model, an unknown effort or provider, or a `-t` id that is not a trial of the job with `invalid_input`; a job with no analyzable trial, or a `--passing`, `--failing` or `-t` selection that leaves none, with `no_analyzable_trials`; a job still running with `job_not_terminal`.
|
|
48
|
-
|
|
49
|
-
On `evolve run` the same knobs are `--analyze-model`, `--analyze-rubric`, `--analyze-prompt`, `--analyze-effort` and `--analyze-provider`. Each of them implies `--analyze`.
|
|
50
|
-
|
|
51
|
-
Running `evolve analyze` again on the same job, with another rubric or model, is the re-analysis path: one wave at a time (`analysis_already_running` meanwhile), and each trial then serves its newest analysis while earlier ones stay under their own ids. Cancelled trials are never analyzed.
|
|
52
|
-
|
|
53
|
-
## The result
|
|
54
|
-
|
|
55
|
-
Each analysis produces a short summary of what happened in the trial, and one entry per criterion:
|
|
56
|
-
|
|
57
|
-
- `outcome`: `pass`, `fail`, `not_applicable` or `unknown`
|
|
58
|
-
- `explanation`: the analyzer's reasoning, in plain words
|
|
59
|
-
- `evidence`: what in the trial record supports it
|
|
60
|
-
|
|
61
|
-
Under the default rubric, the platform derives one label from the outcomes:
|
|
62
|
-
|
|
63
|
-
- `flagged`: a fail on `score_is_earned`, `score_is_correct`, `task_was_fair` or `report_is_truthful`
|
|
64
|
-
- `env_fault`: otherwise, a fail on `environment_worked`
|
|
65
|
-
- `unclear`: otherwise, an unknown on any of those five, or `not_applicable` on `score_is_earned` or `score_is_correct`
|
|
66
|
-
- `clean`: everything else
|
|
67
|
-
|
|
68
|
-
A custom rubric carries its per-criterion outcomes and no label.
|
|
69
|
-
|
|
70
|
-
The analyzer's spend is shown apart from the trial's: `estimated_cost_usd` on the analysis, `stats.analysis.cost_usd` on the job. A failed analysis carries `failure` with a `phase` and a `message`; a run that produces no valid result is retried once.
|
|
71
|
-
|
|
72
|
-
## Read it back
|
|
73
|
-
|
|
74
|
-
```bash
|
|
75
|
-
evolve analysis list --job <job-id> --status failed
|
|
76
|
-
evolve analysis show <analysis-id>
|
|
77
|
-
evolve analysis trace <analysis-id>
|
|
78
|
-
evolve analysis download <analysis-id> -o analyses/
|
|
79
|
-
evolve analysis download <analysis-id> --stream analysis
|
|
80
|
-
```
|
|
81
|
-
|
|
82
|
-
`analysis show` prints the verdict document. Every analysis verb also accepts a trial id, which names that trial's latest analysis. `analysis trace` prints the analyzer's own transcript. `analysis download` saves the run as a trial folder, or streams one artifact: `analysis`, `trace-parsed`, `trace-stdout`, `trace-stderr` or `agent-home`.
|
|
83
|
-
|
|
84
|
-
<Card title="analyze reference" icon="terminal" href="/cli-reference/analyze">
|
|
85
|
-
Every flag of `evolve analyze` and `evolve analysis`.
|
|
86
|
-
</Card>
|
|
@@ -1,64 +0,0 @@
|
|
|
1
|
-
---
|
|
2
|
-
title: "Check"
|
|
3
|
-
description: "Check the quality of tasks against a rubric before you spend a job on them."
|
|
4
|
-
---
|
|
5
|
-
|
|
6
|
-
A check reads a task and, when it can, runs its environment, its reference solution and its verifier, then rules on every criterion of a rubric. Point it at a task directory, or at a directory of tasks:
|
|
7
|
-
|
|
8
|
-
```bash
|
|
9
|
-
evolve check ./tasks \
|
|
10
|
-
-i 'abs-*' \
|
|
11
|
-
-l 5 \
|
|
12
|
-
--watch
|
|
13
|
-
```
|
|
14
|
-
|
|
15
|
-
Or at a published dataset version. Nothing is uploaded; the checker reads the retained task package.
|
|
16
|
-
|
|
17
|
-
```bash
|
|
18
|
-
evolve check -d my-swe@1.0 --watch
|
|
19
|
-
```
|
|
20
|
-
|
|
21
|
-
When the path is a task directory, that task is checked; otherwise every top-level directory that is one, in sorted order. `-i` and `-x` are globs over task names and `-l` caps the count, in that order; a selection that leaves nothing is refused with `no_checkable_tasks`. There is no task-count ceiling.
|
|
22
|
-
|
|
23
|
-
`-r` is a rubric file, `-p` a prompt file. `-m` and `--effort` set the checker's model and effort, `-e` its sandbox provider, `-n` how many checks run at once, beneath the organization's `max_concurrent_analyses`.
|
|
24
|
-
|
|
25
|
-
The default rubric has eleven criteria: `instruction_is_sufficient`, `tests_match_the_instruction`, `verifier_is_correct`, `no_answer_leakage`, `grader_is_out_of_reach`, `reference_solution_is_valid`, `verifier_rejects_non_solutions`, `environment_builds_and_runs`, `verification_is_stable`, `limits_allow_the_task` and `task_is_solvable`. The checker's default model is the analyzer's; a custom rubric, model, prompt and effort follow the same rules and refusals as [analyze](/core-concepts/analyze).
|
|
26
|
-
|
|
27
|
-
## The result
|
|
28
|
-
|
|
29
|
-
Each task gets one entry per criterion, with an `outcome` (`pass`, `fail`, `not_applicable` or `unknown`), an `explanation` and `evidence`. Under the default rubric, one label is derived:
|
|
30
|
-
|
|
31
|
-
- `has_a_problem`: a fail on any criterion
|
|
32
|
-
- `unclear`: otherwise, an unknown on any of the six file-based criteria
|
|
33
|
-
- `no_problem_found`: everything else
|
|
34
|
-
|
|
35
|
-
`executed` says whether the checker actually ran the task's environment, so a reading-only `no_problem_found` is never mistaken for a run. A check never fails as a whole: refusals happen at accept, and a task's failure is on that task. A check is readable by its creator, by every member of the organization and by whoever it was shared with; anyone else reads `check_not_found`.
|
|
36
|
-
|
|
37
|
-
## Read it back
|
|
38
|
-
|
|
39
|
-
```bash
|
|
40
|
-
evolve check list --status running
|
|
41
|
-
evolve check show <check-id>
|
|
42
|
-
evolve check trace <task-check-id> --since 200
|
|
43
|
-
evolve check download <check-id> -o checks/
|
|
44
|
-
evolve check download <task-check-id> --stream task-check
|
|
45
|
-
```
|
|
46
|
-
|
|
47
|
-
`check show` prints the check with one result per task. `check trace` prints the checker's transcript for one task check. `check download` saves the whole check as a check folder by its check id, one task's folder by its task check id, or streams one artifact of a task check: `task-check`, `trace-parsed`, `trace-stdout`, `trace-stderr` or `agent-home`.
|
|
48
|
-
|
|
49
|
-
## Share
|
|
50
|
-
|
|
51
|
-
```bash
|
|
52
|
-
evolve check share <check-id> --link
|
|
53
|
-
evolve check share <check-id> --email alice@example.org
|
|
54
|
-
evolve check shares <check-id>
|
|
55
|
-
evolve check unshare <check-id> --link
|
|
56
|
-
```
|
|
57
|
-
|
|
58
|
-
A check you created shares the two ways a [job](/core-concepts/jobs#share) does, and both may be on at once. `--link` turns on an unlisted link: anyone holding it reads the check's report and downloads its check folder, without signing in. `check shares` prints the link again whenever you need it, and `check unshare --link` kills it at once. `--email` shares the check with a person by address: they get an email with a link and read the check after signing in with that address (a person without an account gets a sign-up link in the same email). A check shared with you appears under `evolve check list --scope shared`.
|
|
59
|
-
|
|
60
|
-
Sharing is read-only: whoever holds the link or the email share reads the check and can never stop one of its task checks; that stays with the creator. Only the creator may share or unshare; another member of the organization is refused with `org_forbidden`. A check's `visibility` field says `PRIVATE` or `LINK`. The dashboard side, and what a link shows, are on [Sharing](/core-concepts/sharing).
|
|
61
|
-
|
|
62
|
-
<Card title="check reference" icon="terminal" href="/cli-reference/check">
|
|
63
|
-
Every flag of `evolve check`.
|
|
64
|
-
</Card>
|