@evolvingmachines/evolve 0.0.65 → 0.0.66

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (152) hide show
  1. package/dist/{chunk-DK2MGUPB.js → chunk-RFLMRPGK.js} +5 -5
  2. package/dist/cli/index.cjs +39 -39
  3. package/dist/cli/index.d.cts +3 -3
  4. package/dist/cli/index.d.ts +3 -3
  5. package/dist/cli/index.js +30 -30
  6. package/dist/index.cjs +1 -1
  7. package/dist/index.d.cts +4 -4
  8. package/dist/index.d.ts +4 -4
  9. package/dist/index.js +1 -1
  10. package/dist/{types-B_qmzlMk.d.cts → types-84M4J_Vm.d.cts} +11 -44
  11. package/dist/{types-B_qmzlMk.d.ts → types-84M4J_Vm.d.ts} +11 -44
  12. package/package.json +4 -4
  13. package/skills/create-adapter/SKILL.md +71 -26
  14. package/skills/create-task/SKILL.md +124 -66
  15. package/skills/evolve/SKILL.md +37 -20
  16. package/skills/evolve-evals/SKILL.md +142 -69
  17. package/skills/evolve-evals/references/cli-reference/agent.md +76 -0
  18. package/skills/evolve-evals/references/cli-reference/analysis.md +78 -0
  19. package/skills/evolve-evals/references/cli-reference/analyze.md +95 -0
  20. package/skills/evolve-evals/references/cli-reference/auth.md +96 -0
  21. package/skills/evolve-evals/references/cli-reference/check.md +156 -0
  22. package/skills/evolve-evals/references/cli-reference/dataset.md +154 -0
  23. package/skills/evolve-evals/references/cli-reference/filesystem.md +168 -0
  24. package/skills/evolve-evals/references/cli-reference/index.md +149 -0
  25. package/skills/evolve-evals/references/cli-reference/job.md +198 -0
  26. package/skills/evolve-evals/references/cli-reference/run.md +187 -0
  27. package/skills/evolve-evals/references/cli-reference/secrets.md +77 -0
  28. package/skills/evolve-evals/references/cli-reference/session.md +56 -0
  29. package/skills/evolve-evals/references/cli-reference/skill.md +52 -0
  30. package/skills/evolve-evals/references/cli-reference/skills.md +85 -0
  31. package/skills/evolve-evals/references/cli-reference/trial.md +115 -0
  32. package/skills/evolve-evals/references/cli-reference/upload.md +72 -0
  33. package/skills/evolve-evals/references/core-concepts/agents.md +96 -0
  34. package/skills/evolve-evals/references/core-concepts/analyze.md +143 -0
  35. package/skills/evolve-evals/references/core-concepts/check.md +116 -0
  36. package/skills/evolve-evals/references/core-concepts/compatibility.md +39 -0
  37. package/skills/evolve-evals/references/core-concepts/compose.md +103 -0
  38. package/skills/evolve-evals/references/core-concepts/datasets.md +234 -0
  39. package/skills/evolve-evals/references/core-concepts/index.md +52 -0
  40. package/skills/evolve-evals/references/core-concepts/jobs.md +223 -0
  41. package/skills/evolve-evals/references/core-concepts/models.md +73 -0
  42. package/skills/evolve-evals/references/core-concepts/multi-step.md +93 -0
  43. package/skills/evolve-evals/references/core-concepts/rewardkit.md +341 -0
  44. package/skills/evolve-evals/references/core-concepts/sandboxes.md +73 -0
  45. package/skills/evolve-evals/references/core-concepts/secrets.md +86 -0
  46. package/skills/evolve-evals/references/core-concepts/sharing.md +81 -0
  47. package/skills/evolve-evals/references/core-concepts/skills.md +100 -0
  48. package/skills/evolve-evals/references/core-concepts/task-config.md +169 -0
  49. package/skills/evolve-evals/references/core-concepts/task-environment.md +127 -0
  50. package/skills/evolve-evals/references/core-concepts/task-reference.md +191 -0
  51. package/skills/evolve-evals/references/core-concepts/task-verifiers.md +209 -0
  52. package/skills/evolve-evals/references/core-concepts/tasks.md +90 -0
  53. package/skills/evolve-evals/references/core-concepts/teams.md +128 -0
  54. package/skills/evolve-evals/references/core-concepts/trial-outputs.md +225 -0
  55. package/skills/evolve-evals/references/core-concepts/trials.md +129 -0
  56. package/skills/evolve-evals/references/core-concepts/upload-sdk-session.md +240 -0
  57. package/skills/evolve-evals/references/core-concepts/upload.md +166 -0
  58. package/skills/evolve-evals/references/dashboard/checks.md +54 -0
  59. package/skills/evolve-evals/references/dashboard/datasets.md +65 -0
  60. package/skills/evolve-evals/references/dashboard/jobs.md +106 -0
  61. package/skills/evolve-evals/references/dashboard/traces.md +54 -0
  62. package/skills/evolve-evals/references/dashboard/trial-viewer.md +77 -0
  63. package/skills/evolve-evals/references/getting-started/first-task.md +228 -0
  64. package/skills/evolve-evals/references/getting-started/installation.md +79 -0
  65. package/skills/evolve-evals/references/getting-started/quick-start.md +100 -0
  66. package/skills/evolve-evals/references/index.md +58 -0
  67. package/skills/evolve-evals/references/sdk/http.md +110 -0
  68. package/skills/evolve-evals/references/sdk/python.md +108 -0
  69. package/skills/evolve-evals/references/sdk/typescript.md +104 -0
  70. package/skills/evolve-evals/references/sdk-reference/agents.md +75 -0
  71. package/skills/evolve-evals/references/sdk-reference/analyses.md +124 -0
  72. package/skills/evolve-evals/references/sdk-reference/auth.md +171 -0
  73. package/skills/evolve-evals/references/sdk-reference/checks.md +145 -0
  74. package/skills/evolve-evals/references/sdk-reference/datasets.md +170 -0
  75. package/skills/evolve-evals/references/sdk-reference/error-codes.md +231 -0
  76. package/skills/evolve-evals/references/sdk-reference/errors.md +98 -0
  77. package/skills/evolve-evals/references/sdk-reference/filesystem.md +166 -0
  78. package/skills/evolve-evals/references/sdk-reference/index.md +161 -0
  79. package/skills/evolve-evals/references/sdk-reference/jobs.md +250 -0
  80. package/skills/evolve-evals/references/sdk-reference/meta.md +54 -0
  81. package/skills/evolve-evals/references/sdk-reference/methods/agents.md +154 -0
  82. package/skills/evolve-evals/references/sdk-reference/methods/analyses.md +211 -0
  83. package/skills/evolve-evals/references/sdk-reference/methods/auth.md +257 -0
  84. package/skills/evolve-evals/references/sdk-reference/methods/checks.md +460 -0
  85. package/skills/evolve-evals/references/sdk-reference/methods/datasets.md +687 -0
  86. package/skills/evolve-evals/references/sdk-reference/methods/filesystem.md +527 -0
  87. package/skills/evolve-evals/references/sdk-reference/methods/jobs.md +1021 -0
  88. package/skills/evolve-evals/references/sdk-reference/methods/meta.md +209 -0
  89. package/skills/evolve-evals/references/sdk-reference/methods/secrets.md +157 -0
  90. package/skills/evolve-evals/references/sdk-reference/methods/skills.md +125 -0
  91. package/skills/evolve-evals/references/sdk-reference/methods/trials.md +371 -0
  92. package/skills/evolve-evals/references/sdk-reference/secrets.md +104 -0
  93. package/skills/evolve-evals/references/sdk-reference/skills.md +66 -0
  94. package/skills/evolve-evals/references/sdk-reference/trials.md +164 -0
  95. package/skills/evolve-evals/references/sdk-reference/types.md +1249 -0
  96. package/skills/publish/SKILL.md +103 -79
  97. package/skills/rewardkit/SKILL.md +78 -23
  98. package/spec/openapi.yaml +19 -74
  99. package/skills/evolve-evals/references/cli-reference/agent.mdx +0 -79
  100. package/skills/evolve-evals/references/cli-reference/analysis.mdx +0 -160
  101. package/skills/evolve-evals/references/cli-reference/analyze.mdx +0 -84
  102. package/skills/evolve-evals/references/cli-reference/auth.mdx +0 -133
  103. package/skills/evolve-evals/references/cli-reference/check.mdx +0 -263
  104. package/skills/evolve-evals/references/cli-reference/dataset.mdx +0 -151
  105. package/skills/evolve-evals/references/cli-reference/index.mdx +0 -108
  106. package/skills/evolve-evals/references/cli-reference/job.mdx +0 -278
  107. package/skills/evolve-evals/references/cli-reference/run.mdx +0 -204
  108. package/skills/evolve-evals/references/cli-reference/secrets.mdx +0 -95
  109. package/skills/evolve-evals/references/cli-reference/skill.mdx +0 -61
  110. package/skills/evolve-evals/references/cli-reference/skills.mdx +0 -81
  111. package/skills/evolve-evals/references/cli-reference/trial.mdx +0 -181
  112. package/skills/evolve-evals/references/cli-reference/upload.mdx +0 -36
  113. package/skills/evolve-evals/references/core-concepts/agents.mdx +0 -106
  114. package/skills/evolve-evals/references/core-concepts/analyze.mdx +0 -86
  115. package/skills/evolve-evals/references/core-concepts/check.mdx +0 -64
  116. package/skills/evolve-evals/references/core-concepts/datasets.mdx +0 -138
  117. package/skills/evolve-evals/references/core-concepts/index.mdx +0 -42
  118. package/skills/evolve-evals/references/core-concepts/jobs.mdx +0 -151
  119. package/skills/evolve-evals/references/core-concepts/models.mdx +0 -68
  120. package/skills/evolve-evals/references/core-concepts/sandboxes.mdx +0 -52
  121. package/skills/evolve-evals/references/core-concepts/secrets.mdx +0 -41
  122. package/skills/evolve-evals/references/core-concepts/sharing.mdx +0 -35
  123. package/skills/evolve-evals/references/core-concepts/skills.mdx +0 -42
  124. package/skills/evolve-evals/references/core-concepts/tasks.mdx +0 -216
  125. package/skills/evolve-evals/references/core-concepts/teams.mdx +0 -36
  126. package/skills/evolve-evals/references/core-concepts/trial-outputs.mdx +0 -117
  127. package/skills/evolve-evals/references/core-concepts/trials.mdx +0 -103
  128. package/skills/evolve-evals/references/core-concepts/upload.mdx +0 -93
  129. package/skills/evolve-evals/references/dashboard/checks.mdx +0 -21
  130. package/skills/evolve-evals/references/dashboard/datasets.mdx +0 -24
  131. package/skills/evolve-evals/references/dashboard/jobs.mdx +0 -65
  132. package/skills/evolve-evals/references/dashboard/traces.mdx +0 -17
  133. package/skills/evolve-evals/references/dashboard/trial-viewer.mdx +0 -30
  134. package/skills/evolve-evals/references/getting-started/installation.mdx +0 -56
  135. package/skills/evolve-evals/references/getting-started/quick-start.mdx +0 -94
  136. package/skills/evolve-evals/references/index.mdx +0 -48
  137. package/skills/evolve-evals/references/sdk/python.mdx +0 -161
  138. package/skills/evolve-evals/references/sdk/typescript.mdx +0 -163
  139. package/skills/evolve-evals/references/sdk-reference/agents.mdx +0 -108
  140. package/skills/evolve-evals/references/sdk-reference/analyses.mdx +0 -222
  141. package/skills/evolve-evals/references/sdk-reference/auth.mdx +0 -116
  142. package/skills/evolve-evals/references/sdk-reference/checks.mdx +0 -300
  143. package/skills/evolve-evals/references/sdk-reference/datasets.mdx +0 -346
  144. package/skills/evolve-evals/references/sdk-reference/errors.mdx +0 -210
  145. package/skills/evolve-evals/references/sdk-reference/index.mdx +0 -60
  146. package/skills/evolve-evals/references/sdk-reference/jobs.mdx +0 -822
  147. package/skills/evolve-evals/references/sdk-reference/meta.mdx +0 -52
  148. package/skills/evolve-evals/references/sdk-reference/skills.mdx +0 -69
  149. package/skills/evolve-evals/references/sdk-reference/trials.mdx +0 -302
  150. package/skills/evolve-evals/references/sdk-reference/types.mdx +0 -274
  151. package/skills/evolve-evals/references/snippets/global-options.mdx +0 -13
  152. package/skills/evolve-evals/references/snippets/list-flags.mdx +0 -23
@@ -1,81 +0,0 @@
1
- ---
2
- title: "evolve skills"
3
- description: "The skills the CLI serves to coding agents."
4
- ---
5
-
6
- import GlobalOptions from '/snippets/global-options.mdx';
7
-
8
- ```bash
9
- evolve skills <command> [options]
10
- ```
11
-
12
- The skills ship with the CLI and match its version. `skills get evals` is the index of these docs and `skills get evals <page>` one page of them. The `evolve` pointer skill that `skills install` writes is served but never listed. The SDK skill is not served; it installs from the repository with `npx skills add evolving-machines-lab/evolve --skill evolve-agents`. `EVOLVE_SKILLS_DIR` names another skills directory to serve.
13
-
14
- ## skills list
15
-
16
- ```bash
17
- evolve skills list
18
- ```
19
-
20
- List the skills the installed version serves, one line each. Bare `evolve skills` does the same.
21
-
22
- ## skills get
23
-
24
- ```bash
25
- evolve skills get <name> [name...]
26
- evolve skills get <name> <page>
27
- ```
28
-
29
- Print one or more skills, or one page. A second word that is not a skill name is a page of the first skill: its path under `references/` without the suffix, as this site spells it (`core-concepts/tasks`).
30
-
31
- <ParamField path="--full">
32
- Also print every file under `references/` and `templates/`, each behind a `--- <path> ---` line.
33
- </ParamField>
34
-
35
- <ParamField path="--all">
36
- Every skill, instead of naming them.
37
- </ParamField>
38
-
39
- ```bash
40
- evolve skills get evals
41
- evolve skills get evals core-concepts/tasks
42
- evolve skills get create-task --full
43
- ```
44
-
45
- ## skills path
46
-
47
- ```bash
48
- evolve skills path [name]
49
- ```
50
-
51
- Print the skills directory, or one skill's folder.
52
-
53
- ## skills install
54
-
55
- ```bash
56
- evolve skills install [options]
57
- ```
58
-
59
- Install the `evolve` pointer skill for your agents. Each target gets `<folder>/evolve/SKILL.md`; the path written is printed. A file already there with different content is left alone unless `--force`.
60
-
61
- <ParamField path="--target" type="claude | codex | cursor | copilot | gemini | opencode | agents | all">
62
- Which agent home: `~/.claude`, `~/.codex`, `~/.cursor`, `~/.copilot`, `~/.gemini`, `~/.config/opencode` or `~/.agents`, each under `skills/`. Defaults to `all`.
63
- </ParamField>
64
-
65
- <ParamField path="--path" type="dir">
66
- A skills directory of your own instead of `--target`.
67
- </ParamField>
68
-
69
- <ParamField path="--force">
70
- Overwrite an installed `SKILL.md` whose content differs.
71
- </ParamField>
72
-
73
- ```bash
74
- evolve skills install
75
- evolve skills install --target claude
76
- evolve skills install --path ./skills
77
- ```
78
-
79
- With the [skills CLI](https://github.com/vercel-labs/skills) instead, the same pointer installs from the repository: `npx skills add evolving-machines-lab/evolve`.
80
-
81
- <GlobalOptions />
@@ -1,181 +0,0 @@
1
- ---
2
- title: "evolve trial"
3
- description: "Inspect, download, and act on single trials."
4
- ---
5
-
6
- import GlobalOptions from '/snippets/global-options.mdx';
7
-
8
- ```bash
9
- evolve trial <command> [options]
10
- ```
11
-
12
- Every command below takes the [global options](/cli-reference#global-options).
13
-
14
- ## trial show
15
-
16
- ```bash
17
- evolve trial show <trial-id>
18
- ```
19
-
20
- Show one trial in full detail: its status, reward, `tokens`, `spent` and `spent (judge)` rows, `gpu compute (est.)` on a GPU trial, the exception when it failed, and its latest analysis with the verdicts, the summary and the typed failure. An uploaded trial prints its provider as `ported` and its reported figures apart from the metered ones.
21
-
22
- The run's own files are under [`trial files`](#trial-files); the platform's stored artifacts (session files, the verifier log) are SDK-only, `trials().files()` and `file()`.
23
-
24
- ## trial trace
25
-
26
- ```bash
27
- evolve trial trace <trial-id> [options]
28
- ```
29
-
30
- Print a trial's parsed trace, filtered server-side.
31
-
32
- <ParamField path="--type" type="event-type">
33
- Only events of exactly this type. At most 100 characters.
34
- </ParamField>
35
-
36
- <ParamField path="--grep" type="pattern">
37
- Only events matching this case-insensitive regex over type and content. A plain string is a plain substring. At most 512 characters; an invalid pattern is refused by the server with `invalid_input`.
38
- </ParamField>
39
-
40
- <ParamField path="--tail" type="n">
41
- Only the last N matching events, 1 to 10000, then paged oldest first.
42
- </ParamField>
43
-
44
- <ParamField path="--cursor" type="seq">
45
- Resume after this sequence number. The filters compose with it.
46
- </ParamField>
47
-
48
- <ParamField path="-l, --limit" type="n" default="200">
49
- Events per page fetch, not a total: the command drains the filtered trace. Max 1000.
50
- </ParamField>
51
-
52
- ```bash
53
- evolve trial trace d1a10c4e-… --grep 'permission denied' --tail 50
54
- ```
55
-
56
- ## trial download
57
-
58
- ```bash
59
- evolve trial download <trial-id> [options]
60
- ```
61
-
62
- Save a trial as its trial tree plus `evolve.json`, or stream one artifact to stdout. The two modes are exclusive.
63
-
64
- <ParamField path="-o, --output-dir" type="dir" default="trials/">
65
- Directory to save under. Files land in `<dir>/<trial-id>/`.
66
- </ParamField>
67
-
68
- <ParamField path="--overwrite">
69
- Replace an existing `<dir>/<trial-id>/`.
70
- </ParamField>
71
-
72
- <ParamField path="--stream" type="artifact">
73
- Print one artifact to stdout instead of saving: `trace-parsed`, `verifier`, `trace-stdout`, `trace-stderr`, `trace-atif` (the ATIF trajectory), `trajectory` (reserved: the harness-native session file), `agent-home`, or `filesystem` (the run's file system as one `.tar.gz`, raw bytes — pipe it to a file; the same archive as `trial files archive`).
74
- </ParamField>
75
-
76
- <ParamField path="--cursor" type="seq">
77
- With `--stream trace-parsed`: resume after this sequence number.
78
- </ParamField>
79
-
80
- <ParamField path="--limit" type="n">
81
- With `--stream trace-parsed`: max events per page.
82
- </ParamField>
83
-
84
- ```bash
85
- evolve trial download d1a10c4e-… --stream trace-stdout
86
- ```
87
-
88
- ## trial files
89
-
90
- ```bash
91
- evolve trial files <status|ls|cat|search|changes|archive> <trial-id> [path] [options]
92
- ```
93
-
94
- The run's file system: while the box runs, `files` reads it as it changes; after the run, the same commands read the kept tree. `files status` says which you get.
95
-
96
- - `files status <trial-id>` — `live`, `captured`, `capturing` (the kept tree is still being written) or `none`; the box, how change events arrive (`watcher`), and the capture record (`ready`, `incomplete` with what was left out, or `failed`).
97
- - `files ls <trial-id> [path]` — one folder, sorted by name (default `/`). Each row shows the type, mode, size, mtime and name; `created (agent)` or `modified (verifier)` marks what the run changed and when, `image only` marks a file the run never touched (it lists but does not open on the kept tree).
98
- - `files cat <trial-id> <path>` — the raw bytes of one file, to stdout. `--range bytes=a-b` (or `bytes=a-`, `bytes=-n`) reads a slice.
99
- - `files search <trial-id> <text>` — every line matching the text, as `path:line: snippet`. `--path` narrows the search to one folder (the whole box takes seconds); `--regex` treats the text as a pattern; a `(truncated)` note means there was more.
100
- - `files changes <trial-id>` — the files the run created, modified or removed, with the phase (`--phase agent|verifier`).
101
- - `files archive <trial-id>` — a `.tar.gz` of one subtree (`--path`, default the whole tree) to stdout, or saved under `-o <dir>`. The kept tree's archive lists the files it does not carry in `MANIFEST.tsv`; while the box runs, a subtree too large to read out in time is refused (`feature_unsupported`).
102
-
103
- <ParamField path="--source" type="live|capture">
104
- Force the source. Without it, whichever the run has. A source that is not there is refused with `filesystem_state`.
105
- </ParamField>
106
-
107
- <ParamField path="-l, --limit" type="n">
108
- Page size on `ls`, `search` and `changes` (`ls`, `changes` default 500; `search` default 200; max 1000).
109
- </ParamField>
110
-
111
- <ParamField path="--cursor" type="c">
112
- Resume a paged `ls` or `changes` after the previous page's `next_cursor`.
113
- </ParamField>
114
-
115
- ```bash
116
- evolve trial files ls d1a10c4e-… /app/work
117
- evolve trial files search d1a10c4e-… 'permission denied' --path /app/work
118
- evolve trial files archive d1a10c4e-… --path /app/work -o archives/
119
- ```
120
-
121
- ## trial logs
122
-
123
- ```bash
124
- evolve trial logs <trial-id> --stream <name> [--follow] [options]
125
- ```
126
-
127
- The box's own streams: `agent` (the harness's stdout and stderr), `verifier` and `system` (the box's system log, only when the job asked for it with `--system-log`); `setup` and `metrics` are named but not recorded today, so their pages are empty and say why. One page by default; `--follow` keeps printing lines as they arrive while the box lives, and returns once it is gone.
128
-
129
- <ParamField path="--stream" type="name" required>
130
- `agent`, `verifier`, `system`, `setup` or `metrics`.
131
- </ParamField>
132
-
133
- <ParamField path="-f, --follow">
134
- Keep printing new lines while the box lives.
135
- </ParamField>
136
-
137
- <ParamField path="-l, --limit" type="n" default="1000">
138
- Lines per page, max 1000.
139
- </ParamField>
140
-
141
- <ParamField path="--cursor" type="seq">
142
- Resume after this line.
143
- </ParamField>
144
-
145
- ```bash
146
- evolve trial logs d1a10c4e-… --stream agent --follow
147
- ```
148
-
149
- ## trial procs
150
-
151
- ```bash
152
- evolve trial procs <trial-id>
153
- ```
154
-
155
- The processes running in the box right now. Live only: once the box is gone the command is refused with `filesystem_state`.
156
-
157
- ## trial retry
158
-
159
- ```bash
160
- evolve trial retry <trial-id>
161
- ```
162
-
163
- Run one settled trial again. The result is a job.
164
-
165
- ## trial regrade
166
-
167
- ```bash
168
- evolve trial regrade <trial-id>
169
- ```
170
-
171
- Verifier-only re-run of one trial. The result is a job.
172
-
173
- ## trial stop
174
-
175
- ```bash
176
- evolve trial stop <trial-id> [trial-id...]
177
- ```
178
-
179
- Stop in-flight trials without cancelling their job.
180
-
181
- <GlobalOptions />
@@ -1,36 +0,0 @@
1
- ---
2
- title: "evolve upload"
3
- description: "Upload a job folder as a finished job."
4
- ---
5
-
6
- import GlobalOptions from '/snippets/global-options.mdx';
7
-
8
- ```bash
9
- evolve upload <job_dir> [options]
10
- ```
11
-
12
- Upload a job folder in the Harbor job layout, its `.tar.gz`, or a public URL of one, as a terminal job. What the folder and each trial's `result.json` must hold is on [Upload a job](/core-concepts/upload). The command follows the import to the job, prints the record with the task linkage (`linked N of M trials to <dataset>` and the not-linked tasks with their reasons), names any skipped trials, and ends with the analyze hint.
13
-
14
- ```bash
15
- evolve upload ./job-2026-08-27__12-00-00 -d terminal-bench-4@4.0
16
- ```
17
-
18
- ## Options
19
-
20
- <ParamField path="-d, --dataset" type="name[@version]">
21
- Link the uploaded trials to a published dataset version by task name. Without it the trials link by their task hash on their own; this is the override, and the way to settle an ambiguous hash. See [upload](/core-concepts/upload#task-linkage).
22
- </ParamField>
23
-
24
- <ParamField path="--from" type="url">
25
- A public https URL of the job archive (`.tar.gz`) that the server fetches itself, instead of `<job_dir>`. No local bytes.
26
- </ParamField>
27
-
28
- <ParamField path="--no-wait">
29
- Return after the import is accepted, with the import id, instead of following it to the job. Re-attach with `evolve job import <id> --watch`.
30
- </ParamField>
31
-
32
- <GlobalOptions />
33
-
34
- ## Following an import
35
-
36
- `evolve job imports` lists your uploads, and `evolve job import <import-id> --watch` follows one. See the [`job`](/cli-reference/job) group.
@@ -1,106 +0,0 @@
1
- ---
2
- title: "Agents"
3
- description: "The built-in harnesses, how to configure an arm, and how to register your own agent."
4
- ---
5
-
6
- An agent is a harness plus a model. `-a` names the harness, `-m` the model it drives. Seven harnesses are built in.
7
-
8
- | Name on `-a` | Harness |
9
- | --- | --- |
10
- | `claude` | Claude Code |
11
- | `codex` | Codex |
12
- | `gemini` | Gemini CLI |
13
- | `qwen` | Qwen Code |
14
- | `kimi` | Kimi Code |
15
- | `opencode` | OpenCode |
16
- | `droid` | Droid |
17
-
18
- ```bash
19
- evolve run \
20
- -d terminal-bench-4@4.0 \
21
- -a claude \
22
- -m fable \
23
- -l 5 \
24
- --watch
25
- ```
26
-
27
- `-a <name[@version]>` can pin a harness version. Omitted, the latest is chosen once, when the job is created, so every trial installs the same version; it is recorded on every trial as `agent_info.version`. A pin that does not exist is refused with `agent_version_not_found`, and a pin on a registered agent is refused.
28
-
29
- ## Configure the arm
30
-
31
- These flags apply to every arm of the job.
32
-
33
- ```bash
34
- evolve run \
35
- -d terminal-bench-4@4.0 \
36
- -a codex \
37
- -m gpt-6-astra \
38
- -l 5 \
39
- --effort high \
40
- --preset no-internet \
41
- --ak config=./codex-settings.json
42
- ```
43
-
44
- `--effort <value>` sets the reasoning effort; see [models](/core-concepts/models#reasoning-effort). `--preset <name>` applies a named settings bundle: `no-internet` turns the vendor's server-side web tools off, `pinned-context` fixes the context window at 200000 tokens. `--ak config=<path|inline JSON>` becomes the harness's native settings file inside the sandbox; your document is the base, the platform's routing sits on top, and a preset wins where they disagree.
45
-
46
- Only `claude` and `codex` take a config or a preset; the [capability document](/sdk-reference/meta) publishes `supports_config` and `presets` per harness. An effort, preset or config a harness cannot honor is refused when the job is created, never silently skipped, and so is a config key that touches billing, base URLs, routing or environment.
47
-
48
- Environment for the agent comes from two places, a task's own `[environment.env]` table and a secret attached with `--secret`; a job cannot add a third. See [secrets](/core-concepts/secrets) and [tasks](/core-concepts/tasks#environment-variables).
49
-
50
- ## Register your own agent
51
-
52
- Any CLI can run as an agent. Register it once, and its name works on `-a` like a built-in.
53
-
54
- ```bash
55
- evolve agent add acme-cli --install-script ./install.sh --run "acme-cli --headless"
56
- evolve run \
57
- -d terminal-bench-4@4.0 \
58
- -a acme-cli \
59
- -m gpt-6-astra \
60
- -l 5 \
61
- --watch
62
- ```
63
-
64
- `--install-script <path>` names a file; its contents are uploaded. `--dir <path>` uploads a local directory instead. `--run <command>` is required and runs with `sh -c`. `--ae KEY=VALUE` injects environment at run time and may repeat.
65
-
66
- ```bash
67
- evolve agent list
68
- evolve agent show acme-cli
69
- evolve agent remove acme-cli
70
- ```
71
-
72
- Removing an agent keeps the record of the past jobs that ran it. A registered agent belongs to its owner and to an organization: teammates may name it in their jobs and open it by name; a name outside your organizations reads as `agent_not_found`. The number you may hold is published as `agent_registration.max_per_user`; past it, `agent_limit_reached`.
73
-
74
- The install script, or the uploaded directory, runs once with internet and no secrets: fetch only public sources and leave the executables in `$PREFIX/bin`. To change an agent without a gap, use the SDK's `upsert`; the CLI removes the registration and adds it again.
75
-
76
- ### The run contract
77
-
78
- Your `run_command` runs with `sh -c` at the task's working directory, and can rely on exactly this:
79
-
80
- - The instruction arrives twice: on stdin, and at the path in `$EVOLVE_INSTRUCTION_FILE`.
81
- - `$EVOLVE_GATEWAY_BASE_URL` is an OpenAI-compatible base URL that already ends in `/v1`, and `$EVOLVE_GATEWAY_API_KEY` is its credential. The same two values are also `$OPENAI_BASE_URL` and `$OPENAI_API_KEY`, so a CLI that reads its endpoint from the environment works unchanged.
82
- - `$EVOLVE_MODEL` is the `model_name` of the arm.
83
- - Your declared env may not override those six keys, and may not look like a credential; both are refused at registration with `agent_invalid_env`.
84
-
85
- A CLI that routes through a config file must write that file inside `run_command`, from the contract values. `codex` is the worked example:
86
-
87
- ```bash
88
- mkdir -p ~/.codex && cat > ~/.codex/config.toml <<EOF
89
- model_provider = "evolve"
90
- [model_providers.evolve]
91
- name = "evolve"
92
- base_url = "$EVOLVE_GATEWAY_BASE_URL"
93
- env_key = "EVOLVE_GATEWAY_API_KEY"
94
- wire_api = "responses"
95
- EOF
96
- codex login --with-api-key <<< "$EVOLVE_GATEWAY_API_KEY"
97
- codex exec --skip-git-repo-check -
98
- ```
99
-
100
- A CLI that ignores `OPENAI_BASE_URL` without this reaches for its vendor's endpoint, finds the box sealed, and spends the agent budget failing to connect. The spend cap is airtight only under `no-network`; see [models](/core-concepts/models).
101
-
102
- Against a built-in, a registered agent has no live trace (there is no parser for an unknown CLI) and no `reasoning_effort` (put the flag in `run_command`); everything else is recorded the same way.
103
-
104
- <Card title="agent reference" icon="terminal" href="/cli-reference/agent">
105
- Every flag of `evolve agent`.
106
- </Card>
@@ -1,86 +0,0 @@
1
- ---
2
- title: "Analyze"
3
- description: "Judge every trial's trace against a rubric, on the platform."
4
- ---
5
-
6
- An analysis reads a settled trial's whole record, the trace and the task, and rules on every criterion of a rubric. It runs on the platform, one analyzer run per trial. Ask for it on a finished job:
7
-
8
- ```bash
9
- evolve analyze <job-id> --watch
10
- ```
11
-
12
- Or arm it when you start the job, so each trial is analyzed the moment it settles:
13
-
14
- ```bash
15
- evolve run \
16
- -d terminal-bench-4@4.0 \
17
- -a codex \
18
- -m gpt-6-astra \
19
- --analyze \
20
- -l 5 \
21
- --watch
22
- ```
23
-
24
- ## Rubrics and prompts
25
-
26
- The default rubric has seven criteria, `score_is_earned` first. Bring your own as TOML, YAML or JSON in the shape below and, if you want, your own prompt for the analyzer.
27
-
28
- ```toml rubric.toml
29
- [[criteria]]
30
- name = "score_is_earned"
31
- description = "The reward reflects work the agent actually did."
32
- guidance = "Read the verifier log and the trajectory side by side."
33
- ```
34
-
35
- ```bash
36
- evolve analyze <job-id> \
37
- -r rubric.toml \
38
- -p prompt.txt \
39
- --failing \
40
- -l 20 \
41
- -n 2 \
42
- --watch
43
- ```
44
-
45
- `-r` is the rubric file, `-p` the prompt file. `--passing` and `--failing` restrict the batch of analyses to trials with reward 1.0, or below it; both together is refused. `-t` names one trial to analyze, repeatable, applied with the filter and before `-l`. `-l` caps the number of trials, `-n` how many analyses run at once, beneath the organization's `max_concurrent_analyses` (fleet default 60). `-m` picks the analyzer's model, `--effort` its reasoning effort, `-e` its sandbox provider. `--show-defaults` prints the defaults a bare `evolve analyze` runs under and exits.
46
-
47
- A result that misses or invents a criterion is a failed analysis, never a partial pass. A rubric with unknown keys, empty or duplicate criteria, or a length bound exceeded is refused with `invalid_rubric`; an off-roster model, an unknown effort or provider, or a `-t` id that is not a trial of the job with `invalid_input`; a job with no analyzable trial, or a `--passing`, `--failing` or `-t` selection that leaves none, with `no_analyzable_trials`; a job still running with `job_not_terminal`.
48
-
49
- On `evolve run` the same knobs are `--analyze-model`, `--analyze-rubric`, `--analyze-prompt`, `--analyze-effort` and `--analyze-provider`. Each of them implies `--analyze`.
50
-
51
- Running `evolve analyze` again on the same job, with another rubric or model, is the re-analysis path: one wave at a time (`analysis_already_running` meanwhile), and each trial then serves its newest analysis while earlier ones stay under their own ids. Cancelled trials are never analyzed.
52
-
53
- ## The result
54
-
55
- Each analysis produces a short summary of what happened in the trial, and one entry per criterion:
56
-
57
- - `outcome`: `pass`, `fail`, `not_applicable` or `unknown`
58
- - `explanation`: the analyzer's reasoning, in plain words
59
- - `evidence`: what in the trial record supports it
60
-
61
- Under the default rubric, the platform derives one label from the outcomes:
62
-
63
- - `flagged`: a fail on `score_is_earned`, `score_is_correct`, `task_was_fair` or `report_is_truthful`
64
- - `env_fault`: otherwise, a fail on `environment_worked`
65
- - `unclear`: otherwise, an unknown on any of those five, or `not_applicable` on `score_is_earned` or `score_is_correct`
66
- - `clean`: everything else
67
-
68
- A custom rubric carries its per-criterion outcomes and no label.
69
-
70
- The analyzer's spend is shown apart from the trial's: `estimated_cost_usd` on the analysis, `stats.analysis.cost_usd` on the job. A failed analysis carries `failure` with a `phase` and a `message`; a run that produces no valid result is retried once.
71
-
72
- ## Read it back
73
-
74
- ```bash
75
- evolve analysis list --job <job-id> --status failed
76
- evolve analysis show <analysis-id>
77
- evolve analysis trace <analysis-id>
78
- evolve analysis download <analysis-id> -o analyses/
79
- evolve analysis download <analysis-id> --stream analysis
80
- ```
81
-
82
- `analysis show` prints the verdict document. Every analysis verb also accepts a trial id, which names that trial's latest analysis. `analysis trace` prints the analyzer's own transcript. `analysis download` saves the run as a trial folder, or streams one artifact: `analysis`, `trace-parsed`, `trace-stdout`, `trace-stderr` or `agent-home`.
83
-
84
- <Card title="analyze reference" icon="terminal" href="/cli-reference/analyze">
85
- Every flag of `evolve analyze` and `evolve analysis`.
86
- </Card>
@@ -1,64 +0,0 @@
1
- ---
2
- title: "Check"
3
- description: "Check the quality of tasks against a rubric before you spend a job on them."
4
- ---
5
-
6
- A check reads a task and, when it can, runs its environment, its reference solution and its verifier, then rules on every criterion of a rubric. Point it at a task directory, or at a directory of tasks:
7
-
8
- ```bash
9
- evolve check ./tasks \
10
- -i 'abs-*' \
11
- -l 5 \
12
- --watch
13
- ```
14
-
15
- Or at a published dataset version. Nothing is uploaded; the checker reads the retained task package.
16
-
17
- ```bash
18
- evolve check -d my-swe@1.0 --watch
19
- ```
20
-
21
- When the path is a task directory, that task is checked; otherwise every top-level directory that is one, in sorted order. `-i` and `-x` are globs over task names and `-l` caps the count, in that order; a selection that leaves nothing is refused with `no_checkable_tasks`. There is no task-count ceiling.
22
-
23
- `-r` is a rubric file, `-p` a prompt file. `-m` and `--effort` set the checker's model and effort, `-e` its sandbox provider, `-n` how many checks run at once, beneath the organization's `max_concurrent_analyses`.
24
-
25
- The default rubric has eleven criteria: `instruction_is_sufficient`, `tests_match_the_instruction`, `verifier_is_correct`, `no_answer_leakage`, `grader_is_out_of_reach`, `reference_solution_is_valid`, `verifier_rejects_non_solutions`, `environment_builds_and_runs`, `verification_is_stable`, `limits_allow_the_task` and `task_is_solvable`. The checker's default model is the analyzer's; a custom rubric, model, prompt and effort follow the same rules and refusals as [analyze](/core-concepts/analyze).
26
-
27
- ## The result
28
-
29
- Each task gets one entry per criterion, with an `outcome` (`pass`, `fail`, `not_applicable` or `unknown`), an `explanation` and `evidence`. Under the default rubric, one label is derived:
30
-
31
- - `has_a_problem`: a fail on any criterion
32
- - `unclear`: otherwise, an unknown on any of the six file-based criteria
33
- - `no_problem_found`: everything else
34
-
35
- `executed` says whether the checker actually ran the task's environment, so a reading-only `no_problem_found` is never mistaken for a run. A check never fails as a whole: refusals happen at accept, and a task's failure is on that task. A check is readable by its creator, by every member of the organization and by whoever it was shared with; anyone else reads `check_not_found`.
36
-
37
- ## Read it back
38
-
39
- ```bash
40
- evolve check list --status running
41
- evolve check show <check-id>
42
- evolve check trace <task-check-id> --since 200
43
- evolve check download <check-id> -o checks/
44
- evolve check download <task-check-id> --stream task-check
45
- ```
46
-
47
- `check show` prints the check with one result per task. `check trace` prints the checker's transcript for one task check. `check download` saves the whole check as a check folder by its check id, one task's folder by its task check id, or streams one artifact of a task check: `task-check`, `trace-parsed`, `trace-stdout`, `trace-stderr` or `agent-home`.
48
-
49
- ## Share
50
-
51
- ```bash
52
- evolve check share <check-id> --link
53
- evolve check share <check-id> --email alice@example.org
54
- evolve check shares <check-id>
55
- evolve check unshare <check-id> --link
56
- ```
57
-
58
- A check you created shares the two ways a [job](/core-concepts/jobs#share) does, and both may be on at once. `--link` turns on an unlisted link: anyone holding it reads the check's report and downloads its check folder, without signing in. `check shares` prints the link again whenever you need it, and `check unshare --link` kills it at once. `--email` shares the check with a person by address: they get an email with a link and read the check after signing in with that address (a person without an account gets a sign-up link in the same email). A check shared with you appears under `evolve check list --scope shared`.
59
-
60
- Sharing is read-only: whoever holds the link or the email share reads the check and can never stop one of its task checks; that stays with the creator. Only the creator may share or unshare; another member of the organization is refused with `org_forbidden`. A check's `visibility` field says `PRIVATE` or `LINK`. The dashboard side, and what a link shows, are on [Sharing](/core-concepts/sharing).
61
-
62
- <Card title="check reference" icon="terminal" href="/cli-reference/check">
63
- Every flag of `evolve check`.
64
- </Card>