@evolvingmachines/evolve 0.0.65 → 0.0.67

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (152) hide show
  1. package/dist/{chunk-DK2MGUPB.js → chunk-RFLMRPGK.js} +5 -5
  2. package/dist/cli/index.cjs +39 -39
  3. package/dist/cli/index.d.cts +3 -3
  4. package/dist/cli/index.d.ts +3 -3
  5. package/dist/cli/index.js +30 -30
  6. package/dist/index.cjs +1 -1
  7. package/dist/index.d.cts +4 -4
  8. package/dist/index.d.ts +4 -4
  9. package/dist/index.js +1 -1
  10. package/dist/{types-B_qmzlMk.d.cts → types-84M4J_Vm.d.cts} +11 -44
  11. package/dist/{types-B_qmzlMk.d.ts → types-84M4J_Vm.d.ts} +11 -44
  12. package/package.json +4 -4
  13. package/skills/create-adapter/SKILL.md +71 -26
  14. package/skills/create-task/SKILL.md +124 -66
  15. package/skills/evolve/SKILL.md +37 -20
  16. package/skills/evolve-evals/SKILL.md +142 -69
  17. package/skills/evolve-evals/references/cli-reference/agent.md +76 -0
  18. package/skills/evolve-evals/references/cli-reference/analysis.md +78 -0
  19. package/skills/evolve-evals/references/cli-reference/analyze.md +95 -0
  20. package/skills/evolve-evals/references/cli-reference/auth.md +96 -0
  21. package/skills/evolve-evals/references/cli-reference/check.md +156 -0
  22. package/skills/evolve-evals/references/cli-reference/dataset.md +154 -0
  23. package/skills/evolve-evals/references/cli-reference/filesystem.md +168 -0
  24. package/skills/evolve-evals/references/cli-reference/index.md +149 -0
  25. package/skills/evolve-evals/references/cli-reference/job.md +198 -0
  26. package/skills/evolve-evals/references/cli-reference/run.md +187 -0
  27. package/skills/evolve-evals/references/cli-reference/secrets.md +77 -0
  28. package/skills/evolve-evals/references/cli-reference/session.md +56 -0
  29. package/skills/evolve-evals/references/cli-reference/skill.md +52 -0
  30. package/skills/evolve-evals/references/cli-reference/skills.md +85 -0
  31. package/skills/evolve-evals/references/cli-reference/trial.md +115 -0
  32. package/skills/evolve-evals/references/cli-reference/upload.md +72 -0
  33. package/skills/evolve-evals/references/core-concepts/agents.md +96 -0
  34. package/skills/evolve-evals/references/core-concepts/analyze.md +143 -0
  35. package/skills/evolve-evals/references/core-concepts/check.md +116 -0
  36. package/skills/evolve-evals/references/core-concepts/compatibility.md +39 -0
  37. package/skills/evolve-evals/references/core-concepts/compose.md +103 -0
  38. package/skills/evolve-evals/references/core-concepts/datasets.md +234 -0
  39. package/skills/evolve-evals/references/core-concepts/index.md +52 -0
  40. package/skills/evolve-evals/references/core-concepts/jobs.md +223 -0
  41. package/skills/evolve-evals/references/core-concepts/models.md +73 -0
  42. package/skills/evolve-evals/references/core-concepts/multi-step.md +93 -0
  43. package/skills/evolve-evals/references/core-concepts/rewardkit.md +341 -0
  44. package/skills/evolve-evals/references/core-concepts/sandboxes.md +73 -0
  45. package/skills/evolve-evals/references/core-concepts/secrets.md +86 -0
  46. package/skills/evolve-evals/references/core-concepts/sharing.md +81 -0
  47. package/skills/evolve-evals/references/core-concepts/skills.md +100 -0
  48. package/skills/evolve-evals/references/core-concepts/task-config.md +169 -0
  49. package/skills/evolve-evals/references/core-concepts/task-environment.md +127 -0
  50. package/skills/evolve-evals/references/core-concepts/task-reference.md +191 -0
  51. package/skills/evolve-evals/references/core-concepts/task-verifiers.md +209 -0
  52. package/skills/evolve-evals/references/core-concepts/tasks.md +90 -0
  53. package/skills/evolve-evals/references/core-concepts/teams.md +128 -0
  54. package/skills/evolve-evals/references/core-concepts/trial-outputs.md +225 -0
  55. package/skills/evolve-evals/references/core-concepts/trials.md +129 -0
  56. package/skills/evolve-evals/references/core-concepts/upload-sdk-session.md +240 -0
  57. package/skills/evolve-evals/references/core-concepts/upload.md +166 -0
  58. package/skills/evolve-evals/references/dashboard/checks.md +54 -0
  59. package/skills/evolve-evals/references/dashboard/datasets.md +65 -0
  60. package/skills/evolve-evals/references/dashboard/jobs.md +106 -0
  61. package/skills/evolve-evals/references/dashboard/traces.md +54 -0
  62. package/skills/evolve-evals/references/dashboard/trial-viewer.md +77 -0
  63. package/skills/evolve-evals/references/getting-started/first-task.md +228 -0
  64. package/skills/evolve-evals/references/getting-started/installation.md +79 -0
  65. package/skills/evolve-evals/references/getting-started/quick-start.md +100 -0
  66. package/skills/evolve-evals/references/index.md +58 -0
  67. package/skills/evolve-evals/references/sdk/http.md +110 -0
  68. package/skills/evolve-evals/references/sdk/python.md +108 -0
  69. package/skills/evolve-evals/references/sdk/typescript.md +104 -0
  70. package/skills/evolve-evals/references/sdk-reference/agents.md +75 -0
  71. package/skills/evolve-evals/references/sdk-reference/analyses.md +124 -0
  72. package/skills/evolve-evals/references/sdk-reference/auth.md +171 -0
  73. package/skills/evolve-evals/references/sdk-reference/checks.md +145 -0
  74. package/skills/evolve-evals/references/sdk-reference/datasets.md +170 -0
  75. package/skills/evolve-evals/references/sdk-reference/error-codes.md +231 -0
  76. package/skills/evolve-evals/references/sdk-reference/errors.md +98 -0
  77. package/skills/evolve-evals/references/sdk-reference/filesystem.md +166 -0
  78. package/skills/evolve-evals/references/sdk-reference/index.md +161 -0
  79. package/skills/evolve-evals/references/sdk-reference/jobs.md +250 -0
  80. package/skills/evolve-evals/references/sdk-reference/meta.md +54 -0
  81. package/skills/evolve-evals/references/sdk-reference/methods/agents.md +154 -0
  82. package/skills/evolve-evals/references/sdk-reference/methods/analyses.md +211 -0
  83. package/skills/evolve-evals/references/sdk-reference/methods/auth.md +257 -0
  84. package/skills/evolve-evals/references/sdk-reference/methods/checks.md +460 -0
  85. package/skills/evolve-evals/references/sdk-reference/methods/datasets.md +687 -0
  86. package/skills/evolve-evals/references/sdk-reference/methods/filesystem.md +527 -0
  87. package/skills/evolve-evals/references/sdk-reference/methods/jobs.md +1021 -0
  88. package/skills/evolve-evals/references/sdk-reference/methods/meta.md +209 -0
  89. package/skills/evolve-evals/references/sdk-reference/methods/secrets.md +157 -0
  90. package/skills/evolve-evals/references/sdk-reference/methods/skills.md +125 -0
  91. package/skills/evolve-evals/references/sdk-reference/methods/trials.md +371 -0
  92. package/skills/evolve-evals/references/sdk-reference/secrets.md +104 -0
  93. package/skills/evolve-evals/references/sdk-reference/skills.md +66 -0
  94. package/skills/evolve-evals/references/sdk-reference/trials.md +164 -0
  95. package/skills/evolve-evals/references/sdk-reference/types.md +1249 -0
  96. package/skills/publish/SKILL.md +103 -79
  97. package/skills/rewardkit/SKILL.md +78 -23
  98. package/spec/openapi.yaml +19 -74
  99. package/skills/evolve-evals/references/cli-reference/agent.mdx +0 -79
  100. package/skills/evolve-evals/references/cli-reference/analysis.mdx +0 -160
  101. package/skills/evolve-evals/references/cli-reference/analyze.mdx +0 -84
  102. package/skills/evolve-evals/references/cli-reference/auth.mdx +0 -133
  103. package/skills/evolve-evals/references/cli-reference/check.mdx +0 -263
  104. package/skills/evolve-evals/references/cli-reference/dataset.mdx +0 -151
  105. package/skills/evolve-evals/references/cli-reference/index.mdx +0 -108
  106. package/skills/evolve-evals/references/cli-reference/job.mdx +0 -278
  107. package/skills/evolve-evals/references/cli-reference/run.mdx +0 -204
  108. package/skills/evolve-evals/references/cli-reference/secrets.mdx +0 -95
  109. package/skills/evolve-evals/references/cli-reference/skill.mdx +0 -61
  110. package/skills/evolve-evals/references/cli-reference/skills.mdx +0 -81
  111. package/skills/evolve-evals/references/cli-reference/trial.mdx +0 -181
  112. package/skills/evolve-evals/references/cli-reference/upload.mdx +0 -36
  113. package/skills/evolve-evals/references/core-concepts/agents.mdx +0 -106
  114. package/skills/evolve-evals/references/core-concepts/analyze.mdx +0 -86
  115. package/skills/evolve-evals/references/core-concepts/check.mdx +0 -64
  116. package/skills/evolve-evals/references/core-concepts/datasets.mdx +0 -138
  117. package/skills/evolve-evals/references/core-concepts/index.mdx +0 -42
  118. package/skills/evolve-evals/references/core-concepts/jobs.mdx +0 -151
  119. package/skills/evolve-evals/references/core-concepts/models.mdx +0 -68
  120. package/skills/evolve-evals/references/core-concepts/sandboxes.mdx +0 -52
  121. package/skills/evolve-evals/references/core-concepts/secrets.mdx +0 -41
  122. package/skills/evolve-evals/references/core-concepts/sharing.mdx +0 -35
  123. package/skills/evolve-evals/references/core-concepts/skills.mdx +0 -42
  124. package/skills/evolve-evals/references/core-concepts/tasks.mdx +0 -216
  125. package/skills/evolve-evals/references/core-concepts/teams.mdx +0 -36
  126. package/skills/evolve-evals/references/core-concepts/trial-outputs.mdx +0 -117
  127. package/skills/evolve-evals/references/core-concepts/trials.mdx +0 -103
  128. package/skills/evolve-evals/references/core-concepts/upload.mdx +0 -93
  129. package/skills/evolve-evals/references/dashboard/checks.mdx +0 -21
  130. package/skills/evolve-evals/references/dashboard/datasets.mdx +0 -24
  131. package/skills/evolve-evals/references/dashboard/jobs.mdx +0 -65
  132. package/skills/evolve-evals/references/dashboard/traces.mdx +0 -17
  133. package/skills/evolve-evals/references/dashboard/trial-viewer.mdx +0 -30
  134. package/skills/evolve-evals/references/getting-started/installation.mdx +0 -56
  135. package/skills/evolve-evals/references/getting-started/quick-start.mdx +0 -94
  136. package/skills/evolve-evals/references/index.mdx +0 -48
  137. package/skills/evolve-evals/references/sdk/python.mdx +0 -161
  138. package/skills/evolve-evals/references/sdk/typescript.mdx +0 -163
  139. package/skills/evolve-evals/references/sdk-reference/agents.mdx +0 -108
  140. package/skills/evolve-evals/references/sdk-reference/analyses.mdx +0 -222
  141. package/skills/evolve-evals/references/sdk-reference/auth.mdx +0 -116
  142. package/skills/evolve-evals/references/sdk-reference/checks.mdx +0 -300
  143. package/skills/evolve-evals/references/sdk-reference/datasets.mdx +0 -346
  144. package/skills/evolve-evals/references/sdk-reference/errors.mdx +0 -210
  145. package/skills/evolve-evals/references/sdk-reference/index.mdx +0 -60
  146. package/skills/evolve-evals/references/sdk-reference/jobs.mdx +0 -822
  147. package/skills/evolve-evals/references/sdk-reference/meta.mdx +0 -52
  148. package/skills/evolve-evals/references/sdk-reference/skills.mdx +0 -69
  149. package/skills/evolve-evals/references/sdk-reference/trials.mdx +0 -302
  150. package/skills/evolve-evals/references/sdk-reference/types.mdx +0 -274
  151. package/skills/evolve-evals/references/snippets/global-options.mdx +0 -13
  152. package/skills/evolve-evals/references/snippets/list-flags.mdx +0 -23
@@ -0,0 +1,198 @@
1
+ ---
2
+ title: "evolve job"
3
+ description: "Inspect results, compare runs, and control or repeat evaluations."
4
+ ---
5
+
6
+ Start with the job ID printed by `evolve run`, or find it in the list. Examples below use `$JOB_ID` for that ID.
7
+
8
+ ```bash
9
+ evolve job list
10
+ evolve job show "$JOB_ID"
11
+ evolve job trials "$JOB_ID"
12
+ ```
13
+
14
+ ## Inspect results
15
+
16
+ ### Jobs
17
+
18
+ ```bash
19
+ evolve job list --search hello
20
+ evolve job list --kind all --scope org
21
+ evolve job show "$JOB_ID" --json
22
+ ```
23
+
24
+ | Option for `job list` | Meaning |
25
+ | --- | --- |
26
+ | `--search <text>` | Match job names and dataset names. |
27
+ | `--kind <job\|check\|all>` | Evaluations, task checks, or both. Default `job`. |
28
+ | `--scope <my\|shared\|org>` | Which records to list. Default `my`. |
29
+
30
+ `job show` accepts one or more IDs. With `--json`, one ID returns an object; several return an array.
31
+
32
+ ### Trials
33
+
34
+ ```bash
35
+ evolve job trials "$JOB_ID" --status INFRASTRUCTURE_ERROR,SCORING_ERROR
36
+ evolve job trials "$JOB_ID" --dataset harbor-examples
37
+ ```
38
+
39
+ | Option for `job trials` | Meaning |
40
+ | --- | --- |
41
+ | `--status <statuses>` | Comma-separated trial statuses. |
42
+ | `--dataset <name>` | Select one dataset in the job. |
43
+
44
+ Statuses: `QUEUED`, `RUNNING`, `SCORING`, `SCORED`, `SCORING_ERROR`, `INFRASTRUCTURE_ERROR`, `BUDGET`, `INDETERMINATE`, `CANCELLED`.
45
+
46
+ Open an individual result with [`evolve trial show`](/cli-reference/trial).
47
+
48
+ ### Tasks
49
+
50
+ ```bash
51
+ evolve job tasks "$JOB_ID"
52
+ ```
53
+
54
+ One row per task: trial count, statuses, reward, spend, and available task-check information.
55
+
56
+ `job list`, `job trials`, and `job tasks` accept the shared [paging and formatting options](/cli-reference/index#list-options): `--limit`, `--cursor`, `--columns`, `--quiet`, `--no-trunc`, and `--no-headers`.
57
+
58
+ A `COMPLETED` job can include failed trials. Inspect the trial statuses as well as the aggregate score.
59
+
60
+ ## Compare jobs
61
+
62
+ Supply two to ten job IDs.
63
+
64
+ ```bash
65
+ evolve job compare "$JOB_ID" "$OTHER_JOB_ID"
66
+ ```
67
+
68
+ ## Search traces
69
+
70
+ Search across the job's trial traces. `$JOB_ID` comes from `job list` or `run`.
71
+
72
+ ```bash
73
+ evolve job grep "$JOB_ID" 'permission denied'
74
+ evolve job grep "$JOB_ID" 'permission denied' --type tool_call --json
75
+ ```
76
+
77
+ | Option | Meaning |
78
+ | --- | --- |
79
+ | `--type <event-type>` | Match only this exact event type. |
80
+ | `-l`, `--limit <n>` | Trial match groups per page. Default `50`, maximum `200`. |
81
+ | `--cursor <cursor>` | Continue from the returned `nextCursor`. |
82
+
83
+ ## Stop work
84
+
85
+ ### Whole job
86
+
87
+ ```bash
88
+ evolve job cancel "$JOB_ID"
89
+ ```
90
+
91
+ Request cancellation. The returned job may still be winding down.
92
+
93
+ ### One dataset
94
+
95
+ ```bash
96
+ evolve job stop "$JOB_ID" --dataset harbor-examples
97
+ ```
98
+
99
+ Stop that dataset's trials while keeping the rest of the job. `--dataset <name>` is required.
100
+
101
+ Output separates stopped trials, already-terminal trials, and missing IDs. If a batch fails, the partial report names how many outcomes remain unreported.
102
+
103
+ For individual trials, use [`trial stop`](/cli-reference/trial#stop-a-trial).
104
+
105
+ ## Run again
106
+
107
+ Each command creates a new linked job. It leaves the source results intact.
108
+
109
+ | Command | Work repeated |
110
+ | --- | --- |
111
+ | `evolve job resume "$JOB_ID"` | Failed or stopped trials selected by the resume policy. |
112
+ | `evolve job retry "$JOB_ID"` | All trials of a settled source job. |
113
+ | `evolve job retry "$JOB_ID" --failed-only` | Trials in `SCORING_ERROR`, `INFRASTRUCTURE_ERROR`, `BUDGET`, or `INDETERMINATE`. |
114
+ | `evolve job retry "$JOB_ID" --trial "$TRIAL_ID"` | Exactly the selected settled trial. Repeat `--trial` for more. |
115
+ | `evolve job regrade "$JOB_ID"` | The recorded verifier, using stored inputs. No replacement verifier. |
116
+
117
+ ### Selection options and requirements
118
+
119
+ **Resume** accepts `-f, --filter-error-type <type>`. Repeat it to select exception types.
120
+
121
+ **Retry** accepts either `--failed-only` or `-t, --trial <id>`, not both. Copy full trial IDs from `evolve job trials "$JOB_ID"`; these flags do not expand prefixes. Explicit settled trials can be retried while other trials in the source job are still running.
122
+
123
+ **Regrade** accepts `--status <s1,s2,...>` and `--task <name>`. It requires eligible stored verifier inputs; shared-verifier, multi-step, and judge-backed trials cannot be regraded through this command. See [Trials](/core-concepts/trials).
124
+
125
+ Uploaded jobs cannot be resumed, retried, or regraded. Their original execution ran outside Evolve.
126
+
127
+ Start a new evaluation with `evolve job start` or its shorter form, [`evolve run`](/cli-reference/run).
128
+
129
+ ## Download results
130
+
131
+ ```bash
132
+ evolve job download "$JOB_ID" -o results/
133
+ ```
134
+
135
+ ```text
136
+ results/
137
+ └── job-<id>/
138
+ ├── config.json
139
+ ├── result.json
140
+ ├── evolve.json Evolve's record
141
+ └── <trial folders>/
142
+ ```
143
+
144
+ | Option | Meaning |
145
+ | --- | --- |
146
+ | `-o`, `--output-dir <dir>` | Parent directory. Default: current directory. |
147
+ | `--overwrite` | Allow replacing an existing job folder. |
148
+
149
+ The download uses the Harbor job layout, with additional `evolve.json` records.
150
+
151
+ ## Share results
152
+
153
+ Only the creator can manage a job's shares.
154
+
155
+ ```bash Create access
156
+ evolve job share "$JOB_ID" --link
157
+ evolve job share "$JOB_ID" --email colleague@example.com
158
+ ```
159
+
160
+ ```bash Inspect or remove access
161
+ evolve job shares "$JOB_ID"
162
+ evolve job unshare "$JOB_ID" --link
163
+ evolve job unshare "$JOB_ID" --email colleague@example.com
164
+ ```
165
+
166
+ Both `share` and `unshare` require `--link`, repeatable `--email <address>`, or both. Email sharing sends the recipient a link. Revoking a link disables it; enabling sharing later creates a new link.
167
+
168
+ See [Sharing](/core-concepts/sharing) for access rules.
169
+
170
+ ## Read an upload's progress
171
+
172
+ [`evolve upload`](/cli-reference/upload) prints an import ID before the imported job is ready.
173
+
174
+ ```bash
175
+ evolve job imports --status RUNNING
176
+ evolve job import "$IMPORT_ID" --watch
177
+ ```
178
+
179
+ | Command | Options |
180
+ | --- | --- |
181
+ | `job imports` | Shared list options; `--status <QUEUED\|RUNNING\|COMPLETED\|FAILED>`. |
182
+ | `job import <id>` | `--watch` to follow it to a job or failure. Use the full import ID. |
183
+
184
+ `job import --watch --json` prints one final job or error document.
185
+
186
+ ## Delete a job
187
+
188
+ ```bash
189
+ evolve job delete "$JOB_ID"
190
+ ```
191
+
192
+ The command asks before permanent deletion. For noninteractive use, pass `-y` or `--yes`.
193
+
194
+ **Warning:**
195
+
196
+ Deleting a job removes its stored trials, traces, analyses, and files. Only the creator can delete it, and the job must be eligible for deletion.
197
+
198
+ [Global options](/cli-reference/index#global-options) apply to every command.
@@ -0,0 +1,187 @@
1
+ ---
2
+ title: "evolve run"
3
+ description: "Choose tasks and agents, set limits, and launch an evaluation."
4
+ ---
5
+
6
+ Start a hosted job. `evolve job start` accepts exactly the same options.
7
+
8
+ ```bash
9
+ evolve run \
10
+ -d harbor-examples@1.0 \
11
+ -i hello-world \
12
+ -a codex -m gpt-5.6-luna \
13
+ --max-trial-spend 0.30 \
14
+ --max-retries 0 \
15
+ --watch
16
+ ```
17
+
18
+ ```text
19
+ Selected tasks
20
+ × Agent/model combinations
21
+ × Attempts
22
+ = Trials
23
+ ```
24
+
25
+ Without `--watch`, the command returns after the job is accepted. Keep the printed job ID to [inspect results](/cli-reference/job).
26
+
27
+ ## Select the work
28
+
29
+ | Option | Meaning | Default |
30
+ | --- | --- | --- |
31
+ | `-d`, `--dataset <name[@version]>` | Dataset to evaluate. Repeat for more datasets. | Required, unless in config. |
32
+ | `-a`, `--agent <name[@version]>` | Built-in or registered agent harness. | Required, unless in config. |
33
+ | `-m`, `--model <name>` | Model for the harness. Repeat for one combination per model. | Required, unless in config. |
34
+ | `-i`, `--include-task-name <glob>` | Include matching tasks in every dataset. Repeatable. | All tasks. |
35
+ | `-x`, `--exclude-task-name <glob>` | Exclude matching tasks after inclusion. Repeatable. | None. |
36
+ | `-l`, `--n-tasks <n>` | Cap tasks per dataset, after filters. | No cap. |
37
+ | `-k`, `--n-attempts <n>` | Attempts per task and agent/model combination. | `1` |
38
+ | `-n`, `--n-concurrent <n>` | Parallel trials, from `1` to `150`. | `4` |
39
+
40
+ A bare dataset name selects its active version. Pin `name@version` for a repeatable selection.
41
+
42
+ ## Save a configuration
43
+
44
+ Use a YAML or JSON file when a job has several agents or per-dataset settings.
45
+
46
+ ```yaml job.yaml
47
+ job_name: hello-world-check
48
+ datasets:
49
+ - name: harbor-examples
50
+ version: "1.0"
51
+ task_names: [hello-world]
52
+ agents:
53
+ - name: codex
54
+ model_name: gpt-5.6-luna
55
+ n_attempts: 1
56
+ n_concurrent_trials: 4
57
+ max_trial_spend_usd: 0.30
58
+ retry:
59
+ max_retries: 0
60
+ ```
61
+
62
+ ```bash Run or inspect
63
+ evolve run -c job.yaml --print-config
64
+ evolve run -c job.yaml --watch
65
+ ```
66
+
67
+ `--print-config` prints the request without creating a job or uploading local skills. It does not resolve catalog names or fill every server default. Inline secret values appear in this output.
68
+
69
+ ### How flags override the file
70
+
71
+ | Flag | What it replaces |
72
+ | --- | --- |
73
+ | `-d` | The entire dataset list. |
74
+ | `-a` and `-m` | The entire agent list. Supply both. |
75
+ | `-i`, `-x`, `-l` | That field on every dataset selector. |
76
+ | `--effort`, `--preset` | That field on every agent combination. |
77
+ | `--ak` | Matching keys inside each agent's `kwargs`. |
78
+ | `--skill` | Every agent's skills list. |
79
+ | `--secret` / `--secret-inline` | The complete secret attachment list. |
80
+ | Retry or analysis options | Only the named field inside `retry` or `analyze`. |
81
+
82
+ Other flags override their matching field. Use `--org` or `auth org use` for organization selection; the current CLI does not retain `org` from the job file.
83
+
84
+ Quote version strings in YAML, for example `version: "1.10"`. Config validation rejects unknown fields and duplicate keys before submission.
85
+
86
+ ## All other options
87
+
88
+ ### Job, organization, and sandbox
89
+
90
+ | Option | Meaning |
91
+ | --- | --- |
92
+ | `--job-name <name>` | Job label. Generated when omitted. |
93
+ | `--org <name>` | Organization to use. Otherwise use the saved CLI default, then your personal organization. `--org personal` overrides a saved team default. |
94
+ | `-e`, `--env <provider>` | Requested sandbox provider: `daytona` (default), `e2b`, or `modal`. |
95
+ | `--system-log` | Record the sandbox's system log. Off by default. |
96
+
97
+ Evolve checks each task's requirements. A task may run on another compatible provider; inspect its trial for the provider actually used. See [Sandboxes](/core-concepts/sandboxes).
98
+
99
+ ### Agent settings and skills
100
+
101
+ These options apply to every agent combination.
102
+
103
+ | Option | Meaning |
104
+ | --- | --- |
105
+ | `--effort <value>` | Reasoning effort supported by the selected harness and model. |
106
+ | `--preset <name>` | Supported settings preset: `no-internet` or `pinned-context`. |
107
+ | `--ak <key=value>`, `--agent-kwarg <key=value>` | Agent setting. Repeatable. `config` accepts a local JSON/TOML file or inline JSON object. |
108
+ | `--skill <ref>`, `--skills <ref>` | Skill reference or local directory. Repeatable. |
109
+
110
+ ```bash
111
+ evolve run -c job.yaml --effort high --ak config=./codex.toml
112
+ evolve run -c job.yaml --skill ./my-skill
113
+ ```
114
+
115
+ Skills accept `skills.sh/owner/repo[/skill]`, `org/repo[@ref]`, a Git URL, `upload:<id>`, or `name:<skill-name>`. Local directories upload before the job starts.
116
+
117
+ Unsupported effort, presets, or settings are refused. See [Agents](/core-concepts/agents) and [Skills](/core-concepts/skills).
118
+
119
+ ### Secrets and verifier environment
120
+
121
+ | Option | Meaning |
122
+ | --- | --- |
123
+ | `--secret <NAME[@LABEL][=ENVNAME]>` | Attach a stored secret to agent runs. Repeatable. `@LABEL` selects a row; `=ENVNAME` renames the environment variable. |
124
+ | `--secret-inline <NAME[@LABEL]:DELIVERY=VALUE>` | Store a secret and attach it. Evaluation jobs require `direct` delivery. Repeatable. |
125
+ | `--verifier-env <KEY=VALUE>`, `--ve <KEY=VALUE>` | Verifier overrides: `REWARDKIT_JUDGE` and `REWARDKIT_MODEL`. Repeatable. |
126
+
127
+ Store sensitive values with [`evolve secrets set`](/cli-reference/secrets), then attach them by name.
128
+
129
+ ```bash
130
+ evolve run -c job.yaml --secret GITHUB_TOKEN
131
+ ```
132
+
133
+ ### Spend and automatic retries
134
+
135
+ | Option | Meaning | Default |
136
+ | --- | --- | --- |
137
+ | `--max-trial-spend <usd>` | Agent model-spend cap per trial attempt. Must be greater than zero. | Platform default, normally `$200`. |
138
+ | `-r`, `--max-retries <n>` | Additional attempts after eligible infrastructure errors. `0` disables these retries. | Platform default, normally `2`. |
139
+ | `--retry-include <exception>` | Retry only these exception types. Repeatable. | Any type not excluded. |
140
+ | `--retry-exclude <exception>` | Never retry these exception types. Repeatable; takes priority over include. | Built-in exclusions. |
141
+
142
+ A low score does not trigger an infrastructure retry. Each retry has the full trial cap. Capacity waits are separate from this retry setting.
143
+
144
+ See [Jobs](/core-concepts/jobs) for the retry and spend rules.
145
+
146
+ ### Analyze traces as trials finish
147
+
148
+ | Option | Meaning |
149
+ | --- | --- |
150
+ | `--analyze` | Enable analysis with the platform defaults. |
151
+ | `--analyze-model <name>` | Analyzer model. |
152
+ | `--analyze-rubric <path>` | TOML, YAML, or JSON rubric. |
153
+ | `--analyze-prompt <path>` | Replacement prompt file. |
154
+ | `--analyze-provider <provider>` | Analyzer sandbox provider. |
155
+ | `--analyze-effort <value>` | Analyzer reasoning effort. |
156
+
157
+ Any `--analyze-*` option enables analysis. Read the current defaults with `evolve analyze --show-defaults`.
158
+
159
+ To analyze an existing job, use [`evolve analyze`](/cli-reference/analyze).
160
+
161
+ ### Timeouts
162
+
163
+ Every multiplier must be greater than zero. A phase-specific value overrides the general multiplier for that phase.
164
+
165
+ | Option | Timeout affected |
166
+ | --- | --- |
167
+ | `--timeout-multiplier <x>` | All task timeouts. Default `1.0`. |
168
+ | `--agent-timeout-multiplier <x>` | Agent execution. |
169
+ | `--verifier-timeout-multiplier <x>` | Verifier execution. |
170
+ | `--agent-setup-timeout-multiplier <x>` | Agent installation/setup. |
171
+ | `--environment-build-timeout-multiplier <x>` | Environment preparation. |
172
+
173
+ Values below `1` shorten a timeout. The task definition is unchanged.
174
+
175
+ ### Configuration and output
176
+
177
+ | Option | Meaning |
178
+ | --- | --- |
179
+ | `-c`, `--config <path>` | Load YAML or JSON job configuration. |
180
+ | `--print-config` | Print the constructed request and exit. |
181
+ | `--watch` | Follow events until the job settles. |
182
+ | `-q`, `--quiet` | With watch, suppress intermediate events. |
183
+ | `-y`, `--yes` | Accepted for compatibility; no effect. |
184
+
185
+ With `--watch --json`, output is one JSON record per line: `job.created`, events, then `job.final`. Quiet mode keeps the initial and final records.
186
+
187
+ [Global options](/cli-reference/index#global-options) apply. Next, [inspect the job](/cli-reference/job).
@@ -0,0 +1,77 @@
1
+ ---
2
+ title: "evolve secrets"
3
+ description: "Store environment secrets and attach them to evaluations by name."
4
+ ---
5
+
6
+ Store a value once, then attach its name to a job. Reads return metadata, never the stored value.
7
+
8
+ ```bash
9
+ printf %s "$GITHUB_TOKEN" | evolve secrets set GITHUB_TOKEN \
10
+ --delivery brokered \
11
+ --allowed-host api.github.com \
12
+ --allowed-path-prefix / \
13
+ --allowed-method GET
14
+ ```
15
+
16
+ Piping the value keeps it out of the command's arguments.
17
+
18
+ ## Choose delivery
19
+
20
+ | Mode | What enters the sandbox |
21
+ | --- | --- |
22
+ | `brokered` | A reference used through the credential proxy. The stored value stays outside the sandbox. |
23
+ | `direct` | The raw value as an environment variable. |
24
+
25
+ Brokered secrets use the allowed-host, path, and method constraints. See [Secrets](/core-concepts/secrets) for delivery behavior.
26
+
27
+ ## Store a secret
28
+
29
+ ```bash
30
+ printf %s "$SERVICE_TOKEN" | evolve secrets set SERVICE_TOKEN --delivery direct
31
+ ```
32
+
33
+ | Option | Meaning |
34
+ | --- | --- |
35
+ | `--value <value>` | Value to store. If omitted, read piped stdin. One trailing newline is removed from stdin. |
36
+ | `--label <label>` | Row label under this name. Default `default`. |
37
+ | `--delivery <brokered\|direct>` | Required delivery mode. |
38
+ | `--allowed-host <host>` | Allowed hostname or wildcard. Repeatable. |
39
+ | `--allowed-path-prefix <path>` | Allowed URL path prefix. Repeatable. |
40
+ | `--allowed-method <method>` | Allowed HTTP method. Repeatable. |
41
+
42
+ Restating the same value can update delivery or scoping. A different value under the same name and label is refused with `secret_exists`; use a new label or delete the old row first.
43
+
44
+ ## Attach it to a job
45
+
46
+ ```bash
47
+ evolve run -c job.yaml --secret GITHUB_TOKEN
48
+ evolve run -c job.yaml --secret SERVICE_TOKEN@staging=APP_TOKEN
49
+ ```
50
+
51
+ The second example selects label `staging` and exposes it under `APP_TOKEN`. See [Run options](/cli-reference/run) for `--secret` and `--secret-inline`.
52
+
53
+ ## List metadata
54
+
55
+ ```bash
56
+ evolve secrets list
57
+ evolve secrets list --json
58
+ ```
59
+
60
+ | Option | Meaning |
61
+ | --- | --- |
62
+ | `--columns <keys\|all\|help>` | Select metadata columns. |
63
+ | `-q`, `--quiet` | Print names, with `:label` for non-default labels. |
64
+ | `--no-trunc` | Keep full terminal table cells. |
65
+ | `--no-headers` | Omit piped TSV headers. |
66
+
67
+ There are no pagination flags for this list.
68
+
69
+ ## Delete
70
+
71
+ ```bash
72
+ evolve secrets delete SERVICE_TOKEN --label staging
73
+ ```
74
+
75
+ `--label <label>` selects a row when a name has multiple labels. Deletion also revokes runtime grants tied to that row. This command has no confirmation prompt.
76
+
77
+ [Global options](/cli-reference/index#global-options) apply.
@@ -0,0 +1,56 @@
1
+ ---
2
+ title: "evolve session"
3
+ description: "Inspect and share the sessions recorded by managed-agent SDK runs."
4
+ ---
5
+
6
+ Sessions come from managed-agent SDK runs. They are separate from evaluation jobs and trials.
7
+
8
+ ```bash
9
+ evolve session list
10
+ evolve session show "$SESSION_ID"
11
+ ```
12
+
13
+ Set `$SESSION_ID` to an ID from the list. These commands inspect recorded sessions; they do not start an agent. To create a session, follow the [agent SDK manual](https://github.com/evolving-machines-lab/evolve/blob/main/docs-agents/index.md).
14
+
15
+ ## Filter the list
16
+
17
+ ```bash
18
+ evolve session list --state ended --tag-prefix qa- --scope org
19
+ ```
20
+
21
+ | Option | Meaning |
22
+ | --- | --- |
23
+ | `--scope <my\|shared\|org>` | Records to list. Default `my`. |
24
+ | `--state <live\|ended>` | Select live or ended sessions. |
25
+ | `--agent <name>` | Select one agent harness. |
26
+ | `--tag-prefix <prefix>` | Select tags beginning with this text. |
27
+
28
+ Shared [list options](/cli-reference/index#list-options) also apply: `--limit`, `--cursor`, `--columns`, `--quiet`, `--no-trunc`, `--no-headers`.
29
+
30
+ ## Share a session
31
+
32
+ The creator can share by unlisted link or email.
33
+
34
+ ```bash
35
+ evolve session share "$SESSION_ID" --link
36
+ evolve session share "$SESSION_ID" --email colleague@example.com
37
+ evolve session shares "$SESSION_ID"
38
+ ```
39
+
40
+ | Option for `share` and `unshare` | Meaning |
41
+ | --- | --- |
42
+ | `--link` | Enable or disable the unlisted link. |
43
+ | `--email <address>` | Add or remove access for this address. Repeatable. |
44
+
45
+ Supply at least one. Email sharing sends the recipient a link. A session share includes the session, transcript, and trace file.
46
+
47
+ ## Remove access
48
+
49
+ ```bash
50
+ evolve session unshare "$SESSION_ID" --link
51
+ evolve session unshare "$SESSION_ID" --email colleague@example.com
52
+ ```
53
+
54
+ Revoking a link disables it. A later link share creates a new link.
55
+
56
+ [Global options and ID prefixes](/cli-reference/index) apply. `sessions` is an alias of `session`.
@@ -0,0 +1,52 @@
1
+ ---
2
+ title: "evolve skill"
3
+ description: "Upload skills for evaluated agents and reuse them by reference."
4
+ ---
5
+
6
+ Upload a skill folder, then attach it to a job.
7
+
8
+ ```bash
9
+ evolve skill upload ./my-skill
10
+ evolve run -c job.yaml --skill name:my-skill
11
+ ```
12
+
13
+ This singular `skill` group manages uploaded content. To read the CLI's bundled manual, use [`evolve skills`](/cli-reference/skills).
14
+
15
+ ## Upload
16
+
17
+ ```bash
18
+ evolve skill upload ./my-skill --org acme
19
+ ```
20
+
21
+ `--org <name>` selects the owning organization. Without it, the CLI uses its saved default, then your personal organization.
22
+
23
+ A directory containing multiple skills can return multiple upload records. Each record has two ways to reference it:
24
+
25
+ | Reference | Use when… |
26
+ | --- | --- |
27
+ | `upload:<id>` | You want this exact upload. |
28
+ | `name:<skill-name>` | You want the current upload under that name when the job is created. |
29
+
30
+ Organization members can reference uploaded skills in their jobs. Only the creator can delete an upload.
31
+
32
+ ## List and inspect
33
+
34
+ ```bash
35
+ evolve skill list --scope org
36
+ evolve skill show name:my-skill
37
+ evolve skill show "$SKILL_ID" --json
38
+ ```
39
+
40
+ `$SKILL_ID` comes from upload or list output. `show` includes metadata and `SKILL.md`.
41
+
42
+ `skill list` accepts `--scope <my|shared|org>` and the shared [list options](/cli-reference/index#list-options).
43
+
44
+ ## Delete
45
+
46
+ ```bash
47
+ evolve skill delete "$SKILL_ID"
48
+ ```
49
+
50
+ A skill referenced by a running job is refused with `skill_in_use`. Past jobs keep their resolved skill records. This command has no confirmation prompt.
51
+
52
+ [Global options](/cli-reference/index#global-options) apply. See [Skills](/core-concepts/skills) for supported sources and how jobs resolve them.
@@ -0,0 +1,85 @@
1
+ ---
2
+ title: "evolve skills"
3
+ description: "Read the bundled manual and install its small entry point for coding agents."
4
+ ---
5
+
6
+ The manual ships with the CLI. An agent can read its index, then load just the page it needs.
7
+
8
+ ```bash
9
+ evolve skills get evals
10
+ evolve skills get evals cli-reference/run
11
+ ```
12
+
13
+ ```text
14
+ Installed pointer: evolve
15
+ ↓
16
+ Documentation index
17
+ evolve skills get evals
18
+ ↓
19
+ One focused page
20
+ evolve skills get evals PAGE
21
+ ```
22
+
23
+ These commands read local package files; they do not call the Evolve API.
24
+
25
+ ## List the bundled skills
26
+
27
+ ```bash
28
+ evolve skills
29
+ evolve skills list --json
30
+ ```
31
+
32
+ | Skill | Purpose |
33
+ | --- | --- |
34
+ | `evals` | Managed evaluations: concepts, CLI, APIs, and SDKs. |
35
+ | `create-task` | Author a Harbor-format task. |
36
+ | `rewardkit` | Write a verifier with Reward Kit. |
37
+ | `create-adapter` | Convert a benchmark into tasks. |
38
+ | `publish` | Publish task datasets or import completed results. |
39
+
40
+ The root `evolve` pointer is available by explicit name but hidden from the list. The managed-agents SDK skill is separate; it is not served by this command.
41
+
42
+ ## Read a skill or page
43
+
44
+ ```bash
45
+ evolve skills get create-task
46
+ evolve skills get create-task rewardkit
47
+ evolve skills get evals core-concepts/tasks
48
+ ```
49
+
50
+ | Option | Meaning |
51
+ | --- | --- |
52
+ | `--full` | Include all files under the skill's references and templates. |
53
+ | `--all` | Read every listed skill without supplying names. |
54
+
55
+ `get <skill> <page>` accepts the page path with or without `.md`/`.mdx`. It cannot combine with `--full`. `--all` cannot combine with skill names.
56
+
57
+ Prefer the index and focused pages. Use `--full` when the complete reference is needed.
58
+
59
+ ## Find files on disk
60
+
61
+ ```bash
62
+ evolve skills path
63
+ evolve skills path evals
64
+ ```
65
+
66
+ The optional skill name selects its folder. `EVOLVE_SKILLS_DIR` can point to a different skills directory for local development.
67
+
68
+ ## Install the pointer
69
+
70
+ ```bash
71
+ evolve skills install
72
+ evolve skills install --target codex
73
+ ```
74
+
75
+ Only the small `evolve/SKILL.md` pointer is installed. The full manual stays in the CLI package.
76
+
77
+ | Option | Meaning |
78
+ | --- | --- |
79
+ | `--target <name>` | `claude`, `codex`, `cursor`, `copilot`, `gemini`, `opencode`, `agents`, or `all`. Default `all`. |
80
+ | `--path <dir>` | Your own skills directory instead of a named target. |
81
+ | `--force` | Replace an installed pointer whose contents differ. |
82
+
83
+ `all` installs into agent homes that already exist. A named target may create its directory. Existing identical pointers are left alone; differing content requires `--force`.
84
+
85
+ `--target` and `--path` cannot be combined. [Global options](/cli-reference/index#global-options), including JSON output, apply.