@evolvingmachines/evolve 0.0.65 → 0.0.67-train-harness-wave-2026-09-25.20260926.e00a8a6d

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (153) hide show
  1. package/dist/{chunk-DK2MGUPB.js → chunk-DI6LPW2B.js} +2 -2
  2. package/dist/cli/index.cjs +29 -29
  3. package/dist/cli/index.d.cts +3 -3
  4. package/dist/cli/index.d.ts +3 -3
  5. package/dist/cli/index.js +18 -18
  6. package/dist/index.cjs +64 -60
  7. package/dist/index.d.cts +297 -18
  8. package/dist/index.d.ts +297 -18
  9. package/dist/index.js +48 -44
  10. package/dist/{types-B_qmzlMk.d.cts → types-Dj0iMl5V.d.cts} +70 -90
  11. package/dist/{types-B_qmzlMk.d.ts → types-Dj0iMl5V.d.ts} +70 -90
  12. package/harness-capabilities.json +285 -0
  13. package/package.json +9 -6
  14. package/skills/create-adapter/SKILL.md +71 -26
  15. package/skills/create-task/SKILL.md +124 -66
  16. package/skills/evolve/SKILL.md +37 -20
  17. package/skills/evolve-evals/SKILL.md +142 -69
  18. package/skills/evolve-evals/references/cli-reference/agent.md +76 -0
  19. package/skills/evolve-evals/references/cli-reference/analysis.md +78 -0
  20. package/skills/evolve-evals/references/cli-reference/analyze.md +97 -0
  21. package/skills/evolve-evals/references/cli-reference/auth.md +96 -0
  22. package/skills/evolve-evals/references/cli-reference/check.md +157 -0
  23. package/skills/evolve-evals/references/cli-reference/dataset.md +154 -0
  24. package/skills/evolve-evals/references/cli-reference/filesystem.md +168 -0
  25. package/skills/evolve-evals/references/cli-reference/index.md +149 -0
  26. package/skills/evolve-evals/references/cli-reference/job.md +198 -0
  27. package/skills/evolve-evals/references/cli-reference/run.md +188 -0
  28. package/skills/evolve-evals/references/cli-reference/secrets.md +77 -0
  29. package/skills/evolve-evals/references/cli-reference/session.md +56 -0
  30. package/skills/evolve-evals/references/cli-reference/skill.md +52 -0
  31. package/skills/evolve-evals/references/cli-reference/skills.md +85 -0
  32. package/skills/evolve-evals/references/cli-reference/trial.md +115 -0
  33. package/skills/evolve-evals/references/cli-reference/upload.md +72 -0
  34. package/skills/evolve-evals/references/core-concepts/agents.md +101 -0
  35. package/skills/evolve-evals/references/core-concepts/analyze.md +143 -0
  36. package/skills/evolve-evals/references/core-concepts/check.md +116 -0
  37. package/skills/evolve-evals/references/core-concepts/compatibility.md +39 -0
  38. package/skills/evolve-evals/references/core-concepts/compose.md +103 -0
  39. package/skills/evolve-evals/references/core-concepts/datasets.md +234 -0
  40. package/skills/evolve-evals/references/core-concepts/index.md +52 -0
  41. package/skills/evolve-evals/references/core-concepts/jobs.md +223 -0
  42. package/skills/evolve-evals/references/core-concepts/models.md +73 -0
  43. package/skills/evolve-evals/references/core-concepts/multi-step.md +93 -0
  44. package/skills/evolve-evals/references/core-concepts/rewardkit.md +341 -0
  45. package/skills/evolve-evals/references/core-concepts/sandboxes.md +73 -0
  46. package/skills/evolve-evals/references/core-concepts/secrets.md +86 -0
  47. package/skills/evolve-evals/references/core-concepts/sharing.md +81 -0
  48. package/skills/evolve-evals/references/core-concepts/skills.md +100 -0
  49. package/skills/evolve-evals/references/core-concepts/task-config.md +169 -0
  50. package/skills/evolve-evals/references/core-concepts/task-environment.md +127 -0
  51. package/skills/evolve-evals/references/core-concepts/task-reference.md +191 -0
  52. package/skills/evolve-evals/references/core-concepts/task-verifiers.md +209 -0
  53. package/skills/evolve-evals/references/core-concepts/tasks.md +90 -0
  54. package/skills/evolve-evals/references/core-concepts/teams.md +128 -0
  55. package/skills/evolve-evals/references/core-concepts/trial-outputs.md +225 -0
  56. package/skills/evolve-evals/references/core-concepts/trials.md +129 -0
  57. package/skills/evolve-evals/references/core-concepts/upload-sdk-session.md +240 -0
  58. package/skills/evolve-evals/references/core-concepts/upload.md +166 -0
  59. package/skills/evolve-evals/references/dashboard/checks.md +54 -0
  60. package/skills/evolve-evals/references/dashboard/datasets.md +65 -0
  61. package/skills/evolve-evals/references/dashboard/jobs.md +106 -0
  62. package/skills/evolve-evals/references/dashboard/traces.md +54 -0
  63. package/skills/evolve-evals/references/dashboard/trial-viewer.md +77 -0
  64. package/skills/evolve-evals/references/getting-started/first-task.md +228 -0
  65. package/skills/evolve-evals/references/getting-started/installation.md +79 -0
  66. package/skills/evolve-evals/references/getting-started/quick-start.md +100 -0
  67. package/skills/evolve-evals/references/index.md +58 -0
  68. package/skills/evolve-evals/references/sdk/http.md +110 -0
  69. package/skills/evolve-evals/references/sdk/python.md +108 -0
  70. package/skills/evolve-evals/references/sdk/typescript.md +104 -0
  71. package/skills/evolve-evals/references/sdk-reference/agents.md +75 -0
  72. package/skills/evolve-evals/references/sdk-reference/analyses.md +125 -0
  73. package/skills/evolve-evals/references/sdk-reference/auth.md +171 -0
  74. package/skills/evolve-evals/references/sdk-reference/checks.md +146 -0
  75. package/skills/evolve-evals/references/sdk-reference/datasets.md +170 -0
  76. package/skills/evolve-evals/references/sdk-reference/error-codes.md +231 -0
  77. package/skills/evolve-evals/references/sdk-reference/errors.md +98 -0
  78. package/skills/evolve-evals/references/sdk-reference/filesystem.md +166 -0
  79. package/skills/evolve-evals/references/sdk-reference/index.md +161 -0
  80. package/skills/evolve-evals/references/sdk-reference/jobs.md +250 -0
  81. package/skills/evolve-evals/references/sdk-reference/meta.md +54 -0
  82. package/skills/evolve-evals/references/sdk-reference/methods/agents.md +154 -0
  83. package/skills/evolve-evals/references/sdk-reference/methods/analyses.md +211 -0
  84. package/skills/evolve-evals/references/sdk-reference/methods/auth.md +257 -0
  85. package/skills/evolve-evals/references/sdk-reference/methods/checks.md +462 -0
  86. package/skills/evolve-evals/references/sdk-reference/methods/datasets.md +687 -0
  87. package/skills/evolve-evals/references/sdk-reference/methods/filesystem.md +527 -0
  88. package/skills/evolve-evals/references/sdk-reference/methods/jobs.md +1023 -0
  89. package/skills/evolve-evals/references/sdk-reference/methods/meta.md +213 -0
  90. package/skills/evolve-evals/references/sdk-reference/methods/secrets.md +157 -0
  91. package/skills/evolve-evals/references/sdk-reference/methods/skills.md +125 -0
  92. package/skills/evolve-evals/references/sdk-reference/methods/trials.md +371 -0
  93. package/skills/evolve-evals/references/sdk-reference/secrets.md +104 -0
  94. package/skills/evolve-evals/references/sdk-reference/skills.md +66 -0
  95. package/skills/evolve-evals/references/sdk-reference/trials.md +164 -0
  96. package/skills/evolve-evals/references/sdk-reference/types.md +1255 -0
  97. package/skills/publish/SKILL.md +103 -79
  98. package/skills/rewardkit/SKILL.md +78 -23
  99. package/spec/openapi.yaml +164 -160
  100. package/skills/evolve-evals/references/cli-reference/agent.mdx +0 -79
  101. package/skills/evolve-evals/references/cli-reference/analysis.mdx +0 -160
  102. package/skills/evolve-evals/references/cli-reference/analyze.mdx +0 -84
  103. package/skills/evolve-evals/references/cli-reference/auth.mdx +0 -133
  104. package/skills/evolve-evals/references/cli-reference/check.mdx +0 -263
  105. package/skills/evolve-evals/references/cli-reference/dataset.mdx +0 -151
  106. package/skills/evolve-evals/references/cli-reference/index.mdx +0 -108
  107. package/skills/evolve-evals/references/cli-reference/job.mdx +0 -278
  108. package/skills/evolve-evals/references/cli-reference/run.mdx +0 -204
  109. package/skills/evolve-evals/references/cli-reference/secrets.mdx +0 -95
  110. package/skills/evolve-evals/references/cli-reference/skill.mdx +0 -61
  111. package/skills/evolve-evals/references/cli-reference/skills.mdx +0 -81
  112. package/skills/evolve-evals/references/cli-reference/trial.mdx +0 -181
  113. package/skills/evolve-evals/references/cli-reference/upload.mdx +0 -36
  114. package/skills/evolve-evals/references/core-concepts/agents.mdx +0 -106
  115. package/skills/evolve-evals/references/core-concepts/analyze.mdx +0 -86
  116. package/skills/evolve-evals/references/core-concepts/check.mdx +0 -64
  117. package/skills/evolve-evals/references/core-concepts/datasets.mdx +0 -138
  118. package/skills/evolve-evals/references/core-concepts/index.mdx +0 -42
  119. package/skills/evolve-evals/references/core-concepts/jobs.mdx +0 -151
  120. package/skills/evolve-evals/references/core-concepts/models.mdx +0 -68
  121. package/skills/evolve-evals/references/core-concepts/sandboxes.mdx +0 -52
  122. package/skills/evolve-evals/references/core-concepts/secrets.mdx +0 -41
  123. package/skills/evolve-evals/references/core-concepts/sharing.mdx +0 -35
  124. package/skills/evolve-evals/references/core-concepts/skills.mdx +0 -42
  125. package/skills/evolve-evals/references/core-concepts/tasks.mdx +0 -216
  126. package/skills/evolve-evals/references/core-concepts/teams.mdx +0 -36
  127. package/skills/evolve-evals/references/core-concepts/trial-outputs.mdx +0 -117
  128. package/skills/evolve-evals/references/core-concepts/trials.mdx +0 -103
  129. package/skills/evolve-evals/references/core-concepts/upload.mdx +0 -93
  130. package/skills/evolve-evals/references/dashboard/checks.mdx +0 -21
  131. package/skills/evolve-evals/references/dashboard/datasets.mdx +0 -24
  132. package/skills/evolve-evals/references/dashboard/jobs.mdx +0 -65
  133. package/skills/evolve-evals/references/dashboard/traces.mdx +0 -17
  134. package/skills/evolve-evals/references/dashboard/trial-viewer.mdx +0 -30
  135. package/skills/evolve-evals/references/getting-started/installation.mdx +0 -56
  136. package/skills/evolve-evals/references/getting-started/quick-start.mdx +0 -94
  137. package/skills/evolve-evals/references/index.mdx +0 -48
  138. package/skills/evolve-evals/references/sdk/python.mdx +0 -161
  139. package/skills/evolve-evals/references/sdk/typescript.mdx +0 -163
  140. package/skills/evolve-evals/references/sdk-reference/agents.mdx +0 -108
  141. package/skills/evolve-evals/references/sdk-reference/analyses.mdx +0 -222
  142. package/skills/evolve-evals/references/sdk-reference/auth.mdx +0 -116
  143. package/skills/evolve-evals/references/sdk-reference/checks.mdx +0 -300
  144. package/skills/evolve-evals/references/sdk-reference/datasets.mdx +0 -346
  145. package/skills/evolve-evals/references/sdk-reference/errors.mdx +0 -210
  146. package/skills/evolve-evals/references/sdk-reference/index.mdx +0 -60
  147. package/skills/evolve-evals/references/sdk-reference/jobs.mdx +0 -822
  148. package/skills/evolve-evals/references/sdk-reference/meta.mdx +0 -52
  149. package/skills/evolve-evals/references/sdk-reference/skills.mdx +0 -69
  150. package/skills/evolve-evals/references/sdk-reference/trials.mdx +0 -302
  151. package/skills/evolve-evals/references/sdk-reference/types.mdx +0 -274
  152. package/skills/evolve-evals/references/snippets/global-options.mdx +0 -13
  153. package/skills/evolve-evals/references/snippets/list-flags.mdx +0 -23
@@ -0,0 +1,166 @@
1
+ ---
2
+ title: "Filesystems and downloads"
3
+ description: "Choose the right files: saved results, a sandbox, or a task package."
4
+ ---
5
+
6
+ **Note:**
7
+
8
+ For every method’s inputs, return values, and language differences, see the [filesystem method reference](/sdk-reference/methods/filesystem).
9
+
10
+ Evolve exposes three file surfaces. They answer different questions.
11
+
12
+ | Need | Client |
13
+ | --- | --- |
14
+ | Read a trial's saved `result.json`, logs, or collected artifact | `trials.files()` / `trials.file()` |
15
+ | Browse sandbox paths during or after a run | `filesystem()` |
16
+ | Inspect the task files supplied to a run | `datasets.taskFiles()` / `task_files()` |
17
+
18
+ ## Open a run's filesystem
19
+
20
+ ```ts TypeScript
21
+ import { trials } from "@evolvingmachines/evolve";
22
+
23
+ const fs = trials().filesystem(trialId);
24
+ const status = await fs.status();
25
+ const folder = await fs.list({ path: status.work_dir });
26
+ console.log(status.state, folder.entries);
27
+ ```
28
+
29
+ ```python Python
30
+ from evolve import trials
31
+
32
+ fs = trials().filesystem(trial_id)
33
+ status = await fs.status()
34
+ folder = await fs.list(path=status.work_dir)
35
+ print(status.state, folder.entries)
36
+ ```
37
+
38
+ The same methods are available from:
39
+
40
+ ### TypeScript
41
+
42
+ | Run | Client |
43
+ | --- | --- |
44
+ | Trial | `trials().filesystem(trialId)` |
45
+ | Analysis | `analyses().filesystem(analysisId)` |
46
+ | Task check | `checks().taskFilesystem(checkId, taskCheckId)` |
47
+
48
+ ### Python
49
+
50
+ | Run | Client |
51
+ | --- | --- |
52
+ | Trial | `trials().filesystem(trial_id)` |
53
+ | Analysis | `analyses().filesystem(analysis_id)` |
54
+ | Task check | `checks().task_filesystem(check_id, task_check_id)` |
55
+
56
+ ### What source will answer?
57
+
58
+ ```text
59
+ 1. Read status
60
+ ├─ live → sandbox
61
+ ├─ captured → saved files
62
+ ├─ capturing → preparing files
63
+ └─ none → unavailable
64
+
65
+ 2. Require a source (optional)
66
+ ├─ "live" → sandbox only
67
+ └─ "capture" → saved files only
68
+ ```
69
+
70
+ Omit `source` to prefer live files, then a capture. A missing required source raises `filesystem_state`.
71
+
72
+ **Note:**
73
+
74
+ A captured listing can include unchanged image files whose bytes were not saved. Check `entry.captured`; reading one with no stored bytes raises `not_captured`. A capture is not a complete disk image.
75
+
76
+ ## Browse, read, search
77
+
78
+ ```ts TypeScript
79
+ const bytes = await fs.read("/app/report.json");
80
+ const hits = await fs.search({ q: "error", path: "/app" });
81
+ const changes = await fs.changes({ phase: "agent" });
82
+ ```
83
+
84
+ ```python Python
85
+ content = await fs.read("/app/report.json")
86
+ hits = await fs.search("error", path="/app")
87
+ changes = await fs.changes(phase="agent")
88
+ ```
89
+
90
+ These paths are examples. Use `status.work_dir` and a listing to find the run's actual files.
91
+
92
+ | Method | Options | Result |
93
+ | --- | --- | --- |
94
+ | `status()` | None | State, box, work directory, watcher, capture summary |
95
+ | `list(...)` | `path`, `source`, `cursor`, `limit` | `entries`, `next_cursor` |
96
+ | `read(path, ...)` | `source`, byte range | `Buffer` / `bytes` |
97
+ | `search(...)` | Query, `path`, `regex`, `limit`, `source` | Hits and `truncated` flag |
98
+ | `changes(...)` | `source`, `phase`, `cursor`, `limit` | Changed files and next cursor |
99
+ | `archive(...)` | `path`, `source`, download options | Archive bytes or saved path |
100
+
101
+ TypeScript search takes `{q, ...}`; Python takes `q` as its first argument. Change phases are `setup`, `agent`, `verifier`, or `all`.
102
+
103
+ Byte ranges: TypeScript `{range: {start, end}}` or `{range: {suffix}}`; Python `start=`, `end=`, or `suffix=`. End is inclusive. Suffix selects the last N bytes.
104
+
105
+ List/change pages default to 500 rows; search defaults to 200. Their maximum is 1,000. A `truncated` search should be narrowed before treating it as complete.
106
+
107
+ ## Follow changes and logs
108
+
109
+ ```ts TypeScript
110
+ await fs.watch(["/app"]);
111
+ for await (const event of fs.events()) {
112
+ console.log(event.event, event.data);
113
+ }
114
+ ```
115
+
116
+ ```python Python
117
+ await fs.watch(["/app"])
118
+ async for event in fs.events():
119
+ print(event.event, event.data)
120
+ ```
121
+
122
+ `watch(paths)` registers up to eight folders. `events()` yields filesystem, state, and heartbeat frames.
123
+
124
+ | Read | TypeScript | Python |
125
+ | --- | --- | --- |
126
+ | One log page | `logs({ stream: "agent", cursor, limit })` | `logs("agent", cursor=..., limit=...)` |
127
+ | Log events | `logEvents(options)` | `log_events(...)` |
128
+ | Live processes | `procs()` | `procs()` |
129
+
130
+ Log streams are `agent`, `verifier`, `system`, `setup`, and `metrics`. System logs require `system_log` at job creation. A stream can be empty with a `reason`; reserved or unavailable streams do not imply missing agent work.
131
+
132
+ Filesystem and log event streams connect once. To reconnect, retain the last event id and pass `lastEventId` / `last_event_id`. TypeScript also accepts `signal`. This is separate from the automatic reconnect behavior of `jobs.watch()`.
133
+
134
+ ## Task package files
135
+
136
+ ```ts TypeScript
137
+ import { datasets } from "@evolvingmachines/evolve";
138
+
139
+ const files = datasets().taskFiles("harbor-examples@1.0", "hello-world");
140
+ const tree = await files.list();
141
+ const instruction = await files.read("instruction.md");
142
+ ```
143
+
144
+ ```python Python
145
+ from evolve import datasets
146
+
147
+ files = datasets().task_files("harbor-examples@1.0", "hello-world")
148
+ tree = await files.list()
149
+ instruction = await files.read("instruction.md")
150
+ ```
151
+
152
+ The dataset reference must pin a version. This client provides `status()`, `list()`, and `read(path)`. List options are `path`, `cursor`, and `limit`: one object in TypeScript, keyword arguments in Python. Reads accept the same byte-range options as above. There is no live source, process list, or log stream.
153
+
154
+ ## Download options
155
+
156
+ Dataset, job, analysis, and check downloads use the same delivery choices. Run-filesystem `archive()` does too.
157
+
158
+ | Choice | TypeScript | Python |
159
+ | --- | --- | --- |
160
+ | Bytes in memory | No options → `Buffer` | No `to` → `bytes` |
161
+ | Save to a directory | `{ to: "./results" }` → path | `to="./results"` → path |
162
+ | Raw response stream | `{ stream: true }` | Not exposed |
163
+
164
+ `to` is a directory. The client chooses the archive filename from the response and returns the saved path.
165
+
166
+ Buffer/bytes and saved-file downloads check declared length and digest when supplied. Saved files appear at their final path only after verification. TypeScript raw streams leave integrity checking to the caller.
@@ -0,0 +1,161 @@
1
+ ---
2
+ title: "SDK clients"
3
+ description: "One configuration. Focused clients for jobs, datasets, trials, and results."
4
+ ---
5
+
6
+ Use the same managed evaluation service from TypeScript or Python. Evolve runs the jobs; your program submits work and reads the results.
7
+
8
+ **Note:**
9
+
10
+ To run agents directly with the Evolve builder, Swarm, or Pipeline, use the separate [agent SDK manual](https://github.com/evolving-machines-lab/evolve/blob/main/docs-agents/index.md).
11
+
12
+ ```ts TypeScript
13
+ import { hosted } from "@evolvingmachines/evolve";
14
+
15
+ const evolve = hosted(); // Reads EVOLVE_API_KEY.
16
+ const page = await evolve.jobs.list();
17
+ ```
18
+
19
+ ```python Python
20
+ from evolve import hosted
21
+
22
+ evolve = hosted() # Reads EVOLVE_API_KEY.
23
+ page = await evolve.jobs.list()
24
+ ```
25
+
26
+ Python examples in this reference belong inside an async function. The [Python guide](/sdk/python) includes a complete runnable program.
27
+
28
+ ## Choose a client
29
+
30
+ **[Jobs](/sdk-reference/jobs)**
31
+
32
+ Start, follow, compare, retry, and download evaluations.
33
+
34
+ **[Trials](/sdk-reference/trials)**
35
+
36
+ Inspect one attempt: reward, trace, and stored files.
37
+
38
+ **[Datasets](/sdk-reference/datasets)**
39
+
40
+ Browse tasks, publish versions, and inspect builds.
41
+
42
+ **[Filesystems](/sdk-reference/filesystem)**
43
+
44
+ Read live sandbox files and saved captures.
45
+
46
+ **[Analyses](/sdk-reference/analyses)**
47
+
48
+ Read the analyzer's results and saved runs.
49
+
50
+ **[Checks](/sdk-reference/checks)**
51
+
52
+ Check task quality before running a benchmark.
53
+
54
+ Also available: [agents](/sdk-reference/agents), [uploaded skills](/sdk-reference/skills), [secrets](/sdk-reference/secrets), [identity and teams](/sdk-reference/auth), and [platform capabilities](/sdk-reference/meta).
55
+
56
+ ## Look up a method
57
+
58
+ Use the client guides above for a workflow. Open a method reference for its inputs, returned fields, and Python and TypeScript examples.
59
+
60
+ | Client | Method reference |
61
+ | --- | --- |
62
+ | Jobs | [Start, watch, compare, retry, and import](/sdk-reference/methods/jobs) |
63
+ | Trials | [Read attempts, traces, and artifacts](/sdk-reference/methods/trials) |
64
+ | Filesystems | [Read files, search, and follow live changes](/sdk-reference/methods/filesystem) |
65
+ | Analyses | [Read analysis runs and outputs](/sdk-reference/methods/analyses) |
66
+ | Checks | [Create and inspect task quality checks](/sdk-reference/methods/checks) |
67
+ | Datasets | [Browse, publish, and manage versions](/sdk-reference/methods/datasets) |
68
+ | Agents | [Register and manage harnesses](/sdk-reference/methods/agents) |
69
+ | Skills | [Upload and manage run context](/sdk-reference/methods/skills) |
70
+ | Secrets | [Store credentials and read metadata](/sdk-reference/methods/secrets) |
71
+ | Identity and teams | [Read identity and manage membership](/sdk-reference/methods/auth) |
72
+ | Capabilities | [Read supported options and limits](/sdk-reference/methods/meta) |
73
+
74
+ For failures, start with [error handling](/sdk-reference/errors) or find the exact name in the [error-code reference](/sdk-reference/error-codes).
75
+
76
+ ## Configure once
77
+
78
+ ```ts TypeScript
79
+ import { hosted } from "@evolvingmachines/evolve";
80
+
81
+ const evolve = hosted({
82
+ apiKey: process.env.EVOLVE_API_KEY,
83
+ org: "my-team",
84
+ });
85
+ ```
86
+
87
+ ```python Python
88
+ import os
89
+ from evolve import HostedClientConfig, hosted
90
+
91
+ evolve = hosted(HostedClientConfig(
92
+ api_key=os.environ["EVOLVE_API_KEY"],
93
+ org="my-team",
94
+ ))
95
+ ```
96
+
97
+ | Setting | TypeScript | Python | Default |
98
+ | --- | --- | --- | --- |
99
+ | API key | `apiKey` | `api_key` | `EVOLVE_API_KEY` |
100
+ | API origin | `baseUrl` | `base_url` | `EVOLVE_DASHBOARD_URL`, then the Evolve dashboard |
101
+ | Owning team | `org` | `org` | Your personal organization |
102
+
103
+ The `org` default applies when you create jobs, publish datasets, register agents, or upload skills. A call's own `org` wins. It does not filter list requests.
104
+
105
+ Prefer a single client when useful: `jobs(config)`, `trials(config)`, and the other factories accept the same configuration. Import `auth` and `managedSecrets` / `managed_secrets` separately; they are not properties of `hosted()`.
106
+
107
+ **Tip:**
108
+
109
+ `meta()` needs no key. Other clients authenticate with your API key. TypeScript checks for a key when the client is created; Python checks when it makes an authenticated request.
110
+
111
+ ## One page or every item
112
+
113
+ ```ts TypeScript
114
+ // One page.
115
+ const page = await evolve.jobs.list({ limit: 20 });
116
+ console.log(page.items, page.nextCursor, page.hasMore);
117
+
118
+ // All pages, one job at a time.
119
+ for await (const job of evolve.jobs.list()) {
120
+ console.log(job.id, job.status);
121
+ }
122
+ ```
123
+
124
+ ```python Python
125
+ # One page.
126
+ page = await evolve.jobs.list(limit=20)
127
+ print(page.items, page.next_cursor, page.has_more)
128
+
129
+ # All pages, one job at a time.
130
+ async for job in evolve.jobs.list():
131
+ print(job.id, job.status)
132
+ ```
133
+
134
+ This pattern applies to list handles, including `jobs.trials()` and `jobs.tasks()`. A trace page, stored-file page, grep result, or dataset's nested task page is a single page; follow its cursor yourself or use the dedicated iterator where available.
135
+
136
+ ## Language differences
137
+
138
+ | Detail | TypeScript | Python |
139
+ | --- | --- | --- |
140
+ | Calls | Input/options objects | Keyword arguments |
141
+ | Most domain fields | `model_name`, `cost_usd` | Same spelling |
142
+ | Returned entities | Objects | Dataclasses; stats, analyses, and checks are dictionaries |
143
+ | Download to memory | `Buffer` | `bytes` |
144
+ | Download to directory | `{ to: "./results" }` | `to="./results"` |
145
+ | Raw download stream | `{ stream: true }` | Not exposed |
146
+ | Wait cancellation | `AbortSignal` on watch methods | `timeout_s` on job/import/check watches |
147
+
148
+ See [types](/sdk-reference/types) for field access and [errors](/sdk-reference/errors) for request failures.
149
+
150
+ ### Python client lifetime
151
+
152
+ Hosted clients support `async with` and `await client.close()`:
153
+
154
+ ```python
155
+ from evolve import hosted
156
+
157
+ async with hosted() as client:
158
+ page = await client.jobs.list(limit=20)
159
+ ```
160
+
161
+ The individual hosted clients currently have no resources to release in `close()`. Closing a client does not cancel an evaluation. Use `jobs().cancel(job_id)` to stop a job.
@@ -0,0 +1,250 @@
1
+ ---
2
+ title: "Jobs"
3
+ description: "Submit evaluations, follow progress, and work with results."
4
+ ---
5
+
6
+ **Note:**
7
+
8
+ For every method’s inputs, return values, and language differences, see the [jobs method reference](/sdk-reference/methods/jobs).
9
+
10
+ A job runs selected tasks against one or more agent/model combinations. Each combination is an **arm**. Each attempt is a **trial**.
11
+
12
+ ```text
13
+ Selected tasks
14
+ × Agent/model arms
15
+ × Attempts
16
+ = Trials
17
+ ```
18
+
19
+ ## Start a job
20
+
21
+ ```ts TypeScript
22
+ import { jobs } from "@evolvingmachines/evolve";
23
+
24
+ const client = jobs();
25
+ const job = await client.start({
26
+ datasets: [{
27
+ name: "harbor-examples",
28
+ version: "1.0",
29
+ task_names: ["hello-world"],
30
+ }],
31
+ agents: [{ name: "codex", model_name: "gpt-5.6-luna" }],
32
+ max_trial_spend_usd: 0.50,
33
+ retry: { max_retries: 0 },
34
+ });
35
+ ```
36
+
37
+ ```python Python
38
+ from evolve import jobs
39
+
40
+ client = jobs()
41
+ job = await client.start(
42
+ datasets=[{
43
+ "name": "harbor-examples",
44
+ "version": "1.0",
45
+ "task_names": ["hello-world"],
46
+ }],
47
+ agents=[{"name": "codex", "model_name": "gpt-5.6-luna"}],
48
+ max_trial_spend_usd=0.50,
49
+ retry={"max_retries": 0},
50
+ )
51
+ ```
52
+
53
+ `start` returns the accepted `Job`. Read `job.id` to address it later.
54
+
55
+ ### Required inputs
56
+
57
+ | Input | Fields |
58
+ | --- | --- |
59
+ | `datasets` | One or more selectors. Required: `name`. Optional: `version`, `task_names`, `exclude_task_names`, `n_tasks`. Omitted version selects the active version. Filters accept glob patterns; the cap applies after filters. |
60
+ | `agents` | One or more arms. Required: `name`, `model_name`. Optional: `version`, `reasoning_effort`, `kwargs`, `preset`, `skills`. Use [capabilities](/sdk-reference/meta) for supported values. |
61
+
62
+ Python also accepts `DatasetSelector` and `AgentArm` dataclass instances in these lists.
63
+
64
+ ### Run settings
65
+
66
+ | Option | Meaning | When omitted |
67
+ | --- | --- | --- |
68
+ | `job_name` | A label for the run | Server-generated name |
69
+ | `org` | Owning organization slug or id | Client default, then personal organization |
70
+ | `n_attempts` | Attempts per task/arm; integer ≥1 | `1` |
71
+ | `n_concurrent_trials` | Requested parallel trials; `1`–`150` | `4` |
72
+ | `sandbox_provider` | Requested `e2b`, `daytona`, or `modal` | Current platform default |
73
+ | `max_trial_spend_usd` | Agent model-spend cap per trial attempt; must be >0 | Current platform default; fallback `$200` |
74
+ | `retry` | Automatic infrastructure retry policy | Current platform policy; fallback two retries |
75
+ | `analyze` | [Analyze each settled trial](/sdk-reference/analyses#start-an-analysis); `{}` enables defaults | Off |
76
+ | `system_log` | Capture sandbox system logs | Off |
77
+ | `secrets` | Stored secret references or inline values | No extra job secrets |
78
+ | `verifier_env` | Judge overrides: `REWARDKIT_JUDGE`, `REWARDKIT_MODEL` only | Task/default judge settings |
79
+
80
+ Organization and fleet capacity also constrain parallel work. A task may use another provider when the requested provider cannot support its requirements; read the trial for its actual provider.
81
+
82
+ ### Timeout multipliers
83
+
84
+ All values are finite numbers greater than zero. They multiply task timeouts; they are not durations in seconds.
85
+
86
+ | Field | Applies to |
87
+ | --- | --- |
88
+ | `timeout_multiplier` | All phases without a specific override; default `1` |
89
+ | `agent_timeout_multiplier` | Agent execution |
90
+ | `verifier_timeout_multiplier` | Verification |
91
+ | `agent_setup_timeout_multiplier` | Harness installation/setup |
92
+ | `environment_build_timeout_multiplier` | Environment build |
93
+
94
+ A phase override replaces the global multiplier for that phase. Values that exceed the runtime's timer limit for the selected tasks are refused.
95
+
96
+ ### Automatic retry policy
97
+
98
+ | Field | Rule |
99
+ | --- | --- |
100
+ | `max_retries` | Integer ≥0. `0` disables configured infrastructure retries. |
101
+ | `include_exceptions` | Exception names to include. Omitted, null, or `[]` means no include filter. |
102
+ | `exclude_exceptions` | Exclusions win over includes. Omitted keeps defaults; null or `[]` removes them. |
103
+ | `wait_multiplier` | Backoff multiplier; default `1` |
104
+ | `min_wait_sec` | Initial wait; default `1` second |
105
+ | `max_wait_sec` | Maximum wait; default `60` seconds |
106
+
107
+ Only eligible infrastructure failures enter this retry policy. Scoring errors and low rewards do not automatically rerun the agent. Each retry receives a fresh per-trial cap.
108
+
109
+ Provider capacity refusals have a separate bounded wait policy. They can wait even when `max_retries` is zero.
110
+
111
+ ### Idempotency and unsupported input
112
+
113
+ Supply an idempotency key when repeating a submission must return the same accepted job:
114
+
115
+ Here, `config` is the job input shown above. `requestId` / `request_id` is a stable string you assign to this submission.
116
+
117
+ ```ts TypeScript
118
+ const job = await client.start(config, { idempotencyKey: requestId });
119
+ ```
120
+
121
+ ```python Python
122
+ job = await client.start(**config, idempotency_key=request_id)
123
+ ```
124
+
125
+ Use one stable request id for the same resolved request. Reusing it for a different request is refused.
126
+
127
+ `agent_env` appears in SDK signatures, but the current platform refuses nonempty values. Use [job secrets](/sdk-reference/secrets) for credentials or [registered-agent environment settings](/sdk-reference/agents) for a custom harness.
128
+
129
+ ## Wait or show progress
130
+
131
+ ```ts TypeScript
132
+ const finished = await client.watch(job.id, {
133
+ onEvent: (event) => console.log(event.type),
134
+ });
135
+ ```
136
+
137
+ ```python Python
138
+ finished = await client.watch(
139
+ job.id,
140
+ on_event=lambda event: print(event.type),
141
+ )
142
+ ```
143
+
144
+ The same handle can instead be consumed with `for await` / `async for` to receive events. Choose one style per handle. Job watches replay history and reconnect from the last event id.
145
+
146
+ | Watch option | TypeScript | Python |
147
+ | --- | --- | --- |
148
+ | Event callback | `onEvent` | `on_event` |
149
+ | Initial reconnect delay | `reconnectDelayMs` (default `1000`) | `reconnect_delay_s` (default `1`) |
150
+ | Maximum reconnect delay | `maxReconnectDelayMs` (default `30000`) | `max_reconnect_delay_s` (default `30`) |
151
+ | Stop waiting | `signal` | `timeout_s` |
152
+
153
+ A terminal job has finished scheduling/executing its trials. **Completed does not mean every trial passed.**
154
+
155
+ ## Read and compare
156
+
157
+ ```ts TypeScript
158
+ const job = await client.get(jobId);
159
+ for await (const trial of client.trials(jobId, { status: ["SCORING_ERROR"] })) {
160
+ console.log(trial.id, trial.exception_info);
161
+ }
162
+ const tasks = await client.tasks(jobId);
163
+ const comparison = await client.compare([firstJobId, secondJobId]);
164
+ ```
165
+
166
+ ```python Python
167
+ job = await client.get(job_id)
168
+ async for trial in client.trials(job_id, status=["SCORING_ERROR"]):
169
+ print(trial.id, trial.exception_info)
170
+ tasks = await client.tasks(job_id)
171
+ comparison = await client.compare([first_job_id, second_job_id])
172
+ ```
173
+
174
+ Use full ids returned by create/list calls.
175
+
176
+ | Method | Options | Returns |
177
+ | --- | --- | --- |
178
+ | `get(id)` | None | `Job` |
179
+ | `list(...)` | `search`, `scope`, `kind`, `limit`, `cursor` | Paginated handle |
180
+ | `trials(id, ...)` | `status` list, `dataset`, `limit`, `cursor` | Trial handle |
181
+ | `tasks(id, ...)` | `limit`, `cursor` | Per-task rollup handle |
182
+ | `compare(ids)` | 2–10 distinct readable job ids | Job aggregates and task matrix |
183
+ | `grep(id, q, ...)` | `type`, `limit`, `cursor` | One page of matches grouped by trial |
184
+
185
+ List scope is `my`, `shared`, or `org`. Job list kind is `job` (default), `check`, or `all`; check rows have a different shape. Branch on `row.kind` before reading job-only fields.
186
+
187
+ `grep` searches parsed traces using a case-insensitive POSIX regular expression. See [trial traces](/sdk-reference/trials#read-the-trace) for per-trial filtering.
188
+
189
+ ## Cancel or run selected work again
190
+
191
+ | Action | TypeScript | Python | Result |
192
+ | --- | --- | --- | --- |
193
+ | Cancel remaining work | `cancel(id)` | `cancel(id)` | Current job |
194
+ | Resume failed/stopped work | `resume(id, { filter_error_types })` | `resume(id, filter_error_types=[...])` | New linked job |
195
+ | Retry selected settled trials | `retry(id, { trial_ids })` | `retry(id, trial_ids=[...])` | New linked job |
196
+ | Retry failed trials | `retry(id, { failed_only: true })` | `retry(id, failed_only=True)` | New linked job |
197
+ | Retry all trials of a terminal job | `retry(id)` | `retry(id)` | New linked job |
198
+ | Re-run eligible verifiers | `regrade(id, { statuses, task_name })` | `regrade(id, statuses=[...], task_name=...)` | New linked job |
199
+
200
+ Resume requires a terminal source job. Explicit retry ids may select settled trials from a job still running. Do not combine `trial_ids` with `failed_only`.
201
+
202
+ Resume and retry accept the same idempotency option as start: a final options object in TypeScript, `idempotency_key` in Python. Regrade has no such SDK option.
203
+
204
+ Regrading uses the recorded verifier snapshot; it cannot replace the verifier or use edited tests. It needs retained inputs from an eligible separate verifier. Shared-verifier, judge-based, multi-step, and uploaded runs cannot be regraded. See [jobs](/core-concepts/jobs) for eligibility.
205
+
206
+ ## Analyze traces
207
+
208
+ `analyze(id, ...)` starts a wave and returns immediately. `watchAnalysis` / `watch_analysis` waits for that wave. Configuration and examples are on [analyses](/sdk-reference/analyses).
209
+
210
+ For a job created with embedded analysis, wait for the job first, then its analyses. A still-running job can temporarily have zero pending analyses.
211
+
212
+ ## Download or import results
213
+
214
+ ```ts TypeScript
215
+ const path = await client.download(jobId, { to: "./results" });
216
+ const imported = await client.upload("./local-job");
217
+ const settled = await client.watchImport(imported.id);
218
+ console.log(settled.status, settled.job_id, settled.failure);
219
+ ```
220
+
221
+ ```python Python
222
+ path = await client.download(job_id, to="./results")
223
+ imported = await client.upload("./local-job")
224
+ settled = await client.watch_import(imported.id)
225
+ print(settled.status, settled.job_id, settled.failure)
226
+ ```
227
+
228
+ Download requires a terminal job. Without `to`, it returns a `Buffer` / `bytes`; TypeScript also supports `{stream: true}`. See [transfer behavior](/sdk-reference/filesystem#download-options).
229
+
230
+ | Import method | Input/options |
231
+ | --- | --- |
232
+ | `upload(source, ...)` | Local job directory or archive path. TypeScript also accepts `{archive_url}`; Python takes `archive_url=` instead of a path. |
233
+ | Upload options | Optional `dataset`, upload progress callback, and import-registration callback. |
234
+ | `getImport` / `get_import` | Import id |
235
+ | `listImports` / `list_imports` | `status`, `limit`, `cursor` |
236
+ | `watchImport` / `watch_import` | TS: `onStatus`, `onProgress`, `pollIntervalMs`, `signal`. Python: `on_status`, `on_progress`, `poll_interval_s`, `timeout_s`. |
237
+
238
+ Callback names and timing use the same [upload/watch conventions as datasets](/sdk-reference/datasets#follow-a-publish). Uploaded results become a terminal record. They can be analyzed, but cannot be resumed, retried, or regraded.
239
+
240
+ ## Share and delete
241
+
242
+ | Action | TypeScript | Python |
243
+ | --- | --- | --- |
244
+ | Enable an unlisted link | `share(id, { link: true })` | `share(id, link=True)` |
245
+ | Share with people | `share(id, { emails: ["person@example.com"] })` | `share(id, emails=["person@example.com"])` |
246
+ | Revoke either form | `unshare(id, { link, emails })` | `unshare(id, link=..., emails=...)` |
247
+ | Read share state | `shares(id)` | `shares(id)` |
248
+ | Permanently delete | `delete(id)` | `delete(id)` |
249
+
250
+ Sharing can send invitation email. Only the creator can manage shares or delete the job; team membership alone does not grant those actions. Deletion requires a terminal job with no running analysis or dependent regrade. The returned receipt counts deleted trials and analyses.
@@ -0,0 +1,54 @@
1
+ ---
2
+ title: "Platform capabilities"
3
+ description: "Discover available harnesses, models, providers, and current limits."
4
+ ---
5
+
6
+ [Method reference: calls, parameters, and response fields](/sdk-reference/methods/meta).
7
+
8
+ `meta()` returns the platform's capability document. It needs no API key.
9
+
10
+ ```ts TypeScript
11
+ import { meta } from "@evolvingmachines/evolve";
12
+
13
+ const capabilities = await meta();
14
+ for (const agent of capabilities.agents) {
15
+ console.log(agent.name, agent.runnable, agent.models.map((m) => m.alias));
16
+ }
17
+ ```
18
+
19
+ ```python Python
20
+ from evolve import meta
21
+
22
+ capabilities = await meta()
23
+ for agent in capabilities.agents:
24
+ print(agent.name, agent.runnable, [m.alias for m in agent.models])
25
+ ```
26
+
27
+ Also available as `hosted().meta()` in both languages.
28
+
29
+ ## What to read
30
+
31
+ | Field | Use it to answer |
32
+ | --- | --- |
33
+ | `agents` | Which harnesses run? Which models, efforts, versions, kwargs, and presets do they support? |
34
+ | `sandbox_providers` | Which providers support the task's resources and features? |
35
+ | `platform_constraints` | Which requirements cannot run on this deployment? |
36
+ | `network_modes` | Which network modes can a task declare? |
37
+ | `analyze` | Which agents can analyses and checks run on, with which models and reasoning efforts? |
38
+ | `limits` | What are the current defaults and bounds for jobs, pages, uploads, and comparisons? |
39
+ | `statuses` | Which states exist, and which are terminal? |
40
+ | `agent_registration` | Which names, environment keys, and upload sizes can a custom agent use? |
41
+ | `error_codes`, `import_warning_codes` | Which failure and warning codes does this deployment publish? |
42
+ | `managed_providers` | Which providers are available to the separate managed-agent runtime? |
43
+
44
+ **Tip:**
45
+
46
+ Read capabilities when building a picker or validating a saved configuration. A documented example is useful; a fixed copy of the platform's model catalog becomes stale.
47
+
48
+ ## Configuration
49
+
50
+ `meta(config?)` accepts the usual [hosted configuration](/sdk-reference/index#configure-once). Only the API origin matters; credentials are not sent.
51
+
52
+ The result includes `schema_version`. Treat it as the document's schema version, not the SDK package version.
53
+
54
+ Read the full editable analyzer policy through [analysis defaults](/sdk-reference/analyses#read-current-defaults) or [check defaults](/sdk-reference/checks#read-current-defaults).