@evolvingmachines/evolve 0.0.65 → 0.0.67-train-harness-wave-2026-09-25.20260926.e00a8a6d

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (153) hide show
  1. package/dist/{chunk-DK2MGUPB.js → chunk-DI6LPW2B.js} +2 -2
  2. package/dist/cli/index.cjs +29 -29
  3. package/dist/cli/index.d.cts +3 -3
  4. package/dist/cli/index.d.ts +3 -3
  5. package/dist/cli/index.js +18 -18
  6. package/dist/index.cjs +64 -60
  7. package/dist/index.d.cts +297 -18
  8. package/dist/index.d.ts +297 -18
  9. package/dist/index.js +48 -44
  10. package/dist/{types-B_qmzlMk.d.cts → types-Dj0iMl5V.d.cts} +70 -90
  11. package/dist/{types-B_qmzlMk.d.ts → types-Dj0iMl5V.d.ts} +70 -90
  12. package/harness-capabilities.json +285 -0
  13. package/package.json +9 -6
  14. package/skills/create-adapter/SKILL.md +71 -26
  15. package/skills/create-task/SKILL.md +124 -66
  16. package/skills/evolve/SKILL.md +37 -20
  17. package/skills/evolve-evals/SKILL.md +142 -69
  18. package/skills/evolve-evals/references/cli-reference/agent.md +76 -0
  19. package/skills/evolve-evals/references/cli-reference/analysis.md +78 -0
  20. package/skills/evolve-evals/references/cli-reference/analyze.md +97 -0
  21. package/skills/evolve-evals/references/cli-reference/auth.md +96 -0
  22. package/skills/evolve-evals/references/cli-reference/check.md +157 -0
  23. package/skills/evolve-evals/references/cli-reference/dataset.md +154 -0
  24. package/skills/evolve-evals/references/cli-reference/filesystem.md +168 -0
  25. package/skills/evolve-evals/references/cli-reference/index.md +149 -0
  26. package/skills/evolve-evals/references/cli-reference/job.md +198 -0
  27. package/skills/evolve-evals/references/cli-reference/run.md +188 -0
  28. package/skills/evolve-evals/references/cli-reference/secrets.md +77 -0
  29. package/skills/evolve-evals/references/cli-reference/session.md +56 -0
  30. package/skills/evolve-evals/references/cli-reference/skill.md +52 -0
  31. package/skills/evolve-evals/references/cli-reference/skills.md +85 -0
  32. package/skills/evolve-evals/references/cli-reference/trial.md +115 -0
  33. package/skills/evolve-evals/references/cli-reference/upload.md +72 -0
  34. package/skills/evolve-evals/references/core-concepts/agents.md +101 -0
  35. package/skills/evolve-evals/references/core-concepts/analyze.md +143 -0
  36. package/skills/evolve-evals/references/core-concepts/check.md +116 -0
  37. package/skills/evolve-evals/references/core-concepts/compatibility.md +39 -0
  38. package/skills/evolve-evals/references/core-concepts/compose.md +103 -0
  39. package/skills/evolve-evals/references/core-concepts/datasets.md +234 -0
  40. package/skills/evolve-evals/references/core-concepts/index.md +52 -0
  41. package/skills/evolve-evals/references/core-concepts/jobs.md +223 -0
  42. package/skills/evolve-evals/references/core-concepts/models.md +73 -0
  43. package/skills/evolve-evals/references/core-concepts/multi-step.md +93 -0
  44. package/skills/evolve-evals/references/core-concepts/rewardkit.md +341 -0
  45. package/skills/evolve-evals/references/core-concepts/sandboxes.md +73 -0
  46. package/skills/evolve-evals/references/core-concepts/secrets.md +86 -0
  47. package/skills/evolve-evals/references/core-concepts/sharing.md +81 -0
  48. package/skills/evolve-evals/references/core-concepts/skills.md +100 -0
  49. package/skills/evolve-evals/references/core-concepts/task-config.md +169 -0
  50. package/skills/evolve-evals/references/core-concepts/task-environment.md +127 -0
  51. package/skills/evolve-evals/references/core-concepts/task-reference.md +191 -0
  52. package/skills/evolve-evals/references/core-concepts/task-verifiers.md +209 -0
  53. package/skills/evolve-evals/references/core-concepts/tasks.md +90 -0
  54. package/skills/evolve-evals/references/core-concepts/teams.md +128 -0
  55. package/skills/evolve-evals/references/core-concepts/trial-outputs.md +225 -0
  56. package/skills/evolve-evals/references/core-concepts/trials.md +129 -0
  57. package/skills/evolve-evals/references/core-concepts/upload-sdk-session.md +240 -0
  58. package/skills/evolve-evals/references/core-concepts/upload.md +166 -0
  59. package/skills/evolve-evals/references/dashboard/checks.md +54 -0
  60. package/skills/evolve-evals/references/dashboard/datasets.md +65 -0
  61. package/skills/evolve-evals/references/dashboard/jobs.md +106 -0
  62. package/skills/evolve-evals/references/dashboard/traces.md +54 -0
  63. package/skills/evolve-evals/references/dashboard/trial-viewer.md +77 -0
  64. package/skills/evolve-evals/references/getting-started/first-task.md +228 -0
  65. package/skills/evolve-evals/references/getting-started/installation.md +79 -0
  66. package/skills/evolve-evals/references/getting-started/quick-start.md +100 -0
  67. package/skills/evolve-evals/references/index.md +58 -0
  68. package/skills/evolve-evals/references/sdk/http.md +110 -0
  69. package/skills/evolve-evals/references/sdk/python.md +108 -0
  70. package/skills/evolve-evals/references/sdk/typescript.md +104 -0
  71. package/skills/evolve-evals/references/sdk-reference/agents.md +75 -0
  72. package/skills/evolve-evals/references/sdk-reference/analyses.md +125 -0
  73. package/skills/evolve-evals/references/sdk-reference/auth.md +171 -0
  74. package/skills/evolve-evals/references/sdk-reference/checks.md +146 -0
  75. package/skills/evolve-evals/references/sdk-reference/datasets.md +170 -0
  76. package/skills/evolve-evals/references/sdk-reference/error-codes.md +231 -0
  77. package/skills/evolve-evals/references/sdk-reference/errors.md +98 -0
  78. package/skills/evolve-evals/references/sdk-reference/filesystem.md +166 -0
  79. package/skills/evolve-evals/references/sdk-reference/index.md +161 -0
  80. package/skills/evolve-evals/references/sdk-reference/jobs.md +250 -0
  81. package/skills/evolve-evals/references/sdk-reference/meta.md +54 -0
  82. package/skills/evolve-evals/references/sdk-reference/methods/agents.md +154 -0
  83. package/skills/evolve-evals/references/sdk-reference/methods/analyses.md +211 -0
  84. package/skills/evolve-evals/references/sdk-reference/methods/auth.md +257 -0
  85. package/skills/evolve-evals/references/sdk-reference/methods/checks.md +462 -0
  86. package/skills/evolve-evals/references/sdk-reference/methods/datasets.md +687 -0
  87. package/skills/evolve-evals/references/sdk-reference/methods/filesystem.md +527 -0
  88. package/skills/evolve-evals/references/sdk-reference/methods/jobs.md +1023 -0
  89. package/skills/evolve-evals/references/sdk-reference/methods/meta.md +213 -0
  90. package/skills/evolve-evals/references/sdk-reference/methods/secrets.md +157 -0
  91. package/skills/evolve-evals/references/sdk-reference/methods/skills.md +125 -0
  92. package/skills/evolve-evals/references/sdk-reference/methods/trials.md +371 -0
  93. package/skills/evolve-evals/references/sdk-reference/secrets.md +104 -0
  94. package/skills/evolve-evals/references/sdk-reference/skills.md +66 -0
  95. package/skills/evolve-evals/references/sdk-reference/trials.md +164 -0
  96. package/skills/evolve-evals/references/sdk-reference/types.md +1255 -0
  97. package/skills/publish/SKILL.md +103 -79
  98. package/skills/rewardkit/SKILL.md +78 -23
  99. package/spec/openapi.yaml +164 -160
  100. package/skills/evolve-evals/references/cli-reference/agent.mdx +0 -79
  101. package/skills/evolve-evals/references/cli-reference/analysis.mdx +0 -160
  102. package/skills/evolve-evals/references/cli-reference/analyze.mdx +0 -84
  103. package/skills/evolve-evals/references/cli-reference/auth.mdx +0 -133
  104. package/skills/evolve-evals/references/cli-reference/check.mdx +0 -263
  105. package/skills/evolve-evals/references/cli-reference/dataset.mdx +0 -151
  106. package/skills/evolve-evals/references/cli-reference/index.mdx +0 -108
  107. package/skills/evolve-evals/references/cli-reference/job.mdx +0 -278
  108. package/skills/evolve-evals/references/cli-reference/run.mdx +0 -204
  109. package/skills/evolve-evals/references/cli-reference/secrets.mdx +0 -95
  110. package/skills/evolve-evals/references/cli-reference/skill.mdx +0 -61
  111. package/skills/evolve-evals/references/cli-reference/skills.mdx +0 -81
  112. package/skills/evolve-evals/references/cli-reference/trial.mdx +0 -181
  113. package/skills/evolve-evals/references/cli-reference/upload.mdx +0 -36
  114. package/skills/evolve-evals/references/core-concepts/agents.mdx +0 -106
  115. package/skills/evolve-evals/references/core-concepts/analyze.mdx +0 -86
  116. package/skills/evolve-evals/references/core-concepts/check.mdx +0 -64
  117. package/skills/evolve-evals/references/core-concepts/datasets.mdx +0 -138
  118. package/skills/evolve-evals/references/core-concepts/index.mdx +0 -42
  119. package/skills/evolve-evals/references/core-concepts/jobs.mdx +0 -151
  120. package/skills/evolve-evals/references/core-concepts/models.mdx +0 -68
  121. package/skills/evolve-evals/references/core-concepts/sandboxes.mdx +0 -52
  122. package/skills/evolve-evals/references/core-concepts/secrets.mdx +0 -41
  123. package/skills/evolve-evals/references/core-concepts/sharing.mdx +0 -35
  124. package/skills/evolve-evals/references/core-concepts/skills.mdx +0 -42
  125. package/skills/evolve-evals/references/core-concepts/tasks.mdx +0 -216
  126. package/skills/evolve-evals/references/core-concepts/teams.mdx +0 -36
  127. package/skills/evolve-evals/references/core-concepts/trial-outputs.mdx +0 -117
  128. package/skills/evolve-evals/references/core-concepts/trials.mdx +0 -103
  129. package/skills/evolve-evals/references/core-concepts/upload.mdx +0 -93
  130. package/skills/evolve-evals/references/dashboard/checks.mdx +0 -21
  131. package/skills/evolve-evals/references/dashboard/datasets.mdx +0 -24
  132. package/skills/evolve-evals/references/dashboard/jobs.mdx +0 -65
  133. package/skills/evolve-evals/references/dashboard/traces.mdx +0 -17
  134. package/skills/evolve-evals/references/dashboard/trial-viewer.mdx +0 -30
  135. package/skills/evolve-evals/references/getting-started/installation.mdx +0 -56
  136. package/skills/evolve-evals/references/getting-started/quick-start.mdx +0 -94
  137. package/skills/evolve-evals/references/index.mdx +0 -48
  138. package/skills/evolve-evals/references/sdk/python.mdx +0 -161
  139. package/skills/evolve-evals/references/sdk/typescript.mdx +0 -163
  140. package/skills/evolve-evals/references/sdk-reference/agents.mdx +0 -108
  141. package/skills/evolve-evals/references/sdk-reference/analyses.mdx +0 -222
  142. package/skills/evolve-evals/references/sdk-reference/auth.mdx +0 -116
  143. package/skills/evolve-evals/references/sdk-reference/checks.mdx +0 -300
  144. package/skills/evolve-evals/references/sdk-reference/datasets.mdx +0 -346
  145. package/skills/evolve-evals/references/sdk-reference/errors.mdx +0 -210
  146. package/skills/evolve-evals/references/sdk-reference/index.mdx +0 -60
  147. package/skills/evolve-evals/references/sdk-reference/jobs.mdx +0 -822
  148. package/skills/evolve-evals/references/sdk-reference/meta.mdx +0 -52
  149. package/skills/evolve-evals/references/sdk-reference/skills.mdx +0 -69
  150. package/skills/evolve-evals/references/sdk-reference/trials.mdx +0 -302
  151. package/skills/evolve-evals/references/sdk-reference/types.mdx +0 -274
  152. package/skills/evolve-evals/references/snippets/global-options.mdx +0 -13
  153. package/skills/evolve-evals/references/snippets/list-flags.mdx +0 -23
@@ -0,0 +1,110 @@
1
+ ---
2
+ title: "HTTP API"
3
+ description: "Submit and inspect evaluations from any language."
4
+ ---
5
+
6
+ The CLI and SDKs call the same HTTP API. Use it directly when you need another language or an operation without an SDK method.
7
+
8
+ ## Connect
9
+
10
+ Use `https://dashboard.evolvingmachines.ai` as the origin. Send your [API key](/getting-started/installation#connect-your-account) in the `Authorization` header. Writes require a full-access key.
11
+
12
+ ```bash
13
+ curl -sS https://dashboard.evolvingmachines.ai/api/auth/status \
14
+ -H "Authorization: Bearer $EVOLVE_API_KEY"
15
+ ```
16
+
17
+ `GET /api/meta` needs no key. It lists the current harnesses, models, limits, and error codes.
18
+
19
+ ## Start one evaluation
20
+
21
+ Save the request as `job.json`:
22
+
23
+ ```json job.json
24
+ {
25
+ "datasets": [{
26
+ "name": "harbor-examples",
27
+ "version": "1.0",
28
+ "task_names": ["hello-world"]
29
+ }],
30
+ "agents": [{
31
+ "name": "codex",
32
+ "model_name": "gpt-5.6-luna"
33
+ }],
34
+ "max_trial_spend_usd": 1,
35
+ "retry": {"max_retries": 0}
36
+ }
37
+ ```
38
+
39
+ ```bash
40
+ curl -sS https://dashboard.evolvingmachines.ai/api/jobs \
41
+ -H "Authorization: Bearer $EVOLVE_API_KEY" \
42
+ -H "Content-Type: application/json" \
43
+ --data-binary @job.json
44
+ ```
45
+
46
+ Success returns `202` and the accepted job. Set `$JOB_ID` to its `id`. Closing the HTTP connection does not cancel the evaluation.
47
+
48
+ The JSON fields match the [SDK job inputs](/sdk-reference/jobs#required-inputs). Add `org` to create the job in a team. Attach stored secrets through the request's [secret references](/sdk-reference/secrets#attach-it-to-a-job).
49
+
50
+ **Note:**
51
+
52
+ To retry the same submission safely after a transport failure, send an `Idempotency-Key` header with a unique value for that intended job. Reuse it only for retries of that submission. A new intended run needs a new key.
53
+
54
+ ## Follow the result
55
+
56
+ ### Read status
57
+
58
+ ```bash
59
+ curl -sS "https://dashboard.evolvingmachines.ai/api/jobs/$JOB_ID" \
60
+ -H "Authorization: Bearer $EVOLVE_API_KEY"
61
+ ```
62
+
63
+ ### Read trials
64
+
65
+ ```bash
66
+ curl -sS "https://dashboard.evolvingmachines.ai/api/jobs/$JOB_ID/trials" \
67
+ -H "Authorization: Bearer $EVOLVE_API_KEY"
68
+ ```
69
+
70
+ ### Stream progress
71
+
72
+ ```bash
73
+ curl -N "https://dashboard.evolvingmachines.ai/api/jobs/$JOB_ID/events" \
74
+ -H "Authorization: Bearer $EVOLVE_API_KEY"
75
+ ```
76
+
77
+ This endpoint returns server-sent events. The SDK watchers handle reconnection for you; `curl` does not.
78
+
79
+ ## Page through collections
80
+
81
+ Collections use `{ items, nextCursor, hasMore }`. Pass the returned `nextCursor` as `cursor` on the next request, keeping the same filters. Treat it as an opaque string.
82
+
83
+ ```bash
84
+ curl -sS --get https://dashboard.evolvingmachines.ai/api/jobs \
85
+ -H "Authorization: Bearer $EVOLVE_API_KEY" \
86
+ --data-urlencode "limit=20" \
87
+ --data-urlencode "cursor=$NEXT_CURSOR"
88
+ ```
89
+
90
+ ## Handle an error
91
+
92
+ An API refusal returns a non-success HTTP status and an `error` object:
93
+
94
+ | Field | Use |
95
+ | --- | --- |
96
+ | `code` | Choose how your client handles the failure |
97
+ | `message` | Explain the problem to the caller |
98
+ | `param`, `details` | Locate invalid input or read failure context, when present |
99
+ | `request_id` | Identify the request when reporting a problem |
100
+ | `retryAfterSec` | Requested delay, when present; also sent as `Retry-After` |
101
+
102
+ The response also includes `X-Request-Id`. See [error handling](/sdk-reference/errors) for common codes and the difference between retrying an HTTP request and retrying a trial.
103
+
104
+ ## Operations beyond the SDK methods
105
+
106
+ | Need | Reference |
107
+ | --- | --- |
108
+ | List or revoke API keys | [Key management](/sdk-reference/auth#manage-api-keys-over-http) |
109
+ | Manage team names, roles, and invitations | [Team management](/sdk-reference/auth#manage-teams-over-http) |
110
+ | Filter or sort the dataset catalog | [Dataset filters](/sdk-reference/datasets#browse-the-catalog) |
@@ -0,0 +1,108 @@
1
+ ---
2
+ title: "Python"
3
+ description: "Run one task from an async Python program. No Node bridge is needed."
4
+ ---
5
+
6
+ Install the package and set your [API key](/getting-started/installation).
7
+
8
+ ```bash
9
+ pip install evolvingmachines-evolve
10
+ export EVOLVE_API_KEY="your-api-key"
11
+ ```
12
+
13
+ ## Run one task
14
+
15
+ The managed evals client calls the API directly. Save this as `eval.py` and run `python eval.py`.
16
+
17
+ ```python eval.py
18
+ import asyncio
19
+ from evolve import hosted
20
+
21
+
22
+ async def main():
23
+ async with hosted() as evolve:
24
+ job = await evolve.jobs.start(
25
+ datasets=[{
26
+ "name": "harbor-examples",
27
+ "version": "1.0",
28
+ "task_names": ["hello-world"],
29
+ }],
30
+ agents=[{"name": "codex", "model_name": "gpt-5.6-luna"}],
31
+ max_trial_spend_usd=0.50,
32
+ retry={"max_retries": 0},
33
+ )
34
+ print("Job:", job.id)
35
+
36
+ finished = await evolve.jobs.watch(job.id)
37
+ print("Status:", finished.status)
38
+ print("Model cost:", finished.stats.get("cost_usd"))
39
+
40
+ async for trial in evolve.jobs.trials(job.id):
41
+ print(trial.task_name, trial.status, trial.reward)
42
+
43
+ archive = await evolve.jobs.download(job.id, to="./results")
44
+ print("Saved:", archive)
45
+
46
+
47
+ asyncio.run(main())
48
+ ```
49
+
50
+ ### 1. Submit
51
+
52
+ `start()` returns the accepted job. It does not wait for the trial.
53
+
54
+ ### 2. Wait
55
+
56
+ `watch()` follows progress and returns the terminal job.
57
+
58
+ ### 3. Inspect
59
+
60
+ Use attributes for jobs and trials, and dictionary keys for stats.
61
+
62
+ ### 4. Keep the results
63
+
64
+ `download()` saves a verified archive and returns its path.
65
+
66
+ The example sets a $0.50 cap for the trial and disables configured infrastructure retries. Provider capacity waits are handled separately. A completed job can contain failed trials; inspect their statuses and rewards.
67
+
68
+ ## Show progress
69
+
70
+ Use the following snippets inside the `async with hosted() as evolve` block in `main()`, after creating `job`.
71
+
72
+ ```python
73
+ finished = await evolve.jobs.watch(
74
+ job.id,
75
+ on_event=lambda event: print(event.seq, event.type),
76
+ )
77
+ ```
78
+
79
+ Or consume the events directly:
80
+
81
+ ```python
82
+ async for event in evolve.jobs.watch(job.id):
83
+ print(event.type, event.data)
84
+ finished = await evolve.jobs.get(job.id)
85
+ ```
86
+
87
+ Choose one consumption style for each watch handle.
88
+
89
+ ## Read a trial
90
+
91
+ This uses the same `evolve` client and `job` from the program above.
92
+
93
+ ```python
94
+ page = await evolve.jobs.trials(job.id, limit=1)
95
+ if page.items:
96
+ trial = page.items[0]
97
+ trajectory = await evolve.trials.artifact(trial.id, "trace-atif")
98
+ files = await evolve.trials.files(trial.id)
99
+ print(trajectory, files.items)
100
+ ```
101
+
102
+ **[Configure a job](/sdk-reference/jobs)**
103
+
104
+ Models, task filters, attempts, limits, and analysis.
105
+
106
+ **[Understand returned objects](/sdk-reference/types)**
107
+
108
+ Dataclasses, dictionaries, nullable values, and status fields.
@@ -0,0 +1,104 @@
1
+ ---
2
+ title: "TypeScript"
3
+ description: "Run one task, follow its progress, and inspect the result."
4
+ ---
5
+
6
+ Install the package and set your [API key](/getting-started/installation).
7
+
8
+ ```bash
9
+ npm install @evolvingmachines/evolve
10
+ export EVOLVE_API_KEY="your-api-key"
11
+ ```
12
+
13
+ ## Run one task
14
+
15
+ Save this as `eval.mts` in your TypeScript project.
16
+
17
+ ```ts eval.mts
18
+ import { hosted } from "@evolvingmachines/evolve";
19
+
20
+ const evolve = hosted();
21
+
22
+ const job = await evolve.jobs.start({
23
+ datasets: [{
24
+ name: "harbor-examples",
25
+ version: "1.0",
26
+ task_names: ["hello-world"],
27
+ }],
28
+ agents: [{ name: "codex", model_name: "gpt-5.6-luna" }],
29
+ max_trial_spend_usd: 0.50,
30
+ retry: { max_retries: 0 },
31
+ });
32
+ console.log("Job:", job.id);
33
+
34
+ const finished = await evolve.jobs.watch(job.id);
35
+ console.log("Status:", finished.status);
36
+ console.log("Model cost:", finished.stats.cost_usd);
37
+
38
+ for await (const trial of evolve.jobs.trials(job.id)) {
39
+ console.log(trial.task_name, trial.status, trial.reward);
40
+ }
41
+
42
+ const archive = await evolve.jobs.download(job.id, { to: "./results" });
43
+ console.log("Saved:", archive);
44
+ ```
45
+
46
+ ### 1. Submit
47
+
48
+ `start()` returns the accepted job. It does not wait for the trial.
49
+
50
+ ### 2. Wait
51
+
52
+ `watch()` follows progress and returns the terminal job.
53
+
54
+ ### 3. Inspect
55
+
56
+ Read each trial's status and reward. A completed job can contain failed trials.
57
+
58
+ ### 4. Keep the results
59
+
60
+ `download()` saves a verified archive and returns its path.
61
+
62
+ The example sets a $0.50 cap for the trial and disables configured infrastructure retries. Provider capacity waits are handled separately. These settings limit the example's work; they do not guarantee a passing result.
63
+
64
+ ## Show progress
65
+
66
+ Use an event callback while awaiting completion:
67
+
68
+ ```ts
69
+ const finished = await evolve.jobs.watch(job.id, {
70
+ onEvent: (event) => console.log(event.seq, event.type),
71
+ });
72
+ ```
73
+
74
+ Or consume events directly:
75
+
76
+ ```ts
77
+ for await (const event of evolve.jobs.watch(job.id)) {
78
+ console.log(event.type, event.data);
79
+ }
80
+ const finished = await evolve.jobs.get(job.id);
81
+ ```
82
+
83
+ Choose one consumption style for each watch handle.
84
+
85
+ ## Read a trial
86
+
87
+ ```ts
88
+ const page = await evolve.jobs.trials(job.id, { limit: 1 });
89
+ const trial = page.items[0];
90
+
91
+ if (trial) {
92
+ const trajectory = await evolve.trials.artifact(trial.id, "trace-atif");
93
+ const files = await evolve.trials.files(trial.id);
94
+ console.log(trajectory, files.items);
95
+ }
96
+ ```
97
+
98
+ **[Configure a job](/sdk-reference/jobs)**
99
+
100
+ Models, task filters, attempts, limits, and analysis.
101
+
102
+ **[Inspect files](/sdk-reference/filesystem)**
103
+
104
+ Live files, saved captures, logs, and processes.
@@ -0,0 +1,75 @@
1
+ ---
2
+ title: "Custom agents"
3
+ description: "Register your own harness and use it in a job arm."
4
+ ---
5
+
6
+ [Method reference: calls, parameters, and response fields](/sdk-reference/methods/agents).
7
+
8
+ Use `agents()` for your own agent registrations. To discover built-in harnesses, use [meta](/sdk-reference/meta).
9
+
10
+ ## Register an agent
11
+
12
+ Provide a local source directory or an install script, plus the command that runs your agent.
13
+
14
+ ```ts TypeScript
15
+ import { agents } from "@evolvingmachines/evolve";
16
+
17
+ const agent = await agents().create({
18
+ name: "my-agent",
19
+ directory: "./my-agent",
20
+ run_command: "python agent.py",
21
+ });
22
+ ```
23
+
24
+ ```python Python
25
+ from evolve import agents
26
+
27
+ agent = await agents().create(
28
+ name="my-agent",
29
+ directory="./my-agent",
30
+ run_command="python agent.py",
31
+ )
32
+ ```
33
+
34
+ The example assumes `./my-agent` contains that runnable program. See the [custom-agent contract](/core-concepts/agents) for how it receives a task and produces output.
35
+
36
+ | Field | Required | Purpose |
37
+ | --- | --- | --- |
38
+ | `name` | Create only | Registration name used by job arms |
39
+ | `run_command` | Yes | Command that starts the agent |
40
+ | `directory` | One source | Local source folder, archived for upload |
41
+ | `install_script` | One source | Script used to install the agent |
42
+ | `env` | No | Declared non-secret environment map |
43
+ | `org` | No | Owning team; overrides the client default |
44
+
45
+ Choose exactly one of `directory` and `install_script`. Discover naming, reserved environment keys, and size limits through `meta().agent_registration`. Store credentials as [secrets](/sdk-reference/secrets).
46
+
47
+ ## Use the registration
48
+
49
+ Name it in a job's `agents` list just like a built-in harness:
50
+
51
+ ```json
52
+ {
53
+ "name": "my-agent",
54
+ "model_name": "your-model"
55
+ }
56
+ ```
57
+
58
+ Use the model your custom agent is written to run. Registration does not make every model compatible with its implementation.
59
+
60
+ ## Read or replace
61
+
62
+ | Action | TypeScript | Python | Returns |
63
+ | --- | --- | --- | --- |
64
+ | List registrations | `list({scope, limit, cursor})` | `list(scope=..., limit=..., cursor=...)` | Paginated handle |
65
+ | Read one | `get(name)` | `get(name)` | Agent registration |
66
+ | Create or replace | `upsert(name, input)` | `upsert(name, run_command=..., ...)` | Agent registration |
67
+ | Delete | `delete(name)` | `delete(name)` | No content |
68
+
69
+ List scopes are `my`, `shared`, and `org`. Upsert accepts the same fields as create except `name`, which is its first argument.
70
+
71
+ **Note:**
72
+
73
+ Upsert is a full replacement. Include every setting you want to retain; omitting `env` clears it. It avoids the gap created by deleting and then registering again.
74
+
75
+ Deleting a registration does not rewrite the agent identity already recorded on past jobs.
@@ -0,0 +1,125 @@
1
+ ---
2
+ title: "Analyses"
3
+ description: "Analyze trial traces, then inspect the result and the analyzer's own run."
4
+ ---
5
+
6
+ **Note:**
7
+
8
+ For every method’s inputs, return values, and language differences, see the [analyses method reference](/sdk-reference/methods/analyses).
9
+
10
+ Analysis asks an agent to read a trial's evidence against a rubric. It produces criterion outcomes and explanations. It does not replace the verifier's reward.
11
+
12
+ ```text
13
+ Trial trace + task + rubric
14
+ │
15
+ ▼
16
+ Analyzer run
17
+ │
18
+ ▼
19
+ Summary + criterion outcomes + evidence
20
+ ```
21
+
22
+ ## Start an analysis
23
+
24
+ Start work through the **jobs** client. Read analyzer runs through the **analyses** client.
25
+
26
+ ```ts TypeScript
27
+ import { jobs, analyses } from "@evolvingmachines/evolve";
28
+
29
+ await jobs().analyze(jobId, { failing: true });
30
+ const job = await jobs().watchAnalysis(jobId);
31
+ console.log(job.stats.analysis);
32
+ ```
33
+
34
+ ```python Python
35
+ from evolve import jobs, analyses
36
+
37
+ await jobs().analyze(job_id, failing=True)
38
+ job = await jobs().watch_analysis(job_id)
39
+ print(job.stats.get("analysis"))
40
+ ```
41
+
42
+ Manual analysis requires a terminal job. It returns immediately after queuing work. Only one analysis wave can run for that job at a time.
43
+
44
+ To analyze trials as they finish, pass `analyze: {}` / `analyze={}` when starting the job. Wait for job completion before calling the analysis watch.
45
+
46
+ ### Configuration
47
+
48
+ The field names are the same in both languages. TypeScript passes an object; Python uses keyword arguments on `analyze()` or a dictionary inside job creation.
49
+
50
+ | Field | Purpose |
51
+ | --- | --- |
52
+ | `agent` | Agent the analyzer runs on; default `claude` |
53
+ | `model_name` | Analyzer model, one of that agent's models |
54
+ | `reasoning_effort` | Effort that agent accepts |
55
+ | `sandbox_provider` | Analyzer's sandbox provider |
56
+ | `rubric` | `{criteria: [{name, description, guidance}, ...]}` |
57
+ | `prompt` | Replacement prompt template text |
58
+ | `n_concurrent` | Requested parallel analyses; integer 1–150, also bounded by team quota |
59
+ | `passing` | Only scored trials with primary reward exactly 1 |
60
+ | `failing` | Other analyzable trials |
61
+ | `n_trials` | Positive cap after selection |
62
+ | `trial_ids` | Explicit trial ids from this job; manual analysis only |
63
+
64
+ `passing` and `failing` cannot both be true. Cancelled trials are excluded. Explicit ids are filtered before the cap; unknown, duplicate, or empty ids are refused.
65
+
66
+ ### Analysis watch options
67
+
68
+ | TypeScript | Python | Purpose |
69
+ | --- | --- | --- |
70
+ | `onStats` | `on_stats` | Called with the job when analysis tally changes |
71
+ | `pollIntervalMs` | `poll_interval_s` | Initial interval; default 2000 ms / 2 s |
72
+ | `signal` | `timeout_s` | Stop waiting / bound the Python wait |
73
+
74
+ Polling slows to at most 30 seconds while the tally stays unchanged. Call the watch after starting analysis; a job never analyzed can have a null tally indefinitely.
75
+
76
+ ## Read current defaults
77
+
78
+ ```ts TypeScript
79
+ const defaults = await analyses().defaults();
80
+ console.log(defaults.model_name, defaults.rubric, defaults.prompt);
81
+ ```
82
+
83
+ ```python Python
84
+ defaults = await analyses().defaults()
85
+ print(defaults["model_name"], defaults["rubric"], defaults["prompt"])
86
+ ```
87
+
88
+ The response includes `agent`, `model_name`, `rubric`, `prompt`, `reasoning_effort`, and `sandbox_provider`. The prompt is the editable template text.
89
+
90
+ A custom analyze prompt may use `{trial_path}`, `{task_section}`, and `{criteria_guidance}`. The required result schema is appended by the platform. See [analysis concepts](/core-concepts/analyze) for rubric design.
91
+
92
+ ## Read results
93
+
94
+ ```ts TypeScript
95
+ for await (const result of analyses().list({ job: jobId })) {
96
+ console.log(result.id, result.status, result.checks);
97
+ }
98
+ ```
99
+
100
+ ```python Python
101
+ async for result in analyses().list(job=job_id):
102
+ print(result["id"], result["status"], result["checks"])
103
+ ```
104
+
105
+ List options: `scope`, `job`, `status` list, `limit`, `cursor`. Status values are `queued`, `running`, `completed`, `failed`.
106
+
107
+ The latest analysis also appears on `trial.analysis`. Each criterion reports `pass`, `fail`, `not_applicable`, or `unknown`, with an explanation and evidence. Evolve derives no verdict from them; compute what you need from the criteria.
108
+
109
+ ## Inspect the analyzer run
110
+
111
+ | Method | TypeScript | Python |
112
+ | --- | --- | --- |
113
+ | List runs | `list(options)` | `list(...)` |
114
+ | Read defaults | `defaults()` | `defaults()` |
115
+ | Download run | `download(analysisId, { to })` | `download(analysis_id, to=...)` |
116
+ | Browse sandbox files | `filesystem(analysisId)` | `filesystem(analysis_id)` |
117
+ | Read one analysis directly | `get(analysisId)` | Use list results or `trial.analysis` |
118
+ | Read analyzer transcript | `transcript(analysisId, { since })` | Not exposed |
119
+ | Read raw stored artifact | `artifact(analysisId, stream)` | Not exposed |
120
+
121
+ `analysisId` identifies the analyzer run, not the trial it read. `since` is an inclusive nonnegative event index; the transcript returns `events`, `total`, run metadata, and separate `gateway_calls`.
122
+
123
+ TypeScript artifact selectors are `trace-stdout`, `trace-stderr`, and `agent-home`. The first two return text or null; `agent-home` returns a file map or null.
124
+
125
+ Downloads return a wrapper-trial archive. See [filesystem and download options](/sdk-reference/filesystem).
@@ -0,0 +1,171 @@
1
+ ---
2
+ title: "Identity and teams"
3
+ description: "Read your identity, choose a team, and manage invitations."
4
+ ---
5
+
6
+ [Method reference: calls, parameters, and response fields](/sdk-reference/methods/auth).
7
+
8
+ Import `auth` for identity and `orgs` for teams. `hosted().orgs` is also available; `hosted().auth` is not.
9
+
10
+ ## Check your identity
11
+
12
+ ```ts TypeScript
13
+ import { auth, orgs } from "@evolvingmachines/evolve";
14
+
15
+ const identity = await auth().status();
16
+ console.log(identity.user_id, identity.email, identity.key.label);
17
+ ```
18
+
19
+ ```python Python
20
+ from evolve import auth, orgs
21
+
22
+ identity = await auth().status()
23
+ print(identity.user_id, identity.email, identity.key.label)
24
+ ```
25
+
26
+ The response includes user id, email, and metadata for the API key used. It does not return the key's secret value. `status()` is the auth client's only method.
27
+
28
+ ## Manage API keys over HTTP
29
+
30
+ Create keys in the [dashboard](https://dashboard.evolvingmachines.ai/api-keys). Choose **Full access** to submit or change work, or **Read-only** to inspect it. Write requests with a read-only key return `403 read_only_key`; full access still respects your account's permissions.
31
+
32
+ Key listing and revocation are HTTP operations, without corresponding methods in either SDK. These examples use the same `EVOLVE_API_KEY` as the clients:
33
+
34
+ ```bash
35
+ EVOLVE_URL="https://dashboard.evolvingmachines.ai"
36
+ ```
37
+
38
+ ### List your keys
39
+
40
+ ```bash
41
+ curl -sS "$EVOLVE_URL/api/auth/keys?limit=50" \
42
+ -H "Authorization: Bearer $EVOLVE_API_KEY"
43
+ ```
44
+
45
+ Returns `{ items, nextCursor, hasMore }`, newest first. `limit` defaults to `50`, with a maximum of `200`; pass `nextCursor` as the next request's `cursor`.
46
+
47
+ Each item has `id`, `label`, `created_at`, and `last_used_at`. Secret values are never returned; `last_used_at` is currently null. Only your own account keys are listed, excluding credentials created for individual runs.
48
+
49
+ ### Revoke a key
50
+
51
+ Set `$KEY_ID` to an `id` from the list, not the secret key value. Use a full-access key belonging to the same account.
52
+
53
+ ```bash
54
+ curl -sS -X DELETE \
55
+ "$EVOLVE_URL/api/auth/keys/$KEY_ID" \
56
+ -H "Authorization: Bearer $EVOLVE_API_KEY"
57
+ ```
58
+
59
+ Success returns `204` with no body. An unknown key, another account's key, or a run credential returns `404 invalid_api_key`.
60
+
61
+ Revocation removes the key, but API servers can continue accepting a cached credential for up to 5 minutes.
62
+
63
+ ## Choose a team
64
+
65
+ ```ts TypeScript
66
+ const teams = await orgs().list();
67
+ for (const team of teams) console.log(team.slug, team.role);
68
+ const team = await orgs().get("my-team");
69
+ console.log(team.quota, team.usage);
70
+ ```
71
+
72
+ ```python Python
73
+ teams = await orgs().list()
74
+ for team in teams:
75
+ print(team.slug, team.role)
76
+ team = await orgs().get("my-team")
77
+ print(team.quota, team.usage)
78
+ ```
79
+
80
+ `list()` returns an array/list, not a paginated handle. `get` accepts an organization slug or id. Detail includes membership count, quotas, and usage.
81
+
82
+ Pass `org` in [client configuration](/sdk-reference/index#configure-once) or a supported create call to make that team own the resource.
83
+
84
+ ## Team methods
85
+
86
+ | Action | TypeScript | Python | Returns |
87
+ | --- | --- | --- | --- |
88
+ | List your teams | `list()` | `list()` | Organizations |
89
+ | Read one team | `get(org)` | `get(org)` | Organization detail |
90
+ | Create a team | `create(name, { displayName })` | `create(name, display_name=...)` | Organization |
91
+ | Create invite token | `invite(org)` | `invite(org)` | Invite metadata and token |
92
+ | Join with token | `join(token)` | `join(token)` | Organization and `already_member` |
93
+ | Read members | `members(org)` | `members(org)` | Member list |
94
+
95
+ `name` in `create` is the slug; display name is optional. Creating an invite requires an organization owner. `invite` uses the server's defaults and takes no expiry/use-limit options in these SDKs.
96
+
97
+ Team membership enables shared access to team-owned work. It does not grant every creator-only action, such as managing a job's external shares or deleting its record. See [teams](/core-concepts/teams) and [sharing](/core-concepts/sharing).
98
+
99
+ ## Manage teams over HTTP
100
+
101
+ The operations below are not methods on either SDK client. Prefix each path with `https://dashboard.evolvingmachines.ai` and send `Authorization: Bearer $EVOLVE_API_KEY`. Writes require a full-access key. Send JSON bodies with `Content-Type: application/json`.
102
+
103
+ Replace `{org}` with the team's slug or id. Get `{userId}` from `orgs().members(org)`; use the `user_id` returned by `auth().status()` when leaving your own membership. Get `{inviteId}` from an invitation's `invite_id`.
104
+
105
+ ### Rename or delete a team
106
+
107
+ An owner can update the slug, display name, or both:
108
+
109
+ **`PATCH`** `/api/orgs/{org}`
110
+
111
+ ```json
112
+ {
113
+ "display_name": "Acme Lab"
114
+ }
115
+ ```
116
+
117
+ Optional `slug`: 1–48 lowercase letters, digits, or hyphens; no leading/trailing hyphen or reserved name. `display_name`: 1–120 characters after trimming. Success returns the updated organization; a taken slug returns `409 org_slug_taken`.
118
+
119
+ `DELETE /api/orgs/{org}` deletes the team and returns `{ deleted: true, org_id }`. It requires an owner and returns `409 org_in_use` while jobs or datasets remain, with their counts in `details`.
120
+
121
+ ### Change a role, remove a member, or leave
122
+
123
+ An owner can set a member's role to `owner` or `member`:
124
+
125
+ **`PATCH`** `/api/orgs/{org}/members/{userId}`
126
+
127
+ ```json
128
+ {
129
+ "role": "owner"
130
+ }
131
+ ```
132
+
133
+ Success returns the updated member. `DELETE` on the same path removes that member and returns `{ removed: true, user_id, role }`. Owners can remove others; members can remove themselves to leave.
134
+
135
+ Demoting or removing the last owner returns `409 org_last_owner`. Promote another owner first. A missing member returns `404 org_member_not_found`.
136
+
137
+ ### Set invitation limits
138
+
139
+ An owner can create a token with a chosen lifetime and use limit:
140
+
141
+ **`POST`** `/api/orgs/{org}/invites`
142
+
143
+ ```json
144
+ {
145
+ "expires_in_sec": 86400,
146
+ "max_uses": 1
147
+ }
148
+ ```
149
+
150
+ | Field | Values and default |
151
+ | --- | --- |
152
+ | `expires_in_sec` | Integer from `60` to `7776000` (90 days), or `null` for no expiry. Omitted: 7 days. |
153
+ | `max_uses` | Positive integer, or `null` for unlimited uses. Omitted: unlimited. |
154
+
155
+ Success returns `201` with invitation metadata and `token`. The token is returned only at creation. CLI and SDK `invite` calls expose neither option and use these defaults.
156
+
157
+ Joining an existing membership returns `already_member: true` without consuming a use. For a new member, an expired, revoked, or exhausted token returns `410 invite_invalid`; an unknown token returns `404 invite_not_found`.
158
+
159
+ ### List or revoke invitations
160
+
161
+ Both operations require an owner:
162
+
163
+ **`GET`** `/api/orgs/{org}/invites`
164
+
165
+ Returns `{ items }`, newest first. Items contain `invite_id`, `expires_at`, `max_uses`, `uses`, `revoked_at`, and `created_at`; tokens are not included.
166
+
167
+ **`DELETE`** `/api/orgs/{org}/invites/{inviteId}`
168
+
169
+ Returns the invitation metadata with `revoked_at` set. Repeating the request leaves it revoked. This prevents new joins without removing existing members. An unknown invitation returns `404 invite_not_found`.
170
+
171
+ Personal organizations cannot be renamed or deleted, have their membership changed, or issue invitations. These actions return `403 org_personal_immutable`.