@evolvingmachines/evolve 0.0.55-project-sable.20260916.db0bc0bb → 0.0.56

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (73) hide show
  1. package/dist/cli/index.cjs +32 -32
  2. package/dist/cli/index.js +27 -27
  3. package/package.json +4 -6
  4. package/skills/evolve/SKILL.md +4 -3
  5. package/skills/evolve-evals/SKILL.md +112 -0
  6. package/{docs-evals → skills/evolve-evals/references}/cli-reference/run.mdx +1 -1
  7. package/{docs-evals → skills/evolve-evals/references}/cli-reference/skills.mdx +4 -4
  8. package/{docs-evals → skills/evolve-evals/references}/core-concepts/agents.mdx +2 -2
  9. package/{docs-evals → skills/evolve-evals/references}/core-concepts/analyze.mdx +1 -1
  10. package/{docs-evals → skills/evolve-evals/references}/core-concepts/jobs.mdx +3 -3
  11. package/{docs-evals → skills/evolve-evals/references}/core-concepts/models.mdx +1 -1
  12. package/{docs-evals → skills/evolve-evals/references}/core-concepts/sandboxes.mdx +2 -2
  13. package/{docs-evals → skills/evolve-evals/references}/core-concepts/secrets.mdx +1 -1
  14. package/{docs-evals → skills/evolve-evals/references}/core-concepts/trial-outputs.mdx +1 -1
  15. package/{docs-evals → skills/evolve-evals/references}/getting-started/quick-start.mdx +1 -1
  16. package/{docs-evals → skills/evolve-evals/references}/index.mdx +1 -1
  17. package/{docs-evals → skills/evolve-evals/references}/sdk/python.mdx +1 -1
  18. package/{docs-evals → skills/evolve-evals/references}/sdk/typescript.mdx +1 -1
  19. package/{docs-evals → skills/evolve-evals/references}/sdk-reference/errors.mdx +2 -2
  20. package/{docs-evals → skills/evolve-evals/references}/sdk-reference/jobs.mdx +2 -2
  21. package/docs-agents/SKILL.md +0 -138
  22. package/docs-agents/index.md +0 -81
  23. package/docs-agents/python/01-getting-started.md +0 -511
  24. package/docs-agents/python/02-configuration.md +0 -855
  25. package/docs-agents/python/03-runtime.md +0 -899
  26. package/docs-agents/python/04-streaming.md +0 -451
  27. package/docs-agents/python/05-swarm-pipeline.md +0 -953
  28. package/docs-agents/python/index.md +0 -122
  29. package/docs-agents/typescript/01-getting-started.md +0 -494
  30. package/docs-agents/typescript/02-configuration.md +0 -841
  31. package/docs-agents/typescript/03-runtime.md +0 -864
  32. package/docs-agents/typescript/04-streaming.md +0 -478
  33. package/docs-agents/typescript/05-swarm-pipeline.md +0 -957
  34. package/docs-agents/typescript/index.md +0 -121
  35. package/docs-evals/SKILL.md +0 -112
  36. package/docs-evals/docs.json +0 -180
  37. /package/{docs-evals → skills/evolve-evals/references}/cli-reference/agent.mdx +0 -0
  38. /package/{docs-evals → skills/evolve-evals/references}/cli-reference/analysis.mdx +0 -0
  39. /package/{docs-evals → skills/evolve-evals/references}/cli-reference/analyze.mdx +0 -0
  40. /package/{docs-evals → skills/evolve-evals/references}/cli-reference/auth.mdx +0 -0
  41. /package/{docs-evals → skills/evolve-evals/references}/cli-reference/check.mdx +0 -0
  42. /package/{docs-evals → skills/evolve-evals/references}/cli-reference/dataset.mdx +0 -0
  43. /package/{docs-evals → skills/evolve-evals/references}/cli-reference/index.mdx +0 -0
  44. /package/{docs-evals → skills/evolve-evals/references}/cli-reference/job.mdx +0 -0
  45. /package/{docs-evals → skills/evolve-evals/references}/cli-reference/secrets.mdx +0 -0
  46. /package/{docs-evals → skills/evolve-evals/references}/cli-reference/skill.mdx +0 -0
  47. /package/{docs-evals → skills/evolve-evals/references}/cli-reference/trial.mdx +0 -0
  48. /package/{docs-evals → skills/evolve-evals/references}/cli-reference/upload.mdx +0 -0
  49. /package/{docs-evals → skills/evolve-evals/references}/core-concepts/check.mdx +0 -0
  50. /package/{docs-evals → skills/evolve-evals/references}/core-concepts/datasets.mdx +0 -0
  51. /package/{docs-evals → skills/evolve-evals/references}/core-concepts/index.mdx +0 -0
  52. /package/{docs-evals → skills/evolve-evals/references}/core-concepts/skills.mdx +0 -0
  53. /package/{docs-evals → skills/evolve-evals/references}/core-concepts/tasks.mdx +0 -0
  54. /package/{docs-evals → skills/evolve-evals/references}/core-concepts/trials.mdx +0 -0
  55. /package/{docs-evals → skills/evolve-evals/references}/core-concepts/upload.mdx +0 -0
  56. /package/{docs-evals → skills/evolve-evals/references}/dashboard/checks.mdx +0 -0
  57. /package/{docs-evals → skills/evolve-evals/references}/dashboard/datasets.mdx +0 -0
  58. /package/{docs-evals → skills/evolve-evals/references}/dashboard/jobs.mdx +0 -0
  59. /package/{docs-evals → skills/evolve-evals/references}/dashboard/traces.mdx +0 -0
  60. /package/{docs-evals → skills/evolve-evals/references}/dashboard/trial-viewer.mdx +0 -0
  61. /package/{docs-evals → skills/evolve-evals/references}/getting-started/installation.mdx +0 -0
  62. /package/{docs-evals → skills/evolve-evals/references}/sdk-reference/agents.mdx +0 -0
  63. /package/{docs-evals → skills/evolve-evals/references}/sdk-reference/analyses.mdx +0 -0
  64. /package/{docs-evals → skills/evolve-evals/references}/sdk-reference/auth.mdx +0 -0
  65. /package/{docs-evals → skills/evolve-evals/references}/sdk-reference/checks.mdx +0 -0
  66. /package/{docs-evals → skills/evolve-evals/references}/sdk-reference/datasets.mdx +0 -0
  67. /package/{docs-evals → skills/evolve-evals/references}/sdk-reference/index.mdx +0 -0
  68. /package/{docs-evals → skills/evolve-evals/references}/sdk-reference/meta.mdx +0 -0
  69. /package/{docs-evals → skills/evolve-evals/references}/sdk-reference/skills.mdx +0 -0
  70. /package/{docs-evals → skills/evolve-evals/references}/sdk-reference/trials.mdx +0 -0
  71. /package/{docs-evals → skills/evolve-evals/references}/sdk-reference/types.mdx +0 -0
  72. /package/{docs-evals → skills/evolve-evals/references}/snippets/global-options.mdx +0 -0
  73. /package/{docs-evals → skills/evolve-evals/references}/snippets/list-flags.mdx +0 -0
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@evolvingmachines/evolve",
3
- "version": "0.0.55-project-sable.20260916.db0bc0bb",
3
+ "version": "0.0.56",
4
4
  "keywords": [
5
5
  "ai",
6
6
  "agents",
@@ -33,8 +33,6 @@
33
33
  "dist",
34
34
  "LICENSE",
35
35
  "spec",
36
- "docs-evals",
37
- "docs-agents",
38
36
  "skills",
39
37
  "hosted-error-codes.json",
40
38
  "harness-capabilities.json"
@@ -131,9 +129,9 @@
131
129
  },
132
130
  "dependencies": {
133
131
  "@agentclientprotocol/sdk": "^0.5.1",
134
- "@evolvingmachines/daytona": "0.0.55-project-sable.20260916.db0bc0bb",
135
- "@evolvingmachines/e2b": "0.0.55-project-sable.20260916.db0bc0bb",
136
- "@evolvingmachines/modal": "0.0.55-project-sable.20260916.db0bc0bb",
132
+ "@evolvingmachines/daytona": "0.0.56",
133
+ "@evolvingmachines/e2b": "0.0.56",
134
+ "@evolvingmachines/modal": "0.0.56",
137
135
  "ajv": "^8.17.1",
138
136
  "p-map": "^7.0.2",
139
137
  "smol-toml": "^1.7.0",
@@ -10,6 +10,8 @@ Hosted evals and the SDK, from the `evolve` command.
10
10
 
11
11
  Install: `npm i -g @evolvingmachines/evolve`, then `export EVOLVE_API_KEY=<your key>` (create a key at https://dashboard.evolvingmachines.ai/api-keys).
12
12
 
13
+ Python SDK: `pip install evolvingmachines-evolve` (`import evolve`). The manual still comes from the `evolve` command.
14
+
13
15
  ## Start here
14
16
 
15
17
  This file is a pointer, not the manual. The manual ships inside the CLI and always matches the installed version. Before running any `evolve` command, load it:
@@ -25,15 +27,14 @@ Read the index first, then the page for your topic, then write the command. Ever
25
27
  ## The other skills
26
28
 
27
29
  ```bash
28
- evolve skills get agents # the SDK: run agents (Claude, Codex, Gemini, ...) in sandboxes from TypeScript or Python
29
30
  evolve skills get create-task # write a new task in the Harbor format, verifier included
30
31
  evolve skills get rewardkit # write a task's verifier with Reward Kit
31
32
  evolve skills get create-adapter # convert an existing benchmark into a folder of Harbor-format tasks
32
33
  evolve skills get publish # publish a dataset of tasks, or upload a job you ran elsewhere
33
34
  ```
34
35
 
35
- `evolve skills list` names everything the installed version serves; `--full` on any skill adds its pages.
36
+ `evolve skills list` names everything the installed version serves; `--full` on any skill adds its pages. The SDK skill (run agents in sandboxes from TypeScript or Python) is not served by the command; install it from the repository: `npx skills add evolving-machines-lab/evolve --skill evolve-agents`.
36
37
 
37
38
  ## Inside the evolve repository
38
39
 
39
- An agent working in a checkout of https://github.com/evolving-machines-lab/evolve reads the same content directly: `docs-evals/SKILL.md` (the evals index; the pages sit beside it), `docs-agents/SKILL.md`, and `skills/<name>/SKILL.md` for the others; nothing needs to be installed.
40
+ An agent working in a checkout of https://github.com/evolving-machines-lab/evolve reads the same content directly from `skills/<name>/SKILL.md` (`evolve-evals` is the evals index, its pages under `references/`); nothing needs to be installed.
@@ -0,0 +1,112 @@
1
+ ---
2
+ # Generated by scripts/generate-skills.ts from docs-evals/. Do not edit; edit the source and run npm run generate:skills.
3
+ name: evolve-evals
4
+ description: "Evolve hosted evals: run any model on any agent harness against datasets of Harbor-format tasks, in cloud sandboxes, from the evolve CLI and the TypeScript and Python SDKs. Use when the user wants to: (1) start, watch, or read a hosted eval job (evolve run, evolve job, jobs()), (2) browse the dataset catalog or publish their own dataset (evolve dataset), (3) check task quality (evolve check) or analyze trial traces (evolve analyze), (4) inspect or download trials, artifacts and traces (evolve trial, trials()), (5) register their own agent, mount skills, attach secrets, or choose models and sandbox providers, (6) upload a job run elsewhere (evolve upload), (7) use the dashboard's jobs, datasets, checks, traces and trial viewer pages. For running agents in sandboxes from code (the SDK's run(), Swarm, Pipeline) install the SDK skill: `npx skills add evolving-machines-lab/evolve --skill evolve-agents`."
5
+ metadata:
6
+ internal: true
7
+ ---
8
+
9
+ # Evolve hosted evals
10
+
11
+ Hosted evaluation for agents: datasets of Harbor-format tasks, jobs that run any model on any agent harness against them in cloud sandboxes, and the trials, checks and analyses they produce — from the `evolve` CLI and the TypeScript and Python SDKs.
12
+
13
+ The pages under `references/` are the documentation site's pages, byte for byte, at the site's paths: a site link to `/core-concepts/tasks` is `references/core-concepts/tasks.mdx`, and `evolve skills get evals core-concepts/tasks` prints it. An `import` of `/snippets/<file>` is `references/snippets/<file>`.
14
+
15
+ ## How to use this skill
16
+
17
+ 1. Find the topic in the index below and read that page before writing any command or code: `evolve skills get evals <page>` prints it, the page named by its site path; `evolve skills get evals --full` prints every page.
18
+ 2. Every CLI verb is documented from its own `--help`; run `evolve <verb> --help` to confirm the flags of the installed version.
19
+ 3. Every command and every SDK client reads `EVOLVE_API_KEY`; the Installation page says where the key comes from.
20
+
21
+ ## Topic index
22
+
23
+ ## Docs
24
+
25
+ ### Getting started
26
+
27
+ | Page | What it covers |
28
+ | --- | --- |
29
+ | [Evolve documentation](references/index.mdx) | Environment-driven agent evaluations: any model, any harness, any sandbox, one API key. Every task versioned and checked, every trajectory analyzed. |
30
+ | [Installation](references/getting-started/installation.mdx) | Install the evolve command and the SDKs, and set your API key. |
31
+ | [Quick start](references/getting-started/quick-start.mdx) | Start your first job, watch it run, and download a trial. |
32
+
33
+ ### Core concepts
34
+
35
+ | Page | What it covers |
36
+ | --- | --- |
37
+ | [Core concepts](references/core-concepts/index.mdx) | How a dataset becomes trials, rewards, analyses and results. |
38
+ | [Tasks](references/core-concepts/tasks.mdx) | The task format: one directory with an instruction, an environment, a verifier, and an optional reference solution. |
39
+ | [Datasets](references/core-concepts/datasets.mdx) | The catalog, and publishing your own dataset. |
40
+ | [Jobs](references/core-concepts/jobs.mdx) | Start a job, watch it, read its results, and derive new jobs from it. |
41
+ | [Trials](references/core-concepts/trials.mdx) | One agent's attempt at one task: its record, its trace, and its artifacts. |
42
+ | [Trial outputs](references/core-concepts/trial-outputs.mdx) | What a finished trial gives you, file by file, and in which formats. |
43
+ | [Agents](references/core-concepts/agents.mdx) | The built-in harnesses, how to configure an arm, and how to register your own agent. |
44
+ | [Models](references/core-concepts/models.mdx) | How a model is named on -m, and why one API key covers every provider. |
45
+ | [Sandboxes](references/core-concepts/sandboxes.mdx) | Where trials run: e2b, daytona, or modal. |
46
+ | [Analyze](references/core-concepts/analyze.mdx) | Judge every trial's trace against a rubric, on the platform. |
47
+ | [Check](references/core-concepts/check.mdx) | Check the quality of tasks against a rubric before you spend a job on them. |
48
+ | [Skills](references/core-concepts/skills.mdx) | Mount skills into every agent run, from a git source or from your uploads. |
49
+ | [Secrets](references/core-concepts/secrets.mdx) | Store env secrets once and attach them to jobs by name. |
50
+ | [Upload a job](references/core-concepts/upload.mdx) | Bring a job you ran elsewhere onto the platform as a finished job. |
51
+
52
+ ### Dashboard
53
+
54
+ | Page | What it covers |
55
+ | --- | --- |
56
+ | [Datasets](references/dashboard/datasets.mdx) | Browse the catalog, and each dataset's tasks and versions, in the dashboard. |
57
+ | [Jobs](references/dashboard/jobs.mdx) | Start a job from the browser, follow it, and read its results. |
58
+ | [Trial viewer](references/dashboard/trial-viewer.mdx) | Read one trial's trace event by event, its tool calls, its spend, and download its artifacts. |
59
+ | [Checks](references/dashboard/checks.mdx) | Read a job's task quality checks in the dashboard. |
60
+ | [Traces](references/dashboard/traces.mdx) | Every recorded run in one list. |
61
+
62
+ ### SDK
63
+
64
+ | Page | What it covers |
65
+ | --- | --- |
66
+ | [TypeScript SDK](references/sdk/typescript.mdx) | Start a job, wait for it, read the result, and download it from TypeScript. |
67
+ | [Python SDK](references/sdk/python.mdx) | Start a job, wait for it, read the result, and download it from Python. |
68
+
69
+ ## CLI reference
70
+
71
+ ### The evolve command
72
+
73
+ | Page | What it covers |
74
+ | --- | --- |
75
+ | [CLI reference](references/cli-reference/index.mdx) | Every verb of the evolve command, from its own help output. |
76
+ | [evolve run](references/cli-reference/run.mdx) | Start a job, and optionally follow it to the end. |
77
+ | [evolve analyze](references/cli-reference/analyze.mdx) | Analyze a finished job's trial traces against a rubric, on the platform. |
78
+ | [evolve check](references/cli-reference/check.mdx) | Check task quality against a rubric, and read checks back. |
79
+ | [evolve upload](references/cli-reference/upload.mdx) | Upload a job directory as a finished job. |
80
+
81
+ ### Command groups
82
+
83
+ | Page | What it covers |
84
+ | --- | --- |
85
+ | [evolve job](references/cli-reference/job.mdx) | Start, follow, and derive jobs. |
86
+ | [evolve trial](references/cli-reference/trial.mdx) | Inspect, download, and act on single trials. |
87
+ | [evolve analysis](references/cli-reference/analysis.mdx) | List, inspect, and download trace-analysis runs. |
88
+ | [evolve dataset](references/cli-reference/dataset.mdx) | Browse and publish the dataset catalog. |
89
+ | [evolve skill](references/cli-reference/skill.mdx) | Upload and manage platform-stored skills. |
90
+ | [evolve skills](references/cli-reference/skills.mdx) | The skills the CLI serves to coding agents. |
91
+ | [evolve agent](references/cli-reference/agent.mdx) | Register and manage your own agents. |
92
+ | [evolve auth](references/cli-reference/auth.mdx) | Identity, API keys, and your organizations. |
93
+ | [evolve secrets](references/cli-reference/secrets.mdx) | Store and manage env secrets. Values are write-only; reads return metadata. |
94
+
95
+ ## SDK reference
96
+
97
+ ### Clients
98
+
99
+ | Page | What it covers |
100
+ | --- | --- |
101
+ | [Clients](references/sdk-reference/index.mdx) | How the hosted clients are built and configured. |
102
+ | [jobs](references/sdk-reference/jobs.mdx) | Start, follow, and derive jobs. |
103
+ | [trials](references/sdk-reference/trials.mdx) | Read, download, and act on single trials. |
104
+ | [datasets](references/sdk-reference/datasets.mdx) | The catalog: list, read, publish, download, activate, delete. |
105
+ | [analyses](references/sdk-reference/analyses.mdx) | Read trace-analysis runs. |
106
+ | [checks](references/sdk-reference/checks.mdx) | Start and read task quality checks. |
107
+ | [skills](references/sdk-reference/skills.mdx) | Upload and manage platform-stored skills. |
108
+ | [agents](references/sdk-reference/agents.mdx) | Register and manage your own agents. |
109
+ | [auth and orgs](references/sdk-reference/auth.mdx) | Who you are, and your organizations. |
110
+ | [meta](references/sdk-reference/meta.mdx) | The platform's capability document, no key needed. |
111
+ | [Types](references/sdk-reference/types.mdx) | Every field of the objects the clients return: name, type, meaning, and when it is null. |
112
+ | [Errors](references/sdk-reference/errors.mdx) | The error class every refused request raises, and every code it can carry. |
@@ -15,7 +15,7 @@ Start a job. `evolve run` is the short form of `evolve job start`, and both take
15
15
  evolve run \
16
16
  -d terminal-bench-4@4.0 \
17
17
  -a codex \
18
- -m gpt-5.5 \
18
+ -m gpt-6-astra \
19
19
  -k 2 \
20
20
  -l 5 \
21
21
  --watch
@@ -9,7 +9,7 @@ import GlobalOptions from '/snippets/global-options.mdx';
9
9
  evolve skills <command> [options]
10
10
  ```
11
11
 
12
- The skills ship with the CLI and match its version. `skills get evals` is the index of these docs and `skills get evals <page>` one page of them. The `evolve` pointer skill that `skills install` writes is served but never listed. `EVOLVE_SKILLS_DIR` names another checkout or package root to serve.
12
+ The skills ship with the CLI and match its version. `skills get evals` is the index of these docs and `skills get evals <page>` one page of them. The `evolve` pointer skill that `skills install` writes is served but never listed. The SDK skill is not served; it installs from the repository with `npx skills add evolving-machines-lab/evolve --skill evolve-agents`. `EVOLVE_SKILLS_DIR` names another skills directory to serve.
13
13
 
14
14
  ## skills list
15
15
 
@@ -26,10 +26,10 @@ evolve skills get <name> [name...]
26
26
  evolve skills get <name> <page>
27
27
  ```
28
28
 
29
- Print one or more skills, or one page. A second word that is not a skill name is a page of the first skill: this site's own path without the suffix (`core-concepts/tasks`), or a file under `references/` for a task-authoring skill.
29
+ Print one or more skills, or one page. A second word that is not a skill name is a page of the first skill: its path under `references/` without the suffix, as this site spells it (`core-concepts/tasks`).
30
30
 
31
31
  <ParamField path="--full">
32
- Also print every page of the skill (a docs folder's pages, or `references/` and `templates/`), each behind a `--- <path> ---` line.
32
+ Also print every file under `references/` and `templates/`, each behind a `--- <path> ---` line.
33
33
  </ParamField>
34
34
 
35
35
  <ParamField path="--all">
@@ -48,7 +48,7 @@ evolve skills get create-task --full
48
48
  evolve skills path [name]
49
49
  ```
50
50
 
51
- Print the root the skills are served from, or one skill's folder.
51
+ Print the skills directory, or one skill's folder.
52
52
 
53
53
  ## skills install
54
54
 
@@ -34,7 +34,7 @@ These flags apply to every arm of the job.
34
34
  evolve run \
35
35
  -d terminal-bench-4@4.0 \
36
36
  -a codex \
37
- -m gpt-5.5 \
37
+ -m gpt-6-astra \
38
38
  -l 5 \
39
39
  --effort high \
40
40
  --preset no-internet \
@@ -56,7 +56,7 @@ evolve agent add acme-cli --install-script ./install.sh --run "acme-cli --headle
56
56
  evolve run \
57
57
  -d terminal-bench-4@4.0 \
58
58
  -a acme-cli \
59
- -m gpt-5.5 \
59
+ -m gpt-6-astra \
60
60
  -l 5 \
61
61
  --watch
62
62
  ```
@@ -15,7 +15,7 @@ Or arm it when you start the job, so each trial is analyzed the moment it settle
15
15
  evolve run \
16
16
  -d terminal-bench-4@4.0 \
17
17
  -a codex \
18
- -m gpt-5.5 \
18
+ -m gpt-6-astra \
19
19
  --analyze \
20
20
  -l 5 \
21
21
  --watch
@@ -9,7 +9,7 @@ A job runs every task of its datasets with every agent arm, a fixed number of at
9
9
  evolve run \
10
10
  -d terminal-bench-4@4.0 \
11
11
  -a codex \
12
- -m gpt-5.5 \
12
+ -m gpt-6-astra \
13
13
  -k 2 \
14
14
  -l 5 \
15
15
  --watch
@@ -25,7 +25,7 @@ evolve run \
25
25
  evolve run \
26
26
  -d terminal-bench-4@4.0 \
27
27
  -a codex \
28
- -m gpt-5.5 \
28
+ -m gpt-6-astra \
29
29
  -i 'auth-*' \
30
30
  -x 'auth-legacy' \
31
31
  -l 20 \
@@ -119,7 +119,7 @@ datasets:
119
119
  version: "1.0"
120
120
  agents:
121
121
  - name: codex
122
- model_name: gpt-5.5
122
+ model_name: gpt-6-astra
123
123
  - name: claude
124
124
  model_name: fable
125
125
  n_attempts: 2
@@ -15,7 +15,7 @@ evolve run \
15
15
  evolve run \
16
16
  -d terminal-bench-4@4.0 \
17
17
  -a codex \
18
- -m gpt-5.5 \
18
+ -m gpt-6-astra \
19
19
  -m gpt-5.6-terra \
20
20
  -l 5 \
21
21
  --watch
@@ -13,7 +13,7 @@ Every trial runs in its own sandbox on one of three providers. Pick the provider
13
13
  evolve run \
14
14
  -d terminal-bench-4@4.0 \
15
15
  -a codex \
16
- -m gpt-5.5 \
16
+ -m gpt-6-astra \
17
17
  -e modal \
18
18
  -l 5 \
19
19
  --watch
@@ -34,7 +34,7 @@ Three refusals appear in the verdicts: multi-container tasks run on `e2b` and `d
34
34
  | Provider | Max vCPUs | Max memory | Disk |
35
35
  | --- | --- | --- | --- |
36
36
  | `e2b` | 8 | 8192 MB | fixed, 20 GB |
37
- | `daytona` | 4 | 8192 MB | sized per task, up to 10 GB |
37
+ | `daytona` | 4 | 8192 MB | sized per task, up to 25 GB |
38
38
  | `modal` | 16 | 32768 MB | fixed, 512 GB |
39
39
 
40
40
  Two limits show only when the sandbox is created, as an `INFRASTRUCTURE_ERROR` with the reason in the trial's failure detail. On `daytona` an `allowlist` holds one kind of entry: IP addresses and CIDRs for a single-container task (at most 10), hostnames for a multi-container task (at most 20, `*.` covering a domain). `e2b` and `modal` take both kinds.
@@ -24,7 +24,7 @@ evolve secrets delete GITHUB_TOKEN --label staging
24
24
  evolve run \
25
25
  -d terminal-bench-4@4.0 \
26
26
  -a codex \
27
- -m gpt-5.5 \
27
+ -m gpt-6-astra \
28
28
  --secret GITHUB_TOKEN \
29
29
  -l 5 \
30
30
  --watch
@@ -74,7 +74,7 @@ Its top-level fields are `schema_version`, `agent` with the harness's name, vers
74
74
  ```json
75
75
  {
76
76
  "schema_version": "ATIF-v1.7",
77
- "agent": {"name": "codex", "version": "0.31.0", "model_name": "gpt-5.5"},
77
+ "agent": {"name": "codex", "version": "0.31.0", "model_name": "gpt-6-astra"},
78
78
  "steps": [
79
79
  {"step_id": 1, "source": "user", "message": "Create hello.txt containing Hello."},
80
80
  {
@@ -34,7 +34,7 @@ This page takes you from an empty terminal to a downloaded trial: one API key, a
34
34
  evolve run \
35
35
  -d terminal-bench-4@4.0 \
36
36
  -a codex \
37
- -m gpt-5.5 \
37
+ -m gpt-6-astra \
38
38
  -l 5 \
39
39
  --watch
40
40
  ```
@@ -15,7 +15,7 @@ export EVOLVE_API_KEY="<your key>" # from https://dashboard.evolvingmachines.a
15
15
  evolve run \
16
16
  -d terminal-bench-4@4.0 \
17
17
  -a codex \
18
- -m gpt-5.5 \
18
+ -m gpt-6-astra \
19
19
  -l 5 \
20
20
  --watch
21
21
  ```
@@ -23,7 +23,7 @@ evals = jobs()
23
23
  ```python
24
24
  job = await evals.start(
25
25
  datasets=[{"name": "terminal-bench-4", "version": "4.0", "n_tasks": 5}],
26
- agents=[{"name": "codex", "model_name": "gpt-5.5"}],
26
+ agents=[{"name": "codex", "model_name": "gpt-6-astra"}],
27
27
  n_attempts=1,
28
28
  n_concurrent_trials=4,
29
29
  max_trial_spend_usd=25,
@@ -20,7 +20,7 @@ const evals = jobs();
20
20
  ```ts
21
21
  const job = await evals.start({
22
22
  datasets: [{ name: "terminal-bench-4", version: "4.0", n_tasks: 5 }],
23
- agents: [{ name: "codex", model_name: "gpt-5.5" }],
23
+ agents: [{ name: "codex", model_name: "gpt-6-astra" }],
24
24
  n_attempts: 1,
25
25
  n_concurrent_trials: 4,
26
26
  max_trial_spend_usd: 25,
@@ -24,7 +24,7 @@ A refused request raises one class. Branch on its `code`, never on the message.
24
24
 
25
25
  ```ts
26
26
  try {
27
- await jobs().start({ datasets: [{ name: "terminal-bench-4" }], agents: [{ name: "codex", model_name: "gpt-5.5" }], sandbox_provider: "modal" });
27
+ await jobs().start({ datasets: [{ name: "terminal-bench-4" }], agents: [{ name: "codex", model_name: "gpt-6-astra" }], sandbox_provider: "modal" });
28
28
  } catch (err) {
29
29
  if (err instanceof EvolveApiError && err.code === "provider_unsupported") {
30
30
  const { refused_tasks } = err.details as { refused_tasks: { task_name: string; reason: string }[] };
@@ -50,7 +50,7 @@ A refused request raises one class. Branch on its `code`, never on the message.
50
50
 
51
51
  ```python
52
52
  try:
53
- await jobs().start(datasets=[{"name": "terminal-bench-4"}], agents=[{"name": "codex", "model_name": "gpt-5.5"}], sandbox_provider="modal")
53
+ await jobs().start(datasets=[{"name": "terminal-bench-4"}], agents=[{"name": "codex", "model_name": "gpt-6-astra"}], sandbox_provider="modal")
54
54
  except EvolveAPIError as err:
55
55
  if err.code == "provider_unsupported":
56
56
  refused = (err.details or {}).get("refused_tasks", [])
@@ -23,7 +23,7 @@ description: "Start, follow, and derive jobs."
23
23
  ```ts
24
24
  const job = await jobs().start({
25
25
  datasets: [{ name: "terminal-bench-4", version: "4.0", n_tasks: 5 }],
26
- agents: [{ name: "codex", model_name: "gpt-5.5" }],
26
+ agents: [{ name: "codex", model_name: "gpt-6-astra" }],
27
27
  });
28
28
  ```
29
29
  </Tab>
@@ -57,7 +57,7 @@ description: "Start, follow, and derive jobs."
57
57
  ```python
58
58
  job = await jobs().start(
59
59
  datasets=[{"name": "terminal-bench-4", "version": "4.0", "n_tasks": 5}],
60
- agents=[{"name": "codex", "model_name": "gpt-5.5"}],
60
+ agents=[{"name": "codex", "model_name": "gpt-6-astra"}],
61
61
  )
62
62
  ```
63
63
  </Tab>
@@ -1,138 +0,0 @@
1
- ---
2
- # Generated by scripts/generate-skills.ts from docs-agents/SKILL.source.md. Do not edit; edit the source and run npm run generate:skills.
3
- name: docs-agents
4
- description: "Evolve SDK development for TypeScript and Python. Use when building applications with Evolve to run AI agents (Claude, Codex, Gemini, Qwen, Kimi, OpenCode, Droid) in secure sandboxes. Triggers: (1) Creating Evolve applications, (2) Configuring agents with skills, Integrations, MCP servers, (3) Using Swarm abstractions (map, filter, reduce, bestOf/best_of, verify), (4) Building Pipelines, (5) Structured output with schemas, (6) Session management, streaming, observability, (7) Checkpointing, storage & StorageClient, (8) Cost tracking (per-run and per-session spend), (9) Historical sessions & trace download via sessions() client. For hosted evals (datasets, jobs, trials, checks, analyses, the evolve CLI) read `evolve skills get evals`."
5
- metadata:
6
- internal: true
7
- ---
8
-
9
- # Evolve SDK
10
-
11
- Build applications that run CLI agents in secure cloud sandboxes.
12
-
13
- **Repo:** https://github.com/evolving-machines-lab/evolve
14
-
15
- Hosted evals — datasets, jobs, trials, checks, analyses and the `evolve` CLI — are `evolve skills get evals`, not this skill.
16
-
17
- ## Language Detection
18
-
19
- Determine the language from (in priority order):
20
-
21
- 1. **User specification** — if the user states a language, use it
22
- 2. **Project signals** — imports, file extensions, package.json vs pyproject.toml
23
- 3. **Ask** — if ambiguous, ask the user
24
-
25
- - **TypeScript** (`@evolvingmachines/evolve`) — read from [typescript/](typescript/)
26
- - **Python** (`evolvingmachines-evolve`) — read from [python/](python/)
27
-
28
- ## Required Reading
29
-
30
- Always read these three references **for the detected language** before writing any Evolve code:
31
-
32
- **TypeScript:**
33
- - [01-getting-started.md](typescript/01-getting-started.md) — Installation, authentication (Gateway, managed BYO provider keys, direct provider-key mode), core lifecycle, streaming basics, agent reference table
34
- - [02-configuration.md](typescript/02-configuration.md) — Sandbox providers, full builder API, agent skills catalog, Managed integrations, MCP servers
35
- - [03-runtime.md](typescript/03-runtime.md) — run(), executeCommand(), upload/download files, session controls, workspace layout, structured output, session management, storage & checkpointing, StorageClient, sessions() client, cost tracking, observability, error handling
36
-
37
- **Python:**
38
- - [01-getting-started.md](python/01-getting-started.md) — Installation, authentication (Gateway, managed BYO provider keys, direct provider-key mode), core lifecycle, streaming basics, agent reference table
39
- - [02-configuration.md](python/02-configuration.md) — Sandbox providers, full constructor API, agent skills catalog, Managed integrations, MCP servers
40
- - [03-runtime.md](python/03-runtime.md) — run(), execute_command(), upload/download files, session controls, workspace layout, structured output, session management, storage & checkpointing, StorageClient, sessions() client, cost tracking, observability, error handling
41
-
42
- ## Critical Constraints
43
-
44
- - **Model names** — Only use exact names from the Agent Reference table. Do not invent or guess model identifiers.
45
- - [TS](typescript/01-getting-started.md#agent-reference) | [PY](python/01-getting-started.md#agent-reference)
46
- - **Cleanup** — Always call `kill()` when done. Sandboxes bill until destroyed.
47
- - [TS](typescript/01-getting-started.md#core-lifecycle) | [PY](python/01-getting-started.md#core-lifecycle)
48
-
49
- ## Additional References
50
-
51
- Read on demand when the user's task requires them:
52
-
53
- | When to read | TypeScript | Python |
54
- |-------------|-----------|--------|
55
- | Building a UI, handling real-time events | [04-streaming.md](typescript/04-streaming.md) | [04-streaming.md](python/04-streaming.md) |
56
- | Parallel agents (map/filter/reduce/bestOf/verify), Pipeline chaining | [05-swarm-pipeline.md](typescript/05-swarm-pipeline.md) | [05-swarm-pipeline.md](python/05-swarm-pipeline.md) |
57
-
58
- ## Topic Index
59
-
60
- ### Getting Started
61
-
62
- | Topic | TypeScript | Python |
63
- |-------|-----------|--------|
64
- | Installation & requirements | [TS](typescript/01-getting-started.md#installation) | [PY](python/01-getting-started.md#installation) |
65
- | Quick start (3 steps) | [TS](typescript/01-getting-started.md#quick-start) | [PY](python/01-getting-started.md#quick-start) |
66
- | Core lifecycle (run, output, kill) | [TS](typescript/01-getting-started.md#core-lifecycle) | [PY](python/01-getting-started.md#core-lifecycle) |
67
- | Streaming basics | [TS](typescript/01-getting-started.md#streaming) | [PY](python/01-getting-started.md#streaming) |
68
- | Gateway, managed BYO provider keys, and direct provider-key mode | [TS](typescript/01-getting-started.md#authentication) | [PY](python/01-getting-started.md#authentication) |
69
- | BYO subscriptions (Claude Max, Codex, Gemini) | [TS](typescript/01-getting-started.md#byo-claude-max-subscription) | [PY](python/01-getting-started.md#byo-claude-max-subscription) |
70
- | Supported agents, models & defaults | [TS](typescript/01-getting-started.md#agent-reference) | [PY](python/01-getting-started.md#agent-reference) |
71
- | Harness and model pairing (per-harness constraints) | [TS](typescript/01-getting-started.md#harness-and-model-pairing) | [PY](python/01-getting-started.md#harness-and-model-pairing) |
72
-
73
- ### Configuration
74
-
75
- | Topic | TypeScript | Python |
76
- |-------|-----------|--------|
77
- | Sandbox providers (E2B, Modal, Daytona) | [TS](typescript/02-configuration.md#sandbox-providers) | [PY](python/02-configuration.md#sandbox-providers) |
78
- | Provider auto-resolution from env | [TS](typescript/02-configuration.md#auto-resolution) | [PY](python/02-configuration.md#auto-resolution) |
79
- | Sandbox create options (image, network, user, homeDir) | [TS](typescript/02-configuration.md#sandbox-create-options) | [PY](python/02-configuration.md#sandbox-create-options) |
80
- | Workspace modes (knowledge / swe) | [TS](typescript/02-configuration.md#workspace-modes) | [PY](python/02-configuration.md#workspace-modes) |
81
- | Full builder/constructor API | [TS](typescript/02-configuration.md#evolve-instance) | [PY](python/02-configuration.md#evolve-instance) |
82
- | Browser automation guide (setup, live view, replay) | [TS](typescript/02-configuration.md#browser-automation) | [PY](python/02-configuration.md#browser-automation) |
83
- | Browser credentials (saved website logins) | [TS](typescript/02-configuration.md#browser-credentials) | [PY](python/02-configuration.md#browser-credentials) |
84
- | Agent plugins/extensions | [TS](typescript/02-configuration.md#agent-plugins) | [PY](python/02-configuration.md#agent-plugins) |
85
- | Agent skills catalog | [TS](typescript/02-configuration.md#agent-skills) | [PY](python/02-configuration.md#agent-skills) |
86
- | Managed secrets (Dashboard-stored env secrets) | [TS](typescript/02-configuration.md#managed-secrets) | [PY](python/02-configuration.md#managed-secrets) |
87
- | Managed integrations (auth paths, tool filtering, types) | [TS](typescript/02-configuration.md#managed-integrations) | [PY](python/02-configuration.md#managed-integrations) |
88
- | MCP server config (STDIO / HTTP / SSE) | [TS](typescript/02-configuration.md#evolve-instance) | [PY](python/02-configuration.md#evolve-instance) |
89
-
90
- ### Runtime
91
-
92
- | Topic | TypeScript | Python |
93
- |-------|-----------|--------|
94
- | run() options (timeout, background, checkpoint) | [TS](typescript/03-runtime.md#run) | [PY](python/03-runtime.md#run) |
95
- | executeCommand() / execute_command() | [TS](typescript/03-runtime.md#executecommand) | [PY](python/03-runtime.md#execute_command) |
96
- | Upload files to sandbox | [TS](typescript/03-runtime.md) | [PY](python/03-runtime.md) |
97
- | Download output files | [TS](typescript/03-runtime.md) | [PY](python/03-runtime.md) |
98
- | Session controls (interrupt, pause, resume, kill) | [TS](typescript/03-runtime.md#session-controls) | [PY](python/03-runtime.md#session-controls) |
99
- | Port forwarding | [TS](typescript/03-runtime.md#gethost) | [PY](python/03-runtime.md#get_host) |
100
- | Workspace filesystem layout | [TS](typescript/03-runtime.md) | [PY](python/03-runtime.md) |
101
- | Structured output (Zod / Pydantic / JSON Schema) | [TS](typescript/03-runtime.md#structured-output) | [PY](python/03-runtime.md#structured-output) |
102
- | Multi-turn conversations | [TS](typescript/03-runtime.md#session-management) | [PY](python/03-runtime.md#session-management) |
103
- | Pause, resume, reconnect, switch sandboxes | [TS](typescript/03-runtime.md#session-management) | [PY](python/03-runtime.md#session-management) |
104
- | Storage & checkpointing (gateway mode) | [TS](typescript/03-runtime.md#storage--checkpointing) | [PY](python/03-runtime.md#storage--checkpointing) |
105
- | StorageClient (list, get, download checkpoints) | [TS](typescript/03-runtime.md#listing--browsing-checkpoints) | [PY](python/03-runtime.md#listing--browsing-checkpoints) |
106
- | Checkpoint lineage & restore | [TS](typescript/03-runtime.md#checkpoint-lineage) | [PY](python/03-runtime.md#checkpoint-lineage) |
107
- | Historical sessions & trace download | [TS](typescript/03-runtime.md#historical-sessions--trace-download) | [PY](python/03-runtime.md#historical-sessions--trace-download) |
108
- | Cost tracking (per-run & per-session spend) | [TS](typescript/03-runtime.md#cost-tracking) | [PY](python/03-runtime.md#cost-tracking) |
109
- | Observability (dashboard + local logs) | [TS](typescript/03-runtime.md#observability) | [PY](python/03-runtime.md#observability) |
110
- | Error handling | [TS](typescript/03-runtime.md#error-handling) | [PY](python/03-runtime.md#error-handling) |
111
-
112
- ### Streaming
113
-
114
- | Topic | TypeScript | Python |
115
- |-------|-----------|--------|
116
- | Event listeners (content, lifecycle, stdout, stderr) | [TS](typescript/04-streaming.md#event-listeners) | [PY](python/04-streaming.md#event-listeners) |
117
- | LifecycleEvent & LifecycleReason | [TS](typescript/04-streaming.md#lifecycleevent) | [PY](python/04-streaming.md#lifecycleevent-typeddict-shape) |
118
- | OutputEvent & SessionUpdate types | [TS](typescript/04-streaming.md#sessionupdate-types) | [PY](python/04-streaming.md#event-types-summary) |
119
- | Tool events (ToolCall, ToolCallUpdate, ToolKind) | [TS](typescript/04-streaming.md#tool-events) | [PY](python/04-streaming.md#toolkind-reference) |
120
- | Browser lifecycle event fields | [TS](typescript/04-streaming.md#browser-automation-streaming) | [PY](python/04-streaming.md#browser-automation-streaming) |
121
- | UI integration example | [TS](typescript/04-streaming.md#ui-integration-example) | [PY](python/04-streaming.md#ui-integration-example) |
122
-
123
- ### Swarm & Pipeline
124
-
125
- | Topic | TypeScript | Python |
126
- |-------|-----------|--------|
127
- | Swarm setup (config, concurrency, retry) | [TS](typescript/05-swarm-pipeline.md) | [PY](python/05-swarm-pipeline.md) |
128
- | Input types (FileMap, folders) | [TS](typescript/05-swarm-pipeline.md#input-types) | [PY](python/05-swarm-pipeline.md#input-types) |
129
- | bestOf / best_of (N candidates + judge) | [TS](typescript/05-swarm-pipeline.md#bestof) | [PY](python/05-swarm-pipeline.md#best_of) |
130
- | map (parallel processing) | [TS](typescript/05-swarm-pipeline.md#map) | [PY](python/05-swarm-pipeline.md#map) |
131
- | filter (evaluate + threshold) | [TS](typescript/05-swarm-pipeline.md#filter) | [PY](python/05-swarm-pipeline.md#filter) |
132
- | reduce (synthesize many to one) | [TS](typescript/05-swarm-pipeline.md#reduce) | [PY](python/05-swarm-pipeline.md#reduce) |
133
- | verify (quality gate with feedback loop) | [TS](typescript/05-swarm-pipeline.md#verify-quality-gate) | [PY](python/05-swarm-pipeline.md#verify-quality-gate) |
134
- | Result types (SwarmResult, ReduceResult, BestOfResult) | [TS](typescript/05-swarm-pipeline.md#result-types) | [PY](python/05-swarm-pipeline.md#result-types) |
135
- | Chaining operations | [TS](typescript/05-swarm-pipeline.md#chaining-operations) | [PY](python/05-swarm-pipeline.md#chaining-operations) |
136
- | Pipeline (fluent chaining, events, terminal) | [TS](typescript/05-swarm-pipeline.md#pipeline) | [PY](python/05-swarm-pipeline.md#pipeline) |
137
-
138
- The installed CLI serves the current version of this skill: `evolve skills get agents --full`.
@@ -1,81 +0,0 @@
1
- ## Documentation
2
-
3
- - [TypeScript SDK](./typescript/)
4
- - [Python SDK](./python/)
5
- - [Cookbooks](https://github.com/evolving-machines-lab/evolve/tree/main/cookbooks)
6
- - [Changelog](https://github.com/evolving-machines-lab/evolve/blob/main/CHANGELOG.md)
7
-
8
- The SDK has two halves. One runs a CLI agent in a sandbox you control — that is the quickstart below. The other scores agents against datasets of tasks on Evolve's own infrastructure: you start a job, and read back per-trial rewards. Hosted evals are documented at https://docs.evolvingmachines.ai: `jobs()`, `datasets()`, and the `evolve` CLI.
9
-
10
- ## Get Started
11
-
12
- ### 1. Install the SDK
13
-
14
- ```bash
15
- npm install @evolvingmachines/evolve # TypeScript
16
- pip install evolvingmachines-evolve # Python
17
- ```
18
-
19
- **Note:** Requires [Node.js 18+](https://nodejs.org/) (the Python SDK uses a lightweight Node.js bridge).
20
-
21
- ### 2. Run your first agent
22
-
23
- Get your Evolve API key at [dashboard.evolvingmachines.ai](https://dashboard.evolvingmachines.ai) ([see 3. below](#evolve-gateway)):
24
- ```bash
25
- # .env - Gateway
26
- EVOLVE_API_KEY=sk-...
27
- ```
28
-
29
- To bring your own provider billing while keeping gateway features, save an Anthropic or OpenAI key in Dashboard → Secrets → BYO Provider Keys — those are the two providers this route serves today. Your app still uses only `EVOLVE_API_KEY`.
30
-
31
- For fully local direct provider keys:
32
- ```bash
33
- # .env - Direct Provider Key Mode (local BYOK)
34
- ANTHROPIC_API_KEY=sk-ant-... # or CLAUDE_CODE_OAUTH_TOKEN (Claude Max), OPENAI_API_KEY, GEMINI_API_KEY
35
- E2B_API_KEY=e2b_... # sandbox provider, get at https://e2b.dev
36
- ```
37
-
38
- Then run:
39
- ```typescript
40
- import { Evolve } from "@evolvingmachines/evolve";
41
-
42
- const evolve = new Evolve(); // auto-resolves env variables
43
- await evolve.run({ prompt: "Create hello.txt with 'Hello World'" });
44
- const output = await evolve.getOutputFiles(); // output.files
45
- ```
46
-
47
- ```python
48
- from evolve import Evolve
49
-
50
- evolve = Evolve() # auto-resolves env variables
51
- await evolve.run(prompt="Create hello.txt with 'Hello World'")
52
- output = await evolve.get_output_files() # output.files
53
- ```
54
-
55
- <a id="evolve-gateway"></a>
56
-
57
- ### 3. Unlock full power with Evolve API key
58
-
59
- Sign up at [dashboard.evolvingmachines.ai](https://dashboard.evolvingmachines.ai/) and get your **Evolve API key** for:
60
- - Agent execution traces, observability and analytics
61
- - Centralized billing across all providers
62
- - Mix any model with any CLI agent
63
- - $10 FREE CREDITS, no CC required
64
-
65
- ### 4. Learn more
66
-
67
- Check out the [documentation](https://github.com/evolving-machines-lab/evolve/tree/main/docs) and [cookbooks](https://github.com/evolving-machines-lab/evolve/tree/main/cookbooks).
68
-
69
- ## Support + Talk with Founders
70
-
71
- - [Community Discord](https://discord.gg/Q36D8dGyNF)
72
- - [Schedule Demo](https://cal.com/brando-magnani/evolve-1-1-onboarding-chat)
73
- - Email: [brandomagnani@evolvingmachines.ai](mailto:brandomagnani@evolvingmachines.ai)
74
-
75
- ## Reporting Bugs
76
-
77
- We welcome your feedback. File a [GitHub issue](https://github.com/evolving-machines-lab/evolve/issues) to report bugs or request features.
78
-
79
- ## License
80
-
81
- See the [LICENSE](https://github.com/evolving-machines-lab/evolve/blob/main/LICENSE) file for full terms and conditions.