@evolvingmachines/evolve 0.0.65 → 0.0.67

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (152) hide show
  1. package/dist/{chunk-DK2MGUPB.js → chunk-RFLMRPGK.js} +5 -5
  2. package/dist/cli/index.cjs +39 -39
  3. package/dist/cli/index.d.cts +3 -3
  4. package/dist/cli/index.d.ts +3 -3
  5. package/dist/cli/index.js +30 -30
  6. package/dist/index.cjs +1 -1
  7. package/dist/index.d.cts +4 -4
  8. package/dist/index.d.ts +4 -4
  9. package/dist/index.js +1 -1
  10. package/dist/{types-B_qmzlMk.d.cts → types-84M4J_Vm.d.cts} +11 -44
  11. package/dist/{types-B_qmzlMk.d.ts → types-84M4J_Vm.d.ts} +11 -44
  12. package/package.json +4 -4
  13. package/skills/create-adapter/SKILL.md +71 -26
  14. package/skills/create-task/SKILL.md +124 -66
  15. package/skills/evolve/SKILL.md +37 -20
  16. package/skills/evolve-evals/SKILL.md +142 -69
  17. package/skills/evolve-evals/references/cli-reference/agent.md +76 -0
  18. package/skills/evolve-evals/references/cli-reference/analysis.md +78 -0
  19. package/skills/evolve-evals/references/cli-reference/analyze.md +95 -0
  20. package/skills/evolve-evals/references/cli-reference/auth.md +96 -0
  21. package/skills/evolve-evals/references/cli-reference/check.md +156 -0
  22. package/skills/evolve-evals/references/cli-reference/dataset.md +154 -0
  23. package/skills/evolve-evals/references/cli-reference/filesystem.md +168 -0
  24. package/skills/evolve-evals/references/cli-reference/index.md +149 -0
  25. package/skills/evolve-evals/references/cli-reference/job.md +198 -0
  26. package/skills/evolve-evals/references/cli-reference/run.md +187 -0
  27. package/skills/evolve-evals/references/cli-reference/secrets.md +77 -0
  28. package/skills/evolve-evals/references/cli-reference/session.md +56 -0
  29. package/skills/evolve-evals/references/cli-reference/skill.md +52 -0
  30. package/skills/evolve-evals/references/cli-reference/skills.md +85 -0
  31. package/skills/evolve-evals/references/cli-reference/trial.md +115 -0
  32. package/skills/evolve-evals/references/cli-reference/upload.md +72 -0
  33. package/skills/evolve-evals/references/core-concepts/agents.md +96 -0
  34. package/skills/evolve-evals/references/core-concepts/analyze.md +143 -0
  35. package/skills/evolve-evals/references/core-concepts/check.md +116 -0
  36. package/skills/evolve-evals/references/core-concepts/compatibility.md +39 -0
  37. package/skills/evolve-evals/references/core-concepts/compose.md +103 -0
  38. package/skills/evolve-evals/references/core-concepts/datasets.md +234 -0
  39. package/skills/evolve-evals/references/core-concepts/index.md +52 -0
  40. package/skills/evolve-evals/references/core-concepts/jobs.md +223 -0
  41. package/skills/evolve-evals/references/core-concepts/models.md +73 -0
  42. package/skills/evolve-evals/references/core-concepts/multi-step.md +93 -0
  43. package/skills/evolve-evals/references/core-concepts/rewardkit.md +341 -0
  44. package/skills/evolve-evals/references/core-concepts/sandboxes.md +73 -0
  45. package/skills/evolve-evals/references/core-concepts/secrets.md +86 -0
  46. package/skills/evolve-evals/references/core-concepts/sharing.md +81 -0
  47. package/skills/evolve-evals/references/core-concepts/skills.md +100 -0
  48. package/skills/evolve-evals/references/core-concepts/task-config.md +169 -0
  49. package/skills/evolve-evals/references/core-concepts/task-environment.md +127 -0
  50. package/skills/evolve-evals/references/core-concepts/task-reference.md +191 -0
  51. package/skills/evolve-evals/references/core-concepts/task-verifiers.md +209 -0
  52. package/skills/evolve-evals/references/core-concepts/tasks.md +90 -0
  53. package/skills/evolve-evals/references/core-concepts/teams.md +128 -0
  54. package/skills/evolve-evals/references/core-concepts/trial-outputs.md +225 -0
  55. package/skills/evolve-evals/references/core-concepts/trials.md +129 -0
  56. package/skills/evolve-evals/references/core-concepts/upload-sdk-session.md +240 -0
  57. package/skills/evolve-evals/references/core-concepts/upload.md +166 -0
  58. package/skills/evolve-evals/references/dashboard/checks.md +54 -0
  59. package/skills/evolve-evals/references/dashboard/datasets.md +65 -0
  60. package/skills/evolve-evals/references/dashboard/jobs.md +106 -0
  61. package/skills/evolve-evals/references/dashboard/traces.md +54 -0
  62. package/skills/evolve-evals/references/dashboard/trial-viewer.md +77 -0
  63. package/skills/evolve-evals/references/getting-started/first-task.md +228 -0
  64. package/skills/evolve-evals/references/getting-started/installation.md +79 -0
  65. package/skills/evolve-evals/references/getting-started/quick-start.md +100 -0
  66. package/skills/evolve-evals/references/index.md +58 -0
  67. package/skills/evolve-evals/references/sdk/http.md +110 -0
  68. package/skills/evolve-evals/references/sdk/python.md +108 -0
  69. package/skills/evolve-evals/references/sdk/typescript.md +104 -0
  70. package/skills/evolve-evals/references/sdk-reference/agents.md +75 -0
  71. package/skills/evolve-evals/references/sdk-reference/analyses.md +124 -0
  72. package/skills/evolve-evals/references/sdk-reference/auth.md +171 -0
  73. package/skills/evolve-evals/references/sdk-reference/checks.md +145 -0
  74. package/skills/evolve-evals/references/sdk-reference/datasets.md +170 -0
  75. package/skills/evolve-evals/references/sdk-reference/error-codes.md +231 -0
  76. package/skills/evolve-evals/references/sdk-reference/errors.md +98 -0
  77. package/skills/evolve-evals/references/sdk-reference/filesystem.md +166 -0
  78. package/skills/evolve-evals/references/sdk-reference/index.md +161 -0
  79. package/skills/evolve-evals/references/sdk-reference/jobs.md +250 -0
  80. package/skills/evolve-evals/references/sdk-reference/meta.md +54 -0
  81. package/skills/evolve-evals/references/sdk-reference/methods/agents.md +154 -0
  82. package/skills/evolve-evals/references/sdk-reference/methods/analyses.md +211 -0
  83. package/skills/evolve-evals/references/sdk-reference/methods/auth.md +257 -0
  84. package/skills/evolve-evals/references/sdk-reference/methods/checks.md +460 -0
  85. package/skills/evolve-evals/references/sdk-reference/methods/datasets.md +687 -0
  86. package/skills/evolve-evals/references/sdk-reference/methods/filesystem.md +527 -0
  87. package/skills/evolve-evals/references/sdk-reference/methods/jobs.md +1021 -0
  88. package/skills/evolve-evals/references/sdk-reference/methods/meta.md +209 -0
  89. package/skills/evolve-evals/references/sdk-reference/methods/secrets.md +157 -0
  90. package/skills/evolve-evals/references/sdk-reference/methods/skills.md +125 -0
  91. package/skills/evolve-evals/references/sdk-reference/methods/trials.md +371 -0
  92. package/skills/evolve-evals/references/sdk-reference/secrets.md +104 -0
  93. package/skills/evolve-evals/references/sdk-reference/skills.md +66 -0
  94. package/skills/evolve-evals/references/sdk-reference/trials.md +164 -0
  95. package/skills/evolve-evals/references/sdk-reference/types.md +1249 -0
  96. package/skills/publish/SKILL.md +103 -79
  97. package/skills/rewardkit/SKILL.md +78 -23
  98. package/spec/openapi.yaml +19 -74
  99. package/skills/evolve-evals/references/cli-reference/agent.mdx +0 -79
  100. package/skills/evolve-evals/references/cli-reference/analysis.mdx +0 -160
  101. package/skills/evolve-evals/references/cli-reference/analyze.mdx +0 -84
  102. package/skills/evolve-evals/references/cli-reference/auth.mdx +0 -133
  103. package/skills/evolve-evals/references/cli-reference/check.mdx +0 -263
  104. package/skills/evolve-evals/references/cli-reference/dataset.mdx +0 -151
  105. package/skills/evolve-evals/references/cli-reference/index.mdx +0 -108
  106. package/skills/evolve-evals/references/cli-reference/job.mdx +0 -278
  107. package/skills/evolve-evals/references/cli-reference/run.mdx +0 -204
  108. package/skills/evolve-evals/references/cli-reference/secrets.mdx +0 -95
  109. package/skills/evolve-evals/references/cli-reference/skill.mdx +0 -61
  110. package/skills/evolve-evals/references/cli-reference/skills.mdx +0 -81
  111. package/skills/evolve-evals/references/cli-reference/trial.mdx +0 -181
  112. package/skills/evolve-evals/references/cli-reference/upload.mdx +0 -36
  113. package/skills/evolve-evals/references/core-concepts/agents.mdx +0 -106
  114. package/skills/evolve-evals/references/core-concepts/analyze.mdx +0 -86
  115. package/skills/evolve-evals/references/core-concepts/check.mdx +0 -64
  116. package/skills/evolve-evals/references/core-concepts/datasets.mdx +0 -138
  117. package/skills/evolve-evals/references/core-concepts/index.mdx +0 -42
  118. package/skills/evolve-evals/references/core-concepts/jobs.mdx +0 -151
  119. package/skills/evolve-evals/references/core-concepts/models.mdx +0 -68
  120. package/skills/evolve-evals/references/core-concepts/sandboxes.mdx +0 -52
  121. package/skills/evolve-evals/references/core-concepts/secrets.mdx +0 -41
  122. package/skills/evolve-evals/references/core-concepts/sharing.mdx +0 -35
  123. package/skills/evolve-evals/references/core-concepts/skills.mdx +0 -42
  124. package/skills/evolve-evals/references/core-concepts/tasks.mdx +0 -216
  125. package/skills/evolve-evals/references/core-concepts/teams.mdx +0 -36
  126. package/skills/evolve-evals/references/core-concepts/trial-outputs.mdx +0 -117
  127. package/skills/evolve-evals/references/core-concepts/trials.mdx +0 -103
  128. package/skills/evolve-evals/references/core-concepts/upload.mdx +0 -93
  129. package/skills/evolve-evals/references/dashboard/checks.mdx +0 -21
  130. package/skills/evolve-evals/references/dashboard/datasets.mdx +0 -24
  131. package/skills/evolve-evals/references/dashboard/jobs.mdx +0 -65
  132. package/skills/evolve-evals/references/dashboard/traces.mdx +0 -17
  133. package/skills/evolve-evals/references/dashboard/trial-viewer.mdx +0 -30
  134. package/skills/evolve-evals/references/getting-started/installation.mdx +0 -56
  135. package/skills/evolve-evals/references/getting-started/quick-start.mdx +0 -94
  136. package/skills/evolve-evals/references/index.mdx +0 -48
  137. package/skills/evolve-evals/references/sdk/python.mdx +0 -161
  138. package/skills/evolve-evals/references/sdk/typescript.mdx +0 -163
  139. package/skills/evolve-evals/references/sdk-reference/agents.mdx +0 -108
  140. package/skills/evolve-evals/references/sdk-reference/analyses.mdx +0 -222
  141. package/skills/evolve-evals/references/sdk-reference/auth.mdx +0 -116
  142. package/skills/evolve-evals/references/sdk-reference/checks.mdx +0 -300
  143. package/skills/evolve-evals/references/sdk-reference/datasets.mdx +0 -346
  144. package/skills/evolve-evals/references/sdk-reference/errors.mdx +0 -210
  145. package/skills/evolve-evals/references/sdk-reference/index.mdx +0 -60
  146. package/skills/evolve-evals/references/sdk-reference/jobs.mdx +0 -822
  147. package/skills/evolve-evals/references/sdk-reference/meta.mdx +0 -52
  148. package/skills/evolve-evals/references/sdk-reference/skills.mdx +0 -69
  149. package/skills/evolve-evals/references/sdk-reference/trials.mdx +0 -302
  150. package/skills/evolve-evals/references/sdk-reference/types.mdx +0 -274
  151. package/skills/evolve-evals/references/snippets/global-options.mdx +0 -13
  152. package/skills/evolve-evals/references/snippets/list-flags.mdx +0 -23
@@ -1,300 +0,0 @@
1
- ---
2
- title: "checks"
3
- description: "Start and read task quality checks."
4
- ---
5
-
6
- `checks()` returns the checks client.
7
-
8
- ## create
9
-
10
- <Tabs>
11
- <Tab title="TypeScript">
12
- ```ts
13
- create(input: { source: { directory: string } | { dataset: string }; name?: string; model_name?: string; rubric?: Rubric; prompt?: string; reasoning_effort?: string; sandbox_provider?: EvalSandboxProvider; n_concurrent?: number; include_task_names?: string[]; exclude_task_names?: string[]; n_tasks?: number; onUploadProgress?: (sentBytes: number, totalBytes: number) => void }): Promise<Check>
14
- ```
15
-
16
- Check a local task directory or a directory of them, uploaded as one archive (`upload_too_large` past the published ceiling), or a published dataset, nothing uploaded. Returns the accepted check with one `queued` result per task; an empty selection is refused with `no_checkable_tasks`, and a dataset version resolves as for a job (`dataset_not_found`, `no_active_version`, `version_not_ready`). A check on a dataset version also appears on every job spanning it, as `check` on `jobs().tasks()` rows.
17
-
18
- ```ts
19
- const check = await checks().create({ source: { dataset: "my-swe@1.0" } });
20
- ```
21
- </Tab>
22
- <Tab title="Python">
23
- ```python
24
- async def create(
25
- directory: Optional[str] = None,
26
- *,
27
- dataset: Optional[str] = None,
28
- name=None,
29
- model_name=None,
30
- rubric: Optional[Rubric] = None,
31
- prompt=None,
32
- reasoning_effort=None,
33
- sandbox_provider=None,
34
- n_concurrent=None,
35
- include_task_names=None,
36
- exclude_task_names=None,
37
- n_tasks=None,
38
- on_upload_progress=None,
39
- ) -> Check
40
- ```
41
-
42
- Check a local task directory or a directory of them, uploaded as one archive (`upload_too_large` past the published ceiling), or a published dataset, nothing uploaded; one source, not both. Returns the accepted check with one `queued` result per task; an empty selection is refused with `no_checkable_tasks`, and a dataset version resolves as for a job. A check on a dataset version also appears on every job spanning it, as `check` on `jobs().tasks()` rows.
43
-
44
- ```python
45
- check = await checks().create(dataset="my-swe@1.0")
46
- ```
47
- </Tab>
48
- </Tabs>
49
-
50
- ## get, list, watch
51
-
52
- <Tabs>
53
- <Tab title="TypeScript">
54
- ```ts
55
- get(checkId: string): Promise<Check>
56
- list(options?: { scope?: "my" | "shared" | "org"; status?: CheckStatus[]; dataset?: string; limit?: number; cursor?: string }): CheckList
57
- watch(checkId: string, options?: { onProgress?: (check: Check) => void; signal?: AbortSignal; pollIntervalMs?: number }): Promise<Check>
58
- ```
59
-
60
- One check with its per-task results, your checks (`scope: "shared"` is your organizations' and the checks shared with your address; `scope: "org"` is every check in your organizations, yours included), or a poll until every task settled. `dataset` keeps only the checks on one dataset: a bare name is every version, `name@version` one version.
61
-
62
- ```ts
63
- const done = await checks().watch(check.id);
64
- const onTb4 = await checks().list({ dataset: "terminal-bench-4@4.0" });
65
- ```
66
- </Tab>
67
- <Tab title="Python">
68
- ```python
69
- async def get(check_id: str) -> Check
70
- def list(
71
- *,
72
- scope: Optional[JobListScope] = None,
73
- status: Optional[List[CheckStatus]] = None,
74
- dataset: Optional[str] = None,
75
- limit: Optional[int] = None,
76
- cursor: Optional[str] = None,
77
- )
78
- async def watch(
79
- check_id: str,
80
- *,
81
- on_progress: Optional[Callable[[Check], None]] = None,
82
- poll_interval_s: float = 2.0,
83
- timeout_s: Optional[float] = None,
84
- ) -> Check
85
- ```
86
-
87
- One check with its per-task results, your checks, or a poll until every task settled. `dataset` keeps only the checks on one dataset: a bare name is every version, `name@version` one version.
88
-
89
- ```python
90
- done = await checks().watch(check.id)
91
- on_tb4 = await checks().list(dataset="terminal-bench-4@4.0")
92
- ```
93
- </Tab>
94
- </Tabs>
95
-
96
- ## defaults
97
-
98
- <Tabs>
99
- <Tab title="TypeScript">
100
- ```ts
101
- defaults(): Promise<CheckDefaults>
102
- ```
103
-
104
- The defaults a check runs under when you name none: `model_name`, `reasoning_effort`, `sandbox_provider`, the platform's `rubric`, and `prompt`, the built-in check prompt as a template with `{task_path}`, `{file_tree}` and `{criteria_guidance}` unrendered. Edit the rubric or the prompt from here and pass it to `create`; passed back unchanged, they run exactly as the defaults do, and the check records them as named, so a later change of the platform default does not move it.
105
-
106
- ```ts
107
- const { rubric } = await checks().defaults();
108
- ```
109
- </Tab>
110
- <Tab title="Python">
111
- ```python
112
- async def defaults() -> CheckDefaults
113
- ```
114
-
115
- The defaults a check runs under when you name none: `model_name`, `reasoning_effort`, `sandbox_provider`, the platform's `rubric`, and `prompt`, the built-in check prompt as a template with `{task_path}`, `{file_tree}` and `{criteria_guidance}` unrendered. Edit the rubric or the prompt from here and pass it to `create`; passed back unchanged, they run exactly as the defaults do, and the check records them as named, so a later change of the platform default does not move it.
116
-
117
- ```python
118
- rubric = (await checks().defaults())["rubric"]
119
- ```
120
- </Tab>
121
- </Tabs>
122
-
123
- ## task, transcript, artifact
124
-
125
- <Tabs>
126
- <Tab title="TypeScript">
127
- ```ts
128
- task(taskCheckId: string): Promise<TaskCheck>
129
- transcript(
130
- taskCheckId: string,
131
- options?: { since?: number },
132
- ): Promise<TaskCheckTranscript>
133
- artifact(
134
- taskCheckId: string,
135
- stream: "trace-stdout" | "trace-stderr",
136
- ): Promise<string | null>
137
- artifact(
138
- taskCheckId: string,
139
- stream: "agent-home",
140
- ): Promise<Record<string, string> | null>
141
- ```
142
-
143
- One task's result by its task check id, the checker's own transcript (everything after `since` in one read), or one stored artifact; a task check has no verifier log and no trajectory. An id you may not read answers `trial_not_found`.
144
-
145
- ```ts
146
- const result = await checks().task(done.results[0].id);
147
- ```
148
- </Tab>
149
- <Tab title="Python">
150
- The Python client has no per-task reads. A task's result is in `Check.results`, returned by `get` and `watch`.
151
- </Tab>
152
- </Tabs>
153
-
154
- ## download
155
-
156
- <Tabs>
157
- <Tab title="TypeScript">
158
- ```ts
159
- download(id: string): Promise<Buffer>
160
- download(
161
- id: string,
162
- options: { to: string },
163
- ): Promise<string>
164
- download(
165
- id: string,
166
- options: { stream: true },
167
- ): Promise<ReadableStream<Uint8Array>>
168
- ```
169
-
170
- The whole check by its check id (`check-<id>/` with `check_report.json` and one folder per task), or one task's folder by its task check id (`check-<task>__<7 chars>/`), as one `.tar.gz`. Each task folder is a trial tree of the checker's own run, with `artifacts/check-result.json` on a completed run. An id that is neither answers `check_not_found`; an unsettled check or task check answers `check_not_terminal`.
171
-
172
- ```ts
173
- const path = await checks().download(
174
- check.id,
175
- { to: "./checks" },
176
- );
177
- ```
178
- </Tab>
179
- <Tab title="Python">
180
- ```python
181
- async def download(
182
- id: str,
183
- *,
184
- to: Optional[str] = None,
185
- )
186
- ```
187
-
188
- The whole check by its check id (`check-<id>/` with `check_report.json` and one folder per task), or one task's folder by its task check id (`check-<task>__<7 chars>/`), as one `.tar.gz`. An id that is neither answers `check_not_found`; an unsettled check or task check answers `check_not_terminal`.
189
-
190
- ```python
191
- path = await checks().download(
192
- check.id,
193
- to="./checks",
194
- )
195
- ```
196
- </Tab>
197
- </Tabs>
198
-
199
- ## share, unshare, shares
200
-
201
- Share a check you created by link, by email, or both — the [job share](/sdk-reference/jobs#share) verbs on a check, with the same arguments and the same `JobShares` answer. `link: true` turns on the check's unlisted link and returns it (the same link on every later call); whoever opens it reads the check's report and downloads its check folder without signing in. Each address in `emails` gets an email with a link and reads the check after signing in with that address; an address without an account gets a sign-up link in the same email. Someone the check is shared with reads the report, each task check's files and the download, and lists it under `scope: "shared"`; they only read, and can never stop one of its task checks. Only the creator may share; another member of the organization is refused with `org_forbidden`. `unshare` takes the same arguments and turns the link off or removes those addresses; `shares` prints the share state. A check's `visibility` field says `PRIVATE` or `LINK`.
202
-
203
- <Tabs>
204
- <Tab title="TypeScript">
205
- ```ts
206
- share(id: string, request: { link?: boolean; emails?: string[] }): Promise<JobShares>
207
- unshare(id: string, request: { link?: boolean; emails?: string[] }): Promise<JobShares>
208
- shares(id: string): Promise<JobShares>
209
- ```
210
-
211
- ```ts
212
- const state = await checks().share(check.id, { link: true, emails: ["alice@example.org"] });
213
- console.log(state.link.url);
214
- await checks().unshare(check.id, { link: true });
215
- ```
216
- </Tab>
217
- <Tab title="Python">
218
- ```python
219
- async def share(id: str, *, link: bool = False, emails: list[str] | None = None) -> JobShares
220
- async def unshare(id: str, *, link: bool = False, emails: list[str] | None = None) -> JobShares
221
- async def shares(id: str) -> JobShares
222
- ```
223
-
224
- ```python
225
- state = await checks().share(check["id"], link=True, emails=["alice@example.org"])
226
- print(state.link.url)
227
- await checks().unshare(check["id"], link=True)
228
- ```
229
- </Tab>
230
- </Tabs>
231
-
232
- ## taskFilesystem(checkId: string, taskCheckId: string)
233
-
234
- <Tabs>
235
- <Tab title="TypeScript">
236
- ```ts
237
- taskFilesystem(checkId: string, taskCheckId: string): RunFilesystem
238
-
239
- interface RunFilesystem {
240
- status(): Promise<FilesystemStatus>
241
- list(options?: { path?: string; source?: "live" | "capture"; cursor?: string; limit?: number }): Promise<FilesystemListing>
242
- read(path: string, options?: { source?: "live" | "capture"; range?: TrialFileRange }): Promise<Buffer>
243
- search(options: { q: string; path?: string; regex?: boolean; limit?: number; source?: "live" | "capture" }): Promise<FilesystemSearchResult>
244
- changes(options?: { source?: "live" | "capture"; phase?: "setup" | "agent" | "verifier" | "all"; cursor?: string; limit?: number }): Promise<FilesystemChanges>
245
- archive(options?: { path?: string; source?: "live" | "capture" }): Promise<Buffer>
246
- archive(options: { path?: string; source?: "live" | "capture"; to: string }): Promise<string>
247
- watch(paths: string[]): Promise<FilesystemWatchResult>
248
- events(options?: { lastEventId?: string; signal?: AbortSignal }): AsyncIterableIterator<FilesystemStreamEvent>
249
- logs(options: { stream: SandboxLogStream; cursor?: string; limit?: number }): Promise<SandboxLogLines>
250
- logEvents(options?: { lastEventId?: string; signal?: AbortSignal }): AsyncIterableIterator<SandboxLogEvent>
251
- procs(): Promise<SandboxProcs>
252
- }
253
- ```
254
-
255
- The task check's file system, sandbox logs and process list. While the box runs, every read sees it as it changes; after the run, the same reads answer from the kept tree. `status()` says which (`state`: `live`, `captured`, `capturing`, `none`), and `source` on a read forces one — a source that is not there is refused with `filesystem_state`.
256
-
257
- `list` pages one folder by name; each entry says whether the run created or modified it and in which phase, and on the kept tree `captured: false` marks a file the run never touched (it lists but does not open — `not_captured`). `read` answers raw bytes, a slice with `range`. `search` finds text under a folder, or over the whole box when `path` is `/` (seconds); `truncated` says there was more. `changes` is the flat list of what the run created, modified or removed. `archive` is a `.tar.gz` of one subtree, in memory or saved under `to`; while the box runs, a subtree too large to read out in time is refused (`feature_unsupported`).
258
-
259
- `events` yields change frames while the box lives (`state`, then one `fs` per change with its `seq`, `path` and `type`); on a provider without a native watcher, `watch(paths)` declares the folders you have open so their changes are reported (at most 8). `logs` pages one stream — `agent`, `verifier`, or `system` when the job asked for it; `setup` and `metrics` are named but not recorded today, so their page is empty with a `reason` — and `logEvents` follows every stream while the box lives. `procs` is the live process list.
260
-
261
- ```ts
262
- const fs = checks().taskFilesystem(check.id, check.results[0].id);
263
- const { state } = await fs.status();
264
- for (const entry of (await fs.list({ path: "/app/work" })).entries) console.log(entry.name, entry.changed);
265
- const main = (await fs.read("/app/work/main.py")).toString("utf8");
266
- for await (const frame of fs.events()) if (frame.event === "fs") console.log(frame.data.path, frame.data.type);
267
- ```
268
- </Tab>
269
- <Tab title="Python">
270
- ```python
271
- def task_filesystem(self, check_id: str, task_check_id: str) -> RunFilesystem
272
-
273
- class RunFilesystem:
274
- async def status(self) -> FilesystemStatus
275
- async def list(self, *, path="/", source=None, cursor=None, limit=None) -> FilesystemListing
276
- async def read(self, path, *, source=None, start=None, end=None, suffix=None) -> bytes
277
- async def search(self, q, *, path="/", regex=False, limit=None, source=None) -> FilesystemSearchResult
278
- async def changes(self, *, source=None, phase=None, cursor=None, limit=None) -> FilesystemChanges
279
- async def archive(self, *, path="/", source=None, to=None) # bytes, or the saved path with to=
280
- async def watch(self, paths) -> FilesystemWatchResult
281
- def events(self, *, last_event_id=None) -> AsyncIterator[FilesystemStreamEvent]
282
- async def logs(self, stream, *, cursor=None, limit=None) -> SandboxLogLines
283
- def log_events(self, *, last_event_id=None) -> AsyncIterator[SandboxLogEvent]
284
- async def procs(self) -> SandboxProcs
285
- ```
286
-
287
- The same surface, the same words: `status().state` says whether the reads answer from the running box (`live`) or the kept tree (`captured`); `source=` forces one. `read` answers bytes, a slice with `start`/`end` or `suffix`. `events` and `log_events` are async iterators over the live streams.
288
-
289
- ```python
290
- fs = checks().task_filesystem(check["id"], check["results"][0]["id"])
291
- print((await fs.status()).state)
292
- for entry in (await fs.list(path="/app/work")).entries:
293
- print(entry.name, entry.changed)
294
- main = (await fs.read("/app/work/main.py")).decode()
295
- async for frame in fs.events():
296
- if frame.event == "fs":
297
- print(frame.data["path"], frame.data["type"])
298
- ```
299
- </Tab>
300
- </Tabs>
@@ -1,346 +0,0 @@
1
- ---
2
- title: "datasets"
3
- description: "The catalog: list, read, publish, download, activate, delete."
4
- ---
5
-
6
- `datasets()` returns the catalog client. A `ref` is `name` or `name@version`.
7
-
8
- ## list, get, getActive
9
-
10
- <Tabs>
11
- <Tab title="TypeScript">
12
- ```ts
13
- list(options?: { search?: string; limit?: number; cursor?: string }): DatasetList
14
- get(
15
- ref: string,
16
- options?: { limit?: number; cursor?: string },
17
- ): Promise<Dataset>
18
- getActive(
19
- name: string,
20
- options?: { limit?: number; cursor?: string },
21
- ): Promise<ActiveDataset>
22
- ```
23
-
24
- The catalog, one dataset with one page of its tasks (default 200, at most 500), or the active version alone. `getActive` throws `NoActiveVersionError` when nothing is runnable. A task row carries public fields only; see [Task](/sdk-reference/types#task).
25
-
26
- ```ts
27
- const dataset = await datasets().get("terminal-bench-4@4.0");
28
- ```
29
- </Tab>
30
- <Tab title="Python">
31
- ```python
32
- def list(
33
- *,
34
- search: Optional[str] = None,
35
- limit: Optional[int] = None,
36
- cursor: Optional[str] = None,
37
- )
38
- async def get(
39
- ref: str,
40
- *,
41
- limit: Optional[int] = None,
42
- cursor: Optional[str] = None,
43
- ) -> Dataset
44
- async def get_active(
45
- name: str,
46
- *,
47
- limit: Optional[int] = None,
48
- cursor: Optional[str] = None,
49
- ) -> ActiveDataset
50
- ```
51
-
52
- The catalog, one dataset with one page of its tasks, or the active version alone. `get_active` raises `NoActiveVersionError` when nothing is runnable.
53
-
54
- ```python
55
- dataset = await datasets().get("terminal-bench-4@4.0")
56
- ```
57
- </Tab>
58
- </Tabs>
59
-
60
- ## getTaskBuild
61
-
62
- <Tabs>
63
- <Tab title="TypeScript">
64
- ```ts
65
- getTaskBuild(
66
- ref: string,
67
- taskName: string,
68
- ): Promise<TaskBuild>
69
- ```
70
-
71
- One task's build record: its outcome and, on failure, the failing step and log.
72
-
73
- ```ts
74
- const build = await datasets().getTaskBuild("my-swe@1.0", "broken-task");
75
- ```
76
- </Tab>
77
- <Tab title="Python">
78
- ```python
79
- async def get_task_build(
80
- ref: str,
81
- task_name: str,
82
- ) -> TaskBuild
83
- ```
84
-
85
- One task's build record: its outcome and, on failure, the failing step and log.
86
-
87
- ```python
88
- build = await datasets().get_task_build("my-swe@1.0", "broken-task")
89
- ```
90
- </Tab>
91
- </Tabs>
92
-
93
- ## taskFiles
94
-
95
- <Tabs>
96
- <Tab title="TypeScript">
97
- ```ts
98
- taskFiles(ref: string, taskName: string): TaskPackageFiles
99
-
100
- interface TaskPackageFiles {
101
- status(): Promise<TaskPackageFilesystemStatus>
102
- list(options?: { path?: string; cursor?: string; limit?: number }): Promise<FilesystemListing>
103
- read(path: string, options?: { range?: TrialFileRange }): Promise<Buffer>
104
- }
105
- ```
106
-
107
- A published task's files, exactly as the platform stored them and as every trial ran against them: `list` pages one folder of the task directory (`/` holds `instruction.md`, `task.toml`, `environment/`, `tests/`), `read` answers one file's bytes. `ref` must name the version (`name@version`); a version imported before packages were kept is refused with `task_package_not_retained` (`status().package_retained` says so first).
108
-
109
- ```ts
110
- const files = datasets().taskFiles("my-swe@1.0", "abs-1");
111
- const tests = await files.list({ path: "/tests" });
112
- const script = (await files.read("/tests/test.sh")).toString("utf8");
113
- ```
114
- </Tab>
115
- <Tab title="Python">
116
- ```python
117
- def task_files(self, ref: str, task_name: str) -> TaskPackageFiles
118
-
119
- class TaskPackageFiles:
120
- async def status(self) -> TaskPackageFilesystemStatus
121
- async def list(self, *, path="/", cursor=None, limit=None) -> FilesystemListing
122
- async def read(self, path, *, start=None, end=None, suffix=None) -> bytes
123
- ```
124
-
125
- ```python
126
- files = datasets().task_files("my-swe@1.0", "abs-1")
127
- tests = await files.list(path="/tests")
128
- script = (await files.read("/tests/test.sh")).decode()
129
- ```
130
- </Tab>
131
- </Tabs>
132
-
133
- ## preflight
134
-
135
- <Tabs>
136
- <Tab title="TypeScript">
137
- ```ts
138
- preflight(input: { source: { directory: string } }): Promise<DatasetPreflight>
139
- ```
140
-
141
- Dry-run a local folder of tasks: only each task's metadata is sent, nothing is uploaded or written.
142
-
143
- ```ts
144
- const verdict = await datasets().preflight({ source: { directory: "./my-swe" } });
145
- ```
146
- </Tab>
147
- <Tab title="Python">
148
- ```python
149
- async def preflight(
150
- *,
151
- directory: str,
152
- ) -> DatasetPreflight
153
- ```
154
-
155
- Dry-run a local folder of tasks: only each task's metadata is sent, nothing is uploaded or written.
156
-
157
- ```python
158
- verdict = await datasets().preflight(directory="./my-swe")
159
- ```
160
- </Tab>
161
- </Tabs>
162
-
163
- ## publish
164
-
165
- <Tabs>
166
- <Tab title="TypeScript">
167
- ```ts
168
- publish(input: { source: DatasetSource; name?: string; version?: string; org?: string }, options?: { onUploadProgress?: (sentBytes: number, totalBytes: number) => void; onRegistered?: (importId: string) => void }): Promise<DatasetImport>
169
- ```
170
-
171
- Publish a version from one `source`: `{ git_url, git_ref, git_path? }` (an `https://` URL, and a subfolder that exists at the ref), `{ directory }`, `{ archive_url }` or `{ hub_package }`. `org` names the organization a new dataset lands in (omitted, the client's `org` default, else your personal one). An existing dataset stays where it is only when no org reaches the server; a named org, the client default included, that is not the dataset's own is refused with `invalid_input`, so a re-publish never moves a dataset. Returns the import record; follow it with `watchImport`.
172
-
173
- `onUploadProgress` reports sent bytes for a directory source, and `onRegistered` fires once with the import id when a large upload begins. What is packed is on [datasets](/core-concepts/datasets#publish-your-own).
174
-
175
- ```ts
176
- const imp = await datasets().publish({
177
- source: { directory: "./my-swe" },
178
- name: "my-swe",
179
- version: "1.0",
180
- });
181
- ```
182
- </Tab>
183
- <Tab title="Python">
184
- ```python
185
- async def publish(
186
- *,
187
- git_url=None,
188
- git_ref=None,
189
- git_path=None,
190
- directory=None,
191
- archive_url=None,
192
- hub_package=None,
193
- name=None,
194
- version=None,
195
- org=None,
196
- on_upload_progress=None,
197
- on_registered=None,
198
- ) -> DatasetImport
199
- ```
200
-
201
- Publish a version from one source: `git_url` with `git_ref` (and `git_path`), `directory`, `archive_url`, or `hub_package`. `org` names the organization a new dataset lands in (omitted, the client's `org` default, else your personal one). An existing dataset stays where it is only when no org reaches the server; a named org, the client default included, that is not the dataset's own is refused with `invalid_input`, so a re-publish never moves a dataset. Returns the import record; follow it with `watch_import`.
202
-
203
- ```python
204
- imp = await datasets().publish(
205
- directory="./my-swe",
206
- name="my-swe",
207
- version="1.0",
208
- )
209
- ```
210
- </Tab>
211
- </Tabs>
212
-
213
- ## getImport, watchImport, listImports
214
-
215
- <Tabs>
216
- <Tab title="TypeScript">
217
- ```ts
218
- getImport(id: string): Promise<DatasetImport>
219
- watchImport(id: string, options?: { onStatus?: (datasetImport: DatasetImport) => void; onProgress?: (progress: DatasetImportProgress, datasetImport: DatasetImport) => void; onVersion?: (version: DatasetVersion, dataset: Dataset) => void; signal?: AbortSignal; pollIntervalMs?: number; settleTimeoutMs?: number }): Promise<DatasetImport>
220
- listImports(options?: { status?: DatasetImportStatus; dataset?: string; limit?: number; cursor?: string }): DatasetImportList
221
- ```
222
-
223
- Read one publish, follow it until the version is READY or FAILED, or list your publishes. `watchImport` polls every `pollIntervalMs` (default 2000): `onStatus` fires on every status change, `onProgress` on every change of the five-phase `progress` (`extracting`, `parsing`, `building`, `copying`, `verifying`), `onVersion` on the version's state. `settleTimeoutMs` (default 30 minutes) bounds the wait; past it the watch throws `ImportSettleError` with the last observed state. `listImports` filters by `status` and `dataset`.
224
-
225
- ```ts
226
- const done = await datasets().watchImport(imp.id);
227
- ```
228
- </Tab>
229
- <Tab title="Python">
230
- ```python
231
- async def get_import(id: str) -> DatasetImport
232
- async def watch_import(
233
- id: str,
234
- *,
235
- on_status=None,
236
- on_progress=None,
237
- on_version=None,
238
- poll_interval_s: float = 2.0,
239
- timeout_s: Optional[float] = None,
240
- settle_timeout_s: float = ...,
241
- ) -> DatasetImport
242
- def list_imports(
243
- *,
244
- status: Optional[str] = None,
245
- dataset: Optional[str] = None,
246
- limit: Optional[int] = None,
247
- cursor: Optional[str] = None,
248
- )
249
- ```
250
-
251
- Read one publish, follow it until the version is READY or FAILED, or list your publishes. `watch_import` polls every `poll_interval_s`: `on_status` fires on every status change, `on_progress` on every change of the five-phase `progress` (`extracting`, `parsing`, `building`, `copying`, `verifying`), `on_version` on the version's state. `settle_timeout_s` (default 30 minutes) bounds the wait; past it the watch raises `ImportSettleError` with the last observed state. `list_imports` filters by `status` and `dataset`.
252
-
253
- ```python
254
- done = await datasets().watch_import(imp.id)
255
- ```
256
- </Tab>
257
- </Tabs>
258
-
259
- ## download
260
-
261
- <Tabs>
262
- <Tab title="TypeScript">
263
- ```ts
264
- download(ref: string): Promise<Buffer>
265
- download(
266
- ref: string,
267
- options: { to: string },
268
- ): Promise<string>
269
- download(
270
- ref: string,
271
- options: { stream: true },
272
- ): Promise<ReadableStream<Uint8Array>>
273
- ```
274
-
275
- The original package of a dataset you own: the bytes, the saved file's path, or a stream. The Buffer and `{ to }` shapes verify the download against the digest the server states and throw `EvolveDigestMismatchError` on a mismatch; `{ stream: true }` is unverified. A version published before packages were retained answers `package_not_retained`. Prefer `{ to }` for anything sizeable.
276
-
277
- ```ts
278
- const path = await datasets().download(
279
- "my-swe@1.0",
280
- { to: "./corpora" },
281
- );
282
- ```
283
- </Tab>
284
- <Tab title="Python">
285
- ```python
286
- async def download(
287
- ref: str,
288
- *,
289
- to: Optional[str] = None,
290
- )
291
- ```
292
-
293
- The original package of a dataset you own: the bytes, or with `to` the saved file's path. Both shapes verify the download against the digest the server states, raising `EvolveDigestMismatchError` on a mismatch. A version published before packages were retained answers `package_not_retained`. Prefer `to` for anything sizeable.
294
-
295
- ```python
296
- path = await datasets().download(
297
- "my-swe@1.0",
298
- to="./corpora",
299
- )
300
- ```
301
- </Tab>
302
- </Tabs>
303
-
304
- ## activate, update, delete
305
-
306
- <Tabs>
307
- <Tab title="TypeScript">
308
- ```ts
309
- activate(
310
- name: string,
311
- version: string,
312
- ): Promise<Dataset>
313
- update(
314
- name: string,
315
- patch: { upstream_auto_import: boolean },
316
- ): Promise<Dataset>
317
- delete(name: string): Promise<void>
318
- ```
319
-
320
- Make a READY version the active one (`version_not_ready` while building, `version_not_activatable` for FAILED or ARCHIVED), switch automatic import of new upstream versions (`upstream_not_watchable` without a moving git ref), or delete a dataset you own (`dataset_not_owned` on a platform dataset, `dataset_in_use` naming the jobs that reference it).
321
-
322
- ```ts
323
- await datasets().activate("my-swe", "1.0");
324
- ```
325
- </Tab>
326
- <Tab title="Python">
327
- ```python
328
- async def activate(
329
- name: str,
330
- version: str,
331
- ) -> Dataset
332
- async def update(
333
- name: str,
334
- *,
335
- upstream_auto_import: bool,
336
- ) -> Dataset
337
- async def delete(name: str) -> None
338
- ```
339
-
340
- Make a READY version the active one (`version_not_ready` while building, `version_not_activatable` for FAILED or ARCHIVED), switch automatic import of new upstream versions (`upstream_not_watchable` without a moving git ref), or delete a dataset you own (`dataset_not_owned` on a platform dataset, `dataset_in_use` naming the jobs that reference it).
341
-
342
- ```python
343
- await datasets().activate("my-swe", "1.0")
344
- ```
345
- </Tab>
346
- </Tabs>