@evolvingmachines/evolve 0.0.65 → 0.0.67-train-harness-wave-2026-09-25.20260926.e00a8a6d

package/skills/evolve-evals/references/sdk-reference/methods/trials.md

@@ -0,0 +1,371 @@
+---
+title: "Trials methods"
+description: "Read attempts, traces, files, and actions by trial id."
+---
+
+Use `trials()` to create `client`. Examples run inside an async context; ids name existing records. The [trial guide](/sdk-reference/trials) shows the full inspection workflow.
+
+| Read | Act |
+| --- | --- |
+| [get](#get), [trace](#trace), [traceEvents](#traceevents) | [retry](#retry), [regrade](#regrade), [stop](#stop) |
+| [files](#files), [file](#file), [artifact](#artifact), [filesystem](#filesystem) | |
+
+## get
+
+Read one attempt by its globally unique id. Returns `Trial`, including `job_id` and the untruncated exception message.
+
+### Signature
+
+```ts TypeScript signature
+get(trialId: string): Promise<Trial>;
+```
+
+```python Python signature
+async def get(trial_id: str) -> Trial: ...
+```
+
+```ts TypeScript
+const trial = await client.get(trialId);
+```
+
+```python Python
+trial = await client.get(trial_id)
+```
+
+See [trial fields](/sdk-reference/types#trial-fields) for rewards, usage, retries, timing, and provenance.
+
+## trace
+
+Read one page of parsed events. Returns `TraceEventPage`.
+
+### Signature
+
+```ts TypeScript signature
+trace(
+  trialId: string,
+  options?: TraceOptions
+): Promise<TraceEventPage>;
+```
+
+```python Python signature
+async def trace(
+    trial_id: str,
+    *,
+    cursor: Optional[str] = None,
+    limit: Optional[int] = None,
+    type: Optional[str] = None,
+    grep: Optional[str] = None,
+    tail: Optional[int] = None,
+) -> TraceEventPage: ...
+```
+
+```ts TypeScript
+const page = await client.trace(trialId, {
+  tail: 20
+});
+```
+
+```python Python
+page = await client.trace(
+    trial_id,
+    tail=20,
+)
+```
+
+| Option | Meaning / default |
+| --- | --- |
+| `cursor` | Events strictly after this sequence number, passed as a string |
+| `limit` | Events per page: default 200, maximum 1,000 |
+| `type` | Exact event type |
+| `grep` | Case-insensitive POSIX regular expression over type and content |
+| `tail` | Last N matching events, still returned oldest first |
+
+The page has `items`, `nextCursor`, `hasMore` / Python `next_cursor`, `has_more`. An event has `seq: number`, `type: string`, and `data: object`. A null next cursor means caught up. To poll again later, retain the last event’s `seq`.
+
+## traceEvents
+
+Python: `trace_events`. Iterate currently available parsed events across pages. It finishes when caught up; it is not a live subscription.
+
+### Signature
+
+```ts TypeScript signature
+traceEvents(
+  trialId: string,
+  options?: TraceOptions
+): AsyncIterableIterator<TraceEvent>;
+```
+
+```python Python signature
+async def trace_events(
+    trial_id: str,
+    *,
+    cursor: Optional[str] = None,
+    limit: Optional[int] = None,
+    type: Optional[str] = None,
+    grep: Optional[str] = None,
+    tail: Optional[int] = None,
+) -> AsyncIterator[TraceEvent]: ...
+```
+
+```ts TypeScript
+for await (const event of client.traceEvents(trialId)) {
+  console.log(event.seq, event.type);
+}
+```
+
+```python Python
+async for event in client.trace_events(trial_id):
+    print(event.seq, event.type)
+```
+
+Accepts the same `cursor`, `limit`, `type`, `grep`, and `tail` options as `trace`. Returns an async iterator of `TraceEvent`.
+
+## files
+
+List the stored trial tree by relative path. Returns one `TrialFilePage`, sorted by path.
+
+### Signature
+
+```ts TypeScript signature
+files(
+  trialId: string,
+  options?: ListTrialFilesOptions
+): Promise<TrialFilePage>;
+```
+
+```python Python signature
+async def files(
+    trial_id: str,
+    *,
+    cursor: Optional[str] = None,
+    limit: Optional[int] = None,
+) -> TrialFilePage: ...
+```
+
+```ts TypeScript
+const page = await client.files(trialId, {
+  limit: 20
+});
+```
+
+```python Python
+page = await client.files(
+    trial_id,
+    limit=20,
+)
+```
+
+Optional `limit` defaults to 200 (maximum 1,000); `cursor` continues the page. Each item contains `path: string` and `size_bytes: number`. An empty tree is a normal result.
+
+## file
+
+Read one path returned by `files`. Returns exact bytes: TypeScript `Buffer`, Python `bytes`.
+
+### Signature
+
+```ts TypeScript signature
+file(
+  trialId: string,
+  path: string,
+  range?: TrialFileRange
+): Promise<Buffer>;
+```
+
+```python Python signature
+async def file(
+    trial_id: str,
+    path: str,
+    *,
+    start: Optional[int] = None,
+    end: Optional[int] = None,
+    suffix: Optional[int] = None,
+) -> bytes: ...
+```
+
+```ts TypeScript
+const bytes = await client.file(
+  trialId,
+  "agent/stderr.log",
+  {
+    suffix: 4096
+  }
+);
+```
+
+```python Python
+data = await client.file(
+    trial_id,
+    'agent/stderr.log',
+    suffix=4096,
+)
+```
+
+Optional byte range: `start` plus inclusive `end`, `start` through EOF, or `suffix` for the last N bytes. Do not combine `suffix` with `start` or `end`. TypeScript passes the range as the third object; Python uses keyword arguments. A missing path is a typed 404.
+
+## artifact
+
+Read a named stored artifact. Returns text, a path-to-text map, or null when nothing was stored.
+
+### Signature
+
+```ts TypeScript signature
+artifact(
+  trialId: string,
+  stream: Exclude<TrialArtifactStream, "trace-parsed" | "agent-home" | "filesystem">
+): Promise<string | null>;
+artifact(
+  trialId: string,
+  stream: "agent-home"
+): Promise<Record<string, string> | null>;
+```
+
+```python Python signature
+async def artifact(
+    trial_id: str,
+    stream: Literal['trace-parsed', 'verifier', 'trace-stdout', 'trace-stderr', 'trace-atif', 'trajectory', 'agent-home', 'filesystem'],
+) -> Optional[Union[str, Dict[str, str]]]: ...
+```
+
+```ts TypeScript
+const trajectory = await client.artifact(
+  trialId,
+  "trace-atif"
+);
+```
+
+```python Python
+trajectory = await client.artifact(
+    trial_id,
+    'trace-atif',
+)
+```
+
+| Selector | Result |
+| --- | --- |
+| `verifier`, `trace-stdout`, `trace-stderr` | `string \| null` / `str \| None` |
+| `trace-atif` | ATIF v1.7 document as JSON text, or null |
+| `agent-home` | Map of sandbox paths to UTF-8 file text, or null |
+| `trajectory` | Reserved native-session selector; currently returns not-found |
+
+Use `trace` for `trace-parsed`, and `filesystem(id).archive()` for `filesystem`. Python’s annotation lists these selectors, but `artifact` rejects both locally. Non-text home files are omitted from the text map and recorded in `/agent-home.json`; the job archive preserves their bytes.
+
+## filesystem
+
+Get a `RunFilesystem` client for this attempt. No request is made until you call a method on it.
+
+### Signature
+
+```ts TypeScript signature
+filesystem(trialId: string): RunFilesystem;
+```
+
+```python Python signature
+def filesystem(trial_id: str) -> 'RunFilesystem': ...
+```
+
+```ts TypeScript
+const fs = client.filesystem(trialId);
+const status = await fs.status();
+```
+
+```python Python
+fs = client.filesystem(trial_id)
+status = await fs.status()
+```
+
+See [every filesystem method](/sdk-reference/methods/filesystem), including live files, captured files, sandbox logs, and processes.
+
+## retry
+
+Run one settled attempt again in a new linked job. Returns `Job`; the source job may still be running.
+
+### Signature
+
+```ts TypeScript signature
+retry(
+  trialId: string,
+  options?: StartJobOptions
+): Promise<Job>;
+```
+
+```python Python signature
+async def retry(
+    trial_id: str,
+    *,
+    idempotency_key: Optional[str] = None,
+) -> Job: ...
+```
+
+```ts TypeScript
+const job = await client.retry(trialId, {
+  idempotencyKey: requestId
+});
+```
+
+```python Python
+job = await client.retry(
+    trial_id,
+    idempotency_key=request_id,
+)
+```
+
+Optional `idempotencyKey` / `idempotency_key` makes a repeated request return the same accepted job. This is the one-trial form of `jobs().retry`. Imported attempts cannot retry.
+
+## regrade
+
+Run only the verifier again against eligible recorded inputs. Returns a new one-trial `Job`.
+
+### Signature
+
+```ts TypeScript signature
+regrade(trialId: string): Promise<Job>;
+```
+
+```python Python signature
+async def regrade(trial_id: str) -> Job: ...
+```
+
+```ts TypeScript
+const job = await client.regrade(trialId);
+```
+
+```python Python
+job = await client.regrade(trial_id)
+```
+
+The attempt must be settled and have retained inputs from separate verification. Shared-mode or pre-persistence trials are refused with `regrade_source_ineligible`; imported jobs are refused.
+
+## stop
+
+Stop selected in-flight trials or trace analyses owned by the caller. Returns `StopResponse`.
+
+Pass 1–100 non-empty IDs in one call. The list can contain trial IDs, analysis IDs, or both.
+
+### Signature
+
+```ts TypeScript signature
+stop(trialIds: string[]): Promise<StopResponse>;
+```
+
+```python Python signature
+async def stop(trial_ids: List[str]) -> StopResponse: ...
+```
+
+```ts TypeScript
+const result = await client.stop([trialId, analysisId]);
+```
+
+```python Python
+result = await client.stop(
+    [trial_id, analysis_id],
+)
+```
+
+| Returned field | Type / meaning |
+| --- | --- |
+| `stopped` | `Trial[]`: attempts settled by this request |
+| `stopped_analyses` | `TrialAnalysis[]`: analyses settled as failed, phase `stopped` |
+| `already_terminal` | `string[]`: untouched settled ids |
+| `not_found` | `string[]`: missing ids or ids owned by someone else |
+
+This does not cancel the whole job. Repeating the request is safe: terminal work stays unchanged.

package/skills/evolve-evals/references/sdk-reference/secrets.md

@@ -0,0 +1,104 @@
+---
+title: "Secrets"
+description: "Store a credential once, then attach it to evaluation jobs by name."
+---
+
+[Method reference: calls, parameters, and response fields](/sdk-reference/methods/secrets).
+
+Managed evals attach **direct** secrets: their values enter the task sandbox as environment variables. Stored values are write-only through the secrets API.
+
+## Store a value
+
+```ts TypeScript
+import { managedSecrets } from "@evolvingmachines/evolve";
+
+const store = managedSecrets();
+await store.set({
+  name: "SERVICE_TOKEN",
+  value: process.env.SERVICE_TOKEN!,
+  delivery: "direct",
+});
+```
+
+```python Python
+import os
+from evolve import managed_secrets
+
+store = managed_secrets()
+await store.set(
+    name="SERVICE_TOKEN",
+    value=os.environ["SERVICE_TOKEN"],
+    delivery="direct",
+)
+```
+
+Set `SERVICE_TOKEN` in your process environment before running this example. A value may contain at most 190 UTF-8 bytes.
+
+## Attach it to a job
+
+Add a `secrets` list to the job creation input:
+
+```ts TypeScript
+const secrets = [{ name: "SERVICE_TOKEN", as: "APP_TOKEN" }];
+// Pass secrets alongside datasets and agents in jobs().start(...).
+```
+
+```python Python
+secrets = [{"name": "SERVICE_TOKEN", "as": "APP_TOKEN"}]
+# Pass secrets=secrets alongside datasets and agents in jobs().start(...).
+```
+
+| Field | Meaning |
+| --- | --- |
+| `name` | Stored secret name |
+| `label` | Optional labeled version |
+| `as` | Optional environment variable name inside the sandbox; defaults to `name` |
+
+Without a label, resolution chooses the `default` row, then the only row if exactly one exists. Multiple remaining labels produce an ambiguity error.
+
+### Inline values
+
+Job creation can store and attach a value in one request:
+
+```json
+{
+  "secrets": [{
+    "name": "SERVICE_TOKEN",
+    "value": "your-value",
+    "delivery": "direct"
+  }]
+}
+```
+
+Optional `label` and `as` work here too. The value is stored as a secret; the job retains a reference. Restating the same identity/value is allowed. A different value under the same name/label is refused instead of silently replacing it.
+
+## Manage stored secrets
+
+| Action | TypeScript | Python | Returns |
+| --- | --- | --- | --- |
+| Read metadata | `list()` | `list()` | Secret metadata list |
+| Store a value | `set(input)` | `set(name=..., value=..., delivery=..., ...)` | `created` / `updated` and metadata |
+| Delete a labeled value | `delete({name, label})` | `delete(name=..., label=...)` | Deletion result |
+
+`set` requires `name`, `value`, and `delivery`; `label` defaults to `default`. A different value under an existing identity returns `secret_exists`. To rotate, use another label or delete and set that identity.
+
+The broader storage API also accepts `delivery: "brokered"`, with required host/path/method scope lists. Managed eval jobs cannot attach those secrets today. Direct delivery refuses those scoping options.
+
+| Scoping option | TypeScript | Python |
+| --- | --- | --- |
+| Hosts | `allowedHosts` | `allowed_hosts` |
+| Path prefixes | `allowedPathPrefixes` | `allowed_path_prefixes` |
+| HTTP methods | `allowedMethods` | `allowed_methods` |
+
+## Configuration and errors
+
+This factory uses its own configuration:
+
+| Setting | TypeScript | Python |
+| --- | --- | --- |
+| API key | `apiKey` | `api_key` |
+| Dashboard origin | `dashboardUrl` | `dashboard_url` |
+
+Python wraps those fields in `ManagedSecretsClientConfig`. Defaults come from `EVOLVE_API_KEY` and `EVOLVE_DASHBOARD_URL` as usual.
+
+Secret-client HTTP failures raise plain `Error` in TypeScript and `RuntimeError` in Python. They are not the hosted `EvolveApiError` / `EvolveAPIError` class.

package/skills/evolve-evals/references/sdk-reference/skills.md

@@ -0,0 +1,66 @@
+---
+title: "Uploaded skills"
+description: "Store skill folders and attach them to evaluation agents."
+---
+
+[Method reference: calls, parameters, and response fields](/sdk-reference/methods/skills).
+
+`skills()` manages content mounted into an agent run. It is separate from `evolve skills`, which serves documentation to coding agents.
+
+## Upload and attach
+
+```text
+my-skill/
+├── SKILL.md
+├── scripts/
+└── references/
+```
+
+```ts TypeScript
+import { skills, jobs } from "@evolvingmachines/evolve";
+
+const [skill] = await skills().upload("./my-skill");
+const job = await jobs().start({
+  datasets: [{ name: "harbor-examples", version: "1.0", task_names: ["hello-world"] }],
+  agents: [{ name: "codex", model_name: "gpt-5.6-luna", skills: [skill.ref] }],
+  max_trial_spend_usd: 0.50,
+  retry: { max_retries: 0 },
+});
+```
+
+```python Python
+from evolve import skills, jobs
+
+uploaded = await skills().upload("./my-skill")
+skill = uploaded[0]
+job = await jobs().start(
+    datasets=[{"name": "harbor-examples", "version": "1.0", "task_names": ["hello-world"]}],
+    agents=[{"name": "codex", "model_name": "gpt-5.6-luna", "skills": [skill.ref]}],
+    max_trial_spend_usd=0.50,
+    retry={"max_retries": 0},
+)
+```
+
+Upload returns a list. A folder containing multiple child skill folders can create multiple records.
+
+## Pick a reference
+
+| Reference | Meaning |
+| --- | --- |
+| `upload:<id>` | This immutable uploaded content |
+| `name:<skill-name>` | The name's current uploaded version, resolved when the job is created |
+
+Uploading identical content under the same name reuses the record. New content creates a new record and moves the name reference. Existing jobs keep their recorded skill locks.
+
+## Methods
+
+| Action | TypeScript | Python | Returns |
+| --- | --- | --- | --- |
+| Upload a directory | `upload(directory, {org})` | `upload(directory, org=...)` | Skill list |
+| List uploads | `list({scope, limit, cursor})` | `list(scope=..., limit=..., cursor=...)` | Paginated handle |
+| Read content/metadata | `get(idOrNameRef)` | `get(id_or_name_ref)` | Skill metadata and `skill_md` |
+| Delete an upload | `delete(id)` | `delete(id)` | No content |
+
+`org` overrides the client default. List scopes are `my`, `shared`, and `org`. Team members can reference visible skills; only their owner can delete them. A skill used by a non-terminal job cannot be deleted.
+
+The metadata includes `id`, `name`, `org`, `digest`, `size_bytes`, `description`, `ref`, and `created_at`. See [skill concepts](/core-concepts/skills) for git-backed references and mounting behavior.

package/skills/evolve-evals/references/sdk-reference/trials.md

@@ -0,0 +1,164 @@
+---
+title: "Trials"
+description: "Inspect one attempt, from its reward to its trace and files."
+---
+
+**Note:**
+
+For every method’s inputs, return values, and language differences, see the [trials method reference](/sdk-reference/methods/trials).
+
+A trial is one task, one agent/model arm, and one attempt. Get its id from `jobs.trials(jobId)`; its record includes the parent `job_id`.
+
+## Read the result
+
+```ts TypeScript
+import { trials } from "@evolvingmachines/evolve";
+
+const client = trials();
+const trial = await client.get(trialId);
+console.log(trial.status, trial.reward, trial.exception_info);
+```
+
+```python Python
+from evolve import trials
+
+client = trials()
+trial = await client.get(trial_id)
+print(trial.status, trial.reward, trial.exception_info)
+```
+
+| Read | What it tells you |
+| --- | --- |
+| `status`, `attempt_phase` | Whether it is waiting, running, scoring, or settled; the current execution phase |
+| `reward`, `verifier_result` | Primary reward and the verifier's full rewards map |
+| `exception_info` | Why a trial ended with an exception |
+| `agent_info` | Harness, version, model, and effort |
+| `usage`, `spend_source` | Metered usage and whether the reading is final |
+| `analysis` | Latest trace-analysis result, when present |
+| `sandbox_provider`, `sandbox_id` | Where it actually ran |
+| `n_retries`, `retries` | Automatic retry history |
+
+A reward of `0` is a scored result. A missing reward is not automatically zero. See [statuses and nullable values](/sdk-reference/types).
+
+## Read the trace
+
+```ts TypeScript
+for await (const event of client.traceEvents(trialId)) {
+  console.log(event.seq, event.type, event.data);
+}
+```
+
+```python Python
+async for event in client.trace_events(trial_id):
+    print(event.seq, event.type, event.data)
+```
+
+This drains stored events. It does not remain connected waiting for future events.
+
+| Method | Returns |
+| --- | --- |
+| `trace(id, options)` | One page of `TraceEvent` records |
+| `traceEvents(id, options)` / `trace_events(id, ...)` | Async iterator over matching stored events |
+
+Both accept `cursor`, `limit`, `type`, `grep`, and `tail`. TypeScript uses one options object; Python uses keywords. `grep` is a case-insensitive POSIX regex over event type and content. `tail` selects the newest matching events; the server caps it at 10,000.
+
+### Read per-call model usage
+
+Use `gatewayUsageOf` / `gateway_usage_of` to find the gateway's token and cost readings. Other events, including the harness's own usage reports, return null / `None`.
+
+```ts TypeScript
+import { gatewayUsageOf, trials } from "@evolvingmachines/evolve";
+
+for await (const event of trials().traceEvents(trialId)) {
+  const call = gatewayUsageOf(event);
+  if (call) console.log(call.callId, call.usage.costUsd);
+}
+```
+
+```python Python
+from evolve import gateway_usage_of, trials
+
+async for event in trials().trace_events(trial_id):
+    call = gateway_usage_of(event)
+    if call is not None:
+        print(call["callId"], call["usage"]["costUsd"])
+```
+
+| Field | Meaning |
+| --- | --- |
+| `promptTokens` | Input tokens, including cached and cache-written tokens |
+| `completionTokens` | Output tokens |
+| `cachedTokens` | Cached portion of input tokens |
+| `costUsd` | Cost recorded by the gateway |
+
+These fields live under `call.usage` / `call["usage"]`. Do not add the harness's usage to these readings: they describe the same run from different sources. Use the trial's `usage` for its aggregate reading.
+
+## Read stored artifacts
+
+```ts TypeScript
+const atif = await client.artifact(trialId, "trace-atif");
+const verifier = await client.artifact(trialId, "verifier");
+const home = await client.artifact(trialId, "agent-home");
+```
+
+```python Python
+atif = await client.artifact(trial_id, "trace-atif")
+verifier = await client.artifact(trial_id, "verifier")
+home = await client.artifact(trial_id, "agent-home")
+```
+
+| Stream | Result |
+| --- | --- |
+| `trace-atif` | Normalized ATIF trajectory as JSON text |
+| `trace-stdout` | Harness stdout |
+| `trace-stderr` | Harness stderr |
+| `verifier` | Verifier output |
+| `agent-home` | Map of captured sandbox paths to text |
+| `trajectory` | Reserved native-session selector; availability is not guaranteed |
+
+Missing stored content returns null / `None`. `trace-parsed` belongs to `trace`; `filesystem` belongs to the [filesystem client](/sdk-reference/filesystem).
+
+## Read individual files
+
+```text
+Stored trial result tree
+├── config.json
+├── result.json
+├── agent/
+│   ├── trajectory.json
+│   └── … harness output
+├── verifier/
+│   └── … verifier output
+└── artifacts/
+    └── … collected files
+```
+
+The actual files depend on what was produced and retained. Use `files()` to discover them.
+
+```ts TypeScript
+const page = await client.files(trialId, { limit: 200 });
+const bytes = await client.file(trialId, "result.json");
+const prefix = await client.file(trialId, "result.json", { start: 0, end: 1023 });
+```
+
+```python Python
+page = await client.files(trial_id, limit=200)
+content = await client.file(trial_id, "result.json")
+prefix = await client.file(trial_id, "result.json", start=0, end=1023)
+```
+
+`files` accepts `limit` and `cursor`. `file` returns `Buffer` / `bytes`; byte ranges accept `start`, inclusive `end`, or `suffix` for the last N bytes. A suffix takes precedence.
+
+These are stored result files. To browse sandbox paths, use `client.filesystem(trialId)` and the [filesystem methods](/sdk-reference/filesystem).
+
+## Stop, retry, or regrade
+
+| Action | TypeScript | Python | Returns |
+| --- | --- | --- | --- |
+| Stop selected runs | `stop([trialId])` | `stop([trial_id])` | Batch result |
+| Run the attempt again | `retry(trialId, { idempotencyKey })` | `retry(trial_id, idempotency_key=...)` | New linked job |
+| Re-run its eligible verifier | `regrade(trialId)` | `regrade(trial_id)` | New linked job |
+
+`stop` accepts up to 100 trial or analysis ids. Its response separates `stopped`, `stopped_analyses`, `already_terminal`, and `not_found`. Check each group; one missing id does not erase the rest of the result.
+
+Retry requires a settled trial. Regrade requires retained inputs from an eligible separate verifier. [Job operations](/sdk-reference/jobs#cancel-or-run-selected-work-again) describe the differences.