@evolvingmachines/evolve 0.0.65 → 0.0.67

package/skills/evolve-evals/references/cli-reference/check.md

@@ -0,0 +1,156 @@
+---
+title: "evolve check"
+description: "Check task quality, inspect the report, and read the checker's trace."
+---
+
+Check a local task folder, a folder of tasks, or a published dataset. Evolve runs the checker remotely.
+
+### Local tasks
+
+```bash
+evolve check ./tasks --watch
+```
+
+### Published tasks
+
+```bash
+evolve check -d harbor-examples@1.0 -i hello-world --watch
+```
+
+Use exactly one source: a local path or `--dataset`. Local tasks are uploaded for this check; they do not need to be published as a dataset first.
+
+**Note:**
+
+`evolve check` runs an agent and incurs usage. For metadata validation before publishing, use [`evolve dataset check`](/cli-reference/dataset#check-before-publishing).
+
+## Choose tasks
+
+| Option | Meaning |
+| --- | --- |
+| `-d`, `--dataset <name[@version]>` | Use tasks from a published dataset instead of a local path. |
+| `-i`, `--include-task-name <glob>` | Include matching tasks. Repeatable. |
+| `-x`, `--exclude-task-name <glob>` | Exclude matching tasks. Repeatable. |
+| `-l`, `--n-tasks <n>` | Maximum tasks after filtering, selected in sorted order. |
+| `-n`, `--n-concurrent <n>` | Concurrent task checks, `1`–`150`; otherwise your organization's ceiling. |
+
+## Configure the checker
+
+Read the actual platform defaults before customizing them.
+
+```bash
+evolve check --show-defaults
+```
+
+| Option | Meaning |
+| --- | --- |
+| `--name <name>` | Check label. Defaults to the acceptance timestamp. |
+| `-m`, `--model <name>` | Checker model. |
+| `--effort <value>` | Reasoning effort. |
+| `-r`, `--rubric <path>` | TOML, YAML, or JSON rubric. |
+| `-p`, `--prompt <path>` | Replacement prompt text file. |
+| `-e`, `--env <provider>` | Checker sandbox provider. |
+| `--show-defaults` | Print the prompt, rubric, model, effort, and provider, then exit. |
+| `--watch` | Wait until every task check settles. |
+| `-q`, `--quiet` | With watch, suppress intermediate progress. |
+
+`--show-defaults` accepts only [global options](/cli-reference/index#global-options). It cannot be combined with a path, dataset, or checker options.
+
+For rubric format and interpretation, see [Check](/core-concepts/check).
+
+## Read the report
+
+The command prints a parent check ID. Use it as `$CHECK_ID` below.
+
+```bash
+evolve check list
+evolve check show "$CHECK_ID"
+```
+
+```text
+Check
+├── Task check: task A
+├── Task check: task B
+└── Task check: task C
+
+Each task check
+├── Criterion results
+└── Trace
+```
+
+The default rubric's execution criteria say whether a task check ran the task; `unknown` on them is not proof that the solution or verifier ran. A clean inspection without execution is not proof that the solution and verifier work.
+
+### List options
+
+`check list` accepts the shared [list options](/cli-reference/index#list-options), plus:
+
+| Option | Meaning |
+| --- | --- |
+| `--scope <my\|shared\|org>` | Records to list. Default `my`. |
+| `--status <statuses>` | Comma-separated parent check statuses: `queued`, `running`, `completed`. |
+| `-d`, `--dataset <name[@version]>` | Filter checks by dataset. A bare name includes all versions. |
+
+`check show` returns exit code `1` if any task-check execution failed. A criterion marked `fail` is a quality finding, not that execution error.
+
+## Inspect one task check
+
+Find the task-check ID in `evolve check show "$CHECK_ID" --json`. Use that ID as `$TASK_CHECK_ID`.
+
+```bash
+evolve check trace "$TASK_CHECK_ID"
+evolve check trace "$TASK_CHECK_ID" --since 200
+```
+
+`--since <n>` skips the first N transcript events. JSON output is one event per line. For files, logs, and live processes, use the [filesystem commands](/cli-reference/filesystem).
+
+**Note:**
+
+The parent check ID selects the whole report. Trace, files, logs, and processes need an individual **task-check ID**.
+
+## Download
+
+### Whole check
+
+```bash
+evolve check download "$CHECK_ID" -o checks/
+```
+
+Saves `check-<id>/` with the report and task-check folders.
+
+### One task check
+
+```bash
+evolve check download "$TASK_CHECK_ID" -o checks/
+```
+
+Saves that task check's folder.
+
+### One artifact
+
+```bash
+evolve check download "$TASK_CHECK_ID" --stream task-check
+evolve check download "$TASK_CHECK_ID" --stream trace-stdout
+```
+
+| Option | Meaning |
+| --- | --- |
+| `-o`, `--output-dir <dir>` | Parent folder. Default `checks/`. |
+| `--overwrite` | Allow replacing an existing folder. |
+| `--stream <artifact>` | Print `task-check`, `trace-parsed`, `trace-stdout`, `trace-stderr`, or `agent-home`. Requires a task-check ID. |
+| `--since <n>` | Skip N transcript events with `--stream trace-parsed`. |
+
+Stream mode cannot be combined with output directory or overwrite.
+
+## Share a report
+
+The creator can share the parent check by link or email.
+
+```bash
+evolve check share "$CHECK_ID" --link
+evolve check share "$CHECK_ID" --email colleague@example.com
+evolve check shares "$CHECK_ID"
+evolve check unshare "$CHECK_ID" --link
+```
+
+`share` and `unshare` accept `--link`, repeatable `--email <address>`, or both; at least one is required. Email sharing sends a link to the recipient.
+
+[Sharing rules](/core-concepts/sharing) and [global options](/cli-reference/index#global-options) apply.

package/skills/evolve-evals/references/cli-reference/dataset.md

@@ -0,0 +1,154 @@
+---
+title: "evolve dataset"
+description: "Find tasks, validate a corpus, and publish a version for evaluation."
+---
+
+A dataset version is a named collection of tasks. Jobs select `name@version`, or the active version when only the name is given.
+
+```bash
+evolve dataset list --search harbor-examples
+evolve dataset show harbor-examples@1.0
+```
+
+## Browse the catalog
+
+| Command | Options |
+| --- | --- |
+| `dataset list` | `--search <text>` matches names and descriptions; all shared [list options](/cli-reference/index#list-options). |
+| `dataset show <name[@version]>` | `-l, --limit <n>` and `--cursor <cursor>` page the task list. |
+
+The detail view includes versions, tasks, build failures, and provider information.
+
+## Check before publishing
+
+```bash
+evolve dataset check ./tasks
+```
+
+This reads local task metadata and sends it to Evolve for validation. It does not publish the corpus or run the tasks.
+
+For an agent-based quality check, use [`evolve check`](/cli-reference/check).
+
+## Publish a version
+
+Choose one source.
+
+### Local directory
+
+```bash
+evolve dataset publish \
+  --dir ./tasks \
+  --name my-benchmark \
+  --version 1.0 \
+  --watch
+```
+
+A `dataset.toml` manifest can supply the name and version. Local publishing runs metadata preflight before uploading.
+
+### Git repository
+
+```bash
+evolve dataset publish \
+  --git https://github.com/acme/tasks.git \
+  --ref v1.0.0 \
+  --path benchmark \
+  --name my-benchmark \
+  --version 1.0 \
+  --watch
+```
+
+Pin a tag or full commit SHA. Branch names are refused. Omit `--path` to use the repository root.
+
+### Public archive
+
+```bash
+evolve dataset publish \
+  --from https://example.com/tasks.tar.gz \
+  --name my-benchmark \
+  --version 1.0 \
+  --watch
+```
+
+Evolve fetches the HTTPS archive. Supply both name and version.
+
+### Harbor package
+
+```bash
+evolve dataset publish --from hub:cookbook/hello-world --watch
+```
+
+Import a public Harbor Hub package using `hub:org/name[@ref]`. Name and version may come from the package.
+
+### Publish options
+
+| Option | Meaning |
+| --- | --- |
+| `--dir <path>` | Local corpus directory. |
+| `--git <url>` | Git repository URL. Requires `--ref`. |
+| `--ref <tag\|sha>` | Pinned Git revision. |
+| `--path <subfolder>` | Corpus folder within the Git repository. |
+| `--from <source>` | Public HTTPS tarball or `hub:org/name[@ref]`. |
+| `--name <name>` | Dataset name. Required for Git and HTTPS sources. |
+| `--version <version>` | Version label. Required for Git and HTTPS sources. |
+| `--org <name>` | Organization for a new dataset; otherwise the saved CLI default, then personal. |
+| `--skip-preflight` | Skip metadata preflight for a local directory. |
+| `--watch` | Follow the import and version build to completion. |
+
+Do not combine source types. `--skip-preflight` skips the early check; it does not remove import validation.
+
+## Follow the build
+
+```text
+1. Upload or fetch the corpus
+                 ↓
+2. Validate tasks and build environments
+                 ↓
+3. Settle the version
+   ├── At least one task built → READY
+   └── No task built          → FAILED
+```
+
+A `READY` version can contain failed tasks. The final report shows how many built and which failed. On your own dataset, the new ready version becomes active.
+
+To reconnect while an import is running:
+
+```bash
+evolve dataset watch my-benchmark
+```
+
+To inspect a specific import, including one that has already settled, use its full ID:
+
+```bash
+evolve dataset watch "$IMPORT_ID"
+```
+
+`$IMPORT_ID` is printed during publishing. Watching by dataset name selects a live import; when none exists, the command returns `nothing_to_watch`.
+
+With `--json`, watching prints progress records followed by `import.final`.
+
+## Change the active version
+
+```bash
+evolve dataset activate my-benchmark 1.0
+```
+
+The chosen version must be `READY`. This changes what a bare `my-benchmark` selects; version-pinned jobs still use their selected version.
+
+## Download the corpus
+
+```bash
+evolve dataset download my-benchmark@1.0 -o corpora/
+```
+
+`-o, --output-dir <dir>` sets the destination; the default is the current directory. This downloads the original stored corpus package. Access is limited to the dataset owner.
+
+## Read one task's files
+
+```bash
+evolve dataset files list harbor-examples@1.0 hello-world /
+evolve dataset files cat harbor-examples@1.0 hello-world /instruction.md
+```
+
+Use [`dataset files status/list/cat`](/cli-reference/filesystem#published-task-files) for package metadata, folder pagination, and byte ranges. These commands require an explicit dataset version.
+
+[Global options](/cli-reference/index#global-options) apply. Next, [run a job](/cli-reference/run) against the published version.

package/skills/evolve-evals/references/cli-reference/filesystem.md

@@ -0,0 +1,168 @@
+---
+title: "Files, logs, and processes"
+description: "Inspect a running sandbox, captured files, or a published task package."
+---
+
+The same read commands work for trials, analysis runs, and individual task checks.
+
+| Resource | Command prefix | ID source |
+| --- | --- | --- |
+| Evaluation trial | `evolve trial` | `evolve job trials "$JOB_ID"` |
+| Analysis run | `evolve analysis` | `evolve analysis list` |
+| Task check | `evolve check` | `evolve check show "$CHECK_ID" --json` |
+
+Use the corresponding full ID as `$TRIAL_ID`, `$ANALYSIS_ID`, or `$TASK_CHECK_ID` below.
+
+## Choose a resource
+
+### Trial
+
+```bash
+evolve trial files status "$TRIAL_ID"
+evolve trial files list "$TRIAL_ID" /
+evolve trial files cat "$TRIAL_ID" /app/result.txt
+evolve trial files search "$TRIAL_ID" 'error' --path /app
+evolve trial files changes "$TRIAL_ID"
+evolve trial files archive "$TRIAL_ID" --path /app -o archives/
+evolve trial logs "$TRIAL_ID" --stream agent --follow
+evolve trial procs "$TRIAL_ID"
+```
+
+### Analysis
+
+```bash
+evolve analysis files status "$ANALYSIS_ID"
+evolve analysis files list "$ANALYSIS_ID" /
+evolve analysis files cat "$ANALYSIS_ID" /app/analysis.json
+evolve analysis files search "$ANALYSIS_ID" 'error' --path /app
+evolve analysis files changes "$ANALYSIS_ID"
+evolve analysis files archive "$ANALYSIS_ID" --path /app -o archives/
+evolve analysis logs "$ANALYSIS_ID" --stream agent --follow
+evolve analysis procs "$ANALYSIS_ID"
+```
+
+Use an analysis ID, not the evaluated trial's ID.
+
+### Task check
+
+```bash
+evolve check files status "$TASK_CHECK_ID"
+evolve check files list "$TASK_CHECK_ID" /
+evolve check files cat "$TASK_CHECK_ID" /app/task/task.toml
+evolve check files search "$TASK_CHECK_ID" 'error' --path /app
+evolve check files changes "$TASK_CHECK_ID"
+evolve check files archive "$TASK_CHECK_ID" --path /app -o archives/
+evolve check logs "$TASK_CHECK_ID" --stream agent --follow
+evolve check procs "$TASK_CHECK_ID"
+```
+
+Use the individual task-check ID, not the parent check ID.
+
+The file paths above are examples. List the folder first and choose a path that exists in your run.
+
+## Live or captured?
+
+`files status` reports what is available.
+
+| State | What you can inspect |
+| --- | --- |
+| `live` | Files in the running sandbox. |
+| `capturing` | Capture is still being written. |
+| `captured` | The stored file tree. |
+| `none` | No live sandbox or available capture. |
+
+Reads normally choose the available source. Add `--source live` or `--source capture` to require one; an unavailable source returns an error.
+
+**Note:**
+
+A capture is not a full copy of the original image. Unchanged image files can appear in listings without stored bytes. Reading one returns `not_captured`. Search also excludes files whose bytes were not captured.
+
+## Browse and read
+
+| Command | Options |
+| --- | --- |
+| `files status <id>` | No command-specific options. |
+| `files list <id> [path]` | Path defaults to `/`; `--source`; `-l, --limit` (default `500`, max `1000`); `--cursor <name>`. |
+| `files cat <id> <path>` | `--source`; `--range <bytes=a-b>`. |
+
+`files ls` is an alias of `files list`. Listing cursors come from `next_cursor`.
+
+### Byte ranges and binary output
+
+| Range | Selects |
+| --- | --- |
+| `bytes=0-1023` | The first 1024 bytes. |
+| `bytes=1024-` | Everything from byte 1024 onward. |
+| `bytes=-1024` | The last 1024 bytes. |
+
+`files cat` writes raw bytes. Redirect them to a local file when needed.
+
+```bash
+evolve trial files cat "$TRIAL_ID" /app/result.txt > result.txt
+```
+
+With `--json`, the response contains base64 `data`, `encoding`, `bytes`, and `path`.
+
+## Search and changes
+
+| Command | Options |
+| --- | --- |
+| `files search <id> <text>` | `--path <folder>` (default `/`); `--regex`; `-l, --limit` (default `200`, max `1000`); `--source`. |
+| `files changes <id>` | `--phase <setup\|agent\|verifier\|all>` (default `all`); `-l, --limit` (default `500`, max `1000`); `--cursor <path>`; `--source`. |
+
+Search reports file paths, line numbers, and snippets. Narrow `--path` when searching the whole sandbox is unnecessary.
+
+Changes report created, modified, and removed files by execution phase.
+
+## Download a subtree
+
+```bash
+evolve trial files archive "$TRIAL_ID" --path /app -o archives/
+```
+
+| Option | Meaning |
+| --- | --- |
+| `--path <path>` | Subtree to archive. Default `/`. |
+| `--source <live\|capture>` | Require a specific source. |
+| `-o`, `--output-dir <dir>` | Save the `.tar.gz` file and print its path. |
+
+Without an output directory, the command writes archive bytes to stdout. With `--json`, those bytes are base64; when saving to disk, JSON returns the saved path.
+
+## Logs
+
+```bash
+evolve trial logs "$TRIAL_ID" --stream agent --follow
+```
+
+| Option | Meaning |
+| --- | --- |
+| `--stream <name>` | Required: `agent`, `verifier`, or `system`. |
+| `-f`, `--follow` | Keep printing new lines while the sandbox lives. |
+| `-l`, `--limit <n>` | Lines per page without follow. Default and maximum `1000`. |
+| `--cursor <seq>` | Resume after this line sequence. |
+
+`system` is recorded only when the job enabled `--system-log`. `setup` and `metrics` are accepted stream names but are not recorded today.
+
+Without follow, JSON is a page of lines. With follow, each line is a separate JSON record. `--limit` does not limit a follow stream.
+
+## Processes
+
+`procs <id>` reads the running sandbox's process list. It accepts only [global options](/cli-reference/index#global-options) and is unavailable after the sandbox ends.
+
+## Published task files
+
+These commands read the **task package**, not a trial sandbox. Pin the dataset version.
+
+```bash
+evolve dataset files status harbor-examples@1.0 hello-world
+evolve dataset files list harbor-examples@1.0 hello-world /
+evolve dataset files cat harbor-examples@1.0 hello-world /instruction.md
+```
+
+| Command | Options |
+| --- | --- |
+| `dataset files status <name@version> <task>` | No command-specific options. |
+| `dataset files list <name@version> <task> [path]` | Path defaults to `/`; `-l, --limit` (default `500`, max `1000`); `--cursor <name>`. |
+| `dataset files cat <name@version> <task> <path>` | `--range <bytes=a-b>`. |
+
+Task-package reads have no live/capture selector. All commands accept [global options](/cli-reference/index#global-options).

package/skills/evolve-evals/references/cli-reference/index.md

@@ -0,0 +1,149 @@
+---
+title: "CLI reference"
+description: "Run evaluations, inspect results, and manage the resources around them."
+---
+
+Use `evolve` to run evaluations in the cloud. Add `--help` to any command for its options.
+
+```bash
+evolve run --help
+evolve job list --json
+```
+
+**[Run an evaluation](/cli-reference/run)**
+
+Select tasks, agents, models, and a spend cap.
+
+**[Inspect results](/cli-reference/job)**
+
+Read scores, compare jobs, and inspect trials.
+
+**[Check and analyze](/cli-reference/check)**
+
+Check task quality or analyze completed traces.
+
+**[Read files and logs](/cli-reference/filesystem)**
+
+Browse a running sandbox or its captured files.
+
+## Choose a command
+
+| You want to… | Command |
+| --- | --- |
+| Start an evaluation | [`run`](/cli-reference/run), also `job start` |
+| Inspect or act on a job | [`job`](/cli-reference/job) |
+| Inspect or act on one trial | [`trial`](/cli-reference/trial) |
+| Analyze completed traces | [`analyze`](/cli-reference/analyze) |
+| Read analysis runs | [`analysis`](/cli-reference/analysis) |
+| Check tasks and read reports | [`check`](/cli-reference/check) |
+| Browse or publish tasks | [`dataset`](/cli-reference/dataset) |
+| Import completed results | [`upload`](/cli-reference/upload) |
+| Register a custom agent | [`agent`](/cli-reference/agent) |
+| Upload skills for evaluated agents | [`skill`](/cli-reference/skill) |
+| Read the bundled agent manual | [`skills`](/cli-reference/skills) |
+| Check identity or manage teams | [`auth`](/cli-reference/auth) |
+| Store environment secrets | [`secrets`](/cli-reference/secrets) |
+| Inspect managed-agent sessions | [`session`](/cli-reference/session) |
+
+`skill` and `skills` are different: one manages uploaded content; the other reads the manual shipped with your CLI.
+
+## Global options
+
+Put options after the command, as in `evolve job list --json`.
+
+| Option | Meaning |
+| --- | --- |
+| `--json` | Machine-readable output. The shape depends on the command; see [Output](#output). |
+| `--api-key <key>` | Override `EVOLVE_API_KEY`. |
+| `--base-url <url>` | Override `EVOLVE_DASHBOARD_URL`, otherwise use the Evolve dashboard API. |
+| `-h`, `--help` | Show help for the command. |
+
+`evolve --version` or `evolve -v` prints the installed version. Bare `evolve` prints help.
+
+## List options
+
+Paged lists return **one page**. Pass the returned cursor to read the next.
+
+```bash
+evolve job list --limit 20 --json
+evolve job list --cursor "$NEXT_CURSOR" --json
+```
+
+| Option | Meaning |
+| --- | --- |
+| `-l`, `--limit <n>` | Page size. |
+| `--cursor <cursor>` | Read the next page. |
+| `--columns <keys>` | Comma-separated column names, in the order you want. |
+| `--columns all` | Show every available column. |
+| `--columns help` | List available columns without an API request. |
+| `-q`, `--quiet` | Print only IDs, or the resource's name where it has no ID column. |
+| `--no-trunc` | Keep full cell text in terminal tables. |
+| `--no-headers` | Omit the header from piped TSV output. |
+
+These options apply to paged `list` commands, `job trials`, `job tasks`, and `job imports`. `auth org list` and `secrets list` use the formatting options only. The local `skills list` has its own simpler output.
+
+### Scope
+
+`job list`, `analysis list`, `check list`, `session list`, `agent list`, and `skill list` accept `--scope`.
+
+| Value | Records selected |
+| --- | --- |
+| `my` | Records you created. Default. |
+| `shared` | Other users’ records available through your organizations or supported email shares. |
+| `org` | Records across your organizations, including your own. |
+
+Email shares apply to jobs, analyses, checks, and sessions. Analyses follow the ownership and sharing of their source job.
+
+Use full IDs when opening records from another scope. Prefix lookup usually searches your own records.
+
+## IDs and values
+
+Most positional UUIDs accept an unambiguous prefix of at least eight characters. A full ID avoids the extra lookup requests.
+
+**Note:**
+
+Use **full trial IDs** with `analyze --trial` and `job retry --trial`. Import IDs must also be copied in full.
+
+A trial prefix may require reading the trials of every job in scope. On large accounts, prefer the full ID from `evolve job trials "$JOB_ID"`.
+
+Repeat a repeatable flag once per value. Do not combine short flags.
+
+```bash
+evolve run -d harbor-examples@1.0 -a codex -m gpt-5.6-luna \
+  -i 'hello-*' -x '*-slow' --max-trial-spend 0.30 --max-retries 0
+```
+
+Values may begin with `-`. If a value is also a recognized option, use `--flag=value` to make it explicit.
+
+### Aliases
+
+`jobs`, `trials`, `analyses`, `sessions`, and `datasets` alias their singular nouns. `secret` aliases `secrets`. `ls` aliases `list`, including `files ls` and `auth org ls`.
+
+`agents` is reserved and is not an alias of `agent`. `skills` is not an alias of `skill`.
+
+## Output
+
+| Command shape | Output with `--json` |
+| --- | --- |
+| Ordinary read or list | One JSON object or page. Multiple `job show` IDs return an array. |
+| Run, analysis, or check with `--watch` | One JSON record per line: accepted, progress, final. |
+| Dataset publish/watch | One JSON record per line while watching. |
+| Upload or `job import --watch` | One final job or error object. |
+| Trace or `logs --follow` | One event or log line per JSON record. |
+| File or archive to stdout | Base64 data with a byte count. Without `--json`, raw bytes. |
+
+Without `--json`, lists use aligned tables in a terminal and tab-separated rows when piped. `-q` is command-specific: on lists it selects IDs; during a watch it suppresses progress.
+
+Runtime and API errors print an error object under `--json`, plus a human message on stderr. Usage errors print only to stderr. A partially completed `job stop` can print its partial result before an error.
+
+## Exit codes
+
+| Code | Meaning |
+| --- | --- |
+| `0` | Command succeeded. An accepted asynchronous request may still be running. |
+| `1` | Runtime/API failure, watched job cancelled or failed, or an analysis/task-check execution failed. |
+| `2` | Invalid command usage or launch quota refusal. |
+
+A completed job can contain failed trials. A check criterion marked `fail` is a quality finding; an errored checker is an execution failure. `check show` also returns `1` when a task checker failed.
+
+For API error codes, see [Errors](/sdk-reference/errors).