@evolvingmachines/evolve 0.0.65 → 0.0.67-train-harness-wave-2026-09-25.20260926.e00a8a6d

package/skills/evolve-evals/references/cli-reference/check.mdx

@@ -1,263 +0,0 @@
----
-title: "evolve check"
-description: "Check task quality against a rubric, and read checks back."
----
-
-import GlobalOptions from '/snippets/global-options.mdx';
-import ListFlags from '/snippets/list-flags.mdx';
-
-```bash
-evolve check [<path>] [options]
-```
-
-Check task quality against a rubric, server-side. `<path>` is one task directory or a directory of task directories, uploaded as one archive (`upload_too_large` past the published ceiling); `-d` names a published dataset instead, one or the other. Without `--watch` the command prints the accepted check with the hint `Follow it with: evolve check show <id>`; `--json` prints the accepted check body. Exit 0 on an accepted check; with `--watch`, and on `check show`, exit 1 when any task errored.
-
-```bash
-evolve check ./tasks \
-  -i 'abs-*' \
-  -l 5 \
-  --watch
-evolve check --show-defaults
-```
-
-## Options
-
-<ParamField path="-d, --dataset" type="name[@version]">
-  Check a published dataset's tasks instead of a local path. The checker reads the version's retained task package, and the job page's CHECK tab lists these checks by task.
-</ParamField>
-
-<ParamField path="--name" type="name">
-  A name for the check, Harbor's `--job-name`. Omitted, the accept timestamp in Harbor's shape, `YYYY-MM-DD__HH-MM-SS`.
-</ParamField>
-
-<ParamField path="-m, --model" type="name" default="openrouter/deepseek/deepseek-v4.1-flash">
-  Model the checker runs. `fireworks/deepseek-v4.1-flash` is the same model on its Fireworks route. Must be on the `claude` roster.
-</ParamField>
-
-<ParamField path="--effort" type="value">
-  Reasoning effort the checker runs at. Defaults to the per-model default.
-</ParamField>
-
-<ParamField path="-r, --rubric" type="path">
-  Rubric file: TOML, YAML or JSON in the `{criteria: [{name, description, guidance}]}` shape. Defaults to the platform's check rubric, eleven criteria.
-</ParamField>
-
-<ParamField path="-p, --prompt" type="path">
-  Prompt file for the evaluator agent. Its text replaces the built-in check prompt, with `{task_path}`, `{file_tree}` and `{criteria_guidance}` rendered. Defaults to the built-in prompt.
-</ParamField>
-
-<ParamField path="-e, --env" type="provider">
-  Sandbox provider the checker runs on. Defaults to the platform's analysis default.
-</ParamField>
-
-<ParamField path="--show-defaults">
-  Print the defaults a check runs under when you name none: model, effort, provider, the rubric's criteria and the built-in prompt template, then exit. With `--json`, one object. Takes no path and no other check flag.
-</ParamField>
-
-<ParamField path="-n, --n-concurrent" type="n">
-  Max concurrent task checks, beneath the organization's ceiling. Defaults to the ceiling.
-</ParamField>
-
-<ParamField path="-i, --include-task-name" type="glob">
-  Only check tasks matching the glob. Repeatable.
-</ParamField>
-
-<ParamField path="-x, --exclude-task-name" type="glob">
-  Skip tasks matching the glob. Repeatable.
-</ParamField>
-
-<ParamField path="-l, --n-tasks" type="n">
-  Max tasks to check, after the globs, in sorted directory order.
-</ParamField>
-
-<ParamField path="--watch">
-  Poll until every task check settles, then print the report. One task prints its label line, a `CHECK | OUTCOME | EXPLANATION` table with the evidence beneath each verdict, and the cost; several tasks print a `TASK | LABEL | PASS | FAIL | N/A | UNKNOWN | COST ($) | TASK CHECK ID` table with each failed task's reason below it. With `--json`, NDJSON envelopes: `check.accepted`, `check.progress` per change, `check.final`.
-</ParamField>
-
-<ParamField path="-q, --quiet">
-  With `--watch`: suppress the progress lines and print the final report only.
-</ParamField>
-
-<GlobalOptions />
-
-## Reading checks back
-
-The same word, followed by `list`, `show`, `trace`, `download`, `share`, `unshare` or `shares`, reads checks back or shares them. A task directory literally named like one of these verbs is written `./list`.
-
-### check list
-
-```bash
-evolve check list [options]
-```
-
-List your task quality checks, newest first.
-
-<ListFlags />
-
-<ParamField path="--scope" type="my | shared">
-  Visibility scope: `my`, what you created (the default), or `shared`, your organizations' rows that teammates created plus the checks shared with your address.
-</ParamField>
-
-<ParamField path="--status" type="s1,s2,...">
-  Filter by check status: `queued`, `running`, `completed`.
-</ParamField>
-
-<ParamField path="-d, --dataset" type="name[@version]">
-  Only checks on this dataset. A bare name is every version, `name@version` one version. A value with an empty half (`name@`, `@version`) is refused as `invalid_input` on `dataset`.
-</ParamField>
-
-```bash
-evolve check list --status running
-evolve check list --dataset terminal-bench-4@4.0
-```
-
-### check show
-
-```bash
-evolve check show <check-id>
-```
-
-Show one task quality check in full, with one result per task. The `visibility` row is `PRIVATE` or `LINK` — whether the check's share link is on ([check shares](#check-shares) prints the link and the addresses).
-
-### check trace
-
-```bash
-evolve check trace <task-check-id> [options]
-```
-
-Print the checker's own parsed transcript for one task check.
-
-<ParamField path="--since" type="n">
-  Skip the first N events. To resume, pass the count you already hold.
-</ParamField>
-
-### check download
-
-```bash
-evolve check download <check-id | task-check-id> [options]
-```
-
-Save a check as a check folder (`check_report.json` plus one folder per task) by its check id, one task check's folder by its task check id, or stream one task check's artifact.
-
-<ParamField path="-o, --output-dir" type="dir" default="checks/">
-  Directory to save under. A check id lands in `<dir>/check-<id>/`, a task check id in `<dir>/check-<task>__<7 chars>/`, the run's own folder name.
-</ParamField>
-
-<ParamField path="--overwrite">
-  Replace an existing folder of the same name.
-</ParamField>
-
-<ParamField path="--stream" type="artifact">
-  Print one artifact of a task check to stdout instead of saving: `task-check` (the result document), `trace-parsed`, `trace-stdout`, `trace-stderr` or `agent-home`.
-</ParamField>
-
-<ParamField path="--since" type="n">
-  With `--stream trace-parsed`: skip the first N events.
-</ParamField>
-
-```bash
-evolve check download 3f9a1c2e-… -o checks/
-```
-
-### check share
-
-```bash
-evolve check share <check-id> --link
-evolve check share <check-id> --email <address> [--email <address>...]
-```
-
-Share a check you created, the way [job share](/cli-reference/job#job-share) shares a job. `--link` turns on the check's unlisted link and prints it; running it again prints the same link. Whoever opens the link reads the check's report and downloads its check folder, without signing in. `--email` shares the check with an address: that person gets an email with a link and reads the check after signing in with the same address (a person without an account gets a sign-up link in the same email; an address already on the waitlist or holding an invite gets the share email without a sign-up link and uses that invite instead). Both flags may ride one command. Someone the check is shared with reads it — the report, each task check's files and logs, the download — and finds it under `evolve check list --scope shared`; they only read, and can never stop one of its task checks. Only the check's creator may share it; another member of the organization is refused with `org_forbidden`.
-
-```bash
-evolve check share 3f9a1c2e-… --link
-evolve check share 3f9a1c2e-… --email alice@example.org
-```
-
-### check unshare
-
-```bash
-evolve check unshare <check-id> --link
-evolve check unshare <check-id> --email <address>
-```
-
-Take a share back. `--link` turns the link off at once — the old link stops working, and a later `check share --link` prints a new one. `--email` removes that address's share. Both are safe to repeat.
-
-### check shares
-
-```bash
-evolve check shares <check-id>
-```
-
-Print who the check is shared with: its visibility, the link while it is on, and every address with who shared it and when. `--json` prints the same as `{"visibility", "link": {"enabled", "url"}, "emails": [{"email", "shared_by", "created_at"}]}`.
-
-## check files
-
-A task check's run, by its task check id (`results[].id` on `evolve check show`).
-
-
-```bash
-evolve check files <status|ls|cat|search|changes|archive> <task-check-id> [path] [options]
-```
-
-The run's file system: while the box runs, `files` reads it as it changes; after the run, the same commands read the kept tree. `files status` says which you get.
-
-- `files status <task-check-id>` — `live`, `captured`, `capturing` (the kept tree is still being written) or `none`; the box, how change events arrive (`watcher`), and the capture record (`ready`, `incomplete` with what was left out, or `failed`).
-- `files ls <task-check-id> [path]` — one folder, sorted by name (default `/`). Each row shows the type, mode, size, mtime and name; `created (agent)` or `modified (verifier)` marks what the run changed and when, `image only` marks a file the run never touched (it lists but does not open on the kept tree).
-- `files cat <task-check-id> <path>` — the raw bytes of one file, to stdout. `--range bytes=a-b` (or `bytes=a-`, `bytes=-n`) reads a slice.
-- `files search <task-check-id> <text>` — every line matching the text, as `path:line: snippet`. `--path` narrows the search to one folder (the whole box takes seconds); `--regex` treats the text as a pattern; a `(truncated)` note means there was more.
-- `files changes <task-check-id>` — the files the run created, modified or removed, with the phase (`--phase agent|verifier`).
-- `files archive <task-check-id>` — a `.tar.gz` of one subtree (`--path`, default the whole tree) to stdout, or saved under `-o <dir>`. The kept tree's archive lists the files it does not carry in `MANIFEST.tsv`; while the box runs, a subtree too large to read out in time is refused (`feature_unsupported`).
-
-<ParamField path="--source" type="live|capture">
-  Force the source. Without it, whichever the run has. A source that is not there is refused with `filesystem_state`.
-</ParamField>
-
-<ParamField path="-l, --limit" type="n">
-  Page size on `ls`, `search` and `changes` (`ls`, `changes` default 500; `search` default 200; max 1000).
-</ParamField>
-
-<ParamField path="--cursor" type="c">
-  Resume a paged `ls` or `changes` after the previous page's `next_cursor`.
-</ParamField>
-
-```bash
-evolve check files ls 3f9a1c2e-… /app/work
-evolve check files search 3f9a1c2e-… 'permission denied' --path /app/work
-evolve check files archive 3f9a1c2e-… --path /app/work -o archives/
-```
-
-## check logs
-
-```bash
-evolve check logs <task-check-id> --stream <name> [--follow] [options]
-```
-
-The box's own streams: `agent` (the harness's stdout and stderr), `verifier` and `system` (the box's system log, only when the job asked for it with `--system-log`); `setup` and `metrics` are named but not recorded today, so their pages are empty and say why. One page by default; `--follow` keeps printing lines as they arrive while the box lives, and returns once it is gone.
-
-<ParamField path="--stream" type="name" required>
-  `agent`, `verifier`, `system`, `setup` or `metrics`.
-</ParamField>
-
-<ParamField path="-f, --follow">
-  Keep printing new lines while the box lives.
-</ParamField>
-
-<ParamField path="-l, --limit" type="n" default="1000">
-  Lines per page, max 1000.
-</ParamField>
-
-<ParamField path="--cursor" type="seq">
-  Resume after this line.
-</ParamField>
-
-```bash
-evolve check logs 3f9a1c2e-… --stream agent --follow
-```
-
-## check procs
-
-```bash
-evolve check procs <task-check-id>
-```
-
-The processes running in the box right now. Live only: once the box is gone the command is refused with `filesystem_state`.

package/skills/evolve-evals/references/cli-reference/dataset.mdx

@@ -1,151 +0,0 @@
----
-title: "evolve dataset"
-description: "Browse and publish the dataset catalog."
----
-
-import GlobalOptions from '/snippets/global-options.mdx';
-import ListFlags from '/snippets/list-flags.mdx';
-
-```bash
-evolve dataset <command> [options]
-```
-
-## dataset list
-
-```bash
-evolve dataset list [options]
-```
-
-List the dataset catalog.
-
-<ListFlags />
-
-<ParamField path="--search" type="text">
-  Free-text filter over name and description.
-</ParamField>
-
-## dataset show
-
-```bash
-evolve dataset show <name[@version]> [options]
-```
-
-Show one dataset: its versions, its manifest identity, its tasks with a NOTES column, the providers each task can run on, and the failed tasks of the shown version with their reasons. One line reports a git dataset whose upstream ref has moved; it never appears in `--json`, where the same fact is the `upstream` field.
-
-<ParamField path="-l, --limit" type="n">
-  Task-list page size.
-</ParamField>
-
-<ParamField path="--cursor" type="c">
-  Resume the task list from a cursor.
-</ParamField>
-
-```bash
-evolve dataset show terminal-bench-4@4.0
-```
-
-## dataset check
-
-```bash
-evolve dataset check <dir>
-```
-
-Pre-flight a local corpus. A dry run: nothing is uploaded, nothing is written. Prints a verdict per task and exits 1 on any refusal.
-
-## dataset publish
-
-```bash
-evolve dataset publish [options]
-```
-
-Publish a dataset version from a git source, a local directory, or a fetchable source: a public tarball URL, or a public hub package. A name owned by someone else is refused before the bytes move, and an archive over the published ceiling is refused with `import_too_large`.
-
-A directory upload prints `upload <sent>/<total> (P%)` once per 10% of the archive; with `--json --watch` the same steps are `upload.progress` events (`sent_bytes`, `total_bytes`, `elapsed_sec`). A large upload prints `Registered import <id>` early and reads `QUEUED (receiving)` until the corpus has arrived. Once the build settles, `--watch` prints every task's outcome and ends with `built N of M tasks`.
-
-<ParamField path="--git" type="url">
-  Git repository URL. With `--ref`.
-</ParamField>
-
-<ParamField path="--ref" type="ref">
-  Pinned git ref: a full 40-hex commit sha, or a tag, resolved to its commit at publish and verified at import. Branch names are refused as `unpinned_git_ref`. With `--git`.
-</ParamField>
-
-<ParamField path="--path" type="subfolder">
-  Repository subfolder holding the corpus. With `--git`; only that folder is imported.
-</ParamField>
-
-<ParamField path="--dir" type="path">
-  Local corpus directory, tarred and uploaded. Pre-flighted first; see `--skip-preflight`.
-</ParamField>
-
-<ParamField path="--from" type="url | hub:org/name[@ref]">
-  Fetchable source the server pulls itself, with no local bytes: a public https tarball URL, or `hub:org/name[@ref]`, a public Harbor hub package. The ref is the latest tag by default, a revision number, or `sha256:<digest>`; it is resolved and digest-pinned when the publish is accepted.
-</ParamField>
-
-<ParamField path="--name" type="dataset">
-  Catalog dataset name to create or extend. Optional with `--dir` when the corpus carries a `dataset.toml` manifest, and with `--from hub:…`, which defaults to the package's short name. Required with `--git` and `--from <url>`.
-</ParamField>
-
-<ParamField path="--version" type="v">
-  Version label for the published version. Optional with `--dir` when `dataset.toml` declares one, and with `--from hub:…`, which defaults to the resolved revision. Required with `--git` and `--from <url>`.
-</ParamField>
-
-<ParamField path="--org" type="name">
-  Organization a new dataset lands in, so your teammates can run it. Omitted, the default set with `evolve auth org use`, else your personal organization; `--org personal` picks your personal organization for this one publish over that default. An existing dataset stays where it is; naming a different organization for it is refused.
-</ParamField>
-
-<ParamField path="--watch">
-  Poll until the publish settles: the version `READY` (built and active) or `FAILED`.
-</ParamField>
-
-<ParamField path="--skip-preflight">
-  Upload without the pre-flight check. A refused task then lands `FAILED` at import instead of being caught here.
-</ParamField>
-
-```bash
-evolve dataset publish --from hub:cookbook/hello-world --watch
-evolve dataset publish --dir ./corpus --name my-swe --version 1.0 --org acme
-```
-
-## dataset watch
-
-```bash
-evolve dataset watch <name | import-id>
-```
-
-Re-attach to a publish and follow it to `READY` or `FAILED`, the same follow `dataset publish --watch` renders. Works after the CLI exited, or from another machine. A name with no live publish is refused, naming the newest settled import. With `--json` the same events stream, opened with `import.attached`; the transfer's own `upload.progress` lines belong to the publishing process and never appear here.
-
-## dataset download
-
-```bash
-evolve dataset download <name[@version]> [options]
-```
-
-Download the original corpus package. Owner only.
-
-<ParamField path="-o, --output-dir" type="dir">
-  Directory to save into. Defaults to the current directory.
-</ParamField>
-
-## dataset files
-
-```bash
-evolve dataset files <status|ls|cat> <name@version> <task> [path] [options]
-```
-
-A published task's files, exactly as the platform stored them and as every trial ran against them: `files ls` lists one folder of the task directory (default `/`: `instruction.md`, `task.toml`, `environment/`, `tests/`), `files cat` prints one file's bytes (`--range bytes=a-b` reads a slice), `files status` says whether the version keeps its package (`package_retained`). A version imported before packages were kept is refused with `task_package_not_retained`. The version must be named: `name@version`.
-
-```bash
-evolve dataset files ls my-swe@1.0 abs-1 /tests
-evolve dataset files cat my-swe@1.0 abs-1 /tests/test.sh
-```
-
-## dataset activate
-
-```bash
-evolve dataset activate <name> <version>
-```
-
-Make a `READY` version the dataset's active version.
-
-<GlobalOptions />

package/skills/evolve-evals/references/cli-reference/index.mdx

@@ -1,108 +0,0 @@
----
-title: "CLI reference"
-description: "Every verb of the evolve command, from its own help output."
----
-
-```bash
-evolve <command> [options]
-```
-
-The `evolve` command starts jobs and reads everything back. Every verb accepts `--json` for machine-readable output, and `-h` or `--help` prints its flags under short headings, with defaults, and one to three examples.
-
-## Verbs
-
-| Verb | What it does |
-| --- | --- |
-| [`run`](/cli-reference/run) | Start a job. Add `--watch` to follow it. The short form of `job start`. |
-| [`analyze`](/cli-reference/analyze) | Analyze a finished job's trial traces against a rubric. |
-| [`check`](/cli-reference/check) | Check task quality against a rubric. |
-| [`upload`](/cli-reference/upload) | Upload a job directory as a finished job. |
-
-## Command groups
-
-| Group | Commands |
-| --- | --- |
-| [`skills`](/cli-reference/skills) | `list`, `get`, `path`, `install` |
-| [`job`](/cli-reference/job) | `start`, `list`, `show`, `trials`, `tasks`, `compare`, `cancel`, `delete`, `stop`, `resume`, `retry`, `regrade`, `imports`, `import`, `download`, `grep` |
-| [`trial`](/cli-reference/trial) | `show`, `trace`, `download`, `retry`, `regrade`, `stop` |
-| [`analysis`](/cli-reference/analysis) | `list`, `show`, `trace`, `download` |
-| [`check`](/cli-reference/check) | `list`, `show`, `trace`, `download` |
-| [`dataset`](/cli-reference/dataset) | `list`, `show`, `check`, `publish`, `watch`, `download`, `activate` |
-| [`skill`](/cli-reference/skill) | `list`, `upload`, `show`, `delete` |
-| [`agent`](/cli-reference/agent) | `list`, `show`, `add`, `remove` |
-| [`auth`](/cli-reference/auth) | `status`, `org list`, `org show` |
-| [`secrets`](/cli-reference/secrets) | `set`, `list`, `delete` |
-
-The singular noun is canonical; `jobs`, `trials`, `analyses` and `datasets` are hidden aliases, as are `secret` for `secrets` and `ls` for `list`. `skills` is not the plural of `skill`: it is the group that serves the bundled skills to your coding agent. `agents` is not an alias: it is reserved for the managed-agents CLI and refuses, pointing at `evolve agent`.
-
-## Global options
-
-These work on every command.
-
-<ParamField path="--json">
-  Machine-readable JSON output. A refusal then prints one `{"error": {...}}` object on stdout with the server's `code`, `message`, `param`, `details`, `retryAfterSec` and `request_id` when present.
-</ParamField>
-
-<ParamField path="--api-key" type="key">
-  API key. Defaults to `$EVOLVE_API_KEY`.
-</ParamField>
-
-<ParamField path="--base-url" type="url">
-  API base URL. Defaults to the Evolve dashboard API.
-</ParamField>
-
-<ParamField path="-v, --version">
-  Print the CLI version.
-</ParamField>
-
-## Ids
-
-Every verb that takes an id (`job`, `trial`, `analysis`, `check` with a check id or a task check id, `skill`) also takes an unambiguous prefix of at least 8 characters. A prefix that matches nothing or more than one row is refused naming the candidates. A full id is never prefix-matched, `name:<skill-name>` is a name, and a job import id is taken as printed.
-
-A trial prefix is resolved against every trial of every job in scope, so on an account with many jobs the lookup can hit the rate limit; the full id, which `evolve job trials <job>` prints, always works.
-
-## Output
-
-Human tables on a TTY, tab-separated rows when piped, `--json` for the machine shape (NDJSON under `--watch`), and `-q` for ids only. `--no-trunc` disables cell truncation and `--no-headers` drops the header row from piped output. A money cell carries its state: `$0.06` is final, `at least $0.06` is still being written, `-` means not measured yet.
-
-`--columns` chooses and orders list columns, comma-separated; `--columns help` prints the keys. For `job list` they are `id`, `name`, `status`, `datasets`, `agents`, `trials`, `spent`, `started`; for `job trials` `task`, `agent`, `attempt`, `status`, `reward`, `spent`, `tokens`, `gpu`, `id`; for `analysis list` `id`, `status`, `task`, `job`, `trial`, `model`, `attempts`, `spent`, `created`, `finished`.
-
-## Listing flags
-
-Every `list` command, and `job trials`, `job tasks` and `job imports`, share one set of paging and formatting flags. Two take only the formatting flags, not `-l` or `--cursor`: `auth org list` and `secrets list`.
-
-<ParamField path="-l, --limit" type="n">
-  Page size.
-</ParamField>
-
-<ParamField path="--cursor" type="c">
-  Resume from a page cursor.
-</ParamField>
-
-<ParamField path="--columns" type="keys | all | help">
-  Choose and order columns, comma-separated. `help` lists the keys.
-</ParamField>
-
-<ParamField path="-q, --quiet">
-  Print only ids, one per line, for piping.
-</ParamField>
-
-<ParamField path="--no-trunc">
-  Full cell content instead of one-line truncation.
-</ParamField>
-
-<ParamField path="--no-headers">
-  Omit the header row in piped (TSV) output.
-</ParamField>
-
-## Scope
-
-`--scope` on `job list`, `analysis list`, `check list`, `session list`, `skill list` and `agent list` is `my`, what you created and the default; `shared`, your organizations' rows that teammates created; or `org`, everything in your organizations, your own rows included. `all` is refused. Every id printed under any scope resolves on its `show`, `trace` and `download`; an id outside your organizations answers the noun's not-found code, exactly like an id that does not exist. The one exception is a registered agent, addressed by name rather than id: `agent show <name>` opens your own registration of that name, else a teammate's in one of your organizations there — it is usable in a job's `agents[].name`, not on `show`.
-
-## Values
-
-A flag's value may begin with `-`, such as `-x '-*'` or a negative number. Only a token that spells another flag of the same command is refused, and the refusal shows the `--flag=value` form. A typo in `--stream`, `--status`, `--scope` or `run`'s `-e` is a usage error naming the legal values, before any request.
-
-## Exit codes
-
-`0` on success (with `--watch`: the job `COMPLETED`, a publish settled `READY`, every analysis completed, no task check errored). `1` on a runtime or API failure (with `--watch`: `FAILED` or `CANCELLED`, a failed analysis, an errored task check). `2` on a usage error, and on a quota refusal, which prints `Launch quota exceeded:` with the server's sentence. A rate limit prints one line with the wait and exits 1; the SDK's watch loops wait it out on their own.