@evolvingmachines/evolve 0.0.65 → 0.0.67

package/skills/evolve-evals/references/cli-reference/job.mdx

@@ -1,278 +0,0 @@
----
-title: "evolve job"
-description: "Start, follow, and derive jobs."
----
-
-import GlobalOptions from '/snippets/global-options.mdx';
-import ListFlags from '/snippets/list-flags.mdx';
-
-```bash
-evolve job <command> [options]
-```
-
-Every command below takes the [global options](/cli-reference#global-options).
-
-## job start
-
-```bash
-evolve job start [options]
-```
-
-Start a job; add `--watch` to follow it. Exactly the options of [`evolve run`](/cli-reference/run).
-
-## job list
-
-```bash
-evolve job list [options]
-```
-
-List jobs, newest first: yours, `--scope shared` for what teammates created, `--scope org` for everything in your organizations. A task quality check is a job too (Harbor runs `harbor check` as one job): `--kind all` lists jobs and checks in one list, each row marked by its kind, and `--kind check` lists checks alone.
-
-<ListFlags />
-
-<ParamField path="--search" type="text">
-  Free-text filter over job name and dataset names.
-</ParamField>
-
-<ParamField path="--scope" type="my | shared | org">
-  Visibility scope: `my`, what you created (the default); `shared`, your organizations' rows that teammates created; or `org`, every row in your organizations, your own included.
-</ParamField>
-
-<ParamField path="--kind" type="job | check | all">
-  Which rows to list: `job` (the default), `check` for task quality checks, or `all` for both merged newest first. With `check` or `all` a `KIND` column leads each row; a check row reads its id, status, dataset or archive, task check count, cost and accept time, and under `--columns` its `name` and its checker's model and effort (`agents`).
-</ParamField>
-
-```bash
-evolve job list --limit 20 -q
-evolve job list --kind all
-```
-
-## job show
-
-```bash
-evolve job show <id> [id...]
-```
-
-Show one or more jobs in full. It ends with a pass@k block, one line per arm and dataset that has numbers; `--json` carries the raw `stats.evals[].pass_at_k`. An `analyze` row holds the embedded policy and an `analysis` row the tally with the analyzer's spend. An uploaded job prints its provider as `ported` and its spend as `reported $X.XX (N/M trials reporting)`. The `visibility` row is `PRIVATE` or `LINK` — whether the job's share link is on ([job shares](#job-shares) prints the link and the addresses).
-
-## job trials
-
-```bash
-evolve job trials <id> [options]
-```
-
-List a job's trials.
-
-<ListFlags />
-
-<ParamField path="--status" type="s1,s2,...">
-  Filter by trial status, for example `INFRASTRUCTURE_ERROR`.
-</ParamField>
-
-<ParamField path="--dataset" type="name">
-  Filter to one dataset's trials.
-</ParamField>
-
-```bash
-evolve job trials 3e1f9a2c-… --status INFRASTRUCTURE_ERROR,SCORING_ERROR
-```
-
-## job tasks
-
-```bash
-evolve job tasks <id> [options]
-```
-
-Per-task rollup of a job: one row per task.
-
-<ListFlags />
-
-## job compare
-
-```bash
-evolve job compare <id> <id> [...]
-```
-
-Compare two to ten jobs side by side.
-
-## job cancel
-
-```bash
-evolve job cancel <id>
-```
-
-Request cancellation of a job.
-
-## job delete
-
-```bash
-evolve job delete <id> [options]
-```
-
-Permanently delete a job you created: its trials, traces, analyses and stored files. It names the job, then asks. Another member of the organization is refused with `org_forbidden`; a live job, a running analysis wave or a live regrade refuses as on [jobs](/core-concepts/jobs#stop-cancel-delete).
-
-<ParamField path="-y, --yes">
-  Delete without a confirmation prompt. On a non-interactive stdin the command refuses without it, rather than guessing. With `--json` the receipt is `{"job_id", "trials_deleted", "analyses_deleted"}`.
-</ParamField>
-
-## job share
-
-```bash
-evolve job share <id> --link
-evolve job share <id> --email <address> [--email <address>...]
-```
-
-Share a job you created. `--link` turns on the job's unlisted link and prints it; running it again prints the same link. `--email` shares the run with an address: that person gets an email with a link to the run and reads it after signing in with the same address (a person without an account gets a sign-up link in the same email; an address already on the waitlist or holding an invite gets the share email without a sign-up link and uses that invite instead). Both flags may ride one command. Someone the run is shared with can read it — the job, its trials, traces and files — and finds it under `evolve job list --scope shared`; they cannot cancel, retry, regrade, analyze or delete it. Only the job's creator may share it; another member of the organization is refused with `org_forbidden`.
-
-<ParamField path="--link">
-  Enable the job's unlisted link and print it. Anyone holding the link reads the run without signing in.
-</ParamField>
-
-<ParamField path="--email" type="address">
-  Share with this address; repeatable. At most 50 addresses per job.
-</ParamField>
-
-## job unshare
-
-```bash
-evolve job unshare <id> --link
-evolve job unshare <id> --email <address>
-```
-
-Take a share back. `--link` turns the link off at once — the old link stops working, and a later `job share --link` prints a new one. `--email` removes that address's share. Both are safe to repeat.
-
-## job shares
-
-```bash
-evolve job shares <id>
-```
-
-Print who the job is shared with: its visibility, the link while it is on, and every address with who shared it and when. `--json` prints the same as `{"visibility", "link": {"enabled", "url"}, "emails": [{"email", "shared_by", "created_at"}]}`.
-
-## job stop
-
-```bash
-evolve job stop <id> --dataset <name>
-```
-
-Stop one dataset's live trials without cancelling the job. Trials already settled are reported as already terminal, and a dataset the job never spanned is refused. If the command fails part way (a rate limit, say), the report prints first, marked `PARTIAL` with the trials no answer came back for (`partial: true` and `unreported` under `--json`), then the error, exit 1; re-running the same command finishes the rest.
-
-<ParamField path="--dataset" type="name" required>
-  The dataset whose live trials to stop.
-</ParamField>
-
-## job resume
-
-```bash
-evolve job resume <id> [options]
-```
-
-A new linked job over a terminal job's failed or stopped trials.
-
-<ParamField path="-f, --filter-error-type" type="type">
-  Failure types to resume, matched on the trial's exception type. Repeatable. Defaults to the standard failure set plus stopped trials.
-</ParamField>
-
-```bash
-evolve job resume 3e1f9a2c-… -f InfrastructureError
-```
-
-## job retry
-
-```bash
-evolve job retry <id> [options]
-```
-
-A new linked job re-running selected trials: all of them, the failed ones only, or named ids.
-
-<ParamField path="--failed-only">
-  Only retry failed trials: `SCORING_ERROR`, `INFRASTRUCTURE_ERROR`, `BUDGET`, `INDETERMINATE`. Stopped and scored trials are not failures.
-</ParamField>
-
-<ParamField path="-t, --trial" type="trial-id">
-  Retry exactly this trial. Repeatable, all or nothing: each must be settled, and the job may still be running. Not combinable with `--failed-only`.
-</ParamField>
-
-## job regrade
-
-```bash
-evolve job regrade <id> [options]
-```
-
-Verifier-only re-run of a terminal job. The result is a job.
-
-<ParamField path="--status" type="s1,s2,...">
-  Only regrade source trials in these statuses.
-</ParamField>
-
-<ParamField path="--task" type="name">
-  Only regrade source trials of this task.
-</ParamField>
-
-## job imports
-
-```bash
-evolve job imports [options]
-```
-
-List your job imports (uploads), newest first. This is how an import id is found again.
-
-<ListFlags />
-
-<ParamField path="--status" type="QUEUED | RUNNING | COMPLETED | FAILED">
-  Filter by import status.
-</ParamField>
-
-## job import
-
-```bash
-evolve job import <import-id> [options]
-```
-
-Show one job import, an upload's record, with a `skipped` count and one line per skipped trial when the ingest left any out.
-
-<ParamField path="--watch">
-  Poll until the import settles: `COMPLETED`, and the job is printed, or `FAILED`.
-</ParamField>
-
-## job download
-
-```bash
-evolve job download <id> [options]
-```
-
-Download the job's results, unpacked as the standard job-directory tree plus `evolve.json` records.
-
-<ParamField path="-o, --output-dir" type="dir">
-  Directory to unpack into. Defaults to the current directory. The tree lands in `<dir>/job-<id>/`.
-</ParamField>
-
-<ParamField path="--overwrite">
-  Replace an existing `<dir>/job-<id>/`.
-</ParamField>
-
-## job grep
-
-```bash
-evolve job grep <id> <pattern> [options]
-```
-
-Search every trial's parsed trace in one server-side pass and print one line per matching trial with its sampled events. The pattern grammar and the page semantics are on [`jobs().grep()`](/sdk-reference/jobs#grep).
-
-<ParamField path="--type" type="event-type">
-  Only search events of exactly this type.
-</ParamField>
-
-<ParamField path="-l, --limit" type="n" default="50">
-  Per-trial match groups per page. Max 200.
-</ParamField>
-
-<ParamField path="--cursor" type="c">
-  Resume after this trial id, the previous page's `nextCursor`.
-</ParamField>
-
-```bash
-evolve job grep 3e1f9a2c-… 'permission denied'
-```
-
-<GlobalOptions />

package/skills/evolve-evals/references/cli-reference/run.mdx

@@ -1,204 +0,0 @@
----
-title: "evolve run"
-description: "Start a job, and optionally follow it to the end."
----
-
-import GlobalOptions from '/snippets/global-options.mdx';
-
-```bash
-evolve run [options]
-```
-
-Start a job. `evolve run` is the short form of `evolve job start`, and both take the same options.
-
-```bash
-evolve run \
-  -d terminal-bench-4@4.0 \
-  -a codex \
-  -m gpt-6-astra \
-  -k 2 \
-  -l 5 \
-  --watch
-```
-
-To run in a team's organization, add `--org acme`; the first line of the output then reads `org  acme`.
-
-## What to run
-
-<ParamField path="-d, --dataset" type="name[@version]">
-  Dataset to run. Repeatable. A bare name means the active version.
-</ParamField>
-
-<ParamField path="-i, --include-task-name" type="glob">
-  Include filter over task names, applied to every dataset. Repeatable.
-</ParamField>
-
-<ParamField path="-x, --exclude-task-name" type="glob">
-  Exclude filter over task names, applied to every dataset. Repeatable.
-</ParamField>
-
-<ParamField path="-l, --n-tasks" type="n">
-  Cap the task count of each dataset after the filters.
-</ParamField>
-
-<ParamField path="-a, --agent" type="name[@version]">
-  Agent, built-in or registered.
-</ParamField>
-
-<ParamField path="-m, --model" type="name">
-  Model. Repeatable; each model is one arm.
-</ParamField>
-
-<ParamField path="-k, --n-attempts" type="n" default="1">
-  Attempts per task and arm.
-</ParamField>
-
-<ParamField path="-n, --n-concurrent" type="n" default="4">
-  Parallel trials, 1 to 150.
-</ParamField>
-
-<ParamField path="-e, --env" type="provider" default="daytona">
-  Sandbox provider: `e2b`, `daytona` or `modal`.
-</ParamField>
-
-<ParamField path="--job-name" type="name">
-  User-facing label. Generated by the server when omitted.
-</ParamField>
-
-<ParamField path="--org" type="name">
-  Organization the job lands in, so your teammates see it. Omitted, the default set with `evolve auth org use`, else your personal organization; `--org personal` picks your personal organization for this one job over that default. The first line of the output names it, `org  acme` or `org  personal`; with `--json` it is the job's `org` field.
-</ParamField>
-
-## The arm
-
-Each of these applies to every arm of the job.
-
-<ParamField path="--effort" type="value">
-  Reasoning effort for every arm, applied verbatim. The values each harness accepts are on the [models](/core-concepts/models#reasoning-effort) page and in the [capability document](/sdk-reference/meta). An agent that cannot honor the value is refused, never silently skipped.
-</ParamField>
-
-<ParamField path="--preset" type="name">
-  Named settings preset for every arm: `no-internet` turns the vendor's server-side web tools off, `pinned-context` fixes the context window at 200000 tokens. Only `claude` and `codex` take one; a preset wins over `--ak config` where they disagree. An unknown name is refused with `invalid_input`, a preset the agent cannot guarantee with `agent_preset_unsupported`.
-</ParamField>
-
-<ParamField path="--agent-kwarg, --ak" type="key=value">
-  Agent kwarg for every arm, as `key=value`, repeatable. The delivered key is `config`: `--ak 'config=<path|inline JSON>'` becomes the harness's native settings file, your document as the base and the platform's routing on top; a path is read locally, as JSON or as TOML for a Codex config. Only `claude` and `codex` take a config. A key other than `config` is refused with `agent_kwarg_unsupported`, a config on another harness with `agent_config_unsupported`, and a key touching billing, base URL, routing or env with `agent_config_key_refused`.
-</ParamField>
-
-<ParamField path="--skills, --skill" type="ref | path">
-  Skill for every agent arm. Repeatable. Accepts `skills.sh/<owner>/<repo>[/<skill>]`, `org/repo[@ref]`, an https git URL, `upload:<id>`, `name:<skill-name>` (your moving name pointer, resolved server-side), or a local folder, which is uploaded first and then referenced; `--print-config` still shows the path you typed.
-</ParamField>
-
-<ParamField path="--ve, --verifier-env" type="KEY=VALUE">
-  Environment for every verifier run. Repeatable. The server honors exactly `REWARDKIT_JUDGE` and `REWARDKIT_MODEL`, rewardkit's judge override, and refuses any other key.
-</ParamField>
-
-<ParamField path="--secret" type="NAME[@LABEL][=ENVNAME]">
-  Attach one of your stored env secrets to every agent run. Repeatable. `NAME` is the stored secret's name.
-
-  `@LABEL` picks a labeled row; omitted, the `default` row or the only row, and several labels with no `default` is refused as ambiguous. `=ENVNAME` renames the env var inside the sandbox. References only: the value never rides the command line or the wire.
-</ParamField>
-
-<ParamField path="--secret-inline" type="NAME[@LABEL]:DELIVERY=VALUE">
-  Save `VALUE` into your vault as an env secret and attach it to this job in one step. Repeatable. `DELIVERY` is `brokered` or `direct` and sits before `=`, so everything after the first `=` is the value, byte for byte. `@LABEL` defaults to `default`.
-
-  Restating an existing secret exactly attaches it, so re-running the same command converges; a different value or delivery is refused as `secret_exists`. The job stores only the reference.
-</ParamField>
-
-## Spend and retries
-
-<ParamField path="--max-trial-spend" type="usd">
-  Model-spend cap for each trial. Defaults to the server's, $200.
-</ParamField>
-
-<ParamField path="-r, --max-retries" type="n">
-  Max automatic retries per trial on infrastructure errors. Defaults to 2; `0` turns retries off. Each attempt carries the full trial cap. Over a `-c` file it overrides only its own field of `retry`.
-</ParamField>
-
-<ParamField path="--retry-include" type="exception">
-  Exception types to retry on. Repeatable. The names are `AgentAuthenticationError` (a gateway credential refusal), `ModelNotFoundError` (a model that never served) and `InfrastructureError` (everything else). Defaults to everything `--retry-exclude` admits.
-</ParamField>
-
-<ParamField path="--retry-exclude" type="exception">
-  Exception types not to retry on. Repeatable, and it wins over `--retry-include`. Defaults to the platform's non-retryable set; an explicit `null` in a `-c` file's `retry` turns exclusions off entirely.
-</ParamField>
-
-## Embedded analysis
-
-<ParamField path="--analyze">
-  Analyze each trial's trace server-side as it settles. Cancelled trials are skipped. Bare, it uses the platform's analyze model, rubric and prompt.
-</ParamField>
-
-<ParamField path="--analyze-model" type="name">
-  Model the analyzer runs. Must be on the `claude` roster. Implies `--analyze`.
-</ParamField>
-
-<ParamField path="--analyze-rubric" type="path">
-  Rubric file for the analyzer: TOML, YAML or JSON in the `{criteria}` shape. Implies `--analyze`.
-</ParamField>
-
-<ParamField path="--analyze-prompt" type="path">
-  Prompt file for the analyzer. Replaces the built-in prompt. Implies `--analyze`.
-</ParamField>
-
-<ParamField path="--analyze-provider" type="provider">
-  Sandbox provider the analyzer runs on. Defaults to the platform's analysis default. Implies `--analyze`.
-</ParamField>
-
-<ParamField path="--analyze-effort" type="value">
-  Reasoning effort the analyzer runs at. Defaults to the per-model default: `high` on `openrouter/deepseek/deepseek-v4.1-flash`, `max` on `glm-5.3-flash`. Implies `--analyze`.
-</ParamField>
-
-## Sandbox logs
-
-<ParamField path="--system-log">
-  Record the box's own system log beside the streams every run keeps (`agent`, `verifier`). Read it back with `evolve trial logs --stream system`. Off by default.
-</ParamField>
-
-## Timeouts
-
-<ParamField path="--timeout-multiplier" type="x" default="1.0">
-  Multiplier for task timeouts, any finite number above 0; below 1 shrinks. It multiplies each task's declared timeouts for this job only; the task is never rewritten. A product past 2,147,483,647 ms is refused at create. Resumed and retried jobs inherit the multipliers.
-</ParamField>
-
-<ParamField path="--agent-timeout-multiplier" type="x">
-  Multiplier for the agent execution timeout. Overrides `--timeout-multiplier`.
-</ParamField>
-
-<ParamField path="--verifier-timeout-multiplier" type="x">
-  Multiplier for the verifier timeout. Overrides `--timeout-multiplier`.
-</ParamField>
-
-<ParamField path="--agent-setup-timeout-multiplier" type="x">
-  Multiplier for the agent setup timeout. Overrides `--timeout-multiplier`.
-</ParamField>
-
-<ParamField path="--environment-build-timeout-multiplier" type="x">
-  Multiplier for the environment build timeout. Overrides `--timeout-multiplier`.
-</ParamField>
-
-## Config file and output
-
-<ParamField path="-c, --config" type="path">
-  Job config file, YAML or JSON with the same field names `jobs().start()` takes. Explicit flags override its fields one at a time, inside the `retry` and `analyze` objects too. The file may be partial: `-d` and `-a`/`-m` can supply what it omits, but a selector inside it needs `name` and an arm needs `name` and `model_name`.
-
-  YAML is read with the 1.1 schema (`yes` and `on` are booleans, `012` is octal), and a duplicate key, a second document or an unknown tag refuse with a line number. The body is checked before any request: an unknown key is refused by name, a string must be a string (quote `version: "1.10"`), `n_attempts` and `n_tasks` are at least 1, `n_concurrent_trials` is 1 to 150, `job_name` is at most 200 characters. A bare date or `.inf` is refused rather than rewritten.
-</ParamField>
-
-<ParamField path="--print-config">
-  Print the resolved job body as JSON and exit without running.
-</ParamField>
-
-<ParamField path="--watch">
-  Stream events until the job finishes.
-</ParamField>
-
-<ParamField path="-q, --quiet">
-  With `--watch`: suppress the event log and print the final block only.
-</ParamField>
-
-<ParamField path="-y, --yes">
-  Accepted for compatibility. Hosted runs have no prompt to confirm.
-</ParamField>
-
-<GlobalOptions />

package/skills/evolve-evals/references/cli-reference/secrets.mdx

@@ -1,95 +0,0 @@
----
-title: "evolve secrets"
-description: "Store and manage env secrets. Values are write-only; reads return metadata."
----
-
-import GlobalOptions from '/snippets/global-options.mdx';
-
-```bash
-evolve secrets <command> [options]
-```
-
-`secrets` is the one plural noun; `secret` is a hidden alias. It is the same vault the dashboard's Secrets page writes, reached with your API key.
-
-The same three operations exist on the SDK's `Evolve.managedSecrets()`.
-
-A read-only API key may `list` but not `set` or `delete` (`read_only_key`). LLM provider keys cannot be stored here; they decide who pays and stay on the dashboard.
-
-## secrets set
-
-```bash
-evolve secrets set <NAME> [options]
-```
-
-Store an env secret, or re-shape its delivery or scoping by restating the same value. A different value under an existing name and label is refused (`secret_exists`, 409): delete first, or use another label. Never a silent overwrite. `NAME` matches `[A-Z_][A-Z0-9_]{0,127}`, uppercased for you; the `EVOLVE_` prefix is reserved.
-
-<ParamField path="--value" type="value">
-  The secret value, at most 190 bytes of UTF-8. Omit it to pipe the value on stdin, which keeps it out of shell history; one trailing newline is stripped from the pipe. A terminal with neither channel is a usage error, never a hang.
-</ParamField>
-
-<ParamField path="--label" type="label" default="default">
-  Labeled-row identity, at most 80 characters of `[A-Za-z0-9._-]`. Several values of one name live side by side.
-</ParamField>
-
-<ParamField path="--delivery" type="mode" required>
-  `brokered`: the value never enters a sandbox, and needs the `--allowed-*` scoping. `direct`: the raw value in the sandbox env; scoping refused.
-</ParamField>
-
-<ParamField path="--allowed-host" type="host">
-  Brokered scoping: a hostname, or a wildcard like `*.example.com`. Repeatable.
-</ParamField>
-
-<ParamField path="--allowed-path-prefix" type="/prefix">
-  Brokered scoping: an allowed URL path prefix. Repeatable.
-</ParamField>
-
-<ParamField path="--allowed-method" type="METHOD">
-  Brokered scoping: an allowed HTTP method. Repeatable.
-</ParamField>
-
-```bash
-printf %s "$GITHUB_TOKEN" | evolve secrets set GITHUB_TOKEN --delivery brokered \
-  --allowed-host api.github.com --allowed-path-prefix / --allowed-method GET
-```
-
-## secrets list
-
-```bash
-evolve secrets list [options]
-```
-
-List your env secrets. Metadata only; values never leave the server.
-
-<ParamField path="--columns" type="keys | all | help">
-  Choose and order columns, comma-separated. `help` lists the keys.
-</ParamField>
-
-<ParamField path="-q, --quiet">
-  Print only `name[:label]`, one per line.
-</ParamField>
-
-<ParamField path="--no-trunc">
-  Full cell content instead of one-line truncation.
-</ParamField>
-
-<ParamField path="--no-headers">
-  Omit the header row in piped (TSV) output.
-</ParamField>
-
-## secrets delete
-
-```bash
-evolve secrets delete <NAME> [options]
-```
-
-Delete an env secret by name. Revokes every runtime grant riding the row. Rotate a value by `delete` then `set`, or store the new value under another label.
-
-<ParamField path="--label" type="label">
-  The labeled row to delete. Omitted, it resolves as an attachment does: the `default` row, else the only row, else refused with `secret_ambiguous` naming every label.
-</ParamField>
-
-```bash
-evolve secrets delete GITHUB_TOKEN --label staging
-```
-
-<GlobalOptions />

package/skills/evolve-evals/references/cli-reference/skill.mdx

@@ -1,61 +0,0 @@
----
-title: "evolve skill"
-description: "Upload and manage platform-stored skills."
----
-
-import GlobalOptions from '/snippets/global-options.mdx';
-import ListFlags from '/snippets/list-flags.mdx';
-
-```bash
-evolve skill <command> [options]
-```
-
-## skill list
-
-```bash
-evolve skill list [options]
-```
-
-List uploaded skills, newest first.
-
-<ParamField path="--scope" type="my | shared | org">
-  Visibility scope: `my`, your own uploads (the default); `shared`, your organizations' other members'; or `org`, every upload in your organizations.
-</ParamField>
-
-<ListFlags />
-
-## skill upload
-
-```bash
-evolve skill upload <dir> [options]
-```
-
-Upload a skill folder. Its name becomes your moving name pointer.
-
-The record belongs to you and to an organization: its members may reference it from their own jobs, and only you can delete it.
-
-<ParamField path="--org" type="name">
-  Organization the record belongs to. Default: the organization from `evolve auth org use`, else your personal one.
-</ParamField>
-
-## skill show
-
-```bash
-evolve skill show <id | name:<skill-name>>
-```
-
-Show one uploaded skill: its metadata and its `SKILL.md`.
-
-```bash
-evolve skill show name:my-skill
-```
-
-## skill delete
-
-```bash
-evolve skill delete <id>
-```
-
-Delete an uploaded skill record. Past jobs keep their locks. Refused (`skill_in_use`, 409) while a running job references it.
-
-<GlobalOptions />