@evolvingmachines/evolve 0.0.67 → 0.0.69

package/skills/evolve-evals/SKILL.md

@@ -39,7 +39,7 @@ Hosted requests use `EVOLVE_API_KEY`. Start with Installation if authentication
 | [Datasets](references/core-concepts/datasets.md) | Browse ready-to-run tasks or publish your own versioned collection. |
 | [Jobs](references/core-concepts/jobs.md) | Run a task set across agents and models. Follow each attempt to its result. |
 | [Agents](references/core-concepts/agents.md) | Choose a built-in coding-agent harness or register your own. |
-| [Models](references/core-concepts/models.md) | Select models, compare them, and control metered spend. |
+| [Models](references/core-concepts/models.md) | Exact model names, per-model reasoning options, and Evolve defaults. |
 | [Sandboxes](references/core-concepts/sandboxes.md) | Choose a cloud provider and understand how task requirements affect placement. |
 | [Agent skills](references/core-concepts/skills.md) | Give evaluated agents reusable instructions and supporting files. |
 | [Secrets](references/core-concepts/secrets.md) | Store a credential once and attach it to evaluation jobs by name. |

package/skills/evolve-evals/references/cli-reference/agent.md

@@ -58,7 +58,7 @@ evolve agent show my-agent
 ```bash
 evolve run \
   -d harbor-examples@1.0 -i hello-world \
-  -a my-agent -m gpt-5.6-luna \
+  -a my-agent -m gpt-6-luna \
   --max-trial-spend 0.30 --max-retries 0 \
   --watch
 ```

package/skills/evolve-evals/references/cli-reference/analyze.md

@@ -15,10 +15,11 @@ Each selected trial gets its own analysis run. Read those runs with [`evolve ana
 
 ```bash
 evolve analyze --show-defaults
+evolve analyze --show-defaults -a codex
 evolve analyze --show-defaults --json
 ```
 
-This returns the current model, reasoning effort, sandbox provider, prompt, and rubric. Defaults mode takes no job ID or other analysis options.
+This returns the configured agent, model, effort, sandbox provider, prompt, and rubric. With `-a`, it returns that agent's defaults. See the [model reference](/core-concepts/models#model-and-effort-reference) for reasoning options. Defaults mode takes no job ID and no other analysis options.
 
 ## Select trials
 
@@ -51,6 +52,7 @@ Cancelled trials are excluded. `--passing` and `--failing` cannot be combined. T
 
 ```bash
 evolve analyze "$JOB_ID" \
+  --agent codex --model gpt-6-sol \
   --rubric rubric.toml \
   --prompt prompt.txt \
   --watch
@@ -58,12 +60,13 @@ evolve analyze "$JOB_ID" \
 
 | Option | Meaning |
 | --- | --- |
-| `-m`, `--model <name>` | Analyzer model. |
-| `--effort <value>` | Reasoning effort. |
+| `-a`, `--agent <name>` | Agent the analyzer runs on. Default `claude`. |
+| `-m`, `--model <name>` | Analyzer model, from that agent's models. Default: the agent's default model. |
+| `--effort <value>` | Reasoning effort for the selected model. See the [model reference](/core-concepts/models#model-and-effort-reference). |
 | `-r`, `--rubric <path>` | TOML, YAML, or JSON rubric. |
 | `-p`, `--prompt <path>` | Text file replacing the default prompt. |
 | `-e`, `--env <provider>` | Sandbox provider for analysis. |
-| `--show-defaults` | Print the default policy and exit. |
+| `--show-defaults` | Print the default policy and exit. `-a` picks the agent. |
 
 A rubric contains a `criteria` list. Every criterion needs `name`, `description`, and `guidance`.
 

package/skills/evolve-evals/references/cli-reference/check.md

@@ -44,16 +44,17 @@ evolve check --show-defaults
 | Option | Meaning |
 | --- | --- |
 | `--name <name>` | Check label. Defaults to the acceptance timestamp. |
-| `-m`, `--model <name>` | Checker model. |
-| `--effort <value>` | Reasoning effort. |
+| `-a`, `--agent <name>` | Agent the checker runs on. Default `claude`. |
+| `-m`, `--model <name>` | Checker model, from that agent's models. Default: the agent's default model. |
+| `--effort <value>` | Reasoning effort for the selected model. See the [model reference](/core-concepts/models#model-and-effort-reference). |
 | `-r`, `--rubric <path>` | TOML, YAML, or JSON rubric. |
 | `-p`, `--prompt <path>` | Replacement prompt text file. |
 | `-e`, `--env <provider>` | Checker sandbox provider. |
-| `--show-defaults` | Print the prompt, rubric, model, effort, and provider, then exit. |
+| `--show-defaults` | Print the agent, model, effort, provider, prompt, and rubric, then exit. `-a` picks the agent. |
 | `--watch` | Wait until every task check settles. |
 | `-q`, `--quiet` | With watch, suppress intermediate progress. |
 
-`--show-defaults` accepts only [global options](/cli-reference/index#global-options). It cannot be combined with a path, dataset, or checker options.
+`--show-defaults` accepts only `-a` and the [global options](/cli-reference/index#global-options). It cannot be combined with a path, dataset, or other checker options.
 
 For rubric format and interpretation, see [Check](/core-concepts/check).
 

package/skills/evolve-evals/references/cli-reference/index.md

@@ -109,7 +109,7 @@ A trial prefix may require reading the trials of every job in scope. On large ac
 Repeat a repeatable flag once per value. Do not combine short flags.
 
 ```bash
-evolve run -d harbor-examples@1.0 -a codex -m gpt-5.6-luna \
+evolve run -d harbor-examples@1.0 -a codex -m gpt-6-luna \
   -i 'hello-*' -x '*-slow' --max-trial-spend 0.30 --max-retries 0
 ```
 

package/skills/evolve-evals/references/cli-reference/run.md

@@ -9,7 +9,7 @@ Start a hosted job. `evolve job start` accepts exactly the same options.
 evolve run \
   -d harbor-examples@1.0 \
   -i hello-world \
-  -a codex -m gpt-5.6-luna \
+  -a codex -m gpt-6-luna \
   --max-trial-spend 0.30 \
   --max-retries 0 \
   --watch
@@ -51,7 +51,7 @@ datasets:
     task_names: [hello-world]
 agents:
   - name: codex
-    model_name: gpt-5.6-luna
+    model_name: gpt-6-luna
 n_attempts: 1
 n_concurrent_trials: 4
 max_trial_spend_usd: 0.30
@@ -102,7 +102,7 @@ These options apply to every agent combination.
 
 | Option | Meaning |
 | --- | --- |
-| `--effort <value>` | Reasoning effort supported by the selected harness and model. |
+| `--effort <value>` | Reasoning effort for the selected harness and model. See the [model reference](/core-concepts/models#model-and-effort-reference). |
 | `--preset <name>` | Supported settings preset: `no-internet` or `pinned-context`. |
 | `--ak <key=value>`, `--agent-kwarg <key=value>` | Agent setting. Repeatable. `config` accepts a local JSON/TOML file or inline JSON object. |
 | `--skill <ref>`, `--skills <ref>` | Skill reference or local directory. Repeatable. |
@@ -114,7 +114,7 @@ evolve run -c job.yaml --skill ./my-skill
 
 Skills accept `skills.sh/owner/repo[/skill]`, `org/repo[@ref]`, a Git URL, `upload:<id>`, or `name:<skill-name>`. Local directories upload before the job starts.
 
-Unsupported effort, presets, or settings are refused. See [Agents](/core-concepts/agents) and [Skills](/core-concepts/skills).
+Unsupported effort, presets, or native configuration settings are refused. See [Agents](/core-concepts/agents) and [Skills](/core-concepts/skills).
 
 ### Secrets and verifier environment
 
@@ -148,7 +148,8 @@ See [Jobs](/core-concepts/jobs) for the retry and spend rules.
 | Option | Meaning |
 | --- | --- |
 | `--analyze` | Enable analysis with the platform defaults. |
-| `--analyze-model <name>` | Analyzer model. |
+| `--analyze-agent <name>` | Agent the analyzer runs on. Default `claude`. |
+| `--analyze-model <name>` | Analyzer model, from that agent's models. |
 | `--analyze-rubric <path>` | TOML, YAML, or JSON rubric. |
 | `--analyze-prompt <path>` | Replacement prompt file. |
 | `--analyze-provider <provider>` | Analyzer sandbox provider. |

package/skills/evolve-evals/references/core-concepts/agents.md

@@ -8,7 +8,7 @@ An evaluation arm pairs an **agent harness** with a **model**. The harness reads
 ```text
 One arm
 ├── Harness: -a codex
-└── Model: -m gpt-5.6-luna
+└── Model: -m gpt-6-luna
 ```
 
 ## Built-in harnesses
@@ -17,15 +17,25 @@ One arm
 | --- | --- |
 | `claude` | Claude Code |
 | `codex` | Codex |
-| `gemini` | Gemini CLI |
 | `qwen` | Qwen Code |
 | `kimi` | Kimi Code |
 | `opencode` | OpenCode |
 | `droid` | Droid |
+| `pi` | Pi |
+| `prime-agent` | Prime Agent |
+| `dsh` | DeepSeek Harness |
+| `zcode` | Z Code |
+| `antigravity` | Antigravity CLI |
+
+Use the [model and effort reference](/core-concepts/models#model-and-effort-reference) to copy the exact strings for each harness. Model names are not interchangeable across harnesses.
+
+**Note:**
+
+`gemini` is retired. Select `antigravity` for new runs; historical Gemini CLI records remain readable.
 
 ```bash
 evolve run -d harbor-examples@1.0 -i hello-world \
-  -a codex -m gpt-5.6-luna \
+  -a codex -m gpt-6-luna \
   --max-trial-spend 1 --max-retries 0 --watch
 ```
 
@@ -35,14 +45,18 @@ Use `-a name@version` to pin a harness release. Without a version, Evolve resolv
 
 | Option | Purpose |
 | --- | --- |
-| `--effort <value>` | Set supported reasoning effort |
+| `--effort <value>` | Set effort on every arm. See the [model reference](/core-concepts/models#model-and-effort-reference) |
 | `--preset no-internet` | Disable vendor server-side web tools |
 | `--preset pinned-context` | Set the supported context window to 200,000 tokens |
 | `--ak config=<path-or-JSON>` | Supply a native harness configuration |
 
+An effort or configuration the harness cannot honor is refused when the job is created.
+
 Presets and native config are supported on Claude Code and Codex. The platform applies routing settings over your config, and preset settings take precedence where they overlap. Routing, billing, credential, and environment overrides are rejected.
 
-The live [capability document](/sdk-reference/meta) lists each harness's supported options. Unsupported effort or configuration is rejected when the job is created.
+The [model table](/core-concepts/models#model-and-effort-reference) shows defaults and reasoning options for each model. Every arm needs an explicit model.
+
+The live [capability document](/sdk-reference/meta) lists accepted configuration options.
 
 **Note:**
 

package/skills/evolve-evals/references/core-concepts/analyze.md

@@ -41,7 +41,7 @@ Repeat `-t` with full trial IDs to select specific trials; selection happens bef
 ```bash
 evolve run \
   -d harbor-examples@1.0 -i hello-world \
-  -a codex -m gpt-5.6-luna \
+  -a codex -m gpt-6-luna \
   --max-trial-spend 1 -r 0 --analyze --watch
 ```
 
@@ -69,12 +69,19 @@ reviewed = await jobs().watch_analysis(finished.id)
 
 ## Rubrics and prompts
 
-Read the current defaults before customizing:
+Read the current rubric, prompt, model, effort, and provider before customizing:
 
-```bash
+```bash Default analyzer
 evolve analyze --show-defaults
 ```
 
+```bash Choose an analyzer
+evolve analyze --show-defaults -a codex
+evolve analyze "$JOB_ID" -a codex --watch
+```
+
+The default analyzer is `claude`. Use `-a` to choose another built-in harness. Omit `-m` to use its default model for analyses, or choose one of its [models](/core-concepts/models#analysis-and-check-models).
+
 The default rubric covers these questions:
 
 | Criterion | Question |
@@ -104,7 +111,7 @@ evolve analyze "$JOB_ID" -r rubric.toml -p prompt.txt --watch
 
 An analyzer prompt can use `{trial_path}`, `{task_section}`, and `{criteria_guidance}`. Evolve appends the required output format.
 
-Use `-m`, `--effort`, `-e`, and `-n` for model, effort, provider, and concurrency. On job creation, use the corresponding `--analyze-*` flags. See the [full reference](/cli-reference/analyze).
+Use `--effort`, `-e`, and `-n` to set reasoning effort, sandbox provider, and concurrency. On job creation, use the corresponding `--analyze-*` flags. See the [full reference](/cli-reference/analyze).
 
 ## The result
 
@@ -117,7 +124,34 @@ Each criterion has an outcome, explanation, and evidence.
 | `not_applicable` | The criterion has no subject in this trial. |
 | `unknown` | The available record cannot decide it. |
 
-The result is this per-criterion JSON and nothing else: Evolve derives no verdict from it. The dashboard shows a summary chip it computes from the outcomes. Under a custom rubric the chip is `flagged` when any criterion fails, `unclear` when none fails but at least one is unknown, and `clean` otherwise. Under a rubric with the default criterion names the chip reads only `score_is_earned`, `score_is_correct`, `task_was_fair`, `report_is_truthful` and `environment_worked`: a fail on the first four is `flagged`, else a failed `environment_worked` is `env_fault`, else an unknown on any of the five or a not applicable score criterion is `unclear`, else `clean`; the other two criteria are facts about the trial and never move the chip. Read the criteria and compute what you need from them. A missing or extra criterion makes the analysis result invalid. A run without a valid result can retry once.
+The API returns the findings without an overall verdict. Use the criterion outcomes and evidence to decide what to do next.
+
+The dashboard computes a short summary label from those outcomes. Its rules depend on whether the result has exactly the default criterion names.
+
+### Default rubric: read the dashboard summary
+
+The two score criteria are `score_is_earned` and `score_is_correct`. Apply the first matching rule:
+
+| Rule | Summary label |
+| --- | --- |
+| A score criterion, `task_was_fair`, or `report_is_truthful` fails. | **Flagged** |
+| `environment_worked` fails. | **Env fault** |
+| Any of these five criteria is unknown, or a score criterion is not applicable. | **Unclear** |
+| None of the above. | **Clean** |
+
+`ended_by_its_own_decision` and `worked_as_for_a_real_user` describe the trial but do not change this label. Read their findings even when the summary is **Clean**.
+
+### Other rubrics: read the dashboard summary
+
+Apply the first matching rule across all criteria:
+
+| Rule | Summary label |
+| --- | --- |
+| Any criterion fails. | **Flagged** |
+| Any criterion is unknown. | **Unclear** |
+| Neither of the above. | **Clean** |
+
+A missing or extra criterion makes the analysis result invalid. A run without a valid result can retry once.
 
 The analysis's `estimated_cost_usd` and the job's `stats.analysis.cost_usd` are separate from evaluation spend.
 

package/skills/evolve-evals/references/core-concepts/check.md

@@ -52,13 +52,31 @@ A check is a hosted model run and incurs cost. `evolve dataset check` is a separ
 | Execution | Valid reference solution; rejection of non-solutions; environment works; stable verification; sufficient limits. |
 | Feasibility | The task is solvable. |
 
-Inspect the default rubric's criteria and their complete guidance, and the current model, effort, provider, and prompt:
+Read the complete rubric guidance and the current agent, model, effort, provider, and prompt:
 
-```bash
+```bash Default checker
 evolve check --show-defaults
 ```
 
-Use `-r rubric.toml` or `-p prompt.txt` to customize. Check prompt tokens are `{task_path}`, `{file_tree}`, and `{criteria_guidance}`. The required result format is appended automatically.
+```bash Choose a checker
+evolve check --show-defaults -a codex
+evolve check ./tasks -a codex --watch
+```
+
+The default checker is `claude`. Use `-a` to choose another built-in harness. Omit `-m` to use its default model for checks, or choose one of its [models](/core-concepts/models#analysis-and-check-models).
+
+### Use your own rubric or prompt
+
+Use `-r rubric.toml` or `-p prompt.txt` to customize the review. A rubric accepts TOML, YAML, or JSON with uniquely named criteria:
+
+```toml rubric.toml
+[[criteria]]
+name = "inputs_are_documented"
+description = "The instructions describe every required input."
+guidance = "Compare the input files with the instructions."
+```
+
+Check prompt tokens are `{task_path}`, `{file_tree}`, and `{criteria_guidance}`. The required result format is appended automatically. See the [check reference](/cli-reference/check#configure-the-checker) for all options.
 
 ## The result
 
@@ -76,11 +94,49 @@ check
     └── ...
 ```
 
-The result is this per-criterion JSON and nothing else: Evolve derives no verdict from it. The dashboard shows a summary chip it computes from the outcomes. Under a custom rubric the chip is `has_a_problem` when any criterion fails, `unclear` when none fails but at least one is unknown, and `no_problem_found` otherwise. Under a rubric with the default criterion names the same rule applies, with two exceptions: an unknown on one of the default rubric's execution criteria does not make the chip unclear but marks it not executed; and `attempt_isolation` counts only when it fails. Read the criteria and compute what you need from them.
+Each criterion reports `pass`, `fail`, `not_applicable`, or `unknown`, with an explanation and evidence. The API returns these findings without an overall verdict or a separate execution flag.
+
+The dashboard computes a short summary label from the outcomes. **No problem found** does not prove that the task runs: read any unresolved execution findings.
+
+### Default rubric: read the dashboard summary
+
+These rules apply when the result has exactly the default criterion names. Apply the first matching rule:
+
+| Rule | Summary label |
+| --- | --- |
+| Any criterion fails. | **Has a problem** |
+| A criterion is unknown, except an execution criterion or `attempt_isolation`. | **Unclear** |
+| Neither of the above. | **No problem found** |
 
-**Note:**
+`attempt_isolation` changes the summary only when it fails.
+
+The execution group contains these five criteria:
+
+- `reference_solution_is_valid`
+
+- `verifier_rejects_non_solutions`
+
+- `environment_builds_and_runs`
+
+- `verification_is_stable`
+
+- `limits_allow_the_task`
+
+If any is `unknown`, the dashboard adds **Not executed**. Otherwise it adds **Executed**. This note summarizes the outcomes; it is not a separate execution audit.
+
+Read the evidence before relying on the note. An unknown means decisive execution evidence was unavailable, such as when the checker could not run the environment.
+
+### Other rubrics: read the dashboard summary
+
+Apply the first matching rule across all criteria:
+
+| Rule | Summary label |
+| --- | --- |
+| Any criterion fails. | **Has a problem** |
+| Any criterion is unknown. | **Unclear** |
+| Neither of the above. | **No problem found** |
 
-**Read the default rubric's execution criteria** (`reference_solution_is_valid`, `verifier_rejects_non_solutions`, `environment_builds_and_runs`, `verification_is_stable`, `limits_allow_the_task`) before treating a result with no failure as proof that the task runs. They are `unknown` when decisive execution evidence is unavailable, for example when the checker could not run the environment, and that leaves execution questions unresolved.
+Other rubrics have no **Executed** or **Not executed** note.
 
 The parent lifecycle is `queued` → `running` → `completed`. A completed check can contain failed task checks; inspect each task's `status` and `failure`.
 

package/skills/evolve-evals/references/core-concepts/datasets.md

@@ -102,7 +102,7 @@ Configuration validation cannot prove that images build or tests are correct. Ru
 
 ```bash
 # Run an exact version.
-evolve run -d my-dataset@1.0 -a codex -m gpt-5.6-luna
+evolve run -d my-dataset@1.0 -a codex -m gpt-6-luna
 
 # Change what the bare name resolves to.
 evolve dataset activate my-dataset 1.0

package/skills/evolve-evals/references/core-concepts/index.md

@@ -20,7 +20,7 @@ flowchart TD
 | --- | --- | --- |
 | [Task](/core-concepts/tasks) | An instruction, environment, and grading logic. | Fix a bug and pass the tests. |
 | [Dataset](/core-concepts/datasets) | A named, versioned collection of tasks. | `harbor-examples@1.0`. |
-| [Agent](/core-concepts/agents) | A harness running a selected model. | Codex with `gpt-5.6-luna`. |
+| [Agent](/core-concepts/agents) | A harness running a selected model. | Codex with `gpt-6-luna`. |
 | [Sandbox](/core-concepts/sandboxes) | The isolated environment where a trial runs. | A task image on Daytona, E2B, or Modal. |
 | [Job](/core-concepts/jobs) | A group of trials with shared run settings. | 10 tasks × 2 configurations × 3 attempts = 60 trials. |
 | [Trial](/core-concepts/trials) | One attempt at one task. | One score and its supporting evidence. |

package/skills/evolve-evals/references/core-concepts/jobs.md

@@ -19,7 +19,7 @@ Each of the three tasks runs twice in Arm A and twice in Arm B.
 ```bash
 evolve run \
   -d harbor-examples@1.0 -i hello-world \
-  -a codex -m gpt-5.6-luna \
+  -a codex -m gpt-6-luna \
   --max-trial-spend 1 -r 0 \
   --watch
 ```
@@ -50,7 +50,7 @@ Use a [config file](#config-files) to compare different harnesses in one job.
 `-i` and `-x` accept task-name globs. Quote them so your shell does not expand them.
 
 ```bash
-evolve run -d my-dataset@1.0 -a codex -m gpt-5.6-luna \
+evolve run -d my-dataset@1.0 -a codex -m gpt-6-luna \
   -i 'auth-*' -x 'auth-legacy' -l 20 \
   --max-trial-spend 2 -r 0 --watch
 ```
@@ -198,7 +198,7 @@ datasets:
     task_names: [hello-world]
 agents:
   - name: codex
-    model_name: gpt-5.6-luna
+    model_name: gpt-6-luna
   - name: claude
     model_name: fable
 n_attempts: 2