@evolvingmachines/evolve 0.0.65 → 0.0.67-train-harness-wave-2026-09-25.20260926.e00a8a6d

package/skills/evolve-evals/references/dashboard/jobs.md

@@ -0,0 +1,106 @@
+---
+title: "Jobs"
+description: "Launch an evaluation, follow its progress, and move from a summary to the evidence."
+---
+
+Open **Jobs** in your selected workspace. The list includes evaluation jobs and task checks; **Kind** tells them apart.
+
+![Jobs and task checks with status, dataset, agent, progress, and cost.](/images/dashboard-jobs.png)
+
+*Evaluation jobs and task checks appear in the same list.*
+
+```text
+Jobs list
+├── Evaluation
+│   └── Job overview
+│       └── Trial
+│           └── Trace and outputs
+└── Check
+    └── Check report
+        └── Task check
+            └── Findings and trace
+```
+
+Search by name or filter by status. Evaluation rows show the task set, agents, progress, and spend. Check rows show the source tasks and checker progress.
+
+## New job
+
+Choose **New job** to configure an evaluation.
+
+![Job launcher with a dataset search, agent, model, reasoning effort, and trial summary.](/images/dashboard-new-job.png)
+
+*Choose sources and agent settings. The summary updates as you configure the run.*
+
+### 1. Choose sources
+
+Select datasets and versions. Add include/exclude task patterns or a task cap.
+
+### 2. Add agents
+
+Choose a harness, model, and reasoning effort for each arm. Expand arm options for a version pin, preset, configuration, or skills.
+
+### 3. Set the run limits
+
+Set attempts, parallel trials, retries, and the per-trial spend cap. Add required secrets. Advanced settings include provider, timeouts, and verifier judge overrides.
+
+### 4. Review and launch
+
+Check the Summary. **Validate** checks the form locally; **Launch** submits it for server validation and execution.
+
+The number of trials is selected tasks × agent arms × attempts. [Job settings](/core-concepts/jobs) explain what each choice changes.
+
+## One job
+
+Start with progress and errors, then open the relevant tab.
+
+![Completed job summary with trials, errors, reward, cost, progress, and per-agent scores.](/images/dashboard-job-overview.png)
+
+*The overview connects run progress to scores for each agent and dataset.*
+
+| Tab | Answers |
+| --- | --- |
+| **Overview** | How did each agent and dataset perform? |
+| **Trials** | Which attempts scored, failed, or are still running? |
+| **Analyze** | Does the trace support the score? |
+| **Check** | What quality checks exist for these tasks? |
+| **Config** | What exact settings and resolved records did this job use? |
+
+A job can be **Completed** while some trials have errors. The top-level average is shown only for a single agent/dataset group; use Overview for comparisons across groups.
+
+### Job actions
+
+| Action | Effect |
+| --- | --- |
+| **Cancel** | Request that the running job stop. |
+| **Resume** | Create a new job from eligible failed or stopped trials. |
+| **Re-run job** | Open the launcher with this job's choices. |
+| **Download** | Retrieve a finished job's result archive. |
+| **Share** | Manage read access to a job you created. |
+
+Actions depend on the job state and your access. Email sharing gives read access; team membership can also permit job operations.
+
+## The ANALYZE tab
+
+The summary separates completed, failed, and pending analyses, plus their cost. An embedded analysis policy shows the configured model, effort, and rubric; manual analyses record their own settings.
+
+![Analysis criteria with pass, fail, not-applicable, and unknown totals.](/images/dashboard-analysis.png)
+
+*Criterion totals help identify which findings need closer inspection.*
+
+1. Read the **analysis summary**.
+
+2. Review **criterion totals**: pass, fail, not applicable, and unknown.
+
+3. Open **per-trial findings**: the summary chip, criterion outcomes, evidence, and summary.
+
+4. Follow the **analyzer trace**.
+
+Start a review with `evolve analyze "$JOB_ID" --watch`, or enable analysis when launching through the CLI or SDK. See [Analyze trials](/core-concepts/analyze).
+
+**[Trial viewer](/dashboard/trial-viewer)**
+
+Inspect one attempt in detail.
+
+**[Checks](/dashboard/checks)**
+
+Read the task-quality reports.

package/skills/evolve-evals/references/dashboard/traces.md

@@ -0,0 +1,54 @@
+---
+title: "Traces"
+description: "Find evaluation trials, analyses, checks, regrades, and agent sessions in one place."
+---
+
+**Traces** lists individual runs. **Jobs** groups evaluation trials; Traces lets you inspect each run directly.
+
+![Individual evaluation and analysis runs with models, providers, costs, and status.](/images/dashboard-traces.png)
+
+*Run tags distinguish the evaluated agent from the reviewer.*
+
+| Tag | Run |
+| --- | --- |
+| **Eval** | One evaluation trial. |
+| **Regrade** | A verifier-only rerun. |
+| **Analyze** | A trial-review run. |
+| **Check** | A task-quality review. |
+| **Agent** | A managed-agent session. |
+
+## Choose the workspace
+
+### Personal
+
+**Mine** shows runs you created, including those in teams.
+
+**Shared with me** shows teammates' runs and runs shared with your email address.
+
+### Team
+
+The list shows runs in that team. **Created by** identifies who started each one.
+
+## Find and inspect a run
+
+### 1. Search by name
+
+Read the kind, task or session name, model, provider, status, and cost to identify the run.
+
+### 2. Open its row
+
+The viewer shows that run's transcript. An Analyze row opens the analyzer's work; an Eval row opens the evaluated agent's work.
+
+### 3. Inspect or act
+
+Read events and downloads. Stop live runs or delete finished records only where your creator permissions allow it.
+
+Sharing lets someone inspect a run. It does not transfer ownership or grant stop/delete permission.
+
+**[Trial viewer](/dashboard/trial-viewer)**
+
+Follow messages, tools, and gateway usage.
+
+**[Sharing](/core-concepts/sharing)**
+
+Email access and unlisted links.

package/skills/evolve-evals/references/dashboard/trial-viewer.md

@@ -0,0 +1,77 @@
+---
+title: "Trial viewer"
+description: "Follow the agent's actions and inspect the evidence behind a result."
+---
+
+Open a trial from a job's **Trials** tab, or select a run in **Traces**.
+
+![Trial viewer with agent messages, tool calls, tool results, and recorded model usage.](/images/dashboard-trial-viewer.png)
+
+*Follow what the agent did and inspect the corresponding usage.*
+
+```text
+1. Run header
+   Task · agent · status · spend
+
+2. Controls
+   Back · search · downloads
+
+3. Event list
+   Select a message or tool call
+            ↓
+4. Read the event
+   Full content and tool result
+```
+
+This schematic shows the reading order. The viewer adapts its controls to the run type and available space.
+
+## Read a run
+
+### 1. Identify the run
+
+Check the run kind, task, model, status, and spend. An analysis or check shows the reviewer's own execution, not the evaluated agent's trace.
+
+### 2. Follow events
+
+Move through the prompt, messages, thoughts, tool calls, and tool results. Select an event to read it in full. New events appear while a live run continues.
+
+### 3. Check the outcome
+
+Inspect the reward or failure. Open the verifier log or review result from Downloads when you need the scoring evidence.
+
+## Move through the trace
+
+| Control | Action |
+| --- | --- |
+| Event row | Select an event. |
+| `↑` / `k` | Previous event. |
+| `↓` / `j` | Next event. |
+| `/` | Focus event search. |
+
+Search highlights matching events. Tool analytics summarizes tool use. Spend views use the recorded accounting; gateway events identify model calls and their token and cost readings.
+
+## Download the right evidence
+
+### Evaluation trial
+
+Parsed trace, ATIF trajectory, stdout, stderr, agent home, verifier log, or the combined download.
+
+### Analysis or task check
+
+Result document, reviewer stdout and stderr, agent home, or the combined download.
+
+A download can only contain what the run recorded. See [Trial outputs](/core-concepts/trial-outputs) for the saved layout and binary-file differences.
+
+## Run actions
+
+**Retry** creates a new one-trial job from a settled evaluation trial. **Stop** ends a live run. Controls appear only when the run type, state, and your permissions allow them.
+
+Use **Back** to return to the job tab or list you came from.
+
+**[Understand trial outcomes](/core-concepts/trials#statuses)**
+
+Scores, errors, budgets, and cancellation.
+
+**[Files and logs](/core-concepts/trial-outputs#sandbox-files)**
+
+Inspect the live sandbox or its retained capture.

package/skills/evolve-evals/references/getting-started/first-task.md

@@ -0,0 +1,228 @@
+---
+title: "Create your first task"
+description: "Write a task, publish it, check its quality, and run one managed evaluation."
+---
+
+Build a task that asks an agent to write one file. Publish it as a dataset, then inspect the agent's answer and score.
+
+First, [install the CLI and set a full-access API key](/getting-started/installation). The task check and evaluation run remotely and incur usage.
+
+## 1. Create the task folder
+
+Start in a fresh working directory:
+
+```bash
+mkdir -p \
+  my-eval/hello-world/environment \
+  my-eval/hello-world/tests \
+  my-eval/hello-world/solution
+cd my-eval
+```
+
+Create all five files below. Keep `tests/` and `solution/` separate from the environment image.
+
+- my-eval/
+  - hello-world/
+    - instruction.md
+    - task.toml
+    - environment/
+      - Dockerfile
+    - tests/
+      - test.sh
+    - solution/
+      - solve.sh
+
+### hello-world/instruction.md
+
+Tell the agent exactly what to produce.
+
+```markdown
+Create this file:
+
+/app/answer.txt
+
+With exactly this text:
+
+Hello, world!
+
+End the file with a newline.
+```
+
+### hello-world/task.toml
+
+Give the agent 120 seconds and the verifier 60 seconds. Shared verification reads the file in the agent's sandbox.
+
+```toml
+schema_version = "1.4"
+
+[agent]
+timeout_sec = 120
+
+[verifier]
+timeout_sec = 60
+environment_mode = "shared"
+
+[environment]
+cpus = 2
+memory_mb = 2048
+storage_mb = 10240
+workdir = "/app"
+```
+
+### hello-world/environment/Dockerfile
+
+Include Python for the verifier. Evolve builds this image when you publish.
+
+```dockerfile
+FROM python:3.12-slim
+WORKDIR /app
+```
+
+### hello-world/tests/test.sh
+
+Write reward `1` for the exact answer, or `0` for a wrong or missing file.
+
+```bash
+#!/bin/bash
+set -uo pipefail
+mkdir -p /logs/verifier
+
+if python3 - <<'PYTEST'
+from pathlib import Path
+answer = Path("/app/answer.txt")
+assert answer.read_text() == "Hello, world!\n"
+PYTEST
+then
+  echo 1 > /logs/verifier/reward.txt
+else
+  echo 0 > /logs/verifier/reward.txt
+fi
+```
+
+The reward file determines the score; the script's exit code alone does not.
+
+### hello-world/solution/solve.sh
+
+Include a reference answer for task-quality review. The evaluated agent does not receive this script as an input.
+
+```bash
+#!/bin/bash
+printf 'Hello, world!\n' > /app/answer.txt
+```
+
+## 2. Validate the metadata
+
+From `my-eval/`, run:
+
+```bash
+evolve dataset check .
+```
+
+This sends task metadata to Evolve for validation. It does not publish the task, build its image, or run the verifier. Fix any reported refusals before continuing.
+
+## 3. Publish and wait for the build
+
+Choose an available dataset name for your account. Keep these variables for the following commands:
+
+```bash
+DATASET="my-first-eval"
+VERSION="1.0"
+
+evolve dataset publish \
+  --dir . \
+  --name "$DATASET" \
+  --version "$VERSION" \
+  --watch
+```
+
+The CLI repeats metadata validation, uploads the task, and follows the import and image build. Continue when the version is `READY` and `hello-world` is listed as built:
+
+```bash
+evolve dataset show "$DATASET@$VERSION"
+```
+
+Publishing prepares the task to run. It does not evaluate the agent.
+
+## 4. Check task quality
+
+```bash
+evolve check \
+  -d "$DATASET@$VERSION" \
+  -i hello-world \
+  --watch
+```
+
+This runs a separate checker against the task's instructions, environment, reference solution, and verifier. Save the printed check ID:
+
+```bash
+CHECK_ID="paste-check-id-here"
+evolve check show "$CHECK_ID"
+```
+
+Read the criterion findings, the default rubric's execution criteria included. A completed check can contain failures; a clean inspection without execution does not establish that the solution and verifier work. See [task checks](/core-concepts/check#the-result) for interpretation.
+
+If you change any task file, publish a new version and use that version in the remaining commands.
+
+## 5. Run one evaluation
+
+```bash
+evolve run \
+  -d "$DATASET@$VERSION" \
+  -i hello-world \
+  -a codex \
+  -m gpt-5.6-luna \
+  --max-trial-spend 1 \
+  --max-retries 0 \
+  --watch
+```
+
+This creates one trial with a $1 model-spend cap. The cap applies to this evaluation, not the earlier task check; an in-flight model call can finish above it. Configured infrastructure retries are disabled, but provider capacity waits can still occur.
+
+## 6. Inspect the answer and score
+
+Set `JOB_ID` to the job ID printed by `run`:
+
+```bash
+JOB_ID="paste-job-id-here"
+evolve job show "$JOB_ID"
+evolve job trials "$JOB_ID"
+```
+
+Set `TRIAL_ID` to the trial ID from that list:
+
+```bash
+TRIAL_ID="paste-trial-id-here"
+evolve trial show "$TRIAL_ID"
+evolve trial trace "$TRIAL_ID"
+evolve trial download "$TRIAL_ID" --stream verifier
+```
+
+Read the answer if the agent produced it:
+
+```bash
+evolve trial files cat "$TRIAL_ID" /app/answer.txt
+```
+
+The saved filesystem can still be finishing after the trial settles. If it is not ready yet, retry the read shortly.
+
+| Result | Meaning for this task |
+| --- | --- |
+| `SCORED`, reward `1` | The answer matched, including the final newline. |
+| `SCORED`, reward `0` | The verifier rejected the answer. |
+| An execution or scoring error | Inspect the trace and verifier log before interpreting the result. |
+
+A completed job means its trials have settled, not that the task passed. Save the trial record and available logs:
+
+```bash
+evolve trial download "$TRIAL_ID" -o ../eval-results/
+```
+
+Keep results outside `my-eval/`, so the dataset folder still contains only task directories.
+
+**[Task configuration](/core-concepts/task-config)**
+
+Add resources, network rules, and artifacts.
+
+**[Task verifiers](/core-concepts/task-verifiers)**
+
+Grade richer outputs or use a separate verifier sandbox.

package/skills/evolve-evals/references/getting-started/installation.md

@@ -0,0 +1,79 @@
+---
+title: "Installation"
+description: "Install your preferred client and connect it to your Evolve account."
+---
+
+Use the CLI, Python, or TypeScript. All three connect to the same managed evaluation service.
+
+## Install a client
+
+```bash CLI
+npm install -g @evolvingmachines/evolve
+evolve --version
+```
+
+```bash Python
+pip install evolvingmachines-evolve
+```
+
+```bash TypeScript
+npm install @evolvingmachines/evolve
+```
+
+The CLI and TypeScript package use Node.js. Python requires **3.10 or newer**; its managed evals clients make HTTP requests directly and do not need Node.js.
+
+**Note:**
+
+For a one-off CLI command, use `npx --package=@evolvingmachines/evolve evolve --help`. Outside a project with Evolve installed, bare `npx evolve` can fetch an unrelated package.
+
+## Connect your account
+
+### 1. Create an API key
+
+Sign in to the [Evolve dashboard](https://dashboard.evolvingmachines.ai), then open [API keys](https://dashboard.evolvingmachines.ai/api-keys) and create a key.
+
+| Access | Use it for |
+| --- | --- |
+| **Full access** | Run evaluations, publish datasets, and make changes your account is allowed to make. Use this for the quick start. |
+| **Read-only** | Inspect existing work. It cannot create, cancel, delete, or change team membership. |
+
+### 2. Set the environment variable
+
+```bash
+export EVOLVE_API_KEY="your-api-key"
+```
+
+Every authenticated client reads this variable. Set it in the environment where your terminal or application runs.
+
+### 3. Check the connection
+
+```bash
+evolve auth status
+```
+
+The result identifies your account and key. See [SDK authentication](/sdk-reference/auth) for the equivalent Python and TypeScript calls.
+
+A read-only key used for a write returns HTTP `403` with code `read_only_key`. Full access still respects team roles and resource permissions. Revoke keys in the dashboard, or use the [HTTP key-management endpoints](/sdk-reference/auth#manage-api-keys-over-http).
+
+## Give your coding agent the manual
+
+```bash
+evolve skills install
+```
+
+By default, this installs the small `evolve` skill into supported agent locations that already exist. To create a specific destination, use a target such as `evolve skills install --target codex`. The skill directs the agent to the manual bundled with your CLI:
+
+```bash
+evolve skills get evals
+evolve skills get evals getting-started/quick-start
+```
+
+See [bundled skills](/cli-reference/skills) for installation targets and task-authoring guides.
+
+**[Quick start](/getting-started/quick-start)**
+
+Run one task from the terminal.
+
+**[SDKs](/sdk-reference/index)**
+
+Create and inspect jobs from code.

package/skills/evolve-evals/references/getting-started/quick-start.md

@@ -0,0 +1,100 @@
+---
+title: "Quick start"
+description: "Run one small task, inspect its result, and download the evidence."
+---
+
+A **trial** is one agent's attempt at one task. A **job** groups trials. Start with a one-trial job.
+
+### 1. Install and authenticate
+
+Create a key in the dashboard's [API keys page](https://dashboard.evolvingmachines.ai/api-keys) after signing in.
+
+```bash
+npm install -g @evolvingmachines/evolve
+export EVOLVE_API_KEY="your-api-key"
+evolve auth status
+```
+
+Already connected? Continue below. Python and TypeScript installation is on the [installation page](/getting-started/installation).
+
+### 2. Inspect the task
+
+```bash
+evolve dataset show harbor-examples@1.0
+```
+
+This shows the dataset's tasks and provider compatibility. We will select only `hello-world`.
+
+### 3. Start the job
+
+```bash
+evolve run \
+  -d harbor-examples@1.0 \
+  -i hello-world \
+  -a codex \
+  -m gpt-5.6-luna \
+  --max-trial-spend 1 \
+  --max-retries 0 \
+  --watch
+```
+
+| Choice | Meaning |
+| --- | --- |
+| `-d` and `-i` | The dataset version and one task name. |
+| `-a` and `-m` | Codex with the selected model. |
+| `--max-trial-spend 1` | A $1 model-spend cap for this trial attempt. |
+| `--max-retries 0` | Disable configured infrastructure retries. Provider capacity waits can still occur. |
+| `--watch` | Follow progress until the job settles. |
+
+The spend cap stops new model calls once reached; a call already in progress can finish above it.
+
+The final output includes the job ID. The job continues remotely if you close the terminal.
+
+### 4. Read the result
+
+Set `JOB_ID` to the full ID printed by the previous command:
+
+```bash
+JOB_ID="paste-job-id-here"
+evolve job show "$JOB_ID"
+evolve job trials "$JOB_ID"
+```
+
+A completed job means all trials have settled. Inspect the trial's status and reward to see whether it succeeded, failed the task, or hit an execution error.
+
+### 5. Inspect the evidence
+
+Set `TRIAL_ID` to the ID from `job trials`:
+
+```bash
+TRIAL_ID="paste-trial-id-here"
+evolve trial show "$TRIAL_ID"
+evolve trial trace "$TRIAL_ID"
+evolve trial download "$TRIAL_ID" --stream verifier
+```
+
+The trace shows the agent's actions. The verifier log explains the grading process.
+
+### 6. Keep the result
+
+```bash
+evolve trial download "$TRIAL_ID" -o ./results/
+```
+
+This writes `./results/$TRIAL_ID/` with the trial record and available trace, logs, and agent session files. See the [output layout](/core-concepts/trial-outputs), including how to download the task filesystem.
+
+**[Compare configurations](/core-concepts/jobs)**
+
+Add models, tasks, or repeated attempts.
+
+**[Create a task](/core-concepts/tasks)**
+
+Evaluate work that matters to your project.
+
+**[Python](/sdk/python)**
+
+Run the same workflow from Python.
+
+**[TypeScript](/sdk/typescript)**
+
+Run the same workflow from TypeScript.

package/skills/evolve-evals/references/index.md

@@ -0,0 +1,58 @@
+---
+title: "Evolve Documentation"
+description: "Environment-driven agent evaluation and optimization."
+---
+
+Evolve is the engine for running reproducible agent evaluations at scale across tasks, sandboxes, harnesses, and models. It provides quality control during task creation, with full observability and trajectory analysis for every run.
+
+```mermaid
+%%{init: {"flowchart": {"nodeSpacing": 16, "rankSpacing": 20, "padding": 10}, "themeVariables": {"fontSize": "14px", "fontFamily": "Google Sans Flex, sans-serif"}}}%%
+flowchart LR
+  T["Tasks"] --> E["Evolve<br/>Cloud runs"]
+  H["Harnesses"] --> E
+  M["Models"] --> E
+  E --> S["Scores"]
+  E --> A["Trajectories"]
+  E --> O["Logs and files"]
+```
+
+**[Task quality checks](/core-concepts/check)**
+
+Check that tasks are solvable, instructions are clear, and verifiers score correctly.
+
+**[Agent optimization](/core-concepts/jobs)**
+
+Compare harnesses, models, and settings on the same tasks. Measure rewards and costs.
+
+**[Trajectory analysis](/core-concepts/analyze)**
+
+Review the agent’s actions, failures, and the evidence behind its score.
+
+**[Full observability](/core-concepts/trial-outputs)**
+
+Inspect traces, files, verifier logs, timing, token usage, and costs.
+
+## One command to start
+
+After [installation and authentication](/getting-started/installation), run one task:
+
+```bash
+evolve run \
+  -d harbor-examples@1.0 -i hello-world \
+  -a codex -m gpt-5.6-luna \
+  --max-trial-spend 1 --max-retries 0 --watch
+```
+
+The job runs on Evolve. Your terminal follows its progress; closing the terminal does not cancel it.
+
+**[First evaluation](/getting-started/quick-start)**
+
+Run, score, and inspect one task.
+
+**[Your own tasks](/core-concepts/tasks)**
+
+Build in the Harbor task format.
+
+**[Python and TypeScript](/sdk-reference/index)**
+
+Run evaluations from code.