@evolvingmachines/evolve 0.0.65 → 0.0.66

package/skills/evolve-evals/references/core-concepts/task-verifiers.md

@@ -0,0 +1,209 @@
+---
+title: "Task verifier"
+description: "Turn the agent's work into a reward you can compare."
+---
+
+The verifier runs after the agent. Its entry point is `tests/test.sh`, and its result comes from a reward file.
+
+```text
+1. Agent output
+   /app/answer.txt
+         ↓
+2. Run the verifier
+   /tests/test.sh
+         ↓
+3. Read the reward
+   /logs/verifier/reward.json
+   or /logs/verifier/reward.txt
+```
+
+## Write a reward
+
+This verifier checks the [example instruction](/core-concepts/tasks#write-a-clear-instruction):
+
+```bash tests/test.sh
+#!/bin/bash
+set -uo pipefail
+mkdir -p /logs/verifier
+
+if python3 - <<'PYTEST'
+from pathlib import Path
+assert Path("/app/answer.txt").read_text() == "Hello, world!\n"
+PYTEST
+then
+  echo 1 > /logs/verifier/reward.txt
+else
+  echo 0 > /logs/verifier/reward.txt
+fi
+```
+
+The reward file determines the score. The script's exit code alone does not.
+
+| File | Shape |
+| --- | --- |
+| `reward.txt` | One finite number from `0` to `1` |
+| `reward.json` | An object of named finite numbers |
+
+A nonempty `reward.json` takes precedence over `reward.txt`. The primary reward is the `reward` field, or the only field in a one-field object. A primary reward must be in `[0, 1]`.
+
+```json /logs/verifier/reward.json
+{"reward": 1, "tests_passed": 12, "tests_total": 12}
+```
+
+Multiple metrics without a `reward` key produce a scored trial with no primary reward. That trial cannot contribute a primary success to pass@k.
+
+| Outcome | Trial status |
+| --- | --- |
+| Valid reward, including `0` | `SCORED` |
+| No reward file | `INDETERMINATE` |
+| Invalid reward | `SCORING_ERROR` |
+
+## Choose where verification runs
+
+### Shared · default
+
+The verifier runs in the agent's sandbox. It can inspect the filesystem and processes left by the agent. Agent credentials are revoked before verification begins.
+
+```text
+One sandbox
+┌─────────────────────┐
+│ 1. Agent works      │
+│         ↓           │
+│ 2. Verifier checks  │
+└─────────────────────┘
+```
+
+```toml task.toml
+[verifier]
+environment_mode = "shared"
+```
+
+Test files are uploaded to `/tests/`. Install verifier dependencies in the task image; a `tests/Dockerfile` is not built in this mode.
+
+### Separate
+
+The verifier starts in a fresh sandbox. Declare the outputs it needs under `artifacts`. Evolve also transfers `/logs/artifacts/` and the collection script’s patch file; it does not copy the whole agent filesystem.
+
+```text
+1. Agent sandbox
+   /app/answer.txt
+         ↓ copy artifact
+2. Fresh verifier sandbox
+   /app/answer.txt
+```
+
+Separate verification supports [regrading eligible trials](/core-concepts/jobs). It is not available for multi-step tasks.
+
+## Separate verification
+
+A verifier with its own Dockerfile keeps grading dependencies out of the agent's image.
+
+```text
+my-task/
+├── instruction.md
+├── task.toml
+├── environment/
+│   └── Dockerfile
+└── tests/
+    ├── Dockerfile
+    └── test.sh
+```
+
+```toml task.toml
+artifacts = ["/app/answer.txt"]
+
+[verifier]
+environment_mode = "separate"
+
+[verifier.environment]
+cpus = 2
+memory_mb = 2048
+workdir = "/app"
+network_mode = "no-network"
+```
+
+```dockerfile tests/Dockerfile
+FROM python:3.12-slim
+WORKDIR /app
+COPY . /tests/
+```
+
+With `environment_mode` omitted, `[verifier.environment]` selects separate mode. An explicit `environment_mode = "shared"` keeps verification in the agent sandbox.
+
+### How the verifier image is selected
+
+The effective configuration is `[verifier.environment]` when present; otherwise, it copies `[environment]`.
+
+| Effective configuration | Behavior |
+| --- | --- |
+| No `docker_image` | Build `tests/Dockerfile`; it must provide `/tests/test.sh` |
+| Same pinned image as the agent | Reuse that image and upload the task's test files |
+| A distinct pinned image | Use that image as-is; it must contain `/tests/test.sh` |
+
+The task source must still contain a nonempty `tests/test.sh`, including when a distinct prebuilt verifier image already contains it.
+
+A `tests/Dockerfile` is not built when the effective configuration pins an image. The import records a `tests_dockerfile_not_built` note if that unused file exists.
+
+## Prepare outputs for collection
+
+Most tasks only need an `artifacts` list. If outputs need preparation after the agent finishes, add `pre_artifacts.sh` at the task root. It runs from the task's working directory.
+
+```bash pre_artifacts.sh
+#!/bin/bash
+set -euo pipefail
+mkdir -p /logs/artifacts
+[ -e /logs/artifacts/model.patch ] || : > /logs/artifacts/model.patch
+if [ -f /app/summary.json ]; then
+  cp /app/summary.json /logs/artifacts/summary.json
+fi
+```
+
+This example assumes the agent produces `/app/summary.json`. A custom collection script must also provide `/logs/artifacts/model.patch`; it can be empty when the task does not produce a patch. If you omit the script, Evolve prepares that file for you.
+
+### Collect from a service
+
+`[[verifier.collect]]` runs a command before artifact collection. For a [multi-container task](/core-concepts/compose), use `service` to choose the container. For a single container, omit it or use `main`.
+
+```toml task.toml
+artifacts = [
+  { source = "/tmp/stats.json", service = "api" }
+]
+
+[[verifier.collect]]
+service = "api"
+command = "curl -fsS http://localhost:8000/stats > /tmp/stats.json"
+timeout_sec = 60
+```
+
+This example assumes an `api` service with a `/stats` endpoint and `curl` installed. Hook failures are logged and do not stop the run. Make the verifier reject missing evidence when it is required for a valid score.
+
+Main-service hooks run before collecting its files. In separate verification, Evolve then stops `main`, runs sidecar hooks, and collects sidecar files. Shared verification keeps the containers running. Sidecar artifacts retain their source paths in the separate verifier.
+
+`timeout_sec` defaults to 60 seconds. Compose hooks can set `user` to choose an existing container user. Single-container hooks do not accept that override.
+
+## Grade with a model
+
+[Reward Kit](/core-concepts/rewardkit) combines reusable checks, custom Python, and model judges.
+
+A verifier can request model access for an LLM judge:
+
+```toml task.toml
+[verifier.env]
+ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"
+```
+
+Evolve supplies a gateway credential and the matching base URL. Recognized credential families are Anthropic and OpenAI; these templates are not job secrets. They also work under `no-network` through the judge gateway.
+
+For a Reward Kit judge, override its agent and model for one job with:
+
+```bash
+evolve run -d my-dataset@1.0 -a codex -m gpt-5.6-luna \
+  --ve REWARDKIT_JUDGE=claude \
+  --ve REWARDKIT_MODEL=sonnet
+```
+
+These are the only accepted `--ve` keys. Judge spend is reported separately from agent spend. Its independent cap is the smaller of the agent’s per-trial cap and $5. Judge-enabled trials are not currently regradable.
+
+**Tip:**
+
+Ask a coding agent to load `evolve skills get rewardkit` for verifier authoring guidance. Use `evolve skills get create-task` for the complete task workflow.

package/skills/evolve-evals/references/core-concepts/tasks.md

@@ -0,0 +1,90 @@
+---
+title: "Tasks"
+description: "Define the work, the environment, and how to score it."
+---
+
+A task is one problem for an agent to solve. It packages an instruction, a Linux environment, and a verifier that scores the result.
+
+Evolve accepts the **Harbor task format**, with the [managed capabilities and limits](/core-concepts/compatibility) described in these guides.
+
+**Tip:**
+
+New to task authoring? [Build your first task](/getting-started/first-task) from five small files.
+
+- my-task/
+  - instruction.md
+  - task.toml
+  - environment/
+    - Dockerfile
+  - tests/
+    - test.sh
+  - solution/
+    - solve.sh
+
+| File | Purpose |
+| --- | --- |
+| `instruction.md` | What the agent must do |
+| `task.toml` | Timeouts, resources, and policies |
+| `environment/Dockerfile` | Tools and files available to the agent |
+| `tests/test.sh` | Checks the work and writes a reward |
+| `solution/solve.sh` | Optional reference solution |
+
+For a single-step task, `instruction.md`, `task.toml`, and a nonempty `tests/test.sh` are required. Define the environment with a Dockerfile, a pinned image, or supported Docker Compose configuration.
+
+## What happens in a trial
+
+```mermaid
+%%{init: {"flowchart": {"nodeSpacing": 16, "rankSpacing": 20, "padding": 10}, "themeVariables": {"fontSize": "14px", "fontFamily": "Google Sans Flex, sans-serif"}}}%%
+flowchart TB
+  A["1. Start the task environment"] --> B["2. Give the agent the instruction"]
+  B --> C["3. Agent works on the task"]
+  C --> D["4. Run the verifier"]
+  D --> E["5. Save reward, logs, and outputs"]
+```
+
+The agent receives the instruction and environment. The verifier and reference solution are not handed to it as task inputs.
+
+## Write a clear instruction
+
+State the requested change, the output path, and the conditions for success. The verifier should check those same conditions.
+
+```markdown instruction.md
+Create /app/answer.txt containing exactly:
+
+Hello, world!
+
+End the file with a newline.
+```
+
+Use absolute paths when the location matters. A benchmark canary marker is removed before the instruction reaches the agent.
+
+## Add a reference solution
+
+`solution/solve.sh` demonstrates a correct answer. For the instruction above:
+
+```bash solution/solve.sh
+#!/bin/bash
+printf 'Hello, world!\n' > /app/answer.txt
+```
+
+Keep it separate from the environment image. [Task checks](/core-concepts/check) can use it to test whether the verifier accepts a valid solution.
+
+## Build the rest
+
+**[Environment](/core-concepts/task-environment)**
+
+Images, working directories, services, and MCP tools.
+
+**[Verifier](/core-concepts/task-verifiers)**
+
+Write a reward and choose where grading runs.
+
+**[Configuration](/core-concepts/task-config)**
+
+Resources, network access, variables, and timeouts.
+
+**[Multiple steps](/core-concepts/multi-step)**
+
+Continue work across ordered instructions in one environment.
+
+When the task is ready, [publish it as a dataset](/core-concepts/datasets).

package/skills/evolve-evals/references/core-concepts/teams.md

@@ -0,0 +1,128 @@
+---
+title: "Teams"
+description: "Put evaluations in a shared workspace and give teammates access through membership."
+---
+
+Every account starts with a personal organization. Create a team organization to run and inspect work together.
+
+```text
+Your account
+├── Personal workspace
+└── Team workspace
+    ├── Members and roles
+    ├── Jobs, checks, and sessions
+    ├── Datasets
+    ├── Registered agents
+    └── Uploaded skills
+```
+
+## Create and join
+
+### 1. Create the team
+
+```bash
+evolve auth org create acme --display-name "Acme Lab"
+```
+
+### 2. Create an invitation
+
+```bash
+evolve auth org invite acme
+```
+
+Only an owner can create invitations. The command prints a token once; give it to the person you want to invite. By default, it expires after 7 days and allows unlimited joins.
+
+### 3. Join from the invited account
+
+Set `$INVITE_TOKEN` to the token from the owner, then run:
+
+```bash
+evolve auth org join "$INVITE_TOKEN"
+```
+
+The invited account joins as a member.
+
+| Role | Access |
+| --- | --- |
+| Member | Read team work and run evaluations. |
+| Owner | Member access, plus team and membership management. |
+
+Membership does not make every record yours to delete or share. Those operations retain their creator-specific rules.
+
+## Manage the team
+
+Use the dashboard's **Members** and **Settings** pages, or the [HTTP team-management endpoints](/sdk-reference/auth#manage-teams-over-http).
+
+| Action | Rule |
+| --- | --- |
+| Rename the team | An owner can change its slug or display name. |
+| Change roles or remove someone | Owner required. The last owner cannot be demoted or removed. |
+| Leave | Any member can leave, except the last owner. Promote another owner first. |
+| Revoke an invitation | Owner required. This stops new joins; it does not remove existing members. |
+| Delete the team | Owner required. Deletion is refused while the team owns jobs or datasets. |
+
+Personal organizations cannot be renamed, deleted, or have their membership changed. For invitation expiry and use limits, use the [HTTP invitation options](/sdk-reference/auth#manage-teams-over-http); the CLI and SDKs use the defaults.
+
+## Run under a team
+
+```bash
+evolve run \
+  -d harbor-examples@1.0 -i hello-world \
+  -a codex -m gpt-5.6-luna \
+  --max-trial-spend 1 -r 0 --org acme
+```
+
+Set a default for later CLI commands, or return to your personal organization:
+
+```bash
+evolve auth org use acme
+evolve auth org use personal
+```
+
+Dataset publishing, custom-agent registration, and skill upload also accept an organization. A new version of an existing dataset stays in that dataset's organization.
+
+```typescript TypeScript
+import { hosted } from "@evolvingmachines/evolve";
+
+const client = hosted({ org: "acme" });
+```
+
+```python Python
+from evolve import hosted, HostedClientConfig
+
+client = hosted(HostedClientConfig(org="acme"))
+```
+
+The SDK client default applies to operations that accept organization ownership, such as job creation and dataset publishing. It is not a filter on every read.
+
+## Choose what to list
+
+| Scope | Shows |
+| --- | --- |
+| `my` | Work you created. The default. |
+| `shared` | Teammates' work and runs shared with your email address. |
+| `org` | Work in your organizations, including your own. |
+
+```bash
+evolve job list --scope org
+```
+
+## In the dashboard
+
+Use the workspace switcher beneath your account to choose **Personal** or a team. Team pages include Datasets, Jobs, Traces, Members, and Settings.
+
+| Page | Use it for |
+| --- | --- |
+| **Members** | Read roles; owners manage membership and invitation links. |
+| **Settings** | Read the workspace name; owners manage the team. |
+| **Traces** | Team runs with a Created by column. |
+
+Skills, Integrations, Secrets, API Keys, Usage, and Billing remain account pages. Selecting a team does not turn these into shared team settings.
+
+**[Organization commands](/cli-reference/auth)**
+
+Create, join, inspect, and select a team.
+
+**[Share one run](/core-concepts/sharing)**
+
+Give access without adding a team member.

package/skills/evolve-evals/references/core-concepts/trial-outputs.md

@@ -0,0 +1,225 @@
+---
+title: "Trial outputs"
+description: "Find the record, trace, logs, and files from an evaluation."
+---
+
+Choose the output that answers your question:
+
+| Question | Open |
+| --- | --- |
+| What score did it earn? | `result.json` and verifier output. |
+| What did the agent do? | Parsed trace or ATIF trajectory. |
+| What did the process print? | Raw stdout and stderr. |
+| What files did it change? | Run filesystem and changes list. |
+| What state did the harness keep? | Captured agent home. |
+
+## The trial directory
+
+```bash
+evolve trial download "$TRIAL_ID" -o trials/
+```
+
+A typical Codex trial looks like this. Files appear only when recorded.
+
+- trials/
+  - TRIAL_ID/
+    - config.json
+    - result.json
+    - evolve.json
+    - exception.txt
+    - agent/
+      - trajectory.json
+      - trace-parsed.jsonl
+      - codex.txt
+      - stderr.log
+      - agent-home.json
+      - .codex/
+    - verifier/
+      - test-stdout.txt
+      - reward.json
+
+### What each file contains
+
+| File | Contents |
+| --- | --- |
+| `config.json` | Task and agent identity |
+| `result.json` | Reward, timing, and exception |
+| `evolve.json` | Evolve's detailed trial record |
+| `exception.txt` | Present when an exception exists |
+| `trajectory.json` | ATIF conversation |
+| `trace-parsed.jsonl` | Parsed events |
+| `codex.txt` | Raw stdout |
+| `stderr.log` | Raw stderr |
+| `agent-home.json` | Capture manifest |
+| `.codex/` | Captured harness state |
+| `test-stdout.txt` | Verifier log |
+| `reward.json` | Recorded rewards |
+
+The stdout filename follows the harness. A registered custom agent uses `stdout.log`.
+
+### Trial download or job archive?
+
+| Download | Includes |
+| --- | --- |
+| `evolve trial download "$TRIAL_ID"` | One trial reconstructed by the CLI, including the text view of its agent home. |
+| `evolve job download "$JOB_ID"` | The full server archive: job records, each trial's locks and logs, retained binary home files, and recorded artifacts. |
+
+A job archive also includes per-step records for multi-step tasks. Platform-specific fields in Harbor-format records use `x_evolve` extensions.
+
+Existing output folders require `--overwrite`. This permits writing into the folder; it does not remove unrelated old files.
+
+### Declared artifacts in the job archive
+
+Each trial in the full job archive contains `artifacts/manifest.json`. For native Evolve runs, artifact payloads come from the retained inputs of **separate verification**. Shared verification does not export the task's artifact list; inspect its live or captured files through [sandbox files](#sandbox-files).
+
+Files normally mirror their absolute source path under `artifacts/`. A task's `destination` changes that downloaded location. For example, one generated manifest entry can be:
+
+```json
+{
+  "source": "/app/reports/result.json",
+  "destination": "artifacts/reports/result.json",
+  "type": "file",
+  "status": "ok",
+  "service": null
+}
+```
+
+### Artifact manifest fields
+
+The manifest is a JSON array of entries.
+
+| Field | Meaning |
+| --- | --- |
+| `source` | Original absolute file path in the sandbox. |
+| `destination` | Export path relative to the trial directory, including `artifacts/`. |
+| `type` | `"file"` for generated entries. Directory declarations expand to individual retained files. |
+| `status` | `"ok"` for an emitted file, or `"skipped"` for a destination conflict. |
+| `service` | Compose service name, or `null` for the main container. |
+
+The first file claiming a destination wins. Later equal or nested destinations are skipped, and `artifacts/manifest.json` is reserved for the manifest itself.
+
+Generated entries describe retained file candidates; missing, unrecorded files have no entry. A native trial with no retained artifact record has `[]`, including shared-mode trials or a separate-mode trial whose artifact record could not be stored. The separate filesystem capture can still be available.
+
+Uploaded archives retain their own artifact files and manifest when present. Their manifest content is preserved rather than rewritten to the generated schema above.
+
+## The parsed trace
+
+Each event has a sequence position, a type, and data. This illustrative event shows a completed tool call:
+
+```json
+{
+  "seq": 3,
+  "type": "tool_call_update",
+  "data": {
+    "update": {
+      "sessionUpdate": "tool_call_update",
+      "toolCallId": "call-1",
+      "status": "completed"
+    }
+  }
+}
+```
+
+```bash
+evolve trial trace "$TRIAL_ID" --json
+evolve trial trace "$TRIAL_ID" --grep 'permission denied' --tail 50
+```
+
+Gateway usage events carry `data.update.source: "gateway"`. Use them for gateway token and cost accounting; harness-reported usage is a different reading.
+
+## The ATIF trajectory
+
+ATIF is Harbor's **Agent Trajectory Interchange Format**. Evolve exports `ATIF-v1.7`, built from the stored parsed trace.
+
+| Field | Contents |
+| --- | --- |
+| `schema_version` | Format version |
+| `agent` | Harness, version, and model |
+| `steps[].source` | User, system, or agent |
+| `steps[].message` | Conversation content |
+| `steps[].tool_calls` | Agent actions |
+| `steps[].observation` | Tool results |
+
+```bash
+evolve trial download "$TRIAL_ID" --stream trace-atif
+```
+
+The `trajectory` stream name is reserved for a native harness session and is not served today. Use `trace-atif` for the portable trajectory.
+
+## Raw streams and the verifier log
+
+```bash
+evolve trial download "$TRIAL_ID" --stream trace-stdout
+evolve trial download "$TRIAL_ID" --stream trace-stderr
+evolve trial download "$TRIAL_ID" --stream verifier
+```
+
+An absent artifact remains absent. It is not replaced with an empty success log.
+
+## The agent's home folder
+
+The captured home contains harness settings, sessions, and subagent transcripts. `agent-home.json` records paths, sizes, digests, and files excluded or skipped.
+
+The `agent-home` stream is a **text view**. Binary files are listed in the manifest but omitted from this view. Download the full job archive to retrieve retained binary files.
+
+## Sandbox files
+
+The run filesystem is separate from the saved trial directory above.
+
+| File in the sandbox | After capture |
+| --- | --- |
+| `/app/report.json` — changed | Metadata and captured bytes |
+| `/app/new.txt` — created | Metadata and captured bytes |
+| `/usr/bin/python` — unchanged | Metadata only |
+
+The capture preserves supported files the run created or changed. It is not a complete copy of the base image. Files omitted during capture carry a reason.
+
+### 1. Check which source is available
+
+```bash
+evolve trial files status "$TRIAL_ID"
+```
+
+The state is `live`, `capturing`, `captured`, or `none`.
+
+### 2. Browse and read
+
+```bash
+evolve trial files ls "$TRIAL_ID" /app
+evolve trial files cat "$TRIAL_ID" /app/report.json
+evolve trial files changes "$TRIAL_ID"
+```
+
+Reads use the live sandbox when available, otherwise its capture. Force one with `--source live` or `--source capture`.
+
+### 3. Keep a subtree
+
+```bash
+evolve trial files archive "$TRIAL_ID" --path /app -o archives/
+```
+
+An unchanged image file may be listed but unreadable after capture (`not_captured`). A forced source that is unavailable returns `filesystem_state`.
+
+### Logs and processes
+
+```bash
+evolve trial logs "$TRIAL_ID" --stream agent --follow
+evolve trial procs "$TRIAL_ID"
+```
+
+| Stream | Availability |
+| --- | --- |
+| `agent` | Recorded harness output. |
+| `verifier` | Recorded verifier output. |
+| `system` | When the job enabled `--system-log`. |
+| `setup`, `metrics` | Reserved; currently empty with an explanation. |
+
+Process inspection requires a live sandbox. Analysis runs and task checks expose the same filesystem operations. Published task files have a separate read-only package browser.
+
+**[Filesystem API](/sdk-reference/filesystem)**
+
+Read ranges, search, changes, and events.
+
+**[Trial viewer](/dashboard/trial-viewer)**
+
+Inspect the trace in the browser.