@evolvingmachines/evolve 0.0.65 → 0.0.66

package/skills/create-task/SKILL.md

@@ -12,8 +12,16 @@ Guide the user through creating a new task end-to-end. Don't just dump commands
 walk them through each decision, especially around the verifier (which is usually the 
 hardest part).
 
-The task format is Harbor's, and Evolve runs it unchanged. The full specification is at 
-https://docs.harborframework.com/core-concepts/tasks/overview.
+Evolve accepts Harbor-format tasks subject to managed-platform support. Read
+`evolve skills get evals core-concepts/tasks` and
+`evolve skills get evals core-concepts/task-config` for supported behavior. The upstream
+format specification is at https://docs.harborframework.com/core-concepts/tasks/overview.
+
+Before the hosted check or publish steps, confirm `evolve --version` and
+`evolve auth status` succeed. If needed, install the CLI with
+`npm install -g @evolvingmachines/evolve` and set `EVOLVE_API_KEY` from the
+dashboard's API keys page. Read `evolve skills get evals getting-started/installation`
+for setup.
 
 ## Step 1: Create the task directory
 
@@ -36,9 +44,6 @@ The files to write, one per step below:
 The directory name is the task's name on the platform: letters, digits, `.`, `_` and `-`, 
 at most 128 characters, starting with a letter or digit; use lowercase (Harbor's convention).
 
-Where Harbor is installed, `harbor task init "<org>/<task-name>"` produces the same layout 
-(optional).
-
 If the user wants a **multi-step task** (ordered steps with per-step
 instructions, tests, and early stopping against a shared container), write
 the single-step layout first, then convert to the `steps/` layout described in
@@ -68,7 +73,7 @@ Edit `environment/Dockerfile` to install dependencies the task needs. The agent
 inside this container.
 
 ```dockerfile
-FROM ubuntu:24.04
+FROM python:3.12-slim
 WORKDIR /app
 
 # Install what the task requires — NOT the solution
@@ -77,7 +82,8 @@ RUN apt-get update && apt-get install -y openssh-client && rm -rf /var/lib/apt/l
 
 For multi-container setups, use `environment/docker-compose.yaml` instead. After 
 publishing, `evolve dataset show "<dataset>@<version>"` prints which sandbox providers can 
-run each task.
+run each task. Read `evolve skills get evals core-concepts/compose` for Evolve's
+supported Compose layout and limits.
 
 **Test the environment interactively** before writing the solution or tests:
 ```bash
@@ -105,18 +111,23 @@ For a separate verifier container with no pinned `[verifier.environment] docker_
 `tests/` is the verifier image's build context and its `tests/Dockerfile` must provide
 `/tests/test.sh`. A verifier that pins the task's own image gets `tests/` uploaded to
 `/tests`; one that pins a distinct image boots as it is with nothing uploaded, so that
-image must carry `/tests/test.sh` itself. A separate verifier judges only what the task
-lists under a top-level `artifacts = ["/app/out.json"]` in `task.toml`, never the agent's
-whole workspace.
+image must carry `/tests/test.sh` itself. Declare the files the verifier needs under
+top-level `artifacts`; Evolve also transfers `/logs/artifacts/` and the collection
+script's patch file, but not the agent's whole workspace.
 
 ```toml
+artifacts = ["/app/out.json"]
+
 [verifier]
 environment_mode = "separate"
 
 [verifier.environment]
-docker_image = "ubuntu:24.04"
+workdir = "/app"
 ```
 
+This configuration builds `tests/Dockerfile`. Read
+`evolve skills get evals core-concepts/task-verifiers` for the complete setup.
+
 ### Option A: Reward Kit (recommended for most cases)
 
 Use when the verifier has multiple criteria, needs partial credit, uses an LLM/agent 
@@ -128,15 +139,25 @@ Good fit signals:
 - Want partial credit rather than pass/fail
 - Want to compose built-ins like `file_contains`, `command_succeeds`, `json_key_equals`
 
-`tests/test.sh`:
+Install Reward Kit in the image that runs verification: the task image for shared
+mode, or the verifier image for separate mode. It requires Python 3.12 or newer.
+
+```dockerfile
+RUN pip install --no-cache-dir 'harbor-rewardkit==0.2.1'
+```
+
+`tests/test.sh` (for a workspace at `/app`):
 ```bash
 #!/bin/bash
-uvx --from 'harbor-rewardkit==0.2.*' rewardkit /tests
+set -euo pipefail
+python3 -m rewardkit /tests \
+  --workspace /app \
+  --output /logs/verifier/reward.json
 ```
 
-Note: the package is named `harbor-rewardkit` but the executable is `rewardkit`,
-hence `--from 'harbor-rewardkit==0.2.*' rewardkit`. Running
-`uvx harbor-rewardkit` directly will fail.
+The package name remains `harbor-rewardkit`; the Python module is `rewardkit`.
+Install any required extras in that image too. Read
+`evolve skills get evals core-concepts/rewardkit` for dependencies and managed judges.
 
 Then add `tests/checks.py` and/or `tests/judge.toml`. Read `evolve skills get rewardkit` to 
 design the criteria.
@@ -145,16 +166,20 @@ design the criteria.
 
 Use when the verification is straightforward assertion-style Python.
 
+Install pytest in the image that runs verification, rather than downloading it
+in `test.sh`. For a Python image:
+
+```dockerfile
+RUN pip install --no-cache-dir 'pytest==8.4.1'
+```
+
 `tests/test.sh`:
 ```bash
 #!/bin/bash
-apt-get update && apt-get install -y curl
-curl -LsSf https://astral.sh/uv/0.9.7/install.sh | sh
-source $HOME/.local/bin/env
-
-uvx --with pytest==8.4.1 pytest /tests/test_outputs.py
+set -uo pipefail
+mkdir -p /logs/verifier
 
-if [ $? -eq 0 ]; then
+if python3 -m pytest /tests/test_outputs.py; then
   echo 1 > /logs/verifier/reward.txt
 else
   echo 0 > /logs/verifier/reward.txt
@@ -183,21 +208,25 @@ fi
 
 ### Reward file format (all options)
 
-- `/logs/verifier/reward.txt` — single number (usually `0` or `1`)
-- `/logs/verifier/reward.json` — `{"accuracy": 0.95, "runtime_sec": 1.2}` for multiple metrics
+- `/logs/verifier/reward.txt` — one finite number from `0` to `1`
+- `/logs/verifier/reward.json` — `{"reward": 0.95, "runtime_sec": 1.2}` for multiple metrics
 
-When both exist, `reward.json` wins. Everything the script prints is kept as the verifier log.
+A nonempty `reward.json` wins when both exist. Its primary score is the `reward`
+key, or the only key in a one-key object, and must be in `[0, 1]`. Multiple keys
+without `reward` produce metrics but no primary score. Everything the script
+prints is kept as the verifier log.
 
 **Always use absolute paths in `test.sh`.**
 
 ## Step 5: Write the solution
 
-Write `solution/solve.sh` — a script that actually solves the task. `evolve check` runs 
-it to confirm the task is solvable and the tests pass on a correct solution. The agent is 
-never given it.
+Write `solution/solve.sh` — a script that actually solves the task. The checker
+reviews it and can run it when its sandbox supports the task's environment.
+Evolve does not give it to the evaluated agent.
 
 ```bash
 #!/bin/bash
+mkdir -p ~/.ssh
 ssh-keygen -t rsa -f ~/.ssh/id_rsa -N ""
 ```
 
@@ -215,8 +244,8 @@ description = "One-line description"
 keywords = ["jax", "mnist", "rewardkit"]  # 3–8 lowercase tokens: domain, verifier style, hardware
 
 [metadata]
-difficulty = "easy" | "medium" | "hard"
-category = "programming" | "machine-learning" | "gpu" | ...
+difficulty = "easy"
+category = "programming"
 tags = ["..."]
 
 [agent]
@@ -247,8 +276,8 @@ Network access has two layers:
 | --- | --- | --- |
 | `[environment].network_mode` | Baseline | Agent env start; shared verifier uses this too |
 | `[verifier.environment].network_mode` | Baseline | Separate verifier env start |
-| `[agent].network_mode`, `[steps.agent].network_mode` | Override | During matching `agent.run()` |
-| `[verifier].network_mode`, `[steps.verifier].network_mode` | Override | During matching `verify()` |
+| `[agent].network_mode` | Override | During the agent phase |
+| `[verifier].network_mode` | Override | Separate verifier policy; shared mode must match the baseline |
 
 Modes: `public`, `no-network`, or `allowlist` with `allowed_hosts = ["pypi.org"]`
 (exact hostnames, IPv4/IPv6 address literals or CIDR ranges, or leading wildcard hostnames, when supported by the selected environment; not URLs,
@@ -288,7 +317,9 @@ environment_mode = "separate"
 network_mode = "public"   # Verifier baseline — not a phase override
 ```
 
-Full reference: https://docs.harborframework.com/core-concepts/tasks/network-policies.
+Evolve rejects per-step network overrides. Read
+`evolve skills get evals core-concepts/task-config` for network access and
+`evolve skills get evals core-concepts/compatibility` for supported task features.
 
 For Reward Kit judges needing API keys:
 ```toml
@@ -301,19 +332,37 @@ the whole value, and the judge's credential is supplied at run time.
 
 ## Step 7: Check the task
 
+First validate the task metadata, then request the hosted rubric review:
+
 ```bash
+evolve dataset check "<task-path>"
 evolve check "<task-path>" --watch
 ```
 
-The check reads the task and, when it can, runs the environment, `solution/solve.sh` and
-the verifier, then rules on every criterion of a rubric (eleven by default); `executed` in
-the result says whether it ran the task. `evolve check show <check-id>` prints one entry
-per criterion, with an `outcome`, an `explanation` and `evidence`, and one label per task:
-`has_a_problem`, `unclear` or `no_problem_found`.
+Set `CHECK_ID` to the parent check ID returned by `evolve check`, then read it:
 
-Where Harbor is installed, `harbor run -p "<task-path>" -a oracle` runs the solution and 
-the verifier locally (optional). Its reward should be `1.0`. If it's not, debug in this order:
-1. Does `solve.sh` actually solve it? (run it by hand inside `docker run --rm -it "<task-name>" bash`)
+```bash
+evolve check show "$CHECK_ID"
+```
+
+`evolve dataset check` sends `task.toml` and any `dataset.toml` to the API for
+validation. It does not upload the full task package or execute the task.
+`evolve check` runs a paid checker agent. It reviews the files against the
+default criteria (`evolve check --show-defaults`) and can run the environment,
+solution, and verifier when its sandbox supports them. It is not a dedicated
+reference-solution runner; the Evolve CLI has no standalone oracle command.
+
+Read each criterion's `outcome`, `explanation`, and `evidence`. The result is
+that per-criterion JSON; Evolve derives no verdict from it. The default rubric's execution
+criteria say whether the checker ran the task: `unknown` on them is not proof
+that the solution or verifier ran. Inspect the evidence and checker trace;
+unresolved execution needs a test in an environment that can run the task.
+
+Read `evolve skills get evals core-concepts/check` and
+`evolve skills get evals cli-reference/check` for results, traces, and custom rubrics.
+
+When a solution or verifier fails, debug in this order:
+1. Does `solve.sh` actually solve it in the task's intended environment?
 2. Does the verifier correctly detect success? (check `/logs/verifier/` output)
 3. Are paths correct? (absolute vs relative)
 4. Are dependencies installed in the Dockerfile?
@@ -329,7 +378,12 @@ evolve run -d "<dataset>@1.0" -a codex -m gpt-5.5 --watch
 ```
 
 If the task is too easy (every model 1.0) or impossible (every model 0.0), consider 
-adjusting difficulty. `evolve skills get publish` covers every publish option.
+adjusting difficulty. `evolve skills get publish` explains the workflow;
+`evolve skills get evals cli-reference/dataset` covers every dataset publish option.
+For TypeScript or Python automation, load
+`evolve skills get evals sdk-reference/methods/datasets`,
+`evolve skills get evals sdk-reference/methods/checks`, and
+`evolve skills get evals sdk-reference/methods/jobs`.
 
 ## Step 9: Write README.md (always the final step)
 
@@ -362,12 +416,10 @@ persist across steps.
 Replace the task-root `instruction.md`, `tests/`, and `solution/` with a
 `steps/` directory containing one sub-directory per step:
 
-Each `[[steps]].name` must match one directory name of at most 255 UTF-8 bytes,
-unique after case folding and Unicode normalization. Avoid path separators,
-control characters, Windows-reserved characters/device names, and trailing dots
-or spaces. Keep all task inputs and linked contents within the task directory;
-validation permits shared links inside the task. Use regular files and directories
-for shared inputs when publishing tasks.
+Each `[[steps]].name` must match a directory under `steps/`. Names must be unique
+and use 1–128 ASCII letters, digits, `.`, `_`, or `-`, starting with a letter or
+digit. Keep task inputs inside the task directory as regular files and
+directories. Copy shared inputs into it; Evolve's importer rejects symbolic links.
 
 ```
 <task-name>/
@@ -401,15 +453,15 @@ with cwd = WORKDIR. Non-zero exit aborts the step and the trial. Have it
 ```toml
 schema_version = "1.4"
 
-[task]
-name = "<org>/<task-name>"
-version = "1.0.0"
-
 # How per-step rewards roll up into the trial-level verifier_result.
 # "mean" (default): per-key mean across steps that produced a result.
 # "final": the last step's verifier_result verbatim.
 multi_step_reward_strategy = "mean"
 
+[task]
+name = "<org>/<task-name>"
+version = "1.0.0"
+
 [[steps]]
 name = "scaffold"              # Must match the directory under steps/
 min_reward = 1.0               # Abort trial if this step's reward < 1.0
@@ -435,11 +487,12 @@ timeout_sec = 60.0
 timeout_sec = 30.0
 ```
 
-Per-step overrides available: `agent.timeout_sec`, `agent.user`,
-`agent.network_mode`, `verifier.timeout_sec`, `verifier.env`, `verifier.user`,
-`verifier.network_mode`, `verifier.environment_mode`, `verifier.environment`,
-`steps.verifier.environment.network_mode`, `healthcheck.*`, `artifacts`. Unset
-fields fall back to the task-level values.
+Evolve supports per-step agent and verifier timeouts, `verifier.env`, and
+healthchecks. Timeouts and verifier variables fall back to task-level values.
+Set the environment, agent user, and network policies once for the task.
+Multi-step tasks require shared verification. Compose, per-step agent users,
+network policies, verifier environments, and collection hooks are rejected.
+Step verifiers run as root; a different verifier user is rejected.
 
 ### Choosing a reward strategy
 
@@ -452,20 +505,25 @@ fields fall back to the task-level values.
 
 ### Artifacts
 
-Step-level `artifacts` are collected into `steps/{name}/artifacts/` after that
-step's verification. Task-level and trial-level artifacts are collected at
-every step in addition to the step-level ones.
+Evolve accepts step-level `artifacts` in the task configuration, but does not
+currently collect per-step artifact snapshots. Do not rely on
+`steps/{name}/artifacts/` being present. Step results and verifier logs are
+recorded separately; see `evolve skills get evals core-concepts/multi-step`.
 
 ### Checking a multi-step task
 
-Where Harbor is installed, `harbor run -p "<task-path>" -a oracle` runs each step's
-`solution/solve.sh`, then each step's verifier, in order (optional). Trial reward
-should be `1.0` across the aggregation strategy. Then publish and run it (Step 8).
+Use the metadata check and hosted review in Step 7. Inspect evidence for each
+step's solution and verifier; the review does not guarantee their execution.
+Then publish and run with an evaluated agent (Step 8). Inspect `step_results`
+and per-step verifier logs, including any threshold that stopped later steps.
 
 ### Full reference + worked example
 
-- Docs: https://docs.harborframework.com/core-concepts/tasks/multi-step
-- Example task: https://github.com/laude-institute/harbor/tree/main/examples/tasks/hello-multi-step-advanced
+- Evolve behavior: `evolve skills get evals core-concepts/multi-step`
+- Optional upstream example: https://github.com/laude-institute/harbor/tree/main/examples/tasks/hello-multi-step-advanced
+
+The upstream example illustrates the format. Apply Evolve's supported options
+above; upstream execution commands do not define the managed workflow.
 
 ## Special features (mention if relevant)
 
@@ -479,7 +537,7 @@ should be `1.0` across the aggregation strategy. Then publish and run it (Step 8
 
 ## Common pitfalls
 
-- Forgetting to write the reward file → task "passes" silently with reward 0
+- Forgetting to write the reward file → trial is `INDETERMINATE`, with no score
 - Using relative paths in `test.sh` → breaks when the verifier runs it from a different cwd
 - Installing the solution into the Dockerfile → agent already gets the answer
 - Test script leaks into `instruction.md` → agent sees the rubric and gaming becomes trivial

package/skills/evolve/SKILL.md

@@ -4,37 +4,54 @@ description: "Evolve runs agent evaluations in the cloud: any model on any codin
 allowed-tools: Bash(evolve:*), Bash(npx evolve:*)
 ---
 
-# evolve
+# Evolve
 
-Hosted evals and the SDK, from the `evolve` command.
+Run evaluations in the cloud and inspect the results. This is the entry point; the versioned manual ships inside the CLI.
 
-Install: `npm i -g @evolvingmachines/evolve`, then `export EVOLVE_API_KEY=<your key>` (create a key at https://dashboard.evolvingmachines.ai/api-keys).
+## Read the relevant page first
 
-Python SDK: `pip install evolvingmachines-evolve` (`import evolve`). The manual still comes from the `evolve` command.
+```bash
+evolve skills get evals                       # documentation index
+evolve skills get evals cli-reference/run     # one focused page
+evolve run --help                            # installed command options
+```
+
+Choose pages from the index. Load `--full` only when the entire reference is needed.
+
+| Task | Starting page |
+| --- | --- |
+| First evaluation | `getting-started/quick-start` |
+| Run or configure a job | `cli-reference/run` |
+| Inspect, compare, stop, or retry | `cli-reference/job` |
+| Read files, logs, or processes | `cli-reference/filesystem` |
+| Check task quality | `cli-reference/check` |
+| Analyze completed traces | `cli-reference/analyze` |
+| Publish a dataset | `cli-reference/dataset` |
+| Import completed results | `cli-reference/upload` |
+| Upload a recorded SDK run | `core-concepts/upload-sdk-session` |
+| Python or TypeScript client | `sdk-reference/index` |
 
-## Start here
+For example: `evolve skills get evals cli-reference/job`.
 
-This file is a pointer, not the manual. The manual ships inside the CLI and always matches the installed version. Keep it current: `npm i -g @evolvingmachines/evolve@latest`, then `evolve skills install --force` refreshes this file. Before running any `evolve` command, load it:
+## Task-authoring skills
 
 ```bash
-evolve skills get evals                        # the index of the documentation: every page, one line each
-evolve skills get evals core-concepts/tasks    # one page, by its site path
-evolve skills get evals --full                 # every page at once; only when everything is needed
+evolve skills get create-task      # write a Harbor-format task
+evolve skills get rewardkit        # write its verifier
+evolve skills get create-adapter   # convert a benchmark
+evolve skills get publish          # publish tasks or import results
 ```
 
-Read the index first, then the page for your topic, then write the command. Every verb also answers `evolve <verb> --help`.
+`evolve skills list` shows the installed bundle. Plural `skills` reads this manual; singular `skill` uploads content for evaluated agents.
 
-## The other skills
+## Setup, when needed
 
-```bash
-evolve skills get create-task      # write a new task in the Harbor format, verifier included
-evolve skills get rewardkit        # write a task's verifier with Reward Kit
-evolve skills get create-adapter   # convert an existing benchmark into a folder of Harbor-format tasks
-evolve skills get publish          # publish a dataset of tasks, or upload a job you ran elsewhere (a Harbor job, or SDK runs packed as one)
-```
+CLI: `npm install -g @evolvingmachines/evolve`. Set `EVOLVE_API_KEY` using a key from https://dashboard.evolvingmachines.ai/api-keys; confirm with `evolve auth status`.
+
+Python: `pip install evolvingmachines-evolve` (`import evolve`). Managed evals references still come from `evolve skills get evals`.
 
-`evolve skills list` names everything the installed version serves; `--full` on any skill adds its pages. The SDK skill (run agents in sandboxes from TypeScript or Python) is not served by the command; install it from the repository: `npx skills add evolving-machines-lab/evolve --skill evolve-agents`.
+To install this pointer into an agent's skills directory, use `evolve skills install --target <agent>`. Do not reinstall or update the CLI merely to read its manual.
 
-## Inside the evolve repository
+The separate managed-agents SDK skill installs from the repository: `npx skills add evolving-machines-lab/evolve --skill evolve-agents`. It is not served by `evolve skills`.
 
-An agent working in a checkout of https://github.com/evolving-machines-lab/evolve reads the same content directly from `skills/<name>/SKILL.md` (`evolve-evals` is the evals index, its pages under `references/`); nothing needs to be installed.
+Inside an Evolve checkout, read `skills/evolve-evals/SKILL.md` and its `references/` directly; no installation is needed.