@evolvingmachines/evolve 0.0.65 → 0.0.67-train-harness-wave-2026-09-25.20260926.e00a8a6d

package/skills/publish/SKILL.md

@@ -1,146 +1,170 @@
 ---
 name: publish
-description: Publish a dataset of Harbor-format tasks to Evolve, or upload a finished job folder — a Harbor job, or Evolve SDK runs packed as one. Use when the user wants to publish, upload, or share tasks, datasets/benchmarks, or job results on Evolve.
+description: Publish Harbor-format task datasets or upload existing results to Evolve, including recorded Evolve SDK runs packaged as jobs. Use when the user wants to publish a benchmark, upload tasks, or turn an SDK session into an evaluation job.
 metadata:
   internal: true
 ---
 
-Help the user publish a dataset of tasks to Evolve, or upload a job they ran elsewhere.
-Walk them through each step, checking prerequisites and confirming before running
-commands that upload.
+# Publish to Evolve
 
-## Prerequisites
+Publish tasks for future evaluations, or import results from a run that already happened. Both workflows use Evolve.
 
-1. **The CLI**: `npm install -g @evolvingmachines/evolve`; `evolve --version` succeeds.
+| What the user has | Start here |
+| --- | --- |
+| A folder or repository of tasks | Publish a dataset below. For the task layout, read `evolve skills get create-task`. |
+| A completed job folder or archive | Import results below. |
+| A recorded Evolve SDK session | Read `evolve skills get evals core-concepts/upload-sdk-session` before packaging it. |
 
-2. **API key**: create a key on the dashboard's API keys page
-   (https://dashboard.evolvingmachines.ai/api-keys) and export it. Every command reads
-   `EVOLVE_API_KEY`.
-   ```bash
-   export EVOLVE_API_KEY="<your key>"
-   evolve auth status
-   ```
-   `auth status` prints who the platform thinks you are and which key is in use.
+## Setup, when needed
 
-3. **Task layout**: a dataset is a folder of task directories, each with `task.toml`,
-   `instruction.md`, `tests/test.sh`, and an `environment/` folder unless `task.toml`
-   names a prebuilt `docker_image` (`evolve skills get create-task` has the format). The
-   directory name is the task's name: letters, digits, `.`, `_` and `-`, at most 128
-   characters, starting with a letter or digit; use lowercase (Harbor's convention). A
-   `dataset.toml` manifest at the root is optional.
+Use the installed CLI; install only if missing: `npm install -g @evolvingmachines/evolve`. Confirm authentication with `evolve auth status`. If needed, set `EVOLVE_API_KEY` using a key from https://dashboard.evolvingmachines.ai/api-keys.
 
-## Publishing a dataset
+Publishing or importing is a remote mutation. Proceed when the user's request authorizes it; otherwise prepare the exact command and explain what it uploads. An upload request does not authorize emailing or sharing results. For explicit sharing requests, read `evolve skills get evals core-concepts/sharing`.
 
-What you publish is private to your organization. There are no tags and no visibility
-flag.
+## Publish a dataset
 
-### 1. Check the folder first
+### 1. Check the tasks
+
+Inspect the task directories and any `dataset.toml`, then validate their metadata:
 
 ```bash
-evolve dataset check "<path/to/tasks>"
+evolve dataset check ./tasks
 ```
 
-A dry run: the pre-flight sends each task's `task.toml`, and the `dataset.toml` if there
-is one, to the server, which answers with a verdict per task and writes nothing. A refused
-task names the field to fix. `evolve check "<path/to/tasks>" --watch` goes further: it
-reads each task and, when it can, runs its environment, reference solution and verifier,
-then rules on a rubric.
+This sends each `task.toml` and the optional manifest to Evolve. It does not publish the corpus or run the tasks. Fix refused fields before continuing.
+
+For an agent-based quality review, use `evolve check ./tasks --watch`. This uploads tasks and incurs agent usage. Read the findings and trace: Evolve derives no verdict from the result, and `unknown` on the default rubric's execution criteria is not proof that the reference solution and verifier were executed successfully. See `evolve skills get evals core-concepts/check`.
 
-### 2. Publish
+### 2. Choose one source and publish
 
-From a local directory:
+**Local directory**
 
 ```bash
 evolve dataset publish \
-  --dir "<path/to/tasks>" \
-  --name "<dataset>" \
+  --dir ./tasks \
+  --name my-benchmark \
   --version 1.0 \
   --watch
 ```
 
-When the folder carries a `dataset.toml` manifest, `--name` and `--version` come from it
-and may be omitted. The pre-flight runs automatically before the upload;
-`--skip-preflight` uploads without it, and a task the check would have refused then fails
-at import instead.
+A `dataset.toml` can supply the name and version. Local publishing runs metadata preflight automatically; `--skip-preflight` skips that early check, not import validation.
 
-From a git repository:
+**Git repository**
 
 ```bash
 evolve dataset publish \
-  --git https://github.com/acme/my-swe.git \
+  --git https://github.com/acme/tasks.git \
   --ref v1.0.0 \
-  --name "<dataset>" \
+  --path benchmark \
+  --name my-benchmark \
   --version 1.0 \
   --watch
 ```
 
-`--ref` must be pinned: a tag, or a full 40-character commit sha. A branch name is
-refused. `--path <subfolder>` imports one folder of a larger repository.
+Pin a tag or a full 40-character commit SHA. Branch names are refused. Omit `--path` to use the repository root.
 
-From a source the server fetches itself:
+**Public HTTPS archive**
 
 ```bash
-evolve dataset publish --from hub:cookbook/hello-world --watch
+evolve dataset publish \
+  --from https://example.com/tasks.tar.gz \
+  --name my-benchmark \
+  --version 1.0 \
+  --watch
 ```
 
-`--from` takes a public https tarball URL, or `hub:org/name[@ref]` for a public package on
-the Harbor hub. For a hub package the name and version default to the package's own.
-
-### 3. Follow the publish
+Evolve fetches the archive. Both Git and HTTPS sources require an explicit name and version.
 
-`--watch` follows the publish until the version is `READY` or `FAILED`. Each task builds
-on its own, so one broken task does not block the others; `--watch` ends with how many
-built. If the terminal is gone, re-attach from any machine:
+**Optional: import a public Harbor Hub package into Evolve**
 
 ```bash
-evolve dataset watch "<dataset>"
+evolve dataset publish \
+  --from hub:cookbook/hello-world \
+  --watch
 ```
 
-The version lands `READY` when at least one task built, and `FAILED` only when none did.
-On your own dataset, `READY` also makes the version active, so the bare name runs it.
+`hub:org/name[@ref]` is an accepted source; the destination is Evolve. Name and version may come from the package. Keep this syntax when importing an actual Harbor package.
 
-## After publishing
+For a new dataset, add `--org <organization>` to select an organization. Otherwise, Evolve uses the saved `evolve auth org use` default, then your personal organization. An existing dataset keeps its organization; an explicit or saved default that conflicts is refused. See `evolve skills get evals cli-reference/dataset` for all options.
+
+### 3. Follow the import and inspect the result
+
+`--watch` follows the import and build. To reconnect, use the complete import ID printed by publishing:
 
 ```bash
-evolve dataset show "<dataset>@1.0"                       # versions, tasks, timeouts, providers per task
-evolve run -d "<dataset>@1.0" -a codex -m gpt-5.5 --watch   # run a job on it
+evolve dataset watch "$IMPORT_ID"
+evolve dataset show my-benchmark@1.0
 ```
 
-Each publish creates a version, named `<dataset>@<version>`; a bare name means the active
-version. To point the bare name at a different `READY` version:
+`evolve dataset watch my-benchmark` also finds a live import. Use the import ID after it has settled.
+
+A `READY` version needs at least one built task, but can contain failed tasks. Report the final state, name, version, built and failed counts, and failure reasons. A new ready version on your own dataset becomes active; a bare dataset name selects that active version.
+
+### 4. Run or manage the version
+
+When the user wants an evaluation, select the harness and model, then run the pinned version:
 
 ```bash
-evolve dataset activate "<dataset>" 1.0
+evolve run \
+  --dataset my-benchmark@1.0 \
+  --agent codex \
+  --model "<model>" \
+  --watch
 ```
 
-The owner of a dataset can download the original package back:
+Replace `<model>` with the selected model. Read `evolve skills get evals cli-reference/run` for job settings.
+
+| Need | Command |
+| --- | --- |
+| Point the bare name at another `READY` version | `evolve dataset activate my-benchmark 1.0` |
+| Download the original corpus package as its owner | `evolve dataset download my-benchmark@1.0 -o corpora/` |
+| Fix tasks after a partial build | Publish the corrected corpus under a new version. |
+
+## Import results
+
+### 1. Prepare the job folder
+
+An existing Harbor-format job folder can be uploaded directly. Before packaging another runner's files, read `evolve skills get evals core-concepts/upload` for the required layout and fields. The root needs `config.json` and `result.json`; each trial has its own `result.json`.
+
+For an SDK session, read `evolve skills get evals core-concepts/upload-sdk-session` first. It shows the local log location, field mapping, and packaging script. A session can contain several runs or commands; do not assume one file is one trial. The example script supports one successful Codex or Claude run; the guide covers other cases.
+
+Preserve recorded rewards in each trial's `verifier_result.rewards`. Without recorded rewards, the import has no score. Do not infer a score from agent completion.
+
+### 2. Upload and follow
 
 ```bash
-evolve dataset download "<dataset>@1.0" -o corpora/
+evolve upload ./completed-job -d my-benchmark@1.0
 ```
 
-## Uploading a job you ran elsewhere
+Use the job directory or its `.tar.gz`. Alternatively, use `evolve upload --from <public-https-archive-url>`. Omit `-d` if no published dataset should be linked; when supplied, it links trial tasks by name.
 
-A job run elsewhere, in the Harbor job layout, uploads as a finished job. Its trials,
-traces and rewards become a job you read like any other. Before packing a folder by
-hand — runs made with the Evolve SDK, or any other runner — read what a trial folder
-and its `result.json` must hold: `evolve skills get evals core-concepts/upload`. The
-score lives in `result.json` as `verifier_result.rewards`; without it a trial arrives
-with no score.
+The CLI waits for the import by default. `--no-wait` returns the import record instead. Reconnect with:
 
 ```bash
-evolve upload "<path/to/job-dir>" -d "<dataset>@1.0"
+evolve job imports --status RUNNING
+evolve job import "$IMPORT_ID" --watch
 ```
 
-`upload` takes the job directory, or its `.tar.gz`. With `--from <url>` it takes instead
-a public https URL of the archive, which the server fetches itself. `-d name[@version]`
-links the uploaded trials to a published dataset version by task name. The command
-follows the import until the job exists; `--no-wait` returns at once with the import id.
+### 3. Inspect the imported job
+
+Use the job ID from the completed import:
 
 ```bash
-evolve job imports --status RUNNING
-evolve job import <import-id> --watch
+evolve job show "$JOB_ID"
+evolve job trials "$JOB_ID"
 ```
 
-`job imports` lists your uploads, newest first. `job import --watch` re-attaches to one and
-follows it to the job, or to its typed failure.
+Choose a trial ID from the listing, then read its transcript with `evolve trial trace "$TRIAL_ID"`. Check the task, harness, model, prompt, agent activity, and recorded scores. Report skipped trial folders and missing task links.
+
+Imported jobs support analysis, but cannot be resumed, retried, or regraded. Read `evolve skills get evals cli-reference/upload` for all upload options.
+
+## Use Python or TypeScript instead
+
+Read `evolve skills get evals sdk-reference/index` for client setup, then the matching method reference:
+
+| Operation | Read |
+| --- | --- |
+| Validate, publish, follow, activate, or download datasets | `evolve skills get evals sdk-reference/methods/datasets` |
+| Upload a job and follow its import | `evolve skills get evals sdk-reference/methods/jobs` |
+
+Both SDKs expose `datasets().preflight()` and `datasets().publish()`, plus `jobs().upload()`. Call preflight separately when publishing through an SDK. Publishing and uploading return import records; follow with `watchImport` in TypeScript or `watch_import` in Python. These calls do not convert SDK session logs into job folders; package the session first.

package/skills/rewardkit/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: rewardkit
-description: Write Harbor-format task verifiers using Reward Kit. Use when creating or editing a 
+description: Write verifiers for Evolve's Harbor-format tasks using Reward Kit. Use when creating or editing a
   task's tests/ directory, adding grading criteria, setting up LLM/agent judges, or designing 
   verifiers that produce a reward score.
 metadata:
@@ -11,6 +11,10 @@ Help the user write task verifiers with Reward Kit. Reward Kit is a lightweight
 package that turns a directory of criteria files into a reward score. Each criterion is a 
 Python function call or a TOML judge file; folders become separate rewards.
 
+The package keeps its upstream name, `harbor-rewardkit`. For Evolve's supported
+setup and a complete example, read
+`evolve skills get evals core-concepts/rewardkit`.
+
 ## Setup in a task
 
 Put criteria alongside `test.sh` in the task's `tests/` directory:
@@ -22,17 +26,33 @@ tests/
 └── judge.toml        # optional LLM/agent judge
 ```
 
-`tests/test.sh`:
+Install the package when the image builds, using Python 3.12 or newer. In shared
+mode, add this to `environment/Dockerfile`; in separate mode, install it in the
+verifier image:
+
+```dockerfile
+RUN pip install --no-cache-dir \
+    'harbor-rewardkit==0.2.1'
+```
+
+Then write `tests/test.sh`:
 ```bash
 #!/bin/bash
-uvx --from 'harbor-rewardkit==0.2.*' rewardkit /tests
+set -euo pipefail
+python3 -m rewardkit /tests \
+  --workspace /app \
+  --output /logs/verifier/reward.json
 ```
 
 This runs all criteria in `/tests/` against the workspace at `/app` and writes 
-`/logs/verifier/reward.json`. Defaults match Harbor's conventions — no extra config needed.
+`/logs/verifier/reward.json`. Adjust `--workspace` if the task uses another directory.
 
-Run `evolve check "<task-path>" --watch` to run the verifier against the task's reference 
-solution before publishing the task.
+Run `evolve check "<task-path>" --watch` for task quality review. The checker
+may run the reference solution and verifier when its environment supports
+them; inspect its findings and evidence. The default rubric's execution
+criteria say whether it ran the task; `unknown` on them is not proof that the
+solution or verifier ran. Read `evolve skills get evals core-concepts/check`
+for the result semantics.
 
 If judge criteria need API keys, request them through `task.toml`:
 ```toml
@@ -40,11 +60,16 @@ If judge criteria need API keys, request them through `task.toml`:
 ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"
 ```
 
-On Evolve you never put a real key in the task: write the template exactly as above, as
-the whole value, and the judge's credential is supplied at run time. The rubric names the
-judge model, or Reward Kit's own default applies. A job can override the judge for every
-verifier with `evolve run --ve REWARDKIT_JUDGE=<judge> --ve REWARDKIT_MODEL=<model>`; no
-other verifier env key is accepted on a job.
+Write the template literally as the whole value. Evolve supplies a scoped gateway
+credential and base URL during verification. Set the judge model explicitly in
+the TOML. Use `OPENAI_API_KEY` instead for an OpenAI-family judge.
+
+On a job, `--ve REWARDKIT_JUDGE=<judge>` overrides the judge and
+`--ve REWARDKIT_MODEL=<model>` selects an agent judge's model. These are the only
+two job-level verifier environment overrides. Read
+`evolve skills get evals cli-reference/run` for CLI options or
+`evolve skills get evals sdk-reference/methods/jobs` for the Python and TypeScript
+`start` inputs.
 
 Ask whether Reward Kit should run in the agent's shared environment or in a
 separate verifier environment. Prefer a separate verifier environment when judge
@@ -53,14 +78,14 @@ available to the agent:
 
 ```toml
 [environment]
-network_mode = "no-network"   # Agent env baseline — offline during agent.run()
+network_mode = "no-network"   # Agent can reach the Evolve model gateway
 
 [verifier]
 environment_mode = "separate"
 
 [verifier.environment]
-network_mode = "public"     # Verifier env baseline — LLM judge API calls
-docker_image = "python:3.12-slim"
+network_mode = "no-network"   # Native judge calls use Evolve's gateway
+# With no docker_image, build the verifier from tests/Dockerfile.
 ```
 
 In shared mode, the verifier runs in the agent container and inherits
@@ -69,9 +94,9 @@ baseline is refused at import on Evolve, because a shared verify cannot switch
 egress. If agent and verifier need different network access, use
 `environment_mode = "separate"` and set `[verifier.environment].network_mode`.
 
-Judge criteria that call external APIs need a `public` baseline or allowlist on
-the verifier environment. Programmatic checks that only read local files can use
-`no-network`.
+Evolve's native judge gateway is reachable with `no-network`. Other external
+APIs need a `public` baseline or the required hosts in the verifier allowlist.
+Programmatic checks that only read local files can also use `no-network`.
 
 In separate mode with no `[verifier.environment].docker_image`, `tests/` is the
 verifier image's build context and its `tests/Dockerfile` must provide
@@ -79,6 +104,12 @@ verifier image's build context and its `tests/Dockerfile` must provide
 uploads `tests/` to `/tests` instead. When it pins a distinct image, that image
 boots as it is with nothing uploaded, so it must carry `/tests/test.sh` itself.
 
+For separate verification, declare the files to grade in the task's `artifacts`
+list. The verifier does not receive the agent's whole workspace. Read
+`evolve skills get evals core-concepts/task-verifiers` and
+`evolve skills get evals core-concepts/task-config` for Evolve's file-transfer and
+network rules.
+
 ## Programmatic criteria
 
 Call built-ins from any `.py` file in `tests/`:
@@ -95,6 +126,10 @@ rk.json_key_equals("result.json", "status", "ok")
 All criteria accept `weight` (default `1.0`) and `isolated` (default `False`, runs in 
 overlayfs so side effects don't leak).
 
+Isolation needs working overlayfs support. If the image needs `fuse-overlayfs`,
+install it at build time; do not rely on its runtime installer with restricted
+network access. Confirm the sandbox permits the required mounts.
+
 ### Available built-ins
 
 - **Files**: `file_exists`, `file_not_exists`, `file_contains`, `file_contains_regex`, 
@@ -102,12 +137,17 @@ overlayfs so side effects don't leak).
 - **Commands**: `command_succeeds`, `command_output_contains`, `command_output_matches`, 
   `command_output_matches_regex` (30s default timeout, optional `cwd`)
 - **Data**: `json_key_equals`, `json_path_equals`, `csv_cell_equals`, `xlsx_cell_equals` 
-  (needs `[office]` extra), `sqlite_query_equals`
+  (needs `[documents]` extra), `sqlite_query_equals`
 - **HTTP**: `http_status_equals`, `http_response_contains`
 - **Images**: `image_similarity`, `image_size_equals` (needs `[image]` extra)
 - **Trajectory**: `trajectory_tool_used`, `trajectory_tool_not_used`, `trajectory_turn_count`
 
-For extras, install with `uv tool install harbor-rewardkit[all]`.
+Install needed extras in that same image: `harbor-rewardkit[documents]==0.2.1`
+or `harbor-rewardkit[image]==0.2.1`. Use `[all]` only if both are needed.
+
+Do not depend on downloading packages during verification. Evolve's offline
+Reward Kit bundle is for judge-enabled runs and does not include optional
+extras. Its `uvx` path cannot fetch versions or extras absent from the bundle.
 
 ## Custom criteria
 
@@ -143,7 +183,7 @@ For subjective checks (quality, readability, edge cases), create a TOML file:
 
 ```toml
 [judge]
-judge = "anthropic/claude-sonnet-5"   # LiteLLM model string
+judge = "anthropic/claude-sonnet-4-6"
 files = ["/app/main.py"]
 
 [[criterion]]
@@ -169,7 +209,7 @@ Agent judges shell out to a CLI and can explore the filesystem:
 ```toml
 [judge]
 judge = "claude-code"
-model = "anthropic/claude-sonnet-5"
+model = "anthropic/claude-sonnet-4-6"
 isolated = true
 
 [[criterion]]
@@ -179,6 +219,10 @@ type = "binary"
 
 Slower and more expensive than LLM judges, but they can run commands and inspect files.
 
+Evolve supports `claude-code` and `codex` agent judges. Install the selected CLI
+in the verifier image and request its matching provider credential. The managed
+judge path does not supply verifier MCP tools or arbitrary provider credentials.
+
 ### Useful `[judge]` options
 
 `timeout` (default 300), `reasoning_effort` (`low`|`medium`|`high`), `reference` (path to 
@@ -289,8 +333,15 @@ weight unless that aggregation's inline map overrides them;
 - `/logs/verifier/reward.json` — per-reward scores
 - `/logs/verifier/reward-details.json` — per-criterion results, judge reasoning, errors
 
+Evolve uses `reward` as the primary score, or the only value when there is one
+key. Multiple dimensions without `reward` remain metrics but have no primary
+score. Keep the primary score in `[0, 1]`.
+
 ## Multi-step tasks
 
+Evolve currently requires shared verification for multi-step tasks; separate
+verifier environments are rejected.
+
 In a multi-step task, each step has its own `tests/` under
 `steps/{name}/tests/`, and the verifier runs once per step. Reward Kit behaves
 the same as in a single-step task: for each step it reads `/tests`, runs the
@@ -311,6 +362,9 @@ separate reward key for that step, and `multi_step_reward_strategy = "mean"`
 averages each key across steps. Use `"final"` when the last step is an
 end-to-end check whose rewards already represent the full task.
 
+Read `evolve skills get evals core-concepts/multi-step` for Evolve's supported
+step configuration and result layout.
+
 ## When to reach for what
 
 - **Use built-ins** for file existence, string matches, command output, JSON/CSV checks, 
@@ -326,5 +380,6 @@ end-to-end check whose rewards already represent the full task.
 
 ## Working example
 
-See https://github.com/laude-institute/harbor/tree/main/examples/tasks/reward-kit-example.
-Reward Kit's own documentation: https://docs.harborframework.com/core-concepts/rewardkit/quick-start.
+Read `evolve skills get evals core-concepts/rewardkit` for a complete verifier,
+or `evolve skills get evals getting-started/first-task` for the task's build,
+publish, check, and evaluation workflow.