@evo-hq/pi-evo 0.4.4-alpha.6 → 0.5.0-alpha.3

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@evo-hq/pi-evo",
-  "version": "0.4.4-alpha.6",
+  "version": "0.5.0-alpha.3",
   "description": "Evo plugin for pi-coding-agent: optimize/discover/subagent skills + mid-run inject extension.",
   "publishConfig": {
     "access": "public"

package/skills/discover/SKILL.md

@@ -2,7 +2,7 @@
 name: discover
 description: Initialize evo for the current repository by exploring the codebase, proposing unexplored optimization dimensions, constructing the benchmark inside a baseline worktree, and running the first experiment. Use when the user invokes /evo:discover, mentions setting up evo, wants to instrument a codebase for autonomous optimization, or asks to start a new evo run on a project.
 argument-hint: <optional context about what to optimize>
-evo_version: 0.4.4-alpha.6
+evo_version: 0.5.0-alpha.3
 ---
 
 # Discover
@@ -40,20 +40,20 @@ evo --version
 The output must be exactly:
 
 ```
-evo-hq-cli 0.4.4-alpha.6
+evo-hq-cli 0.5.0-alpha.3
 ```
 
 Three outcomes:
 
 1. **Matches exactly** — continue to step 1.
 2. **Reports a different version** (`evo-hq-cli 0.4.2`, etc.) — the host refetched a newer/older skill bundle than the CLI on PATH. Drift breaks skills silently. Stop and tell the user:
-   > Your installed evo CLI is on a different version than this skill (`0.4.4-alpha.6`). Run:
+   > Your installed evo CLI is on a different version than this skill (`0.5.0-alpha.3`). Run:
    > ```
-   > uv tool install --force evo-hq-cli==0.4.4-alpha.6
+   > uv tool install --force evo-hq-cli==0.5.0-alpha.3
    > ```
    > Then re-invoke this skill.
 3. **`command not found`, or reports a different package** (commonly `evo 1.x` — the unrelated SLAM tool) — the CLI isn't installed. Tell the user:
-   > `evo-hq-cli` isn't on your PATH. Install it: `uv tool install evo-hq-cli==0.4.4-alpha.6` (or `pipx install evo-hq-cli==0.4.4-alpha.6`). Then re-invoke this skill.
+   > `evo-hq-cli` isn't on your PATH. Install it: `uv tool install evo-hq-cli==0.5.0-alpha.3` (or `pipx install evo-hq-cli==0.5.0-alpha.3`). Then re-invoke this skill.
 
 Do not try to auto-install. Host sandbox + network policy may block it; leaving the install as a user action keeps failure modes clear.
 
@@ -180,7 +180,8 @@ build/
 evo init --name "<short project name>" \
   --target <file> --benchmark "<command using {worktree} and {target}>" --metric <max|min> \
   --host <claude-code|codex|opencode|openclaw|hermes|pi|generic> \
-  --instrumentation-mode <sdk|inline> [--gate "<gate command>"] \
+  --instrumentation-mode <sdk|inline> \
+  --per-exp-timeout <seconds> [--gate "<gate command>"] \
   [--commit-strategy <all|tracked-only>]
 ```
 
@@ -188,6 +189,8 @@ evo init --name "<short project name>" \
 
 **`--name` should be a short human-readable project label** for dashboard display, chosen from the repository/product context. Existing workspaces without a name fall back to the repo directory name; do not hand-edit config just to migrate them.
 
+**`--per-exp-timeout` is required.** Wall-clock seconds for each `evo run` invocation. Becomes the workspace default; override per-call with `evo run --timeout N`. Pick based on what the benchmark actually costs end-to-end on this hardware -- if you don't know yet, time the benchmark once locally and use ~2x that. Typical ranges: a unit-test-style benchmark is 300-900s; a small-model SFT + eval cycle is 1800-3600s; a large-model train run is several hours. Set conservatively -- a too-tight value kills experiments mid-flight; a too-loose value wastes budget only when something actually hangs. Update later with `evo config set per-exp-timeout <seconds>`.
+
 **`--commit-strategy` is optional.** Default is `all`. Override with `--commit-strategy tracked-only` only when you want the stricter shisa-kanko flow where new files must be staged explicitly and acknowledged at `evo run` time.
 
 **Placeholder semantics.** Benchmark and gate commands support two placeholders, resolved lazily at run time by `evo run` / gate evaluation:
@@ -205,7 +208,8 @@ evo init \
   --target agent/solve.py \
   --benchmark "python3 {worktree}/benchmark.py --target {target}" \
   --metric max \
-  --host claude-code
+  --host claude-code \
+  --per-exp-timeout 1800
 ```
 
 Use the same runtime entry point the project already uses, but make sure the command does not assume uncommitted runtime state exists inside the worktree. Worktrees are git checkouts; untracked directories such as local virtualenvs, build caches, and downloaded models are usually not present there. If the benchmark needs setup or a package-manager runner, configure evo's runtime recipe instead of baking local paths into the benchmark command:
@@ -225,6 +229,14 @@ Dashboard live: http://127.0.0.1:8080 (pid 12345)
 
 **Relay that line back to the user verbatim.** If port 8080 is busy, evo auto-increments -- show whatever port prints. The URL is how the user watches the run.
 
+**Benchmark commands must be eval-only.** Do NOT wrap training and evaluation into a single benchmark command. If your benchmark command runs training before scoring, every gate revalidation and every `evo run --check` retrains from scratch, and the experiment budget burns on duplicated training instead of new experiments. Training is a separate step the agent invokes BEFORE `evo run`:
+
+1. The agent makes changes (data curation, hyperparameter selection, technique choice, training code edits).
+2. The agent runs training to produce a checkpoint at the experiment worktree's `final_model/` (or wherever the technique's recipe in `evo:finetuning/references/glue.md` specifies).
+3. THEN `evo run <exp_id>` invokes the registered benchmark command, which loads the produced checkpoint and emits a score.
+
+The registered benchmark command should call `evaluate.py`, `run_eval.py`, or equivalent -- NOT `train.py`. If the project's only existing evaluation tool runs train+eval together with no eval-only mode, wrap it: add a `--skip-train` flag, or have the wrapper detect an existing checkpoint at `final_model/` and short-circuit the train step. Without this, evo's gate-recheck and re-score mechanics retrain repeatedly and the budget evaporates.
+
 **Runtime environment.** If the benchmark needs keys or other runtime variables, configure them through evo rather than copying `.env` into worktrees or hand-editing `config.json`:
 
 ```bash
@@ -318,7 +330,31 @@ Paths below are relative to this `SKILL.md` file (resolve them against the skill
   - Benchmark is Python or Node: paste `references/inline_instrumentation.py` (or `.js`) and call `log_task` / `logTask` per task, `write_result` / `writeResult` once at the end.
   - Benchmark is any other language: port the contract from `references/instrumentation-contract.md` directly into that language (~10-15 lines: read the env vars, write each task trace, atomically publish the result). Do not add a Python/JS wrapper around it.
 
-### 10d. Cheap validation run
+**Per-task emission is load-bearing.** If your benchmark evaluates N independent items (per-question math, per-test-case unit tests, per-document QA, per-sample reasoning trace), emit ONE `log_task` / `report` / trace file PER ITEM -- not one aggregate. Include the item's input, expected output, model output, and any per-item metadata as extras; that detail is what the dashboard's per-task panel + the verifier's reproducibility spot-check + the ideator's failure-clustering all rely on. Wrappers that compute the average score themselves and emit a single aggregate task entry look like they work but lose every diagnostic capability evo provides. The reference files have explicit USAGE EXAMPLES showing the per-item loop AND an ANTI-PATTERN block showing what NOT to do (see `references/inline_instrumentation.{py,js}` and `references/sdk_{python,node}.{py,js}`). Follow them. Single-aggregate emission is only valid when the benchmark really is one indivisible measurement (one e2e workflow, one perf number) -- and even then, attach every observable as extras.
+
+**Runner-library wrappers are the common failure mode.** When the selected benchmark wraps a runner library (e.g. `inspect_evals`, `lm-eval-harness`, `evals`), the per-item loop is hidden inside the runner. The wrapper script's natural shape is to call the runner, read its aggregate output, and write a single `{"score": X}` to `result.json`. This is the anti-pattern. **Even when the runner library handles the per-sample loop internally, the wrapper script MUST parse the runner's per-sample output JSON and emit one `log_task(item_id, score=..., extras={...})` per item.** The runner typically already writes per-sample data to disk (most of these libraries do — `inspect_evals` writes per-sample logs, `lm-eval-harness` writes per-doc records); the wrapper just hasn't been forwarding it to evo. Without per-item forwarding, the dashboard's Tasks tab is empty, the verifier can't spot-check, and the agent has no way to diagnose WHICH items the model fails on — which is required for any RL-on-failures or curriculum strategy.
+
+**Write traces for the future reader.** The only consumers of a committed experiment's traces look BACK at them -- a future orchestrator picking the next move, a verifier auditing for false progress, an ideator clustering failure modes. A trace that records `{score: 0}` and nothing else is unrecoverable: the future reader cannot tell whether the model produced wrong reasoning, unparseable format, or no output at all. Two rules of thumb:
+
+- **`log_task` extras carry context** -- the item's input, expected output, the model's actual output (or first ~500 chars of it), parse outcome, any error. Cost: a few KB per task. Payoff: diagnosis is possible later without re-running the eval.
+- **`evo annotate <exp_id>` is the human-readable summary** -- one or two factual lines after a run commits: which data sources were used, what the score actually represented, the failure mode if any. Annotations get loaded into future orchestrator decisions and ideator briefs, so write them as facts ("model emitted `\boxed{}`, eval prompt requested `ANSWER:`, format mismatch suspected"), not as plans or feelings.
+
+### 10d. Audit with benchmark-reviewer (mandatory before baseline)
+
+Before `evo run` is invoked for the first time, audit the harness via the bundled subagent:
+
+```
+Task(subagent_type="evo:benchmark-reviewer",
+     prompt="workspace=<absolute path to dir containing .evo/>\nbenchmark_command=<the literal --benchmark string from evo init>\nunit=<one-line description of an item, e.g. 'AIME problem', 'HumanEval task', 'BFCL turn'>")
+```
+
+The subagent returns a JSON report with `passed`, `findings[]`. Each finding has `severity: block|warn|note`.
+
+**Gate the baseline on this.** If `passed: false`, address every `block` finding before re-invoking the reviewer. Do **not** proceed to `evo run` until the report comes back clean. Typical `block` findings: aggregate-only emission (most common -- see step 10c), training source that overlaps the held-out set, no real gate registered for a constructed benchmark, harness silently writes `{"score": 0.0}` on error instead of crashing.
+
+`warn` and `note` findings are informational -- record them in `.evo/project.md` and proceed.
+
+### 10e. Cheap validation run
 
 Before the full baseline, validate the toolchain with the cheapest possible end-to-end run (single task, smallest split, dry-run flag -- whatever is fastest). Run the check from the main repo root:
 
@@ -339,7 +375,7 @@ This is the authoritative wiring check, and it is language-agnostic -- it runs t
 
 Fix any issues and re-validate before proceeding.
 
-### 10e. Commit inside the worktree
+### 10f. Commit inside the worktree
 
 Logical commits are ideal but not required. Minimal acceptable:
 
@@ -405,9 +441,33 @@ End the skill by reporting in chat:
 - The baseline experiment ID and score
 - The chosen optimization dimension and why
 - A one-liner on next steps: "Run `/evo:optimize` to start the optimization loop."
+
+**Do not run experiments outside `/evo:optimize`.** Even if the workspace's resource profile forces serial execution (e.g. exclusive-GPU, width 1), you still go through `/evo:optimize` with `subagents=1`. The optimize loop's value isn't just parallelism -- it's the structured loop around every experiment (scan-subagent cross-cutting analysis, brief writing, verifier pre/post hooks, ideator spawning on stall, frontier reconciliation). Bypassing optimize to "drive experiments directly" loses all of that and reverts to ad-hoc iteration. If you are tempted to skip optimize because the workload is serial, read `plugins/evo/skills/optimize/SKILL.md` for how to configure it for serial work -- the answer is `subagents=1`, NOT "don't run optimize."
+
 - **Resume after crash:** if the host, the shell, or the machine restarts mid-flow, re-invoke `evo:optimize`. Evo reads `.evo/` and resumes from the last committed experiment -- no special restore procedure.
 - **State is local to this machine:** experiment commits on branches like `evo/run_0000/exp_*` survive `git push --all`, but orchestration state (graph, annotations, project notes) lives only in `.evo/`. If that history matters to you, back up `.evo/` separately (e.g., `tar -czf evo-state-$(date +%F).tar.gz .evo/`).
 
+## Polling discipline
+
+When waiting on a long-running background process (training, evaluation, batch generation, any externally-spawned long task), do NOT use `while true; do sleep N; tail file; done`. That loop never exits when the underlying process crashes -- the tail keeps reading the same dead file, the agent interprets "no growth" as "still working," and the agent blocks indefinitely.
+
+Use `evo wait`. The CLI is the bounded, structured replacement:
+
+```bash
+# wait for a training subprocess to exit, OR its log to stall, OR the GPU to go idle,
+# whichever first; 60-minute ceiling; structured JSON on stdout
+evo wait --for process=$TRAIN_PID \
+         --for log-growth=$TRAIN_LOG \
+         --for gpu-idle \
+         --timeout 60m --stall-threshold 5m --json
+```
+
+Multiple `--for` flags combine; the wait returns on the first matching condition. The JSON output's `exit_reason` and `triggered_by` identify which condition fired (process-exited / log-stalled / gpu-idle / timed-out). Process / log-growth / gpu-* watches do not require an evo workspace; the workspace-anchored watches (`--for experiments`, `--for ideators`) still work for ideator-proposal and commit waits.
+
+Full surface, exit codes, JSON shape, and examples in `references/evo-wait.md`.
+
+If `evo wait` is not available for some reason (older CLI on PATH), fall back to a bounded poll loop that checks all three signals -- process liveness via `kill -0 $PID`, log growth via `wc -c` delta, GPU via `nvidia-smi --query-gpu=utilization.gpu` -- and exits on any one going negative. NEVER unbounded `while true`.
+
 ## Inspection commands (for debugging, reference only)
 
 ```bash

package/skills/discover/references/inline_instrumentation.js

@@ -6,6 +6,13 @@
  * - Reads EVO_TRACES_DIR, EVO_EXPERIMENT_ID, EVO_RESULT_PATH from process.env.
  * - Writes traces/task_<id>.json per task.
  * - Writes the final result JSON to EVO_RESULT_PATH, or stdout if unset.
+ *
+ * **Per-task emission is the load-bearing discipline.** If your benchmark
+ * evaluates N independent items (per-test-case, per-document, per-sample),
+ * call logTask ONCE PER ITEM with as much detail as you have (input,
+ * expected, model output, timings). Do NOT roll up to one aggregate call --
+ * the dashboard's per-task panel and the verifier's reproducibility check
+ * both rely on per-item traces. See the USAGE EXAMPLE at the bottom.
  */
 
 import {
@@ -95,3 +102,41 @@ export function writeResult(score) {
   }
   return score;
 }
+
+
+// === USAGE EXAMPLE (copy + adapt) ===
+//
+// For a benchmark scoring N independent items, emit one logTask per item.
+// writeResult() with no arg aggregates from the per-task scores.
+//
+// async function main() {
+//   const cases = await loadTestCases();              // N items
+//   for (const [i, c] of cases.entries()) {
+//     const output = await runUnderTest(c.input);
+//     const correct = output === c.expected;
+//     logTask(
+//       `case_${String(i).padStart(2, "0")}`,         // stable id per item
+//       correct ? 1.0 : 0.0,                          // per-item score
+//       {
+//         input: c.input,                             // extras land in the trace JSON
+//         expected: c.expected,                       // for diagnosis later
+//         actual: output,
+//         elapsed_ms: c.elapsed,
+//       }
+//     );
+//   }
+//   // No arg -> writeResult averages the per-task scores
+//   const final = writeResult();
+//   console.log(`final: ${final.toFixed(4)}`);
+// }
+//
+// Anti-pattern: one logTask or writeResult with the aggregate score. Loses
+// the per-item detail the dashboard and verifier rely on. Don't do this:
+//
+//   const score = passed / total;
+//   logTask("eval_total", score);                    // <-- aggregate; useless
+//   writeResult(score);
+//
+// Exception: if the benchmark really is a single indivisible measurement
+// (one e2e workflow, one perf number), emit one task with that score AND
+// pass every observable as extras (timings, allocations, error log).

package/skills/discover/references/inline_instrumentation.py

@@ -5,6 +5,18 @@ Contract:
 - Reads EVO_TRACES_DIR, EVO_EXPERIMENT_ID, EVO_RESULT_PATH from env.
 - Writes traces/task_<id>.json per task.
 - Writes the final result JSON to EVO_RESULT_PATH, or stdout if unset.
+
+**Per-task emission is the load-bearing discipline.** If your benchmark
+evaluates N independent items (per-question math, per-test-case unit
+tests, per-document QA, per-sample reasoning trace), call `log_task`
+ONCE PER ITEM with as much detail as you have (problem text, model
+output, expected answer, intermediate reasoning) -- not one aggregate
+call with the rolled-up score. The dashboard's per-task panel and the
+verifier's reproducibility spot-check both rely on per-item traces;
+emitting one aggregate `log_task("eval_total", score)` makes both
+useless and loses the diagnostic value of the run. `write_result()`
+aggregates from the per-task scores automatically -- you don't need to
+roll up yourself. See the USAGE EXAMPLE at the bottom of this file.
 """
 
 from __future__ import annotations
@@ -107,3 +119,41 @@ def write_result(score: float | None = None) -> float:
     else:
         print(payload)
     return score
+
+
+# === USAGE EXAMPLE (copy + adapt) ===
+#
+# For a benchmark scoring N independent items, emit one log_task per item.
+# write_result() with no arg aggregates _SCORES into the final score.
+#
+# def main():
+#     problems = load_aime_problems()              # list of N items
+#     model = load_model()
+#     for i, problem in enumerate(problems):
+#         output = model.generate(problem.question)
+#         correct = extract_answer(output) == problem.expected
+#         log_task(
+#             f"aime_q{i:02d}",                    # stable id per item
+#             score=1.0 if correct else 0.0,       # per-item score
+#             question=problem.question,           # everything else goes as **extra
+#             expected=problem.expected,           # and lands in the trace JSON
+#             model_output=output,                 # for diagnosis later
+#             tokens_used=len(model.last_tokens),
+#         )
+#     # write_result() with no arg returns mean(_SCORES) over all logged tasks --
+#     # no need to compute the average yourself
+#     final_score = write_result()
+#     print(f"final: {final_score:.4f}")
+#
+# Anti-pattern: ONE log_task or write_result call with the aggregate. The
+# dashboard's per-task panel + the verifier's reproducibility spot-check
+# both need per-item traces; aggregate-only emission makes them useless.
+#
+#     # DO NOT do this:
+#     score = sum(per_problem_correct) / len(per_problem_correct)
+#     log_task("eval_total", score)               # <-- aggregate; loses detail
+#     write_result(score)
+#
+# Exception: if the benchmark really is a single indivisible measurement
+# (one e2e workflow, one perf number), emit one task with that score AND
+# include every observable as **extra (timings, allocations, error log).

package/skills/discover/references/sdk_node.js

@@ -3,6 +3,12 @@
 // The SDK auto-reads $EVO_TRACES_DIR, $EVO_EXPERIMENT_ID, and
 // $EVO_RESULT_PATH. Traces flush on each report() so the dashboard can
 // stream progress live.
+//
+// **Per-task emission is the load-bearing discipline.** Loop over your N
+// independent items and call run.report(id, {score, ...}) ONCE PER ITEM.
+// Do NOT roll up to one aggregate `run.report("eval_total", ...)` call --
+// dashboard panel + verifier reproducibility check both rely on per-item
+// traces. The Anti-pattern at the bottom is what to avoid.
 
 import { Run, Gate } from '@evo-hq/evo-agent';
 
@@ -26,3 +32,16 @@ for (const task of criticalTasks) {
   gate.check(task.id, { score: result.score });
 }
 await gate.finish();
+
+// ---- ANTI-PATTERN (do NOT do this) ----
+//
+// Reporting one aggregate task entry loses every diagnostic value of
+// per-item traces. SDK aggregates from per-task reports automatically.
+//
+//     // WRONG:
+//     const scores = await Promise.all(tasks.map(t => evaluate(t).then(r => r.score)));
+//     run.report("eval_total", { score: scores.reduce((a, b) => a + b) / scores.length });
+//     await run.finish();
+//
+// Exception: if your benchmark really is a single indivisible measurement,
+// report one task AND attach every observable as extras.

package/skills/discover/references/sdk_python.py

@@ -5,6 +5,13 @@ Install `evo-hq-agent` with this project's package manager/runtime, for example
 
 The SDK auto-reads $EVO_TRACES_DIR, $EVO_EXPERIMENT_ID, and $EVO_RESULT_PATH.
 Traces flush on each report() so the dashboard can stream progress live.
+
+**Per-task emission is the load-bearing discipline.** Loop over your N
+independent items and call `run.report(task_id, score=..., ...)` ONCE PER
+ITEM. Do NOT roll up to one aggregate `run.report("eval_total", score=avg)`
+call -- the dashboard's per-task panel and the verifier's reproducibility
+spot-check both rely on per-item traces. The example below shows the right
+pattern. The Anti-pattern at the bottom is what to avoid.
 """
 
 from evo_agent import Run, Gate
@@ -41,3 +48,21 @@ with Gate() as gate:
     for task in critical_tasks:
         result = evaluate(task, agent)
         gate.check(task["id"], score=result.score)
+
+
+# ---- ANTI-PATTERN (do NOT do this) ----
+#
+# Computing the aggregate yourself and reporting it as a single task
+# entry loses every diagnostic value of the per-item traces -- dashboard
+# can't show which problems failed, verifier can't spot-check, ideator
+# can't cluster failure modes. The SDK aggregates from per-task reports
+# automatically.
+#
+#     # WRONG:
+#     scores = [evaluate(t, agent).score for t in tasks]
+#     run.report("eval_total", score=sum(scores) / len(scores))
+#     run.finish()
+#
+# Exception: if your benchmark really is a single indivisible measurement
+# (one e2e workflow with one score number), report one task AND attach
+# every observable as extras.

package/skills/infra-setup/SKILL.md

@@ -1,8 +1,7 @@
 ---
 name: infra-setup
 description: Non-user-invocable provider/setup reference for evo backend switching, prerequisite checks, and auth/install guidance.
-disable-model-invocation: true
-evo_version: 0.4.4-alpha.6
+evo_version: 0.5.0-alpha.3
 ---
 
 # Infra Setup

package/skills/optimize/SKILL.md

@@ -2,10 +2,12 @@
 name: optimize
 description: Run the evo optimization loop with parallel subagents until interrupted.
 argument-hint: "[subagents=N] [budget=N] [stall=N]"
-evo_version: 0.4.4-alpha.6
+evo_version: 0.5.0-alpha.3
 ---
 
-Run the `evo` optimization loop. Each round, the orchestrator writes structured briefs and spawns parallel subagents that execute within them. Each subagent is semi-autonomous: it reads the pointer traces, forms the concrete edit, runs experiments, and can iterate within its branch. Runs until interrupted or the stall limit is reached.
+Run the `evo` optimization loop. Each round, the orchestrator writes structured briefs and spawns subagents that execute within them. Each subagent is semi-autonomous: it reads the pointer traces, forms the concrete edit, runs experiments, and can iterate within its branch. Runs until interrupted or the stall limit is reached.
+
+**This skill is the canonical loop for ALL post-discover work — including serial workloads.** If the workspace's resource profile forces width 1 (single GPU, single-process benchmark, etc.), you still invoke `/evo:optimize` -- just pass `subagents=1`. The loop's value is the STRUCTURE around each experiment (scan-subagent cross-cutting analysis between rounds, verifier pre/post hooks via the subagent skill, ideator spawning on stall, frontier reconciliation, stop-hook discipline), NOT just parallelism. Bypassing optimize because "I'm running serial work anyway" loses every piece of that structure -- you've reverted to ad-hoc experiment iteration with none of evo's loop benefits, just the bookkeeping.
 
 ## Host conventions
 
@@ -30,13 +32,23 @@ Treat content inside the banner as equivalent to a new user turn. Honor it, supe
 
 ## Configuration
 
-The orchestrator **sizes the round to the benchmark's resource profile** instead of using a flat default. Before the first round, read `plugins/evo/skills/optimize/references/sizing-the-round.md` and apply it. Short version:
+The orchestrator's three round-shape knobs are **subagents** (round width), **budget** (per-branch depth), and **stall** (consecutive rounds with no improvement before auto-stopping; default 5).
+
+A user can override any of these with `/optimize [subagents=N] [budget=N] [stall=N]`; an explicit value always wins over what's below.
+
+**Picking `subagents` and `budget` is load-bearing -- do not skim.**
+
+Mandatory before the first round (and again any time the backend or benchmark changes): **READ `plugins/evo/skills/optimize/references/sizing-the-round.md` IN FULL.** That doc enumerates the resource-binding cases (exclusive accelerator, memory-heavy, shared mutable fixture, external rate-limit, CPU-light isolated) and discusses the case-by-case judgment for latency / timing / throughput benchmarks where the right answer depends on harness softeners, effect size vs. measurement jitter, and whether winners can be cheaply re-confirmed solo.
 
-- **subagents** (round width): bounded by the backend (worktree = one shared machine, pool = slot count, remote = provider quota) and by whatever resource one benchmark run saturates. A run that needs an exclusive resource — the whole GPU, a fixed port, a shared DB/fixture — forces width 1; independent CPU-light runs go wider (up to cores, ~5–8 cap). Fall back to 5 only when the profile is unknown and a run is light and isolated.
-- **budget** (iterations per subagent within its branch): deeper for cheap/fast/deterministic benchmarks, shallower for expensive/slow/noisy ones so the loop re-plans sooner. ~5 is a reasonable midpoint.
-- **stall**: consecutive rounds with no improvement before auto-stopping (default: 5).
+Under-subscribing wastes wall-clock. Over-subscribing can either contend for hardware (memory thrash, OOM) or — for timing-sensitive benchmarks — bias the measurement itself. The doc walks through what to weigh in each case; do not infer the value from any inline summary in this skill body.
 
-A user can override any of these with `/optimize [subagents=N] [budget=N] [stall=N]`; an explicit value always wins over the heuristic.
+Common ways agents get this wrong by skimming:
+- "8-core machine, CPU-light → width 5" sounds right but skips the question of whether the metric is corruptible by sibling-process pressure. The doc has the judgment framing.
+- "Worktree backend has no slot cap so I can go higher" — worktree just shifts the cap from infrastructure to the binding resource. Same hardware, no safety net.
+
+If `.evo/project.md` records a resource profile (it should, after `/evo:discover`), START from that. The reference doc is what you use to APPLY it. If the profile is missing or thin, that's a discover-step bug — fix it (write a resource profile that names the binding resource explicitly) before continuing.
+
+In your opening message, state the width/budget you chose AND a one-line reason that references the binding-resource framing FROM THE DOC (e.g. "width 1 — exclusive GPU; budget 8 — runs deterministic"; or "width 3 — CPU-light isolated, but harness reports stable jitter at this concurrency so promoting solo-confirm gate; budget 5"). If your reason doesn't connect to the doc's framing, go back and read it.
 
 - **autonomous**: the keep-going loop. **Default: on** — evo is autoresearch; it runs unattended. Turn off for a run with `evo autonomous off`.
 - **subagents-only**: gate orchestrator edits, pushing all edits through subagents. **Default: on**. Turn off for a run with `evo subagents-only off`.
@@ -297,6 +309,86 @@ Update notes with cross-cutting learnings:
   evo set <exp_id> --note "key insight from round N"
   ```
 
+### 6a. Pattern recognition across history (objective, not narrative)
+
+Step 6 cross-cuts a single round. This step looks across ALL committed experiments in the run, not just this round's. The orchestrator's failure mode is tunnel vision -- iterating on the visible axis (whatever knob the recent rounds touched) while missing the orthogonal axis (the harness itself, the score definition, the environment, the input data, plumbing). The check is cheap, runs between rounds, and is the most reliable signal that you're on the wrong axis.
+
+Four checks via `evo show` + `evo tree`:
+
+1. **Score plateaus across structurally distinct hypotheses.** Read the `hypothesis` strings of committed experiments. If 3+ experiments with materially different hypotheses (not minor parameter sweeps of the same idea) all commit at the same score, the bottleneck is not where the hypotheses were aiming. The next move belongs on an axis none of those hypotheses touched.
+
+2. **Repeated failure class.** Tally failure indicators across discarded + failed nodes in the run so far: `gate_failures` names, non-zero exit codes, shared error-message fragments in `benchmark_err.log`. If 2+ failures share a class, that class is structural -- fix the cause, rather than queuing more experiments that will hit the same wall.
+
+3. **Internal-vs-benchmark delta.** Compare each evaluated node's *internal* indicators (progress signals the experiment's own process produces during a run -- intermediate test pass-rates, training loss, build success, agent self-report metrics, whatever the trace stream carries) to its *committed* benchmark score. Healthy internal signal + flat benchmark score = the experiment is optimizing something the benchmark does not reward. The fix is usually in the harness, output format, or score definition -- not in another hypothesis on the same axis.
+
+4. **Annotate facts, not narratives.** Annotations via `evo annotate <exp_id>` and `evo set <exp_id> --note` should record what HAPPENED -- scores, exact error messages, surprising observations, sources used. Not what you hoped would happen, what you plan to try next, or how you feel about the result. Annotations get loaded into future decision context and into ideator briefs; narrative noise contaminates them. State facts; leave plans to TodoWrite or `evo set --note` on the round itself.
+
+If any check surfaces a structural issue, the next round's subagent briefs should target the orthogonal axis the pattern identifies. Another iteration on a plateaued or systematically-failing axis produces another data point with the same conclusion.
+
+### 6b. Periodically spawn ideators (in parallel)
+
+The optimize loop's scan sub-agents (step 3) read the CURRENT round's evaluated experiments for failure patterns. They don't do deep cross-graph analysis or external literature scans -- that work belongs to the ideator skill (`evo:ideator`).
+
+Spawn ideators in parallel when ANY of these triggers fire:
+
+- **Periodic**: every N=5 committed experiments since the last ideator round
+- **Stall**: best score unchanged for M=3 consecutive committed experiments (the stall counter from step 6)
+- **Failure cluster**: M=3 consecutive discards with related root causes (use the `evo discards` output)
+- **User-triggered**: a directive (`evo direct`) asks for fresh ideas
+
+When a trigger fires, spawn three parallel **evo ideator subagents** via your host's Task tool -- one per brief:
+
+```
+Task(subagent_type="evo:ideator", prompt="workspace=<path>\nbrief=failure_analysis")
+Task(subagent_type="evo:ideator", prompt="workspace=<path>\nbrief=literature")
+Task(subagent_type="evo:ideator", prompt="workspace=<path>\nbrief=frontier_extrapolation")
+```
+
+| Brief | What it does |
+|---|---|
+| `failure_analysis` | Cross-graph clustering of discards/failures |
+| `literature` | Web/arXiv scan for untried techniques in the workspace domain |
+| `frontier_extrapolation` | Deeper variants of the steepest score gradient on the best path |
+
+Each subagent runs the brief in its own context, appends proposals as JSONL lines to `.evo/run_<run_id>/ideator/proposals.jsonl` (single final write), and returns a JSON summary. See `plugins/evo/agents/ideator.md` for the full procedure each ideator follows.
+
+Ideators take 5-10 min while the optimize loop's next round is typically 1-2 min away. If you fire and continue, proposals miss the next round's brief-writing every time. Two patterns work:
+
+- **Block here briefly.** If the trigger was a STALL or FAILURE CLUSTER, the next round's quality depends on fresh ideas -- block until enough proposals land:
+  ```bash
+  evo wait --for ideators --count 3 --timeout 900   # 15 min cap, fail-open
+  ```
+  Exit 0 means the proposals are ready; exit 124 (timeout) means proceed with whatever's available -- proposals.jsonl may have partial results.
+
+- **Fire and continue for periodic spawns** (every-N-commits trigger). The next round can run without proposals; the round after that will read them once they land. Use this when there's plenty of in-graph signal still to extract.
+
+### 6c. Reconcile ideator proposals at brief-writing time
+
+Before writing the next round's briefs (step 4 of the next iteration), check for new ideator proposals:
+
+```bash
+# Read proposals newer than the last round
+test -f .evo/run_*/ideator/proposals.jsonl && \
+  tail -n +1 .evo/run_*/ideator/proposals.jsonl | \
+  jq -s --argjson cutoff "$LAST_ROUND_END_TS" \
+    'map(select(.generated_at > $cutoff))'
+```
+
+If you fired ideators in 6b WITHOUT blocking (periodic trigger) and they haven't landed yet, you can also wait here -- but only a short timeout, since brief-writing should not stall indefinitely:
+
+```bash
+evo wait --for ideators --count 1 --timeout 120   # 2 min cap, fail-open
+```
+
+For each new proposal:
+
+1. Check the workspace graph -- has the proposed config already been tried? Use `evo discards --like "<keyword>"` to scan. If yes, skip.
+2. Score each remaining proposal by `expected_score_uplift × confidence`. Confidence ranking: `frontier_extrapolation > failure_analysis > literature`, all else equal.
+3. The top 1-2 proposals become objectives in the next round's briefs (step 4). Cite the proposal's `hypothesis` and `mechanism` in the brief's *Objective* field.
+4. Leave the rest in the queue -- they may surface as winners after a few more rounds when the frontier shifts.
+
+Proposals are advisory, not mandatory. If none look better than what step 3's scan sub-agents surfaced from in-graph signal, ignore them and proceed with the in-graph briefs. Ideator output complements, doesn't replace.
+
 ### 7. Continue or stop
 
 **Continue** if:
@@ -317,6 +409,27 @@ On stop, print a final summary:
 
 Go back to step 1.
 
+## Polling discipline
+
+When waiting on a long-running background process (a subagent's training subprocess, a long evaluation, a batch job), do NOT use `while true; do sleep N; tail file; done`. That loop never exits when the underlying process crashes -- the tail keeps reading the same dead file, the agent interprets "no growth" as "still working," and the agent blocks indefinitely.
+
+Use `evo wait`. The CLI is the bounded, structured replacement:
+
+```bash
+# wait until the training subprocess exits, OR its log stalls, OR the GPU goes idle,
+# whichever first; 60-minute ceiling; structured JSON on stdout
+evo wait --for process=$TRAIN_PID \
+         --for log-growth=$TRAIN_LOG \
+         --for gpu-idle \
+         --timeout 60m --stall-threshold 5m --json
+```
+
+Multiple `--for` flags combine; the wait returns on the first matching condition. The JSON output's `exit_reason` and `triggered_by` identify which condition fired. Process / log-growth / gpu-* watches do not require an evo workspace context; the workspace-anchored watches (`--for experiments`, `--for ideators`) cover the ideator + commit waits described elsewhere in this skill.
+
+Full surface, exit codes, JSON shape, examples: `references/evo-wait.md` (under `plugins/evo/skills/references/`).
+
+If `evo wait` is not available for some reason (older CLI on PATH, sandbox constraint), fall back to a bounded poll loop that checks all three signals -- process liveness via `kill -0 $PID`, log growth via `wc -c` delta, GPU via `nvidia-smi --query-gpu=utilization.gpu` -- and exits on any one going negative. NEVER unbounded `while true`.
+
 ## Resetting the eval epoch
 
 `evo infra event -m "<reason>" --breaking` bumps `current_eval_epoch` and blocks

package/skills/optimize/references/sizing-the-round.md

@@ -30,6 +30,14 @@ Read `.evo/project.md`'s resource-profile line first (discover records it). If i
 
 When a run needs an exclusive resource, serializing benchmark *execution* (width 1) is correct even though the *edits* are independent — on the worktree backend `evo run` executes the benchmark in-place, so concurrent `evo run` means concurrent benchmark processes on that one resource.
 
+**Latency / timing / throughput benchmarks deserve a per-workspace judgment call, not a fixed answer.** When the metric IS time, jitter, or rate, sibling-process CPU/cache/memory-bandwidth pressure can BIAS the measurement (not just add noise) — and the orchestrator may then promote a "winner" that's just a contention artifact. But this doesn't always happen, and harness softeners (warmup, min-over-N batches, outlier rejection) reduce the risk. Things to weigh case-by-case before picking width:
+- How big is the optimization's expected effect vs. the variance the harness reports under parallel runs? If the effect is much larger than measurement jitter, modest parallelism is fine.
+- How much of the benchmark's wall-clock is the actual timed section? Long edit/compile phases overlap safely; only the timed section needs isolation.
+- Can a winner be cheaply re-confirmed solo before being promoted? If yes, going wider for exploration with a solo-confirm gate is reasonable.
+- Does the harness already filter contention (e.g., reject batches with outlier jitter)?
+
+If unsure, start narrower and widen once you've confirmed measurements are stable. Width 1 is the safe default for *unknown* timing-sensitive benchmarks; don't apply it reflexively when the workspace has data that says otherwise.
+
 ## 3. Depth — `budget` (iterations per subagent within its branch)
 
 Depth trades exploration against spend, and keys off cost per run, not concurrency:

package/skills/report/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: report
 description: Print the dashboard's dot chart (score over experiment order, status colors, best-path stair) inline in the terminal for every run in the workspace. Use when the user invokes /evo:report, asks for a quick score chart without opening the dashboard, or wants the scatter plot in chat output.
-evo_version: 0.4.4-alpha.6
+evo_version: 0.5.0-alpha.3
 ---
 
 # Report

package/skills/subagent/SKILL.md

@@ -1,7 +1,7 @@
 ---
 name: subagent
 description: Internal protocol for evo optimization subagents. Loaded by subagents spawned from /optimize via their host's skill loader. Not for orchestrator use.
-evo_version: 0.4.4-alpha.6
+evo_version: 0.5.0-alpha.3
 ---
 
 # Evo Subagent Protocol
@@ -160,7 +160,22 @@ For multi-line edits, `evo edit --json-stdin` reads `{"old":...,"new":...,"repla
 
 You may edit anything within the target scope. Do NOT modify benchmark, gate, or framework code.
 
-### 4. Run the experiment
+### 4. Verify the experiment design (pre-`evo run`)
+
+Before `evo run` burns compute, invoke the **evo verifier subagent** via your host's Task tool. Static analysis, ~30s.
+
+```
+Task(subagent_type="evo:verifier",
+     prompt="workspace=<workspace abs path>\nexperiment_id=<your exp_id>\nphase=pre")
+```
+
+The verifier checks for test-set leakage in your training data, subsetted eval commands, missing gates for new artifacts, generic hypotheses, and concurrent-resource conflicts. It returns a JSON report (`{passed, verdict, findings}`) and writes the same verdict as an `evo annotation` on the experiment. See `plugins/evo/agents/verifier.md` for the full check list.
+
+If the verifier returns `passed: false` (verdict `fail`), address every flagged `block` finding and re-invoke until it returns `passed: true`. Skipping or fudging a `fail` verdict is a stop-the-line bug -- the verdict is the precondition for compute spend.
+
+If the verifier returns verdict `warn`, you may proceed but address the warnings in your annotation (step 7).
+
+### 5. Run the experiment
 
 ```bash
 evo run <exp_id>
@@ -195,7 +210,7 @@ evo run <exp_id> --i-staged-new-files yes
 
 The ack flag is required when the worktree has any untracked, non-gitignored file. Without it, `evo run` errors closed and lists the files. For each file, decide: source (then `git add`) or warm state (leave untracked -- it persists in the slot for future experiments). Then re-run with `--i-staged-new-files yes`. The flag value must be exactly `yes`. In `commit_strategy=all` workspaces (default for `--backend worktree`) the flag is a silent no-op; safe to always pass.
 
-### 5. Analyze the result
+### 6. Analyze the result
 
 `evo run` prints one of three outcomes:
 
@@ -215,7 +230,7 @@ The ack flag is required when the worktree has any untracked, non-gitignored fil
   - Structural (benchmark broken, evo misconfigured): report to orchestrator and stop.
   - Not worth fixing: `evo discard <id> --reason "..."`.
 
-### 6. Annotate
+### 7. Annotate
 
 ```bash
 evo annotate <exp_id> "<what you changed, what happened, and why>"
@@ -223,7 +238,7 @@ evo annotate <exp_id> "<what you changed, what happened, and why>"
 
 Always annotate so other agents can learn from your experiments.
 
-### 6b. Add gates for fixed behaviors
+### 7b. Add gates for fixed behaviors
 
 When you fix a critical, easy-to-regress behavior, lock it in as a gate so future experiments on this branch can't break it:
 
@@ -233,7 +248,7 @@ evo gate add <exp_id> --name "social_eng_resistance" --command "python3 {worktre
 
 Good candidates: a specific benchmark task that was hard to fix, a test for a critical policy rule, a smoke test for a fragile behavior. The gate command must exit non-zero when the protected behavior regresses; a bare benchmark invocation that prints a low score but exits 0 is decorative and should not be registered. Do NOT gate every passing task -- that over-constrains the search.
 
-### 7. Decide: continue or stop
+### 8. Decide: continue or stop
 
 Continue if budget remains AND (last outcome was committed, OR you have a meaningfully different idea after an evaluated/discarded outcome). When continuing after a committed experiment, update your parent to the newly committed ID.