coder-eval 0.11.7__tar.gz → 0.12.0__tar.gz

{coder_eval-0.11.7 → coder_eval-0.12.0}/.claude/commands/coder-eval-review.md

@@ -24,7 +24,8 @@ The run layout (`runs/<run_id>/<variant_id>/<task_id>/<NN>/…`, `<NN>` a zero-p
 1. Read `<run_path>/run.json` if present (for context — `run_id`, `start_time`).
 2. Glob `<run_path>/*/*/*/task.json` and read each one.
 3. Read `<run_path>/analysis.md` if present — it already diagnoses many failures; lean on its findings rather than re-deriving them.
-4. A task counts as **failed** if `final_status != "SUCCESS"` **or** `weighted_score < 0.9`. Skip passing tasks for now (we may extend to passing tasks later — the schema supports it).
+4. Skip any task whose `final_status` is `"NOT_GRADED"` — `coder-eval execute` produced it, no criterion ran, and `weighted_score` is `null`. It is neither a pass nor a failure, and comparing `null < 0.9` would book every ungraded row as a failure to review.
+5. Of the rest, a task counts as **failed** if `final_status != "SUCCESS"` **or** `weighted_score < 0.9`. Skip passing tasks for now (we may extend to passing tasks later — the schema supports it).
 
 If no `task.json` files exist, write an empty `review_index.json` (`{"reviews": []}`) and exit.
 

{coder_eval-0.11.7 → coder_eval-0.12.0}/.claude/harness-candidates.md

@@ -455,3 +455,12 @@ with the two `action.yml` items above — one considered change to the action's
   `final_status`, which does not exist in `run.json` and would have made a new
   assertion dead on arrival. Guard: assert the key set that non-Python consumers
   depend on, mirroring how CE030 pins doc/schema parity.
+
+- [ ] A `_*TurnState` (agent turn-state) attribute that is written but never read
+  outside its own assignment — CE037-class dead accumulator. Surfaced during the
+  Pi harness port: `_PiTurnState.turns_finished` was copied from OpenCode's
+  `steps_finished` (which drives that agent's `finished_without_tokens` guard) but
+  Pi deliberately dropped that guard, leaving the counter dead. Fixed by hand this
+  run. Guard would need cross-method dataflow over each `Agent`-subclass turn-state
+  class (write sites vs read sites), which the AST-only CExxx runner can't express
+  in ~30 min — deferred. Caught in: Pi harness Phase 2 quality review.

{coder_eval-0.11.7 → coder_eval-0.12.0}/.claude/shared/run-layout.md

@@ -12,6 +12,9 @@ runs/<run_id>/<variant_id>/<task_id>/<NN>/{task.json, task.log, artifacts/}
 - `<NN>` — zero-padded replicate index (e.g. `00`, `01`).
 - `task.json` — the persisted per-replicate result (the consumer contract; carries the large `iterations` array — still accepted under its former name `turns` when reading, but not what current runs write).
 - `task.json.malformed` — present only on the docker degrade path: when an existing `task.json` fails to parse (schema skew from a stale `:latest` image, or a truncated/torn write), the docker runner moves the unparseable original aside to this sidecar and writes a synthetic `final_status=ERROR` `task.json` in its place. Diagnostic-only; `rglob("task.json")` consumers do not match it.
+- `task.execute.json` — present only after a DETACHED grade (`coder-eval evaluate <run_dir>` or `coder-eval run --resume` over a `NOT_GRADED` row). The pre-grade snapshot of `task.json`, written once and never overwritten by a later grade, so "this run was executed separately from grading" stays auditable. Diagnostic-only; `rglob("task.json")` consumers do not match it.
+- `task.json.graded` — present only after `coder-eval execute --driver docker` refused a container's verdict: the runtime image predated `execute` and graded anyway, so the runner quarantines the graded record here rather than leaving it readable as `task.json`, where a later `--resume` / `aggregate` would fold in exactly the row it declined to publish. Diagnostic-only; `rglob("task.json")` consumers do not match it.
+- `grade.log` — present only after a DETACHED grade over this directory (`coder-eval run --resume`). The grading pass's own log. It is a separate file because the log handler truncates whatever file it opens, so writing to `task.log` would destroy the agent trajectory log the run already paid for.
 - `task.log` — the human-readable task log; `artifacts/` — files the agent produced.
 
 **Scope-marker files** (used to detect what a given path represents):

{coder_eval-0.11.7 → coder_eval-0.12.0}/.github/pages-stub/index.html

@@ -12,7 +12,7 @@
     -->
     <meta
       name="description"
-      content="Coder Eval is Playwright for coding agents: an open-source, agent-agnostic framework for evaluating and benchmarking AI coding agents and their skills — sandboxed runs of Claude Code, Codex, Antigravity (Gemini), or OpenCode against declarative YAML tasks, with weighted scoring and CI gates. Documentation: coder-eval.com/docs"
+      content="Coder Eval is Playwright for coding agents: an open-source, agent-agnostic framework for evaluating and benchmarking AI coding agents and their skills — sandboxed runs of Claude Code, Codex, Antigravity (Gemini), OpenCode, or Pi against declarative YAML tasks, with weighted scoring and CI gates. Documentation: coder-eval.com/docs"
     />
 
     <!--
@@ -227,7 +227,7 @@
       <p class="lead">
         <strong>Playwright for coding agents.</strong> An open-source, agent-agnostic framework for
         evaluating and benchmarking AI coding agents and their skills: it runs a real agent — Claude
-        Code, Codex, Antigravity (Gemini), or OpenCode — in a sandbox against declarative YAML
+        Code, Codex, Antigravity (Gemini), OpenCode, or Pi — in a sandbox against declarative YAML
         tasks, then scores the files and commands the agent actually produced.
       </p>
       <p class="notice">

{coder_eval-0.11.7 → coder_eval-0.12.0}/.github/workflows/verify-published-action.yml

@@ -541,6 +541,18 @@ jobs:
                     "uploaded run dir before re-running; this is an unattended paid job.")
               sys.exit(1)
 
+          # NOT_GRADED means a task ran but was never scored. This job invokes the
+          # published action, which runs `coder-eval run` (graded), so reaching it is
+          # impossible unless the action started dispatching `coder-eval execute` --
+          # in which case every score gate below silently measures nothing and the job
+          # goes green having verified no verdict at all. Hard-fail, don't tolerate.
+          ungraded = [s for s in statuses if s == "NOT_GRADED"]
+          if ungraded:
+              print("::error::task(s) reported NOT_GRADED -- the published action ran without "
+                    "grading. `coder-eval run` always grades, so the action is dispatching the "
+                    "wrong command and every score gate in this job is measuring nothing.")
+              sys.exit(1)
+
           # Exit-contract check, conditional on the model having actually performed.
           # Ignoring the step's exit code entirely (see the continue-on-error rationale
           # above) would also hide a REGRESSION in the action's own exit logic -- e.g. a

{coder_eval-0.11.7 → coder_eval-0.12.0}/CHANGELOG.md

@@ -2,6 +2,178 @@
 
 <!-- version list -->
 
+## v0.12.0 (2026-09-09)
+
+### Bug Fixes
+
+- Code review fixes for pi-harness ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+- **container**: Arm the heartbeat watchdog only inside the container
+  ([#154](https://github.com/UiPath/coder_eval/pull/154),
+  [`0fe8cb0`](https://github.com/UiPath/coder_eval/commit/0fe8cb01e6e49dbe1c5e3905ac62fd7fce2851da))
+
+- **criteria**: Refuse an out-of-sandbox criterion path instead of scoring it 0.0
+  ([#154](https://github.com/UiPath/coder_eval/pull/154),
+  [`0fe8cb0`](https://github.com/UiPath/coder_eval/commit/0fe8cb01e6e49dbe1c5e3905ac62fd7fce2851da))
+
+- **eval**: Address the medium and low findings from the branch review
+  ([#154](https://github.com/UiPath/coder_eval/pull/154),
+  [`0fe8cb0`](https://github.com/UiPath/coder_eval/commit/0fe8cb01e6e49dbe1c5e3905ac62fd7fce2851da))
+
+- **eval**: Close the verdict-correctness gaps in detached grading
+  ([#154](https://github.com/UiPath/coder_eval/pull/154),
+  [`0fe8cb0`](https://github.com/UiPath/coder_eval/commit/0fe8cb01e6e49dbe1c5e3905ac62fd7fce2851da))
+
+- **evalboard**: Show per-row cost for open-weight harnesses (Pi)
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+- **execute**: Close the verdict-changing and trust-boundary defects in detached grading
+  ([#154](https://github.com/UiPath/coder_eval/pull/154),
+  [`0fe8cb0`](https://github.com/UiPath/coder_eval/commit/0fe8cb01e6e49dbe1c5e3905ac62fd7fce2851da))
+
+- **execute**: Close the verdict-divergence, trust-gate and fabricated-rate defects
+  ([#154](https://github.com/UiPath/coder_eval/pull/154),
+  [`0fe8cb0`](https://github.com/UiPath/coder_eval/commit/0fe8cb01e6e49dbe1c5e3905ac62fd7fce2851da))
+
+- **execute**: Move post_run to the grading phase and stop the record lying about its driver
+  ([#154](https://github.com/UiPath/coder_eval/pull/154),
+  [`0fe8cb0`](https://github.com/UiPath/coder_eval/commit/0fe8cb01e6e49dbe1c5e3905ac62fd7fce2851da))
+
+- **pi**: Address bai-uipath review — apportionment caveat + token/telemetry minors
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+- **pi**: Do not forward allowed_tools/disallowed_tools to Pi
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+- **pi**: Multi-model review — gate error-crash on intentional cuts + dead dup + doc/guard nits
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+- **pi**: Review blockers — session-id sanitize, error crash, telemetry warn, tool map, CE047
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+- **security**: Close the three CodeQL findings on the detached-grading diff
+  ([#154](https://github.com/UiPath/coder_eval/pull/154),
+  [`0fe8cb0`](https://github.com/UiPath/coder_eval/commit/0fe8cb01e6e49dbe1c5e3905ac62fd7fce2851da))
+
+- **tasks**: Make the two remaining absolute criterion paths reachable, and add CE055
+  ([#154](https://github.com/UiPath/coder_eval/pull/154),
+  [`0fe8cb0`](https://github.com/UiPath/coder_eval/commit/0fe8cb01e6e49dbe1c5e3905ac62fd7fce2851da))
+
+- **tests**: Kill the CodeQL taint source and the Windows mode assertion
+  ([#154](https://github.com/UiPath/coder_eval/pull/154),
+  [`0fe8cb0`](https://github.com/UiPath/coder_eval/commit/0fe8cb01e6e49dbe1c5e3905ac62fd7fce2851da))
+
+### Documentation
+
+- **agents**: 4/4 — add Pi harness docs + enumeration-surface parity
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+- **docker**: Fix env_passthrough model name + list Pi in baked-toolchain docs
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+- **pi**: 3/4 — document docker support for Pi
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+- **pi**: Correct plugins->--skill support + OPENROUTER passthrough claims
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+- **pi**: Fix tool-enforcement claims + cost docstring after review
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+### Features
+
+- **agents**: 2/4 — add PiAgent harness (pi --mode json) + registration
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+- **cli**: `coder-eval execute` + detached grading via `evaluate <run_dir>`
+  ([#154](https://github.com/UiPath/coder_eval/pull/154),
+  [`0fe8cb0`](https://github.com/UiPath/coder_eval/commit/0fe8cb01e6e49dbe1c5e3905ac62fd7fce2851da))
+
+- **cli**: Add `coder-eval execute` — run tasks without grading them
+  ([#154](https://github.com/UiPath/coder_eval/pull/154),
+  [`0fe8cb0`](https://github.com/UiPath/coder_eval/commit/0fe8cb01e6e49dbe1c5e3905ac62fd7fce2851da))
+
+- **cli**: Grade an executed run afterwards — `evaluate <run_dir>` + `Sandbox.adopt`
+  ([#154](https://github.com/UiPath/coder_eval/pull/154),
+  [`0fe8cb0`](https://github.com/UiPath/coder_eval/commit/0fe8cb01e6e49dbe1c5e3905ac62fd7fce2851da))
+
+- **cli**: Make --resume distinguish "executed" from "graded"
+  ([#154](https://github.com/UiPath/coder_eval/pull/154),
+  [`0fe8cb0`](https://github.com/UiPath/coder_eval/commit/0fe8cb01e6e49dbe1c5e3905ac62fd7fce2851da))
+
+- **docker**: 1/4 — bake the pinned Pi CLI into docker/Dockerfile
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+- **evalboard**: Show the Pi logo + "Pi" in the harness view
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+- **models**: 1/4 — add AgentKind.PI + PiAgentConfig model
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+- **pi**: Add Pi harness (`--type pi`) with docker + skill injection
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+- **pi**: Load agent.plugins skills via --skill + harden turn/token handling
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+- **sandbox**: 2/4 — forward OPENROUTER_API_KEY into docker containers
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+- **tasks**: 3/4 — add local pi_smoke_test task
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+### Refactoring
+
+- **agents**: Hoist shared plugins->skills resolver into agents/_skills.py
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+### Testing
+
+- Fix two CI-only failures in the detached-grading tests
+  ([#154](https://github.com/UiPath/coder_eval/pull/154),
+  [`0fe8cb0`](https://github.com/UiPath/coder_eval/commit/0fe8cb01e6e49dbe1c5e3905ac62fd7fce2851da))
+
+- **docker**: 4/4 — guard that the Dockerfile bakes a pinned Pi CLI
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+- **pi**: Assign the awaited cancel result to silence CodeQL 'no effect'
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+- **pi**: Install shutil.which patch so the env-info test doesn't need the real CLI
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+- **pi**: Port OpenCode teardown + cost-fallback matrices (blockers 3, 4)
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+- **pi**: Scrub personal scratchpad path from happy-stream fixture
+  ([#159](https://github.com/UiPath/coder_eval/pull/159),
+  [`57c9e33`](https://github.com/UiPath/coder_eval/commit/57c9e33620e222d9e238983cd9057d2e852015be))
+
+
 ## v0.11.7 (2026-09-08)
 
 ### Bug Fixes

{coder_eval-0.11.7 → coder_eval-0.12.0}/CLAUDE.md

@@ -27,6 +27,13 @@ coder_eval/
 ├── fs_permissions.py              # set_permissions: stacked chmod window (via Sandbox.set_permissions)
 ├── pricing.py                     # Model pricing / cost calculation (ModelPricing, calculate_cost, register_pricing)
 ├── litellm_cost.py                # Join proxy-captured ACTUAL per-call cost/cache onto turns (LiteLLM backend; apply_actual_cost)
+├── reports_html.py                # Single-file HTML report (the evalboard's static twin)
+├── reports_stats.py               # Shared report statistics + the ungraded rendering helpers (format_score, is_env_table_key)
+├── formatting.py                  # Human-readable number/duration formatting shared by the renderers
+├── invocation_log.py              # JSON Lines CLI-invocation log the `cli_called` criterion reads
+├── telemetry.py                   # App Insights / OpenTelemetry emission (CoderEval.Task.End et al.)
+├── isolation/                     # driver: docker — docker_runner.py builds, runs and reaps one container per task
+├── optimize/                      # Prompt/config optimization helpers
 ├── utils.py                       # Version info helpers
 │
 ├── agents/
@@ -80,18 +87,24 @@ coder_eval/
 │   └── timeout.py                 # Timeout handling (TurnTimeoutError carries optional partial TurnRecord)
 │
 ├── orchestration/                 # Batch execution utilities
-│   ├── batch.py                   # Parallel task execution (run_batch + run_batch_resolved)
+│   ├── batch.py                   # Parallel task execution (run_batch) + partition_for_resume/ResumePartition
 │   ├── config.py                  # Batch run configuration
 │   ├── early_stop.py              # validate_early_stop guardrails + EarlyStopWatcher (armed live-verdict observer)
 │   ├── evaluation.py              # Reference dir resolution + per-run private staging
+│   ├── regrade.py                 # Grade an already-executed run in place — shared by `evaluate <run_dir>` and `run --resume`
 │   ├── experiment.py              # ExperimentRunner, resolve_task_for_variant, load_experiment
 │   └── task_loader.py             # YAML task loading
 │
 ├── cli/                           # CLI commands (Typer + Rich)
 │   ├── __init__.py                # Typer app setup (core commands)
-│   ├── run_command.py             # `coder-eval run`
+│   ├── run_command.py             # `coder-eval run` + `run_pipeline` (the body BOTH run and execute share)
+│   ├── execute_command.py         # `coder-eval execute` — Typer signature only; delegates to run_pipeline(grade=False)
 │   ├── plan_command.py            # `coder-eval plan`
+│   ├── evaluate_command.py        # `coder-eval evaluate` (grade a dir, or re-grade a run dir) + `run_evaluation`
+│   ├── evaluate_target.py         # PURE shape detection for evaluate's positionals (run dir ⟺ holds task.json)
 │   ├── report_command.py          # `coder-eval report`
+│   ├── aggregate_command.py       # `coder-eval aggregate` — rebuild run.json from the task.json rows on disk (the step right after `execute`)
+│   ├── run_task_internal_command.py  # `_run-task-internal` — the in-container entry point DockerRunner invokes; never called by a user
 │   ├── run_helpers.py             # CLI helper functions
 │   ├── console.py                 # Rich console instance
 │   └── utils.py                   # CLI utilities
@@ -144,8 +157,12 @@ action.yml                         # Published composite GitHub Action (coder-ev
 - **Sub-agent token accounting**: There is NO separate per-sub-agent field. Every sub-agent generation is captured as a `parent_tool_use_id`-tagged `AssistantMessage` in the turn transcript, so per-sub-agent usage is derived by grouping those messages on that id (the evalboard's `aggregateSubAgentUsage` does exactly this). Claude bubbles its sub-agent's intermediate generations into the parent stream natively, and the **terminal** generation (delivered as the Agent tool result, never streamed) is synthesized into one via `_synthesize_subagent_terminal_message` from `tool_use_result.usage`. Codex reconstructs all child generations from the child rollout (`_recover_subagent_tool_calls`). The turn total already includes sub-agent cost — Claude via the SDK's cumulative `model_usage`; Codex via `_fold_subagent_tokens`, which folds the child messages (their real per-generation tokens) into the parent total. `CommandTelemetry.result_summary` is stored **untruncated** (no 200-char cap) so sub-agent returns are preserved whole. Set `CODER_EVAL_RAW_SDK_LOG=1` to dump every raw SDK event to the task log for inspection.
 - **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The LiteLLM open-weight actual-cost join (`litellm_cost.apply_actual_cost`) deliberately writes cost at the TURN level only (`token_usage.total_cost_usd` = the real OpenRouter bill) plus the per-call `TurnRecord.provider_call_costs` audit record; it does NOT touch the message token buckets, so `EventCollector` stays the single writer and this invariant holds on every backend. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports.
 - **Reference solutions are directory-only, and shielded (partially) from the agent**: `task.reference` is a single required `directory:` (relative to the task YAML) — the inline `code:` / single-file `file:` forms are gone, because a directory is the only shape that can be permission-gated as a unit; a `model_validator(mode="before")` gives the removed forms a migration error. The orchestrator stages a **per-run private copy** (`orchestration/evaluation.py::stage_reference_dir`, symlinks stripped) into a tempdir, removed in `_cleanup` via `path_utils.rmtree_restrictive` (keyed on `_reference_staging_root`, recorded BEFORE the copy so a failed copy still cleans up; `rmtree(ignore_errors=True)` silently declines on a tree left at 000) and deliberately never preserved into `run_dir/artifacts`. That copy is held at mode `000` for the whole of every `agent.communicate` call via **`Sandbox.set_permissions`**, the driver-aware wrapper over `fs_permissions.py::set_permissions`. Windows **stack**: exiting restores the *enclosing* window's mode, only the outermost exit restores the pre-window mode — that is what makes a mid-turn re-grant (`mode=READ_ONLY_MODE`) expressible, and it covers two windows at the same mode so no refcount is needed. The window is enforced **only inside a docker container** (`Sandbox.enforces_permission_windows`) and is a no-op on the host, where the agent shares our uid. **That gate keys on the `CODER_EVAL_IN_CONTAINER` env var, NOT `sandbox.driver`** — `run_task_internal_command` rewrites `driver: docker` → `tempdir` before building the in-container Orchestrator, so a driver-based gate would silently disable the anti-cheat on exactly the path that needs it (regression-guarded by `TestSandboxDriverGate`); `resolve_reference_dir` gates its `/work/references` branch on the same var for the same reason. The task directory is **not** shielded (`:ro` mount → EROFS, and the same YAML is readable at `/work/input`). Criteria address reference files with the `$REFERENCE_DIR` token (same resolver as `$TASK_DIR`) and the `REFERENCE_DIR` env var for `run_command`; `reference_comparison` names one file via `reference_file`. Docker mounts a throwaway **read-write** copy at `/work/references` (a `:ro` mount cannot be chmod'd — EROFS), masks the in-task-dir original with an empty tmpfs, and drops `DAC_OVERRIDE`/`DAC_READ_SEARCH`. `FOWNER`/`CHOWN` are deliberately **NOT** dropped: the in-container orchestrator that applies the window is the same root process with the same caps, so dropping `FOWNER` breaks *the harness's own* chmod wherever the bind mount preserves a non-root owner (native Linux — verified: `chmod: Operation not permitted`), i.e. exactly where the drop would otherwise bite. A window that cannot be applied is now a hard error, not a warning: `Sandbox.set_permissions` passes `strict=True` whenever it enforces, so an unprotected run fails instead of producing a normal-looking score. **KNOWN GAP — this is defense-in-depth, not a boundary**: (a) `chmod(2)` is gated on owner-or-`CAP_FOWNER` and the container runs as root owning the copy, so a deliberate `chmod 755 /work/references` restores access; (b) the window spans `agent.communicate` only, and nothing reaps agent child processes at turn end, so a backgrounded read loop succeeds once the window closes. The **write** half of (b) is closed — `path_utils.digest_tree` hashes the tree at staging and `Orchestrator._verify_reference_integrity` re-checks before grading, raising `ReferenceTamperedError` (→ `FinalStatus.ERROR`) on a mismatch so an agent cannot overwrite the reference to drive `reference_comparison` to 1.0. Passive reads are blocked; an adversarial agent is not. Full containment requires running the agent as a non-root uid AND holding the window for the agent's whole lifetime — follow-up. `tasks/anti_cheat_reference` probes the passive-read half.
-- **Harness run-limit parity**: a shared `BaseAgentConfig` field must mean the same thing on every backend, so a divergence is either fixed or documented — never silent. **`run_limits.max_turns` on Codex/Antigravity counts VISIBLE turns** (resolved tool calls, read live off the shared `EventCollector.visible_turn_count`, the same list `TurnRecord.commands` holds) because one `communicate()` is a single SDK turn on both, so a native counter would clamp at 1; claude-code keeps its native SDK cap, whose unit (an agent-loop turn) absorbs arbitrarily many parallel calls — the same number is NOT the same budget across harnesses. OpenCode likewise keeps a native unit — the CLI streams a real multi-step loop per `communicate()` (`step_start`/`step_finish`), so `max_turns: N` allows N complete steps and cuts cleanly when step N+1 begins. The cap is enforced on the same loop boundary as the cooperative early stop and finalizes cleanly as `max_turns_exhausted` (no crash, no retry); on Antigravity that boundary lives in `_drain()`, so the background-work poll loop honors it too. Known unfixed divergences: `permission_mode` on Codex and Antigravity (both run unconfined — the sandbox driver is the isolation boundary), `disallowed_tools` on Codex (forwarded, not SDK-enforced), `allowed_tools`/`disallowed_tools` on Antigravity (not read at all), `turn_timeout` on Antigravity (bounded by an earlier internal poll deadline at 80% of it), `allowed_tools`/`disallowed_tools`/`system_prompt` on OpenCode (no CLI knob; warned at `start()`, not enforced), and **`agent.plugins[].path` depth** — claude-code REQUIRES a plugin root holding `skills/` and silently loads NOTHING from a bare skills directory, while Codex and Antigravity scan both depths and accept either. That is the costly direction: the wrong depth produces no error, every positive row of an activation suite scores 0, and the suite reports recall 0.0, which reads exactly like a skill that never triggers. Held to the plugin-root shape (for `SKILL_SOURCE_PATH` only) by lint rule CE045. `plugins` on OpenCode is **honored for skills**: each local plugin root is mapped to the `skills` dir its `.claude-plugin/plugin.json` declares (default `<root>/skills`, never the root itself — `skills.paths` is scanned recursively and a plugin root can hold a self-referential symlink) and injected via `OPENCODE_CONFIG_CONTENT`, which `--pure` does not suppress; a plugin's agents/hooks/commands/MCP servers are still dropped. Full table + rationale: docs/agents/HARNESS_PARITY.md.
+- **Harness run-limit parity**: a shared `BaseAgentConfig` field must mean the same thing on every backend, so a divergence is either fixed or documented — never silent. **`run_limits.max_turns` on Codex/Antigravity counts VISIBLE turns** (resolved tool calls, read live off the shared `EventCollector.visible_turn_count`, the same list `TurnRecord.commands` holds) because one `communicate()` is a single SDK turn on both, so a native counter would clamp at 1; claude-code keeps its native SDK cap, whose unit (an agent-loop turn) absorbs arbitrarily many parallel calls — the same number is NOT the same budget across harnesses. OpenCode likewise keeps a native unit — the CLI streams a real multi-step loop per `communicate()` (`step_start`/`step_finish`), so `max_turns: N` allows N complete steps and cuts cleanly when step N+1 begins. **Pi** is the same shape — the CLI (`pi -p --mode json`) streams a real multi-step loop per `communicate()` (`turn_start`/`turn_end`), so `max_turns: N` counts native `turn_start` steps; Pi retries transient/provider errors INTERNALLY (`agent_end.willRetry`), and the reducer finalizes once at `agent_settled`/EOF (not the first `agent_end`), folding the retry cycles into one turn. The cap is enforced on the same loop boundary as the cooperative early stop and finalizes cleanly as `max_turns_exhausted` (no crash, no retry); on Antigravity that boundary lives in `_drain()`, so the background-work poll loop honors it too. Known unfixed divergences: `permission_mode` on Codex, Antigravity, and Pi (all run unconfined — the sandbox driver is the isolation boundary), `disallowed_tools` on Codex (forwarded, not SDK-enforced), `allowed_tools`/`disallowed_tools` on Antigravity (not read at all), `turn_timeout` on Antigravity (bounded by an earlier internal poll deadline at 80% of it), `allowed_tools`/`disallowed_tools`/`system_prompt` on OpenCode (no CLI knob; warned at `start()`, not enforced), `allowed_tools`/`disallowed_tools` on Pi (its built-in tool names are lowercase — `bash`/`read`/… — and cannot map to the Claude-namespaced config default, so forwarding them would strip the agent of ALL tools; warned+ignored like the three agents above) — Pi DOES enforce `system_prompt` (`--append-system-prompt`, a small win over OpenCode) and DOES honor `plugins` for skills (each resolved skills dir → a `--skill <dir>` arg via the shared `_plugin_skill_dirs` resolver, recorded as `pi_skill_paths`, so it CAN run activation suites) but does NOT read `system_prompt_file`, and **`agent.plugins[].path` depth** — claude-code REQUIRES a plugin root holding `skills/` and silently loads NOTHING from a bare skills directory, while Codex and Antigravity scan both depths and accept either. That is the costly direction: the wrong depth produces no error, every positive row of an activation suite scores 0, and the suite reports recall 0.0, which reads exactly like a skill that never triggers. Held to the plugin-root shape (for `SKILL_SOURCE_PATH` only) by lint rule CE045. `plugins` on OpenCode is **honored for skills**: each local plugin root is mapped to the `skills` dir its `.claude-plugin/plugin.json` declares (default `<root>/skills`, never the root itself — `skills.paths` is scanned recursively and a plugin root can hold a self-referential symlink) and injected via `OPENCODE_CONFIG_CONTENT`, which `--pure` does not suppress; a plugin's agents/hooks/commands/MCP servers are still dropped. Full table + rationale: docs/agents/HARNESS_PARITY.md.
 - **sandbox isolation**: Tasks that don't need MCP servers should set `setting_sources: []` in their `agent:` block to isolate the sandbox from the host project's CLAUDE.md and settings. Without this, the host project's CLAUDE.md (often 20 KB+) is injected into every API call, inflating cache-creation tokens and cost significantly.
+- **Execute vs. run (the grading switch)**: `coder-eval execute` is `coder-eval run` with grading removed — the agent runs and the full trajectory is captured, but no criterion is checked, `weighted_score` is `None` (never `0.0`, which would be indistinguishable from "graded and scored zero"), and the row finalizes as **`FinalStatus.NOT_GRADED`**, whose `category` is a **fourth** bucket, `"ungraded"`. Ungraded rows leave BOTH sides of every rate: `RunSummary.pass_rate` / `error_share` and `VariantAggregate.pass_rate` divide by `tasks_graded` (`tasks_run - tasks_not_graded`), and `tasks_not_graded` is part of the sum-to-`tasks_run` invariant, not a `tasks_failed` sub-counter. **Only SUCCESS/FAILURE collapse into it** — `ERROR`, `TIMEOUT`, `BUILD_FAILED`, `MAX_TURNS_EXHAUSTED` and the budget stops are facts about the *run*, not about grading, and still apply (so `execute` still exits non-zero on a crash). The switch is `BatchRunConfig.grade` → `Orchestrator(grade=...)` → the **four** grading call sites (single-shot, evaluate-only, the simulation dialog check, and post-failure diagnostics); it crosses the docker boundary in `context.json` (defaulting to `True` in-container, so a host predating `execute` keeps grading). It is **deliberately not a task-config field** — no 5-layer merge, no `-D` path — because a task YAML must never declare itself ungraded; only the invoking command decides. `run` and `execute` share one body (`run_command.run_pipeline`) and differ solely in that flag, so there is no third code path. Three things are refused rather than degraded: `--junit-xml` (a report of verdicts, and there are none — though `reports_junit` still emits `<skipped>` for an ungraded row it encounters), `--allow-host-grading` (it decides how an ungraded row is GRADED, and `execute` grades nothing), and simulation tasks (their turn-continuation logic reads criteria results, so an ungraded dialog would silently change its own stopping behavior). `stop_early:` blocks are inert under `execute` for the same reason the kill switch exists: the full trajectory is the deliverable. Motivating consumer: an external harness (Harbor / Terminal-Bench 2.0) that builds its own container, calls coder-eval as the agent, and grades with its own tests.
+- **Detached grading (`evaluate` over a run dir) + `Sandbox.adopt`**: `coder-eval evaluate` takes two shapes, told apart by a **pure** resolver (`cli/evaluate_target.py`) on one probe — a target holding `task.json` is a run directory. Run-dir mode rebuilds the task from the run's own `task_config.resolved`, **not** by re-loading the YAML: `resolved` is post-merge, so variant overrides / `-D` / dataset expansion are already baked in and re-loading the source would silently grade a *different* task (fallback to `source_file` only when `resolved` no longer validates, and loudly). It seeds the fresh result from the prior one via `Orchestrator(prior_result=...)` → `_seed_from_prior_result`, which carries the trajectory (every derived figure — tokens, cost, `command_stats`, `model_used` — recomputes from `iterations`), `iteration_count`, execution facts, and **`early_stop` — load-bearing, because gate selection is FIRED-ONLY**: dropping it re-grades a truncated trajectory under the full-run strict-AND gate and can flip the verdict. Carrying it is only half the fix — **both** grading paths select the gate through the single `Orchestrator._select_gate()`; the evaluate-only branch a detached grade actually takes originally called `all_criteria_passed` inline, so the seeded field was written and never read. `tests/test_seed_from_prior_result.py` partitions every `EvaluationResult` field as CARRIED or RECOMPUTED and fails closed on a new one, and asserts the two `_select_gate()` call sites. A prior status that `FinalStatus.is_execution_fact` (TIMEOUT / ERROR / BUILD_FAILED / the budget stops) is **preserved**, never overwritten: grading may only move `NOT_GRADED` to SUCCESS/FAILURE, since it neither repeated nor observed the agent phase. **`MAX_TURNS_EXHAUSTED` is deliberately NOT one of them — anywhere**. `_EXECUTION_FACT_STATUSES` maps it to `False`, and the table and the chain that reads it must agree: it shipped as `True` while `_terminal_status`'s own docstring argued the opposite, and the disagreement pinned a re-graded max-turns row at MAX_TURNS_EXHAUSTED *while holding `weighted_score` 1.000* and exit 1 — a combination `run` can never produce for the same trajectory. Under `execute`: `_terminal_status` puts the `grade=False` arm ABOVE it, because on the graded path it is subordinate to the verdict — `run` returns SUCCESS for a max-turns trajectory whose criteria pass — so it is not knowable without grading. Consuming it first made it terminal AND permanent (the `is_execution_fact` arm then pinned it), so identical agent output scored SUCCESS/1.0 under `run` and MAX_TURNS_EXHAUSTED under `execute` → `evaluate`. The fact survives on `result.max_turns_exhausted`, which `_seed_from_prior_result` carries, so the detached grade walks the identical chain. The CLI must also branch on WHERE a status came from, not on its value: a preserved TIMEOUT exited 0 under "All criteria passed" (a CI wrapper reading the exit code went green on a row run.json counts as failed), and a preserved ERROR printed the ORIGINAL run's crash message as though grading had crashed, claimed the row was "left ungraded" (false — the restored record still read ERROR), and discarded a verdict just computed at 1.000. Grader-host `environment_info` is preserved as flat `graded_by_*` scalars rather than overwriting the run's (flat, not a nested sub-dict: `environment_info` is rendered as a flat map by the HTML report and typed as one by the evalboard, so a nested capture prints as a Python dict repr). The route recorder follows the same rule: on a detached grade it writes `graded_by_api_routing` / `graded_by_eval_routing` and leaves the run's `api_routing` alone — writing in place contradicted the "prior wins" contract and left a self-contradictory record (a direct route named beside the run's stale `aws_region`/`bedrock_model`). Two other parity fixes: `command_base_path` is now persisted into `environment_info` by `_sync_sandbox_command_path_with_agent` and restored in the evaluate-only branch (closing the PATH gap that method's docstring already named), and `_join_litellm_actual_cost` **skips** when `prior_result` is set (its join keys on a per-Orchestrator nonce the prior turns never carried, so it would clobber already-correct costs). The verdict is written back into the run's `task.json`, with the pre-grade record kept as `task.execute.json` — that in-place write is what makes plain `coder-eval aggregate <run_dir>` rebuild a graded `run.json` with **zero** new code. **`Sandbox.adopt(workspace)`** is the grade-in-place primitive: it reuses `setup`'s adoption half but skips every *materializing* step (`_setup_template`, `_generate_cli_recorders`, venv/package installs, the destructive `$HOME` remediation), running only non-mutating derivation (mock-dir `+x`, venv *discovery*, plugin-tools pin); `_cleanup_on_exit` stays False so an adopted tree is never moved or deleted, and `Sandbox.was_adopted` is set — the Orchestrator reads it to SKIP the `pre_run` hook (`run()` calls it unconditionally with `cwd = sandbox_dir`, and several in-tree tasks stage fixtures there with `cp -a /app/[!.]* "$PWD/"`, which would overwrite the agent's deliverables before the criteria read them) and to KEEP `sandbox_path` in the `PreservationMode.NONE` cleanup arm (an adopted tree survives cleanup, so the path is not stale). `pre_run`'s recorded results are carried from the prior run instead. **`post_run` is the opposite case and moved phases**: it is defined as running after the verdict and may mutate the workspace the criteria read (`rm -rf node_modules` is the archetype), so running it under `execute` inverted its own contract and broke round-trip equivalence — the criteria had not read the tree yet, so `execute` + `evaluate` graded a workspace `post_run` had already modified and could return a different verdict than a single `run` for the identical trajectory (the in-tree tasks all escaped it only because their `post_run` touches nothing a criterion reads). `execute` now DEFERS it; whichever command grades runs it, exactly once — `_skip_post_run` skips on `grade=False`, and skips again when the prior row already recorded results, since nothing declares these commands idempotent. That makes it a capability of the in-place path, so `embedded_commands` scans it OUTSIDE `include_setup_phase` (which is False in place) — minus `_operator_baseline_post_run()`, the grading host's own `experiments/default.yaml` contribution, which every task carries and the record therefore did not choose; without that exemption the refusal fired on 100% of run directories, and a refusal that always fires is waved through. In-place is **more correct**, not merely faster: `_setup_template` filters the copy through `_should_ignore_template_file`, which drops `node_modules` / `dist` / `build` / `.venv` / `.git`, so on the copy path a criterion like `test -f dist/bundle.js` fails as a *copying artifact* rather than as a verdict (verified: 0.00 "does not exist" on copy vs 1.00 in place). Defaults: in-place for a run dir, copy for a bare work dir (criteria can mutate it and it is the user's own tree); `--in-place`/`--copy` override. `adopt` hard-errors on `driver: docker` (a container workspace is unreachable from the host), and grading a `driver: docker` task is REFUSED outright unless `--allow-host-grading` is passed — the earlier behavior silently rewrote the driver to `tempdir`, which ran a container task's criteria against a host filesystem lacking `/verifier` and the image's toolchain (FAILURE for a trajectory `run` scored 1.0, plus `rm -rf /verifier` unsandboxed on the grading machine) and neutralized `adopt`'s own docker guard; an opted-in row is stamped `graded_on_host` so it is never silently comparable with a container-graded one (lint rule CE051). A re-grade refuses on a `reference_digest` mismatch — the digest is persisted into `environment_info` at staging time by `_stage_reference` (it shipped once as a read with no writer anywhere, so the guard was dead code; then it shipped with a writer whose value was **discarded before it reached disk**, because `_setup` REBOUND the whole `environment_info` dict from `get_version_info()` a hundred lines later, which CE054 cannot see — a write existed in `src/`, it was just dead. `_setup` now `update()`s that dict rather than rebinding it, and `tests/test_detached_grading_boundaries.py` asserts the key survives a real end-to-end run, not just that `_staged_digest` works in isolation), and `verify_reference_unchanged` now takes the task file it resolves against and RAISES on a vanished or unresolvable reference instead of returning silently. That comparison digests a STAGED copy of the source, not the raw tree: the recorded digest is taken over the staged copy, which `stage_reference_dir` filters through `REFERENCE_COPY_IGNORE` (`.git`), so digesting the source directly compared two differently-filtered trees and reported a permanent false mismatch for any reference that is a git checkout — exactly the case the ignore list exists for. **The recorded config is untrusted input**: `evaluate <run_dir>` rebuilds the task from a shareable artifact, so a rebuilt config that carries shell (`run_command` criteria, `agent_judge`, `llm_judge`, `uipath_eval`, and — only on the `--copy` path, since the in-place path skips them — `pre_run`/`post_run` **and the sandbox's own provisioning**) is REFUSED unless `--allow-recorded-commands` is passed. The provisioning half was the one the gate originally missed, and the worst: `grading_sandbox_config` carries the recorded `sandbox` block through untouched and the `--copy` branch calls `Sandbox.setup`, which reaches `uv pip install <recorded packages>` / `npm install` / `git clone <recorded url>` — arbitrary code at install time — so a shared run dir whose criteria were all `file_exists` sailed through a scan that walked only `success_criteria`. `git clone` now passes `--` before the URL (argv position 2, so a value beginning with `-` was parsed as an option). `Sandbox.resolve_files` is containment-checked for the same reason: criterion paths were the one task-authored path skipping `_resolve_within_sandbox`, and `Path(root) / '/etc/passwd'` is `/etc/passwd`. An escaping LITERAL now raises `CheckerMisuseError` rather than resolving to `[]`: returning no match books an eval-CONFIG error as an agent failure — a gating 0.0 reading "file does not exist" for a file that plainly does exist and that no agent behaviour could place inside the sandbox (CE039's exact distinction). `tasks/byod_smoke_test.yaml` was broken that way for several commits, checking `/opt/byod_marker` baked into the BYOD image, with only a task-log warning to show for it; it now asserts on the container with `run_command: test -f …`, which is what a claim about the IMAGE rather than about the agent's workspace should look like. The guard keys on the escaping path EXISTING, so a merely-absent absolute path stays an ordinary failing verdict, and the GLOB branch still warns-and-drops, since filtering some matches out of a search is its normal behaviour. A warning is not a control: it prints as the command is already being prepared. Passing the task file explicitly (`evaluate <task.yaml> <run_dir>`) also bypasses it, since that config came from the operator. The workspace fallback `artifacts / prior.task_id` is containment-checked like its `sandbox_path` sibling (`task_id` is an unvalidated string, and `"../../.."` joins to a real directory `is_dir()` confirms), `_sanitize_restored_path` drops relative entries (they resolve against the grader's cwd) and anything inside the run dir rather than only the workspace, and `write_text_atomic` opens its temp file `O_EXCL|O_NOFOLLOW` — a pre-planted `task.json.tmp` symlink otherwise bypassed the write-back's destination symlink guard entirely. The record must also describe the task as AUTHORED, not as executed: `run_task_internal_command` rewrites `driver: docker` -> `tempdir` before building the in-container Orchestrator (the one legitimate rewrite — we are already inside the container the driver asked for), and recording that rewrite made a docker run's own `task.json` claim `driver: tempdir`. Since `grading_sandbox_config` reads the driver back OUT of the record, `evaluate <run_dir>` on a container row skipped BOTH the `--allow-host-grading` refusal and the `graded_on_host` stamp and graded a container task against the host filesystem silently — the exact outcome that gate exists to prevent. `Orchestrator(recorded_task=...)` is the seam: what is recorded, as distinct from what is run. NOTE the Typer command is a thin wrapper over `run_evaluation(...)`, which has real Python defaults — calling a Typer command function in-process hands unspecified options an `OptionInfo` sentinel, which silently made `in_place=None` truthy.
+- **`--resume` is command-relative**: `partition_for_resume(tasks, *, grade)` returns a four-way `ResumePartition` (`to_run` / `to_grade` / `prior_results` / `prior_resolved`), because **"finished" is not absolute — it depends on what the resuming command still owes the task**. A `NOT_GRADED` row carries a final status, so the original "has any final status" test called it complete: right for `execute --resume` (it finished executing), and wrong for `run --resume`, which was asked to grade and would instead report "already complete", grade nothing, and **exit 0**. The routing test is the row's **evidence** (`weighted_score is None and not success_criteria_results`), not its category: keying on `category == "ungraded"` missed every `execute` row that ALSO carries an execution fact — a TIMEOUT or budget stop aborts before grading, so it lands unscored with category `error`/`failed`, and resume filed it as complete while `evaluate <run_dir>` graded the identical bytes happily. Under `grade=True` those rows route to `to_grade`, where `_grade_resumed_tasks` runs the criteria against the trajectory and workspace already on disk via `orchestration/regrade.py::regrade_in_place` — reusing the agent spend, which is the entire reason `execute` and `run` are separate. The carve-out is **only** for `NOT_GRADED`: `FAILURE`/`ERROR` stay complete under both commands (resume has never retried failures — delete the task.json), and `clear_rerun_artifacts` deliberately skips `to_grade`, whose artifacts are the very thing being graded. A per-task grading failure is warned, STAMPED onto the folded-back row's `error_message` (the console line alone is not durable), and folded back in with its ORIGINAL ungraded result, so one bad row neither aborts the resume nor vanishes from run.json — and the exit gate counts `tasks_not_graded` **when `grade` is True**, so a `run` that graded nothing exits non-zero instead of telling CI the suite is fine. Under `execute` an ungraded row is the expected outcome and never fails the command. A row is owed a grade only when it was **executed** AND is unscored: evidence of "no verdict" alone routed every dead container and failed image build (`_write_synthetic_task_json` writes those with no verdict either) into grading, where the fold-back replaced the real diagnostic with a wrong-cause grading error and left `task.json` and `run.json` disagreeing about the same row — so the test is `final_status is NOT_GRADED or iteration_count > 0`, and that fold-back now APPENDS to `error_message` instead of replacing it. A re-grade also writes its log to **`grade.log`**, never `task.log`: `task_log_handler` opens `mode="w"`, so grading into the row's own directory truncated the agent trajectory log the run had already paid for — contradicting `_apply_resume`'s own "to_grade is deliberately NOT cleared" contract. `grade` is in `_FINGERPRINT_DIFF_EXEMPT` because `execute` → `run --resume` is a supported flow, not config drift — and the warning's "already-finalized tasks keep their original-config results" text is actively wrong for it. **`orchestration/regrade.py` is the single implementation** shared by that path and `evaluate`'s run-dir mode, which DELEGATES to `regrade_in_place` rather than restating it (it originally hand-built its own Sandbox + Orchestrator and had already drifted — hardcoding `replicate_index=0`, so every replicate but the first was relabelled — which is exactly how two copies become two verdicts for the same run); it raises plain `RegradeError`, which the CLI wraps, since `orchestration/` must not import the CLI layer (CE004). One fidelity rule it enforces: a re-graded row keeps the **agent run's** `started_at`/`duration_seconds`, not the grading pass's — a 10-minute run re-graded in 2s would otherwise report 2s into `average_duration`, the report tables and the evalboard; the grading cost is preserved separately as `environment_info["grading_duration_seconds"]`.
+- **One formula per published rate**: `pass_rate` / `error_share` are published by THREE models (`RunSummary`, `VariantAggregate`, `SuiteRollup`) and all three route through the single `models/results.py::nothing_was_measured(not_graded=, measured=)`. The guard originally shipped on `RunSummary` alone, so the same 10-task `execute` run with one crash rendered "Pass Rate: n/a" in `run.md` and "Pass Rate: 0.0%" in `experiment.md`. `measured` is **counted evidence** (`tasks_measured` / `rows_measured` — rows carrying a `weighted_score`), never a bucket count: the first version tested `tasks_succeeded + tasks_failed == 0`, but `TIMEOUT` and the two budget stops are category `failed` and reachable under `execute` (`_check_run_limits` still runs on the ungraded branch), so ONE timed-out row in a 100-task ungraded night read as "measured" and published `pass_rate: 0.0` — a real 0% point on the evalboard trend for a run that graded nothing. The evalboard mirrors the rule: `TaskTrend.passRate` is `number | null`, and an unmeasured task renders "—" and sorts LAST in the worst-first Trends view rather than to the very top as the worst offender.
 - **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all *task-level* run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block. The one *per-criterion* cap, `stop_early.decide_within`, deliberately lives on `LiveSuccessCriterion` instead (see below) — the watcher must attribute a decision-step timeout to a specific criterion, which `RunLimits` (task-scoped, criterion-agnostic) cannot express.
 - **Early stop on criterion (opt-in, per-criterion arming)**: a `stop_early:` block (`StopEarlyPolicy`) on a criterion ends a single-shot run early once the run's **armed** criteria decide the outcome, so a raised `max_turns` isn't wasted on the smoke flavor. The block's PRESENCE is the arming and alone activates the watcher — there is **no run-level master switch**: `run_limits.stop_early: false` is the run-level KILL SWITCH that force-disarms every block (the one-line experiment-variant/`-D` override for an authoritative full run), and `run_limits.stop_early: true` (the removed master arm) is a hard `EarlyStopConfigError` at resolution. The block exists on `LiveSuccessCriterion` only (currently `skill_triggered`, `command_executed` — so arming an unobservable criterion is unrepresentable, a pydantic extra-forbid error). Arming carries one implicit trigger (a native live-fail may fail-stop the run); its keys refine it: `on_pass: stop` (pass-stop the moment the criterion live-passes; default `continue` just latches) and `decide_within: N` (still undecided after N tool-call steps latches an **effective fail**, fed through the same fail-stop rule, reported as `decision_budget_exceeded` — an ordinary weighted fail, NOT a gate-bypassing force-fail; cumulative across retry attempts of the same turn). A trigger whose polarity the instance can't decide (per the abstract, checker-independent `live_decidable_polarities()`, a pure function of the criterion's own fields, paired with the checker's `live_verdict` override by lint rule CE025, a registry-based whole-tree check) is **inert by design** — one dataset-fanned YAML line serves both positive rows (pass/timeout live) and distractor rows (fail live). Verdicts **latch**: once a criterion decides, its `live_verdict` is never polled again. Stop rule is weighted, not strict-boolean: `run_limits.stop_early_gate_threshold` (default `1.0`, reproducing strict-AND behavior exactly) is the minimum weighted score (`Σ weight·score / Σ weight` over the armed subset) required to pass; a fail-stop fires once the armed set's **ceiling** (best case for everything still undecided) can no longer reach the threshold — so a low-weight fail or timeout that can't doom the gate is absorbed and the run continues — and is **deferred while any pass-capable armed criterion is undecided** (a distractor misfire never truncates a positive row's recall signal); a pass-stop fires once the `on_pass: stop` subset's **floor** (worst case) already meets the threshold, and is symmetrically **deferred while any pass-capable armed criterion outside the `on_pass: stop` subset is undecided** (so an early pass never freezes a sibling `on_pass: continue` criterion's signal out of the trajectory). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a kill-switched (`stop_early: false`) run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` (built when `early_stop_active(task)`: ≥1 armed criterion, kill switch not thrown) through the agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. Gating is **FIRED-ONLY**: a run the watcher actually cut gates on the **armed subset** via the weighted `EvaluationResult.armed_criteria_passed`; a run that completes naturally — armed or not — gates strict-AND via `all_criteria_passed`, so adding a block never changes the verdict of a run it didn't cut. Note the gate keys on the watcher having FIRED (`result.early_stop is not None`), not on confirmed truncation — an agent that ignores `should_stop`, or a stop firing on the final message, still gates armed-only. Every resolution-time guardrail violation is a hard error at resolution (plan *and* run); the one load-time case — a `stop_early:` block on a non-live criterion — is a pydantic schema error at task load, which the run surface reports as a skipped task like any other malformed task. A runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (incl. `gate_threshold` at stop time), report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. No blocks anywhere ⇒ behavior byte-for-byte unchanged.
 
@@ -216,7 +233,7 @@ make plugin-reference  # the plugin's bundled criteria reference from the models
 
 Editing `src/coder_eval/pricing.py` means editing `evalboard/lib/pricing.ts` too — it is a hand-copied mirror, and `evalboard/lib/__tests__/pricing-parity.test.ts` fails the build on drift in either direction.
 
-Recent additions, each traceable to a shipped defect: **CE037** (no unreferenced module-level private helper in `src/` — a helper whose docstring documents a bug the live code still has is worse than none), **CE038** (in an `@asynccontextmanager`, the acquire must sit INSIDE the `try` whose `finally` releases it — `asyncio.shield` protects the inner task, NOT the await, so a cancel on `__aenter__` skips the unwind while the work completes), **CE039** (a criterion checker must not return a gating `score=0.0` from an `except OSError` over a path the *task author* named — that books an eval-config error as an agent failure; raise `CheckerMisuseError` instead, and `# noqa: CE039` the cases that really are the agent's), **CE047** (every onboarding/marketing surface — README, `docs/index.md`, `docs/comparison.md`, `docs/llms.txt`, `mkdocs.yml`'s `site_description`, the Pages stub, and pyproject's `description`/`keywords` — must name every built-in `AgentKind`; OpenCode shipped while four of those seven still listed three harnesses, and nothing failed).
+Recent additions, each traceable to a shipped defect: **CE055** (a criterion `path:` in `tasks/` must be sandbox-relative — an absolute path is joined onto the sandbox root, which DISCARDS the root, so containment refuses it and the criterion can never match whatever the agent does; two in-tree tasks were broken this way and the pair is the argument for a static rule on top of the runtime `CheckerMisuseError`: `byod_smoke_test` IS in a CI bucket and produced only `Results: 7/8 succeeded` plus a gating 0.0 reading "file does not exist" for a file that existed, while `dockerfile_build_example` is in NO bucket, so nothing ran it and no runtime guard was ever reached — the fix is never to relax containment but to say what the criterion means, `run_command: test -f /opt/marker`, a claim about the container IMAGE rather than about the agent's workspace), **CE054** (an `environment_info` key that is READ must be WRITTEN somewhere in `src/` — the bag is `dict[str, Any]`, so nothing connects reader to writer, and the `reference_digest` anti-cheat guard shipped as a read with no writer anywhere: `.get()` returned `None`, the guard took its early return, and CLAUDE.md plus the user guide both described it as protection it never provided), **CE048** (never call a Typer command function in process — its parameter defaults are `OptionInfo` sentinels, not values, and the sentinel is TRUTHY, so `in_place=None` silently selected the wrong branch; the fix is the `run_pipeline` / `run_evaluation` / `run_plan` split, and this rule is the one that also scans `tests/`, since that is the only place the defect occurs), **CE049** (never coalesce a possibly-unmeasured score to a numeric literal — `score or 0.0` publishes "measured and scored zero" while meaning "never measured", which is how an ungraded night reached four unfiltered `avg(Score)` dashboards as a real zero), **CE050** (no untyped `getattr` probe for a discriminated-union field — pyright cannot see the string, so a rename degrades the guard to a permanent no-op; scoped to criterion-shaped receivers because `command`/`tool`/`prompt` are far too common to flag on their own), **CE051** (a sandbox driver may not be rewritten silently — the driver IS the isolation boundary, so a downgrade must be an explicit, stamped, operator-visible decision), **CE053** (no bare run-record filename literal outside `path_utils` — `TASK_JSON_FILENAME` shipped with a rename-safety rationale while twelve exact literals stayed unmigrated, including all three `rglob("task.json")` sites the constant's own comment cites as its reason to exist, so it created the second source of truth it argues against), **CE052** (an `os._exit` must sit inside a branch testing `CODER_EVAL_IN_CONTAINER` — it is the right primitive only for reaping the container's own disposable main process, and `run_task_internal_command` armed its heartbeat watchdog, a daemon thread whose whole authority is `os._exit(137)`, unconditionally: a test that invoked the command in-process left the pytest worker holding that thread, which exited the worker 40s later inside an unrelated test file, naming a different test on each run and on each platform with no traceback — and the dead worker's lost coverage data then failed the gate as `65.13 < 80.00`, naming neither the test nor the cause), **CE037** (no unreferenced module-level private helper in `src/` — a helper whose docstring documents a bug the live code still has is worse than none), **CE038** (in an `@asynccontextmanager`, the acquire must sit INSIDE the `try` whose `finally` releases it — `asyncio.shield` protects the inner task, NOT the await, so a cancel on `__aenter__` skips the unwind while the work completes), **CE039** (a criterion checker must not return a gating `score=0.0` from an `except OSError` over a path the *task author* named — that books an eval-config error as an agent failure; raise `CheckerMisuseError` instead, and `# noqa: CE039` the cases that really are the agent's), **CE047** (every onboarding/marketing surface — README, `docs/index.md`, `docs/comparison.md`, `docs/llms.txt`, `mkdocs.yml`'s `site_description`, the Pages stub, and pyproject's `description`/`keywords` — must name every built-in `AgentKind`; OpenCode shipped while four of those seven still listed three harnesses, and nothing failed).
 
 When fixing a bug, ask: *could a custom lint rule have prevented this?* If the root cause is a mechanically detectable pattern (e.g., "always import from `coder_eval.models`", "never call blocking IO in async"), add a rule to `tests/lint/rules/` following the CE001+ pattern and wire it up in `tests/lint/runner.py`. This turns a one-time fix into permanent enforcement. See `tests/test_custom_lint.py` for how rules are tested. (Doc-surface / whole-tree rules that reason over Markdown/YAML or the entire `src/` tree rather than one `.py` AST at a time — CE026–CE031, CE033–CE036 — are not `BaseRule`s in the runner; they are wired as dedicated `@pytest.mark.lint` test classes. CE036 enforces the `live_verdict` determinism + monotonicity contract (`criteria/base.py`) that `EarlyStopWatcher`'s latching, deferred fail-stop, and flip-attribution silently depend on: monotonicity over arbitrary Python is undecidable, so instead of a static check it REPLAYS each live criterion against every prefix of recorded trajectories (`tests/lint/live_verdict_contract.py::CASES`) — on the authored ordering AND under seeded shuffles (`permuted_violations`, which catch order-sensitive bugs the authored walk misses) — and asserts the property directly, plus registry-derived coverage — every `LiveSuccessCriterion` in the union must have cases, and every polarity its instances claim via `live_decidable_polarities()` must actually be reached by one (otherwise a single always-`undecided` fixture would "cover" a type while proving nothing). Adding a live criterion therefore means adding `ContractCase`s in the same change. CE035 resolves every `steps.<id>.outputs.<key>` / `needs.<job>.outputs.<key>` reference in `.github/workflows/**` to a writer that actually produces that key — GitHub expands an unwritten output to the empty string, so a typo degrades a gate silently and actionlint models `steps.*.outputs` as an open string map. CE034 scans `tasks/` and forces an armed, live-*passable* `command_executed` to set `require_success` — a crashed invocation would otherwise latch a live PASS, fire `on_pass: stop`, and let FIRED-ONLY armed gating report SUCCESS without ever consulting the unarmed criteria (negative assertions are fail-only and are exempt). CE033 keeps the plugin's bundled `reference/criteria.md` in parity with the `SuccessCriterion` union that generates it (`make plugin-reference` writes it; the rule re-renders and diffs — never hand-edit the file). CE031 guards against dead config: a behavior-driving field on `SimulationConfig`/`RunLimits`/`Dataset` that no code reads by name. CE026 keeps the GitHub Action's onboarding surfaces honest — `README.md`, `docs/CI_GATE.md`, `docs/tutorials/02-ci-pipeline.md`, and the plugin's `ci` skill, whose emitted workflow users copy into their own repos: a page's *first* Action snippet must show the agent-runtime prerequisite steps (pinned to the `action-dogfood` job that proves them in CI), a zero-install absolute next to such a snippet must name the channel it means, every `github.com/marketplace/actions/<slug>` link plus the shields badge label must match `action.yml`'s `name:`, and every `with:` key on a snippet's action step must be a real `action.yml` input (GitHub ignores unknown inputs, so a rename would silently degrade every copied workflow). Renaming an action input or changing its runtime prerequisites therefore means updating the skill too.)
 

{coder_eval-0.11.7 → coder_eval-0.12.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.5
 Name: coder-eval
-Version: 0.11.7
-Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode) with sandboxed, reproducible YAML task suites.
+Version: 0.12.0
+Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode, Pi) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval
 Project-URL: Documentation, https://coder-eval.com/docs
@@ -11,7 +11,7 @@ Author-email: UiPath <coder-eval@uipath.com>
 License-Expression: Apache-2.0
 License-File: LICENSE
 License-File: NOTICE
-Keywords: agent,agent-evaluation,agent-skills,agent-testing,ai,ai-evaluation,anthropic,antigravity,benchmark,claude,claude-code,claude-code-skills,claude-skills,code-generation,codex,coding-agent,eval,evals,evaluation,gemini,llm,llm-eval,llm-evaluation,llmops,opencode,sandbox,skills-evaluation,skillsbench,swe-bench
+Keywords: agent,agent-evaluation,agent-skills,agent-testing,ai,ai-evaluation,anthropic,antigravity,benchmark,claude,claude-code,claude-code-skills,claude-skills,code-generation,codex,coding-agent,eval,evals,evaluation,gemini,llm,llm-eval,llm-evaluation,llmops,opencode,pi,sandbox,skills-evaluation,skillsbench,swe-bench
 Classifier: Development Status :: 4 - Beta
 Classifier: Environment :: Console
 Classifier: Intended Audience :: Developers
@@ -61,6 +61,7 @@ Requires-Dist: ruff>=0.15.7; extra == 'dev'
 Provides-Extra: litellm
 Requires-Dist: litellm<2.0.0,>=1.95.0; extra == 'litellm'
 Provides-Extra: opencode
+Provides-Extra: pi
 Provides-Extra: uipath
 Requires-Dist: uipath>=2.10.31; extra == 'uipath'
 Description-Content-Type: text/markdown
@@ -84,14 +85,14 @@ Description-Content-Type: text/markdown
 open-source, **agent-agnostic** framework for **evaluating and benchmarking AI coding
 agents and their skills** — built for benchmark authors, CLI builders, and skill
 builders — with sandboxing, reproducibility, and data-driven analysis. It runs a real
-agent — **Claude Code**, **OpenAI Codex**, **Google Antigravity (Gemini)**, or
-**OpenCode** — in a sandbox against declarative YAML tasks, then scores the files and
+agent — **Claude Code**, **OpenAI Codex**, **Google Antigravity (Gemini)**,
+**OpenCode**, or **Pi** — in a sandbox against declarative YAML tasks, then scores the files and
 commands it actually produced. Changing harness is one field (`agent.type`); the
 tasks, criteria, scoring, telemetry, and reports stay the same.
 
 Reach for it when you want to **benchmark agents on your own domain tasks**,
 **test whether a skill triggers** in the agent you ship for, **A/B-test Claude Code
-vs. Codex vs. Gemini vs. OpenCode** (or model vs. model, prompt vs. prompt), or
+vs. Codex vs. Gemini vs. OpenCode vs. Pi** (or model vs. model, prompt vs. prompt), or
 **gate CI on coding-agent quality**. It is **not a fixed leaderboard**: unlike
 SWE-bench or SkillsBench, which rank models on a shared task set, you bring the tasks
 and you bring the scoring — weighted 0.0–1.0 criteria, a `skill_triggered` activation
@@ -107,7 +108,7 @@ care about. See [How it compares](https://coder-eval.com/docs/comparison).
 - **Sandboxed execution** in isolated environments with resource limits
 - **Weighted, continuous scoring** (0.0–1.0) with fractional credit and thresholds
 - **Many criterion types** — from file checks to code similarity and LLM-graded rubrics
-- **Agent-agnostic by design** — Claude Code, OpenAI Codex, Antigravity (Gemini), and OpenCode today; add your own harness through the plugin SPI
+- **Agent-agnostic by design** — Claude Code, OpenAI Codex, Antigravity (Gemini), OpenCode, and Pi today; add your own harness through the plugin SPI
 - **Experiment layer** — A/B agent configs (models, tools, prompts) side-by-side
 - **Full telemetry** — every tool call, token counts, and cost, with real-time streaming
 
@@ -129,7 +130,7 @@ care about. See [How it compares](https://coder-eval.com/docs/comparison).
 ## What you can do with it
 
 - **Benchmark coding agents** — score an agent across a suite of tasks with weighted scoring and pass/fail thresholds
-- **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini vs. OpenCode, model vs. model, tool-on vs. tool-off, prompt vs. prompt
+- **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini vs. OpenCode vs. Pi, model vs. model, tool-on vs. tool-off, prompt vs. prompt
 - **Evaluate skills** — verify an agent actually engages a target skill (`skill_triggered`) and score skill-driven suites (SkillsBench-style), on whichever harness your users run
 - **Keep skills up to date in CI** — re-validate your skills on every change or on a schedule; catch silent regressions when models, prompts, or the skills themselves drift
 - **Gate CI on agent quality** — run the suite in GitHub Actions and fail the build on regressions
@@ -319,6 +320,7 @@ The step's exit code is coder-eval's own: non-zero on any failed task.
 | [Codex](docs/agents/CODEX.md) | Running the OpenAI Codex agent |
 | [Antigravity (Gemini)](docs/agents/ANTIGRAVITY.md) | Running the Google Antigravity / Gemini agent |
 | [OpenCode](docs/agents/OPENCODE.md) | Running the OpenCode agent on open-weight models |
+| [Pi](docs/agents/PI.md) | Running the Pi agent on open-weight models |
 | [Run-Limit Parity](docs/agents/HARNESS_PARITY.md) | What each run_limits field means on every harness |
 | [A/B Experiments](docs/AB_EXPERIMENTS.md) | Compare models / tools / prompts across the same tasks |
 | [Bring Your Own Dataset](docs/DATASETS.md) | Fan a single task out over a dataset |

{coder_eval-0.11.7 → coder_eval-0.12.0}/README.md

@@ -17,14 +17,14 @@
 open-source, **agent-agnostic** framework for **evaluating and benchmarking AI coding
 agents and their skills** — built for benchmark authors, CLI builders, and skill
 builders — with sandboxing, reproducibility, and data-driven analysis. It runs a real
-agent — **Claude Code**, **OpenAI Codex**, **Google Antigravity (Gemini)**, or
-**OpenCode** — in a sandbox against declarative YAML tasks, then scores the files and
+agent — **Claude Code**, **OpenAI Codex**, **Google Antigravity (Gemini)**,
+**OpenCode**, or **Pi** — in a sandbox against declarative YAML tasks, then scores the files and
 commands it actually produced. Changing harness is one field (`agent.type`); the
 tasks, criteria, scoring, telemetry, and reports stay the same.
 
 Reach for it when you want to **benchmark agents on your own domain tasks**,
 **test whether a skill triggers** in the agent you ship for, **A/B-test Claude Code
-vs. Codex vs. Gemini vs. OpenCode** (or model vs. model, prompt vs. prompt), or
+vs. Codex vs. Gemini vs. OpenCode vs. Pi** (or model vs. model, prompt vs. prompt), or
 **gate CI on coding-agent quality**. It is **not a fixed leaderboard**: unlike
 SWE-bench or SkillsBench, which rank models on a shared task set, you bring the tasks
 and you bring the scoring — weighted 0.0–1.0 criteria, a `skill_triggered` activation
@@ -40,7 +40,7 @@ care about. See [How it compares](https://coder-eval.com/docs/comparison).
 - **Sandboxed execution** in isolated environments with resource limits
 - **Weighted, continuous scoring** (0.0–1.0) with fractional credit and thresholds
 - **Many criterion types** — from file checks to code similarity and LLM-graded rubrics
-- **Agent-agnostic by design** — Claude Code, OpenAI Codex, Antigravity (Gemini), and OpenCode today; add your own harness through the plugin SPI
+- **Agent-agnostic by design** — Claude Code, OpenAI Codex, Antigravity (Gemini), OpenCode, and Pi today; add your own harness through the plugin SPI
 - **Experiment layer** — A/B agent configs (models, tools, prompts) side-by-side
 - **Full telemetry** — every tool call, token counts, and cost, with real-time streaming
 
@@ -62,7 +62,7 @@ care about. See [How it compares](https://coder-eval.com/docs/comparison).
 ## What you can do with it
 
 - **Benchmark coding agents** — score an agent across a suite of tasks with weighted scoring and pass/fail thresholds
-- **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini vs. OpenCode, model vs. model, tool-on vs. tool-off, prompt vs. prompt
+- **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini vs. OpenCode vs. Pi, model vs. model, tool-on vs. tool-off, prompt vs. prompt
 - **Evaluate skills** — verify an agent actually engages a target skill (`skill_triggered`) and score skill-driven suites (SkillsBench-style), on whichever harness your users run
 - **Keep skills up to date in CI** — re-validate your skills on every change or on a schedule; catch silent regressions when models, prompts, or the skills themselves drift
 - **Gate CI on agent quality** — run the suite in GitHub Actions and fail the build on regressions
@@ -252,6 +252,7 @@ The step's exit code is coder-eval's own: non-zero on any failed task.
 | [Codex](docs/agents/CODEX.md) | Running the OpenAI Codex agent |
 | [Antigravity (Gemini)](docs/agents/ANTIGRAVITY.md) | Running the Google Antigravity / Gemini agent |
 | [OpenCode](docs/agents/OPENCODE.md) | Running the OpenCode agent on open-weight models |
+| [Pi](docs/agents/PI.md) | Running the Pi agent on open-weight models |
 | [Run-Limit Parity](docs/agents/HARNESS_PARITY.md) | What each run_limits field means on every harness |
 | [A/B Experiments](docs/AB_EXPERIMENTS.md) | Compare models / tools / prompts across the same tasks |
 | [Bring Your Own Dataset](docs/DATASETS.md) | Fan a single task out over a dataset |

{coder_eval-0.11.7 → coder_eval-0.12.0}/action.yml

@@ -25,7 +25,7 @@ inputs:
   version:
     description: coder-eval version to install from PyPI, or "local" to install from the action checkout
     required: false
-    default: "0.11.7"   # <-- kept in sync with releases by release.yml
+    default: "0.12.0"   # <-- kept in sync with releases by release.yml
   extras:
     description: >-
       Comma-separated coder-eval extras (`codex`, `antigravity,litellm`), composed

{coder_eval-0.11.7 → coder_eval-0.12.0}/docker/Dockerfile

@@ -36,6 +36,12 @@ RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
     && rm -rf /var/lib/apt/lists/* \
     && npm install -g @anthropic-ai/claude-code@${CLAUDE_CODE_VERSION}
 
+# Pi Node CLI, pinned — same rationale as the Claude Code pin above (the agent
+# binary is a dominant non-model driver of results; @latest would freeze
+# nondeterministically under layer caching). Node 22 + npm are already present.
+ARG PI_VERSION=0.84.4
+RUN npm install -g @earendil-works/pi-coding-agent@${PI_VERSION}
+
 # uv: matches host sandbox.py's `uv venv` + `uv pip install` fast path
 RUN curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR=/usr/local/bin sh
 
@@ -58,12 +64,16 @@ COPY experiments/default.yaml ./experiments/default.yaml
 # come from public PyPI, so this needs no private-index credentials. The RUN
 # below always passes `--extra codex --extra antigravity --extra litellm`.
 #
-# NOT every built-in agent ships here: `opencode` is registered unconditionally
-# but its CLI is a Node package (`npm install -g opencode-ai`), absent from this
-# image, and no OpenCode credentials are in SandboxConfig.env_passthrough -- so
-# `--driver docker` does not support it. Adding it means a pinned version that
-# travels with the release tag (as CLAUDE_CODE_VERSION does) plus an
-# env_passthrough block; see docs/agents/OPENCODE.md "Running in Docker".
+# `pi` IS baked above (pinned PI_VERSION) and its OPENROUTER_API_KEY provider
+# credential is in DockerDriverConfig.env_passthrough, so `--driver docker --type pi`
+# is supported. NOT every built-in agent ships here, though: `opencode` is
+# registered unconditionally but its CLI is a Node package
+# (`npm install -g opencode-ai`), absent from this image -- so `--driver docker`
+# does not support it. (OPENROUTER_API_KEY IS now in env_passthrough, added for Pi,
+# so an OpenCode env-auth run would authenticate once the CLI is baked; the CLI is
+# the remaining blocker.) Adding it means a pinned version that travels with the release
+# tag (as CLAUDE_CODE_VERSION / PI_VERSION do) plus an env_passthrough block; see
+# docs/agents/OPENCODE.md "Running in Docker".
 #
 # CODER_EVAL_UV_EXTRAS carries ADDITIONAL opt-in extras on top of those; it
 # defaults to none. `make docker-image-full` passes `--extra uipath`, which

{coder_eval-0.11.7 → coder_eval-0.12.0}/docs/DOCKER_ISOLATION.md

@@ -15,7 +15,7 @@ Set `sandbox.driver: docker` on a task (or pass `--driver docker` on the CLI —
 a thin alias for `-D sandbox.driver=docker`) when you want:
 
 - **Isolation from the host filesystem/network** — agent-generated code can't reach files outside the sandbox.
-- **A pinned toolchain** — the image bakes in Python 3.13, Node 22 LTS, `@anthropic-ai/claude-code`, `uv`, and the matching `coder_eval` version, so results don't drift with host upgrades.
+- **A pinned toolchain** — the image bakes in Python 3.13, Node 22 LTS, `@anthropic-ai/claude-code`, the `pi` CLI (`@earendil-works/pi-coding-agent`), the codex/antigravity/litellm agent SDKs, `uv`, and the matching `coder_eval` version, so results don't drift with host upgrades.
 
 Aggregation (P/R/F1, suite thresholds, reports) always stays on the host. Each container is a sealed "run one task → emit one `task.json`" worker.