coder-eval 0.11.4__tar.gz → 0.11.5__tar.gz

coder_eval-0.11.5/.claude-plugin/marketplace.json

@@ -0,0 +1,35 @@
+{
+  "$schema": "https://json.schemastore.org/claude-code-marketplace.json",
+  "name": "coder-eval",
+  "owner": {
+    "name": "UiPath",
+    "email": "coder-eval@uipath.com",
+    "url": "https://github.com/UiPath/coder_eval"
+  },
+  "description": "Test whether your Claude Code skills actually trigger, and benchmark AI coding agents against your own tasks.",
+  "plugins": [
+    {
+      "name": "coder-eval",
+      "displayName": "Coder Eval",
+      "source": "./plugins/coder-eval",
+      "description": "Test whether your Claude Code skills actually trigger, and benchmark any coding agent — author, run, and analyze the eval suite that proves it, locally or as a CI gate.",
+      "author": { "name": "UiPath", "email": "coder-eval@uipath.com", "url": "https://github.com/UiPath/coder_eval" },
+      "homepage": "https://coder-eval.com",
+      "repository": "https://github.com/UiPath/coder_eval",
+      "license": "Apache-2.0",
+      "category": "testing",
+      "keywords": [
+        "claude-code-skills",
+        "skill-testing",
+        "skill-activation",
+        "evaluation",
+        "benchmark",
+        "agent-testing",
+        "github-actions",
+        "sandbox",
+        "llm-judge",
+        "ci"
+      ]
+    }
+  ]
+}

{coder_eval-0.11.4 → coder_eval-0.11.5}/CHANGELOG.md

@@ -2,6 +2,121 @@
 
 <!-- version list -->
 
+## v0.11.5 (2026-08-28)
+
+### Bug Fixes
+
+- **opencode**: Bound the post-EOF reap, reap the whole process group, test every failure path
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **opencode**: Close the remaining review findings on the harness
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **opencode**: Gate on captured tokens, canonicalize tool args, pin max_turns
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **opencode**: Inject plugin skills so skill suites measure the skills
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **opencode**: Keep the smoke task out of the CI smoke-pass bucket
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **opencode**: Map apply_patch to Write so GPT-family edits are seen by criteria
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **opencode**: Reap the CLI on every turn exit, test the sandbox env contract
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **opencode**: Spread super() in get_environment_info; guard with CE046
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **opencode**: Typecheck on Windows, satisfy both CodeQL findings
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **plugin**: Close the review's verified gaps — regex blind spot, parallel surfaces, runtime guard
+  ([#143](https://github.com/UiPath/coder_eval/pull/143),
+  [`c565ebb`](https://github.com/UiPath/coder_eval/commit/c565ebb8c8c7d895489d2ae6e244959ba12470af))
+
+- **plugin**: Correct the skill-reachability path — every generated activation suite reports recall
+  0.0 ([#143](https://github.com/UiPath/coder_eval/pull/143),
+  [`c565ebb`](https://github.com/UiPath/coder_eval/commit/c565ebb8c8c7d895489d2ae6e244959ba12470af))
+
+- **plugin**: Correct the skill-reachability path, and reuse PR #109's measured descriptions
+  ([#143](https://github.com/UiPath/coder_eval/pull/143),
+  [`c565ebb`](https://github.com/UiPath/coder_eval/commit/c565ebb8c8c7d895489d2ae6e244959ba12470af))
+
+- **routing**: Decouple simulator route from checker_context.api_route
+  ([#144](https://github.com/UiPath/coder_eval/pull/144),
+  [`88ff0f0`](https://github.com/UiPath/coder_eval/commit/88ff0f0ff910f4c3d9c6b81c5b2b88356240a3c4))
+
+- **routing**: Reinstate litellm+agent_judge rejection guard
+  ([#144](https://github.com/UiPath/coder_eval/pull/144),
+  [`88ff0f0`](https://github.com/UiPath/coder_eval/commit/88ff0f0ff910f4c3d9c6b81c5b2b88356240a3c4))
+
+- **routing**: Restore LiteLLM->Claude pin on simulator_route
+  ([#144](https://github.com/UiPath/coder_eval/pull/144),
+  [`88ff0f0`](https://github.com/UiPath/coder_eval/commit/88ff0f0ff910f4c3d9c6b81c5b2b88356240a3c4))
+
+- **test**: Add CE045, document the plugin-path divergence, unpin a test from ordering
+  ([#143](https://github.com/UiPath/coder_eval/pull/143),
+  [`c565ebb`](https://github.com/UiPath/coder_eval/commit/c565ebb8c8c7d895489d2ae6e244959ba12470af))
+
+- **utils**: Use explicit concatenation in the plugin-root warning
+  ([#143](https://github.com/UiPath/coder_eval/pull/143),
+  [`c565ebb`](https://github.com/UiPath/coder_eval/commit/c565ebb8c8c7d895489d2ae6e244959ba12470af))
+
+### Chores
+
+- **opencode**: Standardize on deepseek-v4-pro, drop the flash-0731 rate entry
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **plugin**: Answer "skills only?", fold tags into keywords, add CE044
+  ([#141](https://github.com/UiPath/coder_eval/pull/141),
+  [`2ae9b7b`](https://github.com/UiPath/coder_eval/commit/2ae9b7badb10831a11d81e703aa5e7a981616ace))
+
+- **plugin**: Give both author objects the same contact address
+  ([#141](https://github.com/UiPath/coder_eval/pull/141),
+  [`2ae9b7b`](https://github.com/UiPath/coder_eval/commit/2ae9b7badb10831a11d81e703aa5e7a981616ace))
+
+- **plugin**: Give the marketplace owner a contact address
+  ([#141](https://github.com/UiPath/coder_eval/pull/141),
+  [`2ae9b7b`](https://github.com/UiPath/coder_eval/commit/2ae9b7badb10831a11d81e703aa5e7a981616ace))
+
+- **plugin**: Lead both manifests with skill evaluation, add discovery metadata
+  ([#141](https://github.com/UiPath/coder_eval/pull/141),
+  [`2ae9b7b`](https://github.com/UiPath/coder_eval/commit/2ae9b7badb10831a11d81e703aa5e7a981616ace))
+
+- **plugin**: Pin both manifests to their published JSON schemas
+  ([#141](https://github.com/UiPath/coder_eval/pull/141),
+  [`2ae9b7b`](https://github.com/UiPath/coder_eval/commit/2ae9b7badb10831a11d81e703aa5e7a981616ace))
+
+### Documentation
+
+- **opencode**: Note _TERM_GRACE_SECONDS's second role as the post-EOF exit grace
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+### Features
+
+- **agents**: Add OpenCode harness with opt-in [opencode] extra
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+- **opencode**: Add require_token_telemetry, an escape hatch for the zero-token guard
+  ([#115](https://github.com/UiPath/coder_eval/pull/115),
+  [`5532b87`](https://github.com/UiPath/coder_eval/commit/5532b87ac218f6e413479ad7e2eb8d11d89686b9))
+
+
 ## v0.11.4 (2026-08-27)
 
 ### Chores

{coder_eval-0.11.4 → coder_eval-0.11.5}/CLAUDE.md

@@ -144,7 +144,7 @@ action.yml                         # Published composite GitHub Action (coder-ev
 - **Sub-agent token accounting**: There is NO separate per-sub-agent field. Every sub-agent generation is captured as a `parent_tool_use_id`-tagged `AssistantMessage` in the turn transcript, so per-sub-agent usage is derived by grouping those messages on that id (the evalboard's `aggregateSubAgentUsage` does exactly this). Claude bubbles its sub-agent's intermediate generations into the parent stream natively, and the **terminal** generation (delivered as the Agent tool result, never streamed) is synthesized into one via `_synthesize_subagent_terminal_message` from `tool_use_result.usage`. Codex reconstructs all child generations from the child rollout (`_recover_subagent_tool_calls`). The turn total already includes sub-agent cost — Claude via the SDK's cumulative `model_usage`; Codex via `_fold_subagent_tokens`, which folds the child messages (their real per-generation tokens) into the parent total. `CommandTelemetry.result_summary` is stored **untruncated** (no 200-char cap) so sub-agent returns are preserved whole. Set `CODER_EVAL_RAW_SDK_LOG=1` to dump every raw SDK event to the task log for inspection.
 - **Reconciliation message (stream self-reconciles to the turn total)**: The per-message stream consistently under-reports the authoritative turn total — a fixed prompt slice (~512 input tokens on Claude) is billed on no SDK-emitted message, and sub-agent input/cache only partially bubbles up. So `EventCollector.build_turn_record` appends one synthetic `ReconciliationMessage` (`role="reconciliation"`, in the `TranscriptMessage` union) per turn, carrying the per-bucket residual = `token_usage` − Σ(assistant message buckets). The invariant: **summing the four token buckets across `TurnRecord.messages` (assistant + reconciliation) equals `token_usage` exactly**, for both Claude and Codex (Codex's stream is already complete after `_recover_subagent_tool_calls`, so its residual is usually 0 and no entry is emitted). This is what lets the evalboard SUM the message stream as the source of truth instead of reading a separate aggregate ("agent tokens"): `selectTokenTotals` returns the stream sum whenever a reconciliation entry is present, and the timeline renders it as its own row. It is agent-agnostic (booked at the single `EventCollector` seam), carries no cost (cost stays on `token_usage`), and is excluded from generation/turn counts and the cost simulator. The LiteLLM open-weight actual-cost join (`litellm_cost.apply_actual_cost`) deliberately writes cost at the TURN level only (`token_usage.total_cost_usd` = the real OpenRouter bill) plus the per-call `TurnRecord.provider_call_costs` audit record; it does NOT touch the message token buckets, so `EventCollector` stays the single writer and this invariant holds on every backend. The Python `token_usage`/`total_token_usage` aggregate is unchanged and still authoritative for budget/judges/reports.
 - **Reference solutions are directory-only, and shielded (partially) from the agent**: `task.reference` is a single required `directory:` (relative to the task YAML) — the inline `code:` / single-file `file:` forms are gone, because a directory is the only shape that can be permission-gated as a unit; a `model_validator(mode="before")` gives the removed forms a migration error. The orchestrator stages a **per-run private copy** (`orchestration/evaluation.py::stage_reference_dir`, symlinks stripped) into a tempdir, removed in `_cleanup` via `path_utils.rmtree_restrictive` (keyed on `_reference_staging_root`, recorded BEFORE the copy so a failed copy still cleans up; `rmtree(ignore_errors=True)` silently declines on a tree left at 000) and deliberately never preserved into `run_dir/artifacts`. That copy is held at mode `000` for the whole of every `agent.communicate` call via **`Sandbox.set_permissions`**, the driver-aware wrapper over `fs_permissions.py::set_permissions`. Windows **stack**: exiting restores the *enclosing* window's mode, only the outermost exit restores the pre-window mode — that is what makes a mid-turn re-grant (`mode=READ_ONLY_MODE`) expressible, and it covers two windows at the same mode so no refcount is needed. The window is enforced **only inside a docker container** (`Sandbox.enforces_permission_windows`) and is a no-op on the host, where the agent shares our uid. **That gate keys on the `CODER_EVAL_IN_CONTAINER` env var, NOT `sandbox.driver`** — `run_task_internal_command` rewrites `driver: docker` → `tempdir` before building the in-container Orchestrator, so a driver-based gate would silently disable the anti-cheat on exactly the path that needs it (regression-guarded by `TestSandboxDriverGate`); `resolve_reference_dir` gates its `/work/references` branch on the same var for the same reason. The task directory is **not** shielded (`:ro` mount → EROFS, and the same YAML is readable at `/work/input`). Criteria address reference files with the `$REFERENCE_DIR` token (same resolver as `$TASK_DIR`) and the `REFERENCE_DIR` env var for `run_command`; `reference_comparison` names one file via `reference_file`. Docker mounts a throwaway **read-write** copy at `/work/references` (a `:ro` mount cannot be chmod'd — EROFS), masks the in-task-dir original with an empty tmpfs, and drops `DAC_OVERRIDE`/`DAC_READ_SEARCH`. `FOWNER`/`CHOWN` are deliberately **NOT** dropped: the in-container orchestrator that applies the window is the same root process with the same caps, so dropping `FOWNER` breaks *the harness's own* chmod wherever the bind mount preserves a non-root owner (native Linux — verified: `chmod: Operation not permitted`), i.e. exactly where the drop would otherwise bite. A window that cannot be applied is now a hard error, not a warning: `Sandbox.set_permissions` passes `strict=True` whenever it enforces, so an unprotected run fails instead of producing a normal-looking score. **KNOWN GAP — this is defense-in-depth, not a boundary**: (a) `chmod(2)` is gated on owner-or-`CAP_FOWNER` and the container runs as root owning the copy, so a deliberate `chmod 755 /work/references` restores access; (b) the window spans `agent.communicate` only, and nothing reaps agent child processes at turn end, so a backgrounded read loop succeeds once the window closes. The **write** half of (b) is closed — `path_utils.digest_tree` hashes the tree at staging and `Orchestrator._verify_reference_integrity` re-checks before grading, raising `ReferenceTamperedError` (→ `FinalStatus.ERROR`) on a mismatch so an agent cannot overwrite the reference to drive `reference_comparison` to 1.0. Passive reads are blocked; an adversarial agent is not. Full containment requires running the agent as a non-root uid AND holding the window for the agent's whole lifetime — follow-up. `tasks/anti_cheat_reference` probes the passive-read half.
-- **Harness run-limit parity**: a shared `BaseAgentConfig` field must mean the same thing on every backend, so a divergence is either fixed or documented — never silent. **`run_limits.max_turns` on Codex/Antigravity counts VISIBLE turns** (resolved tool calls, read live off the shared `EventCollector.visible_turn_count`, the same list `TurnRecord.commands` holds) because one `communicate()` is a single SDK turn on both, so a native counter would clamp at 1; claude-code keeps its native SDK cap, whose unit (an agent-loop turn) absorbs arbitrarily many parallel calls — the same number is NOT the same budget across harnesses. The cap is enforced on the same loop boundary as the cooperative early stop and finalizes cleanly as `max_turns_exhausted` (no crash, no retry); on Antigravity that boundary lives in `_drain()`, so the background-work poll loop honors it too. Known unfixed divergences: `permission_mode` on Codex and Antigravity (both run unconfined — the sandbox driver is the isolation boundary), `disallowed_tools` on Codex (forwarded, not SDK-enforced), `allowed_tools`/`disallowed_tools` on Antigravity (not read at all), and `turn_timeout` on Antigravity (bounded by an earlier internal poll deadline at 80% of it). Full table + rationale: docs/agents/HARNESS_PARITY.md.
+- **Harness run-limit parity**: a shared `BaseAgentConfig` field must mean the same thing on every backend, so a divergence is either fixed or documented — never silent. **`run_limits.max_turns` on Codex/Antigravity counts VISIBLE turns** (resolved tool calls, read live off the shared `EventCollector.visible_turn_count`, the same list `TurnRecord.commands` holds) because one `communicate()` is a single SDK turn on both, so a native counter would clamp at 1; claude-code keeps its native SDK cap, whose unit (an agent-loop turn) absorbs arbitrarily many parallel calls — the same number is NOT the same budget across harnesses. OpenCode likewise keeps a native unit — the CLI streams a real multi-step loop per `communicate()` (`step_start`/`step_finish`), so `max_turns: N` allows N complete steps and cuts cleanly when step N+1 begins. The cap is enforced on the same loop boundary as the cooperative early stop and finalizes cleanly as `max_turns_exhausted` (no crash, no retry); on Antigravity that boundary lives in `_drain()`, so the background-work poll loop honors it too. Known unfixed divergences: `permission_mode` on Codex and Antigravity (both run unconfined — the sandbox driver is the isolation boundary), `disallowed_tools` on Codex (forwarded, not SDK-enforced), `allowed_tools`/`disallowed_tools` on Antigravity (not read at all), `turn_timeout` on Antigravity (bounded by an earlier internal poll deadline at 80% of it), `allowed_tools`/`disallowed_tools`/`system_prompt` on OpenCode (no CLI knob; warned at `start()`, not enforced), and **`agent.plugins[].path` depth** — claude-code REQUIRES a plugin root holding `skills/` and silently loads NOTHING from a bare skills directory, while Codex and Antigravity scan both depths and accept either. That is the costly direction: the wrong depth produces no error, every positive row of an activation suite scores 0, and the suite reports recall 0.0, which reads exactly like a skill that never triggers. Held to the plugin-root shape (for `SKILL_SOURCE_PATH` only) by lint rule CE045. `plugins` on OpenCode is **honored for skills**: each local plugin root is mapped to the `skills` dir its `.claude-plugin/plugin.json` declares (default `<root>/skills`, never the root itself — `skills.paths` is scanned recursively and a plugin root can hold a self-referential symlink) and injected via `OPENCODE_CONFIG_CONTENT`, which `--pure` does not suppress; a plugin's agents/hooks/commands/MCP servers are still dropped. Full table + rationale: docs/agents/HARNESS_PARITY.md.
 - **sandbox isolation**: Tasks that don't need MCP servers should set `setting_sources: []` in their `agent:` block to isolate the sandbox from the host project's CLAUDE.md and settings. Without this, the host project's CLAUDE.md (often 20 KB+) is injected into every API call, inflating cache-creation tokens and cost significantly.
 - **Run-time caps (non-criterion enforcement)**: `TaskDefinition.run_limits` (`RunLimits` model) is the single namespace for all *task-level* run-time caps — `max_turns` / `task_timeout` / `turn_timeout` (structural) and `max_input_tokens` / `max_output_tokens` / `max_total_tokens` / `max_usd` (cumulative budget). Token/USD breaches abort with `FinalStatus.TOKEN_BUDGET_EXCEEDED` or `COST_BUDGET_EXCEEDED` (both `category == "failed"`). Structural caps are set from the CLI via `-D run_limits.max_turns=…` / `-D run_limits.task_timeout=…` / `-D run_limits.turn_timeout=…` (field-merged into `run_limits`); budget caps via `-D run_limits.max_usd=…` etc. or YAML. Layered config uses field-merge — a variant block overrides individual keys without replacing the task's block. The one *per-criterion* cap, `stop_early.decide_within`, deliberately lives on `LiveSuccessCriterion` instead (see below) — the watcher must attribute a decision-step timeout to a specific criterion, which `RunLimits` (task-scoped, criterion-agnostic) cannot express.
 - **Early stop on criterion (opt-in, per-criterion arming)**: a `stop_early:` block (`StopEarlyPolicy`) on a criterion ends a single-shot run early once the run's **armed** criteria decide the outcome, so a raised `max_turns` isn't wasted on the smoke flavor. The block's PRESENCE is the arming and alone activates the watcher — there is **no run-level master switch**: `run_limits.stop_early: false` is the run-level KILL SWITCH that force-disarms every block (the one-line experiment-variant/`-D` override for an authoritative full run), and `run_limits.stop_early: true` (the removed master arm) is a hard `EarlyStopConfigError` at resolution. The block exists on `LiveSuccessCriterion` only (currently `skill_triggered`, `command_executed` — so arming an unobservable criterion is unrepresentable, a pydantic extra-forbid error). Arming carries one implicit trigger (a native live-fail may fail-stop the run); its keys refine it: `on_pass: stop` (pass-stop the moment the criterion live-passes; default `continue` just latches) and `decide_within: N` (still undecided after N tool-call steps latches an **effective fail**, fed through the same fail-stop rule, reported as `decision_budget_exceeded` — an ordinary weighted fail, NOT a gate-bypassing force-fail; cumulative across retry attempts of the same turn). A trigger whose polarity the instance can't decide (per the abstract, checker-independent `live_decidable_polarities()`, a pure function of the criterion's own fields, paired with the checker's `live_verdict` override by lint rule CE025, a registry-based whole-tree check) is **inert by design** — one dataset-fanned YAML line serves both positive rows (pass/timeout live) and distractor rows (fail live). Verdicts **latch**: once a criterion decides, its `live_verdict` is never polled again. Stop rule is weighted, not strict-boolean: `run_limits.stop_early_gate_threshold` (default `1.0`, reproducing strict-AND behavior exactly) is the minimum weighted score (`Σ weight·score / Σ weight` over the armed subset) required to pass; a fail-stop fires once the armed set's **ceiling** (best case for everything still undecided) can no longer reach the threshold — so a low-weight fail or timeout that can't doom the gate is absorbed and the run continues — and is **deferred while any pass-capable armed criterion is undecided** (a distractor misfire never truncates a positive row's recall signal); a pass-stop fires once the `on_pass: stop` subset's **floor** (worst case) already meets the threshold, and is symmetrically **deferred while any pass-capable armed criterion outside the `on_pass: stop` subset is undecided** (so an early pass never freezes a sibling `on_pass: continue` criterion's signal out of the trajectory). A fail-stop is therefore verdict-preserving; a pass-stop can miss a *later* distractor misfire, so authoritative P/R/F1 comes from a kill-switched (`stop_early: false`) run. Driven by `orchestration/early_stop.py::EarlyStopWatcher` (built when `early_stop_active(task)`: ≥1 armed criterion, kill switch not thrown) through the agent's cooperative `should_stop` seam (tool-call granularity, no SIGKILL); live verdicts only *trigger* the stop — the standard `check_all_async` on the frozen trajectory is authoritative. Gating is **FIRED-ONLY**: a run the watcher actually cut gates on the **armed subset** via the weighted `EvaluationResult.armed_criteria_passed`; a run that completes naturally — armed or not — gates strict-AND via `all_criteria_passed`, so adding a block never changes the verdict of a run it didn't cut. Note the gate keys on the watcher having FIRED (`result.early_stop is not None`), not on confirmed truncation — an agent that ignores `should_stop`, or a stop firing on the final message, still gates armed-only. Every resolution-time guardrail violation is a hard error at resolution (plan *and* run); the one load-time case — a `stop_early:` block on a non-live criterion — is a pydantic schema error at task load, which the run surface reports as a skipped task like any other malformed task. A runtime verdict bug **fails open** to a full run. Surfaces: `EarlyStopInfo` (incl. `gate_threshold` at stop time), report notes/badges, `stopped_early` run.json rows, `EarlyStopped`/`EarlyStopReason` telemetry dims. Worked rationale: docs/TASK_DEFINITION_GUIDE.md § `stop_early`. No blocks anywhere ⇒ behavior byte-for-byte unchanged.

{coder_eval-0.11.4 → coder_eval-0.11.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: coder-eval
-Version: 0.11.4
+Version: 0.11.5
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval
@@ -60,6 +60,7 @@ Requires-Dist: pytest>=9.0.2; extra == 'dev'
 Requires-Dist: ruff>=0.15.7; extra == 'dev'
 Provides-Extra: litellm
 Requires-Dist: litellm<2.0.0,>=1.95.0; extra == 'litellm'
+Provides-Extra: opencode
 Provides-Extra: uipath
 Requires-Dist: uipath>=2.10.31; extra == 'uipath'
 Description-Content-Type: text/markdown
@@ -77,14 +78,14 @@ Description-Content-Type: text/markdown
 **Coder Eval** (`pip install coder-eval` / `uv tool install coder-eval`) is an open-source framework for
 **evaluating and benchmarking AI coding agents and their skills** — built for CLI
 and skill builders — with sandboxing, reproducibility, and data-driven analysis.
-It runs a real agent (**Claude Code**, **Codex**, or **Google Antigravity /
-Gemini**) in a sandbox against declarative YAML tasks, then scores the files and
+It runs a real agent (**Claude Code**, **Codex**, **Google Antigravity /
+Gemini**, or **OpenCode**) in a sandbox against declarative YAML tasks, then scores the files and
 commands it actually produced. Not an "agentic coding" benchmark: it measures how
 effective your CLI and skills are when used by coding agents.
 
 Reach for it when you want to **test whether a Claude Code skill triggers**,
-**A/B-test Claude Code vs. Codex vs. Gemini** (or model vs. model, prompt vs.
-prompt), or **gate CI on coding-agent quality**. Unlike fixed datasets (SWE-bench,
+**A/B-test Claude Code vs. Codex vs. Gemini vs. OpenCode** (or model vs. model,
+prompt vs. prompt), or **gate CI on coding-agent quality**. Unlike fixed datasets (SWE-bench,
 SkillsBench) that rank models on a shared leaderboard, Coder Eval evaluates the
 tasks, skills, and workflows *you* ship — with weighted 0.0–1.0 criteria, a
 `skill_triggered` activation check, an A/B experiment layer, and per-tool cost
@@ -99,14 +100,14 @@ telemetry. See [How it compares](https://coder-eval.com/docs/comparison).
 - **Sandboxed execution** in isolated environments with resource limits
 - **Weighted, continuous scoring** (0.0–1.0) with fractional credit and thresholds
 - **Many criterion types** — from file checks to code similarity and LLM-graded rubrics
-- **Agent abstraction** — Claude Code, Codex, and Antigravity (Gemini) today, extensible via a plugin SPI
+- **Agent abstraction** — Claude Code, Codex, Antigravity (Gemini), and OpenCode today, extensible via a plugin SPI
 - **Experiment layer** — A/B agent configs (models, tools, prompts) side-by-side
 - **Full telemetry** — every tool call, token counts, and cost, with real-time streaming
 
 ## What you can do with it
 
 - **Benchmark coding agents** — score an agent across a suite of tasks with weighted scoring and pass/fail thresholds
-- **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini, model vs. model, tool-on vs. tool-off, prompt vs. prompt
+- **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini vs. OpenCode, model vs. model, tool-on vs. tool-off, prompt vs. prompt
 - **Evaluate skills** — verify an agent actually engages a target skill (`skill_triggered`) and score skill-driven suites (SkillsBench-style)
 - **Keep skills up to date in CI** — re-validate your skills on every change or on a schedule; catch silent regressions when models, prompts, or the skills themselves drift
 - **Gate CI on agent quality** — run the suite in GitHub Actions and fail the build on regressions
@@ -275,6 +276,7 @@ alone.
 | [Claude Code](docs/agents/CLAUDE_CODE.md) | Configuring and running the default Claude Code agent |
 | [Codex](docs/agents/CODEX.md) | Running the OpenAI Codex agent |
 | [Antigravity (Gemini)](docs/agents/ANTIGRAVITY.md) | Running the Google Antigravity / Gemini agent |
+| [OpenCode](docs/agents/OPENCODE.md) | Running the OpenCode agent on open-weight models |
 | [Run-Limit Parity](docs/agents/HARNESS_PARITY.md) | What each run_limits field means on every harness |
 | [A/B Experiments](docs/AB_EXPERIMENTS.md) | Compare models / tools / prompts across the same tasks |
 | [Bring Your Own Dataset](docs/DATASETS.md) | Fan a single task out over a dataset |

{coder_eval-0.11.4 → coder_eval-0.11.5}/README.md

@@ -11,14 +11,14 @@
 **Coder Eval** (`pip install coder-eval` / `uv tool install coder-eval`) is an open-source framework for
 **evaluating and benchmarking AI coding agents and their skills** — built for CLI
 and skill builders — with sandboxing, reproducibility, and data-driven analysis.
-It runs a real agent (**Claude Code**, **Codex**, or **Google Antigravity /
-Gemini**) in a sandbox against declarative YAML tasks, then scores the files and
+It runs a real agent (**Claude Code**, **Codex**, **Google Antigravity /
+Gemini**, or **OpenCode**) in a sandbox against declarative YAML tasks, then scores the files and
 commands it actually produced. Not an "agentic coding" benchmark: it measures how
 effective your CLI and skills are when used by coding agents.
 
 Reach for it when you want to **test whether a Claude Code skill triggers**,
-**A/B-test Claude Code vs. Codex vs. Gemini** (or model vs. model, prompt vs.
-prompt), or **gate CI on coding-agent quality**. Unlike fixed datasets (SWE-bench,
+**A/B-test Claude Code vs. Codex vs. Gemini vs. OpenCode** (or model vs. model,
+prompt vs. prompt), or **gate CI on coding-agent quality**. Unlike fixed datasets (SWE-bench,
 SkillsBench) that rank models on a shared leaderboard, Coder Eval evaluates the
 tasks, skills, and workflows *you* ship — with weighted 0.0–1.0 criteria, a
 `skill_triggered` activation check, an A/B experiment layer, and per-tool cost
@@ -33,14 +33,14 @@ telemetry. See [How it compares](https://coder-eval.com/docs/comparison).
 - **Sandboxed execution** in isolated environments with resource limits
 - **Weighted, continuous scoring** (0.0–1.0) with fractional credit and thresholds
 - **Many criterion types** — from file checks to code similarity and LLM-graded rubrics
-- **Agent abstraction** — Claude Code, Codex, and Antigravity (Gemini) today, extensible via a plugin SPI
+- **Agent abstraction** — Claude Code, Codex, Antigravity (Gemini), and OpenCode today, extensible via a plugin SPI
 - **Experiment layer** — A/B agent configs (models, tools, prompts) side-by-side
 - **Full telemetry** — every tool call, token counts, and cost, with real-time streaming
 
 ## What you can do with it
 
 - **Benchmark coding agents** — score an agent across a suite of tasks with weighted scoring and pass/fail thresholds
-- **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini, model vs. model, tool-on vs. tool-off, prompt vs. prompt
+- **Compare models & configs** — A/B-test Claude vs. Codex vs. Gemini vs. OpenCode, model vs. model, tool-on vs. tool-off, prompt vs. prompt
 - **Evaluate skills** — verify an agent actually engages a target skill (`skill_triggered`) and score skill-driven suites (SkillsBench-style)
 - **Keep skills up to date in CI** — re-validate your skills on every change or on a schedule; catch silent regressions when models, prompts, or the skills themselves drift
 - **Gate CI on agent quality** — run the suite in GitHub Actions and fail the build on regressions
@@ -209,6 +209,7 @@ alone.
 | [Claude Code](docs/agents/CLAUDE_CODE.md) | Configuring and running the default Claude Code agent |
 | [Codex](docs/agents/CODEX.md) | Running the OpenAI Codex agent |
 | [Antigravity (Gemini)](docs/agents/ANTIGRAVITY.md) | Running the Google Antigravity / Gemini agent |
+| [OpenCode](docs/agents/OPENCODE.md) | Running the OpenCode agent on open-weight models |
 | [Run-Limit Parity](docs/agents/HARNESS_PARITY.md) | What each run_limits field means on every harness |
 | [A/B Experiments](docs/AB_EXPERIMENTS.md) | Compare models / tools / prompts across the same tasks |
 | [Bring Your Own Dataset](docs/DATASETS.md) | Fan a single task out over a dataset |

{coder_eval-0.11.4 → coder_eval-0.11.5}/action.yml

@@ -41,7 +41,7 @@ inputs:
   version:
     description: coder-eval version to install from PyPI, or "local" to install from the action checkout
     required: false
-    default: "0.11.4"   # <-- kept in sync with releases by release.yml
+    default: "0.11.5"   # <-- kept in sync with releases by release.yml
   run-dir:
     description: Run directory (--run-dir)
     required: false

{coder_eval-0.11.4 → coder_eval-0.11.5}/docker/Dockerfile

@@ -49,15 +49,21 @@ COPY src/ ./src/
 COPY experiments/default.yaml ./experiments/default.yaml
 
 # Codex, Antigravity, and litellm are always baked into the image -- codex/
-# antigravity are peers to the claude-code agent installed above, so all
-# built-in agents ship in every build; litellm backs the `checker_context.
-# api_route.route: litellm` judge dispatch (see pyproject.toml's `litellm`
-# extra), which every DockerRunner-isolated task needs available IN-container
-# since the checker runs there too, not just on the host. `openai-codex`
-# (+ its pinned cli-bin), `google-antigravity` (which bundles its
-# `localharness` binary as a manylinux wheel), and `litellm` all come from
-# public PyPI, so this needs no private-index credentials. The RUN below
-# always passes `--extra codex --extra antigravity --extra litellm`.
+# antigravity are peers to the claude-code agent installed above; litellm backs
+# the `checker_context.api_route.route: litellm` judge dispatch (see
+# pyproject.toml's `litellm` extra), which every DockerRunner-isolated task
+# needs available IN-container since the checker runs there too, not just on the
+# host. `openai-codex` (+ its pinned cli-bin), `google-antigravity` (which
+# bundles its `localharness` binary as a manylinux wheel), and `litellm` all
+# come from public PyPI, so this needs no private-index credentials. The RUN
+# below always passes `--extra codex --extra antigravity --extra litellm`.
+#
+# NOT every built-in agent ships here: `opencode` is registered unconditionally
+# but its CLI is a Node package (`npm install -g opencode-ai`), absent from this
+# image, and no OpenCode credentials are in SandboxConfig.env_passthrough -- so
+# `--driver docker` does not support it. Adding it means a pinned version that
+# travels with the release tag (as CLAUDE_CODE_VERSION does) plus an
+# env_passthrough block; see docs/agents/OPENCODE.md "Running in Docker".
 #
 # CODER_EVAL_UV_EXTRAS carries ADDITIONAL opt-in extras on top of those; it
 # defaults to none. `make docker-image-full` passes `--extra uipath`, which

{coder_eval-0.11.4 → coder_eval-0.11.5}/docs/AB_EXPERIMENTS.md

@@ -130,7 +130,7 @@ From `ExperimentVariant` (`coder_eval/models/experiment.py`):
 | `initial_prompt_file` | str                | Prompt replacement loaded from a file                                                  |
 | `run_limits`          | block              | Per-key cap overrides (`max_turns`, `task_timeout`, token/USD budgets)                 |
 | `driver`              | `tempdir`/`docker` | Sandbox driver — enables tempdir-vs-docker arms                                        |
-| `checker_context`     | dict               | Backend/model override for the evaluation side (judge, simulator) — see [Checker Context](TASK_DEFINITION_GUIDE.md#checker-context); **not** currently `-D`-reachable |
+| `checker_context`     | dict               | Backend/model override for the judge side (llm_judge, agent_judge) — has no bearing on the simulator; see [Checker Context](TASK_DEFINITION_GUIDE.md#checker-context); **not** currently `-D`-reachable |
 
 The `agent` dict is the lever for most A/B tests. Anything on `AgentConfig` is
 fair game: `model`, `permission_mode`, `allowed_tools`, `disallowed_tools`,
@@ -181,7 +181,7 @@ variants:
     agent:
       plugins:
         - type: "local"
-          path: "../skills" # skill available
+          path: ".." # PLUGIN ROOT holding skills/ — see note below
 ```
 
 Notes:
@@ -193,8 +193,14 @@ Notes:
   it. Pair the experiment with a [`skill_triggered`](TASK_DEFINITION_GUIDE.md#skill_triggered)
   criterion to measure _whether it fired_ alongside your real success criteria
   that measure _whether outcomes improved_.
+- **`path` must be a plugin ROOT — a directory holding `skills/`** — so the skill
+  resolves at `<path>/skills/<name>/SKILL.md`. Point one level deeper, at the
+  directory of skill directories, and claude-code loads **nothing**: the `with-skill`
+  arm then silently matches the baseline and the A/B compares two identical arms.
+  Codex and Antigravity accept either depth, so this fails on claude-code alone —
+  see [Harness parity](agents/HARNESS_PARITY.md#agentpluginspath-accepts-different-depths-per-harness).
 - Plugin paths are environment-dependent. The shipped example expects a
-  `$PLUGIN_PATH` env var pointing at your plugin directory. See
+  `$PLUGIN_PATH` env var pointing at your plugin **root**. See
   `experiments/plugin-comparison.yaml`.
 
 Run it:

{coder_eval-0.11.4 → coder_eval-0.11.5}/docs/DIALOG_MODE.md

@@ -60,9 +60,11 @@ The mechanics:
 - The simulator is a **tools-disabled Claude Code agent** with `allowed_tools: []`, an explicit
   deny-list, and no plugins or settings sources. It is pure text-in / text-out, and it **cannot see
   the sandbox** — no files, no terminal, no agent reasoning. Only what the agent writes in the chat.
-- The simulator runs on the run's resolved *evaluation* `ApiRoute` — the coding agent's own route
-  (`--backend direct` / `--backend bedrock`) unless `checker_context.api_route.route` overrides it
-  (see [Checker Context](TASK_DEFINITION_GUIDE.md#checker-context)). The **model** is separately
+- The simulator resolves its own `ApiRoute` independently of `checker_context.api_route` (that
+  override is judge-only — see [Checker Context](TASK_DEFINITION_GUIDE.md#checker-context)) — same
+  resolution as the coding agent's own route (`--backend direct` / `--backend bedrock`), including
+  the pin to a constant Claude backend when the agent itself runs on an open-weight LiteLLM route.
+  The **model** is separately
   pinned by `simulation.model` (see [Simulation](TASK_DEFINITION_GUIDE.md#simulation)), not inherited
   from the route — so an A/B varying the subject model doesn't silently vary the simulated user too.
 

{coder_eval-0.11.4 → coder_eval-0.11.5}/docs/EXTENDING.md

@@ -285,8 +285,8 @@ The base package ships **no** plugin rates; only the built-in table.
 ## See also
 
 - [Claude Code](agents/CLAUDE_CODE.md) · [Codex](agents/CODEX.md) ·
-  [Antigravity](agents/ANTIGRAVITY.md) — the built-in agents, each registered via
-  this same SPI
+  [Antigravity](agents/ANTIGRAVITY.md) · [OpenCode](agents/OPENCODE.md) — the
+  built-in agents, each registered via this same SPI
 - [Task Definition Guide](TASK_DEFINITION_GUIDE.md) — the criterion catalogue
 - [CLAUDE.md](https://github.com/UiPath/coder_eval/blob/main/CLAUDE.md) — architecture
   and extension points in depth

{coder_eval-0.11.4 → coder_eval-0.11.5}/docs/PLUGIN.md

@@ -106,10 +106,12 @@ It then:
 One prerequisite the suite cannot infer: the evaluated agent runs in a fresh
 sandbox holding none of your files, so it is offered no skills unless the task
 says where they live. The template reads that location from an environment
-variable — point it at the directory *containing* the skill's own directory:
+variable — point it at a **plugin root**: a directory holding a `skills/`
+subdirectory, so the skill sits at `<path>/skills/<skill-name>/SKILL.md`. For
+`.claude/skills/pdf-forms/SKILL.md` that root is `.claude`, not `.claude/skills`:
 
 ```bash
-export SKILL_SOURCE_PATH="$(pwd)/.claude/skills"
+export SKILL_SOURCE_PATH="$(pwd)/.claude"
 ```
 
 Leave it unset and the skill is simply absent, every positive row scores 0, and

{coder_eval-0.11.4 → coder_eval-0.11.5}/docs/TASK_DEFINITION_GUIDE.md

@@ -64,7 +64,7 @@ reference: { ... }                    # Optional reference solution (a directory
 pre_run: [ ... ]                      # Optional pre-run commands (before agent starts)
 post_run: [ ... ]                     # Optional post-run commands
 dataset: { ... }                      # Optional dataset fan-out (one task -> N row-tasks)
-checker_context: { ... }              # Optional: backend/model for the evaluation side (judge, simulator)
+checker_context: { ... }              # Optional: backend/model for the judge side (llm_judge/agent_judge)
 ```
 
 ### `dataset`
@@ -1308,11 +1308,11 @@ success_criteria:
 
 checker_context:
   api_route:
-    route: bedrock    # which backend the whole eval side (llm_judge/agent_judge/simulator) uses
+    route: bedrock    # which backend the eval side uses
     model: claude-haiku-4-5   # model override for that route
 ```
 
-- `route` selects which backend the WHOLE evaluation side calls (`llm_judge`, `agent_judge`, and the simulator all share one resolved eval route per run — this isn't per-criterion), **decoupled from the agent's own route** (a Claude agent can be graded by a differently-backed judge, or vice versa). This is a backend *name* (`direct` / `bedrock` / `litellm`), not a route object. For `direct`/`bedrock` credentials are never read from the task, always from the matching environment variables (`ANTHROPIC_API_KEY` for `direct`, `AWS_BEARER_TOKEN_BEDROCK`/`AWS_REGION` for `bedrock`). An unconfigured or unknown backend name raises at dispatch rather than silently falling back. **`route: litellm` dispatches `llm_judge` through the `litellm` library** (the `coder-eval[litellm]` extra, `litellm.acompletion`) rather than assuming one wire protocol — a gateway-routed judge model (e.g. an Azure AI `/openai/v1` deployment) rarely speaks Anthropic Messages, so this lets `model` carry its own provider hint (e.g. `azure/gpt-5.6-luna`) and get that provider's actual request/response shape handled by the library. Unlike the other two backends, `route: litellm` has NO implicit env-var fallback — see `params`/`env_params` below, which is how it's configured. `model` is required for `route: litellm` (there is no default open-weight/gateway model).
+- `route` selects which backend `llm_judge` and `agent_judge` call — they share one resolved eval route per run for `direct`/`bedrock`/`litellm` (this isn't per-criterion) — **decoupled from the agent's own route** (a Claude agent can be graded by a differently-backed judge, or vice versa). The simulator is NOT part of this sharing: it always resolves its own route the same way the agent under test does (subprocess-safe by construction), independent of `checker_context.api_route` entirely — see the `route: litellm` note below. This is a backend *name* (`direct` / `bedrock` / `litellm`), not a route object. For `direct`/`bedrock` credentials are never read from the task, always from the matching environment variables (`ANTHROPIC_API_KEY` for `direct`, `AWS_BEARER_TOKEN_BEDROCK`/`AWS_REGION` for `bedrock`). An unconfigured or unknown backend name raises at dispatch rather than silently falling back. **`route: litellm` dispatches `llm_judge` ONLY** through the `litellm` library (the `coder-eval[litellm]` extra, `litellm.acompletion`) rather than assuming one wire protocol — a gateway-routed judge model (e.g. an Azure AI `/openai/v1` deployment) rarely speaks Anthropic Messages, so this lets `model` carry its own provider hint (e.g. `azure/gpt-5.6-luna`) and get that provider's actual request/response shape handled by the library. `agent_judge` spawns a real Claude Code CLI subprocess that only speaks Anthropic Messages — combining `route: litellm` with an enabled `agent_judge` criterion is rejected at resolution time (see below); the simulator is unaffected since it never reads this override. Unlike the other two backends, `route: litellm` has NO implicit env-var fallback — see `params`/`env_params` below, which is how it's configured. `model` is required for `route: litellm` (there is no default open-weight/gateway model).
 - `model` overrides the model that resolved route uses for **`llm_judge` only** — when the criterion itself leaves `model:` unset (precedence: an explicit per-criterion `model:` always wins; below that, `checker_context.api_route.model`; below that, the built-in `DEFAULT_JUDGE_MODEL`). This floor is deliberate and never the agent's own model — an unpinned judge must grade identically regardless of which model the agent under test is using, so `resolve_evaluation_route` never lets the agent's env-configured model (e.g. `BEDROCK_MODEL`) leak into `route.model` on its own; `route.model` is set only when this override was actually given. This works because every `ApiRoute` (`DirectRoute`/`BedrockRoute`/`LiteLLMRoute`) carries its own `model` field; the orchestrator bakes the override into the resolved route's `model` before any criterion runs, so `llm_judge` just reads `context.route.model` — it never reads `checker_context` directly. **`agent_judge` and the simulator do not honor this override** — `agent_judge`'s sub-agent model comes from the criterion's own `agent:` block (defaulted to a fixed judge model), and the simulator's model is pinned by `SimulationConfig.model` (see [Simulation](#simulation) below) — both independent of `checker_context.api_route.model` by design, for the same "measuring instrument stays fixed" reason.
 - `params`/`env_params` (**`route: litellm` only**) are how the call is actually configured — there is no fallback to the agent's own `LITELLM_BASE_URL`/`LITELLM_AUTH_TOKEN` env vars, since a gateway-routed judge model rarely reuses the agent's own LiteLLM proxy/credential. They also cover any of the dozens of other provider-specific kwargs `litellm.acompletion` accepts (`aws_access_key_id`, `vertex_project`, `api_version`, ...), which have no dedicated field on `LiteLLMRoute`:
   ```yaml
@@ -1327,7 +1327,7 @@ checker_context:
         api_key: LITELLM_AUTH_TOKEN
   ```
   `params` is merged straight into the `litellm.acompletion(**kwargs)` call — litellm validates the param names itself, so there's no allowlist to keep in sync here. `env_params` maps a kwarg name to the *name* of an environment variable; the value is resolved right before the call, so no secret is ever written into task/experiment YAML — this is how an arbitrary provider's config, including secrets (IAM keys, an Azure AD token, a service-account path, ...), is representable without a dedicated field per provider. `env_params` is resolved after `params`, so it always wins for the same key. Rejected at task-load time if given without `route: litellm`. For an Azure deployment, pin `api_version` via `params` to whatever API version the agent side is actually configured for (e.g. Codex's `CODEX_API_VERSION`) — the judge has no way to inherit it, and a mismatched version can hit a different shape of the same endpoint.
-  **`route: litellm` is `llm_judge`-only** — `agent_judge` and the simulator run as real Claude Code CLI subprocesses that speak the Anthropic Messages protocol, so pointing them at an arbitrary litellm-fronted gateway (which may speak an entirely different wire protocol) isn't representable. The orchestrator rejects the combination at resolution time (a clear error, not a silent misroute) if the task has an enabled `agent_judge` criterion or `simulation.enabled: true` alongside `route: litellm` — use `route: bedrock`/`direct` for those instead.
+  **`route: litellm` has no bearing on the simulator** — the simulator is a real Claude Code CLI subprocess that speaks the Anthropic Messages protocol, so pointing it at an arbitrary litellm-fronted gateway (which may speak an entirely different wire protocol) isn't representable. Rather than rejecting a task that combines `route: litellm` with `simulation.enabled: true`, the simulator always resolves its own route independently of `checker_context.api_route` — the same way the agent under test resolves its route — so `llm_judge` grades through the gateway named here while the simulator behaves exactly as if this override weren't set. A task can freely combine `route: litellm` with `simulation.enabled: true`; there is nothing to disable or remove. **`agent_judge` is NOT decoupled the same way** — it still shares the `llm_judge` eval route, and `agent_judge`'s Claude Code CLI subprocess has no way to honor a `LiteLLMRoute` safely, so the orchestrator rejects `route: litellm` combined with an enabled `agent_judge` criterion at resolution time (a clear error, not a silent misroute) — use `route: bedrock`/`direct` instead if a task needs both.
 
 `checker_context` merges shallow-per-namespace across `default_experiment.defaults.checker_context` → `experiment.defaults.checker_context` → `task.checker_context` → `variant.checker_context` (same 4-layer precedence as `agent`/`simulation`). So a judge-model A/B, or a judge-backend A/B, is a variant-level config change, not an edit to every task YAML.
 
@@ -1613,7 +1613,7 @@ simulation:
 | `check_criteria` | `end_of_dialog` | `end_of_dialog`, `every_turn`, or `both`. |
 | `model` | `anthropic.claude-sonnet-4-6` | Model that plays the simulated user. Auto-translated to the run's backend (Bedrock inference profile / bare Anthropic alias), the same way [`llm_judge`](#llm_judge)'s `model` is. |
 
-The simulator runs as a tools-disabled Claude Code agent on the run's resolved *evaluation* `ApiRoute` (the coding agent's own route unless [`checker_context.api_route.route`](#checker-context) overrides it), so temperature and sampling are resolved at the route level (same `-b` flag as the coding agent by default) and are not configured on this block. The **model is not**: it is pinned by `model` above. Inheriting it from the route meant `BEDROCK_MODEL` decided who the simulated user was, so an A/B varying the subject model silently varied its interlocutor too. Hold `model` fixed across variants for the same reason you hold a judge model fixed — the simulator is part of the measuring instrument, not the thing being measured.
+The simulator runs as a tools-disabled Claude Code agent on its own resolved `ApiRoute` (the same resolution as the coding agent's own route — [`checker_context.api_route.route`](#checker-context) has no bearing on it), so temperature and sampling are resolved at the route level (same `-b` flag as the coding agent by default) and are not configured on this block. The **model is not**: it is pinned by `model` above. Inheriting it from the route meant `BEDROCK_MODEL` decided who the simulated user was, so an A/B varying the subject model silently varied its interlocutor too. Hold `model` fixed across variants for the same reason you hold a judge model fixed — the simulator is part of the measuring instrument, not the thing being measured.
 
 **Semantics:**
 

{coder_eval-0.11.4 → coder_eval-0.11.5}/docs/USER_GUIDE.md

@@ -39,7 +39,7 @@ coder-eval run tasks/hello_date.yaml --stream full  # live LLM output
 | `-D path=value` / `--set` | Override any resolved task-config field (`agent`/`run_limits`/`sandbox` roots), e.g. `-D run_limits.max_turns=30 -D agent.permission_mode=plan -D agent.sdk_options.effort=high`. Repeatable; schema-validated. This is the way to set permission mode, turn/timeout limits, token/USD budget caps, tools, plugins, and SDK options. |
 | `--model, -m` | Shorthand alias for `-D agent.model=…` (e.g., `claude-sonnet-5`) |
 | `--driver` | Shorthand alias for `-D sandbox.driver=…` (`tempdir` or `docker`) |
-| `--type, -T` | Override agent type for all tasks (`claude-code`, `codex`, `antigravity`, or a plugin kind). |
+| `--type, -T` | Override agent type for all tasks (`claude-code`, `codex`, `antigravity`, `opencode`, or a plugin kind). |
 | `--repeats` | Run each `(task, variant)` N times (≥1); overrides experiment/variant `repeats:`. See [Replicates](#replicates). |
 | `--resume` | Resume an interrupted run: skip tasks already finalized in `--run-dir` and run the rest, folding prior results into `run.json`. Requires `--run-dir`. A task with *any* final status (incl. FAILED/ERROR) counts as finalized, so resume does **not** retry failures — delete a task's `task.json` to force a re-run. A config mismatch is warned, not refused. |
 | `--sample N` | For dataset-backed tasks, run a fixed-seed random N-row sample (reproducible; cheap smoke test). See [Bring Your Own Dataset](DATASETS.md). |

{coder_eval-0.11.4 → coder_eval-0.11.5}/docs/agents/CLAUDE_CODE.md

@@ -98,7 +98,7 @@ agent:
 | `permission_mode` | default **`acceptEdits`** | `default` / `acceptEdits` / `plan` / `bypassPermissions` — semantics come from the Claude Code SDK. `plan` is read-only; `bypassPermissions` grants full autonomy. |
 | `allowed_tools` | `list[str] \| null` | Tool allowlist. Unset ⇒ all tools allowed. |
 | `disallowed_tools` | `list[str] \| null` | Tool denylist. (`ToolSearch` is always appended for Bedrock parity.) |
-| `plugins` | `list[{type: local, path}]` | Local plugin/skill directories; `$VAR` in `path` is expanded and resolved to an absolute path. |
+| `plugins` | `list[{type: local, path}]` | Local plugin roots; `$VAR` in `path` is expanded and resolved to an absolute path. **`path` must hold a `skills/` subdirectory** — a bare directory of skill directories loads nothing here, though Codex and Antigravity accept it. See [Harness parity](HARNESS_PARITY.md#agentpluginspath-accepts-different-depths-per-harness). |
 | `system_prompt` | `str \| null` | **Appended** to the default Claude Code system prompt (via the SDK's `claude_code` preset) — the default's behavioral guidance is kept unless `system_prompt_mode: replace` opts out. Mutually exclusive with `system_prompt_file`. An empty or whitespace-only value is treated as unset. |
 | `system_prompt_mode` | `"append"` (default) / `"replace"` | `replace` sends `system_prompt` as the **entire** system prompt (no preset) and requires a non-blank `system_prompt` / `system_prompt_file` (validated at load). Used by judge sub-agents and the user simulator, which must not carry the coding-agent persona; rarely needed in tasks — see [the migration note](#migrating-tasks-that-set-system_prompt). |
 | `system_prompt_file` | `str \| null` | Path (relative to the task YAML) loaded into `system_prompt` at resolution. Works with either `system_prompt_mode`. |
@@ -183,6 +183,16 @@ simulator force `[]` for the same reason.)
   that call and scores skill-activation suites.
 - **Plugins** are supplied as `plugins: [{type: local, path: …}]`; the path is
   env-expanded and resolved to an absolute path before being handed to the SDK.
+  It must name a **plugin root** — a directory holding `skills/`, so the skill
+  resolves at `<path>/skills/<name>/SKILL.md`. One level deeper loads nothing, with
+  no error: every positive row of an activation suite then scores 0 and the suite
+  reports recall 0.0, which reads exactly like a skill that never triggers. Codex
+  and Antigravity scan both depths, so this is claude-code-only — see
+  [Harness parity](HARNESS_PARITY.md#agentpluginspath-accepts-different-depths-per-harness).
+- A plugin root loads **everything** the plugin declares, not only `skills/`: an
+  `agents/`, `commands/` or `hooks/` directory beside it becomes visible to the
+  evaluated agent too. Point at a minimal root when the suite must measure one
+  skill in isolation.
 - **`PLUGIN_TOOLS_DIR`** pins the canonical `node_modules/@uipath` directory for
   UiPath CLI plugin discovery; when unset the sandbox derives it from the resolved
   `uip` binary. See [User Guide → Environment Variables](../USER_GUIDE.md#environment-variables).