coder-eval 0.12.6__tar.gz → 0.12.8__tar.gz

{coder_eval-0.12.6 → coder_eval-0.12.8}/.claude/harness-candidates.md

@@ -1022,3 +1022,13 @@ re-derive from scratch.
   have, and it is a recurring class ONLY within one file so far — worth promoting to a rule
   if a second subprocess-driven agent repeats it. Caught in: final cross-phase review,
   delegate agent port.
+
+## From 2026-09-26 Antigravity SDK 0.1.18 usage regression
+
+- [ ] Runtime guard: fail an Antigravity turn that ran MODEL steps but booked zero
+  tokens, as OpenCode's `require_token_telemetry` does. SDK 0.1.18 deprecated
+  `Step.usage_metadata` (usage moved to `conversation.total_usage`), so every
+  Antigravity turn from `3aa2db30` on reported no tokens and no cost, and `api_calls`
+  stayed at 1 so `max_turns` never tripped — all silently, with `make verify` green.
+  A static rule cannot see an SDK field go dead; a live-telemetry smoke (one real turn,
+  assert `token_usage` is non-empty) in the harness-bump checklist would have.

{coder_eval-0.12.6 → coder_eval-0.12.8}/.claude/notes/contracts.md

@@ -95,9 +95,14 @@ the raw text alone can stop counting.
 It is memoized because the early-stop watcher re-scans the whole accumulated trajectory on
 every tool-call event, normalizing the same command many times per run.
 
-The regex search window is capped to bound ReDoS on a large command string, and
-normalization runs over that same truncated window, so `shlex` never sees more than the cap
-and needs no separate guard.
+Each regex search is bounded to 2000 characters to limit ReDoS on a large command string,
+but the whole command is searched: the first 2000 characters, then the rest in windows that
+start and end on logical-line boundaries. Agents write long heredoc scripts that end in the
+command a task checks for (`cat > x <<EOF ... EOF` then `uip agent validate`), and a single
+leading window scored those as never run. Normalization runs per window, so `shlex` never
+sees more than the bound and needs no separate guard. Only a Bash command is windowed:
+another tool's params are JSON, and a later window of a Write/Edit body is file content,
+not a command the agent ran, so those keep the single leading window.
 
 ## Recording a CLI invocation
 

{coder_eval-0.12.6 → coder_eval-0.12.8}/.claude/notes/lint-rules.md

@@ -667,7 +667,7 @@ shipped. An id is a permanent documentation anchor: a suppression comment carryi
 an older branch, review or commit message must never start meaning something new. CE023 is
 retired the same way, after the rule was deleted with the package it guarded.
 
-Claim 068 next, and note 065 IS TAKEN without being in `ALL_RULES`: doc-surface and
+Claim 069 next, and note 065 IS TAKEN without being in `ALL_RULES`: doc-surface and
 whole-tree rules are `@pytest.mark.lint` classes in `tests/test_custom_lint.py` rather than
 `BaseRule`s, so `runner.py`'s uniqueness assert cannot see them. Enumerating them in a
 comment is how that note fell behind CE044, so grep instead:

{coder_eval-0.12.6 → coder_eval-0.12.8}/.claude/notes/orchestration.md

@@ -108,6 +108,23 @@ VERDICT, never the facts — the seeding cannot restore a fact the execute phase
 captured. The budget gate runs AFTER the criteria on the graded path purely for
 partial-credit visibility, and there is no partial credit under `execute`.
 
+### Post-failure evidence is declared, not proven
+
+A timed-out run used to score 0.00 on artifacts that every grader accepted, because the
+whole uipath-maestro-flow suite grades through `run_command` and the path ran only file
+checks. `read_only: true` admits a single named command.
+
+The gate could not stay a ClassVar: `run_command` is safe or unsafe per criterion, not
+per type, and no analysis decides which — an arbitrary shell command can start a cloud
+job (`uip maestro flow debug` does). So the flag is the AUTHOR's claim, the ClassVar
+stayed the type-level answer, and `evaluable_after_agent_failure` became the per-instance
+one every caller asks. Widening the flag to the base would let a judge declare itself
+deterministic, which is a different and false claim.
+
+What keeps it honest: results land in `post_failure_criteria_results`, which
+`calculate_weighted_score` never reads. A timed-out run stays ERROR at 0.0 no matter what
+the diagnostic pass finds — the flag buys evidence, never a verdict.
+
 ### Rates need verdict evidence, not bucket counts
 
 A published rate divides by rows that actually carry a verdict, not by a bucket count. The

{coder_eval-0.12.6 → coder_eval-0.12.8}/.claude/notes/permissions.md

@@ -29,10 +29,6 @@ read-write throwaway copy; when it was bind-mounted `:ro` the chmod returned ERO
 no tmpfs mask and has not been one since: see
 [isolation.md](isolation.md) § Why the framework mounts are writable copies.
 
-What the window does NOT hide is the task DEFINITION. `task.yaml` is also staged at
-`/work/input` for the in-container orchestrator and that mount is untouched, so hiding the
-criteria from the agent remains a separate, unsolved problem.
-
 Criteria address reference files with the `$REFERENCE_DIR` token (same resolver as
 `$TASK_DIR`) and the `REFERENCE_DIR` env var for `run_command`; `reference_comparison` names
 one file via `reference_file`.
@@ -64,10 +60,6 @@ criteria and judges run. The task directory is shielded by the same window: unde
 it is mounted as a throwaway COPY at a fixed container path, which is what makes it
 chmod-able without touching the user's checked-out `tasks/` tree.
 
-It shields grading MATERIAL that happens to live in the task directory (a `reference/`
-subdirectory, fixtures), not the task DEFINITION: `task.yaml` is separately staged at
-`/work/input`, which the agent can still read.
-
 ### Why a stack and not a refcount
 
 Windows nest with *different* modes, so what an exit has to restore is the mode of the

{coder_eval-0.12.6 → coder_eval-0.12.8}/.claude/notes/timing.md

@@ -401,8 +401,8 @@ and the tail are already computed.
 
 A reducer's only remaining timing decision is where its window opens, which is the one
 genuinely harness-shaped part: two interleave a tool into a single window outright
-(Antigravity, whose Step for the tool arrives and only a later `usage_metadata` Step cuts
-the message, and Codex, whose `_flush_message` window extends to the last item's
+(Antigravity, whose Step for the tool arrives and only a later rise in the conversation's
+billed usage cuts the message, and Codex, whose `_flush_message` window extends to the last item's
 `completed_at_ms`) while the other three tile the turn contiguously, so a call open at a
 boundary runs inside two windows. Central subtraction handles both without either reducer
 knowing which it is.

{coder_eval-0.12.6 → coder_eval-0.12.8}/.github/workflows/harbor-e2e.yml

@@ -29,6 +29,8 @@ env:
   # Anthropic-credit spend off this path; DirectRoute is exercised elsewhere.
   API_BACKEND: "bedrock"
   CLAUDE_CODE_USE_BEDROCK: "1"
+  # Same expression as pr-checks.yml (see the comment there).
+  SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }},claude-agent-sdk
 
 jobs:
   harbor-e2e:

{coder_eval-0.12.6 → coder_eval-0.12.8}/.github/workflows/pr-checks.yml

@@ -38,11 +38,12 @@ env:
   TELEMETRY_ENABLED: "false"
   # The `uipath-*` pool enforces a minimum package-age safe-chain check on installs.
   # Workflow-level so every installing job inherits it; per-job copies are how some
-  # jobs previously ended up with no exclusions at all. The literal is the operative
-  # value — no secret of that name exists at repo or org level, so the bare `secrets.`
-  # reference this replaced resolved to an empty list. (Image builds carry their own
-  # list in docker/Dockerfile; deliberately not the same set.)
-  SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }}
+  # jobs previously ended up with no exclusions at all. A repo secret of this name
+  # replaces the literal when set, so claude-agent-sdk sits outside the `||`: its wheel
+  # bundles the Claude Code CLI, and new-model support ships there faster than the
+  # gate window. (Image builds carry their own list in docker/Dockerfile; deliberately
+  # not the same set.)
+  SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }},claude-agent-sdk
 
 jobs:
   quality-gate:

{coder_eval-0.12.6 → coder_eval-0.12.8}/.github/workflows/publish-testpypi.yml

@@ -38,7 +38,7 @@ jobs:
     timeout-minutes: 10
     env:
       # `uv build` resolves build deps under the pool's safe-chain gate.
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }}
+      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }},claude-agent-sdk
     environment:
       name: testpypi
       url: https://test.pypi.org/project/coder-eval/

{coder_eval-0.12.6 → coder_eval-0.12.8}/.github/workflows/release.yml

@@ -75,7 +75,7 @@ jobs:
       # Load-bearing on the release path: the pool enforces a package-age safe-chain
       # check on uv installs. Same expression as pr-checks.yml (see the comment there),
       # so a package can't pass PR CI and then fail the release install.
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }}
+      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }},claude-agent-sdk
 
     steps:
       # Only a real release (main) needs the app token: semantic-release pushes the

{coder_eval-0.12.6 → coder_eval-0.12.8}/CHANGELOG.md

@@ -2,6 +2,148 @@
 
 <!-- version list -->
 
+## v0.12.8 (2026-09-29)
+
+### Bug Fixes
+
+- Also delete context.json in-container (source_yaml leaked criteria)
+  ([#179](https://github.com/UiPath/coder_eval/pull/179),
+  [`ddf63e9`](https://github.com/UiPath/coder_eval/commit/ddf63e9b356a6c40a83ca5771a3ffd3a1e8dfd68))
+
+- **anti-cheat**: Apply multi-model review findings M1/M2/M3 + doc drift
+  ([#179](https://github.com/UiPath/coder_eval/pull/179),
+  [`ddf63e9`](https://github.com/UiPath/coder_eval/commit/ddf63e9b356a6c40a83ca5771a3ffd3a1e8dfd68))
+
+- **anti-cheat**: Apply review Low findings L1/L2/L3
+  ([#179](https://github.com/UiPath/coder_eval/pull/179),
+  [`ddf63e9`](https://github.com/UiPath/coder_eval/commit/ddf63e9b356a6c40a83ca5771a3ffd3a1e8dfd68))
+
+- **antigravity**: Read token usage from the conversation, not the Step
+  ([#201](https://github.com/UiPath/coder_eval/pull/201),
+  [`101bb5c`](https://github.com/UiPath/coder_eval/commit/101bb5c85c5af97f8027388611dd70e7e7e0ecc8))
+
+- **codex**: Record provider error notifications on the turn
+  ([#203](https://github.com/UiPath/coder_eval/pull/203),
+  [`894f742`](https://github.com/UiPath/coder_eval/commit/894f74282c8f32918cc995fb4b2517f8984c4ed2))
+
+- **docker**: Close two driver:docker channels that leaked grading material to the agent
+  ([#179](https://github.com/UiPath/coder_eval/pull/179),
+  [`ddf63e9`](https://github.com/UiPath/coder_eval/commit/ddf63e9b356a6c40a83ca5771a3ffd3a1e8dfd68))
+
+- **simulation**: Send the simulated user's utterance, not its tagged transcript
+  ([#204](https://github.com/UiPath/coder_eval/pull/204),
+  [`ffb9a81`](https://github.com/UiPath/coder_eval/commit/ffb9a8198a1af73904fabb0c8b3251a978351ab0))
+
+### Documentation
+
+- **harness**: Log deferred Low findings from anti-cheat review
+  ([#179](https://github.com/UiPath/coder_eval/pull/179),
+  [`ddf63e9`](https://github.com/UiPath/coder_eval/commit/ddf63e9b356a6c40a83ca5771a3ffd3a1e8dfd68))
+
+### Features
+
+- **lint**: 3/3 — CE065 flags eval material inside a skill dir
+  ([#179](https://github.com/UiPath/coder_eval/pull/179),
+  [`ddf63e9`](https://github.com/UiPath/coder_eval/commit/ddf63e9b356a6c40a83ca5771a3ffd3a1e8dfd68))
+
+- **sandbox**: 1/3 — delete staged task.yaml in-container after load (anti-cheat)
+  ([#179](https://github.com/UiPath/coder_eval/pull/179),
+  [`ddf63e9`](https://github.com/UiPath/coder_eval/commit/ddf63e9b356a6c40a83ca5771a3ffd3a1e8dfd68))
+
+- **sandbox**: 2/3 — allowlist tmpfs-mask of auto-mounted plugin trees
+  ([#179](https://github.com/UiPath/coder_eval/pull/179),
+  [`ddf63e9`](https://github.com/UiPath/coder_eval/commit/ddf63e9b356a6c40a83ca5771a3ffd3a1e8dfd68))
+
+
+## v0.12.7 (2026-09-25)
+
+### Bug Fixes
+
+- **criteria**: Search past the 2000-char bound only in Bash commands
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+- **criteria**: Search the whole command in command_executed and bump harness SDKs for Opus 5.5
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+- **criteria**: Search the whole command in command_executed, not just the first 2000 chars
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+- **docs**: Drop the budget-breach trigger, cut read_only to its contract
+  ([#197](https://github.com/UiPath/coder_eval/pull/197),
+  [`28c846e`](https://github.com/UiPath/coder_eval/commit/28c846ea6858fef04d366d245e5f7b679c2b75a1))
+
+- **evalboard**: List date-named runs flagged adhoc in the Ad-hoc section
+  ([#200](https://github.com/UiPath/coder_eval/pull/200),
+  [`5bf4a72`](https://github.com/UiPath/coder_eval/commit/5bf4a72693f007e93da678b58e0ddb84fa1bf63c))
+
+- **harbor**: Declare SUPPORTS_ATIF as a ClassVar to match harbor 0.23
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+### Build System
+
+- **deps**: Bump harness SDKs and CLIs to latest
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+- **deps**: Exempt claude-agent-sdk from the 48h package-age gate and take the latest SDK
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+- **deps**: Hold the SDK bump to releases past CI's 48h package-age gate
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+### Continuous Integration
+
+- **harbor**: Exempt claude-agent-sdk from the package-age gate in Harbor E2E
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+### Documentation
+
+- **agents**: Update Pi and Antigravity references for the harness bump
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+- **criteria**: Name the budget-breach trigger on the post-failure path
+  ([#197](https://github.com/UiPath/coder_eval/pull/197),
+  [`28c846e`](https://github.com/UiPath/coder_eval/commit/28c846ea6858fef04d366d245e5f7b679c2b75a1))
+
+### Features
+
+- **criteria**: Let a run_command criterion opt in to post-failure grading
+  ([#197](https://github.com/UiPath/coder_eval/pull/197),
+  [`28c846e`](https://github.com/UiPath/coder_eval/commit/28c846ea6858fef04d366d245e5f7b679c2b75a1))
+
+- **evalboard**: Count agent time apart from eval overhead, and filter runs by variant
+  ([#198](https://github.com/UiPath/coder_eval/pull/198),
+  [`b04829c`](https://github.com/UiPath/coder_eval/commit/b04829c4dec86772822a28ef1b3fa7316c1f3c8d))
+
+- **evalboard**: Headline agent time per passed task, not full task wall clock
+  ([#198](https://github.com/UiPath/coder_eval/pull/198),
+  [`b04829c`](https://github.com/UiPath/coder_eval/commit/b04829c4dec86772822a28ef1b3fa7316c1f3c8d))
+
+- **evalboard**: Show agent time on the run page and filter it by variant
+  ([#198](https://github.com/UiPath/coder_eval/pull/198),
+  [`b04829c`](https://github.com/UiPath/coder_eval/commit/b04829c4dec86772822a28ef1b3fa7316c1f3c8d))
+
+- **evalboard**: Show summed agent time in the overview window tile
+  ([#198](https://github.com/UiPath/coder_eval/pull/198),
+  [`b04829c`](https://github.com/UiPath/coder_eval/commit/b04829c4dec86772822a28ef1b3fa7316c1f3c8d))
+
+- **evalboard**: Simplify the task timing strip into one bar and two groups
+  ([#198](https://github.com/UiPath/coder_eval/pull/198),
+  [`b04829c`](https://github.com/UiPath/coder_eval/commit/b04829c4dec86772822a28ef1b3fa7316c1f3c8d))
+
+- **evalboard**: Split the task timeline into agent time and eval overhead
+  ([#198](https://github.com/UiPath/coder_eval/pull/198),
+  [`b04829c`](https://github.com/UiPath/coder_eval/commit/b04829c4dec86772822a28ef1b3fa7316c1f3c8d))
+
+
 ## v0.12.6 (2026-09-23)
 
 ### Bug Fixes

{coder_eval-0.12.6 → coder_eval-0.12.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: coder-eval
-Version: 0.12.6
+Version: 0.12.8
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode, Pi, Delegate) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval
@@ -26,7 +26,7 @@ Requires-Python: >=3.13
 Requires-Dist: anthropic<2.0.0,>=1.0.0
 Requires-Dist: anyio>=4.14.2
 Requires-Dist: azure-monitor-opentelemetry-exporter<1.1.0,>=1.0.0b30
-Requires-Dist: claude-agent-sdk>=0.2.124
+Requires-Dist: claude-agent-sdk>=0.2.157
 Requires-Dist: click>=8.3.3
 Requires-Dist: httpx2<3.0.0,>=2.12.0
 Requires-Dist: jmespath>=1.1.0
@@ -43,9 +43,9 @@ Requires-Dist: tomli-w>=1.0.0
 Requires-Dist: tqdm>=4.67.3
 Requires-Dist: typer>=0.24.1
 Provides-Extra: antigravity
-Requires-Dist: google-antigravity==0.1.8; extra == 'antigravity'
+Requires-Dist: google-antigravity==0.1.18; extra == 'antigravity'
 Provides-Extra: codex
-Requires-Dist: openai-codex>=0.144.4; extra == 'codex'
+Requires-Dist: openai-codex>=0.155.1; extra == 'codex'
 Provides-Extra: dev
 Requires-Dist: bandit[toml]>=1.9.4; extra == 'dev'
 Requires-Dist: defusedxml>=0.7.1; extra == 'dev'
@@ -60,7 +60,7 @@ Requires-Dist: pytest-xdist[psutil]>=3.0.0; extra == 'dev'
 Requires-Dist: pytest>=9.0.2; extra == 'dev'
 Requires-Dist: ruff>=0.15.7; extra == 'dev'
 Provides-Extra: harbor
-Requires-Dist: harbor==0.22.0; extra == 'harbor'
+Requires-Dist: harbor==0.23.0; extra == 'harbor'
 Provides-Extra: litellm
 Requires-Dist: litellm<2.0.0,>=1.95.0; extra == 'litellm'
 Provides-Extra: opencode

{coder_eval-0.12.6 → coder_eval-0.12.8}/action.yml

@@ -25,7 +25,7 @@ inputs:
   version:
     description: coder-eval version to install from PyPI, or "local" to install from the action checkout
     required: false
-    default: "0.12.6"   # <-- kept in sync with releases by release.yml
+    default: "0.12.8"   # <-- kept in sync with releases by release.yml
   extras:
     description: >-
       Comma-separated coder-eval extras (`codex`, `antigravity,litellm`), composed

{coder_eval-0.12.6 → coder_eval-0.12.8}/docker/Dockerfile

@@ -30,7 +30,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
 # tag and is bumped deliberately -- mirrors the codex CLI pin
 # (`openai-codex-cli-bin==…` in pyproject). Leaving it at `@latest` let Docker
 # layer caching freeze it nondeterministically across rebuilds.
-ARG CLAUDE_CODE_VERSION=2.1.177
+ARG CLAUDE_CODE_VERSION=2.1.281
 RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
     && apt-get install -y --no-install-recommends nodejs \
     && rm -rf /var/lib/apt/lists/* \
@@ -39,7 +39,7 @@ RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
 # Pi Node CLI, pinned — same rationale as the Claude Code pin above (the agent
 # binary is a dominant non-model driver of results; @latest would freeze
 # nondeterministically under layer caching). Node 22 + npm are already present.
-ARG PI_VERSION=0.84.4
+ARG PI_VERSION=0.87.1
 RUN npm install -g @earendil-works/pi-coding-agent@${PI_VERSION}
 
 # uv: matches host sandbox.py's `uv venv` + `uv pip install` fast path
@@ -84,12 +84,12 @@ COPY experiments/default.yaml ./experiments/default.yaml
 # pulls the `uipath` SDK from public PyPI (no credentials required).
 ARG CODER_EVAL_UV_EXTRAS=""
 
-# safe-chain min-age exclusions for the codex cli-bin pin (see pyproject [tool.uv])
-# and the pinned Antigravity harness. Codex and Antigravity are always installed,
-# so the default excludes their pinned packages from the min-age gate (the
-# google-antigravity release tracks the rapidly-moving Gemini harness and may be
+# safe-chain min-age exclusions for the codex cli-bin pin (see pyproject [tool.uv]),
+# the pinned Antigravity harness, and claude-agent-sdk (its wheel bundles the Claude
+# Code CLI). All three are always installed, so the default excludes their pinned
+# packages from the min-age gate (each tracks a rapidly-moving harness and may be
 # newer than the gate window); callers may override.
-ARG SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS="openai-codex-cli-bin,openai-codex,google-antigravity"
+ARG SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS="openai-codex-cli-bin,openai-codex,google-antigravity,claude-agent-sdk"
 
 # All extras (codex, antigravity, litellm, and the opt-in uipath) resolve from
 # public PyPI per uv.lock, so the build needs no private-index credentials.

{coder_eval-0.12.6 → coder_eval-0.12.8}/docker/Dockerfile.runtime

@@ -53,7 +53,7 @@ COPY pyproject.toml uv.lock README.md ./
 COPY src/ ./src/
 COPY experiments/default.yaml ./experiments/default.yaml
 
-ARG SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS="openai-codex-cli-bin,openai-codex"
+ARG SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS="openai-codex-cli-bin,openai-codex,claude-agent-sdk"
 # Install the locked deps, then coder_eval itself NON-editable. An editable/path
 # install would leave the package source at /src (outside the kit), so the
 # `COPY --from /opt/coder-eval` overlay would ship a venv that can't import
@@ -66,7 +66,7 @@ RUN export SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS="${SAFE_CHAIN_MINIMUM_PACKA
 
 # --- 3. Node LTS + the Claude Code CLI, under the kit dir. -------------------
 ARG NODE_VERSION=22.14.0
-ARG CLAUDE_CODE_VERSION=2.1.177
+ARG CLAUDE_CODE_VERSION=2.1.281
 # Download to a file first so a bad URL fails the build loudly (a piped
 # `curl | tar` would mask a 404 and only surface later as a missing npm), and
 # verify the tarball against nodejs.org's published SHASUMS256 before extracting
@@ -109,6 +109,6 @@ RUN chmod +x /usr/local/bin/coder_eval_entrypoint.sh
 # label; labels don't survive `COPY --from`). The claude-code pin mirrors the
 # framework image — a parity test (tests/test_image_from_dockerfiles.py) enforces it.
 ARG CODER_EVAL_VERSION=unknown
-ARG CLAUDE_CODE_VERSION=2.1.177
+ARG CLAUDE_CODE_VERSION=2.1.281
 LABEL org.coder-eval.version="${CODER_EVAL_VERSION}"
 LABEL org.coder-eval.claude-code-version="${CLAUDE_CODE_VERSION}"

{coder_eval-0.12.6 → coder_eval-0.12.8}/docs/DOCKER_ISOLATION.md

@@ -339,6 +339,45 @@ If the task declares a `reference:` block, a throwaway **copy** of its directory
 
 See [Reference Solutions](TASK_DEFINITION_GUIDE.md#reference-solutions).
 
+### Two more passive-read anti-cheat blocks
+
+These are `driver: docker` only. `driver: tempdir` shares the host uid and has no
+filesystem isolation, so neither applies there (nor can — there is nothing to
+mask). Both are defense-in-depth passive-read blocks, consistent with the
+reference window's posture above; neither contains an adversarial agent.
+
+- **The staged grading inputs are deleted after load.** The host stages the
+  post-override `TaskDefinition` (with `success_criteria`) at `/work/input/task.yaml`
+  **and** a `context.json` whose `source_yaml` is the raw task text — criteria
+  verbatim, both at the top level and inside every `config_lineage` entry — for the
+  in-container orchestrator to load once at startup. The agent runs in the same
+  container, so leaving *either* readable would hand it the grading answer key
+  (deleting only `task.yaml` leaves the identical criteria one file over in
+  `context.json`). The in-container entry point deletes **both** immediately after
+  they are consumed — `context.json` is parsed into memory in the command body and
+  `task.yaml` by `load_task`, both before the delete (gated on
+  `CODER_EVAL_IN_CONTAINER`). They are read exactly once — grading reads criteria
+  from the in-memory task, never from disk. The `/work/input` mount is therefore
+  read-write (a `:ro` mount rejects `rm` with EROFS). `prior.json` is kept: it is
+  read later on the regrade path, and a regrade runs no agent so it is not a leak.
+
+- **Auto-mounted plugin trees are default-deny masked.** An `agent.plugins[].path`
+  (or a `TemplateDirSource.path` that is itself a plugin root) is auto-mounted at
+  its host path `:ro` so the plugin loads. Eval material colocated under that tree
+  as siblings of the skills dir — sibling task YAMLs, reference solutions, test
+  fixtures — would otherwise be readable. So the runner keeps the whole root
+  mounted but layers an empty `--tmpfs` over every child dir OUTSIDE the keep-set
+  (`.claude-plugin` + the manifest-declared skill dirs). Everything that is not
+  the plugin surface is masked by default, so an unknown or new eval layout can
+  never leak; `tests/`, `node_modules/`, and reference solutions are masked for
+  free. A root agent cannot `umount` a tmpfs (`CAP_SYS_ADMIN` is not in Docker's
+  default set), so this mask is *stronger* than the mode-000 reference window. Two
+  residuals the mask cannot cover — an eval def or reference COLOCATED inside a
+  skill dir (masking it would hide the skill), and a `task_id:` YAML **file** loose
+  at the plugin root (a tmpfs masks a directory, not a single file) — are caught by
+  lint rule CE068 (keep eval material out of skill dirs and off the plugin root;
+  put it under a sibling `tests/`).
+
 Inside the container, the entrypoint invokes `coder-eval _run-task-internal` (hidden subcommand), which loads the staged YAML + context, runs the standard in-process Orchestrator (driver auto-coerced back to `tempdir`), and writes `task.json` to the output mount. Host reads it and feeds the existing aggregation pipeline.
 
 A `result_kind` discriminator on `CriterionResult` ensures `ClassificationCriterionResult` subclasses survive the JSON round-trip — without it, host-side aggregation would silently lose `observed_label`/`expected_label`.

{coder_eval-0.12.6 → coder_eval-0.12.8}/docs/REPORT_SCHEMA.md

@@ -188,14 +188,17 @@ fields so subclass keys round-trip.
 
 ### Post-failure criterion evidence
 
-When an agent crashes or its turn times out, coder-eval runs only deterministic,
-read-only artifact criteria while the sandbox is still live: `file_exists`,
+When an agent crashes or its turn times out on a graded run, coder-eval runs only
+deterministic, read-only artifact criteria while the sandbox is still live: `file_exists`,
 `file_contains`, `file_matches_regex`, `file_check`, `json_check`,
-`reference_comparison`, and `classification_match`. Judges, trajectory checks,
-`run_command`, and `uipath_eval` are recorded with
-`evaluation_status="not_evaluated"`; they are not invoked on this recovery path.
+`reference_comparison`, and `classification_match`. A `run_command` criterion joins them
+only when the task author sets [`read_only: true`](TASK_DEFINITION_GUIDE.md#run_command)
+on it. Judges, trajectory checks, plain `run_command`, and `uipath_eval` are recorded
+with `evaluation_status="not_evaluated"`; they are not invoked on this recovery path.
 The diagnostic list is additive evidence. An `ERROR` run remains `ERROR`, and its
-canonical score remains 0.0.
+canonical score remains 0.0. The list stays empty under `coder-eval execute`, and on a
+token/cost budget breach, which fires only after every criterion is already scored in
+`success_criteria_results`.
 
 ### TurnRecord
 
@@ -207,7 +210,10 @@ canonical score remains 0.0.
 cache buckets, captured proxy-side on the LiteLLM open-weight backend and rendered
 by the evalboard as a per-call table; empty on every other
 backend), `num_turns`, `max_turns_exhausted`,
-`result_summary` (`{is_error, subtype, stop_reason, result}`), `crashed`,
+`result_summary` (`{is_error, subtype, stop_reason, result}`), `provider_errors`
+(`list[ProviderError]` — `{at, message, kind, http_status, will_retry, details}`, one
+row per model-provider error the agent's own client reported, including errors it
+retried itself; filled by the codex adapter, empty on other backends), `crashed`,
 `crash_reason`.
 
 > **Token invariant.** Summing the four token buckets across `messages`

{coder_eval-0.12.6 → coder_eval-0.12.8}/docs/TASK_DEFINITION_GUIDE.md

@@ -904,6 +904,12 @@ Runs a command and checks the exit code, with optional stdout matching. **Binary
   expected_stdout: "Hello, World!"    # Optional: check stdout content
   stdout_match: "exact"               # "exact" (default), "contains", or "regex"
   description: "Script must output the correct text"
+
+# Also graded when the turn times out or the agent crashes
+- type: "run_command"
+  command: "python graders/check_flow.py"
+  read_only: true                     # declaration, not enforcement -- see below
+  description: "Flow must contain the approval node"
 ```
 
 | Field | Default | Description |
@@ -914,6 +920,19 @@ Runs a command and checks the exit code, with optional stdout matching. **Binary
 | `expected_stdout` | `null` | When set, stdout is also checked |
 | `stdout_match` | `"exact"` | Match mode: `exact` (stripped), `contains` (substring), `regex` (pattern) |
 | `score_from_stdout` | `false` | Read a float score (0.0–1.0) from the first stdout line (remaining lines become details); a non-zero exit code or a parse failure scores 0.0. Mutually exclusive with `expected_stdout`. |
+| `read_only` | `false` | Declares the command inspects artifacts only. Its sole effect: a graded run also runs the criterion after a turn timeout or an agent crash — see [Post-failure criterion evidence](REPORT_SCHEMA.md#post-failure-criterion-evidence). |
+
+`read_only` is an author declaration, **not** a restriction. coder-eval cannot decide
+whether a shell command is pure, so it verifies nothing and confines nothing: the
+command runs exactly as it always does. Set it only when the command reads artifacts and
+nothing else. Leave it `false` when the command writes state or calls a live service (a
+`uip maestro flow debug` grader starts a real cloud job, so it must stay `false`).
+
+Keep a `read_only` criterion's `timeout` short. The diagnostic pass runs after the agent
+is gone, and the commands are not interruptible: a `task_timeout` that expires mid-pass
+cancels the await, not the shell subprocess, so it keeps running while the sandbox is
+torn down. Without a `task_timeout` the pass is bounded only by the sum of these
+timeouts.
 
 ### `file_matches_regex`
 
@@ -978,6 +997,8 @@ Checks whether the agent executed specific tools/commands during evaluation. Ins
 
 **Shell normalization.** For a Bash command, both `command_pattern` and `exclude_pattern` are matched against the raw command text **and** its shell-normalized form — the `bash`/`sh`/`zsh -lc "..."` wrapper stripped and shell quoting resolved with `shlex` — and a hit on *either* form counts. So a pattern like `curated_channels` matches whether the agent wrote the argument bare, `'single'`-quoted, `"double"`-quoted, or `\"escaped\"`; you do **not** hand-encode shell quoting. Because the same haystacks also feed `exclude_pattern` and the `max_count` gate, normalization is **not** purely additive: a quote-obfuscated call can now be caught by an exclusion or a `max_count: 0` gate that the raw text alone would have missed — and, conversely, an unedited `exclude_pattern` may now exclude a call it previously let through. Cross-repo suites that hand-encoded quote tolerance in their patterns should re-baseline.
 
+**Long commands.** A Bash command is searched in full: the first 2,000 characters, then the rest in windows of at most 2,000 characters that start and end on line boundaries. So a command that follows a long heredoc in the same Bash call still counts. Other tools match only on the first 2,000 characters of their JSON parameters, so a long Write or Edit body is not read as a command. This is not purely additive either: an `exclude_pattern` or a `max_count: 0` gate can now catch a call late in a long script that it did not see before, so suites should re-baseline. A match that crosses a window boundary is not guaranteed, and `^` and `$` can also match at a window edge.
+
 **Codex limitation.** Codex agents map `Read`, `Grep`, and `Glob` tools to `shell` commands (they execute via bash), so `tool_name: "Read"` on Codex returns no matches. Use `tool_name: "Bash"` or `tool_name: null` (any tool) for Codex-compatible checks. This criterion works correctly on Claude Code agents, which emit separate `Read`/`Grep`/`Glob` telemetry.
 
 ### `cli_called`

{coder_eval-0.12.6 → coder_eval-0.12.8}/docs/agents/ANTIGRAVITY.md

@@ -27,7 +27,7 @@ working directory — both required for an unattended eval.
 pip install 'coder-eval[antigravity]'
 ```
 
-This pulls in `google-antigravity` (pinned to `0.1.8`), whose wheel bundles the
+This pulls in `google-antigravity` (pinned to `0.1.18`), whose wheel bundles the
 platform `localharness` binary. As with the other agents the SDK is imported lazily
 — a base install without the extra still runs end-to-end; Antigravity tasks fail at
 dispatch with a clear hint to install the extra.

{coder_eval-0.12.6 → coder_eval-0.12.8}/docs/agents/HARNESS_PARITY.md

@@ -67,8 +67,8 @@ All five harnesses can have tool execution inside a generation window, and it is
 subtracted out of every one of them — **once, centrally**, by
 `timing.py::subtract_tool_time`. No reducer does it itself; each
 publishes the raw window (see the two sections below). Every harness has the
-problem: Antigravity reports a `Step` for the tool and only a later
-`usage_metadata` `Step` cuts the message; Codex's message window is seeded from
+problem: Antigravity reports a `Step` for the tool and only a later rise in
+the conversation's billed usage cuts the message; Codex's message window is seeded from
 the first item's start and extended to the last item's completion; OpenCode, Pi
 and claude-code tile, each window opening where the previous one closed and
 running to the next, with every tool call in between running inside. In each the
@@ -559,8 +559,9 @@ Each harness finds the call boundary in its own stream (the table above):
   first item would let one tool of call N+1 act. A call that ran tools therefore
   opens the next call at its `tokenUsage`, since the results always go back to the
   model, and the cap fires before call N+1 can run anything.
-- **Antigravity** attaches `usage_metadata` to one step per call, and the next call
-  opens with a MODEL step at a new `step_index`.
+- **Antigravity** bills each call on the conversation's cumulative usage, not on a
+  step, so a rise in that total closes the call, and the next call opens with a MODEL
+  step at a new `step_index`.
 - **OpenCode** and **Pi** stream one `step_start` or `turn_start` per call.
 - **Delegate**'s SDK has no round-trip marker, and a tool-only reply streams only
   its tool call, with no text before it. So the next call opens when every tool the

{coder_eval-0.12.6 → coder_eval-0.12.8}/evalboard/app/_overview/__tests__/efficiency-charts.test.tsx

@@ -32,7 +32,7 @@ describe("EfficiencyCharts", () => {
     test("opens on the wall-clock metric", () => {
         renderCharts();
         expect(screen.getByRole("heading")).toHaveTextContent(
-            "Time per Passed Task",
+            "Agent Time per Passed Task",
         );
         expect(
             screen.getByRole("tab", { name: "Time" }),

{coder_eval-0.12.6 → coder_eval-0.12.8}/evalboard/app/_overview/efficiency-charts.tsx

@@ -40,10 +40,10 @@ const TABS: Array<{
     {
         key: "time",
         label: "Time",
-        heading: "Time per Passed Task",
-        title: "Total wall clock ÷ tasks passed. Every task's seconds count, failures included; only passes count in the denominator, so a run that fails more reads slower.",
+        heading: "Agent Time per Passed Task",
+        title: "Agent wall clock ÷ tasks passed. Counts the agent's turns only: sandbox setup, pre_run, grading and cleanup are excluded. Every task's agent seconds count, failures included; only passes count in the denominator, so a run that fails more reads slower.",
         blurb: (scoped) =>
-            "Seconds of every task that ran ÷ the number that passed · hover a point for the share within 2× expected" +
+            "Agent seconds of every task that ran (setup and grading excluded) ÷ the number that passed · hover a point for the share within 2× expected" +
             (scoped ? " · scoped to the active filter" : ""),
         render: (props) => <TimePerPassedTaskChart {...props} />,
     },

{coder_eval-0.12.6 → coder_eval-0.12.8}/evalboard/app/_overview/window-summary.tsx

@@ -36,7 +36,7 @@ function Tile({
 // Front-page window rollup: total spend + shape of the runs in scope. Every
 // tile is summed over the same set — `runCount` and `totals` both come out of
 // getOverview's single pass, so the Runs tile can never disagree with the
-// Cost/Tasks/Pass/Compute tiles or with the charts. The totals are scoped to
+// Cost/Tasks/Pass/Agent time tiles or with the charts. The totals are scoped to
 // matching tasks whenever a filter is active.
 //
 // This describes the window the charts plot, NOT however far the run table below
@@ -91,9 +91,9 @@ export function WindowSummary({
                     valueClass={passClass(pct)}
                 />
                 <Tile
-                    label="Compute time"
-                    value={fmtDuration(totals.durationSeconds)}
-                    sub={totals.durationPartial ? "some runs missing" : undefined}
+                    label="Agent time"
+                    value={fmtDuration(totals.agentSeconds)}
+                    sub={totals.agentPartial ? "some runs missing" : undefined}
                 />
             </div>
         </section>