coder-eval 0.12.5__tar.gz → 0.12.7__tar.gz

{coder_eval-0.12.5 → coder_eval-0.12.7}/.claude/notes/agents.md

@@ -61,14 +61,12 @@ intentionally brief and out of scope; trimming for DISPLAY belongs in the render
 
 - **Harness run-limit parity**: a shared `BaseAgentConfig` field must mean the same
   thing on every backend, so a divergence is either fixed or documented — never silent.
-  **`run_limits.max_turns` on Codex/Antigravity counts VISIBLE turns** (resolved tool
-  calls, read live off the shared `EventCollector.visible_turn_count`, the same list
-  `TurnRecord.commands` holds) because one `communicate()` is a single SDK turn on both,
-  so a native counter would clamp at 1; claude-code keeps its native SDK cap, whose unit
-  (an agent-loop turn) absorbs arbitrarily many parallel calls — the same number is NOT
-  the same budget across harnesses. OpenCode and Pi each keep a native unit too, because
-  their CLIs stream a real multi-step loop per `communicate()`
-  (`step_start`/`step_finish`, `turn_start`/`turn_end`). The cap is enforced on the same
+  **`run_limits.max_turns` counts main-thread model API calls on every harness**, per
+  iteration (each retry and dialog exchange starts at zero). Codex and Antigravity run one
+  SDK turn per `communicate()`, so each counts calls from its own stream (Codex
+  `thread/tokenUsage/updated`, Antigravity MODEL steps). claude-code keeps the CLI's
+  `--max-turns` plus a backstop that counts main-thread `message_id`s. OpenCode and Pi
+  stream one `step_start` / `turn_start` per call. The cap is enforced on the same
   loop boundary as the cooperative early stop and finalizes cleanly as
   `max_turns_exhausted` (no crash, no retry); on Antigravity that boundary lives in
   `_drain()`, so the background-work poll loop honors it too.

{coder_eval-0.12.5 → coder_eval-0.12.7}/.claude/notes/contracts.md

@@ -95,9 +95,14 @@ the raw text alone can stop counting.
 It is memoized because the early-stop watcher re-scans the whole accumulated trajectory on
 every tool-call event, normalizing the same command many times per run.
 
-The regex search window is capped to bound ReDoS on a large command string, and
-normalization runs over that same truncated window, so `shlex` never sees more than the cap
-and needs no separate guard.
+Each regex search is bounded to 2000 characters to limit ReDoS on a large command string,
+but the whole command is searched: the first 2000 characters, then the rest in windows that
+start and end on logical-line boundaries. Agents write long heredoc scripts that end in the
+command a task checks for (`cat > x <<EOF ... EOF` then `uip agent validate`), and a single
+leading window scored those as never run. Normalization runs per window, so `shlex` never
+sees more than the bound and needs no separate guard. Only a Bash command is windowed:
+another tool's params are JSON, and a later window of a Write/Edit body is file content,
+not a command the agent ran, so those keep the single leading window.
 
 ## Recording a CLI invocation
 

{coder_eval-0.12.5 → coder_eval-0.12.7}/.claude/notes/orchestration.md

@@ -108,6 +108,23 @@ VERDICT, never the facts — the seeding cannot restore a fact the execute phase
 captured. The budget gate runs AFTER the criteria on the graded path purely for
 partial-credit visibility, and there is no partial credit under `execute`.
 
+### Post-failure evidence is declared, not proven
+
+A timed-out run used to score 0.00 on artifacts that every grader accepted, because the
+whole uipath-maestro-flow suite grades through `run_command` and the path ran only file
+checks. `read_only: true` admits a single named command.
+
+The gate could not stay a ClassVar: `run_command` is safe or unsafe per criterion, not
+per type, and no analysis decides which — an arbitrary shell command can start a cloud
+job (`uip maestro flow debug` does). So the flag is the AUTHOR's claim, the ClassVar
+stayed the type-level answer, and `evaluable_after_agent_failure` became the per-instance
+one every caller asks. Widening the flag to the base would let a judge declare itself
+deterministic, which is a different and false claim.
+
+What keeps it honest: results land in `post_failure_criteria_results`, which
+`calculate_weighted_score` never reads. A timed-out run stays ERROR at 0.0 no matter what
+the diagnostic pass finds — the flag buys evidence, never a verdict.
+
 ### Rates need verdict evidence, not bucket counts
 
 A published rate divides by rows that actually carry a verdict, not by a bucket count. The

{coder_eval-0.12.5 → coder_eval-0.12.7}/.github/workflows/harbor-e2e.yml

@@ -29,6 +29,8 @@ env:
   # Anthropic-credit spend off this path; DirectRoute is exercised elsewhere.
   API_BACKEND: "bedrock"
   CLAUDE_CODE_USE_BEDROCK: "1"
+  # Same expression as pr-checks.yml (see the comment there).
+  SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }},claude-agent-sdk
 
 jobs:
   harbor-e2e:

{coder_eval-0.12.5 → coder_eval-0.12.7}/.github/workflows/pr-checks.yml

@@ -38,11 +38,12 @@ env:
   TELEMETRY_ENABLED: "false"
   # The `uipath-*` pool enforces a minimum package-age safe-chain check on installs.
   # Workflow-level so every installing job inherits it; per-job copies are how some
-  # jobs previously ended up with no exclusions at all. The literal is the operative
-  # value — no secret of that name exists at repo or org level, so the bare `secrets.`
-  # reference this replaced resolved to an empty list. (Image builds carry their own
-  # list in docker/Dockerfile; deliberately not the same set.)
-  SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }}
+  # jobs previously ended up with no exclusions at all. A repo secret of this name
+  # replaces the literal when set, so claude-agent-sdk sits outside the `||`: its wheel
+  # bundles the Claude Code CLI, and new-model support ships there faster than the
+  # gate window. (Image builds carry their own list in docker/Dockerfile; deliberately
+  # not the same set.)
+  SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }},claude-agent-sdk
 
 jobs:
   quality-gate:

{coder_eval-0.12.5 → coder_eval-0.12.7}/.github/workflows/publish-testpypi.yml

@@ -38,7 +38,7 @@ jobs:
     timeout-minutes: 10
     env:
       # `uv build` resolves build deps under the pool's safe-chain gate.
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }}
+      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }},claude-agent-sdk
     environment:
       name: testpypi
       url: https://test.pypi.org/project/coder-eval/

{coder_eval-0.12.5 → coder_eval-0.12.7}/.github/workflows/release.yml

@@ -75,7 +75,7 @@ jobs:
       # Load-bearing on the release path: the pool enforces a package-age safe-chain
       # check on uv installs. Same expression as pr-checks.yml (see the comment there),
       # so a package can't pass PR CI and then fail the release install.
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }}
+      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }},claude-agent-sdk
 
     steps:
       # Only a real release (main) needs the app token: semantic-release pushes the

{coder_eval-0.12.5 → coder_eval-0.12.7}/CHANGELOG.md

@@ -2,6 +2,158 @@
 
 <!-- version list -->
 
+## v0.12.7 (2026-09-25)
+
+### Bug Fixes
+
+- **criteria**: Search past the 2000-char bound only in Bash commands
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+- **criteria**: Search the whole command in command_executed and bump harness SDKs for Opus 5.5
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+- **criteria**: Search the whole command in command_executed, not just the first 2000 chars
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+- **docs**: Drop the budget-breach trigger, cut read_only to its contract
+  ([#197](https://github.com/UiPath/coder_eval/pull/197),
+  [`28c846e`](https://github.com/UiPath/coder_eval/commit/28c846ea6858fef04d366d245e5f7b679c2b75a1))
+
+- **evalboard**: List date-named runs flagged adhoc in the Ad-hoc section
+  ([#200](https://github.com/UiPath/coder_eval/pull/200),
+  [`5bf4a72`](https://github.com/UiPath/coder_eval/commit/5bf4a72693f007e93da678b58e0ddb84fa1bf63c))
+
+- **harbor**: Declare SUPPORTS_ATIF as a ClassVar to match harbor 0.23
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+### Build System
+
+- **deps**: Bump harness SDKs and CLIs to latest
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+- **deps**: Exempt claude-agent-sdk from the 48h package-age gate and take the latest SDK
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+- **deps**: Hold the SDK bump to releases past CI's 48h package-age gate
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+### Continuous Integration
+
+- **harbor**: Exempt claude-agent-sdk from the package-age gate in Harbor E2E
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+### Documentation
+
+- **agents**: Update Pi and Antigravity references for the harness bump
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+- **criteria**: Name the budget-breach trigger on the post-failure path
+  ([#197](https://github.com/UiPath/coder_eval/pull/197),
+  [`28c846e`](https://github.com/UiPath/coder_eval/commit/28c846ea6858fef04d366d245e5f7b679c2b75a1))
+
+### Features
+
+- **criteria**: Let a run_command criterion opt in to post-failure grading
+  ([#197](https://github.com/UiPath/coder_eval/pull/197),
+  [`28c846e`](https://github.com/UiPath/coder_eval/commit/28c846ea6858fef04d366d245e5f7b679c2b75a1))
+
+- **evalboard**: Count agent time apart from eval overhead, and filter runs by variant
+  ([#198](https://github.com/UiPath/coder_eval/pull/198),
+  [`b04829c`](https://github.com/UiPath/coder_eval/commit/b04829c4dec86772822a28ef1b3fa7316c1f3c8d))
+
+- **evalboard**: Headline agent time per passed task, not full task wall clock
+  ([#198](https://github.com/UiPath/coder_eval/pull/198),
+  [`b04829c`](https://github.com/UiPath/coder_eval/commit/b04829c4dec86772822a28ef1b3fa7316c1f3c8d))
+
+- **evalboard**: Show agent time on the run page and filter it by variant
+  ([#198](https://github.com/UiPath/coder_eval/pull/198),
+  [`b04829c`](https://github.com/UiPath/coder_eval/commit/b04829c4dec86772822a28ef1b3fa7316c1f3c8d))
+
+- **evalboard**: Show summed agent time in the overview window tile
+  ([#198](https://github.com/UiPath/coder_eval/pull/198),
+  [`b04829c`](https://github.com/UiPath/coder_eval/commit/b04829c4dec86772822a28ef1b3fa7316c1f3c8d))
+
+- **evalboard**: Simplify the task timing strip into one bar and two groups
+  ([#198](https://github.com/UiPath/coder_eval/pull/198),
+  [`b04829c`](https://github.com/UiPath/coder_eval/commit/b04829c4dec86772822a28ef1b3fa7316c1f3c8d))
+
+- **evalboard**: Split the task timeline into agent time and eval overhead
+  ([#198](https://github.com/UiPath/coder_eval/pull/198),
+  [`b04829c`](https://github.com/UiPath/coder_eval/commit/b04829c4dec86772822a28ef1b3fa7316c1f3c8d))
+
+
+## v0.12.6 (2026-09-23)
+
+### Bug Fixes
+
+- **claude-code**: End a turn the CLI did not cap at max_turns
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **claude-code**: Kill the CLI on a cooperative stop
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **claude-code**: Kill the CLI on a max_turns backstop stop without a turn timeout
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **codex**: End a capped turn before the next call can run a tool
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **delegate**: Count a tool-only reply as a model call
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **delegate**: Keep counting model calls when a tool never returns
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **evalboard**: Compare visible turns, not SDK num_turns, against expected_turns
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **pricing**: Add Opus 5.5 and GPT-6, correct rates that moved publicly
+  ([#195](https://github.com/UiPath/coder_eval/pull/195),
+  [`11b902f`](https://github.com/UiPath/coder_eval/commit/11b902fa9bc0d307bcab22801dc56523fcc1d92a))
+
+- **run-limits**: Count max_turns as model API calls on every harness
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **run-limits**: Count max_turns as model API calls on every harness and enforce it on claude-code
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **run-limits**: Stop a turn mid-flight once a token or USD budget is crossed
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **simulation**: End a dialog with agent_max_turns when the agent hits its own cap
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+### Documentation
+
+- **parity**: Record the claude-code max_turns backstop
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **run-limits**: Say max_turns restarts per iteration and expected_turns counts tool calls
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+
 ## v0.12.5 (2026-09-22)
 
 ### Bug Fixes

{coder_eval-0.12.5 → coder_eval-0.12.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: coder-eval
-Version: 0.12.5
+Version: 0.12.7
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode, Pi, Delegate) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval
@@ -26,7 +26,7 @@ Requires-Python: >=3.13
 Requires-Dist: anthropic<2.0.0,>=1.0.0
 Requires-Dist: anyio>=4.14.2
 Requires-Dist: azure-monitor-opentelemetry-exporter<1.1.0,>=1.0.0b30
-Requires-Dist: claude-agent-sdk>=0.2.124
+Requires-Dist: claude-agent-sdk>=0.2.157
 Requires-Dist: click>=8.3.3
 Requires-Dist: httpx2<3.0.0,>=2.12.0
 Requires-Dist: jmespath>=1.1.0
@@ -43,9 +43,9 @@ Requires-Dist: tomli-w>=1.0.0
 Requires-Dist: tqdm>=4.67.3
 Requires-Dist: typer>=0.24.1
 Provides-Extra: antigravity
-Requires-Dist: google-antigravity==0.1.8; extra == 'antigravity'
+Requires-Dist: google-antigravity==0.1.18; extra == 'antigravity'
 Provides-Extra: codex
-Requires-Dist: openai-codex>=0.144.4; extra == 'codex'
+Requires-Dist: openai-codex>=0.155.1; extra == 'codex'
 Provides-Extra: dev
 Requires-Dist: bandit[toml]>=1.9.4; extra == 'dev'
 Requires-Dist: defusedxml>=0.7.1; extra == 'dev'
@@ -60,7 +60,7 @@ Requires-Dist: pytest-xdist[psutil]>=3.0.0; extra == 'dev'
 Requires-Dist: pytest>=9.0.2; extra == 'dev'
 Requires-Dist: ruff>=0.15.7; extra == 'dev'
 Provides-Extra: harbor
-Requires-Dist: harbor==0.22.0; extra == 'harbor'
+Requires-Dist: harbor==0.23.0; extra == 'harbor'
 Provides-Extra: litellm
 Requires-Dist: litellm<2.0.0,>=1.95.0; extra == 'litellm'
 Provides-Extra: opencode

{coder_eval-0.12.5 → coder_eval-0.12.7}/action.yml

@@ -25,7 +25,7 @@ inputs:
   version:
     description: coder-eval version to install from PyPI, or "local" to install from the action checkout
     required: false
-    default: "0.12.5"   # <-- kept in sync with releases by release.yml
+    default: "0.12.7"   # <-- kept in sync with releases by release.yml
   extras:
     description: >-
       Comma-separated coder-eval extras (`codex`, `antigravity,litellm`), composed

{coder_eval-0.12.5 → coder_eval-0.12.7}/docker/Dockerfile

@@ -30,7 +30,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
 # tag and is bumped deliberately -- mirrors the codex CLI pin
 # (`openai-codex-cli-bin==…` in pyproject). Leaving it at `@latest` let Docker
 # layer caching freeze it nondeterministically across rebuilds.
-ARG CLAUDE_CODE_VERSION=2.1.177
+ARG CLAUDE_CODE_VERSION=2.1.281
 RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
     && apt-get install -y --no-install-recommends nodejs \
     && rm -rf /var/lib/apt/lists/* \
@@ -39,7 +39,7 @@ RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
 # Pi Node CLI, pinned — same rationale as the Claude Code pin above (the agent
 # binary is a dominant non-model driver of results; @latest would freeze
 # nondeterministically under layer caching). Node 22 + npm are already present.
-ARG PI_VERSION=0.84.4
+ARG PI_VERSION=0.87.1
 RUN npm install -g @earendil-works/pi-coding-agent@${PI_VERSION}
 
 # uv: matches host sandbox.py's `uv venv` + `uv pip install` fast path
@@ -84,12 +84,12 @@ COPY experiments/default.yaml ./experiments/default.yaml
 # pulls the `uipath` SDK from public PyPI (no credentials required).
 ARG CODER_EVAL_UV_EXTRAS=""
 
-# safe-chain min-age exclusions for the codex cli-bin pin (see pyproject [tool.uv])
-# and the pinned Antigravity harness. Codex and Antigravity are always installed,
-# so the default excludes their pinned packages from the min-age gate (the
-# google-antigravity release tracks the rapidly-moving Gemini harness and may be
+# safe-chain min-age exclusions for the codex cli-bin pin (see pyproject [tool.uv]),
+# the pinned Antigravity harness, and claude-agent-sdk (its wheel bundles the Claude
+# Code CLI). All three are always installed, so the default excludes their pinned
+# packages from the min-age gate (each tracks a rapidly-moving harness and may be
 # newer than the gate window); callers may override.
-ARG SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS="openai-codex-cli-bin,openai-codex,google-antigravity"
+ARG SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS="openai-codex-cli-bin,openai-codex,google-antigravity,claude-agent-sdk"
 
 # All extras (codex, antigravity, litellm, and the opt-in uipath) resolve from
 # public PyPI per uv.lock, so the build needs no private-index credentials.

{coder_eval-0.12.5 → coder_eval-0.12.7}/docker/Dockerfile.runtime

@@ -53,7 +53,7 @@ COPY pyproject.toml uv.lock README.md ./
 COPY src/ ./src/
 COPY experiments/default.yaml ./experiments/default.yaml
 
-ARG SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS="openai-codex-cli-bin,openai-codex"
+ARG SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS="openai-codex-cli-bin,openai-codex,claude-agent-sdk"
 # Install the locked deps, then coder_eval itself NON-editable. An editable/path
 # install would leave the package source at /src (outside the kit), so the
 # `COPY --from /opt/coder-eval` overlay would ship a venv that can't import
@@ -66,7 +66,7 @@ RUN export SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS="${SAFE_CHAIN_MINIMUM_PACKA
 
 # --- 3. Node LTS + the Claude Code CLI, under the kit dir. -------------------
 ARG NODE_VERSION=22.14.0
-ARG CLAUDE_CODE_VERSION=2.1.177
+ARG CLAUDE_CODE_VERSION=2.1.281
 # Download to a file first so a bad URL fails the build loudly (a piped
 # `curl | tar` would mask a 404 and only surface later as a missing npm), and
 # verify the tarball against nodejs.org's published SHASUMS256 before extracting
@@ -109,6 +109,6 @@ RUN chmod +x /usr/local/bin/coder_eval_entrypoint.sh
 # label; labels don't survive `COPY --from`). The claude-code pin mirrors the
 # framework image — a parity test (tests/test_image_from_dockerfiles.py) enforces it.
 ARG CODER_EVAL_VERSION=unknown
-ARG CLAUDE_CODE_VERSION=2.1.177
+ARG CLAUDE_CODE_VERSION=2.1.281
 LABEL org.coder-eval.version="${CODER_EVAL_VERSION}"
 LABEL org.coder-eval.claude-code-version="${CLAUDE_CODE_VERSION}"

{coder_eval-0.12.5 → coder_eval-0.12.7}/docs/DIALOG_MODE.md

@@ -121,13 +121,15 @@ After each exchange the driver evaluates the stop conditions **in this order**,
 2. **`stop_on_criteria_pass`** (`criteria_passed`) — every success criterion passes. Requires
    per-turn checking (`check_criteria: every_turn` or `both`); pairing it with the default
    `end_of_dialog` is rejected at load time, since there would be nothing to check against.
-3. **`max_turns`** (`max_turns`) — the hard cap on exchanges. The agent exhausting its *own* inner
-   `max_turns` mid-exchange ends the dialog with the same reason.
+3. **`max_turns`** (`max_turns`) — the hard cap on exchanges.
 4. **`max_total_tokens`** (`budget`) — the dialog-wide budget across simulator **and** agent. The
    dialog ends and the task is **still scored** — unlike
    [`run_limits.max_total_tokens`](TASK_DEFINITION_GUIDE.md#run-limits), which covers the subject
    agent only and aborts.
-5. **`stop_token`** (`stop_token`) — only if none of the above fired is the simulator asked for
+5. **`run_limits.max_turns`** (`agent_max_turns`): the agent used up its own model-call cap inside
+   one exchange. That cap restarts on every exchange, so this reason means one exchange ran long, not
+   that the dialog ran out of exchanges.
+6. **`stop_token`** (`stop_token`) — only if none of the above fired is the simulator asked for
    another message; the sentinel token in *that fresh utterance* ends the dialog. This is the
    workhorse in practice — the simulator decides, in character, that it got what it wanted — but it
    is evaluated **last**, so a turn that trips `max_turns` or the budget never gets the chance to

{coder_eval-0.12.5 → coder_eval-0.12.7}/docs/REPORT_SCHEMA.md

@@ -188,14 +188,17 @@ fields so subclass keys round-trip.
 
 ### Post-failure criterion evidence
 
-When an agent crashes or its turn times out, coder-eval runs only deterministic,
-read-only artifact criteria while the sandbox is still live: `file_exists`,
+When an agent crashes or its turn times out on a graded run, coder-eval runs only
+deterministic, read-only artifact criteria while the sandbox is still live: `file_exists`,
 `file_contains`, `file_matches_regex`, `file_check`, `json_check`,
-`reference_comparison`, and `classification_match`. Judges, trajectory checks,
-`run_command`, and `uipath_eval` are recorded with
-`evaluation_status="not_evaluated"`; they are not invoked on this recovery path.
+`reference_comparison`, and `classification_match`. A `run_command` criterion joins them
+only when the task author sets [`read_only: true`](TASK_DEFINITION_GUIDE.md#run_command)
+on it. Judges, trajectory checks, plain `run_command`, and `uipath_eval` are recorded
+with `evaluation_status="not_evaluated"`; they are not invoked on this recovery path.
 The diagnostic list is additive evidence. An `ERROR` run remains `ERROR`, and its
-canonical score remains 0.0.
+canonical score remains 0.0. The list stays empty under `coder-eval execute`, and on a
+token/cost budget breach, which fires only after every criterion is already scored in
+`success_criteria_results`.
 
 ### TurnRecord
 

{coder_eval-0.12.5 → coder_eval-0.12.7}/docs/TASK_DEFINITION_GUIDE.md

@@ -240,7 +240,7 @@ valid and an empty block is legal — every field defaults to "no limit".
 ```yaml
 run_limits:
   # Structural caps
-  max_turns: 20                       # hard cap on agent inner-loop turns per iteration
+  max_turns: 20                       # hard cap on model API calls per iteration
   expected_turns: 8                   # SOFT efficiency budget (visible turns) — never aborts
   task_timeout: 300                   # wall-clock cap for the full run envelope, seconds
   turn_timeout: 300                   # per-communicate() timeout, seconds
@@ -255,8 +255,8 @@ run_limits:
 
 | Field | Default | Constraint | Description |
 |-------|---------|------------|-------------|
-| `max_turns` | *unset* | `> 0` | Hard cap on agent inner-loop turns per iteration. Unset uses the SDK default. |
-| `expected_turns` | *unset* | `>= 1` | **Soft** target for cumulative visible turns. Exceeding it warns and badges the report; it never aborts. See [`expected_turns`](#expected_turns-soft-efficiency-budget). |
+| `max_turns` | *unset* | `> 0` | Hard cap on main-thread model API calls per iteration, Claude Code's turn, counted the same on every harness. The tools the last allowed call asks for still run; the turn ends when the next call begins. Each retry and each dialog exchange starts a fresh count. Unset uses the SDK default. See [HARNESS_PARITY.md](agents/HARNESS_PARITY.md). |
+| `expected_turns` | *unset* | `>= 1` | **Soft** target for visible turns (tool calls plus the final reply) summed over the whole task, a different unit from `max_turns`. Exceeding it warns and badges the report; it never aborts. See [`expected_turns`](#expected_turns-soft-efficiency-budget). |
 | `task_timeout` | *unset* | `>= 30` | Max seconds for the full run envelope, including agent work, grading, and post-run work. |
 | `turn_timeout` | *unset* | `>= 10` | Max seconds for the agent's single `communicate()` iteration. |
 | `max_input_tokens` | *unset* | `>= 1` | Max cumulative input (prompt) tokens. |
@@ -327,8 +327,8 @@ that did: a budgeted task that failed counts as over budget, while tasks with no
 `expected_turns` budget are excluded entirely (success or fail).
 
 The count compared against the budget is **visible turns** — one per tool call
-plus one for the agent's final reply — *not* the SDK's `total_turns` (which
-counts assistant messages and can bundle several tool calls into one).
+plus one for the agent's final reply. It is *not* `total_turns`, which counts
+model API calls (the `max_turns` unit), and one call can batch several tool calls.
 
 Set it to the number of turns a competent agent should need for the task. Pick
 budgets consistently across a suite — the headline % is only comparable when
@@ -904,6 +904,12 @@ Runs a command and checks the exit code, with optional stdout matching. **Binary
   expected_stdout: "Hello, World!"    # Optional: check stdout content
   stdout_match: "exact"               # "exact" (default), "contains", or "regex"
   description: "Script must output the correct text"
+
+# Also graded when the turn times out or the agent crashes
+- type: "run_command"
+  command: "python graders/check_flow.py"
+  read_only: true                     # declaration, not enforcement -- see below
+  description: "Flow must contain the approval node"
 ```
 
 | Field | Default | Description |
@@ -914,6 +920,19 @@ Runs a command and checks the exit code, with optional stdout matching. **Binary
 | `expected_stdout` | `null` | When set, stdout is also checked |
 | `stdout_match` | `"exact"` | Match mode: `exact` (stripped), `contains` (substring), `regex` (pattern) |
 | `score_from_stdout` | `false` | Read a float score (0.0–1.0) from the first stdout line (remaining lines become details); a non-zero exit code or a parse failure scores 0.0. Mutually exclusive with `expected_stdout`. |
+| `read_only` | `false` | Declares the command inspects artifacts only. Its sole effect: a graded run also runs the criterion after a turn timeout or an agent crash — see [Post-failure criterion evidence](REPORT_SCHEMA.md#post-failure-criterion-evidence). |
+
+`read_only` is an author declaration, **not** a restriction. coder-eval cannot decide
+whether a shell command is pure, so it verifies nothing and confines nothing: the
+command runs exactly as it always does. Set it only when the command reads artifacts and
+nothing else. Leave it `false` when the command writes state or calls a live service (a
+`uip maestro flow debug` grader starts a real cloud job, so it must stay `false`).
+
+Keep a `read_only` criterion's `timeout` short. The diagnostic pass runs after the agent
+is gone, and the commands are not interruptible: a `task_timeout` that expires mid-pass
+cancels the await, not the shell subprocess, so it keeps running while the sandbox is
+torn down. Without a `task_timeout` the pass is bounded only by the sum of these
+timeouts.
 
 ### `file_matches_regex`
 
@@ -978,6 +997,8 @@ Checks whether the agent executed specific tools/commands during evaluation. Ins
 
 **Shell normalization.** For a Bash command, both `command_pattern` and `exclude_pattern` are matched against the raw command text **and** its shell-normalized form — the `bash`/`sh`/`zsh -lc "..."` wrapper stripped and shell quoting resolved with `shlex` — and a hit on *either* form counts. So a pattern like `curated_channels` matches whether the agent wrote the argument bare, `'single'`-quoted, `"double"`-quoted, or `\"escaped\"`; you do **not** hand-encode shell quoting. Because the same haystacks also feed `exclude_pattern` and the `max_count` gate, normalization is **not** purely additive: a quote-obfuscated call can now be caught by an exclusion or a `max_count: 0` gate that the raw text alone would have missed — and, conversely, an unedited `exclude_pattern` may now exclude a call it previously let through. Cross-repo suites that hand-encoded quote tolerance in their patterns should re-baseline.
 
+**Long commands.** A Bash command is searched in full: the first 2,000 characters, then the rest in windows of at most 2,000 characters that start and end on line boundaries. So a command that follows a long heredoc in the same Bash call still counts. Other tools match only on the first 2,000 characters of their JSON parameters, so a long Write or Edit body is not read as a command. This is not purely additive either: an `exclude_pattern` or a `max_count: 0` gate can now catch a call late in a long script that it did not see before, so suites should re-baseline. A match that crosses a window boundary is not guaranteed, and `^` and `$` can also match at a window edge.
+
 **Codex limitation.** Codex agents map `Read`, `Grep`, and `Glob` tools to `shell` commands (they execute via bash), so `tool_name: "Read"` on Codex returns no matches. Use `tool_name: "Bash"` or `tool_name: null` (any tool) for Codex-compatible checks. This criterion works correctly on Claude Code agents, which emit separate `Read`/`Grep`/`Glob` telemetry.
 
 ### `cli_called`
@@ -1721,7 +1742,7 @@ The simulator runs as a tools-disabled Claude Code agent on its own resolved `Ap
 **Semantics:**
 
 - The task's `initial_prompt` is the user's *opening* message; the simulator picks up from turn 2.
-- `max_turns` is the intra-dialog cap (the worst-case agent call budget per trial). Use `n_trials` for variance sampling.
+- `max_turns` caps exchanges. Each exchange also gets a fresh `run_limits.max_turns` of model API calls, so the worst case per trial is the product of the two. Use `n_trials` for variance sampling.
 - The `reference` solution, if present, is hidden from the simulator (same security posture as for the coding agent).
 - When `n_trials > 1`, each trial becomes its own `ResolvedTask` with its own zero-padded replicate directory (`runs/<ts>/<variant_id>/<task_id>/<NN>/`) and its own `task.json` — the same fan-out mechanism as experiment `repeats`, which `n_trials` takes precedence over when simulation is enabled. Trial-level metadata appears under `simulation.replicate_index` / `simulation.n_trials` on the `EvaluationResult`.
 

{coder_eval-0.12.5 → coder_eval-0.12.7}/docs/agents/ANTIGRAVITY.md

@@ -27,7 +27,7 @@ working directory — both required for an unattended eval.
 pip install 'coder-eval[antigravity]'
 ```
 
-This pulls in `google-antigravity` (pinned to `0.1.8`), whose wheel bundles the
+This pulls in `google-antigravity` (pinned to `0.1.18`), whose wheel bundles the
 platform `localharness` binary. As with the other agents the SDK is imported lazily
 — a base install without the extra still runs end-to-end; Antigravity tasks fail at
 dispatch with a clear hint to install the extra.
@@ -190,8 +190,9 @@ as every other agent.
 5. **`allowed_tools` / `disallowed_tools` are not read.** The harness runs with its
    full builtin tool set, so an Antigravity run has tools (web search, subagents,
    URL fetch) that the same task file denies on Claude Code and Codex.
-6. **`max_turns` counts visible turns.** One `communicate()` is a single SDK turn here,
-   so the cap counts resolved tool calls instead, enforced on the step loop. See
+6. **`max_turns` is counted by the harness.** One `communicate()` is a single SDK turn
+   here, so the harness counts model API calls itself (a MODEL step at a new
+   `step_index` opens one) and enforces the cap on the step loop. See
    [Run-Limit Parity](HARNESS_PARITY.md).
 7. **Shell commands over ~10s are moved to the background.** The localharness has a
    10-second maximum synchronous wait; past it the command becomes a background task

{coder_eval-0.12.5 → coder_eval-0.12.7}/docs/agents/CODEX.md

@@ -217,7 +217,7 @@ The Codex SDK is synchronous. The agent uses `_run_async()` helper to detect and
 | **Session Resume** | `--resume {session_id}` | Via thread ID |
 | **Permissions** | `permission_mode` + `allowed_tools` | `permission_mode` → sandbox/approval + `allowed_tools`/`disallowed_tools` → thread config |
 | **Tool Enforcement** | Not enforced by Coder Eval wrapper | `enabled_tools` honored; `disabled_tools` NOT enforced by the SDK |
-| **`max_turns`** | Native SDK turn cap (assistant messages) | Visible-turn cap (tool calls), enforced on the notification pump |
+| **`max_turns`** | Model API calls: the CLI's `--max-turns`, plus a harness backstop | Model API calls, counted per `thread/tokenUsage/updated` and enforced on the notification pump |
 | **Early stop** | Supported (cooperative `should_stop`, polled between messages) | Supported — polled after each streamed notification; the in-flight turn is interrupted best-effort |
 
 Run-limit semantics per harness: [Run-Limit Parity](HARNESS_PARITY.md).