coder-eval 0.12.6__tar.gz → 0.12.7__tar.gz

{coder_eval-0.12.6 → coder_eval-0.12.7}/.claude/notes/contracts.md

@@ -95,9 +95,14 @@ the raw text alone can stop counting.
 It is memoized because the early-stop watcher re-scans the whole accumulated trajectory on
 every tool-call event, normalizing the same command many times per run.
 
-The regex search window is capped to bound ReDoS on a large command string, and
-normalization runs over that same truncated window, so `shlex` never sees more than the cap
-and needs no separate guard.
+Each regex search is bounded to 2000 characters to limit ReDoS on a large command string,
+but the whole command is searched: the first 2000 characters, then the rest in windows that
+start and end on logical-line boundaries. Agents write long heredoc scripts that end in the
+command a task checks for (`cat > x <<EOF ... EOF` then `uip agent validate`), and a single
+leading window scored those as never run. Normalization runs per window, so `shlex` never
+sees more than the bound and needs no separate guard. Only a Bash command is windowed:
+another tool's params are JSON, and a later window of a Write/Edit body is file content,
+not a command the agent ran, so those keep the single leading window.
 
 ## Recording a CLI invocation
 

{coder_eval-0.12.6 → coder_eval-0.12.7}/.claude/notes/orchestration.md

@@ -108,6 +108,23 @@ VERDICT, never the facts — the seeding cannot restore a fact the execute phase
 captured. The budget gate runs AFTER the criteria on the graded path purely for
 partial-credit visibility, and there is no partial credit under `execute`.
 
+### Post-failure evidence is declared, not proven
+
+A timed-out run used to score 0.00 on artifacts that every grader accepted, because the
+whole uipath-maestro-flow suite grades through `run_command` and the path ran only file
+checks. `read_only: true` admits a single named command.
+
+The gate could not stay a ClassVar: `run_command` is safe or unsafe per criterion, not
+per type, and no analysis decides which — an arbitrary shell command can start a cloud
+job (`uip maestro flow debug` does). So the flag is the AUTHOR's claim, the ClassVar
+stayed the type-level answer, and `evaluable_after_agent_failure` became the per-instance
+one every caller asks. Widening the flag to the base would let a judge declare itself
+deterministic, which is a different and false claim.
+
+What keeps it honest: results land in `post_failure_criteria_results`, which
+`calculate_weighted_score` never reads. A timed-out run stays ERROR at 0.0 no matter what
+the diagnostic pass finds — the flag buys evidence, never a verdict.
+
 ### Rates need verdict evidence, not bucket counts
 
 A published rate divides by rows that actually carry a verdict, not by a bucket count. The

{coder_eval-0.12.6 → coder_eval-0.12.7}/.github/workflows/harbor-e2e.yml

@@ -29,6 +29,8 @@ env:
   # Anthropic-credit spend off this path; DirectRoute is exercised elsewhere.
   API_BACKEND: "bedrock"
   CLAUDE_CODE_USE_BEDROCK: "1"
+  # Same expression as pr-checks.yml (see the comment there).
+  SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }},claude-agent-sdk
 
 jobs:
   harbor-e2e:

{coder_eval-0.12.6 → coder_eval-0.12.7}/.github/workflows/pr-checks.yml

@@ -38,11 +38,12 @@ env:
   TELEMETRY_ENABLED: "false"
   # The `uipath-*` pool enforces a minimum package-age safe-chain check on installs.
   # Workflow-level so every installing job inherits it; per-job copies are how some
-  # jobs previously ended up with no exclusions at all. The literal is the operative
-  # value — no secret of that name exists at repo or org level, so the bare `secrets.`
-  # reference this replaced resolved to an empty list. (Image builds carry their own
-  # list in docker/Dockerfile; deliberately not the same set.)
-  SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }}
+  # jobs previously ended up with no exclusions at all. A repo secret of this name
+  # replaces the literal when set, so claude-agent-sdk sits outside the `||`: its wheel
+  # bundles the Claude Code CLI, and new-model support ships there faster than the
+  # gate window. (Image builds carry their own list in docker/Dockerfile; deliberately
+  # not the same set.)
+  SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }},claude-agent-sdk
 
 jobs:
   quality-gate:

{coder_eval-0.12.6 → coder_eval-0.12.7}/.github/workflows/publish-testpypi.yml

@@ -38,7 +38,7 @@ jobs:
     timeout-minutes: 10
     env:
       # `uv build` resolves build deps under the pool's safe-chain gate.
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }}
+      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }},claude-agent-sdk
     environment:
       name: testpypi
       url: https://test.pypi.org/project/coder-eval/

{coder_eval-0.12.6 → coder_eval-0.12.7}/.github/workflows/release.yml

@@ -75,7 +75,7 @@ jobs:
       # Load-bearing on the release path: the pool enforces a package-age safe-chain
       # check on uv installs. Same expression as pr-checks.yml (see the comment there),
       # so a package can't pass PR CI and then fail the release install.
-      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }}
+      SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS: ${{ secrets.SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS || 'openai-codex-cli-bin,openai-codex' }},claude-agent-sdk
 
     steps:
       # Only a real release (main) needs the app token: semantic-release pushes the

{coder_eval-0.12.6 → coder_eval-0.12.7}/CHANGELOG.md

@@ -2,6 +2,95 @@
 
 <!-- version list -->
 
+## v0.12.7 (2026-09-25)
+
+### Bug Fixes
+
+- **criteria**: Search past the 2000-char bound only in Bash commands
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+- **criteria**: Search the whole command in command_executed and bump harness SDKs for Opus 5.5
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+- **criteria**: Search the whole command in command_executed, not just the first 2000 chars
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+- **docs**: Drop the budget-breach trigger, cut read_only to its contract
+  ([#197](https://github.com/UiPath/coder_eval/pull/197),
+  [`28c846e`](https://github.com/UiPath/coder_eval/commit/28c846ea6858fef04d366d245e5f7b679c2b75a1))
+
+- **evalboard**: List date-named runs flagged adhoc in the Ad-hoc section
+  ([#200](https://github.com/UiPath/coder_eval/pull/200),
+  [`5bf4a72`](https://github.com/UiPath/coder_eval/commit/5bf4a72693f007e93da678b58e0ddb84fa1bf63c))
+
+- **harbor**: Declare SUPPORTS_ATIF as a ClassVar to match harbor 0.23
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+### Build System
+
+- **deps**: Bump harness SDKs and CLIs to latest
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+- **deps**: Exempt claude-agent-sdk from the 48h package-age gate and take the latest SDK
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+- **deps**: Hold the SDK bump to releases past CI's 48h package-age gate
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+### Continuous Integration
+
+- **harbor**: Exempt claude-agent-sdk from the package-age gate in Harbor E2E
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+### Documentation
+
+- **agents**: Update Pi and Antigravity references for the harness bump
+  ([#196](https://github.com/UiPath/coder_eval/pull/196),
+  [`3aa2db3`](https://github.com/UiPath/coder_eval/commit/3aa2db30cdb07e47ca2cce98b47385eedb2de9cd))
+
+- **criteria**: Name the budget-breach trigger on the post-failure path
+  ([#197](https://github.com/UiPath/coder_eval/pull/197),
+  [`28c846e`](https://github.com/UiPath/coder_eval/commit/28c846ea6858fef04d366d245e5f7b679c2b75a1))
+
+### Features
+
+- **criteria**: Let a run_command criterion opt in to post-failure grading
+  ([#197](https://github.com/UiPath/coder_eval/pull/197),
+  [`28c846e`](https://github.com/UiPath/coder_eval/commit/28c846ea6858fef04d366d245e5f7b679c2b75a1))
+
+- **evalboard**: Count agent time apart from eval overhead, and filter runs by variant
+  ([#198](https://github.com/UiPath/coder_eval/pull/198),
+  [`b04829c`](https://github.com/UiPath/coder_eval/commit/b04829c4dec86772822a28ef1b3fa7316c1f3c8d))
+
+- **evalboard**: Headline agent time per passed task, not full task wall clock
+  ([#198](https://github.com/UiPath/coder_eval/pull/198),
+  [`b04829c`](https://github.com/UiPath/coder_eval/commit/b04829c4dec86772822a28ef1b3fa7316c1f3c8d))
+
+- **evalboard**: Show agent time on the run page and filter it by variant
+  ([#198](https://github.com/UiPath/coder_eval/pull/198),
+  [`b04829c`](https://github.com/UiPath/coder_eval/commit/b04829c4dec86772822a28ef1b3fa7316c1f3c8d))
+
+- **evalboard**: Show summed agent time in the overview window tile
+  ([#198](https://github.com/UiPath/coder_eval/pull/198),
+  [`b04829c`](https://github.com/UiPath/coder_eval/commit/b04829c4dec86772822a28ef1b3fa7316c1f3c8d))
+
+- **evalboard**: Simplify the task timing strip into one bar and two groups
+  ([#198](https://github.com/UiPath/coder_eval/pull/198),
+  [`b04829c`](https://github.com/UiPath/coder_eval/commit/b04829c4dec86772822a28ef1b3fa7316c1f3c8d))
+
+- **evalboard**: Split the task timeline into agent time and eval overhead
+  ([#198](https://github.com/UiPath/coder_eval/pull/198),
+  [`b04829c`](https://github.com/UiPath/coder_eval/commit/b04829c4dec86772822a28ef1b3fa7316c1f3c8d))
+
+
 ## v0.12.6 (2026-09-23)
 
 ### Bug Fixes

{coder_eval-0.12.6 → coder_eval-0.12.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: coder-eval
-Version: 0.12.6
+Version: 0.12.7
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode, Pi, Delegate) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval
@@ -26,7 +26,7 @@ Requires-Python: >=3.13
 Requires-Dist: anthropic<2.0.0,>=1.0.0
 Requires-Dist: anyio>=4.14.2
 Requires-Dist: azure-monitor-opentelemetry-exporter<1.1.0,>=1.0.0b30
-Requires-Dist: claude-agent-sdk>=0.2.124
+Requires-Dist: claude-agent-sdk>=0.2.157
 Requires-Dist: click>=8.3.3
 Requires-Dist: httpx2<3.0.0,>=2.12.0
 Requires-Dist: jmespath>=1.1.0
@@ -43,9 +43,9 @@ Requires-Dist: tomli-w>=1.0.0
 Requires-Dist: tqdm>=4.67.3
 Requires-Dist: typer>=0.24.1
 Provides-Extra: antigravity
-Requires-Dist: google-antigravity==0.1.8; extra == 'antigravity'
+Requires-Dist: google-antigravity==0.1.18; extra == 'antigravity'
 Provides-Extra: codex
-Requires-Dist: openai-codex>=0.144.4; extra == 'codex'
+Requires-Dist: openai-codex>=0.155.1; extra == 'codex'
 Provides-Extra: dev
 Requires-Dist: bandit[toml]>=1.9.4; extra == 'dev'
 Requires-Dist: defusedxml>=0.7.1; extra == 'dev'
@@ -60,7 +60,7 @@ Requires-Dist: pytest-xdist[psutil]>=3.0.0; extra == 'dev'
 Requires-Dist: pytest>=9.0.2; extra == 'dev'
 Requires-Dist: ruff>=0.15.7; extra == 'dev'
 Provides-Extra: harbor
-Requires-Dist: harbor==0.22.0; extra == 'harbor'
+Requires-Dist: harbor==0.23.0; extra == 'harbor'
 Provides-Extra: litellm
 Requires-Dist: litellm<2.0.0,>=1.95.0; extra == 'litellm'
 Provides-Extra: opencode

{coder_eval-0.12.6 → coder_eval-0.12.7}/action.yml

@@ -25,7 +25,7 @@ inputs:
   version:
     description: coder-eval version to install from PyPI, or "local" to install from the action checkout
     required: false
-    default: "0.12.6"   # <-- kept in sync with releases by release.yml
+    default: "0.12.7"   # <-- kept in sync with releases by release.yml
   extras:
     description: >-
       Comma-separated coder-eval extras (`codex`, `antigravity,litellm`), composed

{coder_eval-0.12.6 → coder_eval-0.12.7}/docker/Dockerfile

@@ -30,7 +30,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
 # tag and is bumped deliberately -- mirrors the codex CLI pin
 # (`openai-codex-cli-bin==…` in pyproject). Leaving it at `@latest` let Docker
 # layer caching freeze it nondeterministically across rebuilds.
-ARG CLAUDE_CODE_VERSION=2.1.177
+ARG CLAUDE_CODE_VERSION=2.1.281
 RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
     && apt-get install -y --no-install-recommends nodejs \
     && rm -rf /var/lib/apt/lists/* \
@@ -39,7 +39,7 @@ RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
 # Pi Node CLI, pinned — same rationale as the Claude Code pin above (the agent
 # binary is a dominant non-model driver of results; @latest would freeze
 # nondeterministically under layer caching). Node 22 + npm are already present.
-ARG PI_VERSION=0.84.4
+ARG PI_VERSION=0.87.1
 RUN npm install -g @earendil-works/pi-coding-agent@${PI_VERSION}
 
 # uv: matches host sandbox.py's `uv venv` + `uv pip install` fast path
@@ -84,12 +84,12 @@ COPY experiments/default.yaml ./experiments/default.yaml
 # pulls the `uipath` SDK from public PyPI (no credentials required).
 ARG CODER_EVAL_UV_EXTRAS=""
 
-# safe-chain min-age exclusions for the codex cli-bin pin (see pyproject [tool.uv])
-# and the pinned Antigravity harness. Codex and Antigravity are always installed,
-# so the default excludes their pinned packages from the min-age gate (the
-# google-antigravity release tracks the rapidly-moving Gemini harness and may be
+# safe-chain min-age exclusions for the codex cli-bin pin (see pyproject [tool.uv]),
+# the pinned Antigravity harness, and claude-agent-sdk (its wheel bundles the Claude
+# Code CLI). All three are always installed, so the default excludes their pinned
+# packages from the min-age gate (each tracks a rapidly-moving harness and may be
 # newer than the gate window); callers may override.
-ARG SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS="openai-codex-cli-bin,openai-codex,google-antigravity"
+ARG SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS="openai-codex-cli-bin,openai-codex,google-antigravity,claude-agent-sdk"
 
 # All extras (codex, antigravity, litellm, and the opt-in uipath) resolve from
 # public PyPI per uv.lock, so the build needs no private-index credentials.

{coder_eval-0.12.6 → coder_eval-0.12.7}/docker/Dockerfile.runtime

@@ -53,7 +53,7 @@ COPY pyproject.toml uv.lock README.md ./
 COPY src/ ./src/
 COPY experiments/default.yaml ./experiments/default.yaml
 
-ARG SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS="openai-codex-cli-bin,openai-codex"
+ARG SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS="openai-codex-cli-bin,openai-codex,claude-agent-sdk"
 # Install the locked deps, then coder_eval itself NON-editable. An editable/path
 # install would leave the package source at /src (outside the kit), so the
 # `COPY --from /opt/coder-eval` overlay would ship a venv that can't import
@@ -66,7 +66,7 @@ RUN export SAFE_CHAIN_MINIMUM_PACKAGE_AGE_EXCLUSIONS="${SAFE_CHAIN_MINIMUM_PACKA
 
 # --- 3. Node LTS + the Claude Code CLI, under the kit dir. -------------------
 ARG NODE_VERSION=22.14.0
-ARG CLAUDE_CODE_VERSION=2.1.177
+ARG CLAUDE_CODE_VERSION=2.1.281
 # Download to a file first so a bad URL fails the build loudly (a piped
 # `curl | tar` would mask a 404 and only surface later as a missing npm), and
 # verify the tarball against nodejs.org's published SHASUMS256 before extracting
@@ -109,6 +109,6 @@ RUN chmod +x /usr/local/bin/coder_eval_entrypoint.sh
 # label; labels don't survive `COPY --from`). The claude-code pin mirrors the
 # framework image — a parity test (tests/test_image_from_dockerfiles.py) enforces it.
 ARG CODER_EVAL_VERSION=unknown
-ARG CLAUDE_CODE_VERSION=2.1.177
+ARG CLAUDE_CODE_VERSION=2.1.281
 LABEL org.coder-eval.version="${CODER_EVAL_VERSION}"
 LABEL org.coder-eval.claude-code-version="${CLAUDE_CODE_VERSION}"

{coder_eval-0.12.6 → coder_eval-0.12.7}/docs/REPORT_SCHEMA.md

@@ -188,14 +188,17 @@ fields so subclass keys round-trip.
 
 ### Post-failure criterion evidence
 
-When an agent crashes or its turn times out, coder-eval runs only deterministic,
-read-only artifact criteria while the sandbox is still live: `file_exists`,
+When an agent crashes or its turn times out on a graded run, coder-eval runs only
+deterministic, read-only artifact criteria while the sandbox is still live: `file_exists`,
 `file_contains`, `file_matches_regex`, `file_check`, `json_check`,
-`reference_comparison`, and `classification_match`. Judges, trajectory checks,
-`run_command`, and `uipath_eval` are recorded with
-`evaluation_status="not_evaluated"`; they are not invoked on this recovery path.
+`reference_comparison`, and `classification_match`. A `run_command` criterion joins them
+only when the task author sets [`read_only: true`](TASK_DEFINITION_GUIDE.md#run_command)
+on it. Judges, trajectory checks, plain `run_command`, and `uipath_eval` are recorded
+with `evaluation_status="not_evaluated"`; they are not invoked on this recovery path.
 The diagnostic list is additive evidence. An `ERROR` run remains `ERROR`, and its
-canonical score remains 0.0.
+canonical score remains 0.0. The list stays empty under `coder-eval execute`, and on a
+token/cost budget breach, which fires only after every criterion is already scored in
+`success_criteria_results`.
 
 ### TurnRecord
 

{coder_eval-0.12.6 → coder_eval-0.12.7}/docs/TASK_DEFINITION_GUIDE.md

@@ -904,6 +904,12 @@ Runs a command and checks the exit code, with optional stdout matching. **Binary
   expected_stdout: "Hello, World!"    # Optional: check stdout content
   stdout_match: "exact"               # "exact" (default), "contains", or "regex"
   description: "Script must output the correct text"
+
+# Also graded when the turn times out or the agent crashes
+- type: "run_command"
+  command: "python graders/check_flow.py"
+  read_only: true                     # declaration, not enforcement -- see below
+  description: "Flow must contain the approval node"
 ```
 
 | Field | Default | Description |
@@ -914,6 +920,19 @@ Runs a command and checks the exit code, with optional stdout matching. **Binary
 | `expected_stdout` | `null` | When set, stdout is also checked |
 | `stdout_match` | `"exact"` | Match mode: `exact` (stripped), `contains` (substring), `regex` (pattern) |
 | `score_from_stdout` | `false` | Read a float score (0.0–1.0) from the first stdout line (remaining lines become details); a non-zero exit code or a parse failure scores 0.0. Mutually exclusive with `expected_stdout`. |
+| `read_only` | `false` | Declares the command inspects artifacts only. Its sole effect: a graded run also runs the criterion after a turn timeout or an agent crash — see [Post-failure criterion evidence](REPORT_SCHEMA.md#post-failure-criterion-evidence). |
+
+`read_only` is an author declaration, **not** a restriction. coder-eval cannot decide
+whether a shell command is pure, so it verifies nothing and confines nothing: the
+command runs exactly as it always does. Set it only when the command reads artifacts and
+nothing else. Leave it `false` when the command writes state or calls a live service (a
+`uip maestro flow debug` grader starts a real cloud job, so it must stay `false`).
+
+Keep a `read_only` criterion's `timeout` short. The diagnostic pass runs after the agent
+is gone, and the commands are not interruptible: a `task_timeout` that expires mid-pass
+cancels the await, not the shell subprocess, so it keeps running while the sandbox is
+torn down. Without a `task_timeout` the pass is bounded only by the sum of these
+timeouts.
 
 ### `file_matches_regex`
 
@@ -978,6 +997,8 @@ Checks whether the agent executed specific tools/commands during evaluation. Ins
 
 **Shell normalization.** For a Bash command, both `command_pattern` and `exclude_pattern` are matched against the raw command text **and** its shell-normalized form — the `bash`/`sh`/`zsh -lc "..."` wrapper stripped and shell quoting resolved with `shlex` — and a hit on *either* form counts. So a pattern like `curated_channels` matches whether the agent wrote the argument bare, `'single'`-quoted, `"double"`-quoted, or `\"escaped\"`; you do **not** hand-encode shell quoting. Because the same haystacks also feed `exclude_pattern` and the `max_count` gate, normalization is **not** purely additive: a quote-obfuscated call can now be caught by an exclusion or a `max_count: 0` gate that the raw text alone would have missed — and, conversely, an unedited `exclude_pattern` may now exclude a call it previously let through. Cross-repo suites that hand-encoded quote tolerance in their patterns should re-baseline.
 
+**Long commands.** A Bash command is searched in full: the first 2,000 characters, then the rest in windows of at most 2,000 characters that start and end on line boundaries. So a command that follows a long heredoc in the same Bash call still counts. Other tools match only on the first 2,000 characters of their JSON parameters, so a long Write or Edit body is not read as a command. This is not purely additive either: an `exclude_pattern` or a `max_count: 0` gate can now catch a call late in a long script that it did not see before, so suites should re-baseline. A match that crosses a window boundary is not guaranteed, and `^` and `$` can also match at a window edge.
+
 **Codex limitation.** Codex agents map `Read`, `Grep`, and `Glob` tools to `shell` commands (they execute via bash), so `tool_name: "Read"` on Codex returns no matches. Use `tool_name: "Bash"` or `tool_name: null` (any tool) for Codex-compatible checks. This criterion works correctly on Claude Code agents, which emit separate `Read`/`Grep`/`Glob` telemetry.
 
 ### `cli_called`

{coder_eval-0.12.6 → coder_eval-0.12.7}/docs/agents/ANTIGRAVITY.md

@@ -27,7 +27,7 @@ working directory — both required for an unattended eval.
 pip install 'coder-eval[antigravity]'
 ```
 
-This pulls in `google-antigravity` (pinned to `0.1.8`), whose wheel bundles the
+This pulls in `google-antigravity` (pinned to `0.1.18`), whose wheel bundles the
 platform `localharness` binary. As with the other agents the SDK is imported lazily
 — a base install without the extra still runs end-to-end; Antigravity tasks fail at
 dispatch with a clear hint to install the extra.

{coder_eval-0.12.6 → coder_eval-0.12.7}/evalboard/app/_overview/__tests__/efficiency-charts.test.tsx

@@ -32,7 +32,7 @@ describe("EfficiencyCharts", () => {
     test("opens on the wall-clock metric", () => {
         renderCharts();
         expect(screen.getByRole("heading")).toHaveTextContent(
-            "Time per Passed Task",
+            "Agent Time per Passed Task",
         );
         expect(
             screen.getByRole("tab", { name: "Time" }),

{coder_eval-0.12.6 → coder_eval-0.12.7}/evalboard/app/_overview/efficiency-charts.tsx

@@ -40,10 +40,10 @@ const TABS: Array<{
     {
         key: "time",
         label: "Time",
-        heading: "Time per Passed Task",
-        title: "Total wall clock ÷ tasks passed. Every task's seconds count, failures included; only passes count in the denominator, so a run that fails more reads slower.",
+        heading: "Agent Time per Passed Task",
+        title: "Agent wall clock ÷ tasks passed. Counts the agent's turns only: sandbox setup, pre_run, grading and cleanup are excluded. Every task's agent seconds count, failures included; only passes count in the denominator, so a run that fails more reads slower.",
         blurb: (scoped) =>
-            "Seconds of every task that ran ÷ the number that passed · hover a point for the share within 2× expected" +
+            "Agent seconds of every task that ran (setup and grading excluded) ÷ the number that passed · hover a point for the share within 2× expected" +
             (scoped ? " · scoped to the active filter" : ""),
         render: (props) => <TimePerPassedTaskChart {...props} />,
     },

{coder_eval-0.12.6 → coder_eval-0.12.7}/evalboard/app/_overview/window-summary.tsx

@@ -36,7 +36,7 @@ function Tile({
 // Front-page window rollup: total spend + shape of the runs in scope. Every
 // tile is summed over the same set — `runCount` and `totals` both come out of
 // getOverview's single pass, so the Runs tile can never disagree with the
-// Cost/Tasks/Pass/Compute tiles or with the charts. The totals are scoped to
+// Cost/Tasks/Pass/Agent time tiles or with the charts. The totals are scoped to
 // matching tasks whenever a filter is active.
 //
 // This describes the window the charts plot, NOT however far the run table below
@@ -91,9 +91,9 @@ export function WindowSummary({
                     valueClass={passClass(pct)}
                 />
                 <Tile
-                    label="Compute time"
-                    value={fmtDuration(totals.durationSeconds)}
-                    sub={totals.durationPartial ? "some runs missing" : undefined}
+                    label="Agent time"
+                    value={fmtDuration(totals.agentSeconds)}
+                    sub={totals.agentPartial ? "some runs missing" : undefined}
                 />
             </div>
         </section>

{coder_eval-0.12.6 → coder_eval-0.12.7}/plugins/coder-eval/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "coder-eval",
   "displayName": "Coder Eval",
-  "version": "0.12.6",
+  "version": "0.12.7",
   "description": "Test whether your Claude Code skills actually trigger, and benchmark any coding agent — author, run, and analyze the eval suite that proves it, locally or as a CI gate.",
   "author": { "name": "UiPath", "email": "coder-eval@uipath.com", "url": "https://github.com/UiPath/coder_eval" },
   "homepage": "https://coder-eval.com",

{coder_eval-0.12.6 → coder_eval-0.12.7}/plugins/coder-eval/reference/criteria.md

@@ -249,6 +249,7 @@ Optional:
 | `expected_stdout` | Expected stdout content. When set, stdout is also checked. |
 | `stdout_match` | How to match stdout: 'exact' (stripped), 'contains' (substring), 'regex' (pattern) |
 | `score_from_stdout` | When true, read a float score (0.0-1.0) from the first line of stdout. Remaining lines are captured as details. Non-zero exit code or parse failure -> score 0.0. Mutually exclusive with expected_stdout. |
+| `read_only` | Declares this command an artifact-only check: it writes nothing and reaches no live service. Its one effect is that a graded run also runs it after an agent crash or turn timeout, on the diagnostic path, where the result is recorded but never scored. Nothing verifies the declaration; see the Task Definition Guide for when to set it. |
 
 ### `skill_triggered`
 

{coder_eval-0.12.6 → coder_eval-0.12.7}/plugins/coder-eval/skills/task/SKILL.md

@@ -120,6 +120,10 @@ Rules that matter:
   filename — a criterion matching that literal is a **smoke check**, not evidence: it
   only proves the agent typed back what it was told. Keep it if you like, at a low
   weight, and put the weight on a criterion that checks the resulting *behaviour*.
+- Set `read_only: true` on a `run_command` grader that only inspects artifacts: a run
+  whose agent crashed or timed out then still records what the artifacts were worth,
+  instead of scoring nothing. Never set it on a command that writes state or calls a
+  live service — nothing verifies the claim, and the command runs unchanged.
 - `weight` reflects importance: `0.5` nice-to-have, `1.0` standard, `1.5`–`2.0` critical.
   `weight: 0` makes a criterion informational (reported, but excluded from the score and
   the pass/fail gate).

{coder_eval-0.12.6 → coder_eval-0.12.7}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "coder-eval"
-version = "0.12.6"
+version = "0.12.7"
 description = "Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode, Pi, Delegate) with sandboxed, reproducible YAML task suites."
 readme = "README.md"
 license = "Apache-2.0"
@@ -43,7 +43,7 @@ dependencies = [
     # explicitly since our code/tests construct httpx2 types directly. Capped to
     # mirror anthropic's own bound on it (`httpx2<3,>=2.0.0`).
     "httpx2>=2.12.0,<3.0.0",
-    "claude-agent-sdk>=0.2.124",
+    "claude-agent-sdk>=0.2.157",
     "anyio>=4.14.2",
     "radon>=6.0.1",
     "tqdm>=4.67.3",
@@ -112,7 +112,7 @@ litellm = [
 # Without this extra, the framework still installs and runs; Codex-dependent
 # code paths fail at dispatch with a clear hint pointing back here.
 codex = [
-    "openai-codex>=0.144.4",
+    "openai-codex>=0.155.1",
 ]
 # Optional extra that enables Antigravity agent support:
 #   - AntigravityAgent implementation using Google's official google-antigravity SDK
@@ -131,7 +131,7 @@ codex = [
 # Without this extra the framework still installs and runs; Antigravity-dependent
 # code paths fail at start() with a clear hint pointing back here.
 antigravity = [
-    "google-antigravity==0.1.8",
+    "google-antigravity==0.1.18",
 ]
 # Optional extra that enables OpenCode agent support.
 #
@@ -180,7 +180,7 @@ pi = []
 # back here (it is not needed to EXPORT a task to Harbor format, only to run coder-eval
 # itself as Harbor's agent).
 harbor = [
-    "harbor==0.22.0",
+    "harbor==0.23.0",
 ]
 
 [project.scripts]
@@ -212,14 +212,14 @@ packages = ["src/coder_eval"]
 allow-direct-references = true
 
 [tool.uv]
-# openai-codex 0.144.4 hardpins `openai-codex-cli-bin==0.144.4` (a stable
+# openai-codex 0.156.1 hardpins `openai-codex-cli-bin==0.156.1` (a stable
 # release; the SDK version now tracks the codex CLI version line). Naming it
 # here documents and holds the pinned cli-bin build — the harness binary is a
 # dominant non-model driver of eval results, so it travels with the coder_eval
 # release tag and is bumped deliberately (mirrors the antigravity localharness
-# and claude-code CLI pins). 0.144.4 publishes manylinux wheels (x86_64 +
+# and claude-code CLI pins). 0.156.1 publishes manylinux wheels (x86_64 +
 # aarch64), so CI/Linux installs work.
-override-dependencies = ["openai-codex-cli-bin==0.144.4"]
+override-dependencies = ["openai-codex-cli-bin==0.156.1"]
 
 constraint-dependencies = [
     # Fix known CVEs in transitive dependencies
@@ -443,6 +443,8 @@ filterwarnings = [
     "ignore::DeprecationWarning",        # Ignore deprecation warnings from dependencies
     "ignore::PendingDeprecationWarning",
     "ignore::ResourceWarning",           # Ignore unclosed resources (handled by cleanup)
+    # litellm>=1.102 TypedDicts mark fields ReadOnly; pydantic warns when it builds their schema
+    "ignore:Item '.*' on TypedDict class '.*' is using the `ReadOnly` qualifier:UserWarning",
 ]
 
 # Coverage integration (when using --cov)

{coder_eval-0.12.6 → coder_eval-0.12.7}/src/coder_eval/__init__.py

@@ -1,3 +1,3 @@
 """coder_eval - A framework for evaluating AI coding agents."""
 
-__version__ = "0.12.6"
+__version__ = "0.12.7"

{coder_eval-0.12.6 → coder_eval-0.12.7}/src/coder_eval/agents/antigravity_agent.py

@@ -275,7 +275,7 @@ class AntigravityAgent(Agent[AntigravityAgentConfig]):
         return roots
 
     def _resolve_workspaces(self, skills_paths: list[str]) -> list[str]:
-        """Workspace roots for the harness's ``workspace_only`` file-tool policy.
+        """Workspace roots the harness confines its file tools to.
 
         The sandbox working directory (the write target) plus the resolved skill
         roots. ``skills_paths`` drives DISCOVERY only; the file-tool allowlist is
@@ -341,9 +341,9 @@ class AntigravityAgent(Agent[AntigravityAgentConfig]):
             cfg = LocalAgentConfig(
                 model=self._effective_model(),
                 api_key=api_key,
-                # File tools are confined to ``workspaces`` by the auto-prepended
-                # workspace_only policy — see _resolve_workspaces for why the skill
-                # roots must be in here and not only in ``skills_paths``.
+                # The harness confines file tools to ``workspaces``; see
+                # _resolve_workspaces for why the skill roots must be in here and
+                # not only in ``skills_paths``.
                 workspaces=self._resolve_workspaces(skills_paths),
                 # Autonomous execution: approve every tool call, which the default
                 # policy would deny. ``permission_mode`` is deliberately NOT mapped

{coder_eval-0.12.6 → coder_eval-0.12.7}/src/coder_eval/agents/pi_agent.py

@@ -5,7 +5,7 @@ events on stdout, and reduces that stream into the standardized coder_eval event
 protocol so :class:`EventCollector` builds the ``TurnRecord``. The design mirrors
 :mod:`coder_eval.agents.opencode_agent`.
 
-Three grammar facts that are not obvious from the event names (``pi`` 0.84.4):
+Three grammar facts that are not obvious from the event names (``pi`` 0.87.1):
 
 - ``agent_start`` can appear MORE THAN ONCE per invocation — Pi auto-retries a
   transient provider error internally — and ``agent_end`` is therefore NOT
@@ -149,7 +149,7 @@ _UNSUPPORTED_CONFIG_FIELDS: tuple[str, ...] = (
     "disallowed_tools",
 )
 
-# The full recognized Pi vocabulary (from `pi` 0.84.4). A clean exit that
+# The full recognized Pi vocabulary (from `pi` 0.87.1). A clean exit that
 # recognized NOTHING from this set is vocabulary drift and is crashed, not scored.
 # Rationale: .claude/notes/agents.md § Why a clean exit can still be a crash
 _RECOGNIZED_EVENTS = frozenset(