coder-eval 0.12.14__tar.gz → 0.12.16__tar.gz

{coder_eval-0.12.14 → coder_eval-0.12.16}/.claude/notes/agents.md

@@ -267,10 +267,15 @@ matter how much the run actually billed. So the CLI harnesses crash rather than
   arm keys on a step the CLI reported as FINISHED, its own claim that a generation
   completed, rather than on `usage.is_empty()` alone, which would also condemn a stream
   cut before any step could finish.
-- **A terminal provider error** (Pi, OpenCode) — `pi -p` exits 0 after exhausting its
+- **A terminal provider error** (Pi, OpenCode, Codex) — `pi -p` exits 0 after exhausting its
   internal retries, so without the crash the turn books as a clean COMPLETED
   (`FinalStatus.FAILURE`, category "failed"), silently depressing the measured pass rate.
   Crashing routes it to `FinalStatus.ERROR`, which is excluded from outcomes.
+  Codex reports the same thing differently: its `turn/completed` arrives normally but
+  carries `turn.status == failed` and a `TurnError` (for example `rateLimitExceeded`
+  after its own stream retries ran out). The harness reads that status and crashes, with
+  the provider's message, so `categorize_error` routes a rate limit to
+  `AGENT_RATE_LIMIT` and its retry budget like any other agent crash.
 - **A CLI that closed its stream but would not exit** within the grace period.
 
 Every arm is gated on `stopped_early` / `max_turns_exhausted`, because an intentional cut
@@ -373,7 +378,8 @@ Codex runs every sub-agent on its own child thread whose events never reach the
 stream, and that thread persists with *Limited* rollout policy, which drops
 `commandExecution` events. So neither the live stream nor `thread.read` surfaces the
 sub-agent's shell commands, and `thread/tokenUsage/updated` only ever reports the PARENT
-thread.
+thread. (The commands a code-mode script runs are the exception: they persist as
+`item_completed` records, below.)
 
 But the child rollout ALWAYS persists the raw `function_call` / `local_shell_call` /
 `custom_tool_call` ResponseItems and a `token_count` event with the child's cumulative
@@ -395,6 +401,72 @@ The rollout file can lag the parent's `turn/completed` by a beat (the recorder f
 a background task), so the lookup polls ~2 s — but bails immediately when
 `<home>/sessions` does not exist at all, since no flush can ever land there.
 
+Codex multi-agent v2 moved the spawn off the collab item. A v1 spawn is a
+`collabAgentToolCall` that names the child in `receiver_thread_ids`; a v2 spawn is a
+`subAgentActivity` item with `kind: started`, `id` = the spawn call id and
+`agent_thread_id` = the child, and its `wait` names no child at all. `_tool_item_root`
+maps a `started` item onto the v1 spawn shape where items enter the turn state, so one
+path registers both, and registers a child once: a repeated spawn item would otherwise
+mine the rollout twice and double the child's rows and tokens. The other kinds
+(`completed`, `interrupted`, `interacted`) are notices about a spawned child, dropped
+there, so they neither become rows nor open a parent model call.
+
+A child's tool calls come in two shapes. A direct call is a function call, and its output
+header carries the exit code. A code-mode call is a custom tool `exec` whose input is a
+JavaScript script, often several shell calls in a `Promise.all`. Its output tells only
+whether the SCRIPT ran to the end: `tools.exec_command` hands a non-zero exit code back to
+the script and does not throw, so `Script completed` is not a command result. The script
+source is not parsed — a command built from a variable or a loop defeats any parser, and
+the source holds no exit code. Instead, while a script runs, Codex writes an
+`item_completed` record per shell command (`CommandExecution`: argv, exit code, output)
+and per patch (`FileChange`), and those records ARE the script's rows, each with its own
+result. A script that left no record stays one `exec` row, not a `Bash` row: its
+JavaScript is not a shell command. The argv's login-shell wrapper (`/bin/zsh -lc <cmd>`)
+is stripped, so a record's row reads like a direct call's.
+
+A record names no script, so it goes to the newest script whose source holds its
+command. A command can outlive its script: `tools.exec_command` returns at the call's own
+`yield_time_ms` with a session id, the script ends, and a later script polls the command
+with `tools.write_stdin`. The text match then gives the record to the script that started
+the command, so that script is not also left as an `exec` row that repeats the command.
+The match decodes the source's string escapes and then ignores backslashes. A multi-line
+command is a `"a\nb"` literal in the source but holds real newlines in its record; in a
+recorded 0.147 TUI thread, 3 of 10 such commands outlived their script and, undecoded,
+went to a later one. The source also escapes the backslashes and double quotes that the
+command holds; without ignoring backslashes, no record of a real 0.156.1 Windows e2e child
+matched. A record that matches no source goes to the newest script: dropping it
+would lose a command built from a variable, the miss this recovery exists to prevent,
+while a wrong guess only moves the row to another generation. A record whose id is a
+direct call's is dropped, so a direct call is never counted twice. Seen on real CLI
+0.156.1 runs, v1 and v2 children alike: the records sit between the script call and its
+output, before the generation's `token_count`, and a 45-second command was polled as
+described. Headless threads through 0.147 (the 0.144 SDK, 0.147 `codex exec`) write no
+`CommandExecution` records, so there a code-mode script stays one `exec` row and its
+commands are not recoverable. The 0.147 TUI and 0.155.1 (the SDK floor) write them, as do
+0.156.1 and 0.162, on Windows too.
+
+A call with no result is `unknown`, never a success, so `require_success` does not credit
+it: an interrupt cut it off, it still ran after its yield time, or `wait` stopped it. The
+abort output is matched whole, so a command that prints `aborted by user` keeps its exit
+code. The `wait` or `write_stdin` call that later returns the result is a row of its own
+and does not change the yielded call: the records already carry each command's exit code,
+so only a yielded script that ran no command, or a direct `exec_command` that a direct
+`write_stdin` polls (no recorded run has one), stays `unknown`. A resuming call stays a
+row because a generation that only resumed would otherwise hold no block, and the recovery
+drops such a generation with its tokens; so each poll is one row, where the parent stream
+records a polled command once. The abort forms come from `abort_message` in
+`core/src/tools/parallel.rs` (only `exec_command` gets the wall time). A script yields
+after `DEFAULT_EXEC_YIELD_TIME_MS` (10 s); about 8% of the scripts in recorded 0.144 to
+0.147 threads yielded.
+
+A spawn with `fork_turns` copies the parent's history, tool calls included, into the head
+of the child's rollout. In a forked rollout (`session_meta` carries `forked_from_id`),
+parsing starts at the first `thread_settings_applied` with the child's thread id, so the
+parent's commands are not credited to the child. A non-forked rollout is parsed whole:
+that event also appears mid-thread on a settings change. A fork without that event is
+also parsed whole, with a warning, so a format change that would credit the parent's
+commands to the child shows in the logs.
+
 The thread-cumulative baseline is the other half. `ThreadTokenUsage.total` counts the whole
 THREAD, and the Codex thread is created once per task and reused, so by turn N it still
 carries turns 1..N-1. The orchestrator sums per-turn usages, so handing it the cumulative

{coder_eval-0.12.14 → coder_eval-0.12.16}/CHANGELOG.md

@@ -2,6 +2,59 @@
 
 <!-- version list -->
 
+## v0.12.16 (2026-10-21)
+
+### Bug Fixes
+
+- **codex**: Crash a turn codex ended as failed instead of grading it
+  ([#230](https://github.com/UiPath/coder_eval/pull/230),
+  [`30c62ce`](https://github.com/UiPath/coder_eval/commit/30c62ce212efd1bf517332497422327cf34bf7a7))
+
+
+## v0.12.15 (2026-10-21)
+
+### Bug Fixes
+
+- **codex**: Match a multi-line command record to its exec script
+  ([#227](https://github.com/UiPath/coder_eval/pull/227),
+  [`b3242f9`](https://github.com/UiPath/coder_eval/commit/b3242f97540ac743c8c159b479a77bcac05809a7))
+
+- **codex**: Read a sub-agent's commands from its rollout records
+  ([#227](https://github.com/UiPath/coder_eval/pull/227),
+  [`b3242f9`](https://github.com/UiPath/coder_eval/commit/b3242f97540ac743c8c159b479a77bcac05809a7))
+
+- **codex**: Recover v2 sub-agent tool calls ([#227](https://github.com/UiPath/coder_eval/pull/227),
+  [`b3242f9`](https://github.com/UiPath/coder_eval/commit/b3242f97540ac743c8c159b479a77bcac05809a7))
+
+- **deps**: Bump aiohttp 3.14.3 -> 3.14.4 for CVE-2026-111063 and CVE-2026-111064
+  ([#227](https://github.com/UiPath/coder_eval/pull/227),
+  [`b3242f9`](https://github.com/UiPath/coder_eval/commit/b3242f97540ac743c8c159b479a77bcac05809a7))
+
+- **deps**: Bump certifi and virtualenv to clear pip-audit advisories
+  ([#232](https://github.com/UiPath/coder_eval/pull/232),
+  [`7628641`](https://github.com/UiPath/coder_eval/commit/7628641e42a743dc85dc8299178ea87297bb07e2))
+
+- **deps**: Bump mcp 1.28.1 -> 1.30.0 for CVE-2026-59951
+  ([#228](https://github.com/UiPath/coder_eval/pull/228),
+  [`ea0e436`](https://github.com/UiPath/coder_eval/commit/ea0e436ec736f5bece0ed460f7f4c854de106d96))
+
+- **deps**: Bump next to 15.5.28 (Next.js security update, Oct 14 2026)
+  ([#226](https://github.com/UiPath/coder_eval/pull/226),
+  [`872f857`](https://github.com/UiPath/coder_eval/commit/872f85764ad4f4d07c9b0659ee17941df0a3b9c9))
+
+- **deps**: Bump next to patched release (Next.js security update, Oct 14 2026)
+  ([#226](https://github.com/UiPath/coder_eval/pull/226),
+  [`872f857`](https://github.com/UiPath/coder_eval/commit/872f85764ad4f4d07c9b0659ee17941df0a3b9c9))
+
+- **deps**: Pin evalboard sharp to 0.35.5 and clear prod audit findings
+  ([#226](https://github.com/UiPath/coder_eval/pull/226),
+  [`872f857`](https://github.com/UiPath/coder_eval/commit/872f85764ad4f4d07c9b0659ee17941df0a3b9c9))
+
+- **docker**: Pin uv in the agent image — 0.12.11+ copies cross-fs ~40x slower
+  ([#209](https://github.com/UiPath/coder_eval/pull/209),
+  [`634005d`](https://github.com/UiPath/coder_eval/commit/634005d2f0ee1848d96d08f080e79cf2bd43d56e))
+
+
 ## v0.12.14 (2026-10-15)
 
 ### Bug Fixes

{coder_eval-0.12.14 → coder_eval-0.12.16}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: coder-eval
-Version: 0.12.14
+Version: 0.12.16
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode, Pi, Delegate) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval
@@ -49,7 +49,7 @@ Requires-Dist: openai-codex>=0.155.1; extra == 'codex'
 Provides-Extra: dev
 Requires-Dist: bandit[toml]>=1.9.4; extra == 'dev'
 Requires-Dist: defusedxml>=0.7.1; extra == 'dev'
-Requires-Dist: mcp>=1.28.1; extra == 'dev'
+Requires-Dist: mcp>=1.30.0; extra == 'dev'
 Requires-Dist: pip-audit>=2.10.0; extra == 'dev'
 Requires-Dist: pre-commit>=4.5.1; extra == 'dev'
 Requires-Dist: pyright>=1.1.408; extra == 'dev'

{coder_eval-0.12.14 → coder_eval-0.12.16}/action.yml

@@ -25,7 +25,7 @@ inputs:
   version:
     description: coder-eval version to install from PyPI, or "local" to install from the action checkout
     required: false
-    default: "0.12.14"   # <-- kept in sync with releases by release.yml
+    default: "0.12.16"   # <-- kept in sync with releases by release.yml
   extras:
     description: >-
       Comma-separated coder-eval extras (`codex`, `antigravity,litellm`), composed

{coder_eval-0.12.14 → coder_eval-0.12.16}/docker/Dockerfile

@@ -42,8 +42,18 @@ RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
 ARG PI_VERSION=0.87.1
 RUN npm install -g @earendil-works/pi-coding-agent@${PI_VERSION}
 
-# uv: matches host sandbox.py's `uv venv` + `uv pip install` fast path
-RUN curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR=/usr/local/bin sh
+# uv: matches host sandbox.py's `uv venv` + `uv pip install` fast path.
+# Pinned for the same reason as the CLIs above: agents run `uv` themselves
+# (coded-agent / function tasks build a venv in the workspace), so its speed is
+# part of every such task's wall clock. The workspace is a bind mount and uv's
+# cache is not, so uv cannot hardlink and copies every file instead. From 0.12.11
+# that cross-filesystem copy is ~40x slower (146-package `uipath-langchain`
+# install: 0.3s on 0.12.10, 10-14s on 0.12.11/.12/.19/.21; 60-130s on the CI
+# host), and the unpinned install picked it up silently in the 0.12.1 image
+# (2026-09-12, uv 0.12.12). Bump deliberately, after timing an install into a
+# bind-mounted dir.
+ARG UV_VERSION=0.12.10
+RUN curl -LsSf https://astral.sh/uv/${UV_VERSION}/install.sh | env UV_INSTALL_DIR=/usr/local/bin sh
 
 WORKDIR /opt/coder_eval
 

{coder_eval-0.12.14 → coder_eval-0.12.16}/docs/agents/CODEX.md

@@ -210,6 +210,30 @@ The agent sets up SKILL.md files (Agent Skills open standard) in `.agents/skills
 5. Codex CLI auto-discovers skills in `.agents/skills/`
 6. Removes the copies when the turn ends, so criteria and run artifacts never see them
 
+### Sub-agents
+
+A sub-agent runs on its own Codex thread, and its tool calls never reach the parent's
+stream. After each turn the agent reads the child's rollout file under
+`$CODEX_HOME/sessions` and records the child's calls as telemetry, nested under the
+spawning `Agent` row. Spawns from multi-agent v1 (`spawnAgent`) and v2
+(`subAgentActivity`) are both recovered.
+
+- Each shell command the child ran is one `Bash` row with its own exit code, also when it
+  runs longer than its script. A code-mode script that runs several commands gives one row
+  per command.
+- A non-zero exit is `result_status: error`. A call that Codex reports with no result is
+  `result_status: unknown`: an interrupt stopped it, or it still ran when its tool call
+  returned. `command_executed` with `require_success: true` counts neither. A later `wait`
+  or `write_stdin` call is a row of its own and does not change that status.
+- Each file the child patched is a `Write` row.
+- A code-mode `exec` script that ran no command and changed no file (for example, one
+  that only reads the tool list) is one `exec` row, not a `Bash` row.
+- Headless Codex through 0.147 writes no command records; 0.155.1 (the SDK floor),
+  0.156.1 and 0.162 do. On a CLI without records a code-mode script is one `exec`
+  row, and the commands inside it are not recovered.
+- A multi-agent v2 spawn names no model, so the child's generations are priced at the
+  parent's model.
+
 ### Async Integration
 
 The Codex SDK is synchronous. The agent uses `_run_async()` helper to detect and await coroutines, preserving the async interface.

{coder_eval-0.12.14 → coder_eval-0.12.16}/docs/agents/HARNESS_PARITY.md

@@ -531,6 +531,18 @@ what that cell means: measured, but not placeable. On historical codex runs, thi
 moves ~8 h in aggregate out of Tool exec and into Unaccounted. No built-in agent
 writes this shape now.
 
+## Sub-agent tool calls
+
+Only claude-code and codex nest a sub-agent's work under its spawn row
+(`parent_tool_use_id`); the other harnesses record no sub-agent path.
+
+| | claude-code | codex |
+|---|---|---|
+| spawn row | `Agent`, with the brief in `description` / `prompt` | `Agent` (`operation: spawnagent`), from a v1 `collabAgentToolCall` or a v2 `subAgentActivity` `started` item; on v2, no prompt (the spawn message is encrypted) and no model |
+| the child's generations | nested live, with the child's real model | nested after the turn, rebuilt from the child's rollout under `$CODEX_HOME/sessions`, priced at the parent's model on v2 |
+| the child's shell commands | one `Bash` row per call, status and `duration_ms` measured live | one `Bash` row per command, with its exit code, rebuilt from the child's rollout (CLI 0.156.1 and later); no `duration_ms`. See [Codex § Sub-agents](CODEX.md#sub-agents) |
+| when the rows exist | during the turn, so live criteria see them | after the turn, so live criteria and `stop_early` do not see them |
+
 ## `max_turns` counts model API calls on every harness
 
 One turn is one main-thread model API call, the unit Claude Code's `--max-turns`

{coder_eval-0.12.14 → coder_eval-0.12.16}/evalboard/package.json

@@ -12,7 +12,7 @@
         "verify": "tsc --noEmit && vitest run && next build"
     },
     "dependencies": {
-        "next": "^15.5.20",
+        "next": "^15.5.28",
         "react": "^19.0.0",
         "react-dom": "^19.0.0",
         "react-markdown": "^10.1.0",
@@ -44,7 +44,10 @@
             "postcss": "^8.5.10",
             "fast-xml-builder": "^1.1.7",
             "@babel/core": "^7.29.6",
-            "uuid": "^11.1.1"
+            "uuid": "^11.1.1",
+            "sharp": "^0.35.5",
+            "nanoid": "^3.3.18",
+            "source-map-js": "^1.2.2"
         }
     }
 }