coder-eval 0.12.13__tar.gz → 0.12.15__tar.gz

{coder_eval-0.12.13 → coder_eval-0.12.15}/.claude/notes/agents.md

@@ -373,7 +373,8 @@ Codex runs every sub-agent on its own child thread whose events never reach the
 stream, and that thread persists with *Limited* rollout policy, which drops
 `commandExecution` events. So neither the live stream nor `thread.read` surfaces the
 sub-agent's shell commands, and `thread/tokenUsage/updated` only ever reports the PARENT
-thread.
+thread. (The commands a code-mode script runs are the exception: they persist as
+`item_completed` records, below.)
 
 But the child rollout ALWAYS persists the raw `function_call` / `local_shell_call` /
 `custom_tool_call` ResponseItems and a `token_count` event with the child's cumulative
@@ -395,6 +396,72 @@ The rollout file can lag the parent's `turn/completed` by a beat (the recorder f
 a background task), so the lookup polls ~2 s — but bails immediately when
 `<home>/sessions` does not exist at all, since no flush can ever land there.
 
+Codex multi-agent v2 moved the spawn off the collab item. A v1 spawn is a
+`collabAgentToolCall` that names the child in `receiver_thread_ids`; a v2 spawn is a
+`subAgentActivity` item with `kind: started`, `id` = the spawn call id and
+`agent_thread_id` = the child, and its `wait` names no child at all. `_tool_item_root`
+maps a `started` item onto the v1 spawn shape where items enter the turn state, so one
+path registers both, and registers a child once: a repeated spawn item would otherwise
+mine the rollout twice and double the child's rows and tokens. The other kinds
+(`completed`, `interrupted`, `interacted`) are notices about a spawned child, dropped
+there, so they neither become rows nor open a parent model call.
+
+A child's tool calls come in two shapes. A direct call is a function call, and its output
+header carries the exit code. A code-mode call is a custom tool `exec` whose input is a
+JavaScript script, often several shell calls in a `Promise.all`. Its output tells only
+whether the SCRIPT ran to the end: `tools.exec_command` hands a non-zero exit code back to
+the script and does not throw, so `Script completed` is not a command result. The script
+source is not parsed — a command built from a variable or a loop defeats any parser, and
+the source holds no exit code. Instead, while a script runs, Codex writes an
+`item_completed` record per shell command (`CommandExecution`: argv, exit code, output)
+and per patch (`FileChange`), and those records ARE the script's rows, each with its own
+result. A script that left no record stays one `exec` row, not a `Bash` row: its
+JavaScript is not a shell command. The argv's login-shell wrapper (`/bin/zsh -lc <cmd>`)
+is stripped, so a record's row reads like a direct call's.
+
+A record names no script, so it goes to the newest script whose source holds its
+command. A command can outlive its script: `tools.exec_command` returns at the call's own
+`yield_time_ms` with a session id, the script ends, and a later script polls the command
+with `tools.write_stdin`. The text match then gives the record to the script that started
+the command, so that script is not also left as an `exec` row that repeats the command.
+The match decodes the source's string escapes and then ignores backslashes. A multi-line
+command is a `"a\nb"` literal in the source but holds real newlines in its record; in a
+recorded 0.147 TUI thread, 3 of 10 such commands outlived their script and, undecoded,
+went to a later one. The source also escapes the backslashes and double quotes that the
+command holds; without ignoring backslashes, no record of a real 0.156.1 Windows e2e child
+matched. A record that matches no source goes to the newest script: dropping it
+would lose a command built from a variable, the miss this recovery exists to prevent,
+while a wrong guess only moves the row to another generation. A record whose id is a
+direct call's is dropped, so a direct call is never counted twice. Seen on real CLI
+0.156.1 runs, v1 and v2 children alike: the records sit between the script call and its
+output, before the generation's `token_count`, and a 45-second command was polled as
+described. Headless threads through 0.147 (the 0.144 SDK, 0.147 `codex exec`) write no
+`CommandExecution` records, so there a code-mode script stays one `exec` row and its
+commands are not recoverable. The 0.147 TUI and 0.155.1 (the SDK floor) write them, as do
+0.156.1 and 0.162, on Windows too.
+
+A call with no result is `unknown`, never a success, so `require_success` does not credit
+it: an interrupt cut it off, it still ran after its yield time, or `wait` stopped it. The
+abort output is matched whole, so a command that prints `aborted by user` keeps its exit
+code. The `wait` or `write_stdin` call that later returns the result is a row of its own
+and does not change the yielded call: the records already carry each command's exit code,
+so only a yielded script that ran no command, or a direct `exec_command` that a direct
+`write_stdin` polls (no recorded run has one), stays `unknown`. A resuming call stays a
+row because a generation that only resumed would otherwise hold no block, and the recovery
+drops such a generation with its tokens; so each poll is one row, where the parent stream
+records a polled command once. The abort forms come from `abort_message` in
+`core/src/tools/parallel.rs` (only `exec_command` gets the wall time). A script yields
+after `DEFAULT_EXEC_YIELD_TIME_MS` (10 s); about 8% of the scripts in recorded 0.144 to
+0.147 threads yielded.
+
+A spawn with `fork_turns` copies the parent's history, tool calls included, into the head
+of the child's rollout. In a forked rollout (`session_meta` carries `forked_from_id`),
+parsing starts at the first `thread_settings_applied` with the child's thread id, so the
+parent's commands are not credited to the child. A non-forked rollout is parsed whole:
+that event also appears mid-thread on a settings change. A fork without that event is
+also parsed whole, with a warning, so a format change that would credit the parent's
+commands to the child shows in the logs.
+
 The thread-cumulative baseline is the other half. `ThreadTokenUsage.total` counts the whole
 THREAD, and the Codex thread is created once per task and reused, so by turn N it still
 carries turns 1..N-1. The orchestrator sums per-turn usages, so handing it the cumulative
@@ -717,8 +784,15 @@ being hardcoded, so a plugin that relocates its skills keeps working.
   skips external *plugins*, not configured skill paths. An inherited value is appended to
   rather than clobbered, since the host may legitimately configure OpenCode the same way.
 - **Pi** passes each as `--skill <dir>`.
-- **Codex** symlinks (or copies, on Windows) each skill dir into `.agents/skills/`, which
-  the CLI auto-discovers from the working directory upward.
+- **Codex** copies each skill dir into `.agents/skills/`, which the CLI auto-discovers from
+  the working directory upward. Copies, not symlinks, for two reasons that compound. Codex
+  resolves a symlinked skill to its source, finds the plugin manifest there and lists the
+  skill as `<plugin>:<skill>`; models sometimes turn that into the nonexistent path
+  `.agents/skills/<plugin>/<skill>/SKILL.md`. The search that follows (`rg --files`,
+  `find`) skips symlinks and returns nothing, so the agent works without the skill. The
+  copies are made before each turn and removed after it, the same concern that kept
+  OpenCode out of the sandbox: criteria and run artifacts must see only what the agent
+  left. Skill dirs the workspace already has are never touched.
 - **Antigravity** takes search paths natively via `skills_paths` — but those only drive
   DISCOVERY. The file-tool allowlist is `workspaces` alone, so the skill roots must appear
   there too, or the agent discovers a skill and every read of its `SKILL.md` is denied as

{coder_eval-0.12.13 → coder_eval-0.12.15}/CHANGELOG.md

@@ -2,6 +2,59 @@
 
 <!-- version list -->
 
+## v0.12.15 (2026-10-21)
+
+### Bug Fixes
+
+- **codex**: Match a multi-line command record to its exec script
+  ([#227](https://github.com/UiPath/coder_eval/pull/227),
+  [`b3242f9`](https://github.com/UiPath/coder_eval/commit/b3242f97540ac743c8c159b479a77bcac05809a7))
+
+- **codex**: Read a sub-agent's commands from its rollout records
+  ([#227](https://github.com/UiPath/coder_eval/pull/227),
+  [`b3242f9`](https://github.com/UiPath/coder_eval/commit/b3242f97540ac743c8c159b479a77bcac05809a7))
+
+- **codex**: Recover v2 sub-agent tool calls ([#227](https://github.com/UiPath/coder_eval/pull/227),
+  [`b3242f9`](https://github.com/UiPath/coder_eval/commit/b3242f97540ac743c8c159b479a77bcac05809a7))
+
+- **deps**: Bump aiohttp 3.14.3 -> 3.14.4 for CVE-2026-111063 and CVE-2026-111064
+  ([#227](https://github.com/UiPath/coder_eval/pull/227),
+  [`b3242f9`](https://github.com/UiPath/coder_eval/commit/b3242f97540ac743c8c159b479a77bcac05809a7))
+
+- **deps**: Bump certifi and virtualenv to clear pip-audit advisories
+  ([#232](https://github.com/UiPath/coder_eval/pull/232),
+  [`7628641`](https://github.com/UiPath/coder_eval/commit/7628641e42a743dc85dc8299178ea87297bb07e2))
+
+- **deps**: Bump mcp 1.28.1 -> 1.30.0 for CVE-2026-59951
+  ([#228](https://github.com/UiPath/coder_eval/pull/228),
+  [`ea0e436`](https://github.com/UiPath/coder_eval/commit/ea0e436ec736f5bece0ed460f7f4c854de106d96))
+
+- **deps**: Bump next to 15.5.28 (Next.js security update, Oct 14 2026)
+  ([#226](https://github.com/UiPath/coder_eval/pull/226),
+  [`872f857`](https://github.com/UiPath/coder_eval/commit/872f85764ad4f4d07c9b0659ee17941df0a3b9c9))
+
+- **deps**: Bump next to patched release (Next.js security update, Oct 14 2026)
+  ([#226](https://github.com/UiPath/coder_eval/pull/226),
+  [`872f857`](https://github.com/UiPath/coder_eval/commit/872f85764ad4f4d07c9b0659ee17941df0a3b9c9))
+
+- **deps**: Pin evalboard sharp to 0.35.5 and clear prod audit findings
+  ([#226](https://github.com/UiPath/coder_eval/pull/226),
+  [`872f857`](https://github.com/UiPath/coder_eval/commit/872f85764ad4f4d07c9b0659ee17941df0a3b9c9))
+
+- **docker**: Pin uv in the agent image — 0.12.11+ copies cross-fs ~40x slower
+  ([#209](https://github.com/UiPath/coder_eval/pull/209),
+  [`634005d`](https://github.com/UiPath/coder_eval/commit/634005d2f0ee1848d96d08f080e79cf2bd43d56e))
+
+
+## v0.12.14 (2026-10-15)
+
+### Bug Fixes
+
+- **codex**: Copy skills into .agents/skills instead of symlinking
+  ([#225](https://github.com/UiPath/coder_eval/pull/225),
+  [`1895a2c`](https://github.com/UiPath/coder_eval/commit/1895a2cfb0c2f0d8a171b2e445b0b034440209b2))
+
+
 ## v0.12.13 (2026-10-13)
 
 ### Bug Fixes

{coder_eval-0.12.13 → coder_eval-0.12.15}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: coder-eval
-Version: 0.12.13
+Version: 0.12.15
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode, Pi, Delegate) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval
@@ -49,7 +49,7 @@ Requires-Dist: openai-codex>=0.155.1; extra == 'codex'
 Provides-Extra: dev
 Requires-Dist: bandit[toml]>=1.9.4; extra == 'dev'
 Requires-Dist: defusedxml>=0.7.1; extra == 'dev'
-Requires-Dist: mcp>=1.28.1; extra == 'dev'
+Requires-Dist: mcp>=1.30.0; extra == 'dev'
 Requires-Dist: pip-audit>=2.10.0; extra == 'dev'
 Requires-Dist: pre-commit>=4.5.1; extra == 'dev'
 Requires-Dist: pyright>=1.1.408; extra == 'dev'

{coder_eval-0.12.13 → coder_eval-0.12.15}/action.yml

@@ -25,7 +25,7 @@ inputs:
   version:
     description: coder-eval version to install from PyPI, or "local" to install from the action checkout
     required: false
-    default: "0.12.13"   # <-- kept in sync with releases by release.yml
+    default: "0.12.15"   # <-- kept in sync with releases by release.yml
   extras:
     description: >-
       Comma-separated coder-eval extras (`codex`, `antigravity,litellm`), composed

{coder_eval-0.12.13 → coder_eval-0.12.15}/docker/Dockerfile

@@ -42,8 +42,18 @@ RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \
 ARG PI_VERSION=0.87.1
 RUN npm install -g @earendil-works/pi-coding-agent@${PI_VERSION}
 
-# uv: matches host sandbox.py's `uv venv` + `uv pip install` fast path
-RUN curl -LsSf https://astral.sh/uv/install.sh | env UV_INSTALL_DIR=/usr/local/bin sh
+# uv: matches host sandbox.py's `uv venv` + `uv pip install` fast path.
+# Pinned for the same reason as the CLIs above: agents run `uv` themselves
+# (coded-agent / function tasks build a venv in the workspace), so its speed is
+# part of every such task's wall clock. The workspace is a bind mount and uv's
+# cache is not, so uv cannot hardlink and copies every file instead. From 0.12.11
+# that cross-filesystem copy is ~40x slower (146-package `uipath-langchain`
+# install: 0.3s on 0.12.10, 10-14s on 0.12.11/.12/.19/.21; 60-130s on the CI
+# host), and the unpinned install picked it up silently in the 0.12.1 image
+# (2026-09-12, uv 0.12.12). Bump deliberately, after timing an install into a
+# bind-mounted dir.
+ARG UV_VERSION=0.12.10
+RUN curl -LsSf https://astral.sh/uv/${UV_VERSION}/install.sh | env UV_INSTALL_DIR=/usr/local/bin sh
 
 WORKDIR /opt/coder_eval
 

{coder_eval-0.12.13 → coder_eval-0.12.15}/docs/TASK_DEFINITION_GUIDE.md

@@ -1345,7 +1345,7 @@ Like `skill_triggered`, this criterion emits a `ClassificationCriterionResult`,
 
 ### `skill_triggered`
 
-Binary classifier: **did the agent engage the target skill during the run?** Agent-agnostic — scans the run's `turn_records` for either signal: Claude's explicit `Skill` tool call whose `skill` parameter matches `skill_name` (namespace prefixes like `plugin:skill` are stripped, so `skill_name: uipath-agents` matches `Skill(skill="uipath-coded-agents:uipath-agents")`), or — for an agent with no `Skill` tool, e.g. Codex — a command that reads the skill's files off disk (a parameter contains `skills/<skill_name>/`, matching both the repo path and the `.agents/skills/` symlink).
+Binary classifier: **did the agent engage the target skill during the run?** Agent-agnostic — scans the run's `turn_records` for either signal: Claude's explicit `Skill` tool call whose `skill` parameter matches `skill_name` (namespace prefixes like `plugin:skill` are stripped, so `skill_name: uipath-agents` matches `Skill(skill="uipath-coded-agents:uipath-agents")`), or — for an agent with no `Skill` tool, e.g. Codex — a command that reads the skill's files off disk (a parameter contains `skills/<skill_name>/`, matching both the repo path and the `.agents/skills/` copy).
 
 Observed label is `"yes"` when either signal is found, else `"no"`. Expected label is `"yes"` iff `expected_skill == skill_name`. **Binary scoring:** `1.0` when observed matches expected, else `0.0`.
 

{coder_eval-0.12.13 → coder_eval-0.12.15}/docs/agents/ANTIGRAVITY.md

@@ -127,7 +127,7 @@ Antigravity supports [Agent Skills](https://agentskills.io/specification)
 2. the runtime plugin directory passed to `start()`.
 
 For each source, the harness is handed whichever of `<source>/skills` or `<source>`
-directly contains subdirectories with a `SKILL.md`. Unlike Codex — which symlinks
+directly contains subdirectories with a `SKILL.md`. Unlike Codex — which copies
 skills into `.agents/skills/` — Antigravity is given the search-path roots directly
 via the SDK's `skills_paths`, and those roots are also added to the harness
 `workspaces` allowlist (otherwise the agent would find a skill but be denied the

{coder_eval-0.12.13 → coder_eval-0.12.15}/docs/agents/CODEX.md

@@ -115,7 +115,7 @@ CodexAgent supports SKILL.md files following the [Agent Skills open standard](ht
 1. **config.plugins** - Local plugins with `type: local` and `path` pointing to a skills directory
 2. **plugin_tools_dir** parameter - Runtime plugin directory passed to `start()`
 
-Skills are symlinked (or copied) to `.agents/skills/` where the Codex CLI auto-discovers them. Environment variables in plugin paths (`$VAR`, `${VAR}`) are expanded at runtime.
+Skills are copied to `.agents/skills/` for each turn, where the Codex CLI auto-discovers them, and removed when the turn ends. Environment variables in plugin paths (`$VAR`, `${VAR}`) are expanded at runtime.
 
 Example with environment variable:
 ```yaml
@@ -206,8 +206,33 @@ The agent sets up SKILL.md files (Agent Skills open standard) in `.agents/skills
 1. Scans `config.plugins` for local plugins with `path` field
 2. Checks `plugin_tools_dir` parameter passed to `start()`
 3. Expands environment variables in paths (`$PLUGIN_PATH`, `${PLUGIN_PATH}`)
-4. Symlinks skill directories (falls back to copying if symlink fails)
+4. Copies each skill directory into `.agents/skills/` before every turn (real files, not symlinks), skipping any skill dir the workspace already has
 5. Codex CLI auto-discovers skills in `.agents/skills/`
+6. Removes the copies when the turn ends, so criteria and run artifacts never see them
+
+### Sub-agents
+
+A sub-agent runs on its own Codex thread, and its tool calls never reach the parent's
+stream. After each turn the agent reads the child's rollout file under
+`$CODEX_HOME/sessions` and records the child's calls as telemetry, nested under the
+spawning `Agent` row. Spawns from multi-agent v1 (`spawnAgent`) and v2
+(`subAgentActivity`) are both recovered.
+
+- Each shell command the child ran is one `Bash` row with its own exit code, also when it
+  runs longer than its script. A code-mode script that runs several commands gives one row
+  per command.
+- A non-zero exit is `result_status: error`. A call that Codex reports with no result is
+  `result_status: unknown`: an interrupt stopped it, or it still ran when its tool call
+  returned. `command_executed` with `require_success: true` counts neither. A later `wait`
+  or `write_stdin` call is a row of its own and does not change that status.
+- Each file the child patched is a `Write` row.
+- A code-mode `exec` script that ran no command and changed no file (for example, one
+  that only reads the tool list) is one `exec` row, not a `Bash` row.
+- Headless Codex through 0.147 writes no command records; 0.155.1 (the SDK floor),
+  0.156.1 and 0.162 do. On a CLI without records a code-mode script is one `exec`
+  row, and the commands inside it are not recovered.
+- A multi-agent v2 spawn names no model, so the child's generations are priced at the
+  parent's model.
 
 ### Async Integration
 

{coder_eval-0.12.13 → coder_eval-0.12.15}/docs/agents/HARNESS_PARITY.md

@@ -531,6 +531,18 @@ what that cell means: measured, but not placeable. On historical codex runs, thi
 moves ~8 h in aggregate out of Tool exec and into Unaccounted. No built-in agent
 writes this shape now.
 
+## Sub-agent tool calls
+
+Only claude-code and codex nest a sub-agent's work under its spawn row
+(`parent_tool_use_id`); the other harnesses record no sub-agent path.
+
+| | claude-code | codex |
+|---|---|---|
+| spawn row | `Agent`, with the brief in `description` / `prompt` | `Agent` (`operation: spawnagent`), from a v1 `collabAgentToolCall` or a v2 `subAgentActivity` `started` item; on v2, no prompt (the spawn message is encrypted) and no model |
+| the child's generations | nested live, with the child's real model | nested after the turn, rebuilt from the child's rollout under `$CODEX_HOME/sessions`, priced at the parent's model on v2 |
+| the child's shell commands | one `Bash` row per call, status and `duration_ms` measured live | one `Bash` row per command, with its exit code, rebuilt from the child's rollout (CLI 0.156.1 and later); no `duration_ms`. See [Codex § Sub-agents](CODEX.md#sub-agents) |
+| when the rows exist | during the turn, so live criteria see them | after the turn, so live criteria and `stop_early` do not see them |
+
 ## `max_turns` counts model API calls on every harness
 
 One turn is one main-thread model API call, the unit Claude Code's `--max-turns`
@@ -684,7 +696,7 @@ This field breaks it silently.
 claude-code hands the value to the SDK as a *plugin directory*, and a plugin's skills
 live at `<plugin>/skills/<name>/SKILL.md`. Point it at the directory that directly
 parents the skill directories and no skill loads. Codex
-(`codex_agent._setup_skills`) and Antigravity (`antigravity_agent._resolve_skills_paths`)
+(`codex_agent._collect_skills`) and Antigravity (`antigravity_agent._resolve_skills_paths`)
 both scan **both** layouts and take whichever actually holds a `<skill>/SKILL.md`.
 
 † Pi uses the shared `_plugin_skill_dirs` resolver, whose bare-dir fallback resolves a

{coder_eval-0.12.13 → coder_eval-0.12.15}/evalboard/package.json

@@ -12,7 +12,7 @@
         "verify": "tsc --noEmit && vitest run && next build"
     },
     "dependencies": {
-        "next": "^15.5.20",
+        "next": "^15.5.28",
         "react": "^19.0.0",
         "react-dom": "^19.0.0",
         "react-markdown": "^10.1.0",
@@ -44,7 +44,10 @@
             "postcss": "^8.5.10",
             "fast-xml-builder": "^1.1.7",
             "@babel/core": "^7.29.6",
-            "uuid": "^11.1.1"
+            "uuid": "^11.1.1",
+            "sharp": "^0.35.5",
+            "nanoid": "^3.3.18",
+            "source-map-js": "^1.2.2"
         }
     }
 }