coder-eval 0.12.5__tar.gz → 0.12.6__tar.gz

{coder_eval-0.12.5 → coder_eval-0.12.6}/.claude/notes/agents.md

@@ -61,14 +61,12 @@ intentionally brief and out of scope; trimming for DISPLAY belongs in the render
 
 - **Harness run-limit parity**: a shared `BaseAgentConfig` field must mean the same
   thing on every backend, so a divergence is either fixed or documented — never silent.
-  **`run_limits.max_turns` on Codex/Antigravity counts VISIBLE turns** (resolved tool
-  calls, read live off the shared `EventCollector.visible_turn_count`, the same list
-  `TurnRecord.commands` holds) because one `communicate()` is a single SDK turn on both,
-  so a native counter would clamp at 1; claude-code keeps its native SDK cap, whose unit
-  (an agent-loop turn) absorbs arbitrarily many parallel calls — the same number is NOT
-  the same budget across harnesses. OpenCode and Pi each keep a native unit too, because
-  their CLIs stream a real multi-step loop per `communicate()`
-  (`step_start`/`step_finish`, `turn_start`/`turn_end`). The cap is enforced on the same
+  **`run_limits.max_turns` counts main-thread model API calls on every harness**, per
+  iteration (each retry and dialog exchange starts at zero). Codex and Antigravity run one
+  SDK turn per `communicate()`, so each counts calls from its own stream (Codex
+  `thread/tokenUsage/updated`, Antigravity MODEL steps). claude-code keeps the CLI's
+  `--max-turns` plus a backstop that counts main-thread `message_id`s. OpenCode and Pi
+  stream one `step_start` / `turn_start` per call. The cap is enforced on the same
   loop boundary as the cooperative early stop and finalizes cleanly as
   `max_turns_exhausted` (no crash, no retry); on Antigravity that boundary lives in
   `_drain()`, so the background-work poll loop honors it too.

{coder_eval-0.12.5 → coder_eval-0.12.6}/CHANGELOG.md

@@ -2,6 +2,69 @@
 
 <!-- version list -->
 
+## v0.12.6 (2026-09-23)
+
+### Bug Fixes
+
+- **claude-code**: End a turn the CLI did not cap at max_turns
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **claude-code**: Kill the CLI on a cooperative stop
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **claude-code**: Kill the CLI on a max_turns backstop stop without a turn timeout
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **codex**: End a capped turn before the next call can run a tool
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **delegate**: Count a tool-only reply as a model call
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **delegate**: Keep counting model calls when a tool never returns
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **evalboard**: Compare visible turns, not SDK num_turns, against expected_turns
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **pricing**: Add Opus 5.5 and GPT-6, correct rates that moved publicly
+  ([#195](https://github.com/UiPath/coder_eval/pull/195),
+  [`11b902f`](https://github.com/UiPath/coder_eval/commit/11b902fa9bc0d307bcab22801dc56523fcc1d92a))
+
+- **run-limits**: Count max_turns as model API calls on every harness
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **run-limits**: Count max_turns as model API calls on every harness and enforce it on claude-code
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **run-limits**: Stop a turn mid-flight once a token or USD budget is crossed
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **simulation**: End a dialog with agent_max_turns when the agent hits its own cap
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+### Documentation
+
+- **parity**: Record the claude-code max_turns backstop
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+- **run-limits**: Say max_turns restarts per iteration and expected_turns counts tool calls
+  ([#194](https://github.com/UiPath/coder_eval/pull/194),
+  [`17fb10f`](https://github.com/UiPath/coder_eval/commit/17fb10f1250b23df0aa80d33e568996b892711dd))
+
+
 ## v0.12.5 (2026-09-22)
 
 ### Bug Fixes

{coder_eval-0.12.5 → coder_eval-0.12.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: coder-eval
-Version: 0.12.5
+Version: 0.12.6
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode, Pi, Delegate) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval

{coder_eval-0.12.5 → coder_eval-0.12.6}/action.yml

@@ -25,7 +25,7 @@ inputs:
   version:
     description: coder-eval version to install from PyPI, or "local" to install from the action checkout
     required: false
-    default: "0.12.5"   # <-- kept in sync with releases by release.yml
+    default: "0.12.6"   # <-- kept in sync with releases by release.yml
   extras:
     description: >-
       Comma-separated coder-eval extras (`codex`, `antigravity,litellm`), composed

{coder_eval-0.12.5 → coder_eval-0.12.6}/docs/DIALOG_MODE.md

@@ -121,13 +121,15 @@ After each exchange the driver evaluates the stop conditions **in this order**,
 2. **`stop_on_criteria_pass`** (`criteria_passed`) — every success criterion passes. Requires
    per-turn checking (`check_criteria: every_turn` or `both`); pairing it with the default
    `end_of_dialog` is rejected at load time, since there would be nothing to check against.
-3. **`max_turns`** (`max_turns`) — the hard cap on exchanges. The agent exhausting its *own* inner
-   `max_turns` mid-exchange ends the dialog with the same reason.
+3. **`max_turns`** (`max_turns`) — the hard cap on exchanges.
 4. **`max_total_tokens`** (`budget`) — the dialog-wide budget across simulator **and** agent. The
    dialog ends and the task is **still scored** — unlike
    [`run_limits.max_total_tokens`](TASK_DEFINITION_GUIDE.md#run-limits), which covers the subject
    agent only and aborts.
-5. **`stop_token`** (`stop_token`) — only if none of the above fired is the simulator asked for
+5. **`run_limits.max_turns`** (`agent_max_turns`): the agent used up its own model-call cap inside
+   one exchange. That cap restarts on every exchange, so this reason means one exchange ran long, not
+   that the dialog ran out of exchanges.
+6. **`stop_token`** (`stop_token`) — only if none of the above fired is the simulator asked for
    another message; the sentinel token in *that fresh utterance* ends the dialog. This is the
    workhorse in practice — the simulator decides, in character, that it got what it wanted — but it
    is evaluated **last**, so a turn that trips `max_turns` or the budget never gets the chance to

{coder_eval-0.12.5 → coder_eval-0.12.6}/docs/TASK_DEFINITION_GUIDE.md

@@ -240,7 +240,7 @@ valid and an empty block is legal — every field defaults to "no limit".
 ```yaml
 run_limits:
   # Structural caps
-  max_turns: 20                       # hard cap on agent inner-loop turns per iteration
+  max_turns: 20                       # hard cap on model API calls per iteration
   expected_turns: 8                   # SOFT efficiency budget (visible turns) — never aborts
   task_timeout: 300                   # wall-clock cap for the full run envelope, seconds
   turn_timeout: 300                   # per-communicate() timeout, seconds
@@ -255,8 +255,8 @@ run_limits:
 
 | Field | Default | Constraint | Description |
 |-------|---------|------------|-------------|
-| `max_turns` | *unset* | `> 0` | Hard cap on agent inner-loop turns per iteration. Unset uses the SDK default. |
-| `expected_turns` | *unset* | `>= 1` | **Soft** target for cumulative visible turns. Exceeding it warns and badges the report; it never aborts. See [`expected_turns`](#expected_turns-soft-efficiency-budget). |
+| `max_turns` | *unset* | `> 0` | Hard cap on main-thread model API calls per iteration, Claude Code's turn, counted the same on every harness. The tools the last allowed call asks for still run; the turn ends when the next call begins. Each retry and each dialog exchange starts a fresh count. Unset uses the SDK default. See [HARNESS_PARITY.md](agents/HARNESS_PARITY.md). |
+| `expected_turns` | *unset* | `>= 1` | **Soft** target for visible turns (tool calls plus the final reply) summed over the whole task, a different unit from `max_turns`. Exceeding it warns and badges the report; it never aborts. See [`expected_turns`](#expected_turns-soft-efficiency-budget). |
 | `task_timeout` | *unset* | `>= 30` | Max seconds for the full run envelope, including agent work, grading, and post-run work. |
 | `turn_timeout` | *unset* | `>= 10` | Max seconds for the agent's single `communicate()` iteration. |
 | `max_input_tokens` | *unset* | `>= 1` | Max cumulative input (prompt) tokens. |
@@ -327,8 +327,8 @@ that did: a budgeted task that failed counts as over budget, while tasks with no
 `expected_turns` budget are excluded entirely (success or fail).
 
 The count compared against the budget is **visible turns** — one per tool call
-plus one for the agent's final reply — *not* the SDK's `total_turns` (which
-counts assistant messages and can bundle several tool calls into one).
+plus one for the agent's final reply. It is *not* `total_turns`, which counts
+model API calls (the `max_turns` unit), and one call can batch several tool calls.
 
 Set it to the number of turns a competent agent should need for the task. Pick
 budgets consistently across a suite — the headline % is only comparable when
@@ -1721,7 +1721,7 @@ The simulator runs as a tools-disabled Claude Code agent on its own resolved `Ap
 **Semantics:**
 
 - The task's `initial_prompt` is the user's *opening* message; the simulator picks up from turn 2.
-- `max_turns` is the intra-dialog cap (the worst-case agent call budget per trial). Use `n_trials` for variance sampling.
+- `max_turns` caps exchanges. Each exchange also gets a fresh `run_limits.max_turns` of model API calls, so the worst case per trial is the product of the two. Use `n_trials` for variance sampling.
 - The `reference` solution, if present, is hidden from the simulator (same security posture as for the coding agent).
 - When `n_trials > 1`, each trial becomes its own `ResolvedTask` with its own zero-padded replicate directory (`runs/<ts>/<variant_id>/<task_id>/<NN>/`) and its own `task.json` — the same fan-out mechanism as experiment `repeats`, which `n_trials` takes precedence over when simulation is enabled. Trial-level metadata appears under `simulation.replicate_index` / `simulation.n_trials` on the `EvaluationResult`.
 

{coder_eval-0.12.5 → coder_eval-0.12.6}/docs/agents/ANTIGRAVITY.md

@@ -190,8 +190,9 @@ as every other agent.
 5. **`allowed_tools` / `disallowed_tools` are not read.** The harness runs with its
    full builtin tool set, so an Antigravity run has tools (web search, subagents,
    URL fetch) that the same task file denies on Claude Code and Codex.
-6. **`max_turns` counts visible turns.** One `communicate()` is a single SDK turn here,
-   so the cap counts resolved tool calls instead, enforced on the step loop. See
+6. **`max_turns` is counted by the harness.** One `communicate()` is a single SDK turn
+   here, so the harness counts model API calls itself (a MODEL step at a new
+   `step_index` opens one) and enforces the cap on the step loop. See
    [Run-Limit Parity](HARNESS_PARITY.md).
 7. **Shell commands over ~10s are moved to the background.** The localharness has a
    10-second maximum synchronous wait; past it the command becomes a background task

{coder_eval-0.12.5 → coder_eval-0.12.6}/docs/agents/CODEX.md

@@ -217,7 +217,7 @@ The Codex SDK is synchronous. The agent uses `_run_async()` helper to detect and
 | **Session Resume** | `--resume {session_id}` | Via thread ID |
 | **Permissions** | `permission_mode` + `allowed_tools` | `permission_mode` → sandbox/approval + `allowed_tools`/`disallowed_tools` → thread config |
 | **Tool Enforcement** | Not enforced by Coder Eval wrapper | `enabled_tools` honored; `disabled_tools` NOT enforced by the SDK |
-| **`max_turns`** | Native SDK turn cap (assistant messages) | Visible-turn cap (tool calls), enforced on the notification pump |
+| **`max_turns`** | Model API calls: the CLI's `--max-turns`, plus a harness backstop | Model API calls, counted per `thread/tokenUsage/updated` and enforced on the notification pump |
 | **Early stop** | Supported (cooperative `should_stop`, polled between messages) | Supported — polled after each streamed notification; the in-flight turn is interrupted best-effort |
 
 Run-limit semantics per harness: [Run-Limit Parity](HARNESS_PARITY.md).

{coder_eval-0.12.5 → coder_eval-0.12.6}/docs/agents/HARNESS_PARITY.md

@@ -12,7 +12,7 @@ This page is the contract for what each run limit means per harness, plus the sh
 
 | Limit | claude-code | codex | antigravity | opencode | pi | delegate |
 |---|---|---|---|---|---|---|
-| `run_limits.max_turns` | native SDK cap (agent-loop turns) | visible-turn cap (resolved tool calls) | visible-turn cap (resolved tool calls) | native step cap (the CLI's own agent-loop steps) | native turn cap (the CLI's own `turn_start` agent-loop steps) | message-event cap (forwarded `message`-type SDK events, NOT tool calls or backend round-trips — the host exposes no round-trip boundary) |
+| `run_limits.max_turns` (main-thread model API calls on every harness, see below) | native CLI cap, plus a harness backstop when call N+1 begins | a call runs from its first item to its `thread/tokenUsage/updated`, and one that ran tools opens the next call there | a call runs from its first new MODEL step to the step carrying its usage | one `step_start` step | one `turn_start` turn | a call opens when the previous call's tools have all returned |
 | `run_limits.turn_timeout` | watchdog, SIGKILL on the CLI subprocess | watchdog + cooperative interrupt | watchdog, plus an earlier internal poll deadline at 80% of it (see below) | deadline enforced in-loop and on the final reap; SIGTERM→SIGKILL on the CLI's whole process group | deadline enforced in-loop and on the final reap; SIGTERM→SIGKILL on the CLI's whole process group | deadline checked both between reads and while blocked inside one (`asyncio.wait_for`); force-kills the host subprocess and drops the handle so the next turn respawns |
 | `run_limits.task_timeout` | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic |
 | `run_limits.stop_early` | cooperative `should_stop` | cooperative `should_stop` | cooperative `should_stop` | cooperative `should_stop` (event granularity) | cooperative `should_stop` (event granularity — Pi streams incrementally) | cooperative `should_stop`, polled per forwarded SDK event; the host is abandoned (no interrupt command exists) and a fresh one spawns for the next turn |
@@ -531,8 +531,7 @@ needed to drive it.
   when a generation begins. Nothing in the timing accounting reads it — the
   head and tail are measured from the first and last `AssistantMessage`
   instead, which is uniform across all five — so this is recorded rather than
-  fixed. It is NOT a `max_turns` hazard: `EventCollector.visible_turn_count` is
-  `len(self._commands)`, derived from `ToolEndEvent`, and `_turn_starts` feeds
+  fixed. It is NOT a `max_turns` hazard: no cap reads it, and `_turn_starts` feeds
   only `assistant_turn_count` on the no-`AgentEndEvent` fallback path. The real
   cost of normalizing it is that the event drives the live renderers, so moving
   it changes the turn boundaries users watch during a run.
@@ -540,50 +539,49 @@ needed to drive it.
 All three are deliberately deferred; see `c/time-bugs-audit.md` for the
 measurements.
 
-## `max_turns` counts visible turns on Codex and Antigravity
-
-A "visible turn" is one entry in the run's timeline: one resolved tool call. It is
-the unit `result_metrics.visible_turn_count` reports and the unit that lands in
-`TurnRecord.commands`. Both backends count it live off the shared
-`EventCollector.visible_turn_count`, so one `max_turns` value means one thing on
-both.
-
-They need their own counter because a native one would be meaningless: Codex and
-Antigravity each deliver exactly **one SDK turn per `communicate()` call**, so an
-SDK-level cap would clamp at 1 no matter what the task asked for.
-
-The cap is enforced on the same loop boundary as the cooperative early stop: the
-step or notification that reaches the cap is processed whole, and the next one is
-never pulled. The in-flight turn is then cancelled server-side (best effort) so
-the cap actually stops spend. A run cut this way finalizes cleanly as
-`max_turns_exhausted` — it is not a crash, and it is not retried.
-
-**claude-code keeps its native SDK cap.** That is a real, honored cap, so it is
-left alone rather than reimplemented in a different unit. Its unit is the SDK's own
-agent-loop turn, which absorbs an arbitrary number of *parallel* tool calls, so the
-same number bounds very different amounts of work: under a prompt that encourages
-batching, a cap of N here permits many more than N tool calls, where it buys exactly
-N on the other two.
-
-**OpenCode also keeps a native unit — its stream's own steps.** Unlike Codex and
-Antigravity, `opencode run` executes a real multi-step agent loop per invocation
-and streams it (`step_start` / `step_finish`), so the natural agent-loop unit
-exists and is honored: `max_turns: N` allows N complete steps and cuts the run
-when step N+1 begins, with the completed steps' tokens intact. A step is one
-assistant generation and may carry several tool calls — so, as with claude-code,
-the same number is a looser tool-call budget than on the visible-turn backends.
-
-**Pi keeps a native unit too — its `turn_start` agent-loop steps.** Like OpenCode,
-`pi -p --mode json` runs a real multi-step agent loop per invocation and streams it
-(`turn_start` / `turn_end`), so `max_turns: N` allows N complete turns and cuts the
-run when turn N+1 begins, with the completed turns' tokens intact. Pi streams
-incrementally, so the cut genuinely stops spend mid-run. A Pi turn is one assistant
-generation and may carry several tool calls — the same looser budget as claude-code
-and OpenCode.
-
-**So holding `max_turns` constant across harnesses does not hold the budget
-constant.** If you are A/B-ing across backends and the cap is close to binding, that
-is the number to distrust.
+## `max_turns` counts model API calls on every harness
+
+One turn is one main-thread model API call, the unit Claude Code's `--max-turns`
+counts. `max_turns: N` lets the agent make N calls and still runs the tools the Nth
+call asked for. The turn ends when call N+1 begins, so a reply that finishes within
+N calls completes normally. Sub-agent calls do not count. `TurnRecord.num_turns`
+reports the same count, and a turn the cap ended reads N+1, as the Claude Code CLI
+reports it.
+
+Each harness finds the call boundary in its own stream (the table above):
+
+- **claude-code** applies the cap in the CLI, and the harness reads the CLI's
+  `error_max_turns` stop. The CLI does not apply it on every route, so the harness
+  also counts distinct main-thread `message_id`s and ends the turn itself when call
+  N+1 begins.
+- **Codex** sends `thread/tokenUsage/updated` once per call, after that call's
+  tools finish. An item starts as its tool runs, so waiting for the next call's
+  first item would let one tool of call N+1 act. A call that ran tools therefore
+  opens the next call at its `tokenUsage`, since the results always go back to the
+  model, and the cap fires before call N+1 can run anything.
+- **Antigravity** attaches `usage_metadata` to one step per call, and the next call
+  opens with a MODEL step at a new `step_index`.
+- **OpenCode** and **Pi** stream one `step_start` or `turn_start` per call.
+- **Delegate**'s SDK has no round-trip marker, and a tool-only reply streams only
+  its tool call, with no text before it. So the next call opens when every tool the
+  previous call announced has returned, since those results go back to the model,
+  and the cap fires before call N+1 can run anything. A reply that announces
+  several tools before their results counts once. If a tool never returns, the
+  next call opens at the model's next text or new tool call instead.
+
+Every harness enforces the cap on the same loop boundary as the
+cooperative early stop, then kills or cancels the in-flight turn so the cap stops
+spend. A run cut this way finalizes cleanly as `max_turns_exhausted`. It is not a
+crash, and it is not retried.
+
+One call can carry several parallel tool calls, so `max_turns` bounds model calls,
+not tool calls. A model that batches does more work per turn, on every harness
+alike.
+
+The count is per iteration: each retry and each dialog exchange starts at zero. A
+dialog whose agent hits the cap inside an exchange ends with `stop_reason:
+agent_max_turns`. That is distinct from `max_turns`, the simulator's cap on
+exchanges.
 
 ### What a capped run looks like
 
@@ -595,12 +593,12 @@ The signals a capped run leaves behind, on every backend:
   `MAX_TURNS_EXHAUSTED` (reporting category `failed`, icon `M`). Never `ERROR`,
   and never retried.
 - `max_turns_exhausted: true` on the task record.
-- On Codex and Antigravity, the count of *resolved* tool calls the model itself
-  issued equals the cap. Two things can add a further *recorded* command, and
-  neither means the cap leaked:
-    - A tool call already in flight when the cap fires is force-closed and recorded
-      with `result_status: unknown` rather than dropped, so the trajectory shows what
-      was interrupted.
+- `num_turns` is the cap plus one.
+- The calls under the cap are recorded whole. Two things can add a further
+  *recorded* command, and neither means the cap leaked:
+    - A tool call from call N+1 that the harness saw before it stopped is
+      force-closed and recorded with `result_status: unknown` rather than dropped,
+      so the trajectory shows what was interrupted.
     - On Codex, a sub-agent's inner tool calls are recovered from its rollout after
       the pump stops, so the child's work and its tokens still reach the record. The
       cap bounds what the model was allowed to do, not what the record may explain.

{coder_eval-0.12.5 → coder_eval-0.12.6}/plugins/coder-eval/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "coder-eval",
   "displayName": "Coder Eval",
-  "version": "0.12.5",
+  "version": "0.12.6",
   "description": "Test whether your Claude Code skills actually trigger, and benchmark any coding agent — author, run, and analyze the eval suite that proves it, locally or as a CI gate.",
   "author": { "name": "UiPath", "email": "coder-eval@uipath.com", "url": "https://github.com/UiPath/coder_eval" },
   "homepage": "https://coder-eval.com",

{coder_eval-0.12.5 → coder_eval-0.12.6}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "coder-eval"
-version = "0.12.5"
+version = "0.12.6"
 description = "Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode, Pi, Delegate) with sandboxed, reproducible YAML task suites."
 readme = "README.md"
 license = "Apache-2.0"

{coder_eval-0.12.5 → coder_eval-0.12.6}/src/coder_eval/__init__.py

@@ -1,3 +1,3 @@
 """coder_eval - A framework for evaluating AI coding agents."""
 
-__version__ = "0.12.5"
+__version__ = "0.12.6"

{coder_eval-0.12.5 → coder_eval-0.12.6}/src/coder_eval/agents/antigravity_agent.py

@@ -408,7 +408,7 @@ class AntigravityAgent(Agent[AntigravityAgentConfig]):
                         if state.max_turns_reached():
                             state.max_turns_hit = True
                             self._log.debug(
-                                "max_turns (%s visible turns) reached; ending step loop",
+                                "max_turns (%s API calls) reached; ending step loop",
                                 state.max_turns,
                             )
                             break
@@ -438,9 +438,8 @@ class AntigravityAgent(Agent[AntigravityAgentConfig]):
         conversation is cancelled (best-effort) and the turn finalizes cleanly as
         ``STOPPED_EARLY`` (``crashed=False``).
 
-        ``max_turns`` caps VISIBLE turns — resolved tool calls — enforced in-stream
-        on the same boundary as the cooperative stop: one ``communicate()`` here is
-        a single SDK turn, so a native counter would cap at 1 and mean nothing.
+        ``max_turns`` caps main-thread model API calls, Claude Code's unit,
+        enforced in-stream on the same boundary as the cooperative stop.
         See docs/agents/HARNESS_PARITY.md.
 
         Drives one logical turn: ``conversation.send(prompt)`` then iterate
@@ -723,6 +722,10 @@ class _AntigravityTurnState:
         self.commands: list[CommandTelemetry] = []
         self._output_parts: list[str] = []
         self._assistant_turns = 0
+        # Main-thread API calls begun; one spans its first new MODEL step to its usage.
+        self.api_calls = 0
+        self._in_api_call = False
+        self._seen_steps: set[tuple[str, Any]] = set()
 
         # ToolStart on first sight of an id; ToolEnd at DONE.
         self._next_seq = 0
@@ -754,14 +757,12 @@ class _AntigravityTurnState:
         return self.stopped_early_hit or self.max_turns_hit
 
     def max_turns_reached(self) -> bool:
-        """True once this turn has produced ``max_turns`` visible turns.
+        """True once the model begins API call ``max_turns + 1``, the unit Claude Code's ``--max-turns`` caps.
 
-        Delegates to ``EventCollector.visible_turn_count``, the single
-        agent-agnostic capture path, so one ``max_turns`` means the same thing here
-        and on Codex. It counts RESOLVED tool calls, so the call that reaches the
-        cap keeps its result instead of being force-closed as unresolved.
+        The next call opens only after the previous call's tools finish, so every
+        call under the cap keeps its tool results.
         """
-        return self.max_turns is not None and self.collector.visible_turn_count >= self.max_turns
+        return self.max_turns is not None and self.api_calls > self.max_turns
 
     def _seed_first_generation_window(self, source: Any) -> None:
         """Move the first window's mark to the first observed MODEL output.
@@ -789,6 +790,11 @@ class _AntigravityTurnState:
         sstatus = _enum_value(step.status)
         ssource = _enum_value(step.source)
         self._seed_first_generation_window(ssource)
+        step_key = (getattr(step, "trajectory_id", "") or "", step.step_index)
+        if ssource == _SOURCE_MODEL and step_key not in self._seen_steps and not self._in_api_call:
+            self._in_api_call = True
+            self.api_calls += 1
+        self._seen_steps.add(step_key)
         starget = _enum_value(step.target)
         done = sstatus in (_STATUS_DONE, _STATUS_ERROR)
 
@@ -810,6 +816,9 @@ class _AntigravityTurnState:
 
         # Per-generation usage: fold into the turn total and cut an AssistantMessage.
         if step.usage_metadata is not None:
+            if not self._in_api_call:
+                self.api_calls += 1
+            self._in_api_call = False
             gen = _to_token_usage(step.usage_metadata, self.model)
             self.total_usage = self.total_usage + gen
             self._flush_generation(gen, getattr(step.usage_metadata, "thoughts_token_count", 0) or 0)
@@ -1013,7 +1022,7 @@ class _AntigravityTurnState:
                 model_used=self.model,
                 assistant_turn_count=self._assistant_turns,
                 messages=self.messages,
-                num_turns=self._assistant_turns,
+                num_turns=self.api_calls,
                 crashed=crashed,
                 crash_reason=crash_reason,
                 max_turns_exhausted=status is AgentEndStatus.MAX_TURNS_EXHAUSTED,

{coder_eval-0.12.5 → coder_eval-0.12.6}/src/coder_eval/agents/claude_code_agent.py

@@ -223,6 +223,8 @@ class _ClaudeTurnState:
         # Set True by the in-loop cooperative-stop break (early-stop-on-criterion).
         # Distinct from timeout_hit: a clean, non-crash stop that must NOT raise.
         self.stopped_early_hit = False
+        self.max_turns_hit = False
+        self.main_turn_ids: set[str] = set()
         # Resolved by _build_claude_query, set on the state before any finalize
         # path. Stays None if we crash before setup (finalize reads it for cost
         # backfill).
@@ -410,6 +412,8 @@ class _ClaudeTurnState:
         if isinstance(message_id, str):
             self.seen_message_ids.add(message_id)
             self.last_message_had_id = True
+            if not isinstance(parent_tool_use_id, str):
+                self.main_turn_ids.add(message_id)
         else:
             self.last_message_had_id = False
 
@@ -635,6 +639,7 @@ class _ClaudeTurnState:
         max_turns_exhausted = not crashed and (
             self._agent._is_max_turns_result(self.sdk_result_summary)
             or (self.max_turns is not None and self.num_turns is not None and self.num_turns > self.max_turns)
+            or self.max_turns_hit
         )
         if max_turns_exhausted and status == AgentEndStatus.COMPLETED:
             status = AgentEndStatus.MAX_TURNS_EXHAUSTED
@@ -982,7 +987,7 @@ class ClaudeCodeAgent(Agent[ClaudeCodeAgentConfig]):
 
         try:
             options, transport, effective_model = self._build_claude_query(
-                user_input, timeout, max_turns, capture_stderr
+                user_input, timeout, max_turns, capture_stderr, can_stop=should_stop is not None
             )
             # Set on the state BEFORE the AgentStart emit and any finalize path
             # (finalize reads it for cost backfill); stays None if setup crashed.
@@ -1116,12 +1121,13 @@ class ClaudeCodeAgent(Agent[ClaudeCodeAgentConfig]):
         ruff's statement cap. ``query`` is still resolved as a module global at
         call time, so ``patch("...claude_code_agent.query", ...)`` mocks work.
 
-        Two break conditions, and the ORDER MATTERS:
+        Three break conditions, and the ORDER MATTERS:
 
         - The wall-clock guard runs at the TOP, so an over-deadline message is
           DISCARDED — no append, no events. Do NOT move it to a post-loop check.
-        - The cooperative stop runs AFTER ``state.dispatch(message)``, so a watcher
-          can flip its flag on THIS message and the next is never pulled.
+        - The max_turns backstop and the cooperative stop run AFTER
+          ``state.dispatch(message)``, so the message that trips them is recorded
+          and the next is never pulled.
         """
         async for message in query(**query_kwargs):
             if deadline is not None and time.monotonic() > deadline:
@@ -1129,9 +1135,19 @@ class ClaudeCodeAgent(Agent[ClaudeCodeAgentConfig]):
                 self._log.warning("Turn timeout reached mid-stream; breaking out of message loop")
                 break
             state.dispatch(message)
+            # The CLI stops after max_turns main-thread API calls; one more means it ignored the cap.
+            if state.max_turns is not None and len(state.main_turn_ids) > state.max_turns:
+                state.max_turns_hit = True
+                state.num_turns = len(state.main_turn_ids)
+                self._log.warning(
+                    "CLI began API call %d past max_turns=%d; ending the turn", state.num_turns, state.max_turns
+                )
+                self._kill_transport(self._active_transport)
+                break
             if should_stop is not None and should_stop():
                 state.stopped_early_hit = True
                 self._log.debug("Cooperative stop requested; ending message loop at this boundary")
+                self._kill_transport(self._active_transport)
                 break
 
     def _build_claude_query(
@@ -1140,11 +1156,14 @@ class ClaudeCodeAgent(Agent[ClaudeCodeAgentConfig]):
         timeout: float | None,
         max_turns: int | None,
         stderr_callback: Callable[[str], None],
+        *,
+        can_stop: bool = False,
     ) -> tuple[ClaudeAgentOptions, SubprocessCLITransport | None, str | None]:
-        """Build the SDK options (+ a timeout-only transport) for one turn.
+        """Build the SDK options (+ a killable transport) for one turn.
 
-        ``transport`` is None unless a ``timeout`` is set: it is pre-constructed
-        only so the watchdog can hard-kill the subprocess. ``effective_model`` may
+        ``transport`` is None unless a timeout, a turn cap or a cooperative stop
+        (``can_stop``) can end the turn: it is pre-constructed only so the harness
+        can hard-kill the subprocess. ``effective_model`` may
         be None on a DirectRoute with no configured model. ``stderr_callback`` is
         wired in here but owned by ``communicate``.
         """
@@ -1211,11 +1230,12 @@ class ClaudeCodeAgent(Agent[ClaudeCodeAgentConfig]):
         # For later inspection: captures every field, defaults included.
         self._sdk_options_dump = dump_dataclass(options)
 
-        # Pre-constructed only under a timeout, to retain the subprocess handle
-        # for hard-kill. None otherwise, so the SDK uses its own default and tests
-        # can mock query() without a real CLI.
+        # Pre-constructed only when the harness may end the turn, to retain the
+        # subprocess handle for hard-kill. None otherwise, so the SDK uses its own default and
+        # tests can mock query() without a real CLI. Closing the SDK's query()
+        # stream does not end the CLI: it never closes the generator it wraps.
         transport: SubprocessCLITransport | None = None
-        if timeout is not None:
+        if timeout is not None or max_turns is not None or can_stop:
             transport = SubprocessCLITransport(prompt=user_input, options=options)
 
         return options, transport, effective_model