coder-eval 0.12.12__tar.gz → 0.12.13__tar.gz

{coder_eval-0.12.12 → coder_eval-0.12.13}/.claude/notes/README.md

@@ -13,6 +13,7 @@ rule docstrings in `tests/lint/rules/`, then the guides under `docs/`.
 ## Contents
 
 - [agents.md](agents.md) — agent adapters, the turn lifecycle, token reconciliation, harness parity
+- [context-window.md](context-window.md) — the `agent.context_window` cap, per harness
 - [contracts.md](contracts.md) — criteria, datasets, aggregation, judging
 - [isolation.md](isolation.md) — the docker driver, the sandbox, detached grading
 - [lint-rules.md](lint-rules.md) — why each CE lint rule exists

{coder_eval-0.12.12 → coder_eval-0.12.13}/.claude/notes/agents.md

@@ -306,6 +306,15 @@ cost bills it at the input rate and the cache buckets separately.
   id), and the `usage` snapshot the last resort. Cost is backfilled from the rate card
   when a turn was timed out or killed, so there is no terminal `ResultMessage` — the
   tokens are already captured, so this is pure pricing.
+  `model_usage` and `total_cost_usd` count the whole SESSION: a CLI resumed with
+  `resume=` restores the running totals it saved on exit, so a dialog's turn N reports
+  turns 1..N (`usage` and `num_turns` stay per-turn). `_turn_usage_slice` subtracts what
+  the same session last reported, from any ResultMessage, error or not, since the CLI
+  saves on every exit. Summing unsliced snapshots shows as a reconciliation row equal to
+  the previous turn's total (200/200 multi-turn tasks of a 2026-10-07 Haiku 5.5 run),
+  and inflates the simulator's tokens the same way. A counter that moved
+  backwards means no restore, and the snapshot is booked whole. A killed turn reports
+  nothing, so if its CLI saved totals, the next turn absorbs its usage.
 - **Codex/OpenAI** report `input_tokens` INCLUSIVE of the cached prefix, and bill no
   separate cache-write fee. So the fresh slice is `input - cached`, `cache_creation` is 0,
   and `cache_read` is `cached`.
@@ -846,12 +855,29 @@ shared `agents/_skills.py` resolver: the SDK wants ONE parent directory, not a l
 ran 7 steps). The adapter counts calls from the event stream and abandons the host when call N+1
 opens; `len(turnUsages)` from the `result` then replaces the estimate as `num_turns`.
 
+**A call opens on its first text, or on its `usage` frame when it streams none — never on a tool
+event.** The host streams a call's text, flushes the call's `usage` frame, then runs the call's
+tools, and it runs a batched reply's tools one at a time: `tool_call`, `tool_result`, `tool_call`,
+`tool_result`. The adapter once opened call N+1 when every open tool had returned. With that order
+the open set empties after each tool, so every tool counted as a call: a downstream suite saw a turn
+with 16 round-trips and 96 tool calls cut at `max_turns: 80`. Now a `thinking` or `message` event
+opens a call unless the open call has already streamed text, and a `usage` frame opens one only for
+a call that streamed none. A crossing call that streams text is cut at that text, before any of its
+tools. A tool-only crossing call is cut at its own frame, and its first tool can start before the kill:
+the host starts a call's tools right after the frame, and the stream gives no earlier boundary. A host
+without `usage` frames counts only the calls that stream text; such a host is below the 1.203.0 floor,
+which is also the first release that takes the `auth` init option.
+Only `_TurnState.on_text` / `on_tool_event` / `on_usage_frame` / `on_result` write the count, so the rule
+lives in one place. `on_result` compares the live count with `len(turnUsages)` and warns when they
+differ: the order is read from the 1.203.1 host and SDK source, not from a live transcript, so a host
+release that reorders the frames shows up in the logs instead of silently moving the cap.
+
 **A cut turn keeps the usage of its finished calls.** A turn cut by `max_turns` or a stop gets no
 `result`, so the adapter sums the per-round-trip `usage` frames; the `result`'s total replaces the sum
-when it arrives. Ordering (from SDK/backend source, not a live transcript): call N's `usage` frame
-always precedes the `tool_result` that opens call N+1. The frame is NOT a call boundary — it arrives
-before its call's tools run — so the cap stays on "the previous call's tools have all returned". The
-adapter warns when a cut turn had finished calls but no usage.
+when it arrives. Ordering (from the host and SDK source, not a live transcript): call N's `usage` frame
+precedes every tool event of call N. A crossing call that streams text is cut at that text, so its usage
+is dropped; a tool-only crossing call is cut at its own frame, so its usage is booked. The adapter warns
+when a cut turn had finished calls but no usage.
 
 **One `AssistantMessage` per `communicate()`.** `isStepStart` and `turnUsages` would allow a
 per-round-trip split; today `isStepStart` only merges streamed deltas. `close_window` opens the one
@@ -948,8 +974,9 @@ introduced or should silently "fix" by picking one number over the other.
 
 ### Delegate agent golden-master coverage
 
-`tests/_fixtures/golden_streams/delegate_fixtures.py` scripts the host's stdout frames for four
-scenarios: a tool call, a `max_turns` cut, an `error`-frame crash and an orphaned tool. The `event`,
+`tests/_fixtures/golden_streams/delegate_fixtures.py` scripts the host's stdout frames for six
+scenarios: a tool call, a `max_turns` cut, an `error`-frame crash, an orphaned tool, a tool-only call
+that crosses the cap, and a batched reply whose tools run one at a time. The `event`,
 `result` and `error` frames mirror the live `1.202.1` transcript. The `usage` frame and the `result`
 fields new in `1.203.0` follow the host source of UiPath/Autopilot#6656 (`src/usage.ts` and
 `handleSend`): every usage object carries all four buckets with `input_tokens` exclusive of cache,

coder_eval-0.12.13/.claude/notes/context-window.md

@@ -0,0 +1,170 @@
+# Capping the agent's context window
+
+Analysis and design for `agent.context_window`: one harness-neutral field that caps the
+context window an agent works within, set per agent config or per experiment variant.
+
+## Problem and evidence
+
+A benchmark of Claude Code on Sonnet 5 (Bedrock, 1M-token window) let runs grow to
+250k-480k tokens of context. Above about 150k tokens each extended-thinking step got
+about 10x slower, so most runs hit the 2-hour task limit. The operator wants to measure
+pass rate and cost at several context caps, for example one variant at 200k and one at
+400k, on the same tasks.
+
+Before this change no harness-neutral way existed:
+
+- **claude-code**: `claude_settings` (`--settings`) could carry `autoCompactWindow`. That
+  works for one harness only, and when `claude_settings` is a file path a variant cannot
+  merge one key into it.
+- **codex**: `CodexAgentConfig` has no pass-through. `CodexAgent._build_thread_options`
+  builds the thread `config` dict itself, so no value reached Codex.
+- **antigravity, opencode, pi, delegate**: no knob is wired.
+
+## How each harness controls its window
+
+### claude-code
+
+Verified in the CLI the SDK spawns. `claude-agent-sdk` runs its **bundled** CLI before it
+looks for `claude` on `PATH` (`SubprocessCLITransport._find_cli`), so the CLI that runs is
+the one the SDK wheel bundles, not the one the agent image installs with npm:
+
+| coder_eval | claude-agent-sdk (uv.lock) | bundled CLI | image npm CLI |
+|---|---|---|---|
+| 0.12.1 | 0.2.124 | 2.1.216 | 2.1.177 |
+| 0.12.12 | 0.2.159 | 2.1.281 | 2.1.281 |
+
+`environment_info.claude_code_cli` records `claude -v` from `PATH`, so a 0.12.1 run
+records 2.1.177 while 2.1.216 ran. That is a separate defect and is not fixed here.
+
+Three inputs set the auto-compact window. Both bundled CLIs (2.1.216 and 2.1.281) contain
+the same resolution function. The highest precedence comes first:
+
+1. `CLAUDE_CODE_AUTO_COMPACT_WINDOW` environment variable: a plain token count. It
+   outranks the flag and every settings scope.
+2. `--autocompact <auto|tokens>`: added in 2.1.221 (CLI reference). It is absent from
+   2.1.216 as a parsed option.
+3. `autoCompactWindow` settings key. The settings schema is
+   `int().min(100000).max(1000000).optional().catch(undefined)`, so an out-of-range
+   value is **silently dropped**.
+
+The effective window is `min(model context window, configured value)`. Auto-compaction
+starts when the context approaches that window.
+
+### codex
+
+Verified in the `openai-codex-cli-bin` 0.156.1 binary (the pin) and its source at tag
+`rust-v0.156.1`:
+
+- `model_context_window` ("Size of the context window for the model, in tokens"):
+  `with_config_overrides` sets the model's `context_window` to
+  `min(value, max_context_window)`. Three things follow from it. The default
+  auto-compact limit becomes 90% of it. The hard cap, which forces compaction, becomes
+  `context_window * effective_context_window_percent / 100`. The remaining-window figure
+  the model is shown also changes.
+- `model_auto_compact_token_limit` ("Token usage threshold triggering auto-compaction"):
+  the trigger only, clamped to 90% of the resolved window. It does not change the window.
+- `model_auto_compact_token_limit_scope` (`total` by default, or `body_after_prefix`) and
+  `model_post_turn_compact_threshold_percent` (turn-end compaction, off by default) tune
+  when compaction starts. They are not caps.
+
+The thread `config` dict `thread_start` takes is the same override surface that
+coder_eval already uses for `enabled_tools` and `model_providers`.
+
+### antigravity, opencode, pi, delegate
+
+None of them has a context-window knob wired in coder_eval. They are out of scope, and
+they reject the field.
+
+## Options considered
+
+**Per-harness pass-through** (a Codex `config` pass-through next to `claude_settings`).
+It is flexible, but a variant would have to name a different key per harness, and the
+recorded value would mean something different on each. It also opens a free-form
+pass-through on Codex with no denylist. Rejected.
+
+**A field under `run_limits`.** Run limits are caps that the orchestrator or the agent
+enforces on a run (turns, time, tokens, USD), and the parity table lists them. A context
+window is not a spend cap. It is a harness setting that changes what the agent does, and
+it depends on the agent type, which `run_limits` does not know. The guide also says that
+`agent:` holds no run-time caps. Rejected.
+
+**A field on the agent config.** One name, accepted on a type-less config so that
+experiment defaults and variants can set it, and validated against the concrete type
+when the config resolves. **Chosen.**
+
+Naming: `context_window`, in tokens. It names the quantity the operator caps, matches
+Codex's `model_context_window`, and avoids "auto-compact", which describes a mechanism of
+one harness. `max_context_tokens` was considered, but it reads as a per-request input
+limit, which neither harness enforces.
+
+## Recommended design
+
+`BaseAgentConfig.context_window: int | None` (default `None`, `gt=0`).
+
+- Each config class declares `_context_window_range: ClassVar[tuple[int, int | None] |
+  None]`. `None` (the base default) means that the harness cannot apply the field.
+  `ClaudeCodeAgentConfig` declares `(100_000, 1_000_000)` and `CodexAgentConfig` declares
+  `(32_768, None)`.
+- `check_context_window_supported` (model validator) rejects an unsupported type or an
+  out-of-range value as soon as the type is known. A plugin agent inherits `None`, so it
+  also rejects the field until it opts in.
+- The field uses the default `replace` merge strategy. It is set at any of the five
+  layers, and with `-D agent.context_window=N`.
+
+Per-harness mapping:
+
+- **claude-code**: `CLAUDE_CODE_AUTO_COMPACT_WINDOW=<N>` in the SDK `env`. The
+  environment variable was chosen over the settings key and the flag for four reasons.
+  It has the highest precedence, so host, project and `claude_settings` values cannot
+  override the cap. It works whether `claude_settings` is a dict or a file path. Both
+  bundled CLIs read it, which the flag does not. `extra_args` stays framework-owned.
+  `claude_settings.autoCompactWindow` together with `context_window` is a load error,
+  because the environment variable would silently override it.
+- **codex**: `model_context_window: <N>` in the thread `config`. No trigger key is set,
+  so Codex compacts at its default 90% of the window.
+
+## Validation
+
+- `gt=0` on every config, so a type-less config rejects nonsense as well.
+- claude-code: 100000-1000000, the CLI's documented range. Outside that range the CLI
+  drops a settings value silently, so coder_eval enforces the range itself.
+- codex: 32768 and up. Codex clamps the value to the model's catalog maximum, and the
+  floor rejects a window too small to work in, such as a YAML `yes` coerced to 1.
+- every other type: rejected at load, naming the supported types. A variant that
+  switches `type` to an unsupported harness fails when the experiment resolves.
+
+## Recording
+
+The resolved agent config is persisted as `agent_config` on every task row (`run.json`
+and `task.json`). `agent.context_window` therefore appears per task, and config lineage
+records which layer set it (for example, the variant). The experiment report groups by
+variant, so a cap per variant groups without new report code. On claude-code the
+`sdk_options` dump also shows the environment variable in `env`.
+
+## Open questions
+
+- **A silent clamp to the model's window.** Neither harness reports that the model's
+  window lowered the cap. A cap above the model's window runs at the model's window.
+  Recording the effective window would need a per-model catalog in coder_eval.
+- **Trigger points differ.** Codex compacts at 90% of the window. Claude Code compacts
+  when the context approaches the window, minus an internal buffer. Equal caps therefore
+  do not compact at exactly the same token count. A separate harness-neutral trigger
+  field is possible, but it is not added until a benchmark needs it.
+- **A host `CLAUDE_CODE_AUTO_COMPACT_WINDOW` and uncapped runs.** Decided: the SDK starts
+  the CLI with `os.environ` underneath `options.env`, so when the field is unset the
+  claude-code agent blanks a host value. A run recorded as uncapped is never capped by
+  the host. This changes every claude-code run on a host that sets the variable,
+  including runs that never set `context_window`. Codex has no equivalent: an unset
+  field leaves a host `model_context_window` in `$CODEX_HOME/config.toml` in effect.
+- **The recorded CLI version is wrong.** See the version table above:
+  `environment_info.claude_code_cli` should record the bundled CLI version.
+- **Other harnesses.** Pi can lower a model's window with
+  `providers.<provider>.modelOverrides.<model>.contextWindow` in `models.json`. A mapping
+  was written and then reverted (d1757a60), because no live run backed it and no task
+  uses `type: pi`; it comes back with a live test. OpenCode (`limit.context` and
+  `limit.input` in provider config) needs a `limit.output` coder_eval does not know, and
+  an unpinned CLI. Antigravity's `CompactionConfig(token_threshold=N)` only moves the
+  trigger. Delegate keeps its history in the backend. Each needs its own verification
+  before it opts in.
+- **How often a run compacted is not recorded.** A cap A/B cannot yet tell "cap never
+  reached" from "compacted N times" from the task row alone.

{coder_eval-0.12.12 → coder_eval-0.12.13}/.claude/notes/lint-rules.md

@@ -845,7 +845,7 @@ assertion for each. A defaulted `AgentStartEvent` lands ~365 days before the clo
 first window, so the head blows any sane UPPER bound by the whole offset. A defaulted
 `AgentEndEvent` lands ~365 days BEFORE its own last message, so `decompose_turn` clamps the
 negative to `0.0` — "measured, and instant", which passes an upper bound. Only a strict
-`> 0.0` catches it. It holds on all three clocked harnesses because real work separates a
+`> 0.0` catches it. It holds on every clocked harness because real work separates a
 turn's last flush from its end event. The margin is smallest on antigravity, which holds
 its process across turns: it measures 0.007-0.035 ms there, 7-35 ticks of the 1 us
 resolution that both `datetime` and `time.monotonic()` have on Linux, macOS and Windows.

{coder_eval-0.12.12 → coder_eval-0.12.13}/.claude/notes/timing.md

@@ -16,6 +16,11 @@ removes:
   transition or an NTP step inside a turn lands directly in a generation window — an
   hour-long jump in a millisecond field. Nightly runs start at 04:18 and run for hours,
   so it is reachable rather than theoretical. A monotonic-derived stamp cannot express it.
+- Delegate stamped its one window and its tool spans with raw `datetime.now()` but measured
+  `duration_seconds` with `time.monotonic()`. On Windows the wall clock moved 0.24-0.37 ms
+  further than the monotonic one inside a sub-millisecond turn, so the tail alone came out
+  longer than the whole turn and `test_delegate_golden[a_tool_call]` failed at random. One
+  window has no inter-window drift to correct, but it still shares the turn's duration.
 
 It is an EXTRACTION, not an invention: antigravity already captured this exact pair at
 the top of `communicate` and simply did not use it for later stamps.
@@ -205,7 +210,7 @@ Magnitudes are only real where a scripted clock makes them real, so each case dr
 harness's own reducer with a hand-moved clock and feeds its output through a real
 `EventCollector`, the seam production uses to compute the head and the tail. The module
 reuses the three clock-injection idioms the per-harness suites already use rather than
-inventing a fourth: an injected `TurnClock` stand-in (pi, antigravity, claude-code), a
+inventing a fourth: an injected `TurnClock` stand-in (pi, antigravity, claude-code, delegate), a
 `datetime` subclass patched onto the module (opencode), and scripted SDK epoch-millisecond
 stamps (codex). OpenCode needs a `datetime` SUBCLASS rather than a stub because the
 reducer also calls `datetime.fromtimestamp` through the same module global to convert the

{coder_eval-0.12.12 → coder_eval-0.12.13}/CHANGELOG.md

@@ -2,6 +2,65 @@
 
 <!-- version list -->
 
+## v0.12.13 (2026-10-13)
+
+### Bug Fixes
+
+- **agent**: Address the context_window review
+  ([#219](https://github.com/UiPath/coder_eval/pull/219),
+  [`ae02485`](https://github.com/UiPath/coder_eval/commit/ae02485a7a21944e2ee15a0f72131def5f2e425a))
+
+- **agents**: Count Delegate batched tool calls once under max_turns and stamp turns off one
+  TurnClock ([#222](https://github.com/UiPath/coder_eval/pull/222),
+  [`db2067e`](https://github.com/UiPath/coder_eval/commit/db2067e94e458796c10cfe5f2b0c0383a3c8db03))
+
+- **claude-code**: Book only the turn's own usage on a resumed session
+  ([#221](https://github.com/UiPath/coder_eval/pull/221),
+  [`c9456e5`](https://github.com/UiPath/coder_eval/commit/c9456e58b49174c29f48577507f249eb9e98a94b))
+
+- **delegate**: Count a batched reply's tools as one model call under max_turns
+  ([#222](https://github.com/UiPath/coder_eval/pull/222),
+  [`db2067e`](https://github.com/UiPath/coder_eval/commit/db2067e94e458796c10cfe5f2b0c0383a3c8db03))
+
+- **delegate**: Keep the max_turns count in one place and warn when it drifts
+  ([#222](https://github.com/UiPath/coder_eval/pull/222),
+  [`db2067e`](https://github.com/UiPath/coder_eval/commit/db2067e94e458796c10cfe5f2b0c0383a3c8db03))
+
+- **delegate**: Stamp the whole turn off one TurnClock
+  ([#222](https://github.com/UiPath/coder_eval/pull/222),
+  [`db2067e`](https://github.com/UiPath/coder_eval/commit/db2067e94e458796c10cfe5f2b0c0383a3c8db03))
+
+- **pricing**: Add Haiku 5.5, Sonnet 5.5 and GPT-6.1 Sol, re-verify every row
+  ([#220](https://github.com/UiPath/coder_eval/pull/220),
+  [`b3ae960`](https://github.com/UiPath/coder_eval/commit/b3ae96033d32f3e88bff68a15219e85372762220))
+
+### Chores
+
+- **no-mistakes/document**: Name Delegate among the TurnClock harnesses in timing docs
+  ([#222](https://github.com/UiPath/coder_eval/pull/222),
+  [`db2067e`](https://github.com/UiPath/coder_eval/commit/db2067e94e458796c10cfe5f2b0c0383a3c8db03))
+
+### Documentation
+
+- **delegate**: Count the six Delegate golden-master scenarios in the notes
+  ([#222](https://github.com/UiPath/coder_eval/pull/222),
+  [`db2067e`](https://github.com/UiPath/coder_eval/commit/db2067e94e458796c10cfe5f2b0c0383a3c8db03))
+
+- **delegate**: Document the tool-only cut race and relax the live test count
+  ([#222](https://github.com/UiPath/coder_eval/pull/222),
+  [`db2067e`](https://github.com/UiPath/coder_eval/commit/db2067e94e458796c10cfe5f2b0c0383a3c8db03))
+
+### Features
+
+- **agent**: Cap Pi's context window with agent.context_window
+  ([#219](https://github.com/UiPath/coder_eval/pull/219),
+  [`ae02485`](https://github.com/UiPath/coder_eval/commit/ae02485a7a21944e2ee15a0f72131def5f2e425a))
+
+- **agent**: Cap the agent's context window with agent.context_window
+  ([#219](https://github.com/UiPath/coder_eval/pull/219),
+  [`ae02485`](https://github.com/UiPath/coder_eval/commit/ae02485a7a21944e2ee15a0f72131def5f2e425a))
+
+
 ## v0.12.12 (2026-10-06)
 
 ### Bug Fixes

{coder_eval-0.12.12 → coder_eval-0.12.13}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: coder-eval
-Version: 0.12.12
+Version: 0.12.13
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode, Pi, Delegate) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval

{coder_eval-0.12.12 → coder_eval-0.12.13}/action.yml

@@ -25,7 +25,7 @@ inputs:
   version:
     description: coder-eval version to install from PyPI, or "local" to install from the action checkout
     required: false
-    default: "0.12.12"   # <-- kept in sync with releases by release.yml
+    default: "0.12.13"   # <-- kept in sync with releases by release.yml
   extras:
     description: >-
       Comma-separated coder-eval extras (`codex`, `antigravity,litellm`), composed

{coder_eval-0.12.12 → coder_eval-0.12.13}/docs/AB_EXPERIMENTS.md

@@ -23,6 +23,7 @@ to the orchestrator are involved.
 - [What a Variant Can Override](#what-a-variant-can-override)
 - [Recipe: A/B a Skill](#recipe-ab-a-skill)
 - [Recipe: A/B a Model](#recipe-ab-a-model)
+- [Recipe: A/B a Context Window](#recipe-ab-a-context-window)
 - [Recipe: A/B a Prompt](#recipe-ab-a-prompt)
 - [Recipe: Smoke vs. e2e Flavors (Early Stop)](#recipe-smoke-vs-e2e-flavors-early-stop)
 - [Replicates (Statistical Power)](#replicates-statistical-power)
@@ -222,6 +223,28 @@ variants:
     agent: { model: claude-opus-5 }
 ```
 
+## Recipe: A/B a Context Window
+
+`agent.context_window` caps the context the agent works within, in tokens: the
+harness compacts the conversation before it outgrows the cap. Use it to measure
+pass rate and cost against the cap on a model with a large window. Each task row
+records the cap in `agent_config.context_window`.
+
+```yaml
+experiment_id: context-window
+description: "The same model with a 200k and a 400k context cap"
+
+variants:
+  - variant_id: cap-200k
+    agent: { context_window: 200000 }
+  - variant_id: cap-400k
+    agent: { context_window: 400000 }
+```
+
+An agent type that cannot apply the cap fails the experiment at load.
+[Run-Limit Parity](agents/HARNESS_PARITY.md#the-context-window-cap-per-harness) lists which
+agent types accept it and what the cap does on each.
+
 ## Recipe: A/B a Prompt
 
 Use `prompt_mutations` (transform the task prompt) or `initial_prompt` /

{coder_eval-0.12.12 → coder_eval-0.12.13}/docs/EXTENDING.md

@@ -80,6 +80,22 @@ class MyAgentConfig(BaseAgentConfig):
 The factory `create_agent(kind, config, …)` raises `TypeError` if the passed config
 isn't an instance of the registered `config_class`, so keep them paired.
 
+To accept `agent.context_window`, declare the inclusive range of tokens your harness
+can cap its context window to, and apply `self.config.context_window` when the agent
+starts. A config class that declares nothing rejects the field at load, which is the
+right default: a capped variant must never run uncapped.
+
+```python
+from typing import ClassVar
+
+class MyAgentConfig(BaseAgentConfig):
+    type: Literal["my-agent"] = "my-agent"
+    _context_window_range: ClassVar[tuple[int, int | None] | None] = (32_768, None)
+```
+
+Add your harness to the table in
+[Run-Limit Parity](agents/HARNESS_PARITY.md#the-context-window-cap-per-harness).
+
 ### The `Agent` ABC — implementation checklist
 
 Implement these three abstract methods:

{coder_eval-0.12.12 → coder_eval-0.12.13}/docs/TASK_DEFINITION_GUIDE.md

@@ -170,6 +170,7 @@ agent:
     - "Write"
     - "Bash"
   model: "claude-sonnet-5"            # Optional: specific model
+  context_window: 200000              # Optional: cap the context window, in tokens (claude-code, codex)
   sdk_options:                        # Optional: agent SDK pass-through (keys depend on `type`)
     effort: high                      # claude-code: any non-framework-managed ClaudeAgentOptions field
 ```
@@ -188,6 +189,13 @@ validates its own keys at YAML load:
 Deep-merged across the 5-layer config chain. Override via CLI with the
 repeatable `-D agent.sdk_options.KEY=VALUE`.
 
+**`context_window`** caps the context window the agent works within, in tokens: the
+harness compacts the conversation before it outgrows the cap, and never runs past the
+model's own window. Override via CLI with `-D agent.context_window=400000`. Which agent
+types accept it, in what range, what each does with it and what an unset value means:
+[Run-Limit Parity](agents/HARNESS_PARITY.md#the-context-window-cap-per-harness). An agent
+type that cannot apply it rejects it at load.
+
 **Permission Modes:**
 - `default` — Default permission handling
 - `acceptEdits` — Auto-accept file edits (recommended for evaluations)

{coder_eval-0.12.12 → coder_eval-0.12.13}/docs/agents/CLAUDE_CODE.md

@@ -104,6 +104,7 @@ agent:
 | `system_prompt_file` | `str \| null` | Path (relative to the task YAML) loaded into `system_prompt` at resolution. Works with either `system_prompt_mode`. |
 | `setting_sources` | `list["user"\|"project"\|"local"] \| null` | Which host setting sources the SDK reads. Default resolves to `["project"]`. See [Sandbox isolation](#sandbox-isolation). |
 | `claude_settings` | `str \| dict \| null` | Passed to the SDK `--settings`. A dict is JSON-serialized; a str is a settings file path. Use `permissions.deny` to block tools/paths. |
+| `context_window` | `int \| null` (100000-1000000) | Auto-compact window, in tokens, sent as `CLAUDE_CODE_AUTO_COMPACT_WINDOW`. It outranks `--autocompact` and every settings scope, and is capped to the model's window. Setting `claude_settings.autoCompactWindow` too is a load error. See [Harness parity](HARNESS_PARITY.md#the-context-window-cap-per-harness). |
 | `sdk_options` | `dict` (default `{}`) | Pass-through for `ClaudeAgentOptions` fields Coder Eval doesn't own (e.g. `effort`). Validated at load — an unknown or framework-owned key is a hard error. |
 | `ignore_patterns` | `list[str] \| null` | Gitignore-style overrides for the workspace copy used by judge sub-agents (supports `!` negation). |
 
@@ -212,9 +213,11 @@ resolution for any agent that does not declare `supports_cooperative_stop`. See
 
 Claude Code produces the richest telemetry of the agents:
 
-- **Authoritative billing** comes from the SDK's cumulative `model_usage` on the
+- **Authoritative billing** comes from the SDK's `model_usage` on the
   terminal result message and reconciles to `total_cost_usd` — this already includes
-  sub-agent consumption that the per-message stream under-reports.
+  sub-agent consumption that the per-message stream under-reports. Both count the
+  whole session, so a resumed turn books only what changed since the session's
+  previous result message.
 - **Sub-agent accounting** is derived by grouping `parent_tool_use_id`-tagged
   assistant messages; there is no separate per-sub-agent field. The terminal
   sub-agent generation (delivered as the Agent tool result, never streamed) is

{coder_eval-0.12.12 → coder_eval-0.12.13}/docs/agents/CODEX.md

@@ -192,6 +192,13 @@ The agent maps `permission_mode` to the Codex SDK's `Sandbox`. The approval mode
 
 `allowed_tools` / `disallowed_tools` are normalized (`Bash` → `shell`, `Write`/`Edit` → `apply_patch`, etc.) and passed as `enabled_tools` / `disabled_tools` in the thread `config`. **Note:** the Codex SDK does not currently enforce `disabled_tools`; do not rely on it as a security boundary (the agent logs a warning when it is set).
 
+### Context Window
+
+`agent.context_window` is passed as `model_context_window` in the thread `config`. Codex
+uses it as the model's context window, capped to the model's catalog maximum, and
+auto-compacts at 90% of it. See
+[Harness parity](HARNESS_PARITY.md#the-context-window-cap-per-harness).
+
 ### Skills Discovery
 
 The agent sets up SKILL.md files (Agent Skills open standard) in `.agents/skills/` directory:

{coder_eval-0.12.12 → coder_eval-0.12.13}/docs/agents/DELEGATE.md

@@ -168,8 +168,8 @@ Run-limit semantics per harness: [Run-Limit Parity](HARNESS_PARITY.md).
 ## Known Limitations
 
 1. **No multi-generation transcript splitting.** This agent builds one `AssistantMessage` per `communicate()` call, not one per backend round-trip. The host does send the signals a split needs (`isStepStart` on `message` events and per-round-trip `turnUsages` on `result`), but this agent does not use them yet.
-2. **`max_turns` is enforced by coder_eval, not by the host.** The host's `maxSteps` option on `send` does not stop the turn (confirmed live: `maxSteps: 2` ran 7 steps and only reported `maxStepsReached: true`), so this agent does not send it. See [Run-Limit Parity](HARNESS_PARITY.md).
-   A turn cut at `max_turns`, or by a cooperative early stop, keeps the token usage and cost of every model call that finished before the cut, as on the other agents. The host sends a `usage` frame for each call before that call's tool results. The call in progress at the cut has no usage. A host that does not send `usage` frames reports usage only on its final `result` frame, which a cut turn never gets: that turn has no usage, and the agent logs a warning.
+2. **`max_turns` is enforced by coder_eval, not by the host.** The host's `maxSteps` option on `send` does not stop the turn (confirmed live: `maxSteps: 2` ran 7 steps and only reported `maxStepsReached: true`), so this agent does not send it. It counts a call on the call's first streamed text, or on its `usage` frame when the call streams none. The host runs a batched reply's tools one at a time, so tool events do not mark a call. See [Run-Limit Parity](HARNESS_PARITY.md).
+   A turn cut at `max_turns`, or by a cooperative early stop, keeps the token usage and cost of every model call that finished before the cut, as on the other agents. The host sends a `usage` frame for each call before that call's tool results. Which side of the cut the crossing call lands on depends on its shape. A call that streams text is cut at that text, before its `usage` frame and before any of its tools, so its usage is dropped. A tool-only call is cut at its own `usage` frame, so its usage is booked, and its first tool can start before the kill: the host starts a call's tools right after the frame, and the stream gives no earlier boundary. When the `result` arrives, the agent compares its own call count with the host's `turnUsages` and logs a warning if they differ, because a change in the host's frame order would move the cap. A host that does not send `usage` frames reports usage only on its final `result` frame, which a cut turn never gets: that turn has no usage, and the agent logs a warning.
 3. **Only three backend failures get a specific diagnosis.** A Cloudflare WAF block page is reported as a content-filter failure and is not retried: the same prompt or tool result is blocked again. An SSE connect timeout is reported as a connection failure and is retried. A session conflict ("A reply is already being generated") is retried in a new conversation when no turn of that conversation has finished. After a finished turn, or with a `session_id` set in the config, the retry stays in the same conversation, so the agent never continues without its earlier turns: the retry succeeds once the backend releases the conversation (it stops the reply when the host disconnects), or the task ends as an error. There is no first-response stall detection: a stalled turn ends at its turn timeout. Every other crash ends the turn as an `AgentCrashError`, which is retried unless its message matches a category that is not retried (a timeout, or an auth, billing or content-filter error).
 4. **`sdk_options` accepts only `effort`.** Reasoning effort uses the same `sdk_options.effort` key as Claude Code, so `-D agent.sdk_options.effort=high` works for both agents. Any other key, or an `effort` that is not a string, is a validation error. The host checks the tier itself: it logs a tier it does not recognize and uses the model's default. `project_id`, `session_id` and `enable_computer_use` are typed config fields.
 5. **`enable_computer_use: true` requires local permissions and is unavailable on Linux.** macOS needs Accessibility + Screen Recording grants; the SDK throws unconditionally on Linux when this is enabled.

{coder_eval-0.12.12 → coder_eval-0.12.13}/docs/agents/HARNESS_PARITY.md

@@ -12,7 +12,7 @@ This page is the contract for what each run limit means per harness, plus the sh
 
 | Limit | claude-code | codex | antigravity | opencode | pi | delegate |
 |---|---|---|---|---|---|---|
-| `run_limits.max_turns` (main-thread model API calls on every harness, see below) | native CLI cap, plus a harness backstop when call N+1 begins | a call runs from its first item to its `thread/tokenUsage/updated`, and one that ran tools opens the next call there | a call runs from its first new MODEL step to the step carrying its usage | one `step_start` step | one `turn_start` turn | a call opens when the previous call's tools have all returned |
+| `run_limits.max_turns` (main-thread model API calls on every harness, see below) | native CLI cap, plus a harness backstop when call N+1 begins | a call runs from its first item to its `thread/tokenUsage/updated`, and one that ran tools opens the next call there | a call runs from its first new MODEL step to the step carrying its usage | one `step_start` step | one `turn_start` turn | a call opens on its first streamed text, or on its `usage` frame when it streams none; a tool-only call N+1's first tool can start before the kill |
 | `run_limits.turn_timeout` | watchdog, SIGKILL on the CLI subprocess | watchdog + cooperative interrupt | watchdog, plus an earlier internal poll deadline at 80% of it (see below) | deadline enforced in-loop and on the final reap; SIGTERM→SIGKILL on the CLI's whole process group | deadline enforced in-loop and on the final reap; SIGTERM→SIGKILL on the CLI's whole process group | deadline checked both between reads and while blocked inside one (`asyncio.wait_for`); force-kills the host subprocess and drops the handle so the next turn respawns |
 | `run_limits.task_timeout` | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic | orchestrator-level, agent-agnostic |
 | `run_limits.stop_early` | cooperative `should_stop` | cooperative `should_stop` | cooperative `should_stop` | cooperative `should_stop` (event granularity) | cooperative `should_stop` (event granularity — Pi streams incrementally) | cooperative `should_stop`, polled per forwarded SDK event; the host is abandoned (no interrupt command exists) and a fresh one spawns for the next turn |
@@ -34,8 +34,8 @@ wall clock its numbers account for.
 | `generation_completed_at` | set | `None` — see below | `None` | `None` | `None` | set |
 | `message_id` source | SDK `message_id`; `None` when the stream carries none; `subagent-<tool_use_id>` for a synthesized sub-agent terminal | synthetic `turn_id-msg-N`, shared across the sub-messages of one generation; `turn_id-subagent-N` for recovered sub-agent generations | synthetic `turn_id-msg-N`, one per generation | CLI `messageID`; `None` when absent | CLI `responseId`; `None` when absent | synthetic `<turn_id>-msg-0` (exactly one per turn) |
 | `Σ generation + ∪ tool + head + tail ≈ turn duration` | yes [^identity] | yes [^identity] | yes [^identity] | yes [^identity] | yes [^identity] | yes, trivially — one window IS the turn |
-| clock basis for recorded stamps | one `TurnClock` per turn | SDK epoch ms (`_ms_to_dt`) — the subprocess's own clock, unreachable from the host, for BOTH window bounds and tool spans | one `TurnClock` per turn | **MIXED**: window bounds on the host `datetime.now()` (`:362`, `:696`); tool spans on CLI epoch ms (`_epoch_ms_to_dt`, `:406`/`:462`) | one `TurnClock` per turn | raw `datetime.now()` (no `TurnClock` — CE064 does not require one, since there is only one window and no inter-window drift it could correct) |
-| turn bracket (`AgentStartEvent` / `AgentEndEvent`) stamp | the same `TurnClock` (**CE064**) | raw `datetime.now()` — consistent with its epoch-ms bounds | the same `TurnClock` (**CE064**) | raw `datetime.now()` — consistent with its epoch-ms tool spans | the same `TurnClock` (**CE064**) | raw `datetime.now()` — consistent with its own single-window bounds |
+| clock basis for recorded stamps | one `TurnClock` per turn | SDK epoch ms (`_ms_to_dt`) — the subprocess's own clock, unreachable from the host, for BOTH window bounds and tool spans | one `TurnClock` per turn | **MIXED**: window bounds on the host `datetime.now()` (`:362`, `:696`); tool spans on CLI epoch ms (`_epoch_ms_to_dt`, `:406`/`:462`) | one `TurnClock` per turn | one `TurnClock` per turn (its raw `datetime.now()` bounds drifted from the monotonic `duration_seconds` on Windows and put the tail past the whole turn) |
+| turn bracket (`AgentStartEvent` / `AgentEndEvent`) stamp | the same `TurnClock` (**CE064**) | raw `datetime.now()` — consistent with its epoch-ms bounds | the same `TurnClock` (**CE064**) | raw `datetime.now()` — consistent with its epoch-ms tool spans | the same `TurnClock` (**CE064**) | the same `TurnClock` (**CE064**) |
 | window built by `timing.py::close_window` | yes | yes | yes | yes | yes | yes |
 
 [^identity]: "yes" is load-bearing, and THREE sensors check it, each seeing
@@ -126,8 +126,8 @@ Two consequences worth stating, because both are behaviour changes:
   (not on `message_id`, which OpenCode and Pi can legitimately leave `None`),
   subtracts the overlap once, and re-apportions so the parts still sum.
 
-**Two clock bases remain, and the row above says which.** Antigravity, Pi and
-claude-code derive every recorded wall stamp from one `TurnClock` per turn, so
+**Two clock bases remain, and the row above says which.** Antigravity, Pi,
+claude-code and Delegate derive every recorded wall stamp from one `TurnClock` per turn, so
 a turn's bounds and the tool spans subtracted from them cannot disagree, and
 neither can be moved by a DST transition or an NTP step inside the turn.
 Antigravity needed it first: its span was monotonic while its tool intervals
@@ -555,16 +555,18 @@ Each harness finds the call boundary in its own stream (the table above):
   step, so a rise in that total closes the call, and the next call opens with a MODEL
   step at a new `step_index`.
 - **OpenCode** and **Pi** stream one `step_start` or `turn_start` per call.
-- **Delegate**'s event stream marks a round-trip only on a text reply
-  (`isStepStart`), and a tool-only reply streams only its tool call, with no text
-  before it. The host's own `maxSteps` option does not stop the turn, so the
-  adapter does not send it and enforces the cap itself. So the next call opens when every tool the
-  previous call announced has returned, since those results go back to the model,
-  and the cap fires before call N+1 can run anything. A reply that announces
-  several tools before their results counts once. If a tool never returns, the
-  next call opens at the model's next text or new tool call instead.
-  The host sends each call's `usage` frame before that call's tool results, so
-  the calls under the cap keep their tokens and cost.
+- **Delegate** streams a call's text, then writes the call's `usage` frame, then
+  runs the call's tools. It runs a batched reply's tools one at a time, so one call
+  streams `tool_call`, `tool_result`, `tool_call`, `tool_result`, and a tool event
+  cannot mark a call boundary. A call therefore opens on its first `thinking` or
+  `message` event. A tool-only reply streams no text, so it opens on its `usage`
+  frame. A call N+1 that streams text is cut at that text, before any of its tools.
+  A tool-only call N+1 is cut at its `usage` frame, and its first tool can start
+  before the kill: the host starts a call's tools right after the frame, and the
+  stream gives no earlier boundary. The host's own `maxSteps` option does not stop
+  the turn, so the adapter does not send it and enforces the cap itself. A host that
+  sends no `usage` frames counts only the calls that stream text. The calls under
+  the cap keep their tokens and cost.
 
 Every harness enforces the cap on the same loop boundary as the
 cooperative early stop, then kills or cancels the in-flight turn so the cap stops
@@ -637,6 +639,38 @@ A timeout is a *failure* (partial turn captured, error status); the turn cap is
 *clean stop*. Conflating them is the mistake this page exists to prevent: a task
 whose cap fires should not look like a task whose harness hung.
 
+## The context window cap per harness
+
+`agent.context_window` is not a run limit, but the same promise applies: one value
+must mean the same cap on every harness that accepts it. It is the context window, in
+tokens, that the harness compacts the conversation within. A harness that cannot apply
+it rejects it when the config loads. It is never ignored, because a variant that runs
+uncapped under a "capped" label is a wrong measurement.
+
+| | claude-code | codex | antigravity | opencode | pi | delegate |
+|---|---|---|---|---|---|---|
+| mechanism | `CLAUDE_CODE_AUTO_COMPACT_WINDOW` in the CLI environment | thread config `model_context_window` | rejected | rejected | rejected | rejected |
+| accepted range | 100000-1000000 | 32768 and up | | | | |
+| when compaction starts | when the context approaches the window | at 90% of the window (Codex's default auto-compact limit) | | | | |
+| above the model's own window | capped to the model's window by the CLI | capped to the model's catalog maximum by Codex | | | | |
+| unset | the model's full window: a host `CLAUDE_CODE_AUTO_COMPACT_WINDOW` is blanked | the host's Codex config: a `model_context_window` in `$CODEX_HOME/config.toml` still applies | | | | |
+
+- **claude-code**: the environment variable outranks `--autocompact` and every settings
+  scope, so a host or project `autoCompactWindow` cannot change the cap. Setting
+  `claude_settings.autoCompactWindow` together with `context_window` fails at load when
+  `claude_settings` is an inline dict; a settings file's `autoCompactWindow` is not read,
+  and is overridden the same way. When `context_window` is unset, a host-set
+  `CLAUDE_CODE_AUTO_COMPACT_WINDOW` is blanked, so a run recorded as uncapped is never
+  capped by the host. This applies to every claude-code run, capped or not.
+- **codex**: the value replaces the model's context window, so the hard-cap
+  compaction at the usable window (a per-model share of it, 95% for Codex's fallback
+  model metadata) also moves. The floor of 32768 rejects a window too small to work in,
+  such as a dropped `000` or a YAML `yes`, which would load as 1. When `context_window`
+  is unset, coder_eval sets nothing, so a host `model_context_window` in
+  `$CODEX_HOME/config.toml` still applies.
+- Neither harness reports when the model's own window has lowered the cap. A cap
+  above the model's window runs at the model's window.
+
 ## `agent.plugins[].path` accepts different depths per harness
 
 Not a run limit, but the same promise: one task file, three harnesses, same meaning.

{coder_eval-0.12.12 → coder_eval-0.12.13}/plugins/coder-eval/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "coder-eval",
   "displayName": "Coder Eval",
-  "version": "0.12.12",
+  "version": "0.12.13",
   "description": "Test whether your Claude Code skills actually trigger, and benchmark any coding agent — author, run, and analyze the eval suite that proves it, locally or as a CI gate.",
   "author": { "name": "UiPath", "email": "coder-eval@uipath.com", "url": "https://github.com/UiPath/coder_eval" },
   "homepage": "https://coder-eval.com",

{coder_eval-0.12.12 → coder_eval-0.12.13}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "coder-eval"
-version = "0.12.12"
+version = "0.12.13"
 description = "Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode, Pi, Delegate) with sandboxed, reproducible YAML task suites."
 readme = "README.md"
 license = "Apache-2.0"