coder-eval 0.12.12__tar.gz → 0.12.14__tar.gz

{coder_eval-0.12.12 → coder_eval-0.12.14}/.claude/notes/README.md

@@ -13,6 +13,7 @@ rule docstrings in `tests/lint/rules/`, then the guides under `docs/`.
 ## Contents
 
 - [agents.md](agents.md) — agent adapters, the turn lifecycle, token reconciliation, harness parity
+- [context-window.md](context-window.md) — the `agent.context_window` cap, per harness
 - [contracts.md](contracts.md) — criteria, datasets, aggregation, judging
 - [isolation.md](isolation.md) — the docker driver, the sandbox, detached grading
 - [lint-rules.md](lint-rules.md) — why each CE lint rule exists

{coder_eval-0.12.12 → coder_eval-0.12.14}/.claude/notes/agents.md

@@ -306,6 +306,15 @@ cost bills it at the input rate and the cache buckets separately.
   id), and the `usage` snapshot the last resort. Cost is backfilled from the rate card
   when a turn was timed out or killed, so there is no terminal `ResultMessage` — the
   tokens are already captured, so this is pure pricing.
+  `model_usage` and `total_cost_usd` count the whole SESSION: a CLI resumed with
+  `resume=` restores the running totals it saved on exit, so a dialog's turn N reports
+  turns 1..N (`usage` and `num_turns` stay per-turn). `_turn_usage_slice` subtracts what
+  the same session last reported, from any ResultMessage, error or not, since the CLI
+  saves on every exit. Summing unsliced snapshots shows as a reconciliation row equal to
+  the previous turn's total (200/200 multi-turn tasks of a 2026-10-07 Haiku 5.5 run),
+  and inflates the simulator's tokens the same way. A counter that moved
+  backwards means no restore, and the snapshot is booked whole. A killed turn reports
+  nothing, so if its CLI saved totals, the next turn absorbs its usage.
 - **Codex/OpenAI** report `input_tokens` INCLUSIVE of the cached prefix, and bill no
   separate cache-write fee. So the fresh slice is `input - cached`, `cache_creation` is 0,
   and `cache_read` is `cached`.
@@ -708,8 +717,15 @@ being hardcoded, so a plugin that relocates its skills keeps working.
   skips external *plugins*, not configured skill paths. An inherited value is appended to
   rather than clobbered, since the host may legitimately configure OpenCode the same way.
 - **Pi** passes each as `--skill <dir>`.
-- **Codex** symlinks (or copies, on Windows) each skill dir into `.agents/skills/`, which
-  the CLI auto-discovers from the working directory upward.
+- **Codex** copies each skill dir into `.agents/skills/`, which the CLI auto-discovers from
+  the working directory upward. Copies, not symlinks, for two reasons that compound. Codex
+  resolves a symlinked skill to its source, finds the plugin manifest there and lists the
+  skill as `<plugin>:<skill>`; models sometimes turn that into the nonexistent path
+  `.agents/skills/<plugin>/<skill>/SKILL.md`. The search that follows (`rg --files`,
+  `find`) skips symlinks and returns nothing, so the agent works without the skill. The
+  copies are made before each turn and removed after it, the same concern that kept
+  OpenCode out of the sandbox: criteria and run artifacts must see only what the agent
+  left. Skill dirs the workspace already has are never touched.
 - **Antigravity** takes search paths natively via `skills_paths` — but those only drive
   DISCOVERY. The file-tool allowlist is `workspaces` alone, so the skill roots must appear
   there too, or the agent discovers a skill and every read of its `SKILL.md` is denied as
@@ -846,12 +862,29 @@ shared `agents/_skills.py` resolver: the SDK wants ONE parent directory, not a l
 ran 7 steps). The adapter counts calls from the event stream and abandons the host when call N+1
 opens; `len(turnUsages)` from the `result` then replaces the estimate as `num_turns`.
 
+**A call opens on its first text, or on its `usage` frame when it streams none — never on a tool
+event.** The host streams a call's text, flushes the call's `usage` frame, then runs the call's
+tools, and it runs a batched reply's tools one at a time: `tool_call`, `tool_result`, `tool_call`,
+`tool_result`. The adapter once opened call N+1 when every open tool had returned. With that order
+the open set empties after each tool, so every tool counted as a call: a downstream suite saw a turn
+with 16 round-trips and 96 tool calls cut at `max_turns: 80`. Now a `thinking` or `message` event
+opens a call unless the open call has already streamed text, and a `usage` frame opens one only for
+a call that streamed none. A crossing call that streams text is cut at that text, before any of its
+tools. A tool-only crossing call is cut at its own frame, and its first tool can start before the kill:
+the host starts a call's tools right after the frame, and the stream gives no earlier boundary. A host
+without `usage` frames counts only the calls that stream text; such a host is below the 1.203.0 floor,
+which is also the first release that takes the `auth` init option.
+Only `_TurnState.on_text` / `on_tool_event` / `on_usage_frame` / `on_result` write the count, so the rule
+lives in one place. `on_result` compares the live count with `len(turnUsages)` and warns when they
+differ: the order is read from the 1.203.1 host and SDK source, not from a live transcript, so a host
+release that reorders the frames shows up in the logs instead of silently moving the cap.
+
 **A cut turn keeps the usage of its finished calls.** A turn cut by `max_turns` or a stop gets no
 `result`, so the adapter sums the per-round-trip `usage` frames; the `result`'s total replaces the sum
-when it arrives. Ordering (from SDK/backend source, not a live transcript): call N's `usage` frame
-always precedes the `tool_result` that opens call N+1. The frame is NOT a call boundary — it arrives
-before its call's tools run — so the cap stays on "the previous call's tools have all returned". The
-adapter warns when a cut turn had finished calls but no usage.
+when it arrives. Ordering (from the host and SDK source, not a live transcript): call N's `usage` frame
+precedes every tool event of call N. A crossing call that streams text is cut at that text, so its usage
+is dropped; a tool-only crossing call is cut at its own frame, so its usage is booked. The adapter warns
+when a cut turn had finished calls but no usage.
 
 **One `AssistantMessage` per `communicate()`.** `isStepStart` and `turnUsages` would allow a
 per-round-trip split; today `isStepStart` only merges streamed deltas. `close_window` opens the one
@@ -948,8 +981,9 @@ introduced or should silently "fix" by picking one number over the other.
 
 ### Delegate agent golden-master coverage
 
-`tests/_fixtures/golden_streams/delegate_fixtures.py` scripts the host's stdout frames for four
-scenarios: a tool call, a `max_turns` cut, an `error`-frame crash and an orphaned tool. The `event`,
+`tests/_fixtures/golden_streams/delegate_fixtures.py` scripts the host's stdout frames for six
+scenarios: a tool call, a `max_turns` cut, an `error`-frame crash, an orphaned tool, a tool-only call
+that crosses the cap, and a batched reply whose tools run one at a time. The `event`,
 `result` and `error` frames mirror the live `1.202.1` transcript. The `usage` frame and the `result`
 fields new in `1.203.0` follow the host source of UiPath/Autopilot#6656 (`src/usage.ts` and
 `handleSend`): every usage object carries all four buckets with `input_tokens` exclusive of cache,

coder_eval-0.12.14/.claude/notes/context-window.md

@@ -0,0 +1,170 @@
+# Capping the agent's context window
+
+Analysis and design for `agent.context_window`: one harness-neutral field that caps the
+context window an agent works within, set per agent config or per experiment variant.
+
+## Problem and evidence
+
+A benchmark of Claude Code on Sonnet 5 (Bedrock, 1M-token window) let runs grow to
+250k-480k tokens of context. Above about 150k tokens each extended-thinking step got
+about 10x slower, so most runs hit the 2-hour task limit. The operator wants to measure
+pass rate and cost at several context caps, for example one variant at 200k and one at
+400k, on the same tasks.
+
+Before this change no harness-neutral way existed:
+
+- **claude-code**: `claude_settings` (`--settings`) could carry `autoCompactWindow`. That
+  works for one harness only, and when `claude_settings` is a file path a variant cannot
+  merge one key into it.
+- **codex**: `CodexAgentConfig` has no pass-through. `CodexAgent._build_thread_options`
+  builds the thread `config` dict itself, so no value reached Codex.
+- **antigravity, opencode, pi, delegate**: no knob is wired.
+
+## How each harness controls its window
+
+### claude-code
+
+Verified in the CLI the SDK spawns. `claude-agent-sdk` runs its **bundled** CLI before it
+looks for `claude` on `PATH` (`SubprocessCLITransport._find_cli`), so the CLI that runs is
+the one the SDK wheel bundles, not the one the agent image installs with npm:
+
+| coder_eval | claude-agent-sdk (uv.lock) | bundled CLI | image npm CLI |
+|---|---|---|---|
+| 0.12.1 | 0.2.124 | 2.1.216 | 2.1.177 |
+| 0.12.12 | 0.2.159 | 2.1.281 | 2.1.281 |
+
+`environment_info.claude_code_cli` records `claude -v` from `PATH`, so a 0.12.1 run
+records 2.1.177 while 2.1.216 ran. That is a separate defect and is not fixed here.
+
+Three inputs set the auto-compact window. Both bundled CLIs (2.1.216 and 2.1.281) contain
+the same resolution function. The highest precedence comes first:
+
+1. `CLAUDE_CODE_AUTO_COMPACT_WINDOW` environment variable: a plain token count. It
+   outranks the flag and every settings scope.
+2. `--autocompact <auto|tokens>`: added in 2.1.221 (CLI reference). It is absent from
+   2.1.216 as a parsed option.
+3. `autoCompactWindow` settings key. The settings schema is
+   `int().min(100000).max(1000000).optional().catch(undefined)`, so an out-of-range
+   value is **silently dropped**.
+
+The effective window is `min(model context window, configured value)`. Auto-compaction
+starts when the context approaches that window.
+
+### codex
+
+Verified in the `openai-codex-cli-bin` 0.156.1 binary (the pin) and its source at tag
+`rust-v0.156.1`:
+
+- `model_context_window` ("Size of the context window for the model, in tokens"):
+  `with_config_overrides` sets the model's `context_window` to
+  `min(value, max_context_window)`. Three things follow from it. The default
+  auto-compact limit becomes 90% of it. The hard cap, which forces compaction, becomes
+  `context_window * effective_context_window_percent / 100`. The remaining-window figure
+  the model is shown also changes.
+- `model_auto_compact_token_limit` ("Token usage threshold triggering auto-compaction"):
+  the trigger only, clamped to 90% of the resolved window. It does not change the window.
+- `model_auto_compact_token_limit_scope` (`total` by default, or `body_after_prefix`) and
+  `model_post_turn_compact_threshold_percent` (turn-end compaction, off by default) tune
+  when compaction starts. They are not caps.
+
+The thread `config` dict `thread_start` takes is the same override surface that
+coder_eval already uses for `enabled_tools` and `model_providers`.
+
+### antigravity, opencode, pi, delegate
+
+None of them has a context-window knob wired in coder_eval. They are out of scope, and
+they reject the field.
+
+## Options considered
+
+**Per-harness pass-through** (a Codex `config` pass-through next to `claude_settings`).
+It is flexible, but a variant would have to name a different key per harness, and the
+recorded value would mean something different on each. It also opens a free-form
+pass-through on Codex with no denylist. Rejected.
+
+**A field under `run_limits`.** Run limits are caps that the orchestrator or the agent
+enforces on a run (turns, time, tokens, USD), and the parity table lists them. A context
+window is not a spend cap. It is a harness setting that changes what the agent does, and
+it depends on the agent type, which `run_limits` does not know. The guide also says that
+`agent:` holds no run-time caps. Rejected.
+
+**A field on the agent config.** One name, accepted on a type-less config so that
+experiment defaults and variants can set it, and validated against the concrete type
+when the config resolves. **Chosen.**
+
+Naming: `context_window`, in tokens. It names the quantity the operator caps, matches
+Codex's `model_context_window`, and avoids "auto-compact", which describes a mechanism of
+one harness. `max_context_tokens` was considered, but it reads as a per-request input
+limit, which neither harness enforces.
+
+## Recommended design
+
+`BaseAgentConfig.context_window: int | None` (default `None`, `gt=0`).
+
+- Each config class declares `_context_window_range: ClassVar[tuple[int, int | None] |
+  None]`. `None` (the base default) means that the harness cannot apply the field.
+  `ClaudeCodeAgentConfig` declares `(100_000, 1_000_000)` and `CodexAgentConfig` declares
+  `(32_768, None)`.
+- `check_context_window_supported` (model validator) rejects an unsupported type or an
+  out-of-range value as soon as the type is known. A plugin agent inherits `None`, so it
+  also rejects the field until it opts in.
+- The field uses the default `replace` merge strategy. It is set at any of the five
+  layers, and with `-D agent.context_window=N`.
+
+Per-harness mapping:
+
+- **claude-code**: `CLAUDE_CODE_AUTO_COMPACT_WINDOW=<N>` in the SDK `env`. The
+  environment variable was chosen over the settings key and the flag for four reasons.
+  It has the highest precedence, so host, project and `claude_settings` values cannot
+  override the cap. It works whether `claude_settings` is a dict or a file path. Both
+  bundled CLIs read it, which the flag does not. `extra_args` stays framework-owned.
+  `claude_settings.autoCompactWindow` together with `context_window` is a load error,
+  because the environment variable would silently override it.
+- **codex**: `model_context_window: <N>` in the thread `config`. No trigger key is set,
+  so Codex compacts at its default 90% of the window.
+
+## Validation
+
+- `gt=0` on every config, so a type-less config rejects nonsense as well.
+- claude-code: 100000-1000000, the CLI's documented range. Outside that range the CLI
+  drops a settings value silently, so coder_eval enforces the range itself.
+- codex: 32768 and up. Codex clamps the value to the model's catalog maximum, and the
+  floor rejects a window too small to work in, such as a YAML `yes` coerced to 1.
+- every other type: rejected at load, naming the supported types. A variant that
+  switches `type` to an unsupported harness fails when the experiment resolves.
+
+## Recording
+
+The resolved agent config is persisted as `agent_config` on every task row (`run.json`
+and `task.json`). `agent.context_window` therefore appears per task, and config lineage
+records which layer set it (for example, the variant). The experiment report groups by
+variant, so a cap per variant groups without new report code. On claude-code the
+`sdk_options` dump also shows the environment variable in `env`.
+
+## Open questions
+
+- **A silent clamp to the model's window.** Neither harness reports that the model's
+  window lowered the cap. A cap above the model's window runs at the model's window.
+  Recording the effective window would need a per-model catalog in coder_eval.
+- **Trigger points differ.** Codex compacts at 90% of the window. Claude Code compacts
+  when the context approaches the window, minus an internal buffer. Equal caps therefore
+  do not compact at exactly the same token count. A separate harness-neutral trigger
+  field is possible, but it is not added until a benchmark needs it.
+- **A host `CLAUDE_CODE_AUTO_COMPACT_WINDOW` and uncapped runs.** Decided: the SDK starts
+  the CLI with `os.environ` underneath `options.env`, so when the field is unset the
+  claude-code agent blanks a host value. A run recorded as uncapped is never capped by
+  the host. This changes every claude-code run on a host that sets the variable,
+  including runs that never set `context_window`. Codex has no equivalent: an unset
+  field leaves a host `model_context_window` in `$CODEX_HOME/config.toml` in effect.
+- **The recorded CLI version is wrong.** See the version table above:
+  `environment_info.claude_code_cli` should record the bundled CLI version.
+- **Other harnesses.** Pi can lower a model's window with
+  `providers.<provider>.modelOverrides.<model>.contextWindow` in `models.json`. A mapping
+  was written and then reverted (d1757a60), because no live run backed it and no task
+  uses `type: pi`; it comes back with a live test. OpenCode (`limit.context` and
+  `limit.input` in provider config) needs a `limit.output` coder_eval does not know, and
+  an unpinned CLI. Antigravity's `CompactionConfig(token_threshold=N)` only moves the
+  trigger. Delegate keeps its history in the backend. Each needs its own verification
+  before it opts in.
+- **How often a run compacted is not recorded.** A cap A/B cannot yet tell "cap never
+  reached" from "compacted N times" from the task row alone.

{coder_eval-0.12.12 → coder_eval-0.12.14}/.claude/notes/lint-rules.md

@@ -845,7 +845,7 @@ assertion for each. A defaulted `AgentStartEvent` lands ~365 days before the clo
 first window, so the head blows any sane UPPER bound by the whole offset. A defaulted
 `AgentEndEvent` lands ~365 days BEFORE its own last message, so `decompose_turn` clamps the
 negative to `0.0` — "measured, and instant", which passes an upper bound. Only a strict
-`> 0.0` catches it. It holds on all three clocked harnesses because real work separates a
+`> 0.0` catches it. It holds on every clocked harness because real work separates a
 turn's last flush from its end event. The margin is smallest on antigravity, which holds
 its process across turns: it measures 0.007-0.035 ms there, 7-35 ticks of the 1 us
 resolution that both `datetime` and `time.monotonic()` have on Linux, macOS and Windows.

{coder_eval-0.12.12 → coder_eval-0.12.14}/.claude/notes/timing.md

@@ -16,6 +16,11 @@ removes:
   transition or an NTP step inside a turn lands directly in a generation window — an
   hour-long jump in a millisecond field. Nightly runs start at 04:18 and run for hours,
   so it is reachable rather than theoretical. A monotonic-derived stamp cannot express it.
+- Delegate stamped its one window and its tool spans with raw `datetime.now()` but measured
+  `duration_seconds` with `time.monotonic()`. On Windows the wall clock moved 0.24-0.37 ms
+  further than the monotonic one inside a sub-millisecond turn, so the tail alone came out
+  longer than the whole turn and `test_delegate_golden[a_tool_call]` failed at random. One
+  window has no inter-window drift to correct, but it still shares the turn's duration.
 
 It is an EXTRACTION, not an invention: antigravity already captured this exact pair at
 the top of `communicate` and simply did not use it for later stamps.
@@ -205,7 +210,7 @@ Magnitudes are only real where a scripted clock makes them real, so each case dr
 harness's own reducer with a hand-moved clock and feeds its output through a real
 `EventCollector`, the seam production uses to compute the head and the tail. The module
 reuses the three clock-injection idioms the per-harness suites already use rather than
-inventing a fourth: an injected `TurnClock` stand-in (pi, antigravity, claude-code), a
+inventing a fourth: an injected `TurnClock` stand-in (pi, antigravity, claude-code, delegate), a
 `datetime` subclass patched onto the module (opencode), and scripted SDK epoch-millisecond
 stamps (codex). OpenCode needs a `datetime` SUBCLASS rather than a stub because the
 reducer also calls `datetime.fromtimestamp` through the same module global to convert the

{coder_eval-0.12.12 → coder_eval-0.12.14}/CHANGELOG.md

@@ -2,6 +2,74 @@
 
 <!-- version list -->
 
+## v0.12.14 (2026-10-15)
+
+### Bug Fixes
+
+- **codex**: Copy skills into .agents/skills instead of symlinking
+  ([#225](https://github.com/UiPath/coder_eval/pull/225),
+  [`1895a2c`](https://github.com/UiPath/coder_eval/commit/1895a2cfb0c2f0d8a171b2e445b0b034440209b2))
+
+
+## v0.12.13 (2026-10-13)
+
+### Bug Fixes
+
+- **agent**: Address the context_window review
+  ([#219](https://github.com/UiPath/coder_eval/pull/219),
+  [`ae02485`](https://github.com/UiPath/coder_eval/commit/ae02485a7a21944e2ee15a0f72131def5f2e425a))
+
+- **agents**: Count Delegate batched tool calls once under max_turns and stamp turns off one
+  TurnClock ([#222](https://github.com/UiPath/coder_eval/pull/222),
+  [`db2067e`](https://github.com/UiPath/coder_eval/commit/db2067e94e458796c10cfe5f2b0c0383a3c8db03))
+
+- **claude-code**: Book only the turn's own usage on a resumed session
+  ([#221](https://github.com/UiPath/coder_eval/pull/221),
+  [`c9456e5`](https://github.com/UiPath/coder_eval/commit/c9456e58b49174c29f48577507f249eb9e98a94b))
+
+- **delegate**: Count a batched reply's tools as one model call under max_turns
+  ([#222](https://github.com/UiPath/coder_eval/pull/222),
+  [`db2067e`](https://github.com/UiPath/coder_eval/commit/db2067e94e458796c10cfe5f2b0c0383a3c8db03))
+
+- **delegate**: Keep the max_turns count in one place and warn when it drifts
+  ([#222](https://github.com/UiPath/coder_eval/pull/222),
+  [`db2067e`](https://github.com/UiPath/coder_eval/commit/db2067e94e458796c10cfe5f2b0c0383a3c8db03))
+
+- **delegate**: Stamp the whole turn off one TurnClock
+  ([#222](https://github.com/UiPath/coder_eval/pull/222),
+  [`db2067e`](https://github.com/UiPath/coder_eval/commit/db2067e94e458796c10cfe5f2b0c0383a3c8db03))
+
+- **pricing**: Add Haiku 5.5, Sonnet 5.5 and GPT-6.1 Sol, re-verify every row
+  ([#220](https://github.com/UiPath/coder_eval/pull/220),
+  [`b3ae960`](https://github.com/UiPath/coder_eval/commit/b3ae96033d32f3e88bff68a15219e85372762220))
+
+### Chores
+
+- **no-mistakes/document**: Name Delegate among the TurnClock harnesses in timing docs
+  ([#222](https://github.com/UiPath/coder_eval/pull/222),
+  [`db2067e`](https://github.com/UiPath/coder_eval/commit/db2067e94e458796c10cfe5f2b0c0383a3c8db03))
+
+### Documentation
+
+- **delegate**: Count the six Delegate golden-master scenarios in the notes
+  ([#222](https://github.com/UiPath/coder_eval/pull/222),
+  [`db2067e`](https://github.com/UiPath/coder_eval/commit/db2067e94e458796c10cfe5f2b0c0383a3c8db03))
+
+- **delegate**: Document the tool-only cut race and relax the live test count
+  ([#222](https://github.com/UiPath/coder_eval/pull/222),
+  [`db2067e`](https://github.com/UiPath/coder_eval/commit/db2067e94e458796c10cfe5f2b0c0383a3c8db03))
+
+### Features
+
+- **agent**: Cap Pi's context window with agent.context_window
+  ([#219](https://github.com/UiPath/coder_eval/pull/219),
+  [`ae02485`](https://github.com/UiPath/coder_eval/commit/ae02485a7a21944e2ee15a0f72131def5f2e425a))
+
+- **agent**: Cap the agent's context window with agent.context_window
+  ([#219](https://github.com/UiPath/coder_eval/pull/219),
+  [`ae02485`](https://github.com/UiPath/coder_eval/commit/ae02485a7a21944e2ee15a0f72131def5f2e425a))
+
+
 ## v0.12.12 (2026-10-06)
 
 ### Bug Fixes

{coder_eval-0.12.12 → coder_eval-0.12.14}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: coder-eval
-Version: 0.12.12
+Version: 0.12.14
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode, Pi, Delegate) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval

{coder_eval-0.12.12 → coder_eval-0.12.14}/action.yml

@@ -25,7 +25,7 @@ inputs:
   version:
     description: coder-eval version to install from PyPI, or "local" to install from the action checkout
     required: false
-    default: "0.12.12"   # <-- kept in sync with releases by release.yml
+    default: "0.12.14"   # <-- kept in sync with releases by release.yml
   extras:
     description: >-
       Comma-separated coder-eval extras (`codex`, `antigravity,litellm`), composed

{coder_eval-0.12.12 → coder_eval-0.12.14}/docs/AB_EXPERIMENTS.md

@@ -23,6 +23,7 @@ to the orchestrator are involved.
 - [What a Variant Can Override](#what-a-variant-can-override)
 - [Recipe: A/B a Skill](#recipe-ab-a-skill)
 - [Recipe: A/B a Model](#recipe-ab-a-model)
+- [Recipe: A/B a Context Window](#recipe-ab-a-context-window)
 - [Recipe: A/B a Prompt](#recipe-ab-a-prompt)
 - [Recipe: Smoke vs. e2e Flavors (Early Stop)](#recipe-smoke-vs-e2e-flavors-early-stop)
 - [Replicates (Statistical Power)](#replicates-statistical-power)
@@ -222,6 +223,28 @@ variants:
     agent: { model: claude-opus-5 }
 ```
 
+## Recipe: A/B a Context Window
+
+`agent.context_window` caps the context the agent works within, in tokens: the
+harness compacts the conversation before it outgrows the cap. Use it to measure
+pass rate and cost against the cap on a model with a large window. Each task row
+records the cap in `agent_config.context_window`.
+
+```yaml
+experiment_id: context-window
+description: "The same model with a 200k and a 400k context cap"
+
+variants:
+  - variant_id: cap-200k
+    agent: { context_window: 200000 }
+  - variant_id: cap-400k
+    agent: { context_window: 400000 }
+```
+
+An agent type that cannot apply the cap fails the experiment at load.
+[Run-Limit Parity](agents/HARNESS_PARITY.md#the-context-window-cap-per-harness) lists which
+agent types accept it and what the cap does on each.
+
 ## Recipe: A/B a Prompt
 
 Use `prompt_mutations` (transform the task prompt) or `initial_prompt` /

{coder_eval-0.12.12 → coder_eval-0.12.14}/docs/EXTENDING.md

@@ -80,6 +80,22 @@ class MyAgentConfig(BaseAgentConfig):
 The factory `create_agent(kind, config, …)` raises `TypeError` if the passed config
 isn't an instance of the registered `config_class`, so keep them paired.
 
+To accept `agent.context_window`, declare the inclusive range of tokens your harness
+can cap its context window to, and apply `self.config.context_window` when the agent
+starts. A config class that declares nothing rejects the field at load, which is the
+right default: a capped variant must never run uncapped.
+
+```python
+from typing import ClassVar
+
+class MyAgentConfig(BaseAgentConfig):
+    type: Literal["my-agent"] = "my-agent"
+    _context_window_range: ClassVar[tuple[int, int | None] | None] = (32_768, None)
+```
+
+Add your harness to the table in
+[Run-Limit Parity](agents/HARNESS_PARITY.md#the-context-window-cap-per-harness).
+
 ### The `Agent` ABC — implementation checklist
 
 Implement these three abstract methods:

{coder_eval-0.12.12 → coder_eval-0.12.14}/docs/TASK_DEFINITION_GUIDE.md

@@ -170,6 +170,7 @@ agent:
     - "Write"
     - "Bash"
   model: "claude-sonnet-5"            # Optional: specific model
+  context_window: 200000              # Optional: cap the context window, in tokens (claude-code, codex)
   sdk_options:                        # Optional: agent SDK pass-through (keys depend on `type`)
     effort: high                      # claude-code: any non-framework-managed ClaudeAgentOptions field
 ```
@@ -188,6 +189,13 @@ validates its own keys at YAML load:
 Deep-merged across the 5-layer config chain. Override via CLI with the
 repeatable `-D agent.sdk_options.KEY=VALUE`.
 
+**`context_window`** caps the context window the agent works within, in tokens: the
+harness compacts the conversation before it outgrows the cap, and never runs past the
+model's own window. Override via CLI with `-D agent.context_window=400000`. Which agent
+types accept it, in what range, what each does with it and what an unset value means:
+[Run-Limit Parity](agents/HARNESS_PARITY.md#the-context-window-cap-per-harness). An agent
+type that cannot apply it rejects it at load.
+
 **Permission Modes:**
 - `default` — Default permission handling
 - `acceptEdits` — Auto-accept file edits (recommended for evaluations)
@@ -1337,7 +1345,7 @@ Like `skill_triggered`, this criterion emits a `ClassificationCriterionResult`,
 
 ### `skill_triggered`
 
-Binary classifier: **did the agent engage the target skill during the run?** Agent-agnostic — scans the run's `turn_records` for either signal: Claude's explicit `Skill` tool call whose `skill` parameter matches `skill_name` (namespace prefixes like `plugin:skill` are stripped, so `skill_name: uipath-agents` matches `Skill(skill="uipath-coded-agents:uipath-agents")`), or — for an agent with no `Skill` tool, e.g. Codex — a command that reads the skill's files off disk (a parameter contains `skills/<skill_name>/`, matching both the repo path and the `.agents/skills/` symlink).
+Binary classifier: **did the agent engage the target skill during the run?** Agent-agnostic — scans the run's `turn_records` for either signal: Claude's explicit `Skill` tool call whose `skill` parameter matches `skill_name` (namespace prefixes like `plugin:skill` are stripped, so `skill_name: uipath-agents` matches `Skill(skill="uipath-coded-agents:uipath-agents")`), or — for an agent with no `Skill` tool, e.g. Codex — a command that reads the skill's files off disk (a parameter contains `skills/<skill_name>/`, matching both the repo path and the `.agents/skills/` copy).
 
 Observed label is `"yes"` when either signal is found, else `"no"`. Expected label is `"yes"` iff `expected_skill == skill_name`. **Binary scoring:** `1.0` when observed matches expected, else `0.0`.
 

{coder_eval-0.12.12 → coder_eval-0.12.14}/docs/agents/ANTIGRAVITY.md

@@ -127,7 +127,7 @@ Antigravity supports [Agent Skills](https://agentskills.io/specification)
 2. the runtime plugin directory passed to `start()`.
 
 For each source, the harness is handed whichever of `<source>/skills` or `<source>`
-directly contains subdirectories with a `SKILL.md`. Unlike Codex — which symlinks
+directly contains subdirectories with a `SKILL.md`. Unlike Codex — which copies
 skills into `.agents/skills/` — Antigravity is given the search-path roots directly
 via the SDK's `skills_paths`, and those roots are also added to the harness
 `workspaces` allowlist (otherwise the agent would find a skill but be denied the

{coder_eval-0.12.12 → coder_eval-0.12.14}/docs/agents/CLAUDE_CODE.md

@@ -104,6 +104,7 @@ agent:
 | `system_prompt_file` | `str \| null` | Path (relative to the task YAML) loaded into `system_prompt` at resolution. Works with either `system_prompt_mode`. |
 | `setting_sources` | `list["user"\|"project"\|"local"] \| null` | Which host setting sources the SDK reads. Default resolves to `["project"]`. See [Sandbox isolation](#sandbox-isolation). |
 | `claude_settings` | `str \| dict \| null` | Passed to the SDK `--settings`. A dict is JSON-serialized; a str is a settings file path. Use `permissions.deny` to block tools/paths. |
+| `context_window` | `int \| null` (100000-1000000) | Auto-compact window, in tokens, sent as `CLAUDE_CODE_AUTO_COMPACT_WINDOW`. It outranks `--autocompact` and every settings scope, and is capped to the model's window. Setting `claude_settings.autoCompactWindow` too is a load error. See [Harness parity](HARNESS_PARITY.md#the-context-window-cap-per-harness). |
 | `sdk_options` | `dict` (default `{}`) | Pass-through for `ClaudeAgentOptions` fields Coder Eval doesn't own (e.g. `effort`). Validated at load — an unknown or framework-owned key is a hard error. |
 | `ignore_patterns` | `list[str] \| null` | Gitignore-style overrides for the workspace copy used by judge sub-agents (supports `!` negation). |
 
@@ -212,9 +213,11 @@ resolution for any agent that does not declare `supports_cooperative_stop`. See
 
 Claude Code produces the richest telemetry of the agents:
 
-- **Authoritative billing** comes from the SDK's cumulative `model_usage` on the
+- **Authoritative billing** comes from the SDK's `model_usage` on the
   terminal result message and reconciles to `total_cost_usd` — this already includes
-  sub-agent consumption that the per-message stream under-reports.
+  sub-agent consumption that the per-message stream under-reports. Both count the
+  whole session, so a resumed turn books only what changed since the session's
+  previous result message.
 - **Sub-agent accounting** is derived by grouping `parent_tool_use_id`-tagged
   assistant messages; there is no separate per-sub-agent field. The terminal
   sub-agent generation (delivered as the Agent tool result, never streamed) is

{coder_eval-0.12.12 → coder_eval-0.12.14}/docs/agents/CODEX.md

@@ -115,7 +115,7 @@ CodexAgent supports SKILL.md files following the [Agent Skills open standard](ht
 1. **config.plugins** - Local plugins with `type: local` and `path` pointing to a skills directory
 2. **plugin_tools_dir** parameter - Runtime plugin directory passed to `start()`
 
-Skills are symlinked (or copied) to `.agents/skills/` where the Codex CLI auto-discovers them. Environment variables in plugin paths (`$VAR`, `${VAR}`) are expanded at runtime.
+Skills are copied to `.agents/skills/` for each turn, where the Codex CLI auto-discovers them, and removed when the turn ends. Environment variables in plugin paths (`$VAR`, `${VAR}`) are expanded at runtime.
 
 Example with environment variable:
 ```yaml
@@ -192,6 +192,13 @@ The agent maps `permission_mode` to the Codex SDK's `Sandbox`. The approval mode
 
 `allowed_tools` / `disallowed_tools` are normalized (`Bash` → `shell`, `Write`/`Edit` → `apply_patch`, etc.) and passed as `enabled_tools` / `disabled_tools` in the thread `config`. **Note:** the Codex SDK does not currently enforce `disabled_tools`; do not rely on it as a security boundary (the agent logs a warning when it is set).
 
+### Context Window
+
+`agent.context_window` is passed as `model_context_window` in the thread `config`. Codex
+uses it as the model's context window, capped to the model's catalog maximum, and
+auto-compacts at 90% of it. See
+[Harness parity](HARNESS_PARITY.md#the-context-window-cap-per-harness).
+
 ### Skills Discovery
 
 The agent sets up SKILL.md files (Agent Skills open standard) in `.agents/skills/` directory:
@@ -199,8 +206,9 @@ The agent sets up SKILL.md files (Agent Skills open standard) in `.agents/skills
 1. Scans `config.plugins` for local plugins with `path` field
 2. Checks `plugin_tools_dir` parameter passed to `start()`
 3. Expands environment variables in paths (`$PLUGIN_PATH`, `${PLUGIN_PATH}`)
-4. Symlinks skill directories (falls back to copying if symlink fails)
+4. Copies each skill directory into `.agents/skills/` before every turn (real files, not symlinks), skipping any skill dir the workspace already has
 5. Codex CLI auto-discovers skills in `.agents/skills/`
+6. Removes the copies when the turn ends, so criteria and run artifacts never see them
 
 ### Async Integration
 

{coder_eval-0.12.12 → coder_eval-0.12.14}/docs/agents/DELEGATE.md

@@ -168,8 +168,8 @@ Run-limit semantics per harness: [Run-Limit Parity](HARNESS_PARITY.md).
 ## Known Limitations
 
 1. **No multi-generation transcript splitting.** This agent builds one `AssistantMessage` per `communicate()` call, not one per backend round-trip. The host does send the signals a split needs (`isStepStart` on `message` events and per-round-trip `turnUsages` on `result`), but this agent does not use them yet.
-2. **`max_turns` is enforced by coder_eval, not by the host.** The host's `maxSteps` option on `send` does not stop the turn (confirmed live: `maxSteps: 2` ran 7 steps and only reported `maxStepsReached: true`), so this agent does not send it. See [Run-Limit Parity](HARNESS_PARITY.md).
-   A turn cut at `max_turns`, or by a cooperative early stop, keeps the token usage and cost of every model call that finished before the cut, as on the other agents. The host sends a `usage` frame for each call before that call's tool results. The call in progress at the cut has no usage. A host that does not send `usage` frames reports usage only on its final `result` frame, which a cut turn never gets: that turn has no usage, and the agent logs a warning.
+2. **`max_turns` is enforced by coder_eval, not by the host.** The host's `maxSteps` option on `send` does not stop the turn (confirmed live: `maxSteps: 2` ran 7 steps and only reported `maxStepsReached: true`), so this agent does not send it. It counts a call on the call's first streamed text, or on its `usage` frame when the call streams none. The host runs a batched reply's tools one at a time, so tool events do not mark a call. See [Run-Limit Parity](HARNESS_PARITY.md).
+   A turn cut at `max_turns`, or by a cooperative early stop, keeps the token usage and cost of every model call that finished before the cut, as on the other agents. The host sends a `usage` frame for each call before that call's tool results. Which side of the cut the crossing call lands on depends on its shape. A call that streams text is cut at that text, before its `usage` frame and before any of its tools, so its usage is dropped. A tool-only call is cut at its own `usage` frame, so its usage is booked, and its first tool can start before the kill: the host starts a call's tools right after the frame, and the stream gives no earlier boundary. When the `result` arrives, the agent compares its own call count with the host's `turnUsages` and logs a warning if they differ, because a change in the host's frame order would move the cap. A host that does not send `usage` frames reports usage only on its final `result` frame, which a cut turn never gets: that turn has no usage, and the agent logs a warning.
 3. **Only three backend failures get a specific diagnosis.** A Cloudflare WAF block page is reported as a content-filter failure and is not retried: the same prompt or tool result is blocked again. An SSE connect timeout is reported as a connection failure and is retried. A session conflict ("A reply is already being generated") is retried in a new conversation when no turn of that conversation has finished. After a finished turn, or with a `session_id` set in the config, the retry stays in the same conversation, so the agent never continues without its earlier turns: the retry succeeds once the backend releases the conversation (it stops the reply when the host disconnects), or the task ends as an error. There is no first-response stall detection: a stalled turn ends at its turn timeout. Every other crash ends the turn as an `AgentCrashError`, which is retried unless its message matches a category that is not retried (a timeout, or an auth, billing or content-filter error).
 4. **`sdk_options` accepts only `effort`.** Reasoning effort uses the same `sdk_options.effort` key as Claude Code, so `-D agent.sdk_options.effort=high` works for both agents. Any other key, or an `effort` that is not a string, is a validation error. The host checks the tier itself: it logs a tier it does not recognize and uses the model's default. `project_id`, `session_id` and `enable_computer_use` are typed config fields.
 5. **`enable_computer_use: true` requires local permissions and is unavailable on Linux.** macOS needs Accessibility + Screen Recording grants; the SDK throws unconditionally on Linux when this is enabled.