coder-eval 0.12.11__tar.gz → 0.12.13__tar.gz

{coder_eval-0.12.11 → coder_eval-0.12.13}/.claude/notes/README.md

@@ -13,6 +13,7 @@ rule docstrings in `tests/lint/rules/`, then the guides under `docs/`.
 ## Contents
 
 - [agents.md](agents.md) — agent adapters, the turn lifecycle, token reconciliation, harness parity
+- [context-window.md](context-window.md) — the `agent.context_window` cap, per harness
 - [contracts.md](contracts.md) — criteria, datasets, aggregation, judging
 - [isolation.md](isolation.md) — the docker driver, the sandbox, detached grading
 - [lint-rules.md](lint-rules.md) — why each CE lint rule exists

{coder_eval-0.12.11 → coder_eval-0.12.13}/.claude/notes/agents.md

@@ -306,6 +306,15 @@ cost bills it at the input rate and the cache buckets separately.
   id), and the `usage` snapshot the last resort. Cost is backfilled from the rate card
   when a turn was timed out or killed, so there is no terminal `ResultMessage` — the
   tokens are already captured, so this is pure pricing.
+  `model_usage` and `total_cost_usd` count the whole SESSION: a CLI resumed with
+  `resume=` restores the running totals it saved on exit, so a dialog's turn N reports
+  turns 1..N (`usage` and `num_turns` stay per-turn). `_turn_usage_slice` subtracts what
+  the same session last reported, from any ResultMessage, error or not, since the CLI
+  saves on every exit. Summing unsliced snapshots shows as a reconciliation row equal to
+  the previous turn's total (200/200 multi-turn tasks of a 2026-10-07 Haiku 5.5 run),
+  and inflates the simulator's tokens the same way. A counter that moved
+  backwards means no restore, and the snapshot is booked whole. A killed turn reports
+  nothing, so if its CLI saved totals, the next turn absorbs its usage.
 - **Codex/OpenAI** report `input_tokens` INCLUSIVE of the cached prefix, and bill no
   separate cache-write fee. So the fresh slice is `input - cached`, `cache_creation` is 0,
   and `cache_read` is `cached`.
@@ -558,7 +567,7 @@ shell-aware `parameters["command"]` extraction in `criteria/command_executed.py`
 to raw-JSON matching — so the same task scores differently per harness. Unknown names pass
 through unchanged.
 
-Three cases are worth knowing:
+Four cases are worth knowing:
 
 - **OpenCode's tool set varies by MODEL within the one harness.** A live 174-task run
   showed DeepSeek using `write`/`edit` 199 times and `apply_patch` 0, while GPT-5.6 used
@@ -571,6 +580,11 @@ Three cases are worth knowing:
 - **Pi's search tool is `find`** (glob-by-pattern), not `glob`; there is no `glob` tool in
   its built-in set, so mapping `find` to the canonical `Glob` is what keeps
   `command_executed` and `commands_efficiency` comparable.
+- **Delegate's host maps most names itself, but not `LoadSkill`.** `delegate-stdio` reports
+  `ReadFile` as `Read` and `ExecuteSkillApi` as `Skill`, so `DelegateAgent` maps only
+  `LoadSkill {name}` to `Skill {skill}`; without it `skill_triggered` never sees a Delegate skill
+  load. The rename is keyed by the host's name: keyed by `Skill`, it would also turn
+  ExecuteSkillApi's `name`, an API call, into a skill engagement.
 
 Antigravity additionally strips the result payload out of a tool call's arguments: the
 harness folds result fields into the same `args` dict at DONE. Beyond a static key list,
@@ -801,89 +815,134 @@ release adding an unclassified field fails loudly instead of silently passing th
 
 ## Delegate agent
 
-`DelegateAgent` drives UiPath Autopilot's Delegate agent — reasoning in the UiPath backend,
-tools executing locally through the SDK's bundled interop process. Unlike every other CLI-driven
-agent in this file, there is no vendor-provided stdio protocol host to spawn: `@uipath/delegate-stdio`,
-the internal package a UiPath-only sibling plugin (`coder_eval_uipath`) drives, is not public. Only
-`@uipath/delegate-sdk` (a programmatic library exposing a `DelegateAgent` class with
-`initialize()`/`onEvent()`/`sendMessage()`/`destroy()`) and `@uipath/delegate-cli` (a terminal wrapper
-around it) are. So this agent ships its own first-party Node host, `agents/delegate/delegate_host.mjs`,
-which wraps the SDK class in a newline-JSON stdio protocol this framework designed, not one it had to
-reverse-engineer — verified live against a real `@uipath/delegate-sdk@0.1.12` install with no token,
-confirming both the plain-public-install story and that the SDK's own event vocabulary
-(`session_start`/`thinking`/`message`/`tool_call`/`tool_result`/`error`/`done`/`step`) substantially
-matches the internal sibling's protocol.
-
-**No multi-generation transcript splitting.** The internal sibling reconstructs one `AssistantMessage`
-per backend round-trip from an `isStepStart` flag and a `turnUsages` array on the host's terminal
-message. Neither is present on `DelegateAgent.onEvent`'s payloads (confirmed absent by grepping the
-installed SDK bundle's string literals), so there is no reliable per-round-trip boundary signal at this
-API layer. `DelegateAgent` builds exactly ONE `AssistantMessage` per `communicate()` call instead — a
-deliberate simplification, not an oversight; richer segmentation can be added once a real boundary
-signal is confirmed against a live backend. `timing.close_window` still opens that one window from the
-turn's own start (there is nothing to tile from), so the head and tail both measure ~0.
-
-**Deferred, not ported speculatively.** The internal sibling's module docstring documents several
-hard-won failure-signature-specific recoveries: a Cloudflare WAF-block-page rewrite, an SSE-connect-
-timeout rewrite, session-conflict fresh-host recovery, and first-response stall-timeout+resend. None
-are ported here — porting a marker tuned to the internal sibling's own observed failures risks matching
-nothing (or the wrong thing) against the public SDK/backend's actual error surface. A crash still ends
-the turn correctly as a retryable `AgentCrashError`; it is just not specially diagnosed. Port these once
-the same failures are actually observed running this agent for real.
-
-**The process handle must be cleared on every path that leaves the host dead or dying** — EOF, a host
-`fatal` message, a timeout (both the top-of-loop pre-check AND a timeout elapsing while blocked inside
-`asyncio.wait_for`), cooperative stop, and `max_turns` exhaustion. A real bug shipped once during this
-agent's own development: a timeout elapsing mid-read fell through to the generic crash path instead of
-`TurnTimeoutError`, and left the process handle set, so the NEXT `communicate()` call reused a host with
-a `"send"` still nominally in flight instead of respawning — risking a stale response being consumed as
-the new turn's. `tests/test_delegate_agent.py`'s `test_timeout_elapsing_mid_read_still_raises_turn_timeout_error`
-pins the fix.
-
-**Skills mapping is deliberately NOT the shared `agents/_skills.py` resolver.** That resolver enumerates
-individual skill directories for a repeated `--skill <dir>`-style CLI argument (OpenCode/Pi's shape).
-The Delegate SDK's `bundledSkillsPath` wants exactly ONE parent directory whose children are skill
-folders — a genuinely different shape — so `_resolve_bundled_skills_path` mirrors the internal sibling's
-own mapping (`<plugin.path>/skills`, first plugin wins) instead of force-fitting the shared helper.
-
-**Registered unconditionally**, exactly like `codex`/`antigravity` — there is no conditional-registration
-gate keyed on an optional-dependency extra anywhere in this codebase; `opencode`/`pi` declare an EMPTY
-extra purely as packaging-metadata documentation, and `delegate` follows that same shape (no new pip
-package — Node/`@uipath/delegate-sdk` is the real prerequisite, resolved lazily in `start()`).
-
-**`environment` resolution needs org/tenant SLUGS, not just IDs — confirmed by reading the installed
-SDK's own bundle, not by guessing from its error message.** When `DELEGATE_BACKEND_URL` is absent and
-`DELEGATE_ENV` (-> `environment` init option) is used instead, the SDK resolves the backend URL from
-`organizationName`/`orgLogicalName` and `tenantName` fields on the `auth` object passed to
-`initialize()` — NOT from `ORG_SLUG`/`TENANT_SLUG` read off `process.env` directly, despite that being
-exactly what the SDK's own thrown error suggests ("set ORG_SLUG and TENANT_SLUG env vars alongside
-ORG_ID/TENANT_ID"). That advice describes the separate `@uipath/delegate-cli`/`delegate-stdio` wrapper's
-own env-var-driven bootstrapping, not the `DelegateAgent` class itself, which we drive directly. So
-`_build_init_options` forwards `ORG_SLUG`/`TENANT_SLUG` (when set) into `auth.organizationName`/
-`auth.tenantName` itself — confirmed against `node_modules/@uipath/delegate-sdk/dist/index.mjs`'s own
-`yie()` resolver function live in CI (`pyright`/tests can't catch this class of bug; it only surfaces
-against a real backend). `tests/test_delegate_agent.py::TestStart::test_org_and_tenant_slug_forwarded_into_auth`
-pins it. Passing `DELEGATE_BACKEND_URL` directly instead of `DELEGATE_ENV` skips this whole path.
-
-**Token usage comes from two getters called after `sendMessage()` resolves, not from any event or the
-resolved value itself — confirmed by reading the installed SDK's bundle, not by guessing.** A first
-pass guessed at a flat `usage` dict carried on the resolved `sendMessage()` value or on a forwarded
-event, tried several plausible snake_case/camelCase bucket-name spellings, and silently returned zero
-tokens every turn against a real backend (`tests/test_delegate_agent_live.py::test_delegate_live_token_usage_populated`
-failed live: `record.crashed is False`, real text/tool output, `token_usage=None`). Reading
-`node_modules/@uipath/delegate-sdk/dist/index.mjs` directly settled it: `sendMessage()` always resolves
-to a plain string (never an object), and no event this host forwards via `agent.onEvent()` ever carries
-a `usage` field — the SDK's own internal event vocabulary has no `"usage"` member (that name IS used
-internally, but only inside the SDK's own Zustand store reducer that updates its "Token usage" UI
-panel, never re-emitted through the public `onEvent` bus). The only way to reach it is
-`DelegateAgent.getLastTurnUsage(sessionId?)` (defaults to the just-used session), so
-`delegate_host.mjs`'s `handleSend` calls it — and `getSessionId()` — right after `sendMessage()`
-resolves, and attaches both to the `send_ok` message itself. That getter's shape, read straight off the
-SDK's own `setUsage` store action, is `{promptTokens, completionTokens, promptTokensCached,
-cacheCreationTokens, turnTokenUnits, contextBreakdown}` — `promptTokens` is the TOTAL input token count
-(OpenAI-style, cached + uncached), `promptTokensCached` the cache-READ subset of it, so
-`_parse_usage` computes `uncached_input_tokens = promptTokens - promptTokensCached`. This is now
-CONFIRMED, not a guess, so `_parse_usage`'s docstring no longer marks it `# UNVERIFIED`.
+`DelegateAgent` drives UiPath Autopilot's Delegate agent: reasoning in the UiPath backend, tools run
+locally. It spawns the host from the public `@uipath/delegate-stdio` npm package
+(`dist/delegate_stdio.mjs`), which pulls in `@uipath/delegate-sdk` and the interop binaries. The
+package README is the wire-protocol SSOT. `agents/delegate/package.json` sets the floor at `1.203.0`, the
+first release with the `auth` init option and the `usage` frame (UiPath/Autopilot#6656). The other frame
+shapes were confirmed against a live `1.202.1` transcript; the frame builders in
+`tests/_fixtures/golden_streams/delegate_fixtures.py` replay those shapes. (An earlier first-party
+`delegate_host.mjs` wrapper is gone; see git history.)
+
+**Auth goes to the host as the `auth` init option, not through its env.** `_env` reads the
+`DELEGATE_*` name first, then the bare `AUTH_TOKEN` / `TENANT_ID` / `ORG_ID` / `ORG_SLUG` /
+`TENANT_SLUG`. `DELEGATE_BACKEND_URL` → `backendUrl`, `DELEGATE_ENV` → `env`. We do not adopt the
+host's own names (`ORG_LOGICAL_NAME`, `BACKEND_URL`, …): bare names collide with other tooling, and CI
+secrets use the `DELEGATE_*` names. `_HOST_ENV_REMOVED` strips the host's names and the token from the
+host env, because the agent's shells inherit it (token leak to code under test) and a stray
+`BACKEND_URL` would reroute the host. A refresh source still writes the fresh token into the host's
+own `process.env.AUTH_TOKEN`. `_INIT_CONFIG_ERROR_HINT` names our variables beside the host's message,
+which names the host's own. `env=<slug>` without org/tenant
+slugs fails init.
+
+**`LLMGW_*` leaves the host env only when a token file is set.** Without a token file, the host's
+`selectTokenSource` uses that S2S pair to refresh, so stripping it always would end refresh after an
+hour. `_strip_redundant_gateway_creds` mirrors the host's lookup exactly
+(`DELEGATE_AUTH_TOKEN_FILE ?? AUTH_TOKEN_FILE`, path-delimiter split, blanks ignored).
+
+**Effort rides `sdk_options.effort`** — the same key as Claude Code, so one `-D` drives both.
+`get_sdk_options()` returns the whole `init` dict (so the reports show Model and Effort), with
+credentials redacted: `auth` → its field names, `backendUrl` → its host (`None` when `urlparse` cannot
+parse the URL: the orchestrator reads it while it persists `task.json`, so a raise there would lose
+the file and hide the original error). It is persisted in
+`task.json`.
+
+**`enableSkills` must be sent explicitly**: the host default is `false`, so `bundledSkillsPath` alone
+loads nothing. `_resolve_bundled_skills_path` maps `<plugin.path>/skills` (first plugin wins), not the
+shared `agents/_skills.py` resolver: the SDK wants ONE parent directory, not a list of skill dirs.
+
+**`max_turns` stays client-side.** The host's `maxSteps` does not stop the turn (live: `maxSteps: 2`
+ran 7 steps). The adapter counts calls from the event stream and abandons the host when call N+1
+opens; `len(turnUsages)` from the `result` then replaces the estimate as `num_turns`.
+
+**A call opens on its first text, or on its `usage` frame when it streams none — never on a tool
+event.** The host streams a call's text, flushes the call's `usage` frame, then runs the call's
+tools, and it runs a batched reply's tools one at a time: `tool_call`, `tool_result`, `tool_call`,
+`tool_result`. The adapter once opened call N+1 when every open tool had returned. With that order
+the open set empties after each tool, so every tool counted as a call: a downstream suite saw a turn
+with 16 round-trips and 96 tool calls cut at `max_turns: 80`. Now a `thinking` or `message` event
+opens a call unless the open call has already streamed text, and a `usage` frame opens one only for
+a call that streamed none. A crossing call that streams text is cut at that text, before any of its
+tools. A tool-only crossing call is cut at its own frame, and its first tool can start before the kill:
+the host starts a call's tools right after the frame, and the stream gives no earlier boundary. A host
+without `usage` frames counts only the calls that stream text; such a host is below the 1.203.0 floor,
+which is also the first release that takes the `auth` init option.
+Only `_TurnState.on_text` / `on_tool_event` / `on_usage_frame` / `on_result` write the count, so the rule
+lives in one place. `on_result` compares the live count with `len(turnUsages)` and warns when they
+differ: the order is read from the 1.203.1 host and SDK source, not from a live transcript, so a host
+release that reorders the frames shows up in the logs instead of silently moving the cap.
+
+**A cut turn keeps the usage of its finished calls.** A turn cut by `max_turns` or a stop gets no
+`result`, so the adapter sums the per-round-trip `usage` frames; the `result`'s total replaces the sum
+when it arrives. Ordering (from the host and SDK source, not a live transcript): call N's `usage` frame
+precedes every tool event of call N. A crossing call that streams text is cut at that text, so its usage
+is dropped; a tool-only crossing call is cut at its own frame, so its usage is booked. The adapter warns
+when a cut turn had finished calls but no usage.
+
+**One `AssistantMessage` per `communicate()`.** `isStepStart` and `turnUsages` would allow a
+per-round-trip split; today `isStepStart` only merges streamed deltas. `close_window` opens the one
+window at turn start, so head and tail measure ~0.
+
+**Known host failures are recategorized** in `_describe_host_error` / `_crash_on_host_error`, ported
+from the out-of-tree `delegate-sdk` adapter that drove the same host and backend:
+
+- **Cloudflare WAF block page** (`<title>Continue with UiPath Platform</title>` or "not available in
+  your country"): a WAF rule matched shell-like text in the request, not a geo/auth block. A retry
+  is blocked again, so the reason carries "content filter" (`AGENT_INVALID_OUTPUT`). Two markers,
+  because the host truncates near 50 KB and the country sentence sits after ~48 KB of CSS.
+- **SSE connect timeout**: no response headers, so the backend never started the turn. Rewritten to
+  "connection" (`AGENT_API_ERROR`, retried) with "timeout" removed, which would be `AGENT_TIMEOUT`
+  (not retried).
+- **Session conflict** ("A reply is already being generated", 409): `_session_id` is dropped so the
+  retry starts a new conversation, but only while no turn of that conversation has finished
+  (`_conversation_has_history`). On a later turn a new conversation would continue the task without
+  the earlier turns, and the row would still be graded as one trajectory. So the retry keeps the
+  session. The 409 is short-lived: the backend cancels the reply when the killed host's stream
+  disconnects, then releases its turn claim, so the 5 s and 10 s `AGENT_CRASH` retries usually
+  succeed. A claim it fails to release expires after `TURN_CLAIM_TTL_SECONDS` (900 s), and the task
+  ends as an error. A `session_id` pinned in config counts as history, and dropping it would not
+  help anyway: each respawn sends it in `init`, and the SDK falls back to it when `send` carries no
+  `sessionId` (`sessionId || this.options.sessionId` in the SDK's `sendMessage`).
+
+Not ported: stall-timeout+resend and the S2S token-file refresher. Port them when the need shows.
+
+**No crash reason carries the stderr tail.** Categorization matches substrings of the reason, and the
+tail holds incidental text (the sandbox path, so the task id). In build 13599116 "guardrail" in a task
+path turned a transient error into non-retryable `AGENT_INVALID_OUTPUT`. `_log_stderr_tail` logs the
+tail at WARNING instead.
+
+**Only an init error that a retry cannot fix is non-retryable.** `AgentConfigError` only on
+`_INIT_CONFIG_ERROR_MARKERS` (auth missing/rejected, missing slugs, unknown env) or on a whole-word
+401/403. A bare `"401"` substring also matched ports and GUIDs, such as `127.0.0.1:54013`, and ended a
+transient failure with no retry. Any other init error and the 60 s init deadline raise a retryable
+`AgentCrashError`. The shared categorizer matches status codes as whole words too
+(`errors/categorization.py::_mentions_status`): the `AgentCrashError` goes through `categorize_error`,
+whose raw `"401"` substring made the same port and GUID cases a non-retryable auth error. A mid-run
+respawn keeps the same classification: it once made every `AgentConfigError` retryable, so an expired
+token ended the task at `start()` but was retried later.
+
+**Clear the process handle on every path that leaves the host dead or dying** — EOF, `error` frame,
+timeout (pre-check AND mid-`wait_for`), stop, `max_turns`. A shipped bug left it set after a mid-read
+timeout, so the next turn reused a host with a `send` in flight
+(`test_timeout_elapsing_mid_read_still_raises_turn_timeout_error` pins it). `_force_kill_host` clears
+the handle before its reap. The adapter never reuses a host after a failed `send`, so no late event
+leaks into the retry.
+
+**Registered unconditionally**, like `codex`/`antigravity`; its extra is empty. Node and
+`@uipath/delegate-stdio` are the real prerequisite, resolved lazily in `start()`.
+
+**Host resolution: `DELEGATE_STDIO_PATH`, then local installs, then a global one.** Local means
+the cwd, its ancestors and `agents/delegate/`; a local install wins, as in Node. A global install
+is found through the package's `delegate-stdio` bin shim on `PATH`, not `npm root -g`, so the
+resolver runs no npm subprocess. On POSIX the shim is a symlink to the bundle. On Windows it is a
+`.cmd` file beside the prefix's `node_modules`. We never execute the `.cmd`: that would make
+`cmd.exe` parse the arguments again. The home probe and `DELEGATE_SDK_NODE_MODULES` are gone. The
+global search covers "install once, run from anywhere", and the root override only repeated the
+path override. The explicit path stays for CI: a source build has no install root, and an install
+in a temporary directory for each run keeps the runs on a shared host on their own versions.
+
+**Windows path length**: an install root that pushes the interop binary path past 260 characters makes
+the spawn fail with `ENOENT`. The docs tell users to install into a short path.
 
 ### Delegate agent pricing
 
@@ -913,16 +972,14 @@ un-mirrored gap, with the same caveat comment). The two tables can price the SAM
 differently until someone reconciles them — a known, pre-existing state, not something this port
 introduced or should silently "fix" by picking one number over the other.
 
-### Delegate agent golden-master and timing-identity coverage
-
-`AgentKind.DELEGATE` is excluded from `tests/test_agent_golden_master.py`'s `_NO_GOLDEN_COVERAGE`
-allowlist rather than given fixture scenarios: a golden snapshot pins a byte-identical `TurnRecord`
-for a scripted event stream, and every event field name below the confirmed `type`/`content`/
-`toolName`/`sessionId`/`model`/`usage` set is a guess (see this file's own "Delegate agent" section
-above and `delegate_agent.py`'s module docstring). Snapshotting a guess would make it look verified.
-Add real scenarios once the fields are confirmed against a live backend.
-
-This does NOT extend to `tests/test_timing_identity_contract.py::test_delegate_buckets_tile_the_turn`,
-which IS a real, unexempted case: the ms-exact four-bucket identity depends only on this agent's own
-`close_window`/`_finalize_turn` code (a single window per turn, opened at `_TurnState.__init__` and
-closed at finalize), never on the SDK's field names, so there is nothing unverified for it to guess at.
+### Delegate agent golden-master coverage
+
+`tests/_fixtures/golden_streams/delegate_fixtures.py` scripts the host's stdout frames for six
+scenarios: a tool call, a `max_turns` cut, an `error`-frame crash, an orphaned tool, a tool-only call
+that crosses the cap, and a batched reply whose tools run one at a time. The `event`,
+`result` and `error` frames mirror the live `1.202.1` transcript. The `usage` frame and the `result`
+fields new in `1.203.0` follow the host source of UiPath/Autopilot#6656 (`src/usage.ts` and
+`handleSend`): every usage object carries all four buckets with `input_tokens` exclusive of cache,
+the frames flush before each event and before the `result` or `error`, and `result.usage` is the sum
+of `turnUsages`. No live `1.203.0` transcript has been compared yet. If one differs, change the
+builders and regenerate the snapshots with `GOLDEN_REGEN=1`.

coder_eval-0.12.13/.claude/notes/context-window.md

@@ -0,0 +1,170 @@
+# Capping the agent's context window
+
+Analysis and design for `agent.context_window`: one harness-neutral field that caps the
+context window an agent works within, set per agent config or per experiment variant.
+
+## Problem and evidence
+
+A benchmark of Claude Code on Sonnet 5 (Bedrock, 1M-token window) let runs grow to
+250k-480k tokens of context. Above about 150k tokens each extended-thinking step got
+about 10x slower, so most runs hit the 2-hour task limit. The operator wants to measure
+pass rate and cost at several context caps, for example one variant at 200k and one at
+400k, on the same tasks.
+
+Before this change no harness-neutral way existed:
+
+- **claude-code**: `claude_settings` (`--settings`) could carry `autoCompactWindow`. That
+  works for one harness only, and when `claude_settings` is a file path a variant cannot
+  merge one key into it.
+- **codex**: `CodexAgentConfig` has no pass-through. `CodexAgent._build_thread_options`
+  builds the thread `config` dict itself, so no value reached Codex.
+- **antigravity, opencode, pi, delegate**: no knob is wired.
+
+## How each harness controls its window
+
+### claude-code
+
+Verified in the CLI the SDK spawns. `claude-agent-sdk` runs its **bundled** CLI before it
+looks for `claude` on `PATH` (`SubprocessCLITransport._find_cli`), so the CLI that runs is
+the one the SDK wheel bundles, not the one the agent image installs with npm:
+
+| coder_eval | claude-agent-sdk (uv.lock) | bundled CLI | image npm CLI |
+|---|---|---|---|
+| 0.12.1 | 0.2.124 | 2.1.216 | 2.1.177 |
+| 0.12.12 | 0.2.159 | 2.1.281 | 2.1.281 |
+
+`environment_info.claude_code_cli` records `claude -v` from `PATH`, so a 0.12.1 run
+records 2.1.177 while 2.1.216 ran. That is a separate defect and is not fixed here.
+
+Three inputs set the auto-compact window. Both bundled CLIs (2.1.216 and 2.1.281) contain
+the same resolution function. The highest precedence comes first:
+
+1. `CLAUDE_CODE_AUTO_COMPACT_WINDOW` environment variable: a plain token count. It
+   outranks the flag and every settings scope.
+2. `--autocompact <auto|tokens>`: added in 2.1.221 (CLI reference). It is absent from
+   2.1.216 as a parsed option.
+3. `autoCompactWindow` settings key. The settings schema is
+   `int().min(100000).max(1000000).optional().catch(undefined)`, so an out-of-range
+   value is **silently dropped**.
+
+The effective window is `min(model context window, configured value)`. Auto-compaction
+starts when the context approaches that window.
+
+### codex
+
+Verified in the `openai-codex-cli-bin` 0.156.1 binary (the pin) and its source at tag
+`rust-v0.156.1`:
+
+- `model_context_window` ("Size of the context window for the model, in tokens"):
+  `with_config_overrides` sets the model's `context_window` to
+  `min(value, max_context_window)`. Three things follow from it. The default
+  auto-compact limit becomes 90% of it. The hard cap, which forces compaction, becomes
+  `context_window * effective_context_window_percent / 100`. The remaining-window figure
+  the model is shown also changes.
+- `model_auto_compact_token_limit` ("Token usage threshold triggering auto-compaction"):
+  the trigger only, clamped to 90% of the resolved window. It does not change the window.
+- `model_auto_compact_token_limit_scope` (`total` by default, or `body_after_prefix`) and
+  `model_post_turn_compact_threshold_percent` (turn-end compaction, off by default) tune
+  when compaction starts. They are not caps.
+
+The thread `config` dict `thread_start` takes is the same override surface that
+coder_eval already uses for `enabled_tools` and `model_providers`.
+
+### antigravity, opencode, pi, delegate
+
+None of them has a context-window knob wired in coder_eval. They are out of scope, and
+they reject the field.
+
+## Options considered
+
+**Per-harness pass-through** (a Codex `config` pass-through next to `claude_settings`).
+It is flexible, but a variant would have to name a different key per harness, and the
+recorded value would mean something different on each. It also opens a free-form
+pass-through on Codex with no denylist. Rejected.
+
+**A field under `run_limits`.** Run limits are caps that the orchestrator or the agent
+enforces on a run (turns, time, tokens, USD), and the parity table lists them. A context
+window is not a spend cap. It is a harness setting that changes what the agent does, and
+it depends on the agent type, which `run_limits` does not know. The guide also says that
+`agent:` holds no run-time caps. Rejected.
+
+**A field on the agent config.** One name, accepted on a type-less config so that
+experiment defaults and variants can set it, and validated against the concrete type
+when the config resolves. **Chosen.**
+
+Naming: `context_window`, in tokens. It names the quantity the operator caps, matches
+Codex's `model_context_window`, and avoids "auto-compact", which describes a mechanism of
+one harness. `max_context_tokens` was considered, but it reads as a per-request input
+limit, which neither harness enforces.
+
+## Recommended design
+
+`BaseAgentConfig.context_window: int | None` (default `None`, `gt=0`).
+
+- Each config class declares `_context_window_range: ClassVar[tuple[int, int | None] |
+  None]`. `None` (the base default) means that the harness cannot apply the field.
+  `ClaudeCodeAgentConfig` declares `(100_000, 1_000_000)` and `CodexAgentConfig` declares
+  `(32_768, None)`.
+- `check_context_window_supported` (model validator) rejects an unsupported type or an
+  out-of-range value as soon as the type is known. A plugin agent inherits `None`, so it
+  also rejects the field until it opts in.
+- The field uses the default `replace` merge strategy. It is set at any of the five
+  layers, and with `-D agent.context_window=N`.
+
+Per-harness mapping:
+
+- **claude-code**: `CLAUDE_CODE_AUTO_COMPACT_WINDOW=<N>` in the SDK `env`. The
+  environment variable was chosen over the settings key and the flag for four reasons.
+  It has the highest precedence, so host, project and `claude_settings` values cannot
+  override the cap. It works whether `claude_settings` is a dict or a file path. Both
+  bundled CLIs read it, which the flag does not. `extra_args` stays framework-owned.
+  `claude_settings.autoCompactWindow` together with `context_window` is a load error,
+  because the environment variable would silently override it.
+- **codex**: `model_context_window: <N>` in the thread `config`. No trigger key is set,
+  so Codex compacts at its default 90% of the window.
+
+## Validation
+
+- `gt=0` on every config, so a type-less config rejects nonsense as well.
+- claude-code: 100000-1000000, the CLI's documented range. Outside that range the CLI
+  drops a settings value silently, so coder_eval enforces the range itself.
+- codex: 32768 and up. Codex clamps the value to the model's catalog maximum, and the
+  floor rejects a window too small to work in, such as a YAML `yes` coerced to 1.
+- every other type: rejected at load, naming the supported types. A variant that
+  switches `type` to an unsupported harness fails when the experiment resolves.
+
+## Recording
+
+The resolved agent config is persisted as `agent_config` on every task row (`run.json`
+and `task.json`). `agent.context_window` therefore appears per task, and config lineage
+records which layer set it (for example, the variant). The experiment report groups by
+variant, so a cap per variant groups without new report code. On claude-code the
+`sdk_options` dump also shows the environment variable in `env`.
+
+## Open questions
+
+- **A silent clamp to the model's window.** Neither harness reports that the model's
+  window lowered the cap. A cap above the model's window runs at the model's window.
+  Recording the effective window would need a per-model catalog in coder_eval.
+- **Trigger points differ.** Codex compacts at 90% of the window. Claude Code compacts
+  when the context approaches the window, minus an internal buffer. Equal caps therefore
+  do not compact at exactly the same token count. A separate harness-neutral trigger
+  field is possible, but it is not added until a benchmark needs it.
+- **A host `CLAUDE_CODE_AUTO_COMPACT_WINDOW` and uncapped runs.** Decided: the SDK starts
+  the CLI with `os.environ` underneath `options.env`, so when the field is unset the
+  claude-code agent blanks a host value. A run recorded as uncapped is never capped by
+  the host. This changes every claude-code run on a host that sets the variable,
+  including runs that never set `context_window`. Codex has no equivalent: an unset
+  field leaves a host `model_context_window` in `$CODEX_HOME/config.toml` in effect.
+- **The recorded CLI version is wrong.** See the version table above:
+  `environment_info.claude_code_cli` should record the bundled CLI version.
+- **Other harnesses.** Pi can lower a model's window with
+  `providers.<provider>.modelOverrides.<model>.contextWindow` in `models.json`. A mapping
+  was written and then reverted (d1757a60), because no live run backed it and no task
+  uses `type: pi`; it comes back with a live test. OpenCode (`limit.context` and
+  `limit.input` in provider config) needs a `limit.output` coder_eval does not know, and
+  an unpinned CLI. Antigravity's `CompactionConfig(token_threshold=N)` only moves the
+  trigger. Delegate keeps its history in the backend. Each needs its own verification
+  before it opts in.
+- **How often a run compacted is not recorded.** A cap A/B cannot yet tell "cap never
+  reached" from "compacted N times" from the task row alone.

{coder_eval-0.12.11 → coder_eval-0.12.13}/.claude/notes/lint-rules.md

@@ -845,7 +845,7 @@ assertion for each. A defaulted `AgentStartEvent` lands ~365 days before the clo
 first window, so the head blows any sane UPPER bound by the whole offset. A defaulted
 `AgentEndEvent` lands ~365 days BEFORE its own last message, so `decompose_turn` clamps the
 negative to `0.0` — "measured, and instant", which passes an upper bound. Only a strict
-`> 0.0` catches it. It holds on all three clocked harnesses because real work separates a
+`> 0.0` catches it. It holds on every clocked harness because real work separates a
 turn's last flush from its end event. The margin is smallest on antigravity, which holds
 its process across turns: it measures 0.007-0.035 ms there, 7-35 ticks of the 1 us
 resolution that both `datetime` and `time.monotonic()` have on Linux, macOS and Windows.

{coder_eval-0.12.11 → coder_eval-0.12.13}/.claude/notes/timing.md

@@ -16,6 +16,11 @@ removes:
   transition or an NTP step inside a turn lands directly in a generation window — an
   hour-long jump in a millisecond field. Nightly runs start at 04:18 and run for hours,
   so it is reachable rather than theoretical. A monotonic-derived stamp cannot express it.
+- Delegate stamped its one window and its tool spans with raw `datetime.now()` but measured
+  `duration_seconds` with `time.monotonic()`. On Windows the wall clock moved 0.24-0.37 ms
+  further than the monotonic one inside a sub-millisecond turn, so the tail alone came out
+  longer than the whole turn and `test_delegate_golden[a_tool_call]` failed at random. One
+  window has no inter-window drift to correct, but it still shares the turn's duration.
 
 It is an EXTRACTION, not an invention: antigravity already captured this exact pair at
 the top of `communicate` and simply did not use it for later stamps.
@@ -205,7 +210,7 @@ Magnitudes are only real where a scripted clock makes them real, so each case dr
 harness's own reducer with a hand-moved clock and feeds its output through a real
 `EventCollector`, the seam production uses to compute the head and the tail. The module
 reuses the three clock-injection idioms the per-harness suites already use rather than
-inventing a fourth: an injected `TurnClock` stand-in (pi, antigravity, claude-code), a
+inventing a fourth: an injected `TurnClock` stand-in (pi, antigravity, claude-code, delegate), a
 `datetime` subclass patched onto the module (opencode), and scripted SDK epoch-millisecond
 stamps (codex). OpenCode needs a `datetime` SUBCLASS rather than a stub because the
 reducer also calls `datetime.fromtimestamp` through the same module global to convert the

{coder_eval-0.12.11 → coder_eval-0.12.13}/.env.example

@@ -89,36 +89,33 @@ LOG_TO_FILE=false  # Set to true to enable file logging
 # CODER_EVAL_REMEDIATE_HOME_PLUGINS=0
 
 # Delegate agent settings (UiPath Autopilot's Delegate agent, requires Node.js
-# on PATH plus `npm install @uipath/delegate-sdk` — a genuinely public package,
-# no token needed). See docs/agents/DELEGATE.md.
+# on PATH plus `npm install -g @uipath/delegate-stdio`, a release that accepts the
+# `auth` init option). coder_eval finds the install itself. See docs/agents/DELEGATE.md.
 #
-# DELEGATE_SDK_NODE_MODULES / DELEGATE_SDK_PATH — override the auto-located
-# install (ancestor-walk from cwd, plus home); set at most one.
-# DELEGATE_SDK_NODE_MODULES="/path/to/install/root"
-# DELEGATE_SDK_PATH="/path/to/node_modules/@uipath/delegate-sdk/dist/index.mjs"
-#
-# DELEGATE_ENV — cloud environment slug (alpha/staging/production/localhost),
-# used to derive the backend URL from the auth record's org/tenant slugs.
+# DELEGATE_ENV — cloud environment slug (alpha/staging/production),
+# used to derive the backend URL from the org/tenant slugs.
 # DELEGATE_ENV="alpha"
 #
 # DELEGATE_BACKEND_URL — pin the full agent-service URL directly instead
 # (wins over DELEGATE_ENV when both are set).
 # DELEGATE_BACKEND_URL="https://cloud.uipath.com/<org>/<tenant>/delegate_"
 #
-# Auth: either the token set below, or a prior `delegate-sdk`/`delegate-cli`
-# login that wrote ~/.aria/sdk-auth.json (read by the Node host itself, not
-# by coder_eval). Each var also accepts a DELEGATE_-namespaced spelling
-# (DELEGATE_AUTH_TOKEN, DELEGATE_TENANT_ID, ...), checked first -- prefer that
-# spelling in a shared environment, since the bare names collide with what
-# other tooling (npm, Vault, Terraform) commonly exports.
-# AUTH_TOKEN="..."
-# TENANT_ID="..."
-# ORG_ID="..."
+# Auth: either the token set below, or a prior `delegate-cli login`
+# that wrote ~/.aria/sdk-auth.json (read by the Node host itself, not
+# by coder_eval). coder_eval sends these values to the host as its `auth`
+# init option, so the token stays out of the environment that the agent's
+# shell tools inherit. Each var also accepts the bare spelling (AUTH_TOKEN,
+# TENANT_ID, ORG_ID, ORG_SLUG, TENANT_SLUG) as a fallback -- prefer the
+# DELEGATE_ spelling in a shared environment, since the bare names collide
+# with what other tooling (npm, Vault, Terraform) commonly exports.
+# DELEGATE_AUTH_TOKEN="..."
+# DELEGATE_TENANT_ID="..."
+# DELEGATE_ORG_ID="..."
 # With DELEGATE_ENV (not DELEGATE_BACKEND_URL) also set these two -- the
 # human-readable org/tenant names, not the GUIDs above -- or init fails with
-# "--env alpha needs org/tenant slugs". See docs/agents/DELEGATE.md.
-# ORG_SLUG="..."
-# TENANT_SLUG="..."
+# 'env="alpha" requires org/tenant slugs'. See docs/agents/DELEGATE.md.
+# DELEGATE_ORG_SLUG="..."
+# DELEGATE_TENANT_SLUG="..."
 
 # Usage Telemetry (anonymous; on by default).
 # Disable entirely: