coder-eval 0.12.7__tar.gz → 0.12.9__tar.gz

{coder_eval-0.12.7 → coder_eval-0.12.9}/.claude/harness-candidates.md

@@ -1022,3 +1022,13 @@ re-derive from scratch.
   have, and it is a recurring class ONLY within one file so far — worth promoting to a rule
   if a second subprocess-driven agent repeats it. Caught in: final cross-phase review,
   delegate agent port.
+
+## From 2026-09-26 Antigravity SDK 0.1.18 usage regression
+
+- [ ] Runtime guard: fail an Antigravity turn that ran MODEL steps but booked zero
+  tokens, as OpenCode's `require_token_telemetry` does. SDK 0.1.18 deprecated
+  `Step.usage_metadata` (usage moved to `conversation.total_usage`), so every
+  Antigravity turn from `3aa2db30` on reported no tokens and no cost, and `api_calls`
+  stayed at 1 so `max_turns` never tripped — all silently, with `make verify` green.
+  A static rule cannot see an SDK field go dead; a live-telemetry smoke (one real turn,
+  assert `token_usage` is non-empty) in the harness-bump checklist would have.

{coder_eval-0.12.7 → coder_eval-0.12.9}/.claude/notes/lint-rules.md

@@ -667,7 +667,7 @@ shipped. An id is a permanent documentation anchor: a suppression comment carryi
 an older branch, review or commit message must never start meaning something new. CE023 is
 retired the same way, after the rule was deleted with the package it guarded.
 
-Claim 068 next, and note 065 IS TAKEN without being in `ALL_RULES`: doc-surface and
+Claim 069 next, and note 065 IS TAKEN without being in `ALL_RULES`: doc-surface and
 whole-tree rules are `@pytest.mark.lint` classes in `tests/test_custom_lint.py` rather than
 `BaseRule`s, so `runner.py`'s uniqueness assert cannot see them. Enumerating them in a
 comment is how that note fell behind CE044, so grep instead:

{coder_eval-0.12.7 → coder_eval-0.12.9}/.claude/notes/permissions.md

@@ -29,10 +29,6 @@ read-write throwaway copy; when it was bind-mounted `:ro` the chmod returned ERO
 no tmpfs mask and has not been one since: see
 [isolation.md](isolation.md) § Why the framework mounts are writable copies.
 
-What the window does NOT hide is the task DEFINITION. `task.yaml` is also staged at
-`/work/input` for the in-container orchestrator and that mount is untouched, so hiding the
-criteria from the agent remains a separate, unsolved problem.
-
 Criteria address reference files with the `$REFERENCE_DIR` token (same resolver as
 `$TASK_DIR`) and the `REFERENCE_DIR` env var for `run_command`; `reference_comparison` names
 one file via `reference_file`.
@@ -64,10 +60,6 @@ criteria and judges run. The task directory is shielded by the same window: unde
 it is mounted as a throwaway COPY at a fixed container path, which is what makes it
 chmod-able without touching the user's checked-out `tasks/` tree.
 
-It shields grading MATERIAL that happens to live in the task directory (a `reference/`
-subdirectory, fixtures), not the task DEFINITION: `task.yaml` is separately staged at
-`/work/input`, which the agent can still read.
-
 ### Why a stack and not a refcount
 
 Windows nest with *different* modes, so what an exit has to restore is the mode of the

{coder_eval-0.12.7 → coder_eval-0.12.9}/.claude/notes/timing.md

@@ -401,8 +401,8 @@ and the tail are already computed.
 
 A reducer's only remaining timing decision is where its window opens, which is the one
 genuinely harness-shaped part: two interleave a tool into a single window outright
-(Antigravity, whose Step for the tool arrives and only a later `usage_metadata` Step cuts
-the message, and Codex, whose `_flush_message` window extends to the last item's
+(Antigravity, whose Step for the tool arrives and only a later rise in the conversation's
+billed usage cuts the message, and Codex, whose `_flush_message` window extends to the last item's
 `completed_at_ms`) while the other three tile the turn contiguously, so a call open at a
 boundary runs inside two windows. Central subtraction handles both without either reducer
 knowing which it is.

{coder_eval-0.12.7 → coder_eval-0.12.9}/.github/workflows/pr-checks.yml

@@ -123,7 +123,7 @@ jobs:
 
       # PHASE 3: Security scanning
       - name: Security - Dependency vulnerabilities (pip-audit)
-        run: .venv/bin/pip-audit --desc --skip-editable --ignore-vuln CVE-2026-4539 --ignore-vuln CVE-2026-3219 --ignore-vuln PYSEC-2025-183 # pygments 2.19.2 ReDoS + pip 26.0.1 tar/ZIP ambiguity + pyjwt 2.12.1 weak-encryption (disputed by supplier; key length is application-chosen); no fixes available on PyPI yet — revisit quarterly
+        run: .venv/bin/pip-audit --desc --skip-editable --ignore-vuln CVE-2026-4539 --ignore-vuln CVE-2026-3219 --ignore-vuln CVE-2026-49265 --ignore-vuln CVE-2026-49264 # pygments 2.19.2 ReDoS + pip 26.0.1 tar/ZIP ambiguity; no fixes available on PyPI yet — revisit quarterly. oauthlib 3.3.1 PKCE timing side channel + RevocationEndpoint JSONP callback injection: both server-side OAuth endpoints, unused here (transitive via azure-monitor exporter -> msrest); fix 4.0.0 is a major bump still inside the safe-chain minimum package age — revisit next month
 
       - name: Security - OSV vulnerability scan (osv-scanner)
         # Complements pip-audit: pip-audit queries the PyPI advisory DB;

{coder_eval-0.12.7 → coder_eval-0.12.9}/CHANGELOG.md

@@ -2,6 +2,88 @@
 
 <!-- version list -->
 
+## v0.12.9 (2026-09-29)
+
+### Bug Fixes
+
+- **docker**: Mount plugins at /coder_eval/plugins/<i> and point the staged task there
+  ([#206](https://github.com/UiPath/coder_eval/pull/206),
+  [`03808c0`](https://github.com/UiPath/coder_eval/commit/03808c084cb28d72e5723a2e9fc30c714f398d14))
+
+- **docker**: Mount plugins at /work/plugins/<i> and point the staged task there
+  ([#206](https://github.com/UiPath/coder_eval/pull/206),
+  [`03808c0`](https://github.com/UiPath/coder_eval/commit/03808c084cb28d72e5723a2e9fc30c714f398d14))
+
+### Chores
+
+- **deps**: Bump pyjwt to 2.15.1 and ignore oauthlib CVE-2026-49265
+  ([#206](https://github.com/UiPath/coder_eval/pull/206),
+  [`03808c0`](https://github.com/UiPath/coder_eval/commit/03808c084cb28d72e5723a2e9fc30c714f398d14))
+
+- **deps**: Ignore oauthlib CVE-2026-49264 and drop the stale pyjwt ignore
+  ([#206](https://github.com/UiPath/coder_eval/pull/206),
+  [`03808c0`](https://github.com/UiPath/coder_eval/commit/03808c084cb28d72e5723a2e9fc30c714f398d14))
+
+### Refactoring
+
+- **docker**: Mount plugins under /work/plugins, not /coder_eval/plugins
+  ([#206](https://github.com/UiPath/coder_eval/pull/206),
+  [`03808c0`](https://github.com/UiPath/coder_eval/commit/03808c084cb28d72e5723a2e9fc30c714f398d14))
+
+
+## v0.12.8 (2026-09-29)
+
+### Bug Fixes
+
+- Also delete context.json in-container (source_yaml leaked criteria)
+  ([#179](https://github.com/UiPath/coder_eval/pull/179),
+  [`ddf63e9`](https://github.com/UiPath/coder_eval/commit/ddf63e9b356a6c40a83ca5771a3ffd3a1e8dfd68))
+
+- **anti-cheat**: Apply multi-model review findings M1/M2/M3 + doc drift
+  ([#179](https://github.com/UiPath/coder_eval/pull/179),
+  [`ddf63e9`](https://github.com/UiPath/coder_eval/commit/ddf63e9b356a6c40a83ca5771a3ffd3a1e8dfd68))
+
+- **anti-cheat**: Apply review Low findings L1/L2/L3
+  ([#179](https://github.com/UiPath/coder_eval/pull/179),
+  [`ddf63e9`](https://github.com/UiPath/coder_eval/commit/ddf63e9b356a6c40a83ca5771a3ffd3a1e8dfd68))
+
+- **antigravity**: Read token usage from the conversation, not the Step
+  ([#201](https://github.com/UiPath/coder_eval/pull/201),
+  [`101bb5c`](https://github.com/UiPath/coder_eval/commit/101bb5c85c5af97f8027388611dd70e7e7e0ecc8))
+
+- **codex**: Record provider error notifications on the turn
+  ([#203](https://github.com/UiPath/coder_eval/pull/203),
+  [`894f742`](https://github.com/UiPath/coder_eval/commit/894f74282c8f32918cc995fb4b2517f8984c4ed2))
+
+- **docker**: Close two driver:docker channels that leaked grading material to the agent
+  ([#179](https://github.com/UiPath/coder_eval/pull/179),
+  [`ddf63e9`](https://github.com/UiPath/coder_eval/commit/ddf63e9b356a6c40a83ca5771a3ffd3a1e8dfd68))
+
+- **simulation**: Send the simulated user's utterance, not its tagged transcript
+  ([#204](https://github.com/UiPath/coder_eval/pull/204),
+  [`ffb9a81`](https://github.com/UiPath/coder_eval/commit/ffb9a8198a1af73904fabb0c8b3251a978351ab0))
+
+### Documentation
+
+- **harness**: Log deferred Low findings from anti-cheat review
+  ([#179](https://github.com/UiPath/coder_eval/pull/179),
+  [`ddf63e9`](https://github.com/UiPath/coder_eval/commit/ddf63e9b356a6c40a83ca5771a3ffd3a1e8dfd68))
+
+### Features
+
+- **lint**: 3/3 — CE065 flags eval material inside a skill dir
+  ([#179](https://github.com/UiPath/coder_eval/pull/179),
+  [`ddf63e9`](https://github.com/UiPath/coder_eval/commit/ddf63e9b356a6c40a83ca5771a3ffd3a1e8dfd68))
+
+- **sandbox**: 1/3 — delete staged task.yaml in-container after load (anti-cheat)
+  ([#179](https://github.com/UiPath/coder_eval/pull/179),
+  [`ddf63e9`](https://github.com/UiPath/coder_eval/commit/ddf63e9b356a6c40a83ca5771a3ffd3a1e8dfd68))
+
+- **sandbox**: 2/3 — allowlist tmpfs-mask of auto-mounted plugin trees
+  ([#179](https://github.com/UiPath/coder_eval/pull/179),
+  [`ddf63e9`](https://github.com/UiPath/coder_eval/commit/ddf63e9b356a6c40a83ca5771a3ffd3a1e8dfd68))
+
+
 ## v0.12.7 (2026-09-25)
 
 ### Bug Fixes

{coder_eval-0.12.7 → coder_eval-0.12.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: coder-eval
-Version: 0.12.7
+Version: 0.12.9
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode, Pi, Delegate) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval

{coder_eval-0.12.7 → coder_eval-0.12.9}/action.yml

@@ -25,7 +25,7 @@ inputs:
   version:
     description: coder-eval version to install from PyPI, or "local" to install from the action checkout
     required: false
-    default: "0.12.7"   # <-- kept in sync with releases by release.yml
+    default: "0.12.9"   # <-- kept in sync with releases by release.yml
   extras:
     description: >-
       Comma-separated coder-eval extras (`codex`, `antigravity,litellm`), composed

{coder_eval-0.12.7 → coder_eval-0.12.9}/docs/DOCKER_ISOLATION.md

@@ -339,6 +339,56 @@ If the task declares a `reference:` block, a throwaway **copy** of its directory
 
 See [Reference Solutions](TASK_DEFINITION_GUIDE.md#reference-solutions).
 
+### Two more passive-read anti-cheat blocks
+
+These are `driver: docker` only. `driver: tempdir` shares the host uid and has no
+filesystem isolation, so neither applies there (nor can — there is nothing to
+mask). Both are defense-in-depth passive-read blocks, consistent with the
+reference window's posture above; neither contains an adversarial agent.
+
+- **The staged grading inputs are deleted after load.** The host stages the
+  post-override `TaskDefinition` (with `success_criteria`) at `/work/input/task.yaml`
+  **and** a `context.json` whose `source_yaml` is the raw task text — criteria
+  verbatim, both at the top level and inside every `config_lineage` entry — for the
+  in-container orchestrator to load once at startup. The agent runs in the same
+  container, so leaving *either* readable would hand it the grading answer key
+  (deleting only `task.yaml` leaves the identical criteria one file over in
+  `context.json`). The in-container entry point deletes **both** immediately after
+  they are consumed — `context.json` is parsed into memory in the command body and
+  `task.yaml` by `load_task`, both before the delete (gated on
+  `CODER_EVAL_IN_CONTAINER`). They are read exactly once — grading reads criteria
+  from the in-memory task, never from disk. The `/work/input` mount is therefore
+  read-write (a `:ro` mount rejects `rm` with EROFS). `prior.json` is kept: it is
+  read later on the regrade path, and a regrade runs no agent so it is not a leak.
+
+- **Plugins mount at a fixed container path.** The `i`-th `agent.plugins[]` entry
+  is resolved on the host (a relative path against the task YAML's directory, `$VAR`
+  and `~` expanded) and mounted `:ro` at `/work/plugins/<i>`. The task YAML
+  staged into the container is rewritten to point at that path, so the in-container
+  agent loads the directory the host mounted, whatever form the authored path took
+  and whatever the container's cwd. (Before, the staged YAML kept the authored string:
+  a relative or `$VAR` path the container could not resolve loaded no skill, with only
+  a warning.) An entry that does not resolve to a host directory is neither mounted
+  nor rewritten. It sits under `/work`, so a `sandbox.docker.extra_mounts` destination
+  cannot shadow it (those are refused anywhere under `/work/`).
+
+- **Auto-mounted plugin trees are default-deny masked.** An `agent.plugins[].path`
+  (at `/work/plugins/<i>`) or a `TemplateDirSource.path` that is itself a plugin
+  root (at its host path) is auto-mounted `:ro` so the plugin loads. Eval material colocated under that tree
+  as siblings of the skills dir — sibling task YAMLs, reference solutions, test
+  fixtures — would otherwise be readable. So the runner keeps the whole root
+  mounted but layers an empty `--tmpfs` over every child dir OUTSIDE the keep-set
+  (`.claude-plugin` + the manifest-declared skill dirs). Everything that is not
+  the plugin surface is masked by default, so an unknown or new eval layout can
+  never leak; `tests/`, `node_modules/`, and reference solutions are masked for
+  free. A root agent cannot `umount` a tmpfs (`CAP_SYS_ADMIN` is not in Docker's
+  default set), so this mask is *stronger* than the mode-000 reference window. Two
+  residuals the mask cannot cover — an eval def or reference COLOCATED inside a
+  skill dir (masking it would hide the skill), and a `task_id:` YAML **file** loose
+  at the plugin root (a tmpfs masks a directory, not a single file) — are caught by
+  lint rule CE068 (keep eval material out of skill dirs and off the plugin root;
+  put it under a sibling `tests/`).
+
 Inside the container, the entrypoint invokes `coder-eval _run-task-internal` (hidden subcommand), which loads the staged YAML + context, runs the standard in-process Orchestrator (driver auto-coerced back to `tempdir`), and writes `task.json` to the output mount. Host reads it and feeds the existing aggregation pipeline.
 
 A `result_kind` discriminator on `CriterionResult` ensures `ClassificationCriterionResult` subclasses survive the JSON round-trip — without it, host-side aggregation would silently lose `observed_label`/`expected_label`.

{coder_eval-0.12.7 → coder_eval-0.12.9}/docs/REPORT_SCHEMA.md

@@ -210,7 +210,10 @@ token/cost budget breach, which fires only after every criterion is already scor
 cache buckets, captured proxy-side on the LiteLLM open-weight backend and rendered
 by the evalboard as a per-call table; empty on every other
 backend), `num_turns`, `max_turns_exhausted`,
-`result_summary` (`{is_error, subtype, stop_reason, result}`), `crashed`,
+`result_summary` (`{is_error, subtype, stop_reason, result}`), `provider_errors`
+(`list[ProviderError]` — `{at, message, kind, http_status, will_retry, details}`, one
+row per model-provider error the agent's own client reported, including errors it
+retried itself; filled by the codex adapter, empty on other backends), `crashed`,
 `crash_reason`.
 
 > **Token invariant.** Summing the four token buckets across `messages`

{coder_eval-0.12.7 → coder_eval-0.12.9}/docs/agents/HARNESS_PARITY.md

@@ -67,8 +67,8 @@ All five harnesses can have tool execution inside a generation window, and it is
 subtracted out of every one of them — **once, centrally**, by
 `timing.py::subtract_tool_time`. No reducer does it itself; each
 publishes the raw window (see the two sections below). Every harness has the
-problem: Antigravity reports a `Step` for the tool and only a later
-`usage_metadata` `Step` cuts the message; Codex's message window is seeded from
+problem: Antigravity reports a `Step` for the tool and only a later rise in
+the conversation's billed usage cuts the message; Codex's message window is seeded from
 the first item's start and extended to the last item's completion; OpenCode, Pi
 and claude-code tile, each window opening where the previous one closed and
 running to the next, with every tool call in between running inside. In each the
@@ -559,8 +559,9 @@ Each harness finds the call boundary in its own stream (the table above):
   first item would let one tool of call N+1 act. A call that ran tools therefore
   opens the next call at its `tokenUsage`, since the results always go back to the
   model, and the cap fires before call N+1 can run anything.
-- **Antigravity** attaches `usage_metadata` to one step per call, and the next call
-  opens with a MODEL step at a new `step_index`.
+- **Antigravity** bills each call on the conversation's cumulative usage, not on a
+  step, so a rise in that total closes the call, and the next call opens with a MODEL
+  step at a new `step_index`.
 - **OpenCode** and **Pi** stream one `step_start` or `turn_start` per call.
 - **Delegate**'s SDK has no round-trip marker, and a tool-only reply streams only
   its tool call, with no text before it. So the next call opens when every tool the

coder_eval-0.12.9/osv-scanner.toml

@@ -0,0 +1,15 @@
+# osv-scanner suppressions for advisories without an available fix.
+#
+# Mirrors the inline --ignore-vuln list passed to pip-audit in
+# .github/workflows/pr-checks.yml so both scanners stay in sync. pip-audit
+# accepts ignores as CLI flags; osv-scanner only reads them from this TOML.
+# When an advisory is fixed upstream, drop the entry here AND remove the
+# matching pip-audit flag in the same PR.
+
+[[IgnoredVulns]]
+id = "CVE-2026-49265"
+reason = "oauthlib 3.3.1 PKCE timing side channel in the server-side code_challenge check, which coder-eval never runs (oauthlib is transitive via azure-monitor-opentelemetry-exporter -> msrest -> requests-oauthlib). The fix, 4.0.0, is a major bump still inside the safe-chain minimum package age; revisit next month."
+
+[[IgnoredVulns]]
+id = "GHSA-hj66-6f7g-4r5v"
+reason = "CVE-2026-49264: oauthlib 3.3.1 RevocationEndpoint JSONP callback injection (only with enable_jsonp=True), a server-side OAuth endpoint coder-eval never runs (oauthlib is transitive via azure-monitor-opentelemetry-exporter -> msrest -> requests-oauthlib). The fix, 4.0.0, is a major bump still inside the safe-chain minimum package age; revisit next month."

{coder_eval-0.12.7 → coder_eval-0.12.9}/plugins/coder-eval/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "coder-eval",
   "displayName": "Coder Eval",
-  "version": "0.12.7",
+  "version": "0.12.9",
   "description": "Test whether your Claude Code skills actually trigger, and benchmark any coding agent — author, run, and analyze the eval suite that proves it, locally or as a CI gate.",
   "author": { "name": "UiPath", "email": "coder-eval@uipath.com", "url": "https://github.com/UiPath/coder_eval" },
   "homepage": "https://coder-eval.com",

{coder_eval-0.12.7 → coder_eval-0.12.9}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "coder-eval"
-version = "0.12.7"
+version = "0.12.9"
 description = "Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode, Pi, Delegate) with sandboxed, reproducible YAML task suites."
 readme = "README.md"
 license = "Apache-2.0"
@@ -227,7 +227,7 @@ constraint-dependencies = [
     "langgraph>=1.0.10",
     "langsmith>=0.6.3",
     "orjson>=3.11.6",
-    "pyjwt>=2.13.0",
+    "pyjwt>=2.14.0",
     "python-multipart>=0.0.31",
     "requests>=2.33.0",
     "starlette>=1.3.1",
@@ -305,6 +305,7 @@ external = [
     "CE064",
     "CE065",
     "CE066",
+    "CE068",
 ]  # custom architectural lint rules (tests/lint/)
 
 [tool.ruff.lint.pylint]

{coder_eval-0.12.7 → coder_eval-0.12.9}/src/coder_eval/__init__.py

@@ -1,3 +1,3 @@
 """coder_eval - A framework for evaluating AI coding agents."""
 
-__version__ = "0.12.7"
+__version__ = "0.12.9"

{coder_eval-0.12.7 → coder_eval-0.12.9}/src/coder_eval/agents/_skills.py

@@ -24,7 +24,7 @@ _DEFAULT_PLUGIN_SKILLS_SUBDIR = "skills"
 _SKILL_FILE = "SKILL.md"
 
 
-def _manifest_skill_dirs(root: Path) -> list[Path]:
+def manifest_skill_dirs(root: Path) -> list[Path]:
     """Skill directories a Claude-plugin root declares, in manifest order.
 
     Reads the ``skills`` field of ``<root>/.claude-plugin/plugin.json`` (a string
@@ -81,7 +81,7 @@ def _plugin_skill_dirs(
                 hint,
             )
             continue
-        candidates = [directory for directory in _manifest_skill_dirs(root) if directory.is_dir()]
+        candidates = [directory for directory in manifest_skill_dirs(root) if directory.is_dir()]
         # A path that is ALREADY a bare skills directory has no `skills/` subdir,
         # so use it as-is. Deliberately NOT a fallback for a root that HAS one.
         # Rationale: .claude/notes/agents.md § Skills, per harness

{coder_eval-0.12.7 → coder_eval-0.12.9}/src/coder_eval/agents/antigravity_agent.py

@@ -25,7 +25,7 @@ from collections.abc import Callable
 from contextlib import AsyncExitStack
 from datetime import datetime
 from pathlib import Path
-from typing import Any, ClassVar
+from typing import Any, ClassVar, NamedTuple
 
 from coder_eval.agent import Agent, AgentState
 from coder_eval.agents._logging import PrefixedAdapter
@@ -152,6 +152,22 @@ def _enum_value(x: Any) -> Any:
     return getattr(x, "value", x)
 
 
+class _UsageCounts(NamedTuple):
+    """The four Gemini token counts, named as ``google.antigravity.types.UsageMetadata`` names them."""
+
+    prompt_token_count: int = 0
+    cached_content_token_count: int = 0
+    candidates_token_count: int = 0
+    thoughts_token_count: int = 0
+
+    @classmethod
+    def of(cls, usage: Any) -> "_UsageCounts":
+        return cls(*((getattr(usage, f, 0) or 0) for f in cls._fields))
+
+    def minus(self, other: "_UsageCounts") -> "_UsageCounts":
+        return _UsageCounts(*(a - b for a, b in zip(self, other, strict=True)))
+
+
 def _to_token_usage(usage: Any, model: str | None) -> TokenUsage:
     """Map a ``google.antigravity.types.UsageMetadata`` to coder_eval ``TokenUsage``.
 
@@ -480,6 +496,7 @@ class AntigravityAgent(Agent[AntigravityAgentConfig]):
             model=model,
             turn_start_time=turn_start_time,
             clock=clock,
+            cumulative_usage=lambda: self._sdk_agent.conversation.total_usage,
             max_turns=max_turns,
         )
 
@@ -677,10 +694,13 @@ class _AntigravityTurnState:
     agent's shared crash kernel builds ``pending_turn`` from ``collector``).
 
     Step-stream shape this consumes (observed): each ``step_index`` is yielded
-    repeatedly through ACTIVE -> DONE transitions; ``usage_metadata`` lands once
-    per generation on a DONE/terminal step (summing them == the turn total); a
-    tool call carries a stable ``id`` and its result is folded into expanded
-    ``args`` at DONE.
+    repeatedly through ACTIVE -> DONE transitions; a tool call carries a stable
+    ``id`` and its result is folded into expanded ``args`` at DONE.
+
+    Usage is NOT on the Step: the SDK accumulates it per model invocation on the
+    conversation, which ``cumulative_usage`` reads. Each rise since the last
+    reading is booked as one generation, and ``finalize`` books the remainder, so
+    the per-message buckets always sum to the turn total.
     """
 
     def __init__(
@@ -696,9 +716,12 @@ class _AntigravityTurnState:
         model: str,
         turn_start_time: float,
         clock: TurnClock,
+        cumulative_usage: Callable[[], Any],
         max_turns: int | None = None,
     ) -> None:
         self._agent = agent
+        self._cumulative_usage = cumulative_usage
+        self._usage_booked = _UsageCounts.of(cumulative_usage())
         self.emit = emit
         self.task_id = task_id
         self.turn_id = turn_id
@@ -815,13 +838,22 @@ class _AntigravityTurnState:
                 self._blocks.append(ContentBlock(block_type="text", sequence=0, text=step.content))
 
         # Per-generation usage: fold into the turn total and cut an AssistantMessage.
-        if step.usage_metadata is not None:
+        if self._book_new_usage():
             if not self._in_api_call:
                 self.api_calls += 1
             self._in_api_call = False
-            gen = _to_token_usage(step.usage_metadata, self.model)
-            self.total_usage = self.total_usage + gen
-            self._flush_generation(gen, getattr(step.usage_metadata, "thoughts_token_count", 0) or 0)
+
+    def _book_new_usage(self) -> bool:
+        """Book the usage accrued since the last reading as one generation; False if none."""
+        now = _UsageCounts.of(self._cumulative_usage())
+        accrued = now.minus(self._usage_booked)
+        if not any(accrued):
+            return False
+        self._usage_booked = now
+        gen = _to_token_usage(accrued, self.model)
+        self.total_usage = self.total_usage + gen
+        self._flush_generation(gen, accrued.thoughts_token_count)
+        return True
 
     def _handle_tool_call(self, call: Any, step: Any, done: bool, sstatus: Any, call_index: int) -> None:
         raw_name = _enum_value(call.name)
@@ -995,7 +1027,8 @@ class _AntigravityTurnState:
                 )
             )
 
-        # Flush any trailing blocks not yet attached to a generation (no usage).
+        # Usage that landed after the last Step, then any blocks it did not carry.
+        self._book_new_usage()
         if self._blocks:
             self._flush_generation(TokenUsage(), 0)
 

{coder_eval-0.12.7 → coder_eval-0.12.9}/src/coder_eval/agents/codex_agent.py

@@ -11,6 +11,7 @@ import tempfile
 import time
 from collections.abc import Callable
 from datetime import datetime
+from enum import Enum
 from pathlib import Path
 from typing import Any, ClassVar, NamedTuple
 from urllib.parse import urlparse
@@ -33,6 +34,7 @@ from coder_eval.models import (
     CommandTelemetry,
     ContentBlock,
     DirectRoute,
+    ProviderError,
     SystemPromptSemantics,
     TokenUsage,
     TranscriptMessage,
@@ -274,6 +276,34 @@ _LOGIN_PROFILE_NAMES = (".bash_profile", ".profile", ".zshenv", ".zprofile", ".z
 _ZSH_PROFILE_NAMES = frozenset({".zshenv", ".zprofile", ".zshrc"})
 
 
+def _provider_error_from(payload: Any) -> ProviderError:
+    """Map a codex ``ErrorNotification`` payload onto a ``ProviderError``.
+
+    ``codexErrorInfo`` is either a bare category (enum) or a one-key object whose
+    key is the category and whose value may carry ``httpStatusCode``.
+    """
+    error = getattr(payload, "error", None)
+    kind: str | None = None
+    http_status: int | None = None
+    info = getattr(getattr(error, "codex_error_info", None), "root", None)
+    if isinstance(info, Enum):
+        kind = str(info.value)
+    elif hasattr(info, "model_dump"):
+        dumped = info.model_dump(by_alias=True, mode="json")
+        if len(dumped) == 1:
+            kind, body = next(iter(dumped.items()))
+            if isinstance(body, dict) and isinstance(body.get("httpStatusCode"), int):
+                http_status = body["httpStatusCode"]
+    return ProviderError(
+        at=datetime.now(),
+        message=str(getattr(error, "message", "") or ""),
+        kind=kind,
+        http_status=http_status,
+        will_retry=bool(getattr(payload, "will_retry", False)),
+        details=getattr(error, "additional_details", None),
+    )
+
+
 def _get_item_root(notification: Any) -> Any:
     """Extract the typed item root from a Codex SDK notification.
 
@@ -338,6 +368,7 @@ class _CodexTurnState:
         # Live pump scratch (set during streaming).
         self.turn_result: Any = None
         self.latest_token_usage: Any = None
+        self.provider_errors: list[ProviderError] = []
         self.agent_message_chunks: list[str] = []
         # Sequence per executable item, assigned at item/started and reused at
         # item/completed via this id->seq map.
@@ -546,8 +577,19 @@ class _CodexTurnState:
             self.on_token_usage_updated(notification)
         elif method == "turn/completed":
             return self.on_turn_completed(notification)
+        elif method == "error":
+            self.on_error(notification)
         return False
 
+    def on_error(self, notification: Any) -> None:
+        """Record a provider error; codex retries a stalled stream silently otherwise."""
+        err = _provider_error_from(notification.payload)
+        self.provider_errors.append(err)
+        self._agent._log.warning(
+            f"provider error (will_retry={err.will_retry}, kind={err.kind}, "
+            + f"http_status={err.http_status}): {err.message}"
+        )
+
     def on_item_started(self, notification: Any) -> None:
         """Emit ToolStartEvent + record the tool_use block for every tool-like item."""
         root = _get_item_root(notification)
@@ -765,6 +807,7 @@ class _CodexTurnState:
                 assistant_turn_count=1,
                 messages=self.messages,
                 num_turns=max(self.api_calls, 1),
+                provider_errors=list(self.provider_errors),
                 crashed=crashed,
                 crash_reason=crash_reason,
                 max_turns_exhausted=status is AgentEndStatus.MAX_TURNS_EXHAUSTED,

{coder_eval-0.12.7 → coder_eval-0.12.9}/src/coder_eval/cli/run_task_internal_command.py

@@ -17,6 +17,7 @@ from __future__ import annotations
 import asyncio
 import contextlib
 import logging
+import os
 from pathlib import Path
 from typing import Any
 
@@ -45,6 +46,17 @@ from coder_eval.path_utils import PRIOR_RESULT_FILENAME
 logger = logging.getLogger(__name__)
 
 
+def _scrub_staged_inputs(task_yaml: Path, context_json: Path) -> None:
+    """Delete the staged ``task.yaml`` and ``context.json`` once loaded, inside a container only.
+
+    Both carry the task's ``success_criteria``, and the agent runs in this same container.
+    ``prior.json`` is left for the regrade path, which reads it later and runs no agent.
+    """
+    if os.environ.get(IN_CONTAINER_ENV) == "1":
+        task_yaml.unlink(missing_ok=True)
+        context_json.unlink(missing_ok=True)
+
+
 def heartbeat_is_alive(current: str, last_counter: str, current_mtime: float, last_mtime: float) -> bool:
     """True when the heartbeat shows a fresh signal of life.
 
@@ -176,6 +188,7 @@ def run_task_internal_command(
     # `task_file` is then pointed under the task_dir mount, so the `TASK_DIR` the
     # Orchestrator exposes to `run_command` criteria resolves there, not /work/input.
     task, _ = load_task(task_yaml)
+    _scrub_staged_inputs(task_yaml, context_json)
     # The path below is never re-read; it only seeds Orchestrator's TASK_DIR.
     runtime_task_file = task_dir / "task.yaml" if task_dir.is_dir() else task_yaml