coder-eval 0.12.9__tar.gz → 0.12.10__tar.gz

{coder_eval-0.12.9 → coder_eval-0.12.10}/.claude/notes/agents.md

@@ -434,7 +434,10 @@ agent had produced; bounding it only by a fixed cycle count disconnected from `t
 path unreachable — the watchdog always wins, and the same spurious-orphan turn burns the
 full turn timeout before crashing with zero criteria evaluated.
 
-The cycle cap is the SOLE bound when a task sets no timeout at all. It is deliberately not
+The cycle cap applies alongside that deadline, whichever comes first, and is the SOLE
+bound when a task sets no timeout at all. Under a long `turn_timeout` (1800 s gives a
+1440 s deadline) the deadline alone let a never-ending command (a dev server, an
+unanswered prompt) idle the turn for 24 minutes. It is deliberately not
 "break after N consecutive empty polls": `receive_steps()` returns identically empty
 whether a backgrounded job is still running or will never resolve, and there is no signal
 that tells the two apart except waiting. A count small enough to matter would abort real
@@ -447,6 +450,36 @@ answer in a headless eval), CANCELED and UNKNOWN — none of which the closed se
 done, and none of which the poll loop should wait out, since they will never become DONE
 on their own.
 
+It is also an allowlist on the TOOL: only a `run_command` can be backgrounded. Any other
+tool left ACTIVE never resolved in practice: an `edit_file` on the read-only skill mount,
+a `view_file` paged at a large `content_offset`, a `start_subagent`. In a 370-task
+SkillSpec run, 12 of 21 turns that polled to the deadline were waiting on one of those,
+after the model had already finished. `finalize` force-closes them as unresolved.
+
+## Antigravity tool allowlist
+
+`allowed_tools` / `disallowed_tools` map onto the harness's builtin toolset
+(`CapabilitiesConfig.enabled_tools` / `disabled_tools`) through
+`_CLAUDE_TO_ANTIGRAVITY_TOOL_MAP`, so an experiment's allowlist confines Antigravity the
+way it confines Claude Code. Without it Antigravity ran with every builtin, including
+`start_subagent` and `search_web`, under a list that gave Claude Code neither. `Skill` and
+`TodoWrite` have no builtin (skills load through `skills_paths`) and are skipped. `finish`
+stays on under any allowlist: it returns structured output, not a capability.
+`enable_subagents` is a separate switch from the toolset and follows whether
+`start_subagent` survives. The SDK takes an allowlist OR a denylist, so with both set the
+denied tools are removed from the allowlist.
+
+## Antigravity non-interactive commands
+
+`run_command` runs in a real terminal: a command that blocks becomes a background task the
+model can check on and type into. A command that stops to ask (`npx` installing a missing
+package, git credentials, an apt/pip confirmation, a pager) and that the model then leaves
+behind keeps the turn waiting on a prompt no one answers. `_NONINTERACTIVE_ENV`
+(`CI`, `npm_config_yes`, `GIT_TERMINAL_PROMPT=0`, `DEBIAN_FRONTEND`, `PIP_NO_INPUT`,
+`PAGER`/`GIT_PAGER=cat`) rides the per-agent `env` seam, each variable only where the
+environment does not already set it, so those commands answer themselves or fail fast. It
+cannot close the terminal: a bare `read` or a server still runs until the poll cap.
+
 ## The receive_steps re-entrancy window
 
 `receive_steps()` is two nested async generators: the public one delegates to the

{coder_eval-0.12.9 → coder_eval-0.12.10}/CHANGELOG.md

@@ -2,6 +2,25 @@
 
 <!-- version list -->
 
+## v0.12.10 (2026-10-03)
+
+### Bug Fixes
+
+- **antigravity**: Enforce tool lists, non-interactive commands, bounded polling
+  ([#215](https://github.com/UiPath/coder_eval/pull/215),
+  [`7a5a4d0`](https://github.com/UiPath/coder_eval/commit/7a5a4d03ff962b7e47a10c28051b0ca286e2a046))
+
+- **cli-called**: Report a zero-match max_count: 0 guard as unproven
+  ([#205](https://github.com/UiPath/coder_eval/pull/205),
+  [`33bc3d7`](https://github.com/UiPath/coder_eval/commit/33bc3d70bd988a522801d75900aeb53ef8400225))
+
+### Chores
+
+- **deps**: Bump urllib3 2.8.0 and virtualenv 21.14.2 to clear pip-audit
+  ([#215](https://github.com/UiPath/coder_eval/pull/215),
+  [`7a5a4d0`](https://github.com/UiPath/coder_eval/commit/7a5a4d03ff962b7e47a10c28051b0ca286e2a046))
+
+
 ## v0.12.9 (2026-09-29)
 
 ### Bug Fixes

{coder_eval-0.12.9 → coder_eval-0.12.10}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: coder-eval
-Version: 0.12.9
+Version: 0.12.10
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode, Pi, Delegate) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval
@@ -43,7 +43,7 @@ Requires-Dist: tomli-w>=1.0.0
 Requires-Dist: tqdm>=4.67.3
 Requires-Dist: typer>=0.24.1
 Provides-Extra: antigravity
-Requires-Dist: google-antigravity==0.1.18; extra == 'antigravity'
+Requires-Dist: google-antigravity==0.1.20; extra == 'antigravity'
 Provides-Extra: codex
 Requires-Dist: openai-codex>=0.155.1; extra == 'codex'
 Provides-Extra: dev

{coder_eval-0.12.9 → coder_eval-0.12.10}/action.yml

@@ -25,7 +25,7 @@ inputs:
   version:
     description: coder-eval version to install from PyPI, or "local" to install from the action checkout
     required: false
-    default: "0.12.9"   # <-- kept in sync with releases by release.yml
+    default: "0.12.10"   # <-- kept in sync with releases by release.yml
   extras:
     description: >-
       Comma-separated coder-eval extras (`codex`, `antigravity,litellm`), composed

{coder_eval-0.12.9 → coder_eval-0.12.10}/docs/agents/ANTIGRAVITY.md

@@ -27,7 +27,7 @@ working directory — both required for an unattended eval.
 pip install 'coder-eval[antigravity]'
 ```
 
-This pulls in `google-antigravity` (pinned to `0.1.18`), whose wheel bundles the
+This pulls in `google-antigravity` (pinned to `0.1.20`), whose wheel bundles the
 platform `localharness` binary. As with the other agents the SDK is imported lazily
 — a base install without the extra still runs end-to-end; Antigravity tasks fail at
 dispatch with a clear hint to install the extra.
@@ -136,18 +136,28 @@ can't be resolved or if zero skills are discovered.
 
 ## Permissions & tools — important differences
 
-**Antigravity ignores `permission_mode`, `allowed_tools`, and `disallowed_tools`.**
-The local harness runs in a single unconditional mode: every tool call (including
-`run_command`) is approved via an allow-all policy, and file tools are restricted to
-the configured `workspaces` (the sandbox working directory plus any skill roots).
+**Antigravity ignores `permission_mode`.** The local harness runs in a single
+unconditional mode: every tool call (including `run_command`) is approved via an
+allow-all policy, and file tools are restricted to the configured `workspaces` (the
+sandbox working directory plus any skill roots).
+
+**`allowed_tools` / `disallowed_tools` are enforced** by mapping the Claude tool names
+onto the harness's builtin tools (`Bash` → `run_command`, `Read` → `view_file`, `Write` →
+`create_file`, `Edit` → `edit_file`, `Glob` → `find_file`, `Grep` → `search_directory`,
+`Task` → `start_subagent`, `WebSearch` → `search_web`, `WebFetch` → `read_url_content`).
+`Skill` has no builtin and is skipped; `finish` always stays on.
+
+`run_command` also gets non-interactive environment variables (`CI=1`,
+`npm_config_yes=true`, `GIT_TERMINAL_PROMPT=0`, `DEBIAN_FRONTEND=noninteractive`,
+`PIP_NO_INPUT=1`, `PAGER=cat`), each only where the environment does not already set it.
 
 The trust boundary for an Antigravity run is therefore the **sandbox**, not the
 agent config. Run untrusted tasks under the [Docker driver](../DOCKER_ISOLATION.md);
 the `tempdir` driver is not a security boundary. This mirrors the reality that the
 `bypassPermissions`-equivalent behavior is always on for this backend.
 
-Those inherited fields still exist on the config for schema uniformity but have no
-runtime effect here — don't rely on them to gate Antigravity.
+`permission_mode` still exists on the config for schema uniformity but has no runtime
+effect here — don't rely on it to gate Antigravity.
 
 ## Telemetry
 
@@ -187,19 +197,18 @@ as every other agent.
 4. **`permission_mode` does not confine the harness.** Every mode runs
    `policy.allow_all()`; coder_eval's write boundary is the sandbox driver, and a
    headless eval has no human to approve anything.
-5. **`allowed_tools` / `disallowed_tools` are not read.** The harness runs with its
-   full builtin tool set, so an Antigravity run has tools (web search, subagents,
-   URL fetch) that the same task file denies on Claude Code and Codex.
-6. **`max_turns` is counted by the harness.** One `communicate()` is a single SDK turn
+5. **`max_turns` is counted by the harness.** One `communicate()` is a single SDK turn
    here, so the harness counts model API calls itself (a MODEL step at a new
    `step_index` opens one) and enforces the cap on the step loop. See
    [Run-Limit Parity](HARNESS_PARITY.md).
-7. **Shell commands over ~10s are moved to the background.** The localharness has a
+6. **Shell commands over ~10s are moved to the background.** The localharness has a
    10-second maximum synchronous wait; past it the command becomes a background task
    and the model gets a task id, not a result. The turn polls for that result instead
-   of finalizing on an idle step stream, so slow work does complete — but the wait is
-   bounded by 80% of `turn_timeout`, and a job that outlives it is force-closed as
-   `result_status: unknown` and graded as an ordinary low score rather than a timeout.
+   of finalizing on an idle step stream, so slow work does complete — but only an
+   orphaned `run_command` is waited on, the wait is bounded by 10 minutes or 80% of
+   `turn_timeout` (whichever is shorter), and a job that outlives it (typically a server
+   the model left running) is force-closed as `result_status: unknown` and graded
+   normally rather than as a timeout.
    Measured in [Run-Limit Parity](HARNESS_PARITY.md).
 
 ## Running in Docker

{coder_eval-0.12.9 → coder_eval-0.12.10}/docs/agents/HARNESS_PARITY.md

@@ -718,9 +718,10 @@ both. See [OpenCode](OPENCODE.md) and [Pi § plugins](PI.md#known-limitations).
   lowercase (`bash`/`read`/`write`/`edit`/`grep`/`find`/`ls`), but the shared config
   default (`experiments/default.yaml`) sets Claude-namespaced names
   (`Bash`/`Read`/`Write`/…). Forwarding those to `--tools` would allowlist tools that
-  do not exist in Pi and strip the agent of ALL tools — so, like OpenCode (drops them),
-  Codex (forwards `disallowed_tools` without SDK enforcement), and Antigravity (does not
-  read them), Pi ignores them and runs with its full native toolset. A task that needs a
+  do not exist in Pi and strip the agent of ALL tools — so, like OpenCode (drops them)
+  and Codex (forwards `disallowed_tools` without SDK enforcement), Pi ignores them and
+  runs with its full native toolset. (Antigravity maps them onto its builtin tools; see
+  [Antigravity](ANTIGRAVITY.md).) A task that needs a
   restricted Pi toolset would have to name Pi's lowercase tools — a documented follow-up.
 - **`permission_mode` is NOT enforced** — Pi headless print mode auto-runs tools and
   exposes only project-file trust (`--approve` / `--no-approve`), no tool-approval

{coder_eval-0.12.9 → coder_eval-0.12.10}/plugins/coder-eval/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "coder-eval",
   "displayName": "Coder Eval",
-  "version": "0.12.9",
+  "version": "0.12.10",
   "description": "Test whether your Claude Code skills actually trigger, and benchmark any coding agent — author, run, and analyze the eval suite that proves it, locally or as a CI gate.",
   "author": { "name": "UiPath", "email": "coder-eval@uipath.com", "url": "https://github.com/UiPath/coder_eval" },
   "homepage": "https://coder-eval.com",

{coder_eval-0.12.9 → coder_eval-0.12.10}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "coder-eval"
-version = "0.12.9"
+version = "0.12.10"
 description = "Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode, Pi, Delegate) with sandboxed, reproducible YAML task suites."
 readme = "README.md"
 license = "Apache-2.0"
@@ -131,7 +131,7 @@ codex = [
 # Without this extra the framework still installs and runs; Antigravity-dependent
 # code paths fail at start() with a clear hint pointing back here.
 antigravity = [
-    "google-antigravity==0.1.18",
+    "google-antigravity==0.1.20",
 ]
 # Optional extra that enables OpenCode agent support.
 #

{coder_eval-0.12.9 → coder_eval-0.12.10}/src/coder_eval/__init__.py

@@ -1,3 +1,3 @@
 """coder_eval - A framework for evaluating AI coding agents."""
 
-__version__ = "0.12.9"
+__version__ = "0.12.10"

{coder_eval-0.12.9 → coder_eval-0.12.10}/src/coder_eval/agents/antigravity_agent.py

@@ -96,12 +96,20 @@ _RECEIVE_STEPS_REENTRY_RETRIES = 5
 _POLL_DEADLINE_TIMEOUT_FRACTION = 0.8
 
 # Cap on poll *cycles* -- the SOLE bound when a task sets no timeout at all, and
-# a backstop against a very large one. 120 * 5s = 10 minutes, ~2x the worst real
+# a backstop against a very large one (applied alongside the deadline, whichever
+# is reached first). 120 * 5s = 10 minutes, ~2x the worst real
 # backgrounded-job duration observed (60-300s). Deliberately NOT "break after N
 # consecutive empty polls".
 # Rationale: .claude/notes/agents.md § Antigravity Step interleaving and the background poll
 _MAX_BACKGROUND_POLLS = 120
 
+# The only tool a model can background: a `run_command` left ACTIVE is a job
+# that may still finish. Any other tool left ACTIVE (edit_file on a read-only
+# mount, a paged view_file, start_subagent) never resolves, so it does not
+# justify polling.
+# Rationale: .claude/notes/agents.md § Antigravity Step interleaving and the background poll
+_BACKGROUNDABLE_TOOL = "run_command"
+
 # Antigravity builtin tool name -> the canonical (Claude) vocabulary every
 # criterion is written against. Unmapped names pass through unchanged.
 # Rationale: .claude/notes/agents.md § Tool-name and argument normalization
@@ -120,6 +128,45 @@ _ANTIGRAVITY_TO_CLAUDE_TOOL_MAP: dict[str, str] = {
     "finish": "Finish",
 }
 
+# Canonical (Claude) tool name -> the Antigravity builtin it enables, for
+# `allowed_tools` / `disallowed_tools`. "Skill" and "TodoWrite" have no builtin
+# (skills load through `skills_paths`), so they map to nothing; a name that is
+# already an Antigravity builtin passes through.
+# Rationale: .claude/notes/agents.md § Antigravity tool allowlist
+_CLAUDE_TO_ANTIGRAVITY_TOOL_MAP: dict[str, str] = {
+    "Bash": "run_command",
+    "Read": "view_file",
+    "Write": "create_file",
+    "Edit": "edit_file",
+    "MultiEdit": "edit_file",
+    "Glob": "find_file",
+    "Grep": "search_directory",
+    "LS": "list_directory",
+    "Task": "start_subagent",
+    "WebSearch": "search_web",
+    "WebFetch": "read_url_content",
+    "AskUserQuestion": "ask_question",
+}
+
+# Kept on under any allowlist: `finish` is how a turn returns structured
+# output, not a capability an allowlist is meant to grant or withhold.
+_ALWAYS_ENABLED_TOOLS: frozenset[str] = frozenset({"finish"})
+
+# Set on every run_command unless the environment already sets them, so a
+# command that would stop to ask (`npx` installing a package, git credentials,
+# apt/pip confirmations, a pager) answers itself or fails instead of waiting on
+# a terminal no one types into.
+# Rationale: .claude/notes/agents.md § Antigravity non-interactive commands
+_NONINTERACTIVE_ENV: dict[str, str] = {
+    "CI": "1",
+    "npm_config_yes": "true",
+    "GIT_TERMINAL_PROMPT": "0",
+    "DEBIAN_FRONTEND": "noninteractive",
+    "PIP_NO_INPUT": "1",
+    "PAGER": "cat",
+    "GIT_PAGER": "cat",
+}
+
 # Tool-call arg keys the harness ADDS at completion (the result payload), not
 # model-supplied inputs. The STATIC backstop; ``_params`` also strips any key
 # that first appears at DONE. A leaked result would false-positive
@@ -303,18 +350,57 @@ class AntigravityAgent(Agent[AntigravityAgentConfig]):
     def _harness_env(self) -> dict[str, str] | None:
         """Per-agent environment for the localharness subprocess (``LocalAgentConfig.env``).
 
-        Returns the mock-CLI PATH prepend as a one-key overlay, or ``None`` when no
-        mock dirs are configured. The SDK merges it over ``os.environ`` at spawn,
-        so naming only ``PATH`` leaves every other inherited variable untouched.
+        An overlay of the non-interactive variables (each only where the
+        environment does not already set it) plus, when mock dirs are configured,
+        the mock-CLI PATH prepend; ``None`` when there is nothing to add, so the
+        SDK spawns with a plain inherited env. The SDK merges it over
+        ``os.environ`` at spawn, so every other inherited variable is untouched.
         The same overlay becomes the harness's ``run_command`` environment.
         """
-        if not self._env_path_prepend:
+        env = {k: v for k, v in _NONINTERACTIVE_ENV.items() if k not in os.environ}
+        if self._env_path_prepend:
+            # Match the parent's own casing (Windows exports ``Path``) so the merge
+            # overrides the inherited entry instead of adding a sibling key.
+            path_key = next((k for k in os.environ if k.upper() == "PATH"), "PATH")
+            env[path_key] = os.pathsep.join([*self._env_path_prepend, os.environ.get(path_key) or ""])
+        return env or None
+
+    def _tool_capabilities(self, types: Any) -> Any:
+        """``CapabilitiesConfig`` restricting the builtin tools to the configured
+        ``allowed_tools`` / ``disallowed_tools``, or ``None`` when neither is set.
+
+        Names are mapped through ``_CLAUDE_TO_ANTIGRAVITY_TOOL_MAP``; names with no
+        Antigravity builtin (``Skill``, ``TodoWrite``, MCP tools) are skipped. The
+        SDK takes an allowlist OR a denylist, so with both set the denied tools are
+        removed from the allowlist.
+
+        Rationale: .claude/notes/agents.md § Antigravity tool allowlist
+        """
+        allowed, disallowed = self.config.allowed_tools, self.config.disallowed_tools
+        if not allowed and not disallowed:
             return None
-        # Match the parent's own casing (Windows exports ``Path``) so the merge
-        # overrides the inherited entry instead of adding a sibling key.
-        path_key = next((k for k in os.environ if k.upper() == "PATH"), "PATH")
-        merged = os.pathsep.join([*self._env_path_prepend, os.environ.get(path_key) or ""])
-        return {path_key: merged}
+        builtin = {t.value for t in types.BuiltinTools}
+
+        def to_builtin(names: list[str] | None) -> set[str]:
+            mapped = {_CLAUDE_TO_ANTIGRAVITY_TOOL_MAP.get(n, n) for n in names or []}
+            return mapped & builtin
+
+        # `enable_subagents` is a separate switch from the toolset; it follows
+        # whether `start_subagent` survives the filter.
+        subagent = types.BuiltinTools.START_SUBAGENT.value
+        if allowed:
+            enabled = (to_builtin(allowed) | _ALWAYS_ENABLED_TOOLS) - to_builtin(disallowed)
+            self._log.debug("Enabled builtin tools: %s", ", ".join(sorted(enabled)))
+            return types.CapabilitiesConfig(
+                enabled_tools=[types.BuiltinTools(t) for t in sorted(enabled)],
+                enable_subagents=subagent in enabled,
+            )
+        disabled = to_builtin(disallowed) - _ALWAYS_ENABLED_TOOLS
+        self._log.debug("Disabled builtin tools: %s", ", ".join(sorted(disabled)))
+        return types.CapabilitiesConfig(
+            disabled_tools=[types.BuiltinTools(t) for t in sorted(disabled)],
+            enable_subagents=subagent not in disabled,
+        )
 
     async def start(
         self,
@@ -370,9 +456,11 @@ class AntigravityAgent(Agent[AntigravityAgentConfig]):
                 system_instructions=self.config.system_prompt or None,
                 # Skill discovery: the search-path roots that parent the skill dirs.
                 skills_paths=skills_paths,
-                # Mock-CLI PATH shadowing, per agent: two concurrent tasks never
-                # see each other's mock dirs.
+                # Mock-CLI PATH shadowing plus the non-interactive variables, per
+                # agent: two concurrent tasks never see each other's mock dirs.
                 env=self._harness_env(),
+                # The builtin toolset, narrowed to allowed_tools / disallowed_tools.
+                capabilities=self._tool_capabilities(types),
             )
             # Thinking level onto every resolved model's endpoint. The SDK
             # validates the model list in a model_validator, so options are set on
@@ -547,11 +635,8 @@ class AntigravityAgent(Agent[AntigravityAgentConfig]):
                         and not state.max_turns_hit
                         and not state.timeout_hit
                         and state.has_orphaned_tool_call()
-                        and (
-                            poll_count < _MAX_BACKGROUND_POLLS
-                            if poll_deadline is None
-                            else time.monotonic() < poll_deadline
-                        )
+                        and poll_count < _MAX_BACKGROUND_POLLS
+                        and (poll_deadline is None or time.monotonic() < poll_deadline)
                     ):
                         poll_count += 1
                         self._log.debug("Polling for backgrounded work (orphaned tool call); attempt %d", poll_count)
@@ -577,9 +662,9 @@ class AntigravityAgent(Agent[AntigravityAgentConfig]):
                         # stop/timeout: the call is force-closed as unresolved and
                         # the turn is still graded normally on everything else.
                         bound = (
-                            f"poll_deadline ({_POLL_DEADLINE_TIMEOUT_FRACTION:.0%} of {timeout:g}s turn timeout)"
-                            if poll_deadline is not None
-                            else f"_MAX_BACKGROUND_POLLS ({_MAX_BACKGROUND_POLLS})"
+                            f"_MAX_BACKGROUND_POLLS ({_MAX_BACKGROUND_POLLS})"
+                            if poll_count >= _MAX_BACKGROUND_POLLS
+                            else f"poll_deadline ({_POLL_DEADLINE_TIMEOUT_FRACTION:.0%} of {timeout:g}s turn timeout)"
                         )
                         msg = "Poll budget exhausted (%s, poll_count=%d) with a tool call still ACTIVE."
                         self._log.warning(msg, bound, poll_count)
@@ -761,6 +846,8 @@ class _AntigravityTurnState:
         # Most recently seen StepStatus per tool id, for has_orphaned_tool_call.
         # Separate from _closed_tools, which tracks only DONE/ERROR.
         self._tool_last_status: dict[str, Any] = {}
+        # Raw (Antigravity) tool name per tool id, for has_orphaned_tool_call.
+        self._tool_raw_names: dict[str, Any] = {}
         # Content blocks accumulated since the last per-generation flush.
         self._blocks: list[ContentBlock] = []
         # Where the CURRENT generation started, advanced only by a flush that
@@ -865,6 +952,7 @@ class _AntigravityTurnState:
         step_key = f"{trajectory_id}:{step.step_index}" if trajectory_id else str(step.step_index)
         cid = call.id or f"{raw_name}_{step_key}_{call_index}"
         self._tool_last_status[cid] = sstatus
+        self._tool_raw_names[cid] = raw_name
         if cid not in self._seen_tools:
             self._seen_tools.add(cid)
             seq = self._next_seq
@@ -990,8 +1078,8 @@ class _AntigravityTurnState:
         return ""
 
     def has_orphaned_tool_call(self) -> bool:
-        """True if any NOT-YET-CLOSED tool call's most recently seen status is
-        ACTIVE — the structural signature of a backgrounded task the model went
+        """True if any NOT-YET-CLOSED ``run_command``'s most recently seen status
+        is ACTIVE — the structural signature of a backgrounded task the model went
         idle on without waiting for. See ``communicate``'s poll loop.
 
         An ALLOWLIST on ACTIVE, never a denylist on "not yet closed": the SDK also
@@ -999,9 +1087,18 @@ class _AntigravityTurnState:
         should wait out. The `not in _closed_tools` guard is layered on top as a
         monotonicity backstop, not a substitute.
 
+        Also an allowlist on the TOOL: only ``run_command`` can be backgrounded.
+        Any other tool left ACTIVE never resolves, so waiting on it only burns
+        the poll budget; ``finalize`` force-closes it as unresolved.
+
         Rationale: .claude/notes/agents.md § Antigravity Step interleaving and the background poll
         """
-        return any(cid not in self._closed_tools and s == _STATUS_ACTIVE for cid, s in self._tool_last_status.items())
+        return any(
+            cid not in self._closed_tools
+            and s == _STATUS_ACTIVE
+            and self._tool_raw_names.get(cid) == _BACKGROUNDABLE_TOOL
+            for cid, s in self._tool_last_status.items()
+        )
 
     def finalize(self, status: AgentEndStatus, *, crashed: bool = False, crash_reason: str | None = None) -> None:
         """Close orphaned tools, flush leftover blocks, emit TurnEnd + AgentEnd.

{coder_eval-0.12.9 → coder_eval-0.12.10}/src/coder_eval/criteria/cli_called.py

@@ -170,7 +170,12 @@ class CliCalledChecker(BaseCriterion[CliCalledCriterion]):
             facets.append(f"flags={sorted(criterion.flags)}")
         wanted = ", ".join(facets)
 
-        if score == 1.0:
+        if score == 1.0 and count == 0 and criterion.max_count == 0:
+            details = (
+                f"0 invocation(s) matched ({wanted}); guard unproven: a typo in verb, tool or flags "
+                "would also match nothing"
+            )
+        elif score == 1.0:
             details = f"{count} invocation(s) matched ({wanted}); satisfies {bound}"
         elif not within_lower:
             # A bare count sends the reader to the sandbox; this criterion exists