coder-eval 0.12.9__tar.gz → 0.12.11__tar.gz

{coder_eval-0.12.9 → coder_eval-0.12.11}/.claude/notes/agents.md

@@ -434,7 +434,13 @@ agent had produced; bounding it only by a fixed cycle count disconnected from `t
 path unreachable — the watchdog always wins, and the same spurious-orphan turn burns the
 full turn timeout before crashing with zero criteria evaluated.
 
-The cycle cap is the SOLE bound when a task sets no timeout at all. It is deliberately not
+The cycle cap (120 × 5 s) bounds the wait only when a task sets no timeout at all. With
+one, the deadline alone bounds it, so a never-ending command (a dev server, an unanswered
+prompt) can idle up to 80% of the turn, but a slow job gets the time the task author
+budgeted for it. Applying the cap under a timeout as well force-closed real solvers and
+simulations at 10 minutes: on the SkillsBench Gemini 4-arm campaign it ended 22 of 348
+rows, 4 of which had passed with the deadline alone (an exam-scheduling MIP solve that
+passed at 1211 s among them). It is deliberately not
 "break after N consecutive empty polls": `receive_steps()` returns identically empty
 whether a backgrounded job is still running or will never resolve, and there is no signal
 that tells the two apart except waiting. A count small enough to matter would abort real
@@ -447,6 +453,58 @@ answer in a headless eval), CANCELED and UNKNOWN — none of which the closed se
 done, and none of which the poll loop should wait out, since they will never become DONE
 on their own.
 
+It is also an allowlist on the TOOL: only a `run_command` can be backgrounded. Any other
+tool left ACTIVE never resolved in practice: an `edit_file` on the read-only skill mount,
+a `view_file` paged at a large `content_offset`, a `start_subagent`. In a 370-task
+SkillSpec run, 12 of 21 turns that polled to the deadline were waiting on one of those,
+after the model had already finished. `finalize` force-closes them as unresolved.
+
+## Antigravity tool allowlist
+
+`allowed_tools` / `disallowed_tools` map onto the harness's builtin toolset
+(`CapabilitiesConfig.enabled_tools` / `disabled_tools`) through
+`_CLAUDE_TO_ANTIGRAVITY_TOOL_MAP`, so an experiment's allowlist confines Antigravity the
+way it confines Claude Code. Without it Antigravity ran with every builtin, including
+`start_subagent` and `search_web`, under a list that gave Claude Code neither. `Skill` and
+`TodoWrite` have no builtin (skills load through `skills_paths`) and are skipped. `finish`
+and `schedule` stay on under any allowlist: `finish` returns structured output, and
+`schedule` is the timer the model sleeps on while a backgrounded `run_command` finishes.
+Without `schedule`, Gemini re-checks the background task every few seconds and each check
+is a model call against `max_turns`: on the SkillsBench Gemini 4-arm campaign mean turns
+rose 36.8 → 50.2 and max-turn rows 4 → 16 on the same 110 rows, and the skill arms were hit
+2-3x harder than baseline. With `schedule` on they came back to 35.4 and 1.
+
+An allowlist only narrows the harness's default toolset (`BuiltinTools.default()`). `Glob`,
+`Grep` and `LS` map to `find_file`, `search_directory` and `list_directory`, which the
+harness ships off, so an allowlist naming them leaves them off. Turned on, Gemini used them
+in place of shell scripts (1,429 calls in 333 of 348 SkillsBench rows, none before), often
+one search per turn.
+
+`start_subagent` also stays on under any allowlist, because the harness builds its system
+prompt from the toolset: with `start_subagent` off it drops the whole subagents section
+(4,801 → 2,783 characters), and with it the prompt's only "you do NOT need to poll ... you
+will be notified" guidance. With it on, the system prompt is byte-identical to 0.12.9's.
+The section alone does not stop Gemini polling a long job with status checks: in a 48-row
+SkillsBench A/B it made 13.4 checks per row with the section and 13.7 without (8.5 on
+0.12.9). The tool descriptions are identical either way, and
+`enable_subagents` alone does not restore the section. Subagents the tool lists withhold
+(no `Task`) are denied by policy at `invoke_subagent` instead, which covers both the
+built-in `research` subagent and one the model defines with `define_subagent`; both get
+`search_web` regardless of the parent's toolset, and a parent policy on `search_web` does
+not reach a model-defined one. The SDK takes an allowlist OR a denylist, so with both set
+the denied tools are removed from the allowlist, except the always-enabled ones.
+
+## Antigravity non-interactive commands
+
+`run_command` runs in a real terminal: a command that blocks becomes a background task the
+model can check on and type into. A command that stops to ask (`npx` installing a missing
+package, git credentials, an apt/pip confirmation, a pager) and that the model then leaves
+behind keeps the turn waiting on a prompt no one answers. `_NONINTERACTIVE_ENV`
+(`CI`, `npm_config_yes`, `GIT_TERMINAL_PROMPT=0`, `DEBIAN_FRONTEND`, `PIP_NO_INPUT`,
+`PAGER`/`GIT_PAGER=cat`) rides the per-agent `env` seam, each variable only where the
+environment does not already set it, so those commands answer themselves or fail fast. It
+cannot close the terminal: a bare `read` or a server still runs until the poll deadline.
+
 ## The receive_steps re-entrancy window
 
 `receive_steps()` is two nested async generators: the public one delegates to the

{coder_eval-0.12.9 → coder_eval-0.12.11}/CHANGELOG.md

@@ -2,6 +2,62 @@
 
 <!-- version list -->
 
+## v0.12.11 (2026-10-05)
+
+### Bug Fixes
+
+- **antigravity**: Bound the background poll by the deadline alone when a timeout is set
+  ([#216](https://github.com/UiPath/coder_eval/pull/216),
+  [`de361f9`](https://github.com/UiPath/coder_eval/commit/de361f979657c68452286da6982d1d67a2fa98c7))
+
+- **antigravity**: Keep a tool allowlist within the harness default toolset
+  ([#216](https://github.com/UiPath/coder_eval/pull/216),
+  [`de361f9`](https://github.com/UiPath/coder_eval/commit/de361f979657c68452286da6982d1d67a2fa98c7))
+
+- **antigravity**: Keep schedule enabled under a tool allowlist
+  ([#216](https://github.com/UiPath/coder_eval/pull/216),
+  [`de361f9`](https://github.com/UiPath/coder_eval/commit/de361f979657c68452286da6982d1d67a2fa98c7))
+
+- **antigravity**: Keep start_subagent on and deny withheld subagents by policy
+  ([#216](https://github.com/UiPath/coder_eval/pull/216),
+  [`de361f9`](https://github.com/UiPath/coder_eval/commit/de361f979657c68452286da6982d1d67a2fa98c7))
+
+- **antigravity**: Restore Gemini's background-job waiting under a tool allowlist
+  ([#216](https://github.com/UiPath/coder_eval/pull/216),
+  [`de361f9`](https://github.com/UiPath/coder_eval/commit/de361f979657c68452286da6982d1d67a2fa98c7))
+
+### Documentation
+
+- **antigravity**: State what keeping start_subagent on does and does not fix
+  ([#216](https://github.com/UiPath/coder_eval/pull/216),
+  [`de361f9`](https://github.com/UiPath/coder_eval/commit/de361f979657c68452286da6982d1d67a2fa98c7))
+
+### Features
+
+- **evalboard**: Treat .dmn as a deliverable kind
+  ([#217](https://github.com/UiPath/coder_eval/pull/217),
+  [`7161fc8`](https://github.com/UiPath/coder_eval/commit/7161fc8a11624662fbf6e3f24d6b468fed99c453))
+
+
+## v0.12.10 (2026-10-03)
+
+### Bug Fixes
+
+- **antigravity**: Enforce tool lists, non-interactive commands, bounded polling
+  ([#215](https://github.com/UiPath/coder_eval/pull/215),
+  [`7a5a4d0`](https://github.com/UiPath/coder_eval/commit/7a5a4d03ff962b7e47a10c28051b0ca286e2a046))
+
+- **cli-called**: Report a zero-match max_count: 0 guard as unproven
+  ([#205](https://github.com/UiPath/coder_eval/pull/205),
+  [`33bc3d7`](https://github.com/UiPath/coder_eval/commit/33bc3d70bd988a522801d75900aeb53ef8400225))
+
+### Chores
+
+- **deps**: Bump urllib3 2.8.0 and virtualenv 21.14.2 to clear pip-audit
+  ([#215](https://github.com/UiPath/coder_eval/pull/215),
+  [`7a5a4d0`](https://github.com/UiPath/coder_eval/commit/7a5a4d03ff962b7e47a10c28051b0ca286e2a046))
+
+
 ## v0.12.9 (2026-09-29)
 
 ### Bug Fixes

{coder_eval-0.12.9 → coder_eval-0.12.11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: coder-eval
-Version: 0.12.9
+Version: 0.12.11
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode, Pi, Delegate) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval
@@ -43,7 +43,7 @@ Requires-Dist: tomli-w>=1.0.0
 Requires-Dist: tqdm>=4.67.3
 Requires-Dist: typer>=0.24.1
 Provides-Extra: antigravity
-Requires-Dist: google-antigravity==0.1.18; extra == 'antigravity'
+Requires-Dist: google-antigravity==0.1.20; extra == 'antigravity'
 Provides-Extra: codex
 Requires-Dist: openai-codex>=0.155.1; extra == 'codex'
 Provides-Extra: dev

{coder_eval-0.12.9 → coder_eval-0.12.11}/action.yml

@@ -25,7 +25,7 @@ inputs:
   version:
     description: coder-eval version to install from PyPI, or "local" to install from the action checkout
     required: false
-    default: "0.12.9"   # <-- kept in sync with releases by release.yml
+    default: "0.12.11"   # <-- kept in sync with releases by release.yml
   extras:
     description: >-
       Comma-separated coder-eval extras (`codex`, `antigravity,litellm`), composed

{coder_eval-0.12.9 → coder_eval-0.12.11}/docs/agents/ANTIGRAVITY.md

@@ -27,7 +27,7 @@ working directory — both required for an unattended eval.
 pip install 'coder-eval[antigravity]'
 ```
 
-This pulls in `google-antigravity` (pinned to `0.1.18`), whose wheel bundles the
+This pulls in `google-antigravity` (pinned to `0.1.20`), whose wheel bundles the
 platform `localharness` binary. As with the other agents the SDK is imported lazily
 — a base install without the extra still runs end-to-end; Antigravity tasks fail at
 dispatch with a clear hint to install the extra.
@@ -136,18 +136,30 @@ can't be resolved or if zero skills are discovered.
 
 ## Permissions & tools — important differences
 
-**Antigravity ignores `permission_mode`, `allowed_tools`, and `disallowed_tools`.**
-The local harness runs in a single unconditional mode: every tool call (including
-`run_command`) is approved via an allow-all policy, and file tools are restricted to
-the configured `workspaces` (the sandbox working directory plus any skill roots).
+**Antigravity ignores `permission_mode`.** The local harness runs in a single
+unconditional mode: every tool call (including `run_command`) is approved via an
+allow-all policy, and file tools are restricted to the configured `workspaces` (the
+sandbox working directory plus any skill roots).
+
+**`allowed_tools` / `disallowed_tools` are enforced** by mapping the Claude tool names
+onto the harness's builtin tools (`Bash` → `run_command`, `Read` → `view_file`, `Write` →
+`create_file`, `Edit` → `edit_file`, `Glob` → `find_file`, `Grep` → `search_directory`,
+`Task` → `start_subagent`, `WebSearch` → `search_web`, `WebFetch` → `read_url_content`).
+`Skill` has no builtin and is skipped. `finish`, `schedule` and `start_subagent` always
+stay on, the last so the harness keeps its default system prompt; when the lists do not
+allow `Task`, running a subagent (`invoke_subagent`) is denied by policy instead.
+
+`run_command` also gets non-interactive environment variables (`CI=1`,
+`npm_config_yes=true`, `GIT_TERMINAL_PROMPT=0`, `DEBIAN_FRONTEND=noninteractive`,
+`PIP_NO_INPUT=1`, `PAGER=cat`), each only where the environment does not already set it.
 
 The trust boundary for an Antigravity run is therefore the **sandbox**, not the
 agent config. Run untrusted tasks under the [Docker driver](../DOCKER_ISOLATION.md);
 the `tempdir` driver is not a security boundary. This mirrors the reality that the
 `bypassPermissions`-equivalent behavior is always on for this backend.
 
-Those inherited fields still exist on the config for schema uniformity but have no
-runtime effect here — don't rely on them to gate Antigravity.
+`permission_mode` still exists on the config for schema uniformity but has no runtime
+effect here — don't rely on it to gate Antigravity.
 
 ## Telemetry
 
@@ -187,19 +199,18 @@ as every other agent.
 4. **`permission_mode` does not confine the harness.** Every mode runs
    `policy.allow_all()`; coder_eval's write boundary is the sandbox driver, and a
    headless eval has no human to approve anything.
-5. **`allowed_tools` / `disallowed_tools` are not read.** The harness runs with its
-   full builtin tool set, so an Antigravity run has tools (web search, subagents,
-   URL fetch) that the same task file denies on Claude Code and Codex.
-6. **`max_turns` is counted by the harness.** One `communicate()` is a single SDK turn
+5. **`max_turns` is counted by the harness.** One `communicate()` is a single SDK turn
    here, so the harness counts model API calls itself (a MODEL step at a new
    `step_index` opens one) and enforces the cap on the step loop. See
    [Run-Limit Parity](HARNESS_PARITY.md).
-7. **Shell commands over ~10s are moved to the background.** The localharness has a
+6. **Shell commands over ~10s are moved to the background.** The localharness has a
    10-second maximum synchronous wait; past it the command becomes a background task
    and the model gets a task id, not a result. The turn polls for that result instead
-   of finalizing on an idle step stream, so slow work does complete — but the wait is
-   bounded by 80% of `turn_timeout`, and a job that outlives it is force-closed as
-   `result_status: unknown` and graded as an ordinary low score rather than a timeout.
+   of finalizing on an idle step stream, so slow work does complete — but only an
+   orphaned `run_command` is waited on, the wait is bounded by 80% of `turn_timeout`
+   (10 minutes when the task sets none), and a job that outlives it (typically a server
+   the model left running) is force-closed as `result_status: unknown` and graded
+   normally rather than as a timeout.
    Measured in [Run-Limit Parity](HARNESS_PARITY.md).
 
 ## Running in Docker

{coder_eval-0.12.9 → coder_eval-0.12.11}/docs/agents/HARNESS_PARITY.md

@@ -718,9 +718,10 @@ both. See [OpenCode](OPENCODE.md) and [Pi § plugins](PI.md#known-limitations).
   lowercase (`bash`/`read`/`write`/`edit`/`grep`/`find`/`ls`), but the shared config
   default (`experiments/default.yaml`) sets Claude-namespaced names
   (`Bash`/`Read`/`Write`/…). Forwarding those to `--tools` would allowlist tools that
-  do not exist in Pi and strip the agent of ALL tools — so, like OpenCode (drops them),
-  Codex (forwards `disallowed_tools` without SDK enforcement), and Antigravity (does not
-  read them), Pi ignores them and runs with its full native toolset. A task that needs a
+  do not exist in Pi and strip the agent of ALL tools — so, like OpenCode (drops them)
+  and Codex (forwards `disallowed_tools` without SDK enforcement), Pi ignores them and
+  runs with its full native toolset. (Antigravity maps them onto its builtin tools; see
+  [Antigravity](ANTIGRAVITY.md).) A task that needs a
   restricted Pi toolset would have to name Pi's lowercase tools — a documented follow-up.
 - **`permission_mode` is NOT enforced** — Pi headless print mode auto-runs tools and
   exposes only project-file trust (`--approve` / `--no-approve`), no tool-approval

{coder_eval-0.12.9 → coder_eval-0.12.11}/plugins/coder-eval/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "coder-eval",
   "displayName": "Coder Eval",
-  "version": "0.12.9",
+  "version": "0.12.11",
   "description": "Test whether your Claude Code skills actually trigger, and benchmark any coding agent — author, run, and analyze the eval suite that proves it, locally or as a CI gate.",
   "author": { "name": "UiPath", "email": "coder-eval@uipath.com", "url": "https://github.com/UiPath/coder_eval" },
   "homepage": "https://coder-eval.com",

{coder_eval-0.12.9 → coder_eval-0.12.11}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "coder-eval"
-version = "0.12.9"
+version = "0.12.11"
 description = "Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode, Pi, Delegate) with sandboxed, reproducible YAML task suites."
 readme = "README.md"
 license = "Apache-2.0"
@@ -131,7 +131,7 @@ codex = [
 # Without this extra the framework still installs and runs; Antigravity-dependent
 # code paths fail at start() with a clear hint pointing back here.
 antigravity = [
-    "google-antigravity==0.1.18",
+    "google-antigravity==0.1.20",
 ]
 # Optional extra that enables OpenCode agent support.
 #

{coder_eval-0.12.9 → coder_eval-0.12.11}/src/coder_eval/__init__.py

@@ -1,3 +1,3 @@
 """coder_eval - A framework for evaluating AI coding agents."""
 
-__version__ = "0.12.9"
+__version__ = "0.12.11"

{coder_eval-0.12.9 → coder_eval-0.12.11}/src/coder_eval/agents/antigravity_agent.py

@@ -95,13 +95,19 @@ _RECEIVE_STEPS_REENTRY_RETRIES = 5
 # Rationale: .claude/notes/agents.md § Antigravity Step interleaving and the background poll
 _POLL_DEADLINE_TIMEOUT_FRACTION = 0.8
 
-# Cap on poll *cycles* -- the SOLE bound when a task sets no timeout at all, and
-# a backstop against a very large one. 120 * 5s = 10 minutes, ~2x the worst real
-# backgrounded-job duration observed (60-300s). Deliberately NOT "break after N
-# consecutive empty polls".
+# Cap on poll *cycles*, the bound only when a task sets no timeout at all; with
+# one, the deadline alone bounds the wait. 120 * 5s = 10 minutes. Deliberately
+# NOT "break after N consecutive empty polls".
 # Rationale: .claude/notes/agents.md § Antigravity Step interleaving and the background poll
 _MAX_BACKGROUND_POLLS = 120
 
+# The only tool a model can background: a `run_command` left ACTIVE is a job
+# that may still finish. Any other tool left ACTIVE (edit_file on a read-only
+# mount, a paged view_file, start_subagent) never resolves, so it does not
+# justify polling.
+# Rationale: .claude/notes/agents.md § Antigravity Step interleaving and the background poll
+_BACKGROUNDABLE_TOOL = "run_command"
+
 # Antigravity builtin tool name -> the canonical (Claude) vocabulary every
 # criterion is written against. Unmapped names pass through unchanged.
 # Rationale: .claude/notes/agents.md § Tool-name and argument normalization
@@ -120,6 +126,51 @@ _ANTIGRAVITY_TO_CLAUDE_TOOL_MAP: dict[str, str] = {
     "finish": "Finish",
 }
 
+# Canonical (Claude) tool name -> the Antigravity builtin it enables, for
+# `allowed_tools` / `disallowed_tools`. "Skill" and "TodoWrite" have no builtin
+# (skills load through `skills_paths`), so they map to nothing; a name that is
+# already an Antigravity builtin passes through.
+# Rationale: .claude/notes/agents.md § Antigravity tool allowlist
+_CLAUDE_TO_ANTIGRAVITY_TOOL_MAP: dict[str, str] = {
+    "Bash": "run_command",
+    "Read": "view_file",
+    "Write": "create_file",
+    "Edit": "edit_file",
+    "MultiEdit": "edit_file",
+    "Glob": "find_file",
+    "Grep": "search_directory",
+    "LS": "list_directory",
+    "Task": "start_subagent",
+    "WebSearch": "search_web",
+    "WebFetch": "read_url_content",
+    "AskUserQuestion": "ask_question",
+}
+
+# Kept on under any allowlist: `finish` returns a turn's structured output,
+# `schedule` is how the model waits on a backgrounded command, and the harness
+# drops its "you will be notified, do not poll" guidance with `start_subagent`.
+# Withheld subagents are denied at `_SUBAGENT_CALL` instead.
+# Rationale: .claude/notes/agents.md § Antigravity tool allowlist
+_ALWAYS_ENABLED_TOOLS: frozenset[str] = frozenset({"finish", "schedule", "start_subagent"})
+
+# The one call that runs a subagent, built-in or model-defined.
+_SUBAGENT_CALL = "invoke_subagent"
+
+# Set on every run_command unless the environment already sets them, so a
+# command that would stop to ask (`npx` installing a package, git credentials,
+# apt/pip confirmations, a pager) answers itself or fails instead of waiting on
+# a terminal no one types into.
+# Rationale: .claude/notes/agents.md § Antigravity non-interactive commands
+_NONINTERACTIVE_ENV: dict[str, str] = {
+    "CI": "1",
+    "npm_config_yes": "true",
+    "GIT_TERMINAL_PROMPT": "0",
+    "DEBIAN_FRONTEND": "noninteractive",
+    "PIP_NO_INPUT": "1",
+    "PAGER": "cat",
+    "GIT_PAGER": "cat",
+}
+
 # Tool-call arg keys the harness ADDS at completion (the result payload), not
 # model-supplied inputs. The STATIC backstop; ``_params`` also strips any key
 # that first appears at DONE. A leaked result would false-positive
@@ -303,18 +354,61 @@ class AntigravityAgent(Agent[AntigravityAgentConfig]):
     def _harness_env(self) -> dict[str, str] | None:
         """Per-agent environment for the localharness subprocess (``LocalAgentConfig.env``).
 
-        Returns the mock-CLI PATH prepend as a one-key overlay, or ``None`` when no
-        mock dirs are configured. The SDK merges it over ``os.environ`` at spawn,
-        so naming only ``PATH`` leaves every other inherited variable untouched.
+        An overlay of the non-interactive variables (each only where the
+        environment does not already set it) plus, when mock dirs are configured,
+        the mock-CLI PATH prepend; ``None`` when there is nothing to add, so the
+        SDK spawns with a plain inherited env. The SDK merges it over
+        ``os.environ`` at spawn, so every other inherited variable is untouched.
         The same overlay becomes the harness's ``run_command`` environment.
         """
-        if not self._env_path_prepend:
+        env = {k: v for k, v in _NONINTERACTIVE_ENV.items() if k not in os.environ}
+        if self._env_path_prepend:
+            # Match the parent's own casing (Windows exports ``Path``) so the merge
+            # overrides the inherited entry instead of adding a sibling key.
+            path_key = next((k for k in os.environ if k.upper() == "PATH"), "PATH")
+            env[path_key] = os.pathsep.join([*self._env_path_prepend, os.environ.get(path_key) or ""])
+        return env or None
+
+    def _tool_capabilities(self, types: Any) -> Any:
+        """``CapabilitiesConfig`` restricting the builtin tools to the configured
+        ``allowed_tools`` / ``disallowed_tools``, or ``None`` when neither is set.
+
+        Names are mapped through ``_CLAUDE_TO_ANTIGRAVITY_TOOL_MAP``; names with no
+        Antigravity builtin (``Skill``, ``TodoWrite``, MCP tools) are skipped. An
+        allowlist only narrows the harness's default toolset, so ``Glob`` / ``Grep`` /
+        ``LS`` never turn on the tools the harness ships off (``find_file``,
+        ``search_directory``, ``list_directory``). The SDK takes an allowlist OR a
+        denylist, so with both set the denied tools are removed from the allowlist.
+        ``_ALWAYS_ENABLED_TOOLS`` are never removed.
+
+        Rationale: .claude/notes/agents.md § Antigravity tool allowlist
+        """
+        allowed, disallowed = self.config.allowed_tools, self.config.disallowed_tools
+        if not allowed and not disallowed:
             return None
-        # Match the parent's own casing (Windows exports ``Path``) so the merge
-        # overrides the inherited entry instead of adding a sibling key.
-        path_key = next((k for k in os.environ if k.upper() == "PATH"), "PATH")
-        merged = os.pathsep.join([*self._env_path_prepend, os.environ.get(path_key) or ""])
-        return {path_key: merged}
+        builtin = {t.value for t in types.BuiltinTools}
+        harness_default = {t.value for t in types.BuiltinTools.default()}
+
+        def to_builtin(names: list[str] | None) -> set[str]:
+            mapped = {_CLAUDE_TO_ANTIGRAVITY_TOOL_MAP.get(n, n) for n in names or []}
+            return mapped & builtin
+
+        if allowed:
+            enabled = ((to_builtin(allowed) & harness_default) - to_builtin(disallowed)) | _ALWAYS_ENABLED_TOOLS
+            self._log.debug("Enabled builtin tools: %s", ", ".join(sorted(enabled)))
+            return types.CapabilitiesConfig(enabled_tools=[types.BuiltinTools(t) for t in sorted(enabled)])
+        disabled = to_builtin(disallowed) - _ALWAYS_ENABLED_TOOLS
+        self._log.debug("Disabled builtin tools: %s", ", ".join(sorted(disabled)))
+        return types.CapabilitiesConfig(disabled_tools=[types.BuiltinTools(t) for t in sorted(disabled)])
+
+    def _subagents_allowed(self) -> bool:
+        """Whether ``allowed_tools`` / ``disallowed_tools`` let the model run subagents (``Task``)."""
+
+        def names_task(names: list[str] | None) -> bool:
+            return any(_CLAUDE_TO_ANTIGRAVITY_TOOL_MAP.get(n, n) == "start_subagent" for n in names or [])
+
+        allowed, disallowed = self.config.allowed_tools, self.config.disallowed_tools
+        return (not allowed or names_task(allowed)) and not names_task(disallowed)
 
     async def start(
         self,
@@ -365,14 +459,17 @@ class AntigravityAgent(Agent[AntigravityAgentConfig]):
                 # policy would deny. ``permission_mode`` is deliberately NOT mapped
                 # here — it does not confine this agent, exactly as on Codex, and
                 # docs/agents/HARNESS_PARITY.md says so rather than leaving it
-                # silent. The isolation boundary is the driver.
-                policies=[policy.allow_all()],
+                # silent. The isolation boundary is the driver. Subagents the tool
+                # lists withhold are denied here, since `start_subagent` stays on.
+                policies=[policy.allow_all()] + ([] if self._subagents_allowed() else [policy.deny(_SUBAGENT_CALL)]),
                 system_instructions=self.config.system_prompt or None,
                 # Skill discovery: the search-path roots that parent the skill dirs.
                 skills_paths=skills_paths,
-                # Mock-CLI PATH shadowing, per agent: two concurrent tasks never
-                # see each other's mock dirs.
+                # Mock-CLI PATH shadowing plus the non-interactive variables, per
+                # agent: two concurrent tasks never see each other's mock dirs.
                 env=self._harness_env(),
+                # The builtin toolset, narrowed to allowed_tools / disallowed_tools.
+                capabilities=self._tool_capabilities(types),
             )
             # Thinking level onto every resolved model's endpoint. The SDK
             # validates the model list in a model_validator, so options are set on
@@ -761,6 +858,8 @@ class _AntigravityTurnState:
         # Most recently seen StepStatus per tool id, for has_orphaned_tool_call.
         # Separate from _closed_tools, which tracks only DONE/ERROR.
         self._tool_last_status: dict[str, Any] = {}
+        # Raw (Antigravity) tool name per tool id, for has_orphaned_tool_call.
+        self._tool_raw_names: dict[str, Any] = {}
         # Content blocks accumulated since the last per-generation flush.
         self._blocks: list[ContentBlock] = []
         # Where the CURRENT generation started, advanced only by a flush that
@@ -865,6 +964,7 @@ class _AntigravityTurnState:
         step_key = f"{trajectory_id}:{step.step_index}" if trajectory_id else str(step.step_index)
         cid = call.id or f"{raw_name}_{step_key}_{call_index}"
         self._tool_last_status[cid] = sstatus
+        self._tool_raw_names[cid] = raw_name
         if cid not in self._seen_tools:
             self._seen_tools.add(cid)
             seq = self._next_seq
@@ -990,8 +1090,8 @@ class _AntigravityTurnState:
         return ""
 
     def has_orphaned_tool_call(self) -> bool:
-        """True if any NOT-YET-CLOSED tool call's most recently seen status is
-        ACTIVE — the structural signature of a backgrounded task the model went
+        """True if any NOT-YET-CLOSED ``run_command``'s most recently seen status
+        is ACTIVE — the structural signature of a backgrounded task the model went
         idle on without waiting for. See ``communicate``'s poll loop.
 
         An ALLOWLIST on ACTIVE, never a denylist on "not yet closed": the SDK also
@@ -999,9 +1099,18 @@ class _AntigravityTurnState:
         should wait out. The `not in _closed_tools` guard is layered on top as a
         monotonicity backstop, not a substitute.
 
+        Also an allowlist on the TOOL: only ``run_command`` can be backgrounded.
+        Any other tool left ACTIVE never resolves, so waiting on it only burns
+        the poll budget; ``finalize`` force-closes it as unresolved.
+
         Rationale: .claude/notes/agents.md § Antigravity Step interleaving and the background poll
         """
-        return any(cid not in self._closed_tools and s == _STATUS_ACTIVE for cid, s in self._tool_last_status.items())
+        return any(
+            cid not in self._closed_tools
+            and s == _STATUS_ACTIVE
+            and self._tool_raw_names.get(cid) == _BACKGROUNDABLE_TOOL
+            for cid, s in self._tool_last_status.items()
+        )
 
     def finalize(self, status: AgentEndStatus, *, crashed: bool = False, crash_reason: str | None = None) -> None:
         """Close orphaned tools, flush leftover blocks, emit TurnEnd + AgentEnd.

{coder_eval-0.12.9 → coder_eval-0.12.11}/src/coder_eval/criteria/cli_called.py

@@ -170,7 +170,12 @@ class CliCalledChecker(BaseCriterion[CliCalledCriterion]):
             facets.append(f"flags={sorted(criterion.flags)}")
         wanted = ", ".join(facets)
 
-        if score == 1.0:
+        if score == 1.0 and count == 0 and criterion.max_count == 0:
+            details = (
+                f"0 invocation(s) matched ({wanted}); guard unproven: a typo in verb, tool or flags "
+                "would also match nothing"
+            )
+        elif score == 1.0:
             details = f"{count} invocation(s) matched ({wanted}); satisfies {bound}"
         elif not within_lower:
             # A bare count sends the reader to the sandbox; this criterion exists