coder-eval 0.11.7__tar.gz → 0.12.1__tar.gz

{coder_eval-0.11.7 → coder_eval-0.12.1}/.claude/commands/coder-eval-review.md

@@ -24,7 +24,8 @@ The run layout (`runs/<run_id>/<variant_id>/<task_id>/<NN>/…`, `<NN>` a zero-p
 1. Read `<run_path>/run.json` if present (for context — `run_id`, `start_time`).
 2. Glob `<run_path>/*/*/*/task.json` and read each one.
 3. Read `<run_path>/analysis.md` if present — it already diagnoses many failures; lean on its findings rather than re-deriving them.
-4. A task counts as **failed** if `final_status != "SUCCESS"` **or** `weighted_score < 0.9`. Skip passing tasks for now (we may extend to passing tasks later — the schema supports it).
+4. Skip any task whose `final_status` is `"NOT_GRADED"` — `coder-eval execute` produced it, no criterion ran, and `weighted_score` is `null`. It is neither a pass nor a failure, and comparing `null < 0.9` would book every ungraded row as a failure to review.
+5. Of the rest, a task counts as **failed** if `final_status != "SUCCESS"` **or** `weighted_score < 0.9`. Skip passing tasks for now (we may extend to passing tasks later — the schema supports it).
 
 If no `task.json` files exist, write an empty `review_index.json` (`{"reviews": []}`) and exit.
 

{coder_eval-0.11.7 → coder_eval-0.12.1}/.claude/harness-candidates.md

@@ -455,3 +455,99 @@ with the two `action.yml` items above — one considered change to the action's
   `final_status`, which does not exist in `run.json` and would have made a new
   assertion dead on arrival. Guard: assert the key set that non-Python consumers
   depend on, mirroring how CE030 pins doc/schema parity.
+- [ ] **A probe task's detector must not be satisfiable from the sandbox-readable task
+  YAML.** `docker_runner._stage_inputs` serialises the post-override `TaskDefinition` to
+  `/work/input/task.yaml` and mounts `tasks/` again at `/work/task_dir`, both agent-readable.
+  Two probes now depend on NOT being satisfiable from that text — `anti_cheat_reference` via a
+  regex that cannot match its own source, `record_cli_responses` via a log-derived detector —
+  and nothing enforces it. `record_cli_responses` originally shipped (in review) with
+  `file_contains` needles that were verbatim in its own YAML, which would have let a
+  transcribing agent pass while dispatch was dead. Guard: for every `smoke-pass` task, assert no
+  `file_contains` needle / `file_matches_regex` pattern on a must-match criterion appears in the
+  serialised task YAML. Deferred: needs per-criterion-type handling and a real false-positive
+  pass (paths and generic words will collide), so well over 30 min.
+- [ ] **A new shim failure mode must still RECORD the invocation.** A generated shim that dies
+  before `record()` leaves a log byte-identical to "the agent never ran it", which passes a
+  `max_count: 0` guard. The sidecar import was exactly that, caught only in review. Guard:
+  render each shim shape, break each external dependency in turn, assert the log is non-empty.
+  Deferred: "each external dependency" has no enumeration today, so the rule needs a seam
+  (a declared list of what a shim depends on) before it can be mechanical rather than a
+  hand-maintained list that decays.
+- [ ] **A generated-artifact invariant must be asserted against a real run of that artifact,
+  not against the config that produced it.** `TestRecordCliProbeIntegrity` first shipped
+  comparing the task YAML with itself and hardcoding `"rule": 0` — a spelling `json.dumps`'s
+  default separators own — so a separator change would have left it green while the blocking CI
+  probe failed. Now fixed for this case by running a real shim. Deferred as a general guard:
+  "derives its expectation from the thing it checks" is not mechanically detectable; it belongs
+  in the review rubric rather than a lint rule.
+
+- [ ] A `_*TurnState` (agent turn-state) attribute that is written but never read
+  outside its own assignment — CE037-class dead accumulator. Surfaced during the
+  Pi harness port: `_PiTurnState.turns_finished` was copied from OpenCode's
+  `steps_finished` (which drives that agent's `finished_without_tokens` guard) but
+  Pi deliberately dropped that guard, leaving the counter dead. Fixed by hand this
+  run. Guard would need cross-method dataflow over each `Agent`-subclass turn-state
+  class (write sites vs read sites), which the AST-only CExxx runner can't express
+  in ~30 min — deferred. Caught in: Pi harness Phase 2 quality review.
+
+- [ ] A duration/count aggregate over run-task rows that fails to exclude
+  `mature_skipped`. Nothing guards it: the CExxx runner is Python-AST only and
+  this defect class lives in the evalboard's TypeScript. A codex nightly
+  rendered "1300 tasks · 15h 29m" for the 397 tasks that actually ran. Fixed
+  for the two whole-run sites by extracting `deriveRunDuration`; a general
+  guard needs a TS lint surface the harness does not have.
+  Caught in: timing-capture Phase 1 review.
+
+- [ ] Subtracting a SUM of intervals from a wall span where the intervals can
+  overlap. Semantic, not syntactic — an AST rule cannot tell a sum of
+  durations from a union. Antigravity shipped it: four concurrent 400ms tool
+  calls inside a 1000ms window summed to 1600ms and clamped generation to the
+  0.0 the change existed to remove. The real guard is the replay-based
+  `assert_timing_captured` golden sensor, which now exists; a static rule
+  would not have caught it. Caught in: timing-capture Phase 3 review.
+
+- [ ] A test fixture whose field name does not exist on the type it models.
+  `parseMessages`'s `CommandEntry` keys on `tool_id`; a fixture using
+  `tool_use_id` never resolves, params fall back to `{}`, and every tool
+  weighs exactly 1 — which silently turned a "split by content size" test
+  into a 99%-thinking assertion that passed. TypeScript accepts it because
+  the fixtures are untyped object literals. Typing the fixture factories
+  against the real interfaces would guard the whole class; that is a
+  sweep across the evalboard test suite, not ~30 min.
+  Caught in: timing-capture Phase 5 review.
+
+### Deferred timing divergences (not guardrails — accounting gaps)
+
+Recorded here as well as in `docs/agents/HARNESS_PARITY.md` § Known
+divergences, so the deferred-work record is one place. Measurements in
+`c/time-bugs-audit.md`.
+
+- [ ] **Antigravity books orphan-poll waiting as agent duration** (audit P2-1).
+  A task can spend `0.8 × turn_timeout` waiting on a tool call that never
+  reaches DONE — 14 tasks, 9.6h of one 83h run. Only CLOSED tool intervals are
+  subtracted from a generation window, so that wait stays inside whichever
+  window contains it; the force-close records `execution_completed_at` while
+  leaving `duration_ms` as `None`. Deliberately out of scope of the timing
+  work: closing it means deciding whether a backgrounded tool's elapsed time
+  is model time (the model IS generating while it runs), which is a semantic
+  question, not a bug fix.
+
+- [ ] **Delegate (`delegate-sdk`) records no execution bounds** (audit P3-1).
+  It reports `duration_ms` but neither `execution_started_at` nor
+  `execution_completed_at`, so its tool calls cannot be placed on a timeline.
+  Coverage is ~88%, so it is not urgent. The agent lives in the separate
+  `coder_eval_uipath` repo; mirror the Codex change there
+  (`_item_timing` + threading the SDK stamps through the telemetry builders).
+
+- [ ] **Pre-existing, surfaced by this work's final review: `TokenUsage._adopt_legacy_input_tokens`
+  double-counts the cache buckets.** The validator copies a legacy record's
+  full-prompt `input_tokens` straight into `uncached_input_tokens`, and the
+  computed `input_tokens` then adds `cache_creation` + `cache_read` again. A
+  legacy record with `input_tokens=1000`, `cache_creation=200`, `cache_read=150`
+  reloads as 1350 prompt tokens and bills 1000 at the uncached rate instead of
+  650. Affects every report, budget check and detached regrade over a
+  pre-split run that used prompt caching. NOT touched by the timing work
+  (token accounting was explicitly out of its scope) and not a guardrail
+  candidate — a real bug needing its own change, with a decision about
+  whether legacy records can be distinguished from current ones at all.
+  Caught in: timing-capture final review (gpt-5.6-sol).

{coder_eval-0.11.7 → coder_eval-0.12.1}/.claude/shared/run-layout.md

@@ -12,6 +12,9 @@ runs/<run_id>/<variant_id>/<task_id>/<NN>/{task.json, task.log, artifacts/}
 - `<NN>` — zero-padded replicate index (e.g. `00`, `01`).
 - `task.json` — the persisted per-replicate result (the consumer contract; carries the large `iterations` array — still accepted under its former name `turns` when reading, but not what current runs write).
 - `task.json.malformed` — present only on the docker degrade path: when an existing `task.json` fails to parse (schema skew from a stale `:latest` image, or a truncated/torn write), the docker runner moves the unparseable original aside to this sidecar and writes a synthetic `final_status=ERROR` `task.json` in its place. Diagnostic-only; `rglob("task.json")` consumers do not match it.
+- `task.execute.json` — present only after a DETACHED grade (`coder-eval evaluate <run_dir>` or `coder-eval run --resume` over a `NOT_GRADED` row). The pre-grade snapshot of `task.json`, written once and never overwritten by a later grade, so "this run was executed separately from grading" stays auditable. Diagnostic-only; `rglob("task.json")` consumers do not match it.
+- `task.json.graded` — present only after `coder-eval execute --driver docker` refused a container's verdict: the runtime image predated `execute` and graded anyway, so the runner quarantines the graded record here rather than leaving it readable as `task.json`, where a later `--resume` / `aggregate` would fold in exactly the row it declined to publish. Diagnostic-only; `rglob("task.json")` consumers do not match it.
+- `grade.log` — present only after a DETACHED grade over this directory (`coder-eval run --resume`). The grading pass's own log. It is a separate file because the log handler truncates whatever file it opens, so writing to `task.log` would destroy the agent trajectory log the run already paid for.
 - `task.log` — the human-readable task log; `artifacts/` — files the agent produced.
 
 **Scope-marker files** (used to detect what a given path represents):

{coder_eval-0.11.7 → coder_eval-0.12.1}/.github/pages-stub/index.html

@@ -12,7 +12,7 @@
     -->
     <meta
       name="description"
-      content="Coder Eval is Playwright for coding agents: an open-source, agent-agnostic framework for evaluating and benchmarking AI coding agents and their skills — sandboxed runs of Claude Code, Codex, Antigravity (Gemini), or OpenCode against declarative YAML tasks, with weighted scoring and CI gates. Documentation: coder-eval.com/docs"
+      content="Coder Eval is Playwright for coding agents: an open-source, agent-agnostic framework for evaluating and benchmarking AI coding agents and their skills — sandboxed runs of Claude Code, Codex, Antigravity (Gemini), OpenCode, or Pi against declarative YAML tasks, with weighted scoring and CI gates. Documentation: coder-eval.com/docs"
     />
 
     <!--
@@ -227,7 +227,7 @@
       <p class="lead">
         <strong>Playwright for coding agents.</strong> An open-source, agent-agnostic framework for
         evaluating and benchmarking AI coding agents and their skills: it runs a real agent — Claude
-        Code, Codex, Antigravity (Gemini), or OpenCode — in a sandbox against declarative YAML
+        Code, Codex, Antigravity (Gemini), OpenCode, or Pi — in a sandbox against declarative YAML
         tasks, then scores the files and commands the agent actually produced.
       </p>
       <p class="notice">

coder_eval-0.12.1/.github/scripts/harbor_e2e.py

@@ -0,0 +1,163 @@
+#!/usr/bin/env python3
+"""Harbor end-to-end smoke test.
+
+Exports a handful of coder-eval tasks to Harbor (`coder-eval export --format
+harbor`), runs each with `hb run -a coder_eval.harbor.agent:CoderEvalAgent`
+against real Docker, and asserts that:
+
+  - the verifier phase wrote `reward.json` with `reward == 1.0`
+  - the agent phase wrote a real ATIF `trajectory.json`
+  - both the agent and verifier phases left a `task.json` behind
+
+Invoked by `.github/workflows/harbor-e2e.yml` -- deliberately NOT part of
+`make test`: it shells out to a real `hb` CLI, real Docker builds, and (for
+the llm_judge scenario) a real model call, none of which belong in the fast
+unit-test suite.
+"""
+
+from __future__ import annotations
+
+import json
+import shutil
+import subprocess
+import sys
+from dataclasses import dataclass
+from pathlib import Path
+
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+WORK_DIR = REPO_ROOT / "tmp" / "harbor_e2e"
+AGENT_IMPORT_PATH = "coder_eval.harbor.agent:CoderEvalAgent"
+
+
+@dataclass(frozen=True)
+class Scenario:
+    """One (task.yaml, export flags) pair to round-trip through Harbor."""
+
+    name: str
+    task_file: Path
+    allow_credentials: bool = False
+
+
+SCENARIOS: list[Scenario] = [
+    # "tmpdir" baseline: plain docker driver, default coder-eval-agent image,
+    # no dockerfile_path / template_sources / llm_judge -- exercises the bare
+    # export -> CoderEvalAgent -> --workspace-dir -> verifier round trip.
+    Scenario("baseline", REPO_ROOT / "tests/harbor_e2e/fixtures/docker_baseline.yaml"),
+    # llm_judge: real model call inside the VERIFIER phase, not just the agent.
+    Scenario("llm_judge", REPO_ROOT / "tests/harbor_e2e/fixtures/llm_judge.yaml", allow_credentials=True),
+    # Custom (BYOD) Docker image via dockerfile_path -- reuses the in-tree
+    # byod_smoke_test task/image rather than duplicating it.
+    Scenario("docker_custom_image", REPO_ROOT / "tasks/byod_smoke_test.yaml"),
+    # template_sources: TemplateDirSource copy-in + rewritten path, plus
+    # sandbox.python.env_packages surviving the agent-phase task.yaml merge.
+    Scenario("template_sources", REPO_ROOT / "tests/harbor_e2e/fixtures/template_sources.yaml"),
+]
+
+
+def _run(cmd: list[str]) -> subprocess.CompletedProcess[str]:
+    print(f"+ {' '.join(cmd)}", flush=True)
+    return subprocess.run(cmd, check=False, text=True, capture_output=True)
+
+
+def export_task(scenario: Scenario, out_dir: Path) -> None:
+    if out_dir.exists():
+        shutil.rmtree(out_dir)
+    cmd = ["coder-eval", "export", str(scenario.task_file), "-o", str(out_dir)]
+    if scenario.allow_credentials:
+        cmd.append("--allow-credentials")
+    result = _run(cmd)
+    print(result.stdout)
+    print(result.stderr, file=sys.stderr)
+    if result.returncode != 0:
+        raise RuntimeError(f"[{scenario.name}] `coder-eval export` failed (exit {result.returncode})")
+
+
+def run_harbor(scenario: Scenario, export_dir: Path, jobs_dir: Path) -> Path:
+    if jobs_dir.exists():
+        shutil.rmtree(jobs_dir)
+    cmd = [
+        "hb",
+        "run",
+        "-p",
+        str(export_dir),
+        "-a",
+        AGENT_IMPORT_PATH,
+        "--jobs-dir",
+        str(jobs_dir),
+        "-n",
+        "1",
+        "-y",
+    ]
+    result = _run(cmd)
+    print(result.stdout)
+    print(result.stderr, file=sys.stderr)
+    if result.returncode != 0:
+        raise RuntimeError(f"[{scenario.name}] `hb run` failed (exit {result.returncode})")
+
+    # <jobs_dir>/<job_timestamp>/<trial_name>/{agent,verifier}/... -- glob for
+    # the one directory two levels down that actually holds a verifier/ output,
+    # rather than assuming a fixed trial-name shape Harbor doesn't guarantee.
+    trial_dirs = [d for d in jobs_dir.glob("*/*/") if (d / "verifier").is_dir()]
+    if len(trial_dirs) != 1:
+        raise RuntimeError(
+            f"[{scenario.name}] expected exactly one trial directory under {jobs_dir}, found {len(trial_dirs)}"
+        )
+    return trial_dirs[0]
+
+
+def assert_scenario_artifacts(scenario: Scenario, trial_dir: Path) -> None:
+    reward_path = trial_dir / "verifier" / "reward.json"
+    if not reward_path.is_file():
+        raise RuntimeError(f"[{scenario.name}] missing {reward_path}")
+    reward = json.loads(reward_path.read_text(encoding="utf-8"))
+    if reward.get("reward") != 1.0:
+        raise RuntimeError(f"[{scenario.name}] expected reward 1.0, got {reward!r} ({reward_path})")
+
+    trajectory_path = trial_dir / "agent" / "trajectory.json"
+    if not trajectory_path.is_file():
+        raise RuntimeError(f"[{scenario.name}] missing {trajectory_path}")
+    trajectory = json.loads(trajectory_path.read_text(encoding="utf-8"))
+    if "schema_version" not in trajectory:
+        raise RuntimeError(f"[{scenario.name}] {trajectory_path} is missing 'schema_version'")
+
+    agent_task_jsons = list((trial_dir / "agent").glob("**/task.json"))
+    if not agent_task_jsons:
+        raise RuntimeError(f"[{scenario.name}] no task.json found under {trial_dir / 'agent'}")
+    verifier_task_json = trial_dir / "verifier" / "task.json"
+    if not verifier_task_json.is_file():
+        raise RuntimeError(f"[{scenario.name}] missing {verifier_task_json}")
+
+    print(
+        f"[{scenario.name}] OK: reward=1.0, trajectory.json present, "
+        + f"{len(agent_task_jsons)} agent task.json + verifier/task.json present"
+    )
+
+
+def main() -> int:
+    WORK_DIR.mkdir(parents=True, exist_ok=True)
+    failures: list[str] = []
+    for scenario in SCENARIOS:
+        export_dir = WORK_DIR / scenario.name / "export"
+        jobs_dir = WORK_DIR / scenario.name / "jobs"
+        print(f"\n=== {scenario.name} ===", flush=True)
+        try:
+            export_task(scenario, export_dir)
+            trial_dir = run_harbor(scenario, export_dir, jobs_dir)
+            assert_scenario_artifacts(scenario, trial_dir)
+        except Exception as exc:
+            print(f"[{scenario.name}] FAILED: {exc}", file=sys.stderr)
+            failures.append(scenario.name)
+
+    print("\n=== Summary ===")
+    for scenario in SCENARIOS:
+        print(f"  {scenario.name}: {'FAILED' if scenario.name in failures else 'OK'}")
+
+    if failures:
+        print(f"\n{len(failures)}/{len(SCENARIOS)} scenario(s) failed: {', '.join(failures)}", file=sys.stderr)
+        return 1
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

coder_eval-0.12.1/.github/workflows/harbor-e2e.yml

@@ -0,0 +1,91 @@
+name: Harbor E2E
+
+# Deliberately NOT triggered on pull_request: this exercises real Docker
+# builds, a real `harbor` install, and (for the llm_judge scenario) a real
+# model call, so it is informational rather than a required PR check for now
+# (see .github/scripts/harbor_e2e.py's module docstring). workflow_dispatch
+# lets a maintainer run it on demand; the nightly schedule catches drift
+# between coder-eval's own release and Harbor's own upstream releases without
+# blocking anyone's PR.
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "17 5 * * *" # nightly, off the hour to avoid GitHub's peak-load pile-up
+  push:
+    branches: [main]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+env:
+  TELEMETRY_ENABLED: "false"
+  # Route through Bedrock, mirroring pr-checks.yml's smoke-pass job -- keeps
+  # Anthropic-credit spend off this path; DirectRoute is exercised elsewhere.
+  API_BACKEND: "bedrock"
+  CLAUDE_CODE_USE_BEDROCK: "1"
+  AWS_BEARER_TOKEN_BEDROCK: ${{ secrets.AWS_BEARER_TOKEN_BEDROCK }}
+  AWS_REGION: ${{ secrets.AWS_REGION }}
+  BEDROCK_MODEL: ${{ secrets.BEDROCK_MODEL }}
+
+jobs:
+  harbor-e2e:
+    name: Harbor export + CoderEvalAgent round trip
+    runs-on: uipath-ubuntu-latest
+    timeout-minutes: 20
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
+
+      - name: Set up Python 3.13
+        uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405  # v6.2.0
+        with:
+          python-version: "3.13"
+
+      - name: Set up Node.js 20
+        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020  # v4.4.0
+        with:
+          node-version: "20"
+
+      - name: Install Claude CLI
+        run: npm install -g @anthropic-ai/claude-code
+
+      - name: Cache dependencies
+        uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae  # v5.0.5
+        with:
+          path: |
+            ~/.cache/uv
+            ~/.cache/pip
+          key: ${{ runner.os }}-py3.13-harbor-e2e-${{ hashFiles('pyproject.toml', 'uv.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-py3.13-harbor-e2e-
+
+      - name: Install uv
+        run: |
+          python -m pip install --upgrade "pip>=26.2"
+          pip install uv
+
+      - name: Install project dependencies (hash-verified from uv.lock)
+        # --extra harbor installs `harbor` into the SAME venv as coder-eval:
+        # CoderEvalAgent is resolved by `hb run -a
+        # coder_eval.harbor.agent:CoderEvalAgent` on the HOST process, so
+        # `harbor` and `coder_eval` must be importable from the same
+        # interpreter (see harbor/agent.py's module docstring). The version is
+        # pinned once, in pyproject.toml's `harbor` extra -- bump it there.
+        run: uv sync --frozen --extra dev --extra harbor
+
+      - name: Put the project venv on PATH
+        run: echo "${{ github.workspace }}/.venv/bin" >> "$GITHUB_PATH"
+
+      - name: Build coder-eval-agent base Docker image
+        run: make docker-image
+
+      - name: Build BYOD template Docker image
+        run: docker build -t byod-custom-image:0.1.0 templates/byod_smoke_test/
+
+      - name: Run Harbor E2E scenarios
+        run: python .github/scripts/harbor_e2e.py

{coder_eval-0.11.7 → coder_eval-0.12.1}/.github/workflows/pr-checks.yml

@@ -80,7 +80,9 @@ jobs:
           pip install uv
 
       - name: Install project dependencies (hash-verified from uv.lock)
-        run: uv sync --frozen --extra dev --extra uipath --extra codex --extra litellm
+        # --extra harbor: pyright below type-checks src/coder_eval/harbor/agent.py
+        # against harbor's real types, not a scoped ignore.
+        run: uv sync --frozen --extra dev --extra uipath --extra codex --extra litellm --extra harbor
 
       # PHASE 1: Fast checks (fail early)
       - name: Check code formatting (ruff format)
@@ -385,7 +387,9 @@ jobs:
           pip install uv
 
       - name: Install project dependencies (hash-verified from uv.lock)
-        run: uv sync --frozen --extra dev --extra uipath --extra codex --extra litellm
+        # --extra harbor: pyright below type-checks src/coder_eval/harbor/agent.py
+        # against harbor's real types, not a scoped ignore.
+        run: uv sync --frozen --extra dev --extra uipath --extra codex --extra litellm --extra harbor
 
       - name: Check code formatting (ruff format)
         run: .venv/Scripts/ruff format --check src/ tests/
@@ -469,16 +473,16 @@ jobs:
       AWS_BEARER_TOKEN_BEDROCK: ${{ secrets.AWS_BEARER_TOKEN_BEDROCK }}
       AWS_REGION: ${{ secrets.AWS_REGION }}
       BEDROCK_MODEL: ${{ secrets.BEDROCK_MODEL }}
-      # tasks_run for --tags smoke-pass. 7 task files (hello_date, dataset_example,
+      # tasks_run for --tags smoke-pass. 8 task files (hello_date, dataset_example,
       # smoke_llm_judge, smoke_agent_judge, byod_smoke_test, agentless_smoke_test,
-      # anti_cheat_reference); dataset_example fans out to 2 inline rows, so 8
-      # sub-tasks. If you add/remove a smoke-pass task or change the dataset row
-      # count, bump these.
+      # anti_cheat_reference, record_cli_responses); dataset_example fans out to 2
+      # inline rows, so 9 sub-tasks. If you add/remove a smoke-pass task or change
+      # the dataset row count, bump these.
       #
       # anti_cheat_reference lives in a SUBDIRECTORY, which `tasks/*.yaml` does not
       # match — the smoke-pass step names its path explicitly. Keep that in sync.
-      EXPECTED_SMOKE_PASS_RUN: "8"
-      EXPECTED_SMOKE_PASS_SUCCEEDED: "8"
+      EXPECTED_SMOKE_PASS_RUN: "9"
+      EXPECTED_SMOKE_PASS_SUCCEEDED: "9"
       # smoke-fail bucket: three tasks expected to fail.
       #   1. smoke_negative_path: file_contains criterion is unsatisfiable
       #      (sentinel-string regression detection for success-checker).
@@ -549,6 +553,9 @@ jobs:
       # explicitly. anti_cheat_reference is the adversarial probe that the agent
       # cannot read the reference solution during its turn; it needs the
       # coder-eval-agent image built above (it is a driver: docker task).
+      # record_cli_responses is the record_cli per-invocation-response probe and
+      # is also driver: docker, so it needs that same image; it is flat in
+      # tasks/, so the glob already matches it.
       - name: Run smoke-pass bucket (expect all to succeed)
         run: |
           .venv/bin/coder-eval run tasks/*.yaml tasks/anti_cheat_reference/*.yaml \

{coder_eval-0.11.7 → coder_eval-0.12.1}/.github/workflows/verify-published-action.yml

@@ -541,6 +541,18 @@ jobs:
                     "uploaded run dir before re-running; this is an unattended paid job.")
               sys.exit(1)
 
+          # NOT_GRADED means a task ran but was never scored. This job invokes the
+          # published action, which runs `coder-eval run` (graded), so reaching it is
+          # impossible unless the action started dispatching `coder-eval execute` --
+          # in which case every score gate below silently measures nothing and the job
+          # goes green having verified no verdict at all. Hard-fail, don't tolerate.
+          ungraded = [s for s in statuses if s == "NOT_GRADED"]
+          if ungraded:
+              print("::error::task(s) reported NOT_GRADED -- the published action ran without "
+                    "grading. `coder-eval run` always grades, so the action is dispatching the "
+                    "wrong command and every score gate in this job is measuring nothing.")
+              sys.exit(1)
+
           # Exit-contract check, conditional on the model having actually performed.
           # Ignoring the step's exit code entirely (see the continue-on-error rationale
           # above) would also hide a REGRESSION in the action's own exit logic -- e.g. a