coder-eval 0.12.0__tar.gz → 0.12.1__tar.gz

{coder_eval-0.12.0 → coder_eval-0.12.1}/.claude/harness-candidates.md

@@ -455,6 +455,31 @@ with the two `action.yml` items above — one considered change to the action's
   `final_status`, which does not exist in `run.json` and would have made a new
   assertion dead on arrival. Guard: assert the key set that non-Python consumers
   depend on, mirroring how CE030 pins doc/schema parity.
+- [ ] **A probe task's detector must not be satisfiable from the sandbox-readable task
+  YAML.** `docker_runner._stage_inputs` serialises the post-override `TaskDefinition` to
+  `/work/input/task.yaml` and mounts `tasks/` again at `/work/task_dir`, both agent-readable.
+  Two probes now depend on NOT being satisfiable from that text — `anti_cheat_reference` via a
+  regex that cannot match its own source, `record_cli_responses` via a log-derived detector —
+  and nothing enforces it. `record_cli_responses` originally shipped (in review) with
+  `file_contains` needles that were verbatim in its own YAML, which would have let a
+  transcribing agent pass while dispatch was dead. Guard: for every `smoke-pass` task, assert no
+  `file_contains` needle / `file_matches_regex` pattern on a must-match criterion appears in the
+  serialised task YAML. Deferred: needs per-criterion-type handling and a real false-positive
+  pass (paths and generic words will collide), so well over 30 min.
+- [ ] **A new shim failure mode must still RECORD the invocation.** A generated shim that dies
+  before `record()` leaves a log byte-identical to "the agent never ran it", which passes a
+  `max_count: 0` guard. The sidecar import was exactly that, caught only in review. Guard:
+  render each shim shape, break each external dependency in turn, assert the log is non-empty.
+  Deferred: "each external dependency" has no enumeration today, so the rule needs a seam
+  (a declared list of what a shim depends on) before it can be mechanical rather than a
+  hand-maintained list that decays.
+- [ ] **A generated-artifact invariant must be asserted against a real run of that artifact,
+  not against the config that produced it.** `TestRecordCliProbeIntegrity` first shipped
+  comparing the task YAML with itself and hardcoding `"rule": 0` — a spelling `json.dumps`'s
+  default separators own — so a separator change would have left it green while the blocking CI
+  probe failed. Now fixed for this case by running a real shim. Deferred as a general guard:
+  "derives its expectation from the thing it checks" is not mechanically detectable; it belongs
+  in the review rubric rather than a lint rule.
 
 - [ ] A `_*TurnState` (agent turn-state) attribute that is written but never read
   outside its own assignment — CE037-class dead accumulator. Surfaced during the
@@ -464,3 +489,65 @@ with the two `action.yml` items above — one considered change to the action's
   run. Guard would need cross-method dataflow over each `Agent`-subclass turn-state
   class (write sites vs read sites), which the AST-only CExxx runner can't express
   in ~30 min — deferred. Caught in: Pi harness Phase 2 quality review.
+
+- [ ] A duration/count aggregate over run-task rows that fails to exclude
+  `mature_skipped`. Nothing guards it: the CExxx runner is Python-AST only and
+  this defect class lives in the evalboard's TypeScript. A codex nightly
+  rendered "1300 tasks · 15h 29m" for the 397 tasks that actually ran. Fixed
+  for the two whole-run sites by extracting `deriveRunDuration`; a general
+  guard needs a TS lint surface the harness does not have.
+  Caught in: timing-capture Phase 1 review.
+
+- [ ] Subtracting a SUM of intervals from a wall span where the intervals can
+  overlap. Semantic, not syntactic — an AST rule cannot tell a sum of
+  durations from a union. Antigravity shipped it: four concurrent 400ms tool
+  calls inside a 1000ms window summed to 1600ms and clamped generation to the
+  0.0 the change existed to remove. The real guard is the replay-based
+  `assert_timing_captured` golden sensor, which now exists; a static rule
+  would not have caught it. Caught in: timing-capture Phase 3 review.
+
+- [ ] A test fixture whose field name does not exist on the type it models.
+  `parseMessages`'s `CommandEntry` keys on `tool_id`; a fixture using
+  `tool_use_id` never resolves, params fall back to `{}`, and every tool
+  weighs exactly 1 — which silently turned a "split by content size" test
+  into a 99%-thinking assertion that passed. TypeScript accepts it because
+  the fixtures are untyped object literals. Typing the fixture factories
+  against the real interfaces would guard the whole class; that is a
+  sweep across the evalboard test suite, not ~30 min.
+  Caught in: timing-capture Phase 5 review.
+
+### Deferred timing divergences (not guardrails — accounting gaps)
+
+Recorded here as well as in `docs/agents/HARNESS_PARITY.md` § Known
+divergences, so the deferred-work record is one place. Measurements in
+`c/time-bugs-audit.md`.
+
+- [ ] **Antigravity books orphan-poll waiting as agent duration** (audit P2-1).
+  A task can spend `0.8 × turn_timeout` waiting on a tool call that never
+  reaches DONE — 14 tasks, 9.6h of one 83h run. Only CLOSED tool intervals are
+  subtracted from a generation window, so that wait stays inside whichever
+  window contains it; the force-close records `execution_completed_at` while
+  leaving `duration_ms` as `None`. Deliberately out of scope of the timing
+  work: closing it means deciding whether a backgrounded tool's elapsed time
+  is model time (the model IS generating while it runs), which is a semantic
+  question, not a bug fix.
+
+- [ ] **Delegate (`delegate-sdk`) records no execution bounds** (audit P3-1).
+  It reports `duration_ms` but neither `execution_started_at` nor
+  `execution_completed_at`, so its tool calls cannot be placed on a timeline.
+  Coverage is ~88%, so it is not urgent. The agent lives in the separate
+  `coder_eval_uipath` repo; mirror the Codex change there
+  (`_item_timing` + threading the SDK stamps through the telemetry builders).
+
+- [ ] **Pre-existing, surfaced by this work's final review: `TokenUsage._adopt_legacy_input_tokens`
+  double-counts the cache buckets.** The validator copies a legacy record's
+  full-prompt `input_tokens` straight into `uncached_input_tokens`, and the
+  computed `input_tokens` then adds `cache_creation` + `cache_read` again. A
+  legacy record with `input_tokens=1000`, `cache_creation=200`, `cache_read=150`
+  reloads as 1350 prompt tokens and bills 1000 at the uncached rate instead of
+  650. Affects every report, budget check and detached regrade over a
+  pre-split run that used prompt caching. NOT touched by the timing work
+  (token accounting was explicitly out of its scope) and not a guardrail
+  candidate — a real bug needing its own change, with a decision about
+  whether legacy records can be distinguished from current ones at all.
+  Caught in: timing-capture final review (gpt-5.6-sol).

coder_eval-0.12.1/.github/scripts/harbor_e2e.py

@@ -0,0 +1,163 @@
+#!/usr/bin/env python3
+"""Harbor end-to-end smoke test.
+
+Exports a handful of coder-eval tasks to Harbor (`coder-eval export --format
+harbor`), runs each with `hb run -a coder_eval.harbor.agent:CoderEvalAgent`
+against real Docker, and asserts that:
+
+  - the verifier phase wrote `reward.json` with `reward == 1.0`
+  - the agent phase wrote a real ATIF `trajectory.json`
+  - both the agent and verifier phases left a `task.json` behind
+
+Invoked by `.github/workflows/harbor-e2e.yml` -- deliberately NOT part of
+`make test`: it shells out to a real `hb` CLI, real Docker builds, and (for
+the llm_judge scenario) a real model call, none of which belong in the fast
+unit-test suite.
+"""
+
+from __future__ import annotations
+
+import json
+import shutil
+import subprocess
+import sys
+from dataclasses import dataclass
+from pathlib import Path
+
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+WORK_DIR = REPO_ROOT / "tmp" / "harbor_e2e"
+AGENT_IMPORT_PATH = "coder_eval.harbor.agent:CoderEvalAgent"
+
+
+@dataclass(frozen=True)
+class Scenario:
+    """One (task.yaml, export flags) pair to round-trip through Harbor."""
+
+    name: str
+    task_file: Path
+    allow_credentials: bool = False
+
+
+SCENARIOS: list[Scenario] = [
+    # "tmpdir" baseline: plain docker driver, default coder-eval-agent image,
+    # no dockerfile_path / template_sources / llm_judge -- exercises the bare
+    # export -> CoderEvalAgent -> --workspace-dir -> verifier round trip.
+    Scenario("baseline", REPO_ROOT / "tests/harbor_e2e/fixtures/docker_baseline.yaml"),
+    # llm_judge: real model call inside the VERIFIER phase, not just the agent.
+    Scenario("llm_judge", REPO_ROOT / "tests/harbor_e2e/fixtures/llm_judge.yaml", allow_credentials=True),
+    # Custom (BYOD) Docker image via dockerfile_path -- reuses the in-tree
+    # byod_smoke_test task/image rather than duplicating it.
+    Scenario("docker_custom_image", REPO_ROOT / "tasks/byod_smoke_test.yaml"),
+    # template_sources: TemplateDirSource copy-in + rewritten path, plus
+    # sandbox.python.env_packages surviving the agent-phase task.yaml merge.
+    Scenario("template_sources", REPO_ROOT / "tests/harbor_e2e/fixtures/template_sources.yaml"),
+]
+
+
+def _run(cmd: list[str]) -> subprocess.CompletedProcess[str]:
+    print(f"+ {' '.join(cmd)}", flush=True)
+    return subprocess.run(cmd, check=False, text=True, capture_output=True)
+
+
+def export_task(scenario: Scenario, out_dir: Path) -> None:
+    if out_dir.exists():
+        shutil.rmtree(out_dir)
+    cmd = ["coder-eval", "export", str(scenario.task_file), "-o", str(out_dir)]
+    if scenario.allow_credentials:
+        cmd.append("--allow-credentials")
+    result = _run(cmd)
+    print(result.stdout)
+    print(result.stderr, file=sys.stderr)
+    if result.returncode != 0:
+        raise RuntimeError(f"[{scenario.name}] `coder-eval export` failed (exit {result.returncode})")
+
+
+def run_harbor(scenario: Scenario, export_dir: Path, jobs_dir: Path) -> Path:
+    if jobs_dir.exists():
+        shutil.rmtree(jobs_dir)
+    cmd = [
+        "hb",
+        "run",
+        "-p",
+        str(export_dir),
+        "-a",
+        AGENT_IMPORT_PATH,
+        "--jobs-dir",
+        str(jobs_dir),
+        "-n",
+        "1",
+        "-y",
+    ]
+    result = _run(cmd)
+    print(result.stdout)
+    print(result.stderr, file=sys.stderr)
+    if result.returncode != 0:
+        raise RuntimeError(f"[{scenario.name}] `hb run` failed (exit {result.returncode})")
+
+    # <jobs_dir>/<job_timestamp>/<trial_name>/{agent,verifier}/... -- glob for
+    # the one directory two levels down that actually holds a verifier/ output,
+    # rather than assuming a fixed trial-name shape Harbor doesn't guarantee.
+    trial_dirs = [d for d in jobs_dir.glob("*/*/") if (d / "verifier").is_dir()]
+    if len(trial_dirs) != 1:
+        raise RuntimeError(
+            f"[{scenario.name}] expected exactly one trial directory under {jobs_dir}, found {len(trial_dirs)}"
+        )
+    return trial_dirs[0]
+
+
+def assert_scenario_artifacts(scenario: Scenario, trial_dir: Path) -> None:
+    reward_path = trial_dir / "verifier" / "reward.json"
+    if not reward_path.is_file():
+        raise RuntimeError(f"[{scenario.name}] missing {reward_path}")
+    reward = json.loads(reward_path.read_text(encoding="utf-8"))
+    if reward.get("reward") != 1.0:
+        raise RuntimeError(f"[{scenario.name}] expected reward 1.0, got {reward!r} ({reward_path})")
+
+    trajectory_path = trial_dir / "agent" / "trajectory.json"
+    if not trajectory_path.is_file():
+        raise RuntimeError(f"[{scenario.name}] missing {trajectory_path}")
+    trajectory = json.loads(trajectory_path.read_text(encoding="utf-8"))
+    if "schema_version" not in trajectory:
+        raise RuntimeError(f"[{scenario.name}] {trajectory_path} is missing 'schema_version'")
+
+    agent_task_jsons = list((trial_dir / "agent").glob("**/task.json"))
+    if not agent_task_jsons:
+        raise RuntimeError(f"[{scenario.name}] no task.json found under {trial_dir / 'agent'}")
+    verifier_task_json = trial_dir / "verifier" / "task.json"
+    if not verifier_task_json.is_file():
+        raise RuntimeError(f"[{scenario.name}] missing {verifier_task_json}")
+
+    print(
+        f"[{scenario.name}] OK: reward=1.0, trajectory.json present, "
+        + f"{len(agent_task_jsons)} agent task.json + verifier/task.json present"
+    )
+
+
+def main() -> int:
+    WORK_DIR.mkdir(parents=True, exist_ok=True)
+    failures: list[str] = []
+    for scenario in SCENARIOS:
+        export_dir = WORK_DIR / scenario.name / "export"
+        jobs_dir = WORK_DIR / scenario.name / "jobs"
+        print(f"\n=== {scenario.name} ===", flush=True)
+        try:
+            export_task(scenario, export_dir)
+            trial_dir = run_harbor(scenario, export_dir, jobs_dir)
+            assert_scenario_artifacts(scenario, trial_dir)
+        except Exception as exc:
+            print(f"[{scenario.name}] FAILED: {exc}", file=sys.stderr)
+            failures.append(scenario.name)
+
+    print("\n=== Summary ===")
+    for scenario in SCENARIOS:
+        print(f"  {scenario.name}: {'FAILED' if scenario.name in failures else 'OK'}")
+
+    if failures:
+        print(f"\n{len(failures)}/{len(SCENARIOS)} scenario(s) failed: {', '.join(failures)}", file=sys.stderr)
+        return 1
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

coder_eval-0.12.1/.github/workflows/harbor-e2e.yml

@@ -0,0 +1,91 @@
+name: Harbor E2E
+
+# Deliberately NOT triggered on pull_request: this exercises real Docker
+# builds, a real `harbor` install, and (for the llm_judge scenario) a real
+# model call, so it is informational rather than a required PR check for now
+# (see .github/scripts/harbor_e2e.py's module docstring). workflow_dispatch
+# lets a maintainer run it on demand; the nightly schedule catches drift
+# between coder-eval's own release and Harbor's own upstream releases without
+# blocking anyone's PR.
+on:
+  workflow_dispatch:
+  schedule:
+    - cron: "17 5 * * *" # nightly, off the hour to avoid GitHub's peak-load pile-up
+  push:
+    branches: [main]
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+env:
+  TELEMETRY_ENABLED: "false"
+  # Route through Bedrock, mirroring pr-checks.yml's smoke-pass job -- keeps
+  # Anthropic-credit spend off this path; DirectRoute is exercised elsewhere.
+  API_BACKEND: "bedrock"
+  CLAUDE_CODE_USE_BEDROCK: "1"
+  AWS_BEARER_TOKEN_BEDROCK: ${{ secrets.AWS_BEARER_TOKEN_BEDROCK }}
+  AWS_REGION: ${{ secrets.AWS_REGION }}
+  BEDROCK_MODEL: ${{ secrets.BEDROCK_MODEL }}
+
+jobs:
+  harbor-e2e:
+    name: Harbor export + CoderEvalAgent round trip
+    runs-on: uipath-ubuntu-latest
+    timeout-minutes: 20
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6.0.2
+
+      - name: Set up Python 3.13
+        uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405  # v6.2.0
+        with:
+          python-version: "3.13"
+
+      - name: Set up Node.js 20
+        uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020  # v4.4.0
+        with:
+          node-version: "20"
+
+      - name: Install Claude CLI
+        run: npm install -g @anthropic-ai/claude-code
+
+      - name: Cache dependencies
+        uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae  # v5.0.5
+        with:
+          path: |
+            ~/.cache/uv
+            ~/.cache/pip
+          key: ${{ runner.os }}-py3.13-harbor-e2e-${{ hashFiles('pyproject.toml', 'uv.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-py3.13-harbor-e2e-
+
+      - name: Install uv
+        run: |
+          python -m pip install --upgrade "pip>=26.2"
+          pip install uv
+
+      - name: Install project dependencies (hash-verified from uv.lock)
+        # --extra harbor installs `harbor` into the SAME venv as coder-eval:
+        # CoderEvalAgent is resolved by `hb run -a
+        # coder_eval.harbor.agent:CoderEvalAgent` on the HOST process, so
+        # `harbor` and `coder_eval` must be importable from the same
+        # interpreter (see harbor/agent.py's module docstring). The version is
+        # pinned once, in pyproject.toml's `harbor` extra -- bump it there.
+        run: uv sync --frozen --extra dev --extra harbor
+
+      - name: Put the project venv on PATH
+        run: echo "${{ github.workspace }}/.venv/bin" >> "$GITHUB_PATH"
+
+      - name: Build coder-eval-agent base Docker image
+        run: make docker-image
+
+      - name: Build BYOD template Docker image
+        run: docker build -t byod-custom-image:0.1.0 templates/byod_smoke_test/
+
+      - name: Run Harbor E2E scenarios
+        run: python .github/scripts/harbor_e2e.py

{coder_eval-0.12.0 → coder_eval-0.12.1}/.github/workflows/pr-checks.yml

@@ -80,7 +80,9 @@ jobs:
           pip install uv
 
       - name: Install project dependencies (hash-verified from uv.lock)
-        run: uv sync --frozen --extra dev --extra uipath --extra codex --extra litellm
+        # --extra harbor: pyright below type-checks src/coder_eval/harbor/agent.py
+        # against harbor's real types, not a scoped ignore.
+        run: uv sync --frozen --extra dev --extra uipath --extra codex --extra litellm --extra harbor
 
       # PHASE 1: Fast checks (fail early)
       - name: Check code formatting (ruff format)
@@ -385,7 +387,9 @@ jobs:
           pip install uv
 
       - name: Install project dependencies (hash-verified from uv.lock)
-        run: uv sync --frozen --extra dev --extra uipath --extra codex --extra litellm
+        # --extra harbor: pyright below type-checks src/coder_eval/harbor/agent.py
+        # against harbor's real types, not a scoped ignore.
+        run: uv sync --frozen --extra dev --extra uipath --extra codex --extra litellm --extra harbor
 
       - name: Check code formatting (ruff format)
         run: .venv/Scripts/ruff format --check src/ tests/
@@ -469,16 +473,16 @@ jobs:
       AWS_BEARER_TOKEN_BEDROCK: ${{ secrets.AWS_BEARER_TOKEN_BEDROCK }}
       AWS_REGION: ${{ secrets.AWS_REGION }}
       BEDROCK_MODEL: ${{ secrets.BEDROCK_MODEL }}
-      # tasks_run for --tags smoke-pass. 7 task files (hello_date, dataset_example,
+      # tasks_run for --tags smoke-pass. 8 task files (hello_date, dataset_example,
       # smoke_llm_judge, smoke_agent_judge, byod_smoke_test, agentless_smoke_test,
-      # anti_cheat_reference); dataset_example fans out to 2 inline rows, so 8
-      # sub-tasks. If you add/remove a smoke-pass task or change the dataset row
-      # count, bump these.
+      # anti_cheat_reference, record_cli_responses); dataset_example fans out to 2
+      # inline rows, so 9 sub-tasks. If you add/remove a smoke-pass task or change
+      # the dataset row count, bump these.
       #
       # anti_cheat_reference lives in a SUBDIRECTORY, which `tasks/*.yaml` does not
       # match — the smoke-pass step names its path explicitly. Keep that in sync.
-      EXPECTED_SMOKE_PASS_RUN: "8"
-      EXPECTED_SMOKE_PASS_SUCCEEDED: "8"
+      EXPECTED_SMOKE_PASS_RUN: "9"
+      EXPECTED_SMOKE_PASS_SUCCEEDED: "9"
       # smoke-fail bucket: three tasks expected to fail.
       #   1. smoke_negative_path: file_contains criterion is unsatisfiable
       #      (sentinel-string regression detection for success-checker).
@@ -549,6 +553,9 @@ jobs:
       # explicitly. anti_cheat_reference is the adversarial probe that the agent
       # cannot read the reference solution during its turn; it needs the
       # coder-eval-agent image built above (it is a driver: docker task).
+      # record_cli_responses is the record_cli per-invocation-response probe and
+      # is also driver: docker, so it needs that same image; it is flat in
+      # tasks/, so the glob already matches it.
       - name: Run smoke-pass bucket (expect all to succeed)
         run: |
           .venv/bin/coder-eval run tasks/*.yaml tasks/anti_cheat_reference/*.yaml \

{coder_eval-0.12.0 → coder_eval-0.12.1}/CHANGELOG.md

@@ -2,6 +2,137 @@
 
 <!-- version list -->
 
+## v0.12.1 (2026-09-12)
+
+### Bug Fixes
+
+- Code review fixes for record-cli-review-fixes
+  ([#150](https://github.com/UiPath/coder_eval/pull/150),
+  [`3aca8e8`](https://github.com/UiPath/coder_eval/commit/3aca8e8010d6beb5147116d07e15118009e81991))
+
+- Resolve py/import-and-import-from CodeQL alerts in test_regrade.py +
+  test_detached_grading_boundaries.py ([#161](https://github.com/UiPath/coder_eval/pull/161),
+  [`6ab6b87`](https://github.com/UiPath/coder_eval/commit/6ab6b87b6112e5b9122bca86dc328b26f5c29188))
+
+- **criteria**: 2/4 — a shim rule fault is an eval-config error, not an agent failure
+  ([#150](https://github.com/UiPath/coder_eval/pull/150),
+  [`3aca8e8`](https://github.com/UiPath/coder_eval/commit/3aca8e8010d6beb5147116d07e15118009e81991))
+
+- **evaluate**: Close the PR review findings on container-based detached grading
+  ([#161](https://github.com/UiPath/coder_eval/pull/161),
+  [`6ab6b87`](https://github.com/UiPath/coder_eval/commit/6ab6b87b6112e5b9122bca86dc328b26f5c29188))
+
+- **evaluate**: Close the review blockers on container-based detached grading
+  ([#161](https://github.com/UiPath/coder_eval/pull/161),
+  [`6ab6b87`](https://github.com/UiPath/coder_eval/commit/6ab6b87b6112e5b9122bca86dc328b26f5c29188))
+
+- **harbor**: Address code review findings from full 8-axis review
+  ([#166](https://github.com/UiPath/coder_eval/pull/166),
+  [`9caffcb`](https://github.com/UiPath/coder_eval/commit/9caffcbbae1feeb498e40c83bf679289caab7148))
+
+- **harbor**: Address PR review blockers (score-integrity, security, drops)
+  ([#166](https://github.com/UiPath/coder_eval/pull/166),
+  [`9caffcb`](https://github.com/UiPath/coder_eval/commit/9caffcbbae1feeb498e40c83bf679289caab7148))
+
+- **harbor**: Derive a prebuilt image's real WORKDIR instead of guessing /app
+  ([#166](https://github.com/UiPath/coder_eval/pull/166),
+  [`9caffcb`](https://github.com/UiPath/coder_eval/commit/9caffcbbae1feeb498e40c83bf679289caab7148))
+
+- **harbor**: Suppress pyright reportMissingImports for the harbor package
+  ([#166](https://github.com/UiPath/coder_eval/pull/166),
+  [`9caffcb`](https://github.com/UiPath/coder_eval/commit/9caffcbbae1feeb498e40c83bf679289caab7148))
+
+- **models**: Reference FlagPredicate at runtime so the cast is a real use
+  ([#150](https://github.com/UiPath/coder_eval/pull/150),
+  [`3aca8e8`](https://github.com/UiPath/coder_eval/commit/3aca8e8010d6beb5147116d07e15118009e81991))
+
+- **record_cli**: Close three real holes the Copilot review found
+  ([#150](https://github.com/UiPath/coder_eval/pull/150),
+  [`3aca8e8`](https://github.com/UiPath/coder_eval/commit/3aca8e8010d6beb5147116d07e15118009e81991))
+
+- **record_cli**: Reject an unusable response rule at load, and trace a shim fault
+  ([#150](https://github.com/UiPath/coder_eval/pull/150),
+  [`3aca8e8`](https://github.com/UiPath/coder_eval/commit/3aca8e8010d6beb5147116d07e15118009e81991))
+
+- **sandbox**: Give the sandbox venv system site packages
+  ([#162](https://github.com/UiPath/coder_eval/pull/162),
+  [`e7d8326`](https://github.com/UiPath/coder_eval/commit/e7d8326ddade4d63305f94238897b39db8af4317))
+
+- **test**: Strip ANSI color before asserting --workspace-dir in CLI output
+  ([#166](https://github.com/UiPath/coder_eval/pull/166),
+  [`9caffcb`](https://github.com/UiPath/coder_eval/commit/9caffcbbae1feeb498e40c83bf679289caab7148))
+
+- **tests**: Harden harbor CLI-error assertions and Windows chmod checks
+  ([#166](https://github.com/UiPath/coder_eval/pull/166),
+  [`9caffcb`](https://github.com/UiPath/coder_eval/commit/9caffcbbae1feeb498e40c83bf679289caab7148))
+
+- **timing**: Account for generation and tool time on every harness
+  ([#164](https://github.com/UiPath/coder_eval/pull/164),
+  [`87102a3`](https://github.com/UiPath/coder_eval/commit/87102a357546672ae58c9c8474e609a4be3cb681))
+
+### Build System
+
+- **docker**: Bake the harbor extra into the coder-eval-agent image
+  ([#166](https://github.com/UiPath/coder_eval/pull/166),
+  [`9caffcb`](https://github.com/UiPath/coder_eval/commit/9caffcbbae1feeb498e40c83bf679289caab7148))
+
+### Continuous Integration
+
+- **harbor**: Add Harbor E2E workflow (export + CoderEvalAgent round trip)
+  ([#166](https://github.com/UiPath/coder_eval/pull/166),
+  [`9caffcb`](https://github.com/UiPath/coder_eval/commit/9caffcbbae1feeb498e40c83bf679289caab7148))
+
+### Features
+
+- **evaluate**: Grade a `driver: docker` row inside a container of its own image
+  ([#161](https://github.com/UiPath/coder_eval/pull/161),
+  [`6ab6b87`](https://github.com/UiPath/coder_eval/commit/6ab6b87b6112e5b9122bca86dc328b26f5c29188))
+
+- **harbor**: Coder-eval as a Harbor agent (C1.2)
+  ([#166](https://github.com/UiPath/coder_eval/pull/166),
+  [`9caffcb`](https://github.com/UiPath/coder_eval/commit/9caffcbbae1feeb498e40c83bf679289caab7148))
+
+- **harbor**: Export coder-eval tasks to Harbor, with coder-eval as the grader
+  ([#166](https://github.com/UiPath/coder_eval/pull/166),
+  [`9caffcb`](https://github.com/UiPath/coder_eval/commit/9caffcbbae1feeb498e40c83bf679289caab7148))
+
+- **harbor**: Export experiment.yaml variants to Harbor task directories
+  ([#166](https://github.com/UiPath/coder_eval/pull/166),
+  [`9caffcb`](https://github.com/UiPath/coder_eval/commit/9caffcbbae1feeb498e40c83bf679289caab7148))
+
+- **harbor**: Fix agent workspace alignment, env passthrough, and template sources
+  ([#166](https://github.com/UiPath/coder_eval/pull/166),
+  [`9caffcb`](https://github.com/UiPath/coder_eval/commit/9caffcbbae1feeb498e40c83bf679289caab7148))
+
+- **harbor**: Task.yaml/experiment.yaml → Harbor export + coder-eval as a Harbor agent
+  ([#166](https://github.com/UiPath/coder_eval/pull/166),
+  [`9caffcb`](https://github.com/UiPath/coder_eval/commit/9caffcbbae1feeb498e40c83bf679289caab7148))
+
+- **record_cli**: Serve a different canned response per invocation
+  ([#150](https://github.com/UiPath/coder_eval/pull/150),
+  [`3aca8e8`](https://github.com/UiPath/coder_eval/commit/3aca8e8010d6beb5147116d07e15118009e81991))
+
+### Refactoring
+
+- **record_cli**: 1/4 — copy argv_match beside the shim instead of splicing it
+  ([#150](https://github.com/UiPath/coder_eval/pull/150),
+  [`3aca8e8`](https://github.com/UiPath/coder_eval/commit/3aca8e8010d6beb5147116d07e15118009e81991))
+
+### Testing
+
+- Close three harness gaps this review surfaced
+  ([#150](https://github.com/UiPath/coder_eval/pull/150),
+  [`3aca8e8`](https://github.com/UiPath/coder_eval/commit/3aca8e8010d6beb5147116d07e15118009e81991))
+
+- **lint**: 4/4 — put the record_cli authoring surface under CE030 doc parity
+  ([#150](https://github.com/UiPath/coder_eval/pull/150),
+  [`3aca8e8`](https://github.com/UiPath/coder_eval/commit/3aca8e8010d6beb5147116d07e15118009e81991))
+
+- **tasks**: 3/4 — add the record_cli per-invocation-response probe
+  ([#150](https://github.com/UiPath/coder_eval/pull/150),
+  [`3aca8e8`](https://github.com/UiPath/coder_eval/commit/3aca8e8010d6beb5147116d07e15118009e81991))
+
+
 ## v0.12.0 (2026-09-09)
 
 ### Bug Fixes