coder-eval 0.12.3__tar.gz → 0.12.4__tar.gz

{coder_eval-0.12.3 → coder_eval-0.12.4}/.claude/notes/orchestration.md

@@ -759,6 +759,25 @@ It RAISES rather than guessing when neither is conclusive. Guessing is worse tha
 grading the wrong directory makes every path-relative criterion fail as a locating
 artifact rather than as a verdict, and reports that as an ordinary score.
 
+A resolved `artifacts_dir_template` is passed in as a SECOND TRUSTED ROOT and preferred as
+the candidate, since that template may place artifacts outside `run_dir` entirely. Note the
+direction: the roots were WIDENED, never the check relaxed — roots stay operator-supplied,
+candidates stay untrusted. Without it, `--resume` over a decoupled layout failed the
+containment check, fell through to a `run_dir/artifacts` that did not exist, and raised
+`RegradeError`. An artifacts dir that exists outranks the recorded `sandbox_path`, which is
+merely what the run claimed.
+
+Only `--resume` (`run_command.py`'s `_grade_resumed_tasks`) passes the resolved
+`artifacts_dir` — it has the run's own `BatchRunConfig` in hand, so `resolve_artifacts_dir`
+needs no new state. Detached `coder-eval evaluate <run_dir>` does not: it has no
+`BatchRunConfig` to resolve a template from, only the untrusted `task.json` it's grading, and
+trusting THAT to widen its own containment root would defeat the check the widening exists
+to keep intact. A run made with a non-default `artifacts_dir_template` therefore still needs
+`--workspace` passed explicitly to `evaluate` — the same escape hatch `RegradeError` already
+names. Not a regression: `evaluate` never auto-located such a workspace before this template
+existed either, and the alternative (trusting the recorded path) reopens the class of bug the
+previous paragraph describes.
+
 **Every return goes through the containment check, rooted at the RUN DIRECTORY.** Both
 `sandbox_path` and `task_id` are unvalidated strings out of the run's own `task.json`, so
 `"../../../../home/victim"` joins to a real directory `is_dir()` happily confirms. The

{coder_eval-0.12.3 → coder_eval-0.12.4}/.claude/notes/persistence.md

@@ -69,6 +69,94 @@ trajectory log the run had already paid for. `grade.docker.log` exists for the s
 one layer down: on the `run --resume` path `docker.log` is already the executed
 container's log.
 
+## The run layout as two directory templates
+
+`logging_dir_template` and `artifacts_dir_template` (`BatchRunConfig`, resolved by
+`path_utils.resolve_dir_template`) describe where a task's bookkeeping and its artifacts
+go. They are resolved LATE — per task, which is the only point where `${variant}`,
+`${task}` and `${repeat}` exist at all — and they are INDEPENDENT: artifacts nesting under
+the logging dir is a default, not a law.
+
+That independence is the point. Harbor puts logs in its own agent logs dir and artifacts at
+the container's WORKDIR, two unrelated parts of the filesystem, and because artifacts were
+never a child of the logging dir there is nothing to copy and nothing lands twice. The
+previous design reached the same end state with a `--workspace-dir`/`--artifacts-dir` pair
+that had to be passed the SAME path so an equality check could infer "don't copy"; a
+coincidence standing in for an intention.
+
+An override needs no special-casing anywhere, because substituting a template that contains
+no placeholders is the identity function: `/work/output` in, `/work/output` out, down the
+same code path as the default. The defaults spell out the historical layout, so an
+unspecified run writes byte-identical paths — asserted in `test_path_utils.py` against the
+literal old layout rather than against `build_task_run_dir`, which now calls the resolver
+and would make the test vacuous.
+
+Two implementation constraints, both load-bearing:
+
+- **`string.Template`, never `re.sub`.** `re.sub` interprets backslashes in the
+  REPLACEMENT, so a Windows `run_dir` of `C:\runs\2026` comes out mangled.
+- **`${task}` may contain a separator.** A dataset-expanded `task_id` is
+  `"<task>/<row>"` (`expand_dataset`, whose row ids are validated precisely because they
+  become directories), so it nests — on Windows too, where `pathlib` splits on both
+  separators.
+
+`${task}` appears twice in the artifacts default (`.../${task}/${repeat}/artifacts/${task}`)
+because that IS the layout on disk: the per-task run dir carries it, and
+`preserve_to`/`capture_to`/DIRECT_WRITE each appended it again. Kept for compatibility, and
+now one string to change rather than five call sites. The `*_as` variants
+(`preserve_as`/`capture_as`) exist so a caller-supplied artifacts dir is used as the FINAL
+path instead of having `task_id` appended to it a second time; `preserve_to`/`capture_to`
+remain as the parent-relative wrappers.
+
+### What run_dir still owns
+
+`${run_dir}` is only a placeholder VALUE. But run-LEVEL files — `run.json`, `run.md`,
+`experiment.*`, `resume_fingerprint.json` — still follow `--run-dir`, and its default is
+CWD-RELATIVE (`runs/<timestamp>`, `config.py`). Omitting `--run-dir` therefore does not
+leave those files harmlessly uncollected; when cwd is the agent's WORKDIR it writes them
+INSIDE the workspace, polluting the very directory `artifacts_dir_template` names.
+Confirmed live: `artifacts/work/runs/<timestamp>/run.json` in a collected Harbor trial.
+Harbor consequently passes `--run-dir /tmp/coder-eval-run`, a throwaway path outside the
+workspace.
+
+Anything that used to DISCOVER per-task files by walking `run_dir` had to be given the
+resolved logging dirs instead, because they need not live under `run_dir` at all and the
+walk silently finds nothing: `atif_emit.emit_trajectories_for_run` (which would emit no
+trajectory, leaving Harbor's token/cost totals empty) and
+`logging_config.aggregate_task_logs` (which would write an empty `experiment.log`).
+
+### A static template is single-task only
+
+A placeholder-free template is the identity function, so every task in a multi-task
+`run`/`execute` would resolve `--logging-dir`/`--artifacts-dir` to the SAME directory and
+overwrite each other's `task.json`/artifacts. `run_batch` refuses this loud
+(`ValueError`) when `len(resolved_tasks) > 1` and either template has no `${...}`
+placeholders (`path_utils.dir_template_is_static`), mirroring the pre-existing
+`--workspace-dir` + multi-task rejection. Static paths exist for exactly the single-task
+case below.
+
+### A flat run_dir needs no special case in run_batch
+
+Harbor's single-task, static-template mode writes `task.json`/`task.html`/`task.log`/
+artifacts flat at the top-level `run_dir` instead of the usual
+`<variant>/<task_id>/<NN>` nesting -- the multi-task guard above already guarantees
+exactly one resolved task here, so that nesting only exists to disambiguate siblings that
+can't occur. A flat `run_dir` also means `trajectory.json` (`emit_trajectories_for_run`'s
+sibling write) lands at a fixed, predictable path instead of requiring a recursive glob.
+`run_batch`'s `run_single` needs no `workspace_dir` special case for this: `rt.run_dir` IS
+the resolved `logging_dir_template`, so "flat" is simply what a static template resolves
+to, not a mode this seam has to detect.
+
+### CoderEvalAgent passes both templates as static paths
+
+`harbor/agent.py`'s `CoderEvalAgent.run()` passes `--logging-dir`/`--artifacts-dir` as two
+STATIC paths (Harbor's own agent logs dir, and the container's WORKDIR via `$(pwd)`) rather
+than templates with placeholders — the case the identity-function property above exists for.
+Because artifacts are no longer a child of the logging dir, nothing lands twice; because the
+artifacts destination IS the workspace, `capture_as`'s self-referential guard makes the copy
+a no-op. `--run-dir` still points at `_THROWAWAY_RUN_DIR` (`/tmp/coder-eval-run`), never
+substituted into either template, for the reason in "What run_dir still owns" above.
+
 ## Judge persistence
 
 A judge transcript — tool calls, raw verdict, rendered prompt, system prompt — runs 10-100

{coder_eval-0.12.3 → coder_eval-0.12.4}/.claude/notes/reporting.md

@@ -453,6 +453,25 @@ container, never where the verifier looks. Confirmed live — the agent's output
 every criterion scored 0 as "file does not exist". `$(pwd)` is resolved by the container's
 shell at exec time and equals the WORKDIR because the exec is given no explicit cwd.
 
+### The artifacts default is CONTAINER_WORK_DIR, not a required field
+
+Harbor's own artifact collection runs after the agent phase but before the verifier and
+container teardown, snapshotting `task.toml`'s `artifacts` source to
+`<trial>/artifacts/<source stripped of its leading slash>/` on the host.
+`CoderEvalAgent`'s `--workspace-dir "$(pwd)"` runs the agent in-place at the container's
+WORKDIR and skips coder-eval's own copy-out, so without an `artifacts` entry nothing the
+agent produced would ever be visible on the host.
+
+Defaults to `CONTAINER_WORK_DIR` (`/work`, coder-eval's own image WORKDIR) rather than
+requiring every task/experiment to restate it — a package exported by coder-eval needs
+coder-eval installed in the image, which in practice means derived from
+`coder-eval-agent` (a mismatch already warns; see `_MISSING_CODER_EVAL_WARNING`). Unlike
+`[environment].workdir` above — deliberately left unset so the container's own WORKDIR
+decides `docker exec -w` — guessing wrong here is not fatal: a nonexistent source is a
+best-effort collection miss recorded in the artifact manifest, not an exit 127. Declared
+as a plain string, not an `ArtifactConfig` table, because Harbor normalizes
+`artifacts = ["/x"]` to `ArtifactConfig(source="/x")` itself.
+
 ### What the export carries, and what it refuses to carry
 
 No Dockerfile is written unless the task sets `sandbox.docker.dockerfile_path` — only

{coder_eval-0.12.3 → coder_eval-0.12.4}/CHANGELOG.md

@@ -2,6 +2,29 @@
 
 <!-- version list -->
 
+## v0.12.4 (2026-09-18)
+
+### Bug Fixes
+
+- **ci**: Silence bandit B108 on the throwaway Harbor run-dir path
+  ([#189](https://github.com/UiPath/coder_eval/pull/189),
+  [`db7f27c`](https://github.com/UiPath/coder_eval/commit/db7f27c7137fdb173c71948af8ec0389becd84e9))
+
+- **orchestration**: Close blockers from the dir-template review round 2
+  ([#189](https://github.com/UiPath/coder_eval/pull/189),
+  [`db7f27c`](https://github.com/UiPath/coder_eval/commit/db7f27c7137fdb173c71948af8ec0389becd84e9))
+
+- **orchestration**: Close code-review gaps in the dir-template split
+  ([#189](https://github.com/UiPath/coder_eval/pull/189),
+  [`db7f27c`](https://github.com/UiPath/coder_eval/commit/db7f27c7137fdb173c71948af8ec0389becd84e9))
+
+### Features
+
+- **orchestration**: Describe the run layout as two independent directory templates
+  ([#189](https://github.com/UiPath/coder_eval/pull/189),
+  [`db7f27c`](https://github.com/UiPath/coder_eval/commit/db7f27c7137fdb173c71948af8ec0389becd84e9))
+
+
 ## v0.12.3 (2026-09-17)
 
 ### Bug Fixes

{coder_eval-0.12.3 → coder_eval-0.12.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: coder-eval
-Version: 0.12.3
+Version: 0.12.4
 Summary: Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode, Pi) with sandboxed, reproducible YAML task suites.
 Project-URL: Homepage, https://coder-eval.com
 Project-URL: Repository, https://github.com/UiPath/coder_eval

{coder_eval-0.12.3 → coder_eval-0.12.4}/action.yml

@@ -25,7 +25,7 @@ inputs:
   version:
     description: coder-eval version to install from PyPI, or "local" to install from the action checkout
     required: false
-    default: "0.12.3"   # <-- kept in sync with releases by release.yml
+    default: "0.12.4"   # <-- kept in sync with releases by release.yml
   extras:
     description: >-
       Comma-separated coder-eval extras (`codex`, `antigravity,litellm`), composed

{coder_eval-0.12.3 → coder_eval-0.12.4}/plugins/coder-eval/.claude-plugin/plugin.json

@@ -2,7 +2,7 @@
   "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
   "name": "coder-eval",
   "displayName": "Coder Eval",
-  "version": "0.12.3",
+  "version": "0.12.4",
   "description": "Test whether your Claude Code skills actually trigger, and benchmark any coding agent — author, run, and analyze the eval suite that proves it, locally or as a CI gate.",
   "author": { "name": "UiPath", "email": "coder-eval@uipath.com", "url": "https://github.com/UiPath/coder_eval" },
   "homepage": "https://coder-eval.com",

{coder_eval-0.12.3 → coder_eval-0.12.4}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "coder-eval"
-version = "0.12.3"
+version = "0.12.4"
 description = "Evaluate, benchmark, and A/B-test AI coding agents (Claude Code, Codex, Gemini/Antigravity, OpenCode, Pi) with sandboxed, reproducible YAML task suites."
 readme = "README.md"
 license = "Apache-2.0"

{coder_eval-0.12.3 → coder_eval-0.12.4}/src/coder_eval/__init__.py

@@ -1,3 +1,3 @@
 """coder_eval - A framework for evaluating AI coding agents."""
 
-__version__ = "0.12.3"
+__version__ = "0.12.4"

{coder_eval-0.12.3 → coder_eval-0.12.4}/src/coder_eval/cli/execute_command.py

@@ -189,13 +189,38 @@ def execute_command(
         "--workspace-dir",
         help=(
             "Run the single resolved task's agent in-place at this absolute path instead of the "
-            "standard run_dir/artifacts workspace (copied out to run_dir/artifacts/<task> at "
+            "standard artifacts workspace named by --artifacts-dir (copied out there at "
             "cleanup). Requires exactly one resolved task; refused for sandbox.driver: docker "
             "(the docker driver already aligns automatically via sandbox.docker.working_dir). "
             "Meant for a Harbor `CoderEvalAgent` invocation, so the agent's writes land at the "
             "container's own WORKDIR, where Harbor's verifier phase looks for them."
         ),
     ),
+    logging_dir: str | None = typer.Option(
+        None,
+        "--logging-dir",
+        help=(
+            "Where task.json/task.log go, as a path template. Placeholders: ${run_dir}, "
+            "${variant}, ${task}, ${repeat}. Default reproduces <run_dir>/<variant>/<task>/<NN>. "
+            "A static path (e.g. /logs/agent) resolves every task to itself, so it is only for a "
+            "single-task run (e.g. Harbor); refused for sandbox.driver: docker and for more than "
+            "one resolved task."
+        ),
+    ),
+    artifacts_dir: str | None = typer.Option(
+        None,
+        "--artifacts-dir",
+        help=(
+            "Where the agent's artifacts go -- the FINAL directory, same placeholders as "
+            "--logging-dir, and independent of it (Harbor puts logs at /logs/agent and "
+            "artifacts at the container's WORKDIR). When it already holds the workspace "
+            "there is nothing to copy. Default reproduces <run_dir>/<variant>/<task>/<NN>/"
+            "artifacts/<task>. A static path is only for a single-task run; refused for "
+            "sandbox.driver: docker (the in-container Orchestrator has no way to receive it) "
+            "and for more than one resolved task, and refused together with --resume (it would "
+            "clear an operator-supplied tree the harness did not create)."
+        ),
+    ),
 ) -> None:
     """Run evaluation tasks WITHOUT checking their success criteria.
 
@@ -250,4 +275,6 @@ def execute_command(
         set_overrides=set_overrides,
         format=format,
         workspace_dir=workspace_dir,
+        logging_dir=logging_dir,
+        artifacts_dir=artifacts_dir,
     )

{coder_eval-0.12.3 → coder_eval-0.12.4}/src/coder_eval/cli/run_command.py

@@ -3,7 +3,6 @@
 import asyncio
 import logging
 import os
-import shutil
 import sys
 import urllib.error
 import urllib.parse
@@ -365,13 +364,38 @@ def run_command(
         "--workspace-dir",
         help=(
             "Run the single resolved task's agent in-place at this absolute path instead of the "
-            "standard run_dir/artifacts workspace (copied out to run_dir/artifacts/<task> at "
+            "standard artifacts workspace named by --artifacts-dir (copied out there at "
             "cleanup). Requires exactly one resolved task; refused for sandbox.driver: docker "
             "(the docker driver already aligns automatically via sandbox.docker.working_dir). "
             "Meant for a Harbor `CoderEvalAgent` invocation, so the agent's writes land at the "
             "container's own WORKDIR, where Harbor's verifier phase looks for them."
         ),
     ),
+    logging_dir: str | None = typer.Option(
+        None,
+        "--logging-dir",
+        help=(
+            "Where task.json/task.log go, as a path template. Placeholders: ${run_dir}, "
+            "${variant}, ${task}, ${repeat}. Default reproduces <run_dir>/<variant>/<task>/<NN>. "
+            "A static path (e.g. /logs/agent) resolves every task to itself, so it is only for a "
+            "single-task run (e.g. Harbor); refused for sandbox.driver: docker and for more than "
+            "one resolved task."
+        ),
+    ),
+    artifacts_dir: str | None = typer.Option(
+        None,
+        "--artifacts-dir",
+        help=(
+            "Where the agent's artifacts go -- the FINAL directory, same placeholders as "
+            "--logging-dir, and independent of it (Harbor puts logs at /logs/agent and "
+            "artifacts at the container's WORKDIR). When it already holds the workspace "
+            "there is nothing to copy. Default reproduces <run_dir>/<variant>/<task>/<NN>/"
+            "artifacts/<task>. A static path is only for a single-task run; refused for "
+            "sandbox.driver: docker (the in-container Orchestrator has no way to receive it) "
+            "and for more than one resolved task, and refused together with --resume (it would "
+            "clear an operator-supplied tree the harness did not create)."
+        ),
+    ),
 ) -> None:
     """Run evaluation tasks (optionally in parallel).
 
@@ -424,9 +448,25 @@ def run_command(
         set_overrides=set_overrides,
         format=format,
         workspace_dir=workspace_dir,
+        logging_dir=logging_dir,
+        artifacts_dir=artifacts_dir,
     )
 
 
+def _dir_template_overrides(logging_dir: str | None, artifacts_dir: str | None) -> dict[str, str]:
+    """Map the two CLI flags onto BatchRunConfig's template fields.
+
+    ``None`` is omitted rather than forwarded, so an unpassed flag leaves the
+    model default (today's layout) authoritative in exactly one place.
+    """
+    overrides: dict[str, str] = {}
+    if logging_dir is not None:
+        overrides["logging_dir_template"] = logging_dir
+    if artifacts_dir is not None:
+        overrides["artifacts_dir_template"] = artifacts_dir
+    return overrides
+
+
 def run_pipeline(
     *,
     grade: bool,
@@ -454,6 +494,8 @@ def run_pipeline(
     set_overrides: list[str],
     format: str | None = None,
     workspace_dir: Path | None = None,
+    logging_dir: str | None = None,
+    artifacts_dir: str | None = None,
 ) -> None:
     """The shared body of ``coder-eval run`` and ``coder-eval execute``.
 
@@ -474,6 +516,16 @@ def run_pipeline(
     # resumed workspace-dir run would never recognize its own prior result.
     if resume and workspace_dir is not None:
         raise typer.BadParameter("--resume is not supported together with --workspace-dir.")
+    # clear_rerun_artifacts rmtree's whatever artifacts_dir_template resolves to for a
+    # re-running task. The default template is always a directory the harness itself
+    # created (run_dir/.../artifacts/<task>), which is what makes that safe -- an
+    # operator-supplied --artifacts-dir points at a pre-existing tree the harness did
+    # not create and has no way to prove it owns.
+    if resume and artifacts_dir is not None:
+        raise typer.BadParameter(
+            "--resume is not supported together with --artifacts-dir -- clearing a re-running "
+            + "task's stale artifacts would rmtree an operator-supplied tree the harness didn't create."
+        )
     # Without --resume this flag parsed, was accepted, and did nothing at all. Its
     # sibling mode-scoped flag (`evaluate --workspace`) hard-errors on exactly this.
     if allow_host_grading and not resume:
@@ -539,6 +591,8 @@ def run_pipeline(
                 grade=grade,
                 format=format,
                 workspace_dir=workspace_dir,
+                logging_dir=logging_dir,
+                artifacts_dir=artifacts_dir,
             )
         )
     except KeyboardInterrupt:
@@ -568,6 +622,8 @@ async def _run_all_tasks(
     grade: bool = True,
     format: str | None = None,
     workspace_dir: Path | None = None,
+    logging_dir: str | None = None,
+    artifacts_dir: str | None = None,
 ) -> None:
     """Async entry point for running all tasks (optionally in parallel).
 
@@ -590,6 +646,7 @@ async def _run_all_tasks(
             promotes it to `<run_dir>/trajectory.json` when the run wrote exactly one
             (a multi-task run is left nested; there is nothing to promote)
         workspace_dir: Run the agent here instead of run_dir/artifacts
+        logging_dir / artifacts_dir: path templates for the run layout (see run_command options)
     """
     # Prepare run directory
     run_dir = prepare_run_directory(run_dir)
@@ -617,6 +674,7 @@ async def _run_all_tasks(
         include_skipped=include_skipped,
         grade=grade,
         workspace_dir=workspace_dir,
+        **_dir_template_overrides(logging_dir, artifacts_dir),
     )
 
     from ..telemetry import flush_telemetry, track_event
@@ -657,20 +715,32 @@ async def _run_all_tasks(
         )
 
         # Aggregate task logs into run.log
+
+        # Over the resolved logging dirs, not a run_dir walk: an overridden
+        # logging_dir_template need not live under run_dir, and the walk would
+        # silently aggregate nothing (empty experiment.log, no error).
+        # summary.task_results rows are plain dicts (RunSummary persists them
+        # that way), carrying the same variant/task/replicate the logging dir was
+        # built from.
+        task_dirs = [
+            config.resolve_logging_dir(
+                row.get("variant_id") or "default",
+                row["task_id"],
+                row.get("replicate_index") or 0,
+            )
+            for row in summary.task_results
+            if row.get("task_id")
+        ]
+
         from ..logging_config import aggregate_task_logs
 
-        aggregate_task_logs(run_dir)
+        aggregate_task_logs(run_dir, task_dirs=task_dirs)
 
         if format == "harbor":
             from ..harbor.atif_emit import emit_trajectories_for_run
 
-            written = emit_trajectories_for_run(run_dir)
-            console.print(f"[dim]Wrote {len(written)} trajectory.json (ATIF) file(s) under {run_dir}[/dim]")
-
-            flat_trajectory_path = run_dir / "trajectory.json"
-            if len(written) == 1 and written[0] != flat_trajectory_path:
-                await asyncio.to_thread(shutil.copy2, written[0], flat_trajectory_path)
-                console.print(f"[dim]Copied the single trajectory to {flat_trajectory_path}[/dim]")
+            written = emit_trajectories_for_run(task_dirs)
+            console.print(f"[dim]Wrote {len(written)} trajectory.json (ATIF) file(s)[/dim]")
 
         # Print execution summary
         print_execution_summary(run_dir, summary)
@@ -764,7 +834,7 @@ def _unreadable_row_placeholder(rt: ResolvedTask, error: Exception) -> Evaluatio
 
 
 async def _grade_resumed_tasks(
-    to_grade: list[ResolvedTask], *, allow_host_grading: bool = False
+    to_grade: list[ResolvedTask], *, config: BatchRunConfig, allow_host_grading: bool = False
 ) -> list[tuple[ResolvedTask, TaskResult]]:
     """Grade the rows ``coder-eval execute`` left NOT_GRADED, in place.
 
@@ -799,7 +869,15 @@ async def _grade_resumed_tasks(
             prior = load_prior_result(rt.run_dir)
             # The reference check lives inside regrade_in_place, so a caller
             # cannot forget it.
-            workspace = default_workspace(rt.run_dir, prior)
+            # The same template the run wrote with, so a workspace the template
+            # placed OUTSIDE run_dir is a trusted root rather than a containment
+            # failure -- which is what previously made --resume unusable alongside
+            # an overridden artifacts dir.
+            workspace = default_workspace(
+                rt.run_dir,
+                prior,
+                artifacts_dir=config.resolve_artifacts_dir(rt.variant_id, rt.task.task_id, rt.replicate_index),
+            )
             # Preserve the ungraded record BEFORE the orchestrator overwrites
             # task.json in this same directory.
             back_up_pre_grade_record(rt.run_dir)
@@ -860,7 +938,7 @@ async def _grade_resumed_tasks(
 
 
 async def _apply_resume(
-    resolved: list[ResolvedTask], *, grade: bool, allow_host_grading: bool
+    resolved: list[ResolvedTask], *, config: BatchRunConfig, grade: bool, allow_host_grading: bool
 ) -> tuple[list[ResolvedTask], list[TaskResult], list[ResolvedTask]]:
     """Split a resumed run into what still needs running, and what is carried in.
 
@@ -879,7 +957,7 @@ async def _apply_resume(
     # Leftover artifacts from a partial run could let a file-based criterion pass on
     # the old output. to_grade is deliberately NOT cleared: its artifacts are what is
     # being graded.
-    cleared = clear_rerun_artifacts(part.to_run)
+    cleared = clear_rerun_artifacts(part.to_run, config=config)
     # Checked explicitly rather than left to regrade_in_place's own per-row guard,
     # so the whole batch is refused up front instead of one row at a time.
     # Rationale: .claude/notes/orchestration.md § Refusing a criteria-free task under grade
@@ -892,7 +970,7 @@ async def _apply_resume(
     )
     # Reusing the trajectory and workspace already on disk rather than paying for
     # the agent twice. Folded in as prior_results so the summary covers them.
-    for rt, tr in await _grade_resumed_tasks(part.to_grade, allow_host_grading=allow_host_grading):
+    for rt, tr in await _grade_resumed_tasks(part.to_grade, config=config, allow_host_grading=allow_host_grading):
         prior_results.append(tr)
         prior_resolved.append(rt)
     return part.to_run, prior_results, prior_resolved
@@ -1051,7 +1129,7 @@ async def _run_with_experiment(
     prior_resolved: list[ResolvedTask] = []
     if resume:
         to_run, prior_results, prior_resolved = await _apply_resume(
-            resolved, grade=grade, allow_host_grading=allow_host_grading
+            resolved, config=config, grade=grade, allow_host_grading=allow_host_grading
         )
 
     # Against `to_run`, not `resolved`: an already-finalized row is never re-graded,

{coder_eval-0.12.3 → coder_eval-0.12.4}/src/coder_eval/harbor/agent.py

@@ -45,6 +45,13 @@ except ImportError:  # pragma: no cover - defensive; coder_eval always defines t
     __version__ = "0.0.0"
 
 
+# Run-level bookkeeping goes here and is discarded with the container. Anywhere
+# outside the WORKDIR would do; /tmp is the one path guaranteed writable in every
+# task image. NOT a bare "/tmp": a dedicated subdirectory keeps run.json/run.md/
+# experiment.* from littering a directory tasks themselves use.
+_THROWAWAY_RUN_DIR = "/tmp/coder-eval-run"  # nosec B108 -- predictable path is fine: single-use, single-tenant container, no other process/user shares /tmp to race or symlink-plant it
+
+
 class CoderEvalAgent(BaseInstalledAgent):
     """Runs coder-eval's own agent loop as a Harbor agent.
 
@@ -76,20 +83,22 @@ class CoderEvalAgent(BaseInstalledAgent):
     async def run(self, instruction: str, environment: BaseEnvironment, context: AgentContext) -> None:
         """Run ``coder-eval execute --format harbor`` inside the environment.
 
-        ``instruction`` is NOT forwarded: the agent-phase task.yaml already carries
-        the identical resolved prompt. Token and cost totals are filled in afterward
-        by ``populate_context_post_run``.
-
-        ``--workspace-dir "$(pwd)"`` is load-bearing — without it the tempdir sandbox
-        writes the agent's workspace somewhere Harbor's verifier never looks.
+        ``instruction`` is NOT forwarded: the agent-phase task.yaml already carries the
+        identical resolved prompt. Token and cost totals are filled in afterward by
+        ``populate_context_post_run``. ``--workspace-dir "$(pwd)"`` is load-bearing --
+        without it the tempdir sandbox writes the agent's workspace somewhere Harbor's
+        verifier never looks. ``--logging-dir``/``--artifacts-dir`` are static paths, not
+        templates; ``--run-dir`` is a throwaway path outside the workspace.
 
-        Rationale: .claude/notes/reporting.md § The non-obvious constraint in the emitted task.yaml
+        Rationale: .claude/notes/persistence.md § CoderEvalAgent passes both templates as static paths
         """
         del instruction, context  # nothing to forward; context is populated post-run
         run_dir = self.environment_logs_dir.as_posix()
         command = (
             f"coder-eval execute {shlex.quote(AGENT_TASK_YAML_PATH)} --format harbor "
-            f'--run-dir {shlex.quote(run_dir)} --workspace-dir "$(pwd)"'
+            f"--run-dir {_THROWAWAY_RUN_DIR} "
+            f'--workspace-dir "$(pwd)" '
+            f'--logging-dir {shlex.quote(run_dir)} --artifacts-dir "$(pwd)"'
         )
         await self._exec(environment, command)
 

{coder_eval-0.12.3 → coder_eval-0.12.4}/src/coder_eval/harbor/atif_emit.py

@@ -17,6 +17,7 @@ Rationale: .claude/notes/reporting.md § Emitting
 from __future__ import annotations
 
 import logging
+from collections.abc import Iterable
 from pathlib import Path
 from typing import Any
 
@@ -385,21 +386,24 @@ def write_trajectory_json(result: EvaluationResult, path: Path) -> Path | None:
         return None
 
 
-def emit_trajectories_for_run(run_dir: Path) -> list[Path]:
-    """Write a ``trajectory.json`` sibling for every ``task.json`` under ``run_dir``.
+def emit_trajectories_for_run(task_dirs: Iterable[Path]) -> list[Path]:
+    """Write a ``trajectory.json`` sibling for every ``task.json`` in ``task_dirs``.
 
-    The ``--format harbor`` post-pass for ``coder-eval execute``: ATIF emission
-    is opt-in (unlike the old always-on design this module's predecessor
-    shipped), so a plain ``run``/``execute`` never gains a new output file.
-    Walks the run directory rather than hooking the orchestrator's finalize
-    path, keeping this package's "translate coder-eval's own artifacts"
-    scope (see ``coder_eval.harbor``'s module docstring) — it needs no access
-    to orchestrator internals, only the ``task.json`` files a run already
-    wrote. Per-task failures are logged and skipped (see
+    ``task_dirs`` are the per-task logging directories, resolved from the same
+    ``logging_dir_template`` the run wrote ``task.json`` with -- taken as an argument
+    rather than discovered by walking a run directory, since an overridden logging dir
+    need not live under ``run_dir`` at all. The ``--format harbor`` post-pass for
+    ``coder-eval execute``; opt-in, so a plain ``run``/``execute`` never gains a new
+    output file. Per-task failures are logged and skipped (see
     :func:`write_trajectory_json`), never aborting the rest of the run's export.
+
+    Rationale: .claude/notes/persistence.md § What run_dir still owns
     """
     written: list[Path] = []
-    for task_json in sorted(run_dir.glob(f"**/{TASK_JSON_FILENAME}")):
+    for task_json in sorted({d / TASK_JSON_FILENAME for d in task_dirs}):
+        if not task_json.is_file():
+            logger.warning("No %s at %s — skipping ATIF emission", TASK_JSON_FILENAME, task_json)
+            continue
         try:
             result = EvaluationResult.model_validate_json(task_json.read_text(encoding="utf-8"))
         except (OSError, ValueError):