parallax-evals 2.0.0.dev1__tar.gz → 2.0.0.dev3__tar.gz

{parallax_evals-2.0.0.dev1 → parallax_evals-2.0.0.dev3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: parallax-evals
-Version: 2.0.0.dev1
+Version: 2.0.0.dev3
 Summary: Agent evaluations on seeded MCP worlds, recorded and compared in Langfuse.
 License-Expression: MIT
 License-File: LICENSE
@@ -160,7 +160,7 @@ and how comparisons pair runs.
 
 ## Status
 
-`2.0.0.dev1`. The task format, world, checks, controls, statistics and Langfuse
+`2.0.0.dev3`. The task format, world, checks, controls, statistics and Langfuse
 recording are in place. No pilot has been run, so no cross-harness results are
 published yet.
 

{parallax_evals-2.0.0.dev1 → parallax_evals-2.0.0.dev3}/README.md

@@ -137,7 +137,7 @@ and how comparisons pair runs.
 
 ## Status
 
-`2.0.0.dev1`. The task format, world, checks, controls, statistics and Langfuse
+`2.0.0.dev3`. The task format, world, checks, controls, statistics and Langfuse
 recording are in place. No pilot has been run, so no cross-harness results are
 published yet.
 

{parallax_evals-2.0.0.dev1 → parallax_evals-2.0.0.dev3}/docs/adapters.md

@@ -32,8 +32,11 @@ class Harness(Protocol):
 - the per-trial `workspace` and `home`;
 - the meter's OpenAI and Anthropic base URLs and a per-trial key.
 
-Parallax sets no time or cost limit on the agent: a turn lasts until the
-product ends it, under the product's own limits, and the adapter waits for that.
+By default Parallax sets no time or cost limit on the agent: a turn lasts until
+the product ends it, under the product's own limits, and the adapter waits for
+that. With `--turn-timeout`, Parallax cancels a turn still running at the
+deadline and records a timeout. `stop` still runs afterwards, so release
+anything a cancelled turn holds there.
 
 When cost is metered, real provider keys never enter the box.
 
@@ -341,7 +344,7 @@ its provenance; unknown or unpriced usage is never silently treated as free.
 | codex | open | CLI `codex exec --json` | yes | metered | Verified: codex-cli 0.156.1 turns recorded |
 | hermes | open | CLI `hermes chat -Q -q` | yes | metered | 0.22.6 runtime protocol probes against a scripted upstream; see evidence below |
 | openclaw | open | CLI `openclaw agent --local --json` | yes | metered | 2026.10.10 runtime protocol probes; clean Docker install verified; see evidence below |
-| grok-build | product | CLI `grok -p --output-format streaming-json` | yes | harness | Unverified: not installed, synthetic fixture from public docs |
+| grok-build | product | CLI `grok -p --output-format streaming-json` | yes | metered | Flags and config verified against grok 1.0.41 (`fixtures/grok-build/help.txt`) |
 | oracle | control | Replays `solution/` | n/a | none | Control |
 | null-claim / null-refuse | control | Fixed answer, no calls | n/a | none | Control |
 

{parallax_evals-2.0.0.dev1 → parallax_evals-2.0.0.dev3}/docs/methodology.md

@@ -262,7 +262,7 @@ A human moves a flagged task by editing `task.toml` and bumping the task
 This section assesses Parallax against the Agentic Benchmark Checklist (Zhu et
 al., 2025, *Establishing Best Practices for Building Rigorous Agentic
 Benchmarks*). The items are grouped under the checklist's three headings. The
-status is as of `2.0.0.dev1`, before the first pilot:
+status is as of `2.0.0.dev3`, before the first pilot:
 
 - ✅ met by construction and enforced in CI;
 - ◐ designed but not yet exercised or complete;
@@ -272,7 +272,7 @@ status is as of `2.0.0.dev1`, before the first pilot:
 
 | Item | Status | How |
 |---|---|---|
-| Tool and environment versions are pinned | ◐ | `uv.lock`, the box images keyed by install line, and the harness version recorded per row. The Docker images are not yet built or published |
+| Tool and environment versions are pinned | ◐ | `uv.lock`, a sidecar image holding the host's dependency versions, the box images keyed by install line, and the harness version recorded per row. The Docker images are not yet built or published |
 | External tools are deterministic, with no live APIs or rate limits | ✅ | Every service is a seeded in-process mock. Rate limits exist only as authored faults |
 | The environment resets between trials | ✅ | Each trial gets a fresh world from the seed, a fresh token, fresh `workspace` and `home`, and a fresh box |
 | The agent cannot reach ground truth or grader code | ◐ | Checks, oracle and mutants are never mounted into the box, and the world exposes only tool calls. Docker isolation is not yet exercised end to end |

{parallax_evals-2.0.0.dev1 → parallax_evals-2.0.0.dev3}/docs/quickstart.md

@@ -69,7 +69,11 @@ parallax run lite --harness codex --model openai/gpt-5-mini
 
 `--harness` and `--model` take comma-separated lists; every harness runs with
 every model, `--repeats` times (default 3). `--dry-run` lists the selected tasks
-and runs nothing. A run goes through these steps:
+and runs nothing. A turn waits for the harness to finish, however long it takes;
+`--turn-timeout SECONDS` kills a turn still running after that and records the
+trial as a timeout, so a hung harness cannot hold up a paid run. Parallax never
+caps spend itself, so size a paid run by its trial count. A run goes through
+these steps:
 
 1. It confirms Langfuse accepts writes for the configured project, before
    anything is spent.

{parallax_evals-2.0.0.dev1 → parallax_evals-2.0.0.dev3}/docs/roadmap.md

@@ -1,6 +1,6 @@
 # Roadmap
 
-Parallax is at `2.0.0.dev1`. The task format, the simulated world, the checks,
+Parallax is at `2.0.0.dev3`. The task format, the simulated world, the checks,
 the controls, the statistics and Langfuse recording are in place and exercised by CI.
 This page is directional; it is not a commitment to dates.
 

parallax_evals-2.0.0.dev3/parallax/__init__.py

@@ -0,0 +1 @@
+__version__ = "2.0.0.dev3"

{parallax_evals-2.0.0.dev1 → parallax_evals-2.0.0.dev3}/parallax/base.Dockerfile

@@ -4,7 +4,7 @@
 #                       layer stays cached across parallax source changes.
 #   (default)        -> parallax-base:<version>-<src8>: runtime + the parallax wheel; the sidecar.
 # Build context: a directory holding px_clock.c and (default target only) one parallax wheel and
-# requirements.txt, uv.lock's pinned dependency versions exported by box.build_with_wheel.
+# constraints.txt, the dependency versions the host runs, written by box.build_with_wheel.
 ARG NODE_IMAGE=node:24.22.2-bookworm-slim
 FROM ${NODE_IMAGE} AS node
 
@@ -34,10 +34,9 @@ RUN useradd --create-home --home-dir /home/agent --shell /bin/sh agent \
  && mkdir /etc/parallax && echo 0 > /etc/parallax/clock
 
 FROM runtime AS base
-COPY *.whl requirements.txt /tmp/wheels/
-# The locked versions first (hash-checked), then the wheel alone, so pip resolves nothing itself.
-RUN pip install --no-cache-dir --require-hashes -r /tmp/wheels/requirements.txt \
- && pip install --no-cache-dir --no-deps /tmp/wheels/*.whl \
+COPY *.whl constraints.txt /tmp/wheels/
+# Every dependency at the host's version, so the sidecar runs (and prices) what the host does.
+RUN pip install --no-cache-dir -c /tmp/wheels/constraints.txt /tmp/wheels/*.whl \
  && pip check \
  && rm -rf /tmp/wheels
 USER world

{parallax_evals-2.0.0.dev1 → parallax_evals-2.0.0.dev3}/parallax/box.py

@@ -36,9 +36,12 @@ from collections.abc import Mapping, Sequence
 from dataclasses import dataclass
 from datetime import UTC, datetime
 from enum import StrEnum
+from importlib.metadata import distributions
 from pathlib import Path
 from typing import assert_never
 
+import httpx
+
 from parallax import __version__
 from parallax.desktop import EXEC as DESKTOP_EXEC
 from parallax.meter import ADMIN_TOKEN_ENV as METER_ADMIN_ENV
@@ -305,8 +308,8 @@ class ExecProc:
         self, stdin: bytes | None, timeout_s: float | None, *, quiet_s: float | None = None
     ) -> tuple[int | None, str, str]:
         """Returns `(None, out, err)` on timeout, after killing the process tree. `timeout_s` None
-        waits for the process to exit however long it takes (an agent's turn: its product owns the
-        limit). `quiet_s`: see `settle_pipes`."""
+        waits for the process to exit however long it takes, or until the wait is cancelled, which kills
+        the tree (an agent's turn: `--turn-timeout`). `quiet_s`: see `settle_pipes`."""
         out, err = bytearray(), bytearray()
         readers = asyncio.gather(drain(self.proc.stdout, out), drain(self.proc.stderr, err))
         readers.add_done_callback(observe_drains)
@@ -316,7 +319,11 @@ class ExecProc:
                     self.proc.stdin.write(stdin)
                     await self.proc.stdin.drain()
                 self.proc.stdin.close()
-        code = await wait_exit(self.proc, timeout_s)
+        try:
+            code = await wait_exit(self.proc, timeout_s)
+        except asyncio.CancelledError:  # the turn deadline or an interrupt: take the tree with it
+            await self.killer.kill()
+            raise
         if code is None:
             await self.killer.kill()
         self.exited_at = time.monotonic()
@@ -662,12 +669,21 @@ def sha8(text: str) -> str:
     return hashlib.sha256(text.encode()).hexdigest()[:8]
 
 
-def source_digest(source: Path) -> str:
-    """sha8 over the files a wheel of this tree ships, plus `pyproject.toml` and `uv.lock` (the dependencies).
+def source_checkout() -> Path | None:
+    """The tree images are built from, or None for an installed release, whose images come from PyPI."""
+    source = PACKAGE_DIR.parent
+    return source if (source / "pyproject.toml").is_file() else None
 
-    Image tags carry it, so an edit to the world, meter or a worker always reaches the sidecar and
-    the boxes instead of a stale image cached under an unchanged version string.
+
+def build_digest() -> str:
+    """What image tags carry: `pypi` for a release (immutable once published), else a source digest.
+
+    That is sha8 over the files a wheel of the tree ships, plus `pyproject.toml` and `uv.lock` (the
+    dependencies), so an edit to the world, meter or a worker always reaches the sidecar and the
+    boxes instead of a stale image cached under an unchanged version string.
     """
+    if (source := source_checkout()) is None:
+        return "pypi"
     digest = hashlib.sha256()
     package = source / "parallax"
     files = sorted(p for p in package.rglob("*") if p.is_file() and "__pycache__" not in p.parts and p.suffix != ".pyc")
@@ -676,20 +692,12 @@ def source_digest(source: Path) -> str:
     return digest.hexdigest()[:8]
 
 
-def source_checkout() -> Path:
-    """The tree images are built from; an installed wheel has none, so that fails loudly."""
-    source = PACKAGE_DIR.parent
-    if not (source / "pyproject.toml").is_file():
-        raise BoxError("parallax images are built from a source checkout, and this install has none")
-    return source
-
-
 def base_image_tag() -> str:
-    return f"parallax-base:{__version__}-{source_digest(source_checkout())}"
+    return f"parallax-base:{__version__}-{build_digest()}"
 
 
 def box_image_tag(info: HarnessInfo) -> str:
-    return f"parallax-box-{info.name}:{__version__}-{sha8(info.install + source_digest(source_checkout()))}"
+    return f"parallax-box-{info.name}:{__version__}-{sha8(info.install + build_digest())}"
 
 
 def desktop_image_tag(info: HarnessInfo) -> str:
@@ -717,28 +725,43 @@ async def image_exists(tag: str) -> bool:
 
 
 async def build_with_wheel(dockerfile: str, tag: str, *build_args: str, platform_: str | None = None) -> None:
-    """`docker build` of one of the package Dockerfiles, with a fresh wheel of the source as context.
+    """`docker build` of one of the package Dockerfiles, with a parallax wheel as context: a fresh
+    build of the tree in a checkout, else this release's wheel from PyPI.
 
-    The context also carries `requirements.txt`, the dependency versions `uv.lock` pins (hashed),
-    so the sidecar runs what the host does: an unpinned install once gave it a newer genai-prices
-    than the host's, and the two priced the same model differently.
+    The context also carries `constraints.txt`, the versions the host runs, so the sidecar prices as
+    the host does: an unpinned install once gave it a newer genai-prices than the host's, and the
+    two priced the same model differently.
     """
-    source = source_checkout()
     with tempfile.TemporaryDirectory(prefix="px-image-") as ctx:
-        built = await run(("uv", "build", "--wheel", "--out-dir", ctx, str(source)), timeout_s=600)
-        if built.code != 0:
+        if (source := source_checkout()) is None:
+            await fetch_release(Path(ctx))
+        elif (built := await run(("uv", "build", "--wheel", "--out-dir", ctx, str(source)), timeout_s=600)).code:
             raise BoxError(f"uv build failed: {built.err.strip()[-2000:]}")
-        requirements = Path(ctx) / "requirements.txt"
-        export = ("uv", "export", "--locked", "--no-dev", "--no-emit-project", "--quiet", "--project", str(source))
-        exported = await run((*export, "--output-file", str(requirements)), timeout_s=120)
-        if exported.code != 0:
-            raise BoxError(f"uv export failed (is uv.lock current? run `uv lock`): {exported.err.strip()[-2000:]}")
+        # parallax itself is the wheel: an editable install's metadata can lag the version string
+        pins = {
+            f"{n}=={d.version}\n" for d in distributions() if (n := d.metadata["Name"]) not in (None, "parallax-evals")
+        }
+        (Path(ctx) / "constraints.txt").write_text("".join(sorted(pins)))
         shutil.copy(PACKAGE_DIR / "px_clock.c", ctx)
         args = [a for arg in build_args for a in ("--build-arg", arg)]
         target = ["--platform", platform_] if platform_ else []
         await docker("build", "-f", str(PACKAGE_DIR / dockerfile), *target, *args, "-t", tag, ctx, timeout_s=1800)
 
 
+async def fetch_release(ctx: Path) -> None:
+    """This version's wheel from PyPI, checked against its published sha256."""
+    try:
+        async with httpx.AsyncClient(timeout=120, follow_redirects=True) as client:
+            listing = (await client.get(f"https://pypi.org/pypi/parallax-evals/{__version__}/json")).raise_for_status()
+            wheel = next(u for u in listing.json()["urls"] if u["packagetype"] == "bdist_wheel")
+            body = (await client.get(wheel["url"])).raise_for_status().content
+    except (httpx.HTTPError, StopIteration) as exc:
+        raise BoxError(f"no parallax-evals {__version__} wheel on PyPI to build images from: {exc!r}") from exc
+    if hashlib.sha256(body).hexdigest() != wheel["digests"]["sha256"]:
+        raise BoxError(f"{wheel['filename']} from PyPI does not match its published sha256")
+    (ctx / wheel["filename"]).write_bytes(body)
+
+
 async def build_runtime_image(platform_: str | None = None) -> str:
     """`parallax-runtime:<version>`: the base stage without parallax; docker's cache keeps it cheap."""
     tag = f"parallax-runtime:{__version__}{_foreign(platform_)}"
@@ -838,8 +861,10 @@ class Sidecar:
         net = f"px-{run_id}"
         name = f"{net}-sidecar"
         image = await build_base_image()
-        await docker("network", "create", "--internal", "--label", f"parallax.net={net}", net)
         self.network = net
+        await self.down()  # a resumed run reuses its id: clear what a killed attempt left behind
+        self.network = net  # before the create: an interrupt mid-create must still leave it to `down`
+        await docker("network", "create", "--internal", "--label", f"parallax.net={net}", net)
         env_names = [k for k in UPSTREAM_ENV if k in os.environ]
         serve = (
             f"python -m parallax.world serve --http 0.0.0.0:{WORLD_PORT} & "
@@ -904,8 +929,11 @@ class Sidecar:
             ids = (await docker("ps", "-aq", "--filter", f"label=parallax.net={self.network}")).split()
             if ids:
                 await docker("rm", "-f", *ids, timeout_s=120)
-        with contextlib.suppress(BoxError):
+        try:
             await docker("network", "rm", self.network, timeout_s=60)
+        except BoxError as exc:
+            if "not found" not in str(exc):  # none yet is the normal case before `up` creates it
+                logger.warning("box.network_rm network=%s: %s", self.network, exc)
         self.network = None
         self.container = None
 

{parallax_evals-2.0.0.dev1 → parallax_evals-2.0.0.dev3}/parallax/cli.py

@@ -101,7 +101,7 @@ def cmd_run(a: argparse.Namespace) -> int:
         out=out, repeats=a.repeats, isolation=a.isolation, concurrency=a.concurrency,
         world_public_url=a.world_public_url, world_port=a.world_port, judge=csv(a.judge) if a.judge else (),
         include_xfail=a.include_xfail, selection=" ".join(a.select), provenance_extra=extra,
-        pairs=pinned, retry_invalid=a.retry_invalid,
+        pairs=pinned, retry_invalid=a.retry_invalid, turn_timeout_s=a.turn_timeout,
     )  # fmt: skip
     run_dir = run(tasks, harnesses, models, opts)
     # Latest row per key: a transient-invalid trial's failed attempt is evidence, not a second trial.
@@ -275,6 +275,7 @@ def run_parser(sub: argparse._SubParsersAction[argparse.ArgumentParser]) -> None
         "--retry-invalid", type=int, default=1, metavar="N",
         help="re-run a trial that ended INVALID for a transient execution fault, up to N further attempts",
     )  # fmt: skip
+    p.add_argument("--turn-timeout", type=float, metavar="SECONDS", help="kill a turn still running after this")
     p.add_argument("--out", type=Path, help="run directory (default: runs/<timestamp>); deleted once uploaded")
     p.add_argument("--resume", type=Path, metavar="RUN_DIR", help="continue an interrupted run")
     p.add_argument("--concurrency", type=int, default=4)

{parallax_evals-2.0.0.dev1 → parallax_evals-2.0.0.dev3}/parallax/controls.py

@@ -39,6 +39,7 @@ from parallax.runner import (
     TrialRun,
     harness_config_hash,
     judge_setup,
+    run_to_teardown,
     services,
 )
 from parallax.semantic import grade_checks, record_grade
@@ -366,7 +367,7 @@ def control_plan(name: str) -> HarnessPlan:
 
 
 def check(tasks: Sequence[Task], isolation: Isolation) -> ControlsReport:
-    return asyncio.run(check_async(tasks, isolation))
+    return run_to_teardown(check_async(tasks, isolation))
 
 
 def render(report: ControlsReport) -> str:

{parallax_evals-2.0.0.dev1 → parallax_evals-2.0.0.dev3}/parallax/extensions.py

@@ -8,7 +8,7 @@ Kept dependency-light: it must be importable from models, task and runner.
 from __future__ import annotations
 
 import functools
-from collections.abc import Callable, Iterable, Iterator, Mapping, Sequence
+from collections.abc import Callable, Iterable, Mapping, Sequence
 from dataclasses import dataclass
 from typing import TYPE_CHECKING
 
@@ -135,15 +135,10 @@ def grading_policy(task: Task) -> GradingPolicy | None:
     return found[0] if found else None
 
 
-def lane_rules(task: Task) -> Iterator[LaneRules]:
-    for ns in task.extensions:
-        if (lane := task_extensions()[ns].lane) is not None:
-            yield lane
-
-
 def check_lanes(tasks: Iterable[Task], harnesses: Sequence[str], concurrency: int) -> None:
     """Raise for the first lane, by label, whose rules this run breaks."""
-    rules = dict.fromkeys(r for t in tasks for r in lane_rules(t))
+    lanes = (task_extensions()[ns].lane for t in tasks for ns in t.extensions)
+    rules = dict.fromkeys(lane for lane in lanes if lane is not None)
     for rule in sorted(rules, key=lambda r: r.label):
         if rule.violated(tuple(harnesses), concurrency):
             raise ValueError(rule.message())

{parallax_evals-2.0.0.dev1 → parallax_evals-2.0.0.dev3}/parallax/harnesses/base.py

@@ -25,7 +25,7 @@ from parallax.models import (
     TurnStatus,
     Usage,
 )
-from parallax.protocols import BoxHandle, Harness, Proc
+from parallax.protocols import BoxHandle, Harness
 
 STDERR_TAIL_CHARS = 4000
 RAW_OUTPUT_CHARS = 2 * 1024 * 1024  # the newest raw harness output kept per trial for diagnosis
@@ -167,7 +167,6 @@ class SubprocessHarness(ABC):
         self._info = info
         self._trial: Trial | None = None
         self._box: BoxHandle | None = None
-        self._proc: Proc | None = None
         self.session_id: str | None = None
         self.in_session = False
         self.stderr_tail = ""
@@ -228,12 +227,9 @@ class SubprocessHarness(ABC):
         prompt_file = self.trial.home / "turns" / f"{t.index:02d}.md"
         await self.box.write_file(prompt_file, t.text.encode())
         argv = self.argv(t, continuing=self.in_session, session_id=self.session_id, prompt_file=prompt_file)
-        self._proc = await self.box.spawn(argv, env=self.env(), cwd=self.trial.workspace)
-        try:
-            # No deadline: the turn ends when the product ends it, under its own limits.
-            code, out, err = await self._proc.communicate(self.stdin(t), None)
-        finally:
-            self._proc = None
+        proc = await self.box.spawn(argv, env=self.env(), cwd=self.trial.workspace)
+        # No deadline here: the runner's `--turn-timeout` cancels the wait, which kills the process tree.
+        code, out, err = await proc.communicate(self.stdin(t), None)
         self.stderr_tail = err[-STDERR_TAIL_CHARS:]
         self.output_tail.add(f"=== turn {t.index} exit {code} ===\n--- stdout ---\n{out}\n--- stderr ---\n{err}\n")
         self.in_session = True
@@ -249,18 +245,12 @@ class SubprocessHarness(ABC):
         steps = tuple(s.model_copy(update={"turn": index}) for s in out.steps)
         usage = tuple(u if u.model else u.model_copy(update={"model": requested_model(self.trial)}) for u in out.usage)
         if self.cumulative_usage:
-            usage = self.per_turn(usage)
+            totals, usage = usage, tuple(subtract_usage(u, self._usage_seen.get(u.model)) for u in usage)
+            self._usage_seen.update({u.model: u for u in totals})
         return out.model_copy(update={"steps": steps, "usage": usage})
 
-    def per_turn(self, totals: tuple[Usage, ...]) -> tuple[Usage, ...]:
-        deltas = tuple(subtract_usage(u, self._usage_seen.get(u.model)) for u in totals)
-        self._usage_seen.update({u.model: u for u in totals})
-        return deltas
-
     async def stop(self) -> None:
-        proc, self._proc = self._proc, None
-        if proc is not None:
-            await proc.kill()
+        return None  # a turn's process never outlives its turn: a cancelled wait kills it
 
     async def version(self, box: BoxHandle) -> str:
         return await run_version(self.info, box)

{parallax_evals-2.0.0.dev1 → parallax_evals-2.0.0.dev3}/parallax/harnesses/oracle.py

@@ -143,14 +143,16 @@ class OracleHarness:
         self._saved = {}
 
     def step_name(self, index: int) -> str:
-        task = self._require_task()
+        if (task := self._task) is None:
+            raise OracleError("oracle has no task bound")
         names = [t.name for t in task.turns] + [f.name for f in task.meta.fires]
         if not 1 <= index <= len(names):
             raise OracleError(f"turn {index} is outside the task's {len(names)} steps and fires")
         return names[index - 1]
 
     async def turn(self, t: TurnInput) -> TurnOutput:
-        trial = self._require_trial()
+        if (trial := self._trial) is None:
+            raise OracleError("oracle turn() before start()")
         try:
             name = self.step_name(t.index)
             client = WorldClient(trial.mcp.url, trial.env[trial.mcp.token_env])
@@ -188,16 +190,6 @@ class OracleHarness:
                 self._saved[call.save] = result
         return TurnOutput(status=TurnStatus.COMPLETED, answer=step.answer, session_id=None, steps=tuple(steps))
 
-    def _require_task(self) -> Task:
-        if self._task is None:
-            raise OracleError("oracle has no task bound")
-        return self._task
-
-    def _require_trial(self) -> Trial:
-        if self._trial is None:
-            raise OracleError("oracle turn() before start()")
-        return self._trial
-
     async def stop(self) -> None:
         return None
 

{parallax_evals-2.0.0.dev1 → parallax_evals-2.0.0.dev3}/parallax/jev.py

@@ -166,11 +166,6 @@ class JevCall(Model):
     history: tuple[JevAttempt, ...] = ()
 
 
-def answers_match(answers: Mapping[str, BoolAnswer], questions: Mapping[str, BoolQuestion]) -> bool:
-    """Exactly the questions asked; the reply model already rejects any non-boolean answer."""
-    return set(answers) == set(questions)
-
-
 def state_sha(state: Mapping[str, JsonValue]) -> str:
     return hashlib.sha256(json.dumps(state, sort_keys=True).encode()).hexdigest()
 
@@ -321,11 +316,9 @@ def _parsed(
     try:
         wire = _Reply.model_validate_json(content)
         update["warnings"] = wire.warnings
-        error = (
-            None
-            if answers_match(wire.answers, questions)
-            else f"answers do not match the questions asked: {sorted(wire.answers)}"
-        )
+        # Exactly the questions asked; the reply model already rejects any non-boolean answer.
+        match = set(wire.answers) == set(questions)
+        error = None if match else f"answers do not match the questions asked: {sorted(wire.answers)}"
         if error is None:
             update["answers"] = dict(wire.answers)
     except ValidationError as exc:

{parallax_evals-2.0.0.dev1 → parallax_evals-2.0.0.dev3}/parallax/runner.py

@@ -16,10 +16,11 @@ import os
 import re
 import secrets
 import shutil
+import signal
 import socket
 import sys
 import time
-from collections.abc import AsyncIterator, Awaitable, Iterator, Mapping, Sequence
+from collections.abc import AsyncIterator, Awaitable, Coroutine, Iterator, Mapping, Sequence
 from dataclasses import dataclass, field
 from datetime import UTC, datetime
 from pathlib import Path
@@ -156,6 +157,7 @@ class RunOptions(Model):
     include_xfail: bool = False
     # Further attempts for a trial that ended INVALID for a transient execution fault (RETRYABLE_INVALID).
     retry_invalid: int = 1
+    turn_timeout_s: float | None = None  # None: a turn ends when the product ends it
     selection: str = ""
     quiet: bool = False
     # Recorded in provenance.json: values only the caller knows.
@@ -238,6 +240,21 @@ async def finish_uncancelled(cleanup: Awaitable[None]) -> None:
     task.result()
 
 
+def run_to_teardown[T](main: Coroutine[None, None, T]) -> T:
+    """`asyncio.run`, but every SIGINT only cancels `main`. A second one (a repeated Ctrl-C, or `uv run`
+    forwarding its copy of the terminal's) would raise KeyboardInterrupt mid-teardown and strand boxes."""
+
+    async def guarded() -> T:
+        if (task := asyncio.current_task()) is not None:
+            asyncio.get_running_loop().add_signal_handler(signal.SIGINT, task.cancel)
+        return await main
+
+    try:
+        return asyncio.run(guarded())
+    except asyncio.CancelledError:
+        raise KeyboardInterrupt from None
+
+
 # ---------------------------------------------------------------------------
 # redaction, paths, placement
 # ---------------------------------------------------------------------------
@@ -480,7 +497,7 @@ async def services(
         await svc.start()
         yield svc
     finally:
-        await svc.stop()
+        await finish_uncancelled(svc.stop())  # a Ctrl-C landing mid-teardown would strand the run network
 
 
 # ---------------------------------------------------------------------------
@@ -655,11 +672,13 @@ class Session:
         run_dir: Path,
         panel: JudgePanel | None,
         judge_config: JudgeConfig | None,
+        turn_timeout_s: float | None = None,
     ) -> None:
         self.svc = svc
         self.run_dir = run_dir
         self.panel = panel
         self.judge_config = judge_config
+        self.turn_timeout_s = turn_timeout_s
 
     async def trial(self, plan: TrialPlan) -> TrialRun:
         task, hp = plan.task, plan.harness
@@ -899,7 +918,9 @@ class TrialExecution:
                 return
             if job is None:
                 continue
-            await control.patch(run_record(fire, job))
+            # the scheduler's own record of the delivery, so a later `list_runs` sees it
+            row = {"id": f"run-{fire.name}", "job": str(job.get("id")), "at": fire.at.isoformat(), "fire": fire.name}
+            await control.patch(WorldPatch(insert={"scheduler.runs": [row]}))
             text = fire_text(fire, job, self.task.seed.user.timezone)
             if not await self.deliver(harness, index, text, True, ()):
                 return
@@ -927,18 +948,23 @@ class TrialExecution:
         origin: RunOrigin = RunOrigin.MEMBER,
         approval: ApprovalDecision | None = None,
     ) -> bool:
-        """One turn, waited on until the harness returns: the product under test owns its limits.
-        False ends the trial (a non-completed turn)."""
+        """One turn, waited on until the harness returns or `--turn-timeout` passes (none by default:
+        the product under test owns its limits). False ends the trial (a non-completed turn)."""
         t_in = TurnInput(
             index=index, text=text, new_session=new_session, attachments=attached, origin=origin, approval=approval
         )
         self.t.delivered_turns.append(t_in)
         submitted = time.monotonic()
         out: TurnOutput | None = None
+        deadline = asyncio.timeout(self.s.turn_timeout_s)
         try:
-            out = await harness.turn(t_in)
+            async with deadline:
+                out = await harness.turn(t_in)
         except Exception as exc:  # an adapter crash mid-turn is the harness's failure, not the eval's
             self.t.status, self.t.error = TrajectoryStatus.HARNESS_ERROR, f"turn {index}: {type(exc).__name__}: {exc}"
+            if deadline.expired():  # status and error then come from the TIMEOUT turn, below
+                error = f"turn {index}: still running after {self.s.turn_timeout_s:.0f}s, killed"
+                out = TurnOutput(status=TurnStatus.TIMEOUT, answer="", session_id=None, error=error)
         # Every turn's time counts toward agent_ms, a failed one's included.
         measured = int((time.monotonic() - submitted) * 1000)
         if out is None:
@@ -1106,22 +1132,12 @@ class TrialExecution:
         world = ev.world
         self.write(self.dir / "world_evidence.json", world.model_dump_json(indent=2))
         self.write(self.dir / "world_log.jsonl", "".join(c.model_dump_json() + "\n" for c in world.calls))
-        self.write(self.dir / "world_final.json", _json_state(world))
+        self.write(self.dir / "world_final.json", json.dumps(world.final, indent=2, ensure_ascii=False))
         self.write(self.dir / "world_diff.json", world.diff.model_dump_json(indent=2))
         for check_id, outcome in ev.verify.items():
             self.write(self.dir / "verify" / f"{safe_segment(check_id)}.log", outcome.model_dump_json(indent=2))
 
 
-def _json_state(world: WorldEvidence) -> str:
-    return json.dumps(world.final, indent=2, ensure_ascii=False)
-
-
-def run_record(fire: Fire, job: JsonObject) -> WorldPatch:
-    """The scheduler's own record of the delivery, so a later `list_runs` sees it."""
-    row = {"id": f"run-{fire.name}", "job": str(job.get("id")), "at": fire.at.isoformat(), "fire": fire.name}
-    return WorldPatch(insert={"scheduler.runs": [row]})
-
-
 # ---------------------------------------------------------------------------
 # planning, resume and the run
 # ---------------------------------------------------------------------------
@@ -1295,31 +1311,27 @@ def manifest_for(opts: RunOptions, plans: Sequence[TrialPlan], harnesses: Sequen
     )  # fmt: skip
 
 
-def schedulable(tasks: Sequence[Task], include_xfail: bool) -> list[Task]:
-    return [t for t in tasks if include_xfail or t.meta.xfail is None]
-
-
 async def run_async(
     tasks: Sequence[Task], harnesses: Sequence[str], models: Sequence[str | None], opts: RunOptions
 ) -> Path:
-    chosen = schedulable(tasks, opts.include_xfail)
+    chosen = [t for t in tasks if opts.include_xfail or t.meta.xfail is None]
     extensions.check_lanes(chosen, harnesses, opts.concurrency)
     if not chosen:
         raise ValueError("no tasks to run (xfail tasks need --include-xfail)")
     panel, judge_config = judge_setup(opts.judge)
     opts = opts.model_copy(update={"judge": panel.models if panel else ()})
-    opts.out.mkdir(parents=True, exist_ok=True)
     infos = [load_harness(h).info for h in harnesses]
     need_meter = any(i.cost_source is CostSource.METERED for i in infos)
     async with services(opts.isolation, opts.world_public_url, need_meter, opts.out.name, opts.world_port) as svc:
         plans_h = [await plan_harness(h, chosen, opts.isolation, svc.network) for h in harnesses]
         plans = only_pairs(plan_trials(chosen, plans_h, models, opts), opts.pairs)
+        opts.out.mkdir(parents=True, exist_ok=True)  # only now: a run interrupted in setup leaves no husk
         (opts.out / "manifest.json").write_text(manifest_for(opts, plans, plans_h).model_dump_json(indent=2))
         prov = provenance.collect(opts.out, opts.provenance_extra)
         provenance.write(opts.out, prov)
         done = completed_keys(opts.out, plans)
         todo = start_order([p for p in plans if p.key not in done])
-        session = Session(svc, opts.out, panel, judge_config)
+        session = Session(svc, opts.out, panel, judge_config, opts.turn_timeout_s)
         runs = {r.result.key: r for r in await execute_all(session, todo, opts)}
         await retry_transient_invalid(session, todo, runs, opts)
     served = provenance.served_models(read_results(opts.out / RESULTS))
@@ -1384,7 +1396,7 @@ async def execute_all(session: Session, todo: Sequence[TrialPlan], opts: RunOpti
 
 
 def run(tasks: Sequence[Task], harnesses: Sequence[str], models: Sequence[str | None], opts: RunOptions) -> Path:
-    return asyncio.run(run_async(tasks, harnesses, models, opts))
+    return run_to_teardown(run_async(tasks, harnesses, models, opts))
 
 
 def invalid_share(results: Sequence[TrialResult]) -> float: