parallax-evals 2.0.0.dev2__tar.gz → 2.0.0.dev4__tar.gz

{parallax_evals-2.0.0.dev2 → parallax_evals-2.0.0.dev4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: parallax-evals
-Version: 2.0.0.dev2
+Version: 2.0.0.dev4
 Summary: Agent evaluations on seeded MCP worlds, recorded and compared in Langfuse.
 License-Expression: MIT
 License-File: LICENSE
@@ -160,7 +160,7 @@ and how comparisons pair runs.
 
 ## Status
 
-`2.0.0.dev2`. The task format, world, checks, controls, statistics and Langfuse
+`2.0.0.dev4`. The task format, world, checks, controls, statistics and Langfuse
 recording are in place. No pilot has been run, so no cross-harness results are
 published yet.
 

{parallax_evals-2.0.0.dev2 → parallax_evals-2.0.0.dev4}/README.md

@@ -137,7 +137,7 @@ and how comparisons pair runs.
 
 ## Status
 
-`2.0.0.dev2`. The task format, world, checks, controls, statistics and Langfuse
+`2.0.0.dev4`. The task format, world, checks, controls, statistics and Langfuse
 recording are in place. No pilot has been run, so no cross-harness results are
 published yet.
 

{parallax_evals-2.0.0.dev2 → parallax_evals-2.0.0.dev4}/docs/methodology.md

@@ -262,7 +262,7 @@ A human moves a flagged task by editing `task.toml` and bumping the task
 This section assesses Parallax against the Agentic Benchmark Checklist (Zhu et
 al., 2025, *Establishing Best Practices for Building Rigorous Agentic
 Benchmarks*). The items are grouped under the checklist's three headings. The
-status is as of `2.0.0.dev2`, before the first pilot:
+status is as of `2.0.0.dev4`, before the first pilot:
 
 - ✅ met by construction and enforced in CI;
 - ◐ designed but not yet exercised or complete;
@@ -272,7 +272,7 @@ status is as of `2.0.0.dev2`, before the first pilot:
 
 | Item | Status | How |
 |---|---|---|
-| Tool and environment versions are pinned | ◐ | `uv.lock`, the box images keyed by install line, and the harness version recorded per row. The Docker images are not yet built or published |
+| Tool and environment versions are pinned | ◐ | `uv.lock`, a sidecar image holding the host's dependency versions, the box images keyed by install line, and the harness version recorded per row. The Docker images are not yet built or published |
 | External tools are deterministic, with no live APIs or rate limits | ✅ | Every service is a seeded in-process mock. Rate limits exist only as authored faults |
 | The environment resets between trials | ✅ | Each trial gets a fresh world from the seed, a fresh token, fresh `workspace` and `home`, and a fresh box |
 | The agent cannot reach ground truth or grader code | ◐ | Checks, oracle and mutants are never mounted into the box, and the world exposes only tool calls. Docker isolation is not yet exercised end to end |

{parallax_evals-2.0.0.dev2 → parallax_evals-2.0.0.dev4}/docs/roadmap.md

@@ -1,6 +1,6 @@
 # Roadmap
 
-Parallax is at `2.0.0.dev2`. The task format, the simulated world, the checks,
+Parallax is at `2.0.0.dev4`. The task format, the simulated world, the checks,
 the controls, the statistics and Langfuse recording are in place and exercised by CI.
 This page is directional; it is not a commitment to dates.
 

parallax_evals-2.0.0.dev4/parallax/__init__.py

@@ -0,0 +1 @@
+__version__ = "2.0.0.dev4"

{parallax_evals-2.0.0.dev2 → parallax_evals-2.0.0.dev4}/parallax/base.Dockerfile

@@ -4,7 +4,7 @@
 #                       layer stays cached across parallax source changes.
 #   (default)        -> parallax-base:<version>-<src8>: runtime + the parallax wheel; the sidecar.
 # Build context: a directory holding px_clock.c and (default target only) one parallax wheel and
-# requirements.txt, uv.lock's pinned dependency versions exported by box.build_with_wheel.
+# constraints.txt, the dependency versions the host runs, written by box.build_with_wheel.
 ARG NODE_IMAGE=node:24.22.2-bookworm-slim
 FROM ${NODE_IMAGE} AS node
 
@@ -34,10 +34,9 @@ RUN useradd --create-home --home-dir /home/agent --shell /bin/sh agent \
  && mkdir /etc/parallax && echo 0 > /etc/parallax/clock
 
 FROM runtime AS base
-COPY *.whl requirements.txt /tmp/wheels/
-# The locked versions first (hash-checked), then the wheel alone, so pip resolves nothing itself.
-RUN pip install --no-cache-dir --require-hashes -r /tmp/wheels/requirements.txt \
- && pip install --no-cache-dir --no-deps /tmp/wheels/*.whl \
+COPY *.whl constraints.txt /tmp/wheels/
+# Every dependency at the host's version, so the sidecar runs (and prices) what the host does.
+RUN pip install --no-cache-dir -c /tmp/wheels/constraints.txt /tmp/wheels/*.whl \
  && pip check \
  && rm -rf /tmp/wheels
 USER world

{parallax_evals-2.0.0.dev2 → parallax_evals-2.0.0.dev4}/parallax/box.py

@@ -36,9 +36,12 @@ from collections.abc import Mapping, Sequence
 from dataclasses import dataclass
 from datetime import UTC, datetime
 from enum import StrEnum
+from importlib.metadata import distributions
 from pathlib import Path
 from typing import assert_never
 
+import httpx
+
 from parallax import __version__
 from parallax.desktop import EXEC as DESKTOP_EXEC
 from parallax.meter import ADMIN_TOKEN_ENV as METER_ADMIN_ENV
@@ -666,12 +669,21 @@ def sha8(text: str) -> str:
     return hashlib.sha256(text.encode()).hexdigest()[:8]
 
 
-def source_digest(source: Path) -> str:
-    """sha8 over the files a wheel of this tree ships, plus `pyproject.toml` and `uv.lock` (the dependencies).
+def source_checkout() -> Path | None:
+    """The tree images are built from, or None for an installed release, whose images come from PyPI."""
+    source = PACKAGE_DIR.parent
+    return source if (source / "pyproject.toml").is_file() else None
+
 
-    Image tags carry it, so an edit to the world, meter or a worker always reaches the sidecar and
-    the boxes instead of a stale image cached under an unchanged version string.
+def build_digest() -> str:
+    """What image tags carry: `pypi` for a release (immutable once published), else a source digest.
+
+    That is sha8 over the files a wheel of the tree ships, plus `pyproject.toml` and `uv.lock` (the
+    dependencies), so an edit to the world, meter or a worker always reaches the sidecar and the
+    boxes instead of a stale image cached under an unchanged version string.
     """
+    if (source := source_checkout()) is None:
+        return "pypi"
     digest = hashlib.sha256()
     package = source / "parallax"
     files = sorted(p for p in package.rglob("*") if p.is_file() and "__pycache__" not in p.parts and p.suffix != ".pyc")
@@ -680,20 +692,12 @@ def source_digest(source: Path) -> str:
     return digest.hexdigest()[:8]
 
 
-def source_checkout() -> Path:
-    """The tree images are built from; an installed wheel has none, so that fails loudly."""
-    source = PACKAGE_DIR.parent
-    if not (source / "pyproject.toml").is_file():
-        raise BoxError("parallax images are built from a source checkout, and this install has none")
-    return source
-
-
 def base_image_tag() -> str:
-    return f"parallax-base:{__version__}-{source_digest(source_checkout())}"
+    return f"parallax-base:{__version__}-{build_digest()}"
 
 
 def box_image_tag(info: HarnessInfo) -> str:
-    return f"parallax-box-{info.name}:{__version__}-{sha8(info.install + source_digest(source_checkout()))}"
+    return f"parallax-box-{info.name}:{__version__}-{sha8(info.install + build_digest())}"
 
 
 def desktop_image_tag(info: HarnessInfo) -> str:
@@ -721,28 +725,43 @@ async def image_exists(tag: str) -> bool:
 
 
 async def build_with_wheel(dockerfile: str, tag: str, *build_args: str, platform_: str | None = None) -> None:
-    """`docker build` of one of the package Dockerfiles, with a fresh wheel of the source as context.
+    """`docker build` of one of the package Dockerfiles, with a parallax wheel as context: a fresh
+    build of the tree in a checkout, else this release's wheel from PyPI.
 
-    The context also carries `requirements.txt`, the dependency versions `uv.lock` pins (hashed),
-    so the sidecar runs what the host does: an unpinned install once gave it a newer genai-prices
-    than the host's, and the two priced the same model differently.
+    The context also carries `constraints.txt`, the versions the host runs, so the sidecar prices as
+    the host does: an unpinned install once gave it a newer genai-prices than the host's, and the
+    two priced the same model differently.
     """
-    source = source_checkout()
     with tempfile.TemporaryDirectory(prefix="px-image-") as ctx:
-        built = await run(("uv", "build", "--wheel", "--out-dir", ctx, str(source)), timeout_s=600)
-        if built.code != 0:
+        if (source := source_checkout()) is None:
+            await fetch_release(Path(ctx))
+        elif (built := await run(("uv", "build", "--wheel", "--out-dir", ctx, str(source)), timeout_s=600)).code:
             raise BoxError(f"uv build failed: {built.err.strip()[-2000:]}")
-        requirements = Path(ctx) / "requirements.txt"
-        export = ("uv", "export", "--locked", "--no-dev", "--no-emit-project", "--quiet", "--project", str(source))
-        exported = await run((*export, "--output-file", str(requirements)), timeout_s=120)
-        if exported.code != 0:
-            raise BoxError(f"uv export failed (is uv.lock current? run `uv lock`): {exported.err.strip()[-2000:]}")
+        # parallax itself is the wheel: an editable install's metadata can lag the version string
+        pins = {
+            f"{n}=={d.version}\n" for d in distributions() if (n := d.metadata["Name"]) not in (None, "parallax-evals")
+        }
+        (Path(ctx) / "constraints.txt").write_text("".join(sorted(pins)))
         shutil.copy(PACKAGE_DIR / "px_clock.c", ctx)
         args = [a for arg in build_args for a in ("--build-arg", arg)]
         target = ["--platform", platform_] if platform_ else []
         await docker("build", "-f", str(PACKAGE_DIR / dockerfile), *target, *args, "-t", tag, ctx, timeout_s=1800)
 
 
+async def fetch_release(ctx: Path) -> None:
+    """This version's wheel from PyPI, checked against its published sha256."""
+    try:
+        async with httpx.AsyncClient(timeout=120, follow_redirects=True) as client:
+            listing = (await client.get(f"https://pypi.org/pypi/parallax-evals/{__version__}/json")).raise_for_status()
+            wheel = next(u for u in listing.json()["urls"] if u["packagetype"] == "bdist_wheel")
+            body = (await client.get(wheel["url"])).raise_for_status().content
+    except (httpx.HTTPError, StopIteration) as exc:
+        raise BoxError(f"no parallax-evals {__version__} wheel on PyPI to build images from: {exc!r}") from exc
+    if hashlib.sha256(body).hexdigest() != wheel["digests"]["sha256"]:
+        raise BoxError(f"{wheel['filename']} from PyPI does not match its published sha256")
+    (ctx / wheel["filename"]).write_bytes(body)
+
+
 async def build_runtime_image(platform_: str | None = None) -> str:
     """`parallax-runtime:<version>`: the base stage without parallax; docker's cache keeps it cheap."""
     tag = f"parallax-runtime:{__version__}{_foreign(platform_)}"

{parallax_evals-2.0.0.dev2 → parallax_evals-2.0.0.dev4}/parallax/extensions.py

@@ -8,7 +8,7 @@ Kept dependency-light: it must be importable from models, task and runner.
 from __future__ import annotations
 
 import functools
-from collections.abc import Callable, Iterable, Iterator, Mapping, Sequence
+from collections.abc import Callable, Iterable, Mapping, Sequence
 from dataclasses import dataclass
 from typing import TYPE_CHECKING
 
@@ -135,15 +135,10 @@ def grading_policy(task: Task) -> GradingPolicy | None:
     return found[0] if found else None
 
 
-def lane_rules(task: Task) -> Iterator[LaneRules]:
-    for ns in task.extensions:
-        if (lane := task_extensions()[ns].lane) is not None:
-            yield lane
-
-
 def check_lanes(tasks: Iterable[Task], harnesses: Sequence[str], concurrency: int) -> None:
     """Raise for the first lane, by label, whose rules this run breaks."""
-    rules = dict.fromkeys(r for t in tasks for r in lane_rules(t))
+    lanes = (task_extensions()[ns].lane for t in tasks for ns in t.extensions)
+    rules = dict.fromkeys(lane for lane in lanes if lane is not None)
     for rule in sorted(rules, key=lambda r: r.label):
         if rule.violated(tuple(harnesses), concurrency):
             raise ValueError(rule.message())

{parallax_evals-2.0.0.dev2 → parallax_evals-2.0.0.dev4}/parallax/harnesses/base.py

@@ -245,14 +245,10 @@ class SubprocessHarness(ABC):
         steps = tuple(s.model_copy(update={"turn": index}) for s in out.steps)
         usage = tuple(u if u.model else u.model_copy(update={"model": requested_model(self.trial)}) for u in out.usage)
         if self.cumulative_usage:
-            usage = self.per_turn(usage)
+            totals, usage = usage, tuple(subtract_usage(u, self._usage_seen.get(u.model)) for u in usage)
+            self._usage_seen.update({u.model: u for u in totals})
         return out.model_copy(update={"steps": steps, "usage": usage})
 
-    def per_turn(self, totals: tuple[Usage, ...]) -> tuple[Usage, ...]:
-        deltas = tuple(subtract_usage(u, self._usage_seen.get(u.model)) for u in totals)
-        self._usage_seen.update({u.model: u for u in totals})
-        return deltas
-
     async def stop(self) -> None:
         return None  # a turn's process never outlives its turn: a cancelled wait kills it
 

{parallax_evals-2.0.0.dev2 → parallax_evals-2.0.0.dev4}/parallax/harnesses/oracle.py

@@ -151,7 +151,8 @@ class OracleHarness:
         return names[index - 1]
 
     async def turn(self, t: TurnInput) -> TurnOutput:
-        trial = self._require_trial()
+        if (trial := self._trial) is None:
+            raise OracleError("oracle turn() before start()")
         try:
             name = self.step_name(t.index)
             client = WorldClient(trial.mcp.url, trial.env[trial.mcp.token_env])
@@ -189,11 +190,6 @@ class OracleHarness:
                 self._saved[call.save] = result
         return TurnOutput(status=TurnStatus.COMPLETED, answer=step.answer, session_id=None, steps=tuple(steps))
 
-    def _require_trial(self) -> Trial:
-        if self._trial is None:
-            raise OracleError("oracle turn() before start()")
-        return self._trial
-
     async def stop(self) -> None:
         return None
 

{parallax_evals-2.0.0.dev2 → parallax_evals-2.0.0.dev4}/parallax/runner.py

@@ -918,7 +918,9 @@ class TrialExecution:
                 return
             if job is None:
                 continue
-            await control.patch(run_record(fire, job))
+            # the scheduler's own record of the delivery, so a later `list_runs` sees it
+            row = {"id": f"run-{fire.name}", "job": str(job.get("id")), "at": fire.at.isoformat(), "fire": fire.name}
+            await control.patch(WorldPatch(insert={"scheduler.runs": [row]}))
             text = fire_text(fire, job, self.task.seed.user.timezone)
             if not await self.deliver(harness, index, text, True, ()):
                 return
@@ -1136,12 +1138,6 @@ class TrialExecution:
             self.write(self.dir / "verify" / f"{safe_segment(check_id)}.log", outcome.model_dump_json(indent=2))
 
 
-def run_record(fire: Fire, job: JsonObject) -> WorldPatch:
-    """The scheduler's own record of the delivery, so a later `list_runs` sees it."""
-    row = {"id": f"run-{fire.name}", "job": str(job.get("id")), "at": fire.at.isoformat(), "fire": fire.name}
-    return WorldPatch(insert={"scheduler.runs": [row]})
-
-
 # ---------------------------------------------------------------------------
 # planning, resume and the run
 # ---------------------------------------------------------------------------

{parallax_evals-2.0.0.dev2 → parallax_evals-2.0.0.dev4}/parallax/semantic.py

@@ -69,8 +69,9 @@ def _files(root: Path | None) -> dict[str, JsonValue] | None:
     for path in sorted(root.rglob("*")):
         if path.is_symlink():
             raise MissingEvidence(f"cannot judge symlink snapshot: {path.relative_to(root)}")
-        # Bytecode appears when the agent runs Python in the workspace; it is not evidence of what it did.
-        if not path.is_file() or "__pycache__" in path.parts or path.suffix == ".pyc":
+        # Bytecode (the agent ran Python) and a git store (a harness such as openclaw ran `git init`) are
+        # tooling, not evidence of what it did: the working tree is.
+        if not path.is_file() or {"__pycache__", ".git"} & set(path.parts) or path.suffix == ".pyc":
             continue
         size += path.stat().st_size
         if size > PACK_LIMIT:

{parallax_evals-2.0.0.dev2 → parallax_evals-2.0.0.dev4}/parallax/track/sink.py

@@ -233,8 +233,8 @@ def experiment_id_for(project_id: str, dataset_id: str, run_name: str) -> str:
 
 
 def metadata_value(value: object) -> str:
-    """Experiment metadata as Parallax reads it: strings as they are, anything else as compact JSON."""
-    return value if isinstance(value, str) else json.dumps(value, separators=(",", ":"))
+    """Experiment metadata as Parallax reads it: strings as they are, anything else as compact JSON, unescaped as Langfuse holds it."""
+    return value if isinstance(value, str) else json.dumps(value, separators=(",", ":"), ensure_ascii=False)
 
 
 def _score(

{parallax_evals-2.0.0.dev2 → parallax_evals-2.0.0.dev4}/parallax/world/service.py

@@ -128,11 +128,7 @@ class Service:
         try:
             parsed = bound.args_model.model_validate(args)
         except ValidationError as exc:
-            raise ServiceError("validation", _validation_message(exc)) from exc
+            # vendor-style field errors: short, one per field, no pydantic URLs
+            parts = [f"{'.'.join(str(p) for p in err['loc']) or '<args>'}: {err['msg']}" for err in exc.errors()]
+            raise ServiceError("validation", "; ".join(parts)) from exc
         return await bound.fn(self, parsed)
-
-
-def _validation_message(exc: ValidationError) -> str:
-    # Vendor-style field errors: short, one per field, no pydantic URLs.
-    parts = [f"{'.'.join(str(p) for p in err['loc']) or '<args>'}: {err['msg']}" for err in exc.errors()]
-    return "; ".join(parts)

{parallax_evals-2.0.0.dev2 → parallax_evals-2.0.0.dev4}/tasks/conformance/conf-two-turn-carry/solution/oracle.yaml

@@ -2,4 +2,4 @@ steps:
   turn-01:
     answer: "Got it - I'll keep that in mind."
   turn-02:
-    answer: "The side door codeword is amber-falcon-62."
+    answer: "The booking reference is amber-falcon-62."

parallax_evals-2.0.0.dev4/tasks/conformance/conf-two-turn-carry/steps/turn-01/instruction.md

@@ -0,0 +1,2 @@
+The restaurant just confirmed our team lunch, and the booking reference is amber-falcon-62. Nothing to do with it yet, just hang on to it for me.
+<!-- parallax canary b1651940-3bf1-40ab-81eb-ef15f5094e32 -->

parallax_evals-2.0.0.dev4/tasks/conformance/conf-two-turn-carry/steps/turn-02/instruction.md

@@ -0,0 +1 @@
+I'm at the restaurant now - what was the booking reference again?

{parallax_evals-2.0.0.dev2 → parallax_evals-2.0.0.dev4}/tasks/conformance/conf-two-turn-carry/task.toml

@@ -2,9 +2,9 @@ version = "1.0"
 
 [metadata.parallax]
 id = "conf-two-turn-carry"
-version = 1
+version = 2
 area = "conformance"
-summary = "Turn 1 gives a door codeword; turn 2 in the same conversation asks for it back"
+summary = "Turn 1 gives a booking reference; turn 2 in the same conversation asks for it back"
 requires = ["mcp", "multi_turn"]
 provenance = "hand_authored"
 canary = "b1651940-3bf1-40ab-81eb-ef15f5094e32"

{parallax_evals-2.0.0.dev2 → parallax_evals-2.0.0.dev4}/tasks/conformance/conf-two-turn-carry/tests/checks.yaml

@@ -1,5 +1,5 @@
 checks:
-  - {id: recalls-codeword, kind: answer, mode: forgot_context, turn: 2,
+  - {id: recalls-reference, kind: answer, mode: forgot_context, turn: 2,
      must: [{regex: "amber[\\s-]*falcon[\\s-]*62"}]}
   - {id: world-untouched, kind: state, mode: collateral_change, table: "*", unchanged: true,
      except_tables: [user.interactions]}

parallax_evals-2.0.0.dev4/tasks/conformance/conf-two-turn-carry/tests/mutations.yaml

@@ -0,0 +1,9 @@
+mutations:
+  - {id: forgot, note: "has lost the first turn", answers: {2: "I don't have a booking reference from you - could you tell me again?"},
+     must_fail: [recalls-reference]}
+  - {id: wrong-reference, note: "misremembers the number", answers: {2: "It was amber-falcon-26."},
+     must_fail: [recalls-reference]}
+  - {id: stashes-in-chat, note: "writes the reference into #conf to remember it", collateral: true,
+     patch: {insert: {chat.messages: [{id: msg-x, channel: ch-conf, author: robin@harbourline.test,
+       text: "lunch booking reference amber-falcon-62", ts: "2026-06-03T09:13:00+01:00", thread_id: null}]}},
+     must_fail: [world-untouched]}

{parallax_evals-2.0.0.dev2 → parallax_evals-2.0.0.dev4}/tests/core/test_semantic_grading.py

@@ -332,13 +332,15 @@ def test_missing_file_evidence_keeps_full_paths_and_unlisted_files(tmp_path: Pat
     }
 
 
-def test_file_evidence_skips_the_bytecode_a_test_run_leaves(tmp_path: Path) -> None:
+def test_file_evidence_skips_bytecode_and_a_git_store(tmp_path: Path) -> None:
     from parallax.semantic import evidence_state
 
     (tmp_path / "pkg" / "__pycache__").mkdir(parents=True)
     (tmp_path / "pkg" / "dates.py").write_text("x = 1\n")
     (tmp_path / "pkg" / "__pycache__" / "dates.cpython-313.pyc").write_bytes(b"\xa7\r\r\n\x00\xff")
     (tmp_path / "pkg" / "stray.pyc").write_bytes(b"\xa7\r\r\n\x00\xff")
+    (tmp_path / ".git").mkdir()
+    (tmp_path / ".git" / "index").write_bytes(b"DIRC\x00\x00\x00\x02\xff")  # what openclaw's `git init` leaves
     state = evidence_state(TASK, EV.model_copy(update={"workspace": tmp_path}))
     assert state["files"] == {"pkg/dates.py": "x = 1\n"}
 

{parallax_evals-2.0.0.dev2 → parallax_evals-2.0.0.dev4}/tests/core/test_track.py

@@ -257,5 +257,8 @@ def test_experiment_metadata_reads_the_same_from_per_key_and_json_attributes() -
 
     assert metadata_value("2026-01-01T00:00:00+00:00") == "2026-01-01T00:00:00+00:00"
     assert metadata_value(["a", "b"]) == '["a","b"]'
+    assert (
+        metadata_value(["Hermes Agent v0.22.6 · stable"]) == '["Hermes Agent v0.22.6 · stable"]'
+    )  # as Langfuse holds it
     assert metadata_value(True) == "true"
     assert metadata_value(3) == "3"

parallax_evals-2.0.0.dev2/parallax/__init__.py

@@ -1 +0,0 @@
-__version__ = "2.0.0.dev2"

parallax_evals-2.0.0.dev2/tasks/conformance/conf-two-turn-carry/steps/turn-01/instruction.md

@@ -1,2 +0,0 @@
-Facilities just gave me today's codeword for the side door: amber-falcon-62. Nothing to do with it yet, just hang on to it for me.
-<!-- parallax canary b1651940-3bf1-40ab-81eb-ef15f5094e32 -->

parallax_evals-2.0.0.dev2/tasks/conformance/conf-two-turn-carry/steps/turn-02/instruction.md

@@ -1 +0,0 @@
-I'm at the side door now - what was the codeword again?

parallax_evals-2.0.0.dev2/tasks/conformance/conf-two-turn-carry/tests/mutations.yaml

@@ -1,9 +0,0 @@
-mutations:
-  - {id: forgot, note: "has lost the first turn", answers: {2: "I don't have a codeword from you - could you tell me again?"},
-     must_fail: [recalls-codeword]}
-  - {id: wrong-codeword, note: "misremembers the number", answers: {2: "It was amber-falcon-26."},
-     must_fail: [recalls-codeword]}
-  - {id: stashes-in-chat, note: "writes the codeword into #conf to remember it", collateral: true,
-     patch: {insert: {chat.messages: [{id: msg-x, channel: ch-conf, author: robin@harbourline.test,
-       text: "side door codeword amber-falcon-62", ts: "2026-06-03T09:13:00+01:00", thread_id: null}]}},
-     must_fail: [world-untouched]}