kitchenbench 0.3.0__py3-none-any.whl

kitchenbench/CLAUDE.md

@@ -0,0 +1,60 @@
+# `kitchenbench` package — module map
+
+A Inspect Robots plugin. Importing the package registers all 10 tasks, the mock
+`kitchen` embodiment, and the mock policies with the Inspect Robots registry; entry
+points (in `pyproject.toml`) make them discoverable without importing.
+
+## Modules
+
+| Module | Responsibility |
+|--------|----------------|
+| `distributions.py` | Pure-numpy `Distribution` types — `Uniform`, `Categorical` (samples **by index** to preserve dtype), `Normal`, `Constant`. Each `sample(rng)` returns a **builtin** scalar (JSON-native); `describe()` renders methodology notation. |
+| `instances.py` | `TaskInstance` (stochastic setup of named distributions + goal), `Realization` (one sampled environment), `Validation` (per-expert ratings + accept rule), and the `K_REALIZATIONS`/`K_INSTANCES`/`K_EXPERTS` constants. `TaskInstance.realize(seed)` samples; `setup_spec()` describes. |
+| `specs.py` | `TaskSpec` + `SPECS` — the **single source of truth**: 10 tasks, each with exactly `K_INSTANCES` (5) distribution-based `TaskInstance`s. |
+| `tasks.py` | `build_scenes` (one `Scene` per instance), `make_task` (`Epochs(count=5, reducer="mean")`), `realize_scene(scene, seed)` (the run-time seam that recovers + realizes an instance), and the 10 `@task` factories. `TASK_FACTORIES` maps key → factory. |
+| `scoring.py` | `task_success()` — success iff `termination_reason == "success"` (mock/sim privileged signal, or a real embodiment reporting operator-confirmed success) **or** an affirmative recorded operator verdict. |
+| `embodiment.py` | `KitchenEmbodiment` — dependency-free abstract bimanual mock. Models *progress toward the scene goal* (not physics), like Inspect Robots's `CubePick`. Action space is `(8,)` = two arms × `[dx,dy,dz,gripper]`. On reset of a KitchenBench scene (marked `metadata["benchmark"] == "kitchenbench"`) it calls `realize_scene` so the observed instruction reflects the per-epoch realization (a real embodiment would also arrange the sampled setup); the instruction is carried on every step observation since real VLA policies re-condition on it each `act()`. |
+| `policies.py` | `ScriptedKitchenPolicy` (reads privileged `goal_dir`, succeeds), `RandomKitchenPolicy`, `NoopKitchenPolicy`. All emit `ActionChunk`s. |
+| `__init__.py` | Re-exports the public surface: factories, mock, policies, specs, distributions, `TaskInstance`/`Realization`/`Validation`, `realize_scene`, and the `K_*` constants (fenced by `__all__`). |
+
+## Key invariants
+
+- The entry-point name, the `@task(name=...)`, and the returned `Task.name` are
+  **all identical** (`kitchenbench/<key>`). Keep them in sync.
+- The mock's hidden unit `goal_dir` + alignment threshold (0.99) is what makes the
+  scripted oracle succeed and random/no-op fail **deterministically** — don't
+  loosen the threshold or random policies may start passing and break tests.
+- Compatibility is exact action-dim equality: the mock (dim 8) pairs with the
+  mock scripted policy (dim 8); a real YAM arm (higher DoF) pairs with a real VLA
+  — there is no cross-pairing. Tasks carry no action space, so they run on both.
+
+## Methodology mapping (do not break)
+
+- **task instance → `Scene`**, **realization → epoch**, `Epochs(count=5,
+  reducer="mean")` → per-scene reduced `task_success` = **P̂[Yᵢ=1]**.
+- `metadata["task_success"]` at the eval level is the *mean of P̂ over instances* —
+  a **convenience aggregate, not** a methodology quantity (the methodology sorts P̂
+  into quantiles → pTQ / automation-halvings; out of scope here).
+- Scene `metadata` is **strictly JSON-native** (distributions stored as
+  `setup_spec()` strings; the live `TaskInstance` is recovered by looking up
+  `metadata["instance_id"]` within `SPEC_BY_KEY[task].instances`, never stored —
+  id-based so replays of old logs fail loudly instead of silently realizing a
+  reordered instance). `metadata["benchmark"] == "kitchenbench"` is the marker
+  the mock embodiment keys on to decide a scene is realize-able.
+- `Categorical.sample` samples **by index** (never `rng.choice(values)`, which
+  coerces numeric tuples to strings). `derive_seed` comes from `inspect_robots.rollout`
+  (not re-exported at the top level).
+
+## Adding a task
+
+1. Author 5 `TaskInstance`s (distribution-based) and a `TaskSpec(...)`; append to
+   `SPECS` in `specs.py`. Every `{placeholder}` in a goal must be in
+   `language_vars` (and in `setup`).
+2. Add an `@task("kitchenbench/<key>")` factory in `tasks.py`, add it to
+   `TASK_FACTORIES` + the `__init__` re-export.
+3. Add the entry point to `pyproject.toml` under
+   `[project.entry-points."inspect_robots.tasks"]`.
+4. Add the task key to KitchenBench's entry in WorldEvals' `catalog.py`.
+
+`tests/test_specs.py` / `test_realize_all.py` parametrize over `SPECS` and realize
+all 50 instances, so a new spec is exercised automatically — keep coverage at 100%.

kitchenbench/__init__.py

@@ -0,0 +1,91 @@
+"""KitchenBench — a bimanual kitchen-manipulation benchmark for VLA models.
+
+Built on `Inspect Robots <https://github.com/robocurve/inspect-robots>`_; the first member of
+`WorldEvals <https://github.com/robocurve/worldevals>`_. Importing this package
+registers all 10 tasks with the Inspect Robots registry (via the ``@task``
+decorator). The mock :class:`~kitchenbench.embodiment.KitchenEmbodiment` and the
+mock policies are not registered on import — Inspect Robots resolves them through
+this package's entry points when it is installed (as it does the tasks, so no
+import is needed for ``inspect-robots list``).
+"""
+
+from __future__ import annotations
+
+from kitchenbench.distributions import (
+    Categorical,
+    Constant,
+    Distribution,
+    Normal,
+    Uniform,
+)
+from kitchenbench.embodiment import KitchenEmbodiment
+from kitchenbench.instances import (
+    K_EXPERTS,
+    K_INSTANCES,
+    K_REALIZATIONS,
+    Realization,
+    TaskInstance,
+    Validation,
+)
+from kitchenbench.policies import (
+    NoopKitchenPolicy,
+    RandomKitchenPolicy,
+    ScriptedKitchenPolicy,
+)
+from kitchenbench.scoring import task_success
+from kitchenbench.specs import SPEC_BY_KEY, SPECS, TaskSpec
+from kitchenbench.tasks import (
+    TASK_FACTORIES,
+    build_scenes,
+    fold_cloth,
+    handoff,
+    make_task,
+    open_container,
+    place_cutlery,
+    place_in_rack,
+    pour_pasta,
+    realize_scene,
+    scoop_pasta,
+    seal_container,
+    sort_cutlery,
+    stack,
+)
+
+__version__ = "0.3.0"
+
+__all__ = [
+    "K_EXPERTS",
+    "K_INSTANCES",
+    "K_REALIZATIONS",
+    "SPECS",
+    "SPEC_BY_KEY",
+    "TASK_FACTORIES",
+    "Categorical",
+    "Constant",
+    "Distribution",
+    "KitchenEmbodiment",
+    "NoopKitchenPolicy",
+    "Normal",
+    "RandomKitchenPolicy",
+    "Realization",
+    "ScriptedKitchenPolicy",
+    "TaskInstance",
+    "TaskSpec",
+    "Uniform",
+    "Validation",
+    "__version__",
+    "build_scenes",
+    "fold_cloth",
+    "handoff",
+    "make_task",
+    "open_container",
+    "place_cutlery",
+    "place_in_rack",
+    "pour_pasta",
+    "realize_scene",
+    "scoop_pasta",
+    "seal_container",
+    "sort_cutlery",
+    "stack",
+    "task_success",
+]

kitchenbench/distributions.py

@@ -0,0 +1,123 @@
+"""Probability distributions for KitchenBench task-instance setups.
+
+A task instance (see :mod:`kitchenbench.instances`) specifies a *stochastic*
+environment: each setup variable is a :class:`Distribution`. A *realization*
+samples every variable to produce one concrete environment. These types are pure
+NumPy with no Inspect Robots dependency.
+
+Distributions mirror the methodology's notation
+(:doc:`reference/physical-automation-methodology-docs`):
+
+- :class:`Uniform` — ``Uniform[a, b]`` (continuous)
+- :class:`Categorical` — ``Categorical({…})`` (uniform or weighted over a finite set)
+- :class:`Normal` — ``N(μ, σ²)`` (Gaussian; a 2-D jitter is two of these)
+- :class:`Constant` — a fixed (non-random) part of a setup
+
+Every ``sample`` returns a **builtin** ``float``/``int``/``str`` (never a NumPy
+scalar) so realizations are JSON-native and mypy-strict clean.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Protocol, runtime_checkable
+
+import numpy as np
+
+Scalar = float | int | str
+
+
+@runtime_checkable
+class Distribution(Protocol):
+    """A sampleable, self-describing setup variable."""
+
+    def sample(self, rng: np.random.Generator) -> Scalar: ...
+
+    def describe(self) -> str: ...
+
+
+@dataclass(frozen=True)
+class Uniform:
+    """Continuous uniform on ``[low, high]``.
+
+    Sampling follows numpy's half-open convention (``high`` itself has probability
+    zero); ``describe()`` keeps the methodology's closed-interval notation.
+    """
+
+    low: float
+    high: float
+
+    def sample(self, rng: np.random.Generator) -> float:
+        return float(rng.uniform(self.low, self.high))
+
+    def describe(self) -> str:
+        return f"Uniform[{_num(self.low)}, {_num(self.high)}]"
+
+
+@dataclass(frozen=True)
+class Categorical:
+    """Uniform (or weighted) choice over a finite set of values.
+
+    Samples **by index** so the value's type is preserved — ``rng.choice`` over a
+    numeric/mixed tuple would coerce to a string array.
+    """
+
+    values: tuple[Scalar, ...]
+    weights: tuple[float, ...] | None = None
+
+    def __post_init__(self) -> None:
+        if not self.values:
+            raise ValueError("Categorical needs at least one value")
+        if self.weights is not None:
+            if len(self.weights) != len(self.values):
+                raise ValueError("weights must match values in length")
+            if any(w < 0 for w in self.weights):
+                raise ValueError("weights must be non-negative")
+            if sum(self.weights) <= 0:
+                raise ValueError("weights must sum to a positive value")
+
+    def sample(self, rng: np.random.Generator) -> Scalar:
+        probs = None if self.weights is None else np.asarray(self.weights, dtype=np.float64)
+        if probs is not None:
+            probs = probs / probs.sum()
+        idx = int(rng.choice(len(self.values), p=probs))
+        return self.values[idx]
+
+    def describe(self) -> str:
+        body = ", ".join(_num(v) if isinstance(v, int | float) else str(v) for v in self.values)
+        if self.weights is None:
+            return f"Categorical({{{body}}})"
+        wbody = ", ".join(_num(w) for w in self.weights)
+        return f"Categorical({{{body}}}; weights=[{wbody}])"
+
+
+@dataclass(frozen=True)
+class Normal:
+    """Gaussian ``N(mean, std²)``."""
+
+    mean: float
+    std: float
+
+    def sample(self, rng: np.random.Generator) -> float:
+        return float(rng.normal(self.mean, self.std))
+
+    def describe(self) -> str:
+        return f"N({_num(self.mean)}, {_num(self.std)}²)"
+
+
+@dataclass(frozen=True)
+class Constant:
+    """A fixed, non-random setup value."""
+
+    value: Scalar
+
+    def sample(self, rng: np.random.Generator) -> Scalar:
+        return self.value
+
+    def describe(self) -> str:
+        return repr(self.value)
+
+
+def _num(x: float) -> str:
+    """Render a number without a trailing ``.0`` for whole values."""
+    return str(int(x)) if isinstance(x, float) and x.is_integer() else str(x)

kitchenbench/embodiment.py

@@ -0,0 +1,138 @@
+"""KitchenEmbodiment — a dependency-free abstract bimanual mock kitchen.
+
+It does **not** simulate physics; like Inspect Robots's ``CubePick`` it models *progress
+toward the scene goal* so the whole pipeline (scenes → chunked rollout → score →
+log) runs in CI with no hardware. The value of KitchenBench is the task
+definitions; this mock exists to exercise them and to serve as the template for a
+real YAM-arms embodiment.
+
+Each reset draws a hidden unit "goal direction" in the 6-D dual-arm command
+subspace. Progress advances only when a normalized action aligns with that
+direction (cosine ≥ ``align_threshold``) — so the scripted oracle (which reads the
+privileged ``goal_dir``) succeeds, while random/no-op policies do not.
+
+The action space is bimanual on purpose: ``(8,)`` = two arms x ``[dx, dy, dz,
+gripper]``. A real YAM embodiment is a higher-DoF analog (compatibility checking
+matches exact dims, so the real arm pairs with a real VLA, not with this mock).
+"""
+
+from __future__ import annotations
+
+import numpy as np
+from inspect_robots import (
+    Action,
+    ActionSemantics,
+    Box,
+    CameraSpec,
+    EmbodimentInfo,
+    Observation,
+    ObservationSpace,
+    Scene,
+    StepResult,
+)
+from inspect_robots.embodiment import PRIVILEGED_SUCCESS, RENDERABLE, SEEDABLE
+
+from kitchenbench.tasks import realize_scene
+
+_IMG = 24
+
+_ACTION_SPACE = Box(
+    shape=(8,),
+    low=np.full(8, -1.0),
+    high=np.full(8, 1.0),
+    semantics=ActionSemantics(control_mode="eef_delta_pos", gripper="continuous", frame="world"),
+)
+
+
+class KitchenEmbodiment:
+    """Abstract bimanual mock kitchen world."""
+
+    def __init__(
+        self,
+        *,
+        step_size: float = 0.25,
+        align_threshold: float = 0.99,
+        goal_threshold: float = 1.0,
+    ):
+        self.step_size = step_size
+        self.align_threshold = align_threshold
+        self.goal_threshold = goal_threshold
+        self.num_steps = 0
+
+        self._progress = 0.0
+        self._goal = np.zeros(6, dtype=np.float64)
+        self._last = np.zeros(8, dtype=np.float64)
+        self._instruction: str | None = None
+        self._rng = np.random.RandomState(0)
+
+        self.info = EmbodimentInfo(
+            name="kitchen",
+            action_space=_ACTION_SPACE,
+            observation_space=ObservationSpace(
+                cameras=(CameraSpec(name="overhead", height=_IMG, width=_IMG, channels=3),),
+                state_keys=frozenset({"progress", "goal_dir", "left_eef", "right_eef"}),
+            ),
+            control_hz=10.0,
+            is_simulated=True,
+            capabilities=frozenset({SEEDABLE, RENDERABLE, PRIVILEGED_SUCCESS}),
+        )
+
+    def reset(self, scene: Scene, *, seed: int | None = None) -> Observation:
+        self._rng = np.random.RandomState(seed if seed is not None else 0)
+        goal = self._rng.normal(size=6)
+        self._goal = goal / (np.linalg.norm(goal) or 1.0)
+        self._progress = 0.0
+        self._last = np.zeros(8, dtype=np.float64)
+        self.num_steps = 0
+        # For a KitchenBench scene (the "benchmark" marker keeps scenes from other
+        # plugins out), realize the task instance for this seed so the observed
+        # instruction reflects the per-epoch realization (a real embodiment /
+        # operator would also arrange the sampled setup). The realization rng is
+        # independent of the goal_dir rng above. Bare scenes fall back unchanged.
+        # The instruction is carried on every subsequent observation — real VLA
+        # policies re-condition on it at each act() call.
+        self._instruction = scene.instruction
+        if scene.metadata.get("benchmark") == "kitchenbench":
+            self._instruction = realize_scene(scene, seed).instruction
+        return self._observe()
+
+    def step(self, action: Action) -> StepResult:
+        self.num_steps += 1
+        data = np.clip(np.asarray(action.data, dtype=np.float64), -1.0, 1.0)
+        self._last = data
+        arm = data[:6]
+        norm = float(np.linalg.norm(arm))
+        if norm > 0.0:
+            cosine = float(np.dot(arm / norm, self._goal))
+            if cosine >= self.align_threshold:
+                self._progress = min(self.goal_threshold, self._progress + self.step_size)
+        success = self._progress >= self.goal_threshold
+        return StepResult(
+            observation=self._observe(),
+            reward=self._progress - 1.0,
+            terminated=success,
+            termination_reason="success" if success else None,
+            truncated=False,
+            info={"success": success, "progress": self._progress},
+        )
+
+    def close(self) -> None:
+        return None
+
+    def _observe(self) -> Observation:
+        return Observation(
+            images={"overhead": self._render()},
+            state={
+                "progress": np.array([self._progress], dtype=np.float64),
+                "goal_dir": self._goal.copy(),
+                "left_eef": self._last[:3].copy(),
+                "right_eef": self._last[3:6].copy(),
+            },
+            instruction=self._instruction,
+        )
+
+    def _render(self) -> np.ndarray:
+        img = np.zeros((_IMG, _IMG, 3), dtype=np.uint8)
+        filled = round(self._progress * _IMG)
+        img[_IMG - filled :, :, 1] = 200  # a green progress bar rising from the bottom
+        return img

kitchenbench/instances.py

@@ -0,0 +1,87 @@
+"""Task instances — the unit of the physical-automation methodology.
+
+A **task instance** is a self-contained scenario: a *stochastic environment
+setup* (named :class:`~kitchenbench.distributions.Distribution` variables) plus a
+*goal* (a natural-language success criterion that may reference sampled
+variables). A **realization** samples every variable to produce one concrete
+environment; the methodology runs ``K_REALIZATIONS`` realizations of each of
+``K_INSTANCES`` instances per task and estimates the instance success probability
+as the mean binary outcome.
+
+This module is pure (no Inspect Robots import); :mod:`kitchenbench.tasks` maps instances
+onto Inspect Robots ``Scene``/``Epochs``.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any
+
+import numpy as np
+
+from kitchenbench.distributions import Distribution, Scalar
+
+#: Methodology recommendations (see the reference docs).
+K_REALIZATIONS = 5  # independent rollouts per instance -> success probability
+K_INSTANCES = 5  # validated instances per task
+K_EXPERTS = 3  # validators per instance
+
+
+@dataclass(frozen=True)
+class Validation:
+    """Per-expert validation ratings carried with an instance.
+
+    Empty until the human commissioning pipeline is run; :attr:`validated` follows
+    the methodology accept rule (``K_EXPERTS`` ratings, all >= 4 on both axes).
+    """
+
+    representativeness: tuple[int, ...] = ()
+    quality: tuple[int, ...] = ()
+    difficulty: tuple[int, ...] = ()  # optional ("nice to have" in the methodology)
+    source: str = "opus-draft"  # provenance: "opus-draft" (AI) vs "human"
+
+    @property
+    def validated(self) -> bool:
+        return (
+            len(self.representativeness) >= K_EXPERTS
+            and len(self.quality) >= K_EXPERTS
+            and all(r >= 4 for r in self.representativeness)
+            and all(q >= 4 for q in self.quality)
+        )
+
+
+@dataclass(frozen=True)
+class Realization:
+    """One concrete environment sampled from a :class:`TaskInstance`."""
+
+    seed: int
+    values: dict[str, Scalar]
+    instruction: str
+    setup_lines: tuple[str, ...]
+
+
+@dataclass(frozen=True)
+class TaskInstance:
+    """A stochastic scenario: named distributions + a goal."""
+
+    instance_id: str
+    goal: str  # template; ``{var}`` placeholders must be in ``language_vars``
+    setup: dict[str, Distribution]
+    target_kind: str
+    language_vars: tuple[str, ...] = ()
+    static: dict[str, Any] = field(default_factory=dict)
+    validation: Validation = field(default_factory=Validation)
+
+    def realize(self, seed: int) -> Realization:
+        """Sample every setup variable (sorted-key order) for one concrete environment."""
+        rng = np.random.default_rng(seed)
+        values: dict[str, Scalar] = {key: self.setup[key].sample(rng) for key in sorted(self.setup)}
+        instruction = self.goal.format(**{k: values[k] for k in self.language_vars})
+        setup_lines = tuple(f"{key} = {values[key]}" for key in sorted(self.setup))
+        return Realization(
+            seed=seed, values=values, instruction=instruction, setup_lines=setup_lines
+        )
+
+    def setup_spec(self) -> dict[str, str]:
+        """A JSON-native description of the setup (``{var: distribution.describe()}``)."""
+        return {key: self.setup[key].describe() for key in sorted(self.setup)}

kitchenbench/policies.py

@@ -0,0 +1,96 @@
+"""Mock policies for the KitchenEmbodiment.
+
+- :class:`ScriptedKitchenPolicy` — reads the privileged ``goal_dir`` and drives
+  straight to success (the CI oracle / template for a real VLA policy).
+- :class:`RandomKitchenPolicy` — random actions; effectively never aligns, so it
+  fails. Deterministic given the construction seed.
+- :class:`NoopKitchenPolicy` — zero actions; never moves.
+
+All emit :class:`~inspect_robots.ActionChunk`\\ s with ``H > 1`` to exercise open-loop
+chunk execution.
+"""
+
+from __future__ import annotations
+
+import numpy as np
+from inspect_robots import (
+    Action,
+    ActionChunk,
+    ActionSemantics,
+    Box,
+    Observation,
+    ObservationSpace,
+    PolicyConfig,
+    PolicyInfo,
+    Scene,
+)
+
+_ACTION_SPACE = Box(
+    shape=(8,),
+    semantics=ActionSemantics(control_mode="eef_delta_pos", gripper="continuous", frame="world"),
+)
+# The scripted oracle needs the privileged goal direction.
+_SCRIPTED_OBS = ObservationSpace(state_keys=frozenset({"goal_dir"}))
+
+
+class ScriptedKitchenPolicy:
+    """Drive both arms along the privileged ``goal_dir`` (deterministic success)."""
+
+    def __init__(self, *, chunk_size: int = 4):
+        self.chunk_size = chunk_size
+        self.num_inferences = 0
+        self.info = PolicyInfo(
+            name="kitchen_scripted", action_space=_ACTION_SPACE, observation_space=_SCRIPTED_OBS
+        )
+        self.config = PolicyConfig(action_horizon=chunk_size)
+
+    def reset(self, scene: Scene) -> None:
+        self.num_inferences = 0
+
+    def act(self, observation: Observation) -> ActionChunk:
+        self.num_inferences += 1
+        goal = np.asarray(observation.state["goal_dir"], dtype=np.float64)
+        data = np.concatenate([goal, np.array([1.0, 1.0])])  # arms aligned; grippers closed
+        return ActionChunk(actions=[Action(data=data.copy()) for _ in range(self.chunk_size)])
+
+
+class RandomKitchenPolicy:
+    """Emit random actions. Deterministic given the construction seed."""
+
+    def __init__(self, *, chunk_size: int = 4, seed: int = 0):
+        self.chunk_size = chunk_size
+        self.num_inferences = 0
+        self._base_seed = seed
+        self._reset_count = 0
+        self._rng = np.random.RandomState(seed)
+        self.info = PolicyInfo(name="kitchen_random", action_space=_ACTION_SPACE)
+        self.config = PolicyConfig(action_horizon=chunk_size)
+
+    def reset(self, scene: Scene) -> None:
+        self._rng = np.random.RandomState(self._base_seed + self._reset_count)
+        self._reset_count += 1
+        self.num_inferences = 0
+
+    def act(self, observation: Observation) -> ActionChunk:
+        self.num_inferences += 1
+        actions = [
+            Action(data=self._rng.uniform(-1.0, 1.0, size=8)) for _ in range(self.chunk_size)
+        ]
+        return ActionChunk(actions=actions)
+
+
+class NoopKitchenPolicy:
+    """Emit zero actions; never moves."""
+
+    def __init__(self, *, chunk_size: int = 1):
+        self.chunk_size = chunk_size
+        self.num_inferences = 0
+        self.info = PolicyInfo(name="kitchen_noop", action_space=_ACTION_SPACE)
+        self.config = PolicyConfig(action_horizon=chunk_size)
+
+    def reset(self, scene: Scene) -> None:
+        self.num_inferences = 0
+
+    def act(self, observation: Observation) -> ActionChunk:
+        self.num_inferences += 1
+        return ActionChunk(actions=[Action(data=np.zeros(8)) for _ in range(self.chunk_size)])

kitchenbench/scoring.py

@@ -0,0 +1,36 @@
+"""KitchenBench scoring.
+
+``task_success`` works both in the dependency-free mock (which reports privileged
+success via the termination reason) and on real hardware: there is no success
+oracle on a real kitchen, so the real embodiment turns the operator's
+confirmation into either a ``"success"`` termination reason or a recorded
+operator verdict — ``task_success`` accepts either.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+from inspect_robots import Score, Scorer, Target, TrialRecord
+
+# Affirmative operator verdicts (case-insensitive), for real-world runs. Mirrors
+# inspect_robots.scorer._OPERATOR_SUCCESS (private there) — keep the two in sync.
+_AFFIRMATIVE = frozenset({"success", "pass", "yes", "y", "1", "true"})
+
+
+@dataclass(frozen=True)
+class _TaskSuccess:
+    name: str = "task_success"
+
+    def __call__(self, record: TrialRecord, target: Target | None) -> Score:
+        if record.termination_reason == "success":
+            return Score(value=True, explanation="terminated with success")
+        verdict = record.operator_judgement
+        if verdict is not None and verdict.strip().lower() in _AFFIRMATIVE:
+            return Score(value=True, explanation=f"operator verdict: {verdict!r}")
+        return Score(value=False, explanation="no success signal")
+
+
+def task_success() -> Scorer:
+    """Success iff the trial terminated successfully or an operator confirmed it."""
+    return _TaskSuccess()