eclise-phyworld 0.2.0__tar.gz

eclise_phyworld-0.2.0/PKG-INFO

@@ -0,0 +1,16 @@
+Metadata-Version: 2.4
+Name: eclise-phyworld
+Version: 0.2.0
+Requires-Python: >=3.9
+Requires-Dist: numpy<3,>=1.25
+Requires-Dist: box2d-py>=2.3.5
+Requires-Dist: h5py>=3.10
+Requires-Dist: imageio[ffmpeg]>=2.36
+Requires-Dist: av>=13.1
+Requires-Dist: opencv-python-headless>=4.10
+Requires-Dist: huggingface_hub<1.0,>=0.34.0
+Requires-Dist: draccus
+Requires-Dist: tqdm
+Requires-Dist: matplotlib>=3.8
+Provides-Extra: test
+Requires-Dist: pytest~=8.4.0; extra == "test"

eclise_phyworld-0.2.0/README.md

@@ -0,0 +1,267 @@
+# phyworld
+
+Synthetic physics video, reorganized from the code for [*How Far is Video
+Generation from World Model: A Physical Law
+Perspective*](https://phyworld.github.io/). Writes clip shards in the shared
+[`eclise.clips`](../../data/clips) format, so `video_world_models`
+consumes this sim's output through the same reader it will use for real video.
+
+## Families
+
+| Family | Physics | Objects | Needs |
+|---|---|---|---|
+| `uniform_motion` | constant velocity; position linear in time | 1 | Box2D |
+| `parabola` | constant gravity; position quadratic in time | 1 | Box2D |
+| `collision` | elastic 1-D collision, mass from radius | 2 | Box2D |
+| `combinatorial` | PHYRE task templates | varies | PHYRE (see below) |
+
+The first three need nothing beyond the main env. `collision` is the interesting one:
+`density=1`, so mass follows from area and the radius ratio *is* the mass ratio —
+a model has to infer mass from apparent size to predict the outcome.
+
+## Setup
+
+This project has **two** conda envs. PHYRE's only prebuilt wheel is `cp39`, so it
+cannot live in the main 3.10 env; the second env exists solely to host it. Both
+install the same `eclise.clips` and `eclise.phyworld`, so there is one `ShardWriter`
+and one CLI either way.
+
+| Env | Python | Spec | For |
+|---|---|---|---|
+| `.conda` | 3.10 | `environment.yml` | `uniform_motion`, `parabola`, `collision`; downloading; verifying |
+| `.conda-phyre` | 3.9 | `environment-phyre.yml` | `combinatorial` only (adds `phyre==0.2.2`) |
+
+### 1. Main env
+
+```bash
+cd data/phyworld
+SCRATCH="${SCRATCH:-/pa/home/scratch_fcm/$USER}"
+mkdir -p "$SCRATCH/eclise/data/phyworld"/{.conda,data,outputs}
+conda env create -f environment.yml -p "$SCRATCH/eclise/data/phyworld/.conda"
+for d in .conda data outputs; do
+  ln -sfn "$SCRATCH/eclise/data/phyworld/$d" ./$d
+done
+conda activate ./.conda
+```
+
+`data/` and `outputs/` must be scratch symlinks: `/pa/home` has a 100GB cap and
+the published 3M splits are tens of GB each.
+
+### 2. PHYRE env, only for the combinatorial family
+
+Run this after step 1, which is what creates the shared `data/` and `outputs/`
+symlinks. The script places the env on scratch, symlinks it in, and verifies the
+simulator with a real rollout — an import alone passes on a broken extension.
+
+```bash
+bash scripts/setup_phyre.sh            # create .conda-phyre, then verify
+bash scripts/setup_phyre.sh --verify   # re-verify an existing env, create nothing
+conda activate ./.conda-phyre
+```
+
+Skip this entirely if you only need the three Box2D families, or if you plan to
+*read* the published combinatorial HDF5 — that needs no PHYRE at all. See
+[The combinatorial family](#the-combinatorial-family) for why the wheel rather
+than a source build.
+
+## Generating
+
+```bash
+# smoke test
+python scripts/generate.py --family collision --grid in_dist_l0 \
+    --experiment smoke --limit 32 --num_workers 1
+
+# the ~30K training set
+python scripts/generate.py --family uniform_motion --grid in_dist_l0 \
+    --experiment train_30k --num_workers 32
+
+# an OOD evaluation regime
+python scripts/generate.py --family parabola --grid ood_cross_small_fast \
+    --experiment eval_cross
+
+# what grids does a family offer?
+python scripts/generate.py --family collision --list_grids True
+```
+
+Shards go to `data/<family>/<experiment>/`, the run report to
+`outputs/<experiment>/generate.json`. Verification runs by default; pass
+`--no_verify True` to skip it. On a cluster, `sbatch scripts/generate.sbatch` takes
+the same flags.
+
+Boolean options need an explicit value — draccus parses them as typed dataclass
+fields rather than argparse `store_true` flags, so a bare `--no_verify` consumes
+the next token instead of toggling.
+
+### Grids
+
+Parameter sweeps are declarative data in `grids.py`, organized around which axes
+leave the training range — because the question the paper asks is whether a model
+that fits the training distribution generalizes outside it.
+
+- `in_dist_l0` / `l1` / `l2` — training distribution at ~30K / 300K / 3M scenes.
+- `ood_low_*` / `ood_high_*` — one or both axes below / above the training range.
+  `ood_low_velocity` reaches zero: a stationary ball is the easiest possible
+  dynamics, and a model that cannot do it has learned drift rather than motion.
+- `ood_cross_*` — radius from one extreme, velocity from the other. Both marginals
+  appear in some other regime, so a failure here is about the *pairing*. That's the
+  distinction between "never saw this value" and "never saw this combination", and
+  it's the one a compositional generalization claim rests on.
+
+## Downloading instead
+
+The published data is often the better choice: it's what the paper's numbers were
+computed on, so a model trained on it is directly comparable.
+
+```bash
+python scripts/download_phyworld.py --list
+python scripts/download_phyworld.py --family uniform_motion --size 30K --eval
+```
+
+Those files carry **no metadata at all** — no frame rate, no world scale, no record
+of which generator made them. `eclise.clips.readers.phyworld_hdf5` supplies it from
+a per-family table, which is why reading them needs the family named:
+
+```python
+from eclise.clips import ClipDataset
+from eclise.clips.readers import phyworld_hdf5
+
+shards = phyworld_hdf5.open_shard_set("collision_30K.hdf5", family="collision")
+data = ClipDataset(shards, velocity_object=0)
+```
+
+## Verifying
+
+```bash
+python scripts/verify_dataset.py --shard_dir data/collision/train_30k --plot
+```
+
+Recovers object positions from the rendered (and mp4-round-tripped) frames and
+compares them to the stored labels. This catches the bug class that produces video
+which looks fine and labels which are silently mirrored or rescaled — a y-axis flip
+above all, since phyworld's stored positions have y increasing *upward* while image
+rows increase downward. The same parser scores a *model's* generated frames, which
+is how phyworld measures physical plausibility without a learned metric.
+
+## The combinatorial family
+
+PHYRE is a C++ extension whose **only prebuilt wheel is cp39**, so it cannot
+live in the main Python 3.10 env. Building from source is also blocked:
+`thrift-cpp=0.11.0` pins `openssl 1.0.*`, while every conda-forge Python ≥3.8
+links openssl 1.1 or 3.x. The wheel sidesteps that — its extension links only
+libstdc++/libm/libgcc/libc, with thrift/boost/openssl statically baked in —
+which is why this family has a second, small env at Python 3.9 rather than a
+toolchain fight.
+
+Create that env with [Setup step 2](#2-phyre-env-only-for-the-combinatorial-family),
+then generate from it:
+
+```bash
+conda activate ./.conda-phyre
+python scripts/generate.py --family combinatorial --experiment combi_smoke \
+    --limit 8 --num_workers 1 \
+    --combinatorial.task_source upstream \
+    --combinatorial.templates '[0]' \
+    --combinatorial.tasks_per_template 2 \
+    --combinatorial.trials_per_task 2
+```
+
+On a cluster: `CONDA_ENV=.conda-phyre sbatch scripts/generate.sbatch --family combinatorial ...`.
+
+The three Box2D families stay in `.conda`, and the *published* combinatorial
+HDF5 reads with no PHYRE at all.
+
+**Task sources.** `--combinatorial.task_source upstream` (256px) uses the
+templates the wheel already ships. `--combinatorial.task_source fork` (default,
+512px) stages phyworld's precompiled `tasks10.bin.lzma` (templates
+`10000`–`10069`) beside them — a one-time download, no C++ rebuild. Shards
+record `phyre_variant` and `terminates_on_solve`: the upstream wheel still
+stops a rollout when the task is solved, which biases the family toward
+unsolved trials at short `clip_len`. Pass
+`--combinatorial.full_length_rollouts True` for the unbiased (slower) path.
+
+**Its `frame_dt` is unverified.** Upstream computes `STRIDE = int(100/fps)`,
+implying PHYRE steps at 100Hz, while PHYRE documents 60Hz — a factor of 5/3 on
+every velocity derived from `object_features`. Shards record
+`frame_dt_verified=false` until measured; set `--combinatorial.frame_dt` once you
+have. Relative comparisons within the family are unaffected.
+
+## Consuming from `video_world_models`
+
+```python
+from video_world_models.training import shard_data
+
+bundle = shard_data.build_shard_clip_dataset(space, "data/phyworld/data/collision/train")
+bundle.dataset[0]   # (standardized latents [T', h, w, z], velocity [2], index)
+```
+
+`source_fps` is read off the shard (`1 / frame_dt` = 10.0 Hz for the Box2D
+families) rather than guessed. Not `5`, which is what upstream's generators pass to
+imageio for side-channel debug videos, and not the mp4 container header either.
+Feeding the wrong rate to the FPS-aware encoder resamples by the wrong factor and
+silently invalidates the velocity labels.
+
+## Layout
+
+```
+src/eclise/phyworld/
+├── config.py                # draccus dataclasses; every knob, no module globals
+├── grids.py                 # parameter sweeps as data
+├── render.py                # one rasterizer, and the y-flip boundary
+├── generate.py              # parallel driver -> ShardWriter
+├── combinatorial_driver.py  # the PHYRE family's driver (different unit of work)
+├── scenes/                  # one module per family
+└── analysis/                # state recovery, metrics, verification
+scripts/
+├── generate.py  generate.sbatch
+├── download_phyworld.py
+├── verify_dataset.py
+└── setup_phyre.sh           # creates .conda-phyre (no C++ build)
+environment.yml              # main env, python 3.10
+environment-phyre.yml        # PHYRE env, python 3.9 + phyre==0.2.2
+```
+
+## What changed from upstream
+
+Upstream is four generator scripts plus a PHYRE fork, and the reorganization was
+mostly deleting duplication and making implicit things explicit:
+
+- **`frame_dt` is recorded.** Upstream writes clips with no `fps` at all while the
+  real spacing is `timestep * stride = 0.1s`, set in code a reader never sees. It's
+  now required metadata, and `source_fps` derives from it.
+- **One rasterizer.** Upstream carries four copies of `vectorized_draw_balls`
+  differing only in the shape's distance function.
+- **One grid builder.** Upstream has six near-identical sweep functions per
+  generator, each hardcoding bounds as locals (`MIN_OOD_V = 0.0` reassigned twice
+  in one function) and reading training bounds from globals set in `__main__`.
+- **Config travels with the work.** Upstream's `ProcessPoolExecutor` workers read
+  `args`, `WORLD_SCALE`, `STRIDE`, `DIRECTION` from module globals, which resolves
+  only because `fork` copies parent memory. Under `spawn` every one is undefined.
+- **Rejections are counted.** Upstream drops infeasible scenes with a bare
+  `continue`, so a run that asked for 30,000 clips and produced 28,412 gives no
+  hint of the gap. The shortfall is now tallied by reason.
+- **Independent per-shard seeds.** Upstream calls `set_seeds(args.seed)` at the top
+  of every worker with the *same* seed, so every chunk draws the identical sequence
+  of random placements.
+- **Feasibility is computed, not sampled.** The collision family's initial gap
+  follows from closing speed and window length; upstream retries placement 50 times
+  and its `num_try >= 10` check marks scenes failed that had already succeeded.
+- **No merge pass.** Upstream has four inlined copies of `merge_files`, each
+  sorting filenames differently. `ShardSet` reads a directory as one sequence.
+- **Pixel centres, not corners.** Upstream's `linspace(0, 1, size)` biases every
+  shape by half a pixel.
+
+## Tests
+
+```bash
+cd data/phyworld
+./.conda/bin/python -m pytest          # main env
+./.conda-phyre/bin/python -m pytest    # PHYRE env, if created
+```
+
+The suite runs in either env. `test_scenes.py` needs Box2D; `test_generate.py`
+additionally needs `eclise.clips` and OpenCV; the rollout tests in
+`test_phyre_compat.py` need `phyre` and so only run in `.conda-phyre`. All skip
+cleanly when their dependency is absent, which is why the two runs report
+different counts rather than failing. `test_generate.py` includes a guard on the
+guard: it mirrors the labels and asserts the verification *fails*, so the y-flip
+check is known to be testing the convention rather than passing vacuously.

eclise_phyworld-0.2.0/pyproject.toml

@@ -0,0 +1,27 @@
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "eclise-phyworld"
+version = "0.2.0"
+# 3.9 so this package installs into the PHYRE env as well as the main one:
+# PHYRE's only prebuilt wheel is cp39. See README's "The combinatorial family".
+# Consequence for annotations: any dataclass draccus introspects must spell
+# optionals `Optional[X]`, since 3.9 cannot evaluate `X | None` at runtime.
+requires-python = ">=3.9"
+dynamic = ["dependencies"]
+
+[project.optional-dependencies]
+test = ["pytest~=8.4.0"]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["eclise.phyworld*"]
+
+[tool.setuptools.dynamic]
+dependencies = {file = ["requirements.txt"]}
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["src"]

eclise_phyworld-0.2.0/requirements.txt

@@ -0,0 +1,57 @@
+# =============================================================================
+# eclise.phyworld runtime dependencies (synthetic physics video generation)
+#
+# This env owns simulators and writes clip shards. It never trains anything and
+# never loads a video model, so it is deliberately torch-free: upstream's
+# generators import torch only to call `torch.manual_seed`, which numpy's
+# Generator covers. That keeps the env small and lets generation run on
+# CPU-only nodes.
+#
+# Python 3.10 in the main env; the combinatorial family additionally installs
+# into a Python 3.9 env (environment-phyre.yml) because PHYRE's only prebuilt
+# wheel is cp39. See README's "The combinatorial family".
+# =============================================================================
+
+# >=1.25 for Generator.spawn(), which is how each shard gets an independent
+# random stream (see generate.py's seeding note).
+numpy>=1.25,<3
+
+# --- Rigid-body physics ------------------------------------------------------
+# The `Box2D` import name is provided by box2d-py (the maintained pybox2d fork
+# with wheels). Upstream phyworld does `from Box2D import *`; we import the
+# handful of names we use explicitly instead.
+box2d-py>=2.3.5
+
+# --- Shard IO ----------------------------------------------------------------
+# `eclise.clips` (the shared clip contract) is installed editable from
+# environment.yml's pip section (`-e ../clips`); editable path
+# deps are not valid PEP 508 specifiers so they cannot live here. It brings h5py,
+# imageio and av transitively; they are named here too because this project's own
+# scripts use them directly.
+h5py>=3.10
+imageio[ffmpeg]>=2.36
+av>=13.1
+
+# --- State recovery from rendered frames -------------------------------------
+# Hough circle detection for the verification/eval path. Headless: these envs
+# run on compute nodes with no display, and plain opencv-python pulls in GUI
+# libraries that then fail to load.
+opencv-python-headless>=4.10
+
+# --- Downloading the published datasets --------------------------------------
+huggingface_hub>=0.34.0,<1.0
+
+# --- CLI + progress ----------------------------------------------------------
+draccus            # dataclass CLI config (as in models/robotics/libero, hierarchical_robotics)
+tqdm
+
+# --- Figures for the verification report -------------------------------------
+matplotlib>=3.8
+
+# NOTE: PHYRE (the combinatorial family's simulator) is NOT a pip requirement of
+# this file. Its only prebuilt wheel is cp39, so it lives in a second conda env
+# created by `scripts/setup_phyre.sh` (see environment-phyre.yml). That env
+# installs the same editable packages as the main one, so combinatorial
+# generation still runs `scripts/generate.py` and writes through the same
+# ShardWriter. Every other family works without it, and the published
+# combinatorial HDF5 can be *read* with no PHYRE present at all.

eclise_phyworld-0.2.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

eclise_phyworld-0.2.0/src/eclise/phyworld/__init__.py

@@ -0,0 +1,64 @@
+"""Synthetic physics video generation, reorganized from the phyworld paper's code.
+
+Produces clip shards in the shared :mod:`eclise.clips` format, so a training
+project consumes this sim's output through the same reader it will use for real
+video.
+
+Four families:
+
+===================  ==========================================================
+``uniform_motion``   one ball, constant velocity -- position linear in time
+``parabola``         one ball under gravity -- position quadratic in time
+``collision``        two balls, elastic, unequal masses from unequal radii
+``combinatorial``    PHYRE templates (needs ``scripts/setup_phyre.sh``)
+===================  ==========================================================
+
+The first three are pure Box2D and need nothing beyond this env.
+
+    from eclise.phyworld import GenerateConfig, generate
+
+    report = generate(GenerateConfig(family="collision", grid="in_dist_l0"))
+    print(report.realized, "clips;", report.rejected, "rejected")
+
+Reading back is the consumer's job, through :mod:`eclise.clips`:
+
+    from eclise.clips import ClipDataset
+    data = ClipDataset("data/collision/default")
+"""
+
+from __future__ import annotations
+
+from .config import (
+    BOX2D_FAMILIES,
+    EVAL_TEMPLATES,
+    FAMILIES,
+    CombinatorialConfig,
+    GenerateConfig,
+)
+from .generate import GenerationReport, generate
+from .grids import Axis, ParamGrid, all_grids, resolve_grid
+from .render import RenderConfig, Trajectory, render_clip, render_velocity_field
+from .scenes import Collision, Parabola, SimConfig, UniformMotion, build_scene
+
+__all__ = [
+    "BOX2D_FAMILIES",
+    "EVAL_TEMPLATES",
+    "FAMILIES",
+    "Axis",
+    "Collision",
+    "CombinatorialConfig",
+    "GenerateConfig",
+    "GenerationReport",
+    "Parabola",
+    "ParamGrid",
+    "RenderConfig",
+    "SimConfig",
+    "Trajectory",
+    "UniformMotion",
+    "all_grids",
+    "build_scene",
+    "generate",
+    "render_clip",
+    "render_velocity_field",
+    "resolve_grid",
+]

eclise_phyworld-0.2.0/src/eclise/phyworld/analysis/__init__.py

@@ -0,0 +1,41 @@
+"""Recovering state from frames, and scoring it.
+
+Serves two purposes with one implementation. On *our* frames it verifies that
+pixels and labels agree, which is how a coordinate-convention bug gets caught
+before it becomes training data. On a *model's* generated frames the same parser
+yields the positions its output implies, which is phyworld's way of scoring
+physical plausibility without a learned metric.
+
+    from eclise.phyworld.analysis import verify_shards
+    print(verify_shards("data/collision/default").summary_text())
+"""
+
+from __future__ import annotations
+
+from .metrics import ClipError, ErrorSummary, clip_error, compare_to_labels, summarize
+from .state_recovery import (
+    Detection,
+    RecoveredClip,
+    color_mask,
+    detect_by_color,
+    detect_by_hough,
+    recover_clip,
+)
+from .verify import VerifyResult, upstream_opener, verify_shards
+
+__all__ = [
+    "ClipError",
+    "Detection",
+    "ErrorSummary",
+    "RecoveredClip",
+    "VerifyResult",
+    "clip_error",
+    "color_mask",
+    "compare_to_labels",
+    "detect_by_color",
+    "detect_by_hough",
+    "recover_clip",
+    "summarize",
+    "upstream_opener",
+    "verify_shards",
+]

eclise_phyworld-0.2.0/src/eclise/phyworld/analysis/metrics.py

@@ -0,0 +1,201 @@
+"""Comparing recovered state against ground truth.
+
+Two comparisons matter, and they answer different questions. *Position error* asks
+whether objects are where they should be -- sensitive to a coordinate-convention
+mistake, which is exactly the class of bug that produces plausible-looking video
+and unusable labels. *Velocity error* asks whether they move correctly, which is
+what a dynamics model is actually being asked to learn and is insensitive to a
+constant offset.
+
+Everything here works in **pixels with y downward**. Labels arrive in world units
+with y up, so put them through
+:func:`eclise.clips.labels.positions_to_pixels` first;
+:func:`compare_to_labels` does that for you.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+
+import numpy as np
+
+from .state_recovery import RecoveredClip
+
+
+@dataclass
+class ClipError:
+    """Discrepancy between recovered and true state for one clip.
+
+    Attributes:
+        position_rmse: root-mean-square centre distance, pixels, over frames
+            where the object was detected.
+        position_max: worst single-frame distance, pixels. The one that catches a
+            y flip -- a mirrored trajectory can have a modest RMSE while being
+            wrong everywhere.
+        velocity_rmse: RMS error of per-step velocity, px/frame.
+        radius_error: mean absolute radius error, pixels.
+        detection_rate: fraction of (frame, object) slots detected.
+        compared_frames: frames that entered the comparison.
+    """
+
+    position_rmse: float
+    position_max: float
+    velocity_rmse: float
+    radius_error: float
+    detection_rate: float
+    compared_frames: int
+
+    def to_dict(self) -> dict[str, float | int]:
+        return {
+            "position_rmse": round(self.position_rmse, 4),
+            "position_max": round(self.position_max, 4),
+            "velocity_rmse": round(self.velocity_rmse, 4),
+            "radius_error": round(self.radius_error, 4),
+            "detection_rate": round(self.detection_rate, 4),
+            "compared_frames": self.compared_frames,
+        }
+
+
+def clip_error(
+    recovered: RecoveredClip,
+    true_centers: np.ndarray,
+    *,
+    true_radii: np.ndarray | None = None,
+) -> ClipError:
+    """Compare one recovered clip against ground truth in pixel coordinates.
+
+    Args:
+        recovered: output of :func:`~.state_recovery.recover_clip`.
+        true_centers: ``[T, n_objects, 2]`` in pixels, y downward.
+        true_radii: ``[n_objects]`` or ``[T, n_objects]`` in pixels.
+
+    Undetected slots are excluded rather than counted as zero error; the
+    detection rate reports how many there were, so a clip that failed to detect
+    anything cannot masquerade as a perfect one.
+    """
+    true_centers = np.asarray(true_centers, dtype=np.float64)
+    if true_centers.shape != recovered.centers.shape:
+        raise ValueError(
+            f"recovered centers are {recovered.centers.shape} but truth is "
+            f"{true_centers.shape}"
+        )
+    mask = recovered.found
+    if not mask.any():
+        return ClipError(
+            position_rmse=float("nan"),
+            position_max=float("nan"),
+            velocity_rmse=float("nan"),
+            radius_error=float("nan"),
+            detection_rate=0.0,
+            compared_frames=0,
+        )
+
+    distances = np.linalg.norm(recovered.centers - true_centers, axis=-1)
+    valid = distances[mask]
+
+    radius_error = float("nan")
+    if true_radii is not None:
+        tr = np.asarray(true_radii, dtype=np.float64)
+        if tr.ndim == 1:
+            tr = np.broadcast_to(tr, recovered.radii.shape)
+        radius_error = float(np.abs(recovered.radii - tr)[mask].mean())
+
+    return ClipError(
+        position_rmse=float(np.sqrt((valid**2).mean())),
+        position_max=float(valid.max()),
+        velocity_rmse=_velocity_rmse(recovered, true_centers),
+        radius_error=radius_error,
+        detection_rate=recovered.detection_rate,
+        compared_frames=int(mask.sum()),
+    )
+
+
+def _velocity_rmse(recovered: RecoveredClip, true_centers: np.ndarray) -> float:
+    """RMS per-step velocity error over consecutive frames both detected in."""
+    both = recovered.found[:-1] & recovered.found[1:]
+    if not both.any():
+        return float("nan")
+    got = np.diff(recovered.centers, axis=0)
+    want = np.diff(true_centers, axis=0)
+    errors = np.linalg.norm(got - want, axis=-1)[both]
+    return float(np.sqrt((errors**2).mean()))
+
+
+def compare_to_labels(
+    recovered: RecoveredClip,
+    positions: np.ndarray,
+    spec,
+    *,
+    image_size: tuple[int, int],
+    world_scale: float | None,
+    radii: np.ndarray | None = None,
+) -> ClipError:
+    """Compare against stored labels, converting them to pixels first.
+
+    This is the function to use for verification: it routes the labels through
+    :func:`eclise.clips.labels.positions_to_pixels`, so the y flip that
+    ``spec.y_axis_up`` declares is applied rather than assumed either way.
+    """
+    from eclise.clips.labels import positions_to_pixels
+
+    true_centers = positions_to_pixels(
+        positions, spec, image_size=image_size, world_scale=world_scale
+    )
+    true_radii = None
+    if radii is not None and world_scale:
+        # Radii scale with the image, and only isotropically for a square frame.
+        true_radii = np.asarray(radii, dtype=np.float64) * (
+            image_size[1] / world_scale
+        )
+    return clip_error(recovered, true_centers, true_radii=true_radii)
+
+
+@dataclass
+class ErrorSummary:
+    """Aggregate over many clips."""
+
+    clips: int = 0
+    position_rmse: float = float("nan")
+    position_rmse_p95: float = float("nan")
+    position_max: float = float("nan")
+    velocity_rmse: float = float("nan")
+    detection_rate: float = float("nan")
+    failed_clips: int = 0
+    per_clip: list[ClipError] = field(default_factory=list)
+
+    def to_dict(self) -> dict[str, float | int]:
+        return {
+            "clips": self.clips,
+            "position_rmse": round(self.position_rmse, 4),
+            "position_rmse_p95": round(self.position_rmse_p95, 4),
+            "position_max": round(self.position_max, 4),
+            "velocity_rmse": round(self.velocity_rmse, 4),
+            "detection_rate": round(self.detection_rate, 4),
+            "failed_clips": self.failed_clips,
+        }
+
+
+def summarize(errors: list[ClipError]) -> ErrorSummary:
+    """Aggregate per-clip errors, reporting the p95 as well as the mean.
+
+    The p95 is there because a handful of bad clips is a different problem from a
+    uniformly small bias, and a mean cannot tell them apart.
+    """
+    if not errors:
+        return ErrorSummary()
+    usable = [e for e in errors if e.compared_frames > 0]
+    if not usable:
+        return ErrorSummary(clips=len(errors), failed_clips=len(errors), per_clip=errors)
+    rmses = np.array([e.position_rmse for e in usable])
+    return ErrorSummary(
+        clips=len(errors),
+        position_rmse=float(rmses.mean()),
+        position_rmse_p95=float(np.percentile(rmses, 95)),
+        position_max=float(max(e.position_max for e in usable)),
+        velocity_rmse=float(
+            np.nanmean([e.velocity_rmse for e in usable])
+        ),
+        detection_rate=float(np.mean([e.detection_rate for e in errors])),
+        failed_clips=len(errors) - len(usable),
+        per_clip=errors,
+    )