logogram 0.1.1__tar.gz → 0.1.2__tar.gz

{logogram-0.1.1 → logogram-0.1.2}/CHANGELOG.md

@@ -2,6 +2,40 @@
 
 What changed in each version of Logogram.
 
+## 0.1.2 (2026-10-08)
+
+Fixes and checks that came out of validating every method on real weights.
+
+### Changed
+
+- On Apple Silicon, **Automatic** runs models on the CPU. TransformerLens reports that Apple's MPS
+  can give silently wrong results, and Logogram hasn't been checked on it yet, so MPS runs only when
+  you choose it; the system check and the model dialog say why.
+
+### Added
+
+- **Check robustness** reruns a float16 or bfloat16 run in float32 and compares the two.
+- When processing a model's weights is what doesn't fit, the model dialog offers to turn it off.
+- When a generated IOI dataset has names the loaded model splits into several tokens, one click
+  makes a new one with names it doesn't.
+- Steering results say when the direction does no more than its random control at any site and
+  strength.
+- Attribution patching of residual stream sites warns that it can miss effects there, and that
+  **Check robustness** patches every site.
+- `scripts/validate_real_weights.py` runs every method on a real model and checks identities,
+  agreement and bit-identical reruns, on one device or two. It is how a release, or Apple's MPS, is
+  checked.
+
+### Fixed
+
+- The baseline says that the clean prompts prefer the distractor, instead of preferring the answer
+  by a negative amount.
+
+### Development
+
+- Tests run on Python 3.14 too, and the Linux runners are pinned to Ubuntu 24.04.
+- The test client uses httpx2, as Starlette recommends.
+
 ## 0.1.1 (2026-10-08)
 
 Every method has now run end to end on real weights: GPT-2 small with a SAELens and an OpenAI

{logogram-0.1.1 → logogram-0.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: logogram
-Version: 0.1.1
+Version: 0.1.2
 Summary: A local-first workbench for causal experiments inside language models.
 Project-URL: Homepage, https://github.com/Jeevash23/logogram
 Project-URL: Source, https://github.com/Jeevash23/logogram
@@ -17,6 +17,7 @@ Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
 Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
 Requires-Python: >=3.11
 Requires-Dist: anyio>=4
@@ -73,7 +74,7 @@ PyTorch is chosen per machine:
 |---|---|---|
 | Linux with an NVIDIA GPU | `uv tool install logogram` | The default Linux wheels include CUDA. |
 | Windows with an NVIDIA GPU | `uv tool install --torch-backend=auto logogram` | Picks the CUDA build that matches your driver. |
-| Apple Silicon | `uv tool install logogram` | Uses Metal (MPS). Use a native arm64 Python, not one under Rosetta. |
+| Apple Silicon | `uv tool install logogram` | Runs on the CPU unless you choose MPS (see Models). Use a native arm64 Python, not one under Rosetta. |
 | CPU only | `uv tool install --torch-backend=cpu logogram` | Smaller download. GPT-2 small runs well on a CPU. |
 
 Then check the setup:
@@ -114,8 +115,9 @@ This starts a local server on 127.0.0.1, prints its address and opens your brows
 4. Press **A** on a head to see its attention pattern, or right-click any cell for **Patch here**,
    **Ablate here** and **Compare across runs**.
 5. Press **Check robustness** to rerun the sweep with a different baseline, direction or donor
-   count. Logogram reports the rank correlation, the overlap of the top components and the
-   components whose conclusion changed, and flags them on the map.
+   count, or, for a run in float16 or bfloat16, in float32. Logogram reports the rank
+   correlation, the overlap of the top components and the components whose conclusion changed,
+   and flags them on the map.
 
 Keyboard: arrow keys move across the map, **Ctrl/⌘ K** opens the command palette (type `L9H9` to
 jump to a head), **1–7** switch views, **[** and **]** step through prompts, **P**, **B**, **A**
@@ -311,7 +313,7 @@ its effect at every strength with intervals. **Check robustness** offers another
 or the other direction. A mean difference steers only when the pairs differ the same way. In IOI
 prompts that mix the ABBA and BABA orders, the difference at the last token flips with the order,
 so the mean cancels and steering does no more than the control; generate prompts of one order to
-steer.
+steer. When no site and strength does more than the control, the results say so.
 
 **SAE features.** A sparse autoencoder (SAE) rewrites one of the model's activations as a few active
 features out of thousands, each a direction in the model, plus an error it misses. In
@@ -490,7 +492,11 @@ times faster on a GPU, but they round. In the IOI example on GPT-2 small and Qwe
 moved effects by at most 0.003 and bfloat16 by up to 0.03: the strongest sites stayed in place,
 but effects smaller than about 0.01 changed order. On Pythia-70m, bfloat16 doubled the clean
 logit difference and reordered the heads, and float16 overflows. Check results that matter
-against float32.
+against float32: **Check robustness** reruns a 16-bit run in float32 and compares the two.
+
+On Apple Silicon, **Automatic** runs models on the CPU. TransformerLens reports that Apple's MPS
+can give silently wrong results, and Logogram hasn't been checked on it yet, so MPS is used only
+when you choose it under **Device**. It is faster; check results that matter on the CPU.
 
 Rather than trusting a list, Logogram checks every model when it loads, on a short fixed input:
 
@@ -561,10 +567,18 @@ push a tag with the same version, such as `v0.1.0`. The Release workflow runs ev
 that commit, builds the wheel and source distribution, and uploads them to PyPI through Trusted
 Publishing, so no token is stored anywhere. PyPI never accepts the same version twice.
 
-Before tagging, manually check a first GPT-2 download, cancel and retry it, then run the example
-on each supported compute backend. CPU and CUDA are covered by local development checks; MPS
-still needs a check on Apple Silicon. Pythia and Qwen 2.5 have also been run end to end with
-their real weights; other families are checked when they load and in the tiny-model tests.
+Before tagging, manually check a first GPT-2 download, cancel and retry it, then run
+`scripts/validate_real_weights.py` on each supported compute backend. It runs every method on a
+real model in a temporary project and checks exact identities, agreement between methods and
+bit-identical reruns; `--compare-device` reruns the head sweep on a second device, and `--sae`
+adds SAE features. CPU and CUDA pass it. Apple's MPS still needs it run on a Mac:
+
+```bash
+uv run python scripts/validate_real_weights.py --device mps --compare-device cpu
+```
+
+Pythia and Qwen 2.5 have also been run end to end with their real weights; other families are
+checked when they load and in the tiny-model tests.
 
 Model access goes through `logogram.backends.base.ModelBackend`. TransformerLens is the only
 backend today; remote execution and other libraries can be added behind the same interface.

{logogram-0.1.1 → logogram-0.1.2}/README.md

@@ -36,7 +36,7 @@ PyTorch is chosen per machine:
 |---|---|---|
 | Linux with an NVIDIA GPU | `uv tool install logogram` | The default Linux wheels include CUDA. |
 | Windows with an NVIDIA GPU | `uv tool install --torch-backend=auto logogram` | Picks the CUDA build that matches your driver. |
-| Apple Silicon | `uv tool install logogram` | Uses Metal (MPS). Use a native arm64 Python, not one under Rosetta. |
+| Apple Silicon | `uv tool install logogram` | Runs on the CPU unless you choose MPS (see Models). Use a native arm64 Python, not one under Rosetta. |
 | CPU only | `uv tool install --torch-backend=cpu logogram` | Smaller download. GPT-2 small runs well on a CPU. |
 
 Then check the setup:
@@ -77,8 +77,9 @@ This starts a local server on 127.0.0.1, prints its address and opens your brows
 4. Press **A** on a head to see its attention pattern, or right-click any cell for **Patch here**,
    **Ablate here** and **Compare across runs**.
 5. Press **Check robustness** to rerun the sweep with a different baseline, direction or donor
-   count. Logogram reports the rank correlation, the overlap of the top components and the
-   components whose conclusion changed, and flags them on the map.
+   count, or, for a run in float16 or bfloat16, in float32. Logogram reports the rank
+   correlation, the overlap of the top components and the components whose conclusion changed,
+   and flags them on the map.
 
 Keyboard: arrow keys move across the map, **Ctrl/⌘ K** opens the command palette (type `L9H9` to
 jump to a head), **1–7** switch views, **[** and **]** step through prompts, **P**, **B**, **A**
@@ -274,7 +275,7 @@ its effect at every strength with intervals. **Check robustness** offers another
 or the other direction. A mean difference steers only when the pairs differ the same way. In IOI
 prompts that mix the ABBA and BABA orders, the difference at the last token flips with the order,
 so the mean cancels and steering does no more than the control; generate prompts of one order to
-steer.
+steer. When no site and strength does more than the control, the results say so.
 
 **SAE features.** A sparse autoencoder (SAE) rewrites one of the model's activations as a few active
 features out of thousands, each a direction in the model, plus an error it misses. In
@@ -453,7 +454,11 @@ times faster on a GPU, but they round. In the IOI example on GPT-2 small and Qwe
 moved effects by at most 0.003 and bfloat16 by up to 0.03: the strongest sites stayed in place,
 but effects smaller than about 0.01 changed order. On Pythia-70m, bfloat16 doubled the clean
 logit difference and reordered the heads, and float16 overflows. Check results that matter
-against float32.
+against float32: **Check robustness** reruns a 16-bit run in float32 and compares the two.
+
+On Apple Silicon, **Automatic** runs models on the CPU. TransformerLens reports that Apple's MPS
+can give silently wrong results, and Logogram hasn't been checked on it yet, so MPS is used only
+when you choose it under **Device**. It is faster; check results that matter on the CPU.
 
 Rather than trusting a list, Logogram checks every model when it loads, on a short fixed input:
 
@@ -524,10 +529,18 @@ push a tag with the same version, such as `v0.1.0`. The Release workflow runs ev
 that commit, builds the wheel and source distribution, and uploads them to PyPI through Trusted
 Publishing, so no token is stored anywhere. PyPI never accepts the same version twice.
 
-Before tagging, manually check a first GPT-2 download, cancel and retry it, then run the example
-on each supported compute backend. CPU and CUDA are covered by local development checks; MPS
-still needs a check on Apple Silicon. Pythia and Qwen 2.5 have also been run end to end with
-their real weights; other families are checked when they load and in the tiny-model tests.
+Before tagging, manually check a first GPT-2 download, cancel and retry it, then run
+`scripts/validate_real_weights.py` on each supported compute backend. It runs every method on a
+real model in a temporary project and checks exact identities, agreement between methods and
+bit-identical reruns; `--compare-device` reruns the head sweep on a second device, and `--sae`
+adds SAE features. CPU and CUDA pass it. Apple's MPS still needs it run on a Mac:
+
+```bash
+uv run python scripts/validate_real_weights.py --device mps --compare-device cpu
+```
+
+Pythia and Qwen 2.5 have also been run end to end with their real weights; other families are
+checked when they load and in the tiny-model tests.
 
 Model access goes through `logogram.backends.base.ModelBackend`. TransformerLens is the only
 backend today; remote execution and other libraries can be added behind the same interface.

{logogram-0.1.1 → logogram-0.1.2}/pyproject.toml

@@ -20,6 +20,7 @@ classifiers = [
     "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",
     "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
     "Topic :: Scientific/Engineering :: Artificial Intelligence",
 ]
 dependencies = [
@@ -51,7 +52,7 @@ logogram = "logogram.cli:main"
 [dependency-groups]
 dev = [
     "pytest>=8",
-    "httpx>=0.27",
+    "httpx2>=2.13",
     "ruff==0.16.10",
 ]
 

logogram-0.1.2/scripts/validate_real_weights.py

@@ -0,0 +1,459 @@
+#!/usr/bin/env python3
+"""Run every Logogram method on a real model and check what must hold.
+
+A release check, and the check for a compute backend Logogram hasn't been verified on (Apple's
+MPS). It loads a model from Hugging Face (downloading it once into the usual cache), runs each
+method on the bundled IOI example in a temporary project, and checks:
+
+* exact identities: patching the residual stream at the only token that differs restores the
+  whole effect, and so does patching the final residual stream; path patching from the last layer
+  to the logits equals patching; direct effects add up to the logit difference; the same spec
+  twice gives bit-identical results;
+* agreement expected of a trained model: attribution patching ranks heads like patching, and
+  verifying its strongest estimates reverses no sign;
+* that every other method runs (ablations, steering, path patching, layer predictions, and SAE
+  features with --sae).
+
+With --compare-device it runs the head sweep again on a second device and reports how far the two
+disagree. Nothing is written outside a temporary folder, apart from the Hugging Face cache.
+
+    uv run python scripts/validate_real_weights.py
+    uv run python scripts/validate_real_weights.py --device mps --compare-device cpu
+    uv run python scripts/validate_real_weights.py --sae jbloom/GPT2-Small-SAEs-Reformatted \\
+        blocks.8.hook_resid_pre
+
+Exits with status 1 if a check fails.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import shutil
+import sys
+import tempfile
+import time
+from collections.abc import Callable
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any
+
+# The tolerance of an identity in each dtype: float32 rounds in the last digits, 16-bit types in
+# the second or third.
+TOLERANCE = {"float32": 1e-3, "float16": 2e-2, "bfloat16": 5e-2}
+
+
+@dataclass
+class Ran:
+    """A finished run: its summary, its folder, and the spec it ran."""
+
+    summary: dict[str, Any]
+    folder: Path
+    spec: Any
+
+
+@dataclass
+class Check:
+    name: str
+    passed: bool | None  # None: skipped
+    detail: str
+    exact: bool  # holds for any model, not only a trained one
+
+
+def validate(
+    backend: Any,
+    project: Any,
+    dataset: str,
+    *,
+    prepend_bos: bool,
+    sae_ref: Any = None,
+    compare: Any = None,
+    log: Callable[[str], None] = print,
+) -> list[Check]:
+    """Run the methods on ``backend`` with ``dataset`` (a path in ``project``) and check them.
+    ``compare`` is a second backend of the same model on another device, if any."""
+    import numpy as np
+    import pyarrow.parquet as pq
+
+    from logogram.compare import compare_summaries
+    from logogram.datasets import file_sha256
+    from logogram.results import largest_change
+    from logogram.runner import run_spec
+    from logogram.spec import Spec
+    from logogram.verify import verification_spec
+
+    info = backend.info
+    tol = TOLERANCE[info.dtype]
+    checks: list[Check] = []
+    sha = file_sha256(project.root / dataset)
+
+    def check(name: str, passed: bool | None, detail: str, exact: bool = True) -> None:
+        checks.append(Check(name, passed, detail, exact))
+        mark = {True: "PASS", False: "FAIL", None: "SKIP"}[passed]
+        log(f"{mark}  {name}: {detail}")
+
+    def spec(experiment: dict, scope: dict, model: Any = None, **extra: Any) -> Spec:
+        m = (model or backend).info
+        return Spec.model_validate(
+            {
+                "name": "validation",
+                "model": {
+                    "id": m.id,
+                    "revision": m.revision,
+                    "dtype": m.dtype,
+                    "device": m.device,
+                    "process_weights": m.process_weights,
+                },
+                "dataset": {"path": dataset, "sha256": sha},
+                "tokenization": {"prepend_bos": prepend_bos},
+                "experiment": experiment,
+                "scope": scope,
+                "statistics": {"bootstrap": 200, "ci": 0.95, "seed": 0},
+                "execution": {"batch_size": 32},
+                **extra,
+            }
+        )
+
+    def run(label: str, s: Spec, model: Any = None, sae: Any = None) -> Ran | None:
+        started = time.perf_counter()
+        outcome = run_spec(s, project, backend=model or backend, sae=sae)
+        seconds = time.perf_counter() - started
+        if outcome.status != "finished":
+            check(f"{label} runs", False, str(outcome.manifest.get("error")))
+            return None
+        log(f"      {label}: {seconds:.1f} s")
+        return Ran(outcome.summary or {}, outcome.folder, s)
+
+    def effect(outcome: Any, label: str) -> float | None:
+        site = next((s for s in outcome.summary["sites"] if s["label"] == label), None)
+        return None if site is None else site["effect"]["mean"]
+
+    patch = {"kind": "activation_patching", "direction": "clean_to_corrupt"}
+    estimate = {"kind": "attribution_patching", "direction": "clean_to_corrupt"}
+    every_head = {"kind": "heads", "position": {"kind": "all"}}
+    last = {"kind": "last"}
+    n_layers, n_heads = info.n_layers, info.n_heads
+    checks_info = info.extra.get("checks") or {}
+    check(
+        "The model reproduces itself when it loads",
+        True,
+        f"error {checks_info.get('function', 0):.1e}, {info.extra.get('block_structure')} layers",
+    )
+
+    heads = run("Patching every head", spec(patch, every_head))
+    resid = run(
+        "Patching the residual stream at each named position",
+        spec(patch, {"kind": "layer_position", "site": "resid_pre", "positions": "labels"}),
+    )
+    if resid is not None:
+        at_s2, at_end = effect(resid, "L0 resid pre @ S2"), effect(resid, "L0 resid pre @ end")
+        if at_s2 is None:
+            check("Patching the token that differs restores everything", None, "no S2 label")
+        else:
+            check(
+                "Patching the token that differs restores everything",
+                abs(at_s2 - 1) <= tol and abs(at_end or 0) <= tol,
+                f"effect {at_s2:.4f} at S2 and {at_end or 0:.4f} at the end, in layer 0",
+            )
+    final = run(
+        "Patching the final residual stream",
+        spec(
+            patch,
+            {
+                "kind": "sites",
+                "sites": [{"kind": "resid_post", "layer": n_layers - 1, "position": last}],
+            },
+        ),
+    )
+    if final is not None:
+        value = final.summary["sites"][0]["effect"]["mean"]
+        check(
+            "Patching the final residual stream restores everything",
+            abs(value - 1) <= tol,
+            f"effect {value:.4f}",
+        )
+    if heads is not None:
+        again = run("The same head sweep again", heads.spec)
+        if again is not None:
+            change = largest_change(
+                pq.read_table(heads.folder / "results.parquet"),
+                pq.read_table(again.folder / "results.parquet"),
+            )
+            check(
+                "The same spec twice gives bit-identical results",
+                change == 0.0,
+                "identical" if change == 0.0 else f"largest change {change}",
+            )
+
+    components = {
+        "kind": "layer_components",
+        "components": ["attn_out", "mlp_out"],
+        "position": {"kind": "all"},
+    }
+    for name, baseline in (
+        ("Zero ablation", {"kind": "zero"}),
+        ("Mean ablation", {"kind": "mean", "reference": "corrupt"}),
+        ("Resample ablation", {"kind": "resample", "pool": "corrupt", "donors": 4, "seed": 0}),
+    ):
+        if run(name, spec({"kind": "ablation", "baseline": baseline}, components)) is not None:
+            check(f"{name} runs", True, "finished")
+
+    direct = run(
+        "Direct logit attribution",
+        spec(
+            {"kind": "direct_logit_attribution", "prompts": "clean"},
+            {"kind": "layer_components", "components": ["attn_out", "mlp_out"], "position": last},
+        ),
+    )
+    best_heads: list[tuple[int, int]] = []
+    if direct is not None:
+        split = direct.summary["direct"]
+        measured = direct.summary["baseline"]["clean"]["logit_diff"]["mean"]
+        parts = split["embeddings"] + split["attention"] + split["mlp"] + split["biases"]
+        check(
+            "Direct effects add up to the logit difference",
+            abs(split["logit_diff"] - measured) <= tol * max(1.0, abs(measured))
+            and abs(parts - split["logit_diff"]) <= 1e-6 * max(1.0, abs(measured)),
+            f"{split['logit_diff']:.4f} split, {measured:.4f} measured",
+        )
+        per_head = run(
+            "Direct effects of every head",
+            spec(
+                {"kind": "direct_logit_attribution", "prompts": "clean"},
+                {"kind": "heads", "position": last},
+            ),
+        )
+        if per_head is not None:
+            later = [s for s in per_head.summary["sites"] if s["layer"] >= n_layers // 2]
+            later.sort(key=lambda s: -(s["effect"]["mean"] or 0))
+            best_heads = [(s["layer"], s["head"]) for s in later[:3]]
+
+    estimated = run("Attribution patching of every head", spec(estimate, every_head))
+    if estimated is not None and heads is not None:
+        cmp = compare_summaries(heads.summary, estimated.summary, top_k=10)
+        check(
+            "Attribution patching ranks heads like patching",
+            (cmp["spearman"] or 0) >= 0.8,
+            f"rank correlation {cmp['spearman']:.3f}, top 10 overlap {cmp['top_overlap']}",
+            exact=False,
+        )
+        verified = run(
+            "Verifying the top 10 estimates by patching",
+            verification_spec(estimated.spec, estimated.summary, 10),
+        )
+        if verified is not None:
+            cmp = compare_summaries(estimated.summary, verified.summary, top_k=5)
+            check(
+                "Patching reverses none of the top estimates",
+                cmp["n_sign_changes"] == 0,
+                f"{cmp['n_sign_changes']} reversed, rank correlation {cmp['spearman']:.3f}",
+                exact=False,
+            )
+
+    steering = {
+        "kind": "steering",
+        "apply_to": "clean",
+        "coefficients": [-1, 1],
+        "train_fraction": 0.5,
+        "seed": 0,
+        "control": True,
+    }
+    if run(
+        "Steering",
+        spec(steering, {"kind": "layer_components", "components": ["resid_pre"], "position": last}),
+    ):
+        check("Steering runs", True, "finished")
+
+    last_heads = [
+        {"kind": "head", "layer": n_layers - 1, "head": h, "position": {"kind": "all"}}
+        for h in range(n_heads)
+    ]
+    to_logits = {
+        "kind": "path_patching",
+        "direction": "clean_to_corrupt",
+        "receivers": [{"kind": "logits"}],
+        "freeze_mlps": False,
+    }
+    path = run(
+        "Path patching from the last layer to the logits",
+        spec(to_logits, {"kind": "sites", "sites": last_heads}),
+    )
+    patched = run(
+        "Patching the last layer's heads", spec(patch, {"kind": "sites", "sites": last_heads})
+    )
+    if path is not None and patched is not None:
+        a = np.asarray(pq.read_table(path.folder / "results.parquet").column("patched_logit_diff"))
+        b = np.asarray(
+            pq.read_table(patched.folder / "results.parquet").column("patched_logit_diff")
+        )
+        check(
+            "From the last layer, a path to the logits is the whole effect",
+            float(np.abs(a - b).max()) <= tol,
+            f"largest difference {float(np.abs(a - b).max()):.2e}",
+        )
+    if best_heads:
+        receivers = [
+            {"kind": "head", "layer": layer, "head": h, "input": "q"} for layer, h in best_heads
+        ]
+        to_queries = {
+            "kind": "path_patching",
+            "direction": "clean_to_corrupt",
+            "receivers": receivers,
+            "freeze_mlps": False,
+        }
+        if run("Path patching into the strongest heads' queries", spec(to_queries, every_head)):
+            check("Path patching into heads runs", True, f"receivers {best_heads}")
+
+    if info.extra.get("prediction_method"):
+        settings = {
+            "method": "final_norm_logit_lens",
+            "prompt_index": 0,
+            "which": "clean",
+            "position": last,
+            "top_k": 3,
+        }
+        lens = run(
+            "Layer predictions",
+            spec(patch, {"kind": "sites", "sites": last_heads[:1]}, predictions=settings),
+        )
+        if lens is not None:
+            report = json.loads((lens.folder / "predictions.json").read_text(encoding="utf-8"))
+            mine = report["layers"][-1]["answer_prob"]
+            model = lens.summary["per_prompt"]["clean_answer_prob"][0]
+            check(
+                "The logit lens at the last layer is the model's own prediction",
+                abs(mine - model) <= tol * max(model, 1e-6) + 1e-7,
+                f"answer probability {mine:.6f} lens, {model:.6f} model",
+            )
+
+    if sae_ref is not None:
+        from logogram.analysis import sae_fit_report
+        from logogram.datasets import load_dataset
+        from logogram.sae import load_sae
+
+        sae = load_sae(sae_ref, info.device)
+        fit = sae_fit_report(
+            backend,
+            sae,
+            load_dataset(project.root / dataset),
+            prepend_bos=prepend_bos,
+            batch_size=32,
+        )
+        check(
+            "The SAE fits these activations",
+            fit["variance_explained"] > 0.5,
+            f"explains {fit['variance_explained']:.1%} of the variance; check weight processing if not",
+            exact=False,
+        )
+        ref = {"repo": sae.repo, "path": sae.path, "revision": sae.revision}
+        features = run(
+            "Attribution patching of every SAE feature",
+            spec(
+                estimate,
+                {"kind": "features", "position": {"kind": "label", "label": "end"}, "top": 20},
+                sae=ref,
+            ),
+            sae=sae,
+        )
+        if features is not None:
+            verified = run(
+                "Patching the top 10 features",
+                verification_spec(features.spec, features.summary, 10),
+                sae=sae,
+            )
+            if verified is not None:
+                cmp = compare_summaries(features.summary, verified.summary, top_k=5)
+                check(
+                    "Feature estimates match patching",
+                    cmp["n_sign_changes"] == 0 and (cmp["spearman"] or 0) >= 0.8,
+                    f"rank correlation {cmp['spearman']:.3f}, {cmp['n_sign_changes']} reversed",
+                    exact=False,
+                )
+
+    if compare is not None and heads is not None:
+        other = run(
+            f"Patching every head on {compare.info.device}", spec(patch, every_head, model=compare)
+        )
+        if other is not None:
+            x = np.asarray(pq.read_table(heads.folder / "results.parquet").column("effect"))
+            y = np.asarray(pq.read_table(other.folder / "results.parquet").column("effect"))
+            cmp = compare_summaries(heads.summary, other.summary, top_k=10)
+            agree = TOLERANCE[compare.info.dtype] if compare.info.dtype != "float32" else tol
+            check(
+                f"{info.device} and {compare.info.device} agree",
+                float(np.abs(x - y).max()) <= agree and cmp["top_overlap"] == 10,
+                f"largest difference in a per-prompt effect {float(np.abs(x - y).max()):.2e}, "
+                f"top 10 overlap {cmp['top_overlap']}",
+            )
+    return checks
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
+    parser.add_argument("--model", default="openai-community/gpt2")
+    parser.add_argument("--revision", default=None)
+    parser.add_argument("--device", default="auto", choices=["auto", "cpu", "cuda", "mps"])
+    parser.add_argument("--dtype", default="float32", choices=sorted(TOLERANCE))
+    parser.add_argument(
+        "--no-processing", action="store_true", help="Load without weight processing."
+    )
+    parser.add_argument("--compare-device", choices=["cpu", "cuda", "mps"], default=None)
+    parser.add_argument("--sae", nargs=2, metavar=("REPO", "PATH"), default=None)
+    args = parser.parse_args()
+
+    from logogram.backends.transformer_lens import load_model
+    from logogram.datasets import load_dataset, write_dataset
+    from logogram.ioi import generate_ioi
+    from logogram.project import Project, example_source
+    from logogram.prompts import prepare_prompts
+    from logogram.spec import SAERef
+
+    started = time.perf_counter()
+
+    def load(device: str) -> Any:
+        print(f"Loading {args.model} on {device} ({args.dtype})…", flush=True)
+        return load_model(
+            args.model,
+            revision=args.revision,
+            dtype=args.dtype,
+            device=device,
+            process_weights=not args.no_processing,
+        )
+
+    backend = load(args.device)
+    compare = load(args.compare_device) if args.compare_device else None
+    folder = Path(tempfile.mkdtemp(prefix="logogram-validation-"))
+    try:
+        shutil.copytree(example_source(), folder / "project")
+        project = Project.open(folder / "project")
+        bos = bool(backend.info.extra.get("bos"))
+        dataset = "datasets/ioi.jsonl"
+        try:
+            prepare_prompts(backend, load_dataset(project.root / dataset), bos)
+        except ValueError as exc:
+            # The example's names were chosen for GPT-2; make prompts this tokenizer can use.
+            print(f"The example's prompts don't fit this tokenizer ({exc}); generating new ones.")
+            records = generate_ioi(
+                64, seed=0, single_token=lambda w: backend.single_token_id(w) is not None
+            )
+            dataset = "datasets/ioi-validation.jsonl"
+            write_dataset(project.root / dataset, records)
+        sae_ref = SAERef(repo=args.sae[0], path=args.sae[1]) if args.sae else None
+        checks = validate(
+            backend, project, dataset, prepend_bos=bos, sae_ref=sae_ref, compare=compare
+        )
+    finally:
+        shutil.rmtree(folder, ignore_errors=True)
+    failed = [c for c in checks if c.passed is False]
+    print(
+        f"\n{len(checks) - len(failed)} of {len(checks)} checks passed "
+        f"in {time.perf_counter() - started:.0f} s on {backend.info.device_name}."
+    )
+    for c in failed:
+        kind = "an identity" if c.exact else "agreement expected of a trained model"
+        print(f"FAILED ({kind}): {c.name}: {c.detail}")
+    return 1 if failed else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())