logogram 0.1.0__tar.gz → 0.1.2__tar.gz

logogram-0.1.2/CHANGELOG.md

@@ -0,0 +1,94 @@
+# Changelog
+
+What changed in each version of Logogram.
+
+## 0.1.2 (2026-10-08)
+
+Fixes and checks that came out of validating every method on real weights.
+
+### Changed
+
+- On Apple Silicon, **Automatic** runs models on the CPU. TransformerLens reports that Apple's MPS
+  can give silently wrong results, and Logogram hasn't been checked on it yet, so MPS runs only when
+  you choose it; the system check and the model dialog say why.
+
+### Added
+
+- **Check robustness** reruns a float16 or bfloat16 run in float32 and compares the two.
+- When processing a model's weights is what doesn't fit, the model dialog offers to turn it off.
+- When a generated IOI dataset has names the loaded model splits into several tokens, one click
+  makes a new one with names it doesn't.
+- Steering results say when the direction does no more than its random control at any site and
+  strength.
+- Attribution patching of residual stream sites warns that it can miss effects there, and that
+  **Check robustness** patches every site.
+- `scripts/validate_real_weights.py` runs every method on a real model and checks identities,
+  agreement and bit-identical reruns, on one device or two. It is how a release, or Apple's MPS, is
+  checked.
+
+### Fixed
+
+- The baseline says that the clean prompts prefer the distractor, instead of preferring the answer
+  by a negative amount.
+
+### Development
+
+- Tests run on Python 3.14 too, and the Linux runners are pinned to Ubuntu 24.04.
+- The test client uses httpx2, as Starlette recommends.
+
+## 0.1.1 (2026-10-08)
+
+Every method has now run end to end on real weights: GPT-2 small with a SAELens and an OpenAI
+SAE, Pythia-70m with an EleutherAI SAE, and Qwen 2.5 0.5B. This release fixes what that turned up.
+
+### Fixed
+
+- Runs work on Apple Silicon. Every method converted tensors to float64 on the model's device,
+  which Apple's Metal (MPS) doesn't have, so no run got past its baseline on a Mac. This still
+  needs a check on a Mac.
+- A model whose own forward pass overflows in the chosen dtype (Pythia in float16) is refused, with
+  the dtype to use. Its load checks used to read as passed.
+- When processing a model's weights doesn't fit in memory, loading says so and frees the memory,
+  instead of failing with a raw CUDA error and keeping it.
+- The memory estimate counts weight processing, which needs three to four times the float32 size
+  of the weights while the model loads. Qwen 2.5 0.5B on a 6 GB GPU now reads "won't fit" with
+  processing and "fits" without it.
+- Models without a beginning-of-sequence token, such as Qwen, run without one, as documented.
+  TransformerLens 4 had given them its end-of-text token instead.
+- Models and SAEs that Logogram downloaded open offline without a revision, and offline memory
+  estimates use the model's real size.
+- `logogram run` reports an identical rerun of direct logit attribution or attribution patching as
+  identical.
+- Direct logit attribution in float16 and bfloat16 no longer warns about drift that is only
+  rounding.
+
+### Added
+
+- Runs warn when the model prefers the distractor on the clean prompts, so it doesn't show the
+  behavior they test (Pythia-70m on IOI).
+- The model dialog shows the memory that weight processing needs. Qwen 2.5 0.5B is marked as
+  tested.
+- Project links on PyPI, and this changelog, which the update notice links to.
+
+### Documentation
+
+- What has been validated with real weights, and where attribution patching is least reliable:
+  residual stream sites at the token that differs between the prompts.
+- Steering needs prompt pairs that differ the same way; IOI prompts that mix the ABBA and BABA
+  orders cancel out.
+- What float16 and bfloat16 do to results, and how weight processing changes heads' direct effects.
+
+## 0.1.0 (2026-10-08)
+
+First public release.
+
+- Activation patching, ablation (zero, mean or resample), direct logit attribution, attribution
+  patching verified by patching, steering with a random control, path patching, and SAE features,
+  all described by one spec that the app and `logogram run` execute the same way.
+- Every model is checked when it loads: that TransformerLens reproduces its predictions, how its
+  layers add into the residual stream, and whether the logit lens reproduces its output.
+- Per-prompt values with seeded bootstrap intervals, and bit-identical reruns on the same machine.
+- The workbench: Explore, Experiment and Evidence, logograms written by results, robustness checks
+  and run comparisons.
+- Local-first: no account and no telemetry, safetensors weights only, and an update check that
+  runs only when you allow it.

logogram-0.1.0/README.md → logogram-0.1.2/PKG-INFO

@@ -1,3 +1,41 @@
+Metadata-Version: 2.5
+Name: logogram
+Version: 0.1.2
+Summary: A local-first workbench for causal experiments inside language models.
+Project-URL: Homepage, https://github.com/Jeevash23/logogram
+Project-URL: Source, https://github.com/Jeevash23/logogram
+Project-URL: Issues, https://github.com/Jeevash23/logogram/issues
+Project-URL: Changelog, https://github.com/Jeevash23/logogram/blob/main/CHANGELOG.md
+Author: The Logogram contributors
+License-Expression: MIT
+License-File: LICENSE
+Keywords: activation-patching,interpretability,mechanistic-interpretability,transformers
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.11
+Requires-Dist: anyio>=4
+Requires-Dist: fastapi>=0.115
+Requires-Dist: huggingface-hub>=0.23
+Requires-Dist: numpy>=1.26
+Requires-Dist: platformdirs>=4
+Requires-Dist: psutil>=5.9
+Requires-Dist: pyarrow>=15
+Requires-Dist: pydantic>=2.7
+Requires-Dist: torch>=2.4
+Requires-Dist: transformer-lens<5,>=4.0
+Requires-Dist: transformers>=4.45
+Requires-Dist: typer>=0.12
+Requires-Dist: uvicorn>=0.30
+Requires-Dist: websockets>=12
+Description-Content-Type: text/markdown
+
 # Logogram
 
 A local workbench for causal experiments inside language models.
@@ -25,9 +63,10 @@ Logogram needs Python 3.11 or newer and [uv](https://docs.astral.sh/uv/).
 uv tool install logogram
 ```
 
-Until Logogram is published on PyPI, install it from a clone of this repository with
-`uv tool install .` instead. The web app is prebuilt inside the package, so you never need Node.
-The commands in the table below work the same way with `.` in place of `logogram`.
+To run the code in a clone of this repository instead, install it with `uv tool install .`; the
+commands in the table below work the same way with `.` in place of `logogram`. The web app is
+prebuilt inside the package, so you never need Node. What changed in each version is in
+[CHANGELOG.md](CHANGELOG.md).
 
 PyTorch is chosen per machine:
 
@@ -35,7 +74,7 @@ PyTorch is chosen per machine:
 |---|---|---|
 | Linux with an NVIDIA GPU | `uv tool install logogram` | The default Linux wheels include CUDA. |
 | Windows with an NVIDIA GPU | `uv tool install --torch-backend=auto logogram` | Picks the CUDA build that matches your driver. |
-| Apple Silicon | `uv tool install logogram` | Uses Metal (MPS). Use a native arm64 Python, not one under Rosetta. |
+| Apple Silicon | `uv tool install logogram` | Runs on the CPU unless you choose MPS (see Models). Use a native arm64 Python, not one under Rosetta. |
 | CPU only | `uv tool install --torch-backend=cpu logogram` | Smaller download. GPT-2 small runs well on a CPU. |
 
 Then check the setup:
@@ -76,8 +115,9 @@ This starts a local server on 127.0.0.1, prints its address and opens your brows
 4. Press **A** on a head to see its attention pattern, or right-click any cell for **Patch here**,
    **Ablate here** and **Compare across runs**.
 5. Press **Check robustness** to rerun the sweep with a different baseline, direction or donor
-   count. Logogram reports the rank correlation, the overlap of the top components and the
-   components whose conclusion changed, and flags them on the map.
+   count, or, for a run in float16 or bfloat16, in float32. Logogram reports the rank
+   correlation, the overlap of the top components and the components whose conclusion changed,
+   and flags them on the map.
 
 Keyboard: arrow keys move across the map, **Ctrl/⌘ K** opens the command palette (type `L9H9` to
 jump to a head), **1–7** switch views, **[** and **]** step through prompts, **P**, **B**, **A**
@@ -229,7 +269,11 @@ estimate misses saturation (in attention, normalization and the final softmax) a
 even invert an effect, so **Verify top 10 by patching** on the results page patches the sites
 with the largest estimated effects for real and opens the comparison: the rank correlation, and
 any estimate whose sign patching confidently reverses. **Check robustness** can also patch the
-whole sweep.
+whole sweep. Do that for residual stream sweeps: an estimate is least reliable where patching
+replaces a whole token's representation. In the IOI example on GPT-2 small, patching the residual
+stream at the changed name in the first layer restores the whole answer while its estimate is
+slightly negative, so verifying the top estimates would never reach it. Head outputs track
+patching closely.
 
 **Direct logit attribution** splits the logit difference of the clean or the corrupt prompts
 (your choice; there is no default) into what each head, attention output and MLP output writes
@@ -266,7 +310,10 @@ like patching, so 1 means the steered prompts moved as far as switching to the o
 random direction of the same length, at the same strengths, runs alongside as a control, and the
 results show both: a layer × strength map with the control columns muted, and, for a selected site,
 its effect at every strength with intervals. **Check robustness** offers another split of the pairs
-or the other direction.
+or the other direction. A mean difference steers only when the pairs differ the same way. In IOI
+prompts that mix the ABBA and BABA orders, the difference at the last token flips with the order,
+so the mean cancels and steering does no more than the control; generate prompts of one order to
+steer. When no site and strength does more than the control, the results say so.
 
 **SAE features.** A sparse autoencoder (SAE) rewrites one of the model's activations as a few active
 features out of thousands, each a direction in the model, plus an error it misses. In
@@ -335,7 +382,7 @@ Every experiment is a `spec.json`. Nothing that can change a number is left impl
 | Field | Values |
 |---|---|
 | `model.revision` | A commit. `null` resolves the current main branch at run time; the saved spec pins what ran. |
-| `model.process_weights` | Fold LayerNorm and center weights, as TransformerLens does by default. Logit differences don't change; zero ablation of head outputs does, because value biases are folded. |
+| `model.process_weights` | Fold LayerNorm and center weights, as TransformerLens does by default. Logit differences don't change. Value biases are folded into the attention output's bias, so zero ablation of head outputs and heads' direct effects do change (by up to several logits in Qwen 2.5, whose value biases are large); a layer's attention output doesn't. |
 | `dataset.sha256` | If set, the run refuses a dataset file that has changed. |
 | `experiment` | `{"kind": "activation_patching", "direction": "clean_to_corrupt" \| "corrupt_to_clean"}`, `{"kind": "ablation", "baseline": …}` with `{"kind": "zero"}`, `{"kind": "mean", "reference": "clean" \| "corrupt"}` or `{"kind": "resample", "pool": "clean" \| "corrupt", "donors": 10, "seed": 0}`, `{"kind": "attribution_patching", "direction": …}` (same directions as patching), `{"kind": "direct_logit_attribution", "prompts": "clean" \| "corrupt"}`, `{"kind": "path_patching", "direction": …, "receivers": [{"kind": "head", "layer": 9, "head": 9, "input": "q" \| "k" \| "v"}, {"kind": "logits"}], "freeze_mlps": false}`, or `{"kind": "steering", "apply_to": "clean" \| "corrupt", "coefficients": [-1, 1, 2], "train_fraction": 0.5, "seed": 0, "control": true}` with a scope of one residual component per layer (`layer_components`) or residual `sites`, at one token |
 | `scope` | `{"kind": "heads", "position": …}`, `{"kind": "layer_position", "site": "resid_pre", "positions": "each" \| "labels"}`, `{"kind": "layer_components", "components": ["attn_out", "mlp_out"], "position": …}` `{"kind": "sites", "sites": [{"kind": "head", "layer": 9, "head": 9, "position": {"kind": "label", "label": "end"}}]}` (feature sites are `{"kind": "sae_feature", "layer": 8, "feature": 1234, "position": …}`), or, for attribution patching, `{"kind": "features", "position": …, "top": 50}` |
@@ -435,12 +482,27 @@ TransformerLens 4 loads well over a hundred architectures, and Logogram works wi
 decoder-only language models among them: GPT-2, Pythia, Llama, Mistral, SmolLM, Qwen, Gemma,
 OLMo, Phi and more. The model dialog lists starting points, and any other Hugging Face id can be
 typed in. Before downloading, Logogram checks that TransformerLens supports the architecture and
-estimates the memory needed (weights, activations and a margin) against what is free.
+estimates the memory needed (weights, activations and a margin) against what is free. Processing
+the weights needs more for a moment while the model loads: TransformerLens works on float32
+copies, three to four times the float32 size of the weights. The estimate includes them, so a
+model that fits only without processing (Qwen 2.5 0.5B on a 6 GB GPU) says so before it loads.
+
+Use float32 whenever the model fits. float16 and bfloat16 halve the memory and run two to three
+times faster on a GPU, but they round. In the IOI example on GPT-2 small and Qwen 2.5, float16
+moved effects by at most 0.003 and bfloat16 by up to 0.03: the strongest sites stayed in place,
+but effects smaller than about 0.01 changed order. On Pythia-70m, bfloat16 doubled the clean
+logit difference and reordered the heads, and float16 overflows. Check results that matter
+against float32: **Check robustness** reruns a 16-bit run in float32 and compares the two.
+
+On Apple Silicon, **Automatic** runs models on the CPU. TransformerLens reports that Apple's MPS
+can give silently wrong results, and Logogram hasn't been checked on it yet, so MPS is used only
+when you choose it under **Device**. It is faster; check results that matter on the CPU.
 
 Rather than trusting a list, Logogram checks every model when it loads, on a short fixed input:
 
 * TransformerLens's version of the model must predict what the original model predicts. If weight
-  processing changes the predictions, loading with processed weights is refused.
+  processing changes the predictions, loading with processed weights is refused. So is a dtype in
+  which the model overflows (Pythia in float16).
 * It measures how each layer adds attention and the MLP to the residual stream: one after the
   other (sequential), or both from the same input (parallel, as in Pythia, GPT-J and Phi). Only
   sequential layers have a residual stream between attention and MLP (`resid_mid`), and the layer
@@ -448,11 +510,15 @@ Rather than trusting a list, Logogram checks every model when it loads, on a sho
 * Layer predictions are offered when the final normalization and unembedding, with any logit
   soft-capping, reproduce the model's output.
 
-GPT-2 small is the model the bundled example was made for, and the one run end to end with real
-weights. The test suite runs the sanity checks on tiny random models of the Llama, Pythia, Qwen 2,
-Gemma 2 and OLMo 2 families without downloading anything. Pythia publishes checkpoints from
-throughout training as revisions (`step1000` to `step143000`), so an experiment can be rerun at
-several points of training.
+GPT-2 small is the model the bundled example was made for. Every method has been run end to end
+with real weights on GPT-2 small (with a SAELens and an OpenAI SAE), Pythia-70m (with an
+EleutherAI SAE) and Qwen 2.5 0.5B, and the test suite runs the sanity checks on tiny random models
+of the Llama, Pythia, Qwen 2, Gemma 2 and OLMo 2 families without downloading anything. The
+example's names are single tokens for GPT-2 and Qwen but not all for Pythia; for another model,
+generate IOI prompts in the app, which keeps only names that are single tokens for it. A small
+model may not do the task at all (Pythia-70m prefers the repeated name), and its runs then say so.
+Pythia publishes checkpoints from throughout training as revisions (`step1000` to `step143000`),
+so an experiment can be rerun at several points of training.
 
 Some tokenizers, such as Qwen's, have no beginning-of-sequence token. Logogram then runs prompts
 without one, and the spec records it. Answers and distractors must still be single tokens.
@@ -501,10 +567,18 @@ push a tag with the same version, such as `v0.1.0`. The Release workflow runs ev
 that commit, builds the wheel and source distribution, and uploads them to PyPI through Trusted
 Publishing, so no token is stored anywhere. PyPI never accepts the same version twice.
 
-Before tagging, manually check a first GPT-2 download, cancel and retry it, then run the example
-on each supported compute backend. CPU and CUDA are covered by local development checks; MPS
-still needs a check on Apple Silicon. Other model families are checked when they load and in the
-tiny-model tests, but not yet run end to end with their real weights.
+Before tagging, manually check a first GPT-2 download, cancel and retry it, then run
+`scripts/validate_real_weights.py` on each supported compute backend. It runs every method on a
+real model in a temporary project and checks exact identities, agreement between methods and
+bit-identical reruns; `--compare-device` reruns the head sweep on a second device, and `--sae`
+adds SAE features. CPU and CUDA pass it. Apple's MPS still needs it run on a Mac:
+
+```bash
+uv run python scripts/validate_real_weights.py --device mps --compare-device cpu
+```
+
+Pythia and Qwen 2.5 have also been run end to end with their real weights; other families are
+checked when they load and in the tiny-model tests.
 
 Model access goes through `logogram.backends.base.ModelBackend`. TransformerLens is the only
 backend today; remote execution and other libraries can be added behind the same interface.

logogram-0.1.0/PKG-INFO → logogram-0.1.2/README.md

@@ -1,36 +1,3 @@
-Metadata-Version: 2.5
-Name: logogram
-Version: 0.1.0
-Summary: A local-first workbench for causal experiments inside language models.
-Author: The Logogram contributors
-License-Expression: MIT
-License-File: LICENSE
-Keywords: activation-patching,interpretability,mechanistic-interpretability,transformers
-Classifier: Development Status :: 4 - Beta
-Classifier: Intended Audience :: Science/Research
-Classifier: Operating System :: OS Independent
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Classifier: Programming Language :: Python :: 3.13
-Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
-Requires-Python: >=3.11
-Requires-Dist: anyio>=4
-Requires-Dist: fastapi>=0.115
-Requires-Dist: huggingface-hub>=0.23
-Requires-Dist: numpy>=1.26
-Requires-Dist: platformdirs>=4
-Requires-Dist: psutil>=5.9
-Requires-Dist: pyarrow>=15
-Requires-Dist: pydantic>=2.7
-Requires-Dist: torch>=2.4
-Requires-Dist: transformer-lens<5,>=4.0
-Requires-Dist: transformers>=4.45
-Requires-Dist: typer>=0.12
-Requires-Dist: uvicorn>=0.30
-Requires-Dist: websockets>=12
-Description-Content-Type: text/markdown
-
 # Logogram
 
 A local workbench for causal experiments inside language models.
@@ -58,9 +25,10 @@ Logogram needs Python 3.11 or newer and [uv](https://docs.astral.sh/uv/).
 uv tool install logogram
 ```
 
-Until Logogram is published on PyPI, install it from a clone of this repository with
-`uv tool install .` instead. The web app is prebuilt inside the package, so you never need Node.
-The commands in the table below work the same way with `.` in place of `logogram`.
+To run the code in a clone of this repository instead, install it with `uv tool install .`; the
+commands in the table below work the same way with `.` in place of `logogram`. The web app is
+prebuilt inside the package, so you never need Node. What changed in each version is in
+[CHANGELOG.md](CHANGELOG.md).
 
 PyTorch is chosen per machine:
 
@@ -68,7 +36,7 @@ PyTorch is chosen per machine:
 |---|---|---|
 | Linux with an NVIDIA GPU | `uv tool install logogram` | The default Linux wheels include CUDA. |
 | Windows with an NVIDIA GPU | `uv tool install --torch-backend=auto logogram` | Picks the CUDA build that matches your driver. |
-| Apple Silicon | `uv tool install logogram` | Uses Metal (MPS). Use a native arm64 Python, not one under Rosetta. |
+| Apple Silicon | `uv tool install logogram` | Runs on the CPU unless you choose MPS (see Models). Use a native arm64 Python, not one under Rosetta. |
 | CPU only | `uv tool install --torch-backend=cpu logogram` | Smaller download. GPT-2 small runs well on a CPU. |
 
 Then check the setup:
@@ -109,8 +77,9 @@ This starts a local server on 127.0.0.1, prints its address and opens your brows
 4. Press **A** on a head to see its attention pattern, or right-click any cell for **Patch here**,
    **Ablate here** and **Compare across runs**.
 5. Press **Check robustness** to rerun the sweep with a different baseline, direction or donor
-   count. Logogram reports the rank correlation, the overlap of the top components and the
-   components whose conclusion changed, and flags them on the map.
+   count, or, for a run in float16 or bfloat16, in float32. Logogram reports the rank
+   correlation, the overlap of the top components and the components whose conclusion changed,
+   and flags them on the map.
 
 Keyboard: arrow keys move across the map, **Ctrl/⌘ K** opens the command palette (type `L9H9` to
 jump to a head), **1–7** switch views, **[** and **]** step through prompts, **P**, **B**, **A**
@@ -262,7 +231,11 @@ estimate misses saturation (in attention, normalization and the final softmax) a
 even invert an effect, so **Verify top 10 by patching** on the results page patches the sites
 with the largest estimated effects for real and opens the comparison: the rank correlation, and
 any estimate whose sign patching confidently reverses. **Check robustness** can also patch the
-whole sweep.
+whole sweep. Do that for residual stream sweeps: an estimate is least reliable where patching
+replaces a whole token's representation. In the IOI example on GPT-2 small, patching the residual
+stream at the changed name in the first layer restores the whole answer while its estimate is
+slightly negative, so verifying the top estimates would never reach it. Head outputs track
+patching closely.
 
 **Direct logit attribution** splits the logit difference of the clean or the corrupt prompts
 (your choice; there is no default) into what each head, attention output and MLP output writes
@@ -299,7 +272,10 @@ like patching, so 1 means the steered prompts moved as far as switching to the o
 random direction of the same length, at the same strengths, runs alongside as a control, and the
 results show both: a layer × strength map with the control columns muted, and, for a selected site,
 its effect at every strength with intervals. **Check robustness** offers another split of the pairs
-or the other direction.
+or the other direction. A mean difference steers only when the pairs differ the same way. In IOI
+prompts that mix the ABBA and BABA orders, the difference at the last token flips with the order,
+so the mean cancels and steering does no more than the control; generate prompts of one order to
+steer. When no site and strength does more than the control, the results say so.
 
 **SAE features.** A sparse autoencoder (SAE) rewrites one of the model's activations as a few active
 features out of thousands, each a direction in the model, plus an error it misses. In
@@ -368,7 +344,7 @@ Every experiment is a `spec.json`. Nothing that can change a number is left impl
 | Field | Values |
 |---|---|
 | `model.revision` | A commit. `null` resolves the current main branch at run time; the saved spec pins what ran. |
-| `model.process_weights` | Fold LayerNorm and center weights, as TransformerLens does by default. Logit differences don't change; zero ablation of head outputs does, because value biases are folded. |
+| `model.process_weights` | Fold LayerNorm and center weights, as TransformerLens does by default. Logit differences don't change. Value biases are folded into the attention output's bias, so zero ablation of head outputs and heads' direct effects do change (by up to several logits in Qwen 2.5, whose value biases are large); a layer's attention output doesn't. |
 | `dataset.sha256` | If set, the run refuses a dataset file that has changed. |
 | `experiment` | `{"kind": "activation_patching", "direction": "clean_to_corrupt" \| "corrupt_to_clean"}`, `{"kind": "ablation", "baseline": …}` with `{"kind": "zero"}`, `{"kind": "mean", "reference": "clean" \| "corrupt"}` or `{"kind": "resample", "pool": "clean" \| "corrupt", "donors": 10, "seed": 0}`, `{"kind": "attribution_patching", "direction": …}` (same directions as patching), `{"kind": "direct_logit_attribution", "prompts": "clean" \| "corrupt"}`, `{"kind": "path_patching", "direction": …, "receivers": [{"kind": "head", "layer": 9, "head": 9, "input": "q" \| "k" \| "v"}, {"kind": "logits"}], "freeze_mlps": false}`, or `{"kind": "steering", "apply_to": "clean" \| "corrupt", "coefficients": [-1, 1, 2], "train_fraction": 0.5, "seed": 0, "control": true}` with a scope of one residual component per layer (`layer_components`) or residual `sites`, at one token |
 | `scope` | `{"kind": "heads", "position": …}`, `{"kind": "layer_position", "site": "resid_pre", "positions": "each" \| "labels"}`, `{"kind": "layer_components", "components": ["attn_out", "mlp_out"], "position": …}` `{"kind": "sites", "sites": [{"kind": "head", "layer": 9, "head": 9, "position": {"kind": "label", "label": "end"}}]}` (feature sites are `{"kind": "sae_feature", "layer": 8, "feature": 1234, "position": …}`), or, for attribution patching, `{"kind": "features", "position": …, "top": 50}` |
@@ -468,12 +444,27 @@ TransformerLens 4 loads well over a hundred architectures, and Logogram works wi
 decoder-only language models among them: GPT-2, Pythia, Llama, Mistral, SmolLM, Qwen, Gemma,
 OLMo, Phi and more. The model dialog lists starting points, and any other Hugging Face id can be
 typed in. Before downloading, Logogram checks that TransformerLens supports the architecture and
-estimates the memory needed (weights, activations and a margin) against what is free.
+estimates the memory needed (weights, activations and a margin) against what is free. Processing
+the weights needs more for a moment while the model loads: TransformerLens works on float32
+copies, three to four times the float32 size of the weights. The estimate includes them, so a
+model that fits only without processing (Qwen 2.5 0.5B on a 6 GB GPU) says so before it loads.
+
+Use float32 whenever the model fits. float16 and bfloat16 halve the memory and run two to three
+times faster on a GPU, but they round. In the IOI example on GPT-2 small and Qwen 2.5, float16
+moved effects by at most 0.003 and bfloat16 by up to 0.03: the strongest sites stayed in place,
+but effects smaller than about 0.01 changed order. On Pythia-70m, bfloat16 doubled the clean
+logit difference and reordered the heads, and float16 overflows. Check results that matter
+against float32: **Check robustness** reruns a 16-bit run in float32 and compares the two.
+
+On Apple Silicon, **Automatic** runs models on the CPU. TransformerLens reports that Apple's MPS
+can give silently wrong results, and Logogram hasn't been checked on it yet, so MPS is used only
+when you choose it under **Device**. It is faster; check results that matter on the CPU.
 
 Rather than trusting a list, Logogram checks every model when it loads, on a short fixed input:
 
 * TransformerLens's version of the model must predict what the original model predicts. If weight
-  processing changes the predictions, loading with processed weights is refused.
+  processing changes the predictions, loading with processed weights is refused. So is a dtype in
+  which the model overflows (Pythia in float16).
 * It measures how each layer adds attention and the MLP to the residual stream: one after the
   other (sequential), or both from the same input (parallel, as in Pythia, GPT-J and Phi). Only
   sequential layers have a residual stream between attention and MLP (`resid_mid`), and the layer
@@ -481,11 +472,15 @@ Rather than trusting a list, Logogram checks every model when it loads, on a sho
 * Layer predictions are offered when the final normalization and unembedding, with any logit
   soft-capping, reproduce the model's output.
 
-GPT-2 small is the model the bundled example was made for, and the one run end to end with real
-weights. The test suite runs the sanity checks on tiny random models of the Llama, Pythia, Qwen 2,
-Gemma 2 and OLMo 2 families without downloading anything. Pythia publishes checkpoints from
-throughout training as revisions (`step1000` to `step143000`), so an experiment can be rerun at
-several points of training.
+GPT-2 small is the model the bundled example was made for. Every method has been run end to end
+with real weights on GPT-2 small (with a SAELens and an OpenAI SAE), Pythia-70m (with an
+EleutherAI SAE) and Qwen 2.5 0.5B, and the test suite runs the sanity checks on tiny random models
+of the Llama, Pythia, Qwen 2, Gemma 2 and OLMo 2 families without downloading anything. The
+example's names are single tokens for GPT-2 and Qwen but not all for Pythia; for another model,
+generate IOI prompts in the app, which keeps only names that are single tokens for it. A small
+model may not do the task at all (Pythia-70m prefers the repeated name), and its runs then say so.
+Pythia publishes checkpoints from throughout training as revisions (`step1000` to `step143000`),
+so an experiment can be rerun at several points of training.
 
 Some tokenizers, such as Qwen's, have no beginning-of-sequence token. Logogram then runs prompts
 without one, and the spec records it. Answers and distractors must still be single tokens.
@@ -534,10 +529,18 @@ push a tag with the same version, such as `v0.1.0`. The Release workflow runs ev
 that commit, builds the wheel and source distribution, and uploads them to PyPI through Trusted
 Publishing, so no token is stored anywhere. PyPI never accepts the same version twice.
 
-Before tagging, manually check a first GPT-2 download, cancel and retry it, then run the example
-on each supported compute backend. CPU and CUDA are covered by local development checks; MPS
-still needs a check on Apple Silicon. Other model families are checked when they load and in the
-tiny-model tests, but not yet run end to end with their real weights.
+Before tagging, manually check a first GPT-2 download, cancel and retry it, then run
+`scripts/validate_real_weights.py` on each supported compute backend. It runs every method on a
+real model in a temporary project and checks exact identities, agreement between methods and
+bit-identical reruns; `--compare-device` reruns the head sweep on a second device, and `--sae`
+adds SAE features. CPU and CUDA pass it. Apple's MPS still needs it run on a Mac:
+
+```bash
+uv run python scripts/validate_real_weights.py --device mps --compare-device cpu
+```
+
+Pythia and Qwen 2.5 have also been run end to end with their real weights; other families are
+checked when they load and in the tiny-model tests.
 
 Model access goes through `logogram.backends.base.ModelBackend`. TransformerLens is the only
 backend today; remote execution and other libraries can be added behind the same interface.

{logogram-0.1.0 → logogram-0.1.2}/pyproject.toml

@@ -20,6 +20,7 @@ classifiers = [
     "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",
     "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
     "Topic :: Scientific/Engineering :: Artificial Intelligence",
 ]
 dependencies = [
@@ -39,13 +40,19 @@ dependencies = [
     "huggingface-hub>=0.23",
 ]
 
+[project.urls]
+Homepage = "https://github.com/Jeevash23/logogram"
+Source = "https://github.com/Jeevash23/logogram"
+Issues = "https://github.com/Jeevash23/logogram/issues"
+Changelog = "https://github.com/Jeevash23/logogram/blob/main/CHANGELOG.md"
+
 [project.scripts]
 logogram = "logogram.cli:main"
 
 [dependency-groups]
 dev = [
     "pytest>=8",
-    "httpx>=0.27",
+    "httpx2>=2.13",
     "ruff==0.16.10",
 ]
 
@@ -58,7 +65,7 @@ packages = ["src/logogram"]
 artifacts = ["src/logogram/web_dist/**"]
 
 [tool.hatch.build.targets.sdist]
-include = ["src/logogram", "web/src", "web/public", "web/tests", "web/package.json", "web/package-lock.json", "web/index.html", "web/tsconfig.json", "web/vite.config.ts", "web/playwright.config.ts", "uv.lock", "tests", "scripts", "README.md", "LICENSE", "AGENTS.md"]
+include = ["src/logogram", "web/src", "web/public", "web/tests", "web/package.json", "web/package-lock.json", "web/index.html", "web/tsconfig.json", "web/vite.config.ts", "web/playwright.config.ts", "uv.lock", "tests", "scripts", "README.md", "CHANGELOG.md", "LICENSE", "AGENTS.md"]
 artifacts = ["src/logogram/web_dist/**"]
 
 [tool.pytest.ini_options]