circuitkit 0.1.10__tar.gz → 0.2.0__tar.gz

{circuitkit-0.1.10 → circuitkit-0.2.0}/.github/workflows/patch-release.yml

@@ -71,9 +71,11 @@ jobs:
             latest_version=$(python -c \
               'import json,sys; print(json.load(open(sys.argv[1]))["tag_name"])' \
               "$latest_response")
+            # the code's own version when it is ahead of the latest release (a deliberate bump,
+            # e.g. 0.2.0 after 0.1.10); otherwise the next patch of the latest release
             requested_version=$(python -c \
-              'import sys; from scripts.release import next_patch; print(next_patch(sys.argv[1]))' \
-              "$latest_version")
+              'import sys; from scripts.release import release_version; print(release_version(sys.argv[1], sys.argv[2]))' \
+              "$current_version" "$latest_version")
 
             if git show-ref --verify --quiet "refs/tags/$latest_version"; then
               latest_commit=$(git rev-list -n 1 "$latest_version")
@@ -106,7 +108,7 @@ jobs:
             version_tree=$(mktemp -d)
             mkdir -p "$version_tree/src/circuitkit" "$version_tree/scripts"
             cp scripts/release.py "$version_tree/scripts/release.py"
-            for version_file in pyproject.toml src/circuitkit/__init__.py CITATION.cff; do
+            for version_file in src/circuitkit/__init__.py CITATION.cff; do
               git show "$requested_version:$version_file" > "$version_tree/$version_file"
             done
             if ! (cd "$version_tree" && python scripts/release.py validate-version "$requested_version"); then
@@ -216,7 +218,7 @@ jobs:
           current_version=$(python scripts/release.py current-version)
           if [[ "$current_version" != "$RELEASE_TAG" ]]; then
             python scripts/release.py set-version "$RELEASE_TAG"
-            expected_files=$'CITATION.cff\npyproject.toml\nsrc/circuitkit/__init__.py'
+            expected_files=$'CITATION.cff\nsrc/circuitkit/__init__.py'
             actual_files=$(git diff --name-only | sort)
             if [[ "$actual_files" != "$expected_files" ]]; then
               echo "::error::Version helper changed files outside the authoritative version set."
@@ -224,7 +226,7 @@ jobs:
             fi
             git config user.name github-actions[bot]
             git config user.email 41898282+github-actions[bot]@users.noreply.github.com
-            git add CITATION.cff pyproject.toml src/circuitkit/__init__.py
+            git add CITATION.cff src/circuitkit/__init__.py
             git commit -m "Release $RELEASE_TAG"
           fi
           python scripts/release.py validate-version "$RELEASE_TAG"

{circuitkit-0.1.10/src/circuitkit.egg-info → circuitkit-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: circuitkit
-Version: 0.1.10
+Version: 0.2.0
 Summary: Unified Discover, Evaluate, Intervene toolkit for mechanistic interpretability: 13 circuit discovery algorithms, 6-pillar faithfulness evaluation, and pruning / quantization / knowledge editing / unlearning / activation steering / contrastive weight steering interventions behind one API.
 Author: Hem Gosalia, Aditya Kasliwal, Vinay Kumar Sankarapu
 Author-email: Pratinav Seth <pratinav.seth@lexsi.ai>
@@ -12,6 +12,8 @@ Project-URL: Repository, https://github.com/Lexsi-Labs/circuitkit
 Project-URL: Bug Tracker, https://github.com/Lexsi-Labs/circuitkit/issues
 Project-URL: Changelog, https://github.com/Lexsi-Labs/circuitkit/blob/main/CHANGELOG.md
 Project-URL: Contributing, https://github.com/Lexsi-Labs/circuitkit/blob/main/CONTRIBUTING.md
+Project-URL: Discord, https://discord.gg/MhVUGVYE8q
+Project-URL: Lexsi Labs, https://lexsi.ai
 Keywords: mechanistic-interpretability,circuit-discovery,transformers,neural-networks,attention-analysis,model-pruning,faithfulness-evaluation,auto-schema,corruption-robustness,knowledge-editing,model-visualization
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Science/Research
@@ -108,6 +110,15 @@ Dynamic: license-file
   <a href="https://circuitkit.lexsi.ai/"><img src="https://img.shields.io/badge/docs-mkdocs%20material-FF4B0A.svg" alt="Docs"></a>
 </p>
 
+<p align="center">
+  <a href="https://discord.gg/MhVUGVYE8q"><img src="https://img.shields.io/badge/Discord-Join-5865F2?logo=discord&logoColor=white" alt="Discord"></a>
+  <a href="https://github.com/Lexsi-Labs"><img src="https://img.shields.io/badge/GitHub-Lexsi--Labs-171515?logo=github&logoColor=white" alt="Lexsi Labs on GitHub"></a>
+  <a href="https://x.com/Lexsi_labs"><img src="https://img.shields.io/badge/X-Follow-000000?logo=x&logoColor=white" alt="Follow on X"></a>
+  <a href="https://www.linkedin.com/company/lexsi-labs/"><img src="https://img.shields.io/badge/LinkedIn-Lexsi%20Labs-0A66C2?logo=linkedin&logoColor=white" alt="Lexsi Labs on LinkedIn"></a>
+  <a href="https://huggingface.co/Lexsi"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Lexsi-FFD21E" alt="Lexsi on Hugging Face"></a>
+  <a href="https://lexsi.ai"><img src="https://img.shields.io/badge/Website-lexsi.ai-FF4B0A.svg" alt="lexsi.ai"></a>
+</p>
+
 ---
 
 CircuitKIT is a framework for mechanistic interpretability. Given a model and a task, it discovers the circuit driving that behaviour, evaluates how faithful it is, and lets you act on it (prune, quantize, edit, steer, or fine-tune), then export a reloadable HuggingFace checkpoint.
@@ -180,19 +191,33 @@ A **circuit** is the minimal set of attention heads and MLP layers in a transfor
 
 Production support covers Llama-3, Gemma/Gemma-3, Qwen, Mistral, Phi, Falcon,
 and GPT-2 (see [Architecture Registry](https://circuitkit.lexsi.ai/advanced/architecture-registry/)).
-CircuitKIT also supports four cohere-family and Llama-family models that
-TransformerLens doesn't natively support, across all three surfaces —
-discovery, evaluation, and interventions:
+CircuitKIT also supports five cohere-family and Llama-family models that
+TransformerLens doesn't natively support. Four of them (Tiny Aya, Command
+R7B, Aya Expanse 8B, SmolLM3-3B) are validated across all three surfaces —
+discovery, evaluation, and interventions. The fifth, Aya Expanse 32B, is
+registered, parity-validated (truncated depth), and checked for pruning and
+quantization module resolution (see below):
 
 | Model | Arch | HF repo |
 |---|---|---|
 | Tiny Aya | `cohere2` | `CohereLabs/tiny-aya-*` (gated) |
 | Command R7B | `cohere2` | `CohereLabs/c4ai-command-r7b-12-2024` |
 | Aya Expanse 8B | `cohere1` | `CohereLabs/aya-expanse-8b` |
+| Aya Expanse 32B | `cohere1` | `CohereLabs/aya-expanse-32b` |
 | SmolLM3-3B | `smollm3` | `HuggingFaceTB/SmolLM3-3B` |
 
-All four support discovery (5/6 stable algorithms plus parity), faithfulness
-evaluation, and interventions (pruning, quant resolution, steering).
+All of them except Aya Expanse 32B support
+discovery (5/6 stable algorithms plus parity), faithfulness evaluation, and
+interventions (pruning, quant resolution, steering), validated on real
+weights. **Aya Expanse 32B** is registered, with config conversion checked
+against the real checkpoint (truncated-depth parity) and module resolution
+for pruning scores and quantization targets checked on the full checkpoint.
+Full-depth parity, discovery, evaluation and weight steering do not run yet:
+the full 40-layer model needs about 64 GiB for weights alone, so it does not
+fit on most single GPUs, and transformer-lens 3.8.0's multi-GPU (`n_devices`)
+block placement is broken. See
+[Experimental Models](https://circuitkit.lexsi.ai/advanced/experimental-models/)
+for what was and wasn't validated, and the details of the transformer-lens bug.
 
 Two caveats on that support. `acdc` is excluded from the standard gate as
 impractically slow on 3–8B models — its test lives behind an opt-in flag — so
@@ -206,7 +231,7 @@ TransformerLens ports; this is not a claim of end-to-end CircuitKIT support.
 Their hardware and Sarvam loading requirements are documented in
 [Experimental Models](https://github.com/Lexsi-Labs/CircuitKIT/blob/main/docs/advanced/experimental-models.md#gemma-4-and-sarvam-moe-discovery-only-transformerlens-ports).
 
-All four are **experimental** — see [Tiny Aya](https://circuitkit.lexsi.ai/advanced/tiny-aya/)
+All five are **experimental** — see [Tiny Aya](https://circuitkit.lexsi.ai/advanced/tiny-aya/)
 and [Experimental Models](https://circuitkit.lexsi.ai/advanced/experimental-models/)
 for architecture details, config mapping, and the full opt-in test matrix.
 Aya Expanse and Tiny Aya also load from local checkpoint directories and

{circuitkit-0.1.10 → circuitkit-0.2.0}/README.md

@@ -20,6 +20,15 @@
   <a href="https://circuitkit.lexsi.ai/"><img src="https://img.shields.io/badge/docs-mkdocs%20material-FF4B0A.svg" alt="Docs"></a>
 </p>
 
+<p align="center">
+  <a href="https://discord.gg/MhVUGVYE8q"><img src="https://img.shields.io/badge/Discord-Join-5865F2?logo=discord&logoColor=white" alt="Discord"></a>
+  <a href="https://github.com/Lexsi-Labs"><img src="https://img.shields.io/badge/GitHub-Lexsi--Labs-171515?logo=github&logoColor=white" alt="Lexsi Labs on GitHub"></a>
+  <a href="https://x.com/Lexsi_labs"><img src="https://img.shields.io/badge/X-Follow-000000?logo=x&logoColor=white" alt="Follow on X"></a>
+  <a href="https://www.linkedin.com/company/lexsi-labs/"><img src="https://img.shields.io/badge/LinkedIn-Lexsi%20Labs-0A66C2?logo=linkedin&logoColor=white" alt="Lexsi Labs on LinkedIn"></a>
+  <a href="https://huggingface.co/Lexsi"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Lexsi-FFD21E" alt="Lexsi on Hugging Face"></a>
+  <a href="https://lexsi.ai"><img src="https://img.shields.io/badge/Website-lexsi.ai-FF4B0A.svg" alt="lexsi.ai"></a>
+</p>
+
 ---
 
 CircuitKIT is a framework for mechanistic interpretability. Given a model and a task, it discovers the circuit driving that behaviour, evaluates how faithful it is, and lets you act on it (prune, quantize, edit, steer, or fine-tune), then export a reloadable HuggingFace checkpoint.
@@ -92,19 +101,33 @@ A **circuit** is the minimal set of attention heads and MLP layers in a transfor
 
 Production support covers Llama-3, Gemma/Gemma-3, Qwen, Mistral, Phi, Falcon,
 and GPT-2 (see [Architecture Registry](https://circuitkit.lexsi.ai/advanced/architecture-registry/)).
-CircuitKIT also supports four cohere-family and Llama-family models that
-TransformerLens doesn't natively support, across all three surfaces —
-discovery, evaluation, and interventions:
+CircuitKIT also supports five cohere-family and Llama-family models that
+TransformerLens doesn't natively support. Four of them (Tiny Aya, Command
+R7B, Aya Expanse 8B, SmolLM3-3B) are validated across all three surfaces —
+discovery, evaluation, and interventions. The fifth, Aya Expanse 32B, is
+registered, parity-validated (truncated depth), and checked for pruning and
+quantization module resolution (see below):
 
 | Model | Arch | HF repo |
 |---|---|---|
 | Tiny Aya | `cohere2` | `CohereLabs/tiny-aya-*` (gated) |
 | Command R7B | `cohere2` | `CohereLabs/c4ai-command-r7b-12-2024` |
 | Aya Expanse 8B | `cohere1` | `CohereLabs/aya-expanse-8b` |
+| Aya Expanse 32B | `cohere1` | `CohereLabs/aya-expanse-32b` |
 | SmolLM3-3B | `smollm3` | `HuggingFaceTB/SmolLM3-3B` |
 
-All four support discovery (5/6 stable algorithms plus parity), faithfulness
-evaluation, and interventions (pruning, quant resolution, steering).
+All of them except Aya Expanse 32B support
+discovery (5/6 stable algorithms plus parity), faithfulness evaluation, and
+interventions (pruning, quant resolution, steering), validated on real
+weights. **Aya Expanse 32B** is registered, with config conversion checked
+against the real checkpoint (truncated-depth parity) and module resolution
+for pruning scores and quantization targets checked on the full checkpoint.
+Full-depth parity, discovery, evaluation and weight steering do not run yet:
+the full 40-layer model needs about 64 GiB for weights alone, so it does not
+fit on most single GPUs, and transformer-lens 3.8.0's multi-GPU (`n_devices`)
+block placement is broken. See
+[Experimental Models](https://circuitkit.lexsi.ai/advanced/experimental-models/)
+for what was and wasn't validated, and the details of the transformer-lens bug.
 
 Two caveats on that support. `acdc` is excluded from the standard gate as
 impractically slow on 3–8B models — its test lives behind an opt-in flag — so
@@ -118,7 +141,7 @@ TransformerLens ports; this is not a claim of end-to-end CircuitKIT support.
 Their hardware and Sarvam loading requirements are documented in
 [Experimental Models](https://github.com/Lexsi-Labs/CircuitKIT/blob/main/docs/advanced/experimental-models.md#gemma-4-and-sarvam-moe-discovery-only-transformerlens-ports).
 
-All four are **experimental** — see [Tiny Aya](https://circuitkit.lexsi.ai/advanced/tiny-aya/)
+All five are **experimental** — see [Tiny Aya](https://circuitkit.lexsi.ai/advanced/tiny-aya/)
 and [Experimental Models](https://circuitkit.lexsi.ai/advanced/experimental-models/)
 for architecture details, config mapping, and the full opt-in test matrix.
 Aya Expanse and Tiny Aya also load from local checkpoint directories and

{circuitkit-0.1.10 → circuitkit-0.2.0}/docs/API_REFERENCE.md

@@ -175,9 +175,10 @@ Orchestrates the pillars end-to-end, ordered by cost (fast pillars first).
 
 Key parameters:
 
-- `pillars` (List[str], optional): Subset to run. Default: all of
-  `["patching", "ablation", "baselines", "robustness", "stability", "generalization", "intervention_reliability"]`.
-- `n_stability_runs` (int): Discovery runs for Pillar 3. Default 5.
+- `pillars` (List[str], optional): Subset to run. Default of this low-level function: all of
+  `["patching", "ablation", "baselines", "robustness", "stability", "generalization", "intervention_reliability"]`
+  (`Pipeline.evaluate` and `ck.faithfulness` default to `["patching", "ablation"]`).
+- `n_stability_runs` (int): Discovery runs for Pillar 3. Default 3.
 - `n_reliability_seeds` (int): Seeds for the optional intervention-reliability pillar.
   Default 3.
 - `target_task_spec` / `target_dataloader`: Required for Pillar 6; if omitted, Pillar 6

{circuitkit-0.1.10 → circuitkit-0.2.0}/docs/about/faq.md

@@ -22,7 +22,7 @@ The six Stable-tier discovery algorithms (`eap`, `eap-ig`, `eap-gp`, `acdc`, `ib
 
 **Which algorithm should I use?**
 
-Start with `eap-ig` (the default). It is the most accurate stable algorithm. For speed over accuracy, use `eap`. For memory-constrained settings, reduce `ig_steps` to 3. See [Algorithm Selection Guide](../algorithms/overview.md).
+Start with `eap-ig` (the default). It is the most accurate stable algorithm. For speed over accuracy, use `eap`. For memory-constrained settings, lower `batch_size` first (`ig_steps` already defaults to 3). See [Algorithm Selection Guide](../algorithms/overview.md).
 
 **How many examples do I need?**
 
@@ -74,7 +74,7 @@ Production: `llama`, `qwen`, `gemma`. Ready: `mistral`, `phi`, `falcon`, `gpt2`.
 
 **How do I reduce memory usage for large models?**
 
-Set `model.precision="bfloat16"`, `ig_steps=3`, `batch_size=1`, and lower `n_examples`. Combined, these reduce peak VRAM by about 80%. There is no `memory_efficient` or `use_half_precision_activations` config key — those are not read anywhere in the discovery backend. See [Memory Optimization](../advanced/memory-optimization.md).
+Set `model.precision="bfloat16"`, `batch_size=1`, and lower `n_examples` (`ig_steps` already defaults to 3). Measured against the previous defaults (`ig_steps=5`, `batch_size=4`, 128 examples), the combination reduced peak VRAM by about 80%. There is no `memory_efficient` or `use_half_precision_activations` config key — those are not read anywhere in the discovery backend. See [Memory Optimization](../advanced/memory-optimization.md).
 
 **How long does discovery take?**
 

{circuitkit-0.1.10 → circuitkit-0.2.0}/docs/advanced/architecture-registry.md

@@ -16,10 +16,25 @@ CircuitKIT's **architecture registry** provides a unified interface for pruning
 | `phi` | Ready | `microsoft/Phi-3-mini-4k-instruct` |
 | `falcon` | Ready | `tiiuae/falcon-7b` |
 | `gpt2` | Ready | `gpt2`, `gpt2-xl` |
-| `cohere` | Experimental | `CohereLabs/tiny-aya-base`, `CohereLabs/c4ai-command-r7b-12-2024` (`cohere2`); `CohereLabs/aya-expanse-8b` (`cohere1`) |
+| `cohere` | Experimental | `CohereLabs/tiny-aya-base`, `CohereLabs/c4ai-command-r7b-12-2024` (`cohere2`); `CohereLabs/aya-expanse-8b`, `CohereLabs/aya-expanse-32b` (`cohere1`) |
 | `smollm3` | Experimental | `HuggingFaceTB/SmolLM3-3B` |
 
-**Production** = validated in the CircuitKIT paper audit. **Ready** = registry entry exists, high confidence, not in the audit. **Experimental** = discovery, evaluation, and interventions supported via the circuitkit TransformerLens port (`circuitkit.backends._tl_compat`), validated on real weights (see below for exactly what "validated" means for each surface). The `cohere` family now covers three checkpoints across two HF architectures — Tiny Aya and Command R7B (`Cohere2ForCausalLM`, `model_type="cohere2"`) and Aya Expanse (`CohereForCausalLM`, `model_type="cohere"`) — sharing one weight converter and one registry entry, since the pruning/quantization/steering module layout (`self_attn.{q,k,v,o}_proj`, `mlp.{gate,up,down}_proj`) is identical across all three. `smollm3` is a separate, Llama-shaped family (new in this stage). See [Tiny Aya (Cohere2)](tiny-aya.md) and [Experimental Models](experimental-models.md).
+**Production** = validated in the CircuitKIT paper audit. **Ready** = registry entry exists, high confidence, not in the audit. **Experimental** = discovery, evaluation, and interventions supported via the circuitkit TransformerLens port (`circuitkit.backends._tl_compat`), validated on real weights (see below for exactly what "validated" means for each surface — Aya Expanse 32B is a partial exception, noted below). The `cohere` family now covers four checkpoints across two HF architectures — Tiny Aya and Command R7B (`Cohere2ForCausalLM`, `model_type="cohere2"`) and Aya Expanse 8B / 32B (`CohereForCausalLM`, `model_type="cohere"`) — sharing one weight converter and one registry entry, since the pruning/quantization/steering module layout (`self_attn.{q,k,v,o}_proj`, `mlp.{gate,up,down}_proj`) is identical across all four. `smollm3` is a separate, Llama-shaped family (new in this stage). See [Tiny Aya (Cohere2)](tiny-aya.md) and [Experimental Models](experimental-models.md).
+
+**Aya Expanse 32B is only partially validated.** Registration, real-weight
+config conversion (at a truncated depth), and — of the registry-entry
+resolution this page documents — **pruning score extraction and
+quantization pattern matching** are confirmed on the real 32B checkpoint;
+see [Experimental Models](experimental-models.md#tests) for the actual
+numbers. **Weight-steering slices** have not: weight steering needs the
+full 40-layer model loaded through TransformerLens, which needs about
+64 GiB for weights alone and so multi-GPU (`n_devices`) loading, and that
+is broken in transformer-lens==3.8.0. See the warning in
+[Experimental Models](experimental-models.md#tests) for the mechanism.
+Everywhere below that says "all four real loaded
+models" for the `cohere` family, that means Tiny Aya, Command R7B, Aya
+Expanse 8B and SmolLM3-3B — Aya Expanse 32B's coverage is pruning and
+quantization only, not weight steering.
 
 ### Two honest caveats (apply to `cohere` and `smollm3` alike)
 

{circuitkit-0.1.10 → circuitkit-0.2.0}/docs/advanced/experimental-models.md

@@ -13,9 +13,20 @@ rotary (NoPE) policy. Three more model families now ride that seam:
 | [Tiny Aya](tiny-aya.md) | `Cohere2ForCausalLM` | `cohere2` | `_tl_compat/cohere.py` |
 | **Command R7B** | `Cohere2ForCausalLM` | `cohere2` | `_tl_compat/cohere.py` |
 | **Aya Expanse 8B** | `CohereForCausalLM` | `cohere` | `_tl_compat/cohere.py` |
+| **Aya Expanse 32B** | `CohereForCausalLM` | `cohere` | `_tl_compat/cohere.py` |
 | **SmolLM3-3B** | `SmolLM3ForCausalLM` | `smollm3` | `_tl_compat/smollm3.py` |
 
-All four are **experimental** discovery/evaluation/intervention targets:
+Aya Expanse 32B shares Aya Expanse 8B's config/weight converter unchanged —
+registration, real-weight config conversion, and **module resolution for
+pruning score extraction and quantization targets** are validated on the real
+checkpoint (see [Tests](#tests)).
+Full-depth bf16 parity, discovery, evaluation, and weight steering do
+**not** run yet. A loaded TransformerLens copy needs about 64 GiB for
+weights alone, so it does not fit on most single GPUs, and multi-GPU
+loading through `n_devices` is affected by a bug in transformer-lens==3.8.0
+(see the warning in [Tests](#tests)). The other four
+models on this page are **experimental** discovery/evaluation/intervention
+targets:
 TransformerLens 3.8.0 has no native support for any of `cohere`, `cohere2`, or
 `smollm3`; the port adds it additively, at import time, with every patch
 falling through to stock TL for any other architecture. Command R7B and Aya
@@ -56,31 +67,42 @@ support.
 
 - `CohereLabs/c4ai-command-r7b-12-2024` (Command R7B, `cohere2`)
 - `CohereLabs/aya-expanse-8b` (Aya Expanse 8B, `cohere1`/`cohere`)
+- `CohereLabs/aya-expanse-32b` (Aya Expanse 32B, `cohere1`/`cohere`) —
+  registration, config conversion, truncated-depth parity and pruning /
+  quantization module resolution validated on real weights; full-depth
+  parity, discovery, evaluation and weight steering do not run yet (see
+  [Tests](#tests))
 - `HuggingFaceTB/SmolLM3-3B` (SmolLM3-3B, `smollm3`)
 
 ---
 
 ## Architecture at a glance
 
-| Property | Command R7B | Aya Expanse 8B | SmolLM3-3B |
-|---|---|---|---|
-| HF architecture | `Cohere2ForCausalLM` | `CohereForCausalLM` | `SmolLM3ForCausalLM` |
-| Params | ~7B | ~8B | ~3B |
-| `d_model` / layers | 4096 / 32 | 4096 / 32 | 2048 / 36 |
-| Query / KV heads | 32 / 8 (GQA) | 32 / 8 (GQA) | 16 / 4 (GQA) |
-| `d_head` | 128 | 128 | 128 |
-| `d_mlp` (gated SiLU) | 14336 | 14336 | 11008 |
-| Normalization | LayerNorm (bias-less) | LayerNorm (bias-less) | **RMSNorm** |
-| Block topology | **parallel** attn+MLP, one input LN | **parallel** attn+MLP, one input LN | **sequential** (standard Llama) |
-| RoPE style | GPT-J interleaved (`rotary_adjacent_pairs=True`) | interleaved (`rotary_adjacent_pairs=True`) | **standard Llama**, non-interleaved (`rotary_adjacent_pairs=False`) |
-| `rope_theta` | 50000 | 10000 | 5000000 |
-| Attention pattern | **sliding-window (4096) ⨯ full**, interleaved | all full attention | all full attention |
-| NoPE (no rotary) layers | full-attention layers | **none** — rotary everywhere | **per-layer**, driven by config's `no_rope_layers` (every 4th layer on the real checkpoint) |
-| `logit_scale` | 0.25 (real fold) | 0.125 (real fold) | none (no post-multiply at all) |
-| Tied embeddings | yes | yes | yes |
-| Vocab | 256000 | 256000 | 128256 |
-| Max position embeddings (real ckpt) | 132096 (preserved) | 8192 | 65536 (preserved) |
-| Gated on the Hub? | `gated=auto` (license click-through) | `gated=auto` (license click-through) | fully public |
+| Property | Command R7B | Aya Expanse 8B | Aya Expanse 32B | SmolLM3-3B |
+|---|---|---|---|---|
+| HF architecture | `Cohere2ForCausalLM` | `CohereForCausalLM` | `CohereForCausalLM` | `SmolLM3ForCausalLM` |
+| Params | ~7B | ~8B | ~32B | ~3B |
+| `d_model` / layers | 4096 / 32 | 4096 / 32 | 8192 / 40 | 2048 / 36 |
+| Query / KV heads | 32 / 8 (GQA, 4:1) | 32 / 8 (GQA, 4:1) | 64 / 8 (GQA, **8:1**) | 16 / 4 (GQA) |
+| `d_head` | 128 | 128 | 128 (not an explicit config field; `hidden_size // n_heads`) | 128 |
+| `d_mlp` (gated SiLU) | 14336 | 14336 | 24576 | 11008 |
+| Normalization | LayerNorm (bias-less) | LayerNorm (bias-less) | LayerNorm (bias-less) | **RMSNorm** |
+| Block topology | **parallel** attn+MLP, one input LN | **parallel** attn+MLP, one input LN | **parallel** attn+MLP, one input LN | **sequential** (standard Llama) |
+| RoPE style | GPT-J interleaved (`rotary_adjacent_pairs=True`) | interleaved (`rotary_adjacent_pairs=True`) | interleaved (`rotary_adjacent_pairs=True`) | **standard Llama**, non-interleaved (`rotary_adjacent_pairs=False`) |
+| `rope_theta` | 50000 | 10000 | **4,000,000** | 5000000 |
+| Attention pattern | **sliding-window (4096) ⨯ full**, interleaved | all full attention | all full attention | all full attention |
+| NoPE (no rotary) layers | full-attention layers | **none** — rotary everywhere | **none** — rotary everywhere | **per-layer**, driven by config's `no_rope_layers` (every 4th layer on the real checkpoint) |
+| `logit_scale` | 0.25 (real fold) | 0.125 (real fold) | **0.0625** (real fold) | none (no post-multiply at all) |
+| Tied embeddings | yes | yes | yes | yes |
+| Vocab | 256000 | 256000 | 256000 | 128256 |
+| Max position embeddings (real ckpt) | 132096 (preserved) | 8192 | 8192 | 65536 (preserved) |
+| Checkpoint size | — | ~15 GiB (fp16, 4 shards) | ~60.2 GiB (fp16, 14 shards) | — |
+| Gated on the Hub? | `gated=auto` (license click-through) | `gated=auto` (license click-through) | `gated=auto` (license click-through) | fully public |
+
+Aya Expanse 32B's tokenizer is byte-identical to 8B's (confirmed via
+`tokenizer.json` SHA-256, both starting `c69a7ea6c0927dfa`), so the existing
+single-token task guards (`greater_than`'s own operand pool, `ioi`'s name
+check) behave the same way they do for 8B.
 
 For reference, [Tiny Aya](tiny-aya.md) is 36 layers / `d_model=2048` / 16:4
 GQA heads / `d_mlp=11008`, the same `cohere2` shape as Command R7B but with
@@ -145,6 +167,14 @@ explicit `head_dim`, so this is a separate function rather than a branch):
 | `logit_scale=0.125` | folded into `unembed.W_U` |
 | `use_qk_norm=True` (unused today) | raises `NotImplementedError` — the weight converter has no q/k-norm conversion path |
 
+**Aya Expanse 32B** uses the exact same `_convert_cohere1_config` function
+and `convert_cohere2_weights` weight converter as 8B — no new code, only a
+second repo ID in `AYA_EXPANSE_MODEL_NAMES`. Only the config *values* differ:
+`rope_theta=4,000,000`, `logit_scale=0.0625`, 64/8 GQA heads, `d_mlp=24576`,
+40 layers. Validated on real weights at a truncated depth (see
+[Tests](#tests)); the mapping table above is identical in structure to 8B's,
+just with 32B's numbers.
+
 **SmolLM3-3B** — a new Llama-family mapping, via `_convert_smollm3_config`:
 
 | HF / spec | TL `HookedTransformerConfig` |
@@ -245,6 +275,10 @@ Offline unit tests (no network, no GPU, no gated weights) run in ordinary CI:
 - `tests/backends/tl_compat/test_aya_expanse.py` — registration, the cohere1
   config converter (no sliding window / `attn_types`, `head_dim` fallback,
   `use_qk_norm` guard), rotary-everywhere NoPE policy.
+- `tests/backends/tl_compat/test_aya_expanse_32b.py` — the same coverage at
+  32B's real spec (40 layers, 8:1 GQA, `rope_theta=4e6`, `logit_scale=0.0625`),
+  plus a tiny random HF model built with those specific values matching
+  HookedTransformer logits to 1e-5.
 - `tests/backends/tl_compat/test_smollm3.py` (41 cases) — registration,
   config/weight conversion, **per-layer NoPE** (`TestSmolLM3PerLayerNoPE`,
   parametrized across NoPE and non-NoPE layer indices) and
@@ -287,6 +321,62 @@ CIRCUITKIT_RUN_AYA_EXPANSE=1 CIRCUITKIT_RUN_AYA_EXPANSE_ACDC=1 HF_TOKEN=... \
     python -m pytest tests/regression/test_aya_expanse_discovery.py -v -k acdc
 ```
 
+**Aya Expanse 32B** (`CIRCUITKIT_RUN_AYA_EXPANSE_32B=1`, needs `HF_TOKEN`,
+deliberately a separate flag from 8B's so an 8B opt-in never silently
+triggers a 60 GiB download):
+
+```bash
+CIRCUITKIT_RUN_AYA_EXPANSE_32B=1 HF_TOKEN=... python -m pytest \
+    tests/backends/tl_compat/test_aya_expanse_32b_parity.py \
+    tests/regression/test_aya_expanse_32b_interventions.py -v
+```
+
+This runs what passes today: truncated-depth parity and the pruning /
+quantization module-resolution checks. Everything that needs the full
+40-layer model through TransformerLens (full-depth parity, discovery,
+evaluation, weight steering) is skipped unless
+`CIRCUITKIT_RUN_AYA_EXPANSE_32B_FULL=1` is also set, because it fails on
+transformer-lens 3.8.0 (see below).
+
+!!! warning "32B: truncated-depth parity and pruning/quantization resolution pass; full-depth parity, discovery, evaluation, and weight steering are blocked by a transformer-lens bug"
+    Validated against the real checkpoint:
+
+    - **Registration** — the repo ID resolves to the same `cohere1`
+      `ArchPort` as 8B.
+    - **Config conversion** at 32B's real spec (40 layers, 8:1 GQA,
+      `rope_theta=4e6`, `logit_scale=0.0625`).
+    - **Truncated-depth parity** (first 4 of 40 real layers plus embeddings
+      and the final norm): KL(HF‖TL) 6.5e-9 to 5.3e-7, max log-prob
+      deviation 3.8e-6 to 7.6e-6, argmax agreement 4/4 on the same four
+      prompts the other models' gates use.
+    - **Pruning score extraction** (`build_importance_dict`) and
+      **quantization target resolution** — both resolve real modules
+      correctly on the full checkpoint (module-path resolution with
+      synthetic scores; no pruned or quantized model is evaluated)
+      (`tests/regression/test_aya_expanse_32b_interventions.py`).
+
+    What does **not** run yet, and why:
+
+    1. **The model does not fit on most single GPUs.** The weights are
+       about 60 GiB at fp16/bf16, and TransformerLens keeps a separate
+       `W_U` (~3.9 GiB more), so a loaded TransformerLens copy needs about
+       64 GiB for weights alone. Full-depth parity, discovery, evaluation,
+       and weight steering therefore need multi-GPU loading through
+       `n_devices`.
+    2. **`n_devices` loading is broken in transformer-lens==3.8.0.**
+       `HookedTransformer.move_model_modules_to_device` places each block
+       with `get_best_available_device(cfg)`, which picks the device with
+       the most free memory and ignores the block index. The forward pass
+       moves the residual stream per block with the index-based
+       `get_device_for_block_index(i, cfg)`. The two disagree. Symptoms:
+       `RuntimeError: Expected all tensors to be on the same device`
+       inside TransformerLens's `layer_norm.py`, or nearly the whole model
+       placed on one GPU followed by an out-of-memory error.
+
+    This is not a defect in the shared `cohere1` converter. Do not treat
+    32B as having the discovery, evaluation, or weight-steering coverage
+    the other four models on this page have.
+
 **SmolLM3-3B** (`CIRCUITKIT_RUN_SMOLLM3=1`, public repo, no `HF_TOKEN`
 needed):
 
@@ -317,6 +407,12 @@ CIRCUITKIT_RUN_SMOLLM3=1 CIRCUITKIT_RUN_SMOLLM3_ACDC=1 \
   and the exact per-layer NoPE mask (`{3, 7, 11, ..., 35}` skip rotary) both
   confirmed directly on the loaded model. These figures are from manual
   real-weight GPU runs; CI does not reproduce this parity measurement.
+  Aya Expanse 32B's parity gate is a **truncated-depth** variant of this
+  same check (first 4 of 40 real layers plus embeddings and the final norm,
+  built by reading only the safetensors shards those tensors live in):
+  KL(HF‖TL) 6.5e-9 to 5.3e-7, max log-prob deviation 3.8e-6 to 7.6e-6,
+  argmax agreement 4/4. See the warning above for what this covers and
+  why the full-depth variant is blocked.
 - **Discovery** (`test_*_discovery.py`) — end-to-end `discover_circuit` on
   `greater_than` for 5 of the 6 stable algorithms (`eap`, `eap-ig`,
   `eap-gp`, `ibcircuit`, `cdt`); asserts finite, non-degenerate node scores.
@@ -333,8 +429,12 @@ CIRCUITKIT_RUN_SMOLLM3=1 CIRCUITKIT_RUN_SMOLLM3_ACDC=1 \
   CPU fallback discovery needed.
 - **Interventions** (`test_*_interventions.py`) — pruning score extraction,
   quantization target-module resolution, and weight-steering setup, all on
-  real weights. See [Architecture Registry](architecture-registry.md) for
-  exactly what was and wasn't run.
+  real weights for Command R7B/8B/SmolLM3-3B. For Aya Expanse 32B, pruning
+  and quantization resolution also pass on the real checkpoint; weight
+  steering is blocked by the same confirmed transformer-lens multi-GPU bug
+  described in the 32B warning above. See
+  [Architecture Registry](architecture-registry.md) for exactly what was and
+  wasn't run per model.
 
 ### ACDC is excluded from the standard gate
 
@@ -361,6 +461,20 @@ repo's own OOM preflight refuses the GPU attempt loudly rather than crashing
 mid-forward. `cdt` needs neither. SmolLM3-3B (~6 GB bf16) is small enough
 that its entire discovery gate ran on GPU with no CPU fallback.
 
+**Aya Expanse 32B** needs more memory than the rest of this page: about
+60 GiB of weights at fp16/bf16 (14 safetensors shards), and TransformerLens
+keeps a separate `W_U` (~3.9 GiB more), so a loaded TransformerLens copy
+needs about 64 GiB for weights alone. That does not fit on most single
+GPUs, and a full fp32 copy needs about 129 GiB. Multi-GPU loading through
+`n_devices` fails on transformer-lens==3.8.0 however much GPU memory is
+free (see the warning in [Tests](#tests)).
+The parity gate avoids the full load by never materializing more than 4
+real layers through TransformerLens. Pruning and quantization resolution
+load the HF copy directly with `device_map="auto"`, which the
+TransformerLens bug does not affect. Full-depth parity, discovery,
+evaluation, and weight steering need the `n_devices` path and do not run
+yet.
+
 ---
 
 ## See also

{circuitkit-0.1.10 → circuitkit-0.2.0}/docs/advanced/memory-optimization.md

@@ -9,14 +9,14 @@ EAP-IG can be memory-intensive, especially for large models. This page covers al
 Peak VRAM after the `perf/vram-optimizations` fixes (qkv-flag lifecycle,
 per-batch buffer release, embedding-only gradients — see the changelog):
 
-| Model | EAP-IG default (measured) | With all optimizations (measured) |
+| Model | Previous default (measured) | With all optimizations (measured) |
 |-------|----------------------------|-------------------------------------|
 | GPT-2 (124M) | 1.0 GB | 0.5 GB |
 | Llama-3.2-1B-Instruct | 8.7 GB | 4.1 GB |
 | Llama-3B | not re-measured — order-of-magnitude only | not re-measured |
 | Gemma-4B | not re-measured — order-of-magnitude only | not re-measured |
 
-"EAP-IG default" is `ig_steps=5`, `batch_size=4`, `num_examples=128`. "With
+"Previous default" is `ig_steps=5`, `batch_size=4`, `num_examples=128` (`ig_steps` now defaults to 3). "With
 all optimizations" is `ig_steps=3`, `batch_size=1`, `num_examples=64` (the
 combined strategy below), both bf16. (The config key is `num_examples`, under
 `data_params`; prose sometimes calls it `n_examples` — it is the same knob.)
@@ -43,13 +43,13 @@ discover_circuit({
 
 ### 2. Reduce `ig_steps` (linear savings)
 
-`ig_steps` is the number of Integrated Gradients integration steps. The config default (`utils/config.py`) is 5. The EAP-IG backend falls back to 30 only when `ig_steps` is left unset (`ig_steps is None`), but the default config always supplies 5, so that fallback does not apply through `discover_circuit`.
+`ig_steps` is the number of Integrated Gradients integration steps. The default is 3 (`utils/config.py`), which is enough for EAP-IG: the EAP-IG paper (Hanna et al., 2024) finds every value above 2 similarly faithful, and 2 unfaithful on some tasks. Cost grows linearly with the step count, so the previous default of 5 takes about 1.7x as long. The backends fall back to 3 as well when `ig_steps` is left unset (`None`).
 
 ```python
-"discovery": {"algorithm": "eap-ig", "ig_steps": 3, ...}  # 3 is minimum
+"discovery": {"algorithm": "eap-ig", "ig_steps": 3, ...}  # the default; 2 is faster but unfaithful on some tasks
 ```
 
-**Savings:** Linear with step reduction. `ig_steps=3` is ~40% faster and ~30% less memory than `ig_steps=5`.
+**Savings:** Linear with step reduction. `ig_steps=3` (the default) is ~40% faster and ~30% less memory than the previous default of `ig_steps=5`.
 
 ### 3. Reduce `batch_size` (linear savings)
 
@@ -77,7 +77,7 @@ discover_circuit({
         "algorithm": "eap-ig",
         "task": "mmlu",
         "level": "node",
-        "ig_steps": 3,                   # 2. fewer IG steps
+        "ig_steps": 3,                   # 2. IG steps (the default)
         "batch_size": 1,                 # 3. batch size 1 (top-level, not under data_params)
         "data_params": {
             "num_examples": 64,          # 4. fewer examples
@@ -97,7 +97,7 @@ discover_circuit({
 | Baseline — float32 (reference only; not the default) | 0% |
 | Model bfloat16 | ~50% |
 | + Batch size 1 | ~70–75% |
-| + Reduced ig_steps (3) | ~75–80% |
+| + ig_steps 3 (now the default; was 5) | ~75–80% |
 | **All combined** | **~80%** |
 
 There are no `memory_efficient` or `use_half_precision_activations` config keys — neither is read anywhere in the discovery backend, so setting them silently does nothing. The levers above (model precision, `batch_size`, `ig_steps`, `n_examples`) are the only ones that actually affect memory.
@@ -141,7 +141,7 @@ All optimizations have negligible impact on circuit quality:
 
 | Optimization | Speed impact | Circuit quality impact |
 |-------------|-------------|----------------------|
-| `ig_steps=3` | ~40% faster | < 0.5% score diff |
+| `ig_steps=3` (now the default; was 5) | ~40% faster | < 0.5% score diff |
 | `batch_size=1` | ~50% slower | None |
 | `bfloat16` model | Same or faster | < 0.1% score diff |
 

{circuitkit-0.1.10 → circuitkit-0.2.0}/docs/advanced/tiny-aya.md

@@ -10,9 +10,14 @@ additively, without forking or pinning a different TL version.
     Tiny-aya was the first model this port supported, and the seam it
     established (`circuitkit.backends._tl_compat`) has since grown to also
     cover **Command R7B** (same `cohere2` architecture as tiny-aya),
-    **Aya Expanse 8B** (`cohere1`), and **SmolLM3-3B** (a new Llama-family
-    port) — all discovery/evaluation/intervention-capable through the same
-    seam. See [Experimental Models](experimental-models.md) for those three.
+    **Aya Expanse 8B** and **32B** (`cohere1`), and **SmolLM3-3B** (a new
+    Llama-family port) — all discovery/evaluation/intervention-capable
+    through the same seam, except Aya Expanse 32B, which is registered,
+    parity-validated (truncated depth), and validated for pruning/
+    quantization but not discovery/evaluation/weight-steering — those are
+    blocked by a confirmed transformer-lens multi-GPU bug, not left undone
+    for time (see [Experimental Models](experimental-models.md) for what
+    was and wasn't validated and why).
 
 !!! warning "Experimental & gated"
     tiny-aya is an **experimental** discovery target. The checkpoints are
@@ -182,7 +187,7 @@ CIRCUITKIT_RUN_TINY_AYA=1 HF_TOKEN=... \
 ## See also
 
 - [Experimental Models](experimental-models.md) — Command R7B, Aya Expanse
-  8B, and SmolLM3-3B, added through the same `_tl_compat` seam.
+  8B / 32B, and SmolLM3-3B, added through the same `_tl_compat` seam.
 - [Architecture Registry](architecture-registry.md) — the `cohere` family entry
   and `EXPERIMENTAL_FAMILIES`.
 - `circuitkit.backends._tl_compat` — the port source.

{circuitkit-0.1.10 → circuitkit-0.2.0}/docs/algorithms/acdc.md

@@ -32,10 +32,11 @@ circuit = discover_circuit({
         "task": "ioi",
         "level": "node",
         "data_params": {"num_examples": 64},
-        # Optional: scope the tao (threshold) grid ACDC sweeps.
-        # Omit these to use the backend defaults.
-        "tao_bases": [1, 3, 5, 7, 9],
-        "tao_exps": [-5, -4, -3, -2],
+        # Optional: scope the tao (threshold) grid ACDC sweeps. Omit these to use the
+        # backend defaults (bases [1, 5] x exps [-5, -4, -3, -2] = 8 sweeps); this narrower
+        # grid runs 2 x 2 = 4.
+        "tao_bases": [1, 5],
+        "tao_exps": [-3, -2],
         "faithfulness_target": "kl_div",  # or "mse"
     },
     "pruning": {"target_sparsity": 0.3, "scope": "heads"},
@@ -52,12 +53,12 @@ circuit = discover_circuit({
 
 | Parameter | Default | Description |
 |-----------|---------|-------------|
-| `tao_bases` | `[1, 3, 5, 7, 9]` | Bases for the tao (threshold) grid ACDC sweeps. |
+| `tao_bases` | `[1, 5]` | Bases for the tao (threshold) grid ACDC sweeps. |
 | `tao_exps` | `[-5, -4, -3, -2]` | Exponents for the tao grid; each edge is scored by the smallest tao at which it is pruned. |
 | `faithfulness_target` | `"kl_div"` | Metric ACDC optimizes the circuit for; `"kl_div"` or `"mse"`. |
 | `num_examples` | `128` | Example count under `data_params`, used to build the batch ACDC patches over. |
 
-ACDC does not use a single threshold — it sweeps a grid of tao values built from the product of `tao_bases` and `tao_exps` (defaults give 20 sweeps). Narrow the grid to trim runtime:
+ACDC does not use a single threshold — it sweeps a grid of tao values built from the product of `tao_bases` and `tao_exps` (defaults give 8 sweeps). Narrow the grid to trim runtime:
 
 1. Keep the defaults for a full sweep.
 2. To go faster, pass fewer values, e.g. `tao_bases=[1]` and `tao_exps=[-3]` for a single tao.

{circuitkit-0.1.10 → circuitkit-0.2.0}/docs/algorithms/capability-matrix.md

@@ -29,7 +29,7 @@ All runtimes: `n_examples=128`, `batch_size=4`, `level="node"`, A100 40GB.
 | Gemma-4B | ~10 GB | ~16 GB | — | ❌ OOM |
 | Llama-7B | ~18 GB | ~30 GB | — | ❌ OOM |
 
-`bfloat16`, `ig_steps=5`, `batch_size=4`. Reduce batch_size or ig_steps if you hit OOM.
+Measured at `bfloat16`, `ig_steps=5` (the previous default), `batch_size=4`. Reduce `batch_size` if you hit OOM; `ig_steps` now defaults to 3.
 
 ## Task compatibility