interpkit 0.4.0__tar.gz → 0.5.0__tar.gz

{interpkit-0.4.0 → interpkit-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: interpkit
-Version: 0.4.0
+Version: 0.5.0
 Summary: Mech interp for any HuggingFace model.
 Author: Davide Zani
 License-Expression: MIT
@@ -20,7 +20,8 @@ Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: torch>=2.1
-Requires-Dist: transformers>=4.36
+Requires-Dist: numpy>=1.24
+Requires-Dist: transformers<6,>=4.36
 Requires-Dist: safetensors>=0.4
 Requires-Dist: rich>=13.0
 Requires-Dist: rich-gradient>=0.3
@@ -28,6 +29,7 @@ Requires-Dist: typer>=0.9
 Requires-Dist: Pillow>=10.0
 Requires-Dist: matplotlib>=3.8
 Requires-Dist: huggingface-hub>=0.20
+Requires-Dist: sentencepiece>=0.1.99
 Provides-Extra: vision
 Requires-Dist: torchvision>=0.16; extra == "vision"
 Provides-Extra: probe
@@ -60,27 +62,55 @@ Dynamic: license-file
 
 Mechanistic interpretability tooling today is fragmented. Each library supports a narrow set of architectures, and moving to a different model family usually means rewriting hook code from scratch.
 
-InterpKit provides a single, consistent interface for mech interp operations across any HuggingFace model — transformers, SSMs, vision models, and more — with zero annotation required.
+InterpKit provides a single, consistent interface for mech interp operations across a wide range of HuggingFace models — transformers, SSMs, vision models, and more — with automatic architecture discovery and little to no manual setup.
 
 ---
 
 ## Install
 
+We strongly recommend installing into an isolated environment so InterpKit's pinned dependencies (e.g. `typer`, `rich`, `transformers`) don't clash with whatever you already have installed globally
+
+Using [uv](https://docs.astral.sh/uv/) (recommended — fast, handles Python versions for you):
+
+```bash
+uv venv --python 3.11
+source .venv/bin/activate
+uv pip install interpkit
+
+# For linear probe support:
+uv pip install "interpkit[probe]"
+```
+
+Or with plain `venv` + `pip`:
+
 ```bash
+python3.11 -m venv .venv
+source .venv/bin/activate
 pip install interpkit
 
 # For linear probe support:
-pip install interpkit[probe]
+pip install "interpkit[probe]"
 ```
 
-Or install from source for development:
+Or with `conda`:
+
+```bash
+conda create -n interpkit python=3.11 -y
+conda activate interpkit
+pip install interpkit
+```
+
+Installing from source for development:
 
 ```bash
 git clone https://github.com/z4nix/interpkit.git
 cd interpkit
-pip install -e ".[dev]"
+uv venv --python 3.11 && source .venv/bin/activate
+uv pip install -e ".[dev]"
 ```
 
+> Python 3.10+ is required. If you must install into your system Python, use `pip install --user interpkit` and be aware that conflicting versions of `typer`, `rich`, or `transformers` already on your machine can break the CLI.
+
 ---
 
 ## Quickstart
@@ -522,6 +552,30 @@ model.trace(input_a, input_b, top_k=10)
 
 ---
 
+## Known limitations
+
+- **DeBERTa-v3 (DisentangledSelfAttention).** A known broadcast bug in
+  HuggingFace transformers' relative-position-bias path triggers on
+  forward hooks for any DeBERTa-v3 model (e.g.
+  `microsoft/deberta-v3-small`). interpkit detects this at load time
+  and gates `trace`, `decompose`, `attribute`, `head_activations`,
+  `steer`, `probe`, `diff`, `ov_scores`, `qk_scores` with a clean
+  `OperationNotSupportedForArchitecture` rather than the cryptic
+  upstream `RuntimeError: tensor (512) must match (7)`. `lens` and
+  `attention` still work. Use `bert`, `roberta`, `electra`, or
+  `albert` for the gated ops on encoder-only inputs.
+
+- **Integrated-gradients completeness on some modern decoders.** On
+  Qwen2/Qwen2.5/Qwen3 and SmolLM-family models, the trapezoidal Riemann
+  sum does not converge to model-output completeness even at large
+  `n_steps` (the P0b/N-008 empirical finding). Per-token IG scores remain
+  reliable as a token-importance **ranking** but cannot be interpreted as
+  additive contribution **magnitudes** on these models. `attribute()`
+  reports this programmatically: `result["interpretation"]` is
+  `"ranking_only"` in that case (and for `gradient` / `gradient_x_input`,
+  which are saliency methods), versus `"quantitative"` when IG completeness
+  holds. Branch on that field rather than parsing the warning text.
+
 ## Examples
 
 See the [`examples/`](examples/) directory for Jupyter notebooks:

{interpkit-0.4.0 → interpkit-0.5.0}/README.md

@@ -12,27 +12,55 @@
 
 Mechanistic interpretability tooling today is fragmented. Each library supports a narrow set of architectures, and moving to a different model family usually means rewriting hook code from scratch.
 
-InterpKit provides a single, consistent interface for mech interp operations across any HuggingFace model — transformers, SSMs, vision models, and more — with zero annotation required.
+InterpKit provides a single, consistent interface for mech interp operations across a wide range of HuggingFace models — transformers, SSMs, vision models, and more — with automatic architecture discovery and little to no manual setup.
 
 ---
 
 ## Install
 
+We strongly recommend installing into an isolated environment so InterpKit's pinned dependencies (e.g. `typer`, `rich`, `transformers`) don't clash with whatever you already have installed globally
+
+Using [uv](https://docs.astral.sh/uv/) (recommended — fast, handles Python versions for you):
+
+```bash
+uv venv --python 3.11
+source .venv/bin/activate
+uv pip install interpkit
+
+# For linear probe support:
+uv pip install "interpkit[probe]"
+```
+
+Or with plain `venv` + `pip`:
+
 ```bash
+python3.11 -m venv .venv
+source .venv/bin/activate
 pip install interpkit
 
 # For linear probe support:
-pip install interpkit[probe]
+pip install "interpkit[probe]"
 ```
 
-Or install from source for development:
+Or with `conda`:
+
+```bash
+conda create -n interpkit python=3.11 -y
+conda activate interpkit
+pip install interpkit
+```
+
+Installing from source for development:
 
 ```bash
 git clone https://github.com/z4nix/interpkit.git
 cd interpkit
-pip install -e ".[dev]"
+uv venv --python 3.11 && source .venv/bin/activate
+uv pip install -e ".[dev]"
 ```
 
+> Python 3.10+ is required. If you must install into your system Python, use `pip install --user interpkit` and be aware that conflicting versions of `typer`, `rich`, or `transformers` already on your machine can break the CLI.
+
 ---
 
 ## Quickstart
@@ -474,6 +502,30 @@ model.trace(input_a, input_b, top_k=10)
 
 ---
 
+## Known limitations
+
+- **DeBERTa-v3 (DisentangledSelfAttention).** A known broadcast bug in
+  HuggingFace transformers' relative-position-bias path triggers on
+  forward hooks for any DeBERTa-v3 model (e.g.
+  `microsoft/deberta-v3-small`). interpkit detects this at load time
+  and gates `trace`, `decompose`, `attribute`, `head_activations`,
+  `steer`, `probe`, `diff`, `ov_scores`, `qk_scores` with a clean
+  `OperationNotSupportedForArchitecture` rather than the cryptic
+  upstream `RuntimeError: tensor (512) must match (7)`. `lens` and
+  `attention` still work. Use `bert`, `roberta`, `electra`, or
+  `albert` for the gated ops on encoder-only inputs.
+
+- **Integrated-gradients completeness on some modern decoders.** On
+  Qwen2/Qwen2.5/Qwen3 and SmolLM-family models, the trapezoidal Riemann
+  sum does not converge to model-output completeness even at large
+  `n_steps` (the P0b/N-008 empirical finding). Per-token IG scores remain
+  reliable as a token-importance **ranking** but cannot be interpreted as
+  additive contribution **magnitudes** on these models. `attribute()`
+  reports this programmatically: `result["interpretation"]` is
+  `"ranking_only"` in that case (and for `gradient` / `gradient_x_input`,
+  which are saliency methods), versus `"quantitative"` when IG completeness
+  holds. Branch on that field rather than parsing the warning text.
+
 ## Examples
 
 See the [`examples/`](examples/) directory for Jupyter notebooks:

interpkit-0.5.0/interpkit/__init__.py

@@ -0,0 +1,65 @@
+"""interpkit — mech interp for any HuggingFace model."""
+
+from interpkit.core.arch import (
+    ArchFamily,
+    ArchInfo,
+    BlockSpec,
+    LayerInfo,
+    ModuleInfo,
+    resolve_arch,
+)
+from interpkit.core.exceptions import (
+    ArchitectureNotSupported,
+    AttentionBackendUnavailable,
+    InterpkitError,
+    LensPipelineMismatch,
+    OperationNotSupportedForArchitecture,
+    WrongInputType,
+)
+from interpkit.core.loader import load, load_module
+from interpkit.core.model import Model
+from interpkit.core.registry import register
+from interpkit.core.tl_compat import (
+    list_roundtrippable_hooks,
+    list_tl_hooks,
+    to_native_name,
+    to_tl_name,
+)
+
+
+def diff(model_a, model_b, input_data, *, save=None):
+    """Compare activations between two models on the same input."""
+    from interpkit.ops.diff import run_diff
+
+    return run_diff(model_a, model_b, input_data, save=save)
+
+
+__all__ = [
+    # Loaders
+    "load",
+    "load_module",
+    "Model",
+    # Architecture types
+    "ArchInfo",
+    "ArchFamily",
+    "BlockSpec",
+    "resolve_arch",
+    # Per-layer structural types
+    "LayerInfo",
+    "ModuleInfo",
+    # Exception types
+    "InterpkitError",
+    "ArchitectureNotSupported",
+    "AttentionBackendUnavailable",
+    "LensPipelineMismatch",
+    "OperationNotSupportedForArchitecture",
+    "WrongInputType",
+    # Operations
+    "register",
+    "diff",
+    # TL compat
+    "to_tl_name",
+    "to_native_name",
+    "list_tl_hooks",
+    "list_roundtrippable_hooks",
+]

{interpkit-0.4.0 → interpkit-0.5.0}/interpkit/__main__.py

@@ -1,18 +1,22 @@
 """Entry point so ``python -m interpkit`` invokes the Typer CLI.
 
-Mirrors the ``[project.scripts] interpkit = "interpkit.cli.main:app"``
+Mirrors the ``[project.scripts] interpkit = "interpkit.cli.main:run"``
 console script declared in :file:`pyproject.toml`, so users without the
 console script on their ``$PATH`` (e.g. just-installed in a fresh
 environment, vendored copies, ad-hoc subprocess invocations) can still
 reach every CLI command via ``python -m interpkit ...``.
 """
 
-from interpkit.cli.main import app
+from interpkit.cli.main import run
 
 
 def main() -> None:
-    """Invoke the Typer app — separate function makes patching easier in tests."""
-    app()
+    """Invoke the CLI — separate function makes patching easier in tests.
+
+    Uses ``run`` (not ``app`` directly) so interpkit's fail-loud errors are
+    rendered as clean one-line messages instead of tracebacks.
+    """
+    run()
 
 
 if __name__ == "__main__":

{interpkit-0.4.0 → interpkit-0.5.0}/interpkit/cli/main.py

@@ -1,8 +1,17 @@
-"""CLI entry point — Typer app with all interpkit commands."""
+"""CLI entry point — Typer app with all interpkit commands.
+
+When ``--format json`` is set, all status / progress output (rich panels,
+load progress bars, tqdm) is silenced or routed to stderr (F-023). The
+stdout stream stays clean JSON for programmatic consumers — pre-1.0
+``--format json`` interleaved rich panels and tqdm bars with the JSON
+block, breaking ``json.loads(p.stdout)`` for every CLI invocation.
+"""
 
 from __future__ import annotations
 
 import json as _json
+import os as _os
+import sys as _sys
 from importlib.metadata import version as _pkg_version
 
 import typer
@@ -33,11 +42,85 @@ app = typer.Typer(
     no_args_is_help=False,
     add_completion=False,
     rich_markup_mode="rich",
+    # interpkit's own errors (OperationNotSupportedForArchitecture,
+    # WrongInputType, LensPipelineMismatch, …) are deliberate, well-messaged,
+    # user-facing failures — not bugs. Disable Typer's rich-traceback so they
+    # don't reach the user as a scary stack trace; ``run()`` renders them as a
+    # clean one-line error instead.
+    pretty_exceptions_enable=False,
 )
+# F-023: console object — production code should call _make_console() so
+# JSON-mode stderr routing happens uniformly. The module-level singleton
+# is reassigned by main() once --format is parsed.
 console = Console()
 
 _output_format: str = "rich"
 
+
+def _make_console() -> Console:
+    """Construct a Console that respects the active output format.
+
+    In ``json`` mode, status / progress output goes to stderr so stdout
+    remains clean JSON. In ``rich`` mode, behaves identically to the
+    pre-1.0 module-level singleton.
+    """
+    if _output_format == "json":
+        return Console(file=_sys.stderr)
+    return Console()
+
+
+def _silence_third_party_loaders() -> None:
+    """Mute transformers / tqdm / huggingface chatter in JSON mode.
+
+    Pre-1.0 ``--format json`` had model-loading tqdm bars and the
+    "Loaded ... on cpu" rich line interleaved with the actual JSON
+    payload (F-023). Programmatic consumers couldn't json.loads(stdout).
+
+    Also re-binds every op-module console to write to stderr so rich
+    op-level rendering doesn't pollute the JSON stream.
+    """
+    if _output_format != "json":
+        return
+    # Silence HF transformers progress / warnings to stderr-only.
+    try:
+        from transformers import logging as _hf_logging
+        _hf_logging.set_verbosity_error()
+        _hf_logging.disable_progress_bar()
+    except (ImportError, AttributeError):
+        pass
+    # Silence raw tqdm.
+    _os.environ["TRANSFORMERS_VERBOSITY"] = "error"
+    _os.environ["TQDM_DISABLE"] = "1"
+    _os.environ["HF_HUB_DISABLE_PROGRESS_BARS"] = "1"
+
+    # Re-bind op-module consoles to stderr so renders don't pollute stdout.
+    import importlib
+
+    _stderr_console = Console(file=_sys.stderr)
+    for mod_name in (
+        "interpkit.core.render",
+        "interpkit.core.plot",
+        "interpkit.ops.attention",
+        "interpkit.ops.attribute",
+        "interpkit.ops.batch",
+        "interpkit.ops.circuits",
+        "interpkit.ops.diff",
+        "interpkit.ops.find_circuit",
+        "interpkit.ops.lens",
+        "interpkit.ops.probe",
+        "interpkit.ops.report",
+        "interpkit.ops.sae",
+        "interpkit.ops.scan",
+        "interpkit.ops.steer",
+        "interpkit.ops.trace",
+    ):
+        try:
+            mod = importlib.import_module(mod_name)
+            if hasattr(mod, "console"):
+                mod.console = _stderr_console  # type: ignore[attr-defined]
+        except ImportError:
+            continue
+
 _VERSION = _pkg_version("interpkit")
 
 
@@ -63,8 +146,17 @@ def _load_model(
 ):
     from interpkit.core.model import load
 
+    # F-007 fix: don't forward dtype=None — load() now requires explicit
+    # dtype. Defer to its built-in default (fp32) when the CLI user didn't
+    # specify --dtype.
+    kwargs: dict = {"device": device}
+    if dtype is not None:
+        kwargs["dtype"] = dtype
+    if device_map is not None:
+        kwargs["device_map"] = device_map
+
     with console.status(f"  Loading [bold]{model_name}[/bold]..."):
-        m = load(model_name, device=device, dtype=dtype, device_map=device_map)
+        m = load(model_name, **kwargs)
     console.print(f"  [bold green]Loaded[/bold green] [{ACCENT}]{model_name}[/{ACCENT}] on [bold]{m._device}[/bold]")
     return m
 
@@ -417,8 +509,11 @@ def main(
     ),
 ) -> None:
     """Mech interp for any HuggingFace model."""
-    global _output_format
+    global _output_format, console
     _output_format = fmt
+    # F-023: re-bind module-level console so it routes to stderr in JSON mode.
+    console = _make_console()
+    _silence_third_party_loaders()
     if ctx.invoked_subcommand is not None:
         return
     if extensive:
@@ -526,6 +621,38 @@ def inspect(
 ) -> None:
     """Print the model's module tree with types, param counts, and detected roles."""
     m = _load_model(model_name, device=device, dtype=dtype, device_map=device_map)
+    if _output_format == "json":
+        # F-023: inspect previously ignored --format json. Now emits a
+        # structured JSON description of the architecture.
+        arch = m.arch_info
+        result = {
+            "model": model_name,
+            "family": arch.family.value if hasattr(arch.family, "value") else str(arch.family),
+            "arch_family": arch.arch_family,
+            "device": m.device,
+            "dtype": str(m.dtype),
+            "num_layers": arch.num_layers,
+            "hidden_size": arch.hidden_size,
+            "num_attention_heads": arch.num_attention_heads,
+            "vocab_size": arch.vocab_size,
+            "is_encoder_decoder": arch.is_encoder_decoder,
+            "spatial": arch.spatial,
+            "head_path": arch.head_path,
+            "embed_path": arch.embed_path,
+            "pre_head_path": arch.pre_head_path,
+            "project_out_path": arch.project_out_path,
+            "blocks": [
+                {"path": b.path, "stage": b.stage,
+                 "has_attention": b.has_attention, "has_residual": b.has_residual}
+                for b in arch.blocks
+            ],
+            "modules": [
+                {"name": m.name, "type": m.type_name, "param_count": m.param_count, "role": m.role}
+                for m in arch.modules
+            ],
+        }
+        _json_dump(result)
+        return
     with console.status("  Inspecting model..."):
         m.inspect()
 
@@ -1063,5 +1190,38 @@ def chat(
         _json_dump({k: v for k, v in result.items() if k not in {"input_ids", "output_ids"}})
 
 
+def run() -> None:
+    """CLI entry point that renders interpkit's intentional errors cleanly.
+
+    The ``InterpkitError`` family (e.g. ``OperationNotSupportedForArchitecture``,
+    ``WrongInputType``, ``LensPipelineMismatch``) is the project's fail-loud
+    contract — these are clear, actionable, user-facing messages, not crashes.
+    Presenting them as a Python traceback undermines that, so we catch them at
+    the boundary and print a single clean line (JSON object in ``--format json``)
+    + exit non-zero. Unexpected exceptions still propagate as a normal traceback.
+    """
+    from interpkit.core.exceptions import InterpkitError
+
+    try:
+        app()
+    except (InterpkitError, ValueError, KeyError, IndexError) as exc:
+        # interpkit's user-facing validation failures: unsupported op / wrong
+        # input type (InterpkitError family), empty input (ValueError), unknown
+        # module path (KeyError with a "did you mean" hint), out-of-range
+        # position (ValueError / IndexError). These are clear, actionable
+        # messages — render one line, not a traceback. Genuine internal bugs
+        # raise other types (RuntimeError, TypeError, …) and still surface a
+        # full traceback. ``KeyError.__str__`` wraps the message in quotes, so
+        # pull ``args[0]`` for it.
+        msg = exc.args[0] if (isinstance(exc, KeyError) and exc.args) else str(exc)
+        if _output_format == "json":
+            import json as _json
+
+            print(_json.dumps({"error": type(exc).__name__, "message": str(msg)}))
+        else:
+            Console(file=_sys.stderr).print(f"[bold red]Error:[/bold red] {msg}")
+        raise SystemExit(1) from None
+
+
 if __name__ == "__main__":
-    app()
+    run()

interpkit-0.5.0/interpkit/core/arch/__init__.py

@@ -0,0 +1,102 @@
+"""Architecture resolution: one cohesive package.
+
+Consolidates what used to be three entangled modules — ``discovery``,
+the ``resolve`` package, and ``residual`` — into a single
+``interpkit.core.arch`` package with one :class:`ArchInfo` contract.
+
+Submodule layout:
+
+- ``names``    — module-name vocabulary + regexes.
+- ``types``    — ``ArchInfo``, ``ArchFamily``, ``BlockSpec``, ``LayerInfo``, ``ModuleInfo``.
+- ``tree``     — static module-tree primitives + weight extraction.
+- ``probe``    — runtime forward-hook probes.
+- ``family``   — family classification, topology, config parsing.
+- ``blocks``   — block / decoder-block discovery.
+- ``layers``   — per-layer attn/mlp/qkv resolution + role assignment.
+- ``heads``    — head / unembedding / project-out / MLM / pre-head discovery.
+- ``resolve``  — ``resolve_arch`` orchestrator + ``discover`` + overrides.
+- ``residual`` — residual-stream decomposition schemas.
+"""
+
+from __future__ import annotations
+
+from interpkit.core.arch.names import (
+    ALL_QKV_NAMES,
+    ATTN_NAMES,
+    ATTN_RE,
+    FUSED_QKV_NAMES,
+    K_PROJ_NAMES,
+    MLP_NAMES,
+    MLP_RE,
+    O_PROJ_NAMES,
+    Q_PROJ_NAMES,
+    V_PROJ_NAMES,
+    names_to_regex,
+)
+from interpkit.core.arch.residual import (
+    Component,
+    PostLNResidual,
+    PreLNResidual,
+    ResidualSchema,
+    Seq2seqResidual,
+    SharedLayerResidual,
+    residual_schema_for,
+)
+from interpkit.core.arch.resolve import (
+    ARCH_OVERRIDES,
+    apply_overrides,
+    discover,
+    resolve_arch,
+)
+from interpkit.core.arch.tree import (
+    canonical_linear_weight,
+    extract_proj_weight,
+    get_weight,
+    module_at_path,
+)
+from interpkit.core.arch.types import (
+    ArchFamily,
+    ArchInfo,
+    BlockSpec,
+    LayerInfo,
+    ModuleInfo,
+)
+
+__all__ = [
+    # Types
+    "ArchInfo",
+    "ArchFamily",
+    "BlockSpec",
+    "LayerInfo",
+    "ModuleInfo",
+    # Resolution
+    "resolve_arch",
+    "discover",
+    "apply_overrides",
+    "ARCH_OVERRIDES",
+    # Tree / weight helpers
+    "module_at_path",
+    "get_weight",
+    "extract_proj_weight",
+    "canonical_linear_weight",
+    # Module-name vocabulary
+    "ATTN_NAMES",
+    "MLP_NAMES",
+    "FUSED_QKV_NAMES",
+    "Q_PROJ_NAMES",
+    "K_PROJ_NAMES",
+    "V_PROJ_NAMES",
+    "ALL_QKV_NAMES",
+    "O_PROJ_NAMES",
+    "ATTN_RE",
+    "MLP_RE",
+    "names_to_regex",
+    # Residual schemas
+    "Component",
+    "ResidualSchema",
+    "PreLNResidual",
+    "PostLNResidual",
+    "SharedLayerResidual",
+    "Seq2seqResidual",
+    "residual_schema_for",
+]