interp-engine 1.3.2__tar.gz → 1.3.4__tar.gz

{interp_engine-1.3.2 → interp_engine-1.3.4}/.gitignore

@@ -33,4 +33,7 @@ dist/
 .venv-ci/
 
 plans/
-*.plan.md
+*.plan.md
+
+logs/*
+logs/

{interp_engine-1.3.2 → interp_engine-1.3.4}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.5
 Name: interp-engine
-Version: 1.3.2
-Summary: A fast, standardized interpretability engine that supports most modern models and architectures. Powers Neuronpedia.
+Version: 1.3.4
+Summary: A fast, standardized, and easy to use interpretability engine.
 Project-URL: Homepage, https://github.com/decoderesearch/interp-engine
 Project-URL: Repository, https://github.com/decoderesearch/interp-engine
 Project-URL: Issues, https://github.com/decoderesearch/interp-engine/issues
@@ -44,10 +44,13 @@ Description-Content-Type: text/markdown
   <a href="https://join.slack.com/t/opensourcemechanistic/shared_invite/zt-3z9o0hxjl-MDX9pbATO2qESOazNDLpdQ"><img src="https://img.shields.io/badge/Slack-Open%20Source%20Mechanistic%20Interpretability-4A154B?logo=slack&logoColor=white" alt="Join the Slack"></a>
 </p>
 
-`interp-engine` is an interpretability engine that is fast (>40x tok/s vs HF eager), standardized (34 'points'/addresses across architectures), and easy to use and debug. It powers all of [Neuronpedia](https://neuronpedia.org)'s inference and is checked for accuracy against HF Transformers and other engines.
+`interp-engine` is an interpretability engine that is fast, standardized (34 'points'/addresses across architectures), and easy to use and debug. It powers all of [Neuronpedia](https://neuronpedia.org)'s inference and is checked for accuracy against HF Transformers and other engines.
 
 <p align="center">
-  <img src="https://neuronpedia.s3.amazonaws.com/site-assets/ie-perf.png" alt="Tokens per second while capturing and generating: eager against IE-vLLM and IE-vLLM-static on qwen3.8-27b and deepseek-v4-flash-0731, 8 requests in flight" width="100%">
+  <img src="https://neuronpedia.s3.amazonaws.com/site-assets/ie-benchmark.gif" alt="benchmark between ie and transformers" width="100%">
+</p>
+<p align="center">
+  <img src="https://neuronpedia.s3.amazonaws.com/site-assets/toks.png" alt="Tokens per second while capturing and generating: eager against IE-vLLM and IE-vLLM-static on qwen3.8-27b and deepseek-v4-flash-0731, 8 requests in flight" width="100%">
 </p>
 <p align="center">
   <img src="https://neuronpedia.s3.amazonaws.com/site-assets/interp-engine-demo.gif" alt="interp-engine demo gif" width="100%">

{interp_engine-1.3.2 → interp_engine-1.3.4}/README.md

@@ -13,10 +13,13 @@
   <a href="https://join.slack.com/t/opensourcemechanistic/shared_invite/zt-3z9o0hxjl-MDX9pbATO2qESOazNDLpdQ"><img src="https://img.shields.io/badge/Slack-Open%20Source%20Mechanistic%20Interpretability-4A154B?logo=slack&logoColor=white" alt="Join the Slack"></a>
 </p>
 
-`interp-engine` is an interpretability engine that is fast (>40x tok/s vs HF eager), standardized (34 'points'/addresses across architectures), and easy to use and debug. It powers all of [Neuronpedia](https://neuronpedia.org)'s inference and is checked for accuracy against HF Transformers and other engines.
+`interp-engine` is an interpretability engine that is fast, standardized (34 'points'/addresses across architectures), and easy to use and debug. It powers all of [Neuronpedia](https://neuronpedia.org)'s inference and is checked for accuracy against HF Transformers and other engines.
 
 <p align="center">
-  <img src="https://neuronpedia.s3.amazonaws.com/site-assets/ie-perf.png" alt="Tokens per second while capturing and generating: eager against IE-vLLM and IE-vLLM-static on qwen3.8-27b and deepseek-v4-flash-0731, 8 requests in flight" width="100%">
+  <img src="https://neuronpedia.s3.amazonaws.com/site-assets/ie-benchmark.gif" alt="benchmark between ie and transformers" width="100%">
+</p>
+<p align="center">
+  <img src="https://neuronpedia.s3.amazonaws.com/site-assets/toks.png" alt="Tokens per second while capturing and generating: eager against IE-vLLM and IE-vLLM-static on qwen3.8-27b and deepseek-v4-flash-0731, 8 requests in flight" width="100%">
 </p>
 <p align="center">
   <img src="https://neuronpedia.s3.amazonaws.com/site-assets/interp-engine-demo.gif" alt="interp-engine demo gif" width="100%">

{interp_engine-1.3.2 → interp_engine-1.3.4}/docs/INTERNALS.md

@@ -66,6 +66,12 @@ files and why they are where they are.
   - `LD_LIBRARY_PATH`) before the first CUDA call, instead of failing ten frames deep in
     `torch.cuda._lazy_init`. Lives here because every app on the engine inherits the same CUDA
     floor — the `[vllm]` wheels link `libcudart.so.13` directly.
+- `notebook_stdout.py` — `ensure_stdout_descriptor`: gives a notebook kernel's `sys.stdout` the file
+  descriptor vLLM's engine start needs. vLLM silences C-level output by dup'ing over one and forks
+  its EngineCore child, so under ipykernel — whose stream writes to a socket and has none — the
+  child dies before it loads anything, and the caller is told to see a root cause that is in another
+  process. Called from `_ensure_engine`; a no-op in a script, a server, or a kernel that captured
+  the real descriptors itself.
 
 ## Correctness
 

interp_engine-1.3.4/interp_engine/notebook_stdout.py

@@ -0,0 +1,90 @@
+"""Give a notebook kernel's stdout the file descriptor vLLM's engine start needs.
+
+vLLM silences C-level output around ``torch.distributed.new_group`` by dup'ing over a
+descriptor (``suppress_stdout`` in ``vllm/utils/system_utils.py``), and its EngineCore
+child is forked, so the child inherits whichever ``sys.stdout`` the parent had. Under
+Jupyter that is ``ipykernel.iostream.OutStream``, which writes through a ZMQ socket and
+raises ``io.UnsupportedOperation`` from ``fileno()``. The child then dies before it loads
+anything, and what reaches the caller is vLLM's ``Engine core initialization failed. See
+root cause above.`` -- above being a traceback in another process, about a descriptor.
+
+A descriptor is what that stream is supposed to hand back: ipykernel answers ``fileno()``
+with ``_original_stdstream_copy``, a dup of the real stdout it took over, whenever the
+kernel was started capturing the low-level ones. This supplies the one a kernel that was
+not started that way -- Colab's -- did not keep. Colab is the case that matters, because
+this repo's own notebook templates run there.
+
+The descriptor stays on the stream rather than being restored after the engine is built.
+The fork is not the only caller: vLLM suppresses stdout the same way around its stateless
+process groups, from whichever process reaches one.
+
+Forcing ``spawn`` would also give the child a stdout of its own, and is what
+``VLLM_WORKER_MULTIPROC_METHOD`` is for. It is not what this does, because a spawned
+child writes to the kernel's real stdout rather than to the cell -- which takes the engine
+logs, and the root cause behind that message, out of the notebook with it.
+"""
+
+from __future__ import annotations
+
+import logging
+import os
+import sys
+
+logger = logging.getLogger(__name__)
+
+
+def _answers_fileno(stream: object) -> bool:
+    """Whether ``stream.fileno()`` returns rather than raises.
+
+    ``io.UnsupportedOperation`` is both an ``OSError`` and a ``ValueError``, which is what
+    a kernel stream raises; a detached or closed one raises ``ValueError`` on its own.
+    """
+    try:
+        stream.fileno()  # type: ignore[attr-defined]
+    except (AttributeError, OSError, ValueError):
+        return False
+    return True
+
+
+def _process_stdout_fd() -> int | None:
+    """The descriptor this process's stdout is, or None when it has none.
+
+    ``sys.__stdout__`` is the stream Python opened at startup, which Jupyter leaves alone
+    when it replaces ``sys.stdout``, so it is the first place to ask. Fd 1 covers an
+    embedding that replaced both. Either is confirmed with ``fstat`` before being handed
+    out, since a daemonized process may have closed it.
+    """
+    original = sys.__stdout__
+    fd = original.fileno() if original is not None and _answers_fileno(original) else 1
+    try:
+        os.fstat(fd)
+    except OSError:
+        return None
+    return fd
+
+
+def ensure_stdout_descriptor() -> bool:
+    """Give ``sys.stdout`` a ``fileno()`` if it has none, and say whether it needed one.
+
+    Idempotent, and a no-op outside a notebook. A stream that already answers is left
+    alone, and so is a process whose stdout is closed: there is no descriptor to offer
+    there, and vLLM's refusal is then the honest outcome rather than one to work around.
+    """
+    stream = sys.stdout
+    if stream is None or _answers_fileno(stream):
+        return False
+    fd = _process_stdout_fd()
+    if fd is None:
+        return False
+    try:
+        stream.fileno = lambda: fd  # type: ignore[method-assign]
+    except (AttributeError, TypeError):
+        # A stream that takes no new attribute, which a C-level or slotted one does not.
+        # Nothing is lost: this is the refusal vLLM was about to raise anyway.
+        return False
+    logger.info(
+        "%s has no file descriptor, which vLLM's engine start requires; answering fileno() with fd %d.",
+        type(stream).__name__,
+        fd,
+    )
+    return True

{interp_engine-1.3.2 → interp_engine-1.3.4}/interp_engine/vllm_backend.py

@@ -30,6 +30,7 @@ from interp_engine import facts
 from interp_engine._loop import refuse_foreign_loop
 from interp_engine.address import Address, format_address, to_address
 from interp_engine.autograd_support import GradSupport, vllm_grad_support
+from interp_engine.notebook_stdout import ensure_stdout_descriptor
 from interp_engine.points import d_model_wide, hyper_connection_names, refusal_reasons
 from interp_engine.points import steer_refusal_reason as points_steer_refusal
 from interp_engine.residual_basis import ResidualBasis, vllm_residual_basis
@@ -1213,6 +1214,9 @@ class VLLMModel:
                         tuple(getattr(self, "_static_reads", ())),
                         tuple(getattr(self, "_static_writes", ())),
                     )
+                    # The line below forks a child that suppresses stdout by descriptor, which a
+                    # notebook kernel's stdout does not have. See `notebook_stdout`.
+                    ensure_stdout_descriptor()
                     self.engine = AsyncLLM.from_engine_args(AsyncEngineArgs(**self._engine_kwargs))
                     self._engine_loop = asyncio.get_running_loop()
         return self.engine

{interp_engine-1.3.2 → interp_engine-1.3.4}/pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "interp-engine"
-version = "1.3.2"
-description = "A fast, standardized interpretability engine that supports most modern models and architectures. Powers Neuronpedia."
+version = "1.3.4"
+description = "A fast, standardized, and easy to use interpretability engine."
 readme = "README.md"
 license = "Apache-2.0"
 license-files = ["LICENSE"]

interp_engine-1.3.4/tests/test_notebook_stdout.py

@@ -0,0 +1,132 @@
+"""Unit tests for the stdout descriptor a notebook kernel does not provide.
+
+No vLLM and no GPU: what is under test is a property of ``sys.stdout``, and the stream
+that has that property in the field -- ipykernel's ``OutStream`` -- is stood in for by a
+class which raises from ``fileno()`` the same way. The engine-shaped half is a vLLM engine
+started under such a stream, which fails in the EngineCore child rather than here.
+
+``sys.stdout`` is replaced **inside each test body** rather than by a fixture. pytest
+captures at the descriptor level and reassigns ``sys.stdout`` every time it resumes
+capture, including on the way from setup into the call phase -- so a patch applied in a
+fixture is gone by the time the test runs, and the test passes or fails on the runner's
+stream instead of the one it meant to use.
+"""
+
+from __future__ import annotations
+
+import io
+import os
+import sys
+from typing import IO, Any
+
+import pytest
+
+from interp_engine.notebook_stdout import ensure_stdout_descriptor
+
+
+class KernelStdout:
+    """``ipykernel.iostream.OutStream``, as far as this matters.
+
+    Writes go somewhere that is not a file -- a ZMQ socket, in the real one -- so there is
+    no descriptor to answer with, which is the whole of the failure being reproduced.
+    """
+
+    def __init__(self) -> None:
+        self.written: list[str] = []
+
+    def write(self, text: str) -> int:
+        self.written.append(text)
+        return len(text)
+
+    def flush(self) -> None:
+        pass
+
+    def fileno(self) -> int:
+        raise io.UnsupportedOperation("fileno")
+
+
+class SlottedStdout:
+    """A stream that takes no new attribute, as a C-level one does not."""
+
+    __slots__ = ()
+
+    def fileno(self) -> int:
+        raise io.UnsupportedOperation("fileno")
+
+
+def _as_stdout(monkeypatch: pytest.MonkeyPatch, stream: object) -> None:
+    """Make ``stream`` this test's ``sys.stdout``. Call from the test body; see above."""
+    monkeypatch.setattr(sys, "stdout", stream)
+
+
+def test_a_kernel_stream_gains_a_descriptor_the_os_accepts(monkeypatch: pytest.MonkeyPatch) -> None:
+    """The point of the exercise: ``fileno()`` answers, and with a live descriptor.
+
+    ``fstat`` is the assertion rather than a particular number, because what vLLM does
+    with the answer is ``os.dup``, which fails on anything ``fstat`` would reject too.
+    """
+    _as_stdout(monkeypatch, KernelStdout())
+    assert ensure_stdout_descriptor() is True
+    os.fstat(sys.stdout.fileno())
+
+
+def test_writes_still_go_where_they_did(monkeypatch: pytest.MonkeyPatch) -> None:
+    """A descriptor is added; nothing is redirected. The cell keeps its output."""
+    stream = KernelStdout()
+    _as_stdout(monkeypatch, stream)
+    ensure_stdout_descriptor()
+    print("to the notebook")
+    assert "to the notebook" in "".join(stream.written)
+
+
+def test_it_is_idempotent(monkeypatch: pytest.MonkeyPatch) -> None:
+    """Every engine build calls this, and only the first one has anything to do."""
+    _as_stdout(monkeypatch, KernelStdout())
+    assert ensure_stdout_descriptor() is True
+    assert ensure_stdout_descriptor() is False
+
+
+def test_a_real_stream_is_left_alone(monkeypatch: pytest.MonkeyPatch, tmp_path: Any) -> None:
+    """A script's stdout already answers, so this must not touch it.
+
+    The second assertion is what says *untouched* rather than merely still working: a
+    descriptor of our own, put on the instance, would satisfy the first one too.
+    """
+    with open(tmp_path / "out", "w") as real:
+        _as_stdout(monkeypatch, real)
+        assert ensure_stdout_descriptor() is False
+        assert "fileno" not in getattr(real, "__dict__", {})
+
+
+def test_a_stream_that_takes_no_attribute_is_refused(monkeypatch: pytest.MonkeyPatch) -> None:
+    """Refused rather than raised: vLLM's own error is a better one than ours here."""
+    _as_stdout(monkeypatch, SlottedStdout())
+    assert ensure_stdout_descriptor() is False
+
+
+def test_a_closed_process_stdout_offers_nothing(monkeypatch: pytest.MonkeyPatch) -> None:
+    """With none to lend, the answer is no rather than a number that fails later.
+
+    This is the daemonized process, whose stdout was closed before Python started. Both
+    sources are taken away: ``sys.__stdout__``, and the fd 1 fallback that stands in for
+    it, which is open in a test runner and would otherwise be handed out.
+    """
+    _as_stdout(monkeypatch, KernelStdout())
+    monkeypatch.setattr(sys, "__stdout__", None)
+    monkeypatch.setattr("interp_engine.notebook_stdout.os.fstat", _raise_ebadf)
+    assert ensure_stdout_descriptor() is False
+
+
+def _raise_ebadf(fd: int) -> os.stat_result:
+    raise OSError(9, "Bad file descriptor")
+
+
+def test_the_stand_in_matches_ipykernel() -> None:
+    """The premise, asserted where it can be: ``fileno()`` raises what is caught for.
+
+    ``io.UnsupportedOperation`` is both an ``OSError`` and a ``ValueError``, which is why
+    the module catches those two rather than importing ipykernel to name the class.
+    """
+    stream: IO[Any] = KernelStdout()  # type: ignore[assignment]
+    with pytest.raises((OSError, ValueError)):
+        stream.fileno()