innards 0.1.0__tar.gz

innards-0.1.0/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

innards-0.1.0/PKG-INFO

@@ -0,0 +1,401 @@
+Metadata-Version: 2.4
+Name: innards
+Version: 0.1.0
+Summary: See inside open-weight LLMs per message: KV cache, context growth and memory, predicted vs measured, across MHA, GQA, MLA, sliding-window and hybrid models.
+Keywords: llm,kv-cache,transformers,attention,gqa,mla,observability,profiling,inference
+Author: Jai
+License-Expression: Apache-2.0
+License-File: LICENSE
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3.15
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: System :: Monitoring
+Classifier: Typing :: Typed
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: huggingface-hub>=1.5
+Requires-Dist: psutil>=5.9 ; extra == 'hf'
+Requires-Dist: torch>=2.4 ; extra == 'hf'
+Requires-Dist: transformers>=5.0 ; extra == 'hf'
+Requires-Python: >=3.10
+Project-URL: Homepage, https://pypi.org/project/innards/
+Project-URL: Documentation, https://pypi.org/project/innards/#description
+Provides-Extra: hf
+Description-Content-Type: text/markdown
+
+# Innards
+
+![Innards: one input, every architecture, zero slowdown](https://<public-host>/innards/infographic.png)
+
+Innards runs open-weight LLMs and shows, for every message, what happens inside: tokens, context growth, KV cache per layer, memory and timing.
+It predicts KV-cache size from the model config, measures the real cache, and explains what each attention design (MHA, GQA, MLA, sliding window, hybrid) changed from its predecessor.
+Observation runs in background threads beside generation, so the model generates exactly as it would without Innards.
+
+## Install
+
+<!-- test: install (checked by the fresh-environment wheel install in CI) -->
+```bash
+pip install innards            # core: KV calculator, lineage, schema, console sink
+pip install "innards[hf]"      # + torch, transformers, psutil for in-process Hugging Face models
+
+uv add innards                 # inside a uv project
+uv add "innards[hf]"
+uv pip install "innards[hf]"   # into an existing environment
+```
+
+The core install has two small dependencies (`httpx`, `huggingface-hub`) and works on any Python 3.10+. The `hf` extra adds PyTorch and Transformers.
+
+## Quickstart
+
+Chat with a model and get one record per message:
+
+<!-- test: requires=hf,network -->
+```python
+from innards import Session
+
+with Session(
+    "Qwen/Qwen3-0.6B", max_new_tokens=48, sinks=["console"], chat_template_kwargs={"enable_thinking": False}
+) as s:
+    turn = s.chat("Explain paged attention in 3 lines")
+    print(turn.output.text)
+    print(turn.tokens.context_after, turn.kv.measured_bytes, turn.kv.predicted_bytes)
+
+    turn = s.chat("Now in one line")
+    print(turn.tokens.cached, "prompt tokens reused from the KV cache")
+```
+
+Send the same messages to models with different attention designs and compare:
+
+<!-- test: requires=hf,network -->
+```python
+from innards import compare
+
+results = compare(
+    models=["gpt2", "Qwen/Qwen3-0.6B"],
+    script=["What is a KV cache?", "Why does it grow with context?"],
+    max_new_tokens=16,
+)
+print(results.table())
+```
+
+Predict KV size without loading weights:
+
+<!-- test: requires=network check-output -->
+```python
+from innards.kv import predict
+
+p = predict("Qwen/Qwen3-0.6B", context_tokens=32768, kv_dtype="float16")
+print(p.formula)  # 2 × 28 layers × 8 KV heads × 128 head dim × 2 B = 114,688 B/token
+print(p.kv_bytes)  # 3758096384
+```
+
+## Imports
+
+| Import | What it is |
+| --- | --- |
+| `from innards import Session` | Synchronous session: chat with one model, one `TurnRecord` per message |
+| `from innards import AsyncSession` | Async session: streams tokens while metrics arrive from background workers |
+| `from innards import compare` | Sends the same messages to several models and aligns the results |
+| `from innards import CompareResult` | Result of `compare`: `turns[model]`, `rows()`, `table()`, `to_dict()` |
+| `from innards import Turn` | Alias of `TurnRecord`, the return type of `Session.chat` |
+| `from innards.kv import predict` | Architectural KV bytes, capacity and per-layer breakdown from a config, no weights |
+| `from innards.kv import KVPrediction` | Result of `predict` |
+| `from innards.kv import ModelSpec, LayerKV` | Normalized architecture facts and per-layer cache geometry |
+| `from innards.kv import load_config, spec_from_config` | Read a config (Hub id, path, dict) and normalize it |
+| `from innards.kv import dtype_bytes, normalize_dtype` | KV element types: `float32`, `float16`, `bfloat16`, `float8` |
+| `from innards.lineage import explain` | What an architecture changed from its predecessor, and why |
+| `from innards.lineage import LineageEntry, chain, families, get` | Lineage entries, predecessor chain, family list and lookup |
+| `from innards.schema import TurnRecord` | The per-message record (schema v1.0) |
+| `from innards.schema import ModelInfo, RuntimeInfo, TokenCounts, ContextComposition, KVInfo, MemoryInfo, TimingInfo, OutputInfo, ObserverInfo` | The record's sections |
+| `from innards.sinks import Sink, ConsoleSink, CallbackSink, make_sink, register_sink` | Where records go; any `write(record)`/`close()` object or plain callable works |
+| `from innards.observe import Observer, BoundedEventQueue` | The non-blocking observer and its drop-counting queue |
+| `from innards.backends import Backend, get_backend, register_backend, ContextFullError` | Engine adapter interface and registry |
+| `from innards.backends.hf import HFBackend` | In-process Hugging Face Transformers backend (needs `innards[hf]`) |
+| `from innards.strategies import FullHistory` | Context strategy that keeps every turn and reuses the KV cache |
+
+## Functions
+
+### `Session`
+
+```text
+Session(model: str, backend: str = "hf", strategy: str = "full_history",
+        sinks: Iterable[str | Sink | Callable] | None = None, *, system: str | None = None,
+        observe: bool = True, max_new_tokens: int = 256, record_timeout: float = 30.0,
+        queue_size: int = 4096, run_id: str | None = None, session_id: str | None = None,
+        load: bool = True, **backend_kwargs) -> Session
+```
+
+- `model`: Hugging Face Hub repo id or local path.
+- `backend`: `"hf"` (in-process Transformers).
+- `strategy`: `"full_history"`: every turn is kept, and the KV cache from earlier turns is reused.
+- `sinks`: where finished records go, e.g. `["console"]`. Records are also returned by `chat`.
+- `system`: optional system prompt.
+- `observe`: `False` turns observation off entirely (no events, no worker threads).
+- `max_new_tokens`: generation budget per turn (greedy decoding).
+- `record_timeout`: how long `chat` waits for the background record after generation has finished.
+- `**backend_kwargs`: for `hf`: `device` (`"cpu"`, `"cuda"`, `"mps"`), `dtype`, `chat_template_kwargs`, `generation_kwargs`, `revision`, `token`, `trust_remote_code`.
+
+Returns a session bound to one model. Use it as a context manager, or call `close()`.
+
+<!-- test: requires=hf,network check-output -->
+```python
+from innards import Session
+
+s = Session("gpt2", device="cpu", max_new_tokens=16)
+print(s.backend.spec.architecture)  # mha
+s.close()
+```
+
+### `Session.chat`
+
+```text
+Session.chat(message: str, *, timeout: float | None = None) -> TurnRecord
+```
+
+- `message`: the user message.
+- `timeout`: seconds to wait for the background record (default: `record_timeout`).
+
+Returns the `TurnRecord` (also available as `Turn`) with output text, tokens, KV, memory and timing.
+
+<!-- test: requires=hf,network -->
+```python
+from innards import Session
+
+with Session("gpt2", max_new_tokens=16) as s:
+    turn = s.chat("The key-value cache")
+    print(turn.kv.cache_tokens, turn.kv.measured_bytes == turn.kv.predicted_bytes)  # e.g. 23 True
+```
+
+### `Session.stream` and `Session.turn_metrics`
+
+```text
+Session.stream(message: str) -> Iterator[str]
+Session.turn_metrics(timeout: float | None = None) -> TurnRecord
+```
+
+- `stream` yields text as it is generated; closing the iterator early cancels generation.
+- `turn_metrics` returns the record for the most recent turn, waiting for the background worker if needed.
+
+<!-- test: requires=hf,network -->
+```python
+from innards import Session
+
+with Session("gpt2", max_new_tokens=16) as s:
+    for piece in s.stream("Attention is"):
+        print(piece, end="", flush=True)
+    print()
+    print(s.turn_metrics().timing)
+```
+
+### `AsyncSession`
+
+```text
+AsyncSession(model: str, backend: str = "hf", **session_kwargs) -> AsyncSession
+async AsyncSession.stream(message: str) -> AsyncIterator[str]
+await AsyncSession.turn_metrics(timeout: float | None = None) -> TurnRecord
+await AsyncSession.chat(message: str) -> TurnRecord
+```
+
+Takes the same arguments as `Session`. The model loads and generates in worker threads, so the event loop is never blocked.
+
+<!-- test: requires=hf,network -->
+```python
+import asyncio
+from innards import AsyncSession
+
+
+async def main():
+    async with AsyncSession("gpt2", max_new_tokens=16) as s:
+        async for token in s.stream("Explain paged attention"):
+            print(token, end="", flush=True)  # generation never waits on Innards
+        turn = await s.turn_metrics()  # arrives from background workers
+        print()
+        print(turn.kv.measured_bytes)
+
+
+asyncio.run(main())
+```
+
+### `compare`
+
+```text
+compare(models: Sequence[str], script: str | PathLike | Sequence[str], backend: str = "hf",
+        **session_kwargs) -> CompareResult
+```
+
+- `models`: Hub repo ids or local paths, run one at a time.
+- `script`: a list of user messages, one message, or a JSON file with a list of messages (strings or `{"content": ...}` objects, optionally under `"messages"`).
+- `**session_kwargs`: passed to every `Session`.
+
+Returns a `CompareResult`: `turns[model]` is the list of records; `rows()`, `table()` and `to_dict()` align them by turn.
+
+<!-- test: requires=hf,network -->
+```python
+import json, pathlib, tempfile
+from innards import compare
+
+path = pathlib.Path(tempfile.mkdtemp()) / "script.json"
+path.write_text(json.dumps({"messages": ["Hello", "What did I just say?"]}))
+result = compare(["gpt2"], path, max_new_tokens=8)
+print(result.rows()[0]["kv_measured_bytes"])
+```
+
+### `predict`
+
+```text
+predict(model_or_config: str | PathLike | dict | PretrainedConfig | ModelSpec, context_tokens: int,
+        kv_dtype: str = "float16", *, memory_bytes: int | None = None, batch_size: int = 1,
+        revision: str | None = None, token: str | None = None) -> KVPrediction
+```
+
+- `model_or_config`: Hub repo id (only `config.json` is downloaded), a config path or directory, a dict, a `PretrainedConfig` or a `ModelSpec`.
+- `context_tokens`: tokens resident in the cache.
+- `kv_dtype`: `float32`, `float16`, `bfloat16` or `float8` (aliases `fp32`, `fp16`, `bf16`, `fp8`).
+- `memory_bytes`: optional KV budget; sets `capacity_tokens`.
+
+Returns a `KVPrediction` with `bytes_per_token`, `kv_bytes`, `recurrent_state_bytes`, `total_bytes`, `per_layer_bytes`, `by_layer_type`, `capacity_tokens`, `formula` and `notes`. "Architectural" KV is what the attention design needs; an engine may hold more, and Innards reports the gap instead of hiding it. Sliding-window layers hold `window - 1` tokens: the current token attends to itself plus `window - 1` cached ones.
+
+<!-- test: requires=network check-output -->
+```python
+from innards.kv import predict
+
+p = predict("deepseek-ai/DeepSeek-V2-Lite", context_tokens=32768, memory_bytes=8 * 2**30)
+print(p.architecture, p.bytes_per_token, p.capacity_tokens)  # mla 31104 163840
+```
+
+A config dict works offline:
+
+<!-- test: check-output -->
+```python
+from innards.kv import predict
+
+gemma_like = {
+    "model_type": "gemma3_text",
+    "num_hidden_layers": 26,
+    "num_attention_heads": 4,
+    "num_key_value_heads": 1,
+    "head_dim": 256,
+    "sliding_window": 512,
+    "sliding_window_pattern": 6,
+    "max_position_embeddings": 32768,
+}
+p = predict(gemma_like, context_tokens=8192, kv_dtype="bf16")
+print(p.by_layer_type)  # {'global': 33554432, 'sliding': 11511808, 'recurrent': 0}
+```
+
+### `explain`
+
+```text
+explain(model_or_architecture: str | PathLike | dict | ModelSpec) -> LineageEntry
+```
+
+- `model_or_architecture`: a family name or alias (`"mha"`, `"mqa"`, `"gqa"`, `"mla"`, `"sliding_window"`, `"hybrid"`, `"ssm"`, `"fp8"`), a Hub repo id, a config path or dict, or a `ModelSpec`.
+
+Returns a `LineageEntry` with `family`, `predecessor`, `change`, `problem_solved`, `trade_off`, `kv_formula` and `reference`. For a model, `detail` describes that model's own numbers.
+
+<!-- test: check-output -->
+```python
+from innards.lineage import explain
+
+entry = explain("gqa")
+print(entry.predecessor, "->", entry.family)  # mha -> gqa
+print(entry.problem_solved)
+```
+
+<!-- test: requires=network -->
+```python
+from innards.lineage import explain
+
+print(explain("deepseek-ai/DeepSeek-V2-Lite"))  # MLA: what it changed from GQA and why
+```
+
+### `TurnRecord`
+
+```text
+TurnRecord.to_dict() -> dict
+TurnRecord.to_json(**json_kwargs) -> str
+TurnRecord.from_dict(data: dict) -> TurnRecord
+TurnRecord.from_json(text: str) -> TurnRecord
+```
+
+One JSON record per message, schema v1.0: `model`, `runtime`, `tokens`, `context`, `kv`, `memory`, `timing`, `output` and `observer`. Missing measurements are `null`, never zero. Readers accept any 1.x record and reject other major versions.
+
+<!-- test: check-output -->
+```python
+from innards.schema import TurnRecord
+
+record = TurnRecord.from_dict(
+    {
+        "schema_version": "1.0",
+        "session_id": "s-1",
+        "turn": 1,
+        "timestamp": "2026-10-01T10:42:07Z",
+        "model": {"id": "Qwen/Qwen3-0.6B", "architecture": "gqa"},
+        "runtime": {"backend": "hf"},
+        "kv": {"predicted_bytes": 378929152, "measured_bytes": 378929152, "measured_source": "hf_cache"},
+    }
+)
+print(record.kv.measured_bytes, record.memory.peak_bytes)  # 378929152 None
+```
+
+## CLI
+
+<!-- test: cli -->
+```bash
+innards chat --model gpt2 --backend hf                    # chat; per-turn metrics on stderr
+innards chat --model Qwen/Qwen3-0.6B --no-think --json    # TurnRecord JSON after each turn
+innards predict --model Qwen/Qwen3-0.6B --context 32768 --kv-dtype fp8
+innards predict --model deepseek-ai/DeepSeek-V2-Lite --context 32768 --memory 16GiB --json
+innards explain sliding_window
+```
+
+`innards chat` streams the reply to stdout and prints a three-line summary per turn to stderr: prompt, cached and generated tokens; KV measured vs predicted by layer type; TTFT, prefill, decode speed, peak memory and dropped events. Type `/reset` to start over, `/exit` to quit. Options: `--system`, `--max-new-tokens`, `--device`, `--dtype`, `--no-think`, `--json`, `--no-metrics`.
+
+## Supported
+
+**Architectures** (KV calculator and HF measurements):
+
+| Family | What is cached | Examples |
+| --- | --- | --- |
+| MHA | K and V for every head | GPT-2, Phi-3-mini (its config also sets a 2,047-token window on every layer) |
+| MQA | One shared K/V head | GPT-BigCode, Falcon-7B |
+| GQA | K/V per head group | Qwen3, Llama 3.2 |
+| MLA | Compressed latent + RoPE key | DeepSeek-V2, DeepSeek-V3 |
+| Sliding window (+ global) | Local layers keep `window - 1` tokens | Gemma 2/3, GPT-OSS, OLMo 3, Cohere 2, EXAONE 4, Mistral 7B v0.1; Llama 4 chunked layers |
+| Hybrid | Attention KV in a few layers + fixed recurrent state | Granite 4.0-H, LFM2, Qwen3-Next, Jamba, Bamba, Falcon-H1, Nemotron-H |
+| SSM | Fixed recurrent state only | Mamba, Mamba-2 |
+
+Legacy config keys (`n_layer`, `n_head`, `n_embd`, ChatGLM, Falcon, MPT) and nested `text_config` (multimodal checkpoints) are read too. KV dtypes: fp32, fp16, bf16, fp8.
+
+**Backends:** `hf`, in-process Hugging Face Transformers 5.x on CPU, CUDA or Apple MPS. Remote engines (vLLM, Ollama, llama.cpp) and managed APIs use the same `TurnRecord` and plug in through `innards.backends.register_backend`. They are planned, not shipped yet.
+
+**Sinks:** `console` (text or JSON lines). Any object with `write(record)` and `close()`, or a plain callable, also works as a sink.
+
+**Environments:** anywhere Python runs: local terminal, Jupyter and VS Code, Docker, cloud VMs and containers (AWS, GCP, Azure), Kaggle and Colab.
+
+**Python:** 3.10, 3.11, 3.12, 3.13 and 3.14. Python 3.15 is tested against its release candidates, with support from the final release (1 October 2026). The core install runs on 3.15 today. The `hf` extra needs PyTorch wheels for your Python version, and PyTorch has none for 3.15 yet.
+
+## Non-blocking guarantee
+
+Innards never sits in the generation path:
+
+- **Hot path**: the only Innards code inside the token loop takes a timestamp and offers a small event to a bounded queue. It makes no network calls, disk writes, tensor copies or device synchronization.
+- **Bounded queues that never block**: when the per-token queue is full, the event is dropped and counted, never waited on. Every record carries `observer.dropped_events`, so gaps are visible. The two per-turn events (start and end, sent outside the token loop) are never dropped, so every message gets a record.
+- **Background workers**: KV math, cache measurement (tensor shapes and dtypes only, `numel × element_size`), memory sampling and sink writes all run in background threads.
+- **Fail-open**: if a collector or sink fails, the error is logged and counted, and generation continues.
+- **Same output**: greedy output with Innards is identical to plain `model.generate`, which the test suite checks.
+
+**Overhead budget:** throughput within 1% of running without Innards, and no added time to first token. `scripts/overhead_bench.py` measures it (same model and prompts, Innards on vs off, interleaved runs, medians).
+
+## License
+
+Apache-2.0