superlinear 0.1.0__tar.gz

superlinear-0.1.0/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

superlinear-0.1.0/PKG-INFO

@@ -0,0 +1,469 @@
+Metadata-Version: 2.4
+Name: superlinear
+Version: 0.1.0
+Summary: Efficient sparse attention kernels for long-context language models
+Author: Superlinear Authors
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/concavity-ai/superlinear
+Project-URL: Documentation, https://github.com/concavity-ai/superlinear#readme
+Project-URL: Repository, https://github.com/concavity-ai/superlinear
+Project-URL: Issues, https://github.com/concavity-ai/superlinear/issues
+Keywords: attention,transformers,triton,sparse-attention,long-context,machine-learning,deep-learning
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch>=2.0.0
+Requires-Dist: triton>=2.1.0
+Provides-Extra: cli
+Requires-Dist: rank-bm25>=0.2.2; extra == "cli"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Requires-Dist: black>=23.0; extra == "dev"
+Requires-Dist: ruff>=0.1.0; extra == "dev"
+Requires-Dist: mypy>=1.0; extra == "dev"
+Provides-Extra: inference
+Requires-Dist: transformers<5,>=4.38.0; extra == "inference"
+Requires-Dist: accelerate>=0.26.0; extra == "inference"
+Provides-Extra: model
+Requires-Dist: transformers<5,>=4.38.0; extra == "model"
+Requires-Dist: accelerate>=0.26.0; extra == "model"
+Requires-Dist: vllm>=0.15.0; extra == "model"
+Provides-Extra: server
+Requires-Dist: fastapi>=0.110.0; extra == "server"
+Requires-Dist: uvicorn>=0.23.0; extra == "server"
+Provides-Extra: docs
+Requires-Dist: sphinx>=6.0; extra == "docs"
+Requires-Dist: sphinx-rtd-theme>=1.0; extra == "docs"
+Requires-Dist: myst-parser>=1.0; extra == "docs"
+Dynamic: license-file
+
+# Superlinear
+
+**Long-context inference with efficient sparse attention**
+
+[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)
+[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)](https://www.python.org/downloads/)
+[![PyTorch 2.0+](https://img.shields.io/badge/pytorch-2.0+-ee4c2c.svg)](https://pytorch.org/)
+
+Superlinear is an inference engine for long-context language models built on [Superlinear Multi-Step Attention](https://arxiv.org/abs/2601.18401) — a fully trainable multi-step attention architecture that achieves **subquadratic complexity** while preserving **random context access** (no eligible token position is structurally excluded from attention). It comes with an OpenAI-compatible server and a CLI that lets you chat, ingest documents, and manage sessions — with **explicit, stateful KV cache reuse across turns**. Unlike stateless “send the full prompt every turn” setups, Superlinear exposes sessions/snapshots so you can directly manage KV cache lifetime and GPU memory for performance.
+
+> **⚠️ Research Preview — Not for Production Use**
+>
+> This software and its associated model weights are released as an **experimental research artifact** accompanying the [Superlinear Multi-Step Attention](https://arxiv.org/abs/2601.18401) paper. They are provided for research, experimentation, and exploration purposes only. The model has undergone **limited** training and evaluation (primarily NIAH retrieval); it has **not** been evaluated for safety, factual accuracy, or robustness across general tasks. **Do not use this system for any critical, production, or decision-making applications.** The authors and contributors assume no responsibility or liability for any outputs generated or consequences arising from the use of this software.
+
+<p align="center">
+  <img src="assets/superlinear_architecture.png" alt="Superlinear attention architecture" width="900">
+</p>
+
+<p align="center"><em>
+Superlinear attention reformulates causal self-attention as a multi-step search problem with N steps,
+yielding O(L<sup>1+1/N</sup>) complexity. In the baseline N=2 instantiation (analogous to jump search):<br>
+(1) <b>accumulate</b> prefix context via linear recurrence (Mamba-2),
+(2) <b>span-search</b> over O(√L) anchors to route to relevant regions,
+(3) <b>span-attention</b> over the top-k selected spans, and
+(4) <b>combine</b> via softmax-weighted gating —
+achieving O(L<sup>3/2</sup>) complexity while preserving random context access.
+</em></p>
+
+## Installation
+
+### Install the repo (kernels + CLI/server)
+
+```bash
+git clone https://github.com/concavity-ai/superlinear.git
+cd superlinear
+
+# Superlinear itself only depends on torch + triton.
+pip install -e ".[server]"
+```
+
+**Requirements:** Python ≥ 3.9, PyTorch ≥ 2.0, Triton ≥ 2.1, CUDA-capable GPU.
+
+### Run the released model (additional GPU deps)
+
+To run **concavity-ai/superlinear-exp-v0.1**, you need additional model/runtime dependencies and CUDA-compiled extensions.
+
+Copy/paste one-liner (recommended, from repo root):
+
+```bash
+conda env create -f environment.yml \
+    && conda run -n superlinear pip install torch --index-url https://download.pytorch.org/whl/cu128 \
+    && conda run -n superlinear pip install -e ".[server,model]" \
+    && conda run -n superlinear bash -lc 'CUDA_HOME="$CONDA_PREFIX" pip install "mamba-ssm[causal-conv1d]" --no-build-isolation --no-cache-dir --no-binary :all:'
+```
+
+See [docs/installation.md](docs/installation.md) for step-by-step instructions, tested versions, and a pip-only option (if you already have `mamba-ssm[causal-conv1d]` working).
+
+## Memory Management
+
+Superlinear’s main differentiator is **stateful KV cache reuse**: you can keep long contexts “hot” across turns and actively control memory/performance via sessions, snapshots, and chunked prefill (`--chunk-size`).
+
+- **GPU memory budgeting.** The model weights for **concavity-ai/superlinear-exp-v0.1** require ~60 GB VRAM at 16-bit precision. Additionally, the KV cache uses ~6 GB per million tokens of context. Budget extra headroom for chunked prefill working memory. A single 80 GB GPU can run up to ~2M context; 180 GB (e.g., B200) supports 10M+ context.
+
+- **KV cache dominates at long context.** Rough rule of thumb: ~6 GB per 1M tokens *per active session*. If you keep multiple sessions open concurrently, KV cache usage scales roughly linearly with the number of sessions.
+- **Chunked prefill trades throughput for peak VRAM.** Prefill runs in chunks to limit peak memory. The server default is `--chunk-size 8192`.
+    - Larger chunks typically improve throughput (fewer chunk boundaries / overhead).
+    - Smaller chunks reduce peak VRAM during prefill, which can help avoid OOM on smaller GPUs.
+    - If you hit OOM during prefill, try reducing chunk size first:
+        - `spl server start --model concavity-ai/superlinear-exp-v0.1 --chunk-size 4096`
+        - (or increase it on large-memory GPUs for better throughput, e.g. `--chunk-size 32768`, as used in the paper’s throughput measurements)
+- **Use sessions to avoid re-prefill.** Stateful sessions persist the KV cache across turns, so follow-up turns don’t require re-sending/re-prefilling the entire history.
+- **Use snapshots to checkpoint and free GPU memory.** Snapshots save session/workspace state to disk so you can close sessions (freeing VRAM) and later resume from a snapshot without re-ingesting documents.
+
+---
+
+## Quick Start
+
+### 1. Start the server
+
+```bash
+spl server start --model concavity-ai/superlinear-exp-v0.1
+```
+
+This downloads the model (if needed) and launches the inference server in the background (default: `http://127.0.0.1:8787`). Check status with `spl server status`.
+
+### 2. Chat
+
+```bash
+spl chat
+```
+
+Opens an interactive REPL with persistent sessions and thinking mode. Type `/help` for all commands.
+
+```
+spl(chat:chat_20260206_...)> What is the capital of France?
+thinking: The user is asking a simple factual question...
+assistant: The capital of France is Paris.
+```
+
+### 3. Document Q&A
+
+Load documents into a workspace, then ask questions with automatic RAG retrieval:
+
+```bash
+spl docs wiki
+```
+
+```
+spl(docs:wiki:INGEST)> /add examples/docs/wiki -s
+added files=8 bytes=... sources=8
+saved base_snapshot_id=764cb17e...
+
+spl(docs:wiki:INGEST)> How do RNNs and Transformers differ in handling sequences?
+assistant: ...
+```
+
+Save your workspace as a snapshot and resume later:
+
+```
+# Later, resume from the snapshot:
+$ spl docs wiki -l 764cb17e...
+```
+
+---
+
+## CLI Reference
+
+The `spl` command is the primary interface. All commands talk to the server over HTTP.
+
+### Server Management
+
+```bash
+spl server start --model concavity-ai/superlinear-exp-v0.1   # Start server (background)
+spl server start --model concavity-ai/superlinear-exp-v0.1 --foreground  # Start in foreground
+spl server start --model concavity-ai/superlinear-exp-v0.1 --chunk-size 4096  # Reduce peak VRAM during prefill (default: 8192)
+spl server status                 # Check if server is running
+spl server stop                   # Stop server
+spl server connect <url>          # Connect to a remote server
+```
+
+### Chat & Docs REPLs
+
+```bash
+spl chat                          # Interactive chat
+spl chat --new                    # Start fresh session
+spl docs <name>                   # Document Q&A workspace
+spl docs <name> -l <snapshot_id>  # Resume from snapshot
+```
+
+Both REPLs support these commands (type `/help` inside):
+
+| Command | Description |
+|---------|-------------|
+| `/help` | Show all commands |
+| `/ls` | List sessions and snapshots |
+| `/rm` | Delete current session, start fresh |
+| `/rm <id...>` | Delete specific session(s) or snapshot(s) |
+| `/save [title]` | Save session as snapshot |
+| `/load <snap>` | Load snapshot |
+| `/head [n]` / `/tail [n]` | Show first/last messages |
+| `/history [n]` | Show input history |
+| `/stats` | Show last turn metrics |
+| `/info` | Show workspace info (docs only) |
+| `/add <paths...>` | Add documents (docs only) |
+| `/sources` | List loaded sources (docs only) |
+| `/rag ...` | Configure RAG retrieval (docs only) |
+
+### Session & Snapshot Management
+
+```bash
+spl ls                            # List all sessions and snapshots
+spl rm <id...>                    # Remove session(s) or snapshot(s)
+spl session ls                    # List sessions (detailed)
+spl session info <id>             # Show session info
+spl snapshot ls                   # List snapshots (detailed)
+spl snapshot save --session <id>  # Save session as snapshot
+spl snapshot load <snap_id>       # Load snapshot into new session
+spl snapshot rm --all             # Delete all snapshots
+```
+
+---
+
+## Using the Server API Directly
+
+The server exposes an OpenAI-compatible API. Any OpenAI SDK client works out of the box:
+
+```python
+from openai import OpenAI
+
+client = OpenAI(base_url="http://localhost:8787/v1", api_key="not-needed")
+
+response = client.chat.completions.create(
+    model="superlinear-exp-v0.1",
+    messages=[{"role": "user", "content": "Hello!"}],
+    max_tokens=100,
+)
+print(response.choices[0].message.content)
+```
+
+### Stateful Sessions
+
+The key differentiator is **stateful sessions**. Instead of resending the full conversation (and re-prefilling it) each turn, the server maintains the KV cache across turns to make long-context interaction feel closer to an interactive “running process” than a stateless API call.
+
+```python
+import requests
+from openai import OpenAI
+
+BASE = "http://localhost:8787"
+client = OpenAI(base_url=f"{BASE}/v1", api_key="not-needed")
+
+# Create a session
+requests.post(f"{BASE}/v1/sessions", json={"session_id": "my-chat"})
+
+# Turn 1 — ingests and caches
+client.chat.completions.create(
+    model="superlinear-exp-v0.1",
+    messages=[{"role": "user", "content": "My name is Alice."}],
+    extra_body={"session_id": "my-chat"},
+)
+
+# Turn 2 — only the new message is processed; prior turns are cached
+response = client.chat.completions.create(
+    model="superlinear-exp-v0.1",
+    messages=[{"role": "user", "content": "What's my name?"}],
+    extra_body={"session_id": "my-chat"},
+)
+print(response.choices[0].message.content)  # "Alice"
+
+# Cleanup
+requests.delete(f"{BASE}/v1/sessions/my-chat")
+```
+
+### Snapshots
+
+Snapshots save the full session state (KV cache + history) to disk. This lets you **resume conversations instantly** or **fork** a session into multiple branches without re-processing the context.
+
+```python
+# Save a snapshot
+requests.post(f"{BASE}/v1/sessions/my-chat/save", json={"title": "checkpoint-1"})
+# Returns: {"snapshot_id": "764cb17e..."}
+
+# Load it back (into a new or existing session)
+requests.post(f"{BASE}/v1/snapshots/764cb17e.../load", json={"session_id": "my-chat-v2"})
+```
+
+**Use cases:**
+- **Document ingestion caching** — Ingest a large corpus once, snapshot it, and load the snapshot each time you want to ask new questions.
+- **Conversation branching** — Snapshot at a decision point, then explore multiple conversation paths.
+- **Warm starts** — Pre-build a session with a system prompt + reference material, snapshot it, and load it for every new user.
+
+### Reasoning Mode
+
+Enable chain-of-thought reasoning with a token budget:
+
+```python
+response = client.chat.completions.create(
+    model="superlinear-exp-v0.1",
+    messages=[{"role": "user", "content": "Solve: 15 × 17"}],
+    extra_body={"reasoning_budget": 256},
+)
+```
+
+### Repetition Detection
+
+Automatically stop runaway generation loops:
+
+```python
+response = client.chat.completions.create(
+    model="superlinear-exp-v0.1",
+    messages=[{"role": "user", "content": "List all primes"}],
+    max_tokens=50000,
+    extra_body={
+        "repetition_detection": {
+            "enabled": True,
+            "tail_len": 1024,
+            "min_repeats": 3,
+            "max_period": 512,
+        }
+    },
+)
+# finish_reason will be "repetition" if a loop was detected
+```
+
+### API Endpoints
+
+| Endpoint | Method | Description |
+|----------|--------|-------------|
+| `/health` | GET | Health check |
+| `/v1/models` | GET | List available models |
+| `/v1/chat/completions` | POST | Chat completions (streaming supported) |
+| `/v1/sessions` | POST | Create a stateful session |
+| `/v1/sessions` | GET | List active sessions |
+| `/v1/sessions/{id}` | GET | Get session info |
+| `/v1/sessions/{id}` | DELETE | Close session |
+| `/v1/sessions/{id}/history` | GET | Get conversation history |
+| `/v1/sessions/{id}/resize` | POST | Resize session context length |
+| `/v1/sessions/{id}/save` | POST | Save session as snapshot |
+| `/v1/snapshots` | GET | List snapshots |
+| `/v1/snapshots/{id}` | GET | Get snapshot info |
+| `/v1/snapshots/{id}` | DELETE | Delete snapshot |
+| `/v1/snapshots/{id}/load` | POST | Load snapshot into session |
+
+---
+
+## Sparse Attention Kernels
+
+Under the hood, Superlinear uses custom Triton kernels implementing the [Superlinear Multi-Step Attention](https://arxiv.org/abs/2601.18401) architecture. The baseline N=2 mechanism works as follows:
+
+1. **Span-search** — For each query at position *i*, score O(√i) candidate anchors (placed at a power-law stride pattern) using a routing query Q_s against accumulated keys K_a, then select the top-k anchors.
+2. **Span-attention** — Attend to contiguous spans of O(√i) tokens centered at each selected anchor, plus a local sliding window for short-range context.
+3. **Combination** — Merge span outputs via softmax-weighted gating over the search scores, making routing end-to-end trainable.
+
+This yields O(L^{3/2}) total complexity for prefill/training while preserving **random context access**: the union of all candidate spans covers every eligible key position.
+
+A **bucketed kernel design** groups (query, span) pairs by key-block footprint for efficient GPU tiling without global sorting. See [Section 4.3 of the paper](https://arxiv.org/abs/2601.18401) for details.
+
+### Throughput
+
+Measured on a single NVIDIA B200 GPU (180 GB VRAM), batch size 1, on a modified 30B hybrid MoE model ([nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-BF16)). Prefill uses 32K chunked prefilling.
+
+| Context Length | Prefill (tok/s) | Decode (tok/s) | Decode Latency |
+|----------------|-----------------|----------------|----------------|
+| 1M tokens      | ~20,000         | ~109           | ~9.2 ms/tok    |
+| 10M tokens     | ~5,500          | ~76            | ~13.2 ms/tok   |
+
+FlashAttention-2 is faster at short contexts (<60K) but its throughput drops rapidly with length. Superlinear attention overtakes it near 60K tokens and scales into the multi-million-token regime where dense attention is impractical. See [Section 4.4 of the paper](https://arxiv.org/abs/2601.18401) for the full comparison.
+
+### Key Features
+
+- **Content-dependent routing** — Spans are selected by learned search scores, not fixed sparsity patterns.
+- **Sliding window** — Recent tokens are always attended to via a local window.
+- **GQA support** — First-class support for Grouped Query Attention.
+- **CUDA graph compatible** — Optimized for high-throughput inference with graph capture.
+
+### Parameters
+
+| Parameter | Description |
+|-----------|-------------|
+| `sw_index` | Sliding window index. Window size = `(sw_index + 1)² - 1` |
+| `num_spans` | Number of top-k spans to aggregate (typically 2–3) |
+| `backward_factor` | Factor for backward span extension from anchor |
+| `forward_factor` | Factor for forward span extension from anchor |
+| `search_power` | Exponent *p* for anchor stride pattern; balanced at 0.5, upscaled to ~0.55 for robustness |
+| `span_power` | Exponent for span length scaling; span length ∝ i^(1−span_power). Balanced at 0.5, can differ from `search_power` for redundancy |
+
+### Direct Usage
+
+```python
+import torch
+from superlinear.kernels.superlinear.attention import fused_prefill_with_swtriton_gqa
+from superlinear.kernels.superlinear.span import decode_span_attention_staged_gqa
+
+B, H_Q, H_KV, L, D = 1, 32, 8, 1024, 128
+
+Q1 = torch.randn(B, H_Q, L, D, device="cuda", dtype=torch.bfloat16)
+Q2 = torch.randn(B, H_Q, L, D, device="cuda", dtype=torch.bfloat16)
+K = torch.randn(B, H_KV, L, D, device="cuda", dtype=torch.bfloat16)
+V = torch.randn(B, H_KV, L, D, device="cuda", dtype=torch.bfloat16)
+cache_position = torch.arange(L, device="cuda")
+
+# Prefill
+output = fused_prefill_with_swtriton_gqa(
+    Q1, Q2, K, V, cache_position,
+    sw_index=3, num_spans=3,
+    backward_factor=3.0, forward_factor=1.0,
+)
+
+# Decode
+output_decode = decode_span_attention_staged_gqa(
+    Q1[:, :, -1:], Q2[:, :, -1:], K, V, cache_position[-1:],
+    sw_index=3, enable_gqa=True,
+)
+```
+
+## Project Structure
+
+```
+superlinear/
+├── kernels/           # Triton attention kernels
+│   ├── common/        # Shared utilities
+│   └── superlinear/   # Model-specific implementations
+│       ├── attention/ # Prefill and decode
+│       ├── search/    # Span search operations
+│       └── span/      # Low-level span attention
+└── engine/            # Inference engine
+    ├── adapters/      # Model adapters
+    ├── chat_engine.py # Async streaming engine
+    └── chat_types.py  # Type definitions
+
+apps/
+├── server/            # OpenAI-compatible HTTP server
+└── cli/               # Command-line interface (spl)
+```
+
+## Environment Variables
+
+| Variable | Description |
+|----------|-------------|
+| `SPAN_ATTENTION_ENABLE_ASSERTS` | Enable validation checks (default: off for CUDA graphs) |
+| `SPAN_ATTENTION_FORCE_SPARSE_META` | Force sparse bucketing strategy |
+| `SUPERLINEAR_SNAPSHOT_DIR` | Directory to store snapshots (default: `~/.cache/spl/snapshots` or `$XDG_CACHE_HOME/spl/snapshots`) |
+
+## License
+
+[Apache 2.0](LICENSE)
+
+## Citation
+
+If you use Superlinear in your research, please cite:
+
+```bibtex
+@article{huang2026superlinear,
+  title={Superlinear Multi-Step Attention},
+  author={Huang, Yufeng},
+  journal={arXiv preprint arXiv:2601.18401},
+  year={2026}
+}
+```