tokenmizer 0.2.5__tar.gz → 0.3.0__tar.gz

{tokenmizer-0.2.5 → tokenmizer-0.3.0}/CHANGELOG.md

@@ -1,5 +1,27 @@
 # Changelog
 
+## [0.3.0] — 2026-07-02 — true SSE streaming
+
+- **`stream: true` now works** — real passthrough streaming in OpenAI
+  `chat.completion.chunk` SSE format for Anthropic, OpenAI, DeepSeek,
+  Mistral, OpenRouter, Grok (OpenAI-compatible base) and Ollama. Cursor,
+  Continue.dev and every streaming client can now point at TokenMizer
+  without config changes.
+- All input-side layers (file intelligence, compression, graph memory,
+  context injection) apply to streamed requests; output trimming is
+  skipped in stream mode by design. Cache hits stream as a single chunk;
+  post-stream analytics + cache writes preserved.
+- Providers without passthrough support return an explicit 501 (never a
+  fake buffered stream). Mid-stream provider failures emit an SSE `error`
+  event instead of silently truncating.
+
+## [0.2.6] — 2026-07-02 — MCP registry readiness
+
+- NEW console script `tokenmizer-mcp` — runs the MCP stdio server directly
+  (no `python3 -m tokenmizer.mcp.server` incantation needed in client configs).
+- README carries the `mcp-name` ownership marker required by the official
+  MCP registry's PyPI package validation.
+
 ## [0.2.5] — 2026-07-02 — restart no longer breaks checkpoints
 
 ### Critical

{tokenmizer-0.2.5 → tokenmizer-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tokenmizer
-Version: 0.2.5
+Version: 0.3.0
 Summary: Reduce AI context loss by 2x. Graph-backed checkpoint and resume for any LLM session.
 Project-URL: Homepage, https://github.com/Shweta-Mishra-ai/tokenmizer
 Project-URL: Repository, https://github.com/Shweta-Mishra-ai/tokenmizer
@@ -87,12 +87,24 @@ Description-Content-Type: text/markdown
 
   <p>
     <a href="https://pypi.org/project/tokenmizer"><img src="https://img.shields.io/pypi/v/tokenmizer?color=7c6af7&style=flat-square" alt="PyPI"/></a>
-    <a href="https://pypi.org/project/tokenmizer"><img src="https://img.shields.io/pypi/pyversions/tokenmizer?color=5ee7c8&style=flat-square"/></a>
+    <a href="https://pypi.org/project/tokenmizer"><img src="https://img.shields.io/pypi/dm/tokenmizer?color=5ee7c8&style=flat-square" alt="Downloads"/></a>
+    <a href="https://github.com/Shweta-Mishra-ai/tokenmizer/actions"><img src="https://img.shields.io/github/actions/workflow/status/Shweta-Mishra-ai/tokenmizer/ci.yml?branch=main&style=flat-square&color=4ade80" alt="CI"/></a>
+    <a href="https://registry.modelcontextprotocol.io/v0/servers?search=tokenmizer"><img src="https://img.shields.io/badge/MCP%20Registry-published-5ee7c8?style=flat-square" alt="MCP Registry"/></a>
     <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-4ade80?style=flat-square"/></a>
-    <a href="https://github.com/Shweta-Mishra-ai/tokenmizer/actions"><img src="https://img.shields.io/github/actions/workflow/status/Shweta-Mishra-ai/tokenmizer/ci.yml?branch=main&style=flat-square&color=4ade80"/></a>
-    <img src="https://img.shields.io/badge/Claude%20Code-Plugin-7c6af7?style=flat-square&logo=anthropic"/>
-    <img src="https://img.shields.io/badge/MCP-Compatible-5ee7c8?style=flat-square"/>
+    <a href="https://github.com/Shweta-Mishra-ai/tokenmizer/stargazers"><img src="https://img.shields.io/github/stars/Shweta-Mishra-ai/tokenmizer?style=flat-square&color=f9d84a" alt="Stars"/></a>
   </p>
+
+  <p>
+    <a href="#quick-start"><b>Quick Start</b></a> ·
+    <a href="#how-tokenmizer-solves-it"><b>How it works</b></a> ·
+    <a href="#benchmarks"><b>Benchmarks</b></a> ·
+    <a href="#claude-code-integration"><b>Claude Code</b></a> ·
+    <a href="#contributing"><b>Contributing</b></a>
+  </p>
+
+  <img src="docs/assets/demo.gif" width="860" alt="TokenMizer demo: 40-turn session checkpointed at 87% context, resumed next day in 233 tokens"/>
+  <br/>
+  <sub>Real run: 25-node graph, checkpoint <code>ckpt_21a0959c3ddf</code>, 233-token resume. Regenerate with <code>python scripts/gen_demo_gif.py</code>.</sub>
 </div>
 
 ---
@@ -193,13 +205,9 @@ response = client.chat.completions.create(
 )
 ```
 
-> **⚠️ Streaming is not supported yet.** Set `stream: false` (or leave it unset —
-> that's the default) in every request. Requests with `stream: true` return
-> `HTTP 501`. True SSE streaming is planned for v0.3.
->
-> **Cursor:** Settings → Models → your model entry → disable "Stream responses"
-> **Continue.dev:** in `config.json`, set `"streamResponses": false` for the
-> TokenMizer provider entry
+> ✅ **Streaming works** (v0.3+): `stream: true` gives real SSE passthrough for
+> Anthropic, OpenAI, DeepSeek, Mistral, OpenRouter, Grok and Ollama. Cursor and
+> Continue.dev work with default settings — no config changes needed.
 
 ---
 
@@ -227,20 +235,23 @@ Then use skills directly:
 
 ### Option B — MCP server
 
+mcp-name: io.github.Shweta-Mishra-ai/tokenmizer
+
 Add to `~/.claude/settings.json`:
 
 ```json
 {
   "mcpServers": {
     "tokenmizer": {
-      "command": "python3",
-      "args": ["-m", "tokenmizer.mcp.server"],
+      "command": "tokenmizer-mcp",
       "env": { "TOKENMIZER_URL": "http://localhost:8000" }
     }
   }
 }
 ```
 
+(`tokenmizer-mcp` is installed with the package; `python3 -m tokenmizer.mcp.server` also works.)
+
 ---
 
 ## Other Tools
@@ -502,16 +513,49 @@ tokenmizer stats
 
 ---
 
+## Roadmap
+
+| Version | Focus |
+|---|---|
+| **v0.3** | SSE streaming passthrough (checkpoint on stream close) |
+| v0.4 | Cross-session memory · embedding-based edge linking |
+| v0.5 | Per-node storage schema (scale past 200-node graphs) |
+| Research | Real-transcript benchmark suite → paper ([tokenmizer-research](https://github.com/Shweta-Mishra-ai/tokenmizer-research)) |
+
+Have a use case that doesn't fit? [Open an issue](https://github.com/Shweta-Mishra-ai/tokenmizer/issues/new/choose) — extraction misses have their own issue template.
+
+---
+
 ## Contributing
 
-See [CONTRIBUTING.md](CONTRIBUTING.md). Graph extraction contributions are the highest priority.
+Contributions welcome — this project merges fast (median PR review < 1 day).
 
 ```bash
 git clone https://github.com/Shweta-Mishra-ai/tokenmizer
+cd tokenmizer
 pip install -e ".[dev]"
-pytest tests/ -v && ruff check tokenmizer/
+pytest tests/ -v && ruff check tokenmizer/     # 218 tests, must stay green
+python scripts/mcp_e2e_check.py                # full-pipeline e2e check
 ```
 
+**Highest-impact areas right now:**
+
+1. **Graph extraction quality** — real-world transcripts where extraction misses tasks/decisions (file an [extraction-miss issue](.github/ISSUE_TEMPLATE/extraction_miss.md) even if you don't fix it — the failing transcript itself is the contribution)
+2. **SSE streaming** (v0.3 headline feature)
+3. **Benchmark sessions** — add a real session + ground truth to `benchmarks/`
+
+Every PR runs the full CI gauntlet (tests × 3 Python versions, lint, Docker build). See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines and [TESTING.md](TESTING.md) for the test architecture.
+
+---
+
+## Support the project
+
+TokenMizer is built and maintained by one person. If it saved you tokens, time, or a lost session:
+
+- ⭐ **[Star the repo](https://github.com/Shweta-Mishra-ai/tokenmizer)** — the single best way to help others find it
+- 🐛 [Report a bug](https://github.com/Shweta-Mishra-ai/tokenmizer/issues) — especially extraction misses
+- 📣 Share your before/after token numbers (`tokenmizer stats`) — real usage data shapes the roadmap
+
 ---
 
 ## License
@@ -522,4 +566,6 @@ MIT © [Shweta Mishra](https://github.com/Shweta-Mishra-ai)
 
 <div align="center">
   <sub>Built for developers who spend too much time re-explaining their projects to AI.</sub>
+  <br/><br/>
+  <a href="https://github.com/Shweta-Mishra-ai/tokenmizer"><img src="https://img.shields.io/github/stars/Shweta-Mishra-ai/tokenmizer?style=social" alt="GitHub stars"/></a>
 </div>

{tokenmizer-0.2.5 → tokenmizer-0.3.0}/README.md

@@ -12,12 +12,24 @@
 
   <p>
     <a href="https://pypi.org/project/tokenmizer"><img src="https://img.shields.io/pypi/v/tokenmizer?color=7c6af7&style=flat-square" alt="PyPI"/></a>
-    <a href="https://pypi.org/project/tokenmizer"><img src="https://img.shields.io/pypi/pyversions/tokenmizer?color=5ee7c8&style=flat-square"/></a>
+    <a href="https://pypi.org/project/tokenmizer"><img src="https://img.shields.io/pypi/dm/tokenmizer?color=5ee7c8&style=flat-square" alt="Downloads"/></a>
+    <a href="https://github.com/Shweta-Mishra-ai/tokenmizer/actions"><img src="https://img.shields.io/github/actions/workflow/status/Shweta-Mishra-ai/tokenmizer/ci.yml?branch=main&style=flat-square&color=4ade80" alt="CI"/></a>
+    <a href="https://registry.modelcontextprotocol.io/v0/servers?search=tokenmizer"><img src="https://img.shields.io/badge/MCP%20Registry-published-5ee7c8?style=flat-square" alt="MCP Registry"/></a>
     <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-4ade80?style=flat-square"/></a>
-    <a href="https://github.com/Shweta-Mishra-ai/tokenmizer/actions"><img src="https://img.shields.io/github/actions/workflow/status/Shweta-Mishra-ai/tokenmizer/ci.yml?branch=main&style=flat-square&color=4ade80"/></a>
-    <img src="https://img.shields.io/badge/Claude%20Code-Plugin-7c6af7?style=flat-square&logo=anthropic"/>
-    <img src="https://img.shields.io/badge/MCP-Compatible-5ee7c8?style=flat-square"/>
+    <a href="https://github.com/Shweta-Mishra-ai/tokenmizer/stargazers"><img src="https://img.shields.io/github/stars/Shweta-Mishra-ai/tokenmizer?style=flat-square&color=f9d84a" alt="Stars"/></a>
   </p>
+
+  <p>
+    <a href="#quick-start"><b>Quick Start</b></a> ·
+    <a href="#how-tokenmizer-solves-it"><b>How it works</b></a> ·
+    <a href="#benchmarks"><b>Benchmarks</b></a> ·
+    <a href="#claude-code-integration"><b>Claude Code</b></a> ·
+    <a href="#contributing"><b>Contributing</b></a>
+  </p>
+
+  <img src="docs/assets/demo.gif" width="860" alt="TokenMizer demo: 40-turn session checkpointed at 87% context, resumed next day in 233 tokens"/>
+  <br/>
+  <sub>Real run: 25-node graph, checkpoint <code>ckpt_21a0959c3ddf</code>, 233-token resume. Regenerate with <code>python scripts/gen_demo_gif.py</code>.</sub>
 </div>
 
 ---
@@ -118,13 +130,9 @@ response = client.chat.completions.create(
 )
 ```
 
-> **⚠️ Streaming is not supported yet.** Set `stream: false` (or leave it unset —
-> that's the default) in every request. Requests with `stream: true` return
-> `HTTP 501`. True SSE streaming is planned for v0.3.
->
-> **Cursor:** Settings → Models → your model entry → disable "Stream responses"
-> **Continue.dev:** in `config.json`, set `"streamResponses": false` for the
-> TokenMizer provider entry
+> ✅ **Streaming works** (v0.3+): `stream: true` gives real SSE passthrough for
+> Anthropic, OpenAI, DeepSeek, Mistral, OpenRouter, Grok and Ollama. Cursor and
+> Continue.dev work with default settings — no config changes needed.
 
 ---
 
@@ -152,20 +160,23 @@ Then use skills directly:
 
 ### Option B — MCP server
 
+mcp-name: io.github.Shweta-Mishra-ai/tokenmizer
+
 Add to `~/.claude/settings.json`:
 
 ```json
 {
   "mcpServers": {
     "tokenmizer": {
-      "command": "python3",
-      "args": ["-m", "tokenmizer.mcp.server"],
+      "command": "tokenmizer-mcp",
       "env": { "TOKENMIZER_URL": "http://localhost:8000" }
     }
   }
 }
 ```
 
+(`tokenmizer-mcp` is installed with the package; `python3 -m tokenmizer.mcp.server` also works.)
+
 ---
 
 ## Other Tools
@@ -427,16 +438,49 @@ tokenmizer stats
 
 ---
 
+## Roadmap
+
+| Version | Focus |
+|---|---|
+| **v0.3** | SSE streaming passthrough (checkpoint on stream close) |
+| v0.4 | Cross-session memory · embedding-based edge linking |
+| v0.5 | Per-node storage schema (scale past 200-node graphs) |
+| Research | Real-transcript benchmark suite → paper ([tokenmizer-research](https://github.com/Shweta-Mishra-ai/tokenmizer-research)) |
+
+Have a use case that doesn't fit? [Open an issue](https://github.com/Shweta-Mishra-ai/tokenmizer/issues/new/choose) — extraction misses have their own issue template.
+
+---
+
 ## Contributing
 
-See [CONTRIBUTING.md](CONTRIBUTING.md). Graph extraction contributions are the highest priority.
+Contributions welcome — this project merges fast (median PR review < 1 day).
 
 ```bash
 git clone https://github.com/Shweta-Mishra-ai/tokenmizer
+cd tokenmizer
 pip install -e ".[dev]"
-pytest tests/ -v && ruff check tokenmizer/
+pytest tests/ -v && ruff check tokenmizer/     # 218 tests, must stay green
+python scripts/mcp_e2e_check.py                # full-pipeline e2e check
 ```
 
+**Highest-impact areas right now:**
+
+1. **Graph extraction quality** — real-world transcripts where extraction misses tasks/decisions (file an [extraction-miss issue](.github/ISSUE_TEMPLATE/extraction_miss.md) even if you don't fix it — the failing transcript itself is the contribution)
+2. **SSE streaming** (v0.3 headline feature)
+3. **Benchmark sessions** — add a real session + ground truth to `benchmarks/`
+
+Every PR runs the full CI gauntlet (tests × 3 Python versions, lint, Docker build). See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines and [TESTING.md](TESTING.md) for the test architecture.
+
+---
+
+## Support the project
+
+TokenMizer is built and maintained by one person. If it saved you tokens, time, or a lost session:
+
+- ⭐ **[Star the repo](https://github.com/Shweta-Mishra-ai/tokenmizer)** — the single best way to help others find it
+- 🐛 [Report a bug](https://github.com/Shweta-Mishra-ai/tokenmizer/issues) — especially extraction misses
+- 📣 Share your before/after token numbers (`tokenmizer stats`) — real usage data shapes the roadmap
+
 ---
 
 ## License
@@ -447,4 +491,6 @@ MIT © [Shweta Mishra](https://github.com/Shweta-Mishra-ai)
 
 <div align="center">
   <sub>Built for developers who spend too much time re-explaining their projects to AI.</sub>
+  <br/><br/>
+  <a href="https://github.com/Shweta-Mishra-ai/tokenmizer"><img src="https://img.shields.io/github/stars/Shweta-Mishra-ai/tokenmizer?style=social" alt="GitHub stars"/></a>
 </div>

{tokenmizer-0.2.5 → tokenmizer-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "tokenmizer"
-version = "0.2.5"
+version = "0.3.0"
 description = "Reduce AI context loss by 2x. Graph-backed checkpoint and resume for any LLM session."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -73,6 +73,7 @@ all = [
 
 [project.scripts]
 tokenmizer = "tokenmizer.cli:app"
+tokenmizer-mcp = "tokenmizer.mcp.server:run_stdio_server"
 
 [project.urls]
 Homepage = "https://github.com/Shweta-Mishra-ai/tokenmizer"

tokenmizer-0.3.0/scripts/gen_demo_gif.py

@@ -0,0 +1,108 @@
+# -*- coding: utf-8 -*-
+"""Generate docs/assets/demo.gif — terminal-style demo.
+
+All numbers/outputs shown are REAL, captured from an actual run of
+GraphMemory + CheckpointManager on the 40-turn-scale demo session
+(checkpoint ckpt_21a0959c3ddf, 25 nodes, 233-token resume).
+Run: python scripts/gen_demo_gif.py [output_path]
+"""
+import os
+import sys
+
+from PIL import Image, ImageDraw, ImageFont
+
+OUT = sys.argv[1] if len(sys.argv) > 1 else "docs/assets/demo.gif"
+W, H = 920, 500
+BG, FG = (13, 17, 23), (201, 209, 217)
+GREEN, PURPLE, YELLOW = (63, 185, 80), (137, 87, 229), (210, 153, 34)
+DIM, CYAN = (110, 118, 129), (57, 197, 187)
+
+font = ImageFont.truetype("consola.ttf", 17)
+bold = ImageFont.truetype("consolab.ttf", 17)
+big = ImageFont.truetype("consolab.ttf", 26)
+
+
+def frame(lines, title="tokenmizer - demo"):
+    im = Image.new("RGB", (W, H), BG)
+    d = ImageDraw.Draw(im)
+    d.rounded_rectangle([8, 8, W - 8, H - 8], 10, outline=(48, 54, 61), width=2)
+    for i, c in enumerate([(255, 95, 86), (255, 189, 46), (39, 201, 63)]):
+        d.ellipse([24 + i * 24, 20, 38 + i * 24, 34], fill=c)
+    d.text((W // 2 - 100, 18), title, font=font, fill=DIM)
+    y = 56
+    for txt, col, f in lines:
+        d.text((28, y), txt, font=f, fill=col)
+        y += 24
+    return im
+
+
+frames, durs = [], []
+
+
+def add(lines, ms):
+    frames.append(frame(lines))
+    durs.append(ms)
+
+
+add([("$ pip install tokenmizer", FG, bold),
+     ("Successfully installed tokenmizer-0.2.6", GREEN, font),
+     ("", FG, font),
+     ("$ export TOKENMIZER_ANTHROPIC_API_KEY=sk-ant-...", FG, bold),
+     ("$ tokenmizer serve", FG, bold),
+     ("  Proxy:     http://localhost:8000/v1/chat/completions", CYAN, font),
+     ("  Dashboard: http://localhost:8000", CYAN, font),
+     ("", FG, font),
+     ("# point any OpenAI-compatible client at localhost:8000 -", DIM, font),
+     ("# one line changed, zero code rewritten", DIM, font)], 2600)
+
+add([("session: fastapi-auth            model: claude-fable-5", DIM, font),
+     ("", FG, font),
+     ("you > Let's build a FastAPI auth service with JWT + PostgreSQL", FG, font),
+     ("ai  > Decided: PostgreSQL (concurrent writes). Files: api/auth.py ...", DIM, font),
+     ("you > Use bcrypt and Python 3.12", FG, font),
+     ("ai  > Decided: bcrypt (industry standard) ...", DIM, font),
+     ("you > Login returns 422, fix it", FG, font),
+     ("ai  > Fixed: missing email validation in LoginRequest ...", DIM, font),
+     ("you > Add refresh tokens with Redis", FG, font),
+     ("ai  > Decided: Redis for refresh tokens (faster revocation) ...", DIM, font),
+     ("            ... 40 turns later ...", YELLOW, bold),
+     ("", FG, font),
+     ("  context window: 87% full", YELLOW, bold)], 3200)
+
+add([("  context window: 87% full", YELLOW, bold),
+     ("", FG, font),
+     ("  * auto-checkpoint triggered *", PURPLE, bold),
+     ("", FG, font),
+     ("  graph: 25 nodes extracted (tasks / decisions / files / errors)", FG, font),
+     ("  checkpoint: ckpt_21a0959c3ddf   saved to SQLite", GREEN, font),
+     ("  resume size: 233 tokens", GREEN, bold),
+     ("", FG, font),
+     ("  session history: ~5,800 tokens -> 233 tokens", CYAN, bold)], 3000)
+
+add([("  -- next day, fresh session --", DIM, font),
+     ("", FG, font),
+     ("$ tokenmizer resume fastapi-auth", FG, bold),
+     ("", FG, font),
+     ("Goal: a FastAPI auth service with JWT and PostgreSQL", GREEN, font),
+     ("Working on: rate limiting with slowapi", FG, font),
+     ("Done: tests/test_auth.py (12 tests passing) | 422 login fix", FG, font),
+     ("Decided: PostgreSQL | bcrypt | Python 3.12 |", FG, font),
+     ("         Redis for refresh tokens (faster revocation)", FG, font),
+     ("Changes: 'Use Redis' -> 'Redis for refresh token storage'", YELLOW, font),
+     ("Continue from: Write the tests", CYAN, font),
+     ("", FG, font),
+     ("  [233 tokens - paste into any new session and keep going]", DIM, font)], 4200)
+
+im = Image.new("RGB", (W, H), BG)
+d = ImageDraw.Draw(im)
+d.rounded_rectangle([8, 8, W - 8, H - 8], 10, outline=(48, 54, 61), width=2)
+d.text((W // 2 - 260, H // 2 - 70), "never re-explain your project again", font=big, fill=FG)
+d.text((W // 2 - 215, H // 2 - 20), "~5,800 tokens  ->  233 tokens resume", font=big, fill=GREEN)
+d.text((W // 2 - 130, H // 2 + 40), "pip install tokenmizer", font=bold, fill=PURPLE)
+frames.append(im)
+durs.append(3500)
+
+os.makedirs(os.path.dirname(OUT), exist_ok=True)
+frames[0].save(OUT, save_all=True, append_images=frames[1:],
+               duration=durs, loop=0, optimize=True)
+print("demo.gif:", os.path.getsize(OUT) // 1024, "KB,", len(frames), "frames")

tokenmizer-0.3.0/server.json

@@ -0,0 +1,34 @@
+{
+  "$schema": "https://static.modelcontextprotocol.io/schemas/2025-12-11/server.schema.json",
+  "name": "io.github.Shweta-Mishra-ai/tokenmizer",
+  "description": "Graph-backed session memory for LLMs: checkpoint, resume, file analysis, token-savings stats.",
+  "repository": {
+    "url": "https://github.com/Shweta-Mishra-ai/tokenmizer",
+    "source": "github"
+  },
+  "version": "0.2.6",
+  "packages": [
+    {
+      "registryType": "pypi",
+      "identifier": "tokenmizer",
+      "version": "0.2.6",
+      "transport": { "type": "stdio" },
+      "environmentVariables": [
+        {
+          "name": "TOKENMIZER_URL",
+          "description": "Base URL of the running TokenMizer proxy (default http://localhost:8000)",
+          "isRequired": false,
+          "format": "string",
+          "isSecret": false
+        },
+        {
+          "name": "TOKENMIZER_API_KEY",
+          "description": "API key if the proxy has auth enabled",
+          "isRequired": false,
+          "format": "string",
+          "isSecret": true
+        }
+      ]
+    }
+  ]
+}

{tokenmizer-0.2.5 → tokenmizer-0.3.0}/tests/integration/test_api_endpoint.py

@@ -126,7 +126,40 @@ class TestChatCompletionsE2E:
         assert kwargs.get("temperature") == 0.0
         assert kwargs.get("top_p") == 0.9
 
-    def test_stream_returns_501_with_clear_message(self, client):
+    def test_streaming_returns_sse_chunks(self, client):
+        """v0.3: stream=true must return true SSE passthrough in OpenAI
+        chat.completion.chunk format, ending with data: [DONE]."""
+        c, fake = client
+
+        async def fake_stream(**kwargs):
+            for piece in ("Hello", " streamed", " world"):
+                yield piece
+
+        fake.chat_stream = fake_stream
+        r = c.post("/v1/chat/completions", json={
+            "messages": [{"role": "user", "content": "stream me a story please"}],
+            "stream": True,
+        })
+        assert r.status_code == 200, r.text
+        assert "text/event-stream" in r.headers["content-type"]
+        body = r.text
+        assert "chat.completion.chunk" in body
+        assert '"content": "Hello"' in body
+        assert '"content": " streamed"' in body
+        assert '"finish_reason": "stop"' in body
+        assert body.rstrip().endswith("data: [DONE]")
+
+    def test_streaming_unsupported_provider_returns_501(self, client, monkeypatch):
+        """Providers without chat_stream override must get a clear 501,
+        not a fake buffered stream."""
+        from tokenmizer.api import app as app_module
+        from tokenmizer.providers.providers import BaseProvider
+
+        class NoStreamProvider(BaseProvider):
+            async def _call(self, *a, **k):  # pragma: no cover
+                raise NotImplementedError
+
+        monkeypatch.setattr(app_module, "_get_provider", lambda: NoStreamProvider())
         c, _ = client
         r = c.post("/v1/chat/completions", json={
             "messages": [{"role": "user", "content": "hi"}],

{tokenmizer-0.2.5 → tokenmizer-0.3.0}/tokenmizer/__init__.py

@@ -1,6 +1,6 @@
 """TokenMizer — Never lose your AI context again."""
 
-__version__ = "0.2.5"
+__version__ = "0.3.0"
 __all__ = ["GraphMemory", "CheckpointManager", "get_settings"]
 
 

{tokenmizer-0.2.5 → tokenmizer-0.3.0}/tokenmizer/api/app.py

@@ -16,7 +16,7 @@ from typing import Optional
 
 from fastapi import Depends, FastAPI, HTTPException, Request
 from fastapi.middleware.cors import CORSMiddleware
-from fastapi.responses import HTMLResponse
+from fastapi.responses import HTMLResponse, StreamingResponse
 from pydantic import BaseModel
 
 from tokenmizer.analytics.engine import AnalyticsEngine
@@ -261,7 +261,7 @@ async def lifespan(app: FastAPI):
 app = FastAPI(
     title="TokenMizer",
     description="Never lose your AI context again.",
-    version="0.2.5",
+    version="0.3.0",
     lifespan=lifespan,
     docs_url="/docs",
     redoc_url="/redoc",
@@ -558,17 +558,6 @@ async def _call_provider(
             output_tokens = count_tokens(cached.response, model)
             return cached.response, 0, output_tokens, 0.0
 
-    # Streaming check
-    if req.stream:
-        raise HTTPException(
-            status_code=501,
-            detail=(
-                "Streaming is not yet supported by the TokenMizer proxy. "
-                "Set stream=False in your request, or connect directly to "
-                "your provider for streaming. True SSE streaming is planned for v0.3."
-            ),
-        )
-
     # LLM call
     # NOTE: `messages` is already redacted — redaction now happens once at
     # ingestion in chat_completions() so every downstream consumer (this call,
@@ -609,6 +598,83 @@ async def _call_provider(
     return response_text, input_tokens, output_tokens, latency_ms
 
 
+def _stream_response(req, messages, model, user_content, session_id,
+                     savings, orig_input_tokens) -> StreamingResponse:
+    """True SSE passthrough (OpenAI chat.completion.chunk format).
+
+    Cache hits stream as a single chunk. After the stream closes, analytics
+    and the semantic-cache write run on the accumulated text — same
+    bookkeeping as the non-stream path, minus output trimming.
+    """
+    import json as _json
+
+    from tokenmizer.providers.providers import BaseProvider, ProviderError
+
+    provider = _get_provider()
+    if getattr(type(provider), "chat_stream", None) is BaseProvider.chat_stream:
+        raise HTTPException(
+            status_code=501,
+            detail=(f"Streaming passthrough is not implemented for provider "
+                    f"'{settings.provider}' yet (supported: anthropic, openai, "
+                    f"deepseek, mistral, openrouter, grok, ollama). "
+                    f"Set stream=false for this provider."),
+        )
+
+    resp_id = f"chatcmpl-{uuid.uuid4().hex[:12]}"
+    created = int(time.time())
+
+    def _chunk(delta: dict, finish: str | None = None) -> str:
+        return "data: " + _json.dumps({
+            "id": resp_id, "object": "chat.completion.chunk",
+            "created": created, "model": model, "session_id": session_id,
+            "choices": [{"index": 0, "delta": delta, "finish_reason": finish}],
+        }) + "\n\n"
+
+    async def _gen():
+        full_text = ""
+        t0 = time.monotonic()
+        yield _chunk({"role": "assistant"})
+        try:
+            cached = (_cache.get(user_content, session_id=session_id)
+                      if settings.cache.enabled and user_content else None)
+            if cached:
+                full_text = cached.response
+                yield _chunk({"content": full_text})
+            else:
+                async for piece in provider.chat_stream(
+                    messages=messages, model=model,
+                    max_tokens=req.max_tokens or 4096,
+                    **_sampling_kwargs(req),
+                ):
+                    full_text += piece
+                    yield _chunk({"content": piece})
+        except ProviderError as e:
+            # Mid-stream failure: SSE can't change the status code anymore —
+            # emit an explicit error event instead of silently truncating.
+            yield "data: " + _json.dumps({"error": {"message": str(e), "type": "provider_error"}}) + "\n\n"
+        yield _chunk({}, finish="stop")
+        yield "data: [DONE]\n\n"
+
+        # Post-stream bookkeeping
+        latency_ms = (time.monotonic() - t0) * 1000
+        output_tokens = count_tokens(full_text, model)
+        input_tokens = count_messages_tokens(messages, model)
+        if settings.cache.enabled and user_content and full_text:
+            _cache.set(user_content, full_text, input_tokens=input_tokens,
+                       output_tokens=output_tokens, session_id=session_id)
+        _analytics.record(
+            session_id=session_id, provider=settings.provider, model=model,
+            input_tokens_original=orig_input_tokens,
+            input_tokens_sent=input_tokens, output_tokens=output_tokens,
+            tokens_saved=sum(savings.values()), latency_ms=latency_ms,
+            cache_hit=False, layer_savings=savings,
+        )
+
+    return StreamingResponse(_gen(), media_type="text/event-stream",
+                             headers={"Cache-Control": "no-cache",
+                                      "X-Accel-Buffering": "no"})
+
+
 @app.post("/v1/chat/completions", dependencies=[Depends(verify_api_key), Depends(injection_guard)])
 async def chat_completions(req: ChatRequest, request: Request):
     """
@@ -662,6 +728,13 @@ async def chat_completions(req: ChatRequest, request: Request):
             session_id, graph, raw_messages, messages, model, savings, user_query
         )
 
+    # Streaming: true SSE passthrough (v0.3). Output-trimming is skipped in
+    # stream mode (can't trim tokens that already left the building) — all
+    # input-side layers (file intel, compression, graph context) still apply.
+    if req.stream:
+        return _stream_response(req, messages, model, user_content,
+                                session_id, savings, orig_input_tokens)
+
     # Layer 5: call provider (or return cache hit)
     response_text, input_tokens_actual, output_tokens, latency_ms = await _call_provider(
         req, messages, model, user_content, session_id, savings

{tokenmizer-0.2.5 → tokenmizer-0.3.0}/tokenmizer/providers/providers.py

@@ -107,6 +107,21 @@ class BaseProvider(ABC):
 
         raise ProviderError(self.__class__.__name__, "max_retries", "All retry attempts exhausted", retryable=False)
 
+    def chat_stream(self, messages: list[dict], model: str = "",
+                    max_tokens: int = 4096, system: str = "", **kwargs):
+        """Async generator yielding text chunks as the provider produces them.
+
+        Providers that support true streaming override this. The base
+        implementation raises so the API layer can return a clear 501 for
+        providers where passthrough streaming isn't implemented yet, instead
+        of silently degrading to a fake (buffered) stream.
+        """
+        raise ProviderError(
+            self.__class__.__name__, "stream_not_supported",
+            f"Streaming passthrough not implemented for {self.__class__.__name__}",
+            retryable=False,
+        )
+
 
 # ── Anthropic ─────────────────────────────────────────────────────────────────
 
@@ -179,6 +194,37 @@ class AnthropicProvider(BaseProvider):
             retryable = e.status_code in (500, 502, 503, 529)
             raise ProviderError("anthropic", f"http_{e.status_code}", str(e), retryable=retryable)
 
+    async def chat_stream(self, messages: list[dict], model: str = "",
+                          max_tokens: int = 4096, system: str = "", **kwargs):
+        """True SSE passthrough — yields text chunks as Anthropic streams them."""
+        try:
+            import anthropic
+        except ImportError:
+            raise ImportError("pip install anthropic")
+        model = model or self.default_model
+        client = anthropic.AsyncAnthropic(api_key=self.api_key)
+
+        sys_parts = [m["content"] for m in messages if m.get("role") == "system"]
+        if system:
+            sys_parts.insert(0, system)
+        conv = [m for m in messages if m.get("role") != "system"]
+        kwargs_clean = _sampling(kwargs)
+        if "stop" in kwargs_clean:
+            kwargs_clean["stop_sequences"] = _as_stop_list(kwargs_clean.pop("stop"))
+        if sys_parts:
+            kwargs_clean["system"] = "\n\n".join(sys_parts)
+
+        try:
+            async with client.messages.stream(
+                model=model, messages=conv, max_tokens=max_tokens, **kwargs_clean
+            ) as s:
+                async for chunk in s.text_stream:
+                    yield chunk
+        except anthropic.RateLimitError as e:
+            raise ProviderError("anthropic", "rate_limit", str(e), retryable=True)
+        except anthropic.APIStatusError as e:
+            raise ProviderError("anthropic", f"http_{e.status_code}", str(e), retryable=False)
+
 
 # ── OpenAI ────────────────────────────────────────────────────────────────────
 
@@ -237,6 +283,37 @@ class OpenAIProvider(BaseProvider):
             retryable = "rate" in err or "server" in err or "timeout" in err
             raise ProviderError("openai", "api_error", str(e), retryable=retryable)
 
+    async def chat_stream(self, messages: list[dict], model: str = "",
+                          max_tokens: int = 4096, system: str = "", **kwargs):
+        """True SSE passthrough for OpenAI and all OpenAI-compatible providers
+        (DeepSeek, Mistral, OpenRouter, Grok inherit this)."""
+        try:
+            from openai import AsyncOpenAI
+        except ImportError:
+            raise ImportError("pip install openai")
+        model = model or self.default_model
+        client = AsyncOpenAI(
+            api_key=self.api_key,
+            **({"base_url": self._base_url} if self._base_url else {}),
+        )
+        all_messages = messages[:]
+        if system:
+            all_messages = [{"role": "system", "content": system}] + all_messages
+        try:
+            stream = await client.chat.completions.create(
+                model=model, messages=all_messages, max_tokens=max_tokens,
+                stream=True, **_sampling(kwargs),
+            )
+            async for chunk in stream:
+                delta = chunk.choices[0].delta.content if chunk.choices else None
+                if delta:
+                    yield delta
+        except Exception as e:
+            err = str(e).lower()
+            retryable = "rate" in err or "server" in err or "timeout" in err
+            raise ProviderError(self.__class__.__name__.lower().replace("provider", ""),
+                                "api_error", str(e), retryable=retryable)
+
 
 # ── DeepSeek ──────────────────────────────────────────────────────────────────
 
@@ -427,6 +504,37 @@ class OllamaProvider(BaseProvider):
             except Exception as e:
                 raise ProviderError("ollama", "api_error", str(e), retryable=True)
 
+    async def chat_stream(self, messages: list[dict], model: str = "",
+                          max_tokens: int = 4096, system: str = "", **kwargs):
+        """Streaming passthrough for local Ollama models."""
+        import json as _json
+
+        import httpx
+        model = model or self.default_model
+        all_messages = messages[:]
+        if system:
+            all_messages = [{"role": "system", "content": system}] + all_messages
+        payload = {"model": model, "messages": all_messages, "stream": True,
+                   "options": {"num_predict": max_tokens}}
+        try:
+            async with httpx.AsyncClient(timeout=120) as client:
+                async with client.stream("POST", f"{self._base_url}/api/chat",
+                                         json=payload) as r:
+                    r.raise_for_status()
+                    async for line in r.aiter_lines():
+                        if not line.strip():
+                            continue
+                        data = _json.loads(line)
+                        chunk = data.get("message", {}).get("content", "")
+                        if chunk:
+                            yield chunk
+                        if data.get("done"):
+                            break
+        except ProviderError:
+            raise
+        except Exception as e:
+            raise ProviderError("ollama", "api_error", str(e), retryable=True)
+
 
 # ── Registry ──────────────────────────────────────────────────────────────────