tokenmizer 0.2.4__tar.gz → 0.2.5__tar.gz

{tokenmizer-0.2.4 → tokenmizer-0.2.5}/CHANGELOG.md

@@ -1,5 +1,28 @@
 # Changelog
 
+## [0.2.5] — 2026-07-02 — restart no longer breaks checkpoints
+
+### Critical
+- **`graph_memory/graph.py`:** `_load()` rebuilt nodes/edges from SQLite
+  without converting `type`/`status` strings back to their str-Enum types.
+  Str-Enum equality kept passing (which hid the bug from every unit test),
+  but any `.value` access crashed — concretely, **after a server restart,
+  checkpointing any reloaded session returned HTTP 500**, and resume paths
+  touching `.value` were equally broken. Enums are now restored on load;
+  nodes/edges with unrecognized types (forward-compat) are skipped with a
+  warning instead of killing the whole load. Found by the new MCP e2e
+  check, not unit tests — reloaded graphs were never checkpoint-ed in tests
+  before. Four regression tests added.
+
+### MCP server
+- `serverInfo.version` no longer hardcoded (was stale "0.2.3").
+- `resume_session` no longer claims "No checkpoint found" when a checkpoint
+  exists but its graph was empty — the two cases now produce distinct,
+  accurate messages.
+- NEW `scripts/mcp_e2e_check.py`: boots the real proxy in-process, spawns
+  the real MCP stdio server, and exercises handshake + all 5 tools
+  end-to-end. This is the check that caught the restart bug.
+
 ## [0.2.4] — 2026-07-02 — launch-readiness fixes
 
 ### Critical — the endpoint did not work

{tokenmizer-0.2.4 → tokenmizer-0.2.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tokenmizer
-Version: 0.2.4
+Version: 0.2.5
 Summary: Reduce AI context loss by 2x. Graph-backed checkpoint and resume for any LLM session.
 Project-URL: Homepage, https://github.com/Shweta-Mishra-ai/tokenmizer
 Project-URL: Repository, https://github.com/Shweta-Mishra-ai/tokenmizer
@@ -86,6 +86,8 @@ Description-Content-Type: text/markdown
   </p>
 
   <p>
+    <a href="https://pypi.org/project/tokenmizer"><img src="https://img.shields.io/pypi/v/tokenmizer?color=7c6af7&style=flat-square" alt="PyPI"/></a>
+    <a href="https://pypi.org/project/tokenmizer"><img src="https://img.shields.io/pypi/pyversions/tokenmizer?color=5ee7c8&style=flat-square"/></a>
     <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-4ade80?style=flat-square"/></a>
     <a href="https://github.com/Shweta-Mishra-ai/tokenmizer/actions"><img src="https://img.shields.io/github/actions/workflow/status/Shweta-Mishra-ai/tokenmizer/ci.yml?branch=main&style=flat-square&color=4ade80"/></a>
     <img src="https://img.shields.io/badge/Claude%20Code-Plugin-7c6af7?style=flat-square&logo=anthropic"/>
@@ -146,22 +148,16 @@ History is **never deleted**. "Why did we switch from React to Next.js?" — alw
 
 ### 1. Install
 
-> Not yet published to PyPI — install from source (PyPI release is on the
-> roadmap; the `pip install tokenmizer` short form will work after that).
-
 ```bash
-git clone https://github.com/Shweta-Mishra-ai/tokenmizer
-cd tokenmizer
-
 # Recommended
-pip install -e ".[anthropic,cache]"
+pip install "tokenmizer[anthropic,cache]"
 
 # All providers
-pip install -e ".[anthropic,openai,gemini,cohere,cache]"
+pip install "tokenmizer[anthropic,openai,gemini,cohere,cache]"
 
 # No key? Use Ollama (free, local)
 brew install ollama && ollama pull llama3
-pip install -e .
+pip install tokenmizer
 ```
 
 ### 2. Set your API key

{tokenmizer-0.2.4 → tokenmizer-0.2.5}/README.md

@@ -11,6 +11,8 @@
   </p>
 
   <p>
+    <a href="https://pypi.org/project/tokenmizer"><img src="https://img.shields.io/pypi/v/tokenmizer?color=7c6af7&style=flat-square" alt="PyPI"/></a>
+    <a href="https://pypi.org/project/tokenmizer"><img src="https://img.shields.io/pypi/pyversions/tokenmizer?color=5ee7c8&style=flat-square"/></a>
     <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-4ade80?style=flat-square"/></a>
     <a href="https://github.com/Shweta-Mishra-ai/tokenmizer/actions"><img src="https://img.shields.io/github/actions/workflow/status/Shweta-Mishra-ai/tokenmizer/ci.yml?branch=main&style=flat-square&color=4ade80"/></a>
     <img src="https://img.shields.io/badge/Claude%20Code-Plugin-7c6af7?style=flat-square&logo=anthropic"/>
@@ -71,22 +73,16 @@ History is **never deleted**. "Why did we switch from React to Next.js?" — alw
 
 ### 1. Install
 
-> Not yet published to PyPI — install from source (PyPI release is on the
-> roadmap; the `pip install tokenmizer` short form will work after that).
-
 ```bash
-git clone https://github.com/Shweta-Mishra-ai/tokenmizer
-cd tokenmizer
-
 # Recommended
-pip install -e ".[anthropic,cache]"
+pip install "tokenmizer[anthropic,cache]"
 
 # All providers
-pip install -e ".[anthropic,openai,gemini,cohere,cache]"
+pip install "tokenmizer[anthropic,openai,gemini,cohere,cache]"
 
 # No key? Use Ollama (free, local)
 brew install ollama && ollama pull llama3
-pip install -e .
+pip install tokenmizer
 ```
 
 ### 2. Set your API key

{tokenmizer-0.2.4 → tokenmizer-0.2.5}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "tokenmizer"
-version = "0.2.4"
+version = "0.2.5"
 description = "Reduce AI context loss by 2x. Graph-backed checkpoint and resume for any LLM session."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -108,6 +108,4 @@ omit = [
 [tool.coverage.report]
 # Honest floor over ALL product code (previously 70, but with api/app.py,
 # auth.py, providers and other core modules excluded — coverage theater).
-# Single source of truth: CI must not override this number.
-fail_under = 50
-show_missing = true
+# Single source of truth: CI m

tokenmizer-0.2.5/scripts/mcp_e2e_check.py

@@ -0,0 +1,158 @@
+"""
+MCP server end-to-end check.
+
+Spawns the real MCP stdio server as a subprocess, performs the MCP
+handshake (initialize → initialized → tools/list), then exercises real
+tool calls against a running TokenMizer proxy (TOKENMIZER_URL) and a
+local file analysis (no proxy needed).
+
+Run:  python scripts/mcp_e2e_check.py
+Exit code 0 = all checks passed.
+"""
+from __future__ import annotations
+
+import json
+import os
+import subprocess
+import sys
+import tempfile
+
+if hasattr(sys.stdout, "reconfigure"):
+    sys.stdout.reconfigure(encoding="utf-8", errors="replace")
+
+FAILURES: list[str] = []
+
+
+def check(name: str, cond: bool, detail: str = ""):
+    status = "PASS" if cond else "FAIL"
+    print(f"[{status}] {name}" + (f" — {detail}" if detail and not cond else ""))
+    if not cond:
+        FAILURES.append(name)
+
+
+def _start_proxy_in_thread(port: int) -> None:
+    """Run the real FastAPI app via uvicorn in a daemon thread."""
+    import threading
+    import time
+
+    import uvicorn
+
+    from tokenmizer.api.app import app
+
+    config = uvicorn.Config(app, host="127.0.0.1", port=port, log_level="error")
+    server = uvicorn.Server(config)
+    t = threading.Thread(target=server.run, daemon=True)
+    t.start()
+    for _ in range(50):  # wait up to 5s for startup
+        if server.started:
+            return
+        time.sleep(0.1)
+    raise RuntimeError("proxy did not start within 5s")
+
+
+def _seed_session_graph(session_id: str) -> None:
+    """Put real nodes into the session graph BEFORE the proxy starts, using
+    the same storage dir the proxy will read — so checkpoint/resume exercise
+    the full real pipeline instead of an empty graph."""
+    from tokenmizer.config.settings import get_settings
+    from tokenmizer.graph_memory.graph import GraphMemory
+
+    storage = get_settings().graph_checkpoint.storage_dir
+    g = GraphMemory(session_id, storage_dir=storage)
+    g.extract_from_messages([
+        {"role": "user", "content": "Let's build a FastAPI auth service with JWT and PostgreSQL"},
+        {"role": "assistant", "content": "Decided: PostgreSQL for storage. Completed: project setup. Working on: login endpoint in api/auth.py"},
+    ], incremental=False)
+    g._persist(force=True)
+
+
+def main() -> int:
+    port = 8765
+    _seed_session_graph("mcp-e2e-test")
+    _start_proxy_in_thread(port)
+    os.environ["TOKENMIZER_URL"] = f"http://127.0.0.1:{port}"
+    print(f"proxy up on :{port}")
+
+    proc = subprocess.Popen(
+        [sys.executable, "-m", "tokenmizer.mcp.server"],
+        stdin=subprocess.PIPE, stdout=subprocess.PIPE,
+        text=True, encoding="utf-8",
+        env={**os.environ},
+    )
+
+    def rpc(method: str, params: dict | None = None, req_id: int | None = 1):
+        msg: dict = {"jsonrpc": "2.0", "method": method}
+        if req_id is not None:
+            msg["id"] = req_id
+        if params is not None:
+            msg["params"] = params
+        proc.stdin.write(json.dumps(msg) + "\n")
+        proc.stdin.flush()
+        if req_id is None:
+            return None
+        line = proc.stdout.readline()
+        return json.loads(line)
+
+    try:
+        # 1. Handshake
+        r = rpc("initialize", {"protocolVersion": "2024-11-05",
+                               "capabilities": {}, "clientInfo": {"name": "e2e", "version": "0"}})
+        check("initialize returns serverInfo",
+              r and r.get("result", {}).get("serverInfo", {}).get("name") == "tokenmizer")
+        check("initialize reports current version",
+              r.get("result", {}).get("serverInfo", {}).get("version") not in ("", "0.2.3"),
+              str(r.get("result", {}).get("serverInfo")))
+        rpc("notifications/initialized", req_id=None)
+
+        # 2. tools/list
+        r = rpc("tools/list", req_id=2)
+        tools = {t["name"] for t in r.get("result", {}).get("tools", [])}
+        expected = {"checkpoint_session", "resume_session", "get_graph_stats",
+                    "analyze_file", "get_savings_stats"}
+        check("tools/list exposes all 5 tools", tools == expected, str(tools))
+
+        # 3. Local tool (no proxy): analyze_file on a real CSV
+        with tempfile.NamedTemporaryFile("w", suffix=".csv", delete=False,
+                                         encoding="utf-8") as f:
+            f.write("region,revenue\nnorth,100\nsouth,250\neast,90\nwest,410\n")
+            csv_path = f.name
+        r = rpc("tools/call", {"name": "analyze_file",
+                               "arguments": {"file_path": csv_path}}, req_id=3)
+        text = r.get("result", {}).get("content", [{}])[0].get("text", "")
+        check("analyze_file returns analysis", "File Analysis" in text, text[:120])
+        os.unlink(csv_path)
+
+        # 4. Proxy-backed tools (require TOKENMIZER_URL server running)
+        r = rpc("tools/call", {"name": "get_savings_stats", "arguments": {}}, req_id=4)
+        text = r.get("result", {}).get("content", [{}])[0].get("text", "")
+        check("get_savings_stats reaches proxy", "Savings Report" in text, text[:120])
+
+        r = rpc("tools/call", {"name": "checkpoint_session",
+                               "arguments": {"session_id": "mcp-e2e-test"}}, req_id=5)
+        text = r.get("result", {}).get("content", [{}])[0].get("text", "")
+        check("checkpoint_session creates checkpoint", "checkpointed" in text, text[:160])
+
+        r = rpc("tools/call", {"name": "resume_session",
+                               "arguments": {"session_id": "mcp-e2e-test"}}, req_id=6)
+        text = r.get("result", {}).get("content", [{}])[0].get("text", "")
+        check("resume_session returns context", "TokenMizer Resume" in text, text[:160])
+
+        # 5. Unknown method → JSON-RPC error, not crash
+        r = rpc("bogus/method", {}, req_id=7)
+        check("unknown method returns -32601 error",
+              r.get("error", {}).get("code") == -32601)
+
+    finally:
+        proc.stdin.close()
+        proc.terminate()
+
+    print()
+    if FAILURES:
+        print(f"E2E RESULT: {len(FAILURES)} FAILURE(S): {FAILURES}")
+        return 1
+    print("E2E RESULT: ALL PASS")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

{tokenmizer-0.2.4 → tokenmizer-0.2.5}/tests/unit/test_graph_persistence.py

@@ -198,3 +198,87 @@ class TestDirectMutationRequiresForce:
             "force=True is required after any direct node mutation, or the "
             "dirty-flag optimization silently drops the write"
         )
+
+
+class TestEnumRestorationOnLoad:
+    """
+    Regression tests for the restart-breaks-everything bug (v0.2.4 e2e find):
+
+    asdict() serializes NodeType/NodeStatus/EdgeType (str-Enums) to plain
+    strings; _load() rebuilt nodes WITHOUT converting back. str-Enum equality
+    (`n.type == NodeType.TASK`) still passed on reloaded graphs — hiding the
+    bug from every existing test — but `.type.value` crashed with
+    "'str' object has no attribute 'value'". Net effect: after any server
+    restart, checkpointing a reloaded session returned HTTP 500.
+    """
+
+    def test_reloaded_node_types_are_real_enums(self, graph, tmp_path):
+        graph.add_node(NodeType.DECISION, "Use PostgreSQL", status=NodeStatus.COMPLETED)
+        graph._persist(force=True)
+
+        reloaded = GraphMemory("dirty-flag-test-session", storage_dir=str(tmp_path))
+        node = next(iter(reloaded._nodes.values()))
+        assert isinstance(node.type, NodeType), type(node.type)
+        assert isinstance(node.status, NodeStatus), type(node.status)
+        # The exact access pattern that crashed checkpoint creation:
+        assert node.type.value == "decision"
+        assert node.status.value == "completed"
+
+    def test_reloaded_edge_types_are_real_enums(self, graph, tmp_path):
+        n1 = graph.add_node(NodeType.TASK, "Implement login")
+        n2 = graph.add_node(NodeType.FILE, "src/auth.py")
+        graph.add_edge(n1, n2, EdgeType.IMPLEMENTS)
+        graph._persist(force=True)
+
+        reloaded = GraphMemory("dirty-flag-test-session", storage_dir=str(tmp_path))
+        assert reloaded._edges, "edge did not survive reload"
+        assert isinstance(reloaded._edges[0].type, EdgeType)
+        assert reloaded._edges[0].type.value == "implements"
+
+    def test_checkpoint_of_reloaded_graph_does_not_crash(self, graph, tmp_path):
+        """Full restart simulation: persist -> fresh GraphMemory ->
+        CheckpointManager.create() — the exact path that 500'd."""
+        from tokenmizer.checkpoints.manager import CheckpointManager
+
+        graph.add_node(NodeType.GOAL, "Build auth service")
+        graph.add_node(NodeType.DECISION, "Use PostgreSQL",
+                       status=NodeStatus.COMPLETED)
+        graph._persist(force=True)
+
+        reloaded = GraphMemory("dirty-flag-test-session", storage_dir=str(tmp_path))
+        mgr = CheckpointManager(storage_dir=str(tmp_path))
+        ckpt = mgr.create(
+            session_id="dirty-flag-test-session",
+            messages=[], graph=reloaded, context_pct=0.5, trigger="manual",
+        )
+        assert ckpt.checkpoint_id
+        snapshot_types = {n["type"] for n in ckpt.graph_snapshot["nodes"]}
+        assert "decision" in snapshot_types
+
+    def test_unknown_node_type_is_skipped_not_fatal(self, graph, tmp_path):
+        """Forward-compat: a node with an unrecognized type (e.g. written by
+        a newer version) must not take down the whole graph load."""
+        import json as _json
+        import sqlite3 as _sqlite3
+
+        graph.add_node(NodeType.TASK, "Implement login")
+        graph._persist(force=True)
+
+        # Corrupt one node's type directly in SQLite
+        conn = _sqlite3.connect(str(graph._db_path))
+        row = conn.execute(
+            "SELECT nodes_json FROM graphs WHERE session_id=?",
+            ("dirty-flag-test-session",),
+        ).fetchone()
+        nodes = _json.loads(row[0])
+        nodes.append({**nodes[0], "id": "corrupt01", "type": "from_the_future"})
+        conn.execute(
+            "UPDATE graphs SET nodes_json=? WHERE session_id=?",
+            (_json.dumps(nodes), "dirty-flag-test-session"),
+        )
+        conn.commit()
+        conn.close()
+
+        reloaded = GraphMemory("dirty-flag-test-session", storage_dir=str(tmp_path))
+        assert "corrupt01" not in reloaded._nodes          # bad node skipped
+        assert len(reloaded._nodes) == 1                   # good node survived

{tokenmizer-0.2.4 → tokenmizer-0.2.5}/tokenmizer/__init__.py

@@ -1,6 +1,6 @@
 """TokenMizer — Never lose your AI context again."""
 
-__version__ = "0.2.4"
+__version__ = "0.2.5"
 __all__ = ["GraphMemory", "CheckpointManager", "get_settings"]
 
 

{tokenmizer-0.2.4 → tokenmizer-0.2.5}/tokenmizer/api/app.py

@@ -261,7 +261,7 @@ async def lifespan(app: FastAPI):
 app = FastAPI(
     title="TokenMizer",
     description="Never lose your AI context again.",
-    version="0.2.4",
+    version="0.2.5",
     lifespan=lifespan,
     docs_url="/docs",
     redoc_url="/redoc",

{tokenmizer-0.2.4 → tokenmizer-0.2.5}/tokenmizer/graph_memory/graph.py

@@ -315,11 +315,37 @@ class GraphMemory:
                 )
                 self._processed_hashes = set()
 
+            # CRITICAL: restore enum types on load. asdict() serializes
+            # NodeType/NodeStatus (str-Enums) to plain strings; without
+            # converting back, every reloaded node has type/status as `str`.
+            # Because they're str-Enums, equality checks still pass — which
+            # HID this bug — but any `.value` access crashes ('str' object
+            # has no attribute 'value'). Concretely: after a server restart,
+            # every checkpoint of a reloaded session returned HTTP 500.
+            # Found by the MCP e2e check, not by unit tests, because unit
+            # tests reloaded graphs but never then called `.type.value`.
             for nd in nodes_data:
                 nd.pop("_evicted", None)
+                try:
+                    nd["type"] = NodeType(nd["type"])
+                    nd["status"] = NodeStatus(nd["status"])
+                except (ValueError, KeyError) as conv_err:
+                    logger.warning(
+                        f"Skipping node with unknown type/status during load "
+                        f"for {self.session_id}: {conv_err} — {nd.get('id')}"
+                    )
+                    continue
                 n = MemoryNode(**{k: v for k, v in nd.items() if k != "_evicted"})
                 self._nodes[n.id] = n
             for ed in edges_data:
+                try:
+                    ed["type"] = EdgeType(ed["type"])
+                except (ValueError, KeyError) as conv_err:
+                    logger.warning(
+                        f"Skipping edge with unknown type during load "
+                        f"for {self.session_id}: {conv_err}"
+                    )
+                    continue
                 self._edges.append(MemoryEdge(**ed))
         except (sqlite3.DatabaseError, sqlite3.OperationalError) as e:
             logger.warning(f"Corrupted DB for {self.session_id} — starting fresh: {e}")

{tokenmizer-0.2.4 → tokenmizer-0.2.5}/tokenmizer/mcp/server.py

@@ -207,7 +207,16 @@ def handle_resume_session(args: dict) -> str:
     ctx = result.get("resume_context", "")
     tokens = result.get("token_count", 0)
     if not ctx:
-        return f"No checkpoint found for session '{session_id}'. Start a session and checkpoint it first."
+        # A checkpoint EXISTS (the API 404s via "error" above when none does) —
+        # its graph was just empty at checkpoint time. Saying "no checkpoint
+        # found" here would be wrong and send the user debugging the wrong
+        # thing.
+        return (
+            f"Checkpoint {result.get('checkpoint_id', '?')} exists for "
+            f"'{session_id}' but its graph was empty (no session activity "
+            f"had been recorded when it was created). Chat through the proxy "
+            f"with this session_id, then checkpoint again."
+        )
     return (
         f"[TokenMizer Resume — session: {session_id} — {tokens} tokens]\n\n"
         f"{ctx}\n\n"
@@ -333,12 +342,13 @@ def run_stdio_server():
         params = req.get("params", {})
 
         if method == "initialize":
+            from tokenmizer import __version__
             send({
                 "jsonrpc": "2.0", "id": req_id,
                 "result": {
                     "protocolVersion": "2024-11-05",
                     "capabilities": {"tools": {}},
-                    "serverInfo": {"name": "tokenmizer", "version": "0.2.3"},
+                    "serverInfo": {"name": "tokenmizer", "version": __version__},
                 },
             })