tokenmizer 0.3.2__tar.gz → 0.4.0__tar.gz

{tokenmizer-0.3.2 → tokenmizer-0.4.0}/.claude-plugin/marketplace.json

@@ -1,6 +1,6 @@
 {
   "name": "tokenmizer",
-  "version": "0.2.3",
+  "version": "0.4.0",
   "description": "Never lose your AI context again. Graph-backed memory, session checkpointing, and file intelligence for any LLM.",
   "homepage": "https://github.com/Shweta-Mishra-ai/tokenmizer",
   "repository": "https://github.com/Shweta-Mishra-ai/tokenmizer",

{tokenmizer-0.3.2 → tokenmizer-0.4.0}/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "tokenmizer",
-  "version": "0.3.2",
+  "version": "0.4.0",
   "description": "Never lose your AI context again. Graph-backed memory, session checkpointing, and file intelligence for Claude Code and any LLM.",
   "author": {
     "name": "Shweta Mishra",

{tokenmizer-0.3.2 → tokenmizer-0.4.0}/.github/workflows/ci.yml

@@ -8,13 +8,9 @@ on:
 
 jobs:
   test:
-    # Windows is in the matrix because the 2026-07-10 audit found THREE
-    # Windows-only bugs Linux CI could never catch: a cp1252
-    # UnicodeEncodeError crashing `tokenmizer --help`, a leaked SQLite
-    # handle blocking corrupt-DB recovery (WinError 32), and console
-    # encoding issues in scripts. One Windows leg (latest Python only —
-    # Windows runners are slow) keeps that whole bug class from ever
-    # shipping again.
+    # Windows is in the matrix to catch platform-specific regressions
+    # (console encoding, file-handle semantics) that Linux legs cannot.
+    # One leg on the latest Python only — Windows runners are slow.
     runs-on: ${{ matrix.os }}
     strategy:
       fail-fast: false
@@ -47,8 +43,8 @@ jobs:
       - name: Run tests with coverage
         run: pytest tests/ --cov=tokenmizer --cov-report=term-missing --cov-report=xml -v
 
-      # Regression guard for the cp1252 crash: --help must work on a
-      # non-UTF-8 Windows console, not just under pytest's captured IO.
+      # --help must work on a non-UTF-8 Windows console, not just under
+      # pytest's captured IO (cp1252 cannot encode the CLI's emoji output).
       - name: CLI smoke test (Windows encoding regression)
         if: runner.os == 'Windows'
         run: python -m tokenmizer.cli --help

{tokenmizer-0.3.2 → tokenmizer-0.4.0}/CHANGELOG.md

@@ -1,5 +1,44 @@
 # Changelog
 
+## [0.4.0] — 2026-07-11 — from storage to reasoning: ontology + graph reasoning
+
+### New — TokenMizer Ontology
+- `tokenmizer/graph_memory/ontology.py`: the formal, machine-readable
+  vocabulary of the graph — every node type with semantics, every edge
+  type with domain/range/semantics, and the status **state machine**
+  (which lifecycle transitions are legal, e.g. COMPLETED→SUPERSEDED→
+  ARCHIVED; SUPERSEDED can never silently become COMPLETED again).
+- Served at `GET /api/ontology` for MCP clients, docs, and tooling.
+- Design principle: the ontology describes and audits, it does not gate
+  writes — graph ingestion stays permissive; violations are surfaced by
+  the consistency audit instead of causing silent data loss.
+
+### New — Graph Reasoning (`tokenmizer/graph_memory/reasoning.py`)
+- **`why()`** — "Why is X the current choice?" Walks the supersession
+  chain in both directions and returns the old→new trail with trigger,
+  reason, and evidence per hop, plus the currently active decision.
+  `GET /api/graph/{id}/why?q=react`
+- **`impact()`** — typed 1-hop neighborhood: which files/tasks/errors
+  connect to a node and via which relation.
+- **`decision_history()`** — decision timeline grouped by topic bucket.
+- **`consistency_check()`** — ontology-based audit: two active decisions
+  sharing a topic (contradictions the tracker missed), SUPERSEDED
+  decisions with no transition record (lost history), transitions
+  referencing pruned nodes.
+- **`GET /api/graph/{id}/reasoning`** — the combined reasoning view.
+- All reasoning is deterministic and local — no LLM calls.
+
+### New — MCP tool `why_decision` (6 tools now)
+- Ask your agent "why did we pick X?" — it traces the decision trail:
+  struck-through old choices, replaced-by hops with reasons/evidence,
+  and the current active choice. Covered in unit tests and the e2e check.
+
+### Changed
+- `glama.json` added (Glama MCP directory maintainer verification).
+- README: "From Storage to Reasoning" section; internal demo-script and
+  planning docs removed from the repository.
+- Version 0.4.0 everywhere (enforced by test_version_consistency).
+
 ## [0.3.2] — 2026-07-10 — full-repo audit: graph memory, MCP server, visualization
 
 ### Critical — the LLM extraction path never worked

{tokenmizer-0.3.2 → tokenmizer-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tokenmizer
-Version: 0.3.2
+Version: 0.4.0
 Summary: Reduce AI context loss by 2x. Graph-backed checkpoint and resume for any LLM session.
 Project-URL: Homepage, https://github.com/Shweta-Mishra-ai/tokenmizer
 Project-URL: Repository, https://github.com/Shweta-Mishra-ai/tokenmizer
@@ -153,13 +153,22 @@ Your App  →  TokenMizer (:8000)  →  Claude / GPT / Gemini / any LLM
 | 🔴 `INVALIDATED` | Explicitly wrong/cancelled | ⚠️ Always (warning) |
 | ⬜ `ARCHIVED` | Superseded >7 days ago — aged out | ❌ Never |
 
-History is **never deleted**. "Why did we switch from React to Next.js?" — always answerable.
+History is **never deleted**. "Why did we switch from React to Next.js?" — always answerable:
+ask `GET /api/graph/{session}/why?q=react` (or the `why_decision` MCP tool) and get the full
+old → new trail with trigger, reason, and evidence per hop.
 
-> Honesty note (2026-07-10 audit): before v0.3.2, `ARCHIVED` was advertised
-> here but **unreachable** — the decay/prune/query logic all handled it, yet
-> no code path ever set it. Superseded decisions now age into `ARCHIVED`
-> automatically after 7 days (`GraphMemory.ARCHIVE_SUPERSEDED_AFTER_DAYS`),
-> which is what makes the "⚠️ 7 days" row above actually true.
+### From Storage to Reasoning
+
+The graph doesn't just store facts — it answers questions over them:
+
+| Capability | Endpoint / Tool | What it answers |
+|---|---|---|
+| **Ontology** | `GET /api/ontology` | The formal vocabulary: node/edge types with semantics, and the status state machine (which lifecycle transitions are legal) |
+| **Causal chains** | `GET /api/graph/{id}/why?q=...` · MCP `why_decision` | "Why is X the current choice?" — walks the supersession chain with trigger/reason/evidence per hop |
+| **Reasoning view** | `GET /api/graph/{id}/reasoning` | Active decisions per topic, recent changes, decision timeline, and a consistency audit |
+| **Consistency audit** | (part of `/reasoning`) | Contradictions the tracker missed, superseded decisions with lost history, dangling references |
+
+All reasoning is deterministic and local — no LLM calls, no extra cost.
 
 ---
 
@@ -360,10 +369,15 @@ env = { TOKENMIZER_URL = "http://localhost:8000" }
 </details>
 
 Then restart the client. Keep `tokenmizer serve` running for the
-checkpoint/resume/stats tools (file analysis works without it).
+checkpoint/resume/stats/reasoning tools (file analysis works without it).
 If `tokenmizer-mcp` isn't on your PATH, use `"command": "python"`,
 `"args": ["-m", "tokenmizer.mcp.server"]` instead.
 
+**Tools exposed (6):** `checkpoint_session`, `resume_session`,
+`get_graph_stats`, `analyze_file`, `get_savings_stats`, and
+`why_decision` — ask your agent *"why did we pick X?"* and it traces the
+decision's supersession chain with reasons and evidence.
+
 ---
 
 ## Other Tools
@@ -461,12 +475,8 @@ graph_checkpoint:
   enabled: true
   trigger_at_percent: 0.85
   use_llm_extraction: false     # true = hybrid LLM+heuristic extraction
-                                # (needs a provider key, ~$0.001/turn).
-                                # NOTE: before v0.3.2 this flag silently did
-                                # nothing — the call site passed provider_fn
-                                # to the wrong function and raised TypeError
-                                # on every turn, falling back to heuristics.
-                                # Fixed + regression-tested; see CHANGELOG.
+                                # (needs a provider key, ~$0.001/turn;
+                                # requires v0.3.2+ — see CHANGELOG)
 
 compression:
   enabled: true
@@ -507,6 +517,9 @@ TOKENMIZER_API_KEY=strong-key docker-compose up
 | `/api/decision/invalidate` | POST | Mark decision as invalid |
 | `/api/graph/{id}` | GET | Session graph stats |
 | `/api/graph/{id}/html` | GET | **Interactive graph page** — decision-history timeline, supersession arcs, type/status filters, search, zoom/pan, PNG export. Zero external dependencies (works offline) |
+| `/api/graph/{id}/why?q=` | GET | **Reasoning:** causal chain behind a decision (old → new with trigger/reason/evidence) |
+| `/api/graph/{id}/reasoning` | GET | **Reasoning view:** active decisions by topic, recent changes, consistency audit |
+| `/api/ontology` | GET | Machine-readable graph ontology (types, relations, status state machine) |
 | `/api/stats` | GET | Token savings analytics |
 | `/health` | GET | Health check |
 | `/docs` | GET | Swagger UI |
@@ -517,15 +530,13 @@ TOKENMIZER_API_KEY=strong-key docker-compose up
 
 - API key auth — `TOKENMIZER_API_KEY` (constant-time comparison)
 - Secret/PII redaction applied once at ingestion, before graph storage,
-  checkpoint storage, AND every LLM call (main chat *and* the background
-  extraction model — these are separate, the redaction gap between them
-  was a real bug, now fixed). Patterns cover Anthropic/OpenAI/Google/
-  GitHub/AWS/Slack/Stripe/JWT/OpenRouter/HF/xAI keys, URL-embedded
-  credentials (`postgres://user:pass@host` — a 2026-07 audit gap, fixed),
-  and generic `key=`/`password=` assignments. Best-effort by nature —
-  an unrecognized format with no keyword context can still slip through.
-  The checkpoint layer independently re-redacts what it persists
-  (defense-in-depth, added in the same audit).
+  checkpoint storage, and every LLM call (main chat and the background
+  extraction model). Patterns cover Anthropic/OpenAI/Google/GitHub/AWS/
+  Slack/Stripe/JWT/OpenRouter/HF/xAI keys, URL-embedded credentials
+  (`postgres://user:pass@host`), and generic `key=`/`password=`
+  assignments. Best-effort by nature — an unrecognized format with no
+  keyword context can still slip through. The checkpoint layer
+  independently re-redacts what it persists (defense in depth).
 - Session-isolated cache (sensitive data never shared across sessions)
 - Basic prompt-injection keyword filter — catches copy-pasted jailbreak
   templates only; **not** a security boundary against a motivated

{tokenmizer-0.3.2 → tokenmizer-0.4.0}/README.md

@@ -78,13 +78,22 @@ Your App  →  TokenMizer (:8000)  →  Claude / GPT / Gemini / any LLM
 | 🔴 `INVALIDATED` | Explicitly wrong/cancelled | ⚠️ Always (warning) |
 | ⬜ `ARCHIVED` | Superseded >7 days ago — aged out | ❌ Never |
 
-History is **never deleted**. "Why did we switch from React to Next.js?" — always answerable.
+History is **never deleted**. "Why did we switch from React to Next.js?" — always answerable:
+ask `GET /api/graph/{session}/why?q=react` (or the `why_decision` MCP tool) and get the full
+old → new trail with trigger, reason, and evidence per hop.
 
-> Honesty note (2026-07-10 audit): before v0.3.2, `ARCHIVED` was advertised
-> here but **unreachable** — the decay/prune/query logic all handled it, yet
-> no code path ever set it. Superseded decisions now age into `ARCHIVED`
-> automatically after 7 days (`GraphMemory.ARCHIVE_SUPERSEDED_AFTER_DAYS`),
-> which is what makes the "⚠️ 7 days" row above actually true.
+### From Storage to Reasoning
+
+The graph doesn't just store facts — it answers questions over them:
+
+| Capability | Endpoint / Tool | What it answers |
+|---|---|---|
+| **Ontology** | `GET /api/ontology` | The formal vocabulary: node/edge types with semantics, and the status state machine (which lifecycle transitions are legal) |
+| **Causal chains** | `GET /api/graph/{id}/why?q=...` · MCP `why_decision` | "Why is X the current choice?" — walks the supersession chain with trigger/reason/evidence per hop |
+| **Reasoning view** | `GET /api/graph/{id}/reasoning` | Active decisions per topic, recent changes, decision timeline, and a consistency audit |
+| **Consistency audit** | (part of `/reasoning`) | Contradictions the tracker missed, superseded decisions with lost history, dangling references |
+
+All reasoning is deterministic and local — no LLM calls, no extra cost.
 
 ---
 
@@ -285,10 +294,15 @@ env = { TOKENMIZER_URL = "http://localhost:8000" }
 </details>
 
 Then restart the client. Keep `tokenmizer serve` running for the
-checkpoint/resume/stats tools (file analysis works without it).
+checkpoint/resume/stats/reasoning tools (file analysis works without it).
 If `tokenmizer-mcp` isn't on your PATH, use `"command": "python"`,
 `"args": ["-m", "tokenmizer.mcp.server"]` instead.
 
+**Tools exposed (6):** `checkpoint_session`, `resume_session`,
+`get_graph_stats`, `analyze_file`, `get_savings_stats`, and
+`why_decision` — ask your agent *"why did we pick X?"* and it traces the
+decision's supersession chain with reasons and evidence.
+
 ---
 
 ## Other Tools
@@ -386,12 +400,8 @@ graph_checkpoint:
   enabled: true
   trigger_at_percent: 0.85
   use_llm_extraction: false     # true = hybrid LLM+heuristic extraction
-                                # (needs a provider key, ~$0.001/turn).
-                                # NOTE: before v0.3.2 this flag silently did
-                                # nothing — the call site passed provider_fn
-                                # to the wrong function and raised TypeError
-                                # on every turn, falling back to heuristics.
-                                # Fixed + regression-tested; see CHANGELOG.
+                                # (needs a provider key, ~$0.001/turn;
+                                # requires v0.3.2+ — see CHANGELOG)
 
 compression:
   enabled: true
@@ -432,6 +442,9 @@ TOKENMIZER_API_KEY=strong-key docker-compose up
 | `/api/decision/invalidate` | POST | Mark decision as invalid |
 | `/api/graph/{id}` | GET | Session graph stats |
 | `/api/graph/{id}/html` | GET | **Interactive graph page** — decision-history timeline, supersession arcs, type/status filters, search, zoom/pan, PNG export. Zero external dependencies (works offline) |
+| `/api/graph/{id}/why?q=` | GET | **Reasoning:** causal chain behind a decision (old → new with trigger/reason/evidence) |
+| `/api/graph/{id}/reasoning` | GET | **Reasoning view:** active decisions by topic, recent changes, consistency audit |
+| `/api/ontology` | GET | Machine-readable graph ontology (types, relations, status state machine) |
 | `/api/stats` | GET | Token savings analytics |
 | `/health` | GET | Health check |
 | `/docs` | GET | Swagger UI |
@@ -442,15 +455,13 @@ TOKENMIZER_API_KEY=strong-key docker-compose up
 
 - API key auth — `TOKENMIZER_API_KEY` (constant-time comparison)
 - Secret/PII redaction applied once at ingestion, before graph storage,
-  checkpoint storage, AND every LLM call (main chat *and* the background
-  extraction model — these are separate, the redaction gap between them
-  was a real bug, now fixed). Patterns cover Anthropic/OpenAI/Google/
-  GitHub/AWS/Slack/Stripe/JWT/OpenRouter/HF/xAI keys, URL-embedded
-  credentials (`postgres://user:pass@host` — a 2026-07 audit gap, fixed),
-  and generic `key=`/`password=` assignments. Best-effort by nature —
-  an unrecognized format with no keyword context can still slip through.
-  The checkpoint layer independently re-redacts what it persists
-  (defense-in-depth, added in the same audit).
+  checkpoint storage, and every LLM call (main chat and the background
+  extraction model). Patterns cover Anthropic/OpenAI/Google/GitHub/AWS/
+  Slack/Stripe/JWT/OpenRouter/HF/xAI keys, URL-embedded credentials
+  (`postgres://user:pass@host`), and generic `key=`/`password=`
+  assignments. Best-effort by nature — an unrecognized format with no
+  keyword context can still slip through. The checkpoint layer
+  independently re-redacts what it persists (defense in depth).
 - Session-isolated cache (sensitive data never shared across sessions)
 - Basic prompt-injection keyword filter — catches copy-pasted jailbreak
   templates only; **not** a security boundary against a motivated

tokenmizer-0.4.0/glama.json

@@ -0,0 +1,6 @@
+{
+  "$schema": "https://glama.ai/mcp/schemas/server.json",
+  "maintainers": [
+    "Shweta-Mishra-ai"
+  ]
+}

{tokenmizer-0.3.2 → tokenmizer-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "tokenmizer"
-version = "0.3.2"
+version = "0.4.0"
 description = "Reduce AI context loss by 2x. Graph-backed checkpoint and resume for any LLM session."
 readme = "README.md"
 requires-python = ">=3.10"

{tokenmizer-0.3.2 → tokenmizer-0.4.0}/scripts/mcp_e2e_check.py

@@ -108,8 +108,8 @@ def main() -> int:
         r = rpc("tools/list", req_id=2)
         tools = {t["name"] for t in r.get("result", {}).get("tools", [])}
         expected = {"checkpoint_session", "resume_session", "get_graph_stats",
-                    "analyze_file", "get_savings_stats"}
-        check("tools/list exposes all 5 tools", tools == expected, str(tools))
+                    "analyze_file", "get_savings_stats", "why_decision"}
+        check("tools/list exposes all 6 tools", tools == expected, str(tools))
 
         # 3. Local tool (no proxy): analyze_file on a real CSV
         with tempfile.NamedTemporaryFile("w", suffix=".csv", delete=False,
@@ -137,8 +137,21 @@ def main() -> int:
         text = r.get("result", {}).get("content", [{}])[0].get("text", "")
         check("resume_session returns context", "TokenMizer Resume" in text, text[:160])
 
+        # Reasoning tool round-trips through the proxy. Whether the test
+        # session happens to contain a matching decision or not, a healthy
+        # server answers with a trail or a clean "no match" — never an error.
+        r = rpc("tools/call", {"name": "why_decision",
+                               "arguments": {"session_id": "mcp-e2e-test",
+                                             "query": "postgres"}}, req_id=7)
+        res = r.get("result", {})
+        text = res.get("content", [{}])[0].get("text", "")
+        check("why_decision answers without error",
+              res.get("isError") is False
+              and ("Decision trail" in text or "No decision matching" in text),
+              text[:160])
+
         # 5. Unknown method → JSON-RPC error, not crash
-        r = rpc("bogus/method", {}, req_id=7)
+        r = rpc("bogus/method", {}, req_id=8)
         check("unknown method returns -32601 error",
               r.get("error", {}).get("code") == -32601)
 

{tokenmizer-0.3.2 → tokenmizer-0.4.0}/server.json

@@ -6,12 +6,12 @@
     "url": "https://github.com/Shweta-Mishra-ai/tokenmizer",
     "source": "github"
   },
-  "version": "0.3.2",
+  "version": "0.4.0",
   "packages": [
     {
       "registryType": "pypi",
       "identifier": "tokenmizer",
-      "version": "0.3.2",
+      "version": "0.4.0",
       "transport": { "type": "stdio" },
       "environmentVariables": [
         {

{tokenmizer-0.3.2 → tokenmizer-0.4.0}/tests/integration/test_api_endpoint.py

@@ -194,10 +194,8 @@ class TestHealthAndDocs:
     def test_graph_share_html(self, client):
         """Shareable graph page: self-contained interactive HTML.
 
-        REDESIGNED (2026-07-10): no more D3-from-CDN — the artifact must
-        work offline. It must also actually render the supersession
-        history (transitions), which the old template computed and then
-        silently never used.
+        The artifact must work offline (no CDN/external loads) and must
+        embed the supersession history (transitions) it renders.
         """
         c, _ = client
         # Put something in the graph via the pipeline first
@@ -223,11 +221,10 @@ class TestHealthAndDocs:
 
 class TestInvalidateDecisionScope:
     """
-    AUDIT FIX (2026-07-10): /api/decision/invalidate previously substring-
-    matched across ALL decision nodes regardless of status — invalidating
-    "postgres" would also flip an already-SUPERSEDED postgres decision,
-    destroying its supersession history. Only ACTIVE (COMPLETED) decisions
-    are now eligible, and the response lists exactly what was affected.
+    /api/decision/invalidate must only match ACTIVE (COMPLETED) decisions:
+    matching across all statuses would overwrite SUPERSEDED history nodes
+    and destroy their supersession record. The response must list every
+    affected node.
     """
 
     def _seed_graph(self, session_id, tmp_path):

{tokenmizer-0.3.2 → tokenmizer-0.4.0}/tests/unit/test_decision_tracker.py

@@ -1,14 +1,10 @@
 """
-Regression tests for the topic classifier (2026-07-10 audit).
-
-Confirmed misclassifications before the fix (all reproduced by execution):
-  - "Go with tRPC for the API layer"  → "language" (imperative "Go" collided
-    with Go-the-language; first-single-word-hit-wins never reached "trpc")
-  - "Use Supabase for the backend"    → None (vocabulary gap; the sibling
-    hybrid_extractor.py knew "supabase" but this file didn't)
-  - "Use Clerk for authentication"    → None (same gap)
-  - "Use FastAPI with SQLAlchemy and PostgreSQL" → only "web_framework";
-    a later Postgres→SQLite switch was never detected as contradicting it.
+Tests for the decision topic classifier and same-decision detection.
+
+Covers the known-hard cases: the imperative "Go with X" vs. Go-the-language
+ambiguity, technology names shared with the extractor vocabulary, multi-topic
+statements, bigram/single-word precedence, and near-duplicate label
+containment.
 """
 from tokenmizer.graph_memory.decision_tracker import (
     classify_topic,
@@ -112,12 +108,12 @@ def test_same_decision_not_superseded(tmp_path):
     assert old_id not in hits
 
 
-# ── AUDIT round 2 (2026-07-10): near-duplicate decision merging ──────────────
+# ── Near-duplicate decision merging ──────────────────────────────────────────
 
 def test_containment_variant_merges_not_supersedes(tmp_path):
-    """The demo bug: 'use React for the frontend.' and 'Use React' were
-    emitted from ONE message, became two nodes, and one superseded the
-    other — a bogus 'Changed:' line in every resume. They must merge."""
+    """Two label variants of one decision (emitted from a single message)
+    must merge into one node rather than supersede each other, which would
+    record a spurious decision change."""
     g = GraphMemory(session_id="t-dup", storage_dir=str(tmp_path))
     id1 = g.add_node(NodeType.DECISION, "use React for the frontend.",
                      NodeStatus.COMPLETED)

{tokenmizer-0.3.2 → tokenmizer-0.4.0}/tests/unit/test_graph.py

@@ -203,11 +203,10 @@ class TestPersistence:
 
 class TestArchivedReachability:
     """
-    AUDIT FIX (2026-07-10): ARCHIVED was a documented, fully-wired status
-    (decay rates, prune rules, query filters all handled it) that NOTHING
-    ever set — the README advertised a 4-state model whose 4th state was
-    unreachable. SUPERSEDED decisions now age into ARCHIVED after
-    GraphMemory.ARCHIVE_SUPERSEDED_AFTER_DAYS days (from supersession time).
+    SUPERSEDED decisions age into ARCHIVED after
+    GraphMemory.ARCHIVE_SUPERSEDED_AFTER_DAYS, measured from supersession
+    time. apply_importance_decay() is the only path that sets ARCHIVED,
+    so these tests guard the state's reachability.
     """
 
     def test_superseded_decision_ages_into_archived(self, graph):

{tokenmizer-0.3.2 → tokenmizer-0.4.0}/tests/unit/test_hybrid_extractor.py

@@ -151,15 +151,10 @@ async def test_extract_without_provider():
 @pytest.mark.asyncio
 async def test_extract_llm_pass_actually_invoked_app_style():
     """
-    REGRESSION (2026-07-10 audit): api/app.py's background extraction did
-      HybridExtractor(provider_fn=_pfn)   → TypeError (no such kwarg), and
-      ext.extract(_msgs)                  → provider_fn=None, LLM pass skipped.
-    So the LLM extraction path NEVER ran in production — every call raised,
-    was caught by the broad except, and logged as a provider failure.
-
-    This test replicates the app.py call pattern exactly as fixed:
-    construct with defaults, pass provider_fn to extract(), and assert
-    the provider was actually called and its output merged in.
+    Regression guard for the api/app.py call pattern: construct with
+    defaults and pass provider_fn to extract(). Asserts the provider is
+    actually invoked and its output is merged — a call-signature drift
+    here silently disables the LLM pass.
     """
     calls = []
 
@@ -183,10 +178,9 @@ async def test_extract_llm_pass_actually_invoked_app_style():
 
 class TestMinConfidenceFilter:
     """
-    AUDIT round 2 (2026-07-10): min_confidence was dead code — stored in
-    __init__, never read. It now filters extract() output by merge()'s
-    confidence tiers (0.95 corroborated / 0.80 LLM-only / 0.65 heuristic-
-    only). Default 0.55 keeps everything (backward compatible).
+    min_confidence filters extract() output by merge()'s confidence tiers
+    (0.95 corroborated / 0.80 LLM-only / 0.65 heuristic-only). The default
+    of 0.55 keeps every tier.
     """
 
     def test_default_keeps_heuristic_only_items(self):

{tokenmizer-0.3.2 → tokenmizer-0.4.0}/tests/unit/test_mcp_server.py

@@ -1,16 +1,13 @@
 """
-Regression tests for the MCP stdio server (2026-07-10 audit).
-
-Bugs these guard against (all confirmed live before the fix):
-  1. isError was computed via result_text.startswith("❌") — a KeyError from
-     a missing required argument surfaced as "Tool error: 'session_id'" with
-     isError FALSE, i.e. clients saw a *successful* result.
-  2. A valid-JSON-but-not-an-object line ([1,2]) crashed the whole server
-     via AttributeError on req.get().
-  3. Any exception in a handler outside tools/call (e.g. initialize) killed
-     the process with no JSON-RPC error — the client hung, then saw the
-     subprocess die.
-  4. Malformed JSON lines were silently dropped (no log, no error response).
+Regression tests for the MCP stdio server.
+
+Invariants under test:
+  1. isError is structural — validation failures and handler crashes are
+     reported with isError: true regardless of message text.
+  2. No input terminates the read loop: malformed JSON returns -32700,
+     non-object messages return -32600, handler exceptions return -32603,
+     and subsequent requests are still served.
+  3. Required arguments are validated with typed, descriptive errors.
 """
 import io
 import json
@@ -81,14 +78,14 @@ def test_analyze_file_success_not_error(tmp_path):
 
 
 def test_handler_crash_is_error_not_exception(monkeypatch):
-    def boom(args):
-        raise RuntimeError("kaboom")
+    def failing_handler(args):
+        raise RuntimeError("simulated handler failure")
     # handle_tool_call builds its dispatch dict from module globals at call
     # time, so patching the module attribute is picked up.
-    monkeypatch.setattr(mcp, "handle_get_savings_stats", boom)
+    monkeypatch.setattr(mcp, "handle_get_savings_stats", failing_handler)
     text, is_error = mcp.handle_tool_call("get_savings_stats", {})
     assert is_error is True
-    assert "kaboom" in text or "internal error" in text.lower()
+    assert "simulated handler failure" in text or "internal error" in text.lower()
 
 
 # ── stdio transport: survives hostile input ──────────────────────────────────
@@ -120,7 +117,7 @@ def test_stdio_survives_malformed_json(monkeypatch):
     ])
     assert out[0]["error"]["code"] == -32700
     assert out[1]["id"] == 2
-    assert len(out[1]["result"]["tools"]) == 5
+    assert len(out[1]["result"]["tools"]) == 6
 
 
 def test_stdio_survives_non_object_json(monkeypatch):
@@ -159,7 +156,7 @@ def test_stdio_handler_exception_yields_jsonrpc_error(monkeypatch):
 
 
 def test_stdio_missing_arg_reports_is_error_true(monkeypatch):
-    """THE bug: missing session_id must reach the client as isError: true."""
+    """A missing required argument must reach the client as isError: true."""
     out = _run_lines(monkeypatch, [
         json.dumps({"jsonrpc": "2.0", "id": 6, "method": "tools/call",
                     "params": {"name": "checkpoint_session", "arguments": {}}}),