knowledge-rag 4.6.0__tar.gz → 4.7.0__tar.gz

{knowledge_rag-4.6.0 → knowledge_rag-4.7.0}/.gitignore

@@ -71,3 +71,4 @@ dist/
 # Path(config.data_dir) reproduces. Preventive: use spec=Config or
 # types.SimpleNamespace when mocking config in tests.
 MagicMock/
+.benchmarks/

{knowledge_rag-4.6.0 → knowledge_rag-4.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: knowledge-rag
-Version: 4.6.0
+Version: 4.7.0
 Summary: Local RAG System for Claude Code — Hybrid search + Cross-encoder Reranking + 13 MCP Tools + 20 Format Parsers. Zero external servers.
 Project-URL: Homepage, https://github.com/lyonzin/knowledge-rag
 Project-URL: Repository, https://github.com/lyonzin/knowledge-rag
@@ -97,7 +97,21 @@ pip install knowledge-rag → restart Claude Code → search_knowledge("your que
 
 ---
 
-## What's New in v4.2.0
+## What's New in v4.6.0
+
+### MCP spec 2026-07-28 via Anthropic Tier 1 SDK (v4.6.0)
+
+Adopts the [**MCP spec 2026-07-28**](https://blog.modelcontextprotocol.io/posts/2026-07-28/) — stateless request/response core, per-request `_meta.io.modelcontextprotocol/protocolVersion` + `clientCapabilities`, `serverInfo` on every response, retirement of the `initialize`/`initialized` handshake and the `Mcp-Session-Id` header. Delivered by moving directly to the **Anthropic Tier 1 Python SDK** (`mcp>=2.0.0,<3.0.0`, published 2026-07-28).
+
+**What changed under the hood**:
+- `mcp>=1.6.0,<2.0.0` → **`mcp>=2.0.0,<3.0.0`**
+- `from mcp.server.fastmcp import FastMCP` → `from mcp.server import MCPServer`
+- `FastMCP(name, host=, port=)` → `MCPServer(name, version="4.6.0")` — `host` / `port` moved to `mcp.run(transport=..., host=..., port=...)` on non-stdio transports (per v2 API)
+- Server now advertises `version` in `serverInfo`, matching the 2026-07-28 spec recommendation
+
+**Zero user-facing changes** — `@mcp.tool()` / `@mcp.resource()` / `@mcp.prompt()` decorator ergonomics are unchanged. All 13 MCP tool signatures frozen. **stdio users see zero behaviour change**. Existing clients (Claude Code, Claude Desktop, Cursor, Windsurf, VS Code Copilot, Cline, Gemini CLI, Zed) negotiate the spec version per request and continue to work.
+
+**Why the Tier 1 SDK and not the third-party `fastmcp` package**: an early experiment with jlowin's [`fastmcp`](https://gofastmcp.com) tripped the Pillar 5 perf gate — `test_bench_orchestrator_idle_rss` +21.9% and `test_bench_query_cache_5000_entries` +17.1% RSS from the ~20 transitive deps its `[client,server]` extras pull in. Cutting the extras broke `@mcp.tool()` at runtime. Anthropic's `mcp 2.x` is the direct upstream reference — no third-party intermediary, no dependency bloat. Full details in the [v4.6.0 changelog entry](#v460-2026-07-30--mcp-spec-2026-07-28-via-anthropic-tier-1-sdk-mcp-2x) and closed PR #134.
 
 ### Search Performance & Output Quality (v4.2.0)
 
@@ -1448,6 +1462,14 @@ Common issues:
 
 ### Unreleased
 
+### v4.7.0 (2026-07-31) — BM25 Fragment Query Support
+
+- **FIX (retrieval)**: BM25 tokenizer now emits **sub-tokens** for hyphenated alphanumeric-code composites, so fragment queries no longer silently return `NO_RESULTS`. Before v4.7.0, `MDR-AD002` was indexed as a single token `mdr-ad002`; queries like `AD002`, `MDR`, `CS005` returned zero even when the composite lived in the corpus. This is not a bug specific to MDR — it hit every hyphenated code taxonomy in typical infosec / doc corpora (`MDR-*`, `CVE-*`, `ADR-*`, `MS17-*`, `MDR-Custom001-xxxx`). The tokenizer now emits both the composite token AND its sub-parts of length ≥ 2 (`mdr-ad002` → `["mdr-ad002", "mdr", "ad002"]`). IDF preserves ranking — composite matches still rank above fragment matches because the composite is rarer. **Heuristic (perf-safety)**: sub-token expansion is triggered ONLY when the composite contains at least one digit — so `pass-the-hash`, `state-of-the-art`, and other natural-language hyphenated phrases stay as single tokens (expanding them would flood the inverted index with stop-word-like sub-parts and hurt concurrent-query throughput). (#140)
+- **NEW (tests)**: `tests/test_bm25_tokenizer_fragment.py` — 25 tests across 4 classes covering unit tokenizer behavior, end-to-end fragment matching, multi-doc disambiguation, and the 5 query patterns explicitly reported (`MDR-AD019`, `MDR-Custom001-xxxx`, `MDR`, `AD019`, `CS005`). Includes retrocompat sentinel + IDF ranking invariant. Baseline: 314 → 339.
+- **UPGRADE NOTE**: run `reindex_documents(force=True)` once after upgrade to rebuild the BM25 inverted index with the new sub-token emission. ChromaDB vectors are untouched. Expect a **~30–50% increase in BM25 index memory** (measured +32.5% on synthetic corpus) — negligible for typical corpora, worth reviewing for deployments with >100K docs.
+- **KNOWN LIMITATION**: alphanumeric-glued tokens without separators (`HARDLAB11`, `6a635623e114b8...`) still require exact match — no sub-token expansion possible without character n-grams (5-10× index overhead). Tracked as follow-up.
+- **VERSION**: 4.6.0 → **4.7.0** (MINOR — tokenization behavior change + reindex requirement, though no API/config break).
+
 ### v4.6.0 (2026-07-30) — MCP spec 2026-07-28 via Anthropic Tier 1 SDK (mcp 2.x)
 
 - **NEW (protocol)**: adopt the [MCP spec `2026-07-28`](https://blog.modelcontextprotocol.io/posts/2026-07-28/) — stateless request/response core, per-request `_meta.io.modelcontextprotocol/protocolVersion` + `clientCapabilities`, `serverInfo` on responses, retirement of the `initialize`/`initialized` handshake and the `Mcp-Session-Id` header. The MCP client Claude Code (and every Tier 1 client) negotiates the spec version per request, so this release stays backwards-compatible for clients speaking earlier spec revisions.

{knowledge_rag-4.6.0 → knowledge_rag-4.7.0}/README.md

@@ -49,7 +49,21 @@ pip install knowledge-rag → restart Claude Code → search_knowledge("your que
 
 ---
 
-## What's New in v4.2.0
+## What's New in v4.6.0
+
+### MCP spec 2026-07-28 via Anthropic Tier 1 SDK (v4.6.0)
+
+Adopts the [**MCP spec 2026-07-28**](https://blog.modelcontextprotocol.io/posts/2026-07-28/) — stateless request/response core, per-request `_meta.io.modelcontextprotocol/protocolVersion` + `clientCapabilities`, `serverInfo` on every response, retirement of the `initialize`/`initialized` handshake and the `Mcp-Session-Id` header. Delivered by moving directly to the **Anthropic Tier 1 Python SDK** (`mcp>=2.0.0,<3.0.0`, published 2026-07-28).
+
+**What changed under the hood**:
+- `mcp>=1.6.0,<2.0.0` → **`mcp>=2.0.0,<3.0.0`**
+- `from mcp.server.fastmcp import FastMCP` → `from mcp.server import MCPServer`
+- `FastMCP(name, host=, port=)` → `MCPServer(name, version="4.6.0")` — `host` / `port` moved to `mcp.run(transport=..., host=..., port=...)` on non-stdio transports (per v2 API)
+- Server now advertises `version` in `serverInfo`, matching the 2026-07-28 spec recommendation
+
+**Zero user-facing changes** — `@mcp.tool()` / `@mcp.resource()` / `@mcp.prompt()` decorator ergonomics are unchanged. All 13 MCP tool signatures frozen. **stdio users see zero behaviour change**. Existing clients (Claude Code, Claude Desktop, Cursor, Windsurf, VS Code Copilot, Cline, Gemini CLI, Zed) negotiate the spec version per request and continue to work.
+
+**Why the Tier 1 SDK and not the third-party `fastmcp` package**: an early experiment with jlowin's [`fastmcp`](https://gofastmcp.com) tripped the Pillar 5 perf gate — `test_bench_orchestrator_idle_rss` +21.9% and `test_bench_query_cache_5000_entries` +17.1% RSS from the ~20 transitive deps its `[client,server]` extras pull in. Cutting the extras broke `@mcp.tool()` at runtime. Anthropic's `mcp 2.x` is the direct upstream reference — no third-party intermediary, no dependency bloat. Full details in the [v4.6.0 changelog entry](#v460-2026-07-30--mcp-spec-2026-07-28-via-anthropic-tier-1-sdk-mcp-2x) and closed PR #134.
 
 ### Search Performance & Output Quality (v4.2.0)
 
@@ -1400,6 +1414,14 @@ Common issues:
 
 ### Unreleased
 
+### v4.7.0 (2026-07-31) — BM25 Fragment Query Support
+
+- **FIX (retrieval)**: BM25 tokenizer now emits **sub-tokens** for hyphenated alphanumeric-code composites, so fragment queries no longer silently return `NO_RESULTS`. Before v4.7.0, `MDR-AD002` was indexed as a single token `mdr-ad002`; queries like `AD002`, `MDR`, `CS005` returned zero even when the composite lived in the corpus. This is not a bug specific to MDR — it hit every hyphenated code taxonomy in typical infosec / doc corpora (`MDR-*`, `CVE-*`, `ADR-*`, `MS17-*`, `MDR-Custom001-xxxx`). The tokenizer now emits both the composite token AND its sub-parts of length ≥ 2 (`mdr-ad002` → `["mdr-ad002", "mdr", "ad002"]`). IDF preserves ranking — composite matches still rank above fragment matches because the composite is rarer. **Heuristic (perf-safety)**: sub-token expansion is triggered ONLY when the composite contains at least one digit — so `pass-the-hash`, `state-of-the-art`, and other natural-language hyphenated phrases stay as single tokens (expanding them would flood the inverted index with stop-word-like sub-parts and hurt concurrent-query throughput). (#140)
+- **NEW (tests)**: `tests/test_bm25_tokenizer_fragment.py` — 25 tests across 4 classes covering unit tokenizer behavior, end-to-end fragment matching, multi-doc disambiguation, and the 5 query patterns explicitly reported (`MDR-AD019`, `MDR-Custom001-xxxx`, `MDR`, `AD019`, `CS005`). Includes retrocompat sentinel + IDF ranking invariant. Baseline: 314 → 339.
+- **UPGRADE NOTE**: run `reindex_documents(force=True)` once after upgrade to rebuild the BM25 inverted index with the new sub-token emission. ChromaDB vectors are untouched. Expect a **~30–50% increase in BM25 index memory** (measured +32.5% on synthetic corpus) — negligible for typical corpora, worth reviewing for deployments with >100K docs.
+- **KNOWN LIMITATION**: alphanumeric-glued tokens without separators (`HARDLAB11`, `6a635623e114b8...`) still require exact match — no sub-token expansion possible without character n-grams (5-10× index overhead). Tracked as follow-up.
+- **VERSION**: 4.6.0 → **4.7.0** (MINOR — tokenization behavior change + reindex requirement, though no API/config break).
+
 ### v4.6.0 (2026-07-30) — MCP spec 2026-07-28 via Anthropic Tier 1 SDK (mcp 2.x)
 
 - **NEW (protocol)**: adopt the [MCP spec `2026-07-28`](https://blog.modelcontextprotocol.io/posts/2026-07-28/) — stateless request/response core, per-request `_meta.io.modelcontextprotocol/protocolVersion` + `clientCapabilities`, `serverInfo` on responses, retirement of the `initialize`/`initialized` handshake and the `Mcp-Session-Id` header. The MCP client Claude Code (and every Tier 1 client) negotiates the spec version per request, so this release stays backwards-compatible for clients speaking earlier spec revisions.

{knowledge_rag-4.6.0 → knowledge_rag-4.7.0}/mcp_server/__init__.py

@@ -8,7 +8,7 @@ import sys  # noqa: I001
 _original_stdout = sys.stdout
 sys.stdout = sys.stderr
 
-__version__ = "4.6.0"
+__version__ = "4.7.0"
 __author__ = "Ailton Rocha (Lyon.)"
 
 from .config import Config  # noqa: E402

{knowledge_rag-4.6.0 → knowledge_rag-4.7.0}/mcp_server/server.py

@@ -696,9 +696,30 @@ class BM25Index:
         self._index_built: bool = False
 
     def _tokenize(self, text: str) -> List[str]:
-        """Simple tokenization: lowercase, split on non-alphanumeric, keep hyphens"""
+        """Tokenize: lowercase, split on non-alphanumeric, emit composite + sub-tokens.
+
+        For alphanumeric-code composites (containing at least one digit — e.g.
+        "mdr-ad002", "cve-2024-1234", "ms17-010"), emit both the composite token
+        AND its sub-parts of length >= 2. This enables fragment queries ("AD002",
+        "CVE", "010") to match while IDF preserves exact-match ranking (composite
+        is rarer).
+
+        Natural-language hyphenated words (e.g. "pass-the-hash", "state-of-the-art")
+        are kept as single tokens — expanding them would flood the inverted index
+        with high-frequency stop-word-like sub-parts and hurt query throughput
+        without helping recall for typical use cases.
+        """
         text_lower = text.lower()
-        tokens = re.findall(r"[a-z0-9][-a-z0-9]*[a-z0-9]|[a-z0-9]", text_lower)
+        composite_tokens = re.findall(r"[a-z0-9][-a-z0-9]*[a-z0-9]|[a-z0-9]", text_lower)
+        tokens: List[str] = []
+        for tok in composite_tokens:
+            tokens.append(tok)
+            # Only expand codes (composites containing at least one digit).
+            # Skips natural-language phrases like "pass-the-hash".
+            if "-" in tok and any(c.isdigit() for c in tok):
+                for part in tok.split("-"):
+                    if len(part) >= 2:
+                        tokens.append(part)
         return tokens
 
     def expand_query(self, query: str) -> str:

{knowledge_rag-4.6.0 → knowledge_rag-4.7.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "knowledge-rag"
-version = "4.6.0"
+version = "4.7.0"
 description = "Local RAG System for Claude Code — Hybrid search + Cross-encoder Reranking + 13 MCP Tools + 20 Format Parsers. Zero external servers."
 readme = "README.md"
 license = {text = "MIT"}