knowledge-rag 4.3.1__tar.gz → 4.5.0__tar.gz

{knowledge_rag-4.3.1 → knowledge_rag-4.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: knowledge-rag
-Version: 4.3.1
+Version: 4.5.0
 Summary: Local RAG System for Claude Code — Hybrid search + Cross-encoder Reranking + 13 MCP Tools + 20 Format Parsers. Zero external servers.
 Project-URL: Homepage, https://github.com/lyonzin/knowledge-rag
 Project-URL: Repository, https://github.com/lyonzin/knowledge-rag
@@ -1449,6 +1449,28 @@ Common issues:
 
 ### Unreleased
 
+### v4.5.0 (2026-07-06) — Hybrid Search Ranking & Routing Fix
+
+- **FIX**: `search_knowledge` now searches the entire index when the user omits `category_filter`. The internal `_route_by_keywords()` heuristic previously acted as a hard where-filter on both the semantic and BM25 branches, so queries whose terms happened to map to a sparsely-populated category could return two documents while thousands of relevant chunks in other categories were silently dropped. The router is now **informational only**: the `routed_by` field is still populated in every result for telemetry (public API unchanged), but the candidate set is never restricted. Users who want a hard filter still get it by passing `category_filter=...` explicitly — which continues to filter BM25 consistently with #109. (#112)
+- **NEW**: Hybrid search adds a small bounded ranking boost when query terms match indexed `source` or `filename` metadata, helping navigational queries surface the canonical file for a topic instead of adjacent files that only cross-reference it. The boost is capped at ~20% of typical RRF magnitudes, so it breaks near-ties without overriding semantic or BM25 signal. Public API unchanged. (#110, thanks @Hohlas)
+- **TEST**: New `TestKeywordRoutingBehavior` (3 tests) pins the routing fix — BM25 and semantic branches must both see the full corpus when no explicit filter is passed, and explicit `category_filter` still overrides the router. New `TestPathAwareRanking` covers the path-metadata boost. Baseline: 266 → 271.
+- **UPGRADE NOTE**: warm `query_cache` entries from before v4.5.0 should be invalidated by restarting the server so cached responses no longer reflect the pre-fix restrictive behavior.
+
+### v4.4.0 (2026-07-06) — Cross-Platform Installer & Hybrid Search Category Filter
+
+- **NEW**: Cross-platform, multi-LLM-client installer (`install.py`) driving both `install.sh` (Linux/macOS) and `install.ps1` (Windows) as thin wrappers. One codebase, one behavior across every OS.
+- **NEW**: Auto-detects and registers `knowledge-rag` in 8 LLM clients — Claude Code, Claude Desktop, Cursor, Windsurf, VS Code (Copilot Chat), Cline, Gemini CLI, Zed — writing to each tool's canonical config path with the correct JSON schema per client (VS Code uses `servers`, Zed uses `context_servers`, everyone else uses `mcpServers`).
+- **NEW**: `--for <clients>` / `--exclude <clients>` opt-in/opt-out selection, `--dry-run` preview, `--list-clients` registry inspection, `--pypi-version <ver>` pinning, `--skip-init` / `--skip-model` for fast reruns. See `python install.py --help`.
+- **FIX**: `install.ps1` no longer writes MCP config to `~/.claude/mcp.json` (a stale secondary path); it now targets `~/.claude.json` — the file Claude Code actually reads — via idempotent JSON merge that preserves every existing MCP server entry with an automatic `.knowledge-rag.bak` backup.
+- **FIX**: `install.ps1` gains PyPI mode (`pip install knowledge-rag`) and runs `mcp_server.server init` on install — feature parity with `install.sh`.
+- **FIX**: MCP server spec no longer uses the fragile `cmd /c cd /d ... && python ...` wrapper on Windows; it emits the standard `command` + `cwd` shape supported natively by every modern client.
+- **FIX**: `install.sh` guards against `sh install.sh` (bash-only features now emit a clear error instead of a cryptic syntax failure).
+- **FIX**: Both scripts now correctly advertise **13 MCP tools** (was outdated at 12; `get_reindex_status` shipped in v4.3.0).
+- **FIX**: Windows-side `install.ps1` prefers `winget install Python.Python.3.12 --scope user` (no admin), falls back to python.org 3.12.7 (was pinned to 3.12.0).
+- **FIX**: Hybrid search now applies the active category filter to BM25 results before RRF fusion, preventing keyword-only BM25 hits from other categories from leaking into filtered searches. Both leak paths are closed: BM25 candidates are metadata-filtered before RRF (with `top_k` widened to `max_results * 20` to compensate for post-filter drop), and the fallback fetch during fusion re-checks `category` before adding a chunk to `combined_scores`. Note: the `category_filter` guard now also applies to the keyword-routed category — `_route_by_keywords()`-inferred routing filters BM25 too, consistent with the semantic branch. Users with warm `query_cache` entries should restart the server to invalidate stale results. (#109, thanks @Hohlas)
+- **TEST**: New `tests/test_installer_no_data_loss.py` (22 tests) locks in the installer's zero-data-loss contract across all three JSON schemas (`mcpServers` / `servers` / `context_servers`): top-level keys preserved, sibling MCP servers byte-identical, `.knowledge-rag.bak` backup written before every mutation, idempotent second run, `--dry-run` writes nothing, atomic `os.replace` write. Baseline: 231 → 266.
+- **TEST**: New `TestHybridCategoryFilter::test_bm25_results_respect_category_filter` in `tests/test_search.py` — deterministic regression covering BM25-only hits with mixed categories. Baseline: 266 → 267.
+
 ### v4.3.1 (2026-06-22) — Hybrid Search Fixes
 
 - **FIX**: Accept `"general"` as a valid category in `search_knowledge`. The parser hardcodes `"general"` as the fallback in `_detect_category` (`ingestion.py`), but the validator only built `valid_categories` from `config.keyword_routes` + `config.category_mappings.values()` — so users who customized `config.yaml` and dropped the default `"general": "general"` mapping hit `Invalid category` even though the index contained `general` documents. Validator now always tolerates `"general"`. (#98, thanks @Hohlas)

{knowledge_rag-4.3.1 → knowledge_rag-4.5.0}/README.md

@@ -1401,6 +1401,28 @@ Common issues:
 
 ### Unreleased
 
+### v4.5.0 (2026-07-06) — Hybrid Search Ranking & Routing Fix
+
+- **FIX**: `search_knowledge` now searches the entire index when the user omits `category_filter`. The internal `_route_by_keywords()` heuristic previously acted as a hard where-filter on both the semantic and BM25 branches, so queries whose terms happened to map to a sparsely-populated category could return two documents while thousands of relevant chunks in other categories were silently dropped. The router is now **informational only**: the `routed_by` field is still populated in every result for telemetry (public API unchanged), but the candidate set is never restricted. Users who want a hard filter still get it by passing `category_filter=...` explicitly — which continues to filter BM25 consistently with #109. (#112)
+- **NEW**: Hybrid search adds a small bounded ranking boost when query terms match indexed `source` or `filename` metadata, helping navigational queries surface the canonical file for a topic instead of adjacent files that only cross-reference it. The boost is capped at ~20% of typical RRF magnitudes, so it breaks near-ties without overriding semantic or BM25 signal. Public API unchanged. (#110, thanks @Hohlas)
+- **TEST**: New `TestKeywordRoutingBehavior` (3 tests) pins the routing fix — BM25 and semantic branches must both see the full corpus when no explicit filter is passed, and explicit `category_filter` still overrides the router. New `TestPathAwareRanking` covers the path-metadata boost. Baseline: 266 → 271.
+- **UPGRADE NOTE**: warm `query_cache` entries from before v4.5.0 should be invalidated by restarting the server so cached responses no longer reflect the pre-fix restrictive behavior.
+
+### v4.4.0 (2026-07-06) — Cross-Platform Installer & Hybrid Search Category Filter
+
+- **NEW**: Cross-platform, multi-LLM-client installer (`install.py`) driving both `install.sh` (Linux/macOS) and `install.ps1` (Windows) as thin wrappers. One codebase, one behavior across every OS.
+- **NEW**: Auto-detects and registers `knowledge-rag` in 8 LLM clients — Claude Code, Claude Desktop, Cursor, Windsurf, VS Code (Copilot Chat), Cline, Gemini CLI, Zed — writing to each tool's canonical config path with the correct JSON schema per client (VS Code uses `servers`, Zed uses `context_servers`, everyone else uses `mcpServers`).
+- **NEW**: `--for <clients>` / `--exclude <clients>` opt-in/opt-out selection, `--dry-run` preview, `--list-clients` registry inspection, `--pypi-version <ver>` pinning, `--skip-init` / `--skip-model` for fast reruns. See `python install.py --help`.
+- **FIX**: `install.ps1` no longer writes MCP config to `~/.claude/mcp.json` (a stale secondary path); it now targets `~/.claude.json` — the file Claude Code actually reads — via idempotent JSON merge that preserves every existing MCP server entry with an automatic `.knowledge-rag.bak` backup.
+- **FIX**: `install.ps1` gains PyPI mode (`pip install knowledge-rag`) and runs `mcp_server.server init` on install — feature parity with `install.sh`.
+- **FIX**: MCP server spec no longer uses the fragile `cmd /c cd /d ... && python ...` wrapper on Windows; it emits the standard `command` + `cwd` shape supported natively by every modern client.
+- **FIX**: `install.sh` guards against `sh install.sh` (bash-only features now emit a clear error instead of a cryptic syntax failure).
+- **FIX**: Both scripts now correctly advertise **13 MCP tools** (was outdated at 12; `get_reindex_status` shipped in v4.3.0).
+- **FIX**: Windows-side `install.ps1` prefers `winget install Python.Python.3.12 --scope user` (no admin), falls back to python.org 3.12.7 (was pinned to 3.12.0).
+- **FIX**: Hybrid search now applies the active category filter to BM25 results before RRF fusion, preventing keyword-only BM25 hits from other categories from leaking into filtered searches. Both leak paths are closed: BM25 candidates are metadata-filtered before RRF (with `top_k` widened to `max_results * 20` to compensate for post-filter drop), and the fallback fetch during fusion re-checks `category` before adding a chunk to `combined_scores`. Note: the `category_filter` guard now also applies to the keyword-routed category — `_route_by_keywords()`-inferred routing filters BM25 too, consistent with the semantic branch. Users with warm `query_cache` entries should restart the server to invalidate stale results. (#109, thanks @Hohlas)
+- **TEST**: New `tests/test_installer_no_data_loss.py` (22 tests) locks in the installer's zero-data-loss contract across all three JSON schemas (`mcpServers` / `servers` / `context_servers`): top-level keys preserved, sibling MCP servers byte-identical, `.knowledge-rag.bak` backup written before every mutation, idempotent second run, `--dry-run` writes nothing, atomic `os.replace` write. Baseline: 231 → 266.
+- **TEST**: New `TestHybridCategoryFilter::test_bm25_results_respect_category_filter` in `tests/test_search.py` — deterministic regression covering BM25-only hits with mixed categories. Baseline: 266 → 267.
+
 ### v4.3.1 (2026-06-22) — Hybrid Search Fixes
 
 - **FIX**: Accept `"general"` as a valid category in `search_knowledge`. The parser hardcodes `"general"` as the fallback in `_detect_category` (`ingestion.py`), but the validator only built `valid_categories` from `config.keyword_routes` + `config.category_mappings.values()` — so users who customized `config.yaml` and dropped the default `"general": "general"` mapping hit `Invalid category` even though the index contained `general` documents. Validator now always tolerates `"general"`. (#98, thanks @Hohlas)

{knowledge_rag-4.3.1 → knowledge_rag-4.5.0}/mcp_server/__init__.py

@@ -8,7 +8,7 @@ import sys  # noqa: I001
 _original_stdout = sys.stdout
 sys.stdout = sys.stderr
 
-__version__ = "4.3.1"
+__version__ = "4.5.0"
 __author__ = "Ailton Rocha (Lyon.)"
 
 from .config import Config  # noqa: E402

{knowledge_rag-4.3.1 → knowledge_rag-4.5.0}/mcp_server/server.py

@@ -637,6 +637,33 @@ class CrossEncoderReranker:
         return documents[:top_k]
 
 
+def _metadata_path_score(query: str, metadata: Dict[str, Any]) -> float:
+    """Return a small generic score boost when query terms match path metadata."""
+    query_terms = re.findall(r"[a-z0-9][-a-z0-9]*[a-z0-9]|[a-z0-9]", query.lower())
+    if not query_terms:
+        return 0.0
+
+    source = str(metadata.get("source", ""))
+    filename = str(metadata.get("filename", ""))
+    path_text = f"{source} {filename}".lower()
+    path_tokens = set(re.findall(r"[a-z0-9][-a-z0-9]*[a-z0-9]|[a-z0-9]", path_text))
+    if not path_tokens:
+        return 0.0
+
+    score = 0.0
+    for term in query_terms:
+        if term in path_tokens:
+            score += 0.0006
+        elif term in path_text:
+            score += 0.0003
+
+    query_phrase = query.strip().lower()
+    if query_phrase and query_phrase in path_text:
+        score += 0.0012
+
+    return min(score, 0.003)
+
+
 # =============================================================================
 # BM25 INDEX
 # =============================================================================
@@ -1466,13 +1493,23 @@ class KnowledgeOrchestrator:
 
         self._ensure_bm25_index()
 
-        # Keyword routing
+        # Keyword routing — informational only.
+        # `routed_category` is surfaced via the `routed_by` field for telemetry,
+        # but MUST NOT restrict the search when the user did not pass an explicit
+        # `category_filter`. Auto-routing to a sparsely-populated category (e.g. one
+        # with only a handful of docs) was hiding relevant material that lived under
+        # the top-level `security` bucket. Users who want a hard filter still get it
+        # by passing `category_filter=...` explicitly.
         routed_category = self._route_by_keywords(query_text)
         where_filter = None
         if category_filter:
             where_filter = {"category": category_filter}
-        elif routed_category:
-            where_filter = {"category": routed_category}
+
+        def _matches_category(metadata: Dict[str, Any]) -> bool:
+            if not where_filter:
+                return True
+            expected_category = where_filter.get("category")
+            return not expected_category or metadata.get("category") == expected_category
 
         # Parallel Semantic + BM25 search (threaded for latency reduction)
         from concurrent.futures import ThreadPoolExecutor
@@ -1507,8 +1544,23 @@ class KnowledgeOrchestrator:
             r = {}
             if hybrid_alpha < 1.0:
                 try:
-                    bm25_hits = self.bm25_index.search(query_text, top_k=max_results * 3)
-                    for rank, (chunk_id, bm25_score) in enumerate(bm25_hits):
+                    bm25_top_k = max_results * (20 if where_filter else 3)
+                    bm25_hits = self.bm25_index.search(query_text, top_k=bm25_top_k)
+
+                    if where_filter:
+                        chunk_ids = [chunk_id for chunk_id, _ in bm25_hits]
+                        metadata_by_id = {}
+                        if chunk_ids:
+                            fetched = self.collection.get(ids=chunk_ids, include=["metadatas"])
+                            metadata_by_id = dict(zip(fetched.get("ids", []), fetched.get("metadatas", [])))
+
+                        bm25_hits = [
+                            (chunk_id, bm25_score)
+                            for chunk_id, bm25_score in bm25_hits
+                            if _matches_category(metadata_by_id.get(chunk_id, {}))
+                        ]
+
+                    for rank, (chunk_id, bm25_score) in enumerate(bm25_hits[: max_results * 3]):
                         r[chunk_id] = {"rank": rank + 1, "bm25_score": bm25_score}
                 except Exception as e:
                     print(f"[WARN] BM25 search failed: {e}")
@@ -1558,8 +1610,11 @@ class KnowledgeOrchestrator:
                 except Exception:
                     continue
 
+            if not _matches_category(data.get("metadata", {})):
+                continue
+
             combined_scores[chunk_id] = {
-                "rrf_score": combined_rrf,
+                "rrf_score": combined_rrf + _metadata_path_score(query_text, data.get("metadata", {})),
                 "semantic_rank": semantic_rank if chunk_id in semantic_results else None,
                 "bm25_rank": bm25_rank if chunk_id in bm25_results else None,
                 "document": data.get("document", ""),

{knowledge_rag-4.3.1 → knowledge_rag-4.5.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "knowledge-rag"
-version = "4.3.1"
+version = "4.5.0"
 description = "Local RAG System for Claude Code — Hybrid search + Cross-encoder Reranking + 13 MCP Tools + 20 Format Parsers. Zero external servers."
 readme = "README.md"
 license = {text = "MIT"}