knowledge-rag 4.1.2__tar.gz → 4.3.0__tar.gz

knowledge_rag-4.1.2/README.md → knowledge_rag-4.3.0/PKG-INFO

@@ -1,3 +1,51 @@
+Metadata-Version: 2.4
+Name: knowledge-rag
+Version: 4.3.0
+Summary: Local RAG System for Claude Code — Hybrid search + Cross-encoder Reranking + 13 MCP Tools + 20 Format Parsers. Zero external servers.
+Project-URL: Homepage, https://github.com/lyonzin/knowledge-rag
+Project-URL: Repository, https://github.com/lyonzin/knowledge-rag
+Project-URL: Issues, https://github.com/lyonzin/knowledge-rag/issues
+Project-URL: Changelog, https://github.com/lyonzin/knowledge-rag/releases
+Author-email: "Lyon." <lyonzin@users.noreply.github.com>
+License: MIT
+License-File: LICENSE
+Keywords: bm25,chromadb,claude-code,embeddings,fastembed,hybrid-search,knowledge-base,local-ai,mcp,rag,reranking,retrieval-augmented-generation,semantic-search
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Indexing
+Requires-Python: >=3.11
+Requires-Dist: beautifulsoup4>=4.12.0
+Requires-Dist: chromadb>=1.4.0
+Requires-Dist: fastembed[reranking]>=0.4.0
+Requires-Dist: mcp>=1.6.0
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: openpyxl>=3.1.0
+Requires-Dist: pymupdf>=1.23.0
+Requires-Dist: python-docx>=1.0.0
+Requires-Dist: python-pptx>=1.0.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: requests>=2.33.0
+Requires-Dist: watchdog>=4.0.0
+Provides-Extra: gpu
+Requires-Dist: nvidia-cublas-cu12; extra == 'gpu'
+Requires-Dist: nvidia-cuda-runtime-cu12; extra == 'gpu'
+Requires-Dist: nvidia-cudnn-cu12; extra == 'gpu'
+Requires-Dist: nvidia-cufft-cu12; extra == 'gpu'
+Requires-Dist: nvidia-curand-cu12; extra == 'gpu'
+Requires-Dist: nvidia-cusolver-cu12; extra == 'gpu'
+Requires-Dist: nvidia-cusparse-cu12; extra == 'gpu'
+Requires-Dist: nvidia-nvjitlink-cu12; extra == 'gpu'
+Requires-Dist: onnxruntime-gpu>=1.14.0; extra == 'gpu'
+Provides-Extra: server
+Requires-Dist: uvicorn>=0.20.0; extra == 'server'
+Description-Content-Type: text/markdown
+
 # Knowledge RAG
 
 <div align="center">
@@ -17,7 +65,7 @@
 ### Your docs, your machine, zero cloud. Claude Code searches them natively.
 
 Drop your PDFs, markdown, code, notebooks — **1800+ files, 39K chunks, indexed in under 3 minutes.**<br/>
-Hybrid search (BM25 + semantic vectors + cross-encoder reranking) through 12 MCP tools.<br/>
+Hybrid search (BM25 + semantic vectors + cross-encoder reranking) through 13 MCP tools.<br/>
 Everything runs locally via ONNX. No Docker, no Ollama, no API keys, no data leaves your machine.
 
 ```
@@ -26,9 +74,9 @@ pip install knowledge-rag → restart Claude Code → search_knowledge("your que
 
 ---
 
-**12 MCP Tools** | **Hybrid Search + Reranking** | **20 File Formats** | **Optional NVIDIA GPU** | **100% Local**
+**13 MCP Tools** | **Hybrid Search + Reranking** | **20 File Formats** | **Optional NVIDIA GPU** | **100% Local**
 
-[What's New](#whats-new-in-v400) | [Supported Formats](#supported-formats) | [Installation](#installation) | [Configuration](#configuration) | [API Reference](#api-reference) | [Architecture](#architecture)
+[What's New](#whats-new-in-v420) | [Supported Formats](#supported-formats) | [Installation](#installation) | [Configuration](#configuration) | [API Reference](#api-reference) | [Architecture](#architecture)
 
 </div>
 
@@ -50,7 +98,17 @@ pip install knowledge-rag → restart Claude Code → search_knowledge("your que
 
 ---
 
-## What's New in v4.0.0
+## What's New in v4.2.0
+
+### Search Performance & Output Quality (v4.2.0)
+
+**128× faster BM25 search** — replaced `rank-bm25` full-corpus scan with a custom **inverted-index** implementation. Only documents containing query terms are scored, using `numpy.argpartition` for O(n) top-k selection. Adjacent chunk fetching now uses a single batched ChromaDB call instead of N round-trips, and an O(1) reverse lookup (`_source_to_docid`) eliminates linear scans.
+
+**Smarter output** — two new parameters on `search_knowledge`:
+- **`snippet_mode`** (default: `true`) — truncates content to ~500 characters at natural break points, reducing token consumption by ~72%. Adds `content_length` field with original size; use `get_document()` for full content.
+- **`min_score`** — filters results below a normalized relevance threshold (0.0–1.0). Eliminates low-quality noise from results. Response includes `filtered_by_score` count for transparency.
+
+Both parameters are fully backwards-compatible (existing callers see no change in behavior).
 
 ### Enterprise Concurrent Access — SSE/HTTP Transport (v4.0.0)
 
@@ -71,7 +129,7 @@ Or via CLI: `knowledge-rag --transport sse`
 - **Prometheus metrics**: `/metrics` endpoint on separate port
 - **Bearer auth**: Token validation for SSE/HTTP connections
 
-All 12 MCP tools are instrumented with `@rate_limited` and `@instrument` decorators — zero overhead when features are disabled. Default transport remains **stdio** for full backwards compatibility.
+All 13 MCP tools are instrumented with `@rate_limited` and `@instrument` decorators — zero overhead when features are disabled. Default transport remains **stdio** for full backwards compatibility.
 
 > **Migration**: Existing users need zero changes. SSE mode is opt-in via `server.transport: "sse"` in config.yaml. See [Configuration](#configuration) for details.
 
@@ -195,7 +253,7 @@ See [Changelog](#changelog) for full history.
 | **MMR Diversification** | Maximal Marginal Relevance reduces redundant results |
 | **Persistent Model Cache** | Embedding models cached in `models_cache/` — survives reboots |
 | **Auto-Migration** | Detects embedding dimension mismatch and rebuilds automatically |
-| **12 MCP Tools** | Full CRUD + search + evaluation via Claude Code |
+| **13 MCP Tools** | Full CRUD + search + evaluation via Claude Code |
 
 ---
 
@@ -207,14 +265,14 @@ See [Changelog](#changelog) for full history.
 flowchart TB
     subgraph MCP["MCP SERVER (FastMCP)"]
         direction TB
-        TOOLS["12 MCP Tools<br/>search | get | add | update | remove<br/>reindex | list | stats | url | similar | evaluate"]
+        TOOLS["13 MCP Tools<br/>search | get | add | update | remove<br/>reindex | reindex_status | list | stats | url | similar | evaluate"]
     end
 
     subgraph SEARCH["HYBRID SEARCH ENGINE"]
         direction LR
         ROUTER["Keyword Router<br/>(word boundaries)"]
         SEMANTIC["Semantic Search<br/>(ChromaDB)"]
-        BM25["BM25 Keyword<br/>(rank-bm25 + expansion)"]
+        BM25["BM25 Keyword<br/>(inverted-index + expansion)"]
         RRF["Reciprocal Rank<br/>Fusion (RRF)"]
         RERANK["Cross-Encoder<br/>Reranker"]
 
@@ -278,15 +336,23 @@ flowchart TB
     subgraph HYBRID["Hybrid Search"]
         direction LR
         SEMANTIC["Semantic Search<br/>(ChromaDB embeddings)<br/>Conceptual similarity"]
-        BM25["BM25 Search<br/>(expanded query)<br/>Exact term matching"]
+        BM25["BM25 Inverted-Index<br/>(posting lists + numpy top-k)<br/>Exact term matching"]
     end
 
     subgraph FUSION["Result Fusion + Reranking"]
         RRF["Reciprocal Rank Fusion<br/>score = alpha * 1/(k+rank_sem)<br/>+ (1-alpha) * 1/(k+rank_bm25)"]
         RERANK["Cross-Encoder Reranker<br/>Re-scores top 3x candidates<br/>query+doc pair scoring"]
         SORT["Sort by Reranker Score<br/>Normalize to 0-1"]
+        ADJ["Adjacent Chunk Expansion<br/>(batch fetch ±1 chunk)"]
 
-        RRF --> RERANK --> SORT
+        RRF --> RERANK --> SORT --> ADJ
+    end
+
+    subgraph OUTPUT["Output Processing"]
+        MINSCORE["min_score Filter<br/>(discard below threshold)"]
+        SNIPPET["snippet_mode Truncation<br/>(~500 chars at natural break)"]
+
+        MINSCORE --> SNIPPET
     end
 
     CATEGORY --> HYBRID
@@ -294,7 +360,8 @@ flowchart TB
     SEMANTIC --> RRF
     BM25 --> RRF
 
-    SORT --> RESULTS["Results<br/>search_method: hybrid|semantic|keyword<br/>score + reranker_score + raw_rrf_score"]
+    ADJ --> MINSCORE
+    SNIPPET --> RESULTS["Results<br/>search_method: hybrid|semantic|keyword<br/>score + filtered_by_score + content_length"]
 ```
 
 ### Document Ingestion Flow
@@ -373,7 +440,53 @@ flowchart LR
 - Claude Code CLI
 - *…or any other MCP client (Claude Desktop, Cursor, VS Code, Antigravity, opencode, Windsurf) — see [Use with other MCP clients](#use-with-other-mcp-clients)*
 - ~200MB disk for model cache (auto-downloaded on first run)
-- *Optional:* NVIDIA GPU + CUDA for accelerated embeddings (`pip install knowledge-rag[gpu]` + `models.embedding.gpu: true` in config)
+- *Optional:* NVIDIA GPU + CUDA 12 for accelerated embeddings (see [GPU Acceleration](#gpu-acceleration) below)
+
+### GPU Acceleration
+
+GPU mode accelerates embedding generation during indexing and search. It requires an NVIDIA GPU with CUDA 12 support. No GPU? No problem — the server runs on CPU by default and GPU is entirely optional.
+
+**Requirements:**
+
+| Component | Minimum | How to check / get it |
+|-----------|---------|----------------------|
+| NVIDIA GPU (Turing+) | RTX 20xx / 30xx / 40xx / 50xx, or Tesla T4+ | `nvidia-smi` |
+| NVIDIA Driver | ≥ 525 | `nvidia-smi` — [nvidia.com/drivers](https://www.nvidia.com/drivers) |
+| CUDA 12 runtime | Provided by pip packages below | Automatic |
+
+**Setup (2 steps):**
+
+```bash
+# 1. Install GPU dependencies (onnxruntime-gpu + all CUDA 12 runtime DLLs)
+pip install knowledge-rag[gpu]
+
+# 2. Enable in config.yaml
+# models:
+#   embedding:
+#     gpu: true
+```
+
+The `[gpu]` extra installs `onnxruntime-gpu` plus 7 NVIDIA CUDA 12 packages (`cublas`, `cudnn`, `cuda-runtime`, `cufft`, `cusparse`, `cusolver`, `curand`, `nvjitlink`) so you don't need a full CUDA Toolkit install.
+
+**Verify GPU is active:**
+
+On server startup, look for the GPU status banner:
+```
+============================================================
+  GPU STATUS: ACTIVE
+  Provider:   CUDAExecutionProvider
+  Device:     NVIDIA GeForce RTX 3080 Ti
+  VRAM:       12.0 GB
+============================================================
+```
+
+Or programmatically:
+```bash
+python -c "import onnxruntime; print(onnxruntime.get_available_providers())"
+# Should include: 'CUDAExecutionProvider'
+```
+
+> **Fallback**: If CUDA is unavailable at runtime (wrong driver, missing DLLs, no GPU), the server falls back to CPU automatically with a `[WARN]` log — it never crashes. The `gpu: true` config is a preference, not a requirement.
 
 ### Install Methods
 
@@ -644,7 +757,7 @@ search_knowledge("lateral movement strategies", hybrid_alpha=1.0)
 
 ### Indexing
 
-Documents are automatically indexed on first startup. To manage the index:
+Documents are automatically indexed on first startup. All reindex operations run **in background** — they return immediately and you poll progress via `get_reindex_status()`:
 
 ```python
 # Incremental: only re-index changed files (fast)
@@ -655,6 +768,10 @@ reindex_documents(force=True)
 
 # Nuclear rebuild: delete everything, re-embed all (use after model change)
 reindex_documents(full_rebuild=True)
+
+# Poll progress (lightweight, no full stats computation)
+get_reindex_status()
+# → {"reindex": {"active": true, "percent": 56, "progress": "2090/3734", ...}}
 ```
 
 ### Evaluating Retrieval Quality
@@ -683,6 +800,8 @@ Hybrid search combining semantic search + BM25 keyword search with cross-encoder
 | `max_results` | int | 5 | Maximum results to return (1-20) |
 | `category` | string | null | Filter by category |
 | `hybrid_alpha` | float | 0.3 | Balance: 0.0 = keyword only, 1.0 = semantic only |
+| `min_score` | float | 0.0 | Minimum relevance score (0.0-1.0) to include a result. Use 0.2-0.4 to cut noise |
+| `snippet_mode` | bool | true | Truncate content to ~500 chars at natural break points. Adds `content_length` field |
 
 **Returns:**
 
@@ -692,6 +811,7 @@ Hybrid search combining semantic search + BM25 keyword search with cross-encoder
   "query": "mimikatz credential dump",
   "hybrid_alpha": 0.5,
   "result_count": 3,
+  "filtered_by_score": 2,
   "cache_hit_rate": "0.0%",
   "results": [
     {
@@ -733,14 +853,39 @@ Retrieve the full content of a specific document.
 
 #### `reindex_documents`
 
-Index or reindex all documents in the knowledge base.
+Index or reindex all documents in the knowledge base. **Runs in background** — returns immediately. Poll progress via `get_reindex_status()`.
 
 | Parameter | Type | Default | Description |
 |-----------|------|---------|-------------|
 | `force` | bool | false | Smart reindex: detects changes, rebuilds BM25. Fast. |
 | `full_rebuild` | bool | false | Nuclear rebuild: deletes everything, re-embeds all documents. Use after model change. |
 
-**Returns:** JSON with indexing statistics (indexed, updated, skipped, deleted, chunks_added, chunks_removed, dedup_skipped, elapsed_seconds).
+**Returns:** `{"status": "started", "operation": "..."}` immediately. If already running, returns `{"status": "already_running", "progress": "1200/3734"}`.
+
+---
+
+#### `get_reindex_status`
+
+Get the current status of a background reindex operation. Lightweight — does not compute full index statistics.
+
+**Returns (active):**
+```json
+{
+  "status": "success",
+  "reindex": {
+    "active": true,
+    "operation": "nuclear_rebuild",
+    "progress": "1200/3734",
+    "percent": 32,
+    "indexed": 1200,
+    "skipped": 0,
+    "errors": 0,
+    "started_at": "2026-06-17T18:29:49"
+  }
+}
+```
+
+**Returns (idle):** `{"status": "success", "reindex": {"active": false}}`
 
 ---
 
@@ -1049,7 +1194,7 @@ For `.md` files, chunking splits at `##` and `###` header boundaries first. Sect
 |-------|---------|-------------|
 | `models.embedding.model` | `BAAI/bge-small-en-v1.5` | Embedding model (ONNX, runs locally) |
 | `models.embedding.dimensions` | 384 | Vector dimensions (must match model) |
-| `models.embedding.gpu` | false | Enable CUDA GPU acceleration. Requires `pip install knowledge-rag[gpu]` |
+| `models.embedding.gpu` | false | Enable CUDA GPU acceleration. See [GPU Acceleration](#gpu-acceleration) for full setup |
 | `models.reranker.enabled` | true | Enable cross-encoder reranking |
 | `models.reranker.model` | `Xenova/ms-marco-MiniLM-L-6-v2` | Reranker model |
 | `models.reranker.top_k_multiplier` | 3 | Fetch N*multiplier candidates for reranking |
@@ -1309,7 +1454,7 @@ Common issues:
 - **NEW**: ChromaDB WAL mode enabled automatically in SSE/HTTP mode for concurrent read performance.
 - **NEW**: Optional rate limiting — sliding-window counter, configurable RPM and burst, disabled by default.
 - **NEW**: Optional Prometheus metrics endpoint — tool call counts, latency histograms, separate port, disabled by default.
-- **NEW**: All 12 MCP tools instrumented with `@rate_limited` and `@instrument` decorators (zero-cost when disabled).
+- **NEW**: All 13 MCP tools instrumented with `@rate_limited` and `@instrument` decorators (zero-cost when disabled).
 - **NEW**: `--transport` CLI override for Docker/systemd deployments.
 - **NEW**: `pip install knowledge-rag[server]` optional dependency for SSE/HTTP (uvicorn).
 - **CHANGED**: SSE/HTTP mode auto-enables single-instance lock (port collision prevention).
@@ -1369,6 +1514,36 @@ Common issues:
 
 ### Unreleased
 
+### v4.3.0 (2026-06-17) — Async Reindex, GPU CUDA 12, 13th MCP Tool
+
+- **NEW**: `get_reindex_status` MCP tool — lightweight reindex progress polling without computing full index stats. Returns active/idle status, percent, processed/total, errors, and last result.
+- **NEW**: `reindex_documents` now runs in background via daemon thread — returns immediately with `{"status": "started"}`. Eliminates MCP timeout on large document sets (5K+ files). Concurrent calls return `already_running` with current progress.
+- **NEW**: GPU acceleration with full CUDA 12 support — `onnxruntime-gpu` + 7 NVIDIA pip packages (`cublas`, `cudnn`, `cuda-runtime`, `cufft`, `cusparse`, `cusolver`, `curand`, `nvjitlink`). Server auto-detects GPU on startup with 4-step verification (providers, DLLs, nvidia-smi, session creation). Falls back to CPU gracefully.
+- **NEW**: `_setup_cuda_dll_paths()` adds NVIDIA pip package DLL directories to `PATH` automatically on Windows — onnxruntime finds CUDA 12 DLLs without a full CUDA Toolkit install.
+- **DEPS**: `[gpu]` extra expanded from 3 to 8 packages (added `cufft`, `cusparse`, `cusolver`, `curand`, `nvjitlink`).
+- **FIX**: GPU status reporting now uses actual ONNX session creation test instead of just checking `get_available_providers()` — prevents false "GPU ACTIVE" when CUDA DLLs are missing.
+- **DOCS**: GPU Acceleration section rewritten with complete requirements table, setup steps, verification instructions, and fallback behavior.
+- **DOCS**: Tool reference updated — `reindex_documents` async behavior documented, `get_reindex_status` reference added.
+- **TEST**: Backwards-compat baseline updated for 13 MCP tools.
+
+### v4.2.0 (2026-06-17) — Search Performance & Output Quality
+
+- **PERF**: Custom inverted-index BM25 replaces `rank-bm25` full-corpus scan — 128× faster keyword search on 50K+ chunk corpora. Only documents containing query terms are scored via posting lists.
+- **PERF**: `numpy.argpartition` for O(n) top-k selection instead of O(n log n) sort.
+- **PERF**: Batched adjacent chunk fetch — single ChromaDB `collection.get()` call replaces N round-trips per result.
+- **PERF**: O(1) reverse lookup via `_source_to_docid` dict eliminates linear scans of `_indexed_docs` in `search_similar`, `update_document`, `remove_document`, and `_expand_with_adjacent_chunks`.
+- **NEW**: `snippet_mode` parameter on `search_knowledge` (default: `true`) — truncates content to ~500 chars at natural break points with `content_length` field. Reduces token consumption by ~72%.
+- **NEW**: `min_score` parameter on `search_knowledge` (default: `0.0`) — filters results below a normalized relevance threshold. Response includes `filtered_by_score` count.
+- **NEW**: `filtered_by_score` field in search response JSON for transparency.
+- **DEPS**: `numpy` added as direct dependency (was transitive via fastembed); `rank-bm25` import removed from server.py.
+- **TEST**: 6 new tests for `min_score` filtering and `snippet_mode` truncation.
+- **TEST**: Updated backwards-compat baseline to include new `search_knowledge` parameters.
+
+### v4.1.2 (2026-06-17)
+
+- **FIX**: `_save_metadata` dict snapshot prevents concurrent modification crash during file watcher events.
+- **STYLE**: ruff format applied to server.py.
+
 ### v4.1.1 (2026-06-17)
 
 - **FIX**: All `_indexed_docs` iterations now use `list()` snapshot, preventing `dictionary changed size during iteration` crash when FileWatcher modifies the index concurrently with MCP tool calls (affects `search_knowledge`, `search_similar`, `update_document`, `remove_document`, `evaluate_retrieval`, `list_categories`, `list_documents`)