knowledge-rag 3.9.0 → 4.2.0

package/README.md

@@ -28,17 +28,66 @@ pip install knowledge-rag → restart Claude Code → search_knowledge("your que
 
 **12 MCP Tools** | **Hybrid Search + Reranking** | **20 File Formats** | **Optional NVIDIA GPU** | **100% Local**
 
-[What's New](#whats-new-in-v390) | [Supported Formats](#supported-formats) | [Installation](#installation) | [Configuration](#configuration) | [API Reference](#api-reference) | [Architecture](#architecture)
+[What's New](#whats-new-in-v420) | [Supported Formats](#supported-formats) | [Installation](#installation) | [Configuration](#configuration) | [API Reference](#api-reference) | [Architecture](#architecture)
 
 </div>
 
 ---
 
-## What's New in v3.9.0
+## Star History
+
+<div align="center">
+
+<a href="https://www.star-history.com/?repos=lyonzin%2Fknowledge-rag&type=date&legend=top-left">
+ <picture>
+   <source media="(prefers-color-scheme: dark)" srcset="https://api.star-history.com/chart?repos=lyonzin/knowledge-rag&type=date&theme=dark&legend=top-left" />
+   <source media="(prefers-color-scheme: light)" srcset="https://api.star-history.com/chart?repos=lyonzin/knowledge-rag&type=date&legend=top-left" />
+   <img alt="Star History Chart" src="https://api.star-history.com/chart?repos=lyonzin/knowledge-rag&type=date&legend=top-left" />
+ </picture>
+</a>
+
+</div>
+
+---
+
+## What's New in v4.2.0
+
+### Search Performance & Output Quality (v4.2.0)
+
+**128× faster BM25 search** — replaced `rank-bm25` full-corpus scan with a custom **inverted-index** implementation. Only documents containing query terms are scored, using `numpy.argpartition` for O(n) top-k selection. Adjacent chunk fetching now uses a single batched ChromaDB call instead of N round-trips, and an O(1) reverse lookup (`_source_to_docid`) eliminates linear scans.
+
+**Smarter output** — two new parameters on `search_knowledge`:
+- **`snippet_mode`** (default: `true`) — truncates content to ~500 characters at natural break points, reducing token consumption by ~72%. Adds `content_length` field with original size; use `get_document()` for full content.
+- **`min_score`** — filters results below a normalized relevance threshold (0.0–1.0). Eliminates low-quality noise from results. Response includes `filtered_by_score` count for transparency.
+
+Both parameters are fully backwards-compatible (existing callers see no change in behavior).
+
+### Enterprise Concurrent Access — SSE/HTTP Transport (v4.0.0)
+
+The server now supports **SSE** and **streamable-http** transport modes. Instead of spawning a separate process per client (stdio), a single server process serves all clients with shared resources — 1 embedding model, 1 ChromaDB, 1 query cache.
+
+```yaml
+# config.yaml
+server:
+  transport: "sse"        # "stdio" | "sse" | "streamable-http"
+  host: "127.0.0.1"
+  port: 8179
+```
+
+Or via CLI: `knowledge-rag --transport sse`
+
+**Optional enterprise features** (all disabled by default):
+- **Rate limiting**: Sliding-window counter, configurable RPM and burst
+- **Prometheus metrics**: `/metrics` endpoint on separate port
+- **Bearer auth**: Token validation for SSE/HTTP connections
+
+All 12 MCP tools are instrumented with `@rate_limited` and `@instrument` decorators — zero overhead when features are disabled. Default transport remains **stdio** for full backwards compatibility.
+
+> **Migration**: Existing users need zero changes. SSE mode is opt-in via `server.transport: "sse"` in config.yaml. See [Configuration](#configuration) for details.
 
 ### Quality Gate — 7-Pillar PR Validation
 
-knowledge-rag is now used daily by 70+ enterprise teams. Every PR (including dependabot bumps and one-line fixes) is now evaluated against **35+ automated checks** spread across 7 pillars before any human review:
+Every PR (including dependabot bumps and one-line fixes) is now evaluated against **35+ automated checks** spread across 7 pillars before any human review:
 
 | Pillar | What it enforces | Tools |
 |---|---|---|
@@ -86,6 +135,7 @@ All methods produce the same MCP server. See [Installation](#installation) for f
 
 ### Recent Highlights
 
+- **v4.0.0** — **Enterprise concurrent access**: SSE/HTTP transport (1 server → N clients), thread-safe shared state, optional rate limiting + Prometheus metrics, ChromaDB WAL mode, `--transport` CLI
 - **v3.9.0** — **Quality Gate** activated: 35+ automated PR checks across 7 pillars (Security, Stability, Memory Leak, Versatility, Scalability, Versioning, Quality) + nightly resilience suite (chaos, soak, determinism, mutation)
 - **v3.8.1** — Critical hotfix: loud-fail embeddings (no more silent zero-vector corruption); Windows CI flake erradicated (HF_HUB_OFFLINE + shell:bash + atexit wrapper)
 - **v3.8.0** — Lazy-load embeddings, opt-in single-instance guard, version sync across PyPI/NPM/Docker
@@ -174,7 +224,7 @@ flowchart TB
         direction LR
         ROUTER["Keyword Router<br/>(word boundaries)"]
         SEMANTIC["Semantic Search<br/>(ChromaDB)"]
-        BM25["BM25 Keyword<br/>(rank-bm25 + expansion)"]
+        BM25["BM25 Keyword<br/>(inverted-index + expansion)"]
         RRF["Reciprocal Rank<br/>Fusion (RRF)"]
         RERANK["Cross-Encoder<br/>Reranker"]
 
@@ -238,15 +288,23 @@ flowchart TB
     subgraph HYBRID["Hybrid Search"]
         direction LR
         SEMANTIC["Semantic Search<br/>(ChromaDB embeddings)<br/>Conceptual similarity"]
-        BM25["BM25 Search<br/>(expanded query)<br/>Exact term matching"]
+        BM25["BM25 Inverted-Index<br/>(posting lists + numpy top-k)<br/>Exact term matching"]
     end
 
     subgraph FUSION["Result Fusion + Reranking"]
         RRF["Reciprocal Rank Fusion<br/>score = alpha * 1/(k+rank_sem)<br/>+ (1-alpha) * 1/(k+rank_bm25)"]
         RERANK["Cross-Encoder Reranker<br/>Re-scores top 3x candidates<br/>query+doc pair scoring"]
         SORT["Sort by Reranker Score<br/>Normalize to 0-1"]
+        ADJ["Adjacent Chunk Expansion<br/>(batch fetch ±1 chunk)"]
 
-        RRF --> RERANK --> SORT
+        RRF --> RERANK --> SORT --> ADJ
+    end
+
+    subgraph OUTPUT["Output Processing"]
+        MINSCORE["min_score Filter<br/>(discard below threshold)"]
+        SNIPPET["snippet_mode Truncation<br/>(~500 chars at natural break)"]
+
+        MINSCORE --> SNIPPET
     end
 
     CATEGORY --> HYBRID
@@ -254,7 +312,8 @@ flowchart TB
     SEMANTIC --> RRF
     BM25 --> RRF
 
-    SORT --> RESULTS["Results<br/>search_method: hybrid|semantic|keyword<br/>score + reranker_score + raw_rrf_score"]
+    ADJ --> MINSCORE
+    SNIPPET --> RESULTS["Results<br/>search_method: hybrid|semantic|keyword<br/>score + filtered_by_score + content_length"]
 ```
 
 ### Document Ingestion Flow
@@ -331,6 +390,7 @@ flowchart LR
 
 - Python 3.11+
 - Claude Code CLI
+- *…or any other MCP client (Claude Desktop, Cursor, VS Code, Antigravity, opencode, Windsurf) — see [Use with other MCP clients](#use-with-other-mcp-clients)*
 - ~200MB disk for model cache (auto-downloaded on first run)
 - *Optional:* NVIDIA GPU + CUDA for accelerated embeddings (`pip install knowledge-rag[gpu]` + `models.embedding.gpu: true` in config)
 
@@ -446,6 +506,94 @@ Add to `~/.claude.json`:
 > Replace `YOUR_USER` with your username, or use the full path from `echo $HOME`.
 </details>
 
+#### Option F: SSE Server Mode (multi-agent)
+
+For multi-agent setups where multiple clients query the same knowledge base simultaneously:
+
+```bash
+pip install knowledge-rag[server]    # Adds uvicorn for SSE/HTTP
+knowledge-rag --transport sse        # Starts on http://127.0.0.1:8179
+```
+
+Then configure each MCP client to connect via SSE:
+
+```json
+{
+  "mcpServers": {
+    "knowledge-rag": {
+      "type": "sse",
+      "url": "http://127.0.0.1:8179/sse"
+    }
+  }
+}
+```
+
+One server process serves all agents — shared embedding model, shared cache, shared ChromaDB. See [Configuration > Server](#server) for rate limiting, metrics, and auth options.
+
+### Use with other MCP clients
+
+`knowledge-rag` supports both **stdio** (default, 1:1) and **SSE** (1:N) transport modes. In stdio mode, it works with any MCP-compatible client, not only Claude Code. The launch command is the same everywhere (the `python -m mcp_server.server` from whichever install method you picked); only the **config file location** and **JSON shape** differ per client.
+
+#### Clients using the standard `mcpServers` format
+
+For **Claude Desktop, Cursor, Antigravity, and Windsurf**, use the same block — only the file location changes:
+
+```json
+{
+  "mcpServers": {
+    "knowledge-rag": {
+      "command": "/home/YOUR_USER/knowledge-rag/venv/bin/python",
+      "args": ["-m", "mcp_server.server"]
+    }
+  }
+}
+```
+
+> **Windows**: set `command` to the full path of `venv\Scripts\python.exe`.
+
+| Client | Config file | Notes |
+|---|---|---|
+| **Claude Code** | use `claude mcp add …` (see install methods above) | The CLI writes `~/.claude.json` for you — manual edits to it aren't reliably picked up. |
+| **Claude Desktop** | macOS: `~/Library/Application Support/Claude/claude_desktop_config.json` · Windows: `%APPDATA%\Claude\claude_desktop_config.json` | Easiest: **Settings → Developer → Edit Config** opens the correct file (avoids the Windows Store/MSIX path quirk). |
+| **Cursor** | `~/.cursor/mcp.json` (global) or `.cursor/mcp.json` (per project) | — |
+| **Antigravity** | macOS/Linux: `~/.gemini/antigravity/mcp_config.json` · Windows: `%USERPROFILE%\.gemini\antigravity\mcp_config.json` | Open via Agent panel → **"…" → Manage MCP Servers → View raw config**. |
+| **Windsurf** | `~/.codeium/windsurf/mcp_config.json` (global only) | Easiest: Cascade panel → MCP → **View raw config**. |
+
+#### VS Code — uses a `servers` key
+
+VS Code (Copilot MCP) nests servers under **`servers`**, not `mcpServers`. Put this in `.vscode/mcp.json` (workspace) or the file opened by the **MCP: Open User Configuration** command:
+
+```json
+{
+  "servers": {
+    "knowledge-rag": {
+      "type": "stdio",
+      "command": "/home/YOUR_USER/knowledge-rag/venv/bin/python",
+      "args": ["-m", "mcp_server.server"]
+    }
+  }
+}
+```
+
+#### opencode — uses an `mcp` key
+
+opencode nests servers under **`mcp`**, takes `command` as a single **array**, and uses `environment` instead of `env`. Put this in `opencode.json` (project root) or `~/.config/opencode/opencode.json` (global):
+
+```jsonc
+{
+  "$schema": "https://opencode.ai/config.json",
+  "mcp": {
+    "knowledge-rag": {
+      "type": "local",
+      "command": ["/home/YOUR_USER/knowledge-rag/venv/bin/python", "-m", "mcp_server.server"],
+      "enabled": true
+    }
+  }
+}
+```
+
+> **Any other MCP client**: point it at the same command + args (`…/venv/bin/python -m mcp_server.server`). If it speaks stdio MCP, knowledge-rag works — only the config file's location and key naming differ. Check your client's docs for the exact path.
+
 ### Verify
 
 ```bash
@@ -554,6 +702,8 @@ Hybrid search combining semantic search + BM25 keyword search with cross-encoder
 | `max_results` | int | 5 | Maximum results to return (1-20) |
 | `category` | string | null | Filter by category |
 | `hybrid_alpha` | float | 0.3 | Balance: 0.0 = keyword only, 1.0 = semantic only |
+| `min_score` | float | 0.0 | Minimum relevance score (0.0-1.0) to include a result. Use 0.2-0.4 to cut noise |
+| `snippet_mode` | bool | true | Truncate content to ~500 chars at natural break points. Adds `content_length` field |
 
 **Returns:**
 
@@ -563,6 +713,7 @@ Hybrid search combining semantic search + BM25 keyword search with cross-encoder
   "query": "mimikatz credential dump",
   "hybrid_alpha": 0.5,
   "result_count": 3,
+  "filtered_by_score": 2,
   "cache_hit_rate": "0.0%",
   "results": [
     {
@@ -660,7 +811,6 @@ Get statistics about the knowledge base index.
   "stats": {
     "total_documents": 75,
     "total_chunks": 9256,
-    "unique_content_hashes": 9100,
     "categories": {"security": 52, "development": 8},
     "supported_formats": [".md", ".txt", ".pdf", ".py", ".json", ".docx", ".xlsx", ".pptx", ".csv", ".ipynb"],
     "embedding_model": "BAAI/bge-small-en-v1.5",
@@ -842,6 +992,21 @@ query_expansions:
   privesc:
     - privilege escalation
     - privesc
+
+# Server — enterprise features (new in v4.0.0)
+server:
+  transport: "stdio"              # "stdio" | "sse" | "streamable-http"
+  host: "127.0.0.1"              # Bind address (SSE/HTTP only)
+  port: 8179                      # Bind port (SSE/HTTP only)
+  auth:
+    bearer_token: ""              # Set a secret to enable auth (SSE/HTTP only)
+  rate_limit:
+    enabled: false
+    requests_per_minute: 60
+    burst: 10
+  metrics:
+    enabled: false
+    port: 9179                    # Separate port for Prometheus scraping
 ```
 
 > See `config.example.yaml` for the fully documented template with explanations for every field.
@@ -861,6 +1026,22 @@ Pre-built configurations for common use cases:
 
 ### Configuration Reference
 
+#### Server
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `server.transport` | `"stdio"` | Transport protocol: `"stdio"`, `"sse"`, or `"streamable-http"` |
+| `server.host` | `"127.0.0.1"` | Bind address for SSE/HTTP mode |
+| `server.port` | `8179` | Bind port for SSE/HTTP mode |
+| `server.auth.bearer_token` | `""` (disabled) | Bearer token for SSE/HTTP auth. Empty = no auth |
+| `server.rate_limit.enabled` | `false` | Enable per-client rate limiting |
+| `server.rate_limit.requests_per_minute` | `60` | Max requests per minute |
+| `server.rate_limit.burst` | `10` | Burst allowance above steady rate |
+| `server.metrics.enabled` | `false` | Enable Prometheus `/metrics` endpoint |
+| `server.metrics.port` | `9179` | Port for metrics scraping |
+
+In stdio mode (default), server settings are ignored. SSE/HTTP mode auto-enables the single-instance lock.
+
 #### Paths
 
 | Field | Default | Description |
@@ -957,6 +1138,22 @@ query_expansions:
 
 Set `query_expansions: {}` for no expansion.
 
+`query_expansions` is directional: only the key on the left triggers the terms on the right. If you need mutual expansion without duplicating entries, use `query_expansion_groups`.
+
+```yaml
+query_expansion_groups:
+  - ["triple barrier", "tb", "trip_barr"]
+  - ["profit factor", "pf"]
+```
+
+Each group is interpreted symmetrically, so every term expands to the rest of the group. The final internal expansion table is built by merging both sources:
+
+1. `query_expansions` entries are loaded as-is.
+2. `query_expansion_groups` adds reciprocal links for every term in each group.
+3. Overlaps are merged by union with duplicate terms removed.
+
+This keeps backward compatibility while allowing concise synonym groups.
+
 ### Hybrid Search Tuning
 
 | hybrid_alpha | Behavior | Best For |
@@ -1098,10 +1295,59 @@ export KNOWLEDGE_RAG_SINGLE_INSTANCE=1
 
 A second instance exits immediately with code 75. Default is OFF (multi-client friendly). Full guide: [docs/single-instance.md](docs/single-instance.md). Sample MCP config: [examples/mcp-config-single-instance.json](examples/mcp-config-single-instance.json).
 
+### SSE server won't start
+
+```bash
+# Check if port 8179 is already in use
+# Windows:
+netstat -aon | findstr :8179
+# Linux/macOS:
+lsof -i :8179
+```
+
+If `uvicorn` is not found, install the server extras: `pip install knowledge-rag[server]`
+
+### Can't connect to SSE server
+
+Verify the server is running and the URL is correct:
+
+```bash
+curl http://127.0.0.1:8179/sse
+```
+
+Common issues:
+- Wrong URL: must end with `/sse` (not just the port)
+- Firewall blocking the port
+- Server started with a different host/port than configured in the MCP client
+
 ---
 
 ## Changelog
 
+### v4.0.0 (2026-06-09) — Enterprise Concurrent Access
+
+- **NEW**: SSE and streamable-http transport modes — 1 server serves N clients (`server.transport: "sse"` in config.yaml or `--transport sse` CLI).
+- **NEW**: Thread-safe shared state for concurrent queries — QueryCache locking, BM25 build lock, orchestrator double-checked locking.
+- **NEW**: ChromaDB WAL mode enabled automatically in SSE/HTTP mode for concurrent read performance.
+- **NEW**: Optional rate limiting — sliding-window counter, configurable RPM and burst, disabled by default.
+- **NEW**: Optional Prometheus metrics endpoint — tool call counts, latency histograms, separate port, disabled by default.
+- **NEW**: All 12 MCP tools instrumented with `@rate_limited` and `@instrument` decorators (zero-cost when disabled).
+- **NEW**: `--transport` CLI override for Docker/systemd deployments.
+- **NEW**: `pip install knowledge-rag[server]` optional dependency for SSE/HTTP (uvicorn).
+- **CHANGED**: SSE/HTTP mode auto-enables single-instance lock (port collision prevention).
+- **CHANGED**: `mcp` dependency bumped to `>=1.6.0` (SSE/streamable-http support).
+- **MIGRATION**: Default transport remains `stdio` — existing users need zero changes. See config.example.yaml for SSE setup.
+
+### v3.9.1 (2026-06-08)
+
+- **FIX**: Expand `~` in `config.yaml` path values (`documents_dir`, `data_dir`, `models_cache_dir`) via `expanduser()` on all platforms (#86).
+- **FIX**: Warn when `documents_dir` resolves to a non-existent path instead of silently indexing zero files.
+- **FIX**: File watcher now uses accumulate-mode debounce — bulk file copies no longer starve the reindex trigger.
+- **FIX**: Concurrent `index_all()` calls are serialized via `_index_lock` to prevent ChromaDB SQLite corruption.
+- **FIX**: `collection.add()` is batched (500 chunks/call) to cap memory usage during large reindex operations.
+- **NEW**: `KNOWLEDGE_RAG_WATCHER_DISABLED=1` env var to disable the file watcher for troubleshooting.
+- **NEW**: Progress logging every 10% for reindex operations with >100 documents.
+
 ### v3.9.0 (2026-05-10) — Quality Gate
 
 **Major governance + CI hardening release. No runtime behavior change in `mcp_server/`. Public API surface unchanged from v3.8.1.**
@@ -1145,6 +1391,38 @@ A second instance exits immediately with code 75. Default is OFF (multi-client f
 
 ### Unreleased
 
+### v4.2.0 (2026-06-17) — Search Performance & Output Quality
+
+- **PERF**: Custom inverted-index BM25 replaces `rank-bm25` full-corpus scan — 128× faster keyword search on 50K+ chunk corpora. Only documents containing query terms are scored via posting lists.
+- **PERF**: `numpy.argpartition` for O(n) top-k selection instead of O(n log n) sort.
+- **PERF**: Batched adjacent chunk fetch — single ChromaDB `collection.get()` call replaces N round-trips per result.
+- **PERF**: O(1) reverse lookup via `_source_to_docid` dict eliminates linear scans of `_indexed_docs` in `search_similar`, `update_document`, `remove_document`, and `_expand_with_adjacent_chunks`.
+- **NEW**: `snippet_mode` parameter on `search_knowledge` (default: `true`) — truncates content to ~500 chars at natural break points with `content_length` field. Reduces token consumption by ~72%.
+- **NEW**: `min_score` parameter on `search_knowledge` (default: `0.0`) — filters results below a normalized relevance threshold. Response includes `filtered_by_score` count.
+- **NEW**: `filtered_by_score` field in search response JSON for transparency.
+- **DEPS**: `numpy` added as direct dependency (was transitive via fastembed); `rank-bm25` import removed from server.py.
+- **TEST**: 6 new tests for `min_score` filtering and `snippet_mode` truncation.
+- **TEST**: Updated backwards-compat baseline to include new `search_knowledge` parameters.
+
+### v4.1.2 (2026-06-17)
+
+- **FIX**: `_save_metadata` dict snapshot prevents concurrent modification crash during file watcher events.
+- **STYLE**: ruff format applied to server.py.
+
+### v4.1.1 (2026-06-17)
+
+- **FIX**: All `_indexed_docs` iterations now use `list()` snapshot, preventing `dictionary changed size during iteration` crash when FileWatcher modifies the index concurrently with MCP tool calls (affects `search_knowledge`, `search_similar`, `update_document`, `remove_document`, `evaluate_retrieval`, `list_categories`, `list_documents`)
+
+### v4.1.0 (2026-06-17)
+
+- **Added:** `query_expansion_groups` config for symmetric synonym expansion (#92)
+- **Improved:** `expand_query()` now returns deterministic expansion order (set → ordered list with dedup)
+
+### v4.0.1 (2026-06-16)
+
+- **FIX**: Orphan cleanup now runs before indexing loop, preventing chunk loss when files are moved (#90).
+- **FIX**: Chunk deduplication is now per-document instead of global, preventing cross-document chunk deletion (#91).
+- **FIX**: Added `on_moved` handler to `DocumentWatcher` for proper file move detection.
 - **FIX**: Startup preflight probes ChromaDB in a child process and moves crashing persistent indexes to `data/backups/auto-repair-*` before MCP initialization.
 - **FIX**: Reranker load failures now fall back to RRF ordering instead of failing `search_knowledge` on offline machines.
 - **FIX**: Virtualenv project-root detection now handles Python symlinks that resolve to the system interpreter.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "knowledge-rag",
-  "version": "3.9.0",
+  "version": "4.2.0",
   "description": "Local RAG System for Claude Code — Hybrid search + Cross-encoder Reranking + 12 MCP Tools + 20 Format Parsers. Zero external servers.",
   "bin": {
     "knowledge-rag": "./bin/cli.js"