knowledge-rag 3.6.1 → 3.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +1165 -29
  2. package/package.json +4 -1
package/README.md CHANGED
@@ -1,61 +1,1197 @@
1
- [![npm version](https://img.shields.io/npm/v/knowledge-rag.svg)](https://www.npmjs.com/package/knowledge-rag)
2
-
3
1
  # Knowledge RAG
4
2
 
5
- Local RAG system for Claude Code. Hybrid BM25 + semantic search with cross-encoder reranking. 12 MCP tools. Zero external servers. Everything runs on your machine.
3
+ <div align="center">
4
+
5
+ [![PyPI](https://img.shields.io/pypi/v/knowledge-rag)](https://pypi.org/project/knowledge-rag/)
6
+ [![NPM](https://img.shields.io/npm/v/knowledge-rag)](https://www.npmjs.com/package/knowledge-rag)
7
+ [![Downloads](https://static.pepy.tech/badge/knowledge-rag/month)](https://pepy.tech/project/knowledge-rag)
8
+ ![Python](https://img.shields.io/badge/python-3.11%2B-green.svg)
9
+ ![License](https://img.shields.io/badge/license-MIT-yellow.svg)
10
+ ![Platform](https://img.shields.io/badge/platform-Windows%20%7C%20Linux%20%7C%20macOS-lightgrey.svg)
11
+ ![GPU](https://img.shields.io/badge/GPU-NVIDIA%20CUDA-76B900.svg?logo=nvidia)
12
+ [![CI](https://github.com/lyonzin/knowledge-rag/actions/workflows/ci.yml/badge.svg)](https://github.com/lyonzin/knowledge-rag/actions/workflows/ci.yml)
13
+ [![CodeQL](https://github.com/lyonzin/knowledge-rag/actions/workflows/security.yml/badge.svg)](https://github.com/lyonzin/knowledge-rag/actions/workflows/security.yml)
14
+ [![Glama Score](https://glama.ai/mcp/servers/lyonzin/knowledge-rag/badges/score.svg)](https://glama.ai/mcp/servers/lyonzin/knowledge-rag)
15
+
16
+ ### Your docs, your machine, zero cloud. Claude Code searches them natively.
17
+
18
+ Drop your PDFs, markdown, code, notebooks — **1800+ files, 39K chunks, indexed in under 3 minutes.**<br/>
19
+ Hybrid search (BM25 + semantic vectors + cross-encoder reranking) through 12 MCP tools.<br/>
20
+ Everything runs locally via ONNX. No Docker, no Ollama, no API keys, no data leaves your machine.
21
+
22
+ ```
23
+ pip install knowledge-rag → restart Claude Code → search_knowledge("your query")
24
+ ```
25
+
26
+ ---
27
+
28
+ **12 MCP Tools** | **Hybrid Search + Reranking** | **20 File Formats** | **Optional NVIDIA GPU** | **100% Local**
29
+
30
+ [What's New](#whats-new-in-v360) | [Supported Formats](#supported-formats) | [Installation](#installation) | [Configuration](#configuration) | [API Reference](#api-reference) | [Architecture](#architecture)
31
+
32
+ </div>
33
+
34
+ ---
35
+
36
+ ## What's New in v3.6.0
37
+
38
+ ### Multi-Language Code Parsing
39
+
40
+ Language-aware extraction for **C**, **C++**, **JavaScript**, **TypeScript**, and **XML** — functions, classes, structs, interfaces, imports, and namespaces are captured as searchable metadata. Total supported formats: **20**.
41
+
42
+ ### 5 Ways to Install
43
+
44
+ ```bash
45
+ npx -y knowledge-rag # NPM — zero setup, auto-manages Python venv
46
+ pip install knowledge-rag # PyPI — classic Python install
47
+ curl -fsSL .../install.sh | bash # One-line installer (Linux/macOS/Windows)
48
+ docker pull ghcr.io/lyonzin/knowledge-rag # Docker — models pre-downloaded
49
+ git clone ... && pip install -r ... # From source
50
+ ```
51
+
52
+ All methods produce the same MCP server. See [Installation](#installation) for full instructions.
53
+
54
+ ### Recent Highlights
55
+
56
+ - **v3.6.0** — Multi-language code parsing (C/C++/JS/TS/XML), NPM wrapper, Docker image, automated release pipeline
57
+ - **v3.5.2** — CUDA DLL auto-discovery from pip packages, graceful GPU→CPU fallback, explicit CPU provider (no CUDA noise when `gpu: false`), BASE_DIR resolution fix for editable installs
58
+ - **v3.5.1** — Remove Python `<3.13` upper bound — 3.13 and 3.14 now supported
59
+ - **v3.5.0** — Optional GPU acceleration, supported formats table, full README rewrite
60
+ - **v3.4.3** — MCP stdout save/restore fix (v3.4.2 broke JSON-RPC responses)
61
+ - **v3.4.0** — Persistent model cache, exclude patterns, Jupyter Notebook parser, inotify resilience, MetaTrader support
62
+
63
+ See [Changelog](#changelog) for full history.
64
+
65
+ ---
66
+
67
+ ## Supported Formats
68
+
69
+ | Format | Extension | Parser | Default | Notes |
70
+ |--------|-----------|--------|---------|-------|
71
+ | Markdown | `.md` | Section-aware (splits at `##`) | Yes | Headers preserved as chunk boundaries |
72
+ | Plain Text | `.txt` | Fixed-size chunking | Yes | 1000 chars + 200 overlap |
73
+ | PDF | `.pdf` | PyMuPDF extraction | Yes | Text-based PDFs only (no OCR) |
74
+ | Python | `.py` | Code-aware parser | Yes | Functions/classes as chunks |
75
+ | JSON | `.json` | Structure-aware | Yes | Flattened key-value extraction |
76
+ | CSV | `.csv` | Row-based parser | Yes | Headers + rows as text |
77
+ | Word | `.docx` | python-docx | Yes | Headings preserved as markdown |
78
+ | Excel | `.xlsx` | openpyxl | Yes | Sheet-by-sheet extraction |
79
+ | PowerPoint | `.pptx` | python-pptx | Yes | Slide-by-slide extraction |
80
+ | Jupyter Notebook | `.ipynb` | Cell-aware parser | Yes | Markdown + code cells only, no outputs/base64 |
81
+ | C Source | `.c` | Code-aware parser | Yes | Functions/structs/includes extracted |
82
+ | C/C++ Header | `.h` | Code-aware parser | Yes | Function declarations/structs extracted |
83
+ | C++ Source | `.cpp` | Code-aware parser | Yes | Classes/structs/includes extracted |
84
+ | JavaScript | `.js` | Code-aware parser | Yes | Functions/classes/imports (ESM + CJS) |
85
+ | React JSX | `.jsx` | Code-aware parser | Yes | Same as JS parser |
86
+ | TypeScript | `.ts` | Code-aware parser | Yes | Functions/classes/interfaces/enums/imports |
87
+ | React TSX | `.tsx` | Code-aware parser | Yes | Same as TS parser |
88
+ | XML | `.xml` | XML parser | Yes | Root element and namespace extraction |
89
+ | MQL4 Header | `.mqh` | Code parser | No | MetaTrader — add to `supported_formats` to enable |
90
+ | MQL4 Source | `.mq4` | Code parser | No | MetaTrader — add to `supported_formats` to enable |
91
+
92
+ > **Tip:** The parser dispatch is extensible. Any format mapped in `_parsers` can be enabled via `supported_formats` in config.yaml.
93
+
94
+ ---
95
+
96
+ ## Features
97
+
98
+ | Feature | Description |
99
+ |---------|-------------|
100
+ | **Hybrid Search** | Semantic + BM25 keyword search with Reciprocal Rank Fusion |
101
+ | **Cross-Encoder Reranker** | Xenova/ms-marco-MiniLM-L-6-v2 re-scores top candidates for precision |
102
+ | **GPU Acceleration** | Optional ONNX CUDA support for 5-10x faster indexing |
103
+ | **YAML Configuration** | Fully customizable via `config.yaml` with domain-specific presets |
104
+ | **Query Expansion** | Configurable synonym mappings (69 security-term defaults) |
105
+ | **Markdown-Aware Chunking** | `.md` files split by `##`/`###` sections instead of fixed windows |
106
+ | **In-Process Embeddings** | FastEmbed ONNX Runtime (BAAI/bge-small-en-v1.5, 384D) |
107
+ | **Keyword Routing** | Word-boundary aware routing for domain-specific queries |
108
+ | **20 Format Parsers** | MD, TXT, PDF, PY, C, H, CPP, JS, JSX, TS, TSX, JSON, XML, CSV, DOCX, XLSX, PPTX, IPYNB + opt-in MQH/MQ4 |
109
+ | **Category Organization** | Organize docs by folder, auto-tagged by path |
110
+ | **Incremental Indexing** | Change detection via mtime/size — only re-indexes modified files |
111
+ | **Chunk Deduplication** | SHA256 content hashing prevents duplicate chunks |
112
+ | **Query Cache** | LRU cache with 5-min TTL for instant repeat queries |
113
+ | **Document CRUD** | Add, update, remove documents via MCP tools |
114
+ | **URL Ingestion** | Fetch URLs, strip HTML, convert to markdown, index |
115
+ | **Similarity Search** | Find documents similar to a reference document |
116
+ | **Retrieval Evaluation** | Built-in MRR@5 and Recall@5 metrics |
117
+ | **File Watcher** | Auto-reindex on document changes via watchdog (5s debounce) |
118
+ | **Exclude Patterns** | Glob-based file/directory exclusion during indexing |
119
+ | **MMR Diversification** | Maximal Marginal Relevance reduces redundant results |
120
+ | **Persistent Model Cache** | Embedding models cached in `models_cache/` — survives reboots |
121
+ | **Auto-Migration** | Detects embedding dimension mismatch and rebuilds automatically |
122
+ | **12 MCP Tools** | Full CRUD + search + evaluation via Claude Code |
123
+
124
+ ---
125
+
126
+ ## Architecture
127
+
128
+ ### System Overview
129
+
130
+ ```mermaid
131
+ flowchart TB
132
+ subgraph MCP["MCP SERVER (FastMCP)"]
133
+ direction TB
134
+ TOOLS["12 MCP Tools<br/>search | get | add | update | remove<br/>reindex | list | stats | url | similar | evaluate"]
135
+ end
136
+
137
+ subgraph SEARCH["HYBRID SEARCH ENGINE"]
138
+ direction LR
139
+ ROUTER["Keyword Router<br/>(word boundaries)"]
140
+ SEMANTIC["Semantic Search<br/>(ChromaDB)"]
141
+ BM25["BM25 Keyword<br/>(rank-bm25 + expansion)"]
142
+ RRF["Reciprocal Rank<br/>Fusion (RRF)"]
143
+ RERANK["Cross-Encoder<br/>Reranker"]
144
+
145
+ ROUTER --> SEMANTIC
146
+ ROUTER --> BM25
147
+ SEMANTIC --> RRF
148
+ BM25 --> RRF
149
+ RRF --> RERANK
150
+ end
151
+
152
+ subgraph STORAGE["STORAGE LAYER"]
153
+ direction LR
154
+ CHROMA[("ChromaDB<br/>Vector Database")]
155
+ COLLECTIONS["Collections<br/>security | ctf<br/>logscale | development"]
156
+ CHROMA --- COLLECTIONS
157
+ end
158
+
159
+ subgraph EMBED["EMBEDDINGS (In-Process)"]
160
+ FASTEMBED["FastEmbed ONNX<br/>BAAI/bge-small-en-v1.5<br/>(384D, CPU or GPU)"]
161
+ CROSSENC["Cross-Encoder<br/>ms-marco-MiniLM-L-6-v2"]
162
+ FASTEMBED --- CROSSENC
163
+ end
164
+
165
+ subgraph INGEST["DOCUMENT INGESTION"]
166
+ PARSERS["20 Parsers<br/>MD | PDF | TXT | PY | C | H | CPP | JS | JSX | TS | TSX | JSON | XML | CSV<br/>DOCX | XLSX | PPTX | IPYNB | MQH | MQ4"]
167
+ CHUNKER["Chunking<br/>MD: section-aware<br/>Other: 1000 chars + 200 overlap"]
168
+ PARSERS --> CHUNKER
169
+ end
170
+
171
+ CLAUDE["Claude Code"] --> MCP
172
+ MCP --> SEARCH
173
+ SEARCH --> STORAGE
174
+ STORAGE --> EMBED
175
+ INGEST --> EMBED
176
+ EMBED --> STORAGE
177
+ ```
178
+
179
+ ### Query Processing Flow
180
+
181
+ ```mermaid
182
+ flowchart TB
183
+ QUERY["User Query<br/>'mimikatz credential dump'"] --> EXPAND
184
+
185
+ subgraph EXPANSION["Query Expansion"]
186
+ EXPAND["Synonym Expansion<br/>mimikatz -> mimikatz, sekurlsa, logonpasswords"]
187
+ end
188
+
189
+ EXPAND --> ROUTER
190
+
191
+ subgraph ROUTING["Keyword Routing"]
192
+ ROUTER["Keyword Router"]
193
+ MATCH{"Word Boundary<br/>Match?"}
194
+ CATEGORY["Filter: redteam"]
195
+ NOFILTER["No Filter"]
196
+
197
+ ROUTER --> MATCH
198
+ MATCH -->|Yes| CATEGORY
199
+ MATCH -->|No| NOFILTER
200
+ end
201
+
202
+ subgraph HYBRID["Hybrid Search"]
203
+ direction LR
204
+ SEMANTIC["Semantic Search<br/>(ChromaDB embeddings)<br/>Conceptual similarity"]
205
+ BM25["BM25 Search<br/>(expanded query)<br/>Exact term matching"]
206
+ end
207
+
208
+ subgraph FUSION["Result Fusion + Reranking"]
209
+ RRF["Reciprocal Rank Fusion<br/>score = alpha * 1/(k+rank_sem)<br/>+ (1-alpha) * 1/(k+rank_bm25)"]
210
+ RERANK["Cross-Encoder Reranker<br/>Re-scores top 3x candidates<br/>query+doc pair scoring"]
211
+ SORT["Sort by Reranker Score<br/>Normalize to 0-1"]
212
+
213
+ RRF --> RERANK --> SORT
214
+ end
215
+
216
+ CATEGORY --> HYBRID
217
+ NOFILTER --> HYBRID
218
+ SEMANTIC --> RRF
219
+ BM25 --> RRF
220
+
221
+ SORT --> RESULTS["Results<br/>search_method: hybrid|semantic|keyword<br/>score + reranker_score + raw_rrf_score"]
222
+ ```
223
+
224
+ ### Document Ingestion Flow
225
+
226
+ ```mermaid
227
+ flowchart LR
228
+ subgraph INPUT["Input"]
229
+ FILES["documents/<br/>├── security/<br/>├── development/<br/>├── ctf/<br/>└── general/"]
230
+ end
231
+
232
+ subgraph PARSE["Parse (20 formats)"]
233
+ MD["Markdown"]
234
+ PDF["PDF<br/>(PyMuPDF)"]
235
+ OFFICE["DOCX | XLSX<br/>PPTX | CSV"]
236
+ CODE["PY | C | H | CPP | JS | JSX<br/>TS | TSX | JSON | XML | IPYNB"]
237
+ end
238
+
239
+ subgraph CHUNK["Chunk"]
240
+ MDSPLIT["MD: Section-Aware<br/>Split at ## headers"]
241
+ TXTSPLIT["Other: Fixed-Size<br/>1000 chars + 200 overlap"]
242
+ DEDUP["SHA256 Dedup<br/>Skip duplicate content"]
243
+ end
244
+
245
+ subgraph EMBED["Embed"]
246
+ FASTEMBED["FastEmbed ONNX<br/>bge-small-en-v1.5<br/>(384D, CPU or GPU)"]
247
+ end
248
+
249
+ subgraph STORE["Store"]
250
+ CHROMADB[("ChromaDB")]
251
+ BM25IDX["BM25 Index"]
252
+ end
253
+
254
+ FILES --> MD & PDF & OFFICE & CODE
255
+ MD --> MDSPLIT
256
+ PDF & OFFICE & CODE --> TXTSPLIT
257
+ MDSPLIT --> DEDUP
258
+ TXTSPLIT --> DEDUP
259
+ DEDUP --> EMBED
260
+ EMBED --> STORE
261
+ ```
262
+
263
+ ### hybrid_alpha Parameter Effect
264
+
265
+ ```mermaid
266
+ flowchart LR
267
+ subgraph ALPHA["hybrid_alpha values"]
268
+ A0["0.0<br/>Pure BM25<br/>Instant"]
269
+ A3["0.3 (default)<br/>Keyword-heavy<br/>Fast"]
270
+ A5["0.5<br/>Balanced"]
271
+ A7["0.7<br/>Semantic-heavy"]
272
+ A10["1.0<br/>Pure Semantic"]
273
+ end
274
+
275
+ subgraph USE["Best For"]
276
+ U0["CVEs, tool names<br/>exact matches"]
277
+ U3["Technical queries<br/>specific terms"]
278
+ U5["General queries"]
279
+ U7["Conceptual queries<br/>related topics"]
280
+ U10["'How to...' questions<br/>conceptual search"]
281
+ end
282
+
283
+ A0 --- U0
284
+ A3 --- U3
285
+ A5 --- U5
286
+ A7 --- U7
287
+ A10 --- U10
288
+ ```
289
+
290
+ ---
291
+
292
+ ## Installation
293
+
294
+ ### Prerequisites
295
+
296
+ - Python 3.11+
297
+ - Claude Code CLI
298
+ - ~200MB disk for model cache (auto-downloaded on first run)
299
+ - *Optional:* NVIDIA GPU + CUDA for accelerated embeddings (`pip install knowledge-rag[gpu]` + `models.embedding.gpu: true` in config)
300
+
301
+ ### Install Methods
302
+
303
+ Pick one — all produce the same running server.
304
+
305
+ #### Option A: NPX (fastest)
306
+
307
+ Requires Node.js 16+. Handles Python venv, pip install, and version upgrades automatically.
308
+
309
+ ```bash
310
+ claude mcp add knowledge-rag -s user -- npx -y knowledge-rag
311
+ ```
312
+
313
+ That's it. On first run, `npx` creates a venv at `~/.knowledge-rag/`, installs the PyPI package, and starts the MCP server. Subsequent runs reuse the cached venv.
314
+
315
+ #### Option B: One-line installer
316
+
317
+ ```bash
318
+ # Linux/macOS:
319
+ curl -fsSL https://raw.githubusercontent.com/lyonzin/knowledge-rag/master/install.sh | bash
320
+
321
+ # Windows (PowerShell):
322
+ irm https://raw.githubusercontent.com/lyonzin/knowledge-rag/master/install.ps1 | iex
323
+ ```
324
+
325
+ Then configure Claude Code:
326
+
327
+ ```bash
328
+ claude mcp add knowledge-rag -s user -- ~/knowledge-rag/venv/bin/python -m mcp_server.server
329
+ ```
330
+
331
+ > **Windows**: `claude mcp add knowledge-rag -s user -- %USERPROFILE%\knowledge-rag\venv\Scripts\python.exe -m mcp_server.server`
332
+
333
+ #### Option C: pip install
334
+
335
+ ```bash
336
+ mkdir ~/knowledge-rag && cd ~/knowledge-rag
337
+ python3 -m venv venv && source venv/bin/activate
338
+ pip install knowledge-rag
339
+ knowledge-rag init # Exports config template, presets, creates documents/
340
+ ```
341
+
342
+ Then configure Claude Code:
343
+
344
+ ```bash
345
+ claude mcp add knowledge-rag -s user -- ~/knowledge-rag/venv/bin/python -m mcp_server.server
346
+ ```
347
+
348
+ > **Windows users**: Use `python` instead of `python3`, `venv\Scripts\activate` instead of `source venv/bin/activate`.
349
+ > **Windows path**: `claude mcp add knowledge-rag -s user -- %USERPROFILE%\knowledge-rag\venv\Scripts\python.exe -m mcp_server.server`
6
350
 
7
- ## Quick Start
351
+ #### Option D: Clone from source
8
352
 
9
353
  ```bash
10
- npx knowledge-rag
354
+ git clone https://github.com/lyonzin/knowledge-rag.git ~/knowledge-rag
355
+ cd ~/knowledge-rag
356
+ python3 -m venv venv && source venv/bin/activate
357
+ pip install -r requirements.txt
11
358
  ```
12
359
 
13
- The wrapper handles Python venv creation, package installation, and server startup automatically.
360
+ Then configure Claude Code:
14
361
 
15
- ## Claude Code Configuration
362
+ ```bash
363
+ claude mcp add knowledge-rag -s user -- ~/knowledge-rag/venv/bin/python -m mcp_server.server
364
+ ```
365
+
366
+ #### Option E: Docker
367
+
368
+ ```bash
369
+ docker pull ghcr.io/lyonzin/knowledge-rag:latest
370
+ ```
16
371
 
17
- Add to your MCP settings (`~/.claude.json` or project `.mcp.json`):
372
+ ```bash
373
+ claude mcp add knowledge-rag -s user -- \
374
+ docker run -i --rm \
375
+ -v ~/knowledge-rag/documents:/app/documents \
376
+ -v ~/knowledge-rag/data:/app/data \
377
+ ghcr.io/lyonzin/knowledge-rag:latest
378
+ ```
379
+
380
+ Models are pre-downloaded in the image — no first-run delay.
381
+
382
+ <details>
383
+ <summary>Alternative: manual JSON config</summary>
384
+
385
+ Add to `~/.claude.json`:
386
+
387
+ **Windows:**
388
+ ```json
389
+ {
390
+ "mcpServers": {
391
+ "knowledge-rag": {
392
+ "command": "C:\\Users\\YOUR_USER\\knowledge-rag\\venv\\Scripts\\python.exe",
393
+ "args": ["-m", "mcp_server.server"]
394
+ }
395
+ }
396
+ }
397
+ ```
18
398
 
399
+ **Linux / macOS:**
19
400
  ```json
20
401
  {
21
402
  "mcpServers": {
22
403
  "knowledge-rag": {
23
- "command": "npx",
24
- "args": ["-y", "knowledge-rag"]
404
+ "command": "/home/YOUR_USER/knowledge-rag/venv/bin/python",
405
+ "args": ["-m", "mcp_server.server"]
25
406
  }
26
407
  }
27
408
  }
28
409
  ```
410
+ > Replace `YOUR_USER` with your username, or use the full path from `echo $HOME`.
411
+ </details>
412
+
413
+ ### Verify
414
+
415
+ ```bash
416
+ claude mcp list
417
+ ```
418
+
419
+ On first start, the server will:
420
+ 1. Download the embedding model (~50MB, cached in `models_cache/`)
421
+ 2. Auto-index any documents in the `documents/` directory
422
+ 3. Start watching for file changes (auto-reindex)
423
+
424
+ ---
29
425
 
30
- ## CLI Flags
426
+ ## Usage
31
427
 
32
- | Flag | Description |
33
- |------|-------------|
34
- | `--version` | Print version and exit |
35
- | `--install-only` | Install the Python package into the venv without starting the server |
428
+ ### Adding Documents
36
429
 
37
- ## Requirements
430
+ Place your documents in the `documents/` directory, organized by category:
38
431
 
39
- - **Node.js** >= 16
40
- - **Python** >= 3.11
432
+ ```
433
+ documents/
434
+ ├── security/ # Pentest, exploit, vulnerability docs
435
+ ├── development/ # Code, APIs, frameworks
436
+ ├── ctf/ # CTF writeups and methodology
437
+ ├── logscale/ # LogScale/LQL documentation
438
+ └── general/ # Everything else
439
+ ```
41
440
 
42
- The wrapper auto-detects your Python installation across platforms (Windows: `python`, `py -3`; Linux/macOS: `python3`, `python`).
441
+ Or add documents programmatically via MCP tools:
43
442
 
44
- ## How It Works
443
+ ```python
444
+ # Add from content
445
+ add_document(
446
+ content="# My Document\n\nContent here...",
447
+ filepath="security/my-technique.md",
448
+ category="security"
449
+ )
45
450
 
46
- 1. Finds a compatible Python 3.11+ interpreter on your system
47
- 2. Creates a persistent virtual environment at `~/.knowledge-rag/venv`
48
- 3. Installs the `knowledge-rag` PyPI package (skips if already up to date)
49
- 4. Starts the MCP server over stdio
451
+ # Add from URL
452
+ add_from_url(
453
+ url="https://example.com/article",
454
+ category="security",
455
+ title="Custom Title"
456
+ )
457
+ ```
50
458
 
51
- The venv and installed package persist between runs. Reinstallation only happens when the NPM package version changes.
459
+ ### Searching
52
460
 
53
- ## Full Documentation
461
+ Claude uses the RAG system automatically when configured. You can also control search behavior:
54
462
 
55
- See the main repository for complete docs, configuration options, and tool reference:
463
+ ```python
464
+ # Pure keyword search — instant, no embedding needed
465
+ search_knowledge("gtfobins suid", hybrid_alpha=0.0)
466
+
467
+ # Keyword-heavy (default) — fast, slight semantic boost
468
+ search_knowledge("mimikatz", hybrid_alpha=0.3)
469
+
470
+ # Balanced hybrid — both engines equally weighted
471
+ search_knowledge("SQL injection techniques", hybrid_alpha=0.5)
472
+
473
+ # Semantic-heavy — better for conceptual queries
474
+ search_knowledge("how to escalate privileges", hybrid_alpha=0.7)
475
+
476
+ # Pure semantic — embedding similarity only
477
+ search_knowledge("lateral movement strategies", hybrid_alpha=1.0)
478
+ ```
479
+
480
+ ### Indexing
481
+
482
+ Documents are automatically indexed on first startup. To manage the index:
483
+
484
+ ```python
485
+ # Incremental: only re-index changed files (fast)
486
+ reindex_documents()
487
+
488
+ # Smart reindex: detect changes + rebuild BM25
489
+ reindex_documents(force=True)
490
+
491
+ # Nuclear rebuild: delete everything, re-embed all (use after model change)
492
+ reindex_documents(full_rebuild=True)
493
+ ```
56
494
 
57
- **https://github.com/lyonzin/knowledge-rag**
495
+ ### Evaluating Retrieval Quality
496
+
497
+ ```python
498
+ evaluate_retrieval(test_cases='[
499
+ {"query": "sql injection", "expected_filepath": "security/sqli-guide.md"},
500
+ {"query": "privilege escalation", "expected_filepath": "security/privesc.md"}
501
+ ]')
502
+ # Returns: MRR@5, Recall@5, per-query results
503
+ ```
504
+
505
+ ---
506
+
507
+ ## API Reference
508
+
509
+ ### Search & Query
510
+
511
+ #### `search_knowledge`
512
+
513
+ Hybrid search combining semantic search + BM25 keyword search with cross-encoder reranking.
514
+
515
+ | Parameter | Type | Default | Description |
516
+ |-----------|------|---------|-------------|
517
+ | `query` | string | required | Search query text (1-3 keywords recommended) |
518
+ | `max_results` | int | 5 | Maximum results to return (1-20) |
519
+ | `category` | string | null | Filter by category |
520
+ | `hybrid_alpha` | float | 0.3 | Balance: 0.0 = keyword only, 1.0 = semantic only |
521
+
522
+ **Returns:**
523
+
524
+ ```json
525
+ {
526
+ "status": "success",
527
+ "query": "mimikatz credential dump",
528
+ "hybrid_alpha": 0.5,
529
+ "result_count": 3,
530
+ "cache_hit_rate": "0.0%",
531
+ "results": [
532
+ {
533
+ "content": "Mimikatz can extract credentials from memory...",
534
+ "source": "documents/security/credential-attacks.md",
535
+ "filename": "credential-attacks.md",
536
+ "category": "security",
537
+ "score": 0.9823,
538
+ "raw_rrf_score": 0.016393,
539
+ "reranker_score": 0.987654,
540
+ "semantic_rank": 2,
541
+ "bm25_rank": 1,
542
+ "search_method": "hybrid",
543
+ "keywords": ["mimikatz", "credential", "lsass"],
544
+ "routed_by": "redteam"
545
+ }
546
+ ]
547
+ }
548
+ ```
549
+
550
+ **Search Method Values:**
551
+ - `hybrid`: Found by both semantic and BM25 search (highest confidence)
552
+ - `semantic`: Found only by semantic search
553
+ - `keyword`: Found only by BM25 keyword search
554
+
555
+ ---
556
+
557
+ #### `get_document`
558
+
559
+ Retrieve the full content of a specific document.
560
+
561
+ | Parameter | Type | Description |
562
+ |-----------|------|-------------|
563
+ | `filepath` | string | Path to the document file |
564
+
565
+ **Returns:** JSON with document content, metadata, keywords, and chunk count.
566
+
567
+ ---
568
+
569
+ #### `reindex_documents`
570
+
571
+ Index or reindex all documents in the knowledge base.
572
+
573
+ | Parameter | Type | Default | Description |
574
+ |-----------|------|---------|-------------|
575
+ | `force` | bool | false | Smart reindex: detects changes, rebuilds BM25. Fast. |
576
+ | `full_rebuild` | bool | false | Nuclear rebuild: deletes everything, re-embeds all documents. Use after model change. |
577
+
578
+ **Returns:** JSON with indexing statistics (indexed, updated, skipped, deleted, chunks_added, chunks_removed, dedup_skipped, elapsed_seconds).
579
+
580
+ ---
581
+
582
+ #### `list_categories`
583
+
584
+ List all document categories with their document counts.
585
+
586
+ **Returns:**
587
+
588
+ ```json
589
+ {
590
+ "status": "success",
591
+ "categories": {
592
+ "security": 52,
593
+ "development": 8,
594
+ "ctf": 12,
595
+ "general": 3
596
+ },
597
+ "total_documents": 75
598
+ }
599
+ ```
600
+
601
+ ---
602
+
603
+ #### `list_documents`
604
+
605
+ List all indexed documents, optionally filtered by category.
606
+
607
+ | Parameter | Type | Description |
608
+ |-----------|------|-------------|
609
+ | `category` | string | Optional category filter |
610
+
611
+ **Returns:** JSON array of documents with id, source, category, format, chunks, and keywords.
612
+
613
+ ---
614
+
615
+ #### `get_index_stats`
616
+
617
+ Get statistics about the knowledge base index.
618
+
619
+ **Returns:**
620
+
621
+ ```json
622
+ {
623
+ "status": "success",
624
+ "stats": {
625
+ "total_documents": 75,
626
+ "total_chunks": 9256,
627
+ "unique_content_hashes": 9100,
628
+ "categories": {"security": 52, "development": 8},
629
+ "supported_formats": [".md", ".txt", ".pdf", ".py", ".json", ".docx", ".xlsx", ".pptx", ".csv", ".ipynb"],
630
+ "embedding_model": "BAAI/bge-small-en-v1.5",
631
+ "embedding_dim": 384,
632
+ "reranker_model": "Xenova/ms-marco-MiniLM-L-6-v2",
633
+ "chunk_size": 1000,
634
+ "chunk_overlap": 200,
635
+ "query_cache": {
636
+ "size": 12,
637
+ "max_size": 100,
638
+ "ttl_seconds": 300,
639
+ "hits": 45,
640
+ "misses": 23,
641
+ "hit_rate": "66.2%"
642
+ }
643
+ }
644
+ }
645
+ ```
646
+
647
+ ---
648
+
649
+ ### Document Management
650
+
651
+ #### `add_document`
652
+
653
+ Add a new document to the knowledge base from raw content. Saves the file to the documents directory and indexes it immediately.
654
+
655
+ | Parameter | Type | Default | Description |
656
+ |-----------|------|---------|-------------|
657
+ | `content` | string | required | Full text content of the document |
658
+ | `filepath` | string | required | Relative path within documents dir (e.g., `security/new-technique.md`) |
659
+ | `category` | string | "general" | Document category |
660
+
661
+ ---
662
+
663
+ #### `update_document`
664
+
665
+ Update an existing document. Removes old chunks from the index and re-indexes with new content.
666
+
667
+ | Parameter | Type | Description |
668
+ |-----------|------|-------------|
669
+ | `filepath` | string | Full path to the document file |
670
+ | `content` | string | New content for the document |
671
+
672
+ ---
673
+
674
+ #### `remove_document`
675
+
676
+ Remove a document from the knowledge base index. Optionally deletes the file from disk.
677
+
678
+ | Parameter | Type | Default | Description |
679
+ |-----------|------|---------|-------------|
680
+ | `filepath` | string | required | Path to the document file |
681
+ | `delete_file` | bool | false | If true, also delete the file from disk |
682
+
683
+ ---
684
+
685
+ #### `add_from_url`
686
+
687
+ Fetch content from a URL, strip HTML (scripts, styles, nav, footer, header), convert to markdown, and add to the knowledge base.
688
+
689
+ | Parameter | Type | Default | Description |
690
+ |-----------|------|---------|-------------|
691
+ | `url` | string | required | URL to fetch content from |
692
+ | `category` | string | "general" | Document category |
693
+ | `title` | string | null | Custom title (auto-detected from `<title>` tag if not provided) |
694
+
695
+ ---
696
+
697
+ #### `search_similar`
698
+
699
+ Find documents similar to a given document using embedding similarity.
700
+
701
+ | Parameter | Type | Default | Description |
702
+ |-----------|------|---------|-------------|
703
+ | `filepath` | string | required | Path to the reference document |
704
+ | `max_results` | int | 5 | Number of similar documents to return (1-20) |
705
+
706
+ ---
707
+
708
+ #### `evaluate_retrieval`
709
+
710
+ Evaluate retrieval quality with test queries. Useful for tuning `hybrid_alpha`, testing query expansion effectiveness, or validating after reindexing.
711
+
712
+ | Parameter | Type | Description |
713
+ |-----------|------|-------------|
714
+ | `test_cases` | string (JSON) | Array of test cases: `[{"query": "...", "expected_filepath": "..."}, ...]` |
715
+
716
+ **Metrics:**
717
+ - **MRR@5** (Mean Reciprocal Rank): Average of 1/rank for expected documents. 1.0 = always first result.
718
+ - **Recall@5**: Fraction of expected documents found in top 5 results. 1.0 = all found.
719
+
720
+ ---
721
+
722
+ ## Configuration
723
+
724
+ Knowledge RAG is fully configurable via a `config.yaml` file in the project root. If no `config.yaml` exists, sensible defaults are used — the system works out of the box with zero configuration.
725
+
726
+ ### Quick Start
727
+
728
+ ```bash
729
+ # Option 1: Use a preset
730
+ cp presets/cybersecurity.yaml config.yaml # Offensive/defensive security, CTFs
731
+ cp presets/developer.yaml config.yaml # Software engineering, APIs, DevOps
732
+ cp presets/research.yaml config.yaml # Academic research, papers, studies
733
+ cp presets/general.yaml config.yaml # Blank slate, pure semantic search
734
+
735
+ # Option 2: Start from the documented template
736
+ cp config.example.yaml config.yaml
737
+ # Edit config.yaml to your needs
738
+ ```
739
+
740
+ Restart Claude Code after changing `config.yaml`.
741
+
742
+ ### config.yaml Structure
743
+
744
+ ```yaml
745
+ # Paths — where your documents live
746
+ paths:
747
+ documents_dir: "./documents" # Scanned recursively
748
+ data_dir: "./data" # Index storage
749
+ models_cache_dir: "./models_cache" # Persistent embedding model cache
750
+
751
+ # Documents — what gets indexed and how
752
+ documents:
753
+ supported_formats: # File types to index
754
+ - .md
755
+ - .txt
756
+ - .pdf
757
+ - .docx
758
+ - .ipynb
759
+ # - .py # Uncomment to index code
760
+ exclude_patterns: # Glob patterns to skip
761
+ - "node_modules"
762
+ - ".venv"
763
+ - "__pycache__"
764
+ chunking:
765
+ chunk_size: 1000 # Max chars per chunk
766
+ chunk_overlap: 200 # Shared chars between chunks
767
+
768
+ # Models — AI models for search (all run locally, no API keys)
769
+ models:
770
+ embedding:
771
+ model: "BAAI/bge-small-en-v1.5" # ONNX, ~33MB, auto-downloaded
772
+ dimensions: 384
773
+ gpu: false # Set true + pip install knowledge-rag[gpu]
774
+ reranker:
775
+ enabled: true # Falls back to RRF if model is unavailable
776
+ model: "Xenova/ms-marco-MiniLM-L-6-v2"
777
+ top_k_multiplier: 3 # Candidates fetched before reranking
778
+
779
+ # Search — result limits and collection name
780
+ search:
781
+ default_results: 5
782
+ max_results: 20
783
+ collection_name: "knowledge_base" # Change for separate knowledge bases
784
+
785
+ # Categories — auto-tag documents by folder path
786
+ # Set to {} to disable categorization entirely
787
+ category_mappings:
788
+ "security/redteam": "redteam"
789
+ "security/blueteam": "blueteam"
790
+ "notes": "notes"
791
+
792
+ # Keyword routing — prioritize categories based on query keywords
793
+ # Set to {} for pure semantic search with no routing bias
794
+ keyword_routes:
795
+ redteam:
796
+ - pentest
797
+ - exploit
798
+ - privilege escalation
799
+
800
+ # Query expansion — expand abbreviations for better BM25 recall
801
+ # Set to {} for no expansion (search terms used as-is)
802
+ query_expansions:
803
+ sqli:
804
+ - sql injection
805
+ - sqli
806
+ privesc:
807
+ - privilege escalation
808
+ - privesc
809
+ ```
810
+
811
+ > See `config.example.yaml` for the fully documented template with explanations for every field.
812
+
813
+ ### Presets
814
+
815
+ Pre-built configurations for common use cases:
816
+
817
+ | Preset | File | Categories | Keywords | Expansions | Best For |
818
+ |--------|------|-----------|----------|-----------|----------|
819
+ | **Cybersecurity** | `presets/cybersecurity.yaml` | 8 | 200+ | 69 | Red/Blue Team, CTFs, threat hunting, exploit dev |
820
+ | **Developer** | `presets/developer.yaml` | 9 | 150+ | 50+ | Full-stack dev, APIs, DevOps, cloud, databases |
821
+ | **Research** | `presets/research.yaml` | 9 | 100+ | 40+ | Academic papers, thesis, lab notebooks, datasets |
822
+ | **General** | `presets/general.yaml` | 0 | 0 | 0 | Blank slate — pure semantic search, no domain logic |
823
+
824
+ **Creating your own preset**: Copy `config.example.yaml`, fill in your categories/keywords/expansions, save to `presets/your-domain.yaml`.
825
+
826
+ ### Configuration Reference
827
+
828
+ #### Paths
829
+
830
+ | Field | Default | Description |
831
+ |-------|---------|-------------|
832
+ | `paths.documents_dir` | `./documents` | Root folder scanned recursively for documents |
833
+ | `paths.data_dir` | `./data` | Internal storage for ChromaDB and index metadata |
834
+ | `paths.models_cache_dir` | `./models_cache` | Persistent cache for embedding models (~250MB). Survives reboots |
835
+
836
+ Relative paths resolve from the project root. Absolute paths work too.
837
+
838
+ #### Documents
839
+
840
+ | Field | Default | Description |
841
+ |-------|---------|-------------|
842
+ | `documents.supported_formats` | .md .txt .pdf .py .json .docx .xlsx .pptx .csv .ipynb | File extensions to index |
843
+ | `documents.exclude_patterns` | `[]` (empty) | Glob patterns for files/dirs to skip during indexing |
844
+ | `documents.chunking.chunk_size` | 1000 | Max characters per chunk |
845
+ | `documents.chunking.chunk_overlap` | 200 | Characters shared between consecutive chunks |
846
+
847
+ **Chunking guidelines**: Short notes → 500/100. General use → 1000/200. Long technical docs → 1500/300.
848
+
849
+ For `.md` files, chunking splits at `##` and `###` header boundaries first. Sections larger than `chunk_size` are sub-chunked with overlap. Non-markdown files use fixed-size chunking.
850
+
851
+ #### Models
852
+
853
+ | Field | Default | Description |
854
+ |-------|---------|-------------|
855
+ | `models.embedding.model` | `BAAI/bge-small-en-v1.5` | Embedding model (ONNX, runs locally) |
856
+ | `models.embedding.dimensions` | 384 | Vector dimensions (must match model) |
857
+ | `models.embedding.gpu` | false | Enable CUDA GPU acceleration. Requires `pip install knowledge-rag[gpu]` |
858
+ | `models.reranker.enabled` | true | Enable cross-encoder reranking |
859
+ | `models.reranker.model` | `Xenova/ms-marco-MiniLM-L-6-v2` | Reranker model |
860
+ | `models.reranker.top_k_multiplier` | 3 | Fetch N*multiplier candidates for reranking |
861
+
862
+ If the reranker model is not available locally and the machine cannot download it, search now falls back to the RRF order from hybrid semantic+BM25 retrieval. This keeps `search_knowledge` available offline, but result ordering may be less precise for ambiguous queries until the reranker model is cached.
863
+
864
+ **Embedding model options** (fastest → most accurate):
865
+ - `BAAI/bge-small-en-v1.5` — 384D, ~33MB (default)
866
+ - `BAAI/bge-base-en-v1.5` — 768D, ~130MB
867
+ - `BAAI/bge-large-en-v1.5` — 1024D, ~335MB
868
+ - `intfloat/multilingual-e5-small` — 384D, 100+ languages
869
+
870
+ > **Warning**: Changing the embedding model after indexing requires `reindex_documents(full_rebuild=True)`.
871
+
872
+ #### Search
873
+
874
+ | Field | Default | Description |
875
+ |-------|---------|-------------|
876
+ | `search.default_results` | 5 | Results returned when no limit specified |
877
+ | `search.max_results` | 20 | Hard cap even if client requests more |
878
+ | `search.collection_name` | `knowledge_base` | ChromaDB collection — change for separate KBs |
879
+
880
+ #### Categories
881
+
882
+ Map folder paths to category names. Documents in matching folders get auto-tagged, enabling filtered searches.
883
+
884
+ ```yaml
885
+ category_mappings:
886
+ "security/redteam": "redteam"
887
+ "security": "security"
888
+ ```
889
+
890
+ Set `category_mappings: {}` to disable — documents are still searchable, just without category filters.
891
+
892
+ #### Keyword Routing
893
+
894
+ Route queries to categories based on keywords. When a query contains listed keywords, results from that category are prioritized (not filtered — other categories still appear, ranked lower).
895
+
896
+ ```yaml
897
+ keyword_routes:
898
+ redteam:
899
+ - pentest
900
+ - exploit
901
+ - sqli
902
+ ```
903
+
904
+ Single-word keywords use regex word boundaries (`\b`) — "api" won't match "RAPID". Multi-word keywords use substring matching.
905
+
906
+ Set `keyword_routes: {}` for pure semantic search.
907
+
908
+ #### Query Expansion
909
+
910
+ Expand search terms with synonyms before BM25 search. Supports single tokens, bigrams, and full query matches.
911
+
912
+ ```yaml
913
+ query_expansions:
914
+ sqli:
915
+ - sql injection
916
+ - sqli
917
+ k8s:
918
+ - kubernetes
919
+ - k8s
920
+ ```
921
+
922
+ Set `query_expansions: {}` for no expansion.
923
+
924
+ ### Hybrid Search Tuning
925
+
926
+ | hybrid_alpha | Behavior | Best For |
927
+ |--------------|----------|----------|
928
+ | 0.0 | Pure BM25 keyword | Exact terms, CVEs, tool names |
929
+ | 0.3 | Keyword-heavy **(default)** | Technical queries with specific terms |
930
+ | 0.5 | Balanced | General queries |
931
+ | 0.7 | Semantic-heavy | Conceptual queries, related topics |
932
+ | 1.0 | Pure semantic | "How to..." questions, abstract concepts |
933
+
934
+ ---
935
+
936
+ ## Project Structure
937
+
938
+ ```
939
+ knowledge-rag/
940
+ ├── mcp_server/
941
+ │ ├── __init__.py # Stdout protection + version
942
+ │ ├── config.py # YAML config loader + defaults
943
+ │ ├── ingestion.py # 20 parsers, chunking, metadata extraction
944
+ │ └── server.py # MCP server, ChromaDB, BM25, reranker, 12 tools
945
+ ├── config.example.yaml # Documented config template (copy to config.yaml)
946
+ ├── config.yaml # Your active configuration (git-ignored)
947
+ ├── presets/ # Ready-to-use domain configurations
948
+ │ ├── cybersecurity.yaml
949
+ │ ├── developer.yaml
950
+ │ ├── research.yaml
951
+ │ └── general.yaml
952
+ ├── documents/ # Your documents (scanned recursively)
953
+ ├── data/
954
+ │ ├── chroma_db/ # ChromaDB vector database
955
+ │ └── index_metadata.json # Incremental indexing state
956
+ ├── models_cache/ # Persistent embedding model cache
957
+ ├── tests/ # Test suite (82 tests)
958
+ ├── install.sh # Linux/macOS installer
959
+ ├── install.ps1 # Windows installer
960
+ ├── venv/ # Python virtual environment
961
+ ├── requirements.txt
962
+ ├── pyproject.toml
963
+ ├── LICENSE
964
+ └── README.md
965
+ ```
966
+
967
+ ---
968
+
969
+ ## Troubleshooting
970
+
971
+ ### Python version mismatch
972
+
973
+ Requires Python 3.11 or newer.
974
+
975
+ ```bash
976
+ python --version # Must be 3.11+
977
+ ```
978
+
979
+ ### FastEmbed model download fails
980
+
981
+ On first run, FastEmbed downloads models to `models_cache/`. If the download fails:
982
+
983
+ ```bash
984
+ # Clear cache and retry
985
+ # Windows:
986
+ rmdir /s /q models_cache
987
+
988
+ # Linux/macOS:
989
+ rm -rf models_cache
990
+
991
+ # Then restart the MCP server
992
+ ```
993
+
994
+ ### Reranker model download fails
995
+
996
+ The reranker is lazy-loaded on the first query. If the model is not cached and the machine is offline, search continues without reranking and uses the RRF order from hybrid retrieval. To keep reranking enabled offline, run one query while online or pre-populate `models_cache/` on the target machine.
997
+
998
+ You can still disable reranking explicitly in `config.yaml`:
999
+
1000
+ ```yaml
1001
+ models:
1002
+ reranker:
1003
+ enabled: false
1004
+ ```
1005
+
1006
+ Disabling reranking reduces memory use and avoids first-query model loading. The tradeoff is lower ranking precision, especially when several chunks match the same terms but only one is the best answer.
1007
+
1008
+ ### ChromaDB index crashes on startup
1009
+
1010
+ Native ChromaDB failures can terminate Python before normal exception handling runs. Startup now probes ChromaDB in a child process before initializing the MCP server. If the probe crashes, the active `chroma_db/` and `index_metadata.json` are moved to `data/backups/auto-repair-*`, and the next startup can rebuild a clean index.
1011
+
1012
+ The same guarded behavior is available through either console script:
1013
+
1014
+ ```bash
1015
+ knowledge-rag
1016
+ knowledge-rag-guarded
1017
+ ```
1018
+
1019
+ ### Index is empty
1020
+
1021
+ ```bash
1022
+ # Check documents directory has files
1023
+ ls documents/
1024
+
1025
+ # Force reindex via Claude Code:
1026
+ # reindex_documents(force=True)
1027
+
1028
+ # Or nuclear rebuild if model changed:
1029
+ # reindex_documents(full_rebuild=True)
1030
+ ```
1031
+
1032
+ ### MCP server not loading
1033
+
1034
+ 1. Check `~/.claude.json` exists and has valid JSON in the `mcpServers` section
1035
+ 2. Verify paths use double backslashes (`\\`) on Windows
1036
+ 3. Restart Claude Code completely
1037
+ 4. Run `claude mcp list` to check connection status
1038
+
1039
+ ### "Failed to connect" error
1040
+
1041
+ The MCP server uses stdout for JSON-RPC communication. If a library prints to stdout during init, the stream gets corrupted. v3.4.3+ includes stdout protection that prevents this. If you're on an older version, upgrade:
1042
+
1043
+ ```bash
1044
+ pip install --upgrade knowledge-rag
1045
+ ```
1046
+
1047
+ ### Slow first query
1048
+
1049
+ The cross-encoder reranker model is lazy-loaded on the first query. This adds a one-time ~2-3 second delay for model download and loading. Subsequent queries are fast. If the model cannot be loaded, search falls back to RRF ordering and does not retry loading the reranker until the server restarts.
1050
+
1051
+ ### Memory usage
1052
+
1053
+ With ~200 documents, expect ~300-500MB RAM. The embedding model (~50MB) and reranker (~25MB) are loaded into memory. For very large knowledge bases (1000+ documents), consider enabling GPU acceleration and using exclude patterns to limit index scope.
1054
+
1055
+ ---
1056
+
1057
+ ## Changelog
1058
+
1059
+ ### Unreleased
1060
+
1061
+ - **FIX**: Startup preflight probes ChromaDB in a child process and moves crashing persistent indexes to `data/backups/auto-repair-*` before MCP initialization.
1062
+ - **FIX**: Reranker load failures now fall back to RRF ordering instead of failing `search_knowledge` on offline machines.
1063
+ - **FIX**: Virtualenv project-root detection now handles Python symlinks that resolve to the system interpreter.
1064
+ - **NEW**: `knowledge-rag-guarded` console script kept as an explicit guarded startup alias.
1065
+
1066
+ ### v3.6.2 (2026-04-23)
1067
+
1068
+ - **INFRA**: NPM provenance attestation (SLSA supply chain security), full README on npm page
1069
+ - **DOCS**: Reorganize Installation section — add NPX and Docker install methods, update What's New to v3.6.0
1070
+
1071
+ ### v3.6.0 (2026-04-23)
1072
+
1073
+ - **NEW**: Multi-language code parsing — C (`.c`), C++ (`.cpp`/`.h`), JavaScript (`.js`/`.jsx`), TypeScript (`.ts`/`.tsx`) with per-language function/class/import extraction
1074
+ - **NEW**: XML parser (`.xml`) — root element and namespace metadata extraction
1075
+ - **NEW**: All 8 new formats default enabled — no config change needed
1076
+ - **NEW**: NPM wrapper (`npx knowledge-rag`) + Docker image (`ghcr.io/lyonzin/knowledge-rag`)
1077
+ - **NEW**: Automated release pipeline — PyPI (Trusted Publishing), NPM, Docker GHCR
1078
+ - **IMPROVED**: Code parser reports correct `language` metadata per file type (was hardcoded to `"python"` for all code files)
1079
+
1080
+ ### v3.5.2 (2026-04-16)
1081
+
1082
+ - **NEW**: Auto-discovery of CUDA 12 DLLs from pip-installed NVIDIA packages — no manual PATH configuration needed
1083
+ - **NEW**: Graceful GPU→CPU fallback with `[WARN]` log when CUDA init fails (missing drivers, wrong version, etc.)
1084
+ - **FIX**: Explicit `CPUExecutionProvider` when `gpu: false` — eliminates noisy CUDA probe errors in logs
1085
+ - **FIX**: BASE_DIR resolution now correctly prefers directories with `config.yaml` over those with only `config.example.yaml` (fixes editable installs)
1086
+
1087
+ ### v3.5.1 (2026-04-16)
1088
+
1089
+ - **FIX**: Removed Python upper bound constraint (`<3.13` → `>=3.11`). Python 3.13 and 3.14 now supported — onnxruntime ships wheels for both.
1090
+
1091
+ ### v3.5.0 (2026-04-16)
1092
+
1093
+ - **NEW**: Optional GPU acceleration for ONNX embeddings — `pip install knowledge-rag[gpu]` + `models.embedding.gpu: true` in config. 5-10x faster indexing on NVIDIA GPUs with automatic CPU fallback.
1094
+ - **DOCS**: Supported formats table added to README (20 formats)
1095
+
1096
+ ### v3.4.3 (2026-04-16)
1097
+
1098
+ - **FIX**: Correct stdout protection via save/restore pattern — `__init__.py` saves original stdout and redirects to stderr during init, `server.py main()` restores it before `mcp.run()`. v3.4.2's global redirect broke MCP JSON-RPC response channel.
1099
+
1100
+ ### v3.4.1 (2026-04-16)
1101
+
1102
+ - **FIX**: `pip install knowledge-rag` now auto-detects project directory from venv location
1103
+ - **NEW**: `install.sh` — Linux/macOS installer with pip and from-source modes
1104
+ - **IMPROVED**: BASE_DIR resolution chain: env var → source dir → venv parent → CWD → fallback
1105
+
1106
+ ### v3.4.0 (2026-04-16)
1107
+
1108
+ - **NEW**: `models_cache_dir` — persistent embedding model cache, prevents re-download after reboots
1109
+ - **NEW**: `exclude_patterns` — glob-based file/directory exclusion during indexing
1110
+ - **NEW**: Jupyter Notebook (.ipynb) parser — extracts markdown and code cell sources only
1111
+ - **NEW**: MCP stdout protection — redirects stdout to stderr before server start
1112
+ - **NEW**: File watcher resilience — graceful fallback when Linux inotify limits are reached
1113
+ - **NEW**: MetaTrader (.mq4, .mqh) support — opt-in code parsing
1114
+ - **NEW**: 23 new tests (exclude patterns, ipynb parser, stdout protection)
1115
+ - Community credit: [@Hohlas](https://github.com/Hohlas) ([PR #18](https://github.com/lyonzin/knowledge-rag/pull/18))
1116
+
1117
+ ### v3.3.x
1118
+
1119
+ - **v3.3.2**: Full type validation on YAML config, bounds checking, version sync
1120
+ - **v3.3.1**: YAML null value crash fix, presets bundled in pip wheel, `knowledge-rag init` CLI
1121
+ - **v3.3.0**: YAML configuration system, 4 domain presets, generic use support
1122
+
1123
+ ### v3.2.x
1124
+
1125
+ - **v3.2.4**: Symlink support with circular loop protection
1126
+ - **v3.2.3**: BASE_DIR smart detection for pip installs
1127
+ - **v3.2.2**: Plug-and-play pip install, `KNOWLEDGE_RAG_DIR` env var
1128
+ - **v3.2.1**: Auto-recovery from corrupted ChromaDB
1129
+ - **v3.2.0**: Parallel BM25 + Semantic search, adjacent chunk retrieval
1130
+
1131
+ ### v3.1.x
1132
+
1133
+ - **v3.1.1**: Code block protection in markdown chunker, AAR category, 14 CVE aliases
1134
+ - **v3.1.0**: DOCX/XLSX/PPTX/CSV support, file watcher, MMR diversification, PyPI publish
1135
+
1136
+ ### v3.0.0 (2026-03-19)
1137
+
1138
+ - Replaced Ollama with FastEmbed (ONNX in-process)
1139
+ - Cross-encoder reranking, markdown-aware chunking, query expansion
1140
+ - 6 new MCP tools (12 total), auto-migration from v2.x
1141
+
1142
+ <details>
1143
+ <summary>v2.x and earlier</summary>
1144
+
1145
+ - **v2.2.0**: `hybrid_alpha=0` skips Ollama, default changed from 0.5 to 0.3
1146
+ - **v2.1.0**: Mermaid architecture diagrams
1147
+ - **v2.0.0**: Hybrid search, RRF fusion, `hybrid_alpha` parameter
1148
+ - **v1.1.0**: Incremental indexing, query cache, chunk deduplication
1149
+ - **v1.0.1**: Auto-cleanup orphan folders, removed hardcoded paths
1150
+ - **v1.0.0**: Initial release
1151
+ </details>
1152
+
1153
+ ---
1154
+
1155
+ ## Contributing
1156
+
1157
+ 1. Fork the repository
1158
+ 2. Create a feature branch (`git checkout -b feature/amazing-feature`)
1159
+ 3. Commit your changes
1160
+ 4. Push to the branch (`git push origin feature/amazing-feature`)
1161
+ 5. Open a Pull Request
1162
+
1163
+ ---
58
1164
 
59
1165
  ## License
60
1166
 
61
- MIT
1167
+ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
1168
+
1169
+ ---
1170
+
1171
+ ## Acknowledgments
1172
+
1173
+ - [ChromaDB](https://www.trychroma.com/) — Vector database
1174
+ - [FastEmbed](https://qdrant.github.io/fastembed/) — ONNX Runtime embeddings
1175
+ - [FastMCP](https://github.com/anthropics/mcp) — Model Context Protocol framework
1176
+ - [PyMuPDF](https://pymupdf.readthedocs.io/) — PDF parsing
1177
+ - [rank-bm25](https://github.com/dorianbrown/rank_bm25) — BM25 Okapi implementation
1178
+ - [Watchdog](https://github.com/gorakhargosh/watchdog) — File system monitoring
1179
+ - [python-docx](https://python-docx.readthedocs.io/) / [openpyxl](https://openpyxl.readthedocs.io/) / [python-pptx](https://python-pptx.readthedocs.io/) — Office document parsing
1180
+ - [PyYAML](https://pyyaml.org/) — YAML configuration parsing
1181
+ - [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/) — HTML parsing for URL ingestion
1182
+
1183
+ ---
1184
+
1185
+ ## Author
1186
+
1187
+ **Lyon.**
1188
+
1189
+ Security Researcher | Developer
1190
+
1191
+ ---
1192
+
1193
+ <div align="center">
1194
+
1195
+ **[Back to Top](#knowledge-rag)**
1196
+
1197
+ </div>
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "knowledge-rag",
3
- "version": "3.6.1",
3
+ "version": "3.7.0",
4
4
  "description": "Local RAG System for Claude Code — Hybrid search + Cross-encoder Reranking + 12 MCP Tools + 20 Format Parsers. Zero external servers.",
5
5
  "bin": {
6
6
  "knowledge-rag": "./bin/cli.js"
@@ -28,6 +28,9 @@
28
28
  "bugs": {
29
29
  "url": "https://github.com/lyonzin/knowledge-rag/issues"
30
30
  },
31
+ "funding": {
32
+ "url": "https://github.com/sponsors/lyonzin"
33
+ },
31
34
  "engines": {
32
35
  "node": ">=16"
33
36
  },