knowledge-rag 3.2.3__tar.gz → 3.3.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {knowledge_rag-3.2.3 → knowledge_rag-3.3.0}/.gitignore +5 -0
- {knowledge_rag-3.2.3 → knowledge_rag-3.3.0}/PKG-INFO +271 -78
- {knowledge_rag-3.2.3 → knowledge_rag-3.3.0}/README.md +269 -77
- knowledge_rag-3.3.0/config.example.yaml +211 -0
- knowledge_rag-3.3.0/mcp_server/config.py +278 -0
- {knowledge_rag-3.2.3 → knowledge_rag-3.3.0}/mcp_server/ingestion.py +16 -3
- knowledge_rag-3.3.0/presets/cybersecurity.yaml +359 -0
- knowledge_rag-3.3.0/presets/developer.yaml +345 -0
- knowledge_rag-3.3.0/presets/general.yaml +109 -0
- knowledge_rag-3.3.0/presets/research.yaml +252 -0
- {knowledge_rag-3.2.3 → knowledge_rag-3.3.0}/pyproject.toml +4 -1
- {knowledge_rag-3.2.3 → knowledge_rag-3.3.0}/requirements.txt +3 -0
- knowledge_rag-3.2.3/mcp_server/config.py +0 -294
- {knowledge_rag-3.2.3 → knowledge_rag-3.3.0}/LICENSE +0 -0
- {knowledge_rag-3.2.3 → knowledge_rag-3.3.0}/documents/examples/sample-document.md +0 -0
- {knowledge_rag-3.2.3 → knowledge_rag-3.3.0}/mcp_server/__init__.py +0 -0
- {knowledge_rag-3.2.3 → knowledge_rag-3.3.0}/mcp_server/server.py +0 -0
|
@@ -4,6 +4,9 @@ __pycache__/
|
|
|
4
4
|
*.pyc
|
|
5
5
|
data/
|
|
6
6
|
|
|
7
|
+
# User config (personal settings — use presets/ as starting point)
|
|
8
|
+
config.yaml
|
|
9
|
+
|
|
7
10
|
# Personal documents (NEVER commit — user-populated content)
|
|
8
11
|
documents/aar/
|
|
9
12
|
documents/security/
|
|
@@ -50,3 +53,5 @@ desktop.ini
|
|
|
50
53
|
*.swp
|
|
51
54
|
*.swo
|
|
52
55
|
dist/
|
|
56
|
+
.ruff_cache/
|
|
57
|
+
.pytest_cache/
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: knowledge-rag
|
|
3
|
-
Version: 3.
|
|
3
|
+
Version: 3.3.0
|
|
4
4
|
Summary: Local RAG System for Claude Code — Hybrid search + Cross-encoder Reranking + 12 MCP Tools. Zero external servers.
|
|
5
5
|
Project-URL: Homepage, https://github.com/lyonzin/knowledge-rag
|
|
6
6
|
Project-URL: Repository, https://github.com/lyonzin/knowledge-rag
|
|
@@ -28,6 +28,7 @@ Requires-Dist: openpyxl>=3.1.0
|
|
|
28
28
|
Requires-Dist: pymupdf>=1.23.0
|
|
29
29
|
Requires-Dist: python-docx>=1.0.0
|
|
30
30
|
Requires-Dist: python-pptx>=1.0.0
|
|
31
|
+
Requires-Dist: pyyaml>=6.0
|
|
31
32
|
Requires-Dist: rank-bm25>=0.2.2
|
|
32
33
|
Requires-Dist: requests>=2.31.0
|
|
33
34
|
Requires-Dist: watchdog>=4.0.0
|
|
@@ -37,7 +38,7 @@ Description-Content-Type: text/markdown
|
|
|
37
38
|
|
|
38
39
|
<div align="center">
|
|
39
40
|
|
|
40
|
-

|
|
41
42
|

|
|
42
43
|

|
|
43
44
|

|
|
@@ -60,7 +61,7 @@ Your documents become instantly searchable inside Claude Code — with reranking
|
|
|
60
61
|
|
|
61
62
|
**12 MCP Tools** | **Hybrid Search + Cross-Encoder Reranking** | **Markdown-Aware Chunking** | **100% Local, Zero Cloud**
|
|
62
63
|
|
|
63
|
-
[What's New](#whats-new-in-
|
|
64
|
+
[What's New](#whats-new-in-v330) | [Installation](#installation) | [Configuration](#configuration) | [API Reference](#api-reference) | [Architecture](#architecture)
|
|
64
65
|
|
|
65
66
|
</div>
|
|
66
67
|
|
|
@@ -107,6 +108,31 @@ After the initial rebuild, startup and queries are faster than v2.x because ther
|
|
|
107
108
|
|
|
108
109
|
---
|
|
109
110
|
|
|
111
|
+
## What's New in v3.3.0
|
|
112
|
+
|
|
113
|
+
### YAML Configuration System
|
|
114
|
+
|
|
115
|
+
All settings are now customizable via `config.yaml` — no more editing Python code. Categories, keyword routing, query expansions, models, chunking, and paths are all configurable through a single YAML file.
|
|
116
|
+
|
|
117
|
+
### Domain Presets
|
|
118
|
+
|
|
119
|
+
Four ready-to-use presets ship with the project. Copy one to `config.yaml` and you're done:
|
|
120
|
+
|
|
121
|
+
- **Cybersecurity** — 8 categories, 200+ keywords, 69 query expansions (red team, blue team, CTFs, threat hunting)
|
|
122
|
+
- **Developer** — 9 categories, 150+ keywords, 50+ expansions (full-stack, APIs, DevOps, cloud, databases)
|
|
123
|
+
- **Research** — 9 categories, 100+ keywords, 40+ expansions (academic papers, thesis, lab notebooks)
|
|
124
|
+
- **General** — Zero routing, zero expansions. Pure semantic search for any domain.
|
|
125
|
+
|
|
126
|
+
### Generic Use Support
|
|
127
|
+
|
|
128
|
+
With `category_mappings: {}`, `keyword_routes: {}`, and `query_expansions: {}`, the system operates as a domain-agnostic semantic search engine. No security-specific logic unless you want it.
|
|
129
|
+
|
|
130
|
+
### Backwards Compatible
|
|
131
|
+
|
|
132
|
+
No `config.yaml`? The system uses built-in defaults — identical behavior to v3.2.x. Zero migration required.
|
|
133
|
+
|
|
134
|
+
---
|
|
135
|
+
|
|
110
136
|
## What's New in v3.1.0
|
|
111
137
|
|
|
112
138
|
### Office Document Support (DOCX, XLSX, PPTX, CSV)
|
|
@@ -139,7 +165,7 @@ After hybrid RRF fusion produces initial candidates, a cross-encoder (Xenova/ms-
|
|
|
139
165
|
|
|
140
166
|
### Query Expansion
|
|
141
167
|
|
|
142
|
-
|
|
168
|
+
69 security-term synonym mappings expand abbreviated queries before BM25 search. Searching for "sqli" automatically includes "sql injection"; "privesc" includes "privilege escalation"; "pth" includes "pass-the-hash". Customize or replace these in `config.yaml` (see [Configuration](#configuration)).
|
|
143
169
|
|
|
144
170
|
### 6 New MCP Tools (12 Total)
|
|
145
171
|
|
|
@@ -167,7 +193,7 @@ Knowledge RAG is a **100% local** hybrid search system that integrates with Clau
|
|
|
167
193
|
- **Zero External Dependencies**: Everything runs in-process. No Ollama, no API keys, no servers to manage.
|
|
168
194
|
- **Hybrid Search + Reranking**: Semantic embeddings + BM25 keywords fused with RRF, then reranked by a cross-encoder for maximum precision.
|
|
169
195
|
- **Markdown-Aware**: `.md` files are chunked by section headers, preserving semantic coherence.
|
|
170
|
-
- **Query Expansion**:
|
|
196
|
+
- **Query Expansion**: Customizable synonym mappings ensure abbreviated queries find relevant content (69 security terms included as preset).
|
|
171
197
|
- **Privacy First**: All processing happens locally. No data leaves your machine.
|
|
172
198
|
- **Multi-Format**: Supports MD, PDF, DOCX, XLSX, PPTX, CSV, TXT, Python, JSON files.
|
|
173
199
|
- **Smart Routing**: Keyword-based routing with word boundaries for accurate category filtering.
|
|
@@ -182,7 +208,8 @@ Knowledge RAG is a **100% local** hybrid search system that integrates with Clau
|
|
|
182
208
|
|---------|-------------|
|
|
183
209
|
| **Hybrid Search** | Semantic + BM25 keyword search with Reciprocal Rank Fusion |
|
|
184
210
|
| **Cross-Encoder Reranker** | Xenova/ms-marco-MiniLM-L-6-v2 re-scores top candidates for precision |
|
|
185
|
-
| **
|
|
211
|
+
| **YAML Configuration** | Fully customizable via `config.yaml` with domain-specific presets |
|
|
212
|
+
| **Query Expansion** | 69 security-term synonym mappings (sqli, privesc, pth, etc.) — customizable |
|
|
186
213
|
| **Markdown-Aware Chunking** | `.md` files split by `##`/`###` sections instead of fixed windows |
|
|
187
214
|
| **In-Process Embeddings** | FastEmbed ONNX Runtime (BAAI/bge-small-en-v1.5, 384D) |
|
|
188
215
|
| **Keyword Routing** | Word-boundary aware routing for domain-specific queries |
|
|
@@ -387,10 +414,15 @@ flowchart LR
|
|
|
387
414
|
# Clone to your home directory
|
|
388
415
|
git clone https://github.com/lyonzin/knowledge-rag.git ~/knowledge-rag
|
|
389
416
|
cd ~/knowledge-rag
|
|
417
|
+
|
|
418
|
+
# Create virtual environment and install
|
|
419
|
+
python3 -m venv venv
|
|
420
|
+
source venv/bin/activate # Linux/macOS
|
|
421
|
+
# .\venv\Scripts\activate # Windows
|
|
390
422
|
pip install -r requirements.txt
|
|
391
423
|
```
|
|
392
424
|
|
|
393
|
-
> **Windows users**: `~/knowledge-rag` becomes `C:\Users\YourName\knowledge-rag`
|
|
425
|
+
> **Windows users**: `~/knowledge-rag` becomes `C:\Users\YourName\knowledge-rag`. Use `python` instead of `python3`.
|
|
394
426
|
|
|
395
427
|
**Step 2: Configure Claude Code**
|
|
396
428
|
|
|
@@ -398,9 +430,11 @@ From inside the cloned folder, run:
|
|
|
398
430
|
|
|
399
431
|
```bash
|
|
400
432
|
cd ~/knowledge-rag
|
|
401
|
-
claude mcp add knowledge-rag -s user -- python -m mcp_server.server
|
|
433
|
+
claude mcp add knowledge-rag -s user -- ~/knowledge-rag/venv/bin/python -m mcp_server.server
|
|
402
434
|
```
|
|
403
435
|
|
|
436
|
+
> **Windows**: `claude mcp add knowledge-rag -s user -- cmd /c "cd /d %USERPROFILE%\knowledge-rag && venv\Scripts\python -m mcp_server.server"`
|
|
437
|
+
|
|
404
438
|
That's it. Claude Code now knows about your RAG server.
|
|
405
439
|
|
|
406
440
|
<details>
|
|
@@ -428,14 +462,15 @@ Add to `~/.claude.json`:
|
|
|
428
462
|
"mcpServers": {
|
|
429
463
|
"knowledge-rag": {
|
|
430
464
|
"type": "stdio",
|
|
431
|
-
"command": "python",
|
|
465
|
+
"command": "/home/YOUR_USER/knowledge-rag/venv/bin/python",
|
|
432
466
|
"args": ["-m", "mcp_server.server"],
|
|
433
|
-
"cwd": "
|
|
467
|
+
"cwd": "/home/YOUR_USER/knowledge-rag",
|
|
434
468
|
"env": {}
|
|
435
469
|
}
|
|
436
470
|
}
|
|
437
471
|
}
|
|
438
472
|
```
|
|
473
|
+
> Replace `YOUR_USER` with your username, or use the full path from `echo $HOME`.
|
|
439
474
|
</details>
|
|
440
475
|
|
|
441
476
|
**Step 3: Restart Claude Code**
|
|
@@ -535,7 +570,7 @@ evaluate_retrieval(test_cases='[
|
|
|
535
570
|
|
|
536
571
|
## API Reference
|
|
537
572
|
|
|
538
|
-
###
|
|
573
|
+
### Search & Query
|
|
539
574
|
|
|
540
575
|
#### `search_knowledge`
|
|
541
576
|
|
|
@@ -675,7 +710,7 @@ Get statistics about the knowledge base index.
|
|
|
675
710
|
|
|
676
711
|
---
|
|
677
712
|
|
|
678
|
-
###
|
|
713
|
+
### Document Management
|
|
679
714
|
|
|
680
715
|
#### `add_document`
|
|
681
716
|
|
|
@@ -826,76 +861,210 @@ Evaluate retrieval quality with test queries. Useful for tuning `hybrid_alpha`,
|
|
|
826
861
|
|
|
827
862
|
## Configuration
|
|
828
863
|
|
|
829
|
-
|
|
864
|
+
Knowledge RAG is fully configurable via a `config.yaml` file in the project root. If no `config.yaml` exists, sensible defaults are used — the system works out of the box with zero configuration.
|
|
830
865
|
|
|
831
|
-
###
|
|
866
|
+
### Quick Start
|
|
832
867
|
|
|
833
|
-
```
|
|
834
|
-
|
|
835
|
-
|
|
868
|
+
```bash
|
|
869
|
+
# Option 1: Use a preset
|
|
870
|
+
cp presets/cybersecurity.yaml config.yaml # Offensive/defensive security, CTFs
|
|
871
|
+
cp presets/developer.yaml config.yaml # Software engineering, APIs, DevOps
|
|
872
|
+
cp presets/research.yaml config.yaml # Academic research, papers, studies
|
|
873
|
+
cp presets/general.yaml config.yaml # Blank slate, pure semantic search
|
|
874
|
+
|
|
875
|
+
# Option 2: Start from the documented template
|
|
876
|
+
cp config.example.yaml config.yaml
|
|
877
|
+
# Edit config.yaml to your needs
|
|
836
878
|
```
|
|
837
879
|
|
|
838
|
-
|
|
880
|
+
Restart Claude Code after changing `config.yaml`.
|
|
881
|
+
|
|
882
|
+
### config.yaml Structure
|
|
883
|
+
|
|
884
|
+
```yaml
|
|
885
|
+
# Paths — where your documents live
|
|
886
|
+
paths:
|
|
887
|
+
documents_dir: "./documents" # Scanned recursively
|
|
888
|
+
data_dir: "./data" # Index storage
|
|
889
|
+
|
|
890
|
+
# Documents — what gets indexed and how
|
|
891
|
+
documents:
|
|
892
|
+
supported_formats: # File types to index
|
|
893
|
+
- .md
|
|
894
|
+
- .txt
|
|
895
|
+
- .pdf
|
|
896
|
+
- .docx
|
|
897
|
+
# - .py # Uncomment to index code
|
|
898
|
+
chunking:
|
|
899
|
+
chunk_size: 1000 # Max chars per chunk
|
|
900
|
+
chunk_overlap: 200 # Shared chars between chunks
|
|
901
|
+
|
|
902
|
+
# Models — AI models for search (all run locally, no API keys)
|
|
903
|
+
models:
|
|
904
|
+
embedding:
|
|
905
|
+
model: "BAAI/bge-small-en-v1.5" # ONNX, ~33MB, auto-downloaded
|
|
906
|
+
dimensions: 384
|
|
907
|
+
reranker:
|
|
908
|
+
enabled: true # Set false on low-resource machines
|
|
909
|
+
model: "Xenova/ms-marco-MiniLM-L-6-v2"
|
|
910
|
+
top_k_multiplier: 3 # Candidates fetched before reranking
|
|
911
|
+
|
|
912
|
+
# Search — result limits and collection name
|
|
913
|
+
search:
|
|
914
|
+
default_results: 5
|
|
915
|
+
max_results: 20
|
|
916
|
+
collection_name: "knowledge_base" # Change for separate knowledge bases
|
|
917
|
+
|
|
918
|
+
# Categories — auto-tag documents by folder path
|
|
919
|
+
# Set to {} to disable categorization entirely
|
|
920
|
+
category_mappings:
|
|
921
|
+
"security/redteam": "redteam"
|
|
922
|
+
"security/blueteam": "blueteam"
|
|
923
|
+
"notes": "notes"
|
|
924
|
+
|
|
925
|
+
# Keyword routing — prioritize categories based on query keywords
|
|
926
|
+
# Set to {} for pure semantic search with no routing bias
|
|
927
|
+
keyword_routes:
|
|
928
|
+
redteam:
|
|
929
|
+
- pentest
|
|
930
|
+
- exploit
|
|
931
|
+
- privilege escalation
|
|
932
|
+
|
|
933
|
+
# Query expansion — expand abbreviations for better BM25 recall
|
|
934
|
+
# Set to {} for no expansion (search terms used as-is)
|
|
935
|
+
query_expansions:
|
|
936
|
+
sqli:
|
|
937
|
+
- sql injection
|
|
938
|
+
- sqli
|
|
939
|
+
privesc:
|
|
940
|
+
- privilege escalation
|
|
941
|
+
- privesc
|
|
942
|
+
```
|
|
839
943
|
|
|
840
|
-
|
|
944
|
+
> See `config.example.yaml` for the fully documented template with explanations for every field.
|
|
841
945
|
|
|
842
|
-
|
|
843
|
-
reranker_model: str = "Xenova/ms-marco-MiniLM-L-6-v2"
|
|
844
|
-
reranker_enabled: bool = True
|
|
845
|
-
reranker_top_k_multiplier: int = 3 # Retrieve 3x candidates for reranking
|
|
846
|
-
```
|
|
946
|
+
### Presets
|
|
847
947
|
|
|
848
|
-
|
|
948
|
+
Pre-built configurations for common use cases. Each preset is a complete `config.yaml` ready to use:
|
|
849
949
|
|
|
850
|
-
|
|
950
|
+
| Preset | File | Categories | Keywords | Expansions | Best For |
|
|
951
|
+
|--------|------|-----------|----------|-----------|----------|
|
|
952
|
+
| **Cybersecurity** | `presets/cybersecurity.yaml` | 8 | 200+ | 69 | Red/Blue Team, CTFs, threat hunting, exploit dev |
|
|
953
|
+
| **Developer** | `presets/developer.yaml` | 9 | 150+ | 50+ | Full-stack dev, APIs, DevOps, cloud, databases |
|
|
954
|
+
| **Research** | `presets/research.yaml` | 9 | 100+ | 40+ | Academic papers, thesis, lab notebooks, datasets |
|
|
955
|
+
| **General** | `presets/general.yaml` | 0 | 0 | 0 | Blank slate — pure semantic search, no domain logic |
|
|
851
956
|
|
|
852
|
-
|
|
853
|
-
|
|
854
|
-
|
|
855
|
-
|
|
856
|
-
|
|
857
|
-
|
|
858
|
-
|
|
859
|
-
|
|
957
|
+
**Creating your own preset**: Copy `config.example.yaml`, fill in your categories/keywords/expansions, save to `presets/your-domain.yaml`. Share it with the community via PR.
|
|
958
|
+
|
|
959
|
+
### Configuration Reference
|
|
960
|
+
|
|
961
|
+
#### Paths
|
|
962
|
+
|
|
963
|
+
| Field | Default | Description |
|
|
964
|
+
|-------|---------|-------------|
|
|
965
|
+
| `paths.documents_dir` | `./documents` | Root folder scanned recursively for documents |
|
|
966
|
+
| `paths.data_dir` | `./data` | Internal storage for ChromaDB and index metadata |
|
|
967
|
+
|
|
968
|
+
Relative paths resolve from the project root. Absolute paths work too. The `KNOWLEDGE_RAG_DIR` environment variable overrides the project root.
|
|
969
|
+
|
|
970
|
+
#### Documents
|
|
971
|
+
|
|
972
|
+
| Field | Default | Description |
|
|
973
|
+
|-------|---------|-------------|
|
|
974
|
+
| `documents.supported_formats` | .md .txt .pdf .py .json .docx .xlsx .pptx .csv | File extensions to index |
|
|
975
|
+
| `documents.chunking.chunk_size` | 1000 | Max characters per chunk |
|
|
976
|
+
| `documents.chunking.chunk_overlap` | 200 | Characters shared between consecutive chunks |
|
|
977
|
+
|
|
978
|
+
**Chunking guidelines**: Short notes → 500/100. General use → 1000/200. Long technical docs → 1500/300.
|
|
979
|
+
|
|
980
|
+
For `.md` files, chunking splits at `##` and `###` header boundaries first. Sections larger than `chunk_size` are sub-chunked with overlap. Non-markdown files use fixed-size chunking.
|
|
981
|
+
|
|
982
|
+
#### Models
|
|
983
|
+
|
|
984
|
+
| Field | Default | Description |
|
|
985
|
+
|-------|---------|-------------|
|
|
986
|
+
| `models.embedding.model` | `BAAI/bge-small-en-v1.5` | Embedding model (ONNX, runs locally) |
|
|
987
|
+
| `models.embedding.dimensions` | 384 | Vector dimensions (must match model) |
|
|
988
|
+
| `models.reranker.enabled` | true | Enable cross-encoder reranking |
|
|
989
|
+
| `models.reranker.model` | `Xenova/ms-marco-MiniLM-L-6-v2` | Reranker model |
|
|
990
|
+
| `models.reranker.top_k_multiplier` | 3 | Fetch N*multiplier candidates for reranking |
|
|
991
|
+
|
|
992
|
+
**Embedding model options** (fastest → most accurate):
|
|
993
|
+
- `BAAI/bge-small-en-v1.5` — 384D, ~33MB (default)
|
|
994
|
+
- `BAAI/bge-base-en-v1.5` — 768D, ~130MB
|
|
995
|
+
- `BAAI/bge-large-en-v1.5` — 1024D, ~335MB
|
|
996
|
+
- `intfloat/multilingual-e5-small` — 384D, 100+ languages
|
|
997
|
+
|
|
998
|
+
> **Warning**: Changing the embedding model after indexing requires `reindex_documents(full_rebuild=True)`.
|
|
999
|
+
|
|
1000
|
+
The reranker fetches `max_results * top_k_multiplier` candidates from RRF fusion, re-scores them with the cross-encoder, and returns the top `max_results`. Set `enabled: false` to disable and use RRF scores directly.
|
|
1001
|
+
|
|
1002
|
+
#### Search
|
|
1003
|
+
|
|
1004
|
+
| Field | Default | Description |
|
|
1005
|
+
|-------|---------|-------------|
|
|
1006
|
+
| `search.default_results` | 5 | Results returned when no limit specified |
|
|
1007
|
+
| `search.max_results` | 20 | Hard cap even if client requests more |
|
|
1008
|
+
| `search.collection_name` | `knowledge_base` | ChromaDB collection — change for separate KBs |
|
|
1009
|
+
|
|
1010
|
+
#### Categories
|
|
1011
|
+
|
|
1012
|
+
Map folder paths to category names. Documents in matching folders get auto-tagged, enabling filtered searches.
|
|
1013
|
+
|
|
1014
|
+
```yaml
|
|
1015
|
+
# Most specific match wins:
|
|
1016
|
+
# documents/security/redteam/exploit.md → "redteam"
|
|
1017
|
+
# documents/security/overview.md → "security"
|
|
1018
|
+
category_mappings:
|
|
1019
|
+
"security/redteam": "redteam"
|
|
1020
|
+
"security": "security"
|
|
860
1021
|
```
|
|
861
1022
|
|
|
862
|
-
|
|
1023
|
+
Set `category_mappings: {}` to disable — documents are still searchable, just without category filters.
|
|
863
1024
|
|
|
864
|
-
|
|
1025
|
+
#### Keyword Routing
|
|
865
1026
|
|
|
866
|
-
|
|
867
|
-
|
|
868
|
-
|
|
1027
|
+
Route queries to categories based on keywords. When a query contains listed keywords, results from that category are prioritized (not filtered — other categories still appear, ranked lower).
|
|
1028
|
+
|
|
1029
|
+
```yaml
|
|
1030
|
+
keyword_routes:
|
|
1031
|
+
redteam:
|
|
1032
|
+
- pentest
|
|
1033
|
+
- exploit
|
|
1034
|
+
- sqli
|
|
869
1035
|
```
|
|
870
1036
|
|
|
871
|
-
|
|
1037
|
+
Single-word keywords use regex word boundaries (`\b`) — "api" won't match "RAPID". Multi-word keywords use substring matching. When multiple keywords match, the category with the most matches wins.
|
|
872
1038
|
|
|
873
|
-
|
|
1039
|
+
Set `keyword_routes: {}` for pure semantic search.
|
|
874
1040
|
|
|
875
|
-
|
|
876
|
-
|
|
877
|
-
|
|
878
|
-
|
|
879
|
-
|
|
880
|
-
|
|
881
|
-
|
|
882
|
-
|
|
883
|
-
|
|
1041
|
+
#### Query Expansion
|
|
1042
|
+
|
|
1043
|
+
Expand search terms with synonyms before BM25 search. Supports single tokens, bigrams, and full query matches.
|
|
1044
|
+
|
|
1045
|
+
```yaml
|
|
1046
|
+
query_expansions:
|
|
1047
|
+
sqli:
|
|
1048
|
+
- sql injection
|
|
1049
|
+
- sqli
|
|
1050
|
+
k8s:
|
|
1051
|
+
- kubernetes
|
|
1052
|
+
- k8s
|
|
884
1053
|
```
|
|
885
1054
|
|
|
886
|
-
|
|
1055
|
+
Set `query_expansions: {}` for no expansion.
|
|
887
1056
|
|
|
888
1057
|
### Hybrid Search Tuning
|
|
889
1058
|
|
|
890
|
-
| hybrid_alpha | Behavior |
|
|
891
|
-
|
|
892
|
-
| 0.0 | Pure BM25 keyword |
|
|
893
|
-
| 0.3 | Keyword-heavy **(default)** |
|
|
894
|
-
| 0.5 | Balanced |
|
|
895
|
-
| 0.7 | Semantic-heavy |
|
|
896
|
-
| 1.0 | Pure semantic |
|
|
1059
|
+
| hybrid_alpha | Behavior | Best For |
|
|
1060
|
+
|--------------|----------|----------|
|
|
1061
|
+
| 0.0 | Pure BM25 keyword | Exact terms, CVEs, tool names |
|
|
1062
|
+
| 0.3 | Keyword-heavy **(default)** | Technical queries with specific terms |
|
|
1063
|
+
| 0.5 | Balanced | General queries |
|
|
1064
|
+
| 0.7 | Semantic-heavy | Conceptual queries, related topics |
|
|
1065
|
+
| 1.0 | Pure semantic | "How to..." questions, abstract concepts |
|
|
897
1066
|
|
|
898
|
-
>
|
|
1067
|
+
> All `hybrid_alpha` values have similar speed because FastEmbed runs in-process.
|
|
899
1068
|
|
|
900
1069
|
---
|
|
901
1070
|
|
|
@@ -905,25 +1074,25 @@ Single-word keywords use regex word boundaries (`\b`) to prevent false positives
|
|
|
905
1074
|
knowledge-rag/
|
|
906
1075
|
├── mcp_server/
|
|
907
1076
|
│ ├── __init__.py
|
|
908
|
-
│ ├── config.py
|
|
909
|
-
│ ├── ingestion.py
|
|
910
|
-
│ └── server.py
|
|
911
|
-
├──
|
|
912
|
-
|
|
913
|
-
|
|
914
|
-
│ ├──
|
|
915
|
-
│ ├──
|
|
916
|
-
│
|
|
1077
|
+
│ ├── config.py # YAML config loader + defaults
|
|
1078
|
+
│ ├── ingestion.py # Document parsing, chunking, metadata extraction
|
|
1079
|
+
│ └── server.py # MCP server, ChromaDB, BM25, reranker, 12 tools
|
|
1080
|
+
├── config.example.yaml # Documented config template (copy to config.yaml)
|
|
1081
|
+
├── config.yaml # Your active configuration (git-ignored)
|
|
1082
|
+
├── presets/ # Ready-to-use domain configurations
|
|
1083
|
+
│ ├── cybersecurity.yaml # Red/Blue Team, CTFs, threat hunting
|
|
1084
|
+
│ ├── developer.yaml # Software engineering, APIs, DevOps
|
|
1085
|
+
│ ├── research.yaml # Academic research, papers, studies
|
|
1086
|
+
│ └── general.yaml # Blank slate, zero domain logic
|
|
1087
|
+
├── documents/ # Your documents (scanned recursively)
|
|
917
1088
|
├── data/
|
|
918
|
-
│ ├── chroma_db/
|
|
919
|
-
│ └── index_metadata.json
|
|
920
|
-
├──
|
|
921
|
-
|
|
922
|
-
├──
|
|
923
|
-
├──
|
|
924
|
-
|
|
925
|
-
├── LICENSE # MIT License
|
|
926
|
-
└── README.md # This file
|
|
1089
|
+
│ ├── chroma_db/ # ChromaDB vector database
|
|
1090
|
+
│ └── index_metadata.json # Incremental indexing state
|
|
1091
|
+
├── tests/ # Test suite (55 tests)
|
|
1092
|
+
├── venv/ # Python virtual environment
|
|
1093
|
+
├── requirements.txt # Python dependencies
|
|
1094
|
+
├── LICENSE # MIT License
|
|
1095
|
+
└── README.md
|
|
927
1096
|
```
|
|
928
1097
|
|
|
929
1098
|
---
|
|
@@ -1024,6 +1193,29 @@ With ~200 documents, expect ~300-500MB RAM. The embedding model (~50MB) and rera
|
|
|
1024
1193
|
|
|
1025
1194
|
## Changelog
|
|
1026
1195
|
|
|
1196
|
+
### v3.3.0 (2026-04-05)
|
|
1197
|
+
|
|
1198
|
+
- **NEW**: YAML configuration system — fully customizable via `config.yaml`
|
|
1199
|
+
- **NEW**: Domain presets — `presets/cybersecurity.yaml`, `presets/developer.yaml`, `presets/research.yaml`, `presets/general.yaml`
|
|
1200
|
+
- **NEW**: `config.example.yaml` — documented template with explanations for every field
|
|
1201
|
+
- **NEW**: Categories, keyword routing, and query expansions are now user-configurable (no more editing `config.py`)
|
|
1202
|
+
- **NEW**: Empty config = pure semantic search with zero domain logic (ideal for generic use)
|
|
1203
|
+
- **NEW**: Warning log for empty files during indexing (previously silent skip)
|
|
1204
|
+
- **IMPROVED**: README rewritten — full configuration reference, preset documentation, updated project structure
|
|
1205
|
+
- **IMPROVED**: `pyyaml` added as dependency
|
|
1206
|
+
- **BREAKING**: None — system works identically without `config.yaml` (backwards compatible)
|
|
1207
|
+
|
|
1208
|
+
### v3.2.4 (2026-04-03)
|
|
1209
|
+
|
|
1210
|
+
- **NEW**: Symlink support — `documents/` directory now follows symbolic links recursively ([#13](https://github.com/lyonzin/knowledge-rag/issues/13))
|
|
1211
|
+
- **NEW**: Circular symlink loop protection via realpath deduplication
|
|
1212
|
+
- **IMPROVED**: `_has_documents()` detection now validates against supported formats only (stricter than before)
|
|
1213
|
+
|
|
1214
|
+
### v3.2.3 (2026-03-22)
|
|
1215
|
+
|
|
1216
|
+
- **FIX**: BASE_DIR smart detection — checks for actual files in `documents/` (not just dir existence)
|
|
1217
|
+
- **FIX**: Prevents false positive when `site-packages/documents/` exists as empty dir
|
|
1218
|
+
|
|
1027
1219
|
### v3.2.2 (2026-03-22)
|
|
1028
1220
|
|
|
1029
1221
|
- **FIX**: `pip install knowledge-rag` now works as plug-and-play (BASE_DIR falls back to CWD when installed from PyPI)
|
|
@@ -1068,7 +1260,7 @@ With ~200 documents, expect ~300-500MB RAM. The embedding model (~50MB) and rera
|
|
|
1068
1260
|
- **BREAKING**: Changed embedding model from nomic-embed-text (768D) to BAAI/bge-small-en-v1.5 (384D)
|
|
1069
1261
|
- **NEW**: Cross-encoder reranker (Xenova/ms-marco-MiniLM-L-6-v2) applied after RRF fusion
|
|
1070
1262
|
- **NEW**: Markdown-aware chunking — `.md` files split by `##`/`###` sections
|
|
1071
|
-
- **NEW**: Query expansion with
|
|
1263
|
+
- **NEW**: Query expansion with 69 security-term synonym mappings
|
|
1072
1264
|
- **NEW**: `add_document` — add document from raw content string
|
|
1073
1265
|
- **NEW**: `update_document` — update existing document (re-chunks and re-indexes)
|
|
1074
1266
|
- **NEW**: `remove_document` — remove document from index (optionally delete file)
|
|
@@ -1152,6 +1344,7 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
|
|
|
1152
1344
|
- [FastMCP](https://github.com/anthropics/mcp) — Model Context Protocol framework
|
|
1153
1345
|
- [PyMuPDF](https://pymupdf.readthedocs.io/) — PDF parsing
|
|
1154
1346
|
- [rank-bm25](https://github.com/dorianbrown/rank_bm25) — BM25 Okapi implementation
|
|
1347
|
+
- [PyYAML](https://pyyaml.org/) — YAML configuration parsing
|
|
1155
1348
|
- [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/) — HTML parsing for URL ingestion
|
|
1156
1349
|
|
|
1157
1350
|
---
|