knowledge-rag 3.2.3__tar.gz → 3.3.1__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {knowledge_rag-3.2.3 → knowledge_rag-3.3.1}/.gitignore +5 -0
- {knowledge_rag-3.2.3 → knowledge_rag-3.3.1}/PKG-INFO +281 -80
- {knowledge_rag-3.2.3 → knowledge_rag-3.3.1}/README.md +279 -79
- knowledge_rag-3.3.1/config.example.yaml +211 -0
- knowledge_rag-3.3.1/mcp_server/config.py +441 -0
- {knowledge_rag-3.2.3 → knowledge_rag-3.3.1}/mcp_server/ingestion.py +16 -3
- {knowledge_rag-3.2.3 → knowledge_rag-3.3.1}/mcp_server/server.py +45 -2
- knowledge_rag-3.3.1/presets/cybersecurity.yaml +359 -0
- knowledge_rag-3.3.1/presets/developer.yaml +345 -0
- knowledge_rag-3.3.1/presets/general.yaml +109 -0
- knowledge_rag-3.3.1/presets/research.yaml +252 -0
- {knowledge_rag-3.2.3 → knowledge_rag-3.3.1}/pyproject.toml +11 -1
- {knowledge_rag-3.2.3 → knowledge_rag-3.3.1}/requirements.txt +3 -0
- knowledge_rag-3.2.3/mcp_server/config.py +0 -294
- {knowledge_rag-3.2.3 → knowledge_rag-3.3.1}/LICENSE +0 -0
- {knowledge_rag-3.2.3 → knowledge_rag-3.3.1}/documents/examples/sample-document.md +0 -0
- {knowledge_rag-3.2.3 → knowledge_rag-3.3.1}/mcp_server/__init__.py +0 -0
|
@@ -4,6 +4,9 @@ __pycache__/
|
|
|
4
4
|
*.pyc
|
|
5
5
|
data/
|
|
6
6
|
|
|
7
|
+
# User config (personal settings — use presets/ as starting point)
|
|
8
|
+
config.yaml
|
|
9
|
+
|
|
7
10
|
# Personal documents (NEVER commit — user-populated content)
|
|
8
11
|
documents/aar/
|
|
9
12
|
documents/security/
|
|
@@ -50,3 +53,5 @@ desktop.ini
|
|
|
50
53
|
*.swp
|
|
51
54
|
*.swo
|
|
52
55
|
dist/
|
|
56
|
+
.ruff_cache/
|
|
57
|
+
.pytest_cache/
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: knowledge-rag
|
|
3
|
-
Version: 3.
|
|
3
|
+
Version: 3.3.1
|
|
4
4
|
Summary: Local RAG System for Claude Code — Hybrid search + Cross-encoder Reranking + 12 MCP Tools. Zero external servers.
|
|
5
5
|
Project-URL: Homepage, https://github.com/lyonzin/knowledge-rag
|
|
6
6
|
Project-URL: Repository, https://github.com/lyonzin/knowledge-rag
|
|
@@ -28,6 +28,7 @@ Requires-Dist: openpyxl>=3.1.0
|
|
|
28
28
|
Requires-Dist: pymupdf>=1.23.0
|
|
29
29
|
Requires-Dist: python-docx>=1.0.0
|
|
30
30
|
Requires-Dist: python-pptx>=1.0.0
|
|
31
|
+
Requires-Dist: pyyaml>=6.0
|
|
31
32
|
Requires-Dist: rank-bm25>=0.2.2
|
|
32
33
|
Requires-Dist: requests>=2.31.0
|
|
33
34
|
Requires-Dist: watchdog>=4.0.0
|
|
@@ -37,7 +38,7 @@ Description-Content-Type: text/markdown
|
|
|
37
38
|
|
|
38
39
|
<div align="center">
|
|
39
40
|
|
|
40
|
-

|
|
41
42
|

|
|
42
43
|

|
|
43
44
|

|
|
@@ -60,7 +61,7 @@ Your documents become instantly searchable inside Claude Code — with reranking
|
|
|
60
61
|
|
|
61
62
|
**12 MCP Tools** | **Hybrid Search + Cross-Encoder Reranking** | **Markdown-Aware Chunking** | **100% Local, Zero Cloud**
|
|
62
63
|
|
|
63
|
-
[What's New](#whats-new-in-
|
|
64
|
+
[What's New](#whats-new-in-v330) | [Installation](#installation) | [Configuration](#configuration) | [API Reference](#api-reference) | [Architecture](#architecture)
|
|
64
65
|
|
|
65
66
|
</div>
|
|
66
67
|
|
|
@@ -107,6 +108,31 @@ After the initial rebuild, startup and queries are faster than v2.x because ther
|
|
|
107
108
|
|
|
108
109
|
---
|
|
109
110
|
|
|
111
|
+
## What's New in v3.3.0
|
|
112
|
+
|
|
113
|
+
### YAML Configuration System
|
|
114
|
+
|
|
115
|
+
All settings are now customizable via `config.yaml` — no more editing Python code. Categories, keyword routing, query expansions, models, chunking, and paths are all configurable through a single YAML file.
|
|
116
|
+
|
|
117
|
+
### Domain Presets
|
|
118
|
+
|
|
119
|
+
Four ready-to-use presets ship with the project. Copy one to `config.yaml` and you're done:
|
|
120
|
+
|
|
121
|
+
- **Cybersecurity** — 8 categories, 200+ keywords, 69 query expansions (red team, blue team, CTFs, threat hunting)
|
|
122
|
+
- **Developer** — 9 categories, 150+ keywords, 50+ expansions (full-stack, APIs, DevOps, cloud, databases)
|
|
123
|
+
- **Research** — 9 categories, 100+ keywords, 40+ expansions (academic papers, thesis, lab notebooks)
|
|
124
|
+
- **General** — Zero routing, zero expansions. Pure semantic search for any domain.
|
|
125
|
+
|
|
126
|
+
### Generic Use Support
|
|
127
|
+
|
|
128
|
+
With `category_mappings: {}`, `keyword_routes: {}`, and `query_expansions: {}`, the system operates as a domain-agnostic semantic search engine. No security-specific logic unless you want it.
|
|
129
|
+
|
|
130
|
+
### Backwards Compatible
|
|
131
|
+
|
|
132
|
+
No `config.yaml`? The system uses built-in defaults — identical behavior to v3.2.x. Zero migration required.
|
|
133
|
+
|
|
134
|
+
---
|
|
135
|
+
|
|
110
136
|
## What's New in v3.1.0
|
|
111
137
|
|
|
112
138
|
### Office Document Support (DOCX, XLSX, PPTX, CSV)
|
|
@@ -139,7 +165,7 @@ After hybrid RRF fusion produces initial candidates, a cross-encoder (Xenova/ms-
|
|
|
139
165
|
|
|
140
166
|
### Query Expansion
|
|
141
167
|
|
|
142
|
-
|
|
168
|
+
69 security-term synonym mappings expand abbreviated queries before BM25 search. Searching for "sqli" automatically includes "sql injection"; "privesc" includes "privilege escalation"; "pth" includes "pass-the-hash". Customize or replace these in `config.yaml` (see [Configuration](#configuration)).
|
|
143
169
|
|
|
144
170
|
### 6 New MCP Tools (12 Total)
|
|
145
171
|
|
|
@@ -167,7 +193,7 @@ Knowledge RAG is a **100% local** hybrid search system that integrates with Clau
|
|
|
167
193
|
- **Zero External Dependencies**: Everything runs in-process. No Ollama, no API keys, no servers to manage.
|
|
168
194
|
- **Hybrid Search + Reranking**: Semantic embeddings + BM25 keywords fused with RRF, then reranked by a cross-encoder for maximum precision.
|
|
169
195
|
- **Markdown-Aware**: `.md` files are chunked by section headers, preserving semantic coherence.
|
|
170
|
-
- **Query Expansion**:
|
|
196
|
+
- **Query Expansion**: Customizable synonym mappings ensure abbreviated queries find relevant content (69 security terms included as preset).
|
|
171
197
|
- **Privacy First**: All processing happens locally. No data leaves your machine.
|
|
172
198
|
- **Multi-Format**: Supports MD, PDF, DOCX, XLSX, PPTX, CSV, TXT, Python, JSON files.
|
|
173
199
|
- **Smart Routing**: Keyword-based routing with word boundaries for accurate category filtering.
|
|
@@ -182,7 +208,8 @@ Knowledge RAG is a **100% local** hybrid search system that integrates with Clau
|
|
|
182
208
|
|---------|-------------|
|
|
183
209
|
| **Hybrid Search** | Semantic + BM25 keyword search with Reciprocal Rank Fusion |
|
|
184
210
|
| **Cross-Encoder Reranker** | Xenova/ms-marco-MiniLM-L-6-v2 re-scores top candidates for precision |
|
|
185
|
-
| **
|
|
211
|
+
| **YAML Configuration** | Fully customizable via `config.yaml` with domain-specific presets |
|
|
212
|
+
| **Query Expansion** | 69 security-term synonym mappings (sqli, privesc, pth, etc.) — customizable |
|
|
186
213
|
| **Markdown-Aware Chunking** | `.md` files split by `##`/`###` sections instead of fixed windows |
|
|
187
214
|
| **In-Process Embeddings** | FastEmbed ONNX Runtime (BAAI/bge-small-en-v1.5, 384D) |
|
|
188
215
|
| **Keyword Routing** | Word-boundary aware routing for domain-specific queries |
|
|
@@ -381,16 +408,23 @@ flowchart LR
|
|
|
381
408
|
|
|
382
409
|
### Quick Start (3 steps)
|
|
383
410
|
|
|
384
|
-
**Step 1:
|
|
411
|
+
**Step 1: Install**
|
|
385
412
|
|
|
386
413
|
```bash
|
|
387
|
-
#
|
|
414
|
+
# Option A: pip install (recommended)
|
|
415
|
+
pip install knowledge-rag
|
|
416
|
+
knowledge-rag init # Exports config template, presets, creates documents/
|
|
417
|
+
|
|
418
|
+
# Option B: Clone from source
|
|
388
419
|
git clone https://github.com/lyonzin/knowledge-rag.git ~/knowledge-rag
|
|
389
420
|
cd ~/knowledge-rag
|
|
421
|
+
python3 -m venv venv
|
|
422
|
+
source venv/bin/activate # Linux/macOS
|
|
423
|
+
# .\venv\Scripts\activate # Windows
|
|
390
424
|
pip install -r requirements.txt
|
|
391
425
|
```
|
|
392
426
|
|
|
393
|
-
> **Windows users**:
|
|
427
|
+
> **Windows users**: Use `python` instead of `python3`.
|
|
394
428
|
|
|
395
429
|
**Step 2: Configure Claude Code**
|
|
396
430
|
|
|
@@ -398,9 +432,11 @@ From inside the cloned folder, run:
|
|
|
398
432
|
|
|
399
433
|
```bash
|
|
400
434
|
cd ~/knowledge-rag
|
|
401
|
-
claude mcp add knowledge-rag -s user -- python -m mcp_server.server
|
|
435
|
+
claude mcp add knowledge-rag -s user -- ~/knowledge-rag/venv/bin/python -m mcp_server.server
|
|
402
436
|
```
|
|
403
437
|
|
|
438
|
+
> **Windows**: `claude mcp add knowledge-rag -s user -- cmd /c "cd /d %USERPROFILE%\knowledge-rag && venv\Scripts\python -m mcp_server.server"`
|
|
439
|
+
|
|
404
440
|
That's it. Claude Code now knows about your RAG server.
|
|
405
441
|
|
|
406
442
|
<details>
|
|
@@ -428,14 +464,15 @@ Add to `~/.claude.json`:
|
|
|
428
464
|
"mcpServers": {
|
|
429
465
|
"knowledge-rag": {
|
|
430
466
|
"type": "stdio",
|
|
431
|
-
"command": "python",
|
|
467
|
+
"command": "/home/YOUR_USER/knowledge-rag/venv/bin/python",
|
|
432
468
|
"args": ["-m", "mcp_server.server"],
|
|
433
|
-
"cwd": "
|
|
469
|
+
"cwd": "/home/YOUR_USER/knowledge-rag",
|
|
434
470
|
"env": {}
|
|
435
471
|
}
|
|
436
472
|
}
|
|
437
473
|
}
|
|
438
474
|
```
|
|
475
|
+
> Replace `YOUR_USER` with your username, or use the full path from `echo $HOME`.
|
|
439
476
|
</details>
|
|
440
477
|
|
|
441
478
|
**Step 3: Restart Claude Code**
|
|
@@ -535,7 +572,7 @@ evaluate_retrieval(test_cases='[
|
|
|
535
572
|
|
|
536
573
|
## API Reference
|
|
537
574
|
|
|
538
|
-
###
|
|
575
|
+
### Search & Query
|
|
539
576
|
|
|
540
577
|
#### `search_knowledge`
|
|
541
578
|
|
|
@@ -675,7 +712,7 @@ Get statistics about the knowledge base index.
|
|
|
675
712
|
|
|
676
713
|
---
|
|
677
714
|
|
|
678
|
-
###
|
|
715
|
+
### Document Management
|
|
679
716
|
|
|
680
717
|
#### `add_document`
|
|
681
718
|
|
|
@@ -826,76 +863,210 @@ Evaluate retrieval quality with test queries. Useful for tuning `hybrid_alpha`,
|
|
|
826
863
|
|
|
827
864
|
## Configuration
|
|
828
865
|
|
|
829
|
-
|
|
866
|
+
Knowledge RAG is fully configurable via a `config.yaml` file in the project root. If no `config.yaml` exists, sensible defaults are used — the system works out of the box with zero configuration.
|
|
830
867
|
|
|
831
|
-
###
|
|
868
|
+
### Quick Start
|
|
832
869
|
|
|
833
|
-
```
|
|
834
|
-
|
|
835
|
-
|
|
870
|
+
```bash
|
|
871
|
+
# Option 1: Use a preset
|
|
872
|
+
cp presets/cybersecurity.yaml config.yaml # Offensive/defensive security, CTFs
|
|
873
|
+
cp presets/developer.yaml config.yaml # Software engineering, APIs, DevOps
|
|
874
|
+
cp presets/research.yaml config.yaml # Academic research, papers, studies
|
|
875
|
+
cp presets/general.yaml config.yaml # Blank slate, pure semantic search
|
|
876
|
+
|
|
877
|
+
# Option 2: Start from the documented template
|
|
878
|
+
cp config.example.yaml config.yaml
|
|
879
|
+
# Edit config.yaml to your needs
|
|
836
880
|
```
|
|
837
881
|
|
|
838
|
-
|
|
882
|
+
Restart Claude Code after changing `config.yaml`.
|
|
883
|
+
|
|
884
|
+
### config.yaml Structure
|
|
885
|
+
|
|
886
|
+
```yaml
|
|
887
|
+
# Paths — where your documents live
|
|
888
|
+
paths:
|
|
889
|
+
documents_dir: "./documents" # Scanned recursively
|
|
890
|
+
data_dir: "./data" # Index storage
|
|
891
|
+
|
|
892
|
+
# Documents — what gets indexed and how
|
|
893
|
+
documents:
|
|
894
|
+
supported_formats: # File types to index
|
|
895
|
+
- .md
|
|
896
|
+
- .txt
|
|
897
|
+
- .pdf
|
|
898
|
+
- .docx
|
|
899
|
+
# - .py # Uncomment to index code
|
|
900
|
+
chunking:
|
|
901
|
+
chunk_size: 1000 # Max chars per chunk
|
|
902
|
+
chunk_overlap: 200 # Shared chars between chunks
|
|
903
|
+
|
|
904
|
+
# Models — AI models for search (all run locally, no API keys)
|
|
905
|
+
models:
|
|
906
|
+
embedding:
|
|
907
|
+
model: "BAAI/bge-small-en-v1.5" # ONNX, ~33MB, auto-downloaded
|
|
908
|
+
dimensions: 384
|
|
909
|
+
reranker:
|
|
910
|
+
enabled: true # Set false on low-resource machines
|
|
911
|
+
model: "Xenova/ms-marco-MiniLM-L-6-v2"
|
|
912
|
+
top_k_multiplier: 3 # Candidates fetched before reranking
|
|
913
|
+
|
|
914
|
+
# Search — result limits and collection name
|
|
915
|
+
search:
|
|
916
|
+
default_results: 5
|
|
917
|
+
max_results: 20
|
|
918
|
+
collection_name: "knowledge_base" # Change for separate knowledge bases
|
|
919
|
+
|
|
920
|
+
# Categories — auto-tag documents by folder path
|
|
921
|
+
# Set to {} to disable categorization entirely
|
|
922
|
+
category_mappings:
|
|
923
|
+
"security/redteam": "redteam"
|
|
924
|
+
"security/blueteam": "blueteam"
|
|
925
|
+
"notes": "notes"
|
|
926
|
+
|
|
927
|
+
# Keyword routing — prioritize categories based on query keywords
|
|
928
|
+
# Set to {} for pure semantic search with no routing bias
|
|
929
|
+
keyword_routes:
|
|
930
|
+
redteam:
|
|
931
|
+
- pentest
|
|
932
|
+
- exploit
|
|
933
|
+
- privilege escalation
|
|
934
|
+
|
|
935
|
+
# Query expansion — expand abbreviations for better BM25 recall
|
|
936
|
+
# Set to {} for no expansion (search terms used as-is)
|
|
937
|
+
query_expansions:
|
|
938
|
+
sqli:
|
|
939
|
+
- sql injection
|
|
940
|
+
- sqli
|
|
941
|
+
privesc:
|
|
942
|
+
- privilege escalation
|
|
943
|
+
- privesc
|
|
944
|
+
```
|
|
839
945
|
|
|
840
|
-
|
|
946
|
+
> See `config.example.yaml` for the fully documented template with explanations for every field.
|
|
841
947
|
|
|
842
|
-
|
|
843
|
-
reranker_model: str = "Xenova/ms-marco-MiniLM-L-6-v2"
|
|
844
|
-
reranker_enabled: bool = True
|
|
845
|
-
reranker_top_k_multiplier: int = 3 # Retrieve 3x candidates for reranking
|
|
846
|
-
```
|
|
948
|
+
### Presets
|
|
847
949
|
|
|
848
|
-
|
|
950
|
+
Pre-built configurations for common use cases. Each preset is a complete `config.yaml` ready to use:
|
|
849
951
|
|
|
850
|
-
|
|
952
|
+
| Preset | File | Categories | Keywords | Expansions | Best For |
|
|
953
|
+
|--------|------|-----------|----------|-----------|----------|
|
|
954
|
+
| **Cybersecurity** | `presets/cybersecurity.yaml` | 8 | 200+ | 69 | Red/Blue Team, CTFs, threat hunting, exploit dev |
|
|
955
|
+
| **Developer** | `presets/developer.yaml` | 9 | 150+ | 50+ | Full-stack dev, APIs, DevOps, cloud, databases |
|
|
956
|
+
| **Research** | `presets/research.yaml` | 9 | 100+ | 40+ | Academic papers, thesis, lab notebooks, datasets |
|
|
957
|
+
| **General** | `presets/general.yaml` | 0 | 0 | 0 | Blank slate — pure semantic search, no domain logic |
|
|
851
958
|
|
|
852
|
-
|
|
853
|
-
|
|
854
|
-
|
|
855
|
-
|
|
856
|
-
|
|
857
|
-
|
|
858
|
-
|
|
859
|
-
|
|
959
|
+
**Creating your own preset**: Copy `config.example.yaml`, fill in your categories/keywords/expansions, save to `presets/your-domain.yaml`. Share it with the community via PR.
|
|
960
|
+
|
|
961
|
+
### Configuration Reference
|
|
962
|
+
|
|
963
|
+
#### Paths
|
|
964
|
+
|
|
965
|
+
| Field | Default | Description |
|
|
966
|
+
|-------|---------|-------------|
|
|
967
|
+
| `paths.documents_dir` | `./documents` | Root folder scanned recursively for documents |
|
|
968
|
+
| `paths.data_dir` | `./data` | Internal storage for ChromaDB and index metadata |
|
|
969
|
+
|
|
970
|
+
Relative paths resolve from the project root. Absolute paths work too. The `KNOWLEDGE_RAG_DIR` environment variable overrides the project root.
|
|
971
|
+
|
|
972
|
+
#### Documents
|
|
973
|
+
|
|
974
|
+
| Field | Default | Description |
|
|
975
|
+
|-------|---------|-------------|
|
|
976
|
+
| `documents.supported_formats` | .md .txt .pdf .py .json .docx .xlsx .pptx .csv | File extensions to index |
|
|
977
|
+
| `documents.chunking.chunk_size` | 1000 | Max characters per chunk |
|
|
978
|
+
| `documents.chunking.chunk_overlap` | 200 | Characters shared between consecutive chunks |
|
|
979
|
+
|
|
980
|
+
**Chunking guidelines**: Short notes → 500/100. General use → 1000/200. Long technical docs → 1500/300.
|
|
981
|
+
|
|
982
|
+
For `.md` files, chunking splits at `##` and `###` header boundaries first. Sections larger than `chunk_size` are sub-chunked with overlap. Non-markdown files use fixed-size chunking.
|
|
983
|
+
|
|
984
|
+
#### Models
|
|
985
|
+
|
|
986
|
+
| Field | Default | Description |
|
|
987
|
+
|-------|---------|-------------|
|
|
988
|
+
| `models.embedding.model` | `BAAI/bge-small-en-v1.5` | Embedding model (ONNX, runs locally) |
|
|
989
|
+
| `models.embedding.dimensions` | 384 | Vector dimensions (must match model) |
|
|
990
|
+
| `models.reranker.enabled` | true | Enable cross-encoder reranking |
|
|
991
|
+
| `models.reranker.model` | `Xenova/ms-marco-MiniLM-L-6-v2` | Reranker model |
|
|
992
|
+
| `models.reranker.top_k_multiplier` | 3 | Fetch N*multiplier candidates for reranking |
|
|
993
|
+
|
|
994
|
+
**Embedding model options** (fastest → most accurate):
|
|
995
|
+
- `BAAI/bge-small-en-v1.5` — 384D, ~33MB (default)
|
|
996
|
+
- `BAAI/bge-base-en-v1.5` — 768D, ~130MB
|
|
997
|
+
- `BAAI/bge-large-en-v1.5` — 1024D, ~335MB
|
|
998
|
+
- `intfloat/multilingual-e5-small` — 384D, 100+ languages
|
|
999
|
+
|
|
1000
|
+
> **Warning**: Changing the embedding model after indexing requires `reindex_documents(full_rebuild=True)`.
|
|
1001
|
+
|
|
1002
|
+
The reranker fetches `max_results * top_k_multiplier` candidates from RRF fusion, re-scores them with the cross-encoder, and returns the top `max_results`. Set `enabled: false` to disable and use RRF scores directly.
|
|
1003
|
+
|
|
1004
|
+
#### Search
|
|
1005
|
+
|
|
1006
|
+
| Field | Default | Description |
|
|
1007
|
+
|-------|---------|-------------|
|
|
1008
|
+
| `search.default_results` | 5 | Results returned when no limit specified |
|
|
1009
|
+
| `search.max_results` | 20 | Hard cap even if client requests more |
|
|
1010
|
+
| `search.collection_name` | `knowledge_base` | ChromaDB collection — change for separate KBs |
|
|
1011
|
+
|
|
1012
|
+
#### Categories
|
|
1013
|
+
|
|
1014
|
+
Map folder paths to category names. Documents in matching folders get auto-tagged, enabling filtered searches.
|
|
1015
|
+
|
|
1016
|
+
```yaml
|
|
1017
|
+
# Most specific match wins:
|
|
1018
|
+
# documents/security/redteam/exploit.md → "redteam"
|
|
1019
|
+
# documents/security/overview.md → "security"
|
|
1020
|
+
category_mappings:
|
|
1021
|
+
"security/redteam": "redteam"
|
|
1022
|
+
"security": "security"
|
|
860
1023
|
```
|
|
861
1024
|
|
|
862
|
-
|
|
1025
|
+
Set `category_mappings: {}` to disable — documents are still searchable, just without category filters.
|
|
863
1026
|
|
|
864
|
-
|
|
1027
|
+
#### Keyword Routing
|
|
865
1028
|
|
|
866
|
-
|
|
867
|
-
|
|
868
|
-
|
|
1029
|
+
Route queries to categories based on keywords. When a query contains listed keywords, results from that category are prioritized (not filtered — other categories still appear, ranked lower).
|
|
1030
|
+
|
|
1031
|
+
```yaml
|
|
1032
|
+
keyword_routes:
|
|
1033
|
+
redteam:
|
|
1034
|
+
- pentest
|
|
1035
|
+
- exploit
|
|
1036
|
+
- sqli
|
|
869
1037
|
```
|
|
870
1038
|
|
|
871
|
-
|
|
1039
|
+
Single-word keywords use regex word boundaries (`\b`) — "api" won't match "RAPID". Multi-word keywords use substring matching. When multiple keywords match, the category with the most matches wins.
|
|
872
1040
|
|
|
873
|
-
|
|
1041
|
+
Set `keyword_routes: {}` for pure semantic search.
|
|
874
1042
|
|
|
875
|
-
|
|
876
|
-
|
|
877
|
-
|
|
878
|
-
|
|
879
|
-
|
|
880
|
-
|
|
881
|
-
|
|
882
|
-
|
|
883
|
-
|
|
1043
|
+
#### Query Expansion
|
|
1044
|
+
|
|
1045
|
+
Expand search terms with synonyms before BM25 search. Supports single tokens, bigrams, and full query matches.
|
|
1046
|
+
|
|
1047
|
+
```yaml
|
|
1048
|
+
query_expansions:
|
|
1049
|
+
sqli:
|
|
1050
|
+
- sql injection
|
|
1051
|
+
- sqli
|
|
1052
|
+
k8s:
|
|
1053
|
+
- kubernetes
|
|
1054
|
+
- k8s
|
|
884
1055
|
```
|
|
885
1056
|
|
|
886
|
-
|
|
1057
|
+
Set `query_expansions: {}` for no expansion.
|
|
887
1058
|
|
|
888
1059
|
### Hybrid Search Tuning
|
|
889
1060
|
|
|
890
|
-
| hybrid_alpha | Behavior |
|
|
891
|
-
|
|
892
|
-
| 0.0 | Pure BM25 keyword |
|
|
893
|
-
| 0.3 | Keyword-heavy **(default)** |
|
|
894
|
-
| 0.5 | Balanced |
|
|
895
|
-
| 0.7 | Semantic-heavy |
|
|
896
|
-
| 1.0 | Pure semantic |
|
|
1061
|
+
| hybrid_alpha | Behavior | Best For |
|
|
1062
|
+
|--------------|----------|----------|
|
|
1063
|
+
| 0.0 | Pure BM25 keyword | Exact terms, CVEs, tool names |
|
|
1064
|
+
| 0.3 | Keyword-heavy **(default)** | Technical queries with specific terms |
|
|
1065
|
+
| 0.5 | Balanced | General queries |
|
|
1066
|
+
| 0.7 | Semantic-heavy | Conceptual queries, related topics |
|
|
1067
|
+
| 1.0 | Pure semantic | "How to..." questions, abstract concepts |
|
|
897
1068
|
|
|
898
|
-
>
|
|
1069
|
+
> All `hybrid_alpha` values have similar speed because FastEmbed runs in-process.
|
|
899
1070
|
|
|
900
1071
|
---
|
|
901
1072
|
|
|
@@ -905,25 +1076,25 @@ Single-word keywords use regex word boundaries (`\b`) to prevent false positives
|
|
|
905
1076
|
knowledge-rag/
|
|
906
1077
|
├── mcp_server/
|
|
907
1078
|
│ ├── __init__.py
|
|
908
|
-
│ ├── config.py
|
|
909
|
-
│ ├── ingestion.py
|
|
910
|
-
│ └── server.py
|
|
911
|
-
├──
|
|
912
|
-
|
|
913
|
-
|
|
914
|
-
│ ├──
|
|
915
|
-
│ ├──
|
|
916
|
-
│
|
|
1079
|
+
│ ├── config.py # YAML config loader + defaults
|
|
1080
|
+
│ ├── ingestion.py # Document parsing, chunking, metadata extraction
|
|
1081
|
+
│ └── server.py # MCP server, ChromaDB, BM25, reranker, 12 tools
|
|
1082
|
+
├── config.example.yaml # Documented config template (copy to config.yaml)
|
|
1083
|
+
├── config.yaml # Your active configuration (git-ignored)
|
|
1084
|
+
├── presets/ # Ready-to-use domain configurations
|
|
1085
|
+
│ ├── cybersecurity.yaml # Red/Blue Team, CTFs, threat hunting
|
|
1086
|
+
│ ├── developer.yaml # Software engineering, APIs, DevOps
|
|
1087
|
+
│ ├── research.yaml # Academic research, papers, studies
|
|
1088
|
+
│ └── general.yaml # Blank slate, zero domain logic
|
|
1089
|
+
├── documents/ # Your documents (scanned recursively)
|
|
917
1090
|
├── data/
|
|
918
|
-
│ ├── chroma_db/
|
|
919
|
-
│ └── index_metadata.json
|
|
920
|
-
├──
|
|
921
|
-
|
|
922
|
-
├──
|
|
923
|
-
├──
|
|
924
|
-
|
|
925
|
-
├── LICENSE # MIT License
|
|
926
|
-
└── README.md # This file
|
|
1091
|
+
│ ├── chroma_db/ # ChromaDB vector database
|
|
1092
|
+
│ └── index_metadata.json # Incremental indexing state
|
|
1093
|
+
├── tests/ # Test suite (55 tests)
|
|
1094
|
+
├── venv/ # Python virtual environment
|
|
1095
|
+
├── requirements.txt # Python dependencies
|
|
1096
|
+
├── LICENSE # MIT License
|
|
1097
|
+
└── README.md
|
|
927
1098
|
```
|
|
928
1099
|
|
|
929
1100
|
---
|
|
@@ -1024,6 +1195,35 @@ With ~200 documents, expect ~300-500MB RAM. The embedding model (~50MB) and rera
|
|
|
1024
1195
|
|
|
1025
1196
|
## Changelog
|
|
1026
1197
|
|
|
1198
|
+
### v3.3.1 (2026-04-06)
|
|
1199
|
+
|
|
1200
|
+
- **FIX**: YAML null values (`category_mappings:` without value) no longer crash the server — falls back to defaults
|
|
1201
|
+
- **FIX**: Presets and config template now bundled in pip wheel (`knowledge-rag init` exports them)
|
|
1202
|
+
- **NEW**: `knowledge-rag init` CLI command — exports config template, presets, and creates documents/ in current directory
|
|
1203
|
+
|
|
1204
|
+
### v3.3.0 (2026-04-05)
|
|
1205
|
+
|
|
1206
|
+
- **NEW**: YAML configuration system — fully customizable via `config.yaml`
|
|
1207
|
+
- **NEW**: Domain presets — `presets/cybersecurity.yaml`, `presets/developer.yaml`, `presets/research.yaml`, `presets/general.yaml`
|
|
1208
|
+
- **NEW**: `config.example.yaml` — documented template with explanations for every field
|
|
1209
|
+
- **NEW**: Categories, keyword routing, and query expansions are now user-configurable (no more editing `config.py`)
|
|
1210
|
+
- **NEW**: Empty config = pure semantic search with zero domain logic (ideal for generic use)
|
|
1211
|
+
- **NEW**: Warning log for empty files during indexing (previously silent skip)
|
|
1212
|
+
- **IMPROVED**: README rewritten — full configuration reference, preset documentation, updated project structure
|
|
1213
|
+
- **IMPROVED**: `pyyaml` added as dependency
|
|
1214
|
+
- **BREAKING**: None — system works identically without `config.yaml` (backwards compatible)
|
|
1215
|
+
|
|
1216
|
+
### v3.2.4 (2026-04-03)
|
|
1217
|
+
|
|
1218
|
+
- **NEW**: Symlink support — `documents/` directory now follows symbolic links recursively ([#13](https://github.com/lyonzin/knowledge-rag/issues/13))
|
|
1219
|
+
- **NEW**: Circular symlink loop protection via realpath deduplication
|
|
1220
|
+
- **IMPROVED**: `_has_documents()` detection now validates against supported formats only (stricter than before)
|
|
1221
|
+
|
|
1222
|
+
### v3.2.3 (2026-03-22)
|
|
1223
|
+
|
|
1224
|
+
- **FIX**: BASE_DIR smart detection — checks for actual files in `documents/` (not just dir existence)
|
|
1225
|
+
- **FIX**: Prevents false positive when `site-packages/documents/` exists as empty dir
|
|
1226
|
+
|
|
1027
1227
|
### v3.2.2 (2026-03-22)
|
|
1028
1228
|
|
|
1029
1229
|
- **FIX**: `pip install knowledge-rag` now works as plug-and-play (BASE_DIR falls back to CWD when installed from PyPI)
|
|
@@ -1068,7 +1268,7 @@ With ~200 documents, expect ~300-500MB RAM. The embedding model (~50MB) and rera
|
|
|
1068
1268
|
- **BREAKING**: Changed embedding model from nomic-embed-text (768D) to BAAI/bge-small-en-v1.5 (384D)
|
|
1069
1269
|
- **NEW**: Cross-encoder reranker (Xenova/ms-marco-MiniLM-L-6-v2) applied after RRF fusion
|
|
1070
1270
|
- **NEW**: Markdown-aware chunking — `.md` files split by `##`/`###` sections
|
|
1071
|
-
- **NEW**: Query expansion with
|
|
1271
|
+
- **NEW**: Query expansion with 69 security-term synonym mappings
|
|
1072
1272
|
- **NEW**: `add_document` — add document from raw content string
|
|
1073
1273
|
- **NEW**: `update_document` — update existing document (re-chunks and re-indexes)
|
|
1074
1274
|
- **NEW**: `remove_document` — remove document from index (optionally delete file)
|
|
@@ -1152,6 +1352,7 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
|
|
|
1152
1352
|
- [FastMCP](https://github.com/anthropics/mcp) — Model Context Protocol framework
|
|
1153
1353
|
- [PyMuPDF](https://pymupdf.readthedocs.io/) — PDF parsing
|
|
1154
1354
|
- [rank-bm25](https://github.com/dorianbrown/rank_bm25) — BM25 Okapi implementation
|
|
1355
|
+
- [PyYAML](https://pyyaml.org/) — YAML configuration parsing
|
|
1155
1356
|
- [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/) — HTML parsing for URL ingestion
|
|
1156
1357
|
|
|
1157
1358
|
---
|