knowledge-rag 3.2.3__tar.gz → 3.3.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -4,6 +4,9 @@ __pycache__/
4
4
  *.pyc
5
5
  data/
6
6
 
7
+ # User config (personal settings — use presets/ as starting point)
8
+ config.yaml
9
+
7
10
  # Personal documents (NEVER commit — user-populated content)
8
11
  documents/aar/
9
12
  documents/security/
@@ -50,3 +53,5 @@ desktop.ini
50
53
  *.swp
51
54
  *.swo
52
55
  dist/
56
+ .ruff_cache/
57
+ .pytest_cache/
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: knowledge-rag
3
- Version: 3.2.3
3
+ Version: 3.3.0
4
4
  Summary: Local RAG System for Claude Code — Hybrid search + Cross-encoder Reranking + 12 MCP Tools. Zero external servers.
5
5
  Project-URL: Homepage, https://github.com/lyonzin/knowledge-rag
6
6
  Project-URL: Repository, https://github.com/lyonzin/knowledge-rag
@@ -28,6 +28,7 @@ Requires-Dist: openpyxl>=3.1.0
28
28
  Requires-Dist: pymupdf>=1.23.0
29
29
  Requires-Dist: python-docx>=1.0.0
30
30
  Requires-Dist: python-pptx>=1.0.0
31
+ Requires-Dist: pyyaml>=6.0
31
32
  Requires-Dist: rank-bm25>=0.2.2
32
33
  Requires-Dist: requests>=2.31.0
33
34
  Requires-Dist: watchdog>=4.0.0
@@ -37,7 +38,7 @@ Description-Content-Type: text/markdown
37
38
 
38
39
  <div align="center">
39
40
 
40
- ![Version](https://img.shields.io/badge/version-3.2.2-blue.svg)
41
+ ![Version](https://img.shields.io/badge/version-3.3.0-blue.svg)
41
42
  ![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12-green.svg)
42
43
  ![License](https://img.shields.io/badge/license-MIT-yellow.svg)
43
44
  ![Platform](https://img.shields.io/badge/platform-Windows%20%7C%20Linux%20%7C%20macOS-lightgrey.svg)
@@ -60,7 +61,7 @@ Your documents become instantly searchable inside Claude Code — with reranking
60
61
 
61
62
  **12 MCP Tools** | **Hybrid Search + Cross-Encoder Reranking** | **Markdown-Aware Chunking** | **100% Local, Zero Cloud**
62
63
 
63
- [What's New](#whats-new-in-v310) | [Installation](#installation) | [API Reference](#api-reference) | [Architecture](#architecture)
64
+ [What's New](#whats-new-in-v330) | [Installation](#installation) | [Configuration](#configuration) | [API Reference](#api-reference) | [Architecture](#architecture)
64
65
 
65
66
  </div>
66
67
 
@@ -107,6 +108,31 @@ After the initial rebuild, startup and queries are faster than v2.x because ther
107
108
 
108
109
  ---
109
110
 
111
+ ## What's New in v3.3.0
112
+
113
+ ### YAML Configuration System
114
+
115
+ All settings are now customizable via `config.yaml` — no more editing Python code. Categories, keyword routing, query expansions, models, chunking, and paths are all configurable through a single YAML file.
116
+
117
+ ### Domain Presets
118
+
119
+ Four ready-to-use presets ship with the project. Copy one to `config.yaml` and you're done:
120
+
121
+ - **Cybersecurity** — 8 categories, 200+ keywords, 69 query expansions (red team, blue team, CTFs, threat hunting)
122
+ - **Developer** — 9 categories, 150+ keywords, 50+ expansions (full-stack, APIs, DevOps, cloud, databases)
123
+ - **Research** — 9 categories, 100+ keywords, 40+ expansions (academic papers, thesis, lab notebooks)
124
+ - **General** — Zero routing, zero expansions. Pure semantic search for any domain.
125
+
126
+ ### Generic Use Support
127
+
128
+ With `category_mappings: {}`, `keyword_routes: {}`, and `query_expansions: {}`, the system operates as a domain-agnostic semantic search engine. No security-specific logic unless you want it.
129
+
130
+ ### Backwards Compatible
131
+
132
+ No `config.yaml`? The system uses built-in defaults — identical behavior to v3.2.x. Zero migration required.
133
+
134
+ ---
135
+
110
136
  ## What's New in v3.1.0
111
137
 
112
138
  ### Office Document Support (DOCX, XLSX, PPTX, CSV)
@@ -139,7 +165,7 @@ After hybrid RRF fusion produces initial candidates, a cross-encoder (Xenova/ms-
139
165
 
140
166
  ### Query Expansion
141
167
 
142
- 54 security-term synonym mappings expand abbreviated queries before BM25 search. Searching for "sqli" automatically includes "sql injection"; "privesc" includes "privilege escalation"; "pth" includes "pass-the-hash". The full expansion table is in `config.py`.
168
+ 69 security-term synonym mappings expand abbreviated queries before BM25 search. Searching for "sqli" automatically includes "sql injection"; "privesc" includes "privilege escalation"; "pth" includes "pass-the-hash". Customize or replace these in `config.yaml` (see [Configuration](#configuration)).
143
169
 
144
170
  ### 6 New MCP Tools (12 Total)
145
171
 
@@ -167,7 +193,7 @@ Knowledge RAG is a **100% local** hybrid search system that integrates with Clau
167
193
  - **Zero External Dependencies**: Everything runs in-process. No Ollama, no API keys, no servers to manage.
168
194
  - **Hybrid Search + Reranking**: Semantic embeddings + BM25 keywords fused with RRF, then reranked by a cross-encoder for maximum precision.
169
195
  - **Markdown-Aware**: `.md` files are chunked by section headers, preserving semantic coherence.
170
- - **Query Expansion**: 54 security-term synonyms ensure abbreviated queries find relevant content.
196
+ - **Query Expansion**: Customizable synonym mappings ensure abbreviated queries find relevant content (69 security terms included as preset).
171
197
  - **Privacy First**: All processing happens locally. No data leaves your machine.
172
198
  - **Multi-Format**: Supports MD, PDF, DOCX, XLSX, PPTX, CSV, TXT, Python, JSON files.
173
199
  - **Smart Routing**: Keyword-based routing with word boundaries for accurate category filtering.
@@ -182,7 +208,8 @@ Knowledge RAG is a **100% local** hybrid search system that integrates with Clau
182
208
  |---------|-------------|
183
209
  | **Hybrid Search** | Semantic + BM25 keyword search with Reciprocal Rank Fusion |
184
210
  | **Cross-Encoder Reranker** | Xenova/ms-marco-MiniLM-L-6-v2 re-scores top candidates for precision |
185
- | **Query Expansion** | 54 security-term synonym mappings (sqli, privesc, pth, etc.) |
211
+ | **YAML Configuration** | Fully customizable via `config.yaml` with domain-specific presets |
212
+ | **Query Expansion** | 69 security-term synonym mappings (sqli, privesc, pth, etc.) — customizable |
186
213
  | **Markdown-Aware Chunking** | `.md` files split by `##`/`###` sections instead of fixed windows |
187
214
  | **In-Process Embeddings** | FastEmbed ONNX Runtime (BAAI/bge-small-en-v1.5, 384D) |
188
215
  | **Keyword Routing** | Word-boundary aware routing for domain-specific queries |
@@ -387,10 +414,15 @@ flowchart LR
387
414
  # Clone to your home directory
388
415
  git clone https://github.com/lyonzin/knowledge-rag.git ~/knowledge-rag
389
416
  cd ~/knowledge-rag
417
+
418
+ # Create virtual environment and install
419
+ python3 -m venv venv
420
+ source venv/bin/activate # Linux/macOS
421
+ # .\venv\Scripts\activate # Windows
390
422
  pip install -r requirements.txt
391
423
  ```
392
424
 
393
- > **Windows users**: `~/knowledge-rag` becomes `C:\Users\YourName\knowledge-rag`
425
+ > **Windows users**: `~/knowledge-rag` becomes `C:\Users\YourName\knowledge-rag`. Use `python` instead of `python3`.
394
426
 
395
427
  **Step 2: Configure Claude Code**
396
428
 
@@ -398,9 +430,11 @@ From inside the cloned folder, run:
398
430
 
399
431
  ```bash
400
432
  cd ~/knowledge-rag
401
- claude mcp add knowledge-rag -s user -- python -m mcp_server.server
433
+ claude mcp add knowledge-rag -s user -- ~/knowledge-rag/venv/bin/python -m mcp_server.server
402
434
  ```
403
435
 
436
+ > **Windows**: `claude mcp add knowledge-rag -s user -- cmd /c "cd /d %USERPROFILE%\knowledge-rag && venv\Scripts\python -m mcp_server.server"`
437
+
404
438
  That's it. Claude Code now knows about your RAG server.
405
439
 
406
440
  <details>
@@ -428,14 +462,15 @@ Add to `~/.claude.json`:
428
462
  "mcpServers": {
429
463
  "knowledge-rag": {
430
464
  "type": "stdio",
431
- "command": "python",
465
+ "command": "/home/YOUR_USER/knowledge-rag/venv/bin/python",
432
466
  "args": ["-m", "mcp_server.server"],
433
- "cwd": "~/knowledge-rag",
467
+ "cwd": "/home/YOUR_USER/knowledge-rag",
434
468
  "env": {}
435
469
  }
436
470
  }
437
471
  }
438
472
  ```
473
+ > Replace `YOUR_USER` with your username, or use the full path from `echo $HOME`.
439
474
  </details>
440
475
 
441
476
  **Step 3: Restart Claude Code**
@@ -535,7 +570,7 @@ evaluate_retrieval(test_cases='[
535
570
 
536
571
  ## API Reference
537
572
 
538
- ### Existing Tools (6)
573
+ ### Search & Query
539
574
 
540
575
  #### `search_knowledge`
541
576
 
@@ -675,7 +710,7 @@ Get statistics about the knowledge base index.
675
710
 
676
711
  ---
677
712
 
678
- ### New Tools (6)
713
+ ### Document Management
679
714
 
680
715
  #### `add_document`
681
716
 
@@ -826,76 +861,210 @@ Evaluate retrieval quality with test queries. Useful for tuning `hybrid_alpha`,
826
861
 
827
862
  ## Configuration
828
863
 
829
- All configuration lives in `mcp_server/config.py` via the `Config` dataclass.
864
+ Knowledge RAG is fully configurable via a `config.yaml` file in the project root. If no `config.yaml` exists, sensible defaults are used — the system works out of the box with zero configuration.
830
865
 
831
- ### Embedding Model
866
+ ### Quick Start
832
867
 
833
- ```python
834
- embedding_model: str = "BAAI/bge-small-en-v1.5"
835
- embedding_dim: int = 384
868
+ ```bash
869
+ # Option 1: Use a preset
870
+ cp presets/cybersecurity.yaml config.yaml # Offensive/defensive security, CTFs
871
+ cp presets/developer.yaml config.yaml # Software engineering, APIs, DevOps
872
+ cp presets/research.yaml config.yaml # Academic research, papers, studies
873
+ cp presets/general.yaml config.yaml # Blank slate, pure semantic search
874
+
875
+ # Option 2: Start from the documented template
876
+ cp config.example.yaml config.yaml
877
+ # Edit config.yaml to your needs
836
878
  ```
837
879
 
838
- FastEmbed supports any model from its [model list](https://qdrant.github.io/fastembed/examples/Supported_Models/). To change the model, update `embedding_model` and `embedding_dim` in `config.py`, then run `reindex_documents(full_rebuild=True)` to rebuild with the new model.
880
+ Restart Claude Code after changing `config.yaml`.
881
+
882
+ ### config.yaml Structure
883
+
884
+ ```yaml
885
+ # Paths — where your documents live
886
+ paths:
887
+ documents_dir: "./documents" # Scanned recursively
888
+ data_dir: "./data" # Index storage
889
+
890
+ # Documents — what gets indexed and how
891
+ documents:
892
+ supported_formats: # File types to index
893
+ - .md
894
+ - .txt
895
+ - .pdf
896
+ - .docx
897
+ # - .py # Uncomment to index code
898
+ chunking:
899
+ chunk_size: 1000 # Max chars per chunk
900
+ chunk_overlap: 200 # Shared chars between chunks
901
+
902
+ # Models — AI models for search (all run locally, no API keys)
903
+ models:
904
+ embedding:
905
+ model: "BAAI/bge-small-en-v1.5" # ONNX, ~33MB, auto-downloaded
906
+ dimensions: 384
907
+ reranker:
908
+ enabled: true # Set false on low-resource machines
909
+ model: "Xenova/ms-marco-MiniLM-L-6-v2"
910
+ top_k_multiplier: 3 # Candidates fetched before reranking
911
+
912
+ # Search — result limits and collection name
913
+ search:
914
+ default_results: 5
915
+ max_results: 20
916
+ collection_name: "knowledge_base" # Change for separate knowledge bases
917
+
918
+ # Categories — auto-tag documents by folder path
919
+ # Set to {} to disable categorization entirely
920
+ category_mappings:
921
+ "security/redteam": "redteam"
922
+ "security/blueteam": "blueteam"
923
+ "notes": "notes"
924
+
925
+ # Keyword routing — prioritize categories based on query keywords
926
+ # Set to {} for pure semantic search with no routing bias
927
+ keyword_routes:
928
+ redteam:
929
+ - pentest
930
+ - exploit
931
+ - privilege escalation
932
+
933
+ # Query expansion — expand abbreviations for better BM25 recall
934
+ # Set to {} for no expansion (search terms used as-is)
935
+ query_expansions:
936
+ sqli:
937
+ - sql injection
938
+ - sqli
939
+ privesc:
940
+ - privilege escalation
941
+ - privesc
942
+ ```
839
943
 
840
- ### Cross-Encoder Reranker
944
+ > See `config.example.yaml` for the fully documented template with explanations for every field.
841
945
 
842
- ```python
843
- reranker_model: str = "Xenova/ms-marco-MiniLM-L-6-v2"
844
- reranker_enabled: bool = True
845
- reranker_top_k_multiplier: int = 3 # Retrieve 3x candidates for reranking
846
- ```
946
+ ### Presets
847
947
 
848
- The reranker fetches `max_results * reranker_top_k_multiplier` candidates from RRF fusion, re-scores them with the cross-encoder, and returns the top `max_results`. Set `reranker_enabled = False` to disable reranking and use RRF scores directly.
948
+ Pre-built configurations for common use cases. Each preset is a complete `config.yaml` ready to use:
849
949
 
850
- ### Query Expansion
950
+ | Preset | File | Categories | Keywords | Expansions | Best For |
951
+ |--------|------|-----------|----------|-----------|----------|
952
+ | **Cybersecurity** | `presets/cybersecurity.yaml` | 8 | 200+ | 69 | Red/Blue Team, CTFs, threat hunting, exploit dev |
953
+ | **Developer** | `presets/developer.yaml` | 9 | 150+ | 50+ | Full-stack dev, APIs, DevOps, cloud, databases |
954
+ | **Research** | `presets/research.yaml` | 9 | 100+ | 40+ | Academic papers, thesis, lab notebooks, datasets |
955
+ | **General** | `presets/general.yaml` | 0 | 0 | 0 | Blank slate — pure semantic search, no domain logic |
851
956
 
852
- ```python
853
- query_expansions: Dict[str, List[str]] = {
854
- "sqli": ["sql injection", "sqli"],
855
- "privesc": ["privilege escalation", "privesc"],
856
- "pth": ["pass-the-hash", "pth"],
857
- "mimikatz": ["mimikatz", "sekurlsa", "logonpasswords"],
858
- # ... 54 total mappings
859
- }
957
+ **Creating your own preset**: Copy `config.example.yaml`, fill in your categories/keywords/expansions, save to `presets/your-domain.yaml`. Share it with the community via PR.
958
+
959
+ ### Configuration Reference
960
+
961
+ #### Paths
962
+
963
+ | Field | Default | Description |
964
+ |-------|---------|-------------|
965
+ | `paths.documents_dir` | `./documents` | Root folder scanned recursively for documents |
966
+ | `paths.data_dir` | `./data` | Internal storage for ChromaDB and index metadata |
967
+
968
+ Relative paths resolve from the project root. Absolute paths work too. The `KNOWLEDGE_RAG_DIR` environment variable overrides the project root.
969
+
970
+ #### Documents
971
+
972
+ | Field | Default | Description |
973
+ |-------|---------|-------------|
974
+ | `documents.supported_formats` | .md .txt .pdf .py .json .docx .xlsx .pptx .csv | File extensions to index |
975
+ | `documents.chunking.chunk_size` | 1000 | Max characters per chunk |
976
+ | `documents.chunking.chunk_overlap` | 200 | Characters shared between consecutive chunks |
977
+
978
+ **Chunking guidelines**: Short notes → 500/100. General use → 1000/200. Long technical docs → 1500/300.
979
+
980
+ For `.md` files, chunking splits at `##` and `###` header boundaries first. Sections larger than `chunk_size` are sub-chunked with overlap. Non-markdown files use fixed-size chunking.
981
+
982
+ #### Models
983
+
984
+ | Field | Default | Description |
985
+ |-------|---------|-------------|
986
+ | `models.embedding.model` | `BAAI/bge-small-en-v1.5` | Embedding model (ONNX, runs locally) |
987
+ | `models.embedding.dimensions` | 384 | Vector dimensions (must match model) |
988
+ | `models.reranker.enabled` | true | Enable cross-encoder reranking |
989
+ | `models.reranker.model` | `Xenova/ms-marco-MiniLM-L-6-v2` | Reranker model |
990
+ | `models.reranker.top_k_multiplier` | 3 | Fetch N*multiplier candidates for reranking |
991
+
992
+ **Embedding model options** (fastest → most accurate):
993
+ - `BAAI/bge-small-en-v1.5` — 384D, ~33MB (default)
994
+ - `BAAI/bge-base-en-v1.5` — 768D, ~130MB
995
+ - `BAAI/bge-large-en-v1.5` — 1024D, ~335MB
996
+ - `intfloat/multilingual-e5-small` — 384D, 100+ languages
997
+
998
+ > **Warning**: Changing the embedding model after indexing requires `reindex_documents(full_rebuild=True)`.
999
+
1000
+ The reranker fetches `max_results * top_k_multiplier` candidates from RRF fusion, re-scores them with the cross-encoder, and returns the top `max_results`. Set `enabled: false` to disable and use RRF scores directly.
1001
+
1002
+ #### Search
1003
+
1004
+ | Field | Default | Description |
1005
+ |-------|---------|-------------|
1006
+ | `search.default_results` | 5 | Results returned when no limit specified |
1007
+ | `search.max_results` | 20 | Hard cap even if client requests more |
1008
+ | `search.collection_name` | `knowledge_base` | ChromaDB collection — change for separate KBs |
1009
+
1010
+ #### Categories
1011
+
1012
+ Map folder paths to category names. Documents in matching folders get auto-tagged, enabling filtered searches.
1013
+
1014
+ ```yaml
1015
+ # Most specific match wins:
1016
+ # documents/security/redteam/exploit.md → "redteam"
1017
+ # documents/security/overview.md → "security"
1018
+ category_mappings:
1019
+ "security/redteam": "redteam"
1020
+ "security": "security"
860
1021
  ```
861
1022
 
862
- Query expansion runs on BM25 queries only (before tokenization). Add custom mappings to `query_expansions` in `config.py` for your domain. Supports single tokens, bigrams, and full query matches.
1023
+ Set `category_mappings: {}` to disable — documents are still searchable, just without category filters.
863
1024
 
864
- ### Chunking
1025
+ #### Keyword Routing
865
1026
 
866
- ```python
867
- chunk_size: int = 1000 # Characters per chunk (for non-markdown files)
868
- chunk_overlap: int = 200 # Overlap between chunks
1027
+ Route queries to categories based on keywords. When a query contains listed keywords, results from that category are prioritized (not filtered — other categories still appear, ranked lower).
1028
+
1029
+ ```yaml
1030
+ keyword_routes:
1031
+ redteam:
1032
+ - pentest
1033
+ - exploit
1034
+ - sqli
869
1035
  ```
870
1036
 
871
- For `.md` files, chunking splits at `##` and `###` header boundaries first. Sections larger than `chunk_size` are sub-chunked with overlap. Non-markdown files use fixed-size chunking with overlap.
1037
+ Single-word keywords use regex word boundaries (`\b`) — "api" won't match "RAPID". Multi-word keywords use substring matching. When multiple keywords match, the category with the most matches wins.
872
1038
 
873
- ### Keyword Routing
1039
+ Set `keyword_routes: {}` for pure semantic search.
874
1040
 
875
- ```python
876
- keyword_routes: Dict[str, List[str]] = {
877
- "redteam": ["pentest", "exploit", "mimikatz", "sqli", "xss", ...],
878
- "security": ["anti-bot", "waf bypass", "cloudflare", ...],
879
- "logscale": ["logscale", "lql", "formattime", ...],
880
- "ctf": ["ctf", "flag", "hackthebox", "tryhackme", ...],
881
- "development": ["python", "typescript", "api", ...],
882
- "blueteam": ["detection", "sigma", "yara", ...],
883
- }
1041
+ #### Query Expansion
1042
+
1043
+ Expand search terms with synonyms before BM25 search. Supports single tokens, bigrams, and full query matches.
1044
+
1045
+ ```yaml
1046
+ query_expansions:
1047
+ sqli:
1048
+ - sql injection
1049
+ - sqli
1050
+ k8s:
1051
+ - kubernetes
1052
+ - k8s
884
1053
  ```
885
1054
 
886
- Single-word keywords use regex word boundaries (`\b`) to prevent false positives (e.g., "api" won't match "RAPID"). Multi-word keywords use substring matching. When multiple keywords match, the category with the most matches wins.
1055
+ Set `query_expansions: {}` for no expansion.
887
1056
 
888
1057
  ### Hybrid Search Tuning
889
1058
 
890
- | hybrid_alpha | Behavior | Speed | Best For |
891
- |--------------|----------|-------|----------|
892
- | 0.0 | Pure BM25 keyword | **Instant** | Exact terms, CVEs, tool names |
893
- | 0.3 | Keyword-heavy **(default)** | Fast | Technical queries with specific terms |
894
- | 0.5 | Balanced | Medium | General queries |
895
- | 0.7 | Semantic-heavy | Medium | Conceptual queries, related topics |
896
- | 1.0 | Pure semantic | Medium | "How to..." questions, abstract concepts |
1059
+ | hybrid_alpha | Behavior | Best For |
1060
+ |--------------|----------|----------|
1061
+ | 0.0 | Pure BM25 keyword | Exact terms, CVEs, tool names |
1062
+ | 0.3 | Keyword-heavy **(default)** | Technical queries with specific terms |
1063
+ | 0.5 | Balanced | General queries |
1064
+ | 0.7 | Semantic-heavy | Conceptual queries, related topics |
1065
+ | 1.0 | Pure semantic | "How to..." questions, abstract concepts |
897
1066
 
898
- > In v3.0, all `hybrid_alpha` values have similar speed because FastEmbed runs in-process. The speed difference between 0.0 and 1.0 is minimal compared to v2.x where Ollama added network overhead.
1067
+ > All `hybrid_alpha` values have similar speed because FastEmbed runs in-process.
899
1068
 
900
1069
  ---
901
1070
 
@@ -905,25 +1074,25 @@ Single-word keywords use regex word boundaries (`\b`) to prevent false positives
905
1074
  knowledge-rag/
906
1075
  ├── mcp_server/
907
1076
  │ ├── __init__.py
908
- │ ├── config.py # Configuration: models, chunking, routing, expansion
909
- │ ├── ingestion.py # Document parsing, markdown-aware chunking, metadata
910
- │ └── server.py # MCP server, ChromaDB, BM25, reranker, 12 tools
911
- ├── documents/ # Your documents go here
912
- │ ├── security/ # Security, pentest, exploits
913
- │ ├── development/ # Code, APIs, frameworks
914
- │ ├── ctf/ # CTF writeups
915
- │ ├── logscale/ # LogScale/LQL docs
916
- │ └── general/ # Everything else
1077
+ │ ├── config.py # YAML config loader + defaults
1078
+ │ ├── ingestion.py # Document parsing, chunking, metadata extraction
1079
+ │ └── server.py # MCP server, ChromaDB, BM25, reranker, 12 tools
1080
+ ├── config.example.yaml # Documented config template (copy to config.yaml)
1081
+ ├── config.yaml # Your active configuration (git-ignored)
1082
+ ├── presets/ # Ready-to-use domain configurations
1083
+ │ ├── cybersecurity.yaml # Red/Blue Team, CTFs, threat hunting
1084
+ │ ├── developer.yaml # Software engineering, APIs, DevOps
1085
+ │ ├── research.yaml # Academic research, papers, studies
1086
+ │ └── general.yaml # Blank slate, zero domain logic
1087
+ ├── documents/ # Your documents (scanned recursively)
917
1088
  ├── data/
918
- │ ├── chroma_db/ # ChromaDB vector database storage
919
- │ └── index_metadata.json # Incremental indexing metadata
920
- ├── .claude/
921
- │ └── mcp.json # Project MCP config (optional)
922
- ├── venv/ # Python virtual environment
923
- ├── requirements.txt # Python dependencies
924
- ├── CHANGELOG.md # Version history
925
- ├── LICENSE # MIT License
926
- └── README.md # This file
1089
+ │ ├── chroma_db/ # ChromaDB vector database
1090
+ │ └── index_metadata.json # Incremental indexing state
1091
+ ├── tests/ # Test suite (55 tests)
1092
+ ├── venv/ # Python virtual environment
1093
+ ├── requirements.txt # Python dependencies
1094
+ ├── LICENSE # MIT License
1095
+ └── README.md
927
1096
  ```
928
1097
 
929
1098
  ---
@@ -1024,6 +1193,29 @@ With ~200 documents, expect ~300-500MB RAM. The embedding model (~50MB) and rera
1024
1193
 
1025
1194
  ## Changelog
1026
1195
 
1196
+ ### v3.3.0 (2026-04-05)
1197
+
1198
+ - **NEW**: YAML configuration system — fully customizable via `config.yaml`
1199
+ - **NEW**: Domain presets — `presets/cybersecurity.yaml`, `presets/developer.yaml`, `presets/research.yaml`, `presets/general.yaml`
1200
+ - **NEW**: `config.example.yaml` — documented template with explanations for every field
1201
+ - **NEW**: Categories, keyword routing, and query expansions are now user-configurable (no more editing `config.py`)
1202
+ - **NEW**: Empty config = pure semantic search with zero domain logic (ideal for generic use)
1203
+ - **NEW**: Warning log for empty files during indexing (previously silent skip)
1204
+ - **IMPROVED**: README rewritten — full configuration reference, preset documentation, updated project structure
1205
+ - **IMPROVED**: `pyyaml` added as dependency
1206
+ - **BREAKING**: None — system works identically without `config.yaml` (backwards compatible)
1207
+
1208
+ ### v3.2.4 (2026-04-03)
1209
+
1210
+ - **NEW**: Symlink support — `documents/` directory now follows symbolic links recursively ([#13](https://github.com/lyonzin/knowledge-rag/issues/13))
1211
+ - **NEW**: Circular symlink loop protection via realpath deduplication
1212
+ - **IMPROVED**: `_has_documents()` detection now validates against supported formats only (stricter than before)
1213
+
1214
+ ### v3.2.3 (2026-03-22)
1215
+
1216
+ - **FIX**: BASE_DIR smart detection — checks for actual files in `documents/` (not just dir existence)
1217
+ - **FIX**: Prevents false positive when `site-packages/documents/` exists as empty dir
1218
+
1027
1219
  ### v3.2.2 (2026-03-22)
1028
1220
 
1029
1221
  - **FIX**: `pip install knowledge-rag` now works as plug-and-play (BASE_DIR falls back to CWD when installed from PyPI)
@@ -1068,7 +1260,7 @@ With ~200 documents, expect ~300-500MB RAM. The embedding model (~50MB) and rera
1068
1260
  - **BREAKING**: Changed embedding model from nomic-embed-text (768D) to BAAI/bge-small-en-v1.5 (384D)
1069
1261
  - **NEW**: Cross-encoder reranker (Xenova/ms-marco-MiniLM-L-6-v2) applied after RRF fusion
1070
1262
  - **NEW**: Markdown-aware chunking — `.md` files split by `##`/`###` sections
1071
- - **NEW**: Query expansion with 54 security-term synonym mappings
1263
+ - **NEW**: Query expansion with 69 security-term synonym mappings
1072
1264
  - **NEW**: `add_document` — add document from raw content string
1073
1265
  - **NEW**: `update_document` — update existing document (re-chunks and re-indexes)
1074
1266
  - **NEW**: `remove_document` — remove document from index (optionally delete file)
@@ -1152,6 +1344,7 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
1152
1344
  - [FastMCP](https://github.com/anthropics/mcp) — Model Context Protocol framework
1153
1345
  - [PyMuPDF](https://pymupdf.readthedocs.io/) — PDF parsing
1154
1346
  - [rank-bm25](https://github.com/dorianbrown/rank_bm25) — BM25 Okapi implementation
1347
+ - [PyYAML](https://pyyaml.org/) — YAML configuration parsing
1155
1348
  - [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/) — HTML parsing for URL ingestion
1156
1349
 
1157
1350
  ---