knowledge-rag 3.2.3__tar.gz → 3.3.1__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -4,6 +4,9 @@ __pycache__/
4
4
  *.pyc
5
5
  data/
6
6
 
7
+ # User config (personal settings — use presets/ as starting point)
8
+ config.yaml
9
+
7
10
  # Personal documents (NEVER commit — user-populated content)
8
11
  documents/aar/
9
12
  documents/security/
@@ -50,3 +53,5 @@ desktop.ini
50
53
  *.swp
51
54
  *.swo
52
55
  dist/
56
+ .ruff_cache/
57
+ .pytest_cache/
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: knowledge-rag
3
- Version: 3.2.3
3
+ Version: 3.3.1
4
4
  Summary: Local RAG System for Claude Code — Hybrid search + Cross-encoder Reranking + 12 MCP Tools. Zero external servers.
5
5
  Project-URL: Homepage, https://github.com/lyonzin/knowledge-rag
6
6
  Project-URL: Repository, https://github.com/lyonzin/knowledge-rag
@@ -28,6 +28,7 @@ Requires-Dist: openpyxl>=3.1.0
28
28
  Requires-Dist: pymupdf>=1.23.0
29
29
  Requires-Dist: python-docx>=1.0.0
30
30
  Requires-Dist: python-pptx>=1.0.0
31
+ Requires-Dist: pyyaml>=6.0
31
32
  Requires-Dist: rank-bm25>=0.2.2
32
33
  Requires-Dist: requests>=2.31.0
33
34
  Requires-Dist: watchdog>=4.0.0
@@ -37,7 +38,7 @@ Description-Content-Type: text/markdown
37
38
 
38
39
  <div align="center">
39
40
 
40
- ![Version](https://img.shields.io/badge/version-3.2.2-blue.svg)
41
+ ![Version](https://img.shields.io/badge/version-3.3.1-blue.svg)
41
42
  ![Python](https://img.shields.io/badge/python-3.11%20%7C%203.12-green.svg)
42
43
  ![License](https://img.shields.io/badge/license-MIT-yellow.svg)
43
44
  ![Platform](https://img.shields.io/badge/platform-Windows%20%7C%20Linux%20%7C%20macOS-lightgrey.svg)
@@ -60,7 +61,7 @@ Your documents become instantly searchable inside Claude Code — with reranking
60
61
 
61
62
  **12 MCP Tools** | **Hybrid Search + Cross-Encoder Reranking** | **Markdown-Aware Chunking** | **100% Local, Zero Cloud**
62
63
 
63
- [What's New](#whats-new-in-v310) | [Installation](#installation) | [API Reference](#api-reference) | [Architecture](#architecture)
64
+ [What's New](#whats-new-in-v330) | [Installation](#installation) | [Configuration](#configuration) | [API Reference](#api-reference) | [Architecture](#architecture)
64
65
 
65
66
  </div>
66
67
 
@@ -107,6 +108,31 @@ After the initial rebuild, startup and queries are faster than v2.x because ther
107
108
 
108
109
  ---
109
110
 
111
+ ## What's New in v3.3.0
112
+
113
+ ### YAML Configuration System
114
+
115
+ All settings are now customizable via `config.yaml` — no more editing Python code. Categories, keyword routing, query expansions, models, chunking, and paths are all configurable through a single YAML file.
116
+
117
+ ### Domain Presets
118
+
119
+ Four ready-to-use presets ship with the project. Copy one to `config.yaml` and you're done:
120
+
121
+ - **Cybersecurity** — 8 categories, 200+ keywords, 69 query expansions (red team, blue team, CTFs, threat hunting)
122
+ - **Developer** — 9 categories, 150+ keywords, 50+ expansions (full-stack, APIs, DevOps, cloud, databases)
123
+ - **Research** — 9 categories, 100+ keywords, 40+ expansions (academic papers, thesis, lab notebooks)
124
+ - **General** — Zero routing, zero expansions. Pure semantic search for any domain.
125
+
126
+ ### Generic Use Support
127
+
128
+ With `category_mappings: {}`, `keyword_routes: {}`, and `query_expansions: {}`, the system operates as a domain-agnostic semantic search engine. No security-specific logic unless you want it.
129
+
130
+ ### Backwards Compatible
131
+
132
+ No `config.yaml`? The system uses built-in defaults — identical behavior to v3.2.x. Zero migration required.
133
+
134
+ ---
135
+
110
136
  ## What's New in v3.1.0
111
137
 
112
138
  ### Office Document Support (DOCX, XLSX, PPTX, CSV)
@@ -139,7 +165,7 @@ After hybrid RRF fusion produces initial candidates, a cross-encoder (Xenova/ms-
139
165
 
140
166
  ### Query Expansion
141
167
 
142
- 54 security-term synonym mappings expand abbreviated queries before BM25 search. Searching for "sqli" automatically includes "sql injection"; "privesc" includes "privilege escalation"; "pth" includes "pass-the-hash". The full expansion table is in `config.py`.
168
+ 69 security-term synonym mappings expand abbreviated queries before BM25 search. Searching for "sqli" automatically includes "sql injection"; "privesc" includes "privilege escalation"; "pth" includes "pass-the-hash". Customize or replace these in `config.yaml` (see [Configuration](#configuration)).
143
169
 
144
170
  ### 6 New MCP Tools (12 Total)
145
171
 
@@ -167,7 +193,7 @@ Knowledge RAG is a **100% local** hybrid search system that integrates with Clau
167
193
  - **Zero External Dependencies**: Everything runs in-process. No Ollama, no API keys, no servers to manage.
168
194
  - **Hybrid Search + Reranking**: Semantic embeddings + BM25 keywords fused with RRF, then reranked by a cross-encoder for maximum precision.
169
195
  - **Markdown-Aware**: `.md` files are chunked by section headers, preserving semantic coherence.
170
- - **Query Expansion**: 54 security-term synonyms ensure abbreviated queries find relevant content.
196
+ - **Query Expansion**: Customizable synonym mappings ensure abbreviated queries find relevant content (69 security terms included as preset).
171
197
  - **Privacy First**: All processing happens locally. No data leaves your machine.
172
198
  - **Multi-Format**: Supports MD, PDF, DOCX, XLSX, PPTX, CSV, TXT, Python, JSON files.
173
199
  - **Smart Routing**: Keyword-based routing with word boundaries for accurate category filtering.
@@ -182,7 +208,8 @@ Knowledge RAG is a **100% local** hybrid search system that integrates with Clau
182
208
  |---------|-------------|
183
209
  | **Hybrid Search** | Semantic + BM25 keyword search with Reciprocal Rank Fusion |
184
210
  | **Cross-Encoder Reranker** | Xenova/ms-marco-MiniLM-L-6-v2 re-scores top candidates for precision |
185
- | **Query Expansion** | 54 security-term synonym mappings (sqli, privesc, pth, etc.) |
211
+ | **YAML Configuration** | Fully customizable via `config.yaml` with domain-specific presets |
212
+ | **Query Expansion** | 69 security-term synonym mappings (sqli, privesc, pth, etc.) — customizable |
186
213
  | **Markdown-Aware Chunking** | `.md` files split by `##`/`###` sections instead of fixed windows |
187
214
  | **In-Process Embeddings** | FastEmbed ONNX Runtime (BAAI/bge-small-en-v1.5, 384D) |
188
215
  | **Keyword Routing** | Word-boundary aware routing for domain-specific queries |
@@ -381,16 +408,23 @@ flowchart LR
381
408
 
382
409
  ### Quick Start (3 steps)
383
410
 
384
- **Step 1: Clone and install**
411
+ **Step 1: Install**
385
412
 
386
413
  ```bash
387
- # Clone to your home directory
414
+ # Option A: pip install (recommended)
415
+ pip install knowledge-rag
416
+ knowledge-rag init # Exports config template, presets, creates documents/
417
+
418
+ # Option B: Clone from source
388
419
  git clone https://github.com/lyonzin/knowledge-rag.git ~/knowledge-rag
389
420
  cd ~/knowledge-rag
421
+ python3 -m venv venv
422
+ source venv/bin/activate # Linux/macOS
423
+ # .\venv\Scripts\activate # Windows
390
424
  pip install -r requirements.txt
391
425
  ```
392
426
 
393
- > **Windows users**: `~/knowledge-rag` becomes `C:\Users\YourName\knowledge-rag`
427
+ > **Windows users**: Use `python` instead of `python3`.
394
428
 
395
429
  **Step 2: Configure Claude Code**
396
430
 
@@ -398,9 +432,11 @@ From inside the cloned folder, run:
398
432
 
399
433
  ```bash
400
434
  cd ~/knowledge-rag
401
- claude mcp add knowledge-rag -s user -- python -m mcp_server.server
435
+ claude mcp add knowledge-rag -s user -- ~/knowledge-rag/venv/bin/python -m mcp_server.server
402
436
  ```
403
437
 
438
+ > **Windows**: `claude mcp add knowledge-rag -s user -- cmd /c "cd /d %USERPROFILE%\knowledge-rag && venv\Scripts\python -m mcp_server.server"`
439
+
404
440
  That's it. Claude Code now knows about your RAG server.
405
441
 
406
442
  <details>
@@ -428,14 +464,15 @@ Add to `~/.claude.json`:
428
464
  "mcpServers": {
429
465
  "knowledge-rag": {
430
466
  "type": "stdio",
431
- "command": "python",
467
+ "command": "/home/YOUR_USER/knowledge-rag/venv/bin/python",
432
468
  "args": ["-m", "mcp_server.server"],
433
- "cwd": "~/knowledge-rag",
469
+ "cwd": "/home/YOUR_USER/knowledge-rag",
434
470
  "env": {}
435
471
  }
436
472
  }
437
473
  }
438
474
  ```
475
+ > Replace `YOUR_USER` with your username, or use the full path from `echo $HOME`.
439
476
  </details>
440
477
 
441
478
  **Step 3: Restart Claude Code**
@@ -535,7 +572,7 @@ evaluate_retrieval(test_cases='[
535
572
 
536
573
  ## API Reference
537
574
 
538
- ### Existing Tools (6)
575
+ ### Search & Query
539
576
 
540
577
  #### `search_knowledge`
541
578
 
@@ -675,7 +712,7 @@ Get statistics about the knowledge base index.
675
712
 
676
713
  ---
677
714
 
678
- ### New Tools (6)
715
+ ### Document Management
679
716
 
680
717
  #### `add_document`
681
718
 
@@ -826,76 +863,210 @@ Evaluate retrieval quality with test queries. Useful for tuning `hybrid_alpha`,
826
863
 
827
864
  ## Configuration
828
865
 
829
- All configuration lives in `mcp_server/config.py` via the `Config` dataclass.
866
+ Knowledge RAG is fully configurable via a `config.yaml` file in the project root. If no `config.yaml` exists, sensible defaults are used — the system works out of the box with zero configuration.
830
867
 
831
- ### Embedding Model
868
+ ### Quick Start
832
869
 
833
- ```python
834
- embedding_model: str = "BAAI/bge-small-en-v1.5"
835
- embedding_dim: int = 384
870
+ ```bash
871
+ # Option 1: Use a preset
872
+ cp presets/cybersecurity.yaml config.yaml # Offensive/defensive security, CTFs
873
+ cp presets/developer.yaml config.yaml # Software engineering, APIs, DevOps
874
+ cp presets/research.yaml config.yaml # Academic research, papers, studies
875
+ cp presets/general.yaml config.yaml # Blank slate, pure semantic search
876
+
877
+ # Option 2: Start from the documented template
878
+ cp config.example.yaml config.yaml
879
+ # Edit config.yaml to your needs
836
880
  ```
837
881
 
838
- FastEmbed supports any model from its [model list](https://qdrant.github.io/fastembed/examples/Supported_Models/). To change the model, update `embedding_model` and `embedding_dim` in `config.py`, then run `reindex_documents(full_rebuild=True)` to rebuild with the new model.
882
+ Restart Claude Code after changing `config.yaml`.
883
+
884
+ ### config.yaml Structure
885
+
886
+ ```yaml
887
+ # Paths — where your documents live
888
+ paths:
889
+ documents_dir: "./documents" # Scanned recursively
890
+ data_dir: "./data" # Index storage
891
+
892
+ # Documents — what gets indexed and how
893
+ documents:
894
+ supported_formats: # File types to index
895
+ - .md
896
+ - .txt
897
+ - .pdf
898
+ - .docx
899
+ # - .py # Uncomment to index code
900
+ chunking:
901
+ chunk_size: 1000 # Max chars per chunk
902
+ chunk_overlap: 200 # Shared chars between chunks
903
+
904
+ # Models — AI models for search (all run locally, no API keys)
905
+ models:
906
+ embedding:
907
+ model: "BAAI/bge-small-en-v1.5" # ONNX, ~33MB, auto-downloaded
908
+ dimensions: 384
909
+ reranker:
910
+ enabled: true # Set false on low-resource machines
911
+ model: "Xenova/ms-marco-MiniLM-L-6-v2"
912
+ top_k_multiplier: 3 # Candidates fetched before reranking
913
+
914
+ # Search — result limits and collection name
915
+ search:
916
+ default_results: 5
917
+ max_results: 20
918
+ collection_name: "knowledge_base" # Change for separate knowledge bases
919
+
920
+ # Categories — auto-tag documents by folder path
921
+ # Set to {} to disable categorization entirely
922
+ category_mappings:
923
+ "security/redteam": "redteam"
924
+ "security/blueteam": "blueteam"
925
+ "notes": "notes"
926
+
927
+ # Keyword routing — prioritize categories based on query keywords
928
+ # Set to {} for pure semantic search with no routing bias
929
+ keyword_routes:
930
+ redteam:
931
+ - pentest
932
+ - exploit
933
+ - privilege escalation
934
+
935
+ # Query expansion — expand abbreviations for better BM25 recall
936
+ # Set to {} for no expansion (search terms used as-is)
937
+ query_expansions:
938
+ sqli:
939
+ - sql injection
940
+ - sqli
941
+ privesc:
942
+ - privilege escalation
943
+ - privesc
944
+ ```
839
945
 
840
- ### Cross-Encoder Reranker
946
+ > See `config.example.yaml` for the fully documented template with explanations for every field.
841
947
 
842
- ```python
843
- reranker_model: str = "Xenova/ms-marco-MiniLM-L-6-v2"
844
- reranker_enabled: bool = True
845
- reranker_top_k_multiplier: int = 3 # Retrieve 3x candidates for reranking
846
- ```
948
+ ### Presets
847
949
 
848
- The reranker fetches `max_results * reranker_top_k_multiplier` candidates from RRF fusion, re-scores them with the cross-encoder, and returns the top `max_results`. Set `reranker_enabled = False` to disable reranking and use RRF scores directly.
950
+ Pre-built configurations for common use cases. Each preset is a complete `config.yaml` ready to use:
849
951
 
850
- ### Query Expansion
952
+ | Preset | File | Categories | Keywords | Expansions | Best For |
953
+ |--------|------|-----------|----------|-----------|----------|
954
+ | **Cybersecurity** | `presets/cybersecurity.yaml` | 8 | 200+ | 69 | Red/Blue Team, CTFs, threat hunting, exploit dev |
955
+ | **Developer** | `presets/developer.yaml` | 9 | 150+ | 50+ | Full-stack dev, APIs, DevOps, cloud, databases |
956
+ | **Research** | `presets/research.yaml` | 9 | 100+ | 40+ | Academic papers, thesis, lab notebooks, datasets |
957
+ | **General** | `presets/general.yaml` | 0 | 0 | 0 | Blank slate — pure semantic search, no domain logic |
851
958
 
852
- ```python
853
- query_expansions: Dict[str, List[str]] = {
854
- "sqli": ["sql injection", "sqli"],
855
- "privesc": ["privilege escalation", "privesc"],
856
- "pth": ["pass-the-hash", "pth"],
857
- "mimikatz": ["mimikatz", "sekurlsa", "logonpasswords"],
858
- # ... 54 total mappings
859
- }
959
+ **Creating your own preset**: Copy `config.example.yaml`, fill in your categories/keywords/expansions, save to `presets/your-domain.yaml`. Share it with the community via PR.
960
+
961
+ ### Configuration Reference
962
+
963
+ #### Paths
964
+
965
+ | Field | Default | Description |
966
+ |-------|---------|-------------|
967
+ | `paths.documents_dir` | `./documents` | Root folder scanned recursively for documents |
968
+ | `paths.data_dir` | `./data` | Internal storage for ChromaDB and index metadata |
969
+
970
+ Relative paths resolve from the project root. Absolute paths work too. The `KNOWLEDGE_RAG_DIR` environment variable overrides the project root.
971
+
972
+ #### Documents
973
+
974
+ | Field | Default | Description |
975
+ |-------|---------|-------------|
976
+ | `documents.supported_formats` | .md .txt .pdf .py .json .docx .xlsx .pptx .csv | File extensions to index |
977
+ | `documents.chunking.chunk_size` | 1000 | Max characters per chunk |
978
+ | `documents.chunking.chunk_overlap` | 200 | Characters shared between consecutive chunks |
979
+
980
+ **Chunking guidelines**: Short notes → 500/100. General use → 1000/200. Long technical docs → 1500/300.
981
+
982
+ For `.md` files, chunking splits at `##` and `###` header boundaries first. Sections larger than `chunk_size` are sub-chunked with overlap. Non-markdown files use fixed-size chunking.
983
+
984
+ #### Models
985
+
986
+ | Field | Default | Description |
987
+ |-------|---------|-------------|
988
+ | `models.embedding.model` | `BAAI/bge-small-en-v1.5` | Embedding model (ONNX, runs locally) |
989
+ | `models.embedding.dimensions` | 384 | Vector dimensions (must match model) |
990
+ | `models.reranker.enabled` | true | Enable cross-encoder reranking |
991
+ | `models.reranker.model` | `Xenova/ms-marco-MiniLM-L-6-v2` | Reranker model |
992
+ | `models.reranker.top_k_multiplier` | 3 | Fetch N*multiplier candidates for reranking |
993
+
994
+ **Embedding model options** (fastest → most accurate):
995
+ - `BAAI/bge-small-en-v1.5` — 384D, ~33MB (default)
996
+ - `BAAI/bge-base-en-v1.5` — 768D, ~130MB
997
+ - `BAAI/bge-large-en-v1.5` — 1024D, ~335MB
998
+ - `intfloat/multilingual-e5-small` — 384D, 100+ languages
999
+
1000
+ > **Warning**: Changing the embedding model after indexing requires `reindex_documents(full_rebuild=True)`.
1001
+
1002
+ The reranker fetches `max_results * top_k_multiplier` candidates from RRF fusion, re-scores them with the cross-encoder, and returns the top `max_results`. Set `enabled: false` to disable and use RRF scores directly.
1003
+
1004
+ #### Search
1005
+
1006
+ | Field | Default | Description |
1007
+ |-------|---------|-------------|
1008
+ | `search.default_results` | 5 | Results returned when no limit specified |
1009
+ | `search.max_results` | 20 | Hard cap even if client requests more |
1010
+ | `search.collection_name` | `knowledge_base` | ChromaDB collection — change for separate KBs |
1011
+
1012
+ #### Categories
1013
+
1014
+ Map folder paths to category names. Documents in matching folders get auto-tagged, enabling filtered searches.
1015
+
1016
+ ```yaml
1017
+ # Most specific match wins:
1018
+ # documents/security/redteam/exploit.md → "redteam"
1019
+ # documents/security/overview.md → "security"
1020
+ category_mappings:
1021
+ "security/redteam": "redteam"
1022
+ "security": "security"
860
1023
  ```
861
1024
 
862
- Query expansion runs on BM25 queries only (before tokenization). Add custom mappings to `query_expansions` in `config.py` for your domain. Supports single tokens, bigrams, and full query matches.
1025
+ Set `category_mappings: {}` to disable — documents are still searchable, just without category filters.
863
1026
 
864
- ### Chunking
1027
+ #### Keyword Routing
865
1028
 
866
- ```python
867
- chunk_size: int = 1000 # Characters per chunk (for non-markdown files)
868
- chunk_overlap: int = 200 # Overlap between chunks
1029
+ Route queries to categories based on keywords. When a query contains listed keywords, results from that category are prioritized (not filtered — other categories still appear, ranked lower).
1030
+
1031
+ ```yaml
1032
+ keyword_routes:
1033
+ redteam:
1034
+ - pentest
1035
+ - exploit
1036
+ - sqli
869
1037
  ```
870
1038
 
871
- For `.md` files, chunking splits at `##` and `###` header boundaries first. Sections larger than `chunk_size` are sub-chunked with overlap. Non-markdown files use fixed-size chunking with overlap.
1039
+ Single-word keywords use regex word boundaries (`\b`) — "api" won't match "RAPID". Multi-word keywords use substring matching. When multiple keywords match, the category with the most matches wins.
872
1040
 
873
- ### Keyword Routing
1041
+ Set `keyword_routes: {}` for pure semantic search.
874
1042
 
875
- ```python
876
- keyword_routes: Dict[str, List[str]] = {
877
- "redteam": ["pentest", "exploit", "mimikatz", "sqli", "xss", ...],
878
- "security": ["anti-bot", "waf bypass", "cloudflare", ...],
879
- "logscale": ["logscale", "lql", "formattime", ...],
880
- "ctf": ["ctf", "flag", "hackthebox", "tryhackme", ...],
881
- "development": ["python", "typescript", "api", ...],
882
- "blueteam": ["detection", "sigma", "yara", ...],
883
- }
1043
+ #### Query Expansion
1044
+
1045
+ Expand search terms with synonyms before BM25 search. Supports single tokens, bigrams, and full query matches.
1046
+
1047
+ ```yaml
1048
+ query_expansions:
1049
+ sqli:
1050
+ - sql injection
1051
+ - sqli
1052
+ k8s:
1053
+ - kubernetes
1054
+ - k8s
884
1055
  ```
885
1056
 
886
- Single-word keywords use regex word boundaries (`\b`) to prevent false positives (e.g., "api" won't match "RAPID"). Multi-word keywords use substring matching. When multiple keywords match, the category with the most matches wins.
1057
+ Set `query_expansions: {}` for no expansion.
887
1058
 
888
1059
  ### Hybrid Search Tuning
889
1060
 
890
- | hybrid_alpha | Behavior | Speed | Best For |
891
- |--------------|----------|-------|----------|
892
- | 0.0 | Pure BM25 keyword | **Instant** | Exact terms, CVEs, tool names |
893
- | 0.3 | Keyword-heavy **(default)** | Fast | Technical queries with specific terms |
894
- | 0.5 | Balanced | Medium | General queries |
895
- | 0.7 | Semantic-heavy | Medium | Conceptual queries, related topics |
896
- | 1.0 | Pure semantic | Medium | "How to..." questions, abstract concepts |
1061
+ | hybrid_alpha | Behavior | Best For |
1062
+ |--------------|----------|----------|
1063
+ | 0.0 | Pure BM25 keyword | Exact terms, CVEs, tool names |
1064
+ | 0.3 | Keyword-heavy **(default)** | Technical queries with specific terms |
1065
+ | 0.5 | Balanced | General queries |
1066
+ | 0.7 | Semantic-heavy | Conceptual queries, related topics |
1067
+ | 1.0 | Pure semantic | "How to..." questions, abstract concepts |
897
1068
 
898
- > In v3.0, all `hybrid_alpha` values have similar speed because FastEmbed runs in-process. The speed difference between 0.0 and 1.0 is minimal compared to v2.x where Ollama added network overhead.
1069
+ > All `hybrid_alpha` values have similar speed because FastEmbed runs in-process.
899
1070
 
900
1071
  ---
901
1072
 
@@ -905,25 +1076,25 @@ Single-word keywords use regex word boundaries (`\b`) to prevent false positives
905
1076
  knowledge-rag/
906
1077
  ├── mcp_server/
907
1078
  │ ├── __init__.py
908
- │ ├── config.py # Configuration: models, chunking, routing, expansion
909
- │ ├── ingestion.py # Document parsing, markdown-aware chunking, metadata
910
- │ └── server.py # MCP server, ChromaDB, BM25, reranker, 12 tools
911
- ├── documents/ # Your documents go here
912
- │ ├── security/ # Security, pentest, exploits
913
- │ ├── development/ # Code, APIs, frameworks
914
- │ ├── ctf/ # CTF writeups
915
- │ ├── logscale/ # LogScale/LQL docs
916
- │ └── general/ # Everything else
1079
+ │ ├── config.py # YAML config loader + defaults
1080
+ │ ├── ingestion.py # Document parsing, chunking, metadata extraction
1081
+ │ └── server.py # MCP server, ChromaDB, BM25, reranker, 12 tools
1082
+ ├── config.example.yaml # Documented config template (copy to config.yaml)
1083
+ ├── config.yaml # Your active configuration (git-ignored)
1084
+ ├── presets/ # Ready-to-use domain configurations
1085
+ │ ├── cybersecurity.yaml # Red/Blue Team, CTFs, threat hunting
1086
+ │ ├── developer.yaml # Software engineering, APIs, DevOps
1087
+ │ ├── research.yaml # Academic research, papers, studies
1088
+ │ └── general.yaml # Blank slate, zero domain logic
1089
+ ├── documents/ # Your documents (scanned recursively)
917
1090
  ├── data/
918
- │ ├── chroma_db/ # ChromaDB vector database storage
919
- │ └── index_metadata.json # Incremental indexing metadata
920
- ├── .claude/
921
- │ └── mcp.json # Project MCP config (optional)
922
- ├── venv/ # Python virtual environment
923
- ├── requirements.txt # Python dependencies
924
- ├── CHANGELOG.md # Version history
925
- ├── LICENSE # MIT License
926
- └── README.md # This file
1091
+ │ ├── chroma_db/ # ChromaDB vector database
1092
+ │ └── index_metadata.json # Incremental indexing state
1093
+ ├── tests/ # Test suite (55 tests)
1094
+ ├── venv/ # Python virtual environment
1095
+ ├── requirements.txt # Python dependencies
1096
+ ├── LICENSE # MIT License
1097
+ └── README.md
927
1098
  ```
928
1099
 
929
1100
  ---
@@ -1024,6 +1195,35 @@ With ~200 documents, expect ~300-500MB RAM. The embedding model (~50MB) and rera
1024
1195
 
1025
1196
  ## Changelog
1026
1197
 
1198
+ ### v3.3.1 (2026-04-06)
1199
+
1200
+ - **FIX**: YAML null values (`category_mappings:` without value) no longer crash the server — falls back to defaults
1201
+ - **FIX**: Presets and config template now bundled in pip wheel (`knowledge-rag init` exports them)
1202
+ - **NEW**: `knowledge-rag init` CLI command — exports config template, presets, and creates documents/ in current directory
1203
+
1204
+ ### v3.3.0 (2026-04-05)
1205
+
1206
+ - **NEW**: YAML configuration system — fully customizable via `config.yaml`
1207
+ - **NEW**: Domain presets — `presets/cybersecurity.yaml`, `presets/developer.yaml`, `presets/research.yaml`, `presets/general.yaml`
1208
+ - **NEW**: `config.example.yaml` — documented template with explanations for every field
1209
+ - **NEW**: Categories, keyword routing, and query expansions are now user-configurable (no more editing `config.py`)
1210
+ - **NEW**: Empty config = pure semantic search with zero domain logic (ideal for generic use)
1211
+ - **NEW**: Warning log for empty files during indexing (previously silent skip)
1212
+ - **IMPROVED**: README rewritten — full configuration reference, preset documentation, updated project structure
1213
+ - **IMPROVED**: `pyyaml` added as dependency
1214
+ - **BREAKING**: None — system works identically without `config.yaml` (backwards compatible)
1215
+
1216
+ ### v3.2.4 (2026-04-03)
1217
+
1218
+ - **NEW**: Symlink support — `documents/` directory now follows symbolic links recursively ([#13](https://github.com/lyonzin/knowledge-rag/issues/13))
1219
+ - **NEW**: Circular symlink loop protection via realpath deduplication
1220
+ - **IMPROVED**: `_has_documents()` detection now validates against supported formats only (stricter than before)
1221
+
1222
+ ### v3.2.3 (2026-03-22)
1223
+
1224
+ - **FIX**: BASE_DIR smart detection — checks for actual files in `documents/` (not just dir existence)
1225
+ - **FIX**: Prevents false positive when `site-packages/documents/` exists as empty dir
1226
+
1027
1227
  ### v3.2.2 (2026-03-22)
1028
1228
 
1029
1229
  - **FIX**: `pip install knowledge-rag` now works as plug-and-play (BASE_DIR falls back to CWD when installed from PyPI)
@@ -1068,7 +1268,7 @@ With ~200 documents, expect ~300-500MB RAM. The embedding model (~50MB) and rera
1068
1268
  - **BREAKING**: Changed embedding model from nomic-embed-text (768D) to BAAI/bge-small-en-v1.5 (384D)
1069
1269
  - **NEW**: Cross-encoder reranker (Xenova/ms-marco-MiniLM-L-6-v2) applied after RRF fusion
1070
1270
  - **NEW**: Markdown-aware chunking — `.md` files split by `##`/`###` sections
1071
- - **NEW**: Query expansion with 54 security-term synonym mappings
1271
+ - **NEW**: Query expansion with 69 security-term synonym mappings
1072
1272
  - **NEW**: `add_document` — add document from raw content string
1073
1273
  - **NEW**: `update_document` — update existing document (re-chunks and re-indexes)
1074
1274
  - **NEW**: `remove_document` — remove document from index (optionally delete file)
@@ -1152,6 +1352,7 @@ This project is licensed under the MIT License - see the [LICENSE](LICENSE) file
1152
1352
  - [FastMCP](https://github.com/anthropics/mcp) — Model Context Protocol framework
1153
1353
  - [PyMuPDF](https://pymupdf.readthedocs.io/) — PDF parsing
1154
1354
  - [rank-bm25](https://github.com/dorianbrown/rank_bm25) — BM25 Okapi implementation
1355
+ - [PyYAML](https://pyyaml.org/) — YAML configuration parsing
1155
1356
  - [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/) — HTML parsing for URL ingestion
1156
1357
 
1157
1358
  ---