knowledge-rag 3.5.1__tar.gz → 3.6.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {knowledge_rag-3.5.1 → knowledge_rag-3.6.0}/.gitignore +59 -59
- {knowledge_rag-3.5.1 → knowledge_rag-3.6.0}/LICENSE +21 -21
- {knowledge_rag-3.5.1 → knowledge_rag-3.6.0}/PKG-INFO +59 -22
- {knowledge_rag-3.5.1 → knowledge_rag-3.6.0}/README.md +57 -20
- {knowledge_rag-3.5.1 → knowledge_rag-3.6.0}/config.example.yaml +247 -239
- {knowledge_rag-3.5.1 → knowledge_rag-3.6.0}/documents/examples/sample-document.md +36 -36
- {knowledge_rag-3.5.1 → knowledge_rag-3.6.0}/mcp_server/__init__.py +17 -17
- {knowledge_rag-3.5.1 → knowledge_rag-3.6.0}/mcp_server/config.py +595 -531
- {knowledge_rag-3.5.1 → knowledge_rag-3.6.0}/mcp_server/ingestion.py +919 -801
- {knowledge_rag-3.5.1 → knowledge_rag-3.6.0}/mcp_server/server.py +1968 -1925
- knowledge_rag-3.6.0/npm/README.md +61 -0
- {knowledge_rag-3.5.1 → knowledge_rag-3.6.0}/presets/cybersecurity.yaml +367 -363
- {knowledge_rag-3.5.1 → knowledge_rag-3.6.0}/presets/developer.yaml +364 -356
- {knowledge_rag-3.5.1 → knowledge_rag-3.6.0}/presets/general.yaml +112 -112
- {knowledge_rag-3.5.1 → knowledge_rag-3.6.0}/presets/research.yaml +256 -256
- {knowledge_rag-3.5.1 → knowledge_rag-3.6.0}/pyproject.toml +114 -114
- {knowledge_rag-3.5.1 → knowledge_rag-3.6.0}/requirements.txt +49 -49
|
@@ -1,59 +1,59 @@
|
|
|
1
|
-
# Runtime
|
|
2
|
-
venv/
|
|
3
|
-
__pycache__/
|
|
4
|
-
*.pyc
|
|
5
|
-
data/
|
|
6
|
-
|
|
7
|
-
# User config (personal settings — use presets/ as starting point)
|
|
8
|
-
config.yaml
|
|
9
|
-
|
|
10
|
-
# Personal documents (NEVER commit — user-populated content)
|
|
11
|
-
documents/aar/
|
|
12
|
-
documents/security/
|
|
13
|
-
documents/logscale/
|
|
14
|
-
documents/general/
|
|
15
|
-
documents/ctf/
|
|
16
|
-
documents/development/
|
|
17
|
-
documents/documents/
|
|
18
|
-
|
|
19
|
-
# Sensitive file formats (extra safety layer)
|
|
20
|
-
documents/**/*.pdf
|
|
21
|
-
documents/**/*.docx
|
|
22
|
-
documents/**/*.xlsx
|
|
23
|
-
documents/**/*.pptx
|
|
24
|
-
documents/**/*.csv
|
|
25
|
-
|
|
26
|
-
# Keep example docs
|
|
27
|
-
!documents/examples/
|
|
28
|
-
!documents/examples/**
|
|
29
|
-
|
|
30
|
-
# FastEmbed model cache
|
|
31
|
-
.cache/
|
|
32
|
-
models_cache/
|
|
33
|
-
*.onnx
|
|
34
|
-
|
|
35
|
-
# Local scripts (not part of distribution)
|
|
36
|
-
scripts/
|
|
37
|
-
setup-notebook.ps1
|
|
38
|
-
demo-real.yml
|
|
39
|
-
documents/.sync-log.txt
|
|
40
|
-
documents/README-CATEGORIES.md
|
|
41
|
-
|
|
42
|
-
# Temp/junk
|
|
43
|
-
*.b64
|
|
44
|
-
*.tar.gz
|
|
45
|
-
*.bak
|
|
46
|
-
|
|
47
|
-
# OS files
|
|
48
|
-
.DS_Store
|
|
49
|
-
Thumbs.db
|
|
50
|
-
desktop.ini
|
|
51
|
-
|
|
52
|
-
# IDE
|
|
53
|
-
.vscode/
|
|
54
|
-
.idea/
|
|
55
|
-
*.swp
|
|
56
|
-
*.swo
|
|
57
|
-
dist/
|
|
58
|
-
.ruff_cache/
|
|
59
|
-
.pytest_cache/
|
|
1
|
+
# Runtime
|
|
2
|
+
venv/
|
|
3
|
+
__pycache__/
|
|
4
|
+
*.pyc
|
|
5
|
+
data/
|
|
6
|
+
|
|
7
|
+
# User config (personal settings — use presets/ as starting point)
|
|
8
|
+
config.yaml
|
|
9
|
+
|
|
10
|
+
# Personal documents (NEVER commit — user-populated content)
|
|
11
|
+
documents/aar/
|
|
12
|
+
documents/security/
|
|
13
|
+
documents/logscale/
|
|
14
|
+
documents/general/
|
|
15
|
+
documents/ctf/
|
|
16
|
+
documents/development/
|
|
17
|
+
documents/documents/
|
|
18
|
+
|
|
19
|
+
# Sensitive file formats (extra safety layer)
|
|
20
|
+
documents/**/*.pdf
|
|
21
|
+
documents/**/*.docx
|
|
22
|
+
documents/**/*.xlsx
|
|
23
|
+
documents/**/*.pptx
|
|
24
|
+
documents/**/*.csv
|
|
25
|
+
|
|
26
|
+
# Keep example docs
|
|
27
|
+
!documents/examples/
|
|
28
|
+
!documents/examples/**
|
|
29
|
+
|
|
30
|
+
# FastEmbed model cache
|
|
31
|
+
.cache/
|
|
32
|
+
models_cache/
|
|
33
|
+
*.onnx
|
|
34
|
+
|
|
35
|
+
# Local scripts (not part of distribution)
|
|
36
|
+
scripts/
|
|
37
|
+
setup-notebook.ps1
|
|
38
|
+
demo-real.yml
|
|
39
|
+
documents/.sync-log.txt
|
|
40
|
+
documents/README-CATEGORIES.md
|
|
41
|
+
|
|
42
|
+
# Temp/junk
|
|
43
|
+
*.b64
|
|
44
|
+
*.tar.gz
|
|
45
|
+
*.bak
|
|
46
|
+
|
|
47
|
+
# OS files
|
|
48
|
+
.DS_Store
|
|
49
|
+
Thumbs.db
|
|
50
|
+
desktop.ini
|
|
51
|
+
|
|
52
|
+
# IDE
|
|
53
|
+
.vscode/
|
|
54
|
+
.idea/
|
|
55
|
+
*.swp
|
|
56
|
+
*.swo
|
|
57
|
+
dist/
|
|
58
|
+
.ruff_cache/
|
|
59
|
+
.pytest_cache/
|
|
@@ -1,21 +1,21 @@
|
|
|
1
|
-
MIT License
|
|
2
|
-
|
|
3
|
-
Copyright (c) 2025 Ailton Rocha (Lyon)
|
|
4
|
-
|
|
5
|
-
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
-
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
-
in the Software without restriction, including without limitation the rights
|
|
8
|
-
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
-
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
-
furnished to do so, subject to the following conditions:
|
|
11
|
-
|
|
12
|
-
The above copyright notice and this permission notice shall be included in all
|
|
13
|
-
copies or substantial portions of the Software.
|
|
14
|
-
|
|
15
|
-
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
-
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
-
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
-
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
-
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
-
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
-
SOFTWARE.
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2025 Ailton Rocha (Lyon)
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: knowledge-rag
|
|
3
|
-
Version: 3.
|
|
4
|
-
Summary: Local RAG System for Claude Code — Hybrid search + Cross-encoder Reranking + 12 MCP Tools. Zero external servers.
|
|
3
|
+
Version: 3.6.0
|
|
4
|
+
Summary: Local RAG System for Claude Code — Hybrid search + Cross-encoder Reranking + 12 MCP Tools + 20 Format Parsers. Zero external servers.
|
|
5
5
|
Project-URL: Homepage, https://github.com/lyonzin/knowledge-rag
|
|
6
6
|
Project-URL: Repository, https://github.com/lyonzin/knowledge-rag
|
|
7
7
|
Project-URL: Issues, https://github.com/lyonzin/knowledge-rag/issues
|
|
@@ -40,58 +40,71 @@ Description-Content-Type: text/markdown
|
|
|
40
40
|
|
|
41
41
|
<div align="center">
|
|
42
42
|
|
|
43
|
-

|
|
44
44
|

|
|
45
45
|

|
|
46
46
|

|
|
47
|
+

|
|
47
48
|
[](https://github.com/lyonzin/knowledge-rag/actions/workflows/ci.yml)
|
|
48
49
|
[](https://github.com/lyonzin/knowledge-rag/actions/workflows/security.yml)
|
|
49
50
|
[](https://glama.ai/mcp/servers/lyonzin/knowledge-rag)
|
|
50
51
|
[](https://pypi.org/project/knowledge-rag/)
|
|
51
52
|
|
|
52
|
-
###
|
|
53
|
+
### Your docs, your machine, zero cloud. Claude Code searches them natively.
|
|
53
54
|
|
|
54
|
-
|
|
55
|
-
|
|
55
|
+
Drop your PDFs, markdown, code, notebooks — **1800+ files, 39K chunks, indexed in under 3 minutes.**<br/>
|
|
56
|
+
Hybrid search (BM25 + semantic vectors + cross-encoder reranking) through 12 MCP tools.<br/>
|
|
57
|
+
Everything runs locally via ONNX. No Docker, no Ollama, no API keys, no data leaves your machine.
|
|
56
58
|
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
`pip install → restart Claude Code → done.`
|
|
59
|
+
```
|
|
60
|
+
pip install knowledge-rag → restart Claude Code → search_knowledge("your query")
|
|
61
|
+
```
|
|
61
62
|
|
|
62
63
|
---
|
|
63
64
|
|
|
64
|
-
**12 MCP Tools** | **Hybrid Search +
|
|
65
|
+
**12 MCP Tools** | **Hybrid Search + Reranking** | **20 File Formats** | **Optional NVIDIA GPU** | **100% Local**
|
|
65
66
|
|
|
66
|
-
[What's New](#whats-new-in-
|
|
67
|
+
[What's New](#whats-new-in-v352) | [Supported Formats](#supported-formats) | [Installation](#installation) | [Configuration](#configuration) | [API Reference](#api-reference) | [Architecture](#architecture)
|
|
67
68
|
|
|
68
69
|
</div>
|
|
69
70
|
|
|
70
71
|
---
|
|
71
72
|
|
|
72
|
-
## What's New in v3.5.
|
|
73
|
+
## What's New in v3.5.2
|
|
73
74
|
|
|
74
75
|
### GPU-Accelerated Embeddings (Optional)
|
|
75
76
|
|
|
76
|
-
ONNX embeddings can
|
|
77
|
+
ONNX embeddings can run on NVIDIA GPUs for **5-10x faster indexing**. Opt-in — CPU remains the default.
|
|
77
78
|
|
|
78
79
|
```bash
|
|
80
|
+
# NVIDIA GPU (requires CUDA 12.x drivers)
|
|
79
81
|
pip install knowledge-rag[gpu]
|
|
82
|
+
|
|
83
|
+
# Also install CUDA 12 runtime libraries (if not using CUDA Toolkit 12.x)
|
|
84
|
+
pip install nvidia-cublas-cu12 nvidia-cudnn-cu12 nvidia-cuda-runtime-cu12
|
|
80
85
|
```
|
|
81
86
|
|
|
82
87
|
```yaml
|
|
83
88
|
# config.yaml
|
|
84
89
|
models:
|
|
85
90
|
embedding:
|
|
86
|
-
gpu: true #
|
|
91
|
+
gpu: true # Automatic CPU fallback if CUDA is unavailable
|
|
87
92
|
```
|
|
88
93
|
|
|
94
|
+
**How it works:**
|
|
95
|
+
- Sets `CUDAExecutionProvider` as primary, `CPUExecutionProvider` as fallback
|
|
96
|
+
- Auto-discovers CUDA 12 DLLs from pip-installed NVIDIA packages (no manual PATH config)
|
|
97
|
+
- If GPU init fails for any reason, falls back to CPU silently with a `[WARN]` log
|
|
98
|
+
- `gpu: false` (default) forces CPU-only mode — zero CUDA overhead, clean logs
|
|
99
|
+
|
|
89
100
|
Ideal for large knowledge bases (1000+ documents) where full rebuilds take minutes on CPU. After the initial index, incremental reindexing (`force: true`) takes seconds regardless.
|
|
90
101
|
|
|
91
102
|
### Recent Highlights
|
|
92
103
|
|
|
104
|
+
- **v3.5.2** — CUDA DLL auto-discovery from pip packages, graceful GPU→CPU fallback, explicit CPU provider (no CUDA noise when `gpu: false`), BASE_DIR resolution fix for editable installs
|
|
105
|
+
- **v3.5.1** — Remove Python `<3.13` upper bound — 3.13 and 3.14 now supported
|
|
106
|
+
- **v3.5.0** — Optional GPU acceleration, supported formats table, full README rewrite
|
|
93
107
|
- **v3.4.3** — MCP stdout save/restore fix (v3.4.2 broke JSON-RPC responses)
|
|
94
|
-
- **v3.4.1** — `pip install` auto-detects project dir from venv location, Linux/macOS `install.sh`
|
|
95
108
|
- **v3.4.0** — Persistent model cache, exclude patterns, Jupyter Notebook parser, inotify resilience, MetaTrader support
|
|
96
109
|
|
|
97
110
|
See [Changelog](#changelog) for full history.
|
|
@@ -112,6 +125,14 @@ See [Changelog](#changelog) for full history.
|
|
|
112
125
|
| Excel | `.xlsx` | openpyxl | Yes | Sheet-by-sheet extraction |
|
|
113
126
|
| PowerPoint | `.pptx` | python-pptx | Yes | Slide-by-slide extraction |
|
|
114
127
|
| Jupyter Notebook | `.ipynb` | Cell-aware parser | Yes | Markdown + code cells only, no outputs/base64 |
|
|
128
|
+
| C Source | `.c` | Code-aware parser | Yes | Functions/structs/includes extracted |
|
|
129
|
+
| C/C++ Header | `.h` | Code-aware parser | Yes | Function declarations/structs extracted |
|
|
130
|
+
| C++ Source | `.cpp` | Code-aware parser | Yes | Classes/structs/includes extracted |
|
|
131
|
+
| JavaScript | `.js` | Code-aware parser | Yes | Functions/classes/imports (ESM + CJS) |
|
|
132
|
+
| React JSX | `.jsx` | Code-aware parser | Yes | Same as JS parser |
|
|
133
|
+
| TypeScript | `.ts` | Code-aware parser | Yes | Functions/classes/interfaces/enums/imports |
|
|
134
|
+
| React TSX | `.tsx` | Code-aware parser | Yes | Same as TS parser |
|
|
135
|
+
| XML | `.xml` | XML parser | Yes | Root element and namespace extraction |
|
|
115
136
|
| MQL4 Header | `.mqh` | Code parser | No | MetaTrader — add to `supported_formats` to enable |
|
|
116
137
|
| MQL4 Source | `.mq4` | Code parser | No | MetaTrader — add to `supported_formats` to enable |
|
|
117
138
|
|
|
@@ -131,7 +152,7 @@ See [Changelog](#changelog) for full history.
|
|
|
131
152
|
| **Markdown-Aware Chunking** | `.md` files split by `##`/`###` sections instead of fixed windows |
|
|
132
153
|
| **In-Process Embeddings** | FastEmbed ONNX Runtime (BAAI/bge-small-en-v1.5, 384D) |
|
|
133
154
|
| **Keyword Routing** | Word-boundary aware routing for domain-specific queries |
|
|
134
|
-
| **
|
|
155
|
+
| **20 Format Parsers** | MD, TXT, PDF, PY, C, H, CPP, JS, JSX, TS, TSX, JSON, XML, CSV, DOCX, XLSX, PPTX, IPYNB + opt-in MQH/MQ4 |
|
|
135
156
|
| **Category Organization** | Organize docs by folder, auto-tagged by path |
|
|
136
157
|
| **Incremental Indexing** | Change detection via mtime/size — only re-indexes modified files |
|
|
137
158
|
| **Chunk Deduplication** | SHA256 content hashing prevents duplicate chunks |
|
|
@@ -189,7 +210,7 @@ flowchart TB
|
|
|
189
210
|
end
|
|
190
211
|
|
|
191
212
|
subgraph INGEST["DOCUMENT INGESTION"]
|
|
192
|
-
PARSERS["
|
|
213
|
+
PARSERS["20 Parsers<br/>MD | PDF | TXT | PY | C | H | CPP | JS | JSX | TS | TSX | JSON | XML | CSV<br/>DOCX | XLSX | PPTX | IPYNB | MQH | MQ4"]
|
|
193
214
|
CHUNKER["Chunking<br/>MD: section-aware<br/>Other: 1000 chars + 200 overlap"]
|
|
194
215
|
PARSERS --> CHUNKER
|
|
195
216
|
end
|
|
@@ -255,11 +276,11 @@ flowchart LR
|
|
|
255
276
|
FILES["documents/<br/>├── security/<br/>├── development/<br/>├── ctf/<br/>└── general/"]
|
|
256
277
|
end
|
|
257
278
|
|
|
258
|
-
subgraph PARSE["Parse (
|
|
279
|
+
subgraph PARSE["Parse (20 formats)"]
|
|
259
280
|
MD["Markdown"]
|
|
260
281
|
PDF["PDF<br/>(PyMuPDF)"]
|
|
261
282
|
OFFICE["DOCX | XLSX<br/>PPTX | CSV"]
|
|
262
|
-
CODE["PY |
|
|
283
|
+
CODE["PY | C | H | CPP | JS | JSX<br/>TS | TSX | JSON | XML | IPYNB"]
|
|
263
284
|
end
|
|
264
285
|
|
|
265
286
|
subgraph CHUNK["Chunk"]
|
|
@@ -920,7 +941,7 @@ knowledge-rag/
|
|
|
920
941
|
├── mcp_server/
|
|
921
942
|
│ ├── __init__.py # Stdout protection + version
|
|
922
943
|
│ ├── config.py # YAML config loader + defaults
|
|
923
|
-
│ ├── ingestion.py #
|
|
944
|
+
│ ├── ingestion.py # 20 parsers, chunking, metadata extraction
|
|
924
945
|
│ └── server.py # MCP server, ChromaDB, BM25, reranker, 12 tools
|
|
925
946
|
├── config.example.yaml # Documented config template (copy to config.yaml)
|
|
926
947
|
├── config.yaml # Your active configuration (git-ignored)
|
|
@@ -1011,6 +1032,22 @@ With ~200 documents, expect ~300-500MB RAM. The embedding model (~50MB) and rera
|
|
|
1011
1032
|
|
|
1012
1033
|
## Changelog
|
|
1013
1034
|
|
|
1035
|
+
### v3.6.0 (2026-04-23)
|
|
1036
|
+
|
|
1037
|
+
- **NEW**: Multi-language code parsing — C (`.c`), C++ (`.cpp`/`.h`), JavaScript (`.js`/`.jsx`), TypeScript (`.ts`/`.tsx`) with per-language function/class/import extraction
|
|
1038
|
+
- **NEW**: XML parser (`.xml`) — root element and namespace metadata extraction
|
|
1039
|
+
- **NEW**: All 8 new formats default enabled — no config change needed
|
|
1040
|
+
- **NEW**: NPM wrapper (`npx knowledge-rag`) + Docker image (`ghcr.io/lyonzin/knowledge-rag`)
|
|
1041
|
+
- **NEW**: Automated release pipeline — PyPI (Trusted Publishing), NPM, Docker GHCR
|
|
1042
|
+
- **IMPROVED**: Code parser reports correct `language` metadata per file type (was hardcoded to `"python"` for all code files)
|
|
1043
|
+
|
|
1044
|
+
### v3.5.2 (2026-04-16)
|
|
1045
|
+
|
|
1046
|
+
- **NEW**: Auto-discovery of CUDA 12 DLLs from pip-installed NVIDIA packages — no manual PATH configuration needed
|
|
1047
|
+
- **NEW**: Graceful GPU→CPU fallback with `[WARN]` log when CUDA init fails (missing drivers, wrong version, etc.)
|
|
1048
|
+
- **FIX**: Explicit `CPUExecutionProvider` when `gpu: false` — eliminates noisy CUDA probe errors in logs
|
|
1049
|
+
- **FIX**: BASE_DIR resolution now correctly prefers directories with `config.yaml` over those with only `config.example.yaml` (fixes editable installs)
|
|
1050
|
+
|
|
1014
1051
|
### v3.5.1 (2026-04-16)
|
|
1015
1052
|
|
|
1016
1053
|
- **FIX**: Removed Python upper bound constraint (`<3.13` → `>=3.11`). Python 3.13 and 3.14 now supported — onnxruntime ships wheels for both.
|
|
@@ -1018,7 +1055,7 @@ With ~200 documents, expect ~300-500MB RAM. The embedding model (~50MB) and rera
|
|
|
1018
1055
|
### v3.5.0 (2026-04-16)
|
|
1019
1056
|
|
|
1020
1057
|
- **NEW**: Optional GPU acceleration for ONNX embeddings — `pip install knowledge-rag[gpu]` + `models.embedding.gpu: true` in config. 5-10x faster indexing on NVIDIA GPUs with automatic CPU fallback.
|
|
1021
|
-
- **DOCS**: Supported formats table added to README (
|
|
1058
|
+
- **DOCS**: Supported formats table added to README (20 formats)
|
|
1022
1059
|
|
|
1023
1060
|
### v3.4.3 (2026-04-16)
|
|
1024
1061
|
|
|
@@ -2,58 +2,71 @@
|
|
|
2
2
|
|
|
3
3
|
<div align="center">
|
|
4
4
|
|
|
5
|
-

|
|
6
6
|

|
|
7
7
|

|
|
8
8
|

|
|
9
|
+

|
|
9
10
|
[](https://github.com/lyonzin/knowledge-rag/actions/workflows/ci.yml)
|
|
10
11
|
[](https://github.com/lyonzin/knowledge-rag/actions/workflows/security.yml)
|
|
11
12
|
[](https://glama.ai/mcp/servers/lyonzin/knowledge-rag)
|
|
12
13
|
[](https://pypi.org/project/knowledge-rag/)
|
|
13
14
|
|
|
14
|
-
###
|
|
15
|
+
### Your docs, your machine, zero cloud. Claude Code searches them natively.
|
|
15
16
|
|
|
16
|
-
|
|
17
|
-
|
|
17
|
+
Drop your PDFs, markdown, code, notebooks — **1800+ files, 39K chunks, indexed in under 3 minutes.**<br/>
|
|
18
|
+
Hybrid search (BM25 + semantic vectors + cross-encoder reranking) through 12 MCP tools.<br/>
|
|
19
|
+
Everything runs locally via ONNX. No Docker, no Ollama, no API keys, no data leaves your machine.
|
|
18
20
|
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
`pip install → restart Claude Code → done.`
|
|
21
|
+
```
|
|
22
|
+
pip install knowledge-rag → restart Claude Code → search_knowledge("your query")
|
|
23
|
+
```
|
|
23
24
|
|
|
24
25
|
---
|
|
25
26
|
|
|
26
|
-
**12 MCP Tools** | **Hybrid Search +
|
|
27
|
+
**12 MCP Tools** | **Hybrid Search + Reranking** | **20 File Formats** | **Optional NVIDIA GPU** | **100% Local**
|
|
27
28
|
|
|
28
|
-
[What's New](#whats-new-in-
|
|
29
|
+
[What's New](#whats-new-in-v352) | [Supported Formats](#supported-formats) | [Installation](#installation) | [Configuration](#configuration) | [API Reference](#api-reference) | [Architecture](#architecture)
|
|
29
30
|
|
|
30
31
|
</div>
|
|
31
32
|
|
|
32
33
|
---
|
|
33
34
|
|
|
34
|
-
## What's New in v3.5.
|
|
35
|
+
## What's New in v3.5.2
|
|
35
36
|
|
|
36
37
|
### GPU-Accelerated Embeddings (Optional)
|
|
37
38
|
|
|
38
|
-
ONNX embeddings can
|
|
39
|
+
ONNX embeddings can run on NVIDIA GPUs for **5-10x faster indexing**. Opt-in — CPU remains the default.
|
|
39
40
|
|
|
40
41
|
```bash
|
|
42
|
+
# NVIDIA GPU (requires CUDA 12.x drivers)
|
|
41
43
|
pip install knowledge-rag[gpu]
|
|
44
|
+
|
|
45
|
+
# Also install CUDA 12 runtime libraries (if not using CUDA Toolkit 12.x)
|
|
46
|
+
pip install nvidia-cublas-cu12 nvidia-cudnn-cu12 nvidia-cuda-runtime-cu12
|
|
42
47
|
```
|
|
43
48
|
|
|
44
49
|
```yaml
|
|
45
50
|
# config.yaml
|
|
46
51
|
models:
|
|
47
52
|
embedding:
|
|
48
|
-
gpu: true #
|
|
53
|
+
gpu: true # Automatic CPU fallback if CUDA is unavailable
|
|
49
54
|
```
|
|
50
55
|
|
|
56
|
+
**How it works:**
|
|
57
|
+
- Sets `CUDAExecutionProvider` as primary, `CPUExecutionProvider` as fallback
|
|
58
|
+
- Auto-discovers CUDA 12 DLLs from pip-installed NVIDIA packages (no manual PATH config)
|
|
59
|
+
- If GPU init fails for any reason, falls back to CPU silently with a `[WARN]` log
|
|
60
|
+
- `gpu: false` (default) forces CPU-only mode — zero CUDA overhead, clean logs
|
|
61
|
+
|
|
51
62
|
Ideal for large knowledge bases (1000+ documents) where full rebuilds take minutes on CPU. After the initial index, incremental reindexing (`force: true`) takes seconds regardless.
|
|
52
63
|
|
|
53
64
|
### Recent Highlights
|
|
54
65
|
|
|
66
|
+
- **v3.5.2** — CUDA DLL auto-discovery from pip packages, graceful GPU→CPU fallback, explicit CPU provider (no CUDA noise when `gpu: false`), BASE_DIR resolution fix for editable installs
|
|
67
|
+
- **v3.5.1** — Remove Python `<3.13` upper bound — 3.13 and 3.14 now supported
|
|
68
|
+
- **v3.5.0** — Optional GPU acceleration, supported formats table, full README rewrite
|
|
55
69
|
- **v3.4.3** — MCP stdout save/restore fix (v3.4.2 broke JSON-RPC responses)
|
|
56
|
-
- **v3.4.1** — `pip install` auto-detects project dir from venv location, Linux/macOS `install.sh`
|
|
57
70
|
- **v3.4.0** — Persistent model cache, exclude patterns, Jupyter Notebook parser, inotify resilience, MetaTrader support
|
|
58
71
|
|
|
59
72
|
See [Changelog](#changelog) for full history.
|
|
@@ -74,6 +87,14 @@ See [Changelog](#changelog) for full history.
|
|
|
74
87
|
| Excel | `.xlsx` | openpyxl | Yes | Sheet-by-sheet extraction |
|
|
75
88
|
| PowerPoint | `.pptx` | python-pptx | Yes | Slide-by-slide extraction |
|
|
76
89
|
| Jupyter Notebook | `.ipynb` | Cell-aware parser | Yes | Markdown + code cells only, no outputs/base64 |
|
|
90
|
+
| C Source | `.c` | Code-aware parser | Yes | Functions/structs/includes extracted |
|
|
91
|
+
| C/C++ Header | `.h` | Code-aware parser | Yes | Function declarations/structs extracted |
|
|
92
|
+
| C++ Source | `.cpp` | Code-aware parser | Yes | Classes/structs/includes extracted |
|
|
93
|
+
| JavaScript | `.js` | Code-aware parser | Yes | Functions/classes/imports (ESM + CJS) |
|
|
94
|
+
| React JSX | `.jsx` | Code-aware parser | Yes | Same as JS parser |
|
|
95
|
+
| TypeScript | `.ts` | Code-aware parser | Yes | Functions/classes/interfaces/enums/imports |
|
|
96
|
+
| React TSX | `.tsx` | Code-aware parser | Yes | Same as TS parser |
|
|
97
|
+
| XML | `.xml` | XML parser | Yes | Root element and namespace extraction |
|
|
77
98
|
| MQL4 Header | `.mqh` | Code parser | No | MetaTrader — add to `supported_formats` to enable |
|
|
78
99
|
| MQL4 Source | `.mq4` | Code parser | No | MetaTrader — add to `supported_formats` to enable |
|
|
79
100
|
|
|
@@ -93,7 +114,7 @@ See [Changelog](#changelog) for full history.
|
|
|
93
114
|
| **Markdown-Aware Chunking** | `.md` files split by `##`/`###` sections instead of fixed windows |
|
|
94
115
|
| **In-Process Embeddings** | FastEmbed ONNX Runtime (BAAI/bge-small-en-v1.5, 384D) |
|
|
95
116
|
| **Keyword Routing** | Word-boundary aware routing for domain-specific queries |
|
|
96
|
-
| **
|
|
117
|
+
| **20 Format Parsers** | MD, TXT, PDF, PY, C, H, CPP, JS, JSX, TS, TSX, JSON, XML, CSV, DOCX, XLSX, PPTX, IPYNB + opt-in MQH/MQ4 |
|
|
97
118
|
| **Category Organization** | Organize docs by folder, auto-tagged by path |
|
|
98
119
|
| **Incremental Indexing** | Change detection via mtime/size — only re-indexes modified files |
|
|
99
120
|
| **Chunk Deduplication** | SHA256 content hashing prevents duplicate chunks |
|
|
@@ -151,7 +172,7 @@ flowchart TB
|
|
|
151
172
|
end
|
|
152
173
|
|
|
153
174
|
subgraph INGEST["DOCUMENT INGESTION"]
|
|
154
|
-
PARSERS["
|
|
175
|
+
PARSERS["20 Parsers<br/>MD | PDF | TXT | PY | C | H | CPP | JS | JSX | TS | TSX | JSON | XML | CSV<br/>DOCX | XLSX | PPTX | IPYNB | MQH | MQ4"]
|
|
155
176
|
CHUNKER["Chunking<br/>MD: section-aware<br/>Other: 1000 chars + 200 overlap"]
|
|
156
177
|
PARSERS --> CHUNKER
|
|
157
178
|
end
|
|
@@ -217,11 +238,11 @@ flowchart LR
|
|
|
217
238
|
FILES["documents/<br/>├── security/<br/>├── development/<br/>├── ctf/<br/>└── general/"]
|
|
218
239
|
end
|
|
219
240
|
|
|
220
|
-
subgraph PARSE["Parse (
|
|
241
|
+
subgraph PARSE["Parse (20 formats)"]
|
|
221
242
|
MD["Markdown"]
|
|
222
243
|
PDF["PDF<br/>(PyMuPDF)"]
|
|
223
244
|
OFFICE["DOCX | XLSX<br/>PPTX | CSV"]
|
|
224
|
-
CODE["PY |
|
|
245
|
+
CODE["PY | C | H | CPP | JS | JSX<br/>TS | TSX | JSON | XML | IPYNB"]
|
|
225
246
|
end
|
|
226
247
|
|
|
227
248
|
subgraph CHUNK["Chunk"]
|
|
@@ -882,7 +903,7 @@ knowledge-rag/
|
|
|
882
903
|
├── mcp_server/
|
|
883
904
|
│ ├── __init__.py # Stdout protection + version
|
|
884
905
|
│ ├── config.py # YAML config loader + defaults
|
|
885
|
-
│ ├── ingestion.py #
|
|
906
|
+
│ ├── ingestion.py # 20 parsers, chunking, metadata extraction
|
|
886
907
|
│ └── server.py # MCP server, ChromaDB, BM25, reranker, 12 tools
|
|
887
908
|
├── config.example.yaml # Documented config template (copy to config.yaml)
|
|
888
909
|
├── config.yaml # Your active configuration (git-ignored)
|
|
@@ -973,6 +994,22 @@ With ~200 documents, expect ~300-500MB RAM. The embedding model (~50MB) and rera
|
|
|
973
994
|
|
|
974
995
|
## Changelog
|
|
975
996
|
|
|
997
|
+
### v3.6.0 (2026-04-23)
|
|
998
|
+
|
|
999
|
+
- **NEW**: Multi-language code parsing — C (`.c`), C++ (`.cpp`/`.h`), JavaScript (`.js`/`.jsx`), TypeScript (`.ts`/`.tsx`) with per-language function/class/import extraction
|
|
1000
|
+
- **NEW**: XML parser (`.xml`) — root element and namespace metadata extraction
|
|
1001
|
+
- **NEW**: All 8 new formats default enabled — no config change needed
|
|
1002
|
+
- **NEW**: NPM wrapper (`npx knowledge-rag`) + Docker image (`ghcr.io/lyonzin/knowledge-rag`)
|
|
1003
|
+
- **NEW**: Automated release pipeline — PyPI (Trusted Publishing), NPM, Docker GHCR
|
|
1004
|
+
- **IMPROVED**: Code parser reports correct `language` metadata per file type (was hardcoded to `"python"` for all code files)
|
|
1005
|
+
|
|
1006
|
+
### v3.5.2 (2026-04-16)
|
|
1007
|
+
|
|
1008
|
+
- **NEW**: Auto-discovery of CUDA 12 DLLs from pip-installed NVIDIA packages — no manual PATH configuration needed
|
|
1009
|
+
- **NEW**: Graceful GPU→CPU fallback with `[WARN]` log when CUDA init fails (missing drivers, wrong version, etc.)
|
|
1010
|
+
- **FIX**: Explicit `CPUExecutionProvider` when `gpu: false` — eliminates noisy CUDA probe errors in logs
|
|
1011
|
+
- **FIX**: BASE_DIR resolution now correctly prefers directories with `config.yaml` over those with only `config.example.yaml` (fixes editable installs)
|
|
1012
|
+
|
|
976
1013
|
### v3.5.1 (2026-04-16)
|
|
977
1014
|
|
|
978
1015
|
- **FIX**: Removed Python upper bound constraint (`<3.13` → `>=3.11`). Python 3.13 and 3.14 now supported — onnxruntime ships wheels for both.
|
|
@@ -980,7 +1017,7 @@ With ~200 documents, expect ~300-500MB RAM. The embedding model (~50MB) and rera
|
|
|
980
1017
|
### v3.5.0 (2026-04-16)
|
|
981
1018
|
|
|
982
1019
|
- **NEW**: Optional GPU acceleration for ONNX embeddings — `pip install knowledge-rag[gpu]` + `models.embedding.gpu: true` in config. 5-10x faster indexing on NVIDIA GPUs with automatic CPU fallback.
|
|
983
|
-
- **DOCS**: Supported formats table added to README (
|
|
1020
|
+
- **DOCS**: Supported formats table added to README (20 formats)
|
|
984
1021
|
|
|
985
1022
|
### v3.4.3 (2026-04-16)
|
|
986
1023
|
|