chunksilo 2.3.4__tar.gz → 2.4.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {chunksilo-2.3.4/src/chunksilo.egg-info → chunksilo-2.4.0}/PKG-INFO +11 -10
- {chunksilo-2.3.4 → chunksilo-2.4.0}/README.md +8 -7
- {chunksilo-2.3.4 → chunksilo-2.4.0}/pyproject.toml +6 -3
- {chunksilo-2.3.4 → chunksilo-2.4.0}/requirements.txt +2 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/src/chunksilo/__init__.py +1 -1
- {chunksilo-2.3.4 → chunksilo-2.4.0}/src/chunksilo/cfgload.py +19 -7
- {chunksilo-2.3.4 → chunksilo-2.4.0/src/chunksilo.egg-info}/PKG-INFO +11 -10
- {chunksilo-2.3.4 → chunksilo-2.4.0}/LICENSE +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/NOTICE +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/setup.cfg +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/src/chunksilo/__main__.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/src/chunksilo/cli.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/src/chunksilo/confluence_html_formatter.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/src/chunksilo/docx_utils.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/src/chunksilo/index.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/src/chunksilo/models.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/src/chunksilo/search.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/src/chunksilo/server.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/src/chunksilo/ui.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/src/chunksilo.egg-info/SOURCES.txt +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/src/chunksilo.egg-info/dependency_links.txt +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/src/chunksilo.egg-info/entry_points.txt +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/src/chunksilo.egg-info/requires.txt +2 -2
- {chunksilo-2.3.4 → chunksilo-2.4.0}/src/chunksilo.egg-info/top_level.txt +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/test/test_chunk_location.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/test/test_cli.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/test/test_confluence_html_formatter.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/test/test_docx_utils.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/test/test_error_handling.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/test/test_heading_path_integration.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/test/test_incremental_ingest.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/test/test_indexing_benchmark.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/test/test_indexing_ui.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/test/test_jira_integration.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/test/test_quoted_phrases.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/test/test_rag_metrics.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/test/test_retrieval_only.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/test/test_scan_timeouts.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/test/test_server.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/test/test_system.py +0 -0
- {chunksilo-2.3.4 → chunksilo-2.4.0}/test/test_utils.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: chunksilo
|
|
3
|
-
Version: 2.
|
|
3
|
+
Version: 2.4.0
|
|
4
4
|
Summary: Local RAG-based semantic document search with MCP server interface
|
|
5
5
|
Author: Fredrik Reveny
|
|
6
6
|
License-Expression: Apache-2.0
|
|
@@ -30,10 +30,10 @@ Requires-Dist: huggingface-hub<2,>=0.22.0
|
|
|
30
30
|
Requires-Dist: flashrank<1,>=0.1.0
|
|
31
31
|
Requires-Dist: fastembed<1,>=0.5.0
|
|
32
32
|
Requires-Dist: pyyaml<7,>=6.0
|
|
33
|
+
Requires-Dist: llama-index-readers-confluence<1,>=0.6.0
|
|
34
|
+
Requires-Dist: jira<4,>=3.5.0
|
|
33
35
|
Provides-Extra: confluence
|
|
34
|
-
Requires-Dist: llama-index-readers-confluence<1,>=0.6.0; extra == "confluence"
|
|
35
36
|
Provides-Extra: jira
|
|
36
|
-
Requires-Dist: jira<4,>=3.5.0; extra == "jira"
|
|
37
37
|
Provides-Extra: test
|
|
38
38
|
Requires-Dist: pytest<9,>=7.4.0; extra == "test"
|
|
39
39
|
Requires-Dist: pytest-cov>=4.0; extra == "test"
|
|
@@ -93,11 +93,12 @@ Requires Python 3.11 or later. Models are downloaded automatically on first run
|
|
|
93
93
|
|
|
94
94
|
```bash
|
|
95
95
|
pip install chunksilo
|
|
96
|
-
|
|
97
|
-
# Or with Confluence and Jira support:
|
|
98
|
-
pip install chunksilo[confluence,jira]
|
|
99
96
|
```
|
|
100
97
|
|
|
98
|
+
Confluence and Jira support is included by default — just provide a config file
|
|
99
|
+
to enable them. (`pip install chunksilo[confluence,jira]` still works as an alias
|
|
100
|
+
for backward compatibility.)
|
|
101
|
+
|
|
101
102
|
Then:
|
|
102
103
|
1. **Create** a config file at `~/.config/chunksilo/config.yaml` (see [Configuration](#configuration))
|
|
103
104
|
2. **Build** the index: `chunksilo --build-index`
|
|
@@ -226,7 +227,7 @@ All settings are optional and have sensible defaults.
|
|
|
226
227
|
|
|
227
228
|
#### Confluence Settings (optional)
|
|
228
229
|
|
|
229
|
-
> **Note:** Confluence
|
|
230
|
+
> **Note:** Confluence support is installed by default; just set the values below to enable it.
|
|
230
231
|
|
|
231
232
|
| Setting | Description |
|
|
232
233
|
| :--- | :--- |
|
|
@@ -244,7 +245,7 @@ All settings are optional and have sensible defaults.
|
|
|
244
245
|
|
|
245
246
|
#### Jira Settings (optional)
|
|
246
247
|
|
|
247
|
-
> **Note:** Jira
|
|
248
|
+
> **Note:** Jira support is installed by default; just set the values below to enable it.
|
|
248
249
|
|
|
249
250
|
| Setting | Description |
|
|
250
251
|
| :--- | :--- |
|
|
@@ -419,8 +420,8 @@ Add to `mcp_settings.json` (typically in `~/.config/Code/User/globalStorage/roov
|
|
|
419
420
|
- **Index missing**: Run `chunksilo --build-index` (PyPI install) or `./venv/bin/chunksilo --build-index` (offline bundle).
|
|
420
421
|
- **Retrieval errors**: Check paths in your MCP client configuration.
|
|
421
422
|
- **Offline mode**: PyPI installs default to `offline: false` (models auto-download). The offline bundle includes pre-downloaded models and sets `offline: true`. Set `retrieval.offline: true` in `config.yaml` to prevent network calls after initial model download.
|
|
422
|
-
- **Confluence Integration**:
|
|
423
|
-
- **Jira Integration**:
|
|
423
|
+
- **Confluence Integration**: Included by default — set `confluence.url`, `confluence.username`, and `confluence.api_token` in `config.yaml`.
|
|
424
|
+
- **Jira Integration**: Included by default — set `jira.url`, `jira.username`, and `jira.api_token` in `config.yaml`. Optionally configure `jira.projects` to restrict search to specific project keys.
|
|
424
425
|
- **Custom CA Bundle**: Set `ssl.ca_bundle_path` in `config.yaml` for custom certificates.
|
|
425
426
|
- **Network mounts**: Unavailable directories are skipped with a warning; indexing continues with available directories.
|
|
426
427
|
- **Legacy .doc files**: Requires LibreOffice to be installed for automatic conversion to .docx. If LibreOffice is not found, .doc files are skipped with a warning. Full heading extraction is supported.
|
|
@@ -51,11 +51,12 @@ Requires Python 3.11 or later. Models are downloaded automatically on first run
|
|
|
51
51
|
|
|
52
52
|
```bash
|
|
53
53
|
pip install chunksilo
|
|
54
|
-
|
|
55
|
-
# Or with Confluence and Jira support:
|
|
56
|
-
pip install chunksilo[confluence,jira]
|
|
57
54
|
```
|
|
58
55
|
|
|
56
|
+
Confluence and Jira support is included by default — just provide a config file
|
|
57
|
+
to enable them. (`pip install chunksilo[confluence,jira]` still works as an alias
|
|
58
|
+
for backward compatibility.)
|
|
59
|
+
|
|
59
60
|
Then:
|
|
60
61
|
1. **Create** a config file at `~/.config/chunksilo/config.yaml` (see [Configuration](#configuration))
|
|
61
62
|
2. **Build** the index: `chunksilo --build-index`
|
|
@@ -184,7 +185,7 @@ All settings are optional and have sensible defaults.
|
|
|
184
185
|
|
|
185
186
|
#### Confluence Settings (optional)
|
|
186
187
|
|
|
187
|
-
> **Note:** Confluence
|
|
188
|
+
> **Note:** Confluence support is installed by default; just set the values below to enable it.
|
|
188
189
|
|
|
189
190
|
| Setting | Description |
|
|
190
191
|
| :--- | :--- |
|
|
@@ -202,7 +203,7 @@ All settings are optional and have sensible defaults.
|
|
|
202
203
|
|
|
203
204
|
#### Jira Settings (optional)
|
|
204
205
|
|
|
205
|
-
> **Note:** Jira
|
|
206
|
+
> **Note:** Jira support is installed by default; just set the values below to enable it.
|
|
206
207
|
|
|
207
208
|
| Setting | Description |
|
|
208
209
|
| :--- | :--- |
|
|
@@ -377,8 +378,8 @@ Add to `mcp_settings.json` (typically in `~/.config/Code/User/globalStorage/roov
|
|
|
377
378
|
- **Index missing**: Run `chunksilo --build-index` (PyPI install) or `./venv/bin/chunksilo --build-index` (offline bundle).
|
|
378
379
|
- **Retrieval errors**: Check paths in your MCP client configuration.
|
|
379
380
|
- **Offline mode**: PyPI installs default to `offline: false` (models auto-download). The offline bundle includes pre-downloaded models and sets `offline: true`. Set `retrieval.offline: true` in `config.yaml` to prevent network calls after initial model download.
|
|
380
|
-
- **Confluence Integration**:
|
|
381
|
-
- **Jira Integration**:
|
|
381
|
+
- **Confluence Integration**: Included by default — set `confluence.url`, `confluence.username`, and `confluence.api_token` in `config.yaml`.
|
|
382
|
+
- **Jira Integration**: Included by default — set `jira.url`, `jira.username`, and `jira.api_token` in `config.yaml`. Optionally configure `jira.projects` to restrict search to specific project keys.
|
|
382
383
|
- **Custom CA Bundle**: Set `ssl.ca_bundle_path` in `config.yaml` for custom certificates.
|
|
383
384
|
- **Network mounts**: Unavailable directories are skipped with a warning; indexing continues with available directories.
|
|
384
385
|
- **Legacy .doc files**: Requires LibreOffice to be installed for automatic conversion to .docx. If LibreOffice is not found, .doc files are skipped with a warning. Full heading extraction is supported.
|
|
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
|
|
|
4
4
|
|
|
5
5
|
[project]
|
|
6
6
|
name = "chunksilo"
|
|
7
|
-
version = "2.
|
|
7
|
+
version = "2.4.0"
|
|
8
8
|
description = "Local RAG-based semantic document search with MCP server interface"
|
|
9
9
|
license = "Apache-2.0"
|
|
10
10
|
requires-python = ">=3.11"
|
|
@@ -25,8 +25,11 @@ classifiers = [
|
|
|
25
25
|
dynamic = ["dependencies"]
|
|
26
26
|
|
|
27
27
|
[project.optional-dependencies]
|
|
28
|
-
|
|
29
|
-
|
|
28
|
+
# Confluence and Jira support is now installed by default (see requirements.txt).
|
|
29
|
+
# These extras are kept as empty aliases so existing `pip install chunksilo[confluence,jira]`
|
|
30
|
+
# commands keep working without a warning.
|
|
31
|
+
confluence = []
|
|
32
|
+
jira = []
|
|
30
33
|
test = ["pytest>=7.4.0,<9", "pytest-cov>=4.0", "requests>=2.31.0,<3"]
|
|
31
34
|
|
|
32
35
|
[project.scripts]
|
|
@@ -142,6 +142,14 @@ def _deep_merge(base: dict[str, Any], override: dict[str, Any]) -> dict[str, Any
|
|
|
142
142
|
return result
|
|
143
143
|
|
|
144
144
|
|
|
145
|
+
def _has_embedding_cache(cache_dir: Path) -> bool:
|
|
146
|
+
"""True if ``cache_dir`` holds a HuggingFace-style embedding model cache."""
|
|
147
|
+
try:
|
|
148
|
+
return any(cache_dir.glob("models--*"))
|
|
149
|
+
except OSError:
|
|
150
|
+
return False
|
|
151
|
+
|
|
152
|
+
|
|
145
153
|
def _resolve_bundled_models(config: dict[str, Any]) -> dict[str, Any]:
|
|
146
154
|
"""Transparently fall back to models bundled in the package.
|
|
147
155
|
|
|
@@ -150,16 +158,20 @@ def _resolve_bundled_models(config: dict[str, Any]) -> dict[str, Any]:
|
|
|
150
158
|
offline mode. This lets a pip-installed wheel run in an air-gapped environment
|
|
151
159
|
with zero extra configuration. A user-configured cache that already holds the
|
|
152
160
|
models always wins, so explicit setups are never overridden.
|
|
153
|
-
"""
|
|
154
|
-
embed_name = config["retrieval"]["embed_model_name"]
|
|
155
|
-
hf_dirname = f"models--{embed_name.replace('/', '--')}"
|
|
156
161
|
|
|
157
|
-
|
|
158
|
-
|
|
162
|
+
Detection looks for any HuggingFace-style ``models--*`` cache directory rather
|
|
163
|
+
than matching the configured embedding name: fastembed stores the model under
|
|
164
|
+
its upstream source repo (e.g. ``models--qdrant--bge-small-en-v1.5-onnx-q``),
|
|
165
|
+
not under the configured alias ``BAAI/bge-small-en-v1.5``.
|
|
166
|
+
"""
|
|
167
|
+
if not _has_embedding_cache(_BUNDLED_MODELS_DIR):
|
|
168
|
+
return config # No models bundled in this install.
|
|
159
169
|
|
|
160
170
|
configured = Path(config["storage"]["model_cache_dir"]).expanduser()
|
|
161
|
-
if (
|
|
162
|
-
|
|
171
|
+
if configured.resolve() != _BUNDLED_MODELS_DIR.resolve() and _has_embedding_cache(
|
|
172
|
+
configured
|
|
173
|
+
):
|
|
174
|
+
return config # Configured cache already has models; respect it.
|
|
163
175
|
|
|
164
176
|
result = copy.deepcopy(config)
|
|
165
177
|
result["storage"]["model_cache_dir"] = str(_BUNDLED_MODELS_DIR)
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: chunksilo
|
|
3
|
-
Version: 2.
|
|
3
|
+
Version: 2.4.0
|
|
4
4
|
Summary: Local RAG-based semantic document search with MCP server interface
|
|
5
5
|
Author: Fredrik Reveny
|
|
6
6
|
License-Expression: Apache-2.0
|
|
@@ -30,10 +30,10 @@ Requires-Dist: huggingface-hub<2,>=0.22.0
|
|
|
30
30
|
Requires-Dist: flashrank<1,>=0.1.0
|
|
31
31
|
Requires-Dist: fastembed<1,>=0.5.0
|
|
32
32
|
Requires-Dist: pyyaml<7,>=6.0
|
|
33
|
+
Requires-Dist: llama-index-readers-confluence<1,>=0.6.0
|
|
34
|
+
Requires-Dist: jira<4,>=3.5.0
|
|
33
35
|
Provides-Extra: confluence
|
|
34
|
-
Requires-Dist: llama-index-readers-confluence<1,>=0.6.0; extra == "confluence"
|
|
35
36
|
Provides-Extra: jira
|
|
36
|
-
Requires-Dist: jira<4,>=3.5.0; extra == "jira"
|
|
37
37
|
Provides-Extra: test
|
|
38
38
|
Requires-Dist: pytest<9,>=7.4.0; extra == "test"
|
|
39
39
|
Requires-Dist: pytest-cov>=4.0; extra == "test"
|
|
@@ -93,11 +93,12 @@ Requires Python 3.11 or later. Models are downloaded automatically on first run
|
|
|
93
93
|
|
|
94
94
|
```bash
|
|
95
95
|
pip install chunksilo
|
|
96
|
-
|
|
97
|
-
# Or with Confluence and Jira support:
|
|
98
|
-
pip install chunksilo[confluence,jira]
|
|
99
96
|
```
|
|
100
97
|
|
|
98
|
+
Confluence and Jira support is included by default — just provide a config file
|
|
99
|
+
to enable them. (`pip install chunksilo[confluence,jira]` still works as an alias
|
|
100
|
+
for backward compatibility.)
|
|
101
|
+
|
|
101
102
|
Then:
|
|
102
103
|
1. **Create** a config file at `~/.config/chunksilo/config.yaml` (see [Configuration](#configuration))
|
|
103
104
|
2. **Build** the index: `chunksilo --build-index`
|
|
@@ -226,7 +227,7 @@ All settings are optional and have sensible defaults.
|
|
|
226
227
|
|
|
227
228
|
#### Confluence Settings (optional)
|
|
228
229
|
|
|
229
|
-
> **Note:** Confluence
|
|
230
|
+
> **Note:** Confluence support is installed by default; just set the values below to enable it.
|
|
230
231
|
|
|
231
232
|
| Setting | Description |
|
|
232
233
|
| :--- | :--- |
|
|
@@ -244,7 +245,7 @@ All settings are optional and have sensible defaults.
|
|
|
244
245
|
|
|
245
246
|
#### Jira Settings (optional)
|
|
246
247
|
|
|
247
|
-
> **Note:** Jira
|
|
248
|
+
> **Note:** Jira support is installed by default; just set the values below to enable it.
|
|
248
249
|
|
|
249
250
|
| Setting | Description |
|
|
250
251
|
| :--- | :--- |
|
|
@@ -419,8 +420,8 @@ Add to `mcp_settings.json` (typically in `~/.config/Code/User/globalStorage/roov
|
|
|
419
420
|
- **Index missing**: Run `chunksilo --build-index` (PyPI install) or `./venv/bin/chunksilo --build-index` (offline bundle).
|
|
420
421
|
- **Retrieval errors**: Check paths in your MCP client configuration.
|
|
421
422
|
- **Offline mode**: PyPI installs default to `offline: false` (models auto-download). The offline bundle includes pre-downloaded models and sets `offline: true`. Set `retrieval.offline: true` in `config.yaml` to prevent network calls after initial model download.
|
|
422
|
-
- **Confluence Integration**:
|
|
423
|
-
- **Jira Integration**:
|
|
423
|
+
- **Confluence Integration**: Included by default — set `confluence.url`, `confluence.username`, and `confluence.api_token` in `config.yaml`.
|
|
424
|
+
- **Jira Integration**: Included by default — set `jira.url`, `jira.username`, and `jira.api_token` in `config.yaml`. Optionally configure `jira.projects` to restrict search to specific project keys.
|
|
424
425
|
- **Custom CA Bundle**: Set `ssl.ca_bundle_path` in `config.yaml` for custom certificates.
|
|
425
426
|
- **Network mounts**: Unavailable directories are skipped with a warning; indexing continues with available directories.
|
|
426
427
|
- **Legacy .doc files**: Requires LibreOffice to be installed for automatic conversion to .docx. If LibreOffice is not found, .doc files are skipped with a warning. Full heading extraction is supported.
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
@@ -10,12 +10,12 @@ huggingface-hub<2,>=0.22.0
|
|
|
10
10
|
flashrank<1,>=0.1.0
|
|
11
11
|
fastembed<1,>=0.5.0
|
|
12
12
|
pyyaml<7,>=6.0
|
|
13
|
+
llama-index-readers-confluence<1,>=0.6.0
|
|
14
|
+
jira<4,>=3.5.0
|
|
13
15
|
|
|
14
16
|
[confluence]
|
|
15
|
-
llama-index-readers-confluence<1,>=0.6.0
|
|
16
17
|
|
|
17
18
|
[jira]
|
|
18
|
-
jira<4,>=3.5.0
|
|
19
19
|
|
|
20
20
|
[test]
|
|
21
21
|
pytest<9,>=7.4.0
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|