chunksilo 2.4.0__tar.gz → 3.0.0__tar.gz

chunksilo-3.0.0/PKG-INFO

@@ -0,0 +1,174 @@
+Metadata-Version: 2.4
+Name: chunksilo
+Version: 3.0.0
+Summary: Local RAG-based semantic document search with MCP server interface
+Author: Fredrik Reveny
+License-Expression: Apache-2.0
+Project-URL: Repository, https://github.com/Chetic/chunksilo
+Project-URL: Issues, https://github.com/Chetic/chunksilo/issues
+Keywords: rag,mcp,semantic-search,document-search,llm
+Classifier: Development Status :: 4 - Beta
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Text Processing :: Indexing
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: NOTICE
+Requires-Dist: llama-index<1,>=0.10.0
+Requires-Dist: llama-index-readers-file<1,>=0.1.0
+Requires-Dist: llama-index-embeddings-fastembed<1,>=0.5.0
+Requires-Dist: llama-index-retrievers-bm25<1,>=0.1.0
+Requires-Dist: pillow<11,>=10.3.0
+Requires-Dist: pypdf<7,>=5.1.0
+Requires-Dist: python-docx<2,>=1.1.0
+Requires-Dist: mcp<2,>=1.10.0
+Requires-Dist: huggingface-hub<2,>=0.22.0
+Requires-Dist: flashrank<1,>=0.1.0
+Requires-Dist: fastembed<1,>=0.5.0
+Requires-Dist: pyyaml<7,>=6.0
+Requires-Dist: llama-index-readers-confluence<1,>=0.6.0
+Requires-Dist: jira<4,>=3.5.0
+Provides-Extra: confluence
+Provides-Extra: jira
+Provides-Extra: test
+Requires-Dist: pytest<9,>=7.4.0; extra == "test"
+Requires-Dist: pytest-cov>=4.0; extra == "test"
+Requires-Dist: requests<3,>=2.31.0; extra == "test"
+Dynamic: license-file
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/Chetic/chunksilo/main/chunksilo.png" alt="ChunkSilo Logo" width="500">
+</p>
+
+<p align="center">
+  <img src="demo/demo.gif" alt="ChunkSilo terminal demo" width="720">
+</p>
+
+# ChunkSilo MCP Server
+
+ChunkSilo is like a local Google for your documents. It uses semantic search — matching by meaning rather than exact keywords — so your LLM can find relevant information across all your files even when the wording differs from your query. Point it at your PDFs, Word docs, Markdown, and text files, and it builds a fully searchable index locally on your machine.
+
+- Runs entirely on your machine — no servers, no infrastructure
+- Semantic search + keyword filename matching across PDF, DOCX, DOC, Markdown, and TXT
+- Incremental indexing — only reprocesses new or changed files, and never writes inside an indexed directory
+- Revision-aware indexing — files that are revisions of one document (`Spec_v2.pdf`, `.../Rev B/Spec.docx`) are grouped and only the newest is indexed; review-comment copies are skipped
+- Heading-aware results with source links back to the original file, spread across documents instead of one document's every chunk
+- Date filtering and recency boosting
+- Files indexed from a mounted network share can be presented at the share's own location (an `smb://` URI plus the Windows UNC path)
+- Serves MCP over stdio to a local client, or over streamable-http behind a reverse proxy of your own (oauth2-proxy + KeyCloak walkthrough included)
+- Optional Confluence and Jira integrations (supports Cloud and Server/Data Center)
+
+## Installation
+
+### Option A: Install from PyPI (Recommended)
+
+Requires Python 3.11 or later. Models are downloaded automatically on first run (~250MB). The first run may appear to pause while models download — this is normal.
+
+```bash
+pip install chunksilo
+```
+
+Confluence and Jira support is included by default — just provide a config file
+to enable them. (`pip install chunksilo[confluence,jira]` still works as an alias
+for backward compatibility.)
+
+Then:
+1. **Create** a config file at `~/.config/chunksilo/config.yaml` (see [Configuration](#configuration))
+2. **Build** the index: `chunksilo --build-index`
+3. **Configure** your MCP client (see [docs/mcp-clients.md](docs/mcp-clients.md))
+
+### Option B: Offline Bundle
+
+A self-contained package with pre-downloaded models, ideal for air-gapped environments or systems without Python installed.
+
+Download from the [Releases page](https://github.com/Chetic/chunksilo/releases):
+
+1. **Download** the `chunksilo-vX.Y.Z-manylinux_2_34_x86_64.tar.gz` file
+2. **Extract** and install:
+
+```bash
+tar -xzf chunksilo-vX.Y.Z-manylinux_2_34_x86_64.tar.gz
+cd chunksilo
+./setup.sh
+```
+
+3. **Edit** `config.yaml` to set your document directories
+4. **Build** the index: `./venv/bin/chunksilo --build-index`
+5. **Configure** your MCP client (see [docs/mcp-clients.md](docs/mcp-clients.md))
+
+## Configuration
+
+ChunkSilo reads one file, `config.yaml`, taken from `--config`, else the
+`CHUNKSILO_CONFIG` environment variable, else `./config.yaml`, else
+`~/.config/chunksilo/config.yaml`. Every setting is optional and
+`chunksilo --dump-defaults` prints them all with their defaults.
+
+```yaml
+indexing:
+  directories:
+    - "./data"
+    - "/mnt/docs"
+
+retrieval:
+  rerank_top_k: 5
+
+storage:
+  storage_dir: "./storage"
+  model_cache_dir: "./models"
+```
+
+The full reference — per-directory filters, revision handling, retrieval
+tuning, Confluence and Jira, share locations, the HTTP transport, and what
+changed for users upgrading from 2.x — is in
+[docs/configuration.md](docs/configuration.md).
+
+## The `search_docs` tool
+
+`search_docs` answers with the files whose names matched and the individual
+chunks that carry the answer, each located by a URI, the page or line it came
+from, and its heading path:
+
+```json
+{
+  "matched_files": [
+    { "uri": "file:///docs/database-configuration.docx", "unc": null, "score": 0.8432 }
+  ],
+  "num_matched_files": 1,
+  "chunks": [
+    {
+      "text": "To configure the database connection, set the DATABASE_URL environment variable...",
+      "score": 0.912,
+      "location": {
+        "uri": "file:///docs/setup-guide.pdf",
+        "unc": null,
+        "page": 12,
+        "line": null,
+        "heading_path": ["Getting Started", "Configuration", "Database"]
+      }
+    }
+  ],
+  "num_chunks": 1,
+  "query": "how to configure the database",
+  "retrieval_time": "0.42s"
+}
+```
+
+See [docs/tools.md](docs/tools.md) for every field, including how files on a
+mounted network share are presented.
+
+## Documentation
+
+- [docs/configuration.md](docs/configuration.md) — the config file, where it is discovered, and a reference for every setting.
+- [docs/cli.md](docs/cli.md) — `chunksilo` and `chunksilo-mcp` options, including `--check-files` for explaining why a file is or is not indexed.
+- [docs/tools.md](docs/tools.md) — the `search_docs` result shape.
+- [docs/mcp-clients.md](docs/mcp-clients.md) — client setup for Claude Code, Claude Desktop, Cline, Roo Code and opencode, over stdio or HTTP.
+- [docs/reverse-proxy.md](docs/reverse-proxy.md) — sharing an instance behind an authenticating reverse proxy, with oauth2-proxy and KeyCloak group-based access.
+- [docs/troubleshooting.md](docs/troubleshooting.md) — indexing surprises, network mounts, offline mode and more.
+
+## License
+
+Apache-2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE).

chunksilo-3.0.0/README.md

@@ -0,0 +1,132 @@
+<p align="center">
+  <img src="https://raw.githubusercontent.com/Chetic/chunksilo/main/chunksilo.png" alt="ChunkSilo Logo" width="500">
+</p>
+
+<p align="center">
+  <img src="demo/demo.gif" alt="ChunkSilo terminal demo" width="720">
+</p>
+
+# ChunkSilo MCP Server
+
+ChunkSilo is like a local Google for your documents. It uses semantic search — matching by meaning rather than exact keywords — so your LLM can find relevant information across all your files even when the wording differs from your query. Point it at your PDFs, Word docs, Markdown, and text files, and it builds a fully searchable index locally on your machine.
+
+- Runs entirely on your machine — no servers, no infrastructure
+- Semantic search + keyword filename matching across PDF, DOCX, DOC, Markdown, and TXT
+- Incremental indexing — only reprocesses new or changed files, and never writes inside an indexed directory
+- Revision-aware indexing — files that are revisions of one document (`Spec_v2.pdf`, `.../Rev B/Spec.docx`) are grouped and only the newest is indexed; review-comment copies are skipped
+- Heading-aware results with source links back to the original file, spread across documents instead of one document's every chunk
+- Date filtering and recency boosting
+- Files indexed from a mounted network share can be presented at the share's own location (an `smb://` URI plus the Windows UNC path)
+- Serves MCP over stdio to a local client, or over streamable-http behind a reverse proxy of your own (oauth2-proxy + KeyCloak walkthrough included)
+- Optional Confluence and Jira integrations (supports Cloud and Server/Data Center)
+
+## Installation
+
+### Option A: Install from PyPI (Recommended)
+
+Requires Python 3.11 or later. Models are downloaded automatically on first run (~250MB). The first run may appear to pause while models download — this is normal.
+
+```bash
+pip install chunksilo
+```
+
+Confluence and Jira support is included by default — just provide a config file
+to enable them. (`pip install chunksilo[confluence,jira]` still works as an alias
+for backward compatibility.)
+
+Then:
+1. **Create** a config file at `~/.config/chunksilo/config.yaml` (see [Configuration](#configuration))
+2. **Build** the index: `chunksilo --build-index`
+3. **Configure** your MCP client (see [docs/mcp-clients.md](docs/mcp-clients.md))
+
+### Option B: Offline Bundle
+
+A self-contained package with pre-downloaded models, ideal for air-gapped environments or systems without Python installed.
+
+Download from the [Releases page](https://github.com/Chetic/chunksilo/releases):
+
+1. **Download** the `chunksilo-vX.Y.Z-manylinux_2_34_x86_64.tar.gz` file
+2. **Extract** and install:
+
+```bash
+tar -xzf chunksilo-vX.Y.Z-manylinux_2_34_x86_64.tar.gz
+cd chunksilo
+./setup.sh
+```
+
+3. **Edit** `config.yaml` to set your document directories
+4. **Build** the index: `./venv/bin/chunksilo --build-index`
+5. **Configure** your MCP client (see [docs/mcp-clients.md](docs/mcp-clients.md))
+
+## Configuration
+
+ChunkSilo reads one file, `config.yaml`, taken from `--config`, else the
+`CHUNKSILO_CONFIG` environment variable, else `./config.yaml`, else
+`~/.config/chunksilo/config.yaml`. Every setting is optional and
+`chunksilo --dump-defaults` prints them all with their defaults.
+
+```yaml
+indexing:
+  directories:
+    - "./data"
+    - "/mnt/docs"
+
+retrieval:
+  rerank_top_k: 5
+
+storage:
+  storage_dir: "./storage"
+  model_cache_dir: "./models"
+```
+
+The full reference — per-directory filters, revision handling, retrieval
+tuning, Confluence and Jira, share locations, the HTTP transport, and what
+changed for users upgrading from 2.x — is in
+[docs/configuration.md](docs/configuration.md).
+
+## The `search_docs` tool
+
+`search_docs` answers with the files whose names matched and the individual
+chunks that carry the answer, each located by a URI, the page or line it came
+from, and its heading path:
+
+```json
+{
+  "matched_files": [
+    { "uri": "file:///docs/database-configuration.docx", "unc": null, "score": 0.8432 }
+  ],
+  "num_matched_files": 1,
+  "chunks": [
+    {
+      "text": "To configure the database connection, set the DATABASE_URL environment variable...",
+      "score": 0.912,
+      "location": {
+        "uri": "file:///docs/setup-guide.pdf",
+        "unc": null,
+        "page": 12,
+        "line": null,
+        "heading_path": ["Getting Started", "Configuration", "Database"]
+      }
+    }
+  ],
+  "num_chunks": 1,
+  "query": "how to configure the database",
+  "retrieval_time": "0.42s"
+}
+```
+
+See [docs/tools.md](docs/tools.md) for every field, including how files on a
+mounted network share are presented.
+
+## Documentation
+
+- [docs/configuration.md](docs/configuration.md) — the config file, where it is discovered, and a reference for every setting.
+- [docs/cli.md](docs/cli.md) — `chunksilo` and `chunksilo-mcp` options, including `--check-files` for explaining why a file is or is not indexed.
+- [docs/tools.md](docs/tools.md) — the `search_docs` result shape.
+- [docs/mcp-clients.md](docs/mcp-clients.md) — client setup for Claude Code, Claude Desktop, Cline, Roo Code and opencode, over stdio or HTTP.
+- [docs/reverse-proxy.md](docs/reverse-proxy.md) — sharing an instance behind an authenticating reverse proxy, with oauth2-proxy and KeyCloak group-based access.
+- [docs/troubleshooting.md](docs/troubleshooting.md) — indexing surprises, network mounts, offline mode and more.
+
+## License
+
+Apache-2.0. See [LICENSE](LICENSE) and [NOTICE](NOTICE).

{chunksilo-2.4.0 → chunksilo-3.0.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "chunksilo"
-version = "2.4.0"
+version = "3.0.0"
 description = "Local RAG-based semantic document search with MCP server interface"
 license = "Apache-2.0"
 requires-python = ">=3.11"

{chunksilo-2.4.0 → chunksilo-3.0.0}/requirements.txt

@@ -5,7 +5,7 @@ llama-index-retrievers-bm25>=0.1.0,<1
 pillow>=10.3.0,<11
 pypdf>=5.1.0,<7
 python-docx>=1.1.0,<2
-mcp>=1.0.0,<2
+mcp>=1.10.0,<2
 huggingface-hub>=0.22.0,<2
 flashrank>=0.1.0,<1
 fastembed>=0.5.0,<1

{chunksilo-2.4.0 → chunksilo-3.0.0}/src/chunksilo/__init__.py

@@ -1,4 +1,4 @@
 # SPDX-License-Identifier: Apache-2.0
 """ChunkSilo - Local RAG-based semantic document search."""
 
-__version__ = "2.4.0"
+__version__ = "3.0.0"

chunksilo-3.0.0/src/chunksilo/cfgload.py

@@ -0,0 +1,350 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: Apache-2.0
+"""
+Shared configuration loading for ChunkSilo.
+
+Loads configuration from config.yaml, searching in standard locations. The
+first load with an explicit path (``--config``) becomes the process-wide
+active configuration: every later no-argument ``load_config()`` or ``get()``
+call answers from it, so a path given on the command line governs the whole
+process rather than only the call sites it was threaded through.
+"""
+import copy
+import logging
+import os
+from pathlib import Path
+from typing import Any
+
+import yaml
+
+logger = logging.getLogger(__name__)
+
+# Retrieval models bundled inside the installed package (populated at wheel-build
+# time). Present in air-gapped wheel installs; absent in plain source checkouts.
+_BUNDLED_MODELS_DIR = Path(__file__).resolve().parent / "_bundled_models"
+
+
+def _find_config() -> Path:
+    """Find config.yaml using a priority-based search.
+
+    Search order:
+    1. CHUNKSILO_CONFIG environment variable
+    2. ./config.yaml (current working directory)
+    3. ~/.config/chunksilo/config.yaml (XDG standard)
+    """
+    env_path = os.environ.get("CHUNKSILO_CONFIG")
+    if env_path:
+        return Path(env_path)
+
+    cwd_path = Path.cwd() / "config.yaml"
+    if cwd_path.exists():
+        return cwd_path
+
+    xdg_path = Path.home() / ".config" / "chunksilo" / "config.yaml"
+    if xdg_path.exists():
+        return xdg_path
+
+    # Return cwd path as default (will fall through to defaults if not found)
+    return cwd_path
+
+
+CONFIG_PATH = _find_config()
+
+# Default file type patterns - the single definition; index.py imports these.
+DEFAULT_INCLUDE_PATTERNS = ["**/*.pdf", "**/*.md", "**/*.txt", "**/*.docx", "**/*.doc"]
+
+DEFAULT_EXCLUDE_PATTERNS = [
+    "**/.git/**",
+    "**/node_modules/**",
+    "**/__pycache__/**",
+    "**/.venv/**",
+    "**/venv/**",
+    "**/.tox/**",
+    "**/.mypy_cache/**",
+    "**/.pytest_cache/**",
+    "**/.eggs/**",
+    "**/*.egg-info/**",
+    "**/.DS_Store",
+]
+
+# Filename markers of copies made to collect review comments, matched
+# case-insensitively against the filename without its extension. Setting
+# indexing.versioning.review_copy_patterns replaces this list wholesale;
+# [] turns detection off.
+DEFAULT_REVIEW_COPY_PATTERNS = [
+    r"\breview\b",
+    r"\bcomments?\b",
+    r"\bkommentar(?:er)?\b",
+    r"\bgranskning\b",
+    r"\bcopy\b",  # "Copy of Spec", "Spec - Copy (2)"
+    r"\bkopia\b",  # "Kopia av Spec", "Spec - kopia"
+]
+
+_DEFAULTS: dict[str, Any] = {
+    "indexing": {
+        "directories": ["./data"],
+        "defaults": {
+            "include": DEFAULT_INCLUDE_PATTERNS,
+            "exclude": DEFAULT_EXCLUDE_PATTERNS,
+            "recursive": True,
+            "case_sensitive": False,
+        },
+        "chunk_size": 512,
+        "chunk_overlap": 50,
+        "batch_size": 200,  # upper bound; adaptive sizing may shrink a batch
+        "parallel_workers": 8,  # 1 = load files serially
+        "checkpoint_interval_files": 500,
+        "checkpoint_interval_seconds": 300,
+        "per_file_seconds": 300,  # give up on a single file; 0 disables
+        "scan_item_seconds": 30,  # timeout for stat/hash/walk during scanning
+        "slow_file_threshold_seconds": 30,  # warn when one file takes longer
+        "versioning": {
+            # Group files that are revisions of one document (revision-named
+            # directories or filename tokens like "Spec PB3.docx") and index
+            # only the newest member of each group, by modification time.
+            "enabled": True,
+            # Regexes tried in order against the full path; files sharing an
+            # extracted document ID form one group. The first capture group
+            # (or the whole match) is the ID. Empty = group by normalized
+            # name/path only.
+            "doc_id_patterns": [],
+            # Extra revision-token regex fragments (case-insensitive), added
+            # to the built-in grammar for directory names and filename
+            # suffixes alike.
+            "extra_revision_tokens": [],
+            "review_copy_patterns": DEFAULT_REVIEW_COPY_PATTERNS,
+            # Index older revisions too, stamped superseded and ranked below
+            # current documents in results.
+            "index_superseded": False,
+        },
+    },
+    "retrieval": {
+        "embed_model_name": "BAAI/bge-small-en-v1.5",
+        "embed_top_k": 20,
+        "rerank_model_name": "ms-marco-MiniLM-L-12-v2",
+        "rerank_top_k": 5,
+        "rerank_candidates": 100,
+        "score_threshold": 0.1,
+        "recency_boost": 0.3,
+        "recency_half_life_days": 365,
+        # At most this many chunks of one document in the final results;
+        # freed slots go to the next-best other documents. 0 disables.
+        "max_chunks_per_doc": 2,
+        "offline": False,
+    },
+    "confluence": {
+        "url": "",
+        "username": "",
+        "api_token": "",
+        "timeout": 10.0,
+        "max_results": 30,
+    },
+    "jira": {
+        "url": "",
+        "username": "",
+        "api_token": "",
+        "timeout": 10.0,
+        "max_results": 30,
+        "projects": [],  # Empty list = all accessible projects
+        "include_comments": False,
+        "include_custom_fields": False,
+    },
+    "ssl": {
+        # CA bundle for environments with TLS interception or a private CA:
+        # used for model downloads and for the Confluence/Jira APIs.
+        "ca_bundle_path": "",
+    },
+    "storage": {
+        "storage_dir": "./storage",
+        "model_cache_dir": "./models",
+    },
+    "server": {
+        "transport": "stdio",  # stdio | streamable-http
+        # Loopback by default. The HTTP transport performs no authentication
+        # of its own: exposing the port is a deliberate act, taken together
+        # with TLS termination and a login layer in front of this server.
+        "host": "127.0.0.1",
+        "port": 8400,
+        # The URL clients reach this server on when a reverse proxy sits in
+        # front, e.g. "https://search.example.com". The MCP transport rejects
+        # any Host header other than the bind address unless it is allowed
+        # here, and a proxy forwards the public hostname. Empty = only the
+        # bind address is accepted.
+        "public_url": "",
+        # Further Host header values and Origins to accept ("host:*" = any port).
+        "allowed_hosts": [],
+        "allowed_origins": [],
+        # Issuer URLs of the authorization servers (e.g. a KeyCloak realm) MCP
+        # clients should obtain a token from. Published as RFC 9728
+        # protected-resource metadata at /.well-known/oauth-protected-resource
+        # so clients can discover them; this server never verifies tokens -
+        # that is the reverse proxy's job. Empty = the metadata is not served.
+        "authorization_servers": [],
+    },
+    # Present files indexed from a mounted network share at the share's own
+    # location (an smb:// URI plus the Windows UNC path) instead of the
+    # server-local mount path, so a client on another machine gets a location
+    # it can open. Uncovered files keep their file:// URIs.
+    # [{prefix: "/mnt/docs", unc: "//nas/docs"}]
+    "shares": [],
+}
+
+# Options that no longer exist, mapped to what replaced them. A config file
+# that still sets one gets a warning instead of silently falling back to the
+# built-in default.
+_REMOVED_KEYS = {
+    "indexing.timeout": "indexing.per_file_seconds (0 disables) and indexing.scan_item_seconds",
+    "indexing.logging": "indexing.slow_file_threshold_seconds",
+    "indexing.enable_parallel_loading": "indexing.parallel_workers (1 = serial)",
+    "indexing.enable_adaptive_batching": "indexing.batch_size (an upper bound; sizing adapts automatically)",
+    "indexing.max_memory_mb": "indexing.batch_size (an upper bound; sizing adapts automatically)",
+    "retrieval.bm25_similarity_top_k": "nothing; the number of file-name matches is fixed",
+}
+
+# The active configuration: the merged result of the last load, plus the path
+# it came from. An explicit-path load replaces it; no-argument loads reuse it.
+_config_cache: dict[str, Any] | None = None
+_active_path: Path | None = None
+
+
+def _warn_removed_keys(user_config: dict[str, Any], path: Path) -> None:
+    """Warn about options the file sets that the schema no longer has."""
+    for dotted, replacement in _REMOVED_KEYS.items():
+        section, _, key = dotted.partition(".")
+        block = user_config.get(section)
+        if isinstance(block, dict) and key in block:
+            logger.warning(
+                "%s: '%s' is no longer used and is ignored; use %s instead",
+                path, dotted, replacement,
+            )
+
+
+def _deep_merge(base: dict[str, Any], override: dict[str, Any]) -> dict[str, Any]:
+    """Deep merge override into base, returning a new dict."""
+    result = base.copy()
+    for key, value in override.items():
+        if value is None and key in result:
+            # An empty YAML value ("chunk_size:" or a commented-out section)
+            # parses as None; keep the default instead of poisoning it.
+            continue
+        if key in result and isinstance(result[key], dict) and isinstance(value, dict):
+            result[key] = _deep_merge(result[key], value)
+        else:
+            result[key] = value
+    return result
+
+
+def _has_embedding_cache(cache_dir: Path) -> bool:
+    """True if ``cache_dir`` holds a HuggingFace-style embedding model cache."""
+    try:
+        return any(cache_dir.glob("models--*"))
+    except OSError:
+        return False
+
+
+def _resolve_bundled_models(config: dict[str, Any]) -> dict[str, Any]:
+    """Transparently fall back to models bundled in the package.
+
+    If the wheel ships the retrieval models and the configured cache directory
+    does not already contain them, point the cache at the bundled copy and enable
+    offline mode. This lets a pip-installed wheel run in an air-gapped environment
+    with zero extra configuration. A user-configured cache that already holds the
+    models always wins, so explicit setups are never overridden.
+
+    Detection looks for any HuggingFace-style ``models--*`` cache directory rather
+    than matching the configured embedding name: fastembed stores the model under
+    its upstream source repo (e.g. ``models--qdrant--bge-small-en-v1.5-onnx-q``),
+    not under the configured alias ``BAAI/bge-small-en-v1.5``.
+    """
+    if not _has_embedding_cache(_BUNDLED_MODELS_DIR):
+        return config  # No models bundled in this install.
+
+    configured = Path(config["storage"]["model_cache_dir"]).expanduser()
+    if configured.resolve() != _BUNDLED_MODELS_DIR.resolve() and _has_embedding_cache(
+        configured
+    ):
+        return config  # Configured cache already has models; respect it.
+
+    result = copy.deepcopy(config)
+    result["storage"]["model_cache_dir"] = str(_BUNDLED_MODELS_DIR)
+    result["retrieval"]["offline"] = True
+    logger.info(
+        "Using bundled retrieval models at %s (offline mode)", _BUNDLED_MODELS_DIR
+    )
+    return result
+
+
+def load_config(config_path: Path | str | None = None) -> dict[str, Any]:
+    """Load configuration from YAML file with defaults.
+
+    Args:
+        config_path: Optional path to config file. When given, the result
+                    becomes the process-wide active configuration that
+                    no-argument calls (and :func:`get`) answer from. When
+                    None, returns the active configuration, loading from the
+                    auto-discovered CONFIG_PATH on first use.
+
+    Returns:
+        Configuration dictionary with defaults merged in.
+    """
+    global _config_cache, _active_path
+
+    if _config_cache is not None and (
+        config_path is None or Path(config_path) == _active_path
+    ):
+        return _config_cache
+
+    path = Path(config_path) if config_path else CONFIG_PATH
+
+    # Callers may mutate the returned dict; never hand out the defaults themselves.
+    defaults = copy.deepcopy(_DEFAULTS)
+
+    if not path.exists():
+        logger.info("Config file not found at %s; using built-in defaults", path)
+        result = _resolve_bundled_models(defaults)
+    else:
+        logger.info("Using config: %s", path)
+        with open(path, encoding="utf-8") as f:
+            user_config = yaml.safe_load(f) or {}
+        _warn_removed_keys(user_config, path)
+        result = _deep_merge(defaults, user_config)
+        result = _resolve_bundled_models(result)
+
+    _config_cache = result
+    _active_path = path
+    return result
+
+
+def get(key: str, default: Any = None) -> Any:
+    """Get a value from the active configuration by dot-notation key.
+
+    Args:
+        key: Dot-separated key path (e.g., 'retrieval.embed_top_k')
+        default: Value to return if key not found
+
+    Returns:
+        Configuration value or default.
+
+    Example:
+        >>> get('retrieval.embed_top_k')
+        20
+        >>> get('storage.storage_dir')
+        './storage'
+    """
+    config = load_config()
+    keys = key.split(".")
+    value: Any = config
+    for k in keys:
+        if isinstance(value, dict) and k in value:
+            value = value[k]
+        else:
+            return default
+    return value
+
+
+def reload_config() -> dict[str, Any]:
+    """Force reload configuration from disk, clearing the active config."""
+    global _config_cache, _active_path
+    _config_cache = None
+    _active_path = None
+    return load_config()