knowledge-rag 4.11.0__tar.gz → 5.0.0__tar.gz

{knowledge_rag-4.11.0 → knowledge_rag-5.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: knowledge-rag
-Version: 4.11.0
+Version: 5.0.0
 Summary: Local RAG System for Claude Code — Hybrid search + Cross-encoder Reranking + 13 MCP Tools + 20 Format Parsers. Zero external servers.
 Project-URL: Homepage, https://github.com/lyonzin/knowledge-rag
 Project-URL: Repository, https://github.com/lyonzin/knowledge-rag
@@ -87,7 +87,6 @@ Structured data (schema.org SoftwareApplication) — indexed by search engines,
 ![License](https://img.shields.io/badge/license-MIT-yellow.svg)
 ![Platform](https://img.shields.io/badge/platform-Windows%20%7C%20Linux%20%7C%20macOS-lightgrey.svg)
 ![GPU](https://img.shields.io/badge/GPU-NVIDIA%20CUDA-76B900.svg?logo=nvidia)
-[![CI](https://github.com/lyonzin/knowledge-rag/actions/workflows/ci.yml/badge.svg)](https://github.com/lyonzin/knowledge-rag/actions/workflows/ci.yml)
 [![CodeQL](https://github.com/lyonzin/knowledge-rag/actions/workflows/security.yml/badge.svg)](https://github.com/lyonzin/knowledge-rag/actions/workflows/security.yml)
 [![Quality Gate](https://github.com/lyonzin/knowledge-rag/actions/workflows/quality-gate.yml/badge.svg)](https://github.com/lyonzin/knowledge-rag/actions/workflows/quality-gate.yml)
 [![Glama Score](https://glama.ai/mcp/servers/lyonzin/knowledge-rag/badges/score.svg)](https://glama.ai/mcp/servers/lyonzin/knowledge-rag)

{knowledge_rag-4.11.0 → knowledge_rag-5.0.0}/README.md

@@ -38,7 +38,6 @@ Structured data (schema.org SoftwareApplication) — indexed by search engines,
 ![License](https://img.shields.io/badge/license-MIT-yellow.svg)
 ![Platform](https://img.shields.io/badge/platform-Windows%20%7C%20Linux%20%7C%20macOS-lightgrey.svg)
 ![GPU](https://img.shields.io/badge/GPU-NVIDIA%20CUDA-76B900.svg?logo=nvidia)
-[![CI](https://github.com/lyonzin/knowledge-rag/actions/workflows/ci.yml/badge.svg)](https://github.com/lyonzin/knowledge-rag/actions/workflows/ci.yml)
 [![CodeQL](https://github.com/lyonzin/knowledge-rag/actions/workflows/security.yml/badge.svg)](https://github.com/lyonzin/knowledge-rag/actions/workflows/security.yml)
 [![Quality Gate](https://github.com/lyonzin/knowledge-rag/actions/workflows/quality-gate.yml/badge.svg)](https://github.com/lyonzin/knowledge-rag/actions/workflows/quality-gate.yml)
 [![Glama Score](https://glama.ai/mcp/servers/lyonzin/knowledge-rag/badges/score.svg)](https://glama.ai/mcp/servers/lyonzin/knowledge-rag)

{knowledge_rag-4.11.0 → knowledge_rag-5.0.0}/bench/dashboard/README.md

@@ -51,13 +51,13 @@ python -m http.server 8000 --directory /tmp/dashboard-preview
 | **FTS5** | `fts5 / lexical / fast[_-]?path` | 10 ms / 30 ms |
 | **Reindex** | `reindex / rebuild / swap` | 10 s / 60 s |
 | **Indexing** | `index / ingest / parse / chunk` | 100 ms / 500 ms |
-| **Memory** | Explicit `measurement_unit: bytes` overrides name matching | Per-workload RSS budget in bytes |
+| **Memory** | Explicit `measurement_unit: bytes` overrides name matching | Per-workload allocation budget in bytes |
 | **Concurrent** | `concurrent / thread / parallel` | 100 ms / 500 ms |
 | **Other** | — | 100 ms / 1 s |
 
 New benchmark files under `bench/` are automatically categorized as long as the test name contains one of these keywords. The first matching pattern wins. Explicit byte measurements always use the memory category. A cache-hit latency benchmark remains a timing measurement rather than a RAM measurement. Timing thresholds are display heuristics, not service-level guarantees.
 
-Memory rows read `rss_peak_delta_bytes`, `rss_retained_delta_bytes`, `rss_budget_bytes` and `rss_sampling_interval_ms` from the benchmark's `extra_info`. Each workload runs five times, sampling process RSS every 5 ms and applying an absolute budget to every run. Older artifacts without byte samples display `not measured` / `unavailable`; elapsed garbage-collection time is never relabeled as memory. Shorter allocation peaks can fall between samples, and RSS includes native allocations and allocator retention.
+Memory rows read `tracemalloc_peak_bytes` and `memory_budget_bytes` from the benchmark's `extra_info`. Each workload runs five times under `tracemalloc` and applies an absolute budget to every run. The peak counts every Python allocation the workload makes, so it does not depend on pages the allocator already holds; native allocations outside Python's allocator are not counted. Older artifacts without traced bytes display `not measured` / `unavailable`; elapsed garbage-collection time is never relabeled as memory.
 
 ## Data schema (`data.json`)
 
@@ -82,6 +82,6 @@ Memory rows read `rss_peak_delta_bytes`, `rss_retained_delta_bytes`, `rss_budget
 }
 ```
 
-For memory rows, `measurement_unit` is `bytes`; `median_ns`, `stddev_ns`, `ops` and `iqr_ratio` are null. `measurement_value` contains sampled peak RSS growth or null when the legacy artifact has no byte sample. The RSS budget and sampling interval are also retained. Sorting groups measurements by unit instead of treating bytes as nanoseconds.
+For memory rows, `measurement_unit` is `bytes`; `median_ns`, `stddev_ns`, `ops` and `iqr_ratio` are null. `measurement_value` contains the traced allocation peak or null when the legacy artifact has no traced bytes. `budget_bytes` is also kept. Sorting groups measurements by unit instead of treating bytes as nanoseconds.
 
 `dashboard.js` enriches each row with `category`, `category_label`, `category_color`, `status`, and `stddev_pct` at render time. `workload_version` identifies changes to the benchmark scenario; the CI gate reports different versions as incomparable and keeps enforcing its 10% threshold for unchanged, comparable scenarios.

{knowledge_rag-4.11.0 → knowledge_rag-5.0.0}/config.example.yaml

@@ -221,11 +221,11 @@ models:
 
     # Query / passage prefixes — prepended to text BEFORE embedding.
     # Required by some model families (intfloat/e5, thenlper/gte, ...);
-    # leave empty for bge / mxbai. When a profile ships prefixes, this
-    # override wins ONLY if the key is present here (empty string "" IS
-    # an explicit override).
-    query_prefix: ""
-    passage_prefix: ""
+    # not used by bge / mxbai. Keep them commented out with a profile: it
+    # sets its own prefixes, and any value here, even "", overrides them.
+    # Uncomment only for a custom model that needs them (e5 shown).
+    # query_prefix: "query: "
+    # passage_prefix: "passage: "
 
     # GPU acceleration mode (v4.8.0+):
     #   "auto"  → probe CUDA at first embedding; use if ready, fall back to CPU (default)

{knowledge_rag-4.11.0 → knowledge_rag-5.0.0}/mcp_server/__init__.py

@@ -1,6 +1,6 @@
 """Knowledge RAG MCP Server - Local Retrieval-Augmented Generation System"""
 
-__version__ = "4.11.0"
+__version__ = "5.0.0"
 __author__ = "Ailton Rocha (Lyon.)"
 
 from .config import Config  # noqa: E402

{knowledge_rag-4.11.0 → knowledge_rag-5.0.0}/mcp_server/config.py

@@ -1,5 +1,6 @@
 """Configuration for Knowledge RAG System v4.0.0 — YAML-configurable"""
 
+import datetime
 import os
 import sys
 from dataclasses import dataclass, field
@@ -559,6 +560,11 @@ def _yaml_embedding_has(key: str) -> bool:
     return key in emb
 
 
+# YAML scalars that keyword_routes / category_mappings convert to text (GH #261).
+# YAML reads an unquoted 2024-01-01 as a date; str() gives the folder name back.
+_TEXT_SCALAR_TYPES = (str, int, float, bool, datetime.date)
+
+
 # ============================================================================
 # CONFIG DATACLASS
 # ============================================================================
@@ -939,7 +945,11 @@ class Config:
         self._apply_embedding_profile(profile)
 
     def _apply_embedding_profile(self, profile: dict) -> None:
-        """Copy profile model/dim into config; prefixes only if user did not declare them."""
+        """Copy profile model/dim into config; prefixes only if user did not declare them.
+
+        A declared prefix that differs from a non-empty profile prefix still wins,
+        with a warning, so an empty one left from an old template is visible (GH #261).
+        """
         if _yaml_embedding_has("model"):
             print(
                 f"[WARN] Both models.embedding.model and profile="
@@ -953,6 +963,13 @@ class Config:
             self.query_prefix = profile["query_prefix"]
         if not _yaml_embedding_has("passage_prefix"):
             self.passage_prefix = profile["passage_prefix"]
+        for key in ("query_prefix", "passage_prefix"):
+            if profile[key] and getattr(self, key) != profile[key]:
+                print(
+                    f"[WARN] models.embedding.{key}={getattr(self, key)!r} overrides profile="
+                    f"'{self.embedding_profile}' prefix {profile[key]!r}; remove the key to use the profile's",
+                    file=sys.stderr,
+                )
 
     def _validate_embedding_types(self) -> None:
         """Type-check prefixes + embedding_dim + reranker settings (runs AFTER profile)."""
@@ -1036,22 +1053,47 @@ class Config:
         self.supported_formats = list(_DEFAULT_SUPPORTED_FORMATS)
 
     def _validate_lists_and_maps(self) -> None:
-        """Type-check exclude_patterns + keyword_routes + query_expansions in order."""
+        """Type-check exclude_patterns + keyword_routes + category_mappings + query_expansions in order."""
         self._validate_exclude_and_routes()
         self._validate_query_expansions()
 
     def _validate_exclude_and_routes(self) -> None:
-        """Ensure exclude_patterns is a str-only list and keyword_routes values are lists."""
+        """Keep exclude_patterns str-only; convert keyword_routes and category_mappings scalars to text.
+
+        YAML parses a bare ``4624`` as int, which breaks the str methods that
+        indexing and query routing call on these tables (GH #261). YAML has
+        already converted the scalar, so ``0123`` or ``yes`` must be quoted to
+        keep their spelling. Null, list and mapping entries are skipped with a
+        warning rather than turned into names such as ``"['a', 'b']"``. New
+        dicts are built so the module-level defaults are never mutated.
+        """
         if not isinstance(self.exclude_patterns, list):
             print(f"[WARN] exclude_patterns={self.exclude_patterns!r} invalid, using []", file=sys.stderr)
             self.exclude_patterns = []
         else:
             self.exclude_patterns = [p for p in self.exclude_patterns if isinstance(p, str)]
 
-        for cat, keywords in list(self.keyword_routes.items()):
-            if not isinstance(keywords, list):
+        routes: Dict[str, List[str]] = {}
+        for cat, keywords in self.keyword_routes.items():
+            if not isinstance(cat, _TEXT_SCALAR_TYPES):
+                print(f"[WARN] keyword_routes: ignoring key {cat!r}", file=sys.stderr)
+            elif not isinstance(keywords, list):
                 print(f"[WARN] keyword_routes.{cat} is not a list, removing", file=sys.stderr)
-                del self.keyword_routes[cat]
+            else:
+                routes[str(cat)] = [str(k) for k in keywords if isinstance(k, _TEXT_SCALAR_TYPES)]
+                for ignored in (k for k in keywords if not isinstance(k, _TEXT_SCALAR_TYPES)):
+                    print(f"[WARN] keyword_routes.{cat}: ignoring {ignored!r}", file=sys.stderr)
+        self.keyword_routes = routes
+        mappings: Dict[str, str] = {}
+        for path, category in self.category_mappings.items():
+            if not isinstance(path, _TEXT_SCALAR_TYPES):
+                print(f"[WARN] category_mappings: ignoring key {path!r}", file=sys.stderr)
+            elif not isinstance(category, _TEXT_SCALAR_TYPES):
+                print(f"[WARN] category_mappings.{path}: ignoring {category!r}", file=sys.stderr)
+            else:
+                # _detect_category matches against the lower-cased relative path.
+                mappings[str(path).lower()] = str(category)
+        self.category_mappings = mappings
 
     def _validate_query_expansions(self) -> None:
         """Type-check + merge query_expansions with query_expansion_groups."""

{knowledge_rag-4.11.0 → knowledge_rag-5.0.0}/mcp_server/data/config.example.yaml

@@ -221,11 +221,11 @@ models:
 
     # Query / passage prefixes — prepended to text BEFORE embedding.
     # Required by some model families (intfloat/e5, thenlper/gte, ...);
-    # leave empty for bge / mxbai. When a profile ships prefixes, this
-    # override wins ONLY if the key is present here (empty string "" IS
-    # an explicit override).
-    query_prefix: ""
-    passage_prefix: ""
+    # not used by bge / mxbai. Keep them commented out with a profile: it
+    # sets its own prefixes, and any value here, even "", overrides them.
+    # Uncomment only for a custom model that needs them (e5 shown).
+    # query_prefix: "query: "
+    # passage_prefix: "passage: "
 
     # GPU acceleration mode (v4.8.0+):
     #   "auto"  → probe CUDA at first embedding; use if ready, fall back to CPU (default)

{knowledge_rag-4.11.0 → knowledge_rag-5.0.0}/mcp_server/file_transaction.py

@@ -8,9 +8,33 @@ import tempfile
 from collections.abc import Iterator
 from contextlib import contextmanager
 from pathlib import Path
+from time import sleep
 
 logger = logging.getLogger(__name__)
 
+# ERROR_ACCESS_DENIED / ERROR_SHARING_VIOLATION: a scanner, indexer or reader holds
+# the path without FILE_SHARE_DELETE, and Windows cannot replace over it until it lets go.
+_WINDOWS_HELD_ERRORS = frozenset({5, 32})
+_REPLACE_RETRY_DELAYS = (0.05, 0.1, 0.2, 0.4, 0.8)
+
+
+def replace_with_retry(source: str | os.PathLike[str], destination: str | os.PathLike[str]) -> None:
+    """``os.replace`` that waits out a transient Windows handle on either path (GH #249).
+
+    Only PermissionError carrying winerror 5 or 32 is retried, for about 1.5 s; then
+    it is re-raised, so a real permission problem still surfaces. Elsewhere this is
+    a plain ``os.replace``.
+    """
+    for delay in _REPLACE_RETRY_DELAYS:
+        try:
+            os.replace(source, destination)
+            return
+        except PermissionError as exc:
+            if getattr(exc, "winerror", None) not in _WINDOWS_HELD_ERRORS:
+                raise
+        sleep(delay)
+    os.replace(source, destination)
+
 
 class StagedTextFile:
     """Prepare beside the destination so rename works on every supported OS."""
@@ -47,19 +71,19 @@ class StagedTextFile:
             ) as stream:
                 backup = Path(stream.name)
             try:
-                os.replace(self.destination, backup)
+                replace_with_retry(self.destination, backup)
             except BaseException:
                 backup.unlink(missing_ok=True)
                 raise
             self.backup = backup
-        os.replace(self.path, self.destination)
+        replace_with_retry(self.path, self.destination)
         self.published = True
 
     def close(self) -> None:
         """Restore old bytes after failure; never delete a backup whose restore failed."""
         if not self.committed:
             if self.backup is not None:
-                os.replace(self.backup, self.destination)
+                replace_with_retry(self.backup, self.destination)
             elif self.published:
                 self.destination.unlink(missing_ok=True)
         for temporary in (self.path, self.backup):

{knowledge_rag-4.11.0 → knowledge_rag-5.0.0}/mcp_server/fts5_index.py

@@ -27,6 +27,8 @@ from itertools import islice
 from pathlib import Path
 from typing import Any, Callable, Iterable, List, Optional, Sequence, Tuple
 
+from .file_transaction import replace_with_retry
+
 ChunkRow = Tuple[str, str, str, str]
 ChunkIterFactory = Callable[[], Iterable[ChunkRow]]
 ProgressCallback = Callable[[int, int], None]
@@ -137,7 +139,7 @@ class Fts5MigrationState:
                     json.dump(payload, tmp)
                     tmp.flush()
                     os.fsync(tmp.fileno())
-                os.replace(tmp_name, self._path)
+                replace_with_retry(tmp_name, self._path)
             except Exception:
                 try:
                     os.unlink(tmp_name)

{knowledge_rag-4.11.0 → knowledge_rag-5.0.0}/mcp_server/ingestion.py

@@ -406,7 +406,7 @@ class DocumentParser:
             ]
 
             for fname in files:
-                if fname.startswith(".rag-pending-"):
+                if fname.startswith((".rag-pending-", "~$")):  # staging file or Office owner file
                     continue
                 filepath = Path(root) / fname
                 # supported_formats holds suffixes (".go") and exact filenames ("Dockerfile")

{knowledge_rag-4.11.0 → knowledge_rag-5.0.0}/mcp_server/preflight.py

@@ -10,8 +10,10 @@ fixes below were validated on their environment before merging.
 
 from __future__ import annotations
 
+import contextlib
 import os
 import re
+import shutil
 import subprocess
 import sys
 import time
@@ -23,6 +25,8 @@ from .config import BASE_DIR, _get, config
 PROBE_ATTEMPTS = 3
 PROBE_RETRY_DELAY_SECONDS = 5
 PROBE_TIMEOUT_SECONDS = 30
+AUTO_REPAIR_BACKUPS_KEPT = 3  # GH #267: unbounded auto-repair backups once filled a disk (197 copies, 76 GB)
+_AUTO_REPAIR_DIR_NAME = re.compile(r"auto-repair-[0-9]{8}-[0-9]{6}")  # what _backup_active_index creates
 
 # Cloud-sync folders break ChromaDB (HNSW mmap) and SQLite WAL/SHM:
 # the provider opens handles on live files to upload them, racing with
@@ -134,32 +138,50 @@ def _backup_active_index(reason: str) -> Path:
     metadata_file = config.data_dir / "index_metadata.json"
     metadata_backup = backup_dir / f"index_metadata.{reason}.json"
     metadata_moved = False
-    if metadata_file.exists():
-        os.rename(str(metadata_file), str(metadata_backup))
-        metadata_moved = True
-
-    if config.chroma_dir.exists():
-        try:
+    try:
+        if metadata_file.exists():
+            os.rename(str(metadata_file), str(metadata_backup))
+            metadata_moved = True
+        if config.chroma_dir.exists():
             os.rename(str(config.chroma_dir), str(backup_dir / f"chroma_db.{reason}"))
-        except OSError:
-            if metadata_moved:
-                # Rollback the metadata move so the two files stay consistent.
-                # If the rollback itself fails (unlikely — dst dir is fresh)
-                # we still re-raise the original OSError so the caller can
-                # exit cleanly instead of leaving the system in a wedged state.
-                try:
-                    os.rename(str(metadata_backup), str(metadata_file))
-                except OSError:
-                    pass
-            raise
+    except OSError:
+        if metadata_moved:
+            # Rollback the metadata move so the two files stay consistent.
+            # If the rollback itself fails (unlikely — dst dir is fresh)
+            # we still re-raise the original OSError so the caller can
+            # exit cleanly instead of leaving the system in a wedged state.
+            with contextlib.suppress(OSError):
+                os.rename(str(metadata_backup), str(metadata_file))
+        raise
+    finally:
+        with contextlib.suppress(OSError):
+            backup_dir.rmdir()  # GH #267: an empty folder would take a kept slot; rmdir spares any holding data
 
     return backup_dir
 
 
+def _prune_auto_repair_backups(new_backup: Path) -> None:
+    """Keep only the newest AUTO_REPAIR_BACKUPS_KEPT auto-repair backups, ``new_backup`` included."""
+    with os.scandir(new_backup.parent) as entries:
+        # Symlinks are skipped here; rmtree refuses Windows junctions, so no link is ever followed.
+        other_dirs = {entry.name for entry in entries if entry.is_dir(follow_symlinks=False)} - {new_backup.name}
+    older = sorted((name for name in other_dirs if _AUTO_REPAIR_DIR_NAME.fullmatch(name)), reverse=True)
+    removed = 0
+    # new_backup takes one kept slot even when a clock change made its stamp sort below the others.
+    for name in older[AUTO_REPAIR_BACKUPS_KEPT - 1 :]:
+        try:
+            shutil.rmtree(new_backup.parent / name)
+            removed += 1
+        except OSError as exc:  # e.g. a file still held open on Windows
+            _log(f"[RECOVERY] could not delete old backup {new_backup.parent / name} ({exc}); will retry next repair")
+    _log(f"[RECOVERY] removed {removed} old auto-repair backup(s)")
+
+
 def _probe_chroma(timeout_seconds: int = PROBE_TIMEOUT_SECONDS) -> subprocess.CompletedProcess[str]:
     """Check Chroma in a child process so native crashes do not kill MCP startup."""
     code = r"""
 import chromadb
+from chromadb.errors import NotFoundError
 
 from mcp_server.config import config
 
@@ -168,7 +190,13 @@ if not config.chroma_dir.exists():
     raise SystemExit(0)
 
 client = chromadb.PersistentClient(path=str(config.chroma_dir))
-collection = client.get_or_create_collection(name=config.collection_name)
+try:
+    # GH #248/#252: read only. Creating it here persisted Chroma's default
+    # embedding function, which the server then refused to open.
+    collection = client.get_collection(name=config.collection_name, embedding_function=None)
+except NotFoundError:
+    print("missing")
+    raise SystemExit(0)
 print(collection.count())
 """
     env = os.environ.copy()
@@ -235,7 +263,12 @@ def run_preflight(timeout_seconds: int = PROBE_TIMEOUT_SECONDS) -> bool:
             f"to avoid partial corruption. Retry after freeing file handles."
         )
         raise SystemExit(75) from exc
-    _log(f"[RECOVERY] moved active index to {backup_dir}")
+    try:
+        if backup_dir.exists():  # GH #267: the empty folder is dropped when there was nothing to move
+            _log(f"[RECOVERY] moved active index to {backup_dir}")
+            _prune_auto_repair_backups(backup_dir)
+    except OSError as exc:  # best effort: old backups never fail the repair
+        _log(f"[RECOVERY] could not prune old backups in {backup_dir.parent} ({exc})")
     return True
 
 

{knowledge_rag-4.11.0 → knowledge_rag-5.0.0}/mcp_server/security.py

@@ -272,6 +272,10 @@ def neutralize_injection_sentinels(text: str) -> str:
     return text
 
 
+#: Longest source identifier the provenance marker keeps; longer ones are cut.
+_MAX_SOURCE_CHARS: Final[int] = 512
+
+
 def _escape_attribute(value: str) -> str:
     """Make a source identifier safe to embed in an XML-ish attribute.
 
@@ -284,11 +288,11 @@ def _escape_attribute(value: str) -> str:
 
     Returns:
         str: Value with quotes, angle brackets and control characters removed,
-        truncated to 512 characters.
+        truncated to ``_MAX_SOURCE_CHARS`` (512) characters.
     """
     cleaned = "".join(ch for ch in value if ch.isprintable())
     cleaned = cleaned.replace('"', "").replace("'", "").replace("<", "").replace(">", "")
-    return cleaned[:512]
+    return cleaned[:_MAX_SOURCE_CHARS]
 
 
 def wrap_external_content(content: str, source: str, digest: str | None = None) -> str:

{knowledge_rag-4.11.0 → knowledge_rag-5.0.0}/mcp_server/server.py

@@ -64,7 +64,7 @@ from watchdog.observers import Observer
 # Local imports
 from . import __version__
 from .config import config
-from .file_transaction import staged_text_file
+from .file_transaction import replace_with_retry, staged_text_file
 from .filesystem import io_path
 from .fts5_index import Fts5LexicalIndex, Fts5MigrationError, Fts5NotReadyError
 from .ingestion import Document, DocumentParser
@@ -85,8 +85,12 @@ from .ratelimit import rate_limited
 from .search_candidates import select_rerank_candidates
 from .search_scope import FolderScope, ScopeKey
 from .security import (
+    _MAX_SOURCE_CHARS,
     BearerAuthMiddleware,
     PathEscapeError,
+    _escape_attribute,
+    content_sha256,
+    detect_external_marker,
     is_path_within,
     sanitize_external_content,
     validate_path_within,
@@ -1468,7 +1472,7 @@ class DocumentWatcher(FileSystemEventHandler):
         Suffixes are lowercased to match DocumentParser.parse_directory().
         """
         p = Path(path)
-        if p.name.startswith(".rag-pending-"):
+        if p.name.startswith((".rag-pending-", "~$")):  # staging file or Office owner file
             return False
         return p.suffix.lower() in config.supported_formats or p.name in config.supported_formats
 
@@ -1541,6 +1545,30 @@ class DocumentWatcher(FileSystemEventHandler):
 # distinguishes from ``None`` (no existing doc yet, index as fresh).
 _SKIP_DOC = object()
 
+# Characters read to find a saved page's provenance marker. It is the first line,
+# and its tag, source (at most 512 characters) and SHA-256 take at most 614.
+_MARKER_HEAD_CHARS = 1024
+
+
+def _is_saved_from(path: Path, url: str) -> bool:
+    """Whether the provenance marker at the top of ``path`` proves it holds ``url``'s page.
+
+    A file without the marker, or one that cannot be read, proves nothing. The
+    marker keeps the URL as ``_escape_attribute`` leaves it (quotes, angle brackets
+    and non-printable characters removed), so two URLs that differ only in those
+    characters count as the same page. A marker URL cut at ``_MAX_SOURCE_CHARS``
+    proves nothing either, so each re-add of a URL that long takes the next free
+    name, until the full-digest name is reached and then refreshed.
+    """
+    try:
+        with path.open(encoding="utf-8", errors="replace") as page:
+            marker = detect_external_marker(page.read(_MARKER_HEAD_CHARS))
+    except OSError as exc:
+        # Held by another program or an offline cloud placeholder: owner unproven.
+        print(f"[WARN] add_from_url: cannot read {path}, saving under a hashed name: {exc}", file=sys.stderr)
+        return False
+    return marker is not None and len(marker[0]) < _MAX_SOURCE_CHARS and marker[0] == _escape_attribute(url)
+
 
 class KnowledgeOrchestrator:
     """Main orchestrator for knowledge retrieval with semantic search + keyword routing"""
@@ -1642,17 +1670,41 @@ class KnowledgeOrchestrator:
     def _safe_get_collection(self):
         """Open the collection without discarding data on initialization errors."""
         try:
-            return self.chroma_client.get_or_create_collection(
-                name=config.collection_name,
-                embedding_function=self.embed_fn,
-                metadata={"description": "Knowledge base for RAG"},
-            )
+            return self._open_collection()
         except Exception as exc:
-            raise RuntimeError(
-                f"Could not open ChromaDB collection {config.collection_name!r} "
-                f"at {config.chroma_dir}: {exc}. Existing index preserved. "
-                "Check database access and embedding configuration before retrying."
-            ) from exc
+            if not self._drop_empty_default_ef_collection():
+                raise RuntimeError(
+                    f"Could not open ChromaDB collection {config.collection_name!r} "
+                    f"at {config.chroma_dir}: {exc}. Existing index preserved. "
+                    "Check database access and embedding configuration before retrying."
+                ) from exc
+        return self._open_collection()
+
+    def _open_collection(self) -> Any:
+        """Get or create the production collection bound to the FastEmbed function."""
+        return self.chroma_client.get_or_create_collection(
+            name=config.collection_name,
+            embedding_function=self.embed_fn,
+            metadata={"description": "Knowledge base for RAG"},
+        )
+
+    def _drop_empty_default_ef_collection(self) -> bool:
+        """Drop the collection only when it is empty and bound to Chroma's default EF.
+
+        GH #248/#252: the preflight probe used to create it without an embedding
+        function, so Chroma persisted ``default`` and refused FastEmbed on every
+        later open. An empty one holds nothing; one with data is never touched.
+        """
+        try:
+            stale = self.chroma_client.get_collection(name=config.collection_name, embedding_function=None)
+            ef_name = (stale.configuration_json.get("embedding_function") or {}).get("name")
+            if ef_name != "default" or stale.count() != 0:
+                return False
+            self.chroma_client.delete_collection(name=config.collection_name)
+        except Exception:
+            return False
+        print("[RECOVERY] Recreating empty collection bound to Chroma's default embedding (GH #248)", file=sys.stderr)
+        return True
 
     def _check_dimension_mismatch(self) -> bool:
         """Check if stored embeddings have different dimension than current config.
@@ -1721,8 +1773,10 @@ class KnowledgeOrchestrator:
             try:
                 count = self.collection.count()
                 if count == 0:
-                    # Empty collection — nothing to build. Leave flag False so
-                    # a later call retries once documents become available.
+                    # Empty collection — nothing to build, and the old generation must
+                    # not outlive its chunks (#259). Leave flag False so a later
+                    # call retries once documents become available.
+                    self.bm25_index.clear()
                     return
 
                 # Batch to avoid SQLite "too many SQL variables" (chromadb 1.x
@@ -1851,8 +1905,17 @@ class KnowledgeOrchestrator:
 
         Done BEFORE indexing so moved files are not blocked by stale content
         hashes. Mutates ``stats`` (chunks_removed, deleted) and evicts from
-        both ``_indexed_docs`` and ``_source_to_docid``.
+        both ``_indexed_docs`` and ``_source_to_docid``. An unavailable or empty
+        ``documents_dir`` (unplugged drive, unmounted share, folder recreated at
+        startup) prunes nothing; remove_document removes on purpose (#259).
         """
+        if not io_path(config.documents_dir).is_dir() or (not documents and self._indexed_docs):
+            print(
+                f"[WARN] documents_dir unavailable or empty, skipping orphan pruning: {config.documents_dir}",
+                file=sys.stderr,
+            )
+            stats["errors"] += 1
+            return
         current_paths = {str(Path(getattr(doc, "source", doc)).resolve()) for doc in documents}
         for doc_id, info in list(self._indexed_docs.items()):
             source = Path(info.get("source", ""))
@@ -1945,11 +2008,20 @@ class KnowledgeOrchestrator:
                 if not is_path_within(config.documents_dir, doc):
                     raise PathEscapeError(f"File escaped documents directory: {doc}")
                 doc = self.parser.parse_file(doc)
-                if doc is None:
-                    raise ValueError("Document has no indexable content")
                 after_parse = io_path(source).stat()
                 if (source_stat.st_mtime_ns, source_stat.st_size) != (after_parse.st_mtime_ns, after_parse.st_size):
                     raise RuntimeError("Source changed during parsing; retry indexing this document")
+                # Empty or text-less files are skipped, never an error that aborts a rebuild (#259).
+                if doc is None:
+                    if repair_ids is not None:
+                        self._remove_repair_chunks(source, [value for value in repair_ids if value != existing])
+                    if existing:
+                        self._evict_stale_doc(existing, stats)
+                    else:
+                        stats["skipped"] += 1
+                    if repair_ids is not None:
+                        tracking["repaired_sources"].add(str(source.resolve()))
+                    return
             existing_doc_id = self._resolve_existing_or_skip(doc, force, path_to_docid, stats, tracking)
             if existing_doc_id is _SKIP_DOC:
                 return
@@ -2042,14 +2114,15 @@ class KnowledgeOrchestrator:
         return stored_mtime == current_mtime and stored_size == current_size
 
     def _evict_stale_doc(self, existing_doc_id: str, stats: Dict[str, Any]) -> None:
-        """Remove chunks + metadata for a doc that needs reindexing. Bumps updated."""
+        """Remove chunks + metadata of a doc whose file has no indexable content left. Bumps deleted."""
         removed = self._remove_document_chunks(existing_doc_id)
         stats["chunks_removed"] += removed
         src = self._indexed_docs[existing_doc_id].get("source", "")
         if src:
             self._source_to_docid.pop(str(Path(src).resolve()), None)
         del self._indexed_docs[existing_doc_id]
-        stats["updated"] += 1
+        stats["deleted"] += 1
+        print(f"[WARN] Removed {removed} chunks of {src}: the file has no indexable content", file=sys.stderr)
 
     def _register_indexed_doc(self, doc, chunks_added: int, source_stat=None) -> None:
         """Record the source version actually parsed, before potentially slow inference."""
@@ -2387,6 +2460,17 @@ class KnowledgeOrchestrator:
         if self._reindex_progress.get("active"):
             return {"status": "already_running", "progress": dict(self._reindex_progress)}
 
+        # GH #249: index_all refuses at once while a watcher pass or a document
+        # write holds the index lock, so "started" would hide the refusal.
+        # nuclear_rebuild waits for the lock instead.
+        # ponytail: a probe, not admission. A pass that takes the lock between this
+        # check and the worker's index_all is still refused, as before, and shows up
+        # as last_result.skipped_reason; move admission into the worker if it matters.
+        if mode != "nuclear_rebuild":
+            if not self._index_lock.acquire(blocking=False):
+                return {"status": "already_running", "progress": {**self._reindex_progress, "operation": None}}
+            self._index_lock.release()
+
         self._reindex_progress = self._fresh_reindex_progress(mode, resume_state)
 
         # GH #163 (v4.8.3): propagate force through smart_reindex so
@@ -2440,7 +2524,7 @@ class KnowledgeOrchestrator:
         resume_state: Optional[Dict[str, Any]] = None,
         force: bool = False,
     ) -> Dict[str, Any]:
-        """Smart reindex: incremental detection + BM25 rebuild + orphan cleanup.
+        """Smart reindex: index_all (unchanged files skipped unless ``force``) + BM25 rebuild + orphan cleanup.
 
         When ``resume_state`` is provided (v4.8.0 Fase 4), docs listed in
         it are skipped so the interrupted previous run picks up where it
@@ -2471,7 +2555,7 @@ class KnowledgeOrchestrator:
             return stats
 
         print("[REINDEX] Rebuilding BM25 index...", file=sys.stderr)
-        self.bm25_index.clear()
+        # No clear(): queries keep the published generation until the rebuild replaces it (#259).
         self._bm25_initialized = False
         self._ensure_bm25_index()
 
@@ -3704,7 +3788,8 @@ class KnowledgeOrchestrator:
             window: Number of adjacent chunks to fetch on each side (default: 1)
 
         Returns:
-            Results with expanded content field
+            Results with expanded content field; ``match_offset`` marks where the
+            matched chunk starts inside it, so snippets can skip the context before it.
         """
         if not results:
             return results
@@ -3757,6 +3842,7 @@ class KnowledgeOrchestrator:
                 expanded = "\n\n".join(parts_before + [results[result_idx]["content"]] + parts_after)
                 results[result_idx]["content"] = expanded
                 results[result_idx]["context_expanded"] = True
+                results[result_idx]["match_offset"] = sum(len(part) + len("\n\n") for part in parts_before)
 
         return results
 
@@ -4064,14 +4150,36 @@ class KnowledgeOrchestrator:
         lines = [line.strip() for line in text.splitlines() if line.strip()]
         clean_text = f"# {title}\n\nSource: {url}\n\n" + "\n\n".join(lines)
 
-        safe_title = re.sub(r"[^\w\s-]", "", title).strip().replace(" ", "-").lower()[:60]
-        filename = f"{safe_title}.md"
-        filepath = f"{category}/{filename}"
+        safe_title = re.sub(r"[^\w\s-]", "", title).strip().replace(" ", "-").lower()[:60] or "page"
 
         # The body was fetched from an untrusted URL; the sanitizer inserts a
         # provenance fence and defuses model control tokens before the text
         # ever reaches the parser or the LLM that will consume the RAG output.
-        return self.add_document_from_content(clean_text, filepath, category, external_source=url)
+        # The lock spans name check and write, so parallel adds cannot both claim a free name.
+        with self._index_lock:
+            filepath = self._url_document_path(url, category, safe_title)
+            return self.add_document_from_content(clean_text, filepath, category, external_source=url)
+
+    @staticmethod
+    def _url_document_path(url: str, category: str, safe_title: str) -> str:
+        """Pick the first name that no other source owns for ``url``'s page (GH #262).
+
+        Titles are cut to 60 characters, so different pages can map to one name.
+        Tried in order: the title-based name, then that name plus the first 8 and
+        16 hex characters of the URL's SHA-256. The first that is free, or that
+        ``_is_saved_from`` shows already holds this URL's page, wins. The name with
+        the full digest is the last resort and is used without a check.
+        """
+        digest = content_sha256(url)
+        for stem in (safe_title, f"{safe_title}-{digest[:8]}", f"{safe_title}-{digest[:16]}"):
+            filepath = f"{category}/{stem}.md"
+            try:
+                existing = validate_path_within(config.documents_dir, filepath)
+            except PathEscapeError:
+                return filepath  # add_document_from_content rejects it and reports why
+            if not existing.is_file() or _is_saved_from(existing, url):
+                return filepath
+        return f"{category}/{safe_title}-{digest}.md"
 
     @collection_reader
     def search_similar(self, filepath: str, max_results: int = 5) -> List[Dict[str, Any]]:
@@ -4097,16 +4205,21 @@ class KnowledgeOrchestrator:
             if not results["ids"] or embeddings is None or len(embeddings) == 0:
                 return []
             query_embedding = embeddings[0]
-        except Exception:
+        except Exception as exc:
+            print(f"[WARN] search_similar query failed for {filepath!r}: {exc}", file=sys.stderr)
             return []
 
         try:
+            # Exclude the reference in the query: filtering afterwards let a long
+            # document's own chunks take every candidate slot.
             similar = self.collection.query(
                 query_embeddings=[query_embedding],
                 n_results=max_results + 20,
+                where={"doc_id": {"$ne": doc_id}},
                 include=["documents", "metadatas", "distances"],
             )
-        except Exception:
+        except Exception as exc:
+            print(f"[WARN] search_similar query failed for {filepath!r}: {exc}", file=sys.stderr)
             return []
 
         if not similar["ids"] or not similar["ids"][0]:
@@ -4118,8 +4231,6 @@ class KnowledgeOrchestrator:
             meta = (similar["metadatas"][0][i] or {}) if similar["metadatas"] else {}
             source = meta.get("source", "")
 
-            if meta.get("doc_id") == doc_id:
-                continue
             if source in seen_sources:
                 continue
             seen_sources.add(source)
@@ -4367,7 +4478,7 @@ class KnowledgeOrchestrator:
                 temporary.unlink(missing_ok=True)
                 raise
         try:
-            os.replace(temporary, path)
+            replace_with_retry(temporary, path)
         finally:
             temporary.unlink(missing_ok=True)
 
@@ -4497,7 +4608,7 @@ class KnowledgeOrchestrator:
                 temporary.unlink(missing_ok=True)
                 raise
         try:
-            os.replace(temporary, self._metadata_file)
+            replace_with_retry(temporary, self._metadata_file)
         finally:
             temporary.unlink(missing_ok=True)
 
@@ -4556,7 +4667,7 @@ class KnowledgeOrchestrator:
             json.dumps(payload, indent=2, ensure_ascii=False),
             encoding="utf-8",
         )
-        os.replace(tmp_path, self._checkpoint_file)
+        replace_with_retry(tmp_path, self._checkpoint_file)
         # Reflect in progress dict for status polling.
         self._reindex_progress["checkpoint_saved_at"] = payload["checkpoint_at"]
 
@@ -4688,7 +4799,7 @@ def _make_snippet(content: str, max_chars: int = 500) -> str:
 def search_knowledge(
     query: str,
     max_results: int = 5,
-    category: str = None,
+    category: Optional[str] = None,
     hybrid_alpha: float = 0.3,
     min_score: float = 0.0,
     snippet_mode: bool = True,
@@ -4712,9 +4823,10 @@ def search_knowledge(
         min_score: Minimum normalized relevance score (0.0–1.0) to include a result. Results scoring
             below this threshold are discarded. Default 0.0 returns all results. Use 0.2–0.4 to cut
             low-relevance noise.
-        snippet_mode: When true (default), truncates content to ~500 characters at a natural break
-            point and adds a content_length field with the original size. Use get_document() to
-            fetch full content when needed. Set to false to return full chunk content.
+        snippet_mode: When true (default), truncates content to ~500 characters from the start of the
+            matching chunk, at a natural break point, and adds a content_length field with the full
+            size. Use get_document() to fetch full content when needed. Set to false to return the full
+            content, which can include adjacent chunks; match_offset, when present, is where the match starts.
         search_method: Dispatch selector (v4.8.2+). One of ``"auto"`` (router picks FTS5 fast-path
             for lexical queries when enabled, hybrid otherwise), ``"hybrid"`` (force hybrid path —
             kill switch for suspected router misclassification), or ``"fts5"`` (force FTS5 fast-path
@@ -4789,7 +4901,11 @@ def search_knowledge(
 
     if snippet_mode:
         results = [
-            {**row, "content": _make_snippet(row["content"]), "content_length": len(row.get("content", ""))}
+            {
+                **row,
+                "content": _make_snippet(row["content"][row.get("match_offset", 0) :]),
+                "content_length": len(row.get("content", "")),
+            }
             for row in results
         ]
 
@@ -4820,15 +4936,15 @@ def get_document(filepath: str) -> str:
     Args:
         filepath: Relative path to the document within the documents directory
             (e.g., "security/technique.md"). Must be an indexed file — use
-            list_documents() to browse available paths, or search_knowledge()
-            to find the filepath by topic first.
+            search_knowledge() to find the filepath by topic first, or browse
+            list_documents(), which returns one page at a time (see its offset).
 
     Returns:
         JSON string with full document content and metadata (filepath, category, size).
 
     Usage: Use when you need the complete text of a known file — search_knowledge()
     returns chunks, not full docs. Use search_knowledge() first to find the filepath
-    if unknown. Use list_documents() to browse all available files by category.
+    if unknown. Use list_documents() to browse files by category, one page at a time.
     """
     orchestrator = get_orchestrator()
     doc = orchestrator.get_document(filepath)
@@ -4914,8 +5030,12 @@ def reindex_documents(
 ) -> str:
     """Index or reindex all documents in the knowledge base (runs in background).
 
-    ``force`` — smart reindex (detect changed files + rebuild BM25). Use after
-    filesystem edits outside add_document/update_document.
+    No flags — incremental pass: index new or changed files, skip unchanged
+    ones, prune deleted sources. Use after filesystem edits outside
+    add_document/update_document.
+    ``force`` — smart reindex: re-embed every file, unchanged ones included,
+    and rebuild BM25. Slow on a large corpus; use after changing the
+    query/passage prefixes or chunking settings.
     ``full_rebuild`` — nuclear rebuild (delete + re-embed). Use only after
     embedding-model change or index corruption. Mutually exclusive with resume.
     ``resume`` — pick up an interrupted smart reindex from
@@ -4983,12 +5103,15 @@ def list_categories() -> str:
     )
 
 
+_LIST_DOCUMENTS_MAX_LIMIT = 200  # ~75 KB of JSON: stays under common MCP client output limits
+
+
 @mcp.tool()
 @rate_limited
 @instrument("list_documents")
-def list_documents(category: str = None) -> str:
+def list_documents(category: Optional[str] = None, limit: int = 100, offset: int = 0) -> str:
     """
-    List all indexed documents, optionally filtered by category.
+    List indexed documents one page at a time, optionally filtered by category.
 
     Read-only. No side effects.
 
@@ -4996,19 +5119,35 @@ def list_documents(category: str = None) -> str:
         category: Optional category filter. Must be a valid category name — call
             list_categories() to see available options (e.g., security, ctf, logscale,
             development, general, redteam, blueteam).
+        limit: Maximum number of documents to return (default: 100, max: 200).
+        offset: Number of matching documents to skip (default: 0). For the next page, call
+            again with offset + count until offset reaches total.
 
     Returns:
-        JSON string with list of document filepaths, categories, and metadata for each indexed file.
+        JSON string with total (documents matching the filter), offset, limit, count (documents
+        in this page) and the filepaths, categories, and metadata of this page, ordered by filepath.
 
-    Usage: Use to browse what's in the index or verify a specific file is indexed. Use
-    list_categories() first to see valid category names. Use search_knowledge() when you
-    want to find documents by topic rather than browsing the full list. Use get_document()
-    to read a specific file once you have its filepath.
+    Usage: Use to browse what's in the index, one page at a time; to see every document, call
+    again with offset + count until offset reaches total. Use list_categories() first to see valid category names. Use
+    search_knowledge() when you want to find documents by topic rather than browsing the full
+    list. Use get_document() to read a specific file once you have its filepath.
     """
+    offset = max(0, offset)
+    limit = max(1, min(limit, _LIST_DOCUMENTS_MAX_LIMIT))
     orchestrator = get_orchestrator()
-    docs = orchestrator.list_documents(category=category)
+    # Source order keeps pages stable: re-indexing an edited file moves it to the end of the index.
+    docs = sorted(orchestrator.list_documents(category=category), key=lambda doc: doc["source"])
+    page = docs[offset : offset + limit]
     return json.dumps(
-        {"status": "success", "filter": category or "all", "count": len(docs), "documents": docs},
+        {
+            "status": "success",
+            "filter": category or "all",
+            "total": len(docs),
+            "offset": offset,
+            "limit": limit,
+            "count": len(page),
+            "documents": page,
+        },
         indent=2,
         ensure_ascii=False,
     )
@@ -5092,7 +5231,7 @@ def update_document(filepath: str, content: str) -> str:
 
     Args:
         filepath: Full or relative path to the document file. Must be an already-indexed
-            file — use list_documents() to find valid paths.
+            file — list_documents() lists indexed paths, one page at a time.
         content: New full-text content to replace the existing content entirely
 
     Returns:
@@ -5127,8 +5266,8 @@ def remove_document(filepath: str, delete_file: bool = False) -> str:
     the file from disk (irreversible, cannot be undone).
 
     Args:
-        filepath: Path to the document file. Must be an indexed document — use
-            list_documents() to find valid paths.
+        filepath: Path to the document file. Must be an indexed document —
+            list_documents() lists indexed paths, one page at a time.
         delete_file: If True, permanently deletes the file from disk in addition to
             removing from the index (default: False).
 
@@ -5155,7 +5294,7 @@ def remove_document(filepath: str, delete_file: bool = False) -> str:
 @mcp.tool()
 @rate_limited
 @instrument("add_from_url")
-def add_from_url(url: str, category: str = "general", title: str = None) -> str:
+def add_from_url(url: str, category: str = "general", title: Optional[str] = None) -> str:
     """
     Fetch content from a URL, convert to markdown, and add to the knowledge base.
 
@@ -5197,8 +5336,8 @@ def search_similar(filepath: str, max_results: int = 5) -> str:
     Read-only. No side effects. Uses the document's embedding for similarity comparison.
 
     Args:
-        filepath: Path to the reference document (must already be indexed — use
-            list_documents() to verify). E.g., "security/technique.md"
+        filepath: Path to the reference document (must already be indexed — list_documents()
+            lists indexed paths, one page at a time). E.g., "security/technique.md"
         max_results: Number of similar documents to return (default: 5, max: 20)
 
     Returns:
@@ -5206,8 +5345,8 @@ def search_similar(filepath: str, max_results: int = 5) -> str:
 
     Usage: Use when you have a specific document and want to discover thematically related
     ones. Use search_knowledge() instead when you have a text query rather than a reference
-    document. The reference document must be indexed — call list_documents() to confirm
-    it exists before calling this tool.
+    document. The reference document must be indexed; list_documents() lists indexed paths
+    one page at a time.
     """
     if not filepath:
         return json.dumps({"status": "error", "message": "Filepath required"})

{knowledge_rag-4.11.0 → knowledge_rag-5.0.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "knowledge-rag"
-version = "4.11.0"
+version = "5.0.0"
 description = "Local RAG System for Claude Code — Hybrid search + Cross-encoder Reranking + 13 MCP Tools + 20 Format Parsers. Zero external servers."
 readme = "README.md"
 license = {text = "MIT"}
@@ -141,7 +141,9 @@ source = ["mcp_server"]
 
 [tool.coverage.report]
 show_missing = true
-fail_under = 35
+# pytest-cov enforces this in every test matrix cell; each cell measured
+# 84.2-84.4% on 2026-11-05, so the floor catches a real drop, not noise.
+fail_under = 80
 
 # ── mypy: strict gradual rollout (Pillar 7) ─────────────────────────────────
 # Strict mode is enabled GLOBALLY but with a per-module exclusion list while