knowledge-rag 4.7.0__tar.gz → 4.8.0__tar.gz

{knowledge_rag-4.7.0 → knowledge_rag-4.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: knowledge-rag
-Version: 4.7.0
+Version: 4.8.0
 Summary: Local RAG System for Claude Code — Hybrid search + Cross-encoder Reranking + 13 MCP Tools + 20 Format Parsers. Zero external servers.
 Project-URL: Homepage, https://github.com/lyonzin/knowledge-rag
 Project-URL: Repository, https://github.com/lyonzin/knowledge-rag
@@ -1462,12 +1462,44 @@ Common issues:
 
 ### Unreleased
 
-### v4.7.0 (2026-07-31) — BM25 Fragment Query Support
+_(vazio — próximas mudanças vão aqui)_
 
-- **FIX (retrieval)**: BM25 tokenizer now emits **sub-tokens** for hyphenated alphanumeric-code composites, so fragment queries no longer silently return `NO_RESULTS`. Before v4.7.0, `MDR-AD002` was indexed as a single token `mdr-ad002`; queries like `AD002`, `MDR`, `CS005` returned zero even when the composite lived in the corpus. This is not a bug specific to MDR — it hit every hyphenated code taxonomy in typical infosec / doc corpora (`MDR-*`, `CVE-*`, `ADR-*`, `MS17-*`, `MDR-Custom001-xxxx`). The tokenizer now emits both the composite token AND its sub-parts of length ≥ 2 (`mdr-ad002` → `["mdr-ad002", "mdr", "ad002"]`). IDF preserves ranking — composite matches still rank above fragment matches because the composite is rarer. **Heuristic (perf-safety)**: sub-token expansion is triggered ONLY when the composite contains at least one digit — so `pass-the-hash`, `state-of-the-art`, and other natural-language hyphenated phrases stay as single tokens (expanding them would flood the inverted index with stop-word-like sub-parts and hurt concurrent-query throughput). (#140)
-- **NEW (tests)**: `tests/test_bm25_tokenizer_fragment.py` — 25 tests across 4 classes covering unit tokenizer behavior, end-to-end fragment matching, multi-doc disambiguation, and the 5 query patterns explicitly reported (`MDR-AD019`, `MDR-Custom001-xxxx`, `MDR`, `AD019`, `CS005`). Includes retrocompat sentinel + IDF ranking invariant. Baseline: 314 → 339.
+### v4.8.0 (2026-08-06) — Multilingual Foundation + Zero-Downtime Reindex
+
+v4.8.0 é a release de fundação para migração pra embeddings multilingual (`intfloat/multilingual-e5-large`) sem forçar downtime nem breaking change. Sete fases entregaram: security wire (7 vetores fechados que existiam desde v4.5.1), embedding profiles com prefix opt-in, GPU auto-detect com dependency chain documentada, batch/parallel indexing configurável + fix da assimetria BM25/semantic candidate pool, progress granular + resume checkpoint, dual-collection swap zero-downtime, e refactor pra manter complexidade Radon C+ na superfície tocada.
+
+Zero breaking change: usuários que atualizarem SEM tocar em `config.yaml` continuam com `bge-small-en-v1.5` byte-for-byte. Opt-in para multilingual: `profile: "multilingual"` + `reindex_documents(force=True)` — durante o reindex, RAG continua servindo da collection antiga (zero downtime via staging swap). Detalhes de upgrade em `docs/migration-v4.8.0.md`.
+
+- **REFACTOR (server + config)**: extracted 40+ helpers from 12 functions touched by v4.8.0 F1–F5 to enforce Core hard limits (function ≤30 lines). Biggest wins — `KnowledgeOrchestrator._index_all_impl` split from 246 → 23 lines (14 focused helpers covering setup / per-doc processing / progress metrics / persistence / teardown), `Config.__post_init__` split from 185 → 12 lines (13 domain validators), `reindex_documents` MCP tool from 100 → 30 lines (mode resolution + resume hydration + response formatting extracted to module helpers). Also refactored `_index_document` (91→19), `_rebuild_via_swap` (66→29), `_validate_staging` (56→24), `_cleanup_stale_staging_collections` (48→26), `get_reindex_status` (49→12), `_print_gpu_banner` (46→20), `start_reindex_background` (46→26), `_swap_collections_atomic` (43→25), `_route_load` (41→9). New sentinel `_SKIP_DOC` distinguishes "caller must skip" from "index as fresh" in the per-doc worker. Radon complexity average dropped from D (23.4) to C (17.2); zero D+ functions remain in F1–F5 touched surface. File-level extract (`server.py` = 4009 lines, `config.py` = 990 lines) deferred to v4.9.0 because `test_backwards_compat.py` validates the MCP contract by AST and moving classes across modules multiplies regression surface. Zero behavior change — full test sweep (`test_config` + `test_embedding_profile` + `test_gpu_auto` + `test_batch_parallel` + `test_reindex_resume` + `test_swap_zero_downtime` + `test_backwards_compat` + `test_bm25_*` + `test_search` + `test_dedup` + `test_ingestion*` + `test_lazy_embeddings` + `test_reranker_fallback` + `tests/security/` + all misc) = 557 passed / 3 skipped / 9 xfailed / 0 failed. Full audit report in `docs/refactor-audit-v4.8.0-fase-6.md`. v4.8.0 Fase 6.
+- **PERF (reindex)**: `nuclear_rebuild(swap=True)` is now the default, replacing the destructive-first rebuild that took the RAG offline for the entire duration (4min–40h depending on corpus + hardware). New workflow — cleanup stale stagings (24h TTL) → snapshot orchestrator state (`self.collection` / `self.bm25_index` / `self._indexed_docs` / `self._source_to_docid`) → create `{collection_name}__staging_{unix_ts}` (timestamped to avoid collisions with concurrent or previously-crashed rebuilds) → populate via temporary orchestrator rebind (`self.collection` points at staging, `self.bm25_index` is a throwaway `BM25Index()` so production BM25 keeps serving queries throughout) → validate three gates (`count >= baseline * 0.9` with 10% loss threshold for parser-skipped docs / 4 of 5 canonical queries `readme/function/import/return/class` return hits / `query()` does not raise, catching embedding dim mismatches) → atomic two-step swap via `Collection.modify(name=)` (prod → `__old_{ts}` frees the name, staging → prod assumes it, delete `__old_{ts}` non-fatally) with rollback-rename on mid-swap failure → post-swap BM25 rebuild from swapped-in ChromaDB contents + query cache invalidation. Any failure in populate/validate/swap triggers rollback via the snapshot and staging cleanup — production sees zero net change. Startup cleanup handles crash-orphaned stagings automatically (also runs before each rebuild). Storage doubles temporarily during rebuild (both collections on disk), back to 1x after swap. `nuclear_rebuild(swap=False)` preserves the legacy destructive behavior byte-for-byte in `_rebuild_destructive` for backwards compat / forced-cleanup edge cases. New `tests/test_swap_zero_downtime.py` (15 tests, ~1.7s, all mocked — no real ChromaDB or FastEmbed touched) covers naming format + populate BM25 isolation + all three validation gates + fresh-install zero-baseline path + post-swap `self.collection` reconnect + BM25 rebuild only after swap success + 100 parallel queries against a captured production handle during populate (zero errors, non-empty results) + stale/recent staging cleanup + non-staging collection preservation + backwards-compat dispatch. Docs in `docs/reindex-operations.md` section "Zero-downtime Rebuild (v4.8.0 Fase 5+)". v4.8.0 Fase 5.
+- **NEW (reindex)**: `reindex_documents(resume=True)` recovers from `data/reindex_checkpoint.json` after an interrupted smart reindex. Checkpoint is written every 500 docs or 30s (whichever first), and stores `indexed_doc_ids` + `chunks_processed` + `config_signature` (SHA256 of `embedding_model | embedding_dim | chunk_size | chunk_overlap`). A config mismatch between checkpoint write and resume load invalidates the checkpoint with a WARN — this prevents a mixed collection (partial old-model vectors + partial new-model vectors) which would silently degrade retrieval. The combination `resume=True + full_rebuild=True` is rejected with a structured error because nuclear rebuild wipes the collection first — resuming into a partial state would leave no coherent index. The `resume=True` flag also implicitly forces `mode='smart_reindex'` even when `force=False` was passed, so an interrupted smart run cannot silently fall back to `incremental` (which would ignore the checkpoint entirely). Existing 2-arg `reindex_documents(force, full_rebuild)` signature preserved — `resume` is an opt-in tail kwarg. Missing / corrupt / future-version / non-dict checkpoints all degrade gracefully to "start fresh" instead of raising. Written to `data/reindex_checkpoint.json` via atomic `.tmp` + `os.replace()` — Windows-safe on NTFS in-directory. Cleared automatically on successful reindex completion. Metadata (`index_metadata.json`) is flushed alongside every checkpoint write so `_indexed_docs` on disk stays in sync with the checkpoint's `indexed_doc_ids` list. New `tests/test_reindex_resume.py` (22 tests, all `object.__new__` isolation — no ChromaDB, no FastEmbed download) covers write cadence + atomic write + schema + load validation (6 failure modes) + resume kwarg happy path + rejection combos + status field flow. v4.8.0 Fase 4.
+- **NEW (reindex)**: `get_reindex_status()` now returns 6 new fields during active reindex — `chunks_processed` (chunks committed to ChromaDB so far), `chunks_total` (rolling estimate, 0 during warmup then running average from completed docs × `total_files`), `throughput_cps` (chunks/sec, sliding window bounded by the smaller of 100 samples or 30 seconds — deque `maxlen=100` combined with age-based pruning each iteration keeps the number stable during transient stalls without diluting current speed with ancient samples), `eta_seconds` (estimated seconds to completion, derived from throughput + remaining chunks, guarded against div-by-zero), `checkpoint_saved_at` (ISO timestamp of last checkpoint write, `null` until first checkpoint at 500 docs / 30s), `resumed` (bool — `true` when the current run was recovered from a checkpoint via `resume=True`, `false` otherwise). Enables real-time progress dashboards and the "should I keep waiting or restart" call. Idle path (`active: false`) unchanged — still returns just `last_result` or `last_error`. v4.8.0 Fase 4.
+- **DOCS (reindex)**: `docs/reindex-operations.md` NEW — documents the sync-vs-async gotcha for `reindex_documents(full_rebuild=True)` (daemon thread dies with process → RAG offline until manual restart), recommends `orchestrator.nuclear_rebuild()` sync for CLI scripts and `reindex_documents(force=True) + resume=True` for long unstable smart rebuilds. Also covers resume semantics (only valid with `full_rebuild=False`, implicitly forces `smart_reindex` mode, graceful degrade to fresh start on missing/corrupt/version/signature mismatch), checkpoint cadence (500 docs OR 30s, whichever first, plus rationale for the OR condition), `config_signature` invalidation semantic, full lifecycle (write on cadence, clear on success, clear on failed resume load), progress fields table with warmup behavior, example recovery flow showing crash → `ls checkpoint` → `resume=True` → poll status with `resumed:true` marker, and escalation rules of thumb for the three most likely operator questions. v4.8.0 Fase 4.
+- **PERF (indexing)**: `documents.batch_size` (default 500) and `documents.parallel_workers` (default 1, opt-in) are now configurable in `config.yaml`. The previously hardcoded `_CHROMA_BATCH_SIZE = 500` in `server.py:_index_document` continues as fallback constant for tests that instantiate the orchestrator without a full `Config` (via `object.__new__`, as `tests/test_search.py` does). When `parallel_workers > 1` AND the document produces more than one batch, the per-batch `collection.add(...)` calls run inside a `ThreadPoolExecutor(max_workers=parallel_workers)`. Single-batch documents skip the pool entirely — no orchestration overhead to extract from a single call. Gain comes from SQLite writes overlapping with the NEXT batch's ONNX inference (embedding itself is serialized by ONNX session lock — never parallel inference). Windows users should monitor stability at `workers > 4` (extra `[WARN]` on startup). `batch_size` clamped to `[1, 5000]`, `parallel_workers` clamped to `[1, 16]` — out-of-range values are clamped with a `[WARN]`. New `bench/test_bench_reindex.py` (3 benchmarks) uses a `_SleepingCollection` stub with 5ms per-`add()` sleep so the parallel path measurably overlaps without depending on a real ChromaDB; local numbers: reindex_batch_2000 5.66ms, reindex_default 5.68ms, reindex_parallel_4 7.30ms (200-chunk doc, 4 concurrent batches, sequential would be ~20ms). New `tests/test_batch_parallel.py` (20 tests, ~1.6s) covers config parsing, clamping, Windows-only warn on `workers > 4`, sequential vs parallel dispatch, single-batch skip, `getattr` fallback to `_CHROMA_BATCH_SIZE`, exception propagation via `.result()`, and empty-doc short-circuit. Defaults preserve v4.7.1 behavior byte-for-byte. v4.8.0 Fase 3.
+- **FIX (search)**: `search.max_results` default raised from `20` to `100`. This eliminates the asymmetric candidate pool in hybrid retrieval where `_do_semantic` clamped candidates to `min(max_results * 3, config.max_results) = min(15, 20) = 20` while BM25 pulled up to `max_results * 20 = 400` — semantic silently starved without any log signal, degrading hybrid retrieval quality on any query where the semantic branch could have contributed better candidates. Callers passing `max_results` explicitly in the MCP tool call (`search_knowledge`, `search_similar`) are unaffected — only the config-loaded default changed. New `TestDoSemanticCandidateMath` class in `tests/test_search.py` (2 tests) pins the fix against future regressions: `test_semantic_asks_chromadb_for_3x_max_results` proves the 3× semantic candidate pool, `test_semantic_pool_is_bounded_by_config_max_results` proves the cap still works when `max_results * 3` exceeds it. v4.8.0 Fase 3.
+- **CHANGE (gpu)**: `models.embedding.gpu` is now tri-state (`"auto"` | `"true"` | `"false"`) with `"auto"` as the new default. In auto mode, `FastEmbedEmbeddings.verify_gpu_readiness()` runs four startup checks (`CUDAExecutionProvider` present in `onnxruntime.get_available_providers()` / required NVIDIA DLLs reachable via `PATH` — `cudart64_12.dll`, `cudnn64_9.dll`, `cublasLt64_12.dll` on Windows and their `.so.12` / `.so.9` counterparts on Linux / `nvidia-smi --query-gpu=name,memory.total` exits 0 / a minimal single-op ONNX `InferenceSession` initializes on the CUDA provider) and uses CUDA when all pass, CPU otherwise. Legacy bool `gpu: true`/`gpu: false` continues to work — normalized to the string form internally. The startup banner now prints on every path (including CPU-only and fallback), reports the mode (`auto-cuda` / `auto-cpu-fallback` / `forced-cpu` / `forced-cuda` / `forced-cuda-fallback`), and surfaces the probe `fallback_reason` on CPU paths with a hint pointing at the full pip install chain. Full dependency chain (onnxruntime-gpu CUDA 12 variant via `--extra-index-url` + `nvidia-cudnn-cu12` + `nvidia-cublas-cu12` + siblings) and troubleshooting for the six most common failures documented in `docs/gpu-setup.md`. Zero behavior change for users who had `gpu: false` explicitly. New coverage in `tests/test_gpu_auto.py` (13 tests, all mocked — no real GPU required to run). v4.8.0 Fase 2. (#149)
+- **NEW (embedding)**: `models.embedding.profile` opt-in preset (`compact` | `quality` | `multilingual` | `custom`) plus `query_prefix` / `passage_prefix` fields. The `multilingual` profile ships `intfloat/multilingual-e5-large` (1024D, 100+ languages) with the required `"query: "` / `"passage: "` prefixes wired into `FastEmbedEmbeddings.__call__`, `.embed_query`, and `.embed_documents`. `compact` preserves the current `bge-small-en-v1.5` default byte-for-byte. Changing profile or prefix requires `reindex_documents(force=True)` because existing chunks were embedded without the new prefix. (#148)
+- **FIX (bm25)**: BM25 tokenizer is now Unicode-aware. Before this fix, the `[a-z0-9]` character class in `BM25Index._tokenize()` and `_metadata_path_score()` (server.py:652, 659, 720) dropped every code point outside Basic Latin — Cyrillic, Greek, CJK, Arabic corpora silently degraded to semantic-only search (empty `bm25_rank`, `search_method: semantic`, no warning). Portuguese/Spanish/French text with diacritics was **fragmented** on the accent: `"informação técnica"` tokenised as `['informa', 'o', 't', 'cnica']` and BM25 scored garbage fragments instead of the actual words. The regex now uses `[^\W_]+(?:-[^\W_]+)*` which keeps hyphenated composites (`CVE-2024-1234`, `rule-a002`, `pass-the-hash`) byte-identical to prior behaviour while also matching letters from any script. All 28 tests in `test_bm25_tokenizer_fragment.py` remain green. New coverage in `tests/test_bm25_unicode_tokenizer.py` (13 tests) exercises Cyrillic / Greek / Arabic / CJK / PT-BR / mixed-script / composite-with-diacritics / underscore-separation. Closes #145. (#147)
+- **DOCS (security)**: rewrite `SECURITY.md` reporting section — remove the `lyonzin@users.noreply.github.com` fallback (GitHub `noreply.github.com` addresses are outbound-only and never receive mail, so that channel was decorative) and direct all reports to the GitHub Security Advisory form at `/security/advisories/new`. Private vulnerability reporting is now enabled on the repo, so the form is available to any GitHub user without collaborator status. Closes #146. (#147)
+- **FIX (security)**: wire `validate_path_within` on all filepath-taking orchestrator tools — `get_document`, `add_document_from_content`, `update_document_content`, `remove_document_by_path`, `search_similar`. Escape attempts return a structured `{"error": "Filepath rejected: ..."}` and touch nothing on disk. Closes 5 latent path traversal / arbitrary-file-{read,write,delete} vectors that the `mcp_server/security.py` primitives had covered since v4.5.1 but which never had callers in the mutation path. (#147)
+- **FIX (security)**: wire `sanitize_external_content` on `add_from_url`. The fetched body now flows through a new `external_source` kwarg on `add_document_from_content` (default `None` — operator ingests stay byte-identical) which fences it in a provenance envelope and defuses known prompt-injection sentinels (`<|im_start|>`, `[INST]`, etc.) before the parser or any downstream LLM sees it. Closes OWASP LLM01:2025 on the URL ingest path. (#147)
+- **FIX (security)**: install `BearerAuthMiddleware` on HTTP transports (`sse`, `streamable-http`) when `config.auth_bearer_token` is set. Wraps the FastMCP ASGI app and serves via uvicorn; unset token keeps prior `mcp.run()` open-port behaviour with a `[WARN]` on stderr; `stdio` transport is never wrapped. Closes the auth-bypass gap where `auth_bearer_token` had existed as a config field since v4.0.0 but was never compared. (#147)
+- **NEW (tests)**: `tests/security/test_add_from_url_sanitized.py` and `tests/security/test_bearer_middleware_attached.py` — integration tests that prove the wire (not just the library primitives). The `@pytest.mark.xfail("integration pending v4.6.0")` markers on 9 tests across `test_path_traversal.py`, `test_bearer_auth.py`, `test_prompt_injection.py` are removed — they now pass. `tests/security/` totals: 138 passed, 7 skipped, 6 xfailed. (#147)
+- **DOCS (perf)**: `docs/perf-baseline-v4.7.1.md` + `perf-baseline-v4.7.1.json` — reference microbenchmarks + `nuclear_rebuild()` timing (100-doc synthetic corpus) captured at the tip of v4.7.1. Non-runtime baseline for the planned Phase 3 batch-parallel embedding work in v4.8.0.
+- **NEW (release automation)**: `release.yml` now has a `yank-superseded` job that reads a `YANKS:` marker from the release body and automatically yanks the listed version from PyPI + deprecates it on NPM after the new version publishes. Format: `YANKS: v4.7.0` or `YANKS: v4.7.0 — some reason`. Requires the `PYPI_API_TOKEN` repo secret (classic scoped token — Trusted Publisher OIDC does not currently support yank). Job runs after `publish-pypi`/`publish-npm`/`publish-docker`, is idempotent, and degrades gracefully when secrets are missing (emits `::warning::` instead of failing). If the marker is absent, the job is a no-op.
+
+### v4.7.1 (2026-08-01) — Documentation Hygiene
+
+- **CHORE (docs, tests)**: sanitize identifiers in test fixtures and CHANGELOG examples. v4.7.0 referenced code names from a real deployment corpus in `tests/test_bm25_tokenizer_fragment.py` and the v4.7.0 CHANGELOG entry; those are replaced with synthetic placeholders (`RULE-A00X`, `PROJECT-Custom001-xxxx`) and well-known public identifiers (`CVE-*`, `MS17-*`, `ADR-*`, `T1078.002`) that carry no organization-specific taxonomy. Zero behavior change — all 28 fragment tests pass unchanged, tokenizer logic identical.
+- **NOTE**: v4.7.0 is **yanked from PyPI** and **deprecated on NPM** because its published README contained the same corporate example strings. The Docker tag `v4.7.0` remains (image tags are immutable) but `latest` now points to v4.7.1. Please upgrade — v4.7.0 and v4.7.1 are functionally equivalent.
+- **VERSION**: 4.7.0 → **4.7.1** (PATCH — documentation hygiene, zero code / API / config change).
+
+### v4.7.0 (2026-07-31) — BM25 Fragment Query Support ⚠️ **YANKED — use v4.7.1**
+
+- **FIX (retrieval)**: BM25 tokenizer now emits **sub-tokens** for hyphenated alphanumeric-code composites, so fragment queries no longer silently return `NO_RESULTS`. Before v4.7.0, `RULE-A002` was indexed as a single token `rule-a002`; queries like `A002`, `RULE`, `B005` returned zero even when the composite lived in the corpus. This is not a bug specific to any naming scheme — it hit every hyphenated code taxonomy in typical infosec / doc corpora (`RULE-*`, `CVE-*`, `ADR-*`, `MS17-*`, `PROJECT-Custom001-xxxx`). The tokenizer now emits both the composite token AND its sub-parts of length ≥ 2 (`rule-a002` → `["rule-a002", "rule", "a002"]`). IDF preserves ranking — composite matches still rank above fragment matches because the composite is rarer. **Heuristic (perf-safety)**: sub-token expansion is triggered ONLY when the composite contains at least one digit — so `pass-the-hash`, `state-of-the-art`, and other natural-language hyphenated phrases stay as single tokens (expanding them would flood the inverted index with stop-word-like sub-parts and hurt concurrent-query throughput). (#140)
+- **NEW (tests)**: `tests/test_bm25_tokenizer_fragment.py` — 28 tests across 4 classes covering unit tokenizer behavior, end-to-end fragment matching, multi-doc disambiguation, and the 5 query patterns explicitly reported (`RULE-A019`, `PROJECT-Custom001-xxxx`, `RULE`, `A019`, `B005`). Includes retrocompat sentinel + IDF ranking invariant. Baseline: 314 → 342.
 - **UPGRADE NOTE**: run `reindex_documents(force=True)` once after upgrade to rebuild the BM25 inverted index with the new sub-token emission. ChromaDB vectors are untouched. Expect a **~30–50% increase in BM25 index memory** (measured +32.5% on synthetic corpus) — negligible for typical corpora, worth reviewing for deployments with >100K docs.
-- **KNOWN LIMITATION**: alphanumeric-glued tokens without separators (`HARDLAB11`, `6a635623e114b8...`) still require exact match — no sub-token expansion possible without character n-grams (5-10× index overhead). Tracked as follow-up.
+- **KNOWN LIMITATION**: alphanumeric-glued tokens without separators (single-token identifiers with no hyphens, hash prefixes) still require exact match — no sub-token expansion possible without character n-grams (5-10× index overhead). Tracked as follow-up.
 - **VERSION**: 4.6.0 → **4.7.0** (MINOR — tokenization behavior change + reindex requirement, though no API/config break).
 
 ### v4.6.0 (2026-07-30) — MCP spec 2026-07-28 via Anthropic Tier 1 SDK (mcp 2.x)

{knowledge_rag-4.7.0 → knowledge_rag-4.8.0}/README.md

@@ -1414,12 +1414,44 @@ Common issues:
 
 ### Unreleased
 
-### v4.7.0 (2026-07-31) — BM25 Fragment Query Support
-
-- **FIX (retrieval)**: BM25 tokenizer now emits **sub-tokens** for hyphenated alphanumeric-code composites, so fragment queries no longer silently return `NO_RESULTS`. Before v4.7.0, `MDR-AD002` was indexed as a single token `mdr-ad002`; queries like `AD002`, `MDR`, `CS005` returned zero even when the composite lived in the corpus. This is not a bug specific to MDR — it hit every hyphenated code taxonomy in typical infosec / doc corpora (`MDR-*`, `CVE-*`, `ADR-*`, `MS17-*`, `MDR-Custom001-xxxx`). The tokenizer now emits both the composite token AND its sub-parts of length ≥ 2 (`mdr-ad002` → `["mdr-ad002", "mdr", "ad002"]`). IDF preserves ranking — composite matches still rank above fragment matches because the composite is rarer. **Heuristic (perf-safety)**: sub-token expansion is triggered ONLY when the composite contains at least one digit — so `pass-the-hash`, `state-of-the-art`, and other natural-language hyphenated phrases stay as single tokens (expanding them would flood the inverted index with stop-word-like sub-parts and hurt concurrent-query throughput). (#140)
-- **NEW (tests)**: `tests/test_bm25_tokenizer_fragment.py` — 25 tests across 4 classes covering unit tokenizer behavior, end-to-end fragment matching, multi-doc disambiguation, and the 5 query patterns explicitly reported (`MDR-AD019`, `MDR-Custom001-xxxx`, `MDR`, `AD019`, `CS005`). Includes retrocompat sentinel + IDF ranking invariant. Baseline: 314 → 339.
+_(vazio — próximas mudanças vão aqui)_
+
+### v4.8.0 (2026-08-06) — Multilingual Foundation + Zero-Downtime Reindex
+
+v4.8.0 é a release de fundação para migração pra embeddings multilingual (`intfloat/multilingual-e5-large`) sem forçar downtime nem breaking change. Sete fases entregaram: security wire (7 vetores fechados que existiam desde v4.5.1), embedding profiles com prefix opt-in, GPU auto-detect com dependency chain documentada, batch/parallel indexing configurável + fix da assimetria BM25/semantic candidate pool, progress granular + resume checkpoint, dual-collection swap zero-downtime, e refactor pra manter complexidade Radon C+ na superfície tocada.
+
+Zero breaking change: usuários que atualizarem SEM tocar em `config.yaml` continuam com `bge-small-en-v1.5` byte-for-byte. Opt-in para multilingual: `profile: "multilingual"` + `reindex_documents(force=True)` — durante o reindex, RAG continua servindo da collection antiga (zero downtime via staging swap). Detalhes de upgrade em `docs/migration-v4.8.0.md`.
+
+- **REFACTOR (server + config)**: extracted 40+ helpers from 12 functions touched by v4.8.0 F1–F5 to enforce Core hard limits (function ≤30 lines). Biggest wins — `KnowledgeOrchestrator._index_all_impl` split from 246 → 23 lines (14 focused helpers covering setup / per-doc processing / progress metrics / persistence / teardown), `Config.__post_init__` split from 185 → 12 lines (13 domain validators), `reindex_documents` MCP tool from 100 → 30 lines (mode resolution + resume hydration + response formatting extracted to module helpers). Also refactored `_index_document` (91→19), `_rebuild_via_swap` (66→29), `_validate_staging` (56→24), `_cleanup_stale_staging_collections` (48→26), `get_reindex_status` (49→12), `_print_gpu_banner` (46→20), `start_reindex_background` (46→26), `_swap_collections_atomic` (43→25), `_route_load` (41→9). New sentinel `_SKIP_DOC` distinguishes "caller must skip" from "index as fresh" in the per-doc worker. Radon complexity average dropped from D (23.4) to C (17.2); zero D+ functions remain in F1–F5 touched surface. File-level extract (`server.py` = 4009 lines, `config.py` = 990 lines) deferred to v4.9.0 because `test_backwards_compat.py` validates the MCP contract by AST and moving classes across modules multiplies regression surface. Zero behavior change — full test sweep (`test_config` + `test_embedding_profile` + `test_gpu_auto` + `test_batch_parallel` + `test_reindex_resume` + `test_swap_zero_downtime` + `test_backwards_compat` + `test_bm25_*` + `test_search` + `test_dedup` + `test_ingestion*` + `test_lazy_embeddings` + `test_reranker_fallback` + `tests/security/` + all misc) = 557 passed / 3 skipped / 9 xfailed / 0 failed. Full audit report in `docs/refactor-audit-v4.8.0-fase-6.md`. v4.8.0 Fase 6.
+- **PERF (reindex)**: `nuclear_rebuild(swap=True)` is now the default, replacing the destructive-first rebuild that took the RAG offline for the entire duration (4min–40h depending on corpus + hardware). New workflow — cleanup stale stagings (24h TTL) → snapshot orchestrator state (`self.collection` / `self.bm25_index` / `self._indexed_docs` / `self._source_to_docid`) → create `{collection_name}__staging_{unix_ts}` (timestamped to avoid collisions with concurrent or previously-crashed rebuilds) → populate via temporary orchestrator rebind (`self.collection` points at staging, `self.bm25_index` is a throwaway `BM25Index()` so production BM25 keeps serving queries throughout) → validate three gates (`count >= baseline * 0.9` with 10% loss threshold for parser-skipped docs / 4 of 5 canonical queries `readme/function/import/return/class` return hits / `query()` does not raise, catching embedding dim mismatches) → atomic two-step swap via `Collection.modify(name=)` (prod → `__old_{ts}` frees the name, staging → prod assumes it, delete `__old_{ts}` non-fatally) with rollback-rename on mid-swap failure → post-swap BM25 rebuild from swapped-in ChromaDB contents + query cache invalidation. Any failure in populate/validate/swap triggers rollback via the snapshot and staging cleanup — production sees zero net change. Startup cleanup handles crash-orphaned stagings automatically (also runs before each rebuild). Storage doubles temporarily during rebuild (both collections on disk), back to 1x after swap. `nuclear_rebuild(swap=False)` preserves the legacy destructive behavior byte-for-byte in `_rebuild_destructive` for backwards compat / forced-cleanup edge cases. New `tests/test_swap_zero_downtime.py` (15 tests, ~1.7s, all mocked — no real ChromaDB or FastEmbed touched) covers naming format + populate BM25 isolation + all three validation gates + fresh-install zero-baseline path + post-swap `self.collection` reconnect + BM25 rebuild only after swap success + 100 parallel queries against a captured production handle during populate (zero errors, non-empty results) + stale/recent staging cleanup + non-staging collection preservation + backwards-compat dispatch. Docs in `docs/reindex-operations.md` section "Zero-downtime Rebuild (v4.8.0 Fase 5+)". v4.8.0 Fase 5.
+- **NEW (reindex)**: `reindex_documents(resume=True)` recovers from `data/reindex_checkpoint.json` after an interrupted smart reindex. Checkpoint is written every 500 docs or 30s (whichever first), and stores `indexed_doc_ids` + `chunks_processed` + `config_signature` (SHA256 of `embedding_model | embedding_dim | chunk_size | chunk_overlap`). A config mismatch between checkpoint write and resume load invalidates the checkpoint with a WARN — this prevents a mixed collection (partial old-model vectors + partial new-model vectors) which would silently degrade retrieval. The combination `resume=True + full_rebuild=True` is rejected with a structured error because nuclear rebuild wipes the collection first — resuming into a partial state would leave no coherent index. The `resume=True` flag also implicitly forces `mode='smart_reindex'` even when `force=False` was passed, so an interrupted smart run cannot silently fall back to `incremental` (which would ignore the checkpoint entirely). Existing 2-arg `reindex_documents(force, full_rebuild)` signature preserved — `resume` is an opt-in tail kwarg. Missing / corrupt / future-version / non-dict checkpoints all degrade gracefully to "start fresh" instead of raising. Written to `data/reindex_checkpoint.json` via atomic `.tmp` + `os.replace()` — Windows-safe on NTFS in-directory. Cleared automatically on successful reindex completion. Metadata (`index_metadata.json`) is flushed alongside every checkpoint write so `_indexed_docs` on disk stays in sync with the checkpoint's `indexed_doc_ids` list. New `tests/test_reindex_resume.py` (22 tests, all `object.__new__` isolation — no ChromaDB, no FastEmbed download) covers write cadence + atomic write + schema + load validation (6 failure modes) + resume kwarg happy path + rejection combos + status field flow. v4.8.0 Fase 4.
+- **NEW (reindex)**: `get_reindex_status()` now returns 6 new fields during active reindex — `chunks_processed` (chunks committed to ChromaDB so far), `chunks_total` (rolling estimate, 0 during warmup then running average from completed docs × `total_files`), `throughput_cps` (chunks/sec, sliding window bounded by the smaller of 100 samples or 30 seconds — deque `maxlen=100` combined with age-based pruning each iteration keeps the number stable during transient stalls without diluting current speed with ancient samples), `eta_seconds` (estimated seconds to completion, derived from throughput + remaining chunks, guarded against div-by-zero), `checkpoint_saved_at` (ISO timestamp of last checkpoint write, `null` until first checkpoint at 500 docs / 30s), `resumed` (bool — `true` when the current run was recovered from a checkpoint via `resume=True`, `false` otherwise). Enables real-time progress dashboards and the "should I keep waiting or restart" call. Idle path (`active: false`) unchanged — still returns just `last_result` or `last_error`. v4.8.0 Fase 4.
+- **DOCS (reindex)**: `docs/reindex-operations.md` NEW — documents the sync-vs-async gotcha for `reindex_documents(full_rebuild=True)` (daemon thread dies with process → RAG offline until manual restart), recommends `orchestrator.nuclear_rebuild()` sync for CLI scripts and `reindex_documents(force=True) + resume=True` for long unstable smart rebuilds. Also covers resume semantics (only valid with `full_rebuild=False`, implicitly forces `smart_reindex` mode, graceful degrade to fresh start on missing/corrupt/version/signature mismatch), checkpoint cadence (500 docs OR 30s, whichever first, plus rationale for the OR condition), `config_signature` invalidation semantic, full lifecycle (write on cadence, clear on success, clear on failed resume load), progress fields table with warmup behavior, example recovery flow showing crash → `ls checkpoint` → `resume=True` → poll status with `resumed:true` marker, and escalation rules of thumb for the three most likely operator questions. v4.8.0 Fase 4.
+- **PERF (indexing)**: `documents.batch_size` (default 500) and `documents.parallel_workers` (default 1, opt-in) are now configurable in `config.yaml`. The previously hardcoded `_CHROMA_BATCH_SIZE = 500` in `server.py:_index_document` continues as fallback constant for tests that instantiate the orchestrator without a full `Config` (via `object.__new__`, as `tests/test_search.py` does). When `parallel_workers > 1` AND the document produces more than one batch, the per-batch `collection.add(...)` calls run inside a `ThreadPoolExecutor(max_workers=parallel_workers)`. Single-batch documents skip the pool entirely — no orchestration overhead to extract from a single call. Gain comes from SQLite writes overlapping with the NEXT batch's ONNX inference (embedding itself is serialized by ONNX session lock — never parallel inference). Windows users should monitor stability at `workers > 4` (extra `[WARN]` on startup). `batch_size` clamped to `[1, 5000]`, `parallel_workers` clamped to `[1, 16]` — out-of-range values are clamped with a `[WARN]`. New `bench/test_bench_reindex.py` (3 benchmarks) uses a `_SleepingCollection` stub with 5ms per-`add()` sleep so the parallel path measurably overlaps without depending on a real ChromaDB; local numbers: reindex_batch_2000 5.66ms, reindex_default 5.68ms, reindex_parallel_4 7.30ms (200-chunk doc, 4 concurrent batches, sequential would be ~20ms). New `tests/test_batch_parallel.py` (20 tests, ~1.6s) covers config parsing, clamping, Windows-only warn on `workers > 4`, sequential vs parallel dispatch, single-batch skip, `getattr` fallback to `_CHROMA_BATCH_SIZE`, exception propagation via `.result()`, and empty-doc short-circuit. Defaults preserve v4.7.1 behavior byte-for-byte. v4.8.0 Fase 3.
+- **FIX (search)**: `search.max_results` default raised from `20` to `100`. This eliminates the asymmetric candidate pool in hybrid retrieval where `_do_semantic` clamped candidates to `min(max_results * 3, config.max_results) = min(15, 20) = 20` while BM25 pulled up to `max_results * 20 = 400` — semantic silently starved without any log signal, degrading hybrid retrieval quality on any query where the semantic branch could have contributed better candidates. Callers passing `max_results` explicitly in the MCP tool call (`search_knowledge`, `search_similar`) are unaffected — only the config-loaded default changed. New `TestDoSemanticCandidateMath` class in `tests/test_search.py` (2 tests) pins the fix against future regressions: `test_semantic_asks_chromadb_for_3x_max_results` proves the 3× semantic candidate pool, `test_semantic_pool_is_bounded_by_config_max_results` proves the cap still works when `max_results * 3` exceeds it. v4.8.0 Fase 3.
+- **CHANGE (gpu)**: `models.embedding.gpu` is now tri-state (`"auto"` | `"true"` | `"false"`) with `"auto"` as the new default. In auto mode, `FastEmbedEmbeddings.verify_gpu_readiness()` runs four startup checks (`CUDAExecutionProvider` present in `onnxruntime.get_available_providers()` / required NVIDIA DLLs reachable via `PATH` — `cudart64_12.dll`, `cudnn64_9.dll`, `cublasLt64_12.dll` on Windows and their `.so.12` / `.so.9` counterparts on Linux / `nvidia-smi --query-gpu=name,memory.total` exits 0 / a minimal single-op ONNX `InferenceSession` initializes on the CUDA provider) and uses CUDA when all pass, CPU otherwise. Legacy bool `gpu: true`/`gpu: false` continues to work — normalized to the string form internally. The startup banner now prints on every path (including CPU-only and fallback), reports the mode (`auto-cuda` / `auto-cpu-fallback` / `forced-cpu` / `forced-cuda` / `forced-cuda-fallback`), and surfaces the probe `fallback_reason` on CPU paths with a hint pointing at the full pip install chain. Full dependency chain (onnxruntime-gpu CUDA 12 variant via `--extra-index-url` + `nvidia-cudnn-cu12` + `nvidia-cublas-cu12` + siblings) and troubleshooting for the six most common failures documented in `docs/gpu-setup.md`. Zero behavior change for users who had `gpu: false` explicitly. New coverage in `tests/test_gpu_auto.py` (13 tests, all mocked — no real GPU required to run). v4.8.0 Fase 2. (#149)
+- **NEW (embedding)**: `models.embedding.profile` opt-in preset (`compact` | `quality` | `multilingual` | `custom`) plus `query_prefix` / `passage_prefix` fields. The `multilingual` profile ships `intfloat/multilingual-e5-large` (1024D, 100+ languages) with the required `"query: "` / `"passage: "` prefixes wired into `FastEmbedEmbeddings.__call__`, `.embed_query`, and `.embed_documents`. `compact` preserves the current `bge-small-en-v1.5` default byte-for-byte. Changing profile or prefix requires `reindex_documents(force=True)` because existing chunks were embedded without the new prefix. (#148)
+- **FIX (bm25)**: BM25 tokenizer is now Unicode-aware. Before this fix, the `[a-z0-9]` character class in `BM25Index._tokenize()` and `_metadata_path_score()` (server.py:652, 659, 720) dropped every code point outside Basic Latin — Cyrillic, Greek, CJK, Arabic corpora silently degraded to semantic-only search (empty `bm25_rank`, `search_method: semantic`, no warning). Portuguese/Spanish/French text with diacritics was **fragmented** on the accent: `"informação técnica"` tokenised as `['informa', 'o', 't', 'cnica']` and BM25 scored garbage fragments instead of the actual words. The regex now uses `[^\W_]+(?:-[^\W_]+)*` which keeps hyphenated composites (`CVE-2024-1234`, `rule-a002`, `pass-the-hash`) byte-identical to prior behaviour while also matching letters from any script. All 28 tests in `test_bm25_tokenizer_fragment.py` remain green. New coverage in `tests/test_bm25_unicode_tokenizer.py` (13 tests) exercises Cyrillic / Greek / Arabic / CJK / PT-BR / mixed-script / composite-with-diacritics / underscore-separation. Closes #145. (#147)
+- **DOCS (security)**: rewrite `SECURITY.md` reporting section — remove the `lyonzin@users.noreply.github.com` fallback (GitHub `noreply.github.com` addresses are outbound-only and never receive mail, so that channel was decorative) and direct all reports to the GitHub Security Advisory form at `/security/advisories/new`. Private vulnerability reporting is now enabled on the repo, so the form is available to any GitHub user without collaborator status. Closes #146. (#147)
+- **FIX (security)**: wire `validate_path_within` on all filepath-taking orchestrator tools — `get_document`, `add_document_from_content`, `update_document_content`, `remove_document_by_path`, `search_similar`. Escape attempts return a structured `{"error": "Filepath rejected: ..."}` and touch nothing on disk. Closes 5 latent path traversal / arbitrary-file-{read,write,delete} vectors that the `mcp_server/security.py` primitives had covered since v4.5.1 but which never had callers in the mutation path. (#147)
+- **FIX (security)**: wire `sanitize_external_content` on `add_from_url`. The fetched body now flows through a new `external_source` kwarg on `add_document_from_content` (default `None` — operator ingests stay byte-identical) which fences it in a provenance envelope and defuses known prompt-injection sentinels (`<|im_start|>`, `[INST]`, etc.) before the parser or any downstream LLM sees it. Closes OWASP LLM01:2025 on the URL ingest path. (#147)
+- **FIX (security)**: install `BearerAuthMiddleware` on HTTP transports (`sse`, `streamable-http`) when `config.auth_bearer_token` is set. Wraps the FastMCP ASGI app and serves via uvicorn; unset token keeps prior `mcp.run()` open-port behaviour with a `[WARN]` on stderr; `stdio` transport is never wrapped. Closes the auth-bypass gap where `auth_bearer_token` had existed as a config field since v4.0.0 but was never compared. (#147)
+- **NEW (tests)**: `tests/security/test_add_from_url_sanitized.py` and `tests/security/test_bearer_middleware_attached.py` — integration tests that prove the wire (not just the library primitives). The `@pytest.mark.xfail("integration pending v4.6.0")` markers on 9 tests across `test_path_traversal.py`, `test_bearer_auth.py`, `test_prompt_injection.py` are removed — they now pass. `tests/security/` totals: 138 passed, 7 skipped, 6 xfailed. (#147)
+- **DOCS (perf)**: `docs/perf-baseline-v4.7.1.md` + `perf-baseline-v4.7.1.json` — reference microbenchmarks + `nuclear_rebuild()` timing (100-doc synthetic corpus) captured at the tip of v4.7.1. Non-runtime baseline for the planned Phase 3 batch-parallel embedding work in v4.8.0.
+- **NEW (release automation)**: `release.yml` now has a `yank-superseded` job that reads a `YANKS:` marker from the release body and automatically yanks the listed version from PyPI + deprecates it on NPM after the new version publishes. Format: `YANKS: v4.7.0` or `YANKS: v4.7.0 — some reason`. Requires the `PYPI_API_TOKEN` repo secret (classic scoped token — Trusted Publisher OIDC does not currently support yank). Job runs after `publish-pypi`/`publish-npm`/`publish-docker`, is idempotent, and degrades gracefully when secrets are missing (emits `::warning::` instead of failing). If the marker is absent, the job is a no-op.
+
+### v4.7.1 (2026-08-01) — Documentation Hygiene
+
+- **CHORE (docs, tests)**: sanitize identifiers in test fixtures and CHANGELOG examples. v4.7.0 referenced code names from a real deployment corpus in `tests/test_bm25_tokenizer_fragment.py` and the v4.7.0 CHANGELOG entry; those are replaced with synthetic placeholders (`RULE-A00X`, `PROJECT-Custom001-xxxx`) and well-known public identifiers (`CVE-*`, `MS17-*`, `ADR-*`, `T1078.002`) that carry no organization-specific taxonomy. Zero behavior change — all 28 fragment tests pass unchanged, tokenizer logic identical.
+- **NOTE**: v4.7.0 is **yanked from PyPI** and **deprecated on NPM** because its published README contained the same corporate example strings. The Docker tag `v4.7.0` remains (image tags are immutable) but `latest` now points to v4.7.1. Please upgrade — v4.7.0 and v4.7.1 are functionally equivalent.
+- **VERSION**: 4.7.0 → **4.7.1** (PATCH — documentation hygiene, zero code / API / config change).
+
+### v4.7.0 (2026-07-31) — BM25 Fragment Query Support ⚠️ **YANKED — use v4.7.1**
+
+- **FIX (retrieval)**: BM25 tokenizer now emits **sub-tokens** for hyphenated alphanumeric-code composites, so fragment queries no longer silently return `NO_RESULTS`. Before v4.7.0, `RULE-A002` was indexed as a single token `rule-a002`; queries like `A002`, `RULE`, `B005` returned zero even when the composite lived in the corpus. This is not a bug specific to any naming scheme — it hit every hyphenated code taxonomy in typical infosec / doc corpora (`RULE-*`, `CVE-*`, `ADR-*`, `MS17-*`, `PROJECT-Custom001-xxxx`). The tokenizer now emits both the composite token AND its sub-parts of length ≥ 2 (`rule-a002` → `["rule-a002", "rule", "a002"]`). IDF preserves ranking — composite matches still rank above fragment matches because the composite is rarer. **Heuristic (perf-safety)**: sub-token expansion is triggered ONLY when the composite contains at least one digit — so `pass-the-hash`, `state-of-the-art`, and other natural-language hyphenated phrases stay as single tokens (expanding them would flood the inverted index with stop-word-like sub-parts and hurt concurrent-query throughput). (#140)
+- **NEW (tests)**: `tests/test_bm25_tokenizer_fragment.py` — 28 tests across 4 classes covering unit tokenizer behavior, end-to-end fragment matching, multi-doc disambiguation, and the 5 query patterns explicitly reported (`RULE-A019`, `PROJECT-Custom001-xxxx`, `RULE`, `A019`, `B005`). Includes retrocompat sentinel + IDF ranking invariant. Baseline: 314 → 342.
 - **UPGRADE NOTE**: run `reindex_documents(force=True)` once after upgrade to rebuild the BM25 inverted index with the new sub-token emission. ChromaDB vectors are untouched. Expect a **~30–50% increase in BM25 index memory** (measured +32.5% on synthetic corpus) — negligible for typical corpora, worth reviewing for deployments with >100K docs.
-- **KNOWN LIMITATION**: alphanumeric-glued tokens without separators (`HARDLAB11`, `6a635623e114b8...`) still require exact match — no sub-token expansion possible without character n-grams (5-10× index overhead). Tracked as follow-up.
+- **KNOWN LIMITATION**: alphanumeric-glued tokens without separators (single-token identifiers with no hyphens, hash prefixes) still require exact match — no sub-token expansion possible without character n-grams (5-10× index overhead). Tracked as follow-up.
 - **VERSION**: 4.6.0 → **4.7.0** (MINOR — tokenization behavior change + reindex requirement, though no API/config break).
 
 ### v4.6.0 (2026-07-30) — MCP spec 2026-07-28 via Anthropic Tier 1 SDK (mcp 2.x)

{knowledge_rag-4.7.0 → knowledge_rag-4.8.0}/config.example.yaml

@@ -108,6 +108,34 @@ documents:
     chunk_size: 1000
     chunk_overlap: 200
 
+  # ─────────────────────────────────────────────────────────────────────
+  # Indexing performance tuning (v4.8.0+)
+  # ─────────────────────────────────────────────────────────────────────
+  #
+  # batch_size — Number of chunks per ChromaDB batch add() call. Higher
+  # values mean fewer SQLite round-trips and faster indexing, at the
+  # cost of RAM (roughly batch_size * embedding_dim * 4 bytes for
+  # float32 vectors sitting in memory while the batch flushes).
+  # Default 500 preserves prior hardcoded behavior byte-for-byte.
+  # Valid range: 1 to 5000 (values outside are clamped with a WARN).
+  batch_size: 500
+
+  # parallel_workers — Opt-in threading around ChromaDB batch adds.
+  # Default 1 = single-threaded = safe on all platforms.
+  #
+  # When set > 1, per-batch ``collection.add(...)`` calls run inside a
+  # ThreadPoolExecutor. ONNX embedding itself is serialized (internal
+  # session lock), so the win comes from SQLite writes overlapping with
+  # the NEXT batch's inference — NOT from parallel inference. Expect
+  # modest gains (10-30%) on multi-batch documents, zero gain on
+  # single-batch docs (the pool is skipped for those).
+  #
+  # WARNING: parallel_workers > 4 on Windows may hit ONNX threading
+  # issues or SQLite lock contention — monitor stability. Recommended
+  # cap: 4 on Windows, 8 on Linux/macOS.
+  # Valid range: 1 to 16 (values outside are clamped with a WARN).
+  parallel_workers: 1
+
 
 # ============================================================================
 # MODELS
@@ -119,23 +147,78 @@ models:
   # Embedding model — converts text into vectors for semantic search.
   # Runs locally via ONNX (no API key needed, no data leaves your machine).
   #
-  # Options (from fastest to most accurate):
-  #   "BAAI/bge-small-en-v1.5"    → 384 dims, ~33MB, fast    (default)
-  #   "BAAI/bge-base-en-v1.5"     → 768 dims, ~130MB, balanced
-  #   "BAAI/bge-large-en-v1.5"    → 1024 dims, ~335MB, most accurate
+  # Two ways to configure (v4.8.0+):
+  #
+  # ---- Option A: named profile (recommended) ----
+  # ``profile`` is a shorthand that fills model + dimensions + prefixes
+  # from a single named preset. Available profiles:
   #
-  # Multilingual alternatives:
-  #   "BAAI/bge-small-zh-v1.5"         → Chinese
-  #   "intfloat/multilingual-e5-small"  → 100+ languages
+  #   compact       — BAAI/bge-small-en-v1.5 (384D, ~33MB, English-only, fastest — default)
+  #   quality       — BAAI/bge-large-en-v1.5 (1024D, ~335MB, English-only, ~2x slower)
+  #   multilingual  — intfloat/multilingual-e5-large (1024D, 100+ languages
+  #                   incl. PT-BR / Spanish / Cyrillic / Arabic / CJK; ships
+  #                   the required "query: " / "passage: " prefixes)
+  #   custom        — opt out: respect the explicit model/dimensions/prefix
+  #                   fields below (default when ``profile`` is omitted)
   #
-  # WARNING: Changing this after indexing requires a full reindex.
+  # When ``profile`` is anything other than "custom", the resolver IGNORES
+  # any explicit ``model`` declared below and emits a
+  # ``[WARN] profile takes precedence`` log on startup.
+  #
+  # ---- Option B: explicit model (custom profile) ----
+  # Legacy shape — set ``profile: "custom"`` (or omit ``profile``) and
+  # declare the model + dimensions directly. Additional community models:
+  #   "BAAI/bge-base-en-v1.5"          → 768 dims, ~130MB, balanced
+  #   "BAAI/bge-small-zh-v1.5"         → 384 dims, Chinese-only
+  #   "intfloat/multilingual-e5-small" → 384 dims, 100+ languages (smaller e5)
+  #
+  # WARNING: changing profile / model / dimensions / prefix after indexing
+  # REQUIRES a full reindex (``reindex_documents(force=True)``). Existing
+  # chunks were embedded with the previous configuration and their vectors
+  # are not comparable to the new ones.
   embedding:
+    # Uncomment to switch to a named profile (see Option A above).
+    # profile: "multilingual"
+
+    # Explicit model / dimensions — respected only when profile is
+    # "custom" (default). Ignored with a WARN under any other profile.
     model: "BAAI/bge-small-en-v1.5"
     dimensions: 384
-    # GPU acceleration for ONNX embeddings (5-10x faster indexing).
-    # Requires: pip install knowledge-rag[gpu]  (installs onnxruntime-gpu)
-    # Falls back to CPU automatically if CUDA is unavailable.
-    gpu: false
+
+    # Query / passage prefixes — prepended to text BEFORE embedding.
+    # Required by some model families (intfloat/e5, thenlper/gte, ...);
+    # leave empty for bge / mxbai. When a profile ships prefixes, this
+    # override wins ONLY if the key is present here (empty string "" IS
+    # an explicit override).
+    query_prefix: ""
+    passage_prefix: ""
+
+    # GPU acceleration mode (v4.8.0+):
+    #   "auto"  → probe CUDA at startup; use if ready, fall back to CPU (default)
+    #   "true"  → force CUDA; if not ready, log WARN and fall back to CPU
+    #   "false" → never probe; runs on CPU with zero startup overhead
+    #
+    # Legacy bool `gpu: true/false` continues to work — it is normalized to the
+    # string form internally, so old configs keep behaving exactly the same.
+    #
+    # GPU dependency chain (all installed via pip in the same venv):
+    #   1. onnxruntime-gpu, CUDA 12 variant (CUDA 13 is NOT supported by FastEmbed):
+    #        pip install onnxruntime-gpu \
+    #          --extra-index-url \
+    #          https://aiinfra.pkgs.visualstudio.com/PublicPackages/_packaging/onnxruntime-cuda-12/pypi/simple/
+    #   2. NVIDIA runtime libs (pulled by pip, no system driver rewrite):
+    #        pip install nvidia-cudnn-cu12 nvidia-cublas-cu12 nvidia-cuda-runtime-cu12 \
+    #          nvidia-cufft-cu12 nvidia-curand-cu12 nvidia-cusolver-cu12 \
+    #          nvidia-cusparse-cu12 nvidia-nvjitlink-cu12 nvidia-cuda-nvrtc-cu12
+    #   3. Host: NVIDIA driver >= 525 (for CUDA 12). cuDNN 9 comes with nvidia-cudnn-cu12.
+    #
+    # Verify readiness manually without loading the embedding model:
+    #   python -c "from mcp_server.server import FastEmbedEmbeddings; \
+    #     import json; print(json.dumps(FastEmbedEmbeddings.verify_gpu_readiness().__dict__, \
+    #     default=str, indent=2))"
+    #
+    # Full setup guide + troubleshooting: docs/gpu-setup.md
+    gpu: "auto"
 
   # Reranker — re-scores search results for better accuracy.
   # Adds ~50-100ms per query but significantly improves result quality.
@@ -158,7 +241,16 @@ search:
   default_results: 5
 
   # Hard maximum — even if a client requests more, cap at this number.
-  max_results: 20
+  # Also bounds the candidate pool size in the semantic branch of the
+  # hybrid retriever (``_do_semantic`` asks ChromaDB for
+  # ``min(max_results * 3, config.max_results)`` candidates).
+  #
+  # v4.8.0 Fase 3 raised the default from 20 to 100. Prior to that fix,
+  # the ``min(...)`` capped semantic candidates at 20 while BM25 pulled
+  # up to ``max_results * 20 = 400`` — semantic silently starved on
+  # hybrid mode. Callers passing ``max_results`` explicitly in the MCP
+  # tool call are unaffected; only the default changed.
+  max_results: 100
 
   # ChromaDB collection name. Change this to maintain separate
   # knowledge bases in the same data directory.

{knowledge_rag-4.7.0 → knowledge_rag-4.8.0}/mcp_server/__init__.py

@@ -8,7 +8,7 @@ import sys  # noqa: I001
 _original_stdout = sys.stdout
 sys.stdout = sys.stderr
 
-__version__ = "4.7.0"
+__version__ = "4.8.0"
 __author__ = "Ailton Rocha (Lyon.)"
 
 from .config import Config  # noqa: E402