knowledge-rag 4.7.1 → 4.8.0

package/README.md

@@ -1414,6 +1414,32 @@ Common issues:
 
 ### Unreleased
 
+_(vazio — próximas mudanças vão aqui)_
+
+### v4.8.0 (2026-08-06) — Multilingual Foundation + Zero-Downtime Reindex
+
+v4.8.0 é a release de fundação para migração pra embeddings multilingual (`intfloat/multilingual-e5-large`) sem forçar downtime nem breaking change. Sete fases entregaram: security wire (7 vetores fechados que existiam desde v4.5.1), embedding profiles com prefix opt-in, GPU auto-detect com dependency chain documentada, batch/parallel indexing configurável + fix da assimetria BM25/semantic candidate pool, progress granular + resume checkpoint, dual-collection swap zero-downtime, e refactor pra manter complexidade Radon C+ na superfície tocada.
+
+Zero breaking change: usuários que atualizarem SEM tocar em `config.yaml` continuam com `bge-small-en-v1.5` byte-for-byte. Opt-in para multilingual: `profile: "multilingual"` + `reindex_documents(force=True)` — durante o reindex, RAG continua servindo da collection antiga (zero downtime via staging swap). Detalhes de upgrade em `docs/migration-v4.8.0.md`.
+
+- **REFACTOR (server + config)**: extracted 40+ helpers from 12 functions touched by v4.8.0 F1–F5 to enforce Core hard limits (function ≤30 lines). Biggest wins — `KnowledgeOrchestrator._index_all_impl` split from 246 → 23 lines (14 focused helpers covering setup / per-doc processing / progress metrics / persistence / teardown), `Config.__post_init__` split from 185 → 12 lines (13 domain validators), `reindex_documents` MCP tool from 100 → 30 lines (mode resolution + resume hydration + response formatting extracted to module helpers). Also refactored `_index_document` (91→19), `_rebuild_via_swap` (66→29), `_validate_staging` (56→24), `_cleanup_stale_staging_collections` (48→26), `get_reindex_status` (49→12), `_print_gpu_banner` (46→20), `start_reindex_background` (46→26), `_swap_collections_atomic` (43→25), `_route_load` (41→9). New sentinel `_SKIP_DOC` distinguishes "caller must skip" from "index as fresh" in the per-doc worker. Radon complexity average dropped from D (23.4) to C (17.2); zero D+ functions remain in F1–F5 touched surface. File-level extract (`server.py` = 4009 lines, `config.py` = 990 lines) deferred to v4.9.0 because `test_backwards_compat.py` validates the MCP contract by AST and moving classes across modules multiplies regression surface. Zero behavior change — full test sweep (`test_config` + `test_embedding_profile` + `test_gpu_auto` + `test_batch_parallel` + `test_reindex_resume` + `test_swap_zero_downtime` + `test_backwards_compat` + `test_bm25_*` + `test_search` + `test_dedup` + `test_ingestion*` + `test_lazy_embeddings` + `test_reranker_fallback` + `tests/security/` + all misc) = 557 passed / 3 skipped / 9 xfailed / 0 failed. Full audit report in `docs/refactor-audit-v4.8.0-fase-6.md`. v4.8.0 Fase 6.
+- **PERF (reindex)**: `nuclear_rebuild(swap=True)` is now the default, replacing the destructive-first rebuild that took the RAG offline for the entire duration (4min–40h depending on corpus + hardware). New workflow — cleanup stale stagings (24h TTL) → snapshot orchestrator state (`self.collection` / `self.bm25_index` / `self._indexed_docs` / `self._source_to_docid`) → create `{collection_name}__staging_{unix_ts}` (timestamped to avoid collisions with concurrent or previously-crashed rebuilds) → populate via temporary orchestrator rebind (`self.collection` points at staging, `self.bm25_index` is a throwaway `BM25Index()` so production BM25 keeps serving queries throughout) → validate three gates (`count >= baseline * 0.9` with 10% loss threshold for parser-skipped docs / 4 of 5 canonical queries `readme/function/import/return/class` return hits / `query()` does not raise, catching embedding dim mismatches) → atomic two-step swap via `Collection.modify(name=)` (prod → `__old_{ts}` frees the name, staging → prod assumes it, delete `__old_{ts}` non-fatally) with rollback-rename on mid-swap failure → post-swap BM25 rebuild from swapped-in ChromaDB contents + query cache invalidation. Any failure in populate/validate/swap triggers rollback via the snapshot and staging cleanup — production sees zero net change. Startup cleanup handles crash-orphaned stagings automatically (also runs before each rebuild). Storage doubles temporarily during rebuild (both collections on disk), back to 1x after swap. `nuclear_rebuild(swap=False)` preserves the legacy destructive behavior byte-for-byte in `_rebuild_destructive` for backwards compat / forced-cleanup edge cases. New `tests/test_swap_zero_downtime.py` (15 tests, ~1.7s, all mocked — no real ChromaDB or FastEmbed touched) covers naming format + populate BM25 isolation + all three validation gates + fresh-install zero-baseline path + post-swap `self.collection` reconnect + BM25 rebuild only after swap success + 100 parallel queries against a captured production handle during populate (zero errors, non-empty results) + stale/recent staging cleanup + non-staging collection preservation + backwards-compat dispatch. Docs in `docs/reindex-operations.md` section "Zero-downtime Rebuild (v4.8.0 Fase 5+)". v4.8.0 Fase 5.
+- **NEW (reindex)**: `reindex_documents(resume=True)` recovers from `data/reindex_checkpoint.json` after an interrupted smart reindex. Checkpoint is written every 500 docs or 30s (whichever first), and stores `indexed_doc_ids` + `chunks_processed` + `config_signature` (SHA256 of `embedding_model | embedding_dim | chunk_size | chunk_overlap`). A config mismatch between checkpoint write and resume load invalidates the checkpoint with a WARN — this prevents a mixed collection (partial old-model vectors + partial new-model vectors) which would silently degrade retrieval. The combination `resume=True + full_rebuild=True` is rejected with a structured error because nuclear rebuild wipes the collection first — resuming into a partial state would leave no coherent index. The `resume=True` flag also implicitly forces `mode='smart_reindex'` even when `force=False` was passed, so an interrupted smart run cannot silently fall back to `incremental` (which would ignore the checkpoint entirely). Existing 2-arg `reindex_documents(force, full_rebuild)` signature preserved — `resume` is an opt-in tail kwarg. Missing / corrupt / future-version / non-dict checkpoints all degrade gracefully to "start fresh" instead of raising. Written to `data/reindex_checkpoint.json` via atomic `.tmp` + `os.replace()` — Windows-safe on NTFS in-directory. Cleared automatically on successful reindex completion. Metadata (`index_metadata.json`) is flushed alongside every checkpoint write so `_indexed_docs` on disk stays in sync with the checkpoint's `indexed_doc_ids` list. New `tests/test_reindex_resume.py` (22 tests, all `object.__new__` isolation — no ChromaDB, no FastEmbed download) covers write cadence + atomic write + schema + load validation (6 failure modes) + resume kwarg happy path + rejection combos + status field flow. v4.8.0 Fase 4.
+- **NEW (reindex)**: `get_reindex_status()` now returns 6 new fields during active reindex — `chunks_processed` (chunks committed to ChromaDB so far), `chunks_total` (rolling estimate, 0 during warmup then running average from completed docs × `total_files`), `throughput_cps` (chunks/sec, sliding window bounded by the smaller of 100 samples or 30 seconds — deque `maxlen=100` combined with age-based pruning each iteration keeps the number stable during transient stalls without diluting current speed with ancient samples), `eta_seconds` (estimated seconds to completion, derived from throughput + remaining chunks, guarded against div-by-zero), `checkpoint_saved_at` (ISO timestamp of last checkpoint write, `null` until first checkpoint at 500 docs / 30s), `resumed` (bool — `true` when the current run was recovered from a checkpoint via `resume=True`, `false` otherwise). Enables real-time progress dashboards and the "should I keep waiting or restart" call. Idle path (`active: false`) unchanged — still returns just `last_result` or `last_error`. v4.8.0 Fase 4.
+- **DOCS (reindex)**: `docs/reindex-operations.md` NEW — documents the sync-vs-async gotcha for `reindex_documents(full_rebuild=True)` (daemon thread dies with process → RAG offline until manual restart), recommends `orchestrator.nuclear_rebuild()` sync for CLI scripts and `reindex_documents(force=True) + resume=True` for long unstable smart rebuilds. Also covers resume semantics (only valid with `full_rebuild=False`, implicitly forces `smart_reindex` mode, graceful degrade to fresh start on missing/corrupt/version/signature mismatch), checkpoint cadence (500 docs OR 30s, whichever first, plus rationale for the OR condition), `config_signature` invalidation semantic, full lifecycle (write on cadence, clear on success, clear on failed resume load), progress fields table with warmup behavior, example recovery flow showing crash → `ls checkpoint` → `resume=True` → poll status with `resumed:true` marker, and escalation rules of thumb for the three most likely operator questions. v4.8.0 Fase 4.
+- **PERF (indexing)**: `documents.batch_size` (default 500) and `documents.parallel_workers` (default 1, opt-in) are now configurable in `config.yaml`. The previously hardcoded `_CHROMA_BATCH_SIZE = 500` in `server.py:_index_document` continues as fallback constant for tests that instantiate the orchestrator without a full `Config` (via `object.__new__`, as `tests/test_search.py` does). When `parallel_workers > 1` AND the document produces more than one batch, the per-batch `collection.add(...)` calls run inside a `ThreadPoolExecutor(max_workers=parallel_workers)`. Single-batch documents skip the pool entirely — no orchestration overhead to extract from a single call. Gain comes from SQLite writes overlapping with the NEXT batch's ONNX inference (embedding itself is serialized by ONNX session lock — never parallel inference). Windows users should monitor stability at `workers > 4` (extra `[WARN]` on startup). `batch_size` clamped to `[1, 5000]`, `parallel_workers` clamped to `[1, 16]` — out-of-range values are clamped with a `[WARN]`. New `bench/test_bench_reindex.py` (3 benchmarks) uses a `_SleepingCollection` stub with 5ms per-`add()` sleep so the parallel path measurably overlaps without depending on a real ChromaDB; local numbers: reindex_batch_2000 5.66ms, reindex_default 5.68ms, reindex_parallel_4 7.30ms (200-chunk doc, 4 concurrent batches, sequential would be ~20ms). New `tests/test_batch_parallel.py` (20 tests, ~1.6s) covers config parsing, clamping, Windows-only warn on `workers > 4`, sequential vs parallel dispatch, single-batch skip, `getattr` fallback to `_CHROMA_BATCH_SIZE`, exception propagation via `.result()`, and empty-doc short-circuit. Defaults preserve v4.7.1 behavior byte-for-byte. v4.8.0 Fase 3.
+- **FIX (search)**: `search.max_results` default raised from `20` to `100`. This eliminates the asymmetric candidate pool in hybrid retrieval where `_do_semantic` clamped candidates to `min(max_results * 3, config.max_results) = min(15, 20) = 20` while BM25 pulled up to `max_results * 20 = 400` — semantic silently starved without any log signal, degrading hybrid retrieval quality on any query where the semantic branch could have contributed better candidates. Callers passing `max_results` explicitly in the MCP tool call (`search_knowledge`, `search_similar`) are unaffected — only the config-loaded default changed. New `TestDoSemanticCandidateMath` class in `tests/test_search.py` (2 tests) pins the fix against future regressions: `test_semantic_asks_chromadb_for_3x_max_results` proves the 3× semantic candidate pool, `test_semantic_pool_is_bounded_by_config_max_results` proves the cap still works when `max_results * 3` exceeds it. v4.8.0 Fase 3.
+- **CHANGE (gpu)**: `models.embedding.gpu` is now tri-state (`"auto"` | `"true"` | `"false"`) with `"auto"` as the new default. In auto mode, `FastEmbedEmbeddings.verify_gpu_readiness()` runs four startup checks (`CUDAExecutionProvider` present in `onnxruntime.get_available_providers()` / required NVIDIA DLLs reachable via `PATH` — `cudart64_12.dll`, `cudnn64_9.dll`, `cublasLt64_12.dll` on Windows and their `.so.12` / `.so.9` counterparts on Linux / `nvidia-smi --query-gpu=name,memory.total` exits 0 / a minimal single-op ONNX `InferenceSession` initializes on the CUDA provider) and uses CUDA when all pass, CPU otherwise. Legacy bool `gpu: true`/`gpu: false` continues to work — normalized to the string form internally. The startup banner now prints on every path (including CPU-only and fallback), reports the mode (`auto-cuda` / `auto-cpu-fallback` / `forced-cpu` / `forced-cuda` / `forced-cuda-fallback`), and surfaces the probe `fallback_reason` on CPU paths with a hint pointing at the full pip install chain. Full dependency chain (onnxruntime-gpu CUDA 12 variant via `--extra-index-url` + `nvidia-cudnn-cu12` + `nvidia-cublas-cu12` + siblings) and troubleshooting for the six most common failures documented in `docs/gpu-setup.md`. Zero behavior change for users who had `gpu: false` explicitly. New coverage in `tests/test_gpu_auto.py` (13 tests, all mocked — no real GPU required to run). v4.8.0 Fase 2. (#149)
+- **NEW (embedding)**: `models.embedding.profile` opt-in preset (`compact` | `quality` | `multilingual` | `custom`) plus `query_prefix` / `passage_prefix` fields. The `multilingual` profile ships `intfloat/multilingual-e5-large` (1024D, 100+ languages) with the required `"query: "` / `"passage: "` prefixes wired into `FastEmbedEmbeddings.__call__`, `.embed_query`, and `.embed_documents`. `compact` preserves the current `bge-small-en-v1.5` default byte-for-byte. Changing profile or prefix requires `reindex_documents(force=True)` because existing chunks were embedded without the new prefix. (#148)
+- **FIX (bm25)**: BM25 tokenizer is now Unicode-aware. Before this fix, the `[a-z0-9]` character class in `BM25Index._tokenize()` and `_metadata_path_score()` (server.py:652, 659, 720) dropped every code point outside Basic Latin — Cyrillic, Greek, CJK, Arabic corpora silently degraded to semantic-only search (empty `bm25_rank`, `search_method: semantic`, no warning). Portuguese/Spanish/French text with diacritics was **fragmented** on the accent: `"informação técnica"` tokenised as `['informa', 'o', 't', 'cnica']` and BM25 scored garbage fragments instead of the actual words. The regex now uses `[^\W_]+(?:-[^\W_]+)*` which keeps hyphenated composites (`CVE-2024-1234`, `rule-a002`, `pass-the-hash`) byte-identical to prior behaviour while also matching letters from any script. All 28 tests in `test_bm25_tokenizer_fragment.py` remain green. New coverage in `tests/test_bm25_unicode_tokenizer.py` (13 tests) exercises Cyrillic / Greek / Arabic / CJK / PT-BR / mixed-script / composite-with-diacritics / underscore-separation. Closes #145. (#147)
+- **DOCS (security)**: rewrite `SECURITY.md` reporting section — remove the `lyonzin@users.noreply.github.com` fallback (GitHub `noreply.github.com` addresses are outbound-only and never receive mail, so that channel was decorative) and direct all reports to the GitHub Security Advisory form at `/security/advisories/new`. Private vulnerability reporting is now enabled on the repo, so the form is available to any GitHub user without collaborator status. Closes #146. (#147)
+- **FIX (security)**: wire `validate_path_within` on all filepath-taking orchestrator tools — `get_document`, `add_document_from_content`, `update_document_content`, `remove_document_by_path`, `search_similar`. Escape attempts return a structured `{"error": "Filepath rejected: ..."}` and touch nothing on disk. Closes 5 latent path traversal / arbitrary-file-{read,write,delete} vectors that the `mcp_server/security.py` primitives had covered since v4.5.1 but which never had callers in the mutation path. (#147)
+- **FIX (security)**: wire `sanitize_external_content` on `add_from_url`. The fetched body now flows through a new `external_source` kwarg on `add_document_from_content` (default `None` — operator ingests stay byte-identical) which fences it in a provenance envelope and defuses known prompt-injection sentinels (`<|im_start|>`, `[INST]`, etc.) before the parser or any downstream LLM sees it. Closes OWASP LLM01:2025 on the URL ingest path. (#147)
+- **FIX (security)**: install `BearerAuthMiddleware` on HTTP transports (`sse`, `streamable-http`) when `config.auth_bearer_token` is set. Wraps the FastMCP ASGI app and serves via uvicorn; unset token keeps prior `mcp.run()` open-port behaviour with a `[WARN]` on stderr; `stdio` transport is never wrapped. Closes the auth-bypass gap where `auth_bearer_token` had existed as a config field since v4.0.0 but was never compared. (#147)
+- **NEW (tests)**: `tests/security/test_add_from_url_sanitized.py` and `tests/security/test_bearer_middleware_attached.py` — integration tests that prove the wire (not just the library primitives). The `@pytest.mark.xfail("integration pending v4.6.0")` markers on 9 tests across `test_path_traversal.py`, `test_bearer_auth.py`, `test_prompt_injection.py` are removed — they now pass. `tests/security/` totals: 138 passed, 7 skipped, 6 xfailed. (#147)
+- **DOCS (perf)**: `docs/perf-baseline-v4.7.1.md` + `perf-baseline-v4.7.1.json` — reference microbenchmarks + `nuclear_rebuild()` timing (100-doc synthetic corpus) captured at the tip of v4.7.1. Non-runtime baseline for the planned Phase 3 batch-parallel embedding work in v4.8.0.
+- **NEW (release automation)**: `release.yml` now has a `yank-superseded` job that reads a `YANKS:` marker from the release body and automatically yanks the listed version from PyPI + deprecates it on NPM after the new version publishes. Format: `YANKS: v4.7.0` or `YANKS: v4.7.0 — some reason`. Requires the `PYPI_API_TOKEN` repo secret (classic scoped token — Trusted Publisher OIDC does not currently support yank). Job runs after `publish-pypi`/`publish-npm`/`publish-docker`, is idempotent, and degrades gracefully when secrets are missing (emits `::warning::` instead of failing). If the marker is absent, the job is a no-op.
+
 ### v4.7.1 (2026-08-01) — Documentation Hygiene
 
 - **CHORE (docs, tests)**: sanitize identifiers in test fixtures and CHANGELOG examples. v4.7.0 referenced code names from a real deployment corpus in `tests/test_bm25_tokenizer_fragment.py` and the v4.7.0 CHANGELOG entry; those are replaced with synthetic placeholders (`RULE-A00X`, `PROJECT-Custom001-xxxx`) and well-known public identifiers (`CVE-*`, `MS17-*`, `ADR-*`, `T1078.002`) that carry no organization-specific taxonomy. Zero behavior change — all 28 fragment tests pass unchanged, tokenizer logic identical.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "knowledge-rag",
-  "version": "4.7.1",
+  "version": "4.8.0",
   "description": "Local RAG System for Claude Code — Hybrid search + Cross-encoder Reranking + 13 MCP Tools + 20 Format Parsers. Zero external servers.",
   "bin": {
     "knowledge-rag": "./bin/cli.js"