knowledge-rag 4.8.0 → 4.8.3

package/README.md

@@ -1252,6 +1252,27 @@ This keeps backward compatibility while allowing concise synonym groups.
 | 0.7 | Semantic-heavy | Conceptual queries, related topics |
 | 1.0 | Pure semantic | "How to..." questions, abstract concepts |
 
+### Search Method (v4.8.2+)
+
+Two paths ship with v4.8.2 — the default `hybrid` path (BM25 + semantic + RRF
++ optional rerank) that has been the sole path since v4.0, and an opt-in
+`fts5` lexical fast-path optimised for exact identifier queries (CVEs, MITRE
+ATT&CK codes, CWEs, file hashes, bug-bounty IDs). Enable via
+`search.lexical_fast_path.enabled: true` in `config.yaml`.
+
+| Method                  | Path              | Latency (typical)     | Best For                                     |
+|-------------------------|-------------------|-----------------------|----------------------------------------------|
+| `auto` **(default)**    | Router decides    | Router adds ~0.1ms    | Mixed workloads — safe default               |
+| `hybrid`                | BM25 + semantic   | 50-150ms (with rerank)| Prose queries, "how does X work", exploration|
+| `fts5`                  | SQLite FTS5 only  | <10ms cold / <2ms hot | CVE / MITRE / CWE / hash lookups             |
+
+The `search_knowledge` MCP tool exposes `search_method: Literal["auto",
+"hybrid", "fts5"] = "auto"` (ADR-006, additive on the LEI 1 contract). The
+default `enabled: false` preserves v4.8.1 behaviour byte-for-byte; the flip
+to `enabled: true` by default is reserved for v4.9.0 pending the CI
+perf-gate adjudication documented in ADR-004 and ADR-009. Full user guide
+in [`docs/features/fts5_fast_path.md`](docs/features/fts5_fast_path.md).
+
 ---
 
 ## Project Structure
@@ -1414,7 +1435,66 @@ Common issues:
 
 ### Unreleased
 
-_(vazio — próximas mudanças vão aqui)_
+### v4.8.3 (2026-08-10) — Critical hotfix: nuclear-rebuild + smart-reindex hardening
+
+**Recommended upgrade for any corpus with >1000 chunks or any user relying on `reindex_documents(force=True)` after a config change.** The v4.8.0 nuclear rebuild + v4.8.2 FTS5 fast-path have several latent state-management bugs that only trigger under a real-world combination (large corpus + rebuild + concurrent queries + FTS5 opt-in). Enterprise users with hacktricks-scale corpora (~50k+ chunks) are affected.
+
+Fixes 3 GitHub issues opened by @grishkovei against v4.8.1:
+
+- **Fixes #161** — `_rebuild_via_swap()` no longer rebinds `self.collection` to the empty staging collection during populate. Writes now route through the new `_write_collection` property (returns `self._staging_target` when set, else production). Queries keep reading `self.collection` unchanged. Zero-downtime is now real, not just a docs promise. `nuclear_rebuild(swap=True)` on 50k-chunk corpora previously served empty results for the entire hours-long populate window; queries now hit production until the atomic swap completes.
+- **Fixes #162** — Smart-reindex checkpoint now serializes only doc-ids the current run actually committed (via new `tracking["committed_this_run"]` set), not the full `_indexed_docs.keys()` snapshot. A document that changed on disk before the reindex reached it is no longer silently skipped on resume — its mtime/size delta is properly detected.
+- **Fixes #163** — `reindex_documents(force=True)` now propagates through to `index_all(force=True)`. The v4.8 migration path (`docs/migration-v4.8.0.md`) documented `force=True` as the way to re-embed after prefix/model changes; the hardcoded `force=False` in `reindex_all()` silently skipped every unchanged file. Prefix-only changes no longer leave mixed old/new embedding conventions.
+
+Additional fixes discovered while triaging production issues on a 5889-doc / 75016-chunk corpus:
+
+- **fix(indexing)** — `_index_lock` moved from class-level to instance-level (`__init__`). The class-level `threading.Lock` was silently shared across all `KnowledgeOrchestrator` instances, so a staging orchestrator from a previous `nuclear_rebuild` swap could be garbage-collected while holding the lock, leaving every subsequent `reindex_documents` returning `reindex_already_running` despite `active: false`. Requires daemon restart to recover pre-fix.
+- **fix(chromadb)** — `_ensure_bm25_initialized` and `_iter_chroma_chunks_for_fts5` now batch-read Chroma via `offset+limit=500` instead of `collection.get(limit=count)`. Chromadb 1.x rebuilds an `IN (?, ?, ...)` clause per returned row, so a single `limit=48184` call blew past SQLite's default `max_variable_number=999` with `Internal error: too many SQL variables`. Any user with >999 chunks who ran nuclear rebuild or triggered the FTS5 lazy migration got a silently failed rebuild.
+- **fix(fts5)** — `Fts5LexicalIndex.is_ready()` now re-reads the marker file when the cached `_ready` is False. Previously an orchestrator instantiated during nuclear-swap while the FTS5 migration was still running cached `_ready=False` forever and never re-checked the marker, permanently silencing the fast-path.
+- **fix(fts5)** — `_maybe_start_fts5_migration` cross-checks FTS5 row count vs Chroma count on startup. A stale `complete` marker paired with a near-empty FTS5 index (post-swap orphan cleanup, manual truncate, disk corruption) previously left the fast-path silent forever. The new `_fts5_marker_matches_reality` helper triggers rebuild when FTS5 holds less than 10% of Chroma's row count.
+- **fix(fts5)** — `_format_fts5_results` skips orphan hits (FTS5 has the `chunk_id` but Chroma doesn't — nuclear-rebuild residue or manual-delete race). Previously emitted empty result items with `source=""` that broke reranking and polluted result lists.
+- **fix(indexing, config)** — Added `Fts5LexicalIndex.count()` for the new sanity check.
+
+All 6 core fixes were confirmed via production reproduction on a 75016-chunk corpus + 2052 newly-added catalog documents; the `nuclear_rebuild(full_rebuild=True)` path now completes cleanly and populates FTS5 with 75016 rows on first attempt.
+
+Migration path: `pip install -U knowledge-rag==4.8.3` (or `npx -y knowledge-rag@4.8.3` / `docker pull ghcr.io/lyonzin/knowledge-rag:4.8.3`) then restart the MCP daemon. No config changes required. If your FTS5 index is corrupted from a prior partial rebuild (marker says `complete` but queries return empty), delete `data/fts5_migration.state` and restart — the daemon will detect the drift and rebuild automatically.
+
+### v4.8.2 (2026-08-10) — FTS5 Lexical Fast-Path Opt-In Release (default OFF)
+
+v4.8.2 bundles Fases 1–5 of the FTS5 Lexical Fast-Path plan
+(`.compozy/tasks/fts5-lexical-fast-path/`) as an **opt-in** release —
+`search.lexical_fast_path.enabled` remains `false` by default, so users
+who do not touch `config.yaml` keep v4.8.1 behaviour byte-for-byte
+(LEI 1 preserved). The v4.9.0 default flip is deferred to the CI
+perf-gate adjudication documented in ADR-009 (bench harness ships in
+this release; the flip PR follows once G1 ≥15× and G2 ≤2% both pass on
+CI hardware).
+
+- **DOCS (search)**: new `docs/features/fts5_fast_path.md` user guide
+  covering overview / quick-start / 5-key config reference / how it
+  works / 5 troubleshooting scenarios / when-to-use guidance. README
+  Configuration section cross-links it.
+- **NEW (bench)**: `bench/test_bench_fts5_lexical.py` ships the
+  BENCH-001 (cold, p95 ≤ 10ms target) + BENCH-002 (hot, p95 ≤ 2ms
+  target) harness against a self-contained 3865-row FTS5 corpus. The
+  gate G2 comparison against `bench/test_bench_search.py` is executed
+  by CI per `.compozy/tasks/fts5-lexical-fast-path/bench_v4_9_0_gate.md`;
+  the two-round A/B was not run locally because the release-authoring
+  workstation cannot fairly baseline against CI hardware (ADR-004
+  §Risks).
+- **DOCS (adr)**: new ADR-009 documenting why the ADR-004 gate is
+  deferred to the CI perf-gate step and what conditions unlock the
+  v4.9.0 flip PR.
+- **CHANGE (config)**: `config.example.yaml` uncomments the
+  `search.lexical_fast_path.*` block so new users see the full
+  configuration surface (still `enabled: false`). Existing installs
+  are unaffected.
+- **VERSION**: 4.8.1 → **4.8.2** (PATCH — opt-in infrastructure + docs,
+  zero behaviour change on the default install).
+- **feat (search)**: FTS5 lazy background migration for existing corpus + CRUD sync (ADR-008). `Fts5LexicalIndex` gains `add_document`/`update_document`/`remove_document` (SQL INSERT/DELETE under `_fts5_lock`, incremental — diverges from BM25's full-rebuild pattern because FTS5 mutations are O(1) native SQL) and `start_migration_background(chunk_iter_factory, docs_total, resume_from=0, on_progress=...)` (daemon thread that persists checkpoint every 100 rows into the marker file). On daemon startup, `_maybe_start_fts5_migration` reads the marker: `complete` = fast-path live, `in_progress` = resume from `docs_indexed`, `failed`/missing = start fresh migration; queries during migration fall back to hybrid + emit `fast_path_fallback_total{reason="disabled"}`. `KnowledgeOrchestrator.{add,update,remove}_document_*` gain best-effort FTS5 sync hooks after the BM25 rebuild — errors are caught, logged, and counted (`fast_path_errors_total{error_class="Fts5CrudSyncError"}`) but never propagate to the CRUD tool (upstream write must keep succeeding). `nuclear_rebuild` and `_rebuild_bm25_post_swap` drop the FTS5 db + WAL/SHM/marker and dispatch a fresh migration from the swapped-in Chroma corpus. Two Prometheus gauges expose migration progress (`knowledge_rag_fast_path_migration_docs_indexed`, `_docs_total`). Standalone rebuild helper `scripts/build_fts5_index.py --data-dir <path> [--force] [--foreground] [--verbose]` for operator-driven repair. Operational runbook in `docs/runbooks/fts5_migration.md` covering enable / wait / verify / manual rebuild / large-corpus caveat. New coverage: `tests/test_fts5_migration.py` (17 tests — TestMigrationLifecycle UT-043..048, TestCRUDSync UT-049..052, TestMigrationIntegration IT-021..023, TestCRUDSyncIntegration IT-CRUD-001..004) + `tests/test_e2e_fts5.py::test_e2e006_forced_fts5_with_migration_pending_raises` (E2E-006). Fase 4 of the FTS5 Lexical Fast-Path plan.
+- **NEW (search)**: Optional cross-encoder rerank toggle on the FTS5 fast-path via `search.lexical_fast_path.rerank_enabled` (default `false` — see ADR-003 for rationale). When `true`, `KnowledgeOrchestrator._run_fts5_search` layers `CrossEncoderReranker.rerank` on top of the FTS5 results before adjacent-chunk expansion; result items keep `search_method="fts5"` (rerank is a rescorer, not a path change) and `reranker_score` is populated. When `false`, the `knowledge_rag_fast_path_rerank_skipped_total` counter increments and the fast-path stays under the <10ms lexical latency budget. Empirical basis: Flash-Rerank MiniLM-L-6-v2 CPU ≈72ms/100 docs vs the <10ms target — keeping rerank OFF by default is the correct trade-off; opt in only when recall on lexical queries matters more than latency. New coverage: `tests/test_search.py::TestRerankToggle` (UT-062, UT-063, IT-024, IT-025). Fase 3.5 of the FTS5 Lexical Fast-Path plan.
+- **NEW (search)**: FTS5 lexical fast-path dispatch wired inside `KnowledgeOrchestrator.query()` behind the opt-in `search.lexical_fast_path.enabled` toggle (default `false` — v4.8.1 behavior byte-for-byte for users who don't touch config). When enabled, the query router (Fase 2) classifies each query; lexical queries dispatch to the FTS5 index (Fase 1) via a new `_maybe_dispatch_fts5` helper, semantic queries flow through the existing hybrid pipeline. Per ADR-003 the cross-encoder reranker is skipped on the fast-path (real opt-in toggle deferred to Task 04). Per ADR-006 the `search_knowledge` MCP tool gains one optional param `search_method: Literal["auto","hybrid","fts5"] = "auto"` — additive-safe on the LEI 1 backward-compat contract. Explicit `"fts5"` when the feature is disabled OR the index is not ready raises `Fts5NotReadyError` and the MCP wrapper surfaces it as a JSON error envelope with a `suggestion` field pointing at `search_method='auto'`. `QueryCache._make_key` gains a 5th `search_method` param so paths cannot share cache entries. `MetricsCollector` gains five FTS5 counters plus a bucketed latency histogram (`knowledge_rag_fast_path_hits_total{path}`, `..._fallback_total{reason}`, `..._latency_seconds`, `..._errors_total{error_class}`, `..._rerank_skipped_total`) exposed through `/metrics`. New coverage: `tests/test_search_method_override.py` (5 IT), `tests/test_metrics.py` (2 UT), `tests/test_e2e_fts5.py` (7 E2E), and extensions to `tests/test_search.py` (16 IT/UT — dispatch, config toggle, cache key, metrics scrape), `tests/test_backwards_compat.py` (3 UT — MCP tool signature + guardrail), `tests/test_fts5_index.py` (3 UT — fallback dispatch, busy_timeout, NotReady message). Fase 3 of the FTS5 Lexical Fast-Path plan.
+- **NEW (search)**: Query router lexical/semantic classifier (`mcp_server/query_router.py`) ships isolated and unused. Static regex classifier per ADR-002 — patterns are pre-compiled at construction (fail-fast on `re.error`) and matched in YAML-declared order (first-match-wins, OQ-2). API surface: `QueryRouter(patterns: Sequence[str])` plus `classify(query: str) -> Literal["lexical", "semantic"]`; empty query classifies as `"semantic"` (safe default). Zero coupling to `mcp_server/fts5_index` — the router is storage-agnostic. Task 03 will wire the dispatch inside `KnowledgeOrchestrator.query()` behind `config.fts5_enabled`; this PR is deliberately unwired so runtime behaviour is preserved. New coverage in `tests/test_query_router.py` (19 tests covering UT-001..UT-013 including a `hypothesis` property test). Fase 2 of the FTS5 Lexical Fast-Path plan (`.compozy/tasks/fts5-lexical-fast-path/`).
+- **NEW (search)**: FTS5 lexical index module (`mcp_server/fts5_index.py`) ships isolated and unused behind the opt-in `search.lexical_fast_path.enabled` config field (default `false`). Encapsulates the dedicated `data/fts5_index.db` SQLite FTS5 store with the ADR-005 tokenizer (`unicode61 remove_diacritics 2 tokenchars '-_.'`) so composite identifiers like `MDR-AD002`, `CVE-2021-4034`, `T1078.001`, `pass-the-hash`, `context.py` survive tokenization intact. Public surface: `Fts5LexicalIndex` (search + connection lifecycle + WAL/busy_timeout=5000ms/synchronous=NORMAL PRAGMAs per ADR-001), `Fts5MigrationState` (atomic JSON marker via `tempfile` + `fsync` + `os.replace`), and three `RuntimeError`-derived exceptions (`Fts5NotReadyError`, `Fts5CorruptError`, `Fts5MigrationError`) per ADR-004/Q4. `Config` gains four `fts5_*` fields (`fts5_enabled`, `fts5_min_hits`, `fts5_rerank_enabled`, `fts5_patterns`) backed by a new `_get_nested()` helper that mirrors the `models.reranker.*` pattern. `config.example.yaml` documents the new `search.lexical_fast_path.*` block (commented out). All FTS5 queries use prepared statements and every user-supplied token is quoted so FTS5 metacharacters (`MATCH`, `NEAR`, `OR`, `AND`, `NOT`, `*`, `:`, `(`, `)`, `"`) cannot leak through. Task 03 will wire the dispatch inside `KnowledgeOrchestrator.query()`; this PR is deliberately unwired so runtime behaviour of v4.8.1 is preserved byte-for-byte. New coverage in `tests/test_fts5_index.py` (25 tests covering UT-020..028, UT-054..058, UT-061, IT-MIG-001) and `tests/test_config_fts5.py` (9 tests covering UT-030..038). Fase 1 of the FTS5 Lexical Fast-Path plan (`.compozy/tasks/fts5-lexical-fast-path/`).
 
 ### v4.8.0 (2026-08-06) — Multilingual Foundation + Zero-Downtime Reindex
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "knowledge-rag",
-  "version": "4.8.0",
+  "version": "4.8.3",
   "description": "Local RAG System for Claude Code — Hybrid search + Cross-encoder Reranking + 13 MCP Tools + 20 Format Parsers. Zero external servers.",
   "bin": {
     "knowledge-rag": "./bin/cli.js"