knowledge-rag 4.8.1 → 4.8.4

package/README.md

@@ -230,19 +230,32 @@ See [Changelog](#changelog) for full history.
 flowchart TB
     subgraph MCP["MCP SERVER (FastMCP)"]
         direction TB
-        TOOLS["13 MCP Tools<br/>search | get | add | update | remove<br/>reindex | reindex_status | list | stats | url | similar | evaluate"]
+        TOOLS["13 MCP Tools (frozen, LEI 1)<br/>search_knowledge | get_document | search_similar<br/>add_document | add_from_url | update_document | remove_document<br/>reindex_documents | get_reindex_status<br/>list_categories | list_documents | get_index_stats | evaluate_retrieval"]
     end
 
-    subgraph SEARCH["HYBRID SEARCH ENGINE"]
+    subgraph SEARCH["SEARCH DISPATCH (v4.8.2+)"]
+        direction TB
+        SM["search_method dispatch<br/>auto | hybrid | fts5"]
+        QROUTER["QueryRouter (regex)<br/>lexical vs semantic<br/>(ADR-002)"]
+        FTS5PATH["FTS5 Fast-Path<br/>(exact identifiers <10ms)"]
+        HYBRIDPATH["Hybrid Pipeline<br/>(conceptual queries)"]
+
+        SM --> QROUTER
+        QROUTER -->|lexical| FTS5PATH
+        QROUTER -->|semantic| HYBRIDPATH
+        FTS5PATH -.->|not_ready / low_hits<br/>fallback| HYBRIDPATH
+    end
+
+    subgraph HYBRID["HYBRID PIPELINE"]
         direction LR
-        ROUTER["Keyword Router<br/>(word boundaries)"]
+        KWROUTER["Keyword Router<br/>(word boundaries → category filter)"]
         SEMANTIC["Semantic Search<br/>(ChromaDB)"]
         BM25["BM25 Keyword<br/>(inverted-index + expansion)"]
         RRF["Reciprocal Rank<br/>Fusion (RRF)"]
         RERANK["Cross-Encoder<br/>Reranker"]
 
-        ROUTER --> SEMANTIC
-        ROUTER --> BM25
+        KWROUTER --> SEMANTIC
+        KWROUTER --> BM25
         SEMANTIC --> RRF
         BM25 --> RRF
         RRF --> RERANK
@@ -250,9 +263,11 @@ flowchart TB
 
     subgraph STORAGE["STORAGE LAYER"]
         direction LR
-        CHROMA[("ChromaDB<br/>Vector Database")]
-        COLLECTIONS["Collections<br/>security | ctf<br/>logscale | development"]
+        CHROMA[("ChromaDB<br/>data/chroma_db/<br/>(vector store)")]
+        FTS5DB[("SQLite FTS5<br/>data/fts5_index.db<br/>(lexical index, ADR-001)")]
+        COLLECTIONS["8 Categories<br/>redteam | blueteam | ctf<br/>security | logscale<br/>development | aar | general"]
         CHROMA --- COLLECTIONS
+        FTS5DB --- COLLECTIONS
     end
 
     subgraph EMBED["EMBEDDINGS (In-Process)"]
@@ -264,36 +279,55 @@ flowchart TB
     subgraph INGEST["DOCUMENT INGESTION"]
         PARSERS["20 Parsers<br/>MD | PDF | TXT | PY | C | H | CPP | JS | JSX | TS | TSX | JSON | XML | CSV<br/>DOCX | XLSX | PPTX | IPYNB | MQH | MQ4"]
         CHUNKER["Chunking<br/>MD: section-aware<br/>Other: 1000 chars + 200 overlap"]
-        PARSERS --> CHUNKER
+        FTS5SYNC["FTS5 CRUD sync<br/>(v4.8.2 task 05)"]
+        PARSERS --> CHUNKER --> FTS5SYNC
     end
 
     CLAUDE["Claude Code"] --> MCP
     MCP --> SEARCH
-    SEARCH --> STORAGE
+    HYBRIDPATH --> HYBRID
+    FTS5PATH --> FTS5DB
+    HYBRID --> STORAGE
     STORAGE --> EMBED
     INGEST --> EMBED
     EMBED --> STORAGE
+    FTS5SYNC --> FTS5DB
 ```
 
 ### Query Processing Flow
 
 ```mermaid
 flowchart TB
-    QUERY["User Query<br/>'mimikatz credential dump'"] --> EXPAND
+    QUERY["User Query<br/>'mimikatz credential dump' | 'CVE-2021-4034'"] --> METHOD
+
+    subgraph DISPATCH["Dispatch (v4.8.2+, ADR-006)"]
+        METHOD{"search_method<br/>auto | hybrid | fts5"}
+        AUTOROUTE["QueryRouter.classify()<br/>regex-based lexical detection"]
+        FASTPATH["FTS5 Fast-Path<br/>SQLite MATCH<br/>(&lt;10ms cold, &lt;2ms hot)"]
+        NOTREADY{"FTS5 ready<br/>AND hits >= min_hits?"}
+
+        METHOD -->|auto| AUTOROUTE
+        METHOD -->|fts5| FASTPATH
+        METHOD -->|hybrid| EXPAND
+        AUTOROUTE -->|lexical| FASTPATH
+        AUTOROUTE -->|semantic| EXPAND
+        FASTPATH --> NOTREADY
+        NOTREADY -->|no, fallback| EXPAND
+    end
 
-    subgraph EXPANSION["Query Expansion"]
+    subgraph EXPANSION["Query Expansion (hybrid path)"]
         EXPAND["Synonym Expansion<br/>mimikatz -> mimikatz, sekurlsa, logonpasswords"]
     end
 
-    EXPAND --> ROUTER
+    EXPAND --> KWROUTER
 
-    subgraph ROUTING["Keyword Routing"]
-        ROUTER["Keyword Router"]
+    subgraph ROUTING["Keyword Routing (category filter)"]
+        KWROUTER["Keyword Router<br/>(word boundaries)"]
         MATCH{"Word Boundary<br/>Match?"}
         CATEGORY["Filter: redteam"]
         NOFILTER["No Filter"]
 
-        ROUTER --> MATCH
+        KWROUTER --> MATCH
         MATCH -->|Yes| CATEGORY
         MATCH -->|No| NOFILTER
     end
@@ -326,7 +360,8 @@ flowchart TB
     BM25 --> RRF
 
     ADJ --> MINSCORE
-    SNIPPET --> RESULTS["Results<br/>search_method: hybrid|semantic|keyword<br/>score + filtered_by_score + content_length"]
+    NOTREADY -->|yes, hit| MINSCORE
+    SNIPPET --> RESULTS["Results<br/>search_method: fts5 | hybrid | semantic | keyword<br/>routed_by: fts5_router | none<br/>score + filtered_by_score + content_length"]
 ```
 
 ### Document Ingestion Flow
@@ -334,7 +369,7 @@ flowchart TB
 ```mermaid
 flowchart LR
     subgraph INPUT["Input"]
-        FILES["documents/<br/>├── security/<br/>├── development/<br/>├── ctf/<br/>└── general/"]
+        FILES["documents/<br/>├── security/<br/>│   ├── redteam/<br/>│   ├── blueteam/<br/>│   └── ctf/<br/>├── aar/<br/>├── logscale/<br/>├── development/<br/>└── general/"]
     end
 
     subgraph PARSE["Parse (20 formats)"]
@@ -354,9 +389,18 @@ flowchart LR
         FASTEMBED["FastEmbed ONNX<br/>bge-small-en-v1.5<br/>(384D, CPU or GPU)"]
     end
 
+    subgraph DISPATCH["Write Dispatch (v4.8.3, #161)"]
+        WCOL{"_write_collection<br/>routes writes"}
+        STAGING["Staging Collection<br/>(nuclear_rebuild only,<br/>keeps prod serving queries)"]
+        PROD["Production Collection<br/>(default)"]
+        WCOL -->|_staging_target set| STAGING
+        WCOL -->|default| PROD
+    end
+
     subgraph STORE["Store"]
-        CHROMADB[("ChromaDB")]
-        BM25IDX["BM25 Index"]
+        CHROMADB[("ChromaDB<br/>data/chroma_db/")]
+        BM25IDX["BM25 Index<br/>(in-memory,<br/>rebuilt on write)"]
+        FTS5DB[("SQLite FTS5<br/>data/fts5_index.db<br/>(CRUD-synced, ADR-008)")]
     end
 
     FILES --> MD & PDF & OFFICE & CODE
@@ -365,14 +409,20 @@ flowchart LR
     MDSPLIT --> DEDUP
     TXTSPLIT --> DEDUP
     DEDUP --> EMBED
-    EMBED --> STORE
+    EMBED --> DISPATCH
+    STAGING --> CHROMADB
+    PROD --> CHROMADB
+    PROD --> BM25IDX
+    PROD --> FTS5DB
 ```
 
 ### hybrid_alpha Parameter Effect
 
+`hybrid_alpha` weights RRF fusion between semantic and BM25 rankings on the **hybrid pipeline only**. When `search_method="auto"` and the QueryRouter classifies the query as lexical (e.g. `CVE-2021-4034`, `MDR-AD002`, `T1078.001`, file hashes), the FTS5 fast-path fires and `hybrid_alpha` is not consulted — FTS5 uses SQLite's native `bm25()` scoring exclusively. Pass `search_method="hybrid"` to force RRF fusion + rerank on every query if you want deterministic hybrid semantics regardless of the query shape.
+
 ```mermaid
 flowchart LR
-    subgraph ALPHA["hybrid_alpha values"]
+    subgraph ALPHA["hybrid_alpha values (hybrid path only)"]
         A0["0.0<br/>Pure BM25<br/>Instant"]
         A3["0.3 (default)<br/>Keyword-heavy<br/>Fast"]
         A5["0.5<br/>Balanced"]
@@ -381,7 +431,7 @@ flowchart LR
     end
 
     subgraph USE["Best For"]
-        U0["CVEs, tool names<br/>exact matches"]
+        U0["CVEs, tool names<br/>exact matches<br/>(consider FTS5 fast-path)"]
         U3["Technical queries<br/>specific terms"]
         U5["General queries"]
         U7["Conceptual queries<br/>related topics"]
@@ -1252,6 +1302,27 @@ This keeps backward compatibility while allowing concise synonym groups.
 | 0.7 | Semantic-heavy | Conceptual queries, related topics |
 | 1.0 | Pure semantic | "How to..." questions, abstract concepts |
 
+### Search Method (v4.8.2+)
+
+Two paths ship with v4.8.2 — the default `hybrid` path (BM25 + semantic + RRF
++ optional rerank) that has been the sole path since v4.0, and an opt-in
+`fts5` lexical fast-path optimised for exact identifier queries (CVEs, MITRE
+ATT&CK codes, CWEs, file hashes, bug-bounty IDs). Enable via
+`search.lexical_fast_path.enabled: true` in `config.yaml`.
+
+| Method                  | Path              | Latency (typical)     | Best For                                     |
+|-------------------------|-------------------|-----------------------|----------------------------------------------|
+| `auto` **(default)**    | Router decides    | Router adds ~0.1ms    | Mixed workloads — safe default               |
+| `hybrid`                | BM25 + semantic   | 50-150ms (with rerank)| Prose queries, "how does X work", exploration|
+| `fts5`                  | SQLite FTS5 only  | <10ms cold / <2ms hot | CVE / MITRE / CWE / hash lookups             |
+
+The `search_knowledge` MCP tool exposes `search_method: Literal["auto",
+"hybrid", "fts5"] = "auto"` (ADR-006, additive on the LEI 1 contract). The
+default `enabled: false` preserves v4.8.1 behaviour byte-for-byte; the flip
+to `enabled: true` by default is reserved for v4.9.0 pending the CI
+perf-gate adjudication documented in ADR-004 and ADR-009. Full user guide
+in [`docs/features/fts5_fast_path.md`](docs/features/fts5_fast_path.md).
+
 ---
 
 ## Project Structure
@@ -1414,7 +1485,83 @@ Common issues:
 
 ### Unreleased
 
-_(vazio — próximas mudanças vão aqui)_
+### v4.8.4 (2026-08-13) — Patch: security + durability + defensive fixes
+
+**Recommended upgrade for anyone running HTTP/SSE transport with bearer auth on a non-loopback host, or relying on `reindex_documents(force=True)`.** Four independent fixes accumulated since v4.8.3: one previously blocked remote-with-auth deployments (#174), one silently corrupted durable index metadata after a failed rebuild (#173), and two defensive fixes hardened `.ipynb` ingestion (#175) and the nightly chaos suite (#164).
+
+**Fixed:**
+
+- **fix(security)** — create the HTTP MCP app with the configured `server_host`. On the bearer-auth path the app factory was called without a host, so it defaulted to `127.0.0.1` and MCP 2.0.0's localhost-only DNS-rebinding allowlist rejected a configured non-loopback host with `421 Invalid Host header` before auth ran — even though uvicorn was bound to that host. Remote/LAN deployments with auth are now reachable (#174).
+- **fix(reindex)** — restore durable metadata on a failed zero-downtime rebuild. The staging populate path persisted the staging map to `index_metadata.json` before validation ran; when validation failed, rollback restored only the in-memory `_indexed_docs`, leaving the file describing staging while memory held production. After a restart `_load_metadata` trusted the stale file, corrupting skip/update and embedding-dimension decisions. Rollback now re-persists the restored production metadata (#173).
+- **fix(ingestion)** — `_parse_ipynb` handled invalid JSON gracefully but then assumed the parsed notebook was a well-formed dict. A valid-JSON notebook with a null `metadata` or `cells` field, a non-dict cell, a bare list, or a dict cell whose `source` is not `str | list[str]` (a number, a dict, or a list with non-string items) crashed with `AttributeError`/`TypeError`. Guarded the notebook, `metadata`, `cells`, cell, and cell `source` lookups so a parseable-but-malformed notebook is handled the same as any other odd input (#175, thanks @eeshsaxena).
+- **fix(ci)** — the nightly chaos job (`pytest -m chaos`) aborted collection with `ModuleNotFoundError: No module named 'hypothesis'` because `tests/test_query_router.py` imported hypothesis at module top-level while the chaos job installs only `pytest psutil numpy`. Guarded the import with `pytest.importorskip("hypothesis")` so the module is skipped cleanly instead of failing the whole nightly run (#164).
+
+**Docs:**
+
+- **docs(architecture)** — sync the 4 Mermaid diagrams in `## Architecture` (System Overview, Query Processing Flow, Document Ingestion Flow, hybrid_alpha Parameter Effect) with the v4.8.2 FTS5 fast-path (ADR-002/003/006), the v4.8.3 `_write_collection` staging dispatcher (#161), and the current 8-category layout. The diagrams previously described only the pre-v4.8 hybrid pipeline and 4-category storage; they now also show FTS5 dispatch, fallback semantics, FTS5 CRUD sync (ADR-008), and the zero-downtime staging write path. `hybrid_alpha` now clarifies that the fast-path bypasses RRF entirely so the weighting only matters on the hybrid path.
+
+**No breaking changes.** Backwards compat verified by `check_api_surface.py` (zero diff vs baseline). All 13 MCP tool parameter names frozen (verified by `tests/test_backwards_compat.py`).
+
+### v4.8.3 (2026-08-10) — Critical hotfix: nuclear-rebuild + smart-reindex hardening
+
+**Recommended upgrade for any corpus with >1000 chunks or any user relying on `reindex_documents(force=True)` after a config change.** The v4.8.0 nuclear rebuild + v4.8.2 FTS5 fast-path have several latent state-management bugs that only trigger under a real-world combination (large corpus + rebuild + concurrent queries + FTS5 opt-in). Enterprise users with hacktricks-scale corpora (~50k+ chunks) are affected.
+
+Fixes 3 GitHub issues opened by @grishkovei against v4.8.1:
+
+- **Fixes #161** — `_rebuild_via_swap()` no longer rebinds `self.collection` to the empty staging collection during populate. Writes now route through the new `_write_collection` property (returns `self._staging_target` when set, else production). Queries keep reading `self.collection` unchanged. Zero-downtime is now real, not just a docs promise. `nuclear_rebuild(swap=True)` on 50k-chunk corpora previously served empty results for the entire hours-long populate window; queries now hit production until the atomic swap completes.
+- **Fixes #162** — Smart-reindex checkpoint now serializes only doc-ids the current run actually committed (via new `tracking["committed_this_run"]` set), not the full `_indexed_docs.keys()` snapshot. A document that changed on disk before the reindex reached it is no longer silently skipped on resume — its mtime/size delta is properly detected.
+- **Fixes #163** — `reindex_documents(force=True)` now propagates through to `index_all(force=True)`. The v4.8 migration path (`docs/migration-v4.8.0.md`) documented `force=True` as the way to re-embed after prefix/model changes; the hardcoded `force=False` in `reindex_all()` silently skipped every unchanged file. Prefix-only changes no longer leave mixed old/new embedding conventions.
+
+Additional fixes discovered while triaging production issues on a 5889-doc / 75016-chunk corpus:
+
+- **fix(indexing)** — `_index_lock` moved from class-level to instance-level (`__init__`). The class-level `threading.Lock` was silently shared across all `KnowledgeOrchestrator` instances, so a staging orchestrator from a previous `nuclear_rebuild` swap could be garbage-collected while holding the lock, leaving every subsequent `reindex_documents` returning `reindex_already_running` despite `active: false`. Requires daemon restart to recover pre-fix.
+- **fix(chromadb)** — `_ensure_bm25_initialized` and `_iter_chroma_chunks_for_fts5` now batch-read Chroma via `offset+limit=500` instead of `collection.get(limit=count)`. Chromadb 1.x rebuilds an `IN (?, ?, ...)` clause per returned row, so a single `limit=48184` call blew past SQLite's default `max_variable_number=999` with `Internal error: too many SQL variables`. Any user with >999 chunks who ran nuclear rebuild or triggered the FTS5 lazy migration got a silently failed rebuild.
+- **fix(fts5)** — `Fts5LexicalIndex.is_ready()` now re-reads the marker file when the cached `_ready` is False. Previously an orchestrator instantiated during nuclear-swap while the FTS5 migration was still running cached `_ready=False` forever and never re-checked the marker, permanently silencing the fast-path.
+- **fix(fts5)** — `_maybe_start_fts5_migration` cross-checks FTS5 row count vs Chroma count on startup. A stale `complete` marker paired with a near-empty FTS5 index (post-swap orphan cleanup, manual truncate, disk corruption) previously left the fast-path silent forever. The new `_fts5_marker_matches_reality` helper triggers rebuild when FTS5 holds less than 10% of Chroma's row count.
+- **fix(fts5)** — `_format_fts5_results` skips orphan hits (FTS5 has the `chunk_id` but Chroma doesn't — nuclear-rebuild residue or manual-delete race). Previously emitted empty result items with `source=""` that broke reranking and polluted result lists.
+- **fix(indexing, config)** — Added `Fts5LexicalIndex.count()` for the new sanity check.
+
+All 6 core fixes were confirmed via production reproduction on a 75016-chunk corpus + 2052 newly-added catalog documents; the `nuclear_rebuild(full_rebuild=True)` path now completes cleanly and populates FTS5 with 75016 rows on first attempt.
+
+Migration path: `pip install -U knowledge-rag==4.8.3` (or `npx -y knowledge-rag@4.8.3` / `docker pull ghcr.io/lyonzin/knowledge-rag:4.8.3`) then restart the MCP daemon. No config changes required. If your FTS5 index is corrupted from a prior partial rebuild (marker says `complete` but queries return empty), delete `data/fts5_migration.state` and restart — the daemon will detect the drift and rebuild automatically.
+
+### v4.8.2 (2026-08-10) — FTS5 Lexical Fast-Path Opt-In Release (default OFF)
+
+v4.8.2 bundles Fases 1–5 of the FTS5 Lexical Fast-Path plan
+(`.compozy/tasks/fts5-lexical-fast-path/`) as an **opt-in** release —
+`search.lexical_fast_path.enabled` remains `false` by default, so users
+who do not touch `config.yaml` keep v4.8.1 behaviour byte-for-byte
+(LEI 1 preserved). The v4.9.0 default flip is deferred to the CI
+perf-gate adjudication documented in ADR-009 (bench harness ships in
+this release; the flip PR follows once G1 ≥15× and G2 ≤2% both pass on
+CI hardware).
+
+- **DOCS (search)**: new `docs/features/fts5_fast_path.md` user guide
+  covering overview / quick-start / 5-key config reference / how it
+  works / 5 troubleshooting scenarios / when-to-use guidance. README
+  Configuration section cross-links it.
+- **NEW (bench)**: `bench/test_bench_fts5_lexical.py` ships the
+  BENCH-001 (cold, p95 ≤ 10ms target) + BENCH-002 (hot, p95 ≤ 2ms
+  target) harness against a self-contained 3865-row FTS5 corpus. The
+  gate G2 comparison against `bench/test_bench_search.py` is executed
+  by CI per `.compozy/tasks/fts5-lexical-fast-path/bench_v4_9_0_gate.md`;
+  the two-round A/B was not run locally because the release-authoring
+  workstation cannot fairly baseline against CI hardware (ADR-004
+  §Risks).
+- **DOCS (adr)**: new ADR-009 documenting why the ADR-004 gate is
+  deferred to the CI perf-gate step and what conditions unlock the
+  v4.9.0 flip PR.
+- **CHANGE (config)**: `config.example.yaml` uncomments the
+  `search.lexical_fast_path.*` block so new users see the full
+  configuration surface (still `enabled: false`). Existing installs
+  are unaffected.
+- **VERSION**: 4.8.1 → **4.8.2** (PATCH — opt-in infrastructure + docs,
+  zero behaviour change on the default install).
+- **feat (search)**: FTS5 lazy background migration for existing corpus + CRUD sync (ADR-008). `Fts5LexicalIndex` gains `add_document`/`update_document`/`remove_document` (SQL INSERT/DELETE under `_fts5_lock`, incremental — diverges from BM25's full-rebuild pattern because FTS5 mutations are O(1) native SQL) and `start_migration_background(chunk_iter_factory, docs_total, resume_from=0, on_progress=...)` (daemon thread that persists checkpoint every 100 rows into the marker file). On daemon startup, `_maybe_start_fts5_migration` reads the marker: `complete` = fast-path live, `in_progress` = resume from `docs_indexed`, `failed`/missing = start fresh migration; queries during migration fall back to hybrid + emit `fast_path_fallback_total{reason="disabled"}`. `KnowledgeOrchestrator.{add,update,remove}_document_*` gain best-effort FTS5 sync hooks after the BM25 rebuild — errors are caught, logged, and counted (`fast_path_errors_total{error_class="Fts5CrudSyncError"}`) but never propagate to the CRUD tool (upstream write must keep succeeding). `nuclear_rebuild` and `_rebuild_bm25_post_swap` drop the FTS5 db + WAL/SHM/marker and dispatch a fresh migration from the swapped-in Chroma corpus. Two Prometheus gauges expose migration progress (`knowledge_rag_fast_path_migration_docs_indexed`, `_docs_total`). Standalone rebuild helper `scripts/build_fts5_index.py --data-dir <path> [--force] [--foreground] [--verbose]` for operator-driven repair. Operational runbook in `docs/runbooks/fts5_migration.md` covering enable / wait / verify / manual rebuild / large-corpus caveat. New coverage: `tests/test_fts5_migration.py` (17 tests — TestMigrationLifecycle UT-043..048, TestCRUDSync UT-049..052, TestMigrationIntegration IT-021..023, TestCRUDSyncIntegration IT-CRUD-001..004) + `tests/test_e2e_fts5.py::test_e2e006_forced_fts5_with_migration_pending_raises` (E2E-006). Fase 4 of the FTS5 Lexical Fast-Path plan.
+- **NEW (search)**: Optional cross-encoder rerank toggle on the FTS5 fast-path via `search.lexical_fast_path.rerank_enabled` (default `false` — see ADR-003 for rationale). When `true`, `KnowledgeOrchestrator._run_fts5_search` layers `CrossEncoderReranker.rerank` on top of the FTS5 results before adjacent-chunk expansion; result items keep `search_method="fts5"` (rerank is a rescorer, not a path change) and `reranker_score` is populated. When `false`, the `knowledge_rag_fast_path_rerank_skipped_total` counter increments and the fast-path stays under the <10ms lexical latency budget. Empirical basis: Flash-Rerank MiniLM-L-6-v2 CPU ≈72ms/100 docs vs the <10ms target — keeping rerank OFF by default is the correct trade-off; opt in only when recall on lexical queries matters more than latency. New coverage: `tests/test_search.py::TestRerankToggle` (UT-062, UT-063, IT-024, IT-025). Fase 3.5 of the FTS5 Lexical Fast-Path plan.
+- **NEW (search)**: FTS5 lexical fast-path dispatch wired inside `KnowledgeOrchestrator.query()` behind the opt-in `search.lexical_fast_path.enabled` toggle (default `false` — v4.8.1 behavior byte-for-byte for users who don't touch config). When enabled, the query router (Fase 2) classifies each query; lexical queries dispatch to the FTS5 index (Fase 1) via a new `_maybe_dispatch_fts5` helper, semantic queries flow through the existing hybrid pipeline. Per ADR-003 the cross-encoder reranker is skipped on the fast-path (real opt-in toggle deferred to Task 04). Per ADR-006 the `search_knowledge` MCP tool gains one optional param `search_method: Literal["auto","hybrid","fts5"] = "auto"` — additive-safe on the LEI 1 backward-compat contract. Explicit `"fts5"` when the feature is disabled OR the index is not ready raises `Fts5NotReadyError` and the MCP wrapper surfaces it as a JSON error envelope with a `suggestion` field pointing at `search_method='auto'`. `QueryCache._make_key` gains a 5th `search_method` param so paths cannot share cache entries. `MetricsCollector` gains five FTS5 counters plus a bucketed latency histogram (`knowledge_rag_fast_path_hits_total{path}`, `..._fallback_total{reason}`, `..._latency_seconds`, `..._errors_total{error_class}`, `..._rerank_skipped_total`) exposed through `/metrics`. New coverage: `tests/test_search_method_override.py` (5 IT), `tests/test_metrics.py` (2 UT), `tests/test_e2e_fts5.py` (7 E2E), and extensions to `tests/test_search.py` (16 IT/UT — dispatch, config toggle, cache key, metrics scrape), `tests/test_backwards_compat.py` (3 UT — MCP tool signature + guardrail), `tests/test_fts5_index.py` (3 UT — fallback dispatch, busy_timeout, NotReady message). Fase 3 of the FTS5 Lexical Fast-Path plan.
+- **NEW (search)**: Query router lexical/semantic classifier (`mcp_server/query_router.py`) ships isolated and unused. Static regex classifier per ADR-002 — patterns are pre-compiled at construction (fail-fast on `re.error`) and matched in YAML-declared order (first-match-wins, OQ-2). API surface: `QueryRouter(patterns: Sequence[str])` plus `classify(query: str) -> Literal["lexical", "semantic"]`; empty query classifies as `"semantic"` (safe default). Zero coupling to `mcp_server/fts5_index` — the router is storage-agnostic. Task 03 will wire the dispatch inside `KnowledgeOrchestrator.query()` behind `config.fts5_enabled`; this PR is deliberately unwired so runtime behaviour is preserved. New coverage in `tests/test_query_router.py` (19 tests covering UT-001..UT-013 including a `hypothesis` property test). Fase 2 of the FTS5 Lexical Fast-Path plan (`.compozy/tasks/fts5-lexical-fast-path/`).
+- **NEW (search)**: FTS5 lexical index module (`mcp_server/fts5_index.py`) ships isolated and unused behind the opt-in `search.lexical_fast_path.enabled` config field (default `false`). Encapsulates the dedicated `data/fts5_index.db` SQLite FTS5 store with the ADR-005 tokenizer (`unicode61 remove_diacritics 2 tokenchars '-_.'`) so composite identifiers like `MDR-AD002`, `CVE-2021-4034`, `T1078.001`, `pass-the-hash`, `context.py` survive tokenization intact. Public surface: `Fts5LexicalIndex` (search + connection lifecycle + WAL/busy_timeout=5000ms/synchronous=NORMAL PRAGMAs per ADR-001), `Fts5MigrationState` (atomic JSON marker via `tempfile` + `fsync` + `os.replace`), and three `RuntimeError`-derived exceptions (`Fts5NotReadyError`, `Fts5CorruptError`, `Fts5MigrationError`) per ADR-004/Q4. `Config` gains four `fts5_*` fields (`fts5_enabled`, `fts5_min_hits`, `fts5_rerank_enabled`, `fts5_patterns`) backed by a new `_get_nested()` helper that mirrors the `models.reranker.*` pattern. `config.example.yaml` documents the new `search.lexical_fast_path.*` block (commented out). All FTS5 queries use prepared statements and every user-supplied token is quoted so FTS5 metacharacters (`MATCH`, `NEAR`, `OR`, `AND`, `NOT`, `*`, `:`, `(`, `)`, `"`) cannot leak through. Task 03 will wire the dispatch inside `KnowledgeOrchestrator.query()`; this PR is deliberately unwired so runtime behaviour of v4.8.1 is preserved byte-for-byte. New coverage in `tests/test_fts5_index.py` (25 tests covering UT-020..028, UT-054..058, UT-061, IT-MIG-001) and `tests/test_config_fts5.py` (9 tests covering UT-030..038). Fase 1 of the FTS5 Lexical Fast-Path plan (`.compozy/tasks/fts5-lexical-fast-path/`).
 
 ### v4.8.0 (2026-08-06) — Multilingual Foundation + Zero-Downtime Reindex
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "knowledge-rag",
-  "version": "4.8.1",
+  "version": "4.8.4",
   "description": "Local RAG System for Claude Code — Hybrid search + Cross-encoder Reranking + 13 MCP Tools + 20 Format Parsers. Zero external servers.",
   "bin": {
     "knowledge-rag": "./bin/cli.js"