md-semantic-search 1.0.3 → 2.0.0

package/README.md

@@ -100,7 +100,7 @@ measurements that shaped its defaults.
 
 | Approach | Stronger than mdss at | mdss advantage for its target segment |
 |---|---|---|
-| **QMD** | Query expansion, HyDE, LLM reranking, MCP/SDK ecosystem | Smaller default footprint, Node 18, zero native C++ runtime (pure ONNX.js), fast incremental chunk reuse |
+| **QMD** | Query expansion, HyDE, LLM reranking, MCP/SDK ecosystem | Smaller default footprint, Node 20.9+, zero native C++ runtime (pure ONNX.js), fast incremental chunk reuse |
 | **Obsidian Hybrid Search** | Obsidian GUI integration, Obsidian vault plugins | Standalone CLI & HTTP API, not tied to Obsidian, crash-safe atomic indexing, robust incremental re-index |
 | **Vector Databases (Qdrant/Pinecone)** | Multi-million chunk scale, distributed vector index, cloud replication | Zero infra, single JSON index, no server/daemon required, fast in-memory sweep for 10k–50k chunks |
 | **Grep / Fzf** | Zero model load time, instant literal string matching | Cross-lingual matching (RU/EN), paraphrase retrieval, semantic understanding while retaining exact BM25 lexical precision |
@@ -378,7 +378,7 @@ mdss search --db /path/to/your/markdown "how do I rotate the api token" --rerank
 ```
 
 - The first pass pulls a wider candidate pool (default `max(20, k*3)` chunks),
-  the cross-encoder (`Xenova/bge-reranker-base`, ~280 MB, downloaded on first
+  the cross-encoder (`gte-reranker-modernbert`, ~144 MB, downloaded on first
   use) re-scores them as query↔passage pairs, and the best `k` survive.
 - Results from a re-ranked search carry a `rerankScore` field in `--json` mode.
 - Programmatic use: `rerank: true` in `search()`/`searchIndex()`, `rerankPool`
@@ -386,9 +386,57 @@ mdss search --db /path/to/your/markdown "how do I rotate the api token" --rerank
   accepts `"rerank": true` in the `/search` body.
 - The reranker is lazy: nothing loads unless `--rerank` is actually requested,
   so searches without it pay zero extra cost.
+- **Reranker registry** (issue #203): `--rerank-model <alias>` picks the model —
+  `gte-reranker-modernbert` (Alibaba gte-reranker-modernbert-base, ~144 MB,
+  ModernBERT 8k context, Apache-2.0; **default since the #203 measurement** —
+  nDCG@10 0.9318 vs 0.8839) or the pre-#203 incumbent `bge-reranker-base`
+  (~278 MB, kept for reproducibility); raw `id@revision` also accepted.
+  Programmatic: `rerankModel: 'bge-reranker-base'`.
+- **Multi-query expansion** (issue #203): `--multi-query <n>` embeds N query
+  variants (the original + N-1 LLM paraphrases from `--llm-endpoint`, one extra
+  embedding each) and fuses the dense rankings with union-RRF — a recall lift
+  on heavy paraphrases, at the cost of N−1 extra embeddings and one LLM call.
+  Without an endpoint (or on any LLM failure) it silently degrades to the
+  single-query baseline. Programmatic: `multiQuery: 3` (+ `llmEndpoint`).
 
+### 3. Ask & Interactive Multi-turn Chat (`mdss ask`, `mdss chat`)
 
-### 3. Serve (daemon, warm index + model)
+`md-semantic-search` includes a grounded RAG QA engine that operates fully offline by default with zero external dependencies, or connects optionally to local/remote LLMs (Ollama, OpenAI-compatible APIs).
+
+> ⚠️ **Data egress**: a non-loopback `--llm-endpoint` (or `--embedder openai`) sends **note content** to that external service. The CLI prints a one-time warning before the first send; see [SECURITY.md → Data egress](./SECURITY.md#data-egress-issue-183). Loopback endpoints (local Ollama/llama.cpp) and the default extractive/local-embedder paths keep everything on your machine.
+
+```bash
+# One-shot grounded QA with citations:
+mdss ask --db ./notes "What is the token expiration policy?"
+
+# Multi-turn persistent chat session:
+mdss chat --db ./notes
+
+# Resume the most recent session or a specific session:
+mdss chat --db ./notes --resume
+mdss chat --db ./notes --session my-session-id
+
+# One-shot query into a persistent session:
+mdss chat --db ./notes --session my-session-id "What about database failover?"
+```
+
+- **Offline Extractive Synthesis**: Zero-dependency default extracts salient factual sentences with strict `[file › heading]` source provenance.
+- **OpenAI-compatible chat with SSE streaming** (issue #199): with `--llm-endpoint`, synthesis speaks the `/chat/completions` lingua franca — llama.cpp, LM Studio, vLLM, and Ollama (`.../v1`) all work **without code** — and answers **stream token-by-token** into `mdss ask`, `mdss chat`, the TUI RAG panel (`mdss search -i --rag`), and the web UI (`mdss serve` → ✦ Ask): the first token renders in milliseconds instead of after the whole generation.
+  ```bash
+  # llama.cpp / LM Studio / vLLM (OpenAI-compatible):
+  mdss ask --db ./notes "What is the token expiration policy?" \
+    --llm-endpoint http://127.0.0.1:8080/v1 --llm-model qwen2.5
+  # Ollama's OpenAI-compatible surface:
+  mdss chat --db ./notes --llm-endpoint http://127.0.0.1:11434/v1 --resume
+  # Ollama's native API (pre-#199 wire format, also streamed):
+  mdss ask --db ./notes "..." --llm-endpoint http://127.0.0.1:11434 --llm-provider ollama
+  ```
+  An endpoint that does not answer `/chat/completions` (404/405) is retried once on the Ollama-native `/api/generate`, so existing Ollama-root setups keep working with default flags; `--llm-provider` pins the wire format explicitly. A hung endpoint fails on the **idle timeout** (`MDSS_FETCH_TIMEOUT_MS` — armed on connect, reset on every token, issue #184), so a long generation is never killed mid-answer, and any failure degrades to the extractive synthesis.
+- **Persistent Sessions**: Multi-turn sessions are stored in `<indexDir>/sessions/<id>.json` preserving full conversation turns, retrieved chunk provenance manifests (file, heading, exact line numbers, chunk hashes), and model metadata.
+- **Context Carry-Over**: Follow-up turns re-rank and expand query tokens using prior conversational context.
+- **Interactive REPL Commands**: `/session` (info), `/sources` (manifest of last turn), `/help`, `/exit`.
+
+### 4. Serve (daemon, warm index + model)
 
 ```bash
 mdss serve --db /path/to/your/markdown [--port 8747] [--host 127.0.0.1] [--watch]
@@ -421,6 +469,22 @@ curl localhost:8747/health          # → {ok, chunks, model, dim, built, watchi
 score, matches, snippet, rerankScore?}]}`. The CLI's `search --json` prints
 the same hit shape.
 
+**`POST /ask` — streamed grounded QA over SSE** (issue #199): start the daemon
+with `--llm-endpoint` (chat backend — llama.cpp, LM Studio, vLLM, Ollama `.../v1`)
+and `{"query": string, "k"?: number}` answers as `text/event-stream` frames:
+`sources` (citations from retrieval) → `delta` (`{"text": "…"}` per token) →
+`done` (`{answer, mode, citations, citationsUsed}`; `mode` is `endpoint` for the
+LLM answer or `extractive` when it degraded, `citationsUsed` lists the citations
+the answer text actually cites). **Multi-turn chat**: pass `"chat": true` to
+start a persistent session or `"sessionId": "…"` to continue one — the same
+`<indexDir>/sessions/` store `mdss chat` uses; a `session` frame opens the
+stream and the `done` frame returns `sessionId`/`turnsCount`. The built-in web
+UI has a **chat** toggle with history and a "new chat" reset. The same fairness gates as `/search` apply —
+rate limit `429`, concurrency `503` — and an LLM call holds one concurrency
+slot for its whole generation. Without `--llm-endpoint` the route answers
+`400` instead of silently degrading. The built-in web UI ships an
+**✦ Ask the notes** button that consumes this stream.
+
 Or from a script / editor extension:
 
 ```js
@@ -514,16 +578,53 @@ mdss mcp --db /path/to/your/markdown
 
 **Streamable HTTP transport (issue #123):** a long-running daemon can expose
 the same tools over HTTP — `mdss serve --db ./notes --mcp` mounts `/mcp`
-(POST JSON-RPC, SSE channel on GET, `Mcp-Session-Id` sessions, protocol
-`2025-03-26`). It sits behind the exact same auth, Host-allowlist and
-rate-limit gates as `/search`, so remote agents inherit the hardening. stdio
-via `mdss mcp` remains the default transport.
+(POST JSON-RPC, SSE channel on GET, `Mcp-Session-Id` sessions). It sits behind
+the exact same auth, Host-allowlist and rate-limit gates as `/search`, so
+remote agents inherit the hardening. stdio via `mdss mcp` remains the default
+transport.
+
+**Protocol negotiation & `ping` (issue #197):** `initialize` speaks
+`2025-11-25`, `2025-06-18`, `2025-03-26`, and `2024-11-05` — the client's
+requested version is echoed when supported, otherwise the server answers with
+its newest (`2025-11-25`), so modern clients never warn about an outdated
+protocol and older clients keep working. `ping` is answered promptly with an
+empty result on both transports.
+
+**Sampling — LLM answers via the client (issue #197):** when the client
+declares the `sampling` capability at `initialize`, `ask_knowledge_base` hands
+the synthesis to the client's own model with a server→client
+`sampling/createMessage` request — grounded passages go out, a cited answer
+comes back — with **no local LLM and no extra user configuration** (Claude
+Desktop sampling, for example). Over stdio the exchange rides the same
+stdin/stdout JSON-RPC lines; over HTTP it rides the GET `/mcp` SSE stream with
+the answer POSTed back. Sampling is strictly best-effort: a client without the
+capability, a refusal, or a timeout (default 60 s, override with
+`MDSS_SAMPLING_TIMEOUT_MS`) degrades to the built-in extractive answer. The
+tool payload reports which path produced the result in `answerMode`
+(`sampling` / `extractive`).
 
 #### Available MCP Tools:
-- **`search_markdown`**: Hybrid semantic vector + BM25 search with optional `path` glob and `tag` filters.
+- **`search_markdown`**: Hybrid semantic vector + BM25 search with optional `path` glob, `tag`, rich `filter`, and `graphBoost`.
 - **`get_chunk`**: Retrieve a specific section by file and heading.
+- **`get_lines`**: Retrieve exact line spans (`fromLine`, `maxLines`) from an indexed Markdown document.
+- **`related_notes`**: Find graph-connected (outgoing, backlinks, 2-hop) and semantically related notes for a document.
 - **`list_files`**: List indexed Markdown files with chunk counts.
 - **`index_status`**: Inspect index health, chunk counts, and model metadata.
+- **`ask_knowledge_base`**: Grounded direct Q&A synthesis with citations; uses the client's own LLM via MCP sampling when available (see below), else the built-in extractive answerer — `answerMode` in the payload tells which.
+- *All tools return `structuredContent` alongside text and include `readOnly: true, idempotent: true` annotations.*
+
+#### Available MCP Resources & Templates:
+- **`mdss://note/{path}`**: Read full content of an indexed Markdown document.
+- **`mdss://note/{path}?fromLine=1&maxLines=50`**: Addressable line-span slices of notes.
+- **`mdss://vault/{vault}`**: List all indexed notes within a vault.
+- **`mdss://status`**: Live index status and chunk count statistics.
+
+#### Available MCP Prompts:
+- **`search-and-cite`**: Synthesizes a grounded answer referencing `mdss://note/{path}` resource URIs.
+- **`summarize-note`**: Produces a structured summary of core concepts and linked notes.
+- **`compare-notes`**: Identifies agreements, divergences, and synergies across two notes.
+- **`find-contradictions`**: Audits notes for conflicting statements or outdated ADRs.
+- **`timeline`**: Builds a chronological milestone / event timeline from notes.
 
 #### Claude Desktop Configuration:
 Add to your `claude_desktop_config.json`:
@@ -571,12 +672,14 @@ Add to your `claude_desktop_config.json`:
 | `--json` | Machine-readable output: `index` → build result JSON, `stats` → index stats JSON, `search` → hit list JSON (each hit includes `matches` — the query terms found in the chunk). |
 | `--semantic` | `search` only: pure vector ranking, skip lexical/RRF fusion (zero lexical tokenization). |
 | `--open [N]` | `search`: open the top (or Nth) hit in your editor at its `startLine` — `$MDSS_EDITOR`/`$VISUAL`/`$EDITOR`, VS Code `--goto`, or the GUI opener. With `--json` on a non-TTY prints the intent instead of launching (issue #110). |
-| `--fix` | `check`: apply safe auto-repairs — stale `.mdss.lock`, SHA-mismatched `vectors.bin` sidecar, corrupt `ivf.json`; rebuilds a corrupt index / recomputes `.hashes.json` when `--db` is available. Never touches source notes (issue #117). |
+| `--fix` | `check`: apply safe auto-repairs — stale `.mdss.lock`, SHA-mismatched `vectors.bin` sidecar, corrupt `ivf.json`/`hnsw.json`; rebuilds a corrupt index / recomputes `.hashes.json` when `--db` is available. Never touches source notes (issue #117). |
 | `--dry-run` | `check --fix`: print the planned repairs without mutating anything. |
-| `--rerank` | Re-rank the candidate pool with a cross-encoder (`Xenova/bge-reranker-base`, ~280 MB model, downloaded on first use). Slower, sharper results — see *Reranking* below. |
+| `--rerank` | Re-rank the candidate pool with a cross-encoder (~144 MB default model, downloaded on first use). Slower, sharper results — see *Reranking* below. |
+| `--rerank-model <m>` | Reranker registry alias (issue #203): `bge-reranker-base` (default) or `gte-reranker-modernbert` (144 MB, 8k ctx, Apache-2.0); raw `id@revision` accepted. |
+| `--multi-query <n>` | Multi-query expansion (issue #203): N query variants (original + N−1 LLM paraphrases) fused by union-RRF; needs `--llm-endpoint`, degrades silently without it. |
 | `--graph-boost <n>` | Weight for Obsidian wikilink graph ranking and PageRank prior (search; default 0). Incorporates document centrality and 2-hop contextual propagation. |
 | `--filter <expr>` | Rich boolean filter across frontmatter tags and properties (e.g. `tag:engineering AND status != archived`). |
-| `--ann` | Approximate nearest neighbor search via IVF clustering (search; auto-enables for large corpora). |
+| `--ann [ivf\|hnsw]` | Approximate nearest neighbor search (search; auto-enables for large corpora). Engines (issue #201): `ivf` (k-means, the default sidecar) or `hnsw` (graph — no training phase, 100% measured recall at equal-or-lower query latency, `hnsw.json` sidecar, corrupt → brute-force fallback). Bare `--ann` prefers an existing sidecar. |
 | `--nprobe <n>` | Number of nearest centroid clusters to probe during ANN search (default 8). |
 | `--port <n>` | HTTP port for `serve` (default 8747, or `MDSS_PORT`). |
 | `--host <ip>` | Bind address for `serve` (default `127.0.0.1` — loopback only; use `0.0.0.0` to expose on the LAN, or `MDSS_HOST`). |
@@ -590,7 +693,7 @@ Add to your `claude_desktop_config.json`:
 | `--watch` | `serve`: re-index incrementally when files change. |
 | `--watch-debug` | `serve --watch`: verbose trace of polls, FS-error classifications and retries (issue #116). |
 | `--recency <days>` | `search`: time-decay half-life — passages get a `0.5^(age/halfLife)` boost after fusion; age from frontmatter `created`/`updated`, then file mtime; surfaced in `explain` (issue #127). |
-| `--embedder <name>` | Use an external embedding provider (`ollama` or `openai`) instead of local transformers.js, together with `--embedder-model` (+ `--embedder-base-url`, `--embedder-key-file`). The provider id/model/dim are recorded in the index — switching providers invalidates vectors with the usual full re-embed. Local remains the zero-config default and the only fully-offline path (issue #124). |
+| `--embedder <name>` | Use an external embedding provider (`ollama` or `openai`) instead of local transformers.js, together with `--embedder-model` (+ `--embedder-base-url`, `--embedder-key-file`). The provider id/model/dim are recorded in the index — switching providers invalidates vectors with the usual full re-embed. Local remains the zero-config default and the only fully-offline path (issue #124). ⚠️ A non-loopback endpoint receives note content as embedding input — see [SECURITY.md → Data egress](./SECURITY.md#data-egress-issue-183). |
 | `--no-query-cache` | `search`: skip the persistent query-embedding cache — repeated queries in fresh processes re-embed otherwise (issue #114). |
 | `--no-ui` | `serve`: disable the built-in web UI at `/` — restores JSON API help there (issue #111). |
 | `--watch-interval <ms>` | `serve --watch`: poll every N ms (default 3000). |
@@ -687,6 +790,8 @@ mdss models
 | `e5-large` | `Xenova/multilingual-e5-large` | 1024 | ~2.2 GB, higher quality. |
 | `bge-m3` | `Xenova/bge-m3` | 1024 | ~2.3 GB. Best cross-lingual separation in tests. |
 | `qwen3-embedding-0.6b` | `onnx-community/Qwen3-Embedding-0.6B-ONNX` | 1024 | ~613 MB q8. Opt-in; instruction-prefixed queries and last-token pooling. |
+| `granite-multilingual-311m` | `onnx-community/granite-embedding-311m-multilingual-r2-ONNX` | 768 | ~298 MB q8. IBM Granite r2 — ModernBERT 8k context, CLS pooling, Apache-2.0 (issue #200). |
+| `embeddinggemma-300m` | `onnx-community/embeddinggemma-300m-ONNX` | 768 | ~300 MB q8. Google EmbeddingGemma — pre-pooled output, MRL dims 64–768, Gemma license (issue #200). |
 
 Switching models invalidates the stored vectors automatically — the next
 `index` run does a full rebuild.
@@ -709,12 +814,16 @@ The adapter declares exactly which embedding semantics it needs — the shared
 `embed()` path never guesses from the model name:
 
 - `queryPrefix` / `passagePrefix` — text prepended to queries/passages (E5 uses
-  `query: `/`passage: `, BGE none, Qwen3 a retrieval instruction on queries);
-- `pooling` — `'mean'`, `'last_token'`, or `'none'`;
+  `query: `/`passage: `, BGE/Granite none, Qwen3 a retrieval instruction on
+  queries, EmbeddingGemma its task/title prefixes);
+- `pooling` — `'mean'`, `'last_token'`, `'cls'`, or `'none'`;
 - `normalize` — whether vectors are L2-normalized;
 - `nativeDim` / `dim` — embedding dimension (and optional `dimensions` for MRL);
 - `dtype` — the ONNX runtime data type loaded at runtime (default `q8`);
 - `maxTokens` — the tokenizer context budget (Qwen3-0.6B is 32768);
+- `pooledOutput` — the ONNX graph already emits a pooled `sentence_embedding`
+  (EmbeddingGemma); embedding reads that output directly instead of the
+  feature-extraction pipeline;
 - `family` — an informational formatting/pooling label surfaced by `stats`.
 
 `mdss models` prints each adapter's pooling, normalization, dimension, and
@@ -760,7 +869,12 @@ section instead of silently producing wrong vectors.
 1. **Walk** `--db` recursively for `.md`/`.markdown` (dotfiles & `--ignore`
    globs skipped).
 2. **Chunk** each file by Markdown headings while retaining the active heading
-   stack; oversized sections split on blank lines (~1400 body chars/chunk).
+   stack; oversized sections split on blank lines. The budget is in **tokens**
+   (issue #202): `min(model maxTokens, 512)` by default with a script-
+   calibrated estimator, so Cyrillic notes get the same real budget as English
+   ones (the legacy English-calibrated `~1400 chars` budget overflowed the
+   model's context for RU ~2×). Override with `--max-chunk-tokens`, or set `0`
+   to restore the legacy char budget.
 3. **Embed** the document title, full heading path, and chunk body (`passage:`
    prefix for E5) → store `{file, title, heading, headingPath, text, vec}` in
    `vectors.json`, plus per-file md5 in `.hashes.json`. The index stores an
@@ -796,7 +910,9 @@ Contributions are welcome — bug reports with reproduction steps, docs fixes,
 tests, and features all help. Start with
 [CONTRIBUTING.md](./CONTRIBUTING.md) (dev setup, project tour, conventions)
 and look for issues labeled **`good first issue`** — they are scoped for a
-first-time contributor. By participating you agree to the
+first-time contributor. For the internals — module map, data-flow diagrams,
+on-disk formats, and extension points — see
+[ARCHITECTURE.md](./ARCHITECTURE.md). By participating you agree to the
 [Code of Conduct](./CODE_OF_CONDUCT.md).
 
 ## License

package/RESEARCH.md

@@ -155,6 +155,13 @@ Baseline on dev (e5-base, hybrid RRF, k=10, 2026-08-14):
 | Hit@10 | 0.9792 |
 | Recall@10 | 1.0000 |
 
+The **committed regression baseline** (issue #198) lives in
+`bench/fixtures/golden-baseline.json` (2026-10-28: nDCG@10 0.8856, Recall 1.0 —
+re-measured against the current ranking stack; the 2026-08-14 numbers above
+predate later fusion/chunking changes). The nightly workflow gates against it
+(`nDCG@10` drop ≤ 0.01, `Recall@10` = 1.0); see *Retrieval quality gate* in
+CONTRIBUTING.md for the local `--gate` / `--update-baseline` workflow.
+
 ## 8. Reproducing
 
 ```bash
@@ -226,7 +233,204 @@ If no candidate meets the multilingual non-inferiority gate, `bge-reranker-base`
 - **Trigger:** WebAssembly SIMD dot-product kernels will be considered **only if** CPU profiling on >50,000 chunks shows JS float multiplication accounts for >40% of warm search latency.
 - **Constraint:** Zero native Node C++ bindings; must maintain pure JS fallback for non-WASM environments.
 
-### ANN / IVF Scale Threshold (Issue #32)
+### ANN / IVF Scale Threshold & K-Means++ Seeding (Issues #32 & #139)
 - **Baseline:** Exact $O(N \cdot d)$ linear sweep.
-- **Trigger:** ANN (Inverted File / HNSW) indexing is a scale contingency. It shall **not** be introduced until corpus size exceeds **100,000+ chunks** and exact warm sweep latency exceeds 50 ms.
-- **Requirement:** Exact pre/post-filter correctness must be preserved; small knowledge bases (<100k chunks) will always use exact linear search.
+- **K-Means++ Seeding:** Uses deterministic pseudo-random $D^2$-weighted centroid seeding (Mulberry32) to prevent initial centroid collapse on correlated notes corpora.
+- **Measured Quality Gate:** `bench/ann-recall.mjs` benchmarks Recall@10 across $nprobe \in \{1, 4, 8, 16\}$ on clustered corpora, demonstrating $\ge 99.5\%$ Recall@10 at default $nprobe=8$ with sub-millisecond candidate pruning.
+- **Trigger:** ANN indexing is auto-engaged above `ANN_THRESHOLD = 500` chunks when `--ann` is passed or IVF index is persisted, while exact pre/post-filter correctness is strictly preserved.
+
+---
+
+## 14. Matryoshka Representation Learning (MRL) Dimension Selection (Issue #143)
+
+### Mechanism
+Models trained with Matryoshka Representation Learning (MRL) loss (such as `onnx-community/Qwen3-Embedding-0.6B-ONNX` and `Xenova/bge-m3`) embed semantic information hierarchically, allowing vectors to be sliced at predefined sub-dimensions (`32`, `64`, `128`, `256`, `512`, `768`, `1024`) and re-normalized to unit L2 norm without significant loss in semantic retrieval fidelity.
+
+### Benefits & Trade-offs
+- **Storage & RAM Savings:** 1024 → 256 dimensions reduces `vectors.bin` size and memory consumption by 4× (from ~19.5 MB to ~4.88 MB per 5,000 chunks).
+- **Sweep Latency:** Dot-product cosine sweep latency is halved (e.g. from ~14.2 ms to ~6.8 ms for 5,000 chunks in single-threaded pure JS).
+- **Integrity & Cache Invalidation:** `embeddingAdapterFingerprint` binds the selected dimension (`model.dim`). Changing `--dimensions` automatically invalidates previous chunk hashes and forces clean re-indexing rather than silent vector dimension mismatch.
+
+### Usage
+```bash
+# Build index truncated to 256 dims (4x smaller vectors)
+mdss index --db ./notes --model qwen3-embedding-0.6b --dimensions 256
+
+# Search with dimension safety validation
+mdss search --db ./notes --dimensions 256 "query"
+```
+
+---
+
+## 15. Research Evaluation: Modern Embedding Adapter Candidates (Issue #200)
+
+### Context & Decision Gate
+
+The registry default (`e5-base`, 2022) predates the 2024-2025 multilingual
+embedding generation. Issue #200 asked for 2-3 measured candidates, each a
+registry entry with a pinned revision + q8 ONNX artifact, evaluated on the
+golden set (#56) before any default change.
+
+**Candidate screening (Hugging Face artifact probe, 2026-11):**
+
+| Candidate | Verdict |
+|---|---|
+| `Xenova/gte-multilingual-base-1` | **Rejected** — no public transformers.js artifact (gte-multilingual needs remote-code modeling, unsupported by the JS runtime) |
+| `Xenova/arctic-embed-m` | **Rejected** — no ONNX conversion published |
+| `Xenova/jina-embeddings-v3` | **Rejected** — no transformers.js-compatible artifact; task-LoRA switch is not expressible through the feature-extraction pipeline |
+| `onnx-community/Qwen3-Embedding-4B-ONNX` | **Rejected** — fp32-only (16 GB); no quantized export; unusable for a CLI embedder |
+| `onnx-community/granite-embedding-311m-multilingual-r2-ONNX` | **Accepted** — q8 298 MB, Apache-2.0, ModernBERT (8k ctx), CLS pooling, no prefixes |
+| `onnx-community/embeddinggemma-300m-ONNX` | **Accepted** — q8 ~300 MB, pre-pooled `sentence_embedding` graph output (needed a new `pooledOutput` adapter path in `core.embed`), MRL 64–768, Gemma license |
+
+### Golden-Set Results (frozen corpus, hash `dev-golden.json`)
+
+| Model (alias) | q8 size | nDCG@10 (dev, n=48) | MRR | Hit@10 | Recall@10 | nDCG (test) | nDCG (holdout) |
+|---|---|---|---|---|---|---|---|
+| `e5-base` (baseline) | ~280 MB | 0.8797 | 0.8628 | 0.979 | 1.000 | 1.0000 | 0.9077 |
+| `granite-multilingual-311m` | ~298 MB | 0.8761 | 0.8559 | 0.979 | 1.000 | 1.0000 | 0.9077 |
+| `embeddinggemma-300m` | ~300 MB | **0.9002** | **0.8837** | **1.000** | 1.000 | 1.0000 | 0.9077 |
+
+- **Non-inferiority (#198 tolerance: drop ≤ 0.01):** granite passes (−0.0036);
+  embeddinggemma **exceeds** the baseline (+0.0205) and is the only candidate
+  that recovers the two dev queries e5-base misses (Hit 1.000).
+- **Discriminating slice is `dev`** — `test` and `holdout` are saturated for
+  both models, so they cannot rank the candidates further.
+
+### Decision
+
+1. **Both candidates ship as opt-in registry entries** with pinned revisions
+   (issue #200 task 1) and are documented in the README Models table.
+2. **The default stays `e5-base`.** EmbeddingGemma's superiority is proven on
+   our golden set, but the default switch is gated on two factors outside this
+   measurement: (a) the **Gemma license** — not MIT/Apache, so making it the
+   npm-distributed default imposes Google's additional terms on every user,
+   which is a project-licensing decision, not an engineering one; (b) one
+   fixture (48 dev queries) is a thin base for overturning a default that the
+   #198 gate has been calibrated on. Re-evaluate if/when the golden set grows.
+   **Until then `embeddinggemma-300m` is the recommended opt-in** (`--model
+   embeddinggemma-300m`) — same size class as e5-base, better measured quality.
+3. **Infrastructure added:** `pooling: 'cls'` (granite) and the
+   `pooledOutput` adapter flag (pre-pooled ONNX graphs bypass the
+   feature-extraction pipeline) — both fingerprint-relevant and unit-tested.
+
+### Reproducing
+
+```bash
+node scripts/run-bench.mjs --model e5-base --slice dev
+node scripts/run-bench.mjs --model granite-multilingual-311m --slice dev
+node scripts/run-bench.mjs --model embeddinggemma-300m --slice dev
+```
+
+---
+
+## 16. Reranker Registry & Multi-Query Expansion (Issue #203)
+
+### Candidate screening
+
+The issue's proposed `bge-reranker-v2-m3` has **no transformers.js artifact**
+(`Xenova/bge-reranker-v2-m3` and `onnx-community/bge-reranker-v2-m3` do not
+exist — 401 on every resolve). The screened replacement:
+
+| Candidate | Verdict |
+|---|---|
+| `BAAI/bge-reranker-v2-m3` | **Rejected** — no JS-runtime ONNX conversion |
+| `Alibaba-NLP/gte-reranker-modernbert-base` | **Accepted** — ONNX exports in-repo, q8 144 MB, Apache-2.0, ModernBERT 8k ctx, single-logit head (same extraction contract as bge-reranker-base) |
+| `Xenova/bge-reranker-large` | available but unmeasured (~2× base size; superseded by gte-modernbert's result) |
+
+### Golden-Set Results (e5-base dense retrieval, frozen corpus, dev slice n=48)
+
+| Config | nDCG@10 | MRR | Hit@10 | Recall@10 |
+|---|---|---|---|---|
+| no rerank (baseline) | 0.8797 | 0.8628 | 0.979 | 1.000 |
+| `--rerank` (bge-reranker-base, ~278 MB) | 0.8839 | 0.8646 | 1.000 | 1.000 |
+| `--rerank --rerank-model gte-reranker-modernbert` (~144 MB) | **0.9318** | **0.9253** | 1.000 | 1.000 |
+
+**gte-reranker-modernbert passes the non-inferiority gate with a decisive
+margin: +0.052 nDCG@10 over the incumbent reranker, at half the size, under
+Apache-2.0.** Unlike the embedding default (#200), no license blocker applies —
+**the default reranker switches to `gte-reranker-modernbert`**; the incumbent
+stays in the registry as `--rerank-model bge-reranker-base` for reproducibility.
+
+### Multi-query expansion
+
+`--multi-query N` (N = total variants: original + N−1 LLM paraphrases, one
+extra embedding each) fuses per-variant dense rankings by union-RRF; ANN
+probing unions the per-variant candidate sets; the `cos` field stays anchored
+to the original query. Quality of the lift depends entirely on the LLM's
+paraphrases, so the pinned contract is the MECHANISM (unit tests with a fake
+LLM): variants embed through the shared query cache, the union never drops
+baseline hits, and — the acceptance criterion — **a paraphrase-only hit enters
+the top-k with the flag on and is absent without it**, while `--multi-query`
+without an endpoint (or on any LLM failure) is byte-identical to the
+single-query baseline. Production lift numbers require a real local LLM and
+are deliberately not claimed from a rigged fixture.
+
+### Reproducing
+
+```bash
+node scripts/run-bench.mjs --rerank
+node scripts/run-bench.mjs --rerank --rerank-model gte-reranker-modernbert
+```
+
+---
+
+## 17. HNSW Graph ANN alongside IVF (Issue #201)
+
+### Implementation
+
+`src/hnsw.ts` — pure JS/TS HNSW (Malkov & Yashunin 2016) sharing the IVF
+contract: build → serialize → `hnsw.json` → search as a candidate pre-filter.
+Key properties:
+
+- **No training phase.** The graph builds by incremental insertion
+  (`insertHnsw`), which is also the watch-rebuild path: tombstoned deletes
+  (`deleteHnsw`) keep removed chunks searchable-through-but-hidden, so a
+  persisted graph can grow without a full rebuild.
+- **Diversity heuristic pruning** (Algorithm 4). The first implementation used
+  blind top-M pruning, which severed the just-added edge at overflowing
+  neighbors — on the 5000-vector clustered corpus recall collapsed to 44.4%
+  and did NOT respond to `ef` (stranded regions). The heuristic restores
+  connectivity: 100% recall at every ef on both bench corpora.
+- **Determinism**: level assignment is seeded per node (`seed + idx`), so
+  full rebuilds from the same insertion order are identical.
+- **Hardened `deserializeHnsw`** with the same discipline as
+  `deserializeIVF` (format, header, per-level link caps, self/duplicate
+  links, entry-point-on-top-layer, tombstone range, count-vs-chunks).
+
+### Measured (bench/ann-recall.mjs, clustered corpus, dim 256)
+
+| n | Engine | Build | Config | Recall@10 | p50 latency |
+|---|---|---|---|---|---|
+| 1000 | IVF | 85 ms | nprobe 4 | 100% | 0.098 ms |
+| 1000 | HNSW | 2605 ms | ef 16 | 100% | 0.212 ms |
+| 5000 | IVF | 1025 ms | nprobe 4 | 100% | 0.252 ms |
+| 5000 | HNSW | 20027 ms | ef 16 | 100% | 0.215 ms |
+
+**Query-side: HNSW ≥ IVF** — 100% recall at equal-or-lower p50 latency
+(satisfying the acceptance criterion's first branch). **Build-side: IVF wins
+~20×** (no training vs O(n log n) graph construction in JS), and the sidecar
+is rebuilt by every full index run — so the automatic sidecar for large
+corpora keeps **IVF as the default engine**; the criterion's second branch
+(documented result, IVF remains default) covers the build-time trade-off.
+
+### Decision
+
+1. `--ann hnsw` opts into the graph engine end-to-end (indexer sidecar +
+   search probe + `check --fix` corrupt-sidecar repair); corrupt `hnsw.json`
+   degrades to brute force silently, like `ivf.json`.
+2. Bare `--ann` and the automatic threshold path stay on IVF until either
+   (a) the JS graph build gets an order of magnitude cheaper (worker
+   parallelism, WASM SIMD — see §13), or (b) incremental watch rebuilds
+   start reusing the persisted graph (the `insertHnsw`/`deleteHnsw` API is
+   already in place for that).
+3. `--ef <n>` tunes the HNSW search breadth (default 64; recall saturates
+   at 100% already at ef 16 on the bench corpora — the default leaves
+   headroom for noisier real-world distributions).
+
+### Reproducing
+
+```bash
+node bench/ann-recall.mjs                 # IVF + HNSW curves side by side
+mdss index --db ./notes --ann hnsw        # hnsw.json sidecar
+mdss search --db ./notes --ann hnsw "query"
+```