md-semantic-search 0.3.0 → 0.4.0

package/README.md

@@ -34,8 +34,11 @@ measurements that shaped its defaults.
 - 🔌 **Any folder, anywhere.** Point `--db` at any directory of `.md`/`.markdown`
   files. It does **not** have to live inside this project. Recursive by default.
 - 🌍 **Cross-lingual.** Multilingual embeddings (default `multilingual-e5-base`).
+- 🧭 **Markdown-aware context.** Each embedding includes the full active heading
+  path (`title > h1 > h2 > ...`), so a deeply nested passage remains searchable
+  by its parent topics without an LLM-generated context step.
 - 🧠 **Hybrid ranking.** Reciprocal Rank Fusion of vector similarity (meaning)
-  and lexical overlap (exact names like `win32`, `TextIOWrapper`).
+  and persisted BM25 postings (exact names like `win32`, `TextIOWrapper`).
 - ⚡ **Two-level incremental.** Per-file md5 (unchanged files skip work) **plus**
   per-chunk hash — inside a *changed* file, sections whose text is unchanged
   reuse their stored vector. Editing one place in a long file re-embeds only the
@@ -44,11 +47,65 @@ measurements that shaped its defaults.
   uploaded anywhere.
 - 📦 **Zero infra.** One JSON index, brute-force cosine in memory. No Pinecone,
   no Qdrant, no pgvector. Scales fine to thousands of chunks.
+- 🗜️ **Compact index.** Vectors are stored as base64 `Float32Array` blobs
+  instead of decimal JSON — ~3.2× smaller index files and ~12× faster loads
+  on a real corpus. Old decimal indexes are read transparently and migrated
+  on the next `index` run.
+- 🖥️ **Daemon mode.** `mdss serve` keeps the parsed index AND the embedding
+  model warm in memory across queries (no ~280 MB reload per search), with an
+  optional `--watch` flag that re-indexes incrementally when your notes change.
+- 🔏 **Crash-safe concurrent writes.** Every index build takes a
+  `<index>/.mdss.lock` (self-healing by pid-liveness), so `mdss index` in one
+  terminal plus `mdss serve --watch` never interleave into a torn index. Long
+  builds also checkpoint embedding progress and resume after interruption.
 
-## Requirements
+---
+
+## Architecture & Retrieval Pipeline
+
+`md-semantic-search` implements a fully local, deterministic Markdown-native retrieval pipeline designed for small-to-medium RU/EN engineering knowledge bases.
+
+```
+┌────────────────┐     ┌───────────────────────┐     ┌──────────────────────┐
+│ Markdown Notes │ ──> │ Structural AST Chunker│ ──> │ Typed Frontmatter &  │
+│  (.md files)   │     │ (src/markdown-parser) │     │ Canonical Identity   │
+└────────────────┘     └───────────────────────┘     └──────────────────────┘
+                                                                │
+     ┌──────────────────────────────────────────────────────────┘
+     ▼
+┌──────────────────────┐     ┌──────────────────────┐     ┌──────────────────────┐
+│ Contextual Embeddings│     │ Field BM25F Postings │     │ Bounded Wikilink /   │
+│   (src/core.mjs)     │     │  (src/lexical.mjs)   │     │ Backlink Graph       │
+└──────────────────────┘     └──────────────────────┘     └──────────────────────┘
+            │                            │                            │
+            └────────────────────┬───────┴────────────────────────────┘
+                                 ▼
+                     ┌──────────────────────┐
+                     │ Reciprocal Rank (RRF)│
+                     │  + Result Collapse   │
+                     │  (src/search.mjs)    │
+                     └──────────────────────┘
+```
+
+1. **Structural AST Parsing** ([`src/markdown-parser.mjs`](./src/markdown-parser.mjs)): Markdown is parsed into structural blocks (headings, fenced code blocks, tables, blockquotes, lists). Block boundaries are preserved atomically.
+2. **Typed Frontmatter & Canonical Identity** ([`src/frontmatter.mjs`](./src/frontmatter.mjs)): YAML frontmatter is parsed into typed metadata (`title`, `aliases`, `tags`, `project`, `type`, `status`, `canonical`, `canonicalRef`).
+3. **Contextual Embeddings** ([`src/core.mjs`](./src/core.mjs)): Passages combine title, full heading hierarchy (`Title > H1 > H2`), and body text, following Anthropic's Contextual Retrieval design to eliminate lost context without an LLM expansion step.
+4. **Field-Aware BM25F & Fuzzy Lookup** ([`src/lexical.mjs`](./src/lexical.mjs)): Inverted lexical postings score `title`, `aliases`, `headingPath`, and `body` with dedicated field weights, complemented by bounded Damerau-Levenshtein fuzzy title matching.
+5. **Relationship Graph** ([`src/wikilinks.mjs`](./src/wikilinks.mjs)): Wikilinks (`[[note]]`) and relative Markdown links resolve to canonical target notes to compute outgoing links and backlinks.
+6. **Reciprocal Rank Fusion & Collapse** ([`src/search.mjs`](./src/search.mjs), [`src/collapse.mjs`](./src/collapse.mjs)): Fuses dense cosine similarity, BM25F lexical scoring, and fuzzy candidates with RRF, followed by optional document collapse (`--max-per-file`) and cross-encoder reranking ([`src/rerank.mjs`](./src/rerank.mjs)).
 
-- Node.js ≥ 18
-- ~280 MB disk for the default model (downloaded once into a cache dir)
+---
+
+## Comparison with Alternatives
+
+| Approach | Stronger than mdss at | mdss advantage for its target segment |
+|---|---|---|
+| **QMD** | Query expansion, HyDE, LLM reranking, MCP/SDK ecosystem | Smaller default footprint, Node 18, zero native C++ runtime (pure ONNX.js), fast incremental chunk reuse |
+| **Obsidian Hybrid Search** | Obsidian GUI integration, Obsidian vault plugins | Standalone CLI & HTTP API, not tied to Obsidian, crash-safe atomic indexing, robust incremental re-index |
+| **Vector Databases (Qdrant/Pinecone)** | Multi-million chunk scale, distributed vector index, cloud replication | Zero infra, single JSON index, no server/daemon required, fast in-memory sweep for 10k–50k chunks |
+| **Grep / Fzf** | Zero model load time, instant literal string matching | Cross-lingual matching (RU/EN), paraphrase retrieval, semantic understanding while retaining exact BM25 lexical precision |
+
+---
 
 ## Install
 
@@ -74,6 +131,44 @@ npm install
 node bin/cli.mjs --help
 ```
 
+---
+
+## Library usage
+
+The package exports a stable programmatic API (no CLI required) — for scripts,
+agents, and editors that want to run many queries in one process.
+
+```js
+import { buildIndex, search, loadIndex, searchIndex, resolveModel, MODELS } from 'md-semantic-search';
+
+// 1. Build (or refresh) an index — same as `mdss index --db ./docs`.
+await buildIndex({ db: './docs', indexDir: './.mdss', cacheDir: '~/.cache/mdss' });
+
+// 2a. One-shot search — parses the index on every call (same as `mdss search`).
+const hits = await search({
+  indexDir: './.mdss', cacheDir: '~/.cache/mdss',
+  query: 'how do I rotate the API token', k: 6,
+});
+for (const h of hits) console.log(h.file, h.heading, h.cosine);
+
+// 2b. Repeated queries in one process — parse the index ONCE and reuse it.
+//     The embedding model is cached in-process too (issue #2).
+const idx = loadIndex('./.mdss');                          // parse once
+const r1 = await searchIndex({ loaded: idx, cacheDir: '~/.cache/mdss', query: 'failover runbook' });
+const r2 = await searchIndex({ loaded: idx, cacheDir: '~/.cache/mdss', query: 'backup schedule' });
+//                                                          ^^ reuses parsed chunks + warm model
+```
+
+The extractor pipeline is cached per model id (see `getExtractor`), so the
+second and later queries skip both the file read and the model load.
+
+Also exported: `resolveModel`, `MODELS`, `DEFAULT_MODEL`, `chunkHash`,
+`tokenize`, `keywordScores`, `rrf`, `cosine`, `encodeVec`/`decodeVec`,
+`walkMarkdown`/`parseFile`/`chunkMarkdown`, `splitFrontmatter`, `extractTitle`,
+`globToRegExp`. Types come from JSDoc (`// @ts-check` + `checkJs`).
+`keywordScores` remains the legacy exact-overlap helper for schema-v0/v1/v2
+indexes and existing library consumers; schema-v3 search uses persisted BM25.
+
 ## Usage
 
 ### 1. Build the index
@@ -93,6 +188,107 @@ just the new entry). The index output reports the split, e.g.:
 Indexed 64 file(s) → 725 chunks (725 reused [5 chunk-level, 720 file-level], 0 embedded), ... 0.6s
 ```
 
+Schema-v3 indexes persist each chunk's full Markdown heading path plus a compact
+BM25 inverted index inside `vectors.json`. Search tokenizes only the query;
+unchanged files and a model-independent hash of the exact lexical document
+(`title + ancestor headings + leaf heading + text`) reuse lexical analysis
+independently of vector reuse. Parent heading changes reanalyze only the affected
+lexical subtree, while model-only changes reuse unchanged lexical records.
+New indexes write `bm25-v2`; existing `bm25-v1` indexes remain searchable and
+upgrade on the next index build. Schema-v0/v1/v2 remain searchable through the
+legacy exact-token overlap lane. A v2 rebuild reuses its contextual vectors
+while adding BM25; v0/v1 still re-embed once because they lack contextual
+passages.
+
+During long builds, mdss atomically checkpoints progress every eight embedding
+batches (about 256 chunks) to `<index>/.checkpoint.json`. The canonical
+`vectors.json` and `.hashes.json` remain the last complete searchable generation;
+after an interruption, the next build resumes from the compatible checkpoint.
+The sidecar is removed after the completed generation is published. This is
+automatic and does not change the CLI or canonical index format.
+
+For scripts/CI, `mdss index --json` emits the build result as JSON (the exact
+`buildIndex` return value: `files`, `chunks`, `reused`, `embedded`, `dim`,
+`model`, `vectorsPath`) instead of the prose — e.g. to assert "0 embedded"
+(fully incremental re-index) in a cron job.
+
+### Inspect the index (`mdss stats`)
+
+`mdss stats` prints index statistics by parsing only `vectors.json` +
+`.hashes.json` — **no model load, no network**, so it's a cheap sanity check
+after a re-index or for staleness detection in scripts:
+
+```bash
+mdss stats --db /path/to/your/markdown
+# Index at /path/to/your/markdown/.mdss
+#   format: binary-v1 · model: e5-base (dim 768)
+#   schema: v3 · lexical: bm25-v2
+#   chunks: 725 · files: 64
+#   size: 42.1 KiB (vectors.json)
+#   built: 2026-08-08T09:42:58.331Z (2m 12s ago)
+#   db: /path/to/your/markdown
+```
+
+`mdss stats --json` prints machine-readable JSON: `indexDir`, `schemaVersion`,
+`format` (`binary-v1` vs legacy `decimal`), `lexicalFormat`, `lexicalStatus`,
+`model` (id@revision) / `modelAlias`, `dim`, `chunks`, `files`, `indexBytes`,
+`built`, `ageSeconds`, `db`. Stats validates the same schema-v3 envelope and
+stored vectors as the loader; malformed current metadata, unknown formats, and
+corrupt vector storage exit with an actionable error instead of reporting a
+healthy index.
+
+Indexes also carry an explicit `schemaVersion` (issue #39): reading or
+re-indexing over an index written by a **newer** mdss fails with a clear
+"upgrade md-semantic-search" error instead of silently misparsing, and corrupt
+root/schema/chunk shapes, unknown current formats, malformed v3 lexical data,
+and corrupt vectors (non-canonical/truncated base64, NaN/Infinity, wrong or
+model-inconsistent dimensions) are
+rejected at load with an actionable rebuild/upgrade hint.
+
+Index writes are **serialized across processes** with a lockfile
+(`<index>/.mdss.lock`, issue #37): `mdss serve --watch` re-indexes on a timer
+while a manual `mdss index` (or a second terminal) may run — without a lock two
+builds interleave and can leave `vectors.json`/`.hashes.json` from different
+runs. `buildIndex` acquires the lock for the whole build and releases it in a
+`finally`; a second process gets a clear `index is being written by pid NNN`
+error, and `serve --watch` politely defers its cycle until the other writer
+finishes. A crashed writer can't wedge the index — the lock self-heals by
+pid-liveness + staleness (a dead-pid or abandoned lock is reclaimed).
+
+### Diagnose problems (`mdss check` / `mdss doctor`)
+
+`mdss check` (alias: `mdss doctor`) is an **offline, read-only** diagnostic for
+the index/db/model-cache trio — like `mdss stats`, it never loads the embedding
+model and never touches the network, but its job is answering *"why is search
+broken?"* rather than *"what's in the index?"*:
+
+```bash
+mdss check --db /path/to/your/markdown
+# check: /path/to/your/markdown/.mdss
+#   ok    vectors.json: parses (schema v3, binary-v1)
+#   ok    .hashes.json: parses (64 file(s))
+#   ok    chunks: 725/725 vectors valid (dim from index)
+#   ok    db /path/to/your/markdown: fresh
+#   ok    model cache: e5-base present at ~/.cache/mdss/Xenova/multilingual-e5-base
+# check: healthy (exit 0)
+```
+
+It validates, in order: `vectors.json` exists / parses; root, `schemaVersion`,
+schema-v3 BM25 data, required current chunk metadata, model dimension, and
+`format` are recognized;
+`.hashes.json` parses; **every chunk's decoded vector**
+with the same validator the loader uses (dim mismatch, NaN/Infinity, truncated
+base64, vec-less legacy chunks — the first five offenders are named); db
+staleness (newest file mtime vs `built`, same 5s grace as the stale warning);
+and the Transformers.js FileCache layout (`<cache-dir>/<model-id>` for `main`,
+with a further `<revision>` directory for pinned revisions). A missing model
+cache is a *warning* by default
+(the first online run downloads it) and a *failure* under `--offline`.
+
+Exit code is `0` when healthy and `1` with a problem summary otherwise;
+`--json` prints the full structured report (`healthy`, `index`, `hashes`,
+`chunks`, `db`, `model`) for scripting and CI gates.
+
 ### 2. Search
 
 ```bash
@@ -111,19 +307,101 @@ Top 3 for: "how do I add a new translation language"
 2. ...
 ```
 
+### Reranking (optional, sharper results)
+
+The default ranking scores each chunk *independently* (vector cosine + BM25
+fused with RRF; legacy indexes use exact overlap). A **cross-encoder** reads the query and the
+passage **together**, capturing pairwise relevance that independent scores can
+miss — at the cost of one forward pass per candidate.
+
+```bash
+mdss search --db /path/to/your/markdown "how do I rotate the api token" --rerank
+```
+
+- The first pass pulls a wider candidate pool (default `max(20, k*3)` chunks),
+  the cross-encoder (`Xenova/bge-reranker-base`, ~280 MB, downloaded on first
+  use) re-scores them as query↔passage pairs, and the best `k` survive.
+- Results from a re-ranked search carry a `rerankScore` field in `--json` mode.
+- Programmatic use: `rerank: true` in `search()`/`searchIndex()`, `rerankPool`
+  to size the pool, or `rerankFn` to swap in your own scorer. The API server
+  accepts `"rerank": true` in the `/search` body.
+- The reranker is lazy: nothing loads unless `--rerank` is actually requested,
+  so searches without it pay zero extra cost.
+
+
+### 3. Serve (daemon, warm index + model)
+
+```bash
+mdss serve --db /path/to/your/markdown [--port 8747] [--host 127.0.0.1] [--watch]
+```
+
+`serve` is the long-running mode for editors, scripts, or anything that asks
+many questions in a row: the index is parsed **once** and the embedding model
+stays loaded in memory, so every request skips the ~280 MB model load and the
+full `vectors.json` parse. `--watch` polls the base every **3 s** (configurable
+via `--watch-interval`, no extra dependencies) and runs the incremental
+re-index automatically. Detection is **content-confirmed**, not mtime-based
+(issue #42): each candidate file's md5 is compared against the authoritative
+per-file hashes the indexer wrote to `.hashes.json`, so a no-op write (touch,
+chmod, an editor rewriting identical bytes) never triggers a re-index, and a
+real edit caught inside a coarse mtime window (FAT, older SMB) is still picked
+up. A **1 s settle debounce** (`--watch-delay`) collapses a burst of successive
+saves (editor save → linter → save) into ONE re-index instead of a back-to-back
+burst. New, changed, and removed files are all handled without restarting.
+
+```bash
+# query it with curl…
+curl -X POST localhost:8747/search \
+  -H 'content-type: application/json' \
+  -d '{"query":"rotate api token","k":5}'
+curl localhost:8747/health          # → {ok, chunks, model, dim, built, watching}
+```
+
+`POST /search` accepts `{"query": string, "k"?: number, "semanticOnly"?: bool,
+"rerank"?: bool}` and replies `{"results": [{file, title, heading, cosine,
+score, matches, snippet, rerankScore?}]}`. The CLI's `search --json` prints
+the same hit shape.
+
+Or from a script / editor extension:
+
+```js
+const r = await fetch('http://localhost:8747/search', {
+  method: 'POST',
+  headers: { 'content-type': 'application/json' },
+  body: JSON.stringify({ query: 'failover runbook', k: 3 }),
+});
+const { results } = await r.json();
+```
+
+The API is unauthenticated — by default it binds to **loopback only**
+(`127.0.0.1`), so other machines on your LAN cannot reach it. Pass `--host
+0.0.0.0` (or `MDSS_HOST`) only if you intend to expose it on a network, and put
+a reverse proxy (or at least a firewall rule) in front of it. Request bodies
+are capped at 64 KB (oversized → `413`), and malformed JSON gets a clear `400`
+instead of being silently swallowed.
+
 ### Options
 
 | Flag | Meaning |
 |------|---------|
 | `--db <dir>` | Folder of `.md` files (or set `MDSS_DB`). Can be anywhere on disk. |
 | `--index-dir <dir>` | Where to store the index (default: `<db>/.mdss`). |
-| `--cache-dir <dir>` | Model cache dir (default: package `.cache`, or `MDSS_CACHE_DIR`). |
+| `--cache-dir <dir>` | Model cache dir. Default: `$XDG_CACHE_HOME/mdss` if set, else `~/.cache/mdss` on **all** platforms (so on Windows that's `C:\Users\<you>\.cache\mdss` — *not* `%LOCALAPPDATA%`; override with `MDSS_CACHE_DIR`). |
 | `--model <name\|id>` | Embedding model (default `e5-base`). See `mdss models`. |
 | `--ignore <glob>` | Skip files/paths; repeatable. e.g. `--ignore "log.md" --ignore "**/archive/**"`. |
-| `--k <n>` | Number of results (default 6). |
-| `--json` | Machine-readable output. |
-| `--semantic` | Pure vector ranking, skip lexical fusion. |
+| `--path <glob>` | Search only files matching glob; repeatable. e.g. `--path "docs/**"`. |
+| `--since <date>` | Search only files modified at/after date (`YYYY-MM-DD` or ISO 8601). |
+| `--k <n>` | Number of results, positive integer (default 6). |
+| `--json` | Machine-readable output: `index` → build result JSON, `stats` → index stats JSON, `search` → hit list JSON (each hit includes `matches` — the query terms found in the chunk). |
+| `--semantic` | `search` only: pure vector ranking, skip lexical/RRF fusion (zero lexical tokenization). |
+| `--rerank` | Re-rank the candidate pool with a cross-encoder (`Xenova/bge-reranker-base`, ~280 MB model, downloaded on first use). Slower, sharper results — see *Reranking* below. |
+| `--port <n>` | HTTP port for `serve` (default 8747, or `MDSS_PORT`). |
+| `--host <ip>` | Bind address for `serve` (default `127.0.0.1` — loopback only; use `0.0.0.0` to expose on the LAN, or `MDSS_HOST`). |
+| `--watch` | `serve`: re-index incrementally when files change. |
+| `--watch-interval <ms>` | `serve --watch`: poll every N ms (default 3000). |
+| `--watch-delay <ms>` | `serve --watch`: quiet-period debounce before a burst of saves triggers ONE re-index (default 1000; issue #42). |
 | `--offline` | Never download the model — require a cached one (or `MDSS_OFFLINE=1`). |
+| `--version` | Print the installed version. |
 
 ### The base can live outside the project
 
@@ -156,14 +434,44 @@ mdss models
 | `e5-base` ⭐ | `Xenova/multilingual-e5-base` | 768 | Default. Best balance. |
 | `e5-large` | `Xenova/multilingual-e5-large` | 1024 | ~2.2 GB, higher quality. |
 | `bge-m3` | `Xenova/bge-m3` | 1024 | ~2.3 GB. Best cross-lingual separation in tests. |
+| `qwen3-embedding-0.6b` | `onnx-community/Qwen3-Embedding-0.6B-ONNX` | 1024 | ~613 MB q8. Opt-in; instruction-prefixed queries and last-token pooling. |
 
 Switching models invalidates the stored vectors automatically — the next
 `index` run does a full rebuild.
 
 Weights are downloaded in quantized (q8) form when the model repo ships them
-(all `Xenova/*` repos do — e5-base is ~280 MB, not ~1.1 GB fp32).
-
-You can also pass any raw `Xenova/*` id, optionally pinning a revision:
+(e5-base is ~280 MB instead of ~1.1 GB fp32).
+
+The Qwen3 alias is pinned to a tested model revision and automatically applies
+the retrieval instruction required for queries; indexed passages remain
+unprefixed. Its upstream benchmark is promising, but it has not replaced
+`e5-base` as the default because the project corpus has not been benchmarked
+against it yet. This is experimental enablement rather than completion of issue
+#50: release provenance and the versioned RU/EN benchmark remain tracked by
+#55 and #56 (the explicit model adapter contract landed in #60).
+
+### How to add a model
+
+Every supported model is an *explicit adapter* in `src/models.mjs` (issue #60).
+The adapter declares exactly which embedding semantics it needs — the shared
+`embed()` path never guesses from the model name:
+
+- `queryPrefix` / `passagePrefix` — text prepended to queries/passages (E5 uses
+  `query: `/`passage: `, BGE none, Qwen3 a retrieval instruction on queries);
+- `pooling` — `'mean'`, `'last_token'`, or `'none'`;
+- `normalize` — whether vectors are L2-normalized;
+- `nativeDim` / `dim` — embedding dimension (and optional `dimensions` for MRL);
+- `dtype` — the ONNX runtime data type loaded at runtime (default `q8`);
+- `maxTokens` — the tokenizer context budget (Qwen3-0.6B is 32768);
+- `family` — an informational formatting/pooling label surfaced by `stats`.
+
+`mdss models` prints each adapter's pooling, normalization, dimension, and
+token budget. `mdss stats` and `mdss check` expose the same adapter fields for
+an existing index, plus its `adapterFingerprint`.
+
+**A raw Hugging Face id that is not a registered adapter is rejected at embed
+time — it does NOT silently inherit E5 prefixes and mean pooling.** Use a
+registered alias (or the explicit object descriptor below):
 
 ```bash
 mdss index --db ./docs --model "Xenova/multilingual-e5-small@abc123def"
@@ -173,19 +481,48 @@ Pinned ids invalidate the index too (the revision is part of the model key), so
 a `@revision` bump triggers a full rebuild. Custom ids should have quantized
 weights (a `model_quantized.onnx` file) in the repo.
 
+Library consumers adding a one-off model can pass an explicit adapter object to
+`buildIndex`/`search` instead of editing the registry:
+
+```js
+import { buildIndex } from 'md-semantic-search';
+await buildIndex({
+  db: './docs', indexDir: './.mdss',
+  modelName: {
+    id: 'Xenova/some-embedding-model',
+    nativeDim: 384,
+    queryPrefix: 'query: ',
+    passagePrefix: 'passage: ',
+    pooling: 'mean',
+    normalize: true,
+  },
+});
+```
+
+Without `queryPrefix`/`passagePrefix`/`pooling`/`dim`, a raw id is still treated
+as an unconfigured adapter and indexing fails with a message pointing at this
+section instead of silently producing wrong vectors.
+
 ## How it works
 
 1. **Walk** `--db` recursively for `.md`/`.markdown` (dotfiles & `--ignore`
    globs skipped).
-2. **Chunk** each file by Markdown headings; oversized sections split on blank
-   lines (~1400 chars/chunk).
-3. **Embed** each chunk (`passage:` prefix for E5) → store `{file, heading,
-   text, vec}` in `vectors.json`, plus per-file md5 in `.hashes.json`. Each chunk
-   also stores `chunkHash` (SHA-256 of the exact passage input + model identity),
-   so on re-index an unchanged section inside a modified file reuses its vector
-   instead of being embedded again.
-4. **Search**: embed the query (`query:` prefix), score every chunk by cosine,
-   score by lexical term-overlap, then **fuse with RRF**. Return top-k chunks.
+2. **Chunk** each file by Markdown headings while retaining the active heading
+   stack; oversized sections split on blank lines (~1400 body chars/chunk).
+3. **Embed** the document title, full heading path, and chunk body (`passage:`
+   prefix for E5) → store `{file, title, heading, headingPath, text, vec}` in
+   `vectors.json`, plus per-file md5 in `.hashes.json`. The index stores an
+   adapter fingerprint so formatting, pooling, normalization, or dimension
+   changes invalidate vectors and checkpoints without invalidating lexical
+   records. Each chunk also stores
+   `chunkHash` (SHA-256 of that exact contextual passage + model identity), so a
+   parent rename invalidates its subtree while unchanged sibling branches reuse
+   their vectors.
+4. **Search**: embed the query with the selected model's formatter (`query:` for
+   E5, no prefix for BGE-M3, or the retrieval instruction for Qwen3), score every
+   chunk by cosine,
+   score query terms from the persisted BM25 postings, then **fuse with RRF**.
+   Legacy schema-v0/v1/v2 indexes retain exact token-overlap scoring.
 
 No external services, no database — the whole index is one JSON file and search
 is an in-memory dot-product sweep.

package/RESEARCH.md

@@ -12,7 +12,7 @@ auditable and reproducible.
 >   ranked an irrelevant page *above* the correct one. `e5-base` fixed it.
 > - **Quantization is not the problem** people assume — int8 vs fp32 changed
 >   cosine scores by <0.003 and never changed the ranking.
-> - **Hybrid (vector + keyword via RRF)** beats either alone: vectors catch
+> - **Hybrid (vector + lexical via RRF)** beats either alone: vectors catch
 >   paraphrases, keywords catch exact identifiers.
 
 ---
@@ -98,55 +98,135 @@ This is why the default is `e5-base`, not the tempting lightweight `e5-small`.
 ## 5. Design decisions that follow
 
 ### E5 prefixes are mandatory
-The E5 family is trained with instruction prefixes: documents must be embedded
-as `passage: <text>` and queries as `query: <text>`. Omitting them measurably
-degrades retrieval. The model registry encodes this per-model, because
-**`bge-m3` is the opposite — it wants no prefix at all.** Getting this wrong is
-a silent quality bug.
-
-### Chunk by heading, not by page
-Embedding a whole multi-section page into one vector blurs distinct topics.
-We split on Markdown headings (`#`..`######`); sections over ~1400 chars split
-again on blank lines. Search then returns the relevant *section*, with its
-heading, not just "the page where the word appears somewhere."
-
-### Hybrid ranking via Reciprocal Rank Fusion
-Pure vectors miss exact identifiers (`win32`, `TextIOWrapper`, error codes);
-pure keywords miss paraphrases. We compute two rankings — cosine and lexical
-term-overlap — and fuse them with RRF (`score = Σ 1/(k + rank)`, `k=60`). RRF
-needs no score normalization or weight tuning, which is what makes it robust
-across very different score scales.
-
-### Stop-words protect the lexical lane
-Cross-lingual queries share function words ("при", "the", "кода") with
-irrelevant documents, letting lexical overlap promote noise. A small ru/en
-stop-list is removed before lexical scoring so RRF fuses signal, not filler.
-
-### Incremental by content hash
-Each file's md5 is stored. Re-indexing reuses embeddings for unchanged files
-and only re-embeds what changed — a no-op re-index of 46 files is sub-second.
-Changing the model invalidates all vectors and forces a clean rebuild
-(dimensions and prefix semantics differ).
-
-## 6. Reproducing
+`multilingual-e5-*` models require `"query: "` before queries and `"passage: "`
+before documents. Omitting them degrades cosine separation.
+
+### Lexical + vector fusion (RRF)
+Vector search catches concepts ("frozen console" → "win32 stdin"), but can blur
+exact function names or error codes (`E_FAIL_WIN32`). Combined rank via
+Reciprocal Rank Fusion ($k=60$) yields the best overall precision.
+
+## 6. Reranking architecture (issue #15)
+
+First-pass vector + lexical retrieval is shallow: query and document are
+embedded independently. A cross-encoder reranker (`Xenova/bge-reranker-base`)
+processes the query and document together through a sequence classification
+head.
+
+- **Lazy loading:** The reranker model is loaded only when `--rerank` is passed.
+- **Candidate pool:** First-pass retrieval returns candidate pool $k_{\text{pool}} = \max(20, k \times 3)$, which is then re-ordered by the cross-encoder.
+- **Head semantics:** `bge-reranker-base` uses a single-class output head (`logits[0]`). Softmax must **not** be applied over single-class heads (which would yield 1.0 for all inputs).
+
+## 7. Golden-set benchmark (issue #56)
+
+The experiments above are qualitative. To make quality claims reproducible and
+regression-detectable, the repo ships a **frozen golden set**: a small synthetic
+RU/EN corpus plus graded relevance judgements.
+
+- `bench/corpus/` — 16 markdown files, 8 topics × 2 languages (tokens, i18n,
+  windows stdin, db failover, model cache, obsidian links, backups, search
+  index/server). Content is deliberately plain so embedding quality, not
+  domain vocabulary, drives the scores.
+- `bench/fixtures/dev-golden.json` — 60 queries across six categories
+  (`natural-question`, `paraphrase`, `keyword`, `alias`, `hard-negative`,
+  `identifier`), 48 dev / 8 test / 4 holdout. Grades: 3 = direct answer,
+  2 = useful support, 1 = related, 0 = hard negative. The fixture pins
+  `corpusHash` (sha256 of paths+content) — a changed corpus fails the run.
+- `bench/fixture.mjs` — schema-v1 validator, `loadFixture`, `corpusFingerprint`,
+  deterministic `splitIntoSlices`.
+- `src/metrics.mjs` — nDCG@k (graded, log2 discount), MRR, Hit@k, Recall@k,
+  `queryMetrics`/`aggregateMetrics`, win/tie/loss for paired model comparison.
+- `scripts/run-bench.mjs` — builds the index over the frozen corpus, runs every
+  query of the selected slice, prints aggregate + per-category metrics.
+
+```bash
+node scripts/run-bench.mjs --slice dev          # full dev eval (k=10, e5-base)
+node scripts/run-bench.mjs --fast --json        # CI smoke (k=3)
+node scripts/run-bench.mjs --model bge-m3 --slice dev > /tmp/bge-m3.json
+node scripts/run-bench.mjs --model e5-base --slice dev > /tmp/e5-base.json
+```
+
+Baseline on dev (e5-base, hybrid RRF, k=10, 2026-08-14):
+
+| Metric | Value |
+|--------|-------|
+| nDCG@10 | 0.9254 |
+| MRR | 0.9167 |
+| Hit@10 | 0.9792 |
+| Recall@10 | 1.0000 |
+
+## 8. Reproducing
 
 ```bash
-# Build with the default, then with the heavy model, and compare a known query:
 mdss index  --db ./your-wiki
 mdss search --db ./your-wiki --semantic "a paraphrase of something you know is in there"
 
 mdss index  --db ./your-wiki --model bge-m3
 mdss search --db ./your-wiki --semantic "the same paraphrase"
-```
 
-Use `--semantic` to see the raw vector ranking (no lexical fusion) when
-evaluating a model — that isolates embedding quality from the keyword lane.
+mdss index  --db ./your-wiki --model qwen3-embedding-0.6b
+mdss search --db ./your-wiki --semantic "the same paraphrase"
+```
 
-## 7. Recommendations
+## 9. Recommendations
 
 | If you want… | Use |
 |--------------|-----|
 | Sensible default, fast, ~280 MB | `e5-base` (default) |
 | Maximum cross-lingual quality, disk/time no object | `bge-m3` |
+| Evaluate a newer 0.6B multilingual model locally | `qwen3-embedding-0.6b` (~613 MB q8) |
 | Smallest footprint and your queries are same-language & literal | `e5-small` (with eyes open) |
 | Best precision on exact identifiers | keep hybrid on (don't pass `--semantic`) |
+| Sharpest top-k on a larger corpus | add `--rerank` (cross-encoder, +280 MB) |
+
+---
+
+## 10. Research Evaluation: Lightweight Multilingual Reranker Tier (Issue #51)
+
+### Context & Decision Gate
+- **Baseline:** `Xenova/bge-reranker-base` (278MB q8, XLM-RoBERTa architecture, single-class logit head).
+- **Rejected Candidate:** `cross-encoder/ms-marco-MiniLM-L6-v2` is English-only and collapses on Russian/bilingual technical queries despite smaller size.
+- **Criteria for New Candidate:**
+  1. Verified ONNX/Transformers.js artifact with `q8` quantization.
+  2. Proven multilingual / cross-lingual ranking capability on RU/EN.
+  3. Commercial MIT/Apache-2.0 compatible license.
+  4. Non-inferiority on the golden set (#56): nDCG@10 drop $\le 0.01$ vs BGE-base.
+  5. Single-class raw logit verification (no 1-class softmax wrapper).
+
+If no candidate meets the multilingual non-inferiority gate, `bge-reranker-base` remains the sole default reranker.
+
+---
+
+## 11. Research Evaluation: Persisted Vector int8 & Matryoshka (MRL) Scaling (Issue #52)
+
+### Disambiguation & Rules
+- **Model Inference Quantization (`dtype: q8`):** Already used for transformer weight inference in `src/core.mjs`.
+- **Stored Vector Quantization:** Compressing stored document float vectors (`Float32Array`) to `int8` with per-vector scale factors.
+- **Matryoshka Representation Learning (MRL):** Truncating embedding output dimension (e.g., 1024 → 256).
+- **Contract Rule:** Dimension truncation is allowed **only** for models with explicit MRL training contracts (e.g. `Qwen3-Embedding-0.6B`). Models like `BGE-M3` do **not** support arbitrary dimension slicing without severe recall degradation.
+
+### Scale Trigger
+- Do not introduce persisted vector quantization or MRL index format breaking changes until corpus size exceeds **100,000 chunks** or stored index size dominates disk/RAM budgets beyond 500 MB.
+
+---
+
+## 12. Research Evaluation: Late Chunking vs Heading-Path Context (Issue #53)
+
+### Baseline vs Late Chunking
+- **Current Baseline:** `mdss` prepends document title + full heading hierarchy path (`headingPath: ['Section', 'Subsection']`) to every chunk before tokenization.
+- **Late Chunking Concept:** Passing full documents through a long-context transformer, extracting token-level hidden states, and pooling chunk token spans post-hoc.
+- **Evaluation Gate:** Late chunking adds high memory overhead, token-offset alignment complexity, and custom model batching. It will be prototyped **only if** heading-path contextualization exhibits a measured context-recall deficit on long-document ablation benchmarks.
+
+---
+
+## 13. Research Evaluation: WASM SIMD & ANN/IVF Scale Triggers (Issues #31 & #32)
+
+### WebAssembly SIMD Kernel Gate (Issue #31)
+- **Baseline:** Linear sweep over contiguous `Float32Array` runtime vector buffer (Issue #30).
+- **Trigger:** WebAssembly SIMD dot-product kernels will be considered **only if** CPU profiling on >50,000 chunks shows JS float multiplication accounts for >40% of warm search latency.
+- **Constraint:** Zero native Node C++ bindings; must maintain pure JS fallback for non-WASM environments.
+
+### ANN / IVF Scale Threshold (Issue #32)
+- **Baseline:** Exact $O(N \cdot d)$ linear sweep.
+- **Trigger:** ANN (Inverted File / HNSW) indexing is a scale contingency. It shall **not** be introduced until corpus size exceeds **100,000+ chunks** and exact warm sweep latency exceeds 50 ms.
+- **Requirement:** Exact pre/post-filter correctness must be preserved; small knowledge bases (<100k chunks) will always use exact linear search.