md-semantic-search 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 chelslava
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,162 @@
+# md-semantic-search
+
+**Local, private semantic (vector) search over any folder of Markdown files.**
+
+Find passages by *meaning*, not just keywords — and across languages (ask in
+one language, match documents written in another). Runs fully on your machine
+via [transformers.js](https://github.com/xenova/transformers.js): no API keys,
+no cloud calls, no vector database. Your notes never leave the disk.
+
+```bash
+npx md-semantic-search index  --db ./docs
+npx md-semantic-search search --db ./docs "how do I rotate the API token"
+```
+
+---
+
+## Why
+
+Keyword search misses paraphrases. A query like *"как починить зависший ввод на
+windows"* will never match a page titled *"win32 stdin re-wrap closes buffer"* —
+zero shared words, different language. Semantic search embeds both into the same
+vector space and matches on meaning. This tool was extracted from a real wiki
+where exactly that gap kept biting; see **[RESEARCH.md](./RESEARCH.md)** for the
+measurements that shaped its defaults.
+
+## Features
+
+- 🔌 **Any folder, anywhere.** Point `--db` at any directory of `.md`/`.markdown`
+  files. It does **not** have to live inside this project. Recursive by default.
+- 🌍 **Cross-lingual.** Multilingual embeddings (default `multilingual-e5-base`).
+- 🧠 **Hybrid ranking.** Reciprocal Rank Fusion of vector similarity (meaning)
+  and lexical overlap (exact names like `win32`, `TextIOWrapper`).
+- ⚡ **Incremental.** Per-file md5 — re-indexing only re-embeds changed files.
+- 🔒 **Private & offline.** Model downloads once, then no network. Nothing is
+  uploaded anywhere.
+- 📦 **Zero infra.** One JSON index, brute-force cosine in memory. No Pinecone,
+  no Qdrant, no pgvector. Scales fine to thousands of chunks.
+
+## Requirements
+
+- Node.js ≥ 18
+- ~280 MB disk for the default model (downloaded once into a cache dir)
+
+## Install
+
+Run on demand with `npx` (no install):
+
+```bash
+npx md-semantic-search --help
+```
+
+Or install globally for the short `mdss` alias:
+
+```bash
+npm install -g md-semantic-search
+mdss --help
+```
+
+Or from source:
+
+```bash
+git clone https://github.com/chelslava/md-semantic-search
+cd md-semantic-search
+npm install
+node bin/cli.mjs --help
+```
+
+## Usage
+
+### 1. Build the index
+
+```bash
+mdss index --db /path/to/your/markdown
+```
+
+First run downloads the model (~280 MB). The index is written to `<db>/.mdss/`
+by default (override with `--index-dir`). Re-run after editing your notes — it's
+incremental, so only changed files are re-embedded.
+
+### 2. Search
+
+```bash
+mdss search --db /path/to/your/markdown "your question in plain language"
+```
+
+Example output:
+
+```
+Top 3 for: "how do I add a new translation language"
+
+1. [cos 0.833] i18n Application Analysis
+   i18n-analysis.md › Language status
+   | Language | File | Status | ... English | en/shared.json | complete | ...
+
+2. ...
+```
+
+### Options
+
+| Flag | Meaning |
+|------|---------|
+| `--db <dir>` | Folder of `.md` files (or set `MDSS_DB`). Can be anywhere on disk. |
+| `--index-dir <dir>` | Where to store the index (default: `<db>/.mdss`). |
+| `--cache-dir <dir>` | Model cache dir (default: package `.cache`, or `MDSS_CACHE_DIR`). |
+| `--model <name\|id>` | Embedding model (default `e5-base`). See `mdss models`. |
+| `--ignore <glob>` | Skip files/paths; repeatable. e.g. `--ignore "log.md" --ignore "**/archive/**"`. |
+| `--k <n>` | Number of results (default 6). |
+| `--json` | Machine-readable output. |
+| `--semantic` | Pure vector ranking, skip lexical fusion. |
+
+### The base can live outside the project
+
+The index does not need write access to your notes if they're read-only — just
+point the index somewhere writable:
+
+```bash
+mdss index  --db /mnt/shared/team-wiki --index-dir ~/.cache/team-wiki-index
+mdss search --db /mnt/shared/team-wiki --index-dir ~/.cache/team-wiki-index "incident runbook for db failover"
+```
+
+Or drive everything from environment variables:
+
+```bash
+export MDSS_DB=/mnt/shared/team-wiki
+export MDSS_INDEX_DIR=~/.cache/team-wiki-index
+mdss index
+mdss search "incident runbook for db failover"
+```
+
+## Models
+
+```bash
+mdss models
+```
+
+| Alias | Model | Dim | Notes |
+|-------|-------|-----|-------|
+| `e5-small` | `Xenova/multilingual-e5-small` | 384 | Fastest (~120 MB). **Weak cross-lingual** — see RESEARCH. |
+| `e5-base` ⭐ | `Xenova/multilingual-e5-base` | 768 | Default. Best balance. |
+| `e5-large` | `Xenova/multilingual-e5-large` | 1024 | ~2.2 GB, higher quality. |
+| `bge-m3` | `Xenova/bge-m3` | 1024 | ~2.3 GB. Best cross-lingual separation in tests. |
+
+Switching models invalidates the stored vectors automatically — the next
+`index` run does a full rebuild. You can also pass any raw `Xenova/*` id.
+
+## How it works
+
+1. **Walk** `--db` recursively for `.md`/`.markdown` (dotfiles & `--ignore`
+   globs skipped).
+2. **Chunk** each file by Markdown headings; oversized sections split on blank
+   lines (~1400 chars/chunk).
+3. **Embed** each chunk (`passage:` prefix for E5) → store `{file, heading,
+   text, vec}` in `vectors.json`, plus per-file md5 in `.hashes.json`.
+4. **Search**: embed the query (`query:` prefix), score every chunk by cosine,
+   score by lexical term-overlap, then **fuse with RRF**. Return top-k chunks.
+
+No external services, no database — the whole index is one JSON file and search
+is an in-memory dot-product sweep.
+
+## License
+
+MIT © chelslava

package/RESEARCH.md

@@ -0,0 +1,152 @@
+# RESEARCH — design notes & measurements
+
+This tool's defaults aren't guesses. They come from building semantic search
+over a real, bilingual (Russian/English) engineering wiki and measuring what
+actually worked. This document records those experiments so the choices are
+auditable and reproducible.
+
+> **TL;DR**
+> - For a small corpus you do **not** need a vector database — brute-force
+>   cosine over a JSON file is microseconds.
+> - **`multilingual-e5-small` is not good enough for cross-lingual search.** It
+>   ranked an irrelevant page *above* the correct one. `e5-base` fixed it.
+> - **Quantization is not the problem** people assume — int8 vs fp32 changed
+>   cosine scores by <0.003 and never changed the ranking.
+> - **Hybrid (vector + keyword via RRF)** beats either alone: vectors catch
+>   paraphrases, keywords catch exact identifiers.
+
+---
+
+## 1. The problem
+
+The starting point was a keyword-only wiki search (substring + tag matching).
+It failed on the most common real query shape: a paraphrase, often in a
+different language than the document.
+
+Concrete failure: the query
+
+> «как починить зависший ввод консоли на windows»
+> *(how to fix frozen console input on windows)*
+
+should surface a page documenting a Windows `stdin` bridge bug
+(*"win32 stdin re-wrap closes the shared buffer"*). Keyword search returns
+**nothing** — there is not one shared token, and the document is in English
+while the query is in Russian.
+
+## 2. Corpus scale — why no vector DB
+
+The wiki under test:
+
+| Metric | Value |
+|--------|-------|
+| Pages | 46 markdown files |
+| Raw size | 238 KB (~140,800 chars ≈ 35K tokens) |
+| Chunks (by heading) | 220 |
+| Vector dim (e5-base) | 768 |
+
+220 × 768 floats is a ~1.5 MB JSON file. A full cosine sweep over it is well
+under a millisecond. **Pinecone / Qdrant / Chroma / pgvector would be pure
+overhead here.** The entire index is one file; search is an in-memory
+dot-product loop. This holds comfortably into the low tens of thousands of
+chunks before you'd want an ANN index.
+
+## 3. Experiment A — does quantization hurt?
+
+transformers.js loads **int8-quantized** ONNX weights by default. A common
+worry is that quantization collapses the embedding space. We tested the same
+query against three passages, quantized vs full fp32 (`quantized: false`):
+
+Query: «редактор кода съедает пробелы при наборе» *(code editor eats spaces while typing)*
+
+| Passage | quantized (int8) | full (fp32) |
+|---------|------------------:|------------:|
+| win32 stdin re-wrap closes the shared buffer | 0.816 | 0.818 |
+| Monaco editor drops every space while typing | 0.806 | 0.806 |
+| Excel library writes cells / formats workbooks | 0.779 | 0.767 |
+
+**Conclusion:** differences are ≤ 0.003 and the **ranking is identical**.
+Quantization is not worth disabling — you'd pay 4× the download and memory for
+no ranking change. (Note the *other* problem visible here, addressed next: the
+irrelevant `win32` passage outscores the correct `Monaco` one — that's the
+model, not the quantization.)
+
+## 4. Experiment B — model choice decides cross-lingual quality
+
+Same query («редактор кода съедает пробелы при наборе»), where the **correct**
+answer is the Monaco "spaces" passage. We compared three multilingual models.
+A model passes only if it ranks Monaco **above** the unrelated passages.
+
+| Model | win32 stdin | **Monaco (correct)** | Excel | Correct #1? | Margin over runner-up |
+|-------|------------:|---------------------:|------:|:-----------:|----------------------:|
+| `multilingual-e5-small` | **0.818** | 0.806 | 0.767 | ❌ no | −0.012 |
+| `multilingual-e5-base`  | 0.784 | **0.810** | 0.741 | ✅ yes | +0.026 |
+| `bge-m3`                | 0.700 | **0.765** | 0.664 | ✅ yes | **+0.065** |
+
+**Findings:**
+
+- **`e5-small` fails.** It puts the unrelated `win32` passage *above* the
+  correct one. On a real corpus this means the right page never reaches the top.
+  Its cosine scores also sit in a narrow band (~0.77–0.82 for everything),
+  i.e. it barely separates documents.
+- **`e5-base` fixes the ordering** at ~280 MB and stays fast — chosen as the
+  **default**.
+- **`bge-m3` gives the widest separation** (+0.065 margin), the best quality,
+  but costs ~2.3 GB and is slower. Offered as an opt-in for quality-critical use.
+
+This is why the default is `e5-base`, not the tempting lightweight `e5-small`.
+
+## 5. Design decisions that follow
+
+### E5 prefixes are mandatory
+The E5 family is trained with instruction prefixes: documents must be embedded
+as `passage: <text>` and queries as `query: <text>`. Omitting them measurably
+degrades retrieval. The model registry encodes this per-model, because
+**`bge-m3` is the opposite — it wants no prefix at all.** Getting this wrong is
+a silent quality bug.
+
+### Chunk by heading, not by page
+Embedding a whole multi-section page into one vector blurs distinct topics.
+We split on Markdown headings (`#`..`######`); sections over ~1400 chars split
+again on blank lines. Search then returns the relevant *section*, with its
+heading, not just "the page where the word appears somewhere."
+
+### Hybrid ranking via Reciprocal Rank Fusion
+Pure vectors miss exact identifiers (`win32`, `TextIOWrapper`, error codes);
+pure keywords miss paraphrases. We compute two rankings — cosine and lexical
+term-overlap — and fuse them with RRF (`score = Σ 1/(k + rank)`, `k=60`). RRF
+needs no score normalization or weight tuning, which is what makes it robust
+across very different score scales.
+
+### Stop-words protect the lexical lane
+Cross-lingual queries share function words ("при", "the", "кода") with
+irrelevant documents, letting lexical overlap promote noise. A small ru/en
+stop-list is removed before lexical scoring so RRF fuses signal, not filler.
+
+### Incremental by content hash
+Each file's md5 is stored. Re-indexing reuses embeddings for unchanged files
+and only re-embeds what changed — a no-op re-index of 46 files is sub-second.
+Changing the model invalidates all vectors and forces a clean rebuild
+(dimensions and prefix semantics differ).
+
+## 6. Reproducing
+
+```bash
+# Build with the default, then with the heavy model, and compare a known query:
+mdss index  --db ./your-wiki
+mdss search --db ./your-wiki --semantic "a paraphrase of something you know is in there"
+
+mdss index  --db ./your-wiki --model bge-m3
+mdss search --db ./your-wiki --semantic "the same paraphrase"
+```
+
+Use `--semantic` to see the raw vector ranking (no lexical fusion) when
+evaluating a model — that isolates embedding quality from the keyword lane.
+
+## 7. Recommendations
+
+| If you want… | Use |
+|--------------|-----|
+| Sensible default, fast, ~280 MB | `e5-base` (default) |
+| Maximum cross-lingual quality, disk/time no object | `bge-m3` |
+| Smallest footprint and your queries are same-language & literal | `e5-small` (with eyes open) |
+| Best precision on exact identifiers | keep hybrid on (don't pass `--semantic`) |

package/bin/cli.mjs

@@ -0,0 +1,157 @@
+#!/usr/bin/env node
+/**
+ * md-semantic-search (mdss) — local semantic search over any folder of .md files.
+ *
+ *   mdss index  --db <dir> [--model e5-base] [--index-dir <dir>] [--ignore <glob>]
+ *   mdss search --db <dir> [--k 6] [--json] [--semantic] "query text"
+ *   mdss models
+ *
+ * The markdown base (--db) can live anywhere on disk — it does NOT need to be
+ * inside this project. The index defaults to <db>/.mdss unless --index-dir is given.
+ */
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import fs from 'node:fs';
+import { buildIndex } from '../src/indexer.mjs';
+import { search } from '../src/search.mjs';
+import { MODELS, DEFAULT_MODEL } from '../src/models.mjs';
+
+const HERE = path.dirname(fileURLToPath(import.meta.url));
+const PKG_ROOT = path.resolve(HERE, '..');
+const DEFAULT_CACHE = path.join(PKG_ROOT, '.cache');
+
+function parseArgs(argv) {
+  const opts = { _: [], ignore: [] };
+  for (let i = 0; i < argv.length; i++) {
+    const a = argv[i];
+    if (a === '--db') opts.db = argv[++i];
+    else if (a === '--index-dir') opts.indexDir = argv[++i];
+    else if (a === '--cache-dir') opts.cacheDir = argv[++i];
+    else if (a === '--model') opts.model = argv[++i];
+    else if (a === '--k') opts.k = parseInt(argv[++i], 10);
+    else if (a === '--ignore') opts.ignore.push(argv[++i]);
+    else if (a === '--json') opts.json = true;
+    else if (a === '--semantic') opts.semantic = true;
+    else if (a === '-h' || a === '--help') opts.help = true;
+    else opts._.push(a);
+  }
+  return opts;
+}
+
+function resolveDb(opts) {
+  const db = opts.db || process.env.MDSS_DB;
+  if (!db) die('Missing --db <dir> (or set MDSS_DB). Path to your .md folder.');
+  const abs = path.resolve(db);
+  if (!fs.existsSync(abs) || !fs.statSync(abs).isDirectory()) {
+    die(`--db is not a directory: ${abs}`);
+  }
+  return abs;
+}
+
+function resolveIndexDir(opts, db) {
+  // Default: <db>/.mdss. Override with --index-dir or MDSS_INDEX_DIR.
+  const dir = opts.indexDir || process.env.MDSS_INDEX_DIR || path.join(db, '.mdss');
+  return path.resolve(dir);
+}
+
+const resolveCache = (opts) =>
+  path.resolve(opts.cacheDir || process.env.MDSS_CACHE_DIR || DEFAULT_CACHE);
+
+function die(msg) {
+  process.stderr.write(`error: ${msg}\n`);
+  process.exit(1);
+}
+
+const HELP = `md-semantic-search (mdss) — local, private semantic search over Markdown
+
+Usage:
+  mdss index  --db <dir> [options]            Build/refresh the index
+  mdss search --db <dir> [options] "query"    Search by meaning
+  mdss models                                  List available models
+
+Options:
+  --db <dir>          Folder of .md files (or env MDSS_DB). Can be anywhere.
+  --index-dir <dir>   Where to store the index (default: <db>/.mdss).
+  --cache-dir <dir>   Model cache dir (default: <pkg>/.cache or MDSS_CACHE_DIR).
+  --model <name|id>   Embedding model (default: ${DEFAULT_MODEL}). See \`mdss models\`.
+  --ignore <glob>     Skip files/paths (repeatable). e.g. --ignore "log.md".
+  --k <n>             Number of results (search, default 6).
+  --json              Machine-readable output (search).
+  --semantic          Pure vector ranking, skip lexical/RRF fusion (search).
+  -h, --help          Show this help.
+
+Examples:
+  mdss index  --db ./docs
+  mdss index  --db /abs/path/to/wiki --model bge-m3 --ignore "log.md" --ignore "**/archive/**"
+  mdss search --db ./docs "how do I rotate the api token"
+  MDSS_DB=./docs mdss search "rollback a failed migration" --k 8 --json
+`;
+
+async function cmdIndex(opts) {
+  const db = resolveDb(opts);
+  const indexDir = resolveIndexDir(opts, db);
+  const cacheDir = resolveCache(opts);
+  const t0 = Date.now();
+  const r = await buildIndex({
+    db, indexDir, cacheDir,
+    modelName: opts.model || DEFAULT_MODEL,
+    ignore: opts.ignore,
+    log: s => process.stderr.write(s + '\n'),
+  });
+  const secs = ((Date.now() - t0) / 1000).toFixed(1);
+  process.stderr.write(
+    `\nIndexed ${r.files} file(s) → ${r.chunks} chunks ` +
+    `(${r.reused} reused, ${r.embedded} embedded), dim=${r.dim}, ` +
+    `model=${r.model}, ${secs}s\n→ ${r.vectorsPath}\n`,
+  );
+}
+
+async function cmdSearch(opts) {
+  const db = resolveDb(opts);
+  const indexDir = resolveIndexDir(opts, db);
+  const cacheDir = resolveCache(opts);
+  const query = opts._.join(' ').trim();
+  if (!query) die('Missing query text. e.g. mdss search --db ./docs "your question"');
+
+  const results = await search({
+    indexDir, cacheDir, query,
+    k: opts.k || 6,
+    semanticOnly: !!opts.semantic,
+  });
+
+  if (opts.json) { process.stdout.write(JSON.stringify(results, null, 2) + '\n'); return; }
+  if (results.length === 0) { process.stdout.write('No matches.\n'); return; }
+
+  process.stdout.write(`\nTop ${results.length} for: "${query}"\n\n`);
+  results.forEach((r, i) => {
+    const loc = r.heading ? `${r.file} › ${r.heading}` : r.file;
+    process.stdout.write(`${i + 1}. [cos ${r.cosine}] ${r.title}\n`);
+    process.stdout.write(`   ${loc}\n`);
+    process.stdout.write(`   ${r.snippet}${r.snippet.length >= 220 ? '…' : ''}\n\n`);
+  });
+}
+
+function cmdModels() {
+  process.stdout.write('Available models (alias → id):\n\n');
+  for (const [alias, m] of Object.entries(MODELS)) {
+    const star = alias === DEFAULT_MODEL ? ' (default)' : '';
+    process.stdout.write(`  ${alias}${star}\n    ${m.id} · dim ${m.dim}\n    ${m.note}\n\n`);
+  }
+  process.stdout.write('You can also pass any raw Xenova/* model id to --model.\n');
+}
+
+async function main() {
+  const argv = process.argv.slice(2);
+  const opts = parseArgs(argv);
+  const cmd = opts._.shift();
+
+  if (opts.help || !cmd) { process.stdout.write(HELP); return; }
+  switch (cmd) {
+    case 'index': return cmdIndex(opts);
+    case 'search': return cmdSearch(opts);
+    case 'models': return cmdModels();
+    default: die(`unknown command: ${cmd}. Try \`mdss --help\`.`);
+  }
+}
+
+main().catch(e => die(e.message));

package/package.json

@@ -0,0 +1,48 @@
+{
+  "name": "md-semantic-search",
+  "version": "0.1.0",
+  "description": "Local, private semantic (vector) search over any folder of Markdown files. Finds passages by meaning — cross-lingual — using transformers.js. No API keys, no cloud, no vector database.",
+  "type": "module",
+  "bin": {
+    "md-semantic-search": "bin/cli.mjs",
+    "mdss": "bin/cli.mjs"
+  },
+  "files": [
+    "bin/",
+    "src/",
+    "README.md",
+    "RESEARCH.md",
+    "LICENSE"
+  ],
+  "scripts": {
+    "test": "node --test"
+  },
+  "keywords": [
+    "semantic-search",
+    "vector-search",
+    "embeddings",
+    "markdown",
+    "rag",
+    "transformers",
+    "e5",
+    "local",
+    "offline",
+    "cli"
+  ],
+  "author": "chelslava",
+  "license": "MIT",
+  "engines": {
+    "node": ">=18"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/chelslava/md-semantic-search.git"
+  },
+  "homepage": "https://github.com/chelslava/md-semantic-search#readme",
+  "bugs": {
+    "url": "https://github.com/chelslava/md-semantic-search/issues"
+  },
+  "dependencies": {
+    "@xenova/transformers": "^2.17.2"
+  }
+}

package/src/indexer.mjs

@@ -0,0 +1,107 @@
+/**
+ * Build / refresh the semantic index for a folder of markdown files.
+ * Incremental: per-file md5; unchanged files reuse their stored embeddings.
+ */
+import fs from 'node:fs';
+import path from 'node:path';
+import crypto from 'node:crypto';
+import { walkMarkdown, parseFile, embed, resolveModel } from './core.mjs';
+
+const md5 = s => crypto.createHash('md5').update(s).digest('hex');
+const loadJSON = (p, fb) => {
+  try { return JSON.parse(fs.readFileSync(p, 'utf8')); } catch { return fb; }
+};
+
+/**
+ * @param {object} opts
+ * @param {string} opts.db        - base dir of .md files (absolute)
+ * @param {string} opts.indexDir  - where vectors.json / .hashes.json live
+ * @param {string} opts.cacheDir  - model cache dir
+ * @param {string} opts.modelName - model alias or id
+ * @param {string[]} opts.ignore  - glob patterns to skip
+ * @param {(s:string)=>void} [opts.log]
+ */
+export async function buildIndex(opts) {
+  const { db, indexDir, cacheDir, modelName, ignore = [], log = () => {} } = opts;
+  const model = resolveModel(modelName);
+  const vectorsPath = path.join(indexDir, 'vectors.json');
+  const hashesPath = path.join(indexDir, '.hashes.json');
+
+  fs.mkdirSync(indexDir, { recursive: true });
+
+  const files = walkMarkdown(db, ignore);
+  const oldHashes = loadJSON(hashesPath, {});
+  const oldIndex = loadJSON(vectorsPath, { chunks: [], model: null });
+
+  // If the model changed, all stored vectors are incompatible → full rebuild.
+  const modelChanged = oldIndex.model && oldIndex.model !== model.id;
+
+  const oldByFile = new Map();
+  if (!modelChanged) {
+    for (const c of oldIndex.chunks) {
+      if (!oldByFile.has(c.file)) oldByFile.set(c.file, []);
+      oldByFile.get(c.file).push(c);
+    }
+  }
+
+  const newHashes = {};
+  const chunks = [];
+  const toEmbed = [];
+  let reused = 0, changedFiles = 0;
+
+  for (const abs of files) {
+    const raw = fs.readFileSync(abs, 'utf8');
+    const rel = path.relative(db, abs).split(path.sep).join('/');
+    const h = md5(raw);
+    newHashes[rel] = h;
+
+    if (!modelChanged && oldHashes[rel] === h && oldByFile.has(rel)) {
+      const old = oldByFile.get(rel);
+      chunks.push(...old);
+      reused += old.length;
+      continue;
+    }
+    changedFiles++;
+    const parsed = parseFile(abs, db);
+    for (const c of parsed) { chunks.push(c); toEmbed.push(c); }
+  }
+
+  if (toEmbed.length > 0) {
+    log(`Embedding ${toEmbed.length} chunks from ${changedFiles} changed file(s) ` +
+        `with ${model.id}...`);
+    const BATCH = 32;
+    for (let i = 0; i < toEmbed.length; i += BATCH) {
+      const slice = toEmbed.slice(i, i + BATCH);
+      const vecs = await embed(
+        slice.map(c => `${c.title}\n${c.heading}\n${c.text}`),
+        'passage', model, cacheDir,
+      );
+      slice.forEach((c, j) => { c.vec = vecs[j]; });
+      log(`  ${Math.min(i + BATCH, toEmbed.length)}/${toEmbed.length}`);
+    }
+  }
+
+  const dim = chunks[0]?.vec?.length ?? model.dim ?? 0;
+  const index = {
+    model: model.id,
+    modelAlias: modelName || 'e5-base',
+    dim,
+    db,
+    built: new Date().toISOString(),
+    chunkCount: chunks.length,
+    chunks,
+  };
+
+  fs.writeFileSync(vectorsPath, JSON.stringify(index));
+  fs.writeFileSync(hashesPath, JSON.stringify(newHashes, null, 2));
+
+  return {
+    files: files.length,
+    chunks: chunks.length,
+    reused,
+    embedded: toEmbed.length,
+    dim,
+    model: model.id,
+    vectorsPath,
+  };
+}

package/src/models.mjs

@@ -0,0 +1,55 @@
+/**
+ * Embedding model registry.
+ *
+ * `prefix` matters: the E5 family is trained with instruction prefixes
+ * ("query: " / "passage: ") and ranks poorly without them. BGE-M3 is trained
+ * WITHOUT prefixes, so forcing them in would hurt. Each entry declares how to
+ * format inputs so the rest of the code stays model-agnostic.
+ */
+export const MODELS = {
+  'e5-small': {
+    id: 'Xenova/multilingual-e5-small',
+    dim: 384,
+    queryPrefix: 'query: ',
+    passagePrefix: 'passage: ',
+    note: 'Fastest, ~120MB. Weak cross-lingual ranking — see RESEARCH.md.',
+  },
+  'e5-base': {
+    id: 'Xenova/multilingual-e5-base',
+    dim: 768,
+    queryPrefix: 'query: ',
+    passagePrefix: 'passage: ',
+    note: 'Default. ~280MB. Solid multilingual + cross-lingual balance.',
+  },
+  'e5-large': {
+    id: 'Xenova/multilingual-e5-large',
+    dim: 1024,
+    queryPrefix: 'query: ',
+    passagePrefix: 'passage: ',
+    note: '~2.2GB. Higher quality, slower.',
+  },
+  'bge-m3': {
+    id: 'Xenova/bge-m3',
+    dim: 1024,
+    queryPrefix: '',
+    passagePrefix: '',
+    note: '~2.3GB. Best cross-lingual separation in our tests; no prefixes.',
+  },
+};
+
+export const DEFAULT_MODEL = 'e5-base';
+
+/** Resolve a model alias OR a raw Xenova/... id into a model descriptor. */
+export function resolveModel(name) {
+  if (!name) return MODELS[DEFAULT_MODEL];
+  if (MODELS[name]) return MODELS[name];
+  // Allow passing a raw HF/Xenova id; assume E5-style prefixes unless it's bge.
+  const isBge = /bge/i.test(name);
+  return {
+    id: name,
+    dim: 0,
+    queryPrefix: isBge ? '' : 'query: ',
+    passagePrefix: isBge ? '' : 'passage: ',
+    note: 'Custom model id.',
+  };
+}

package/src/search.mjs

@@ -0,0 +1,94 @@
+/**
+ * Hybrid semantic + lexical search over a prebuilt index.
+ * Ranking = Reciprocal Rank Fusion of cosine similarity (meaning) and
+ * term-overlap (exact names). The model is read from the index, so callers
+ * never have to repeat --model at search time.
+ */
+import fs from 'node:fs';
+import path from 'node:path';
+import { embed, cosine, resolveModel } from './core.mjs';
+
+// Common ru/en function words — they match everywhere and pollute lexical scores.
+const STOP = new Set([
+  'the', 'and', 'for', 'are', 'was', 'has', 'with', 'this', 'that', 'from',
+  'not', 'but', 'you', 'your', 'can', 'all', 'any', 'its',
+  'все', 'как', 'что', 'это', 'при', 'для', 'или', 'был', 'без', 'над',
+  'под', 'так', 'его', 'нет', 'код', 'кода', 'есть',
+]);
+
+function tokenize(text) {
+  const m = text.toLowerCase().match(/[\p{L}\p{N}]{3,}/gu);
+  return m ? m.filter(t => !STOP.has(t)) : [];
+}
+
+function keywordScores(chunks, query) {
+  const qTerms = new Set(tokenize(query));
+  return chunks.map(c => {
+    const hay = `${c.title} ${c.heading} ${c.text}`.toLowerCase();
+    let s = 0;
+    for (const t of qTerms) if (hay.includes(t)) s++;
+    return s;
+  });
+}
+
+/** Reciprocal Rank Fusion. rankings: arrays of {idx, score}; higher = better. */
+function rrf(rankings, k = 60) {
+  const fused = new Map();
+  for (const ranking of rankings) {
+    const sorted = [...ranking].sort((a, b) => b.score - a.score);
+    sorted.forEach((item, rank) => {
+      if (item.score <= 0) return;
+      fused.set(item.idx, (fused.get(item.idx) || 0) + 1 / (k + rank + 1));
+    });
+  }
+  return fused;
+}
+
+/**
+ * @param {object} opts
+ * @param {string} opts.indexDir
+ * @param {string} opts.cacheDir
+ * @param {string} opts.query
+ * @param {number} [opts.k=6]
+ * @param {boolean} [opts.semanticOnly=false]
+ * @returns {Promise<Array>} results with file, title, heading, cosine, score, snippet
+ */
+export async function search(opts) {
+  const { indexDir, cacheDir, query, k = 6, semanticOnly = false } = opts;
+  const vectorsPath = path.join(indexDir, 'vectors.json');
+  if (!fs.existsSync(vectorsPath)) {
+    throw new Error(`No index at ${vectorsPath}. Run \`mdss index\` first.`);
+  }
+  const index = JSON.parse(fs.readFileSync(vectorsPath, 'utf8'));
+  const model = resolveModel(index.modelAlias || index.model);
+
+  const [qVec] = await embed([query], 'query', model, cacheDir);
+
+  const semantic = index.chunks.map((c, idx) => ({ idx, score: cosine(qVec, c.vec) }));
+  const cosByIdx = new Map(semantic.map(s => [s.idx, s.score]));
+
+  let ranked;
+  if (semanticOnly) {
+    ranked = [...semantic].sort((a, b) => b.score - a.score).slice(0, k)
+      .map(s => ({ idx: s.idx, fscore: s.score, cos: s.score }));
+  } else {
+    const kw = keywordScores(index.chunks, query).map((score, idx) => ({ idx, score }));
+    const fused = rrf([semantic, kw]);
+    ranked = [...fused.entries()]
+      .map(([idx, fscore]) => ({ idx, fscore, cos: cosByIdx.get(idx) }))
+      .sort((a, b) => b.fscore - a.fscore)
+      .slice(0, k);
+  }
+
+  return ranked.map(r => {
+    const c = index.chunks[r.idx];
+    return {
+      file: c.file,
+      title: c.title,
+      heading: c.heading,
+      cosine: +r.cos.toFixed(3),
+      score: +r.fscore.toFixed(4),
+      snippet: c.text.replace(/\s+/g, ' ').slice(0, 220),
+    };
+  });
+}