md-semantic-search 1.0.2 → 1.1.0

package/README.md

@@ -109,6 +109,9 @@ measurements that shaped its defaults.
 
 ## Install
 
+Requires **Node.js ≥ 18**. Native dependencies ship as prebuilt binaries — no
+compiler toolchain needed on any platform.
+
 Run on demand with `npx` (no install):
 
 ```bash
@@ -131,6 +134,9 @@ npm install
 node bin/cli.mjs --help
 ```
 
+For headless server deployments there is a Docker path — see
+[docs/deploy.md](./docs/deploy.md).
+
 ---
 
 ## Library usage
@@ -177,7 +183,14 @@ indexes and existing library consumers; schema-v3 search uses persisted BM25.
 mdss index --db /path/to/your/markdown
 ```
 
-First run downloads the model (~280 MB). The index is written to `<db>/.mdss/`
+First run downloads the model (~280 MB) — and the download itself is
+**visible**: an aggregated bar on stderr shows percent across every file of the
+model directory, live MB/s and ETA (the same bar reappears when `--rerank`
+first pulls the cross-encoder). Under `--json` the bar is replaced by throttled
+`{"type":"download-progress", …}` / `{"type":"download-complete", …}` events
+**on stderr**, so stdout stays machine-readable. A warm cache produces no
+download output at all, and `--offline` behaves exactly as before. The index
+is written to `<db>/.mdss/`
 by default (override with `--index-dir`). Re-run after editing your notes — it's
 incremental at two levels: unchanged files are skipped via their md5, and inside
 a changed file only the sections whose text actually changed are re-embedded
@@ -374,8 +387,31 @@ mdss search --db /path/to/your/markdown "how do I rotate the api token" --rerank
 - The reranker is lazy: nothing loads unless `--rerank` is actually requested,
   so searches without it pay zero extra cost.
 
+### 3. Ask & Interactive Multi-turn Chat (`mdss ask`, `mdss chat`)
 
-### 3. Serve (daemon, warm index + model)
+`md-semantic-search` includes a grounded RAG QA engine that operates fully offline by default with zero external dependencies, or connects optionally to local/remote LLMs (Ollama, OpenAI-compatible APIs).
+
+```bash
+# One-shot grounded QA with citations:
+mdss ask --db ./notes "What is the token expiration policy?"
+
+# Multi-turn persistent chat session:
+mdss chat --db ./notes
+
+# Resume the most recent session or a specific session:
+mdss chat --db ./notes --resume
+mdss chat --db ./notes --session my-session-id
+
+# One-shot query into a persistent session:
+mdss chat --db ./notes --session my-session-id "What about database failover?"
+```
+
+- **Offline Extractive Synthesis**: Zero-dependency default extracts salient factual sentences with strict `[file › heading]` source provenance.
+- **Persistent Sessions**: Multi-turn sessions are stored in `<indexDir>/sessions/<id>.json` preserving full conversation turns, retrieved chunk provenance manifests (file, heading, exact line numbers, chunk hashes), and model metadata.
+- **Context Carry-Over**: Follow-up turns re-rank and expand query tokens using prior conversational context.
+- **Interactive REPL Commands**: `/session` (info), `/sources` (manifest of last turn), `/help`, `/exit`.
+
+### 4. Serve (daemon, warm index + model)
 
 ```bash
 mdss serve --db /path/to/your/markdown [--port 8747] [--host 127.0.0.1] [--watch]
@@ -420,11 +456,76 @@ const { results } = await r.json();
 ```
 
 The API is unauthenticated by default (or secured via `--api-key`) — by default it binds to **loopback only**
-(`127.0.0.1`), so other machines on your LAN cannot reach it. Pass `--host
-0.0.0.0` (or `MDSS_HOST`) only if you intend to expose it on a network, and put
-a reverse proxy (or at least a firewall rule) in front of it. Request bodies
-are capped at 64 KB (oversized → `413`), and malformed JSON gets a clear `400`
-instead of being silently swallowed.
+(`127.0.0.1`), so other machines on your LAN cannot reach it. Passing `--host 0.0.0.0` (or `MDSS_HOST`)
+without authentication **refuses to start** — set an API key, or opt in explicitly with `--allow-unsecured`
+(see [Securing remote deployments](#securing-remote-deployments)). Request bodies are capped at 64 KB
+(oversized → `413`), malformed JSON gets a clear `400`, per-client request rate is capped at 60/min with a
+burst of 10 (`429 + Retry-After` above that; tune via `--rate-limit`), and at most `CPU count` searches run
+concurrently with a bounded wait queue (`503` when saturated; tune via `--max-concurrency`). Every response
+carries `X-Content-Type-Options: nosniff` and `Cache-Control: no-store`; the `Host` header is validated
+against a loopback allowlist (`403` otherwise) so DNS-rebinding pages cannot read your notes cross-origin,
+and CORS stays fully off unless you pass `--cors-origin`. `/health` exposes the live counters
+(`in_flight`, `queued`, `rejected_total`).
+
+### Web UI (built-in, `--ui`)
+
+By default `mdss serve` also serves a **zero-dependency web UI at `/`** — a keyboard-first search page
+with live debounced results, term highlighting, score bars, tag/type/status filters, dark/light theme
+(follows the system until you toggle it) and an API-key prompt when Bearer auth is enabled (the key is kept
+in `sessionStorage`, same-origin only). It ships as plain HTML/CSS/JS split into three assets so a strict
+CSP applies (`default-src 'none'; script-src 'self'; style-src 'self'; connect-src 'self';
+frame-ancestors 'none'`) — no inline code, no external calls. Use `--no-ui` to restore pure JSON at `/`;
+the JSON contract (`/search`, `/health`, `/help`) never changes. A Playwright smoke test runs weekly via
+the opt-in `ui-smoke` workflow.
+
+### Securing remote deployments
+
+`mdss serve` is designed to be private-first: loopback bind by default, refuse-to-start for unauthenticated
+network binds, DNS-rebinding protection, and rate limiting built in. To serve a team over the network safely:
+
+1. **Keep TLS + auth at a reverse proxy** — mdss speaks plain HTTP on loopback; nginx/caddy terminate TLS.
+
+   ```nginx
+   # /etc/nginx/sites-available/mdss — TLS termination + Bearer-auth passthrough
+   server {
+     listen 443 ssl;
+     server_name search.example.com;
+     ssl_certificate     /etc/letsencrypt/live/search.example.com/fullchain.pem;
+     ssl_certificate_key /etc/letsencrypt/live/search.example.com/privkey.pem;
+
+     location / {
+       proxy_pass http://127.0.0.1:8747;
+       proxy_set_header Host $host;            # must match --allowed-host if not loopback-named
+       proxy_set_header X-Forwarded-For $remote_addr;
+     }
+   }
+   ```
+
+   ```caddyfile
+   # Caddyfile — automatic HTTPS
+   search.example.com {
+     reverse_proxy 127.0.0.1:8747
+   }
+   ```
+
+2. **Set an API key without leaking it into shell history or CI logs:**
+
+   ```bash
+   echo "$RANDOM_KEY" > /etc/mdss/api-key && chmod 600 /etc/mdss/api-key
+   mdss serve --db /srv/wiki --host 127.0.0.1 --api-key-file /etc/mdss/api-key
+   # or MDSS_API_KEY_FILE=/etc/mdss/api-key
+   ```
+
+3. **If you must expose mdss directly** (no proxy), require the key *and* keep the firewall tight:
+
+   ```bash
+   MDSS_API_KEY=$(openssl rand -hex 32) mdss serve --db /srv/wiki --host 0.0.0.0
+   ```
+
+   The startup banner warns loudly whenever an unsecured non-loopback bind was explicitly allowed.
+
+4. **Browser clients on another origin?** Opt into CORS explicitly, exact origins only:
+   `--cors-origin https://intranet.example.com`.
 
 ### 4. MCP Server (Claude Desktop, Cursor, Copilot, Antigravity)
 
@@ -434,11 +535,35 @@ instead of being silently swallowed.
 mdss mcp --db /path/to/your/markdown
 ```
 
+**Streamable HTTP transport (issue #123):** a long-running daemon can expose
+the same tools over HTTP — `mdss serve --db ./notes --mcp` mounts `/mcp`
+(POST JSON-RPC, SSE channel on GET, `Mcp-Session-Id` sessions, protocol
+`2025-03-26`). It sits behind the exact same auth, Host-allowlist and
+rate-limit gates as `/search`, so remote agents inherit the hardening. stdio
+via `mdss mcp` remains the default transport.
+
 #### Available MCP Tools:
-- **`search_markdown`**: Hybrid semantic vector + BM25 search with optional `path` glob and `tag` filters.
+- **`search_markdown`**: Hybrid semantic vector + BM25 search with optional `path` glob, `tag`, rich `filter`, and `graphBoost`.
 - **`get_chunk`**: Retrieve a specific section by file and heading.
+- **`get_lines`**: Retrieve exact line spans (`fromLine`, `maxLines`) from an indexed Markdown document.
+- **`related_notes`**: Find graph-connected (outgoing, backlinks, 2-hop) and semantically related notes for a document.
 - **`list_files`**: List indexed Markdown files with chunk counts.
 - **`index_status`**: Inspect index health, chunk counts, and model metadata.
+- **`ask_knowledge_base`**: Grounded direct Q&A synthesis with citations.
+- *All tools return `structuredContent` alongside text and include `readOnly: true, idempotent: true` annotations.*
+
+#### Available MCP Resources & Templates:
+- **`mdss://note/{path}`**: Read full content of an indexed Markdown document.
+- **`mdss://note/{path}?fromLine=1&maxLines=50`**: Addressable line-span slices of notes.
+- **`mdss://vault/{vault}`**: List all indexed notes within a vault.
+- **`mdss://status`**: Live index status and chunk count statistics.
+
+#### Available MCP Prompts:
+- **`search-and-cite`**: Synthesizes a grounded answer referencing `mdss://note/{path}` resource URIs.
+- **`summarize-note`**: Produces a structured summary of core concepts and linked notes.
+- **`compare-notes`**: Identifies agreements, divergences, and synergies across two notes.
+- **`find-contradictions`**: Audits notes for conflicting statements or outdated ADRs.
+- **`timeline`**: Builds a chronological milestone / event timeline from notes.
 
 #### Claude Desktop Configuration:
 Add to your `claude_desktop_config.json`:
@@ -485,6 +610,9 @@ Add to your `claude_desktop_config.json`:
 | `--k <n>` | Number of results, positive integer (default 6). |
 | `--json` | Machine-readable output: `index` → build result JSON, `stats` → index stats JSON, `search` → hit list JSON (each hit includes `matches` — the query terms found in the chunk). |
 | `--semantic` | `search` only: pure vector ranking, skip lexical/RRF fusion (zero lexical tokenization). |
+| `--open [N]` | `search`: open the top (or Nth) hit in your editor at its `startLine` — `$MDSS_EDITOR`/`$VISUAL`/`$EDITOR`, VS Code `--goto`, or the GUI opener. With `--json` on a non-TTY prints the intent instead of launching (issue #110). |
+| `--fix` | `check`: apply safe auto-repairs — stale `.mdss.lock`, SHA-mismatched `vectors.bin` sidecar, corrupt `ivf.json`; rebuilds a corrupt index / recomputes `.hashes.json` when `--db` is available. Never touches source notes (issue #117). |
+| `--dry-run` | `check --fix`: print the planned repairs without mutating anything. |
 | `--rerank` | Re-rank the candidate pool with a cross-encoder (`Xenova/bge-reranker-base`, ~280 MB model, downloaded on first use). Slower, sharper results — see *Reranking* below. |
 | `--graph-boost <n>` | Weight for Obsidian wikilink graph ranking and PageRank prior (search; default 0). Incorporates document centrality and 2-hop contextual propagation. |
 | `--filter <expr>` | Rich boolean filter across frontmatter tags and properties (e.g. `tag:engineering AND status != archived`). |
@@ -492,12 +620,50 @@ Add to your `claude_desktop_config.json`:
 | `--nprobe <n>` | Number of nearest centroid clusters to probe during ANN search (default 8). |
 | `--port <n>` | HTTP port for `serve` (default 8747, or `MDSS_PORT`). |
 | `--host <ip>` | Bind address for `serve` (default `127.0.0.1` — loopback only; use `0.0.0.0` to expose on the LAN, or `MDSS_HOST`). |
+| `--api-key <key>` | `serve`: require Bearer auth on every request (`MDSS_API_KEY`). |
+| `--api-key-file <path>` | `serve`: read the API key from a file instead of a literal value — avoids shell-history / CI-log leaks (`MDSS_API_KEY_FILE`; trailing newline trimmed, POSIX permissions checked). |
+| `--allow-unsecured` | `serve`: explicit opt-in to bind non-loopback **without** auth; without it (and without an API key) such a bind refuses to start (issue #121). |
+| `--allowed-host <h>` | `serve`: extra `Host` header value to accept (repeatable); anything else gets `403` — DNS-rebinding guard (issue #120). |
+| `--cors-origin <o>` | `serve`: allow cross-origin browser access from this exact origin (repeatable); CORS is off unless set (issue #120). |
+| `--rate-limit <n>` | `serve`: max `/search` requests per minute per client, burst 10 → `429 + Retry-After` above it; `0` disables (default 60, `MDSS_RATE_LIMIT`; issue #119). |
+| `--max-concurrency <n>` | `serve`: max concurrent in-flight searches; queue cap 2× → `503` when full; `0` disables (default = CPU count, `MDSS_MAX_CONCURRENCY`; issue #119). |
 | `--watch` | `serve`: re-index incrementally when files change. |
+| `--watch-debug` | `serve --watch`: verbose trace of polls, FS-error classifications and retries (issue #116). |
+| `--recency <days>` | `search`: time-decay half-life — passages get a `0.5^(age/halfLife)` boost after fusion; age from frontmatter `created`/`updated`, then file mtime; surfaced in `explain` (issue #127). |
+| `--embedder <name>` | Use an external embedding provider (`ollama` or `openai`) instead of local transformers.js, together with `--embedder-model` (+ `--embedder-base-url`, `--embedder-key-file`). The provider id/model/dim are recorded in the index — switching providers invalidates vectors with the usual full re-embed. Local remains the zero-config default and the only fully-offline path (issue #124). |
+| `--no-query-cache` | `search`: skip the persistent query-embedding cache — repeated queries in fresh processes re-embed otherwise (issue #114). |
+| `--no-ui` | `serve`: disable the built-in web UI at `/` — restores JSON API help there (issue #111). |
 | `--watch-interval <ms>` | `serve --watch`: poll every N ms (default 3000). |
 | `--watch-delay <ms>` | `serve --watch`: quiet-period debounce before a burst of saves triggers ONE re-index (default 1000; issue #42). |
 | `--offline` | Never download the model — require a cached one (or `MDSS_OFFLINE=1`). |
 | `--version` | Print the installed version. |
 
+### Shell completions
+
+`mdss completions <bash|zsh|fish|powershell>` prints a self-contained completion
+script covering every subcommand and flag, with directory completion for
+`--db` / `--index-dir` / `--cache-dir` / `--vault` and file completion for
+`--output` / `--config` / `--api-key-file`:
+
+```bash
+# bash — current session only:
+eval "$(mdss completions bash)"
+# bash — persistent:
+mdss completions bash > ~/.local/share/bash-completions/completions/mdss
+
+# zsh:
+mkdir -p ~/.zfunc && mdss completions zsh > ~/.zfunc/_mdss   # fpath before compinit
+
+# fish:
+mdss completions fish > ~/.config/fish/completions/mdss.fish
+
+# PowerShell — paste into your $PROFILE:
+mdss completions powershell >> $PROFILE
+```
+
+A unit test guards against flag drift: the generated tables must match the
+flags `parseArgs` actually accepts.
+
 ### The base can live outside the project
 
 The index does not need write access to your notes if they're read-only — just
@@ -517,6 +683,13 @@ mdss index
 mdss search "incident runbook for db failover"
 ```
 
+**Supported layouts** (hardened by the issue #118 suite): deep trees beyond
+Windows MAX_PATH (>260 chars) work; unicode and spaces in names work;
+`--index-dir` may live on a different drive than `--db`; NTFS junction loops
+terminate the walker (bounded by path length); `\\?\` extended-length roots are
+accepted for `--db`/`--index-dir`; remote UNC shares (`\\server\share`) are
+**rejected** by design (local-only security boundary).
+
 ### Configuration file (`.mdssrc` / `mdss.config.json`)
 
 Instead of repeating CLI options on every command, create a `.mdssrc.json` or `mdss.config.json` file.
@@ -650,6 +823,23 @@ is an in-memory dot-product sweep.
 
 - **Obsidian**: An official plugin is available in [`integrations/obsidian`](./integrations/obsidian) providing a live semantic search panel directly in your vault connected to `mdss serve`.
 - **Model Context Protocol (MCP)**: Native stdio JSON-RPC 2.0 server (`mdss mcp`) integrating with Claude Desktop, Cursor, Copilot, Antigravity, and AI agents for local knowledge retrieval.
+- **VS Code / Raycast / Alfred**: see [`integrations/README.md`](./integrations/README.md). Releases are automated — tagging `integrations/vscode/vX.Y.Z` builds and attaches the VSIX to a GitHub Release from source (no binaries are committed; issue #112).
+- **Windows tray icon**: [`integrations/windows/`](./integrations/windows/) — a zero-dependency system-tray companion for `mdss serve` (pure PowerShell + .NET WinForms, built into Windows): state-colored icon, indexing stats on hover (chunks / model / index age), a balloon when your notes get re-indexed. One-command start:
+
+```cmd
+integrations\windows\start-mdss.cmd D:\Notes
+```
+
+## Contributing
+
+Contributions are welcome — bug reports with reproduction steps, docs fixes,
+tests, and features all help. Start with
+[CONTRIBUTING.md](./CONTRIBUTING.md) (dev setup, project tour, conventions)
+and look for issues labeled **`good first issue`** — they are scoped for a
+first-time contributor. For the internals — module map, data-flow diagrams,
+on-disk formats, and extension points — see
+[ARCHITECTURE.md](./ARCHITECTURE.md). By participating you agree to the
+[Code of Conduct](./CODE_OF_CONDUCT.md).
 
 ## License
 

package/RESEARCH.md

@@ -226,7 +226,29 @@ If no candidate meets the multilingual non-inferiority gate, `bge-reranker-base`
 - **Trigger:** WebAssembly SIMD dot-product kernels will be considered **only if** CPU profiling on >50,000 chunks shows JS float multiplication accounts for >40% of warm search latency.
 - **Constraint:** Zero native Node C++ bindings; must maintain pure JS fallback for non-WASM environments.
 
-### ANN / IVF Scale Threshold (Issue #32)
+### ANN / IVF Scale Threshold & K-Means++ Seeding (Issues #32 & #139)
 - **Baseline:** Exact $O(N \cdot d)$ linear sweep.
-- **Trigger:** ANN (Inverted File / HNSW) indexing is a scale contingency. It shall **not** be introduced until corpus size exceeds **100,000+ chunks** and exact warm sweep latency exceeds 50 ms.
-- **Requirement:** Exact pre/post-filter correctness must be preserved; small knowledge bases (<100k chunks) will always use exact linear search.
+- **K-Means++ Seeding:** Uses deterministic pseudo-random $D^2$-weighted centroid seeding (Mulberry32) to prevent initial centroid collapse on correlated notes corpora.
+- **Measured Quality Gate:** `bench/ann-recall.mjs` benchmarks Recall@10 across $nprobe \in \{1, 4, 8, 16\}$ on clustered corpora, demonstrating $\ge 99.5\%$ Recall@10 at default $nprobe=8$ with sub-millisecond candidate pruning.
+- **Trigger:** ANN indexing is auto-engaged above `ANN_THRESHOLD = 500` chunks when `--ann` is passed or IVF index is persisted, while exact pre/post-filter correctness is strictly preserved.
+
+---
+
+## 14. Matryoshka Representation Learning (MRL) Dimension Selection (Issue #143)
+
+### Mechanism
+Models trained with Matryoshka Representation Learning (MRL) loss (such as `onnx-community/Qwen3-Embedding-0.6B-ONNX` and `Xenova/bge-m3`) embed semantic information hierarchically, allowing vectors to be sliced at predefined sub-dimensions (`32`, `64`, `128`, `256`, `512`, `768`, `1024`) and re-normalized to unit L2 norm without significant loss in semantic retrieval fidelity.
+
+### Benefits & Trade-offs
+- **Storage & RAM Savings:** 1024 → 256 dimensions reduces `vectors.bin` size and memory consumption by 4× (from ~19.5 MB to ~4.88 MB per 5,000 chunks).
+- **Sweep Latency:** Dot-product cosine sweep latency is halved (e.g. from ~14.2 ms to ~6.8 ms for 5,000 chunks in single-threaded pure JS).
+- **Integrity & Cache Invalidation:** `embeddingAdapterFingerprint` binds the selected dimension (`model.dim`). Changing `--dimensions` automatically invalidates previous chunk hashes and forces clean re-indexing rather than silent vector dimension mismatch.
+
+### Usage
+```bash
+# Build index truncated to 256 dims (4x smaller vectors)
+mdss index --db ./notes --model qwen3-embedding-0.6b --dimensions 256
+
+# Search with dimension safety validation
+mdss search --db ./notes --dimensions 256 "query"
+```