md-semantic-search 1.0.2 → 1.0.3

package/README.md

@@ -109,6 +109,9 @@ measurements that shaped its defaults.
 
 ## Install
 
+Requires **Node.js ≥ 18**. Native dependencies ship as prebuilt binaries — no
+compiler toolchain needed on any platform.
+
 Run on demand with `npx` (no install):
 
 ```bash
@@ -131,6 +134,9 @@ npm install
 node bin/cli.mjs --help
 ```
 
+For headless server deployments there is a Docker path — see
+[docs/deploy.md](./docs/deploy.md).
+
 ---
 
 ## Library usage
@@ -177,7 +183,14 @@ indexes and existing library consumers; schema-v3 search uses persisted BM25.
 mdss index --db /path/to/your/markdown
 ```
 
-First run downloads the model (~280 MB). The index is written to `<db>/.mdss/`
+First run downloads the model (~280 MB) — and the download itself is
+**visible**: an aggregated bar on stderr shows percent across every file of the
+model directory, live MB/s and ETA (the same bar reappears when `--rerank`
+first pulls the cross-encoder). Under `--json` the bar is replaced by throttled
+`{"type":"download-progress", …}` / `{"type":"download-complete", …}` events
+**on stderr**, so stdout stays machine-readable. A warm cache produces no
+download output at all, and `--offline` behaves exactly as before. The index
+is written to `<db>/.mdss/`
 by default (override with `--index-dir`). Re-run after editing your notes — it's
 incremental at two levels: unchanged files are skipped via their md5, and inside
 a changed file only the sections whose text actually changed are re-embedded
@@ -420,11 +433,76 @@ const { results } = await r.json();
 ```
 
 The API is unauthenticated by default (or secured via `--api-key`) — by default it binds to **loopback only**
-(`127.0.0.1`), so other machines on your LAN cannot reach it. Pass `--host
-0.0.0.0` (or `MDSS_HOST`) only if you intend to expose it on a network, and put
-a reverse proxy (or at least a firewall rule) in front of it. Request bodies
-are capped at 64 KB (oversized → `413`), and malformed JSON gets a clear `400`
-instead of being silently swallowed.
+(`127.0.0.1`), so other machines on your LAN cannot reach it. Passing `--host 0.0.0.0` (or `MDSS_HOST`)
+without authentication **refuses to start** — set an API key, or opt in explicitly with `--allow-unsecured`
+(see [Securing remote deployments](#securing-remote-deployments)). Request bodies are capped at 64 KB
+(oversized → `413`), malformed JSON gets a clear `400`, per-client request rate is capped at 60/min with a
+burst of 10 (`429 + Retry-After` above that; tune via `--rate-limit`), and at most `CPU count` searches run
+concurrently with a bounded wait queue (`503` when saturated; tune via `--max-concurrency`). Every response
+carries `X-Content-Type-Options: nosniff` and `Cache-Control: no-store`; the `Host` header is validated
+against a loopback allowlist (`403` otherwise) so DNS-rebinding pages cannot read your notes cross-origin,
+and CORS stays fully off unless you pass `--cors-origin`. `/health` exposes the live counters
+(`in_flight`, `queued`, `rejected_total`).
+
+### Web UI (built-in, `--ui`)
+
+By default `mdss serve` also serves a **zero-dependency web UI at `/`** — a keyboard-first search page
+with live debounced results, term highlighting, score bars, tag/type/status filters, dark/light theme
+(follows the system until you toggle it) and an API-key prompt when Bearer auth is enabled (the key is kept
+in `sessionStorage`, same-origin only). It ships as plain HTML/CSS/JS split into three assets so a strict
+CSP applies (`default-src 'none'; script-src 'self'; style-src 'self'; connect-src 'self';
+frame-ancestors 'none'`) — no inline code, no external calls. Use `--no-ui` to restore pure JSON at `/`;
+the JSON contract (`/search`, `/health`, `/help`) never changes. A Playwright smoke test runs weekly via
+the opt-in `ui-smoke` workflow.
+
+### Securing remote deployments
+
+`mdss serve` is designed to be private-first: loopback bind by default, refuse-to-start for unauthenticated
+network binds, DNS-rebinding protection, and rate limiting built in. To serve a team over the network safely:
+
+1. **Keep TLS + auth at a reverse proxy** — mdss speaks plain HTTP on loopback; nginx/caddy terminate TLS.
+
+   ```nginx
+   # /etc/nginx/sites-available/mdss — TLS termination + Bearer-auth passthrough
+   server {
+     listen 443 ssl;
+     server_name search.example.com;
+     ssl_certificate     /etc/letsencrypt/live/search.example.com/fullchain.pem;
+     ssl_certificate_key /etc/letsencrypt/live/search.example.com/privkey.pem;
+
+     location / {
+       proxy_pass http://127.0.0.1:8747;
+       proxy_set_header Host $host;            # must match --allowed-host if not loopback-named
+       proxy_set_header X-Forwarded-For $remote_addr;
+     }
+   }
+   ```
+
+   ```caddyfile
+   # Caddyfile — automatic HTTPS
+   search.example.com {
+     reverse_proxy 127.0.0.1:8747
+   }
+   ```
+
+2. **Set an API key without leaking it into shell history or CI logs:**
+
+   ```bash
+   echo "$RANDOM_KEY" > /etc/mdss/api-key && chmod 600 /etc/mdss/api-key
+   mdss serve --db /srv/wiki --host 127.0.0.1 --api-key-file /etc/mdss/api-key
+   # or MDSS_API_KEY_FILE=/etc/mdss/api-key
+   ```
+
+3. **If you must expose mdss directly** (no proxy), require the key *and* keep the firewall tight:
+
+   ```bash
+   MDSS_API_KEY=$(openssl rand -hex 32) mdss serve --db /srv/wiki --host 0.0.0.0
+   ```
+
+   The startup banner warns loudly whenever an unsecured non-loopback bind was explicitly allowed.
+
+4. **Browser clients on another origin?** Opt into CORS explicitly, exact origins only:
+   `--cors-origin https://intranet.example.com`.
 
 ### 4. MCP Server (Claude Desktop, Cursor, Copilot, Antigravity)
 
@@ -434,6 +512,13 @@ instead of being silently swallowed.
 mdss mcp --db /path/to/your/markdown
 ```
 
+**Streamable HTTP transport (issue #123):** a long-running daemon can expose
+the same tools over HTTP — `mdss serve --db ./notes --mcp` mounts `/mcp`
+(POST JSON-RPC, SSE channel on GET, `Mcp-Session-Id` sessions, protocol
+`2025-03-26`). It sits behind the exact same auth, Host-allowlist and
+rate-limit gates as `/search`, so remote agents inherit the hardening. stdio
+via `mdss mcp` remains the default transport.
+
 #### Available MCP Tools:
 - **`search_markdown`**: Hybrid semantic vector + BM25 search with optional `path` glob and `tag` filters.
 - **`get_chunk`**: Retrieve a specific section by file and heading.
@@ -485,6 +570,9 @@ Add to your `claude_desktop_config.json`:
 | `--k <n>` | Number of results, positive integer (default 6). |
 | `--json` | Machine-readable output: `index` → build result JSON, `stats` → index stats JSON, `search` → hit list JSON (each hit includes `matches` — the query terms found in the chunk). |
 | `--semantic` | `search` only: pure vector ranking, skip lexical/RRF fusion (zero lexical tokenization). |
+| `--open [N]` | `search`: open the top (or Nth) hit in your editor at its `startLine` — `$MDSS_EDITOR`/`$VISUAL`/`$EDITOR`, VS Code `--goto`, or the GUI opener. With `--json` on a non-TTY prints the intent instead of launching (issue #110). |
+| `--fix` | `check`: apply safe auto-repairs — stale `.mdss.lock`, SHA-mismatched `vectors.bin` sidecar, corrupt `ivf.json`; rebuilds a corrupt index / recomputes `.hashes.json` when `--db` is available. Never touches source notes (issue #117). |
+| `--dry-run` | `check --fix`: print the planned repairs without mutating anything. |
 | `--rerank` | Re-rank the candidate pool with a cross-encoder (`Xenova/bge-reranker-base`, ~280 MB model, downloaded on first use). Slower, sharper results — see *Reranking* below. |
 | `--graph-boost <n>` | Weight for Obsidian wikilink graph ranking and PageRank prior (search; default 0). Incorporates document centrality and 2-hop contextual propagation. |
 | `--filter <expr>` | Rich boolean filter across frontmatter tags and properties (e.g. `tag:engineering AND status != archived`). |
@@ -492,12 +580,50 @@ Add to your `claude_desktop_config.json`:
 | `--nprobe <n>` | Number of nearest centroid clusters to probe during ANN search (default 8). |
 | `--port <n>` | HTTP port for `serve` (default 8747, or `MDSS_PORT`). |
 | `--host <ip>` | Bind address for `serve` (default `127.0.0.1` — loopback only; use `0.0.0.0` to expose on the LAN, or `MDSS_HOST`). |
+| `--api-key <key>` | `serve`: require Bearer auth on every request (`MDSS_API_KEY`). |
+| `--api-key-file <path>` | `serve`: read the API key from a file instead of a literal value — avoids shell-history / CI-log leaks (`MDSS_API_KEY_FILE`; trailing newline trimmed, POSIX permissions checked). |
+| `--allow-unsecured` | `serve`: explicit opt-in to bind non-loopback **without** auth; without it (and without an API key) such a bind refuses to start (issue #121). |
+| `--allowed-host <h>` | `serve`: extra `Host` header value to accept (repeatable); anything else gets `403` — DNS-rebinding guard (issue #120). |
+| `--cors-origin <o>` | `serve`: allow cross-origin browser access from this exact origin (repeatable); CORS is off unless set (issue #120). |
+| `--rate-limit <n>` | `serve`: max `/search` requests per minute per client, burst 10 → `429 + Retry-After` above it; `0` disables (default 60, `MDSS_RATE_LIMIT`; issue #119). |
+| `--max-concurrency <n>` | `serve`: max concurrent in-flight searches; queue cap 2× → `503` when full; `0` disables (default = CPU count, `MDSS_MAX_CONCURRENCY`; issue #119). |
 | `--watch` | `serve`: re-index incrementally when files change. |
+| `--watch-debug` | `serve --watch`: verbose trace of polls, FS-error classifications and retries (issue #116). |
+| `--recency <days>` | `search`: time-decay half-life — passages get a `0.5^(age/halfLife)` boost after fusion; age from frontmatter `created`/`updated`, then file mtime; surfaced in `explain` (issue #127). |
+| `--embedder <name>` | Use an external embedding provider (`ollama` or `openai`) instead of local transformers.js, together with `--embedder-model` (+ `--embedder-base-url`, `--embedder-key-file`). The provider id/model/dim are recorded in the index — switching providers invalidates vectors with the usual full re-embed. Local remains the zero-config default and the only fully-offline path (issue #124). |
+| `--no-query-cache` | `search`: skip the persistent query-embedding cache — repeated queries in fresh processes re-embed otherwise (issue #114). |
+| `--no-ui` | `serve`: disable the built-in web UI at `/` — restores JSON API help there (issue #111). |
 | `--watch-interval <ms>` | `serve --watch`: poll every N ms (default 3000). |
 | `--watch-delay <ms>` | `serve --watch`: quiet-period debounce before a burst of saves triggers ONE re-index (default 1000; issue #42). |
 | `--offline` | Never download the model — require a cached one (or `MDSS_OFFLINE=1`). |
 | `--version` | Print the installed version. |
 
+### Shell completions
+
+`mdss completions <bash|zsh|fish|powershell>` prints a self-contained completion
+script covering every subcommand and flag, with directory completion for
+`--db` / `--index-dir` / `--cache-dir` / `--vault` and file completion for
+`--output` / `--config` / `--api-key-file`:
+
+```bash
+# bash — current session only:
+eval "$(mdss completions bash)"
+# bash — persistent:
+mdss completions bash > ~/.local/share/bash-completions/completions/mdss
+
+# zsh:
+mkdir -p ~/.zfunc && mdss completions zsh > ~/.zfunc/_mdss   # fpath before compinit
+
+# fish:
+mdss completions fish > ~/.config/fish/completions/mdss.fish
+
+# PowerShell — paste into your $PROFILE:
+mdss completions powershell >> $PROFILE
+```
+
+A unit test guards against flag drift: the generated tables must match the
+flags `parseArgs` actually accepts.
+
 ### The base can live outside the project
 
 The index does not need write access to your notes if they're read-only — just
@@ -517,6 +643,13 @@ mdss index
 mdss search "incident runbook for db failover"
 ```
 
+**Supported layouts** (hardened by the issue #118 suite): deep trees beyond
+Windows MAX_PATH (>260 chars) work; unicode and spaces in names work;
+`--index-dir` may live on a different drive than `--db`; NTFS junction loops
+terminate the walker (bounded by path length); `\\?\` extended-length roots are
+accepted for `--db`/`--index-dir`; remote UNC shares (`\\server\share`) are
+**rejected** by design (local-only security boundary).
+
 ### Configuration file (`.mdssrc` / `mdss.config.json`)
 
 Instead of repeating CLI options on every command, create a `.mdssrc.json` or `mdss.config.json` file.
@@ -650,6 +783,21 @@ is an in-memory dot-product sweep.
 
 - **Obsidian**: An official plugin is available in [`integrations/obsidian`](./integrations/obsidian) providing a live semantic search panel directly in your vault connected to `mdss serve`.
 - **Model Context Protocol (MCP)**: Native stdio JSON-RPC 2.0 server (`mdss mcp`) integrating with Claude Desktop, Cursor, Copilot, Antigravity, and AI agents for local knowledge retrieval.
+- **VS Code / Raycast / Alfred**: see [`integrations/README.md`](./integrations/README.md). Releases are automated — tagging `integrations/vscode/vX.Y.Z` builds and attaches the VSIX to a GitHub Release from source (no binaries are committed; issue #112).
+- **Windows tray icon**: [`integrations/windows/`](./integrations/windows/) — a zero-dependency system-tray companion for `mdss serve` (pure PowerShell + .NET WinForms, built into Windows): state-colored icon, indexing stats on hover (chunks / model / index age), a balloon when your notes get re-indexed. One-command start:
+
+```cmd
+integrations\windows\start-mdss.cmd D:\Notes
+```
+
+## Contributing
+
+Contributions are welcome — bug reports with reproduction steps, docs fixes,
+tests, and features all help. Start with
+[CONTRIBUTING.md](./CONTRIBUTING.md) (dev setup, project tour, conventions)
+and look for issues labeled **`good first issue`** — they are scoped for a
+first-time contributor. By participating you agree to the
+[Code of Conduct](./CODE_OF_CONDUCT.md).
 
 ## License