md-semantic-search 0.4.0 → 1.0.1

package/README.md

@@ -255,6 +255,52 @@ error, and `serve --watch` politely defers its cycle until the other writer
 finishes. A crashed writer can't wedge the index — the lock self-heals by
 pid-liveness + staleness (a dead-pid or abandoned lock is reclaimed).
 
+### Export the index (`mdss export`)
+
+Export chunks and decoded embedding vectors to JSONL, CSV, or Parquet for downstream pipelines (LangChain, LlamaIndex, Qdrant, Pandas):
+
+```bash
+# Export all chunks with decoded float vectors as JSONL:
+mdss export --db ./notes --format jsonl > chunks.jsonl
+
+# Export metadata only without vectors (fast & compact):
+mdss export --db ./notes --format jsonl --no-vectors > chunks_meta.jsonl
+
+# Export as CSV (metadata only):
+mdss export --db ./notes --format csv > chunks.csv
+
+# Export directly to file:
+mdss export --db ./notes --format jsonl --output ./data/corpus.jsonl
+```
+
+#### JSONL output schema:
+```json
+{
+  "file": "runbooks/failover.md",
+  "title": "Database Failover",
+  "heading": "Step 2: Promote Replica",
+  "headingPath": ["Database Failover", "Step 2: Promote Replica"],
+  "text": "Run the promote script...",
+  "vec": [0.012, -0.045, 0.089],
+  "chunkHash": "a1b2c3...",
+  "startLine": 14,
+  "endLine": 28
+}
+```
+
+#### Downstream integration example (Python / LangChain / LlamaIndex):
+```python
+import json
+
+# Load into custom vector store / pipeline
+with open("chunks.jsonl") as f:
+    for line in f:
+        chunk = json.loads(line)
+        text = chunk["text"]
+        vector = chunk["vec"]
+        meta = {"file": chunk["file"], "heading": chunk["heading"]}
+```
+
 ### Diagnose problems (`mdss check` / `mdss doctor`)
 
 `mdss check` (alias: `mdss doctor`) is an **offline, read-only** diagnostic for
@@ -373,21 +419,66 @@ const r = await fetch('http://localhost:8747/search', {
 const { results } = await r.json();
 ```
 
-The API is unauthenticated — by default it binds to **loopback only**
+The API is unauthenticated by default (or secured via `--api-key`) — by default it binds to **loopback only**
 (`127.0.0.1`), so other machines on your LAN cannot reach it. Pass `--host
 0.0.0.0` (or `MDSS_HOST`) only if you intend to expose it on a network, and put
 a reverse proxy (or at least a firewall rule) in front of it. Request bodies
 are capped at 64 KB (oversized → `413`), and malformed JSON gets a clear `400`
 instead of being silently swallowed.
 
+### 4. MCP Server (Claude Desktop, Cursor, Copilot, Antigravity)
+
+`md-semantic-search` provides a zero-dependency **Model Context Protocol (MCP)** server over standard I/O (JSON-RPC 2.0). This allows AI agents and IDEs to query your local notes privately by meaning without uploading text to external clouds.
+
+```bash
+mdss mcp --db /path/to/your/markdown
+```
+
+#### Available MCP Tools:
+- **`search_markdown`**: Hybrid semantic vector + BM25 search with optional `path` glob and `tag` filters.
+- **`get_chunk`**: Retrieve a specific section by file and heading.
+- **`list_files`**: List indexed Markdown files with chunk counts.
+- **`index_status`**: Inspect index health, chunk counts, and model metadata.
+
+#### Claude Desktop Configuration:
+Add to your `claude_desktop_config.json`:
+
+```json
+{
+  "mcpServers": {
+    "md-semantic-search": {
+      "command": "npx",
+      "args": ["-y", "md-semantic-search", "mcp", "--db", "C:\\path\\to\\your\\notes"]
+    }
+  }
+}
+```
+
+#### Cursor Configuration (`.cursor/mcp.json`):
+```json
+{
+  "mcpServers": {
+    "md-semantic-search": {
+      "command": "npx",
+      "args": ["-y", "md-semantic-search", "mcp", "--db", "./notes"]
+    }
+  }
+}
+```
+
 ### Options
 
 | Flag | Meaning |
 |------|---------|
+| `--config <file>` | Path to config file (default: `.mdssrc.json`, `mdss.config.json`, `.mdssrc`). |
 | `--db <dir>` | Folder of `.md` files (or set `MDSS_DB`). Can be anywhere on disk. |
 | `--index-dir <dir>` | Where to store the index (default: `<db>/.mdss`). |
 | `--cache-dir <dir>` | Model cache dir. Default: `$XDG_CACHE_HOME/mdss` if set, else `~/.cache/mdss` on **all** platforms (so on Windows that's `C:\Users\<you>\.cache\mdss` — *not* `%LOCALAPPDATA%`; override with `MDSS_CACHE_DIR`). |
 | `--model <name\|id>` | Embedding model (default `e5-base`). See `mdss models`. |
+| `--workers <n>` | Number of parallel batch workers for embedding (default: 1). |
+| `--format <fmt>` | Export format: `jsonl` (default), `csv`, `parquet` (`export`). |
+| `--no-vectors` | Omit embeddings from export (`export`). |
+| `--output <file>` | Output file path (`export`, default: stdout). |
 | `--ignore <glob>` | Skip files/paths; repeatable. e.g. `--ignore "log.md" --ignore "**/archive/**"`. |
 | `--path <glob>` | Search only files matching glob; repeatable. e.g. `--path "docs/**"`. |
 | `--since <date>` | Search only files modified at/after date (`YYYY-MM-DD` or ISO 8601). |
@@ -395,6 +486,10 @@ instead of being silently swallowed.
 | `--json` | Machine-readable output: `index` → build result JSON, `stats` → index stats JSON, `search` → hit list JSON (each hit includes `matches` — the query terms found in the chunk). |
 | `--semantic` | `search` only: pure vector ranking, skip lexical/RRF fusion (zero lexical tokenization). |
 | `--rerank` | Re-rank the candidate pool with a cross-encoder (`Xenova/bge-reranker-base`, ~280 MB model, downloaded on first use). Slower, sharper results — see *Reranking* below. |
+| `--graph-boost <n>` | Weight for Obsidian wikilink graph ranking and PageRank prior (search; default 0). Incorporates document centrality and 2-hop contextual propagation. |
+| `--filter <expr>` | Rich boolean filter across frontmatter tags and properties (e.g. `tag:engineering AND status != archived`). |
+| `--ann` | Approximate nearest neighbor search via IVF clustering (search; auto-enables for large corpora). |
+| `--nprobe <n>` | Number of nearest centroid clusters to probe during ANN search (default 8). |
 | `--port <n>` | HTTP port for `serve` (default 8747, or `MDSS_PORT`). |
 | `--host <ip>` | Bind address for `serve` (default `127.0.0.1` — loopback only; use `0.0.0.0` to expose on the LAN, or `MDSS_HOST`). |
 | `--watch` | `serve`: re-index incrementally when files change. |
@@ -422,6 +517,30 @@ mdss index
 mdss search "incident runbook for db failover"
 ```
 
+### Configuration file (`.mdssrc` / `mdss.config.json`)
+
+Instead of repeating CLI options on every command, create a `.mdssrc.json` or `mdss.config.json` file.
+`mdss` discovers config files by searching:
+1. An explicit path passed via `--config <path>` or the `MDSS_CONFIG` environment variable.
+2. Current working directory up through ancestor directories (`.mdssrc.json`, `mdss.config.json`, `.mdssrc`).
+3. User global config at `~/.mdssrc.json` or `~/.mdssrc`.
+
+CLI flags always override configuration values.
+
+Example `.mdssrc.json`:
+
+```json
+{
+  "db": "./notes",
+  "model": "e5-base",
+  "indexDir": "./notes/.mdss",
+  "ignore": ["archive/**", "drafts/**"],
+  "k": 6
+}
+```
+
+Run `mdss check` to validate your configuration file schema and detect unknown keys.
+
 ## Models
 
 ```bash
@@ -527,6 +646,11 @@ section instead of silently producing wrong vectors.
 No external services, no database — the whole index is one JSON file and search
 is an in-memory dot-product sweep.
 
+## Integrations
+
+- **Obsidian**: An official plugin is available in [`integrations/obsidian`](./integrations/obsidian) providing a live semantic search panel directly in your vault connected to `mdss serve`.
+- **Model Context Protocol (MCP)**: Native stdio JSON-RPC 2.0 server (`mdss mcp`) integrating with Claude Desktop, Cursor, Copilot, Antigravity, and AI agents for local knowledge retrieval.
+
 ## License
 
 MIT © chelslava