@itkoren/sqmd 0.1.0

package/README.md

@@ -0,0 +1,1052 @@
+# sqmd
+
+A fully local, high-performance semantic search engine for Markdown files. Index your notes, documentation, or any collection of `.md` / `.mdx` files and query them with natural language — no external API keys, no cloud services, no data leaving your machine.
+
+Designed to serve both humans (CLI + REST API) and AI agents (MCP server), with a RAG-ready output layer for use as an agent memory backend.
+
+---
+
+## Table of Contents
+
+- [Features](#features)
+- [Architecture Overview](#architecture-overview)
+- [Technology Stack](#technology-stack)
+- [Project Structure](#project-structure)
+- [Getting Started](#getting-started)
+  - [Prerequisites](#prerequisites)
+  - [Installation](#installation)
+  - [Initial Configuration](#initial-configuration)
+  - [First Index](#first-index)
+- [CLI Reference](#cli-reference)
+  - [index](#index)
+  - [search](#search)
+  - [serve](#serve)
+  - [mcp](#mcp)
+  - [status](#status)
+  - [config](#config)
+- [REST API](#rest-api)
+  - [Search](#search-endpoints)
+  - [Documents](#document-endpoints)
+  - [Index Management](#index-management-endpoints)
+  - [Health & Metrics](#health--metrics-endpoints)
+  - [Authentication](#authentication)
+- [MCP Server](#mcp-server)
+  - [Tools](#mcp-tools)
+  - [Resources](#mcp-resources)
+  - [Claude Desktop Integration](#claude-desktop-integration)
+- [RAG Layer](#rag-layer)
+- [Configuration Reference](#configuration-reference)
+- [Architecture Deep Dive](#architecture-deep-dive)
+  - [Chunking Algorithm](#chunking-algorithm)
+  - [Embedding Pipeline](#embedding-pipeline)
+  - [Hybrid Search & RRF](#hybrid-search--rrf)
+  - [Incremental Indexing](#incremental-indexing)
+  - [LanceDB Schema](#lancedb-schema)
+- [Embedding Backends](#embedding-backends)
+  - [Transformers.js (Default)](#transformersjs-default)
+  - [Ollama](#ollama)
+- [Performance](#performance)
+- [Development](#development)
+  - [Running Tests](#running-tests)
+  - [Project Conventions](#project-conventions)
+- [Troubleshooting](#troubleshooting)
+
+---
+
+## Features
+
+- **Fully local** — all embeddings, vector storage, and search run on-device
+- **Hierarchical chunking** — sections are split following the document's heading structure, preserving semantic context
+- **Hybrid search** — combines dense vector search (cosine ANN) and sparse full-text search (BM25/Tantivy) fused via Reciprocal Rank Fusion
+- **Incremental indexing** — SHA-256 fingerprinting skips unchanged files; filesystem watcher triggers re-indexing automatically
+- **Multiple interfaces** — CLI, REST API (Hono), and MCP server for AI agents
+- **RAG output** — context builder assembles ranked chunks into token-budgeted context windows with source attribution
+- **Optional reranking** — cross-encoder reranking (ONNX) for higher-precision results
+- **Two embedding backends** — Transformers.js ONNX (default, bundled) or Ollama HTTP
+- **Type-safe configuration** — Zod-validated YAML config with environment variable overrides
+
+---
+
+## Architecture Overview
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                        Interfaces                               │
+│   CLI (Commander)   REST API (Hono)   MCP Server (stdio/SSE)    │
+└────────────┬──────────────┬──────────────────┬─────────────────┘
+             │              │                  │
+             ▼              ▼                  ▼
+┌─────────────────────────────────────────────────────────────────┐
+│                       Search Layer                              │
+│         Query preprocessing → Hybrid RRF → Reranker            │
+└────────────────────────────┬────────────────────────────────────┘
+                             │
+             ┌───────────────┼───────────────┐
+             ▼               ▼               ▼
+        Vector ANN        BM25 FTS      RAG Context
+        (LanceDB)        (Tantivy)       Builder
+             │               │
+             └───────┬───────┘
+                     ▼
+┌─────────────────────────────────────────────────────────────────┐
+│                       Storage Layer                             │
+│             LanceDB  (chunks + files tables)                    │
+└────────────────────────────┬────────────────────────────────────┘
+                             │
+┌─────────────────────────────────────────────────────────────────┐
+│                      Ingestion Pipeline                         │
+│   Scanner → Parser (remark AST) → Chunker → Embedder → Writer  │
+└──────────────┬───────────────────────────────────────┬──────────┘
+               │                                       │
+          File System                          Transformers.js
+         (chokidar watch)                       ONNX / Ollama
+```
+
+---
+
+## Technology Stack
+
+| Component | Library | Rationale |
+|-----------|---------|-----------|
+| Language | TypeScript (Node.js ≥22) | Type safety; ESM native; no GIL |
+| Vector DB | `@lancedb/lancedb` | Embedded hybrid vector + BM25; no separate process |
+| Embeddings | `@huggingface/transformers` v3 | ONNX runtime, 2–3× faster than PyTorch on CPU |
+| MD Parsing | `remark` / `remark-parse` | Full mdast AST with line positions |
+| REST Server | `hono` + `@hono/node-server` | ~3× faster than Express; excellent TypeScript DX |
+| MCP Server | `@modelcontextprotocol/sdk` | Official Anthropic reference SDK |
+| File Watch | `chokidar` v3 | Native FSEvents on macOS; debounce built-in |
+| CLI | `commander` | Lightweight, typed |
+| Config | `zod` + `js-yaml` | Runtime-validated config |
+| Concurrency | `p-limit` | Bounded parallelism for the indexing pipeline |
+
+---
+
+## Project Structure
+
+```
+sqmd/
+├── src/
+│   ├── index.ts                    # CLI entrypoint
+│   ├── config/
+│   │   ├── schema.ts               # Zod config schemas + TypeScript types
+│   │   └── loader.ts               # YAML loading, ~ expansion, env overrides
+│   ├── ingestion/
+│   │   ├── scanner.ts              # Recursive async file discovery
+│   │   ├── parser.ts               # remark AST → Section[] with line numbers
+│   │   ├── chunker.ts              # Hierarchical token-aware chunking
+│   │   ├── fingerprint.ts          # SHA-256 content + path hashing
+│   │   └── pipeline.ts             # Full index orchestration (scan→chunk→embed→store)
+│   ├── embeddings/
+│   │   ├── types.ts                # Embedder interface
+│   │   ├── transformers.ts         # Transformers.js ONNX backend
+│   │   └── ollama.ts               # Ollama HTTP backend
+│   ├── store/
+│   │   ├── schema.ts               # Apache Arrow schemas + TypeScript record types
+│   │   ├── db.ts                   # LanceDB connection management
+│   │   ├── writer.ts               # Upsert / delete operations
+│   │   └── reader.ts               # Vector search, FTS search, file/chunk queries
+│   ├── search/
+│   │   ├── query.ts                # Query preprocessing and prefix injection
+│   │   ├── hybrid.ts               # RRF fusion (vector + BM25)
+│   │   └── reranker.ts             # Optional cross-encoder reranking
+│   ├── watcher/
+│   │   ├── handler.ts              # chokidar event handler + debounce
+│   │   └── daemon.ts               # Long-running watcher lifecycle
+│   ├── api/
+│   │   ├── app.ts                  # Hono application factory
+│   │   ├── middleware.ts           # Auth, CORS, request logging
+│   │   ├── models.ts               # Zod request/response schemas
+│   │   └── routes/
+│   │       ├── health.ts           # GET /api/v1/health
+│   │       ├── search.ts           # POST|GET /api/v1/search
+│   │       ├── documents.ts        # GET /api/v1/documents[/:id]
+│   │       └── index.ts            # POST /api/v1/index/trigger, GET /api/v1/index/status
+│   ├── mcp/
+│   │   ├── tools.ts                # MCP tool + resource implementations
+│   │   └── server.ts               # MCP server (stdio + SSE transports)
+│   └── rag/
+│       ├── context-builder.ts      # Token-budgeted context window assembly
+│       └── prompt-templates.ts     # System prompts for RAG use
+├── tests/
+│   ├── unit/
+│   │   ├── chunker.test.ts
+│   │   ├── hybrid.test.ts
+│   │   └── config.test.ts
+│   └── integration/
+│       ├── pipeline.test.ts
+│       └── api.test.ts
+├── config.yaml                     # Default configuration (copy to ~/.sqmd/)
+├── package.json
+└── tsconfig.json
+```
+
+---
+
+## Getting Started
+
+### Prerequisites
+
+- **Node.js 22+** — required for native ESM and modern `node:` builtins
+- **pnpm** (recommended) or npm
+
+```bash
+node --version   # must be ≥ 22.0.0
+```
+
+### Installation
+
+**From source:**
+
+```bash
+git clone <repo>
+cd sqmd
+npm install       # or: pnpm install
+npm run build     # compiles TypeScript → dist/
+```
+
+**Global install (after build):**
+
+```bash
+npm install -g .
+sqmd --version
+```
+
+### Initial Configuration
+
+Write the default configuration file:
+
+```bash
+node dist/index.js config --init ~/.sqmd/config.yaml
+```
+
+Then edit `~/.sqmd/config.yaml` to set the directories you want to index:
+
+```yaml
+paths:
+  watch_dirs:
+    - "~/notes"
+    - "~/work/docs"
+  db_path: "~/.sqmd/lancedb"
+```
+
+The tool resolves config in this order:
+1. Path from `--config` flag
+2. `$SQMD_CONFIG` environment variable
+3. `~/.sqmd/config.yaml`
+4. `./config.yaml` (project-local)
+5. Built-in defaults
+
+### First Index
+
+```bash
+node dist/index.js index
+```
+
+On first run, the embedding model (`nomic-ai/nomic-embed-text-v1.5`, ~270 MB) is downloaded and cached to `~/.sqmd/models`. Subsequent runs use the cached model.
+
+---
+
+## CLI Reference
+
+All commands accept `--config <path>` to specify a non-default config file.
+
+### `index`
+
+Scan and index Markdown files.
+
+```
+sqmd index [options]
+
+Options:
+  --path <path>     Directory or single file to index (default: watch_dirs from config)
+  --force           Re-index all files, even if content is unchanged
+  --watch           Keep running and re-index files as they change
+  --config <path>   Config file path
+```
+
+**Examples:**
+
+```bash
+# Index default watch_dirs
+node dist/index.js index
+
+# Index a specific directory
+node dist/index.js index --path ~/work/docs
+
+# Force full re-index (ignores change detection)
+node dist/index.js index --force
+
+# Index then keep watching
+node dist/index.js index --watch
+```
+
+Progress is printed per-file. A summary reports indexed, skipped (unchanged), and errored files.
+
+---
+
+### `search`
+
+Query the index from the terminal.
+
+```
+sqmd search <query> [options]
+
+Arguments:
+  query             Natural language search query (quote multi-word queries)
+
+Options:
+  --top-k <n>       Number of results to return (default: 10)
+  --mode <mode>     hybrid | vector | fts  (default: hybrid)
+  --filter <path>   Restrict results to files whose path contains this substring
+  --config <path>   Config file path
+```
+
+**Examples:**
+
+```bash
+# Semantic search
+node dist/index.js search "how to configure authentication"
+
+# Full-text only
+node dist/index.js search "OAuth token refresh" --mode fts
+
+# Top 5 results scoped to a directory
+node dist/index.js search "deployment strategy" --top-k 5 --filter /work/
+```
+
+Output includes file path, heading breadcrumb, score, line range, and a 200-character snippet.
+
+---
+
+### `serve`
+
+Start the HTTP REST API server.
+
+```
+sqmd serve [options]
+
+Options:
+  --host <host>     Bind address (default: 127.0.0.1)
+  --port <port>     Port (default: 7832)
+  --config <path>   Config file path
+```
+
+```bash
+node dist/index.js serve
+# → Listening on http://127.0.0.1:7832
+```
+
+If `watcher.enabled` is `true` in config, the file watcher starts automatically alongside the API server.
+
+---
+
+### `mcp`
+
+Start the Model Context Protocol server.
+
+```
+sqmd mcp [options]
+
+Options:
+  --transport <transport>   stdio | sse  (default: stdio)
+  --port <port>             Port for SSE transport (default: 7833)
+  --config <path>           Config file path
+```
+
+```bash
+# For Claude Desktop / Claude Code (stdio)
+node dist/index.js mcp
+
+# For HTTP-based agents (SSE)
+node dist/index.js mcp --transport sse --port 7833
+```
+
+---
+
+### `status`
+
+Display index statistics.
+
+```bash
+node dist/index.js status
+```
+
+Output:
+
+```
+sqmd Status
+────────────────────────────────────────
+DB path:       ~/.sqmd/lancedb
+Files indexed: 142
+Chunks stored: 3847
+Last indexed:  3/17/2026, 09:14:32 AM
+Watch dirs:    ~/notes
+Embedder:      transformers / nomic-ai/nomic-embed-text-v1.5
+```
+
+---
+
+### `config`
+
+Manage configuration.
+
+```bash
+# Write default config to a path
+node dist/index.js config --init ~/.sqmd/config.yaml
+```
+
+---
+
+## REST API
+
+Base URL: `http://localhost:7832/api/v1`
+
+All responses are JSON. Errors use `{ "error": "...", "message": "..." }`.
+
+### Search Endpoints
+
+#### `POST /api/v1/search`
+
+```json
+// Request body
+{
+  "query": "how to set up two-factor authentication",
+  "top_k": 10,
+  "mode": "hybrid",
+  "filter_path": "/notes/security",
+  "include_context": false,
+  "rerank": false
+}
+```
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `query` | string | **required** | Natural language query |
+| `top_k` | number | 10 | Number of results |
+| `mode` | `"hybrid"` \| `"vector"` \| `"fts"` | `"hybrid"` | Search algorithm |
+| `filter_path` | string | — | Path substring filter |
+| `include_context` | boolean | false | Include breadcrumb-prefixed `text` field |
+| `rerank` | boolean | config default | Apply cross-encoder reranking |
+
+```json
+// Response
+{
+  "results": [
+    {
+      "chunk_id": "abc123:2:0",
+      "file_id": "sha256-of-path",
+      "file_path": "/notes/security/2fa.md",
+      "heading_path": "Setup > Two-Factor Authentication",
+      "heading_text": "Two-Factor Authentication",
+      "heading_level": 2,
+      "section_index": 2,
+      "chunk_index": 0,
+      "text_raw": "Enable 2FA by navigating to Settings...",
+      "token_count": 87,
+      "score": 0.0312,
+      "line_start": 45,
+      "line_end": 72
+    }
+  ],
+  "query": "how to set up two-factor authentication",
+  "total": 10,
+  "duration_ms": 43
+}
+```
+
+#### `GET /api/v1/search?q=...`
+
+```bash
+curl "http://localhost:7832/api/v1/search?q=configure+auth&top_k=5&mode=vector"
+```
+
+Accepts the same parameters as POST, via query string. Useful for quick browser/curl queries.
+
+---
+
+### Document Endpoints
+
+#### `GET /api/v1/documents`
+
+Returns a paginated list of indexed files.
+
+```bash
+curl "http://localhost:7832/api/v1/documents?limit=20&offset=0"
+```
+
+```json
+{
+  "documents": [
+    {
+      "file_id": "...",
+      "file_path": "/notes/setup.md",
+      "file_hash": "...",
+      "chunk_count": 12,
+      "indexed_at": 1742215200000,
+      "status": "indexed"
+    }
+  ],
+  "total": 142,
+  "limit": 20,
+  "offset": 0
+}
+```
+
+#### `GET /api/v1/documents/:fileId`
+
+Returns metadata and all stored chunks for a specific file.
+
+```bash
+curl "http://localhost:7832/api/v1/documents/<file_id>"
+```
+
+#### `GET /api/v1/documents/:fileId/raw`
+
+Returns the raw Markdown content of the file (read from disk).
+
+---
+
+### Index Management Endpoints
+
+#### `POST /api/v1/index/trigger`
+
+Trigger a re-index operation asynchronously.
+
+```json
+// Request
+{
+  "paths": ["/notes/work"],   // optional; defaults to watch_dirs
+  "force": false              // optional
+}
+```
+
+```json
+// Response 202
+{
+  "job_id": "job-1710000000000",
+  "status": "queued"
+}
+```
+
+#### `GET /api/v1/index/status`
+
+Returns current index statistics and watcher state.
+
+```json
+{
+  "fileCount": 142,
+  "chunkCount": 3847,
+  "watcherRunning": true,
+  "dbPath": "~/.sqmd/lancedb"
+}
+```
+
+#### `GET /api/v1/index/jobs/:jobId`
+
+Returns the progress of a triggered index job.
+
+```json
+{
+  "job_id": "job-1710000000000",
+  "status": "completed",
+  "indexed": 5,
+  "skipped": 137,
+  "errors": 0,
+  "started_at": 1710000000000,
+  "completed_at": 1710000003500
+}
+```
+
+---
+
+### Health & Metrics Endpoints
+
+#### `GET /api/v1/health`
+
+```json
+{
+  "status": "ok",
+  "db": "connected",
+  "embedder": "transformers / nomic-ai/nomic-embed-text-v1.5",
+  "watcher": "running",
+  "uptime_seconds": 3600
+}
+```
+
+#### `GET /api/v1/metrics`
+
+Search latency percentiles and throughput counters.
+
+---
+
+### Authentication
+
+Set `api.api_key` in config to a non-empty string to enable bearer token auth. All `/api/*` requests must include:
+
+```
+Authorization: Bearer <your-api-key>
+```
+
+If `api_key` is empty (the default), authentication is disabled — suitable for local use.
+
+---
+
+## MCP Server
+
+sqmd exposes a full Model Context Protocol server, allowing AI agents like Claude to search your notes directly from conversations.
+
+### MCP Tools
+
+| Tool | Required Args | Optional Args | Description |
+|------|--------------|---------------|-------------|
+| `search_documents` | `query` | `top_k`, `mode`, `filter_path`, `include_context` | Primary semantic/hybrid search |
+| `get_document` | `file_path` | `section` | Fetch a file's metadata and chunks, optionally filtered to a heading |
+| `list_documents` | — | `path_prefix`, `limit` | Browse the indexed file tree |
+| `trigger_index` | — | `paths`, `force` | Request re-indexing |
+| `get_index_status` | — | — | Index health and stats |
+
+**`search_documents` example (Claude tool call):**
+
+```json
+{
+  "query": "database migration strategy",
+  "top_k": 5,
+  "mode": "hybrid",
+  "include_context": true
+}
+```
+
+When `include_context` is `true`, the response includes a pre-assembled `context` string ready to inject into a prompt.
+
+### MCP Resources
+
+Every indexed file is exposed as a resource with URI scheme `md://<absolute-path>`:
+
+```
+md:///Users/alice/notes/architecture.md
+```
+
+Agents can read raw Markdown content directly via the resource protocol without going through the search tool.
+
+### Claude Desktop Integration
+
+Add to `~/Library/Application Support/Claude/claude_desktop_config.json`:
+
+```json
+{
+  "mcpServers": {
+    "sqmd": {
+      "command": "node",
+      "args": ["/path/to/sqmd/dist/index.js", "mcp"],
+      "env": {
+        "SQMD_CONFIG": "/Users/alice/.sqmd/config.yaml"
+      }
+    }
+  }
+}
+```
+
+Or, if installed globally:
+
+```json
+{
+  "mcpServers": {
+    "sqmd": {
+      "command": "sqmd",
+      "args": ["mcp"],
+      "env": {
+        "SQMD_CONFIG": "~/.sqmd/config.yaml"
+      }
+    }
+  }
+}
+```
+
+### Claude Code Integration
+
+Add to your `.mcp.json` or use `claude mcp add`:
+
+```bash
+claude mcp add sqmd -- node /path/to/dist/index.js mcp
+```
+
+---
+
+## RAG Layer
+
+The `src/rag/` module provides utilities for AI agent memory management.
+
+**`buildContext(results, maxTokens)`** assembles search results into a single context string that fits within a token budget. Each chunk is preceded by attribution metadata:
+
+```
+Source: /notes/architecture/decisions.md
+Section: Architecture > Database > Schema Design
+Lines: 45-72
+
+We chose PostgreSQL because it provides...
+
+---
+
+Source: /notes/architecture/decisions.md
+Section: Architecture > Database > Migrations
+Lines: 100-134
+
+All schema changes are managed via...
+```
+
+The `search_documents` MCP tool returns this context when `include_context: true`. Inject it directly into the system prompt or user message of your agent.
+
+**`ragSystemPrompt()`** returns a baseline system prompt for RAG-style agents instructing the model on how to interpret sourced context.
+
+---
+
+## Configuration Reference
+
+All settings live in `config.yaml` (or the file pointed to by `--config` / `$SQMD_CONFIG`).
+
+### `paths`
+
+| Key | Default | Description |
+|-----|---------|-------------|
+| `watch_dirs` | `["~/notes"]` | Directories to index and watch |
+| `db_path` | `~/.sqmd/lancedb` | LanceDB database location |
+| `model_cache_dir` | `~/.sqmd/models` | Directory for cached embedding models |
+
+### `embeddings`
+
+| Key | Default | Description |
+|-----|---------|-------------|
+| `backend` | `"transformers"` | `"transformers"` (ONNX) or `"ollama"` |
+| `model` | `"nomic-ai/nomic-embed-text-v1.5"` | HuggingFace model ID or Ollama model name |
+| `batch_size` | `64` | Texts per embedding batch |
+| `ollama_base_url` | `"http://localhost:11434"` | Ollama server URL (used only when backend is `"ollama"`) |
+
+### `chunking`
+
+| Key | Default | Description |
+|-----|---------|-------------|
+| `max_tokens` | `512` | Maximum tokens per chunk before splitting |
+| `min_chars` | `50` | Minimum characters; shorter chunks are discarded |
+| `include_breadcrumb` | `true` | Prepend `"Section: H1 > H2 > H3\n\n"` to chunk text for richer embeddings |
+| `overlap_tokens` | `64` | Carry-over tokens between adjacent sub-chunks when a section is split |
+
+### `search`
+
+| Key | Default | Description |
+|-----|---------|-------------|
+| `default_top_k` | `10` | Default number of results |
+| `rrf_k` | `60` | RRF constant (`k` in `1/(k + rank)`) — higher values reduce outlier impact |
+| `rerank` | `false` | Enable cross-encoder reranking globally |
+| `rerank_model` | `"cross-encoder/ms-marco-MiniLM-L-6-v2"` | ONNX cross-encoder model |
+| `rerank_top_n` | `20` | Fetch this many candidates before reranking to `top_k` |
+
+### `watcher`
+
+| Key | Default | Description |
+|-----|---------|-------------|
+| `enabled` | `true` | Auto-start file watcher when `serve` runs |
+| `debounce_ms` | `3000` | Wait this long after the last change before re-indexing |
+| `extensions` | `[".md", ".mdx"]` | File extensions to watch |
+| `ignore_patterns` | `["**/.git/**", "**/node_modules/**"]` | Glob patterns to ignore |
+
+### `api`
+
+| Key | Default | Description |
+|-----|---------|-------------|
+| `host` | `"127.0.0.1"` | Bind address |
+| `port` | `7832` | HTTP port |
+| `api_key` | `""` | API key for bearer auth; empty disables auth |
+
+### `mcp`
+
+| Key | Default | Description |
+|-----|---------|-------------|
+| `transport` | `"stdio"` | `"stdio"` or `"sse"` |
+| `sse_port` | `7833` | Port for SSE transport |
+
+### Environment Variable Overrides
+
+| Variable | Config Key |
+|----------|------------|
+| `SQMD_CONFIG` | Config file path |
+| `SQMD_DB_PATH` | `paths.db_path` |
+| `SQMD_API_PORT` | `api.port` |
+
+---
+
+## Architecture Deep Dive
+
+### Chunking Algorithm
+
+The chunker (`src/ingestion/chunker.ts`) implements a hierarchical, token-aware strategy inspired by PageIndex's TOC-based approach:
+
+1. **Parse** — `remark-parse` converts Markdown to an mdast AST with precise line number tracking.
+
+2. **Build section tree** — The AST walker maintains a heading stack. Every content block (paragraphs, lists, code blocks) is assigned to its nearest ancestor heading.
+
+3. **Inject breadcrumb** — When `include_breadcrumb` is enabled, each chunk's `text` field is prefixed with `"Section: H1 > H2 > H3\n\n"`. This prefix is embedded alongside the content, giving the vector model full hierarchical context. The `text_raw` field always contains the unprefixed content for display.
+
+4. **Token-aware splitting** — Sections exceeding `max_tokens` (default 512) are split at paragraph boundaries. The last paragraph of each chunk is carried over into the next when it fits within `overlap_tokens` (default 64), maintaining cross-chunk coherence.
+
+5. **Stub filtering** — Chunks with `text_raw.length < min_chars` (default 50) are discarded.
+
+6. **Preamble handling** — Content before the first heading becomes `heading_level = 0` with the filename stem as the breadcrumb.
+
+Token estimation uses `Math.ceil(words * 1.3)` — a fast approximation that overestimates slightly to avoid over-long chunks.
+
+---
+
+### Embedding Pipeline
+
+The pipeline (`src/ingestion/pipeline.ts`) orchestrates indexing with bounded parallelism:
+
+```
+scanDirectory()
+    │
+    ├── hashFile() → compare with stored hash
+    │   └── skip if unchanged (unless --force)
+    │
+    ├── parseMarkdown() → ParsedDocument
+    ├── chunkDocument() → ChunkRecord[] (vectors empty)
+    │
+    └── [collected into batches of batch_size * 4]
+            │
+            ├── embedder.embed(texts) → number[][]
+            └── upsertChunks() + upsertFile() → LanceDB
+```
+
+Files are processed with `p-limit(4)` concurrency. Embedding batches are flushed when the pending buffer exceeds `batch_size * 4` (default 256 chunks), balancing memory usage and throughput.
+
+After the first bulk index, `createIndexes()` builds:
+- **IVF-PQ vector index** — `num_partitions: 256`, `num_sub_vectors: 96` (cosine metric)
+- **Tantivy FTS index** — on the `text` field
+
+---
+
+### Hybrid Search & RRF
+
+`src/search/hybrid.ts` fuses vector and full-text results using **Reciprocal Rank Fusion**:
+
+```
+query
+  │
+  ├── prepareQueryForEmbedding()  →  "search_query: <query>"  (nomic prefix)
+  │
+  ├── vectorSearch(vector, k*3)   →  ranked list A
+  └── ftsSearch(query, k*3)       →  ranked list B
+                │
+                ▼
+          RRF score(d) = Σ 1 / (60 + rank_i)
+                │
+                ▼
+          top-k by RRF score  →  SearchResult[]
+```
+
+The RRF constant `k=60` (configurable via `search.rrf_k`) controls how steeply rank differences penalise lower-ranked results. Duplicate chunk IDs across lists are merged, summing their RRF scores.
+
+**Search modes:**
+- `hybrid` — RRF fusion of both lists (recommended)
+- `vector` — pure cosine ANN search only
+- `fts` — pure BM25 full-text search only
+
+**Optional reranking:** When enabled, the initial `top_k` result set is expanded to `rerank_top_n` (default 20) and scored by a cross-encoder (`cross-encoder/ms-marco-MiniLM-L-6-v2` ONNX), which jointly processes query + passage for higher-precision ranking.
+
+---
+
+### Incremental Indexing
+
+Change detection uses two layers:
+
+1. **Content hash** (`src/ingestion/fingerprint.ts`) — SHA-256 of file contents stored in the `files` table. On re-scan, the current hash is compared against the stored one; identical hashes skip the file entirely.
+
+2. **File watcher** (`src/watcher/`) — chokidar monitors `watch_dirs` for `add`, `change`, and `unlink` events. Events are debounced (default 3 s) to coalesce rapid saves. On `unlink`, the file's chunks are deleted from both tables.
+
+---
+
+### LanceDB Schema
+
+Two tables are maintained:
+
+**`chunks`** — one row per chunk (core search table):
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `chunk_id` | Utf8 | `"{file_hash}:{section_idx}:{chunk_idx}"` |
+| `file_id` | Utf8 | SHA-256 of the absolute file path |
+| `file_path` | Utf8 | Absolute path |
+| `file_hash` | Utf8 | Content hash (change detection) |
+| `file_mtime` | Float64 | Epoch timestamp |
+| `heading_path` | Utf8 | `"H1 > H2 > H3"` |
+| `heading_level` | Int8 | 0 = preamble, 1–6 = heading depth |
+| `heading_text` | Utf8 | Verbatim heading text |
+| `section_index` | Int32 | Index of section within the file |
+| `chunk_index` | Int32 | Index of chunk within the section |
+| `text` | Utf8 | Breadcrumb-prefixed text (embedded) |
+| `text_raw` | Utf8 | Display text (no breadcrumb) |
+| `token_count` | Int32 | Approximate token count |
+| `parent_headings` | List\<Utf8\> | Ancestor heading texts |
+| `depth` | Int8 | Heading depth |
+| `vector` | FixedSizeList(768, Float32) | Embedding vector |
+| `line_start` | Int32 | First line in the source file |
+| `line_end` | Int32 | Last line in the source file |
+
+**`files`** — one row per indexed file:
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `file_id` | Utf8 | SHA-256 of path |
+| `file_path` | Utf8 | Absolute path |
+| `file_hash` | Utf8 | Content hash |
+| `file_mtime` | Float64 | Last modification time |
+| `chunk_count` | Int32 | Number of chunks |
+| `indexed_at` | Float64 | Indexing timestamp |
+| `status` | Utf8 | `"indexed"` \| `"error"` \| `"skipped"` |
+| `error_msg` | Utf8 | Error details if status is `"error"` |
+
+Vector dimension is `768` for `nomic-embed-text-v1.5`. For `bge-m3`, change `VECTOR_DIM` in `src/store/schema.ts` to `1024` before first index.
+
+---
+
+## Embedding Backends
+
+### Transformers.js (Default)
+
+Uses `@huggingface/transformers` v3 with the ONNX runtime. No Python, no separate process. Models are downloaded once and cached locally.
+
+**Model:** `nomic-ai/nomic-embed-text-v1.5` (768-dim, ~270 MB)
+
+The nomic model uses asymmetric prefixes for higher accuracy:
+- Documents are embedded as `"search_document: <text>"`
+- Queries are embedded as `"search_query: <text>"`
+
+To use `bge-m3` (1024-dim, multilingual):
+
+```yaml
+embeddings:
+  model: "BAAI/bge-m3"
+```
+
+Also update `VECTOR_DIM = 1024` in `src/store/schema.ts` and rebuild.
+
+### Ollama
+
+Point sqmd at a running [Ollama](https://ollama.ai) instance:
+
+```yaml
+embeddings:
+  backend: "ollama"
+  model: "nomic-embed-text"
+  ollama_base_url: "http://localhost:11434"
+```
+
+Ollama must be running with the model already pulled (`ollama pull nomic-embed-text`).
+
+---
+
+## Performance
+
+| Operation | Typical Time | Notes |
+|-----------|-------------|-------|
+| Initial index (50k chunks) | 2–4 min | CPU; ONNX SIMD; batch size 64 |
+| Single file re-index | < 1 s | Hash skip + targeted upsert |
+| Search (hybrid, no rerank) | < 100 ms | IVF-PQ ANN + Tantivy BM25 + RRF |
+| Search (with reranking) | 200–500 ms | Cross-encoder inference per candidate |
+| Memory (idle) | ~400 MB | ONNX model ~200 MB + mmap'd LanceDB |
+
+Embedding throughput scales with CPU core count — the ONNX runtime uses SIMD and will use available threads automatically.
+
+---
+
+## Development
+
+### Running Tests
+
+```bash
+npm test              # run all tests once (vitest)
+npm run test:watch    # watch mode
+```
+
+Test coverage:
+- `tests/unit/chunker.test.ts` — hierarchical chunking, breadcrumbs, overlap, stub filtering
+- `tests/unit/hybrid.test.ts` — RRF fusion logic (mocked DB)
+- `tests/unit/config.test.ts` — config loading, validation, env var overrides
+- `tests/integration/pipeline.test.ts` — full pipeline with temp LanceDB instance
+- `tests/integration/api.test.ts` — Hono app endpoints (health, search, documents, index)
+
+### Building
+
+```bash
+npm run build      # tsc → dist/
+npm run dev        # tsx src/index.ts (no build step, for development)
+```
+
+### Project Conventions
+
+- All source imports use `.js` extension for ESM compatibility (TypeScript resolves to `.ts` at compile time)
+- Node built-ins use the `node:` prefix (`node:fs`, `node:path`, `node:crypto`)
+- `src/config/schema.ts` is the single source of truth for all config types — do not duplicate config fields elsewhere
+- Embedder is lazy-loaded on first use to avoid model download cost at startup for non-indexing commands
+- `p-limit` concurrency default is 4 files; adjust `concurrency` in `pipeline.run()` for I/O-bound vs CPU-bound workloads
+
+---
+
+## Troubleshooting
+
+**`Database may not be initialized. Run sqmd index first.`**
+
+The LanceDB database doesn't exist yet. Run `node dist/index.js index` to create it.
+
+---
+
+**`Path not found: ~/notes`**
+
+The tilde in `watch_dirs` is expanded at runtime. Ensure the directory exists. Use an absolute path to be explicit:
+
+```yaml
+paths:
+  watch_dirs:
+    - "/Users/alice/notes"
+```
+
+---
+
+**First index takes a long time**
+
+The embedding model (~270 MB) is being downloaded on first use. Subsequent runs use the cache at `~/.sqmd/models`. Check disk space and network connectivity if the download stalls.
+
+---
+
+**Search returns no results**
+
+1. Run `node dist/index.js status` to verify files were indexed
+2. Check `errors` count — some files may have failed to parse
+3. Try `--mode fts` first to verify full-text search works independently
+4. Ensure you're using the same model for both indexing and search (config `embeddings.model`)
+
+---
+
+**Vector dimension mismatch error**
+
+If you change the embedding model after an initial index, the stored vector dimension will mismatch the new model. Delete the database and re-index:
+
+```bash
+rm -rf ~/.sqmd/lancedb
+node dist/index.js index --force
+```
+
+---
+
+**Ollama connection refused**
+
+Ensure Ollama is running (`ollama serve`) and the model is pulled (`ollama pull nomic-embed-text`). Verify `ollama_base_url` in config.
+
+---
+
+**Port 7832 already in use**
+
+Override with `--port` or in config:
+
+```yaml
+api:
+  port: 8832
+```