@ssmeo/mcp-local-rag 0.18.14 → 0.19.1

package/README.md

@@ -4,10 +4,14 @@
 
 # MCP Local RAG
 
-[![GitHub stars](https://img.shields.io/github/stars/shinpr/mcp-local-rag?style=social)](https://github.com/shinpr/mcp-local-rag)
-[![npm version](https://img.shields.io/npm/v/mcp-local-rag.svg)](https://www.npmjs.com/package/mcp-local-rag)
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
-[![MCP Registry](https://img.shields.io/badge/MCP-Registry-green.svg)](https://registry.modelcontextprotocol.io/)
+[![GitHub
+stars](https://img.shields.io/github/stars/shinpr/mcp-local-rag?style=social)](https://github.com/shinpr/mcp-local-rag)
+[![npm
+version](https://img.shields.io/npm/v/mcp-local-rag.svg)](https://www.npmjs.com/package/mcp-local-rag)
+[![License:
+MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![MCP
+Registry](https://img.shields.io/badge/MCP-Registry-green.svg)](https://registry.modelcontextprotocol.io/)
 
 <p align="center">
   <strong>English</strong> |
@@ -23,46 +27,46 @@ embedding API.
 
 mcp-local-rag indexes PDF, DOCX, Markdown, and text files on your machine. Search combines
 semantic similarity with keyword matching, so queries can match both intent and exact technical
-terms such as API names, class names, and error codes.
+terms such as API names, class names, and error codes. Results include source passages and,
+where available, headings, line numbers, or page numbers so you can check and cite the original
+document.
 
-## Features
-
-- **Runs locally:** Document parsing, embeddings, storage, and search run on your machine.
-  After the initial model download, text ingestion and search work offline.
-- **Hybrid search:** Semantic retrieval finds related concepts, while keyword matching boosts
-  exact technical terms.
-- **Configurable embeddings:** Choose a Hugging Face embedding model that fits the language and
-  domain of your documents.
-- **Semantic chunking:** Documents are split at topic boundaries instead of fixed character
-  counts. Markdown code blocks stay intact.
-- **MCP and CLI:** Use the same index from an AI coding tool or directly from the terminal.
-
-No API key, Docker, Python, or external database is required.
+No API key, Docker, Python, or external database is required. After the initial model download,
+text ingestion and search work offline.
 
 ## Quick Start
 
 ### Requirements
 
-- Node.js 22 or later
+- Node.js 22.20 or later
 - Internet access on first use to download the npm package and embedding model
 - A directory containing the documents you want to search
 
 Set `BASE_DIR` to that directory. It is also the security boundary for file operations. Replace
 `/absolute/path/to/your/documents` below with the directory's absolute path.
 
-mcp-local-rag uses the standard MCP protocol over a local stdio server, so it works with AI
-coding tools and other MCP hosts that support local MCP servers.
-
 Use one of the examples below, or register `npx -y mcp-local-rag` and set `BASE_DIR` using your
 client's MCP configuration format.
 
-**For Claude Code:** Run this command:
+Set `DB_PATH` and `CACHE_DIR` to absolute paths as well. Relative paths resolve from the
+server's working directory, so starting the server from different projects creates a separate
+index and model cache in each.
+
+<details>
+<summary>Claude Code</summary>
+
+Run this command:
 
 ```bash
 claude mcp add local-rag --scope user --env BASE_DIR=/absolute/path/to/your/documents -- npx -y mcp-local-rag
 ```
 
-**For Codex:** Add to `~/.codex/config.toml`:
+</details>
+
+<details>
+<summary>Codex</summary>
+
+Add to `~/.codex/config.toml`:
 
 ```toml
 [mcp_servers.local-rag]
@@ -73,7 +77,12 @@ args = ["-y", "mcp-local-rag"]
 BASE_DIR = "/absolute/path/to/your/documents"
 ```
 
-**For OpenCode:** Add to `~/.config/opencode/opencode.json` (or `opencode.jsonc`):
+</details>
+
+<details>
+<summary>OpenCode</summary>
+
+Add to `~/.config/opencode/opencode.json` (or `opencode.jsonc`):
 
 ```json
 {
@@ -90,7 +99,12 @@ BASE_DIR = "/absolute/path/to/your/documents"
 }
 ```
 
-**For Cursor:** Add to `~/.cursor/mcp.json`:
+</details>
+
+<details>
+<summary>Cursor</summary>
+
+Add to `~/.cursor/mcp.json`:
 
 ```json
 {
@@ -106,6 +120,8 @@ BASE_DIR = "/absolute/path/to/your/documents"
 }
 ```
 
+</details>
+
 Restart the client, then ask it to build the index:
 
 ```text
@@ -133,31 +149,39 @@ npx mcp-local-rag query "authentication API"
 The CLI uses the current directory as its document root by default. Run both commands from the
 same directory so they use the same default index, or set `BASE_DIR` and `DB_PATH` explicitly.
 
-## Why This Exists
-
-Some document sets cannot be sent to a hosted embedding service because of confidentiality or
-organizational policy. Keeping the index local makes them searchable without adding a per-query
-API cost.
-
-Semantic search alone can miss exact identifiers that matter in technical documentation.
-Keyword reranking keeps those terms visible without giving up natural-language retrieval.
-
 ## Supported Content
 
 | Input | How to ingest |
 |---|---|
 | PDF, DOCX, TXT, Markdown | File ingestion or directory sync |
-| HTML already fetched by the client | `ingest_data`; cleaned with Readability and converted to Markdown |
+| HTML already fetched by the client | `ingest_data` |
 | Plain text or Markdown held in memory | `ingest_data` with a stable source identifier |
 
 HTML fetching is not built into the server. An MCP client can fetch a page and pass its HTML to
 `ingest_data`.
 
 Excel, PowerPoint, standalone images, and source-code file extensions are not supported by file
-ingestion. PDFs can optionally use a local vision model to describe figures, but this is not OCR
-or image search.
+ingestion. PDFs can optionally use a local vision model to describe figures, but this is not
+OCR or image search.
+
+## Using the Index
+
+Sync after adding, editing, or removing documents. For searches and follow-up reading, ask your
+MCP client:
 
-## MCP Tools
+```text
+Find the documented behavior of ERR_CONNECTION_REFUSED.
+Read the surrounding chunks for that result.
+```
+
+You can also ingest a single file or HTML already fetched by the client. To refresh an existing
+entry, ingest the document again using the same path or source identifier. `sync` skips unchanged
+files. MCP file paths must be absolute and inside a configured document root.
+
+PDF section headings may be inaccurate. Check the original page when you need the exact heading.
+
+<details>
+<summary>MCP Tools</summary>
 
 | Tool | Purpose |
 |---|---|
@@ -171,161 +195,14 @@ or image search.
 | `delete_file` | Delete an indexed file or an `ingest_data` item |
 | `status` | Show index and search status |
 
-### Syncing a Document Root
-
-`sync_start` ingests new and changed files, skips byte-identical files, and removes index entries
-for files that no longer exist:
-
-```text
-Sync everything under the configured document roots and wait for completion.
-```
-
-The tool returns a `jobId` immediately. Clients should poll `sync_status` until its state becomes
-`succeeded` or `failed`. A changed PDF keeps the visual profile it was indexed with; `sync_start`
-cannot change it. Set `STORE_IMAGES=true` in the MCP server environment to store supported PDF and
-DOCX images for new or changed files selected by sync; unchanged files remain skipped.
-
-Only one sync job is retained by the server process. A newer job replaces a finished record, and
-restarting the server discards it.
-
-### Ingesting One File
-
-`ingest_file` accepts PDF, DOCX, TXT, and Markdown. MCP file paths must be absolute and must stay
-inside a configured document root:
-
-```text
-Ingest the document at /Users/me/docs/api-spec.pdf.
-```
-
-Re-ingesting the same path replaces its existing chunks.
-
-### Searching and Reading More Context
-
-```text
-What does the API documentation say about authentication?
-Find the documented behavior of ERR_CONNECTION_REFUSED.
-```
-
-Results contain the text, source path, title, chunk index, relevance score, and any images stored
-on that chunk. MCP returns each image as an image content block paired with its result identity;
-CLI `query` includes an `images` array of `{ imageIndex, mimeType, data }` on every result. Pass the
-`chunkIndex` and either `filePath` or `source` from a result to `read_chunk_neighbors` when the
-answer needs more context:
-
-```text
-Read the surrounding chunks for that authentication result.
-```
-
-Both `query_documents` and `list_files` accept an optional absolute `scope` path prefix, or a
-list of prefixes. A prefix matches the exact path and its descendants.
-
-### Ingesting HTML
-
-Use `ingest_data` after the MCP client fetches a page:
-
-```text
-Fetch https://example.com/docs and ingest the HTML.
-```
-
-The server extracts the main article, converts it to Markdown, and stores it under the supplied
-source identifier. Reusing the same source updates the existing content.
-
-Respect the source site's terms and copyright when indexing external content.
-
-### PDF Visual Captions and Stored Images
-
-Visual mode adds a generated caption for figure-heavy PDF pages. It is opt-in and does not load
-a vision model during normal ingestion.
-
-```text
-Ingest /Users/me/docs/research-paper.pdf with visual: true.
-```
-
-```bash
-npx mcp-local-rag ingest ./docs/research-paper.pdf --visual
-```
-
-Image storage is independent of visual captions. Set `STORE_IMAGES=true` for the MCP server, or
-pass `--images` to CLI ingestion and sync:
-
-```bash
-npx mcp-local-rag ingest ./docs/research-paper.pdf --images
-npx mcp-local-rag sync ./docs/ --images
-```
-
-PDF storage uses detected figure/table regions. DOCX storage includes only PNG/JPEG images that
-the existing Mammoth conversion emits as `<img>`; charts, SmartArt, and shapes are not separately
-rendered. Stored images follow their surrounding text into the final semantic chunk and do not
-alter ranking, scores, or result count.
-
-| `visual` / `--visual` | `STORE_IMAGES` / `--images` | PDF behavior |
-|---|---|---|
-| false | false | Text only; no visual captions or returned images. |
-| true | false | Generated captions become searchable text; no images are stored or returned. |
-| true | true | Generated captions become searchable text, and images from matched chunks are returned inline. |
-| false | true | Images are attached to nearby retained PDF text and returned inline for matched chunks; the VLM is not imported, loaded, or run. |
-
-| Profile | Model cache | Use case |
-|---|---:|---|
-| `fast` (default) | about 250 MB | Lightweight visual indexing |
-| `quality` | about 1.7 GB | Figures containing labels, annotations, or other in-image text |
-
-Select the larger model with `visualQuality: "quality"` over MCP or
-`--visual-quality quality` over CLI. Measured CPU inference was about three times as slow as
-`fast`, though results depend on hardware and model updates.
-
-#### Updating Existing `quality` Captions
-
-From 0.18.4 `quality` runs Qwen3.5-2B; earlier versions ran Qwen2.5-VL-3B. Captions already indexed
-keep the wording the old model produced, and `sync` will not redo them, so re-ingest the files you
-want refreshed:
-
-```bash
-npx mcp-local-rag ingest ./docs/research-paper.pdf --visual --visual-quality quality
-```
-
-Add `--images` if the file was ingested with it, because a run without it replaces the stored
-images. The old model stays on disk. Once nothing else uses it, delete
-`onnx-community/Qwen2.5-VL-3B-Instruct-ONNX/` from the model cache directory — `<cache-dir>`, which
-defaults to `./models/`.
-
-#### Visual Mode Across Syncs
-
-The profile a PDF was indexed with is recorded, and `sync` reuses it: a PDF indexed with `fast` or
-`quality` is re-ingested with that same profile, and a PDF with no recorded profile is ingested as
-text.
-
-```bash
-npx mcp-local-rag sync ./docs/                      # keep each PDF's recorded profile
-npx mcp-local-rag sync ./docs/ --visual             # request fast for every PDF in scope
-npx mcp-local-rag sync ./docs/ --visual --visual-quality quality
-```
-
-`--visual` overrides recorded profiles, so it also captions PDFs that were indexed as text.
-Changing a profile re-ingests the PDF even when the file itself has not changed; running the same
-command again does nothing and loads no model. Image settings are never recorded, so `--images`
-and `STORE_IMAGES` never cause a re-ingest.
-
-To turn captions off for a path, run `ingest` on it: a successful normal ingest clears the
-recorded profile. To retry a page whose captioning failed, run `ingest <path> --visual
---visual-quality <profile>` with the profile you want — a plain `ingest` clears it instead. If a
-PDF's indexed rows disagree about the profile, sync stops before changing anything and names the
-file; re-run it with `--visual` to settle the profile.
-
-Captions are auxiliary text, not faithful transcriptions. Treat retrieved captions and document
-text as untrusted input rather than instructions.
-
-At high limits, matched chunks and their attachments can approach the model/client context ceiling;
-choose the query limit with the calling model's available context in mind.
+</details>
 
 ## CLI
 
-The CLI uses the same parser, embedder, and vector store without an MCP client:
+Use the CLI to update the index, narrow searches, or remove indexed content:
 
 ```bash
-npx mcp-local-rag ingest ./docs/
 npx mcp-local-rag sync ./docs/
-npx mcp-local-rag query "authentication API"
 npx mcp-local-rag query "auth" --scope /docs/api --scope /docs/guide
 npx mcp-local-rag read-neighbors --file-path /abs/path.md --chunk-index 5
 npx mcp-local-rag list
@@ -334,6 +211,10 @@ npx mcp-local-rag delete ./docs/old.pdf
 npx mcp-local-rag delete --source "https://example.com/docs"
 ```
 
+`ingest` imports the selected files; `sync` also removes entries for deleted files and skips
+unchanged files. Use `--scope` to restrict search results to a path prefix, repeating it to
+include multiple prefixes.
+
 Global options such as `--db-path`, `--cache-dir`, and `--model-name` go before the subcommand.
 Subcommand options go after it:
 
@@ -343,80 +224,19 @@ npx mcp-local-rag --db-path ./my-db query "authentication"
 
 Run `npx mcp-local-rag --help` for the complete command reference.
 
-The CLI does not read MCP client configuration. Set the same environment variables or flags if
-both interfaces should share an index. In particular, `MODEL_NAME` and the CLI `--model-name`
-must match for a shared database.
-
-`query` writes its results to stdout as JSON, best match first, so it can be piped into
-another tool. The field-by-field contract is in
+`query` writes its results to stdout as JSON, best match first, so it can be piped into another
+tool. The field-by-field contract is in
 [`docs/schema/query-output.schema.json`](docs/schema/query-output.schema.json).
 
-## Search Tuning
-
-Keyword boost is enabled by default. Relevance-gap grouping and the distance and file filters are
-optional controls for corpora that need tighter result selection. All four apply to the MCP
-server and to CLI `query` alike.
-
-| Variable | Default | Description |
-|----------|---------|-------------|
-| `RAG_HYBRID_WEIGHT` | `0.6` | Keyword boost factor (0.0–1.0). 0 disables keyword reranking; 1 applies the maximum boost. |
-| `RAG_GROUPING` | (not set) | `similar` keeps the first relevance group; `related` keeps up to two, using significant vector-distance gaps as boundaries. |
-| `RAG_MAX_DISTANCE` | (not set) | Filter out low-relevance results (e.g., `0.5`). |
-| `RAG_MAX_FILES` | (not set) | Limit results to top N files (e.g., `1` for single best file). |
-| `RAG_RERANK_CMD` | (not set) | MCP server only: external command that reorders results. Your query and the matched text are sent to it. |
-| `RAG_RERANK_TIMEOUT_MS` | `10000` | Time budget per rerank call in milliseconds (100–600000). |
-
-For API specifications and other documents containing many identifiers, a stronger keyword
-weight can improve exact-term ranking:
+If you move a project and want to keep its index, stop the MCP server and other database
+writers, move the files and database together without changing the directory layout, then run:
 
-```json
-"env": {
-  "RAG_HYBRID_WEIGHT": "0.7"
-}
-```
-
-- `0.7`: slightly stronger exact-term reranking than the default
-- `1.0`: maximum keyword boost
-
-### External Reranking (`RAG_RERANK_CMD`)
-
-Name a command here and the server hands it each set of search results to reorder, together with
-your query and the text of the matched chunks. A command that calls a remote service sends all of
-that off this machine.
-
-Give the command and its arguments separated by spaces. It has to be an executable: the server
-runs it without a shell, so an npm-installed `.cmd` shim on Windows will not start.
-
-```json
-"env": {
-  "RAG_RERANK_CMD": "/path/to/reranker",
-  "RAG_RERANK_TIMEOUT_MS": "10000"
-}
+```bash
+npx mcp-local-rag --db-path /new/project/lancedb relocate --from /old/project --to /new/project
 ```
 
-The command receives each result in the form published at
-[`docs/schema/query-output.schema.json`](docs/schema/query-output.schema.json) and has to answer
-in that same form. Within it the command decides everything: what to keep, how to order it, and
-what the text says. Whatever it returns is what you see.
-
-Results keep their original order if the command fails, times out, or answers with something that
-is not that form.
-
-## How It Works
-
-During ingestion:
-
-1. The parser extracts text for the input format.
-2. The semantic chunker finds topic boundaries and preserves Markdown code blocks.
-3. Transformers.js creates embeddings locally.
-4. LanceDB stores the chunks, metadata, vectors, and full-text index.
-
-During search:
-
-1. The query is embedded with the same model.
-2. Vector search retrieves semantically related chunks.
-3. Optional distance and relevance-group filters narrow the candidates when configured.
-4. Full-text matches boost exact query terms.
+Update `BASE_DIR`/`BASE_DIRS`, `DB_PATH`, and any other moved paths in your MCP configuration,
+then restart the client.
 
 ## Agent Skills
 
@@ -431,72 +251,152 @@ npx mcp-local-rag skills install --codex
 Installed skills cover query formulation, result refinement, and HTML ingestion. Ask the
 assistant to use the mcp-local-rag skill explicitly if it does not activate automatically.
 
-## Configuration
+## Advanced Options
+
+Start with the defaults. Open the sections below when you need different document roots, better
+results for your corpus, or searchable PDF figures.
+
+<details>
+<summary>Storage and Document Roots</summary>
 
-The MCP server reads environment variables. The CLI accepts the listed global environment
-variables and flags; image storage on CLI ingestion and sync is enabled only with `--images`.
+The MCP server reads environment variables. The CLI accepts the listed variables and flags.
+Keep the same `DB_PATH` when commands should use the same index.
 
 | Environment Variable | CLI Flag | Default | Description |
 |---------------------|----------|---------|-------------|
 | `BASE_DIR` | `--base-dir` | Current directory | One document root; the CLI flag is repeatable on `ingest`, `list`, and `sync` |
 | `BASE_DIRS` | N/A | (unset) | JSON array of document roots; takes precedence over `BASE_DIR` |
+| `IGNORE_PATHS` | N/A | (unset) | JSON array of glob patterns for files to keep out of the index |
 | `DB_PATH` | `--db-path` | `./lancedb/` | Vector database location |
 | `CACHE_DIR` | `--cache-dir` | `./models/` | Model cache directory |
 | `HF_ENDPOINT` | N/A | `https://huggingface.co` | Hugging Face model download endpoint; use a mirror URL when direct downloads are blocked |
-| `MODEL_NAME` | `--model-name` | `Xenova/all-MiniLM-L6-v2` | Hugging Face embedding model |
 | `MAX_FILE_SIZE` | `--max-file-size` | `104857600` (100MB) | Maximum file size in bytes |
+
+File operations stay within configured roots. For multiple directories, set
+`BASE_DIRS='["/absolute/docs","/absolute/specs"]'` or repeat CLI `--base-dir`. Precedence: CLI
+roots, `BASE_DIRS`, `BASE_DIR`, then the current directory. Only the highest-priority source is
+used; roots from different sources are not merged. Invalid `BASE_DIRS` is an error. Relative
+`DB_PATH` and `CACHE_DIR` are resolved from the working directory.
+
+To keep files out of the index, set `IGNORE_PATHS` to a JSON array of glob patterns, for example
+`IGNORE_PATHS='["plugs/**","**/*.draft.md"]'`. Each pattern is matched against the file's path
+relative to the root that contains it, written with `/` on every OS. `*` and `**` do not match
+names that start with `.`, so write such directories out, as in `".obsidian/**"`. `ingest`,
+`list`, and `sync` skip matching files when they scan a directory, and `sync` removes any that
+are already indexed. Invalid `IGNORE_PATHS` is an error.
+
+</details>
+
+<details>
+<summary>Models and Search Tuning</summary>
+
+Choose an embedding model for your documents’ language and subject. Compare settings using
+questions you actually ask and check which source passages are returned. The model must support
+mean pooling and L2 normalization, which this tool uses to produce embeddings.
+
+| Environment Variable | CLI Flag | Default | Description |
+|---------------------|----------|---------|-------------|
+| `MODEL_NAME` | `--model-name` | `Xenova/all-MiniLM-L6-v2` | Hugging Face embedding model |
 | `CHUNK_MIN_LENGTH` | `--chunk-min-length` | `50` | Minimum length in characters (1–10000) for ordinary chunks; a fragment of content split to fit the model's token limit can be shorter |
-| `STORE_IMAGES` | N/A | `false` | MCP server only: store supported PDF/DOCX images and return them with matched chunks. CLI uses `--images`. |
+| `EMBED_TITLE_PREFIX` | N/A | `false` | Add the document title to each chunk's embedding input |
+| `EMBED_HEADING_PREFIX` | N/A | `false` | Add the heading hierarchy to each chunk's embedding input when it fits |
 | `RAG_DEVICE` | N/A | `cpu` | ONNX Runtime execution device |
 | `RAG_DTYPE` | N/A | `fp32` | Embedding dtype passed to the selected model |
 
-### Document Roots (`BASE_DIR` and `BASE_DIRS`)
+Both prefix options default to `false` and work independently. Try `EMBED_TITLE_PREFIX` when a
+passage needs the document’s overall topic, or `EMBED_HEADING_PREFIX` when it needs its
+section’s topic. Enabling both is not always better. They affect embeddings, not the returned
+text or keyword index; heading context is omitted when it would exceed the input budget.
 
-mcp-local-rag only allows file operations inside configured roots. For multiple roots,
-`BASE_DIRS` must be a JSON array of non-empty paths:
+When changing embedding models, build a fresh index at a new `DB_PATH`. Vectors from different
+models are not comparable, even when their dimensions match. After changing `RAG_DTYPE` or
+either prefix option, re-ingest all indexed documents before searching. `sync` skips unchanged
+files.
 
-```bash
-export BASE_DIRS='["/Users/me/Documents/work","/Users/me/Projects/specs"]'
+The CLI does not read MCP client configuration. When sharing an index, use the same model,
+`RAG_DTYPE`, and prefix settings for ingestion and search. A change to `RAG_DEVICE` alone does
+not require a new index.
+
+### Search Tuning
+
+The first four settings below apply to both MCP and CLI queries. To give exact terms more
+weight, try increasing `RAG_HYBRID_WEIGHT` and compare results on your own questions. External
+reranking is MCP-only.
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `RAG_HYBRID_WEIGHT` | `0.6` | Keyword boost factor (0.0–1.0). 0 disables keyword reranking; 1 applies the maximum boost. |
+| `RAG_GROUPING` | (not set) | `similar` keeps the first relevance group; `related` keeps up to two, using significant vector-distance gaps as boundaries. |
+| `RAG_MAX_DISTANCE` | (not set) | Filter out low-relevance results (e.g., `0.5`). |
+| `RAG_MAX_FILES` | (not set) | Limit results to top N files (e.g., `1` for single best file). |
+| `RAG_RERANK_CMD` | (not set) | MCP only: external command; `{query}` passes the query and `{top}` the requested result count. |
+| `RAG_RERANK_TIMEOUT_MS` | `10000` | Time budget per rerank call in milliseconds (100–600000). |
+
+### External Reranking (`RAG_RERANK_CMD`)
+
+The command reads search results, including matched text, from stdin. If it calls a remote
+service, that text may leave your machine.
+
+Give the executable and its complete argument template. Put `{query}` and `{top}` where the
+command expects the query and result count. Single or double quotes group paths or arguments
+containing spaces, and backslashes stay literal. The server runs the executable without a
+shell, so an npm-installed `.cmd` shim on Windows will not start.
+
+```json
+{
+  "env": {
+    "RAG_RERANK_CMD": "/path/to/reranker --query {query} --top {top}",
+    "RAG_RERANK_TIMEOUT_MS": "10000"
+  }
+}
 ```
 
-Root configuration is resolved in this order:
+The command must read and return results in the format defined by [the query output
+schema](docs/schema/query-output.schema.json). It can remove or reorder results and modify
+their text. The server returns its output.
 
-1. CLI `--base-dir <path>` flags (repeatable on `ingest`, `list`, and `sync`)
-2. `BASE_DIRS`
-3. `BASE_DIR`
-4. Current directory
+Results keep their original order if the command fails, times out, or returns output that does
+not match the schema.
 
-Each source replaces the lower-priority source rather than merging with it. Invalid `BASE_DIRS`
-configuration fails instead of falling back to `BASE_DIR` or the current directory. `status`
-remains available in MCP so the client can report the configuration error.
+</details>
+
+<details>
+<summary>PDF Figures and Stored Images</summary>
+
+By default, ingestion indexes only text. To make PDF figures searchable, enable local caption
+generation with `visual: true` in MCP or `--visual` in the CLI. Captions are generated
+descriptions, not OCR or exact transcriptions.
+
+`fast` (default) downloads about 250 MB on first use. Choose `quality` for labels and text
+within figures; it downloads about 1.7 GB and takes longer to run.
+
+Select the profile with `visualQuality: "quality"` in MCP or `--visual-quality quality` in the
+CLI.
 
 ```bash
-npx mcp-local-rag ingest --base-dir /Users/me/work --base-dir /Users/me/specs /Users/me/work/readme.md
-npx mcp-local-rag list --base-dir /Users/me/work --base-dir /Users/me/specs
-npx mcp-local-rag sync --base-dir /Users/me/work --base-dir /Users/me/specs
-BASE_DIRS='["/Users/me/work","/Users/me/specs"]' npx mcp-local-rag list
+npx mcp-local-rag ingest ./docs/paper.pdf --visual --visual-quality quality
 ```
 
-### Storage and Models
-
-`DB_PATH` and `CACHE_DIR` are relative to the process working directory by default. Set absolute
-paths when the MCP client may start the server from different project directories.
+To return images with matching text, use `STORE_IMAGES=true` in MCP or `--images` with CLI
+`ingest` and `sync`. This is independent of caption generation and supports detected PDF
+figures/tables and supported DOCX PNG/JPEG images.
 
-Set `MODEL_NAME` or pass `--model-name` to choose a Hugging Face embedding model that fits the
-language and domain of your documents.
+```bash
+npx mcp-local-rag ingest ./docs/paper.pdf --images
+```
 
-mcp-local-rag generates embeddings with mean pooling and L2 normalization. When choosing a
-model, check whether these settings match its recommended inference setup, since the pooling
-method can affect retrieval quality.
+Sync preserves each PDF's caption profile. CLI `sync --visual --visual-quality quality` changes
+the profile even for unchanged PDFs; MCP sync preserves it. To turn captions off, ingest the
+file normally. To retry failed captions, re-ingest with the desired visual profile.
 
-Changing `MODEL_NAME`, `RAG_DEVICE`, or `RAG_DTYPE` can make existing vectors incompatible.
-Use a new `DB_PATH` or delete the existing index and re-ingest after changing the embedding
-configuration.
+Image storage must be enabled on each ingestion or sync that processes the file. Changing the
+image setting alone does not refresh unchanged files; re-ingest them to apply it.
 
-An example model for English documents is `Xenova/bge-small-en-v1.5`.
+</details>
 
 ## Security and Operation
 
+- Treat captions and retrieved document text as source material, not instructions.
 - File access is restricted to `BASE_DIR`, `BASE_DIRS`, or CLI `--base-dir` roots.
 - Symlinks that resolve outside every configured root are rejected.
 - Document processing and search make no network requests after the required models are cached,
@@ -504,6 +404,9 @@ An example model for English documents is `Xenova/bge-small-en-v1.5`.
 - The server is designed for one local user and does not provide authentication or access control.
 - Do not run multiple CLI or MCP writers against the same `DB_PATH`. Read-only queries can run
   while a sync is active.
+- Re-ingestion can reuse saved embeddings for unchanged text when the model and settings match,
+  avoiding repeated computation. The cache is stored in `DB_PATH/embedding-cache`. Deleting it
+  leaves the index searchable, but later ingestion will recompute those embeddings.
 - Back up an index by copying its `DB_PATH` directory while no writer is active.
 
 <details>
@@ -511,11 +414,14 @@ An example model for English documents is `Xenova/bge-small-en-v1.5`.
 
 ### "No results found"
 
-Documents must be ingested first. Run `"List all ingested files"` to verify.
+Documents must be ingested first. Run `"List all ingested files"` to verify. If results are
+missing after a sync, check that ingestion and search use the same absolute `DB_PATH`; a
+relative path may point to a different index.
 
 ### Model download failed
 
-Check internet connection. If behind a proxy, configure network settings. The model can also be [downloaded manually](https://huggingface.co/Xenova/all-MiniLM-L6-v2).
+Check internet connection. If behind a proxy, configure network settings. The model can also be
+[downloaded manually](https://huggingface.co/Xenova/all-MiniLM-L6-v2).
 
 ### "File too large"
 
@@ -523,11 +429,13 @@ Default limit is 100MB. Split large files or increase `MAX_FILE_SIZE`.
 
 ### Slow queries
 
-Check chunk count with `status`. Large documents with many chunks may slow queries. Consider splitting very large files.
+Check chunk count with `status`. Large documents with many chunks may slow queries. Consider
+splitting very large files.
 
 ### "Path outside BASE_DIR"
 
-Ensure file paths are within one of the configured roots (`BASE_DIR`, any `BASE_DIRS` entry, or any CLI `--base-dir`). Use absolute paths.
+Ensure file paths are within one of the configured roots (`BASE_DIR`, any `BASE_DIRS` entry, or
+any CLI `--base-dir`). Use absolute paths.
 
 ### "BASE_DIRS must be a JSON array..."
 
@@ -559,4 +467,6 @@ MIT License. Free for personal and commercial use.
 
 ## Acknowledgments
 
-Built with [Model Context Protocol](https://modelcontextprotocol.io/) by Anthropic, [LanceDB](https://lancedb.com/), and [Transformers.js](https://huggingface.co/docs/transformers.js).
+Built with [Model Context Protocol](https://modelcontextprotocol.io/) by Anthropic,
+[LanceDB](https://lancedb.com/), and
+[Transformers.js](https://huggingface.co/docs/transformers.js).