@angri450/zvec-grep 0.2.2

package/docs/03-mcp.md

@@ -0,0 +1,185 @@
+# MCP guide
+
+[Documentation](./README.md) · [Agents](./01-agents.md) ·
+[CLI](./02-cli.md) · [MCP](./03-mcp.md) · [Pipeline](./04-pipeline.md) ·
+[Architecture](./05-architecture.md) · [Server](./06-server.md) ·
+[Embedding](./07-embedding.md) · [Roadmap](./08-roadmap.md)
+
+zvec-grep exposes its local search layer over Streamable HTTP MCP. The normal
+endpoint is:
+
+```text
+http://127.0.0.1:7999/mcp
+```
+
+Run `zg --install` to configure a supported agent automatically. Use this page
+when building another MCP client or when you need the exact boundary between
+the default and compatibility toolsets. See
+[Server and execution modes](./06-server.md) for lifecycle, mode selection,
+refresh, authentication, and logs.
+
+## Default agent toolset
+
+The default `agent` toolset intentionally exposes only search:
+
+Agents first decide whether the requested answer should be grounded in the
+current indexed workspace, then choose exact or semantic retrieval. The same
+rules apply to source code and non-code material such as documentation, books,
+research material, meeting notes, knowledge-base exports, manuals,
+configuration, and data.
+
+Workspace relevance requires a request to inspect, search, or ground the answer
+in local material, prior context that established local material as the intended
+source, or a question about whether relevant local material exists. Negative,
+incidental, or comparative workspace mentions do not establish relevance.
+
+| Tool | Use it when | Index required |
+| --- | --- | --- |
+| `zvec_grep_search` | The answer is workspace-grounded and wording or location is unknown, or semantic, fuzzy, relationship, chronology, causality, comparison, or cross-file synthesis is required | Yes |
+
+Agents use native grep or rg when locating an exact word, quotation, name, date,
+key, filename, path, source fragment, or regex is sufficient. For mixed tasks,
+start with `zvec_grep_search`, then use native grep or rg for focused follow-up.
+When semantic discovery is selected because no sufficient exact anchor is
+available and the user asks whether conceptually related material exists
+locally, agents make at most one focused search probe and stop when its results
+are not relevant. The probe does not apply to exact quotations, configuration
+keys, filenames, regexes, or exhaustive occurrence requests. Unrelated
+open-world knowledge, current external facts, and web content that does not
+depend on local evidence use the appropriate external source instead.
+
+Every workspace tool input uses an absolute `root` visible to the daemon.
+
+## `zvec_grep_search`
+
+The indexed tool supports hybrid, lexical, and vector query groups. At least one
+of `query`, `queries`, `fts`, or `vector` is required.
+
+Minimal conceptual search:
+
+```json
+{
+  "root": "/absolute/path/to/workspace",
+  "query": "decision history behind the launch date",
+  "limit": 5
+}
+```
+
+Explicit query routes and scope:
+
+```json
+{
+  "root": "/absolute/path/to/workspace",
+  "query": "authentication flow",
+  "fts": ["AuthService", "ForbiddenError"],
+  "globs": ["src/**", "!src/generated/**"],
+  "fileTypes": ["ts"],
+  "fuse": true,
+  "limit": 10
+}
+```
+
+Important inputs:
+
+| Input | Meaning |
+| --- | --- |
+| `query` | One hybrid natural-language or exact query |
+| `queries` | One or more hybrid query groups |
+| `fts` | Ranked lexical constraints within indexed search, not exhaustive occurrence lookup |
+| `vector` | Semantic-only query groups |
+| `fuse` | Combine every group into one ranked plan |
+| `limit` | Maximum items per group, up to 50 |
+| `globs` / `insensitiveGlobs` | Ordered path rules |
+| `fileTypes` / `excludedFileTypes` | ripgrep file-type filters |
+| `symbolTypes` / `preferSymbol` | Indexed symbol controls |
+| `modifiedAfter` / `modifiedBefore` | File modification-time bounds |
+| `freshness` | `eventual` or `wait_for_fresh` |
+| `autoUpdate` | Allow an eventual search to schedule a background update |
+
+The response is compact text designed for agent context. It begins with index
+state and then groups ranked results by file:
+
+```text
+freshness: fresh
+src/theme/use-theme.ts:12-36
+matched: 16-18
+source:
+15  export function useTheme() {
+16    const [theme, setTheme] = useState("light");
+17    useEffect(() => saveTheme(theme), [theme]);
+```
+
+When `freshness` is `possibly_stale`, the response may also include current
+indexing state. Agents can use sufficient results immediately rather than
+running a status preflight.
+
+Remote models may cause the tool to request explicit Remote Embedding
+authorization. See
+[Embedding models](./07-embedding.md#remote-embedding-and-authorization).
+
+## `zvec_grep_rg`
+
+This tool is retained in the optional `full` MCP toolset and is not registered
+in the default `agent` toolset. The CLI equivalent, `zg --rg`, remains
+available without changing the MCP toolset.
+
+Pass the ripgrep command you would otherwise run. The command is parsed into
+arguments and is never executed by a shell:
+
+```json
+{
+  "root": "/absolute/path/to/workspace",
+  "command": "rg -n -F 'loadTheme' -g '*.ts' src"
+}
+```
+
+The tool is exhaustive by default. Append `| head -N` only when intentionally
+requesting bounded output:
+
+```json
+{
+  "root": "/absolute/path/to/workspace",
+  "command": "rg -n 'TODO|FIXME' src | head -50"
+}
+```
+
+Scope broad searches with command paths, `-g/--glob`, or `-t/--type`. Managed rg
+supports common ripgrep matching, context, type, glob, ignore, encoding, and
+regex-engine options while preserving zvec-grep's compact result format.
+
+## Full compatibility toolset
+
+The CLI owns index lifecycle and diagnostics, so agents normally do not need
+administrative MCP tools. Clients that require them can restart the server with:
+
+```bash
+zg --server off
+zg --server on --mcp-toolset full
+```
+
+The `full` toolset exposes six tools:
+
+| Tool | Purpose |
+| --- | --- |
+| `zvec_grep_search` | Indexed retrieval |
+| `zvec_grep_rg` | No-index exhaustive search |
+| `zvec_grep_index` | Create, update, rebuild, or explicitly drop an index |
+| `zvec_grep_index_drop` | Explicitly delete an index |
+| `zvec_grep_index_status` | Inspect persisted and active index state |
+| `zvec_grep_server_status` | Inspect daemon, queue, runtime, and model-pool state |
+
+`zvec_grep_index` requires an absolute root. Its `wait` input defaults to
+`false`, returning a background job identifier. Poll `zvec_grep_index_status`
+only when completion, progress, failure diagnosis, or explicit monitoring is
+needed. An agent must never silently create, rebuild, or delete a persistent
+index.
+
+Set `ZVEC_GREP_MCP_TOOLSET=full` as an environment fallback. An explicit
+`--mcp-toolset` flag takes precedence.
+
+## Transport security
+
+The MCP endpoint is loopback-only. Optional Bearer authentication protects the
+local Server but remains independent of Embedding provider credentials and
+Remote Embedding authorization. Configuration examples are in
+[Server authentication](./06-server.md#bearer-authentication).

package/docs/04-pipeline.md

@@ -0,0 +1,234 @@
+# Retrieval pipeline
+
+[Documentation](./README.md) · [Agents](./01-agents.md) ·
+[CLI](./02-cli.md) · [MCP](./03-mcp.md) · [Pipeline](./04-pipeline.md) ·
+[Architecture](./05-architecture.md) · [Server](./06-server.md) ·
+[Embedding](./07-embedding.md) · [Roadmap](./08-roadmap.md)
+
+zg provides one search layer across semantic discovery, ranked lexical search,
+and exhaustive ripgrep. The normal flow is:
+
+```text
+workspace → file discovery → local index → query routes → compact results
+```
+
+The index combines lexical and vector retrieval. Managed ripgrep follows the
+same workspace and output conventions but can run without an index.
+
+## 1. Choose the workspace scope
+
+For the common case, run a search from the repository root. If no index exists,
+zg creates one there with a local embedding model and then completes the search:
+
+```bash
+cd your-repository
+zg "where authentication is validated"
+```
+
+Use `--index` when you need to select a model or constrain the workspace before
+the first search:
+
+```bash
+cd your-repository
+zg --index --embedding local/potion-code-16m-v2
+
+# Equivalent with an explicit root
+zg --index /absolute/path/to/your-repository \
+  --embedding local/potion-code-16m-v2
+```
+
+The workspace index is stored under `<root>/.zvec-grep/`. `.git` and
+`.zvec-grep` are always excluded. Common dependency, build, generated, cache,
+and log directories are excluded by default, as are files ignored by the
+repository's ignore rules.
+
+The main workspace files are `manifest.json`, `files.zvec`, and `index.zvec`.
+The manifest stores index metadata and the workspace Embedding runtime settings,
+including an API key when one was explicitly persisted for that workspace.
+
+Scope large repositories early:
+
+```bash
+zg --index \
+  --embedding local/potion-code-16m-v2 \
+  -g "src/**" \
+  -g "docs/**" \
+  -g "!dist/**" \
+  -t ts
+```
+
+Useful discovery controls include:
+
+| Option | Effect |
+| --- | --- |
+| `-g, --glob <glob>` | Add an ordered include or `!` exclude rule |
+| `--iglob <glob>` | Add a case-insensitive glob rule |
+| `-t, --type <type>` | Include a ripgrep file type |
+| `-T, --type-not <type>` | Exclude a ripgrep file type |
+| `--hidden` | Include hidden paths except `.git` and `.zvec-grep` |
+| `--no-ignore` | Stop applying ignore files |
+| `--ignore-file <path>` | Add an ignore file |
+| `--max-depth <n>` | Limit recursive depth |
+| `--max-filesize <size>` | Limit file size, for example `500K` or `2M` |
+| `-L, --follow` | Follow symbolic links safely |
+
+File-type filters narrow the result after glob rules. For example,
+`-g "docs/**" -t ts` selects TypeScript files inside `docs`, not every file in
+that directory.
+
+Without an explicit `--max-filesize`, indexing uses type-aware safety limits:
+1 MiB for code, 256 MiB for text and Markdown, 16 MiB for structured data, and
+10 MiB for images. An explicit value replaces the type-aware defaults for every
+selected file. Files excluded by these limits remain silent during normal
+indexing; use `zg --index --debug` to print skipped-file counts and samples.
+
+### Supported formats and extraction
+
+The scanner assigns each admitted file to one extraction path. Structure-aware
+extractors preserve useful code symbols or Markdown sections. When structure is
+not available, zg falls back to plain-text chunks so the file can still
+participate in indexed search.
+
+| Files | Formats | Extractor | Indexed representation |
+| --- | --- | --- | --- |
+| Structure-aware code | C/C++ (`.c`, `.cc`, `.cpp`, `.cxx`, `.h`, `.hpp`), Go, Java, JavaScript/JSX, TypeScript/TSX, Python, Rust | `CodeExtractor` | Symbols, signatures, breadcrumbs, and surrounding source |
+| Component scripts | `.vue`, `.svelte` | `CodeExtractor` | JavaScript or TypeScript `<script>` blocks; plain-text fallback when no structure is found |
+| Other recognized code | Ruby, PHP, Swift, Kotlin, C#, Scala, shell, SQL, CSS/SCSS/Less, `Dockerfile`, `Makefile` | `CodeExtractor` | Plain-text chunks until a structural grammar is available |
+| Markdown | `.md`, `.mdx` | `MarkdownExtractor` | Heading sections and breadcrumbs; plain-text fallback for documents without headings |
+| Text documents | `.txt`, `.rst`, `.html`, `.htm`, `.xml` | `TextExtractor` | Plain-text chunks |
+| Text data | `.csv`, `.json`, `.jsonc`, `.toml`, `.yaml`, `.yml` | `TextExtractor` | Plain-text chunks |
+| Other non-binary files | Unrecognized extensions that pass binary detection | `TextExtractor` | Plain-text chunks |
+| Raster images | `.gif`, `.jpeg`, `.jpg`, `.png`, `.webp` | `ImageExtractor` | Image content when explicitly included and the selected Embedding model accepts images |
+
+Raster images are excluded by the default discovery rules and must be selected
+explicitly. A text-only Embedding model cannot add image fragments to its
+vector index.
+
+The following binary formats are currently skipped before extraction:
+
+- documents: `.pdf`, `.doc`, `.docx`, `.ppt`, `.pptx`, `.xls`, `.xlsx`;
+- archives: `.zip`, `.tar`, `.gz`, `.bz2`, `.xz`, `.7z`, `.rar`;
+- executables and compiled artifacts: `.exe`, `.dll`, `.dylib`, `.so`, `.a`,
+  `.o`, `.obj`, `.wasm`, `.class`, `.jar`;
+- media and databases: `.mp3`, `.mp4`, `.mov`, `.avi`, `.mkv`, `.db`,
+  `.sqlite`.
+
+Empty files, files above the configured size limit, and files detected as
+binary are also skipped. Skipped files do not become extraction failures and
+are not included in the current `filesScanned` count, so the index summary does
+not list each skipped path or reason.
+
+<a id="indexing"></a>
+
+## 2. Build and maintain the index
+
+A manually created index resolves its model from explicit `--embedding`,
+`ZVEC_GREP_EMBEDDING`, the configured default, then the built-in local default.
+An implicit first-search index always uses a local model. Existing indexes reuse
+their stored model and file-selection settings:
+
+```bash
+# First build
+zg --index --embedding local/potion-code-16m-v2
+
+# Incremental update with the stored schema
+zg --index
+```
+
+Use `zg --status` to see the root, selected model, file counts, failures,
+truncation, and the suggested next action:
+
+```bash
+zg --status
+zg --status --check-ready
+```
+
+Changing the Embedding model or an incompatible endpoint requires an explicit
+rebuild:
+
+```bash
+zg --index --rebuild --embedding local/jina-embeddings-v2-base-code
+```
+
+Use `--reset-paths` when the existing file-selection settings should be
+replaced rather than reused. Deleting an index is explicit and destructive:
+
+```bash
+zg --index --drop --yes
+```
+
+See [Embedding models](./07-embedding.md) before choosing or changing a model.
+
+<a id="querying"></a>
+
+## 3. Query through one search layer
+
+The shortest query uses hybrid ranked retrieval:
+
+```bash
+zg "where theme preferences are restored"
+```
+
+Choose an explicit route only when you need more control:
+
+| Route | Use it for | Coverage |
+| --- | --- | --- |
+| Positional query or `--hybrid` | Intent plus useful lexical anchors | Ranked sample |
+| `--fts` | Exact terms ranked through the index | Ranked sample |
+| `--vector` | Conceptual similarity without lexical ranking | Ranked sample |
+| `--rg` | Exhaustive literal or regex matching | Exhaustive unless explicitly bounded |
+
+Examples:
+
+```bash
+# Ranked lexical search
+zg --fts "AuthService"
+
+# Explicit semantic search
+zg --vector "where credentials are validated"
+
+# Combine and fuse several query groups
+zg \
+  --hybrid "authentication flow" \
+  --fts "ForbiddenError" \
+  --fuse \
+  --limit 10
+
+# No index required
+zg --rg -n -F "AuthService" -g "*.ts" src
+```
+
+Multiple positional queries remain separate groups unless `--fuse` is set.
+Use `-g`, `--iglob`, `-t`, and `-T` on indexed queries to narrow results. Managed
+rg also accepts common ripgrep matching, context, engine, encoding, discovery,
+glob, and type options.
+
+zg owns the managed-rg result format, so output-changing options such as
+`--json`, `--count`, `--files`, `-l`, `-o`, `--replace`, and `--vimgrep` are
+rejected. Use `-A`, `-B`, or `-C` to add context.
+
+## Freshness
+
+Indexed results report `fresh` or `possibly_stale`. Routine reconciliation stays
+`fresh` until there is evidence of index drift. See
+[Server and execution modes](./06-server.md#refresh-behavior) for the interaction
+between `auto`, `server`, `direct`, and `--refresh`.
+
+## Output for agents and people
+
+When stdout is a terminal, indexed CLI output is human-readable and includes a
+full source preview by default. When stdout is redirected, output is compact,
+grouped by query group, and omits previews unless requested:
+
+```bash
+zg "plugin lifecycle" --preview short --limit 5
+```
+
+Use `--compact` to request the pipe-oriented form even in a terminal:
+
+```bash
+zg --compact "plugin lifecycle" --limit 5
+```
+
+Use `--debug` for query diagnostics and `--trace` for per-hit indexed search
+trace information.

package/docs/05-architecture.md

@@ -0,0 +1,86 @@
+# Architecture
+
+[Documentation](./README.md) · [Agents](./01-agents.md) ·
+[CLI](./02-cli.md) · [MCP](./03-mcp.md) · [Pipeline](./04-pipeline.md) ·
+[Architecture](./05-architecture.md) · [Server](./06-server.md) ·
+[Embedding](./07-embedding.md) · [Roadmap](./08-roadmap.md)
+
+zg presents one search layer to people and agents while keeping execution,
+retrieval, and storage local by default.
+
+## System at a glance
+
+```mermaid
+flowchart LR
+  Human["Human or script"] --> CLI["zg CLI"]
+  Agent["Agent"] --> MCP["MCP client"]
+
+  CLI --> Router{"auto / server / direct"}
+  MCP --> Server["Local Server"]
+  Router --> Server
+  Router --> Direct["Direct runtime"]
+
+  Server --> Engine["zvec-grep engine"]
+  Direct --> Engine
+
+  Engine --> Indexed["Indexed search<br/>BM25 + vector + RRF"]
+  Engine --> RG["Managed ripgrep<br/>exact text + regex"]
+  Engine --> Indexing["Indexing<br/>scan + extract + embed"]
+
+  Files[("Workspace files")] --> RG
+  Files --> Indexing
+  Indexing --> Index[("Workspace index<br/>.zvec-grep/")]
+  Index --> Indexed
+
+  Indexed --> Results["Compact results"]
+  RG --> Results
+```
+
+## Entry and execution
+
+People and scripts enter through the CLI. Agents normally enter through the
+local Streamable HTTP MCP endpoint configured by `zg --install`.
+
+The CLI routes indexed operations through `auto`, `server`, or `direct` mode.
+Both Server and Direct modes call the same engine; the difference is process
+lifetime and coordination, not search behavior. MCP requests always arrive
+through the Server. Managed `zg --rg` can run directly without a Server
+or index.
+
+See [Agent integrations](./01-agents.md), [CLI](./02-cli.md),
+[MCP](./03-mcp.md), and [Server and execution modes](./06-server.md) for the
+individual interfaces.
+
+## Retrieval paths
+
+The engine exposes two complementary paths behind the same product boundary:
+
+| Path | Best for | Data source |
+| --- | --- | --- |
+| Indexed retrieval | Intent, related concepts, and ranked keywords | BM25/FTS and vector data in the workspace index |
+| Managed ripgrep | Known text, symbols, paths, and regular expressions | Workspace files scanned directly |
+
+Indexed search can combine lexical and vector candidates and fuse their ranks
+with reciprocal rank fusion (RRF). Managed ripgrep is exhaustive by default and
+does not require an Embedding model. Both paths apply workspace-aware filtering
+and return file-oriented results suitable for terminal reading or agent context.
+
+The [Retrieval pipeline](./04-pipeline.md) covers indexing, freshness, filters,
+and route selection in detail.
+
+## State and trust boundary
+
+The normal repository index lives under `<workspace>/.zvec-grep/`. Global
+configuration and daemon state live under `~/.zvec-grep/`. Workspace scanning,
+managed ripgrep, index storage, and local Embedding models remain on the local
+machine, and the Server listens on loopback only.
+
+Selecting a remote Embedding provider is the one path that can send query text
+or workspace content outside the machine. zg requests explicit once-only or
+workspace authorization before that transfer. MCP Bearer authentication protects
+the local endpoint; it does not authorize remote Embedding.
+
+See [Embedding models](./07-embedding.md#remote-embedding-and-authorization)
+for provider authorization and
+[Server authentication](./06-server.md#bearer-authentication) for the local
+endpoint boundary.